Bug #64972
openqa: "ceph tell 4.3a deep-scrub" command not found
0%
Description
Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/ceph.py", line 1929, in task osd_scrub_pgs(ctx, config) File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/ceph.py", line 1306, in osd_scrub_pgs manager.raw_cluster_cmd('tell', pgid, 'deep-scrub') File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/ceph_manager.py", line 1638, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks/ceph_manager.py", line 1629, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi026 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell 4.3a deep-scrub'
From: /teuthology/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608816/teuthology.log
The command is not valid in octopus/pacific/quincy/reef(?)/squid(?).
Updated by Patrick Donnelly about 1 month ago
- Related to Bug #63967: qa/tasks/ceph.py: "ceph tell <pgid> deep_scrub" fails added
Updated by Patrick Donnelly about 1 month ago
Updated by Radoslaw Zarzynski about 1 month ago
Patrick, are you posting the PR as a culprit?
Updated by Patrick Donnelly about 1 month ago
Radoslaw Zarzynski wrote:
Patrick, are you posting the PR as a culprit?
yes, is it not?
Updated by Laura Flores about 1 month ago
Strange, the syntax in the text snippet works in a vstart cluster:
$ ./bin/ceph tell 4.74 deep-scrub
{
"deep": true,
"must": true,
"stamp": "0.000000"
}
So far I've verified on main and squid that this works. So it's possible it doesn't work on an older version.
Updated by Patrick Donnelly about 1 month ago
Laura Flores wrote:
Strange, the syntax in the text snippet works in a vstart cluster:
[...]
The issue, I believe, is that the QA suite code is using the new tell syntax on an older octopus cluster.
Updated by Laura Flores about 1 month ago
/a/teuthology-2024-03-22_02:08:13-upgrade-squid-distro-default-smithi/7616010
2024-03-22T07:09:37.612 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell 4.15 deep-scrub
...
2024-03-22T07:09:37.802 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches:
2024-03-22T07:09:37.802 INFO:teuthology.orchestra.run.smithi064.stderr:0
2024-03-22T07:09:37.802 INFO:teuthology.orchestra.run.smithi064.stderr:1
2024-03-22T07:09:37.802 INFO:teuthology.orchestra.run.smithi064.stderr:2
2024-03-22T07:09:37.802 INFO:teuthology.orchestra.run.smithi064.stderr:abort
2024-03-22T07:09:37.802 INFO:teuthology.orchestra.run.smithi064.stderr:assert
2024-03-22T07:09:37.802 INFO:teuthology.orchestra.run.smithi064.stderr:bench [<count:int>] [<size:int>] [<object_size:int>] [<object_num:int>]
2024-03-22T07:09:37.803 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs debug_inject_read_zeros
2024-03-22T07:09:37.803 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs files list
2024-03-22T07:09:37.803 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs stats
2024-03-22T07:09:37.803 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator dump block
2024-03-22T07:09:37.803 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command
2024-03-22T07:09:37.804 DEBUG:teuthology.orchestra.run:got remote process result: 22
2024-03-22T07:09:37.805 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested
yield vars
File "/home/teuthworker/src/git.ceph.com_ceph_768f6cd6e95d0c7a626042719c230df59bdba9d3/qa/tasks/ceph.py", line 1929, in task
osd_scrub_pgs(ctx, config)
File "/home/teuthworker/src/git.ceph.com_ceph_768f6cd6e95d0c7a626042719c230df59bdba9d3/qa/tasks/ceph.py", line 1306, in osd_scrub_pgs
manager.raw_cluster_cmd('tell', pgid, 'deep-scrub')
File "/home/teuthworker/src/git.ceph.com_ceph_768f6cd6e95d0c7a626042719c230df59bdba9d3/qa/tasks/ceph_manager.py", line 1630, in raw_cluster_cmd
return self.run_cluster_cmd(**kwargs).stdout.getvalue()
File "/home/teuthworker/src/git.ceph.com_ceph_768f6cd6e95d0c7a626042719c230df59bdba9d3/qa/tasks/ceph_manager.py", line 1621, in run_cluster_cmd
return self.controller.run(**kwargs)
File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run
r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run
r.wait()
File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait
self._raise_for_status()
File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status
raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi064 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell 4.15 deep-scrub'
description: upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_8.stream
clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health
ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade
3-compat_client/quincy}}
Updated by Laura Flores about 1 month ago
/a/teuthology-2024-03-22_02:08:13-upgrade-squid-distro-default-smithi/7615997
Updated by Ronen Friedman about 1 month ago
Patrick's analysis is correct. The command change did create a problem for all 'upgrade' tests. I am considering the available solutions.
Updated by Ronen Friedman about 1 month ago
- Status changed from New to Fix Under Review
- Pull request ID set to 56618
Updated by Ronen Friedman 24 days ago
- Pull request ID changed from 56618 to 56745
note a new PR suggested as a fix
Updated by Ronen Friedman 24 days ago
- Status changed from Fix Under Review to Pending Backport
Updated by Backport Bot 24 days ago
- Copied to Backport #65374: squid: qa: "ceph tell 4.3a deep-scrub" command not found added