Ceph » RADOS

and https://github.com/ceph/ceph/pull/54214

Patrick, are you posting the PR as a culprit?

Radoslaw Zarzynski wrote:

Patrick, are you posting the PR as a culprit?

yes, is it not?

Strange, the syntax in the text snippet works in a vstart cluster:

$ ./bin/ceph tell 4.74 deep-scrub
{
    "deep": true,
    "must": true,
    "stamp": "0.000000" 
}

So far I've verified on main and squid that this works. So it's possible it doesn't work on an older version.

Laura Flores wrote:

Strange, the syntax in the text snippet works in a vstart cluster:
[...]

The issue, I believe, is that the QA suite code is using the new tell syntax on an older octopus cluster.

/a/teuthology-2024-03-22_02:08:13-upgrade-squid-distro-default-smithi/7616010

2024-03-22T07:09:37.612 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell 4.15 deep-scrub
...
2024-03-22T07:09:37.802 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches:
2024-03-22T07:09:37.802 INFO:teuthology.orchestra.run.smithi064.stderr:0
2024-03-22T07:09:37.802 INFO:teuthology.orchestra.run.smithi064.stderr:1
2024-03-22T07:09:37.802 INFO:teuthology.orchestra.run.smithi064.stderr:2
2024-03-22T07:09:37.802 INFO:teuthology.orchestra.run.smithi064.stderr:abort
2024-03-22T07:09:37.802 INFO:teuthology.orchestra.run.smithi064.stderr:assert
2024-03-22T07:09:37.802 INFO:teuthology.orchestra.run.smithi064.stderr:bench [<count:int>] [<size:int>] [<object_size:int>] [<object_num:int>]
2024-03-22T07:09:37.803 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs debug_inject_read_zeros
2024-03-22T07:09:37.803 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs files list
2024-03-22T07:09:37.803 INFO:teuthology.orchestra.run.smithi064.stderr:bluefs stats
2024-03-22T07:09:37.803 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator dump block
2024-03-22T07:09:37.803 INFO:teuthology.orchestra.run.smithi064.stderr:Error EINVAL: invalid command
2024-03-22T07:09:37.804 DEBUG:teuthology.orchestra.run:got remote process result: 22
2024-03-22T07:09:37.805 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested
    yield vars
  File "/home/teuthworker/src/git.ceph.com_ceph_768f6cd6e95d0c7a626042719c230df59bdba9d3/qa/tasks/ceph.py", line 1929, in task
    osd_scrub_pgs(ctx, config)
  File "/home/teuthworker/src/git.ceph.com_ceph_768f6cd6e95d0c7a626042719c230df59bdba9d3/qa/tasks/ceph.py", line 1306, in osd_scrub_pgs
    manager.raw_cluster_cmd('tell', pgid, 'deep-scrub')
  File "/home/teuthworker/src/git.ceph.com_ceph_768f6cd6e95d0c7a626042719c230df59bdba9d3/qa/tasks/ceph_manager.py", line 1630, in raw_cluster_cmd
    return self.run_cluster_cmd(**kwargs).stdout.getvalue()
  File "/home/teuthworker/src/git.ceph.com_ceph_768f6cd6e95d0c7a626042719c230df59bdba9d3/qa/tasks/ceph_manager.py", line 1621, in run_cluster_cmd
    return self.controller.run(**kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run
    r.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi064 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell 4.15 deep-scrub'

description: upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_8.stream
clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health
ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade
3-compat_client/quincy}}

/a/teuthology-2024-03-22_02:08:13-upgrade-squid-distro-default-smithi/7615997

Patrick's analysis is correct. The command change did create a problem for all 'upgrade' tests. I am considering the available solutions.