Ceph

This is for 16.2.15

Run: https://pulpito.ceph.com/yuriw-2024-01-31_16:13:03-upgrade:octopus-x-pacific-release-distro-default-smithi/
Jobs: 7540432 7540434 7540435
Logs: https://pulpito.ceph.com/yuriw-2024-01-31_16:13:03-upgrade:octopus-x-pacific-release-distro-default-smithi/7540432/

2024-01-31T17:36:17.722 INFO:teuthology.orchestra.run.smithi159.stderr:Error ENOTSUP: Warning: due to ceph-mgr restart, some PG states may not be up to date
2024-01-31T17:36:17.722 INFO:teuthology.orchestra.run.smithi159.stderr:Module 'orchestrator' is not enabled/loaded (required by command 'orch ps'): use `ceph mgr module enable orchestrator` to enable it

024-01-31T17:36:17.933 INFO:journalctl@ceph.mon.a.smithi159.stdout:Jan 31 17:36:17 smithi159 ceph-157b0ed4-c05e-11ee-95b3-87774f69a715-mon-a[70364]: cluster 2024-01-31T17:36:17.656902+0000 mon.a (mon.0) 62 : cluster [INF] Manager daemon y is now available
2024-01-31T17:36:18.034 DEBUG:teuthology.orchestra.run:got remote process result: 95
2024-01-31T17:36:18.035 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 105, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 83, in run_one_task
    return task(**kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/parallel.py", line 56, in task
    p.spawn(_run_spawned, ctx, confg, taskname)
  File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/parallel.py", line 64, in _run_spawned
    mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 83, in run_one_task
    return task(**kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/sequential.py", line 47, in task
    mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg)
  File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 83, in run_one_task
    return task(**kwargs)
  File "/home/teuthworker/src/github.com_ceph_ceph-c_88fb4c6adb4bae6b1cdf34fca7eae2dacb06cc7b/qa/tasks/cephadm.py", line 1058, in shell
    _shell(ctx, cluster_name, remote,
  File "/home/teuthworker/src/github.com_ceph_ceph-c_88fb4c6adb4bae6b1cdf34fca7eae2dacb06cc7b/qa/tasks/cephadm.py", line 34, in _shell
    return remote.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 455, in run
    r.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi159 with status 95: "sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:octopus shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 157b0ed4-c05e-11ee-95b3-87774f69a715 -e sha1=88fb4c6adb4bae6b1cdf34fca7eae2dacb06cc7b -- bash -c 'ceph orch ps'" 
2024-01-31T17:36:18.263 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=05855bfb45164e1fa69621bf999495c7