Ceph » CephFS

Hit the following error in fs/upgrade/volume_client test,

2020-08-19T16:11:42.075 INFO:tasks.ceph.mds.c.smithi062.stderr:2020-08-19 16:11:42.067 7f656496d700 -1 received  signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mds -f --cluster ceph -i c  (PID: 12002) UID: 0
2020-08-19T16:11:42.075 INFO:tasks.ceph.mds.c.smithi062.stderr:2020-08-19 16:11:42.067 7f656496d700 -1 mds.c *** got signal Terminated ***
2020-08-19T16:11:42.075 INFO:tasks.ceph.mon.a.smithi062.stderr:2020-08-19T16:11:42.067+0000 7fb5260fd700 -1 received  signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mon -f --cluster ceph -i a  (PID: 16677) UID: 0
2020-08-19T16:11:42.075 INFO:tasks.ceph.mon.a.smithi062.stderr:2020-08-19T16:11:42.067+0000 7fb5260fd700 -1 mon.a@0(leader) e2 *** Got Signal Terminated ***
2020-08-19T16:11:42.075 INFO:tasks.ceph.mon.c.smithi062.stderr:2020-08-19T16:11:42.071+0000 7f3987edf700 -1 received  signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mon -f --cluster ceph -i c  (PID: 16736) UID: 0
2020-08-19T16:11:42.076 INFO:tasks.ceph.mon.c.smithi062.stderr:2020-08-19T16:11:42.071+0000 7f3987edf700 -1 mon.c@2(peon) e2 *** Got Signal Terminated ***
2020-08-19T16:11:42.076 INFO:tasks.ceph.mon.b.smithi062.stderr:2020-08-19T16:11:42.071+0000 7f2d77518700 -1 received  signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mon -f --cluster ceph -i b  (PID: 16703) UID: 0
2020-08-19T16:11:42.076 INFO:tasks.ceph.mon.b.smithi062.stderr:2020-08-19T16:11:42.071+0000 7f2d77518700 -1 mon.b@1(peon) e2 *** Got Signal Terminated ***
2020-08-19T16:11:42.197 INFO:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json
2020-08-19T16:11:42.276 INFO:tasks.ceph.mgr.y.smithi062.stderr:daemon-helper: command crashed with signal 15

And later,

2020-08-19T16:13:42.262 ERROR:teuthology.run_tasks: Sentry event: http://sentry.ceph.com/sepia/teuthology/?q=3f7baa7951d94ae5976213c5402c7c16
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 93, in run_tasks
    manager.__enter__()
  File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/home/teuthworker/src/github.com_ceph_ceph-c_ajarr-wip-46163/qa/tasks/ceph.py", line 1551, in restart
    ctx.managers[cluster].wait_for_all_osds_up()
  File "/home/teuthworker/src/github.com_ceph_ceph-c_ajarr-wip-46163/qa/tasks/ceph_manager.py", line 2534, in wait_for_all_osds_up
    while not self.are_all_osds_up():
  File "/home/teuthworker/src/github.com_ceph_ceph-c_ajarr-wip-46163/qa/tasks/ceph_manager.py", line 2524, in are_all_osds_up
    x = self.get_osd_dump()
  File "/home/teuthworker/src/github.com_ceph_ceph-c_ajarr-wip-46163/qa/tasks/ceph_manager.py", line 2297, in get_osd_dump
    return self.get_osd_dump_json()['osds']
  File "/home/teuthworker/src/github.com_ceph_ceph-c_ajarr-wip-46163/qa/tasks/ceph_manager.py", line 2289, in get_osd_dump_json
    out = self.raw_cluster_cmd('osd', 'dump', '--format=json')
  File "/home/teuthworker/src/github.com_ceph_ceph-c_ajarr-wip-46163/qa/tasks/ceph_manager.py", line 1357, in raw_cluster_cmd
    stdout=StringIO(),
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 204, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 446, in run
    r.wait()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 160, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 182, in _raise_for_status
    node=self.hostname, label=self.label
teuthology.exceptions.CommandFailedError: Command failed on smithi062 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd dump --format=json'

Possibly an issue with the setup and teardown of the tests?

Logs are here,
http://pulpito.front.sepia.ceph.com/rraja-2020-08-19_13:10:02-fs-ajarr-wip-46163-distro-basic-smithi/5359764/
http://pulpito.front.sepia.ceph.com/rraja-2020-08-19_13:10:02-fs-ajarr-wip-46163-distro-basic-smithi/5359765/
http://pulpito.front.sepia.ceph.com/rraja-2020-08-19_13:10:02-fs-ajarr-wip-46163-distro-basic-smithi/5359766/