Ceph

Run: http://pulpito.ceph.com/teuthology-2020-06-02_11:15:03-powercycle-master-testing-basic-smithi/
Job: 511856
Logs: /a/teuthology-2020-06-02_11:15:03-powercycle-master-testing-basic-smithi/5111856/teuthology.log

2020-06-04T02:30:30.240 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2: {}
2020-06-04T02:30:30.240 INFO:teuthology.orchestra.run.smithi116.stderr:osd.0: osd_enable_op_tracker = 'false'
2020-06-04T02:30:30.241 INFO:teuthology.orchestra.run.smithi116.stderr:Error ENXIO: problem getting command descriptions from osd.1
2020-06-04T02:30:30.241 INFO:teuthology.orchestra.run.smithi116.stderr:osd.1: problem getting command descriptions from osd.1
2020-06-04T02:30:30.241 INFO:teuthology.orchestra.run.smithi116.stderr:osd.2: osd_enable_op_tracker = 'false'
2020-06-04T02:30:30.243 INFO:tasks.ceph.osd.2.smithi003.stderr:2020-06-04T02:30:30.241+0000 7fa068b4a700 -1 received  signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2  (PID: 22070) UID: 0
2020-06-04T02:30:30.251 DEBUG:teuthology.orchestra.run:got remote process result: 6
2020-06-04T02:30:30.251 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring
2020-06-04T02:30:30.344 INFO:tasks.ceph.osd.2.smithi003.stderr:2020-06-04T02:30:30.341+0000 7fa068b4a700 -1 received  signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2  (PID: 22070) UID: 0
2020-06-04T02:30:30.445 INFO:tasks.ceph.osd.0.smithi005.stderr:2020-06-04T02:30:30.443+0000 7f5832534700 -1 received  signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0  (PID: 21975) UID: 0
2020-06-04T02:30:30.471 INFO:teuthology.orchestra.run.smithi005.stdout:ERROR: (22) Invalid argument
2020-06-04T02:30:30.471 INFO:teuthology.orchestra.run.smithi005.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards.
2020-06-04T02:30:30.485 INFO:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight
2020-06-04T02:30:30.547 INFO:tasks.ceph.osd.2.smithi003.stderr:2020-06-04T02:30:30.544+0000 7fa068b4a700 -1 received  signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2  (PID: 22070) UID: 0
2020-06-04T02:30:30.648 INFO:tasks.ceph.osd.2.smithi003.stderr:2020-06-04T02:30:30.645+0000 7fa068b4a700 -1 received  signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2  (PID: 22070) UID: 0
2020-06-04T02:30:30.708 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~303s
2020-06-04T02:30:30.709 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons
2020-06-04T02:30:30.709 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi116.front.sepia.ceph.com...
2020-06-04T02:30:30.710 INFO:teuthology.orchestra.run:Running command with timeout 1800
2020-06-04T02:30:30.710 INFO:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0)
2020-06-04T02:30:30.735 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument
2020-06-04T02:30:30.736 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards.
2020-06-04T02:30:30.746 INFO:tasks.workunit.client.0.smithi116.stdout:Wrote -1 instead of 4096 bytes.
2020-06-04T02:30:30.747 INFO:tasks.workunit.client.0.smithi116.stdout:Probably out of disk space
2020-06-04T02:30:30.747 INFO:tasks.workunit.client.0.smithi116.stderr:write: Transport endpoint is not connected
2020-06-04T02:30:30.748 DEBUG:teuthology.orchestra.run:got remote process result: 1
2020-06-04T02:30:30.749 INFO:tasks.workunit:Stopping ['suites/ffsb.sh'] on client.0...
2020-06-04T02:30:30.749 INFO:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0
2020-06-04T02:30:30.750 INFO:tasks.ceph.osd.0.smithi005.stderr:2020-06-04T02:30:30.746+0000 7f5832534700 -1 received  signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0  (PID: 21975) UID: 0
2020-06-04T02:30:30.751 INFO:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops
2020-06-04T02:30:30.850 INFO:tasks.ceph.osd.2.smithi003.stderr:2020-06-04T02:30:30.847+0000 7fa068b4a700 -1 received  signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2  (PID: 22070) UID: 0
2020-06-04T02:30:30.950 INFO:tasks.ceph.osd.2.smithi003.stderr:2020-06-04T02:30:30.948+0000 7fa068b4a700 -1 received  signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2  (PID: 22070) UID: 0
2020-06-04T02:30:30.988 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument
2020-06-04T02:30:30.989 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards.
2020-06-04T02:30:31.003 INFO:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops
2020-06-04T02:30:31.045 INFO:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd unset nodeep-scrub
2020-06-04T02:30:31.051 INFO:tasks.ceph.osd.2.smithi003.stderr:2020-06-04T02:30:31.048+0000 7fa068b4a700 -1 received  signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2  (PID: 22070) UID: 0
2020-06-04T02:30:31.086 INFO:teuthology.orchestra.run:waiting for 900
2020-06-04T02:30:31.087 INFO:teuthology.orchestra.run:Running command with timeout 300
2020-06-04T02:30:31.087 INFO:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0)
2020-06-04T02:30:31.156 INFO:tasks.ceph.osd.2.smithi003.stderr:2020-06-04T02:30:31.151+0000 7fa068b4a700 -1 received  signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2  (PID: 22070) UID: 0
2020-06-04T02:30:31.227 INFO:tasks.cephfs.mount:Cleaning the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0'
2020-06-04T02:30:31.228 INFO:tasks.cephfs.mount:Removing the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0'
2020-06-04T02:30:31.228 INFO:teuthology.orchestra.run:Running command with timeout 300
2020-06-04T02:30:31.228 INFO:teuthology.orchestra.run.smithi116:> sudo bash -c 'ip link set brx.0 down'
2020-06-04T02:30:31.237 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument
2020-06-04T02:30:31.238 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards.
2020-06-04T02:30:31.253 INFO:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight
2020-06-04T02:30:31.256 INFO:tasks.ceph.osd.2.smithi003.stderr:2020-06-04T02:30:31.253+0000 7fa068b4a700 -1 received  signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2  (PID: 22070) UID: 0
2020-06-04T02:30:31.339 INFO:teuthology.orchestra.run:Running command with timeout 300
2020-06-04T02:30:31.340 INFO:teuthology.orchestra.run.smithi116:> sudo bash -c 'ip link delete brx.0'
2020-06-04T02:30:31.355 INFO:tasks.ceph.osd.2.smithi003.stderr:2020-06-04T02:30:31.353+0000 7fa068b4a700 -1 received  signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2  (PID: 22070) UID: 0
2020-06-04T02:30:31.428 ERROR:teuthology.run_tasks:Saw exception from tasks.
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 91, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 70, in run_one_task
    return task(**kwargs)
  File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/workunit.py", line 140, in task
    cleanup=cleanup)
  File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/workunit.py", line 290, in _spawn_on_all_clients
    timeout=timeout)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 87, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 101, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 37, in resurrect_traceback
    reraise(*exc_info)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/six.py", line 703, in reraise
    raise value
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 24, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks/workunit.py", line 415, in _run_tests
    label="workunit test {workunit}".format(workunit=workunit)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 206, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 475, in run
    r.wait()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 162, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 184, in _raise_for_status
    node=self.hostname, label=self.label
teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/ffsb.sh) on smithi116 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=cca6533da2dbb756769bf3640b19705a1d0ea1fa TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh'
2020-06-04T02:30:31.490 ERROR:teuthology.run_tasks: Sentry event: http://sentry.ceph.com/sepia/teuthology/?q=4fdfbaaaade6448d8cf112d6741b265f