Ceph

Run: http://pulpito.ceph.com/teuthology-2020-03-09_16:18:11-powercycle-octopus-distro-basic-smithi/
Jobs: '4841663', '4841635', '4841551', '4841677', '4841523', '4841649', '4841775', '4841705', '4841593', '4841621', '4841691', '4841565', '4841509', '4841733', '4841719', '4841803', '4841761', '4841747', '4841579', '4841607', '4841537'
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2020-03-09_16:18:11-powercycle-octopus-distro-basic-smithi/4841509/teuthology.log

2020-03-09T23:11:40.597 INFO:tasks.ceph.osd.0.smithi072.stderr:2020-03-09T23:11:40.595+0000 7f0a4d362700 -1 received  signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0  (PID: 6999) UID: 0
2020-03-09T23:11:40.598 INFO:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops
2020-03-09T23:11:40.698 INFO:tasks.ceph.osd.1.smithi040.stderr:2020-03-09T23:11:40.696+0000 7f9a68c5f700 -1 received  signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1  (PID: 3095) UID: 0
2020-03-09T23:11:40.758 INFO:teuthology.orchestra.run.smithi072.stdout:ERROR: (22) Invalid argument
2020-03-09T23:11:40.758 INFO:teuthology.orchestra.run.smithi072.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards.
2020-03-09T23:11:40.759 INFO:tasks.workunit.client.0.smithi165.stderr:Note: checking out '12973fc04fd10d4af086901e10ffa8e48866b735'.
2020-03-09T23:11:40.760 INFO:tasks.workunit.client.0.smithi165.stderr:
2020-03-09T23:11:40.760 INFO:tasks.workunit.client.0.smithi165.stderr:You are in 'detached HEAD' state. You can look around, make experimental
...
020-03-09T23:11:40.761 INFO:tasks.workunit.client.0.smithi165.stderr:do so (now or later) by using -b with the checkout command again. Example:
2020-03-09T23:11:40.761 INFO:tasks.workunit.client.0.smithi165.stderr:
2020-03-09T23:11:40.761 INFO:tasks.workunit.client.0.smithi165.stderr:  git checkout -b <new-branch-name>
2020-03-09T23:11:40.761 INFO:tasks.workunit.client.0.smithi165.stderr:
2020-03-09T23:11:40.761 INFO:tasks.workunit.client.0.smithi165.stderr:HEAD is now at 12973fc0 btrfs/022: Correct seq number in golden output
2020-03-09T23:11:40.762 INFO:tasks.workunit.client.0.smithi165.stderr:+ make -j4
2020-03-09T23:11:40.767 INFO:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops
2020-03-09T23:11:40.777 INFO:tasks.workunit.client.0.smithi165.stdout:libtoolize -cfi
2020-03-09T23:11:40.777 INFO:tasks.workunit.client.0.smithi165.stderr:make: libtoolize: Command not found
2020-03-09T23:11:40.777 INFO:tasks.workunit.client.0.smithi165.stderr:make: *** [Makefile:66: configure] Error 127
2020-03-09T23:11:40.778 DEBUG:teuthology.orchestra.run:got remote process result: 2
2020-03-09T23:11:40.778 INFO:tasks.workunit:Stopping ['suites/fsx.sh'] on client.0...
2020-03-09T23:11:40.779 INFO:teuthology.orchestra.run.smithi165:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0
2020-03-09T23:11:40.799 INFO:tasks.ceph.osd.1.smithi040.stderr:2020-03-09T23:11:40.797+0000 7f9a68c5f700 -1 received  signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1  (PID: 3095) UID: 0
2020-03-09T23:11:40.882 INFO:teuthology.orchestra.run.smithi072.stdout:ERROR: (22) Invalid argument
2020-03-09T23:11:40.882 INFO:teuthology.orchestra.run.smithi072.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards.
2020-03-09T23:11:40.892 INFO:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight
2020-03-09T23:11:40.900 INFO:tasks.ceph.osd.0.smithi072.stderr:2020-03-09T23:11:40.898+0000 7f0a4d362700 -1 received  signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0  (PID: 6999) UID: 0
2020-03-09T23:11:40.952 ERROR:teuthology.run_tasks:Saw exception from tasks.
2020-03-09T23:11:40.952 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 86, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 65, in run_one_task
    return task(**kwargs)
  File "/home/teuthworker/src/git.ceph.com_ceph_octopus/qa/tasks/workunit.py", line 140, in task
    cleanup=cleanup)
  File "/home/teuthworker/src/git.ceph.com_ceph_octopus/qa/tasks/workunit.py", line 290, in _spawn_on_all_clients
    timeout=timeout)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 87, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 101, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 37, in resurrect_traceback
    reraise(*exc_info)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 24, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_ceph_octopus/qa/tasks/workunit.py", line 415, in _run_tests
    label="workunit test {workunit}".format(workunit=workunit)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 198, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 433, in run
    r.wait()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 158, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 180, in _raise_for_status
    node=self.hostname, label=self.label
CommandFailedError: Command failed (workunit test suites/fsx.sh) on smithi165 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5206bfbd48467d551aaf2e0fce29cfcebfa44e33 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh'