Actions
Bug #49592
closed"test_notify.py" is timing out in upgrade-clients:client-upgrade-nautilus-pacific-pacific
% Done:
0%
Source:
Q/A
Tags:
Backport:
nautilus, octopus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
Run: https://pulpito.ceph.com/teuthology-2021-03-03_17:23:34-upgrade-clients:client-upgrade-nautilus-pacific-pacific-distro-basic-gibba/
Jobs: all
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2021-03-03_17:23:34-upgrade-clients:client-upgrade-nautilus-pacific-pacific-distro-basic-gibba/5930144/teuthology.log
2021-03-03T17:41:59.203 DEBUG:teuthology.orchestra.remote:gibba044:/home/ubuntu/cephtest/workunits.list.client.1 is 4KB
2021-03-03T17:41:59.257 INFO:tasks.workunit:Running workunits matching rbd/notify_slave.sh on client.1...
2021-03-03T17:41:59.257 INFO:tasks.workunit:Running workunit rbd/notify_slave.sh...
2021-03-03T17:41:59.258 DEBUG:teuthology.orchestra.run.gibba044:workunit test rbd/notify_slave.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=octopus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 RBD_FEATURES=61 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/rbd/notify_slave.sh
2021-03-03T17:41:59.275 INFO:tasks.workunit.client.1.gibba044.stderr:+ dirname /home/ubuntu/cephtest/clone.client.1/qa/workunits/rbd/notify_slave.sh
2021-03-03T17:41:59.276 INFO:tasks.workunit.client.1.gibba044.stderr:+ relpath=/home/ubuntu/cephtest/clone.client.1/qa/workunits/rbd/../../../src/test/librbd
2021-03-03T17:41:59.276 INFO:tasks.workunit.client.1.gibba044.stderr:+ python3 /home/ubuntu/cephtest/clone.client.1/qa/workunits/rbd/../../../src/test/librbd/test_notify.py slave
2021-03-03T20:41:57.460 DEBUG:teuthology.orchestra.run:got remote process result: 124
2021-03-03T20:41:57.498 INFO:tasks.workunit:Stopping ['rbd/notify_master.sh'] on client.0...
2021-03-03T20:41:57.499 DEBUG:teuthology.orchestra.run.gibba041:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0
2021-03-03T20:41:57.628 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/run_tasks.py", line 91, in run_tasks
manager = run_one_task(taskname, ctx=ctx, config=config)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/run_tasks.py", line 70, in run_one_task
return task(**kwargs)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/task/parallel.py", line 56, in task
p.spawn(_run_spawned, ctx, confg, taskname)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/parallel.py", line 84, in __exit__
for result in self:
File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/parallel.py", line 98, in __next__
resurrect_traceback(result)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/parallel.py", line 30, in resurrect_traceback
raise exc.exc_info[1]
File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/parallel.py", line 23, in capture_traceback
return func(*args, **kwargs)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/task/parallel.py", line 64, in _run_spawned
mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/run_tasks.py", line 70, in run_one_task
return task(**kwargs)
File "/home/teuthworker/src/git.ceph.com_ceph_e713a2fcf55395640035401674cfa3db7100dbbe/qa/tasks/workunit.py", line 125, in task
timeout=timeout,cleanup=cleanup)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/parallel.py", line 84, in __exit__
for result in self:
File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/parallel.py", line 98, in __next__
resurrect_traceback(result)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/parallel.py", line 30, in resurrect_traceback
raise exc.exc_info[1]
File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/parallel.py", line 23, in capture_traceback
return func(*args, **kwargs)
File "/home/teuthworker/src/git.ceph.com_ceph_e713a2fcf55395640035401674cfa3db7100dbbe/qa/tasks/workunit.py", line 411, in _run_tests
label="workunit test {workunit}".format(workunit=workunit)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/orchestra/remote.py", line 215, in run
r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/orchestra/run.py", line 455, in run
r.wait()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/orchestra/run.py", line 161, in wait
self._raise_for_status()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/orchestra/run.py", line 183, in _raise_for_status
node=self.hostname, label=self.label
teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/notify_master.sh) on gibba041 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=nautilus TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 RBD_FEATURES=61 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/notify_master.sh'
Actions