Bug #12006
closed
huge tickets test failure
Added by Sage Weil almost 9 years ago.
Updated almost 7 years ago.
Description
/var/lib/teuthworker/archive/teuthology-2015-06-09_23:06:01-krbd-master-testing-basic-multi/928130
2015-06-12T18:59:16.142 INFO:tasks.workunit.client.0.burnupi36.stderr:imported keyring
2015-06-12T18:59:16.154 INFO:tasks.workunit.client.0.burnupi36.stderr:++ sudo rbd map -n client.bar --keyring /tmp/keyring-bar test
2015-06-12T18:59:21.795 INFO:tasks.workunit.client.0.burnupi36.stderr:+ DEV=/dev/rbd0
2015-06-12T18:59:21.796 INFO:tasks.workunit.client.0.burnupi36.stderr:+ sudo rbd unmap /dev/rbd0
2015-06-12T18:59:21.978 INFO:tasks.workunit.client.0.burnupi36.stderr:+ for user in '{foo,bar,baz}'
2015-06-12T18:59:21.978 INFO:tasks.workunit.client.0.burnupi36.stderr:+ ceph auth import -i /tmp/keyring-baz
2015-06-12T18:59:28.228 INFO:tasks.workunit.client.0.burnupi36.stderr:imported keyring
2015-06-12T18:59:28.241 INFO:tasks.workunit.client.0.burnupi36.stderr:++ sudo rbd map -n client.baz --keyring /tmp/keyring-baz test
2015-06-12T21:59:02.571 INFO:tasks.workunit:Stopping ['rbd/huge-tickets.sh'] on client.0...
- Project changed from rbd to Linux kernel client
- Priority changed from Urgent to Normal
From
CommandFailedError: Command failed (workunit test rbd/huge-tickets.sh) on burnupi36 with status 124: 'mkdir p - /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5a5dc5f0aa79672c3613c08edb2e148196506901 TESTDIR="/home/ubuntu/cephtest" CEPH_ID="0" PATH=$PATH:/usr/sbin adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/workunit.client.0/rbd/huge-tickets.sh'
it looks like huge-tickets workunit got timed out after 3 hours (timestamps confirm it). The command that hung is 'rbd map' and there is no record of a corresponding watch op in osd0 log, which suggests that the kernel got stuck in the libceph initialisation code. That's entirely feasible since that's what huge-tickets hammers on, but unfortunately the is no syslog in burnupi36 archive so no telling what had actually happened - nothing similar recently.
- Assignee set to Ilya Dryomov
- Category set to libceph
- Status changed from New to Closed
I recall several qa/ bugs that left krbd instances behind, leaving test nodes in bad state. teuthology syslog collection issue has been fixed too.
Haven't seen any huge-tickets.sh issues in a long while.
Also available in: Atom
PDF