Project

General

Profile

Actions
Copy link

Bug #11618

closed

test_mon_osd failed

Added by David Zafman almost 9 years ago. Updated almost 9 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Loïc Dachary
Category:
-
Target version:
-
% Done:

0%

Source:
other
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

ubuntu@teuthology:/a/dzafman-2015-05-12_08:43:26-rados-wip-10809-11135-10290---basic-multi/887629

I don't see any other information other than teuthology.log in this directory.

2015-05-12T13:46:49.561 INFO:tasks.workunit.client.0.burnupi61.stdout:max_osd = 105 in epoch 166
2015-05-12T13:46:49.561 INFO:tasks.workunit.client.0.burnupi61.stderr: test_mon_osd: 1020: ceph osd create e9b8daf9-576a-4a25-ad81-77023eec872c 4
2015-05-12T13:46:49.564 INFO:tasks.workunit.client.0.burnupi61.stderr: test_mon_osd: 1020: grep EINVAL
2015-05-12T13:46:49.749 INFO:tasks.workunit.client.0.burnupi61.stdout:Error EINVAL: uuid e9b8daf9-576a-4a25-ad81-77023eec872c already in use for different id 104
2015-05-12T13:46:49.749 INFO:tasks.workunit.client.0.burnupi61.stderr: test_mon_osd: 1022: ceph osd create
2015-05-12T13:46:51.355 INFO:tasks.workunit.client.0.burnupi61.stderr: test_mon_osd: 1022: id=5
2015-05-12T13:46:52.191 INFO:tasks.workunit.client.0.burnupi61.stderr: test_mon_osd: 1023: '[' 5 = 4 ']'
2015-05-12T13:46:52.191 INFO:tasks.workunit.client.0.burnupi61.stderr: test_mon_osd: 1: rm fr /tmp/cephtool7822
2015-05-12T13:46:52.192 INFO:tasks.workunit:Stopping ['cephtool'] on client.0...
2015-05-12T13:46:52.192 INFO:teuthology.orchestra.run.burnupi61:Running: 'rm -rf - /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/workunit.client.0'
2015-05-12T13:46:52.199 ERROR:teuthology.parallel:Exception in parallel execution
Traceback (most recent call last):
File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 82, in exit
for result in self:
File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 101, in next
resurrect_traceback(result)
File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 19, in capture_traceback
return func(*args, **kwargs)
File "/var/lib/teuthworker/src/ceph-qa-suite_wip-11343/tasks/workunit.py", line 361, in _run_tests
label="workunit test {workunit}".format(workunit=workunit)
File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/remote.py", line 156, in run
r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 378, in run
r.wait()
File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 114, in wait
label=self.label)
CommandFailedError: Command failed (workunit test cephtool/test.sh) on burnupi61 with status 1: 'mkdir p - /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8f4927eab2f016edeb5719f67d109e9abb3180f7 TESTDIR="/home/ubuntu/cephtest" CEPH_ID="0" PATH=$PATH:/usr/sbin adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/workunit.client.0/cephtool/test.sh'

Related issues 1 (0 open — 1 closed)

Has duplicate Ceph - Bug #11690: cephtool/test.sh failingDuplicateLoïc Dachary05/20/2015

Actions
Actions
Copy link
 #1

Updated by Loïc Dachary almost 9 years ago

  • Status changed from New to 12
  • Priority changed from Normal to Urgent

Can be reproduced every time with

./virtualenv/bin/teuthology-suite --priority 101 --suite rados/singleton/all/cephtool.yaml --suite-branch master --machine-type plana,burnupi,mira --distro ubuntu --email loic@dachary.org --ceph master

http://pulpito.ceph.com/loic-2015-05-15_23:10:22-rados:singleton:all:cephtool.yaml-wip-11359-tier-tests---basic-multi/

Actions
Copy link
 #2

Updated by Loïc Dachary almost 9 years ago

  • Assignee set to Loïc Dachary
Actions
Copy link
 #3

Updated by Loïc Dachary almost 9 years ago

can probably be traced back to a commit with bisecting

Actions
Copy link
 #6

Updated by Loïc Dachary almost 9 years ago

When running locally it succeeds with

 test_mon_osd: 1030: uuid=c2d59bd2-c877-443e-bd64-92df83594f72
 test_mon_osd: 1031: local gap_start=4
  test_mon_osd: 1032: ceph osd create c2d59bd2-c877-443e-bd64-92df83594f72 104
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
 test_mon_osd: 1032: id=104
 test_mon_osd: 1033: '[' 104 = 104 ']'
 test_mon_osd: 1034: max_osd=105
 test_mon_osd: 1035: ceph osd getmaxosd
 test_mon_osd: 1035: grep 'max_osd = 105'
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
max_osd = 105 in epoch 172
 test_mon_osd: 1037: ceph osd create c2d59bd2-c877-443e-bd64-92df83594f72 4
 test_mon_osd: 1037: grep EINVAL
Error EINVAL: uuid c2d59bd2-c877-443e-bd64-92df83594f72 already in use for different id 104
  test_mon_osd: 1039: ceph osd create
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
 test_mon_osd: 1039: id=4
 test_mon_osd: 1040: '[' 4 = 4 ']'
 test_mon_osd: 1041: gap_start=5
   test_mon_osd: 1043: uuidgen
  test_mon_osd: 1043: ceph osd create 78e61bc1-ebcf-47ee-a421-75e303b4fb68
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
 test_mon_osd: 1043: id=5
 test_mon_osd: 1044: '[' 5 = 5 ']'
 test_mon_osd: 1045: gap_start=6

Actions
Copy link
 #7

Updated by Loïc Dachary almost 9 years ago

When a new osd is created it gets the id 3 because there already are 3 osds, 0, 1, 2. It shows that at this point in the test the cluster had 3 as the highest osd number to allocate. What makes it jump to 5 a few lines later ?

Actions
Copy link
 #8

Updated by Loïc Dachary almost 9 years ago 

./virtualenv/bin/teuthology-suite --priority 101 --suite rados/singleton/all/cephtool.yaml --suite-branch wip-11618 --machine-type plana,burnupi,mira --distro ubuntu --email loic@dachary.org --ceph master

just to run test.sh and nothing else, although I'm pretty sure it's entirely unrelated http://pulpito.ceph.com/loic-2015-05-21_14:44:26-rados:singleton:all:cephtool.yaml-master---basic-multi/. The problem repeats all the time, there is no way it's a race with another test.
Actions
Copy link
 #9

Updated by Loïc Dachary almost 9 years ago

CEPH_CLI_TEST_DUP_COMMAND=1 is set by workunit.py but is not set by cephtool-test-mon.sh

Actions
Copy link
 #10

Updated by Loïc Dachary almost 9 years ago

  • Status changed from 12 to In Progress

Can be reproduced with

$ CEPH_CLI_TEST_DUP_COMMAND=1 test/cephtool-test-mon.sh
...
 test_mon_osd: 1031: local gap_start=4
  test_mon_osd: 1032: ceph osd create 3108a967-ace2-4548-9db4-d5fd03cfbc24 104
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
 test_mon_osd: 1032: id=104
 test_mon_osd: 1033: '[' 104 = 104 ']'
 test_mon_osd: 1034: max_osd=105
 test_mon_osd: 1035: ceph osd getmaxosd
 test_mon_osd: 1035: grep 'max_osd = 105'
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
max_osd = 105 in epoch 222
 test_mon_osd: 1037: ceph osd create 3108a967-ace2-4548-9db4-d5fd03cfbc24 4
 test_mon_osd: 1037: grep EINVAL
Error EINVAL: uuid 3108a967-ace2-4548-9db4-d5fd03cfbc24 already in use for different id 104
  test_mon_osd: 1039: ceph osd create
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
 test_mon_osd: 1039: id=5
 test_mon_osd: 1040: '[' 5 = 4 ']'
 test_mon_osd: 1: rm -fr /tmp/cephtool6526

most probably introduced by mon: osd create: add optional 'id' parameter

Actions
Copy link
 #11

Updated by Loïc Dachary almost 9 years ago

  • Status changed from In Progress to Fix Under Review
Actions
Copy link
 #12

Updated by Loïc Dachary almost 9 years ago

  • Status changed from Fix Under Review to Resolved
Actions
Copy link

Also available in: Atom PDF