Project

General

Profile

Bug #13810

"stderr:rbd.InvalidArgument: error writing to test_notify_clone)" in upgrade:client-upgrade-jewel-distro-basic-multi

Added by Yuri Weinstein about 6 years ago. Updated almost 6 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Jason Dillaman
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
upgrade/client-upgrade
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Run: http://pulpito.ceph.com/teuthology-2015-11-12_23:20:02-upgrade:client-upgrade-jewel-distro-basic-multi/
Job: 1148300
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2015-11-12_23:20:02-upgrade:client-upgrade-jewel-distro-basic-multi/1148300/teuthology.log

2015-11-15T14:49:27.417 INFO:tasks.workunit.client.0.plana48.stderr:+ python test_notify.py master
2015-11-15T14:49:32.700 INFO:tasks.workunit.client.0.plana48.stdout:starting master
2015-11-15T14:49:32.701 INFO:tasks.workunit.client.0.plana48.stdout:acquiring exclusive lock
2015-11-15T14:49:32.701 INFO:tasks.workunit.client.0.plana48.stderr:Traceback (most recent call last):
2015-11-15T14:49:32.701 INFO:tasks.workunit.client.0.plana48.stderr:  File "test_notify.py", line 133, in <module>
2015-11-15T14:49:32.702 INFO:tasks.workunit.client.0.plana48.stderr:    main()
2015-11-15T14:49:32.702 INFO:tasks.workunit.client.0.plana48.stderr:  File "test_notify.py", line 127, in main
2015-11-15T14:49:32.702 INFO:tasks.workunit.client.0.plana48.stderr:    master(ioctx)
2015-11-15T14:49:32.703 INFO:tasks.workunit.client.0.plana48.stderr:  File "test_notify.py", line 66, in master
2015-11-15T14:49:32.703 INFO:tasks.workunit.client.0.plana48.stderr:    image.write(data, offset)
2015-11-15T14:49:32.703 INFO:tasks.workunit.client.0.plana48.stderr:  File "/usr/lib/python2.7/dist-packages/rbd.py", line 794, in write
2015-11-15T14:49:32.703 INFO:tasks.workunit.client.0.plana48.stderr:    raise make_ex(ret, "error writing to %s" % (self.name,))
2015-11-15T14:49:32.704 INFO:tasks.workunit.client.0.plana48.stderr:rbd.InvalidArgument: error writing to test_notify_clone
2015-11-15T14:49:32.708 INFO:tasks.workunit:Stopping ['rbd/notify_master.sh'] on client.0...
2015-11-15T14:49:32.708 INFO:teuthology.orchestra.run.plana48:Running: 'rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/workunit.client.0'
2015-11-15T14:49:32.720 ERROR:teuthology.parallel:Exception in parallel execution
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 82, in __exit__
    for result in self:
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 101, in next
    resurrect_traceback(result)
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 19, in capture_traceback
    return func(*args, **kwargs)
  File "/var/lib/teuthworker/src/ceph-qa-suite_jewel/tasks/workunit.py", line 361, in _run_tests
    label="workunit test {workunit}".format(workunit=workunit)
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/remote.py", line 156, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 378, in run
    r.wait()
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 114, in wait
    label=self.label)
CommandFailedError: Command failed (workunit test rbd/notify_master.sh) on plana48 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=hammer TESTDIR="/home/ubuntu/cephtest" CEPH_ID="0" PATH=$PATH:/usr/sbin RBD_FEATURES=13 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/workunit.client.0/rbd/notify_master.sh'

Associated revisions

Revision 3992d6fe (diff)
Added by Jason Dillaman almost 6 years ago

tests: notification slave needs to wait for master

If the slave instance starts before the master, race
conditions are possible.

Fixes: #13810
Backport: infernalis, hammer
Signed-off-by: Jason Dillaman <>

Revision d4c41d04 (diff)
Added by Jason Dillaman almost 6 years ago

tests: notification slave needs to wait for master

If the slave instance starts before the master, race
conditions are possible.

Fixes: #13810
Backport: infernalis, hammer
Signed-off-by: Jason Dillaman <>
(cherry picked from commit 3992d6fe67bbf82322cedc1582406caaf6d4de60)

Revision 447ab1d7 (diff)
Added by Jason Dillaman almost 6 years ago

tests: notification slave needs to wait for master

If the slave instance starts before the master, race
conditions are possible.

Fixes: #13810
Backport: infernalis, hammer
Signed-off-by: Jason Dillaman <>
(cherry picked from commit 3992d6fe67bbf82322cedc1582406caaf6d4de60)

History

#1 Updated by Sage Weil about 6 years ago

  • Project changed from Ceph to rbd

#4 Updated by Jason Dillaman almost 6 years ago

  • Status changed from New to In Progress
  • Assignee set to Jason Dillaman

#5 Updated by Jason Dillaman almost 6 years ago

  • Status changed from In Progress to Fix Under Review

#6 Updated by Jason Dillaman almost 6 years ago

Changes to test_notify.py need to be backported to hammer and infernalis branches.

#7 Updated by Jason Dillaman almost 6 years ago

hammer PR: https://github.com/ceph/ceph/pull/7226
infernalis PR: https://github.com/ceph/ceph/pull/7225

(these only affect teuthology, so no need for official backport process)

#8 Updated by Jason Dillaman almost 6 years ago

  • Status changed from Fix Under Review to Resolved

Also available in: Atom PDF