Project

General

Profile

Bug #10785

test_librbd: [ FAILED ] TestImageWatcher.RequestLock

Added by Jason Dillaman about 9 years ago. Updated about 9 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Jason Dillaman
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

http://qa-proxy.ceph.com/teuthology/teuthology-2015-02-04_23:00:02-rbd-master-testing-basic-multi/740488/teuthology.log

2015-02-05T13:36:53.555 INFO:tasks.workunit.client.0.plana52.stdout:[ RUN      ] TestImageWatcher.RequestLock
2015-02-05T13:36:53.922 INFO:tasks.workunit.client.0.plana52.stdout:test/librbd/test_ImageWatcher.cc:78: Failure
2015-02-05T13:36:53.923 INFO:tasks.workunit.client.0.plana52.stdout:Value of: static_cast<NotifyOp>(op)
2015-02-05T13:36:53.923 INFO:tasks.workunit.client.0.plana52.stdout:  Actual: 1
2015-02-05T13:36:53.923 INFO:tasks.workunit.client.0.plana52.stdout:Expected: notify_op
2015-02-05T13:36:53.923 INFO:tasks.workunit.client.0.plana52.stdout:Which is: 0
2015-02-05T13:36:56.530 INFO:tasks.thrashosds.thrasher:in_osds:  [3, 0, 5, 4] out_osds:  [2, 1] dead_osds:  [] live_osds:  [0, 3, 2, 5, 4, 1]
2015-02-05T13:36:56.531 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0
2015-02-05T13:36:56.531 INFO:tasks.thrashosds.thrasher:inject_pause on 0
2015-02-05T13:36:56.531 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3
2015-02-05T13:36:56.531 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False
2015-02-05T13:36:56.531 INFO:teuthology.orchestra.run.plana52:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config set filestore_inject_stall 3'
2015-02-05T13:36:58.923 INFO:tasks.workunit.client.0.plana52.stdout:test/librbd/test_ImageWatcher.cc:439: Failure
2015-02-05T13:36:58.923 INFO:tasks.workunit.client.0.plana52.stdout:Value of: m_ioctx.notify2(ictx->header_oid, bl, 5000, __null)
2015-02-05T13:36:58.923 INFO:tasks.workunit.client.0.plana52.stdout:  Actual: -110
2015-02-05T13:36:58.923 INFO:tasks.workunit.client.0.plana52.stdout:Expected: 0
2015-02-05T13:37:01.658 INFO:tasks.thrashosds.thrasher:in_osds:  [3, 0, 5, 4] out_osds:  [2, 1] dead_osds:  [] live_osds:  [0, 3, 2, 5, 4, 1]
2015-02-05T13:37:01.658 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0
2015-02-05T13:37:01.659 INFO:tasks.thrashosds.thrasher:Adding osd 1
2015-02-05T13:37:01.659 INFO:teuthology.orchestra.run.plana52:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd in 1'
2015-02-05T13:37:01.987 INFO:tasks.workunit.client.0.plana52.stdout:[  FAILED  ] TestImageWatcher.RequestLock (8433 ms)

Associated revisions

Revision debd7f30 (diff)
Added by Jason Dillaman about 9 years ago

tests: relax librbd ImageWatcher test case state machine

If librados repeats watch/notify messages, the test case state
machine will get confused about the unexpected (repeated) message.
Relaxed the test case watch/notify state machine to accept
repeated messages.

Fixes: #10785
Signed-off-by: Jason Dillaman <>

History

#1 Updated by Jason Dillaman about 9 years ago

watch/notify will "randomly" replay old notifications when socket errors are introduced (i.e. ms inject socket failures = 500):

NOTIFY: 2, 32160715112464, 72477632, 204102
NOTIFY: 1, 32160715112465, 72477632, 204102
NOTIFY: 1, 32160715112466, 72477632, 204102 <-- client replayed notify 32160715112465
NOTIFY: 1, 32160715112466, 72477632, 204102 <-- server replayed notify 32160715112466
NOTIFY: 1, 32160715112467, 72477632, 204102 <-- client replayed notify 32160715112465/32160715112466
NOTIFY: 0, 32160715112468, 72477632, 204102
NOTIFY: 1, 32160715112466, 72477632, 204102 <-- server replayed notify 32160715112466
NOTIFY: 1, 32160715112467, 72477632, 204102 <-- server replayed notify 32160715112467
NOTIFY: 0, 32160715112468, 72477632, 204102 <-- server replayed notify 32160715112468
NOTIFY: 1, 32160715112469, 72477632, 204102
NOTIFY: 1, 32160715112466, 72477632, 204102
NOTIFY: 1, 32160715112467, 72477632, 204102
NOTIFY: 0, 32160715112468, 72477632, 204102
NOTIFY: 1, 32160715112469, 72477632, 204102
NOTIFY: 0, 32160715112470, 72477632, 204102
NOTIFY: 1, 32160715112471, 72477632, 204102
NOTIFY: 0, 32160715112472, 72477632, 204102

#2 Updated by Jason Dillaman about 9 years ago

  • Status changed from In Progress to Fix Under Review

#3 Updated by Jason Dillaman about 9 years ago

  • Status changed from Fix Under Review to Resolved

Also available in: Atom PDF