Project

General

Profile

Bug #13432

"Mutex.cc: 105: FAILED assert(r == 0)" in rbd run

Added by Yuri Weinstein over 8 years ago. Updated over 8 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Jason Dillaman
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
rbd
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Run: http://pulpito.ceph.com/teuthology-2015-10-05_23:00:07-rbd-master---basic-multi/
Job: 1091223
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2015-10-05_23:00:07-rbd-master---basic-multi/1091223/teuthology.log

2015-10-08T13:36:15.216 INFO:teuthology.orchestra.run.plana09.stdout:rbd engine: RBD version: 0.1.10
2015-10-08T13:36:15.350 INFO:teuthology.orchestra.run.plana09.stdout:rbd engine: RBD version: 0.1.10
2015-10-08T13:42:49.619 INFO:teuthology.orchestra.run.plana09.stderr:common/Mutex.cc: In function 'void Mutex::Lock(bool)' thread 7f9c057ea700 time 2015-10-08 13:42:49.592973
2015-10-08T13:42:49.620 INFO:teuthology.orchestra.run.plana09.stderr:common/Mutex.cc: 105: FAILED assert(r == 0)
2015-10-08T13:42:49.634 INFO:teuthology.orchestra.run.plana09.stderr: ceph version 9.0.3-2146-gd1e6976 (d1e697628931775e7fcdbb79a76ddd8e7dedffd5)
2015-10-08T13:42:49.634 INFO:teuthology.orchestra.run.plana09.stderr: 1: (()+0x18ebcb) [0x7f9c68dfdbcb]
2015-10-08T13:42:49.635 INFO:teuthology.orchestra.run.plana09.stderr: 2: (()+0x144dcc) [0x7f9c68db3dcc]
2015-10-08T13:42:49.635 INFO:teuthology.orchestra.run.plana09.stderr: 3: (()+0x5c4c0) [0x7f9c68ccb4c0]
2015-10-08T13:42:49.635 INFO:teuthology.orchestra.run.plana09.stderr: 4: /home/ubuntu/rbd-fio-test/fio-fio-2.2.9/fio() [0x462904]
2015-10-08T13:42:49.635 INFO:teuthology.orchestra.run.plana09.stderr: 5: (()+0x5c149) [0x7f9c68ccb149]
2015-10-08T13:42:49.635 INFO:teuthology.orchestra.run.plana09.stderr: 6: (()+0x5d145) [0x7f9c68ccc145]
2015-10-08T13:42:49.635 INFO:teuthology.orchestra.run.plana09.stderr: 7: (()+0x5d64d) [0x7f9c68ccc64d]
2015-10-08T13:42:49.636 INFO:teuthology.orchestra.run.plana09.stderr: 8: (()+0x5b899) [0x7f9c68cca899]
2015-10-08T13:42:49.636 INFO:teuthology.orchestra.run.plana09.stderr: 9: (()+0x5de37) [0x7f9c68ccce37]
2015-10-08T13:42:49.636 INFO:teuthology.orchestra.run.plana09.stderr: 10: (()+0x5bd18) [0x7f9c68ccad18]
2015-10-08T13:42:49.636 INFO:teuthology.orchestra.run.plana09.stderr: 11: (()+0x81e62) [0x7f9c68cf0e62]
2015-10-08T13:42:49.636 INFO:teuthology.orchestra.run.plana09.stderr: 12: (()+0x180126) [0x7f9c68def126]
2015-10-08T13:42:49.636 INFO:teuthology.orchestra.run.plana09.stderr: 13: (()+0x181010) [0x7f9c68df0010]
2015-10-08T13:42:49.637 INFO:teuthology.orchestra.run.plana09.stderr: 14: (()+0x8182) [0x7f9c65f39182]
2015-10-08T13:42:49.637 INFO:teuthology.orchestra.run.plana09.stderr: 15: (clone()+0x6d) [0x7f9c65a6247d]
2015-10-08T13:42:49.637 INFO:teuthology.orchestra.run.plana09.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2015-10-08T13:42:49.647 INFO:teuthology.orchestra.run.plana09.stderr:terminate called after throwing an instance of 'ceph::FailedAssertion'
2015-10-08T13:42:49.721 INFO:tasks.rbd_fio:Cleaning up fio install

History

#1 Updated by Yuri Weinstein over 8 years ago

  • Subject changed from "Mutex.cc: 105: FAILED assert(r == 0)" in to "Mutex.cc: 105: FAILED assert(r == 0)" in rbd run

#2 Updated by Vasu Kulkarni over 8 years ago

  • Project changed from Ceph to rbd

Is this a issue with rbd fio engine? If So I can open a ticket at fio github to get it fixed.

#3 Updated by Vasu Kulkarni over 8 years ago

yeah this is from ceph common/mutex.cc not fio

#4 Updated by Jason Dillaman over 8 years ago

  • Status changed from New to In Progress
  • Assignee set to Jason Dillaman

#5 Updated by Jason Dillaman over 8 years ago

Race condition within fio rbd engine where a 'fio_rbd_iou' object is reused prior to retrieving the return value from the completion.

#6 Updated by Jason Dillaman over 8 years ago

  • Status changed from In Progress to 15

#8 Updated by Tamilarasi muthamizhan over 8 years ago

the fix is pending upstream for fio.
not a blocker for ceph. once fio bug is fixed, the teuthology test has to be modified to pull the latest fio.

ps: this is a new test that was added for infernalis.

#9 Updated by Jason Dillaman over 8 years ago

  • Status changed from 15 to Resolved

Fixed included in fio v2.2.11 release

Also available in: Atom PDF