Bug #11236: lock inversions in test/libcephfs flock test - CephFS - Ceph

Actions

Copy link

Bug #11236

closed

lock inversions in test/libcephfs flock test

Added by Greg Farnum about 9 years ago. Updated almost 8 years ago.

Status:

Resolved

Priority:

High

Assignee:

Category:

Target version:

% Done:

Source:

Q/A

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Component(FS):

libcephfs

Labels (FS):

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

2015-03-25T00:07:52.051 INFO:tasks.workunit.client.0.burnupi33.stderr:common/lockdep.cc: In function 'int lockdep_will_lock(const char*, int)' thread 7fd101460700 time 2015-03-25 00:07:52.039716
2015-03-25T00:07:52.052 INFO:tasks.workunit.client.0.burnupi33.stderr:common/lockdep.cc: 192: FAILED assert(0)
2015-03-25T00:07:52.052 INFO:tasks.workunit.client.0.burnupi33.stderr: ceph version 0.93-782-g8dd9f6a (8dd9f6a4ede4ffb75a34d110a1f2a8743851b2e0)
2015-03-25T00:07:52.052 INFO:tasks.workunit.client.0.burnupi33.stderr: 1: (()+0x210a8b) [0x7fd1073f5a8b]
2015-03-25T00:07:52.052 INFO:tasks.workunit.client.0.burnupi33.stderr: 2: (()+0x25706f) [0x7fd10743c06f]
2015-03-25T00:07:52.052 INFO:tasks.workunit.client.0.burnupi33.stderr: 3: (()+0x1c75a8) [0x7fd1073ac5a8]
2015-03-25T00:07:52.052 INFO:tasks.workunit.client.0.burnupi33.stderr: 4: (()+0x1fa1af) [0x7fd1073df1af]
2015-03-25T00:07:52.052 INFO:tasks.workunit.client.0.burnupi33.stderr: 5: (()+0x1fb21d) [0x7fd1073e021d]
2015-03-25T00:07:52.052 INFO:tasks.workunit.client.0.burnupi33.stderr: 6: (()+0x8182) [0x7fd106fcf182]
2015-03-25T00:07:52.053 INFO:tasks.workunit.client.0.burnupi33.stderr: 7: (clone()+0x6d) [0x7fd1067e238d]
2015-03-25T00:07:52.053 INFO:tasks.workunit.client.0.burnupi33.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2015-03-25T00:07:52.053 INFO:tasks.workunit.client.0.burnupi33.stderr:terminate called after throwing an instance of 'ceph::FailedAssertion'
2015-03-25T00:07:57.074 INFO:tasks.workunit.client.0.burnupi33.stdout:test/libcephfs/flock.cc:465: Failure
2015-03-25T00:07:57.074 INFO:tasks.workunit.client.0.burnupi33.stdout:Value of: sem_timedwait(&s.sem[1%2], abstime(ts, waitSlowMs))
2015-03-25T00:07:57.074 INFO:tasks.workunit.client.0.burnupi33.stdout:  Actual: -1
2015-03-25T00:07:57.074 INFO:tasks.workunit.client.0.burnupi33.stdout:Expected: 0
2015-03-25T00:07:57.074 INFO:tasks.workunit.client.0.burnupi33.stdout:[  FAILED  ] LibCephFS.InterProcessLocking (5030 ms)
2015-03-25T00:07:57.074 INFO:tasks.workunit.client.0.burnupi33.stdout:[ RUN      ] LibCephFS.ThreesomeInterProcessLocking
2015-03-25T00:07:57.081 INFO:tasks.workunit.client.0.burnupi33.stderr:common/lockdep.cc: In function 'int lockdep_will_lock(const char*, int)' thread 7fd0b3fff700 time 2015-03-25 00:07:57.069663
2015-03-25T00:07:57.081 INFO:tasks.workunit.client.0.burnupi33.stderr:common/lockdep.cc: 192: FAILED assert(0)
2015-03-25T00:07:57.082 INFO:tasks.workunit.client.0.burnupi33.stderr: ceph version 0.93-782-g8dd9f6a (8dd9f6a4ede4ffb75a34d110a1f2a8743851b2e0)
2015-03-25T00:07:57.082 INFO:tasks.workunit.client.0.burnupi33.stderr: 1: (()+0x210a8b) [0x7fd1073f5a8b]
2015-03-25T00:07:57.082 INFO:tasks.workunit.client.0.burnupi33.stderr: 2: (()+0x25706f) [0x7fd10743c06f]
2015-03-25T00:07:57.082 INFO:tasks.workunit.client.0.burnupi33.stderr: 3: (()+0x1c75a8) [0x7fd1073ac5a8]
2015-03-25T00:07:57.082 INFO:tasks.workunit.client.0.burnupi33.stderr: 4: (()+0x1fa1af) [0x7fd1073df1af]
2015-03-25T00:07:57.082 INFO:tasks.workunit.client.0.burnupi33.stderr: 5: (()+0x1fb21d) [0x7fd1073e021d]
2015-03-25T00:07:57.082 INFO:tasks.workunit.client.0.burnupi33.stderr: 6: (()+0x8182) [0x7fd106fcf182]
2015-03-25T00:07:57.083 INFO:tasks.workunit.client.0.burnupi33.stderr: 7: (clone()+0x6d) [0x7fd1067e238d]
2015-03-25T00:07:57.083 INFO:tasks.workunit.client.0.burnupi33.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2015-03-25T00:07:57.083 INFO:tasks.workunit.client.0.burnupi33.stderr:terminate called after throwing an instance of 'ceph::FailedAssertion'
2015-03-25T00:07:57.084 INFO:tasks.workunit.client.0.burnupi33.stderr:common/lockdep.cc: In function 'int lockdep_will_lock(const char*, int)' thread 7fd0b3fff700 time 2015-03-25 00:07:57.072588
2015-03-25T00:07:57.084 INFO:tasks.workunit.client.0.burnupi33.stderr:common/lockdep.cc: 192: FAILED assert(0)
2015-03-25T00:07:57.085 INFO:tasks.workunit.client.0.burnupi33.stderr: ceph version 0.93-782-g8dd9f6a (8dd9f6a4ede4ffb75a34d110a1f2a8743851b2e0)
2015-03-25T00:07:57.085 INFO:tasks.workunit.client.0.burnupi33.stderr: 1: (()+0x210a8b) [0x7fd1073f5a8b]
2015-03-25T00:07:57.085 INFO:tasks.workunit.client.0.burnupi33.stderr: 2: (()+0x25706f) [0x7fd10743c06f]
2015-03-25T00:07:57.085 INFO:tasks.workunit.client.0.burnupi33.stderr: 3: (()+0x1c75a8) [0x7fd1073ac5a8]
2015-03-25T00:07:57.085 INFO:tasks.workunit.client.0.burnupi33.stderr: 4: (()+0x1fa1af) [0x7fd1073df1af]
2015-03-25T00:07:57.085 INFO:tasks.workunit.client.0.burnupi33.stderr: 5: (()+0x1fb21d) [0x7fd1073e021d]
2015-03-25T00:07:57.085 INFO:tasks.workunit.client.0.burnupi33.stderr: 6: (()+0x8182) [0x7fd106fcf182]
2015-03-25T00:07:57.086 INFO:tasks.workunit.client.0.burnupi33.stderr: 7: (clone()+0x6d) [0x7fd1067e238d]
2015-03-25T00:07:57.086 INFO:tasks.workunit.client.0.burnupi33.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2015-03-25T00:07:57.086 INFO:tasks.workunit.client.0.burnupi33.stderr:terminate called after throwing an instance of 'ceph::FailedAssertion'

We recorded some of these in #11128 as well, but I was hoping this was a result of the synchronization issues we saw there. Apparently it's not, because the run at http://qa-proxy.ceph.com/teuthology/teuthology-2015-03-23_23:04:02-fs-master-testing-basic-multi/819222/ includes all our patches to fix it up. :(

Actions

Copy link

Updated by Zheng Yan about 9 years ago

Status changed from New to Fix Under Review

this failure happens if we do a fork() while ceph is in-use, then open ceph again in child process.

Actions

Copy link

Updated by Greg Farnum about 9 years ago

Merged to master in commit:413da564d4fd479954da7feb5b27b5d8fade1100

Actions

Copy link

Updated by Greg Farnum about 9 years ago

Status changed from Fix Under Review to Resolved

Actions

Copy link

Updated by John Spray almost 9 years ago

Status changed from Resolved to 12
Regression set to No

This appears to have resurfaced here:

http://pulpito.front.sepia.ceph.com/teuthology-2015-05-04_23:04:01-fs-master-testing-basic-multi/875736/

Actions

Copy link

Updated by Zheng Yan almost 9 years ago

John Spray wrote:

This appears to have resurfaced here:

http://pulpito.front.sepia.ceph.com/teuthology-2015-05-04_23:04:01-fs-master-testing-basic-multi/875736/

the failure is side effect of failure of ceph_read(). When a test case fails, it does not release the cmount. lockdep assertion will be triggered if we call fork() while there is in-use cmount.

Actions

Copy link

Updated by John Spray almost 9 years ago

Status changed from 12 to Resolved

Cool, that makes sense to me.

Actions

Copy link

Updated by Greg Farnum almost 9 years ago

Status changed from Resolved to Need More Info

Did we create a ticket for the failed test case(s)? We should perhaps fix the tests so that we don't get semi-fake warnings about lockdep if that's going to be disguising genuine failures!

Actions

Copy link