Bug #13364
closedLibCephFS locking tests are failing and/or lockdep asserting
0%
Description
Run: http://qa-proxy.ceph.com/teuthology/teuthology-2015-10-04_05:00:08-smoke-master-distro-basic-multi
Job: 1088065
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2015-10-04_05:00:08-smoke-master-distro-basic-multi/1088065/teuthology.log
2015-10-04T06:11:27.517 INFO:tasks.workunit.client.0.plana20.stdout:Value of: status 2015-10-04T06:11:27.517 INFO:tasks.workunit.client.0.plana20.stdout: Actual: 139 2015-10-04T06:11:27.518 INFO:tasks.workunit.client.0.plana20.stdout:Expected: 0 2015-10-04T06:11:27.518 INFO:tasks.workunit.client.0.plana20.stdout:[ FAILED ] LibCephFS.InterProcessLocking (269 ms) 2015-10-04T06:11:27.518 INFO:tasks.workunit.client.0.plana20.stdout:[ RUN ] LibCephFS.ThreesomeInterProcessLocking 2015-10-04T06:11:27.867 INFO:tasks.workunit.client.0.plana20.stdout:test/libcephfs/flock.cc:628: Failure 2015-10-04T06:11:27.867 INFO:tasks.workunit.client.0.plana20.stdout:Value of: status 2015-10-04T06:11:27.868 INFO:tasks.workunit.client.0.plana20.stdout: Actual: 139 2015-10-04T06:11:27.868 INFO:tasks.workunit.client.0.plana20.stdout:Expected: 0 2015-10-04T06:11:27.868 INFO:tasks.workunit.client.0.plana20.stdout:[ FAILED ] LibCephFS.ThreesomeInterProcessLocking (350 ms) 2015-10-04T06:11:27.869 INFO:tasks.workunit.client.0.plana20.stdout:[----------] 43 tests from LibCephFS (103928 ms total) 2015-10-04T06:11:27.869 INFO:tasks.workunit.client.0.plana20.stdout: 2015-10-04T06:11:27.869 INFO:tasks.workunit.client.0.plana20.stdout:[----------] 1 test from Caps 2015-10-04T06:11:27.870 INFO:tasks.workunit.client.0.plana20.stdout:[ RUN ] Caps.ReadZero 2015-10-04T06:11:32.252 INFO:tasks.workunit.client.0.plana20.stdout:[ OK ] Caps.ReadZero (4385 ms) 2015-10-04T06:11:32.252 INFO:tasks.workunit.client.0.plana20.stdout:[----------] 1 test from Caps (4385 ms total) 2015-10-04T06:11:32.252 INFO:tasks.workunit.client.0.plana20.stdout: 2015-10-04T06:11:32.252 INFO:tasks.workunit.client.0.plana20.stdout:[----------] 4 tests from AccessTest 2015-10-04T06:11:32.253 INFO:tasks.workunit.client.0.plana20.stdout:[ RUN ] AccessTest.Foo 2015-10-04T06:11:32.278 INFO:tasks.workunit.client.0.plana20.stdout:out: [{"entity":"client.libcephfs_foo_test.1909293361","key":"AQCEJRFWsL8bEBAAQ7NPaIeFgwqeeJtOPI+aIg==","caps":{"mds":"allow rw","mon":"allow *","osd":"allow rw"}}] 2015-10-04T06:11:32.278 INFO:tasks.workunit.client.0.plana20.stdout:key: AQCEJRFWsL8bEBAAQ7NPaIeFgwqeeJtOPI+aIg== 2015-10-04T06:11:37.249 INFO:tasks.workunit.client.0.plana20.stdout:[ OK ] AccessTest.Foo (4997 ms) 2015-10-04T06:11:37.250 INFO:tasks.workunit.client.0.plana20.stdout:[ RUN ] AccessTest.Path 2015-10-04T06:11:37.288 INFO:tasks.workunit.client.0.plana20.stdout:out: [{"entity":"client.libcephfs_path_test.437206246","key":"AQCJJRFWZy3AEBAAf4BIJUCd5yBOVsfnm2OfdQ==","caps":{"mds":"allow r, allow rw path=\/ceph_test_libcephfs.1699450276","mon":"allow r","osd":"allow rwx"}}] 2015-10-04T06:11:37.288 INFO:tasks.workunit.client.0.plana20.stdout:key: AQCJJRFWZy3AEBAAf4BIJUCd5yBOVsfnm2OfdQ== 2015-10-04T06:11:37.343 INFO:tasks.workunit.client.0.plana20.stdout:test/libcephfs/access.cc:173: Failure 2015-10-04T06:11:37.343 INFO:tasks.workunit.client.0.plana20.stdout:Expected: (ceph_ftruncate(cmount, fd, 0)) >= (0), actual: -13 vs 0 2015-10-04T06:11:37.343 INFO:tasks.workunit.client.0.plana20.stdout:[ FAILED ] AccessTest.Path (94 ms) 2015-10-04T06:11:37.343 INFO:tasks.workunit.client.0.plana20.stdout:[ RUN ] AccessTest.ReadOnly 2015-10-04T06:11:37.373 INFO:tasks.workunit.client.0.plana20.stdout:out: [{"entity":"client.libcephfs_readonly_test.119217342","key":"AQCJJRFWEE3aFRAA\/QoZTOI1tepzMZIDsGdXbw==","caps":{"mds":"allow r","mon":"allow r","osd":"allow rw"}}] 2015-10-04T06:11:37.373 INFO:tasks.workunit.client.0.plana20.stdout:key: AQCJJRFWEE3aFRAA/QoZTOI1tepzMZIDsGdXbw== 2015-10-04T06:11:42.250 INFO:tasks.workunit.client.0.plana20.stdout:[ OK ] AccessTest.ReadOnly (4907 ms) 2015-10-04T06:11:42.250 INFO:tasks.workunit.client.0.plana20.stdout:[ RUN ] AccessTest.User 2015-10-04T06:11:42.275 INFO:tasks.workunit.client.0.plana20.stdout:out: [{"entity":"client.libcephfs_user_test.1966880974","key":"AQCOJRFWseP2DxAA2w+x5kqT30z6ogb6ekuRTg==","caps":{"mds":"allow rw uid=123 gids=456,789","mon":"allow *","osd":"allow rw"}}] 2015-10-04T06:11:42.275 INFO:tasks.workunit.client.0.plana20.stdout:key: AQCOJRFWseP2DxAA2w+x5kqT30z6ogb6ekuRTg== 2015-10-04T06:11:49.498 INFO:teuthology.orchestra.run.burnupi14:Running: 'sudo logrotate /etc/logrotate.d/ceph-test.conf' 2015-10-04T06:11:49.501 INFO:teuthology.orchestra.run.mira041:Running: 'sudo logrotate /etc/logrotate.d/ceph-test.conf' 2015-10-04T06:11:49.504 INFO:teuthology.orchestra.run.plana20:Running: 'sudo logrotate /etc/logrotate.d/ceph-test.conf' 2015-10-04T06:11:52.250 INFO:tasks.workunit.client.0.plana20.stdout:[ OK ] AccessTest.User (10000 ms) 2015-10-04T06:11:52.250 INFO:tasks.workunit.client.0.plana20.stdout:[----------] 4 tests from AccessTest (19998 ms total) 2015-10-04T06:11:52.250 INFO:tasks.workunit.client.0.plana20.stdout: 2015-10-04T06:11:52.250 INFO:tasks.workunit.client.0.plana20.stdout:[----------] Global test environment tear-down 2015-10-04T06:11:52.250 INFO:tasks.workunit.client.0.plana20.stdout:[==========] 48 tests from 3 test cases ran. (128311 ms total) 2015-10-04T06:11:52.250 INFO:tasks.workunit.client.0.plana20.stdout:[ PASSED ] 45 tests. 2015-10-04T06:11:52.250 INFO:tasks.workunit.client.0.plana20.stdout:[ FAILED ] 3 tests, listed below: 2015-10-04T06:11:52.251 INFO:tasks.workunit.client.0.plana20.stdout:[ FAILED ] LibCephFS.InterProcessLocking 2015-10-04T06:11:52.251 INFO:tasks.workunit.client.0.plana20.stdout:[ FAILED ] LibCephFS.ThreesomeInterProcessLocking 2015-10-04T06:11:52.251 INFO:tasks.workunit.client.0.plana20.stdout:[ FAILED ] AccessTest.Path 2015-10-04T06:11:52.251 INFO:tasks.workunit.client.0.plana20.stdout: 2015-10-04T06:11:52.251 INFO:tasks.workunit.client.0.plana20.stdout: 3 FAILED TESTS
Updated by Yuri Weinstein over 8 years ago
- Subject changed from "LibCephFS.*" 3 test failed in smoke-master-distro-basic-multi to "LibCephFS.*" tests failed in smoke-master-distro-basic-multi
Updated by Sage Weil over 8 years ago
- Status changed from New to 12
- Assignee set to Sage Weil
Updated by Sage Weil over 8 years ago
https://github.com/ceph/ceph/pull/6200
fixes the path check.
These still fail:
[ RUN ] LibCephFS.InterProcessLocking
test/libcephfs/flock.cc:392: Failure
Value of: ceph_mount(cmount, __null)
Actual: -22
Expected: 0
test/libcephfs/flock.cc:465: Failure
Value of: sem_timedwait(&s.sem[1%2], abstime(ts, waitSlowMs))
Actual: -1
Expected: 0
[ FAILED ] LibCephFS.InterProcessLocking (5108 ms)
2015-10-07 21:15:36,246.246 INFO:tasks.workunit.client.0.plana91.stdout:[ RUN ] LibCephFS.ThreesomeInterProcessLocking
2015-10-07 21:15:36,708.708 INFO:tasks.workunit.client.0.plana91.stdout:test/libcephfs/flock.cc:628: Failure
2015-10-07 21:15:36,709.709 INFO:tasks.workunit.client.0.plana91.stdout:Value of: status
2015-10-07 21:15:36,709.709 INFO:tasks.workunit.client.0.plana91.stdout: Actual: 139
2015-10-07 21:15:36,709.709 INFO:tasks.workunit.client.0.plana91.stdout:Expected: 0
2015-10-07 21:15:36,710.710 INFO:tasks.workunit.client.0.plana91.stdout:[ FAILED ] LibCephFS.ThreesomeInterProcessLocking (464 ms)
but i'm not sure why...
Updated by Greg Farnum over 8 years ago
- Project changed from Ceph to CephFS
- Assignee changed from Sage Weil to Greg Farnum
Sage says this passes when run against a vstart cluster.
Updated by Greg Farnum over 8 years ago
If somebody else wants to investigate this, feel free, but otherwise I'll try and get to it.
Updated by Greg Farnum over 8 years ago
- Status changed from 12 to In Progress
Looking at this now, it's weird.
Updated by Greg Farnum over 8 years ago
- Subject changed from "LibCephFS.*" tests failed in smoke-master-distro-basic-multi to LibCephFS locking tests are failing and/or lockdep asserting
- Status changed from In Progress to 12
- Assignee deleted (
Greg Farnum)
http://pulpito.ceph.com/teuthology-2015-11-09_23:04:02-fs-master---basic-multi/1144848/
We've been hitting lockdep asserts on these tests too; here's one of them ^
Updated by Zheng Yan over 8 years ago
Greg Farnum wrote:
http://pulpito.ceph.com/teuthology-2015-11-09_23:04:02-fs-master---basic-multi/1144848/
We've been hitting lockdep asserts on these tests too; here's one of them ^
still is side effect of previous failure (missing . in readdir result)
Updated by Greg Farnum over 8 years ago
Zheng thinks this is two bugs:
1) Missing "." in list results (fixed in waiting PR)
2) if MDS fragments directory, we "restart" listing and that breaks the test by resetting the dir pointer.
Updated by Zheng Yan over 8 years ago
- Status changed from 12 to Fix Under Review
add a commit to fix the readdir test case to https://github.com/ceph/ceph/pull/6260
Updated by Greg Farnum over 8 years ago
- Status changed from Fix Under Review to Resolved
Merged to jewel branch as of 088b8c2c881c8561b9bc96934e04d030e8de9255.
I created a new feature ticket for the directory resetting thing: #14271.