Actions
Bug #23463
closedsrc/os/bluestore/StupidAllocator.cc: 336: FAILED assert(rm.empty())
Status:
Can't reproduce
Priority:
Normal
Assignee:
-
Target version:
-
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
The ceph-volume nightly tests have seen this failure on one run so far (March 25th) with 2 out of 6 OSDs deployed. We have been unable to replicate or see it again
in any of our tests.
The failure happened for:
- ceph master (235f2119010484c12c5bd29421aeef7d44df38a1)
- CentOS 7
- Using `ceph-volume lvm`
- With dmcrypt
- With bluestore
changed: [osd0] => (item={u'data_vg': u'test_group', u'db': u'journal1', u'db_vg': u'journals', u'data': u'data-lv2'}) => {
"changed": true,
"cmd": [
"ceph-volume",
"--cluster",
"ceph",
"lvm",
"create",
"--bluestore",
"--data",
"test_group/data-lv2",
"--block.db",
"journals/journal1",
"--dmcrypt"
],
"delta": "0:00:41.484166",
"end": "2018-03-25 23:30:31.592695",
"failed": false,
"item": {
"data": "data-lv2",
"data_vg": "test_group",
"db": "journal1",
"db_vg": "journals"
},
"rc": 0,
"start": "2018-03-25 23:29:50.108529"
}
STDOUT:
Running command: /bin/ceph-authtool --gen-print-key
Running command: /bin/ceph-authtool --gen-print-key
Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new d45ce6aa-8a19-42a3-8be4-2832d652cbc5
Running command: /bin/ceph-authtool --gen-print-key
Running command: /usr/sbin/cryptsetup --batch-mode --key-file - luksFormat /dev/test_group/data-lv2
Running command: /usr/sbin/cryptsetup --key-file - luksOpen /dev/test_group/data-lv2 uktWUg-OTJe-hT0L-kqJX-q1Qk-8m3g-vmEtoY
Running command: /usr/sbin/cryptsetup --batch-mode --key-file - luksFormat /dev/journals/journal1
Running command: /usr/sbin/cryptsetup --key-file - luksOpen /dev/journals/journal1 H7sf8A-FbTy-isn3-J1Xe-0bdT-L9se-0ApudF
Running command: /bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-1
Running command: /bin/chown -R ceph:ceph /dev/dm-6
Running command: /bin/ln -s /dev/mapper/uktWUg-OTJe-hT0L-kqJX-q1Qk-8m3g-vmEtoY /var/lib/ceph/osd/ceph-1/block
Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-1/activate.monmap
stderr: got monmap epoch 1
Running command: /bin/ceph-authtool /var/lib/ceph/osd/ceph-1/keyring --create-keyring --name osd.1 --add-key AQDuMLhaM1A1DRAAJvexOz8zvevay+tMktd2qA==
stdout: creating /var/lib/ceph/osd/ceph-1/keyring
stdout: added entity osd.1 auth auth(auid = 18446744073709551615 key=AQDuMLhaM1A1DRAAJvexOz8zvevay+tMktd2qA== with 0 caps)
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1/keyring
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1/
Running command: /bin/chown -R ceph:ceph /dev/dm-7
Running command: /bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 1 --monmap /var/lib/ceph/osd/ceph-1/activate.monmap --keyfile - --bluestore-block-db-path /dev/mapper/H7sf8A-FbTy-isn3-J1Xe-0bdT-L9se-0ApudF --osd-data /var/lib/ceph/osd/ceph-1/ --osd-uuid d45ce6aa-8a19-42a3-8be4-2832d652cbc5 --setuser ceph --setgroup ceph
stderr: 2018-03-25 23:30:25.082 7f38b9406040 -1 bluestore(/var/lib/ceph/osd/ceph-1/) _read_fsid unparsable uuid
stderr: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.0.1-3240-g235f211/rpm/el7/BUILD/ceph-13.0.1-3240-g235f211/src/os/bluestore/StupidAllocator.cc: In function 'virtual void StupidAllocator::init_rm_free(uint64_t, uint64_t)' thread 7f38b9406040 time 2018-03-25 23:30:25.339214
stderr: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.0.1-3240-g235f211/rpm/el7/BUILD/ceph-13.0.1-3240-g235f211/src/os/bluestore/StupidAllocator.cc: 336: FAILED assert(rm.empty())
stderr: ceph version 13.0.1-3240-g235f211 (235f2119010484c12c5bd29421aeef7d44df38a1) mimic (dev)
stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0xff) [0x7f38b0847c8f]
stderr: 2: (()+0x278e77) [0x7f38b0847e77]
stderr: 3: (StupidAllocator::init_rm_free(unsigned long, unsigned long)+0x20c2) [0x563e13931a22]
stderr: 4: (BlueFS::mount()+0x222) [0x563e13911172]
stderr: 5: (BlueStore::_open_db(bool, bool)+0x1531) [0x563e138388d1]
stderr: 6: (BlueStore::mkfs()+0x699) [0x563e1386e429]
stderr: 7: (OSD::mkfs(CephContext*, ObjectStore*, std::string const&, uuid_d, int)+0x177) [0x563e134391a7]
stderr: 8: (main()+0x29fe) [0x563e1331324e]
stderr: 9: (__libc_start_main()+0xf5) [0x7f38ace3ec05]
stderr: 10: (()+0x37ea80) [0x563e133efa80]
stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
stderr: 2018-03-25 23:30:25.343 7f38b9406040 -1 /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.0.1-3240-g235f211/rpm/el7/BUILD/ceph-13.0.1-3240-g235f211/src/os/bluestore/StupidAllocator.cc: In function 'virtual void StupidAllocator::init_rm_free(uint64_t, uint64_t)' thread 7f38b9406040 time 2018-03-25 23:30:25.339214
stderr: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.0.1-3240-g235f211/rpm/el7/BUILD/ceph-13.0.1-3240-g235f211/src/os/bluestore/StupidAllocator.cc: 336: FAILED assert(rm.empty())
stderr: ceph version 13.0.1-3240-g235f211 (235f2119010484c12c5bd29421aeef7d44df38a1) mimic (dev)
stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0xff) [0x7f38b0847c8f]
stderr: 2: (()+0x278e77) [0x7f38b0847e77]
stderr: 3: (StupidAllocator::init_rm_free(unsigned long, unsigned long)+0x20c2) [0x563e13931a22]
stderr: 4: (BlueFS::mount()+0x222) [0x563e13911172]
stderr: 5: (BlueStore::_open_db(bool, bool)+0x1531) [0x563e138388d1]
stderr: 6: (BlueStore::mkfs()+0x699) [0x563e1386e429]
stderr: 7: (OSD::mkfs(CephContext*, ObjectStore*, std::string const&, uuid_d, int)+0x177) [0x563e134391a7]
stderr: 8: (main()+0x29fe) [0x563e1331324e]
stderr: 9: (__libc_start_main()+0xf5) [0x7f38ace3ec05]
stderr: 10: (()+0x37ea80) [0x563e133efa80]
stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
stderr: -21> 2018-03-25 23:30:25.082 7f38b9406040 -1 bluestore(/var/lib/ceph/osd/ceph-1/) _read_fsid unparsable uuid
stderr: 0> 2018-03-25 23:30:25.343 7f38b9406040 -1 /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.0.1-3240-g235f211/rpm/el7/BUILD/ceph-13.0.1-3240-g235f211/src/os/bluestore/StupidAllocator.cc: In function 'virtual void StupidAllocator::init_rm_free(uint64_t, uint64_t)' thread 7f38b9406040 time 2018-03-25 23:30:25.339214
stderr: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.0.1-3240-g235f211/rpm/el7/BUILD/ceph-13.0.1-3240-g235f211/src/os/bluestore/StupidAllocator.cc: 336: FAILED assert(rm.empty())
stderr: ceph version 13.0.1-3240-g235f211 (235f2119010484c12c5bd29421aeef7d44df38a1) mimic (dev)
stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0xff) [0x7f38b0847c8f]
stderr: 2: (()+0x278e77) [0x7f38b0847e77]
stderr: 3: (StupidAllocator::init_rm_free(unsigned long, unsigned long)+0x20c2) [0x563e13931a22]
stderr: 4: (BlueFS::mount()+0x222) [0x563e13911172]
stderr: 5: (BlueStore::_open_db(bool, bool)+0x1531) [0x563e138388d1]
stderr: 6: (BlueStore::mkfs()+0x699) [0x563e1386e429]
stderr: 7: (OSD::mkfs(CephContext*, ObjectStore*, std::string const&, uuid_d, int)+0x177) [0x563e134391a7]
stderr: 8: (main()+0x29fe) [0x563e1331324e]
stderr: 9: (__libc_start_main()+0xf5) [0x7f38ace3ec05]
stderr: 10: (()+0x37ea80) [0x563e133efa80]
stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
stderr: *** Caught signal (Aborted) **
stderr: in thread 7f38b9406040 thread_name:ceph-osd
stderr: ceph version 13.0.1-3240-g235f211 (235f2119010484c12c5bd29421aeef7d44df38a1) mimic (dev)
stderr: 1: (()+0x8e6e00) [0x563e13957e00]
stderr: 2: (()+0xf5e0) [0x7f38ade295e0]
stderr: 3: (gsignal()+0x37) [0x7f38ace521f7]
stderr: 4: (abort()+0x148) [0x7f38ace538e8]
stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x25d) [0x7f38b0847ded]
stderr: 6: (()+0x278e77) [0x7f38b0847e77]
stderr: 7: (StupidAllocator::init_rm_free(unsigned long, unsigned long)+0x20c2) [0x563e13931a22]
stderr: 8: (BlueFS::mount()+0x222) [0x563e13911172]
stderr: 9: (BlueStore::_open_db(bool, bool)+0x1531) [0x563e138388d1]
stderr: 10: (BlueStore::mkfs()+0x699) [0x563e1386e429]
stderr: 11: (OSD::mkfs(CephContext*, ObjectStore*, std::string const&, uuid_d, int)+0x177) [0x563e134391a7]
stderr: 12: (main()+0x29fe) [0x563e1331324e]
stderr: 13: (__libc_start_main()+0xf5) [0x7f38ace3ec05]
stderr: 14: (()+0x37ea80) [0x563e133efa80]
stderr: 2018-03-25 23:30:25.349 7f38b9406040 -1 *** Caught signal (Aborted) **
stderr: in thread 7f38b9406040 thread_name:ceph-osd
stderr: ceph version 13.0.1-3240-g235f211 (235f2119010484c12c5bd29421aeef7d44df38a1) mimic (dev)
stderr: 1: (()+0x8e6e00) [0x563e13957e00]
stderr: 2: (()+0xf5e0) [0x7f38ade295e0]
stderr: 3: (gsignal()+0x37) [0x7f38ace521f7]
stderr: 4: (abort()+0x148) [0x7f38ace538e8]
stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x25d) [0x7f38b0847ded]
stderr: 6: (()+0x278e77) [0x7f38b0847e77]
stderr: 7: (StupidAllocator::init_rm_free(unsigned long, unsigned long)+0x20c2) [0x563e13931a22]
stderr: 8: (BlueFS::mount()+0x222) [0x563e13911172]
stderr: 9: (BlueStore::_open_db(bool, bool)+0x1531) [0x563e138388d1]
stderr: 10: (BlueStore::mkfs()+0x699) [0x563e1386e429]
stderr: 11: (OSD::mkfs(CephContext*, ObjectStore*, std::string const&, uuid_d, int)+0x177) [0x563e134391a7]
stderr: 12: (main()+0x29fe) [0x563e1331324e]
stderr: 13: (__libc_start_main()+0xf5) [0x7f38ace3ec05]
stderr: 14: (()+0x37ea80) [0x563e133efa80]
stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
stderr: 0> 2018-03-25 23:30:25.349 7f38b9406040 -1 *** Caught signal (Aborted) **
stderr: in thread 7f38b9406040 thread_name:ceph-osd
stderr: ceph version 13.0.1-3240-g235f211 (235f2119010484c12c5bd29421aeef7d44df38a1) mimic (dev)
stderr: 1: (()+0x8e6e00) [0x563e13957e00]
stderr: 2: (()+0xf5e0) [0x7f38ade295e0]
stderr: 3: (gsignal()+0x37) [0x7f38ace521f7]
stderr: 4: (abort()+0x148) [0x7f38ace538e8]
stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x25d) [0x7f38b0847ded]
stderr: 6: (()+0x278e77) [0x7f38b0847e77]
stderr: 7: (StupidAllocator::init_rm_free(unsigned long, unsigned long)+0x20c2) [0x563e13931a22]
stderr: 8: (BlueFS::mount()+0x222) [0x563e13911172]
stderr: 9: (BlueStore::_open_db(bool, bool)+0x1531) [0x563e138388d1]
stderr: 10: (BlueStore::mkfs()+0x699) [0x563e1386e429]
stderr: 11: (OSD::mkfs(CephContext*, ObjectStore*, std::string const&, uuid_d, int)+0x177) [0x563e134391a7]
stderr: 12: (main()+0x29fe) [0x563e1331324e]
stderr: 13: (__libc_start_main()+0xf5) [0x7f38ace3ec05]
stderr: 14: (()+0x37ea80) [0x563e133efa80]
stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Files
Actions