Project

General

Profile

Actions

Bug #41744

closed

os/bluestore/BlueStore.cc: 5313: FAILED ceph_assert(bluefs->maybe_verify_layout(bluefs_layout) == 0)

Added by Sage Weil over 4 years ago. Updated over 4 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

2019-09-10T04:10:09.733 INFO:tasks.workunit.client.0.smithi019.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:90: TEST_bluestore:  ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 bluefs-bdev-sizes
2019-09-10T04:10:09.748 INFO:tasks.workunit.client.0.smithi019.stdout:inferring bluefs devices from bluestore path
2019-09-10T04:10:09.756 INFO:tasks.workunit.client.0.smithi019.stdout: slot 2 td/osd-bluefs-volume-ops/0/block -> /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/block
2019-09-10T04:10:09.756 INFO:tasks.workunit.client.0.smithi019.stdout: slot 1 td/osd-bluefs-volume-ops/0/block.db -> /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/block.db
2019-09-10T04:10:09.756 INFO:tasks.workunit.client.0.smithi019.stdout: slot 0 td/osd-bluefs-volume-ops/0/block.wal -> /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/block.wal
2019-09-10T04:10:10.546 INFO:tasks.workunit.client.0.smithi019.stdout:0 : device size 0x20000000 : own 0x[1000~1ffff000] = 0x1ffff000 : using 0x5ff000(6.0 MiB)
2019-09-10T04:10:10.546 INFO:tasks.workunit.client.0.smithi019.stdout:1 : device size 0x40000000 : own 0x[2000~3fffe000] = 0x3fffe000 : using 0x9fe000(10 MiB)
2019-09-10T04:10:10.546 INFO:tasks.workunit.client.0.smithi019.stdout:2 : device size 0x100000000 : own 0x[30000000~20000000] = 0x20000000 : using 0x0(0 B)
2019-09-10T04:10:10.547 INFO:tasks.workunit.client.0.smithi019.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:92: TEST_bluestore:  ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.wal --dev-target td/osd-bluefs-volume-ops/0/block.db --command bluefs-bdev-migrate
2019-09-10T04:10:10.567 INFO:tasks.workunit.client.0.smithi019.stdout:inferring bluefs devices from bluestore path
2019-09-10T04:10:11.360 INFO:tasks.workunit.client.0.smithi019.stdout: device removed:0 td/osd-bluefs-volume-ops/0/block.wal
2019-09-10T04:10:11.364 INFO:tasks.workunit.client.0.smithi019.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:97: TEST_bluestore:  ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck
2019-09-10T04:10:11.388 INFO:tasks.workunit.client.0.smithi019.stderr:/build/ceph-15.0.0-4757-g2553842/src/os/bluestore/BlueStore.cc: In function 'int BlueStore::_open_bluefs(bool)' thread 7fd8bf0a4d80 time 2019-09-10T04:10:11.388751+0000
2019-09-10T04:10:11.389 INFO:tasks.workunit.client.0.smithi019.stderr:/build/ceph-15.0.0-4757-g2553842/src/os/bluestore/BlueStore.cc: 5313: FAILED ceph_assert(bluefs->maybe_verify_layout(bluefs_layout) == 0)
2019-09-10T04:10:11.389 INFO:tasks.workunit.client.0.smithi019.stderr:2019-09-10T04:10:11.388+0000 7fd8bf0a4d80 -1 bluefs maybe_verify_layout memorized layout doesn't fit current one
2019-09-10T04:10:11.391 INFO:tasks.workunit.client.0.smithi019.stderr: ceph version 15.0.0-4757-g2553842 (2553842e6458c797f8eb03866efc07694e6559f4) octopus (dev)
2019-09-10T04:10:11.391 INFO:tasks.workunit.client.0.smithi019.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7fd8b55d7d7a]
2019-09-10T04:10:11.391 INFO:tasks.workunit.client.0.smithi019.stderr: 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7fd8b55d7f55]
2019-09-10T04:10:11.391 INFO:tasks.workunit.client.0.smithi019.stderr: 3: (BlueStore::_open_bluefs(bool)+0x208) [0x55f8ecca3548]
2019-09-10T04:10:11.391 INFO:tasks.workunit.client.0.smithi019.stderr: 4: (BlueStore::_open_db(bool, bool, bool)+0xa14) [0x55f8ecca4764]
2019-09-10T04:10:11.391 INFO:tasks.workunit.client.0.smithi019.stderr: 5: (BlueStore::_open_db_and_around(bool)+0x44) [0x55f8eccbe154]
2019-09-10T04:10:11.392 INFO:tasks.workunit.client.0.smithi019.stderr: 6: (BlueStore::_fsck(bool, bool)+0x1017) [0x55f8eccfff17]
2019-09-10T04:10:11.392 INFO:tasks.workunit.client.0.smithi019.stderr: 7: (main()+0x10a0) [0x55f8ecc2e700]
2019-09-10T04:10:11.392 INFO:tasks.workunit.client.0.smithi019.stderr: 8: (__libc_start_main()+0xe7) [0x7fd8b3b68b97]
2019-09-10T04:10:11.392 INFO:tasks.workunit.client.0.smithi019.stderr: 9: (_start()+0x2a) [0x55f8ecc3578a]
2019-09-10T04:10:11.392 INFO:tasks.workunit.client.0.smithi019.stderr:*** Caught signal (Aborted) **

/a/sage-2019-09-09_21:29:24-rados-wip-sage2-testing-2019-09-09-1115-distro-basic-smithi/4293417
Actions #1

Updated by Igor Fedotov over 4 years ago

looks like one needs to update bluefs layout when doing volume add/removal.

Actions #2

Updated by Radoslaw Zarzynski over 4 years ago

Early, untested fix candidate: https://github.com/ceph/ceph/commit/d5b56c665a7f0ed7725e485ed05393a8b821ce7b.

Igor, I would to get your opinion on the approach the patch follows. I can send an RFC PR if you prefer.

Actions #3

Updated by Radoslaw Zarzynski over 4 years ago

  • Status changed from 12 to In Progress
Actions #4

Updated by Kefu Chai over 4 years ago

/a/http://pulpito.ceph.com/kchai-2019-09-13_04:19:52-rados-wip-kefu-testing-2019-09-11-2224-distro-basic-mira/4301801/

Actions #5

Updated by Kefu Chai over 4 years ago

/a/kchai-2019-09-21_17:17:30-rados-wip-kefu-testing-2019-09-20-1944-distro-basic-mira/4324219/

Actions #6

Updated by Radoslaw Zarzynski over 4 years ago

  • Status changed from In Progress to Fix Under Review
Actions #7

Updated by Igor Fedotov over 4 years ago

  • Status changed from Fix Under Review to Resolved
Actions

Also available in: Atom PDF