Project

General

Profile

Bug #22535

OSD crushes with FAILED assert(used_blocks.size() > count) during the first start after upgrade 12.2.1 -> 12.2.2

Added by Vladimir Drobyshevskiy about 1 year ago. Updated 12 months ago.

Status:
Resolved
Priority:
Urgent
Assignee:
-
Target version:
-
Start date:
12/24/2017
Due date:
% Done:

0%

Source:
Tags:
Backport:
luminous
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:

Description

Hello!

I'm trying to upgrade Ceph Luminous cluster from 12.2.1 to 12.2.2. mon/mds/mgr restarts went flawlessly, but after the first osd restart I've got a problem.

I have "bluestore fsck on mount = true", so osd initiates fsck process on start. It takes a lot of time, but crashes with:

@
2017-12-24 18:54:07.158729 7fa6baf27e00 1 bluefs fsck
2017-12-24 18:54:07.158744 7fa6baf27e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _fsck walking object keyspace
2017-12-24 19:40:08.939488 7fa6baf27e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _fsck checking shared_blobs
2017-12-24 19:40:08.940645 7fa6baf27e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _fsck checking for stray omap data
2017-12-24 19:40:14.739627 7fa6baf27e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _fsck checking deferred events
2017-12-24 19:40:14.744003 7fa6baf27e00 1 bluestore(/var/lib/ceph/osd/ceph-0) _fsck checking freelist vs allocated
2017-12-24 19:40:26.217813 7fa6baf27e00 -1 /build/ceph-12.2.2/src/os/bluestore/BlueStore.cc: In function 'int BlueStore::_fsck(bool, bool)' thread 7fa6baf27e00 time 2017-12-24 19:40:26.215431
/build/ceph-12.2.2/src/os/bluestore/BlueStore.cc: 6122: FAILED assert(used_blocks.size() > count)

ceph version 12.2.2 (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x55e7f9f7e892]
2: (BlueStore::_fsck(bool, bool)+0x7e3c) [0x55e7f9e41c8c]
3: (BlueStore::_mount(bool)+0x20a) [0x55e7f9e4284a]
4: (OSD::init()+0x3e2) [0x55e7f99ae4e2]
5: (main()+0x2f07) [0x55e7f98c01d7]
6: (__libc_start_main()+0xf0) [0x7fa6b838f830]
7: (_start()+0x29) [0x55e7f994b7f9]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
@

Full ceph-osd.log from the "systemctl ceph-osd@x restart" is in the attachment. I believe it may be connected with #22464 bug.

If I disable fsck then osd starts normally.

I stopped upgrade process before I know my data is safe. What do I have to do in this situation?

Thanks!

It's a small all-flash cluster using Ubuntu 16.04, kernel 4.10.0-38.

@root@e001n02:/var/log# ceph -s
cluster:
id: 6a8ffc55-fa2e-48dc-a71c-647e1fff749b
health: HEALTH_WARN
too many PGs per OSD (252 > max 200)
mon e001n01 is low on available space

services:
mon: 3 daemons, quorum e001n01,e001n02,e001n03
mgr: e001n01(active), standbys: e001n02
mds: onefs-1/1/1 up {0=e001n03=up:active}, 2 up:standby
osd: 4 osds: 4 up, 4 in
data:
pools: 3 pools, 336 pgs
objects: 383k objects, 1497 GB
usage: 4306 GB used, 2421 GB / 6728 GB avail
pgs: 336 active+clean@

Pools use snappy compression.

ceph-osd.0.log View (221 KB) Vladimir Drobyshevskiy, 12/24/2017 06:24 PM


Related issues

Copied to bluestore - Backport #22633: luminous: OSD crushes with FAILED assert(used_blocks.size() > count) during the first start after upgrade 12.2.1 -> 12.2.2 Resolved

History

#1 Updated by Igor Fedotov about 1 year ago

  • Status changed from New to Need More Info

Could you please set bluestore debug level to 10 and provide a log for reproduced issue.

#2 Updated by Vladimir Drobyshevskiy about 1 year ago

Igor Fedotov wrote:

Could you please set bluestore debug level to 10 and provide a log for reproduced issue.

Sure. Here is a link to log file: https://cloud.mail.ru/public/BQEo/vsVhQqL8A

Please beware it is 47Mb bz2 and around 780Mb raw.

#3 Updated by Igor Fedotov about 1 year ago

Submitted https://github.com/ceph/ceph/pull/19718 to fix that in master.
IMO the root cause is a bug in out-of-range access to fsck's used_blocks bitmap. Looks like legacy store's could initialize freelist manager with block device size (=4K) while fsck in v12.2.2+ uses min_alloc_size (=16K) to estimate amount of used blocks. Additionally device size isn't aligned with that new min_alloc_size. For unknown reason boost doesn't 100% properly handle out-of-range access on dynamic_biteset resulting that later odd assert when size < count.

#4 Updated by Sage Weil about 1 year ago

  • Backport set to luminous

#5 Updated by Sage Weil about 1 year ago

  • Priority changed from Normal to Urgent

#6 Updated by Kefu Chai about 1 year ago

  • Status changed from Need More Info to Pending Backport
  • Target version deleted (v12.2.2)

#7 Updated by Nathan Cutler about 1 year ago

  • Copied to Backport #22633: luminous: OSD crushes with FAILED assert(used_blocks.size() > count) during the first start after upgrade 12.2.1 -> 12.2.2 added

#8 Updated by Nathan Cutler 12 months ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF