Project

General

Profile

Actions

Bug #15335

closed

with bluestore enable osd failed on restart osd/OSD.h: 886: FAILED assert(ret)

Added by chee zhu about 8 years ago. Updated about 7 years ago.

Status:
Can't reproduce
Priority:
Normal
Assignee:
-
Category:
OSD
Target version:
-
% Done:

0%

Source:
Community (user)
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

A cluster witch 3 nodes and 3monitors and 12 osds ; enable the experimental features:bluestore,rocksdb
When restartone node; the osd.6 on this node can not start with the error:

###############################
_* DB Stats **
Uptime(secs): 0.1 total, 0.1 interval
Cumulative writes: 0 writes, 0 keys, 0 batches, 0.0 writes per batch, ingest: 0.00 GB, 0.00 MB/s
Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s
Cumulative compaction: 0.00 GB write, 3.88 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
Interval writes: 0 writes, 0 keys, 0 batches, 0.0 writes per batch, ingest: 0.00 MB, 0.00 MB/s
Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s
Interval compaction: 0.00 GB write, 3.88 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
Interval stall: 00:00:0.000 H:M:S, 0.0 percent

-6> 2016-03-31 11:03:19.530213 7f3ff50d3700 4 rocksdb: [default] [JOB 3] Compacting 4@0 files to L1, score 1.00
-5> 2016-03-31 11:03:19.530219 7f3ff50d3700 4 rocksdb: [default] Compaction start summary: Base version 2 Base level 0, inputs: [17(549KB) 15(862KB) 7(820B) 4(597B)]

-4> 2016-03-31 11:03:19.530240 7f3ff50d3700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1459393399530222, "job": 3, "event": "compaction_started", "files_L0": [17, 15, 7, 4], "score": 1, "input_data_size": 1447519}
-3> 2016-03-31 11:03:19.534981 7f4001115800 2 osd.6 0 boot
-2> 2016-03-31 11:03:19.535420 7f4001115800 0 <cls> cls/cephfs/cls_cephfs.cc:202: loading cephfs_size_scan
-1> 2016-03-31 11:03:19.535576 7f4001115800 0 <cls> cls/hello/cls_hello.cc:305: loading cls_hello
0> 2016-03-31 11:03:19.543536 7f4001115800 -1 osd/OSD.h: In function 'OSDMapRef OSDService::get_map(epoch_t)' thread 7f4001115800 time 2016-03-31 11:03:19.541793
osd/OSD.h: 886: FAILED assert(ret)

ceph version 10.1.0 (96ae8bd25f31862dbd5302f304ebf8bf1166aba6)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0x7f4001b45a55]
2: (OSDService::get_map(unsigned int)+0x3d) [0x7f40015326fd]
3: (OSD::init()+0xf22) [0x7f40014e5172]
4: (main()+0x2aab) [0x7f400144ba2b]
5: (_libc_start_main()+0xf5) [0x7f3ffe046b15]
6: (()+0x349689) [0x7f4001495689]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

##############################

The same thing happened on the version 10.0.5 also. details please see the attached log.

Enabled the bluestore with the config:
enable_experimental_unrecoverable_data_corrupting_features = bluestore rocksdb
osd_objectstore = bluestore
bluestore_block_size = 1073741824000
#bluestore_block_db_size = 1073741824
#bluestore_block_wal_size = 1073741824
bluestore_bluefs = false


Files

ceph.conf.bk1 (4 KB) ceph.conf.bk1 chee zhu, 03/31/2016 07:12 AM
ceph-osd.6.log (443 KB) ceph-osd.6.log chee zhu, 03/31/2016 09:17 AM
Actions #1

Updated by chee zhu about 8 years ago

attache the osd log;

Actions #3

Updated by Sage Weil about 7 years ago

  • Status changed from New to Can't reproduce
Actions

Also available in: Atom PDF