Actions
Bug #12651
closedosd crashes at startup if current/commit_op_seq is not readable
Status:
Won't Fix
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:
0%
Source:
other
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
on master 281d45d2ad449d38a0ec3be668d9bba7735b9d21
2015-08-06 16:13:34.663402 7f0e105a37c0 0 ceph version (), process ceph-osd, pid 2427
starting osd.19 at :/0 osd_data /var/lib/ceph/osd/by-user/ceph-osd1 /var/lib/ceph/osd/by-user/ceph-osd1/journal
2015-08-06 16:13:34.673120 7f0e105a37c0 0 filestore(/var/lib/ceph/osd/by-user/ceph-osd1) backend btrfs (magic 0x9123683e)
2015-08-06 16:13:34.674727 7f0e105a37c0 0 genericfilestorebackend(/var/lib/ceph/osd/by-user/ceph-osd1) detect_features: FIEMAP ioctl is supported and appears to work
2015-08-06 16:13:34.674732 7f0e105a37c0 0 genericfilestorebackend(/var/lib/ceph/osd/by-user/ceph-osd1) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
2015-08-06 16:13:34.953972 7f0e105a37c0 0 genericfilestorebackend(/var/lib/ceph/osd/by-user/ceph-osd1) detect_features: syncfs(2) syscall fully supported (by glibc and kernel)
2015-08-06 16:13:34.954073 7f0e105a37c0 0 btrfsfilestorebackend(/var/lib/ceph/osd/by-user/ceph-osd1) detect_feature: CLONE_RANGE ioctl is supported
2015-08-06 16:13:35.212241 7f0e105a37c0 0 btrfsfilestorebackend(/var/lib/ceph/osd/by-user/ceph-osd1) detect_feature: SNAP_CREATE is supported
2015-08-06 16:13:35.212370 7f0e105a37c0 0 btrfsfilestorebackend(/var/lib/ceph/osd/by-user/ceph-osd1) detect_feature: SNAP_DESTROY is supported
2015-08-06 16:13:35.213477 7f0e105a37c0 0 btrfsfilestorebackend(/var/lib/ceph/osd/by-user/ceph-osd1) detect_feature: START_SYNC is supported (transid 17832)
2015-08-06 16:13:35.345677 7f0e105a37c0 0 btrfsfilestorebackend(/var/lib/ceph/osd/by-user/ceph-osd1) detect_feature: WAIT_SYNC is supported
2015-08-06 16:13:35.349348 7f0e105a37c0 0 btrfsfilestorebackend(/var/lib/ceph/osd/by-user/ceph-osd1) detect_feature: SNAP_CREATE_V2 is supported
2015-08-06 16:13:35.762381 7f0e105a37c0 -1 filestore(/var/lib/ceph/osd/by-user/ceph-osd1) FileStore::mount: read_op_seq failed
2015-08-06 16:13:35.762483 7f0e105a37c0 -1 osd.19 0 OSD::init() : unable to read osd superblock
common/Thread.cc: In function 'int Thread::join(void**)' thread 7f0e105a37c0 time 2015-08-06 16:13:35.762488
common/Thread.cc: 136: FAILED assert("join on thread that was never started" == 0)
ceph version ()
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x78) [0xb13da8]
2: (Thread::join(void**)+0x41) [0xaf8b41]
3: (FileStore::umount()+0x18d) [0x8a398d]
4: (OSD::init()+0x845) [0x6718e5]
5: (main()+0x2a49) [0x60f0b9]
6: (__libc_start_main()+0xf5) [0x7f0e0df29995]
7: (_start()+0x29) [0x6153a9]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2015-08-06 16:13:35.762977 7f0e105a37c0 -1 common/Thread.cc: In function 'int Thread::join(void**)' thread 7f0e105a37c0 time 2015-08-06 16:13:35.762488
common/Thread.cc: 136: FAILED assert("join on thread that was never started" == 0)
ceph version ()
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x78) [0xb13da8]
2: (Thread::join(void**)+0x41) [0xaf8b41]
3: (FileStore::umount()+0x18d) [0x8a398d]
4: (OSD::init()+0x845) [0x6718e5]
5: (main()+0x2a49) [0x60f0b9]
6: (__libc_start_main()+0xf5) [0x7f0e0df29995]
7: (_start()+0x29) [0x6153a9]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- begin dump of recent events ---
-51> 2015-08-06 16:13:34.662075 7f0e105a37c0 5 asok(0x7f0e0d046100) register_command perfcounters_dump hook 0x7f0e0d03d060
-50> 2015-08-06 16:13:34.662094 7f0e105a37c0 5 asok(0x7f0e0d046100) register_command 1 hook 0x7f0e0d03d060
-49> 2015-08-06 16:13:34.662098 7f0e105a37c0 5 asok(0x7f0e0d046100) register_command perf dump hook 0x7f0e0d03d060
-48> 2015-08-06 16:13:34.662101 7f0e105a37c0 5 asok(0x7f0e0d046100) register_command perfcounters_schema hook 0x7f0e0d03d060
-47> 2015-08-06 16:13:34.662103 7f0e105a37c0 5 asok(0x7f0e0d046100) register_command 2 hook 0x7f0e0d03d060
-46> 2015-08-06 16:13:34.662105 7f0e105a37c0 5 asok(0x7f0e0d046100) register_command perf schema hook 0x7f0e0d03d060
-45> 2015-08-06 16:13:34.662112 7f0e105a37c0 5 asok(0x7f0e0d046100) register_command perf reset hook 0x7f0e0d03d060
-44> 2015-08-06 16:13:34.662116 7f0e105a37c0 5 asok(0x7f0e0d046100) register_command config show hook 0x7f0e0d03d060
-43> 2015-08-06 16:13:34.662119 7f0e105a37c0 5 asok(0x7f0e0d046100) register_command config set hook 0x7f0e0d03d060
-42> 2015-08-06 16:13:34.662121 7f0e105a37c0 5 asok(0x7f0e0d046100) register_command config get hook 0x7f0e0d03d060
-41> 2015-08-06 16:13:34.662124 7f0e105a37c0 5 asok(0x7f0e0d046100) register_command config diff hook 0x7f0e0d03d060
-40> 2015-08-06 16:13:34.662126 7f0e105a37c0 5 asok(0x7f0e0d046100) register_command log flush hook 0x7f0e0d03d060
-39> 2015-08-06 16:13:34.662129 7f0e105a37c0 5 asok(0x7f0e0d046100) register_command log dump hook 0x7f0e0d03d060
-38> 2015-08-06 16:13:34.662131 7f0e105a37c0 5 asok(0x7f0e0d046100) register_command log reopen hook 0x7f0e0d03d060
-37> 2015-08-06 16:13:34.663402 7f0e105a37c0 0 ceph version (), process ceph-osd, pid 2427
-36> 2015-08-06 16:13:34.663661 7f0e105a37c0 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6809/2427 need_addr=1
-35> 2015-08-06 16:13:34.663677 7f0e105a37c0 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6810/2427 need_addr=1
-34> 2015-08-06 16:13:34.663685 7f0e105a37c0 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6811/2427 need_addr=1
-33> 2015-08-06 16:13:34.663695 7f0e105a37c0 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6812/2427 need_addr=1
-32> 2015-08-06 16:13:34.665240 7f0e105a37c0 5 asok(0x7f0e0d046100) init /var/run/ceph/ceph-osd.19.asok
-31> 2015-08-06 16:13:34.665248 7f0e105a37c0 5 asok(0x7f0e0d046100) bind_and_listen /var/run/ceph/ceph-osd.19.asok
-30> 2015-08-06 16:13:34.665303 7f0e105a37c0 5 asok(0x7f0e0d046100) register_command 0 hook 0x7f0e0d03c108
-29> 2015-08-06 16:13:34.665307 7f0e105a37c0 5 asok(0x7f0e0d046100) register_command version hook 0x7f0e0d03c108
-28> 2015-08-06 16:13:34.665310 7f0e105a37c0 5 asok(0x7f0e0d046100) register_command git_version hook 0x7f0e0d03c108
-27> 2015-08-06 16:13:34.665312 7f0e105a37c0 5 asok(0x7f0e0d046100) register_command help hook 0x7f0e0d03d120
-26> 2015-08-06 16:13:34.665315 7f0e105a37c0 5 asok(0x7f0e0d046100) register_command get_command_descriptions hook 0x7f0e0d03d130
-25> 2015-08-06 16:13:34.665338 7f0e105a37c0 10 monclient(hunting): build_initial_monmap
-24> 2015-08-06 16:13:34.665336 7f0e0b2b8700 5 asok(0x7f0e0d046100) entry start
-23> 2015-08-06 16:13:34.672859 7f0e105a37c0 5 adding auth protocol: cephx
-22> 2015-08-06 16:13:34.672863 7f0e105a37c0 5 adding auth protocol: cephx
-21> 2015-08-06 16:13:34.672913 7f0e105a37c0 5 asok(0x7f0e0d046100) register_command objecter_requests hook 0x7f0e0d03d1c0
-20> 2015-08-06 16:13:34.672956 7f0e105a37c0 1 -- 0.0.0.0:6809/2427 messenger.start
-19> 2015-08-06 16:13:34.672971 7f0e105a37c0 1 -- :/0 messenger.start
-18> 2015-08-06 16:13:34.672988 7f0e105a37c0 1 -- 0.0.0.0:6812/2427 messenger.start
-17> 2015-08-06 16:13:34.673007 7f0e105a37c0 1 -- 0.0.0.0:6811/2427 messenger.start
-16> 2015-08-06 16:13:34.673026 7f0e105a37c0 1 -- 0.0.0.0:6810/2427 messenger.start
-15> 2015-08-06 16:13:34.673035 7f0e105a37c0 1 -- :/0 messenger.start
-14> 2015-08-06 16:13:34.673062 7f0e105a37c0 2 osd.19 0 mounting /var/lib/ceph/osd/by-user/ceph-osd1 /var/lib/ceph/osd/by-user/ceph-osd1/journal
-13> 2015-08-06 16:13:34.673120 7f0e105a37c0 0 filestore(/var/lib/ceph/osd/by-user/ceph-osd1) backend btrfs (magic 0x9123683e)
-12> 2015-08-06 16:13:34.674727 7f0e105a37c0 0 genericfilestorebackend(/var/lib/ceph/osd/by-user/ceph-osd1) detect_features: FIEMAP ioctl is supported and appears to work
-11> 2015-08-06 16:13:34.674732 7f0e105a37c0 0 genericfilestorebackend(/var/lib/ceph/osd/by-user/ceph-osd1) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
-10> 2015-08-06 16:13:34.953972 7f0e105a37c0 0 genericfilestorebackend(/var/lib/ceph/osd/by-user/ceph-osd1) detect_features: syncfs(2) syscall fully supported (by glibc and kernel)
-9> 2015-08-06 16:13:34.954073 7f0e105a37c0 0 btrfsfilestorebackend(/var/lib/ceph/osd/by-user/ceph-osd1) detect_feature: CLONE_RANGE ioctl is supported
-8> 2015-08-06 16:13:35.212241 7f0e105a37c0 0 btrfsfilestorebackend(/var/lib/ceph/osd/by-user/ceph-osd1) detect_feature: SNAP_CREATE is supported
-7> 2015-08-06 16:13:35.212370 7f0e105a37c0 0 btrfsfilestorebackend(/var/lib/ceph/osd/by-user/ceph-osd1) detect_feature: SNAP_DESTROY is supported
-6> 2015-08-06 16:13:35.213477 7f0e105a37c0 0 btrfsfilestorebackend(/var/lib/ceph/osd/by-user/ceph-osd1) detect_feature: START_SYNC is supported (transid 17832)
-5> 2015-08-06 16:13:35.345677 7f0e105a37c0 0 btrfsfilestorebackend(/var/lib/ceph/osd/by-user/ceph-osd1) detect_feature: WAIT_SYNC is supported
-4> 2015-08-06 16:13:35.349348 7f0e105a37c0 0 btrfsfilestorebackend(/var/lib/ceph/osd/by-user/ceph-osd1) detect_feature: SNAP_CREATE_V2 is supported
-3> 2015-08-06 16:13:35.762381 7f0e105a37c0 -1 filestore(/var/lib/ceph/osd/by-user/ceph-osd1) FileStore::mount: read_op_seq failed
-2> 2015-08-06 16:13:35.762407 7f0e105a37c0 2 osd.19 0 boot
-1> 2015-08-06 16:13:35.762483 7f0e105a37c0 -1 osd.19 0 OSD::init() : unable to read osd superblock
0> 2015-08-06 16:13:35.762977 7f0e105a37c0 -1 common/Thread.cc: In function 'int Thread::join(void**)' thread 7f0e105a37c0 time 2015-08-06 16:13:35.762488
common/Thread.cc: 136: FAILED assert("join on thread that was never started" == 0)
ceph version ()
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x78) [0xb13da8]
2: (Thread::join(void**)+0x41) [0xaf8b41]
3: (FileStore::umount()+0x18d) [0x8a398d]
4: (OSD::init()+0x845) [0x6718e5]
5: (main()+0x2a49) [0x60f0b9]
6: (__libc_start_main()+0xf5) [0x7f0e0df29995]
7: (_start()+0x29) [0x6153a9]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
0/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 keyvaluestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
-2/-2 (syslog threshold)
99/99 (stderr threshold)
max_recent 10000
max_new 1000
log_file
--- end dump of recent events ---
terminate called after throwing an instance of 'ceph::FailedAssertion'
*** Caught signal (Aborted) **
in thread 7f0e105a37c0
ceph version ()
1: ceph-osd() [0xa1fd54]
2: (()+0x103e0) [0x7f0e0fae23e0]
3: (gsignal()+0x37) [0x7f0e0df3c3e7]
4: (abort()+0x16a) [0x7f0e0df3d7ea]
5: (__gnu_cxx::__verbose_terminate_handler()+0x15d) [0x7f0e0ea53e0d]
6: (()+0x8ce56) [0x7f0e0ea51e56]
7: (()+0x8cea1) [0x7f0e0ea51ea1]
8: (()+0x8d0b8) [0x7f0e0ea520b8]
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x25f) [0xb13f8f]
10: (Thread::join(void**)+0x41) [0xaf8b41]
11: (FileStore::umount()+0x18d) [0x8a398d]
12: (OSD::init()+0x845) [0x6718e5]
13: (main()+0x2a49) [0x60f0b9]
14: (__libc_start_main()+0xf5) [0x7f0e0df29995]
15: (_start()+0x29) [0x6153a9]
2015-08-06 16:13:35.764061 7f0e105a37c0 -1 *** Caught signal (Aborted) **
in thread 7f0e105a37c0
ceph version ()
1: ceph-osd() [0xa1fd54]
2: (()+0x103e0) [0x7f0e0fae23e0]
3: (gsignal()+0x37) [0x7f0e0df3c3e7]
4: (abort()+0x16a) [0x7f0e0df3d7ea]
5: (__gnu_cxx::__verbose_terminate_handler()+0x15d) [0x7f0e0ea53e0d]
6: (()+0x8ce56) [0x7f0e0ea51e56]
7: (()+0x8cea1) [0x7f0e0ea51ea1]
8: (()+0x8d0b8) [0x7f0e0ea520b8]
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x25f) [0xb13f8f]
10: (Thread::join(void**)+0x41) [0xaf8b41]
11: (FileStore::umount()+0x18d) [0x8a398d]
12: (OSD::init()+0x845) [0x6718e5]
13: (main()+0x2a49) [0x60f0b9]
14: (__libc_start_main()+0xf5) [0x7f0e0df29995]
15: (_start()+0x29) [0x6153a9]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- begin dump of recent events ---
0> 2015-08-06 16:13:35.764061 7f0e105a37c0 -1 *** Caught signal (Aborted) **
in thread 7f0e105a37c0
ceph version ()
1: ceph-osd() [0xa1fd54]
2: (()+0x103e0) [0x7f0e0fae23e0]
3: (gsignal()+0x37) [0x7f0e0df3c3e7]
4: (abort()+0x16a) [0x7f0e0df3d7ea]
5: (__gnu_cxx::__verbose_terminate_handler()+0x15d) [0x7f0e0ea53e0d]
6: (()+0x8ce56) [0x7f0e0ea51e56]
7: (()+0x8cea1) [0x7f0e0ea51ea1]
8: (()+0x8d0b8) [0x7f0e0ea520b8]
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x25f) [0xb13f8f]
10: (Thread::join(void**)+0x41) [0xaf8b41]
11: (FileStore::umount()+0x18d) [0x8a398d]
12: (OSD::init()+0x845) [0x6718e5]
13: (main()+0x2a49) [0x60f0b9]
14: (__libc_start_main()+0xf5) [0x7f0e0df29995]
15: (_start()+0x29) [0x6153a9]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
0/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 keyvaluestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
-2/-2 (syslog threshold)
99/99 (stderr threshold)
max_recent 10000
max_new 1000
log_file
--- end dump of recent events ---
Actions