2015-04-28 00:34:38.101603 7f5d9a681900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 4012680 2015-04-28 00:34:38.123941 7f5d9a681900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. 2015-04-28 00:34:38.124182 7f5d9a681900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) 2015-04-28 00:34:38.125700 7f5d9a681900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work 2015-04-28 00:34:38.125724 7f5d9a681900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-04-28 00:34:38.131888 7f5d9a681900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2015-04-28 00:34:38.132075 7f5d9a681900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.9-031809-generic >= 3.5 2015-04-28 00:34:38.452728 7f5d9a681900 1 filestore upgrade_to_v2 start 2015-04-28 00:34:38.545159 7f5d9a681900 1 filestore upgrade_to_v2 done 2015-04-28 00:34:38.550103 7f5d9a681900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f5d9a681900 time 2015-04-28 00:34:38.545257 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f5d977c4ec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -50> 2015-04-28 00:34:38.097365 7f5d9a681900 5 asok(0x46c6000) register_command perfcounters_dump hook 0x4666050 -49> 2015-04-28 00:34:38.097437 7f5d9a681900 5 asok(0x46c6000) register_command 1 hook 0x4666050 -48> 2015-04-28 00:34:38.097448 7f5d9a681900 5 asok(0x46c6000) register_command perf dump hook 0x4666050 -47> 2015-04-28 00:34:38.097463 7f5d9a681900 5 asok(0x46c6000) register_command perfcounters_schema hook 0x4666050 -46> 2015-04-28 00:34:38.097472 7f5d9a681900 5 asok(0x46c6000) register_command 2 hook 0x4666050 -45> 2015-04-28 00:34:38.097480 7f5d9a681900 5 asok(0x46c6000) register_command perf schema hook 0x4666050 -44> 2015-04-28 00:34:38.097489 7f5d9a681900 5 asok(0x46c6000) register_command perf reset hook 0x4666050 -43> 2015-04-28 00:34:38.097514 7f5d9a681900 5 asok(0x46c6000) register_command config show hook 0x4666050 -42> 2015-04-28 00:34:38.097524 7f5d9a681900 5 asok(0x46c6000) register_command config set hook 0x4666050 -41> 2015-04-28 00:34:38.097533 7f5d9a681900 5 asok(0x46c6000) register_command config get hook 0x4666050 -40> 2015-04-28 00:34:38.097541 7f5d9a681900 5 asok(0x46c6000) register_command config diff hook 0x4666050 -39> 2015-04-28 00:34:38.097549 7f5d9a681900 5 asok(0x46c6000) register_command log flush hook 0x4666050 -38> 2015-04-28 00:34:38.097558 7f5d9a681900 5 asok(0x46c6000) register_command log dump hook 0x4666050 -37> 2015-04-28 00:34:38.097567 7f5d9a681900 5 asok(0x46c6000) register_command log reopen hook 0x4666050 -36> 2015-04-28 00:34:38.101603 7f5d9a681900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 4012680 -35> 2015-04-28 00:34:38.103114 7f5d9a681900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -34> 2015-04-28 00:34:38.103136 7f5d9a681900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6802/4012680 need_addr=0 -33> 2015-04-28 00:34:38.103182 7f5d9a681900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -32> 2015-04-28 00:34:38.103192 7f5d9a681900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6802/4012680 need_addr=0 -31> 2015-04-28 00:34:38.103211 7f5d9a681900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -30> 2015-04-28 00:34:38.103220 7f5d9a681900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6803/4012680 need_addr=0 -29> 2015-04-28 00:34:38.103240 7f5d9a681900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -28> 2015-04-28 00:34:38.103250 7f5d9a681900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6803/4012680 need_addr=0 -27> 2015-04-28 00:34:38.105576 7f5d9a681900 5 asok(0x46c6000) init /var/run/ceph/ceph-osd.23.asok -26> 2015-04-28 00:34:38.105601 7f5d9a681900 5 asok(0x46c6000) bind_and_listen /var/run/ceph/ceph-osd.23.asok -25> 2015-04-28 00:34:38.105647 7f5d9a681900 5 asok(0x46c6000) register_command 0 hook 0x46620a8 -24> 2015-04-28 00:34:38.105658 7f5d9a681900 5 asok(0x46c6000) register_command version hook 0x46620a8 -23> 2015-04-28 00:34:38.105667 7f5d9a681900 5 asok(0x46c6000) register_command git_version hook 0x46620a8 -22> 2015-04-28 00:34:38.105678 7f5d9a681900 5 asok(0x46c6000) register_command help hook 0x4666150 -21> 2015-04-28 00:34:38.105687 7f5d9a681900 5 asok(0x46c6000) register_command get_command_descriptions hook 0x4666140 -20> 2015-04-28 00:34:38.105746 7f5d9478c700 5 asok(0x46c6000) entry start -19> 2015-04-28 00:34:38.105755 7f5d9a681900 10 monclient(hunting): build_initial_monmap -18> 2015-04-28 00:34:38.122855 7f5d9a681900 5 adding auth protocol: cephx -17> 2015-04-28 00:34:38.122874 7f5d9a681900 5 adding auth protocol: cephx -16> 2015-04-28 00:34:38.123039 7f5d9a681900 5 asok(0x46c6000) register_command objecter_requests hook 0x4666180 -15> 2015-04-28 00:34:38.123162 7f5d9a681900 1 -- 192.168.30.73:6802/4012680 messenger.start -14> 2015-04-28 00:34:38.123219 7f5d9a681900 1 -- :/0 messenger.start -13> 2015-04-28 00:34:38.123265 7f5d9a681900 1 -- 192.168.30.73:6803/4012680 messenger.start -12> 2015-04-28 00:34:38.123303 7f5d9a681900 1 -- 192.168.32.73:6803/4012680 messenger.start -11> 2015-04-28 00:34:38.123338 7f5d9a681900 1 -- 192.168.32.73:6802/4012680 messenger.start -10> 2015-04-28 00:34:38.123381 7f5d9a681900 1 -- :/0 messenger.start -9> 2015-04-28 00:34:38.123473 7f5d9a681900 2 osd.23 0 mounting /var/lib/ceph/osd/ceph-23 /var/lib/ceph/osd/ceph-23/journal -8> 2015-04-28 00:34:38.123941 7f5d9a681900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. -7> 2015-04-28 00:34:38.124182 7f5d9a681900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) -6> 2015-04-28 00:34:38.125700 7f5d9a681900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work -5> 2015-04-28 00:34:38.125724 7f5d9a681900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option -4> 2015-04-28 00:34:38.131888 7f5d9a681900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) -3> 2015-04-28 00:34:38.132075 7f5d9a681900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.9-031809-generic >= 3.5 -2> 2015-04-28 00:34:38.452728 7f5d9a681900 1 filestore upgrade_to_v2 start -1> 2015-04-28 00:34:38.545159 7f5d9a681900 1 filestore upgrade_to_v2 done 0> 2015-04-28 00:34:38.550103 7f5d9a681900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f5d9a681900 time 2015-04-28 00:34:38.545257 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f5d977c4ec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 00:34:38.556895 7f5d9a681900 -1 *** Caught signal (Aborted) ** in thread 7f5d9a681900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f5d9933a340] 3: (gsignal()+0x39) [0x7f5d977d9cc9] 4: (abort()+0x148) [0x7f5d977dd0d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f5d980e46b5] 6: (()+0x5e836) [0x7f5d980e2836] 7: (()+0x5e863) [0x7f5d980e2863] 8: (()+0x5eaa2) [0x7f5d980e2aa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f5d977c4ec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2015-04-28 00:34:38.556895 7f5d9a681900 -1 *** Caught signal (Aborted) ** in thread 7f5d9a681900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f5d9933a340] 3: (gsignal()+0x39) [0x7f5d977d9cc9] 4: (abort()+0x148) [0x7f5d977dd0d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f5d980e46b5] 6: (()+0x5e836) [0x7f5d980e2836] 7: (()+0x5e863) [0x7f5d980e2863] 8: (()+0x5eaa2) [0x7f5d980e2aa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f5d977c4ec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 00:34:39.194475 7f96914d0900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 4013454 2015-04-28 00:34:39.207361 7f96914d0900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. 2015-04-28 00:34:39.207393 7f96914d0900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) 2015-04-28 00:34:39.208373 7f96914d0900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work 2015-04-28 00:34:39.208387 7f96914d0900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-04-28 00:34:39.213003 7f96914d0900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2015-04-28 00:34:39.213125 7f96914d0900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.9-031809-generic >= 3.5 2015-04-28 00:34:39.220919 7f96914d0900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f96914d0900 time 2015-04-28 00:34:39.218350 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f968e613ec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -48> 2015-04-28 00:34:39.192053 7f96914d0900 5 asok(0x3d66000) register_command perfcounters_dump hook 0x3d06050 -47> 2015-04-28 00:34:39.192099 7f96914d0900 5 asok(0x3d66000) register_command 1 hook 0x3d06050 -46> 2015-04-28 00:34:39.192103 7f96914d0900 5 asok(0x3d66000) register_command perf dump hook 0x3d06050 -45> 2015-04-28 00:34:39.192111 7f96914d0900 5 asok(0x3d66000) register_command perfcounters_schema hook 0x3d06050 -44> 2015-04-28 00:34:39.192116 7f96914d0900 5 asok(0x3d66000) register_command 2 hook 0x3d06050 -43> 2015-04-28 00:34:39.192141 7f96914d0900 5 asok(0x3d66000) register_command perf schema hook 0x3d06050 -42> 2015-04-28 00:34:39.192145 7f96914d0900 5 asok(0x3d66000) register_command perf reset hook 0x3d06050 -41> 2015-04-28 00:34:39.192158 7f96914d0900 5 asok(0x3d66000) register_command config show hook 0x3d06050 -40> 2015-04-28 00:34:39.192165 7f96914d0900 5 asok(0x3d66000) register_command config set hook 0x3d06050 -39> 2015-04-28 00:34:39.192168 7f96914d0900 5 asok(0x3d66000) register_command config get hook 0x3d06050 -38> 2015-04-28 00:34:39.192170 7f96914d0900 5 asok(0x3d66000) register_command config diff hook 0x3d06050 -37> 2015-04-28 00:34:39.192174 7f96914d0900 5 asok(0x3d66000) register_command log flush hook 0x3d06050 -36> 2015-04-28 00:34:39.192179 7f96914d0900 5 asok(0x3d66000) register_command log dump hook 0x3d06050 -35> 2015-04-28 00:34:39.192182 7f96914d0900 5 asok(0x3d66000) register_command log reopen hook 0x3d06050 -34> 2015-04-28 00:34:39.194475 7f96914d0900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 4013454 -33> 2015-04-28 00:34:39.195444 7f96914d0900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -32> 2015-04-28 00:34:39.195458 7f96914d0900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6808/4013454 need_addr=0 -31> 2015-04-28 00:34:39.195477 7f96914d0900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -30> 2015-04-28 00:34:39.195490 7f96914d0900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6808/4013454 need_addr=0 -29> 2015-04-28 00:34:39.195504 7f96914d0900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -28> 2015-04-28 00:34:39.195510 7f96914d0900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6809/4013454 need_addr=0 -27> 2015-04-28 00:34:39.195523 7f96914d0900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -26> 2015-04-28 00:34:39.195530 7f96914d0900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6809/4013454 need_addr=0 -25> 2015-04-28 00:34:39.197066 7f96914d0900 5 asok(0x3d66000) init /var/run/ceph/ceph-osd.23.asok -24> 2015-04-28 00:34:39.197084 7f96914d0900 5 asok(0x3d66000) bind_and_listen /var/run/ceph/ceph-osd.23.asok -23> 2015-04-28 00:34:39.197201 7f96914d0900 5 asok(0x3d66000) register_command 0 hook 0x3d020a8 -22> 2015-04-28 00:34:39.197212 7f96914d0900 5 asok(0x3d66000) register_command version hook 0x3d020a8 -21> 2015-04-28 00:34:39.197215 7f96914d0900 5 asok(0x3d66000) register_command git_version hook 0x3d020a8 -20> 2015-04-28 00:34:39.197219 7f96914d0900 5 asok(0x3d66000) register_command help hook 0x3d06150 -19> 2015-04-28 00:34:39.197227 7f96914d0900 5 asok(0x3d66000) register_command get_command_descriptions hook 0x3d06140 -18> 2015-04-28 00:34:39.197272 7f96914d0900 10 monclient(hunting): build_initial_monmap -17> 2015-04-28 00:34:39.197278 7f968b5db700 5 asok(0x3d66000) entry start -16> 2015-04-28 00:34:39.206851 7f96914d0900 5 adding auth protocol: cephx -15> 2015-04-28 00:34:39.206865 7f96914d0900 5 adding auth protocol: cephx -14> 2015-04-28 00:34:39.206987 7f96914d0900 5 asok(0x3d66000) register_command objecter_requests hook 0x3d06180 -13> 2015-04-28 00:34:39.207087 7f96914d0900 1 -- 192.168.30.73:6808/4013454 messenger.start -12> 2015-04-28 00:34:39.207127 7f96914d0900 1 -- :/0 messenger.start -11> 2015-04-28 00:34:39.207161 7f96914d0900 1 -- 192.168.30.73:6809/4013454 messenger.start -10> 2015-04-28 00:34:39.207193 7f96914d0900 1 -- 192.168.32.73:6809/4013454 messenger.start -9> 2015-04-28 00:34:39.207219 7f96914d0900 1 -- 192.168.32.73:6808/4013454 messenger.start -8> 2015-04-28 00:34:39.207244 7f96914d0900 1 -- :/0 messenger.start -7> 2015-04-28 00:34:39.207311 7f96914d0900 2 osd.23 0 mounting /var/lib/ceph/osd/ceph-23 /var/lib/ceph/osd/ceph-23/journal -6> 2015-04-28 00:34:39.207361 7f96914d0900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. -5> 2015-04-28 00:34:39.207393 7f96914d0900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) -4> 2015-04-28 00:34:39.208373 7f96914d0900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work -3> 2015-04-28 00:34:39.208387 7f96914d0900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option -2> 2015-04-28 00:34:39.213003 7f96914d0900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) -1> 2015-04-28 00:34:39.213125 7f96914d0900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.9-031809-generic >= 3.5 0> 2015-04-28 00:34:39.220919 7f96914d0900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f96914d0900 time 2015-04-28 00:34:39.218350 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f968e613ec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 00:34:39.224575 7f96914d0900 -1 *** Caught signal (Aborted) ** in thread 7f96914d0900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f9690189340] 3: (gsignal()+0x39) [0x7f968e628cc9] 4: (abort()+0x148) [0x7f968e62c0d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f968ef336b5] 6: (()+0x5e836) [0x7f968ef31836] 7: (()+0x5e863) [0x7f968ef31863] 8: (()+0x5eaa2) [0x7f968ef31aa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f968e613ec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2015-04-28 00:34:39.224575 7f96914d0900 -1 *** Caught signal (Aborted) ** in thread 7f96914d0900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f9690189340] 3: (gsignal()+0x39) [0x7f968e628cc9] 4: (abort()+0x148) [0x7f968e62c0d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f968ef336b5] 6: (()+0x5e836) [0x7f968ef31836] 7: (()+0x5e863) [0x7f968ef31863] 8: (()+0x5eaa2) [0x7f968ef31aa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f968e613ec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 00:34:39.787969 7ff0a6a76900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 4013968 2015-04-28 00:34:39.809464 7ff0a6a76900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. 2015-04-28 00:34:39.809511 7ff0a6a76900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) 2015-04-28 00:34:39.810964 7ff0a6a76900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work 2015-04-28 00:34:39.810987 7ff0a6a76900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-04-28 00:34:39.817233 7ff0a6a76900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2015-04-28 00:34:39.817433 7ff0a6a76900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.9-031809-generic >= 3.5 2015-04-28 00:34:39.827345 7ff0a6a76900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7ff0a6a76900 time 2015-04-28 00:34:39.822462 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7ff0a3bb9ec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -48> 2015-04-28 00:34:39.783529 7ff0a6a76900 5 asok(0x4810000) register_command perfcounters_dump hook 0x47b0050 -47> 2015-04-28 00:34:39.783606 7ff0a6a76900 5 asok(0x4810000) register_command 1 hook 0x47b0050 -46> 2015-04-28 00:34:39.783613 7ff0a6a76900 5 asok(0x4810000) register_command perf dump hook 0x47b0050 -45> 2015-04-28 00:34:39.783628 7ff0a6a76900 5 asok(0x4810000) register_command perfcounters_schema hook 0x47b0050 -44> 2015-04-28 00:34:39.783637 7ff0a6a76900 5 asok(0x4810000) register_command 2 hook 0x47b0050 -43> 2015-04-28 00:34:39.783642 7ff0a6a76900 5 asok(0x4810000) register_command perf schema hook 0x47b0050 -42> 2015-04-28 00:34:39.783647 7ff0a6a76900 5 asok(0x4810000) register_command perf reset hook 0x47b0050 -41> 2015-04-28 00:34:39.783667 7ff0a6a76900 5 asok(0x4810000) register_command config show hook 0x47b0050 -40> 2015-04-28 00:34:39.783677 7ff0a6a76900 5 asok(0x4810000) register_command config set hook 0x47b0050 -39> 2015-04-28 00:34:39.783681 7ff0a6a76900 5 asok(0x4810000) register_command config get hook 0x47b0050 -38> 2015-04-28 00:34:39.783686 7ff0a6a76900 5 asok(0x4810000) register_command config diff hook 0x47b0050 -37> 2015-04-28 00:34:39.783690 7ff0a6a76900 5 asok(0x4810000) register_command log flush hook 0x47b0050 -36> 2015-04-28 00:34:39.783694 7ff0a6a76900 5 asok(0x4810000) register_command log dump hook 0x47b0050 -35> 2015-04-28 00:34:39.783704 7ff0a6a76900 5 asok(0x4810000) register_command log reopen hook 0x47b0050 -34> 2015-04-28 00:34:39.787969 7ff0a6a76900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 4013968 -33> 2015-04-28 00:34:39.789485 7ff0a6a76900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -32> 2015-04-28 00:34:39.789507 7ff0a6a76900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6808/4013968 need_addr=0 -31> 2015-04-28 00:34:39.789540 7ff0a6a76900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -30> 2015-04-28 00:34:39.789546 7ff0a6a76900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6808/4013968 need_addr=0 -29> 2015-04-28 00:34:39.789571 7ff0a6a76900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -28> 2015-04-28 00:34:39.789581 7ff0a6a76900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6809/4013968 need_addr=0 -27> 2015-04-28 00:34:39.789604 7ff0a6a76900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -26> 2015-04-28 00:34:39.789614 7ff0a6a76900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6809/4013968 need_addr=0 -25> 2015-04-28 00:34:39.791939 7ff0a6a76900 5 asok(0x4810000) init /var/run/ceph/ceph-osd.23.asok -24> 2015-04-28 00:34:39.791965 7ff0a6a76900 5 asok(0x4810000) bind_and_listen /var/run/ceph/ceph-osd.23.asok -23> 2015-04-28 00:34:39.792135 7ff0a6a76900 5 asok(0x4810000) register_command 0 hook 0x47ac0a8 -22> 2015-04-28 00:34:39.792151 7ff0a6a76900 5 asok(0x4810000) register_command version hook 0x47ac0a8 -21> 2015-04-28 00:34:39.792158 7ff0a6a76900 5 asok(0x4810000) register_command git_version hook 0x47ac0a8 -20> 2015-04-28 00:34:39.792169 7ff0a6a76900 5 asok(0x4810000) register_command help hook 0x47b0150 -19> 2015-04-28 00:34:39.792175 7ff0a6a76900 5 asok(0x4810000) register_command get_command_descriptions hook 0x47b0140 -18> 2015-04-28 00:34:39.792246 7ff0a6a76900 10 monclient(hunting): build_initial_monmap -17> 2015-04-28 00:34:39.792275 7ff0a0b81700 5 asok(0x4810000) entry start -16> 2015-04-28 00:34:39.808794 7ff0a6a76900 5 adding auth protocol: cephx -15> 2015-04-28 00:34:39.808811 7ff0a6a76900 5 adding auth protocol: cephx -14> 2015-04-28 00:34:39.808975 7ff0a6a76900 5 asok(0x4810000) register_command objecter_requests hook 0x47b0180 -13> 2015-04-28 00:34:39.809088 7ff0a6a76900 1 -- 192.168.30.73:6808/4013968 messenger.start -12> 2015-04-28 00:34:39.809149 7ff0a6a76900 1 -- :/0 messenger.start -11> 2015-04-28 00:34:39.809190 7ff0a6a76900 1 -- 192.168.30.73:6809/4013968 messenger.start -10> 2015-04-28 00:34:39.809229 7ff0a6a76900 1 -- 192.168.32.73:6809/4013968 messenger.start -9> 2015-04-28 00:34:39.809267 7ff0a6a76900 1 -- 192.168.32.73:6808/4013968 messenger.start -8> 2015-04-28 00:34:39.809305 7ff0a6a76900 1 -- :/0 messenger.start -7> 2015-04-28 00:34:39.809395 7ff0a6a76900 2 osd.23 0 mounting /var/lib/ceph/osd/ceph-23 /var/lib/ceph/osd/ceph-23/journal -6> 2015-04-28 00:34:39.809464 7ff0a6a76900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. -5> 2015-04-28 00:34:39.809511 7ff0a6a76900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) -4> 2015-04-28 00:34:39.810964 7ff0a6a76900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work -3> 2015-04-28 00:34:39.810987 7ff0a6a76900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option -2> 2015-04-28 00:34:39.817233 7ff0a6a76900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) -1> 2015-04-28 00:34:39.817433 7ff0a6a76900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.9-031809-generic >= 3.5 0> 2015-04-28 00:34:39.827345 7ff0a6a76900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7ff0a6a76900 time 2015-04-28 00:34:39.822462 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7ff0a3bb9ec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 00:34:39.834133 7ff0a6a76900 -1 *** Caught signal (Aborted) ** in thread 7ff0a6a76900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7ff0a572f340] 3: (gsignal()+0x39) [0x7ff0a3bcecc9] 4: (abort()+0x148) [0x7ff0a3bd20d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7ff0a44d96b5] 6: (()+0x5e836) [0x7ff0a44d7836] 7: (()+0x5e863) [0x7ff0a44d7863] 8: (()+0x5eaa2) [0x7ff0a44d7aa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7ff0a3bb9ec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2015-04-28 00:34:39.834133 7ff0a6a76900 -1 *** Caught signal (Aborted) ** in thread 7ff0a6a76900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7ff0a572f340] 3: (gsignal()+0x39) [0x7ff0a3bcecc9] 4: (abort()+0x148) [0x7ff0a3bd20d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7ff0a44d96b5] 6: (()+0x5e836) [0x7ff0a44d7836] 7: (()+0x5e863) [0x7ff0a44d7863] 8: (()+0x5eaa2) [0x7ff0a44d7aa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7ff0a3bb9ec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 00:34:40.378002 7f0088abc900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 4014545 2015-04-28 00:34:40.399101 7f0088abc900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. 2015-04-28 00:34:40.399145 7f0088abc900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) 2015-04-28 00:34:40.400641 7f0088abc900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work 2015-04-28 00:34:40.400662 7f0088abc900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-04-28 00:34:40.407502 7f0088abc900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2015-04-28 00:34:40.407671 7f0088abc900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.9-031809-generic >= 3.5 2015-04-28 00:34:40.417181 7f0088abc900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f0088abc900 time 2015-04-28 00:34:40.412227 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f0085bffec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -48> 2015-04-28 00:34:40.373853 7f0088abc900 5 asok(0x43c4000) register_command perfcounters_dump hook 0x4364050 -47> 2015-04-28 00:34:40.373906 7f0088abc900 5 asok(0x43c4000) register_command 1 hook 0x4364050 -46> 2015-04-28 00:34:40.373917 7f0088abc900 5 asok(0x43c4000) register_command perf dump hook 0x4364050 -45> 2015-04-28 00:34:40.373928 7f0088abc900 5 asok(0x43c4000) register_command perfcounters_schema hook 0x4364050 -44> 2015-04-28 00:34:40.373933 7f0088abc900 5 asok(0x43c4000) register_command 2 hook 0x4364050 -43> 2015-04-28 00:34:40.373938 7f0088abc900 5 asok(0x43c4000) register_command perf schema hook 0x4364050 -42> 2015-04-28 00:34:40.373947 7f0088abc900 5 asok(0x43c4000) register_command perf reset hook 0x4364050 -41> 2015-04-28 00:34:40.373958 7f0088abc900 5 asok(0x43c4000) register_command config show hook 0x4364050 -40> 2015-04-28 00:34:40.373968 7f0088abc900 5 asok(0x43c4000) register_command config set hook 0x4364050 -39> 2015-04-28 00:34:40.373973 7f0088abc900 5 asok(0x43c4000) register_command config get hook 0x4364050 -38> 2015-04-28 00:34:40.373981 7f0088abc900 5 asok(0x43c4000) register_command config diff hook 0x4364050 -37> 2015-04-28 00:34:40.373985 7f0088abc900 5 asok(0x43c4000) register_command log flush hook 0x4364050 -36> 2015-04-28 00:34:40.373993 7f0088abc900 5 asok(0x43c4000) register_command log dump hook 0x4364050 -35> 2015-04-28 00:34:40.374000 7f0088abc900 5 asok(0x43c4000) register_command log reopen hook 0x4364050 -34> 2015-04-28 00:34:40.378002 7f0088abc900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 4014545 -33> 2015-04-28 00:34:40.379434 7f0088abc900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -32> 2015-04-28 00:34:40.379456 7f0088abc900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6810/4014545 need_addr=0 -31> 2015-04-28 00:34:40.379489 7f0088abc900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -30> 2015-04-28 00:34:40.379496 7f0088abc900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6810/4014545 need_addr=0 -29> 2015-04-28 00:34:40.379520 7f0088abc900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -28> 2015-04-28 00:34:40.379530 7f0088abc900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6811/4014545 need_addr=0 -27> 2015-04-28 00:34:40.379555 7f0088abc900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -26> 2015-04-28 00:34:40.379564 7f0088abc900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6811/4014545 need_addr=0 -25> 2015-04-28 00:34:40.381806 7f0088abc900 5 asok(0x43c4000) init /var/run/ceph/ceph-osd.23.asok -24> 2015-04-28 00:34:40.381828 7f0088abc900 5 asok(0x43c4000) bind_and_listen /var/run/ceph/ceph-osd.23.asok -23> 2015-04-28 00:34:40.381993 7f0088abc900 5 asok(0x43c4000) register_command 0 hook 0x43600a8 -22> 2015-04-28 00:34:40.382008 7f0088abc900 5 asok(0x43c4000) register_command version hook 0x43600a8 -21> 2015-04-28 00:34:40.382013 7f0088abc900 5 asok(0x43c4000) register_command git_version hook 0x43600a8 -20> 2015-04-28 00:34:40.382020 7f0088abc900 5 asok(0x43c4000) register_command help hook 0x4364150 -19> 2015-04-28 00:34:40.382025 7f0088abc900 5 asok(0x43c4000) register_command get_command_descriptions hook 0x4364140 -18> 2015-04-28 00:34:40.382090 7f0088abc900 10 monclient(hunting): build_initial_monmap -17> 2015-04-28 00:34:40.382121 7f0082bc7700 5 asok(0x43c4000) entry start -16> 2015-04-28 00:34:40.398458 7f0088abc900 5 adding auth protocol: cephx -15> 2015-04-28 00:34:40.398474 7f0088abc900 5 adding auth protocol: cephx -14> 2015-04-28 00:34:40.398641 7f0088abc900 5 asok(0x43c4000) register_command objecter_requests hook 0x4364180 -13> 2015-04-28 00:34:40.398749 7f0088abc900 1 -- 192.168.30.73:6810/4014545 messenger.start -12> 2015-04-28 00:34:40.398805 7f0088abc900 1 -- :/0 messenger.start -11> 2015-04-28 00:34:40.398841 7f0088abc900 1 -- 192.168.30.73:6811/4014545 messenger.start -10> 2015-04-28 00:34:40.398878 7f0088abc900 1 -- 192.168.32.73:6811/4014545 messenger.start -9> 2015-04-28 00:34:40.398915 7f0088abc900 1 -- 192.168.32.73:6810/4014545 messenger.start -8> 2015-04-28 00:34:40.398950 7f0088abc900 1 -- :/0 messenger.start -7> 2015-04-28 00:34:40.399034 7f0088abc900 2 osd.23 0 mounting /var/lib/ceph/osd/ceph-23 /var/lib/ceph/osd/ceph-23/journal -6> 2015-04-28 00:34:40.399101 7f0088abc900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. -5> 2015-04-28 00:34:40.399145 7f0088abc900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) -4> 2015-04-28 00:34:40.400641 7f0088abc900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work -3> 2015-04-28 00:34:40.400662 7f0088abc900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option -2> 2015-04-28 00:34:40.407502 7f0088abc900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) -1> 2015-04-28 00:34:40.407671 7f0088abc900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.9-031809-generic >= 3.5 0> 2015-04-28 00:34:40.417181 7f0088abc900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f0088abc900 time 2015-04-28 00:34:40.412227 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f0085bffec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 00:34:40.424004 7f0088abc900 -1 *** Caught signal (Aborted) ** in thread 7f0088abc900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f0087775340] 3: (gsignal()+0x39) [0x7f0085c14cc9] 4: (abort()+0x148) [0x7f0085c180d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f008651f6b5] 6: (()+0x5e836) [0x7f008651d836] 7: (()+0x5e863) [0x7f008651d863] 8: (()+0x5eaa2) [0x7f008651daa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f0085bffec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2015-04-28 00:34:40.424004 7f0088abc900 -1 *** Caught signal (Aborted) ** in thread 7f0088abc900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f0087775340] 3: (gsignal()+0x39) [0x7f0085c14cc9] 4: (abort()+0x148) [0x7f0085c180d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f008651f6b5] 6: (()+0x5e836) [0x7f008651d836] 7: (()+0x5e863) [0x7f008651d863] 8: (()+0x5eaa2) [0x7f008651daa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f0085bffec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 00:34:41.010155 7f54430be900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 4015051 2015-04-28 00:34:41.032386 7f54430be900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. 2015-04-28 00:34:41.032432 7f54430be900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) 2015-04-28 00:34:41.033923 7f54430be900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work 2015-04-28 00:34:41.033939 7f54430be900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-04-28 00:34:41.040814 7f54430be900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2015-04-28 00:34:41.040970 7f54430be900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.9-031809-generic >= 3.5 2015-04-28 00:34:41.050604 7f54430be900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f54430be900 time 2015-04-28 00:34:41.045711 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f5440201ec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -48> 2015-04-28 00:34:41.005764 7f54430be900 5 asok(0x45c0000) register_command perfcounters_dump hook 0x4560050 -47> 2015-04-28 00:34:41.005815 7f54430be900 5 asok(0x45c0000) register_command 1 hook 0x4560050 -46> 2015-04-28 00:34:41.005835 7f54430be900 5 asok(0x45c0000) register_command perf dump hook 0x4560050 -45> 2015-04-28 00:34:41.005860 7f54430be900 5 asok(0x45c0000) register_command perfcounters_schema hook 0x4560050 -44> 2015-04-28 00:34:41.005871 7f54430be900 5 asok(0x45c0000) register_command 2 hook 0x4560050 -43> 2015-04-28 00:34:41.005886 7f54430be900 5 asok(0x45c0000) register_command perf schema hook 0x4560050 -42> 2015-04-28 00:34:41.005894 7f54430be900 5 asok(0x45c0000) register_command perf reset hook 0x4560050 -41> 2015-04-28 00:34:41.005911 7f54430be900 5 asok(0x45c0000) register_command config show hook 0x4560050 -40> 2015-04-28 00:34:41.005921 7f54430be900 5 asok(0x45c0000) register_command config set hook 0x4560050 -39> 2015-04-28 00:34:41.005930 7f54430be900 5 asok(0x45c0000) register_command config get hook 0x4560050 -38> 2015-04-28 00:34:41.005935 7f54430be900 5 asok(0x45c0000) register_command config diff hook 0x4560050 -37> 2015-04-28 00:34:41.005943 7f54430be900 5 asok(0x45c0000) register_command log flush hook 0x4560050 -36> 2015-04-28 00:34:41.005948 7f54430be900 5 asok(0x45c0000) register_command log dump hook 0x4560050 -35> 2015-04-28 00:34:41.005956 7f54430be900 5 asok(0x45c0000) register_command log reopen hook 0x4560050 -34> 2015-04-28 00:34:41.010155 7f54430be900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 4015051 -33> 2015-04-28 00:34:41.012712 7f54430be900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -32> 2015-04-28 00:34:41.012737 7f54430be900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6802/4015051 need_addr=0 -31> 2015-04-28 00:34:41.012766 7f54430be900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -30> 2015-04-28 00:34:41.012773 7f54430be900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6802/4015051 need_addr=0 -29> 2015-04-28 00:34:41.012791 7f54430be900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -28> 2015-04-28 00:34:41.012822 7f54430be900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6803/4015051 need_addr=0 -27> 2015-04-28 00:34:41.012842 7f54430be900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -26> 2015-04-28 00:34:41.012849 7f54430be900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6803/4015051 need_addr=0 -25> 2015-04-28 00:34:41.015163 7f54430be900 5 asok(0x45c0000) init /var/run/ceph/ceph-osd.23.asok -24> 2015-04-28 00:34:41.015186 7f54430be900 5 asok(0x45c0000) bind_and_listen /var/run/ceph/ceph-osd.23.asok -23> 2015-04-28 00:34:41.015348 7f54430be900 5 asok(0x45c0000) register_command 0 hook 0x455c0a8 -22> 2015-04-28 00:34:41.015363 7f54430be900 5 asok(0x45c0000) register_command version hook 0x455c0a8 -21> 2015-04-28 00:34:41.015369 7f54430be900 5 asok(0x45c0000) register_command git_version hook 0x455c0a8 -20> 2015-04-28 00:34:41.015376 7f54430be900 5 asok(0x45c0000) register_command help hook 0x4560150 -19> 2015-04-28 00:34:41.015381 7f54430be900 5 asok(0x45c0000) register_command get_command_descriptions hook 0x4560140 -18> 2015-04-28 00:34:41.015449 7f54430be900 10 monclient(hunting): build_initial_monmap -17> 2015-04-28 00:34:41.015468 7f543d1c9700 5 asok(0x45c0000) entry start -16> 2015-04-28 00:34:41.031717 7f54430be900 5 adding auth protocol: cephx -15> 2015-04-28 00:34:41.031734 7f54430be900 5 adding auth protocol: cephx -14> 2015-04-28 00:34:41.031889 7f54430be900 5 asok(0x45c0000) register_command objecter_requests hook 0x4560180 -13> 2015-04-28 00:34:41.031996 7f54430be900 1 -- 192.168.30.73:6802/4015051 messenger.start -12> 2015-04-28 00:34:41.032053 7f54430be900 1 -- :/0 messenger.start -11> 2015-04-28 00:34:41.032087 7f54430be900 1 -- 192.168.30.73:6803/4015051 messenger.start -10> 2015-04-28 00:34:41.032126 7f54430be900 1 -- 192.168.32.73:6803/4015051 messenger.start -9> 2015-04-28 00:34:41.032162 7f54430be900 1 -- 192.168.32.73:6802/4015051 messenger.start -8> 2015-04-28 00:34:41.032219 7f54430be900 1 -- :/0 messenger.start -7> 2015-04-28 00:34:41.032317 7f54430be900 2 osd.23 0 mounting /var/lib/ceph/osd/ceph-23 /var/lib/ceph/osd/ceph-23/journal -6> 2015-04-28 00:34:41.032386 7f54430be900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. -5> 2015-04-28 00:34:41.032432 7f54430be900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) -4> 2015-04-28 00:34:41.033923 7f54430be900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work -3> 2015-04-28 00:34:41.033939 7f54430be900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option -2> 2015-04-28 00:34:41.040814 7f54430be900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) -1> 2015-04-28 00:34:41.040970 7f54430be900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.9-031809-generic >= 3.5 0> 2015-04-28 00:34:41.050604 7f54430be900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f54430be900 time 2015-04-28 00:34:41.045711 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f5440201ec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 00:34:41.057418 7f54430be900 -1 *** Caught signal (Aborted) ** in thread 7f54430be900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f5441d77340] 3: (gsignal()+0x39) [0x7f5440216cc9] 4: (abort()+0x148) [0x7f544021a0d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f5440b216b5] 6: (()+0x5e836) [0x7f5440b1f836] 7: (()+0x5e863) [0x7f5440b1f863] 8: (()+0x5eaa2) [0x7f5440b1faa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f5440201ec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2015-04-28 00:34:41.057418 7f54430be900 -1 *** Caught signal (Aborted) ** in thread 7f54430be900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f5441d77340] 3: (gsignal()+0x39) [0x7f5440216cc9] 4: (abort()+0x148) [0x7f544021a0d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f5440b216b5] 6: (()+0x5e836) [0x7f5440b1f836] 7: (()+0x5e863) [0x7f5440b1f863] 8: (()+0x5eaa2) [0x7f5440b1faa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f5440201ec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 00:34:41.515094 7f2714149900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 4015314 2015-04-28 00:34:41.536222 7f2714149900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. 2015-04-28 00:34:41.536267 7f2714149900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) 2015-04-28 00:34:41.537766 7f2714149900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work 2015-04-28 00:34:41.537781 7f2714149900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-04-28 00:34:41.544645 7f2714149900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2015-04-28 00:34:41.544827 7f2714149900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.9-031809-generic >= 3.5 2015-04-28 00:34:41.554116 7f2714149900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f2714149900 time 2015-04-28 00:34:41.549285 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f271128cec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -48> 2015-04-28 00:34:41.510819 7f2714149900 5 asok(0x3822000) register_command perfcounters_dump hook 0x37c2050 -47> 2015-04-28 00:34:41.510896 7f2714149900 5 asok(0x3822000) register_command 1 hook 0x37c2050 -46> 2015-04-28 00:34:41.510903 7f2714149900 5 asok(0x3822000) register_command perf dump hook 0x37c2050 -45> 2015-04-28 00:34:41.510915 7f2714149900 5 asok(0x3822000) register_command perfcounters_schema hook 0x37c2050 -44> 2015-04-28 00:34:41.510924 7f2714149900 5 asok(0x3822000) register_command 2 hook 0x37c2050 -43> 2015-04-28 00:34:41.510930 7f2714149900 5 asok(0x3822000) register_command perf schema hook 0x37c2050 -42> 2015-04-28 00:34:41.510940 7f2714149900 5 asok(0x3822000) register_command perf reset hook 0x37c2050 -41> 2015-04-28 00:34:41.510962 7f2714149900 5 asok(0x3822000) register_command config show hook 0x37c2050 -40> 2015-04-28 00:34:41.510973 7f2714149900 5 asok(0x3822000) register_command config set hook 0x37c2050 -39> 2015-04-28 00:34:41.510978 7f2714149900 5 asok(0x3822000) register_command config get hook 0x37c2050 -38> 2015-04-28 00:34:41.510982 7f2714149900 5 asok(0x3822000) register_command config diff hook 0x37c2050 -37> 2015-04-28 00:34:41.510987 7f2714149900 5 asok(0x3822000) register_command log flush hook 0x37c2050 -36> 2015-04-28 00:34:41.510992 7f2714149900 5 asok(0x3822000) register_command log dump hook 0x37c2050 -35> 2015-04-28 00:34:41.510996 7f2714149900 5 asok(0x3822000) register_command log reopen hook 0x37c2050 -34> 2015-04-28 00:34:41.515094 7f2714149900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 4015314 -33> 2015-04-28 00:34:41.516538 7f2714149900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -32> 2015-04-28 00:34:41.516558 7f2714149900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6802/4015314 need_addr=0 -31> 2015-04-28 00:34:41.516585 7f2714149900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -30> 2015-04-28 00:34:41.516594 7f2714149900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6802/4015314 need_addr=0 -29> 2015-04-28 00:34:41.516613 7f2714149900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -28> 2015-04-28 00:34:41.516618 7f2714149900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6803/4015314 need_addr=0 -27> 2015-04-28 00:34:41.516637 7f2714149900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -26> 2015-04-28 00:34:41.516647 7f2714149900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6803/4015314 need_addr=0 -25> 2015-04-28 00:34:41.518994 7f2714149900 5 asok(0x3822000) init /var/run/ceph/ceph-osd.23.asok -24> 2015-04-28 00:34:41.519015 7f2714149900 5 asok(0x3822000) bind_and_listen /var/run/ceph/ceph-osd.23.asok -23> 2015-04-28 00:34:41.519168 7f2714149900 5 asok(0x3822000) register_command 0 hook 0x37be0a8 -22> 2015-04-28 00:34:41.519182 7f2714149900 5 asok(0x3822000) register_command version hook 0x37be0a8 -21> 2015-04-28 00:34:41.519194 7f2714149900 5 asok(0x3822000) register_command git_version hook 0x37be0a8 -20> 2015-04-28 00:34:41.519201 7f2714149900 5 asok(0x3822000) register_command help hook 0x37c2150 -19> 2015-04-28 00:34:41.519207 7f2714149900 5 asok(0x3822000) register_command get_command_descriptions hook 0x37c2140 -18> 2015-04-28 00:34:41.519281 7f2714149900 10 monclient(hunting): build_initial_monmap -17> 2015-04-28 00:34:41.519335 7f270e254700 5 asok(0x3822000) entry start -16> 2015-04-28 00:34:41.535533 7f2714149900 5 adding auth protocol: cephx -15> 2015-04-28 00:34:41.535546 7f2714149900 5 adding auth protocol: cephx -14> 2015-04-28 00:34:41.535693 7f2714149900 5 asok(0x3822000) register_command objecter_requests hook 0x37c2180 -13> 2015-04-28 00:34:41.535796 7f2714149900 1 -- 192.168.30.73:6802/4015314 messenger.start -12> 2015-04-28 00:34:41.535855 7f2714149900 1 -- :/0 messenger.start -11> 2015-04-28 00:34:41.535901 7f2714149900 1 -- 192.168.30.73:6803/4015314 messenger.start -10> 2015-04-28 00:34:41.535952 7f2714149900 1 -- 192.168.32.73:6803/4015314 messenger.start -9> 2015-04-28 00:34:41.535992 7f2714149900 1 -- 192.168.32.73:6802/4015314 messenger.start -8> 2015-04-28 00:34:41.536034 7f2714149900 1 -- :/0 messenger.start -7> 2015-04-28 00:34:41.536136 7f2714149900 2 osd.23 0 mounting /var/lib/ceph/osd/ceph-23 /var/lib/ceph/osd/ceph-23/journal -6> 2015-04-28 00:34:41.536222 7f2714149900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. -5> 2015-04-28 00:34:41.536267 7f2714149900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) -4> 2015-04-28 00:34:41.537766 7f2714149900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work -3> 2015-04-28 00:34:41.537781 7f2714149900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option -2> 2015-04-28 00:34:41.544645 7f2714149900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) -1> 2015-04-28 00:34:41.544827 7f2714149900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.9-031809-generic >= 3.5 0> 2015-04-28 00:34:41.554116 7f2714149900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f2714149900 time 2015-04-28 00:34:41.549285 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f271128cec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 00:34:41.560957 7f2714149900 -1 *** Caught signal (Aborted) ** in thread 7f2714149900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f2712e02340] 3: (gsignal()+0x39) [0x7f27112a1cc9] 4: (abort()+0x148) [0x7f27112a50d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f2711bac6b5] 6: (()+0x5e836) [0x7f2711baa836] 7: (()+0x5e863) [0x7f2711baa863] 8: (()+0x5eaa2) [0x7f2711baaaa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f271128cec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2015-04-28 00:34:41.560957 7f2714149900 -1 *** Caught signal (Aborted) ** in thread 7f2714149900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f2712e02340] 3: (gsignal()+0x39) [0x7f27112a1cc9] 4: (abort()+0x148) [0x7f27112a50d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f2711bac6b5] 6: (()+0x5e836) [0x7f2711baa836] 7: (()+0x5e863) [0x7f2711baa863] 8: (()+0x5eaa2) [0x7f2711baaaa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f271128cec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 00:43:24.134484 7f3dab82d900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 4035276 2015-04-28 00:43:24.155934 7f3dab82d900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. 2015-04-28 00:43:24.155982 7f3dab82d900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) 2015-04-28 00:43:24.157604 7f3dab82d900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work 2015-04-28 00:43:24.157627 7f3dab82d900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-04-28 00:43:24.164636 7f3dab82d900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2015-04-28 00:43:24.164797 7f3dab82d900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.9-031809-generic >= 3.5 2015-04-28 00:43:24.174135 7f3dab82d900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f3dab82d900 time 2015-04-28 00:43:24.169311 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f3da8970ec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -48> 2015-04-28 00:43:24.130169 7f3dab82d900 5 asok(0x46b8000) register_command perfcounters_dump hook 0x4658050 -47> 2015-04-28 00:43:24.130241 7f3dab82d900 5 asok(0x46b8000) register_command 1 hook 0x4658050 -46> 2015-04-28 00:43:24.130248 7f3dab82d900 5 asok(0x46b8000) register_command perf dump hook 0x4658050 -45> 2015-04-28 00:43:24.130283 7f3dab82d900 5 asok(0x46b8000) register_command perfcounters_schema hook 0x4658050 -44> 2015-04-28 00:43:24.130294 7f3dab82d900 5 asok(0x46b8000) register_command 2 hook 0x4658050 -43> 2015-04-28 00:43:24.130299 7f3dab82d900 5 asok(0x46b8000) register_command perf schema hook 0x4658050 -42> 2015-04-28 00:43:24.130325 7f3dab82d900 5 asok(0x46b8000) register_command perf reset hook 0x4658050 -41> 2015-04-28 00:43:24.130365 7f3dab82d900 5 asok(0x46b8000) register_command config show hook 0x4658050 -40> 2015-04-28 00:43:24.130379 7f3dab82d900 5 asok(0x46b8000) register_command config set hook 0x4658050 -39> 2015-04-28 00:43:24.130388 7f3dab82d900 5 asok(0x46b8000) register_command config get hook 0x4658050 -38> 2015-04-28 00:43:24.130396 7f3dab82d900 5 asok(0x46b8000) register_command config diff hook 0x4658050 -37> 2015-04-28 00:43:24.130404 7f3dab82d900 5 asok(0x46b8000) register_command log flush hook 0x4658050 -36> 2015-04-28 00:43:24.130413 7f3dab82d900 5 asok(0x46b8000) register_command log dump hook 0x4658050 -35> 2015-04-28 00:43:24.130421 7f3dab82d900 5 asok(0x46b8000) register_command log reopen hook 0x4658050 -34> 2015-04-28 00:43:24.134484 7f3dab82d900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 4035276 -33> 2015-04-28 00:43:24.135965 7f3dab82d900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -32> 2015-04-28 00:43:24.135986 7f3dab82d900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6804/4035276 need_addr=0 -31> 2015-04-28 00:43:24.136035 7f3dab82d900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -30> 2015-04-28 00:43:24.136045 7f3dab82d900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6804/4035276 need_addr=0 -29> 2015-04-28 00:43:24.136066 7f3dab82d900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -28> 2015-04-28 00:43:24.136073 7f3dab82d900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6805/4035276 need_addr=0 -27> 2015-04-28 00:43:24.136093 7f3dab82d900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -26> 2015-04-28 00:43:24.136103 7f3dab82d900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6805/4035276 need_addr=0 -25> 2015-04-28 00:43:24.138498 7f3dab82d900 5 asok(0x46b8000) init /var/run/ceph/ceph-osd.23.asok -24> 2015-04-28 00:43:24.138523 7f3dab82d900 5 asok(0x46b8000) bind_and_listen /var/run/ceph/ceph-osd.23.asok -23> 2015-04-28 00:43:24.138693 7f3dab82d900 5 asok(0x46b8000) register_command 0 hook 0x46540a8 -22> 2015-04-28 00:43:24.138709 7f3dab82d900 5 asok(0x46b8000) register_command version hook 0x46540a8 -21> 2015-04-28 00:43:24.138714 7f3dab82d900 5 asok(0x46b8000) register_command git_version hook 0x46540a8 -20> 2015-04-28 00:43:24.138729 7f3dab82d900 5 asok(0x46b8000) register_command help hook 0x4658150 -19> 2015-04-28 00:43:24.138739 7f3dab82d900 5 asok(0x46b8000) register_command get_command_descriptions hook 0x4658140 -18> 2015-04-28 00:43:24.138811 7f3dab82d900 10 monclient(hunting): build_initial_monmap -17> 2015-04-28 00:43:24.138841 7f3da5938700 5 asok(0x46b8000) entry start -16> 2015-04-28 00:43:24.155268 7f3dab82d900 5 adding auth protocol: cephx -15> 2015-04-28 00:43:24.155288 7f3dab82d900 5 adding auth protocol: cephx -14> 2015-04-28 00:43:24.155455 7f3dab82d900 5 asok(0x46b8000) register_command objecter_requests hook 0x4658180 -13> 2015-04-28 00:43:24.155574 7f3dab82d900 1 -- 192.168.30.73:6804/4035276 messenger.start -12> 2015-04-28 00:43:24.155632 7f3dab82d900 1 -- :/0 messenger.start -11> 2015-04-28 00:43:24.155670 7f3dab82d900 1 -- 192.168.30.73:6805/4035276 messenger.start -10> 2015-04-28 00:43:24.155707 7f3dab82d900 1 -- 192.168.32.73:6805/4035276 messenger.start -9> 2015-04-28 00:43:24.155743 7f3dab82d900 1 -- 192.168.32.73:6804/4035276 messenger.start -8> 2015-04-28 00:43:24.155779 7f3dab82d900 1 -- :/0 messenger.start -7> 2015-04-28 00:43:24.155862 7f3dab82d900 2 osd.23 0 mounting /var/lib/ceph/osd/ceph-23 /var/lib/ceph/osd/ceph-23/journal -6> 2015-04-28 00:43:24.155934 7f3dab82d900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. -5> 2015-04-28 00:43:24.155982 7f3dab82d900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) -4> 2015-04-28 00:43:24.157604 7f3dab82d900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work -3> 2015-04-28 00:43:24.157627 7f3dab82d900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option -2> 2015-04-28 00:43:24.164636 7f3dab82d900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) -1> 2015-04-28 00:43:24.164797 7f3dab82d900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.9-031809-generic >= 3.5 0> 2015-04-28 00:43:24.174135 7f3dab82d900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f3dab82d900 time 2015-04-28 00:43:24.169311 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f3da8970ec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 00:43:24.181029 7f3dab82d900 -1 *** Caught signal (Aborted) ** in thread 7f3dab82d900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f3daa4e6340] 3: (gsignal()+0x39) [0x7f3da8985cc9] 4: (abort()+0x148) [0x7f3da89890d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f3da92906b5] 6: (()+0x5e836) [0x7f3da928e836] 7: (()+0x5e863) [0x7f3da928e863] 8: (()+0x5eaa2) [0x7f3da928eaa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f3da8970ec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2015-04-28 00:43:24.181029 7f3dab82d900 -1 *** Caught signal (Aborted) ** in thread 7f3dab82d900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f3daa4e6340] 3: (gsignal()+0x39) [0x7f3da8985cc9] 4: (abort()+0x148) [0x7f3da89890d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f3da92906b5] 6: (()+0x5e836) [0x7f3da928e836] 7: (()+0x5e863) [0x7f3da928e863] 8: (()+0x5eaa2) [0x7f3da928eaa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f3da8970ec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 00:43:24.886597 7f357cb13900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 4035826 2015-04-28 00:43:24.903834 7f357cb13900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. 2015-04-28 00:43:24.903873 7f357cb13900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) 2015-04-28 00:43:24.905149 7f357cb13900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work 2015-04-28 00:43:24.905166 7f357cb13900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-04-28 00:43:24.911827 7f357cb13900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2015-04-28 00:43:24.911992 7f357cb13900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.9-031809-generic >= 3.5 2015-04-28 00:43:24.919605 7f357cb13900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f357cb13900 time 2015-04-28 00:43:24.915726 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f3579c56ec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -48> 2015-04-28 00:43:24.883207 7f357cb13900 5 asok(0x5790000) register_command perfcounters_dump hook 0x5730050 -47> 2015-04-28 00:43:24.883251 7f357cb13900 5 asok(0x5790000) register_command 1 hook 0x5730050 -46> 2015-04-28 00:43:24.883258 7f357cb13900 5 asok(0x5790000) register_command perf dump hook 0x5730050 -45> 2015-04-28 00:43:24.883270 7f357cb13900 5 asok(0x5790000) register_command perfcounters_schema hook 0x5730050 -44> 2015-04-28 00:43:24.883286 7f357cb13900 5 asok(0x5790000) register_command 2 hook 0x5730050 -43> 2015-04-28 00:43:24.883290 7f357cb13900 5 asok(0x5790000) register_command perf schema hook 0x5730050 -42> 2015-04-28 00:43:24.883295 7f357cb13900 5 asok(0x5790000) register_command perf reset hook 0x5730050 -41> 2015-04-28 00:43:24.883305 7f357cb13900 5 asok(0x5790000) register_command config show hook 0x5730050 -40> 2015-04-28 00:43:24.883310 7f357cb13900 5 asok(0x5790000) register_command config set hook 0x5730050 -39> 2015-04-28 00:43:24.883314 7f357cb13900 5 asok(0x5790000) register_command config get hook 0x5730050 -38> 2015-04-28 00:43:24.883318 7f357cb13900 5 asok(0x5790000) register_command config diff hook 0x5730050 -37> 2015-04-28 00:43:24.883322 7f357cb13900 5 asok(0x5790000) register_command log flush hook 0x5730050 -36> 2015-04-28 00:43:24.883327 7f357cb13900 5 asok(0x5790000) register_command log dump hook 0x5730050 -35> 2015-04-28 00:43:24.883339 7f357cb13900 5 asok(0x5790000) register_command log reopen hook 0x5730050 -34> 2015-04-28 00:43:24.886597 7f357cb13900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 4035826 -33> 2015-04-28 00:43:24.887804 7f357cb13900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -32> 2015-04-28 00:43:24.887822 7f357cb13900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6800/4035826 need_addr=0 -31> 2015-04-28 00:43:24.887843 7f357cb13900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -30> 2015-04-28 00:43:24.887848 7f357cb13900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6800/4035826 need_addr=0 -29> 2015-04-28 00:43:24.887861 7f357cb13900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -28> 2015-04-28 00:43:24.887882 7f357cb13900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6801/4035826 need_addr=0 -27> 2015-04-28 00:43:24.887897 7f357cb13900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -26> 2015-04-28 00:43:24.887905 7f357cb13900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6801/4035826 need_addr=0 -25> 2015-04-28 00:43:24.889748 7f357cb13900 5 asok(0x5790000) init /var/run/ceph/ceph-osd.23.asok -24> 2015-04-28 00:43:24.889768 7f357cb13900 5 asok(0x5790000) bind_and_listen /var/run/ceph/ceph-osd.23.asok -23> 2015-04-28 00:43:24.889873 7f357cb13900 5 asok(0x5790000) register_command 0 hook 0x572c0a8 -22> 2015-04-28 00:43:24.889885 7f357cb13900 5 asok(0x5790000) register_command version hook 0x572c0a8 -21> 2015-04-28 00:43:24.889889 7f357cb13900 5 asok(0x5790000) register_command git_version hook 0x572c0a8 -20> 2015-04-28 00:43:24.889895 7f357cb13900 5 asok(0x5790000) register_command help hook 0x5730150 -19> 2015-04-28 00:43:24.889899 7f357cb13900 5 asok(0x5790000) register_command get_command_descriptions hook 0x5730140 -18> 2015-04-28 00:43:24.889955 7f357cb13900 10 monclient(hunting): build_initial_monmap -17> 2015-04-28 00:43:24.890173 7f3576c1e700 5 asok(0x5790000) entry start -16> 2015-04-28 00:43:24.903235 7f357cb13900 5 adding auth protocol: cephx -15> 2015-04-28 00:43:24.903251 7f357cb13900 5 adding auth protocol: cephx -14> 2015-04-28 00:43:24.903391 7f357cb13900 5 asok(0x5790000) register_command objecter_requests hook 0x5730180 -13> 2015-04-28 00:43:24.903496 7f357cb13900 1 -- 192.168.30.73:6800/4035826 messenger.start -12> 2015-04-28 00:43:24.903546 7f357cb13900 1 -- :/0 messenger.start -11> 2015-04-28 00:43:24.903579 7f357cb13900 1 -- 192.168.30.73:6801/4035826 messenger.start -10> 2015-04-28 00:43:24.903621 7f357cb13900 1 -- 192.168.32.73:6801/4035826 messenger.start -9> 2015-04-28 00:43:24.903673 7f357cb13900 1 -- 192.168.32.73:6800/4035826 messenger.start -8> 2015-04-28 00:43:24.903706 7f357cb13900 1 -- :/0 messenger.start -7> 2015-04-28 00:43:24.903778 7f357cb13900 2 osd.23 0 mounting /var/lib/ceph/osd/ceph-23 /var/lib/ceph/osd/ceph-23/journal -6> 2015-04-28 00:43:24.903834 7f357cb13900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. -5> 2015-04-28 00:43:24.903873 7f357cb13900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) -4> 2015-04-28 00:43:24.905149 7f357cb13900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work -3> 2015-04-28 00:43:24.905166 7f357cb13900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option -2> 2015-04-28 00:43:24.911827 7f357cb13900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) -1> 2015-04-28 00:43:24.911992 7f357cb13900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.9-031809-generic >= 3.5 0> 2015-04-28 00:43:24.919605 7f357cb13900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f357cb13900 time 2015-04-28 00:43:24.915726 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f3579c56ec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 00:43:24.925054 7f357cb13900 -1 *** Caught signal (Aborted) ** in thread 7f357cb13900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f357b7cc340] 3: (gsignal()+0x39) [0x7f3579c6bcc9] 4: (abort()+0x148) [0x7f3579c6f0d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f357a5766b5] 6: (()+0x5e836) [0x7f357a574836] 7: (()+0x5e863) [0x7f357a574863] 8: (()+0x5eaa2) [0x7f357a574aa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f3579c56ec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2015-04-28 00:43:24.925054 7f357cb13900 -1 *** Caught signal (Aborted) ** in thread 7f357cb13900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f357b7cc340] 3: (gsignal()+0x39) [0x7f3579c6bcc9] 4: (abort()+0x148) [0x7f3579c6f0d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f357a5766b5] 6: (()+0x5e836) [0x7f357a574836] 7: (()+0x5e863) [0x7f357a574863] 8: (()+0x5eaa2) [0x7f357a574aa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f3579c56ec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 00:43:25.609076 7f81d4b2a900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 4036419 2015-04-28 00:43:25.630527 7f81d4b2a900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. 2015-04-28 00:43:25.630574 7f81d4b2a900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) 2015-04-28 00:43:25.632067 7f81d4b2a900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work 2015-04-28 00:43:25.632089 7f81d4b2a900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-04-28 00:43:25.639411 7f81d4b2a900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2015-04-28 00:43:25.639589 7f81d4b2a900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.9-031809-generic >= 3.5 2015-04-28 00:43:25.649353 7f81d4b2a900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f81d4b2a900 time 2015-04-28 00:43:25.644457 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f81d1c6dec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -48> 2015-04-28 00:43:25.604832 7f81d4b2a900 5 asok(0x472a000) register_command perfcounters_dump hook 0x46ca050 -47> 2015-04-28 00:43:25.604899 7f81d4b2a900 5 asok(0x472a000) register_command 1 hook 0x46ca050 -46> 2015-04-28 00:43:25.604910 7f81d4b2a900 5 asok(0x472a000) register_command perf dump hook 0x46ca050 -45> 2015-04-28 00:43:25.604931 7f81d4b2a900 5 asok(0x472a000) register_command perfcounters_schema hook 0x46ca050 -44> 2015-04-28 00:43:25.604940 7f81d4b2a900 5 asok(0x472a000) register_command 2 hook 0x46ca050 -43> 2015-04-28 00:43:25.604953 7f81d4b2a900 5 asok(0x472a000) register_command perf schema hook 0x46ca050 -42> 2015-04-28 00:43:25.604961 7f81d4b2a900 5 asok(0x472a000) register_command perf reset hook 0x46ca050 -41> 2015-04-28 00:43:25.604991 7f81d4b2a900 5 asok(0x472a000) register_command config show hook 0x46ca050 -40> 2015-04-28 00:43:25.605001 7f81d4b2a900 5 asok(0x472a000) register_command config set hook 0x46ca050 -39> 2015-04-28 00:43:25.605010 7f81d4b2a900 5 asok(0x472a000) register_command config get hook 0x46ca050 -38> 2015-04-28 00:43:25.605018 7f81d4b2a900 5 asok(0x472a000) register_command config diff hook 0x46ca050 -37> 2015-04-28 00:43:25.605026 7f81d4b2a900 5 asok(0x472a000) register_command log flush hook 0x46ca050 -36> 2015-04-28 00:43:25.605035 7f81d4b2a900 5 asok(0x472a000) register_command log dump hook 0x46ca050 -35> 2015-04-28 00:43:25.605041 7f81d4b2a900 5 asok(0x472a000) register_command log reopen hook 0x46ca050 -34> 2015-04-28 00:43:25.609076 7f81d4b2a900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 4036419 -33> 2015-04-28 00:43:25.610618 7f81d4b2a900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -32> 2015-04-28 00:43:25.610640 7f81d4b2a900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6800/4036419 need_addr=0 -31> 2015-04-28 00:43:25.610666 7f81d4b2a900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -30> 2015-04-28 00:43:25.610673 7f81d4b2a900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6800/4036419 need_addr=0 -29> 2015-04-28 00:43:25.610690 7f81d4b2a900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -28> 2015-04-28 00:43:25.610708 7f81d4b2a900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6801/4036419 need_addr=0 -27> 2015-04-28 00:43:25.610727 7f81d4b2a900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -26> 2015-04-28 00:43:25.610736 7f81d4b2a900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6801/4036419 need_addr=0 -25> 2015-04-28 00:43:25.613018 7f81d4b2a900 5 asok(0x472a000) init /var/run/ceph/ceph-osd.23.asok -24> 2015-04-28 00:43:25.613041 7f81d4b2a900 5 asok(0x472a000) bind_and_listen /var/run/ceph/ceph-osd.23.asok -23> 2015-04-28 00:43:25.613202 7f81d4b2a900 5 asok(0x472a000) register_command 0 hook 0x46c60a8 -22> 2015-04-28 00:43:25.613217 7f81d4b2a900 5 asok(0x472a000) register_command version hook 0x46c60a8 -21> 2015-04-28 00:43:25.613223 7f81d4b2a900 5 asok(0x472a000) register_command git_version hook 0x46c60a8 -20> 2015-04-28 00:43:25.613230 7f81d4b2a900 5 asok(0x472a000) register_command help hook 0x46ca150 -19> 2015-04-28 00:43:25.613235 7f81d4b2a900 5 asok(0x472a000) register_command get_command_descriptions hook 0x46ca140 -18> 2015-04-28 00:43:25.613302 7f81d4b2a900 10 monclient(hunting): build_initial_monmap -17> 2015-04-28 00:43:25.613326 7f81cec35700 5 asok(0x472a000) entry start -16> 2015-04-28 00:43:25.629819 7f81d4b2a900 5 adding auth protocol: cephx -15> 2015-04-28 00:43:25.629839 7f81d4b2a900 5 adding auth protocol: cephx -14> 2015-04-28 00:43:25.630007 7f81d4b2a900 5 asok(0x472a000) register_command objecter_requests hook 0x46ca180 -13> 2015-04-28 00:43:25.630119 7f81d4b2a900 1 -- 192.168.30.73:6800/4036419 messenger.start -12> 2015-04-28 00:43:25.630179 7f81d4b2a900 1 -- :/0 messenger.start -11> 2015-04-28 00:43:25.630222 7f81d4b2a900 1 -- 192.168.30.73:6801/4036419 messenger.start -10> 2015-04-28 00:43:25.630263 7f81d4b2a900 1 -- 192.168.32.73:6801/4036419 messenger.start -9> 2015-04-28 00:43:25.630303 7f81d4b2a900 1 -- 192.168.32.73:6800/4036419 messenger.start -8> 2015-04-28 00:43:25.630362 7f81d4b2a900 1 -- :/0 messenger.start -7> 2015-04-28 00:43:25.630456 7f81d4b2a900 2 osd.23 0 mounting /var/lib/ceph/osd/ceph-23 /var/lib/ceph/osd/ceph-23/journal -6> 2015-04-28 00:43:25.630527 7f81d4b2a900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. -5> 2015-04-28 00:43:25.630574 7f81d4b2a900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) -4> 2015-04-28 00:43:25.632067 7f81d4b2a900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work -3> 2015-04-28 00:43:25.632089 7f81d4b2a900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option -2> 2015-04-28 00:43:25.639411 7f81d4b2a900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) -1> 2015-04-28 00:43:25.639589 7f81d4b2a900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.9-031809-generic >= 3.5 0> 2015-04-28 00:43:25.649353 7f81d4b2a900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f81d4b2a900 time 2015-04-28 00:43:25.644457 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f81d1c6dec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 00:43:25.656294 7f81d4b2a900 -1 *** Caught signal (Aborted) ** in thread 7f81d4b2a900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f81d37e3340] 3: (gsignal()+0x39) [0x7f81d1c82cc9] 4: (abort()+0x148) [0x7f81d1c860d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f81d258d6b5] 6: (()+0x5e836) [0x7f81d258b836] 7: (()+0x5e863) [0x7f81d258b863] 8: (()+0x5eaa2) [0x7f81d258baa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f81d1c6dec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2015-04-28 00:43:25.656294 7f81d4b2a900 -1 *** Caught signal (Aborted) ** in thread 7f81d4b2a900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f81d37e3340] 3: (gsignal()+0x39) [0x7f81d1c82cc9] 4: (abort()+0x148) [0x7f81d1c860d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f81d258d6b5] 6: (()+0x5e836) [0x7f81d258b836] 7: (()+0x5e863) [0x7f81d258b863] 8: (()+0x5eaa2) [0x7f81d258baa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f81d1c6dec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 00:43:26.348919 7fbc8d20e900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 4036923 2015-04-28 00:43:26.370502 7fbc8d20e900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. 2015-04-28 00:43:26.370551 7fbc8d20e900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) 2015-04-28 00:43:26.372166 7fbc8d20e900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work 2015-04-28 00:43:26.372183 7fbc8d20e900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-04-28 00:43:26.379522 7fbc8d20e900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2015-04-28 00:43:26.379687 7fbc8d20e900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.9-031809-generic >= 3.5 2015-04-28 00:43:26.388974 7fbc8d20e900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7fbc8d20e900 time 2015-04-28 00:43:26.384094 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7fbc8a351ec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -48> 2015-04-28 00:43:26.344408 7fbc8d20e900 5 asok(0x4dd6000) register_command perfcounters_dump hook 0x4d76050 -47> 2015-04-28 00:43:26.344484 7fbc8d20e900 5 asok(0x4dd6000) register_command 1 hook 0x4d76050 -46> 2015-04-28 00:43:26.344500 7fbc8d20e900 5 asok(0x4dd6000) register_command perf dump hook 0x4d76050 -45> 2015-04-28 00:43:26.344522 7fbc8d20e900 5 asok(0x4dd6000) register_command perfcounters_schema hook 0x4d76050 -44> 2015-04-28 00:43:26.344535 7fbc8d20e900 5 asok(0x4dd6000) register_command 2 hook 0x4d76050 -43> 2015-04-28 00:43:26.344546 7fbc8d20e900 5 asok(0x4dd6000) register_command perf schema hook 0x4d76050 -42> 2015-04-28 00:43:26.344558 7fbc8d20e900 5 asok(0x4dd6000) register_command perf reset hook 0x4d76050 -41> 2015-04-28 00:43:26.344578 7fbc8d20e900 5 asok(0x4dd6000) register_command config show hook 0x4d76050 -40> 2015-04-28 00:43:26.344592 7fbc8d20e900 5 asok(0x4dd6000) register_command config set hook 0x4d76050 -39> 2015-04-28 00:43:26.344604 7fbc8d20e900 5 asok(0x4dd6000) register_command config get hook 0x4d76050 -38> 2015-04-28 00:43:26.344615 7fbc8d20e900 5 asok(0x4dd6000) register_command config diff hook 0x4d76050 -37> 2015-04-28 00:43:26.344627 7fbc8d20e900 5 asok(0x4dd6000) register_command log flush hook 0x4d76050 -36> 2015-04-28 00:43:26.344639 7fbc8d20e900 5 asok(0x4dd6000) register_command log dump hook 0x4d76050 -35> 2015-04-28 00:43:26.344651 7fbc8d20e900 5 asok(0x4dd6000) register_command log reopen hook 0x4d76050 -34> 2015-04-28 00:43:26.348919 7fbc8d20e900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 4036923 -33> 2015-04-28 00:43:26.350479 7fbc8d20e900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -32> 2015-04-28 00:43:26.350502 7fbc8d20e900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6800/4036923 need_addr=0 -31> 2015-04-28 00:43:26.350527 7fbc8d20e900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -30> 2015-04-28 00:43:26.350538 7fbc8d20e900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6800/4036923 need_addr=0 -29> 2015-04-28 00:43:26.350556 7fbc8d20e900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -28> 2015-04-28 00:43:26.350565 7fbc8d20e900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6801/4036923 need_addr=0 -27> 2015-04-28 00:43:26.350582 7fbc8d20e900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -26> 2015-04-28 00:43:26.350592 7fbc8d20e900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6801/4036923 need_addr=0 -25> 2015-04-28 00:43:26.352901 7fbc8d20e900 5 asok(0x4dd6000) init /var/run/ceph/ceph-osd.23.asok -24> 2015-04-28 00:43:26.352924 7fbc8d20e900 5 asok(0x4dd6000) bind_and_listen /var/run/ceph/ceph-osd.23.asok -23> 2015-04-28 00:43:26.353091 7fbc8d20e900 5 asok(0x4dd6000) register_command 0 hook 0x4d720a8 -22> 2015-04-28 00:43:26.353107 7fbc8d20e900 5 asok(0x4dd6000) register_command version hook 0x4d720a8 -21> 2015-04-28 00:43:26.353113 7fbc8d20e900 5 asok(0x4dd6000) register_command git_version hook 0x4d720a8 -20> 2015-04-28 00:43:26.353120 7fbc8d20e900 5 asok(0x4dd6000) register_command help hook 0x4d76150 -19> 2015-04-28 00:43:26.353125 7fbc8d20e900 5 asok(0x4dd6000) register_command get_command_descriptions hook 0x4d76140 -18> 2015-04-28 00:43:26.353198 7fbc8d20e900 10 monclient(hunting): build_initial_monmap -17> 2015-04-28 00:43:26.353225 7fbc87319700 5 asok(0x4dd6000) entry start -16> 2015-04-28 00:43:26.369768 7fbc8d20e900 5 adding auth protocol: cephx -15> 2015-04-28 00:43:26.369791 7fbc8d20e900 5 adding auth protocol: cephx -14> 2015-04-28 00:43:26.369964 7fbc8d20e900 5 asok(0x4dd6000) register_command objecter_requests hook 0x4d76180 -13> 2015-04-28 00:43:26.370093 7fbc8d20e900 1 -- 192.168.30.73:6800/4036923 messenger.start -12> 2015-04-28 00:43:26.370156 7fbc8d20e900 1 -- :/0 messenger.start -11> 2015-04-28 00:43:26.370198 7fbc8d20e900 1 -- 192.168.30.73:6801/4036923 messenger.start -10> 2015-04-28 00:43:26.370238 7fbc8d20e900 1 -- 192.168.32.73:6801/4036923 messenger.start -9> 2015-04-28 00:43:26.370276 7fbc8d20e900 1 -- 192.168.32.73:6800/4036923 messenger.start -8> 2015-04-28 00:43:26.370326 7fbc8d20e900 1 -- :/0 messenger.start -7> 2015-04-28 00:43:26.370426 7fbc8d20e900 2 osd.23 0 mounting /var/lib/ceph/osd/ceph-23 /var/lib/ceph/osd/ceph-23/journal -6> 2015-04-28 00:43:26.370502 7fbc8d20e900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. -5> 2015-04-28 00:43:26.370551 7fbc8d20e900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) -4> 2015-04-28 00:43:26.372166 7fbc8d20e900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work -3> 2015-04-28 00:43:26.372183 7fbc8d20e900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option -2> 2015-04-28 00:43:26.379522 7fbc8d20e900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) -1> 2015-04-28 00:43:26.379687 7fbc8d20e900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.9-031809-generic >= 3.5 0> 2015-04-28 00:43:26.388974 7fbc8d20e900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7fbc8d20e900 time 2015-04-28 00:43:26.384094 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7fbc8a351ec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 00:43:26.395941 7fbc8d20e900 -1 *** Caught signal (Aborted) ** in thread 7fbc8d20e900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7fbc8bec7340] 3: (gsignal()+0x39) [0x7fbc8a366cc9] 4: (abort()+0x148) [0x7fbc8a36a0d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7fbc8ac716b5] 6: (()+0x5e836) [0x7fbc8ac6f836] 7: (()+0x5e863) [0x7fbc8ac6f863] 8: (()+0x5eaa2) [0x7fbc8ac6faa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7fbc8a351ec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2015-04-28 00:43:26.395941 7fbc8d20e900 -1 *** Caught signal (Aborted) ** in thread 7fbc8d20e900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7fbc8bec7340] 3: (gsignal()+0x39) [0x7fbc8a366cc9] 4: (abort()+0x148) [0x7fbc8a36a0d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7fbc8ac716b5] 6: (()+0x5e836) [0x7fbc8ac6f836] 7: (()+0x5e863) [0x7fbc8ac6f863] 8: (()+0x5eaa2) [0x7fbc8ac6faa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7fbc8a351ec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 00:43:27.091269 7fe8f2e26900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 4037545 2015-04-28 00:43:27.110600 7fe8f2e26900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. 2015-04-28 00:43:27.110645 7fe8f2e26900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) 2015-04-28 00:43:27.112059 7fe8f2e26900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work 2015-04-28 00:43:27.112076 7fe8f2e26900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-04-28 00:43:27.117744 7fe8f2e26900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2015-04-28 00:43:27.117907 7fe8f2e26900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.9-031809-generic >= 3.5 2015-04-28 00:43:27.126425 7fe8f2e26900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7fe8f2e26900 time 2015-04-28 00:43:27.122164 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7fe8eff69ec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -48> 2015-04-28 00:43:27.087454 7fe8f2e26900 5 asok(0x4054000) register_command perfcounters_dump hook 0x3ff4050 -47> 2015-04-28 00:43:27.087523 7fe8f2e26900 5 asok(0x4054000) register_command 1 hook 0x3ff4050 -46> 2015-04-28 00:43:27.087533 7fe8f2e26900 5 asok(0x4054000) register_command perf dump hook 0x3ff4050 -45> 2015-04-28 00:43:27.087543 7fe8f2e26900 5 asok(0x4054000) register_command perfcounters_schema hook 0x3ff4050 -44> 2015-04-28 00:43:27.087551 7fe8f2e26900 5 asok(0x4054000) register_command 2 hook 0x3ff4050 -43> 2015-04-28 00:43:27.087555 7fe8f2e26900 5 asok(0x4054000) register_command perf schema hook 0x3ff4050 -42> 2015-04-28 00:43:27.087561 7fe8f2e26900 5 asok(0x4054000) register_command perf reset hook 0x3ff4050 -41> 2015-04-28 00:43:27.087581 7fe8f2e26900 5 asok(0x4054000) register_command config show hook 0x3ff4050 -40> 2015-04-28 00:43:27.087589 7fe8f2e26900 5 asok(0x4054000) register_command config set hook 0x3ff4050 -39> 2015-04-28 00:43:27.087593 7fe8f2e26900 5 asok(0x4054000) register_command config get hook 0x3ff4050 -38> 2015-04-28 00:43:27.087600 7fe8f2e26900 5 asok(0x4054000) register_command config diff hook 0x3ff4050 -37> 2015-04-28 00:43:27.087605 7fe8f2e26900 5 asok(0x4054000) register_command log flush hook 0x3ff4050 -36> 2015-04-28 00:43:27.087612 7fe8f2e26900 5 asok(0x4054000) register_command log dump hook 0x3ff4050 -35> 2015-04-28 00:43:27.087616 7fe8f2e26900 5 asok(0x4054000) register_command log reopen hook 0x3ff4050 -34> 2015-04-28 00:43:27.091269 7fe8f2e26900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 4037545 -33> 2015-04-28 00:43:27.092622 7fe8f2e26900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -32> 2015-04-28 00:43:27.092641 7fe8f2e26900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6802/4037545 need_addr=0 -31> 2015-04-28 00:43:27.092665 7fe8f2e26900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -30> 2015-04-28 00:43:27.092683 7fe8f2e26900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6802/4037545 need_addr=0 -29> 2015-04-28 00:43:27.092700 7fe8f2e26900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -28> 2015-04-28 00:43:27.092708 7fe8f2e26900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6803/4037545 need_addr=0 -27> 2015-04-28 00:43:27.092723 7fe8f2e26900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -26> 2015-04-28 00:43:27.092729 7fe8f2e26900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6803/4037545 need_addr=0 -25> 2015-04-28 00:43:27.094807 7fe8f2e26900 5 asok(0x4054000) init /var/run/ceph/ceph-osd.23.asok -24> 2015-04-28 00:43:27.094830 7fe8f2e26900 5 asok(0x4054000) bind_and_listen /var/run/ceph/ceph-osd.23.asok -23> 2015-04-28 00:43:27.094983 7fe8f2e26900 5 asok(0x4054000) register_command 0 hook 0x3ff00a8 -22> 2015-04-28 00:43:27.094996 7fe8f2e26900 5 asok(0x4054000) register_command version hook 0x3ff00a8 -21> 2015-04-28 00:43:27.095005 7fe8f2e26900 5 asok(0x4054000) register_command git_version hook 0x3ff00a8 -20> 2015-04-28 00:43:27.095014 7fe8f2e26900 5 asok(0x4054000) register_command help hook 0x3ff4150 -19> 2015-04-28 00:43:27.095022 7fe8f2e26900 5 asok(0x4054000) register_command get_command_descriptions hook 0x3ff4140 -18> 2015-04-28 00:43:27.095089 7fe8f2e26900 10 monclient(hunting): build_initial_monmap -17> 2015-04-28 00:43:27.095086 7fe8ecf31700 5 asok(0x4054000) entry start -16> 2015-04-28 00:43:27.109971 7fe8f2e26900 5 adding auth protocol: cephx -15> 2015-04-28 00:43:27.109988 7fe8f2e26900 5 adding auth protocol: cephx -14> 2015-04-28 00:43:27.110138 7fe8f2e26900 5 asok(0x4054000) register_command objecter_requests hook 0x3ff4180 -13> 2015-04-28 00:43:27.110259 7fe8f2e26900 1 -- 192.168.30.73:6802/4037545 messenger.start -12> 2015-04-28 00:43:27.110318 7fe8f2e26900 1 -- :/0 messenger.start -11> 2015-04-28 00:43:27.110358 7fe8f2e26900 1 -- 192.168.30.73:6803/4037545 messenger.start -10> 2015-04-28 00:43:27.110391 7fe8f2e26900 1 -- 192.168.32.73:6803/4037545 messenger.start -9> 2015-04-28 00:43:27.110424 7fe8f2e26900 1 -- 192.168.32.73:6802/4037545 messenger.start -8> 2015-04-28 00:43:27.110456 7fe8f2e26900 1 -- :/0 messenger.start -7> 2015-04-28 00:43:27.110532 7fe8f2e26900 2 osd.23 0 mounting /var/lib/ceph/osd/ceph-23 /var/lib/ceph/osd/ceph-23/journal -6> 2015-04-28 00:43:27.110600 7fe8f2e26900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. -5> 2015-04-28 00:43:27.110645 7fe8f2e26900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) -4> 2015-04-28 00:43:27.112059 7fe8f2e26900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work -3> 2015-04-28 00:43:27.112076 7fe8f2e26900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option -2> 2015-04-28 00:43:27.117744 7fe8f2e26900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) -1> 2015-04-28 00:43:27.117907 7fe8f2e26900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.9-031809-generic >= 3.5 0> 2015-04-28 00:43:27.126425 7fe8f2e26900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7fe8f2e26900 time 2015-04-28 00:43:27.122164 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7fe8eff69ec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 00:43:27.132558 7fe8f2e26900 -1 *** Caught signal (Aborted) ** in thread 7fe8f2e26900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7fe8f1adf340] 3: (gsignal()+0x39) [0x7fe8eff7ecc9] 4: (abort()+0x148) [0x7fe8eff820d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7fe8f08896b5] 6: (()+0x5e836) [0x7fe8f0887836] 7: (()+0x5e863) [0x7fe8f0887863] 8: (()+0x5eaa2) [0x7fe8f0887aa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7fe8eff69ec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2015-04-28 00:43:27.132558 7fe8f2e26900 -1 *** Caught signal (Aborted) ** in thread 7fe8f2e26900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7fe8f1adf340] 3: (gsignal()+0x39) [0x7fe8eff7ecc9] 4: (abort()+0x148) [0x7fe8eff820d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7fe8f08896b5] 6: (()+0x5e836) [0x7fe8f0887836] 7: (()+0x5e863) [0x7fe8f0887863] 8: (()+0x5eaa2) [0x7fe8f0887aa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7fe8eff69ec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 00:43:27.775761 7fa97b1ba900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 4038045 2015-04-28 00:43:27.797921 7fa97b1ba900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. 2015-04-28 00:43:27.797969 7fa97b1ba900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) 2015-04-28 00:43:27.799528 7fa97b1ba900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work 2015-04-28 00:43:27.799552 7fa97b1ba900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-04-28 00:43:27.805839 7fa97b1ba900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2015-04-28 00:43:27.806017 7fa97b1ba900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.9-031809-generic >= 3.5 2015-04-28 00:43:27.815676 7fa97b1ba900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7fa97b1ba900 time 2015-04-28 00:43:27.810763 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7fa9782fdec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -48> 2015-04-28 00:43:27.771468 7fa97b1ba900 5 asok(0x4c80000) register_command perfcounters_dump hook 0x4c20050 -47> 2015-04-28 00:43:27.771551 7fa97b1ba900 5 asok(0x4c80000) register_command 1 hook 0x4c20050 -46> 2015-04-28 00:43:27.771559 7fa97b1ba900 5 asok(0x4c80000) register_command perf dump hook 0x4c20050 -45> 2015-04-28 00:43:27.771599 7fa97b1ba900 5 asok(0x4c80000) register_command perfcounters_schema hook 0x4c20050 -44> 2015-04-28 00:43:27.771610 7fa97b1ba900 5 asok(0x4c80000) register_command 2 hook 0x4c20050 -43> 2015-04-28 00:43:27.771615 7fa97b1ba900 5 asok(0x4c80000) register_command perf schema hook 0x4c20050 -42> 2015-04-28 00:43:27.771620 7fa97b1ba900 5 asok(0x4c80000) register_command perf reset hook 0x4c20050 -41> 2015-04-28 00:43:27.771639 7fa97b1ba900 5 asok(0x4c80000) register_command config show hook 0x4c20050 -40> 2015-04-28 00:43:27.771649 7fa97b1ba900 5 asok(0x4c80000) register_command config set hook 0x4c20050 -39> 2015-04-28 00:43:27.771654 7fa97b1ba900 5 asok(0x4c80000) register_command config get hook 0x4c20050 -38> 2015-04-28 00:43:27.771659 7fa97b1ba900 5 asok(0x4c80000) register_command config diff hook 0x4c20050 -37> 2015-04-28 00:43:27.771663 7fa97b1ba900 5 asok(0x4c80000) register_command log flush hook 0x4c20050 -36> 2015-04-28 00:43:27.771673 7fa97b1ba900 5 asok(0x4c80000) register_command log dump hook 0x4c20050 -35> 2015-04-28 00:43:27.771679 7fa97b1ba900 5 asok(0x4c80000) register_command log reopen hook 0x4c20050 -34> 2015-04-28 00:43:27.775761 7fa97b1ba900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 4038045 -33> 2015-04-28 00:43:27.777268 7fa97b1ba900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -32> 2015-04-28 00:43:27.777290 7fa97b1ba900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6800/4038045 need_addr=0 -31> 2015-04-28 00:43:27.777314 7fa97b1ba900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -30> 2015-04-28 00:43:27.777325 7fa97b1ba900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6800/4038045 need_addr=0 -29> 2015-04-28 00:43:27.777342 7fa97b1ba900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -28> 2015-04-28 00:43:27.777352 7fa97b1ba900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6801/4038045 need_addr=0 -27> 2015-04-28 00:43:27.777371 7fa97b1ba900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -26> 2015-04-28 00:43:27.777380 7fa97b1ba900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6801/4038045 need_addr=0 -25> 2015-04-28 00:43:27.779831 7fa97b1ba900 5 asok(0x4c80000) init /var/run/ceph/ceph-osd.23.asok -24> 2015-04-28 00:43:27.779855 7fa97b1ba900 5 asok(0x4c80000) bind_and_listen /var/run/ceph/ceph-osd.23.asok -23> 2015-04-28 00:43:27.780029 7fa97b1ba900 5 asok(0x4c80000) register_command 0 hook 0x4c1c0a8 -22> 2015-04-28 00:43:27.780043 7fa97b1ba900 5 asok(0x4c80000) register_command version hook 0x4c1c0a8 -21> 2015-04-28 00:43:27.780048 7fa97b1ba900 5 asok(0x4c80000) register_command git_version hook 0x4c1c0a8 -20> 2015-04-28 00:43:27.780054 7fa97b1ba900 5 asok(0x4c80000) register_command help hook 0x4c20150 -19> 2015-04-28 00:43:27.780060 7fa97b1ba900 5 asok(0x4c80000) register_command get_command_descriptions hook 0x4c20140 -18> 2015-04-28 00:43:27.780129 7fa97b1ba900 10 monclient(hunting): build_initial_monmap -17> 2015-04-28 00:43:27.780164 7fa9752c5700 5 asok(0x4c80000) entry start -16> 2015-04-28 00:43:27.797200 7fa97b1ba900 5 adding auth protocol: cephx -15> 2015-04-28 00:43:27.797220 7fa97b1ba900 5 adding auth protocol: cephx -14> 2015-04-28 00:43:27.797390 7fa97b1ba900 5 asok(0x4c80000) register_command objecter_requests hook 0x4c20180 -13> 2015-04-28 00:43:27.797506 7fa97b1ba900 1 -- 192.168.30.73:6800/4038045 messenger.start -12> 2015-04-28 00:43:27.797566 7fa97b1ba900 1 -- :/0 messenger.start -11> 2015-04-28 00:43:27.797623 7fa97b1ba900 1 -- 192.168.30.73:6801/4038045 messenger.start -10> 2015-04-28 00:43:27.797667 7fa97b1ba900 1 -- 192.168.32.73:6801/4038045 messenger.start -9> 2015-04-28 00:43:27.797706 7fa97b1ba900 1 -- 192.168.32.73:6800/4038045 messenger.start -8> 2015-04-28 00:43:27.797752 7fa97b1ba900 1 -- :/0 messenger.start -7> 2015-04-28 00:43:27.797847 7fa97b1ba900 2 osd.23 0 mounting /var/lib/ceph/osd/ceph-23 /var/lib/ceph/osd/ceph-23/journal -6> 2015-04-28 00:43:27.797921 7fa97b1ba900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. -5> 2015-04-28 00:43:27.797969 7fa97b1ba900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) -4> 2015-04-28 00:43:27.799528 7fa97b1ba900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work -3> 2015-04-28 00:43:27.799552 7fa97b1ba900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option -2> 2015-04-28 00:43:27.805839 7fa97b1ba900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) -1> 2015-04-28 00:43:27.806017 7fa97b1ba900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.9-031809-generic >= 3.5 0> 2015-04-28 00:43:27.815676 7fa97b1ba900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7fa97b1ba900 time 2015-04-28 00:43:27.810763 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7fa9782fdec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 00:43:27.822604 7fa97b1ba900 -1 *** Caught signal (Aborted) ** in thread 7fa97b1ba900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7fa979e73340] 3: (gsignal()+0x39) [0x7fa978312cc9] 4: (abort()+0x148) [0x7fa9783160d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7fa978c1d6b5] 6: (()+0x5e836) [0x7fa978c1b836] 7: (()+0x5e863) [0x7fa978c1b863] 8: (()+0x5eaa2) [0x7fa978c1baa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7fa9782fdec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2015-04-28 00:43:27.822604 7fa97b1ba900 -1 *** Caught signal (Aborted) ** in thread 7fa97b1ba900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7fa979e73340] 3: (gsignal()+0x39) [0x7fa978312cc9] 4: (abort()+0x148) [0x7fa9783160d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7fa978c1d6b5] 6: (()+0x5e836) [0x7fa978c1b836] 7: (()+0x5e863) [0x7fa978c1b863] 8: (()+0x5eaa2) [0x7fa978c1baa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7fa9782fdec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 00:48:00.883340 7fb37ebb9900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 4556 2015-04-28 00:48:00.909794 7fb37ebb9900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. 2015-04-28 00:48:00.910034 7fb37ebb9900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) 2015-04-28 00:48:00.939633 7fb37ebb9900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work 2015-04-28 00:48:00.939664 7fb37ebb9900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-04-28 00:48:00.940325 7fb37ebb9900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2015-04-28 00:48:00.943774 7fb37ebb9900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.9-031809-generic >= 3.5 2015-04-28 00:48:01.083564 7fb37ebb9900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7fb37ebb9900 time 2015-04-28 00:48:01.078616 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7fb37bcfcec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -48> 2015-04-28 00:48:00.878823 7fb37ebb9900 5 asok(0x3dbc000) register_command perfcounters_dump hook 0x3d5c050 -47> 2015-04-28 00:48:00.878879 7fb37ebb9900 5 asok(0x3dbc000) register_command 1 hook 0x3d5c050 -46> 2015-04-28 00:48:00.878893 7fb37ebb9900 5 asok(0x3dbc000) register_command perf dump hook 0x3d5c050 -45> 2015-04-28 00:48:00.878906 7fb37ebb9900 5 asok(0x3dbc000) register_command perfcounters_schema hook 0x3d5c050 -44> 2015-04-28 00:48:00.878920 7fb37ebb9900 5 asok(0x3dbc000) register_command 2 hook 0x3d5c050 -43> 2015-04-28 00:48:00.878925 7fb37ebb9900 5 asok(0x3dbc000) register_command perf schema hook 0x3d5c050 -42> 2015-04-28 00:48:00.878934 7fb37ebb9900 5 asok(0x3dbc000) register_command perf reset hook 0x3d5c050 -41> 2015-04-28 00:48:00.878945 7fb37ebb9900 5 asok(0x3dbc000) register_command config show hook 0x3d5c050 -40> 2015-04-28 00:48:00.878954 7fb37ebb9900 5 asok(0x3dbc000) register_command config set hook 0x3d5c050 -39> 2015-04-28 00:48:00.878960 7fb37ebb9900 5 asok(0x3dbc000) register_command config get hook 0x3d5c050 -38> 2015-04-28 00:48:00.878967 7fb37ebb9900 5 asok(0x3dbc000) register_command config diff hook 0x3d5c050 -37> 2015-04-28 00:48:00.878972 7fb37ebb9900 5 asok(0x3dbc000) register_command log flush hook 0x3d5c050 -36> 2015-04-28 00:48:00.878980 7fb37ebb9900 5 asok(0x3dbc000) register_command log dump hook 0x3d5c050 -35> 2015-04-28 00:48:00.878985 7fb37ebb9900 5 asok(0x3dbc000) register_command log reopen hook 0x3d5c050 -34> 2015-04-28 00:48:00.883340 7fb37ebb9900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 4556 -33> 2015-04-28 00:48:00.884811 7fb37ebb9900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -32> 2015-04-28 00:48:00.884828 7fb37ebb9900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6802/4556 need_addr=0 -31> 2015-04-28 00:48:00.884853 7fb37ebb9900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -30> 2015-04-28 00:48:00.884859 7fb37ebb9900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6802/4556 need_addr=0 -29> 2015-04-28 00:48:00.884877 7fb37ebb9900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -28> 2015-04-28 00:48:00.884883 7fb37ebb9900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6803/4556 need_addr=0 -27> 2015-04-28 00:48:00.884900 7fb37ebb9900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -26> 2015-04-28 00:48:00.884906 7fb37ebb9900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6803/4556 need_addr=0 -25> 2015-04-28 00:48:00.890112 7fb37ebb9900 5 asok(0x3dbc000) init /var/run/ceph/ceph-osd.23.asok -24> 2015-04-28 00:48:00.890148 7fb37ebb9900 5 asok(0x3dbc000) bind_and_listen /var/run/ceph/ceph-osd.23.asok -23> 2015-04-28 00:48:00.890227 7fb37ebb9900 5 asok(0x3dbc000) register_command 0 hook 0x3d580a8 -22> 2015-04-28 00:48:00.890246 7fb37ebb9900 5 asok(0x3dbc000) register_command version hook 0x3d580a8 -21> 2015-04-28 00:48:00.890271 7fb37ebb9900 5 asok(0x3dbc000) register_command git_version hook 0x3d580a8 -20> 2015-04-28 00:48:00.890292 7fb37ebb9900 5 asok(0x3dbc000) register_command help hook 0x3d5c150 -19> 2015-04-28 00:48:00.890308 7fb37ebb9900 5 asok(0x3dbc000) register_command get_command_descriptions hook 0x3d5c140 -18> 2015-04-28 00:48:00.890405 7fb37ebb9900 10 monclient(hunting): build_initial_monmap -17> 2015-04-28 00:48:00.890567 7fb378cc4700 5 asok(0x3dbc000) entry start -16> 2015-04-28 00:48:00.908679 7fb37ebb9900 5 adding auth protocol: cephx -15> 2015-04-28 00:48:00.908704 7fb37ebb9900 5 adding auth protocol: cephx -14> 2015-04-28 00:48:00.908906 7fb37ebb9900 5 asok(0x3dbc000) register_command objecter_requests hook 0x3d5c180 -13> 2015-04-28 00:48:00.909046 7fb37ebb9900 1 -- 192.168.30.73:6802/4556 messenger.start -12> 2015-04-28 00:48:00.909116 7fb37ebb9900 1 -- :/0 messenger.start -11> 2015-04-28 00:48:00.909173 7fb37ebb9900 1 -- 192.168.30.73:6803/4556 messenger.start -10> 2015-04-28 00:48:00.909229 7fb37ebb9900 1 -- 192.168.32.73:6803/4556 messenger.start -9> 2015-04-28 00:48:00.909284 7fb37ebb9900 1 -- 192.168.32.73:6802/4556 messenger.start -8> 2015-04-28 00:48:00.909334 7fb37ebb9900 1 -- :/0 messenger.start -7> 2015-04-28 00:48:00.909436 7fb37ebb9900 2 osd.23 0 mounting /var/lib/ceph/osd/ceph-23 /var/lib/ceph/osd/ceph-23/journal -6> 2015-04-28 00:48:00.909794 7fb37ebb9900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. -5> 2015-04-28 00:48:00.910034 7fb37ebb9900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) -4> 2015-04-28 00:48:00.939633 7fb37ebb9900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work -3> 2015-04-28 00:48:00.939664 7fb37ebb9900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option -2> 2015-04-28 00:48:00.940325 7fb37ebb9900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) -1> 2015-04-28 00:48:00.943774 7fb37ebb9900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.9-031809-generic >= 3.5 0> 2015-04-28 00:48:01.083564 7fb37ebb9900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7fb37ebb9900 time 2015-04-28 00:48:01.078616 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7fb37bcfcec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 00:48:01.090488 7fb37ebb9900 -1 *** Caught signal (Aborted) ** in thread 7fb37ebb9900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7fb37d872340] 3: (gsignal()+0x39) [0x7fb37bd11cc9] 4: (abort()+0x148) [0x7fb37bd150d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7fb37c61c6b5] 6: (()+0x5e836) [0x7fb37c61a836] 7: (()+0x5e863) [0x7fb37c61a863] 8: (()+0x5eaa2) [0x7fb37c61aaa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7fb37bcfcec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2015-04-28 00:48:01.090488 7fb37ebb9900 -1 *** Caught signal (Aborted) ** in thread 7fb37ebb9900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7fb37d872340] 3: (gsignal()+0x39) [0x7fb37bd11cc9] 4: (abort()+0x148) [0x7fb37bd150d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7fb37c61c6b5] 6: (()+0x5e836) [0x7fb37c61a836] 7: (()+0x5e863) [0x7fb37c61a863] 8: (()+0x5eaa2) [0x7fb37c61aaa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7fb37bcfcec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 00:48:02.315269 7f53b4203900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 5007 2015-04-28 00:48:02.339946 7f53b4203900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. 2015-04-28 00:48:02.339994 7f53b4203900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) 2015-04-28 00:48:02.341518 7f53b4203900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work 2015-04-28 00:48:02.341536 7f53b4203900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-04-28 00:48:02.342310 7f53b4203900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2015-04-28 00:48:02.342455 7f53b4203900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.9-031809-generic >= 3.5 2015-04-28 00:48:02.359830 7f53b4203900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f53b4203900 time 2015-04-28 00:48:02.354946 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f53b1346ec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -48> 2015-04-28 00:48:02.311035 7f53b4203900 5 asok(0x4fb0000) register_command perfcounters_dump hook 0x4f50050 -47> 2015-04-28 00:48:02.311087 7f53b4203900 5 asok(0x4fb0000) register_command 1 hook 0x4f50050 -46> 2015-04-28 00:48:02.311094 7f53b4203900 5 asok(0x4fb0000) register_command perf dump hook 0x4f50050 -45> 2015-04-28 00:48:02.311106 7f53b4203900 5 asok(0x4fb0000) register_command perfcounters_schema hook 0x4f50050 -44> 2015-04-28 00:48:02.311111 7f53b4203900 5 asok(0x4fb0000) register_command 2 hook 0x4f50050 -43> 2015-04-28 00:48:02.311115 7f53b4203900 5 asok(0x4fb0000) register_command perf schema hook 0x4f50050 -42> 2015-04-28 00:48:02.311121 7f53b4203900 5 asok(0x4fb0000) register_command perf reset hook 0x4f50050 -41> 2015-04-28 00:48:02.311129 7f53b4203900 5 asok(0x4fb0000) register_command config show hook 0x4f50050 -40> 2015-04-28 00:48:02.311135 7f53b4203900 5 asok(0x4fb0000) register_command config set hook 0x4f50050 -39> 2015-04-28 00:48:02.311140 7f53b4203900 5 asok(0x4fb0000) register_command config get hook 0x4f50050 -38> 2015-04-28 00:48:02.311144 7f53b4203900 5 asok(0x4fb0000) register_command config diff hook 0x4f50050 -37> 2015-04-28 00:48:02.311149 7f53b4203900 5 asok(0x4fb0000) register_command log flush hook 0x4f50050 -36> 2015-04-28 00:48:02.311153 7f53b4203900 5 asok(0x4fb0000) register_command log dump hook 0x4f50050 -35> 2015-04-28 00:48:02.311158 7f53b4203900 5 asok(0x4fb0000) register_command log reopen hook 0x4f50050 -34> 2015-04-28 00:48:02.315269 7f53b4203900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 5007 -33> 2015-04-28 00:48:02.317262 7f53b4203900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -32> 2015-04-28 00:48:02.317293 7f53b4203900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6806/5007 need_addr=0 -31> 2015-04-28 00:48:02.317338 7f53b4203900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -30> 2015-04-28 00:48:02.317356 7f53b4203900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6806/5007 need_addr=0 -29> 2015-04-28 00:48:02.317396 7f53b4203900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -28> 2015-04-28 00:48:02.317411 7f53b4203900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6807/5007 need_addr=0 -27> 2015-04-28 00:48:02.317444 7f53b4203900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -26> 2015-04-28 00:48:02.317457 7f53b4203900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6807/5007 need_addr=0 -25> 2015-04-28 00:48:02.320578 7f53b4203900 5 asok(0x4fb0000) init /var/run/ceph/ceph-osd.23.asok -24> 2015-04-28 00:48:02.320606 7f53b4203900 5 asok(0x4fb0000) bind_and_listen /var/run/ceph/ceph-osd.23.asok -23> 2015-04-28 00:48:02.320823 7f53b4203900 5 asok(0x4fb0000) register_command 0 hook 0x4f4c0a8 -22> 2015-04-28 00:48:02.320843 7f53b4203900 5 asok(0x4fb0000) register_command version hook 0x4f4c0a8 -21> 2015-04-28 00:48:02.320851 7f53b4203900 5 asok(0x4fb0000) register_command git_version hook 0x4f4c0a8 -20> 2015-04-28 00:48:02.320861 7f53b4203900 5 asok(0x4fb0000) register_command help hook 0x4f50150 -19> 2015-04-28 00:48:02.320869 7f53b4203900 5 asok(0x4fb0000) register_command get_command_descriptions hook 0x4f50140 -18> 2015-04-28 00:48:02.320941 7f53ae30e700 5 asok(0x4fb0000) entry start -17> 2015-04-28 00:48:02.320954 7f53b4203900 10 monclient(hunting): build_initial_monmap -16> 2015-04-28 00:48:02.339276 7f53b4203900 5 adding auth protocol: cephx -15> 2015-04-28 00:48:02.339293 7f53b4203900 5 adding auth protocol: cephx -14> 2015-04-28 00:48:02.339444 7f53b4203900 5 asok(0x4fb0000) register_command objecter_requests hook 0x4f50180 -13> 2015-04-28 00:48:02.339556 7f53b4203900 1 -- 192.168.30.73:6806/5007 messenger.start -12> 2015-04-28 00:48:02.339625 7f53b4203900 1 -- :/0 messenger.start -11> 2015-04-28 00:48:02.339667 7f53b4203900 1 -- 192.168.30.73:6807/5007 messenger.start -10> 2015-04-28 00:48:02.339706 7f53b4203900 1 -- 192.168.32.73:6807/5007 messenger.start -9> 2015-04-28 00:48:02.339743 7f53b4203900 1 -- 192.168.32.73:6806/5007 messenger.start -8> 2015-04-28 00:48:02.339785 7f53b4203900 1 -- :/0 messenger.start -7> 2015-04-28 00:48:02.339873 7f53b4203900 2 osd.23 0 mounting /var/lib/ceph/osd/ceph-23 /var/lib/ceph/osd/ceph-23/journal -6> 2015-04-28 00:48:02.339946 7f53b4203900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. -5> 2015-04-28 00:48:02.339994 7f53b4203900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) -4> 2015-04-28 00:48:02.341518 7f53b4203900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work -3> 2015-04-28 00:48:02.341536 7f53b4203900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option -2> 2015-04-28 00:48:02.342310 7f53b4203900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) -1> 2015-04-28 00:48:02.342455 7f53b4203900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.9-031809-generic >= 3.5 0> 2015-04-28 00:48:02.359830 7f53b4203900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f53b4203900 time 2015-04-28 00:48:02.354946 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f53b1346ec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 00:48:02.366836 7f53b4203900 -1 *** Caught signal (Aborted) ** in thread 7f53b4203900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f53b2ebc340] 3: (gsignal()+0x39) [0x7f53b135bcc9] 4: (abort()+0x148) [0x7f53b135f0d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f53b1c666b5] 6: (()+0x5e836) [0x7f53b1c64836] 7: (()+0x5e863) [0x7f53b1c64863] 8: (()+0x5eaa2) [0x7f53b1c64aa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f53b1346ec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2015-04-28 00:48:02.366836 7f53b4203900 -1 *** Caught signal (Aborted) ** in thread 7f53b4203900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f53b2ebc340] 3: (gsignal()+0x39) [0x7f53b135bcc9] 4: (abort()+0x148) [0x7f53b135f0d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f53b1c666b5] 6: (()+0x5e836) [0x7f53b1c64836] 7: (()+0x5e863) [0x7f53b1c64863] 8: (()+0x5eaa2) [0x7f53b1c64aa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f53b1346ec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 00:48:03.228067 7ffb62831900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 5532 2015-04-28 00:48:03.249568 7ffb62831900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. 2015-04-28 00:48:03.249614 7ffb62831900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) 2015-04-28 00:48:03.251244 7ffb62831900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work 2015-04-28 00:48:03.251267 7ffb62831900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-04-28 00:48:03.252238 7ffb62831900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2015-04-28 00:48:03.252388 7ffb62831900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.9-031809-generic >= 3.5 2015-04-28 00:48:03.262178 7ffb62831900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7ffb62831900 time 2015-04-28 00:48:03.257334 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7ffb5f974ec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -48> 2015-04-28 00:48:03.223910 7ffb62831900 5 asok(0x44c2000) register_command perfcounters_dump hook 0x4462050 -47> 2015-04-28 00:48:03.223970 7ffb62831900 5 asok(0x44c2000) register_command 1 hook 0x4462050 -46> 2015-04-28 00:48:03.223986 7ffb62831900 5 asok(0x44c2000) register_command perf dump hook 0x4462050 -45> 2015-04-28 00:48:03.223998 7ffb62831900 5 asok(0x44c2000) register_command perfcounters_schema hook 0x4462050 -44> 2015-04-28 00:48:03.224008 7ffb62831900 5 asok(0x44c2000) register_command 2 hook 0x4462050 -43> 2015-04-28 00:48:03.224016 7ffb62831900 5 asok(0x44c2000) register_command perf schema hook 0x4462050 -42> 2015-04-28 00:48:03.224023 7ffb62831900 5 asok(0x44c2000) register_command perf reset hook 0x4462050 -41> 2015-04-28 00:48:03.224045 7ffb62831900 5 asok(0x44c2000) register_command config show hook 0x4462050 -40> 2015-04-28 00:48:03.224055 7ffb62831900 5 asok(0x44c2000) register_command config set hook 0x4462050 -39> 2015-04-28 00:48:03.224064 7ffb62831900 5 asok(0x44c2000) register_command config get hook 0x4462050 -38> 2015-04-28 00:48:03.224072 7ffb62831900 5 asok(0x44c2000) register_command config diff hook 0x4462050 -37> 2015-04-28 00:48:03.224080 7ffb62831900 5 asok(0x44c2000) register_command log flush hook 0x4462050 -36> 2015-04-28 00:48:03.224090 7ffb62831900 5 asok(0x44c2000) register_command log dump hook 0x4462050 -35> 2015-04-28 00:48:03.224098 7ffb62831900 5 asok(0x44c2000) register_command log reopen hook 0x4462050 -34> 2015-04-28 00:48:03.228067 7ffb62831900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 5532 -33> 2015-04-28 00:48:03.229581 7ffb62831900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -32> 2015-04-28 00:48:03.229614 7ffb62831900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6802/5532 need_addr=0 -31> 2015-04-28 00:48:03.229659 7ffb62831900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -30> 2015-04-28 00:48:03.229674 7ffb62831900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6802/5532 need_addr=0 -29> 2015-04-28 00:48:03.229694 7ffb62831900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -28> 2015-04-28 00:48:03.229704 7ffb62831900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6803/5532 need_addr=0 -27> 2015-04-28 00:48:03.229722 7ffb62831900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -26> 2015-04-28 00:48:03.229732 7ffb62831900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6803/5532 need_addr=0 -25> 2015-04-28 00:48:03.232062 7ffb62831900 5 asok(0x44c2000) init /var/run/ceph/ceph-osd.23.asok -24> 2015-04-28 00:48:03.232084 7ffb62831900 5 asok(0x44c2000) bind_and_listen /var/run/ceph/ceph-osd.23.asok -23> 2015-04-28 00:48:03.232246 7ffb62831900 5 asok(0x44c2000) register_command 0 hook 0x445e0a8 -22> 2015-04-28 00:48:03.232260 7ffb62831900 5 asok(0x44c2000) register_command version hook 0x445e0a8 -21> 2015-04-28 00:48:03.232265 7ffb62831900 5 asok(0x44c2000) register_command git_version hook 0x445e0a8 -20> 2015-04-28 00:48:03.232272 7ffb62831900 5 asok(0x44c2000) register_command help hook 0x4462150 -19> 2015-04-28 00:48:03.232277 7ffb62831900 5 asok(0x44c2000) register_command get_command_descriptions hook 0x4462140 -18> 2015-04-28 00:48:03.232342 7ffb62831900 10 monclient(hunting): build_initial_monmap -17> 2015-04-28 00:48:03.232350 7ffb5c93c700 5 asok(0x44c2000) entry start -16> 2015-04-28 00:48:03.248849 7ffb62831900 5 adding auth protocol: cephx -15> 2015-04-28 00:48:03.248869 7ffb62831900 5 adding auth protocol: cephx -14> 2015-04-28 00:48:03.249036 7ffb62831900 5 asok(0x44c2000) register_command objecter_requests hook 0x4462180 -13> 2015-04-28 00:48:03.249162 7ffb62831900 1 -- 192.168.30.73:6802/5532 messenger.start -12> 2015-04-28 00:48:03.249223 7ffb62831900 1 -- :/0 messenger.start -11> 2015-04-28 00:48:03.249266 7ffb62831900 1 -- 192.168.30.73:6803/5532 messenger.start -10> 2015-04-28 00:48:03.249305 7ffb62831900 1 -- 192.168.32.73:6803/5532 messenger.start -9> 2015-04-28 00:48:03.249343 7ffb62831900 1 -- 192.168.32.73:6802/5532 messenger.start -8> 2015-04-28 00:48:03.249384 7ffb62831900 1 -- :/0 messenger.start -7> 2015-04-28 00:48:03.249493 7ffb62831900 2 osd.23 0 mounting /var/lib/ceph/osd/ceph-23 /var/lib/ceph/osd/ceph-23/journal -6> 2015-04-28 00:48:03.249568 7ffb62831900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. -5> 2015-04-28 00:48:03.249614 7ffb62831900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) -4> 2015-04-28 00:48:03.251244 7ffb62831900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work -3> 2015-04-28 00:48:03.251267 7ffb62831900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option -2> 2015-04-28 00:48:03.252238 7ffb62831900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) -1> 2015-04-28 00:48:03.252388 7ffb62831900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.9-031809-generic >= 3.5 0> 2015-04-28 00:48:03.262178 7ffb62831900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7ffb62831900 time 2015-04-28 00:48:03.257334 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7ffb5f974ec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 00:48:03.269164 7ffb62831900 -1 *** Caught signal (Aborted) ** in thread 7ffb62831900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7ffb614ea340] 3: (gsignal()+0x39) [0x7ffb5f989cc9] 4: (abort()+0x148) [0x7ffb5f98d0d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7ffb602946b5] 6: (()+0x5e836) [0x7ffb60292836] 7: (()+0x5e863) [0x7ffb60292863] 8: (()+0x5eaa2) [0x7ffb60292aa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7ffb5f974ec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2015-04-28 00:48:03.269164 7ffb62831900 -1 *** Caught signal (Aborted) ** in thread 7ffb62831900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7ffb614ea340] 3: (gsignal()+0x39) [0x7ffb5f989cc9] 4: (abort()+0x148) [0x7ffb5f98d0d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7ffb602946b5] 6: (()+0x5e836) [0x7ffb60292836] 7: (()+0x5e863) [0x7ffb60292863] 8: (()+0x5eaa2) [0x7ffb60292aa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7ffb5f974ec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 00:48:05.291486 7f03fadb2900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 6335 2015-04-28 00:48:05.327949 7f03fadb2900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. 2015-04-28 00:48:05.327995 7f03fadb2900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) 2015-04-28 00:48:05.329412 7f03fadb2900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work 2015-04-28 00:48:05.329436 7f03fadb2900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-04-28 00:48:05.330360 7f03fadb2900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2015-04-28 00:48:05.330506 7f03fadb2900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.9-031809-generic >= 3.5 2015-04-28 00:48:05.339694 7f03fadb2900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f03fadb2900 time 2015-04-28 00:48:05.334824 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f03f7ef5ec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -48> 2015-04-28 00:48:05.286840 7f03fadb2900 5 asok(0x5342000) register_command perfcounters_dump hook 0x52e2050 -47> 2015-04-28 00:48:05.286909 7f03fadb2900 5 asok(0x5342000) register_command 1 hook 0x52e2050 -46> 2015-04-28 00:48:05.286922 7f03fadb2900 5 asok(0x5342000) register_command perf dump hook 0x52e2050 -45> 2015-04-28 00:48:05.286932 7f03fadb2900 5 asok(0x5342000) register_command perfcounters_schema hook 0x52e2050 -44> 2015-04-28 00:48:05.286941 7f03fadb2900 5 asok(0x5342000) register_command 2 hook 0x52e2050 -43> 2015-04-28 00:48:05.286945 7f03fadb2900 5 asok(0x5342000) register_command perf schema hook 0x52e2050 -42> 2015-04-28 00:48:05.286955 7f03fadb2900 5 asok(0x5342000) register_command perf reset hook 0x52e2050 -41> 2015-04-28 00:48:05.286978 7f03fadb2900 5 asok(0x5342000) register_command config show hook 0x52e2050 -40> 2015-04-28 00:48:05.286990 7f03fadb2900 5 asok(0x5342000) register_command config set hook 0x52e2050 -39> 2015-04-28 00:48:05.286996 7f03fadb2900 5 asok(0x5342000) register_command config get hook 0x52e2050 -38> 2015-04-28 00:48:05.287004 7f03fadb2900 5 asok(0x5342000) register_command config diff hook 0x52e2050 -37> 2015-04-28 00:48:05.287009 7f03fadb2900 5 asok(0x5342000) register_command log flush hook 0x52e2050 -36> 2015-04-28 00:48:05.287019 7f03fadb2900 5 asok(0x5342000) register_command log dump hook 0x52e2050 -35> 2015-04-28 00:48:05.287024 7f03fadb2900 5 asok(0x5342000) register_command log reopen hook 0x52e2050 -34> 2015-04-28 00:48:05.291486 7f03fadb2900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 6335 -33> 2015-04-28 00:48:05.293098 7f03fadb2900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -32> 2015-04-28 00:48:05.293121 7f03fadb2900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6810/6335 need_addr=0 -31> 2015-04-28 00:48:05.293154 7f03fadb2900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -30> 2015-04-28 00:48:05.293164 7f03fadb2900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6810/6335 need_addr=0 -29> 2015-04-28 00:48:05.293190 7f03fadb2900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -28> 2015-04-28 00:48:05.293199 7f03fadb2900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6811/6335 need_addr=0 -27> 2015-04-28 00:48:05.293225 7f03fadb2900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -26> 2015-04-28 00:48:05.293235 7f03fadb2900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6811/6335 need_addr=0 -25> 2015-04-28 00:48:05.295596 7f03fadb2900 5 asok(0x5342000) init /var/run/ceph/ceph-osd.23.asok -24> 2015-04-28 00:48:05.295618 7f03fadb2900 5 asok(0x5342000) bind_and_listen /var/run/ceph/ceph-osd.23.asok -23> 2015-04-28 00:48:05.295775 7f03fadb2900 5 asok(0x5342000) register_command 0 hook 0x52de0a8 -22> 2015-04-28 00:48:05.295789 7f03fadb2900 5 asok(0x5342000) register_command version hook 0x52de0a8 -21> 2015-04-28 00:48:05.295794 7f03fadb2900 5 asok(0x5342000) register_command git_version hook 0x52de0a8 -20> 2015-04-28 00:48:05.295803 7f03fadb2900 5 asok(0x5342000) register_command help hook 0x52e2150 -19> 2015-04-28 00:48:05.295812 7f03fadb2900 5 asok(0x5342000) register_command get_command_descriptions hook 0x52e2140 -18> 2015-04-28 00:48:05.295885 7f03fadb2900 10 monclient(hunting): build_initial_monmap -17> 2015-04-28 00:48:05.295879 7f03f4ebd700 5 asok(0x5342000) entry start -16> 2015-04-28 00:48:05.327250 7f03fadb2900 5 adding auth protocol: cephx -15> 2015-04-28 00:48:05.327267 7f03fadb2900 5 adding auth protocol: cephx -14> 2015-04-28 00:48:05.327435 7f03fadb2900 5 asok(0x5342000) register_command objecter_requests hook 0x52e2180 -13> 2015-04-28 00:48:05.327561 7f03fadb2900 1 -- 192.168.30.73:6810/6335 messenger.start -12> 2015-04-28 00:48:05.327620 7f03fadb2900 1 -- :/0 messenger.start -11> 2015-04-28 00:48:05.327662 7f03fadb2900 1 -- 192.168.30.73:6811/6335 messenger.start -10> 2015-04-28 00:48:05.327702 7f03fadb2900 1 -- 192.168.32.73:6811/6335 messenger.start -9> 2015-04-28 00:48:05.327745 7f03fadb2900 1 -- 192.168.32.73:6810/6335 messenger.start -8> 2015-04-28 00:48:05.327788 7f03fadb2900 1 -- :/0 messenger.start -7> 2015-04-28 00:48:05.327880 7f03fadb2900 2 osd.23 0 mounting /var/lib/ceph/osd/ceph-23 /var/lib/ceph/osd/ceph-23/journal -6> 2015-04-28 00:48:05.327949 7f03fadb2900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. -5> 2015-04-28 00:48:05.327995 7f03fadb2900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) -4> 2015-04-28 00:48:05.329412 7f03fadb2900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work -3> 2015-04-28 00:48:05.329436 7f03fadb2900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option -2> 2015-04-28 00:48:05.330360 7f03fadb2900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) -1> 2015-04-28 00:48:05.330506 7f03fadb2900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.9-031809-generic >= 3.5 0> 2015-04-28 00:48:05.339694 7f03fadb2900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f03fadb2900 time 2015-04-28 00:48:05.334824 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f03f7ef5ec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 00:48:05.346461 7f03fadb2900 -1 *** Caught signal (Aborted) ** in thread 7f03fadb2900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f03f9a6b340] 3: (gsignal()+0x39) [0x7f03f7f0acc9] 4: (abort()+0x148) [0x7f03f7f0e0d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f03f88156b5] 6: (()+0x5e836) [0x7f03f8813836] 7: (()+0x5e863) [0x7f03f8813863] 8: (()+0x5eaa2) [0x7f03f8813aa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f03f7ef5ec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2015-04-28 00:48:05.346461 7f03fadb2900 -1 *** Caught signal (Aborted) ** in thread 7f03fadb2900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f03f9a6b340] 3: (gsignal()+0x39) [0x7f03f7f0acc9] 4: (abort()+0x148) [0x7f03f7f0e0d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f03f88156b5] 6: (()+0x5e836) [0x7f03f8813836] 7: (()+0x5e863) [0x7f03f8813863] 8: (()+0x5eaa2) [0x7f03f8813aa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f03f7ef5ec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 00:48:06.022205 7f1657c9d900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 6842 2015-04-28 00:48:06.045071 7f1657c9d900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. 2015-04-28 00:48:06.045118 7f1657c9d900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) 2015-04-28 00:48:06.046605 7f1657c9d900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work 2015-04-28 00:48:06.046624 7f1657c9d900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-04-28 00:48:06.047375 7f1657c9d900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2015-04-28 00:48:06.047497 7f1657c9d900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.9-031809-generic >= 3.5 2015-04-28 00:48:06.057857 7f1657c9d900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f1657c9d900 time 2015-04-28 00:48:06.052750 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f1654de0ec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -48> 2015-04-28 00:48:06.017550 7f1657c9d900 5 asok(0x4df6000) register_command perfcounters_dump hook 0x4d96050 -47> 2015-04-28 00:48:06.017618 7f1657c9d900 5 asok(0x4df6000) register_command 1 hook 0x4d96050 -46> 2015-04-28 00:48:06.017625 7f1657c9d900 5 asok(0x4df6000) register_command perf dump hook 0x4d96050 -45> 2015-04-28 00:48:06.017662 7f1657c9d900 5 asok(0x4df6000) register_command perfcounters_schema hook 0x4d96050 -44> 2015-04-28 00:48:06.017672 7f1657c9d900 5 asok(0x4df6000) register_command 2 hook 0x4d96050 -43> 2015-04-28 00:48:06.017677 7f1657c9d900 5 asok(0x4df6000) register_command perf schema hook 0x4d96050 -42> 2015-04-28 00:48:06.017686 7f1657c9d900 5 asok(0x4df6000) register_command perf reset hook 0x4d96050 -41> 2015-04-28 00:48:06.017706 7f1657c9d900 5 asok(0x4df6000) register_command config show hook 0x4d96050 -40> 2015-04-28 00:48:06.017715 7f1657c9d900 5 asok(0x4df6000) register_command config set hook 0x4d96050 -39> 2015-04-28 00:48:06.017720 7f1657c9d900 5 asok(0x4df6000) register_command config get hook 0x4d96050 -38> 2015-04-28 00:48:06.017724 7f1657c9d900 5 asok(0x4df6000) register_command config diff hook 0x4d96050 -37> 2015-04-28 00:48:06.017729 7f1657c9d900 5 asok(0x4df6000) register_command log flush hook 0x4d96050 -36> 2015-04-28 00:48:06.017737 7f1657c9d900 5 asok(0x4df6000) register_command log dump hook 0x4d96050 -35> 2015-04-28 00:48:06.017743 7f1657c9d900 5 asok(0x4df6000) register_command log reopen hook 0x4d96050 -34> 2015-04-28 00:48:06.022205 7f1657c9d900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 6842 -33> 2015-04-28 00:48:06.024005 7f1657c9d900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -32> 2015-04-28 00:48:06.024036 7f1657c9d900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6806/6842 need_addr=0 -31> 2015-04-28 00:48:06.024073 7f1657c9d900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -30> 2015-04-28 00:48:06.024086 7f1657c9d900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6806/6842 need_addr=0 -29> 2015-04-28 00:48:06.024150 7f1657c9d900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -28> 2015-04-28 00:48:06.024165 7f1657c9d900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6807/6842 need_addr=0 -27> 2015-04-28 00:48:06.024188 7f1657c9d900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -26> 2015-04-28 00:48:06.024195 7f1657c9d900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6807/6842 need_addr=0 -25> 2015-04-28 00:48:06.026749 7f1657c9d900 5 asok(0x4df6000) init /var/run/ceph/ceph-osd.23.asok -24> 2015-04-28 00:48:06.026774 7f1657c9d900 5 asok(0x4df6000) bind_and_listen /var/run/ceph/ceph-osd.23.asok -23> 2015-04-28 00:48:06.026947 7f1657c9d900 5 asok(0x4df6000) register_command 0 hook 0x4d920a8 -22> 2015-04-28 00:48:06.026962 7f1657c9d900 5 asok(0x4df6000) register_command version hook 0x4d920a8 -21> 2015-04-28 00:48:06.026968 7f1657c9d900 5 asok(0x4df6000) register_command git_version hook 0x4d920a8 -20> 2015-04-28 00:48:06.026974 7f1657c9d900 5 asok(0x4df6000) register_command help hook 0x4d96150 -19> 2015-04-28 00:48:06.026979 7f1657c9d900 5 asok(0x4df6000) register_command get_command_descriptions hook 0x4d96140 -18> 2015-04-28 00:48:06.027046 7f1657c9d900 10 monclient(hunting): build_initial_monmap -17> 2015-04-28 00:48:06.027044 7f1651da8700 5 asok(0x4df6000) entry start -16> 2015-04-28 00:48:06.044356 7f1657c9d900 5 adding auth protocol: cephx -15> 2015-04-28 00:48:06.044378 7f1657c9d900 5 adding auth protocol: cephx -14> 2015-04-28 00:48:06.044540 7f1657c9d900 5 asok(0x4df6000) register_command objecter_requests hook 0x4d96180 -13> 2015-04-28 00:48:06.044667 7f1657c9d900 1 -- 192.168.30.73:6806/6842 messenger.start -12> 2015-04-28 00:48:06.044726 7f1657c9d900 1 -- :/0 messenger.start -11> 2015-04-28 00:48:06.044768 7f1657c9d900 1 -- 192.168.30.73:6807/6842 messenger.start -10> 2015-04-28 00:48:06.044808 7f1657c9d900 1 -- 192.168.32.73:6807/6842 messenger.start -9> 2015-04-28 00:48:06.044851 7f1657c9d900 1 -- 192.168.32.73:6806/6842 messenger.start -8> 2015-04-28 00:48:06.044893 7f1657c9d900 1 -- :/0 messenger.start -7> 2015-04-28 00:48:06.044998 7f1657c9d900 2 osd.23 0 mounting /var/lib/ceph/osd/ceph-23 /var/lib/ceph/osd/ceph-23/journal -6> 2015-04-28 00:48:06.045071 7f1657c9d900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. -5> 2015-04-28 00:48:06.045118 7f1657c9d900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) -4> 2015-04-28 00:48:06.046605 7f1657c9d900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work -3> 2015-04-28 00:48:06.046624 7f1657c9d900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option -2> 2015-04-28 00:48:06.047375 7f1657c9d900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) -1> 2015-04-28 00:48:06.047497 7f1657c9d900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.9-031809-generic >= 3.5 0> 2015-04-28 00:48:06.057857 7f1657c9d900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f1657c9d900 time 2015-04-28 00:48:06.052750 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f1654de0ec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 00:48:06.065014 7f1657c9d900 -1 *** Caught signal (Aborted) ** in thread 7f1657c9d900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f1656956340] 3: (gsignal()+0x39) [0x7f1654df5cc9] 4: (abort()+0x148) [0x7f1654df90d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f16557006b5] 6: (()+0x5e836) [0x7f16556fe836] 7: (()+0x5e863) [0x7f16556fe863] 8: (()+0x5eaa2) [0x7f16556feaa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f1654de0ec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2015-04-28 00:48:06.065014 7f1657c9d900 -1 *** Caught signal (Aborted) ** in thread 7f1657c9d900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f1656956340] 3: (gsignal()+0x39) [0x7f1654df5cc9] 4: (abort()+0x148) [0x7f1654df90d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f16557006b5] 6: (()+0x5e836) [0x7f16556fe836] 7: (()+0x5e863) [0x7f16556fe863] 8: (()+0x5eaa2) [0x7f16556feaa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f1654de0ec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 00:48:06.721471 7f29dc050900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 7484 2015-04-28 00:48:06.740147 7f29dc050900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. 2015-04-28 00:48:06.740187 7f29dc050900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) 2015-04-28 00:48:06.741477 7f29dc050900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work 2015-04-28 00:48:06.741491 7f29dc050900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-04-28 00:48:06.742158 7f29dc050900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2015-04-28 00:48:06.742272 7f29dc050900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.9-031809-generic >= 3.5 2015-04-28 00:48:06.749824 7f29dc050900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f29dc050900 time 2015-04-28 00:48:06.745703 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f29d9193ec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -48> 2015-04-28 00:48:06.717776 7f29dc050900 5 asok(0x5318000) register_command perfcounters_dump hook 0x52b8050 -47> 2015-04-28 00:48:06.717829 7f29dc050900 5 asok(0x5318000) register_command 1 hook 0x52b8050 -46> 2015-04-28 00:48:06.717838 7f29dc050900 5 asok(0x5318000) register_command perf dump hook 0x52b8050 -45> 2015-04-28 00:48:06.717849 7f29dc050900 5 asok(0x5318000) register_command perfcounters_schema hook 0x52b8050 -44> 2015-04-28 00:48:06.717857 7f29dc050900 5 asok(0x5318000) register_command 2 hook 0x52b8050 -43> 2015-04-28 00:48:06.717860 7f29dc050900 5 asok(0x5318000) register_command perf schema hook 0x52b8050 -42> 2015-04-28 00:48:06.717868 7f29dc050900 5 asok(0x5318000) register_command perf reset hook 0x52b8050 -41> 2015-04-28 00:48:06.717890 7f29dc050900 5 asok(0x5318000) register_command config show hook 0x52b8050 -40> 2015-04-28 00:48:06.717898 7f29dc050900 5 asok(0x5318000) register_command config set hook 0x52b8050 -39> 2015-04-28 00:48:06.717906 7f29dc050900 5 asok(0x5318000) register_command config get hook 0x52b8050 -38> 2015-04-28 00:48:06.717913 7f29dc050900 5 asok(0x5318000) register_command config diff hook 0x52b8050 -37> 2015-04-28 00:48:06.717917 7f29dc050900 5 asok(0x5318000) register_command log flush hook 0x52b8050 -36> 2015-04-28 00:48:06.717924 7f29dc050900 5 asok(0x5318000) register_command log dump hook 0x52b8050 -35> 2015-04-28 00:48:06.717930 7f29dc050900 5 asok(0x5318000) register_command log reopen hook 0x52b8050 -34> 2015-04-28 00:48:06.721471 7f29dc050900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 7484 -33> 2015-04-28 00:48:06.722753 7f29dc050900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -32> 2015-04-28 00:48:06.722772 7f29dc050900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6806/7484 need_addr=0 -31> 2015-04-28 00:48:06.722815 7f29dc050900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -30> 2015-04-28 00:48:06.722824 7f29dc050900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6806/7484 need_addr=0 -29> 2015-04-28 00:48:06.722843 7f29dc050900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -28> 2015-04-28 00:48:06.722851 7f29dc050900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6807/7484 need_addr=0 -27> 2015-04-28 00:48:06.722870 7f29dc050900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -26> 2015-04-28 00:48:06.722878 7f29dc050900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6807/7484 need_addr=0 -25> 2015-04-28 00:48:06.724880 7f29dc050900 5 asok(0x5318000) init /var/run/ceph/ceph-osd.23.asok -24> 2015-04-28 00:48:06.724902 7f29dc050900 5 asok(0x5318000) bind_and_listen /var/run/ceph/ceph-osd.23.asok -23> 2015-04-28 00:48:06.725045 7f29dc050900 5 asok(0x5318000) register_command 0 hook 0x52b40a8 -22> 2015-04-28 00:48:06.725058 7f29dc050900 5 asok(0x5318000) register_command version hook 0x52b40a8 -21> 2015-04-28 00:48:06.725066 7f29dc050900 5 asok(0x5318000) register_command git_version hook 0x52b40a8 -20> 2015-04-28 00:48:06.725075 7f29dc050900 5 asok(0x5318000) register_command help hook 0x52b8150 -19> 2015-04-28 00:48:06.725083 7f29dc050900 5 asok(0x5318000) register_command get_command_descriptions hook 0x52b8140 -18> 2015-04-28 00:48:06.725155 7f29dc050900 10 monclient(hunting): build_initial_monmap -17> 2015-04-28 00:48:06.725191 7f29d615b700 5 asok(0x5318000) entry start -16> 2015-04-28 00:48:06.739562 7f29dc050900 5 adding auth protocol: cephx -15> 2015-04-28 00:48:06.739576 7f29dc050900 5 adding auth protocol: cephx -14> 2015-04-28 00:48:06.739723 7f29dc050900 5 asok(0x5318000) register_command objecter_requests hook 0x52b8180 -13> 2015-04-28 00:48:06.739822 7f29dc050900 1 -- 192.168.30.73:6806/7484 messenger.start -12> 2015-04-28 00:48:06.739871 7f29dc050900 1 -- :/0 messenger.start -11> 2015-04-28 00:48:06.739905 7f29dc050900 1 -- 192.168.30.73:6807/7484 messenger.start -10> 2015-04-28 00:48:06.739937 7f29dc050900 1 -- 192.168.32.73:6807/7484 messenger.start -9> 2015-04-28 00:48:06.739971 7f29dc050900 1 -- 192.168.32.73:6806/7484 messenger.start -8> 2015-04-28 00:48:06.740004 7f29dc050900 1 -- :/0 messenger.start -7> 2015-04-28 00:48:06.740086 7f29dc050900 2 osd.23 0 mounting /var/lib/ceph/osd/ceph-23 /var/lib/ceph/osd/ceph-23/journal -6> 2015-04-28 00:48:06.740147 7f29dc050900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. -5> 2015-04-28 00:48:06.740187 7f29dc050900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) -4> 2015-04-28 00:48:06.741477 7f29dc050900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work -3> 2015-04-28 00:48:06.741491 7f29dc050900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option -2> 2015-04-28 00:48:06.742158 7f29dc050900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) -1> 2015-04-28 00:48:06.742272 7f29dc050900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.9-031809-generic >= 3.5 0> 2015-04-28 00:48:06.749824 7f29dc050900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f29dc050900 time 2015-04-28 00:48:06.745703 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f29d9193ec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 00:48:06.755624 7f29dc050900 -1 *** Caught signal (Aborted) ** in thread 7f29dc050900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f29dad09340] 3: (gsignal()+0x39) [0x7f29d91a8cc9] 4: (abort()+0x148) [0x7f29d91ac0d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f29d9ab36b5] 6: (()+0x5e836) [0x7f29d9ab1836] 7: (()+0x5e863) [0x7f29d9ab1863] 8: (()+0x5eaa2) [0x7f29d9ab1aa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f29d9193ec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2015-04-28 00:48:06.755624 7f29dc050900 -1 *** Caught signal (Aborted) ** in thread 7f29dc050900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f29dad09340] 3: (gsignal()+0x39) [0x7f29d91a8cc9] 4: (abort()+0x148) [0x7f29d91ac0d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f29d9ab36b5] 6: (()+0x5e836) [0x7f29d9ab1836] 7: (()+0x5e863) [0x7f29d9ab1863] 8: (()+0x5eaa2) [0x7f29d9ab1aa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f29d9193ec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 01:20:09.685867 7f867bb85900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 3549 2015-04-28 01:20:09.909457 7f867bb85900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. 2015-04-28 01:20:09.909658 7f867bb85900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) 2015-04-28 01:20:09.934609 7f867bb85900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work 2015-04-28 01:20:09.934633 7f867bb85900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-04-28 01:20:09.935501 7f867bb85900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2015-04-28 01:20:09.938816 7f867bb85900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.12-031812-generic >= 3.5 2015-04-28 01:20:10.072699 7f867bb85900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f867bb85900 time 2015-04-28 01:20:10.067550 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f8678cc8ec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -48> 2015-04-28 01:20:09.675665 7f867bb85900 5 asok(0x55e8000) register_command perfcounters_dump hook 0x5588050 -47> 2015-04-28 01:20:09.675712 7f867bb85900 5 asok(0x55e8000) register_command 1 hook 0x5588050 -46> 2015-04-28 01:20:09.675735 7f867bb85900 5 asok(0x55e8000) register_command perf dump hook 0x5588050 -45> 2015-04-28 01:20:09.675747 7f867bb85900 5 asok(0x55e8000) register_command perfcounters_schema hook 0x5588050 -44> 2015-04-28 01:20:09.675756 7f867bb85900 5 asok(0x55e8000) register_command 2 hook 0x5588050 -43> 2015-04-28 01:20:09.675760 7f867bb85900 5 asok(0x55e8000) register_command perf schema hook 0x5588050 -42> 2015-04-28 01:20:09.675766 7f867bb85900 5 asok(0x55e8000) register_command perf reset hook 0x5588050 -41> 2015-04-28 01:20:09.675788 7f867bb85900 5 asok(0x55e8000) register_command config show hook 0x5588050 -40> 2015-04-28 01:20:09.675798 7f867bb85900 5 asok(0x55e8000) register_command config set hook 0x5588050 -39> 2015-04-28 01:20:09.675803 7f867bb85900 5 asok(0x55e8000) register_command config get hook 0x5588050 -38> 2015-04-28 01:20:09.675807 7f867bb85900 5 asok(0x55e8000) register_command config diff hook 0x5588050 -37> 2015-04-28 01:20:09.675816 7f867bb85900 5 asok(0x55e8000) register_command log flush hook 0x5588050 -36> 2015-04-28 01:20:09.675825 7f867bb85900 5 asok(0x55e8000) register_command log dump hook 0x5588050 -35> 2015-04-28 01:20:09.675833 7f867bb85900 5 asok(0x55e8000) register_command log reopen hook 0x5588050 -34> 2015-04-28 01:20:09.685867 7f867bb85900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 3549 -33> 2015-04-28 01:20:09.763727 7f867bb85900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -32> 2015-04-28 01:20:09.763765 7f867bb85900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6803/3549 need_addr=0 -31> 2015-04-28 01:20:09.763809 7f867bb85900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -30> 2015-04-28 01:20:09.763847 7f867bb85900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6804/3549 need_addr=0 -29> 2015-04-28 01:20:09.763894 7f867bb85900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -28> 2015-04-28 01:20:09.763907 7f867bb85900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6807/3549 need_addr=0 -27> 2015-04-28 01:20:09.763929 7f867bb85900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -26> 2015-04-28 01:20:09.763937 7f867bb85900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6807/3549 need_addr=0 -25> 2015-04-28 01:20:09.766854 7f867bb85900 5 asok(0x55e8000) init /var/run/ceph/ceph-osd.23.asok -24> 2015-04-28 01:20:09.766883 7f867bb85900 5 asok(0x55e8000) bind_and_listen /var/run/ceph/ceph-osd.23.asok -23> 2015-04-28 01:20:09.766939 7f867bb85900 5 asok(0x55e8000) register_command 0 hook 0x55840a8 -22> 2015-04-28 01:20:09.766961 7f867bb85900 5 asok(0x55e8000) register_command version hook 0x55840a8 -21> 2015-04-28 01:20:09.766982 7f867bb85900 5 asok(0x55e8000) register_command git_version hook 0x55840a8 -20> 2015-04-28 01:20:09.767001 7f867bb85900 5 asok(0x55e8000) register_command help hook 0x5588150 -19> 2015-04-28 01:20:09.767021 7f867bb85900 5 asok(0x55e8000) register_command get_command_descriptions hook 0x5588140 -18> 2015-04-28 01:20:09.767131 7f8675c90700 5 asok(0x55e8000) entry start -17> 2015-04-28 01:20:09.767248 7f867bb85900 10 monclient(hunting): build_initial_monmap -16> 2015-04-28 01:20:09.907857 7f867bb85900 5 adding auth protocol: cephx -15> 2015-04-28 01:20:09.907884 7f867bb85900 5 adding auth protocol: cephx -14> 2015-04-28 01:20:09.908360 7f867bb85900 5 asok(0x55e8000) register_command objecter_requests hook 0x5588180 -13> 2015-04-28 01:20:09.908492 7f867bb85900 1 -- 192.168.30.73:6803/3549 messenger.start -12> 2015-04-28 01:20:09.908576 7f867bb85900 1 -- :/0 messenger.start -11> 2015-04-28 01:20:09.908646 7f867bb85900 1 -- 192.168.30.73:6807/3549 messenger.start -10> 2015-04-28 01:20:09.908707 7f867bb85900 1 -- 192.168.32.73:6807/3549 messenger.start -9> 2015-04-28 01:20:09.908775 7f867bb85900 1 -- 192.168.32.73:6804/3549 messenger.start -8> 2015-04-28 01:20:09.908837 7f867bb85900 1 -- :/0 messenger.start -7> 2015-04-28 01:20:09.908960 7f867bb85900 2 osd.23 0 mounting /var/lib/ceph/osd/ceph-23 /var/lib/ceph/osd/ceph-23/journal -6> 2015-04-28 01:20:09.909457 7f867bb85900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. -5> 2015-04-28 01:20:09.909658 7f867bb85900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) -4> 2015-04-28 01:20:09.934609 7f867bb85900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work -3> 2015-04-28 01:20:09.934633 7f867bb85900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option -2> 2015-04-28 01:20:09.935501 7f867bb85900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) -1> 2015-04-28 01:20:09.938816 7f867bb85900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.12-031812-generic >= 3.5 0> 2015-04-28 01:20:10.072699 7f867bb85900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f867bb85900 time 2015-04-28 01:20:10.067550 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f8678cc8ec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 01:20:10.079952 7f867bb85900 -1 *** Caught signal (Aborted) ** in thread 7f867bb85900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f867a83e340] 3: (gsignal()+0x39) [0x7f8678cddcc9] 4: (abort()+0x148) [0x7f8678ce10d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f86795e86b5] 6: (()+0x5e836) [0x7f86795e6836] 7: (()+0x5e863) [0x7f86795e6863] 8: (()+0x5eaa2) [0x7f86795e6aa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f8678cc8ec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2015-04-28 01:20:10.079952 7f867bb85900 -1 *** Caught signal (Aborted) ** in thread 7f867bb85900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f867a83e340] 3: (gsignal()+0x39) [0x7f8678cddcc9] 4: (abort()+0x148) [0x7f8678ce10d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f86795e86b5] 6: (()+0x5e836) [0x7f86795e6836] 7: (()+0x5e863) [0x7f86795e6863] 8: (()+0x5eaa2) [0x7f86795e6aa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f8678cc8ec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 01:20:10.502105 7f72d526a900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 3940 2015-04-28 01:20:10.523752 7f72d526a900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. 2015-04-28 01:20:10.523794 7f72d526a900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) 2015-04-28 01:20:10.525257 7f72d526a900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work 2015-04-28 01:20:10.525279 7f72d526a900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-04-28 01:20:10.526216 7f72d526a900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2015-04-28 01:20:10.526363 7f72d526a900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.12-031812-generic >= 3.5 2015-04-28 01:20:10.540461 7f72d526a900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f72d526a900 time 2015-04-28 01:20:10.535597 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f72d23adec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -48> 2015-04-28 01:20:10.497827 7f72d526a900 5 asok(0x4460000) register_command perfcounters_dump hook 0x4400050 -47> 2015-04-28 01:20:10.497900 7f72d526a900 5 asok(0x4460000) register_command 1 hook 0x4400050 -46> 2015-04-28 01:20:10.497914 7f72d526a900 5 asok(0x4460000) register_command perf dump hook 0x4400050 -45> 2015-04-28 01:20:10.497929 7f72d526a900 5 asok(0x4460000) register_command perfcounters_schema hook 0x4400050 -44> 2015-04-28 01:20:10.497938 7f72d526a900 5 asok(0x4460000) register_command 2 hook 0x4400050 -43> 2015-04-28 01:20:10.497943 7f72d526a900 5 asok(0x4460000) register_command perf schema hook 0x4400050 -42> 2015-04-28 01:20:10.497949 7f72d526a900 5 asok(0x4460000) register_command perf reset hook 0x4400050 -41> 2015-04-28 01:20:10.497971 7f72d526a900 5 asok(0x4460000) register_command config show hook 0x4400050 -40> 2015-04-28 01:20:10.497979 7f72d526a900 5 asok(0x4460000) register_command config set hook 0x4400050 -39> 2015-04-28 01:20:10.497985 7f72d526a900 5 asok(0x4460000) register_command config get hook 0x4400050 -38> 2015-04-28 01:20:10.497989 7f72d526a900 5 asok(0x4460000) register_command config diff hook 0x4400050 -37> 2015-04-28 01:20:10.497993 7f72d526a900 5 asok(0x4460000) register_command log flush hook 0x4400050 -36> 2015-04-28 01:20:10.498003 7f72d526a900 5 asok(0x4460000) register_command log dump hook 0x4400050 -35> 2015-04-28 01:20:10.498011 7f72d526a900 5 asok(0x4460000) register_command log reopen hook 0x4400050 -34> 2015-04-28 01:20:10.502105 7f72d526a900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 3940 -33> 2015-04-28 01:20:10.503607 7f72d526a900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -32> 2015-04-28 01:20:10.503628 7f72d526a900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6804/3940 need_addr=0 -31> 2015-04-28 01:20:10.503656 7f72d526a900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -30> 2015-04-28 01:20:10.503663 7f72d526a900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6804/3940 need_addr=0 -29> 2015-04-28 01:20:10.503683 7f72d526a900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -28> 2015-04-28 01:20:10.503693 7f72d526a900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6805/3940 need_addr=0 -27> 2015-04-28 01:20:10.503713 7f72d526a900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -26> 2015-04-28 01:20:10.503722 7f72d526a900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6805/3940 need_addr=0 -25> 2015-04-28 01:20:10.506162 7f72d526a900 5 asok(0x4460000) init /var/run/ceph/ceph-osd.23.asok -24> 2015-04-28 01:20:10.506184 7f72d526a900 5 asok(0x4460000) bind_and_listen /var/run/ceph/ceph-osd.23.asok -23> 2015-04-28 01:20:10.506359 7f72d526a900 5 asok(0x4460000) register_command 0 hook 0x43fc0a8 -22> 2015-04-28 01:20:10.506372 7f72d526a900 5 asok(0x4460000) register_command version hook 0x43fc0a8 -21> 2015-04-28 01:20:10.506378 7f72d526a900 5 asok(0x4460000) register_command git_version hook 0x43fc0a8 -20> 2015-04-28 01:20:10.506385 7f72d526a900 5 asok(0x4460000) register_command help hook 0x4400150 -19> 2015-04-28 01:20:10.506391 7f72d526a900 5 asok(0x4460000) register_command get_command_descriptions hook 0x4400140 -18> 2015-04-28 01:20:10.506456 7f72d526a900 10 monclient(hunting): build_initial_monmap -17> 2015-04-28 01:20:10.506463 7f72cf375700 5 asok(0x4460000) entry start -16> 2015-04-28 01:20:10.523117 7f72d526a900 5 adding auth protocol: cephx -15> 2015-04-28 01:20:10.523133 7f72d526a900 5 adding auth protocol: cephx -14> 2015-04-28 01:20:10.523283 7f72d526a900 5 asok(0x4460000) register_command objecter_requests hook 0x4400180 -13> 2015-04-28 01:20:10.523388 7f72d526a900 1 -- 192.168.30.73:6804/3940 messenger.start -12> 2015-04-28 01:20:10.523441 7f72d526a900 1 -- :/0 messenger.start -11> 2015-04-28 01:20:10.523475 7f72d526a900 1 -- 192.168.30.73:6805/3940 messenger.start -10> 2015-04-28 01:20:10.523513 7f72d526a900 1 -- 192.168.32.73:6805/3940 messenger.start -9> 2015-04-28 01:20:10.523554 7f72d526a900 1 -- 192.168.32.73:6804/3940 messenger.start -8> 2015-04-28 01:20:10.523590 7f72d526a900 1 -- :/0 messenger.start -7> 2015-04-28 01:20:10.523679 7f72d526a900 2 osd.23 0 mounting /var/lib/ceph/osd/ceph-23 /var/lib/ceph/osd/ceph-23/journal -6> 2015-04-28 01:20:10.523752 7f72d526a900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. -5> 2015-04-28 01:20:10.523794 7f72d526a900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) -4> 2015-04-28 01:20:10.525257 7f72d526a900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work -3> 2015-04-28 01:20:10.525279 7f72d526a900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option -2> 2015-04-28 01:20:10.526216 7f72d526a900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) -1> 2015-04-28 01:20:10.526363 7f72d526a900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.12-031812-generic >= 3.5 0> 2015-04-28 01:20:10.540461 7f72d526a900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f72d526a900 time 2015-04-28 01:20:10.535597 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f72d23adec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 01:20:10.547340 7f72d526a900 -1 *** Caught signal (Aborted) ** in thread 7f72d526a900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f72d3f23340] 3: (gsignal()+0x39) [0x7f72d23c2cc9] 4: (abort()+0x148) [0x7f72d23c60d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f72d2ccd6b5] 6: (()+0x5e836) [0x7f72d2ccb836] 7: (()+0x5e863) [0x7f72d2ccb863] 8: (()+0x5eaa2) [0x7f72d2ccbaa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f72d23adec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2015-04-28 01:20:10.547340 7f72d526a900 -1 *** Caught signal (Aborted) ** in thread 7f72d526a900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f72d3f23340] 3: (gsignal()+0x39) [0x7f72d23c2cc9] 4: (abort()+0x148) [0x7f72d23c60d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f72d2ccd6b5] 6: (()+0x5e836) [0x7f72d2ccb836] 7: (()+0x5e863) [0x7f72d2ccb863] 8: (()+0x5eaa2) [0x7f72d2ccbaa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f72d23adec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 01:20:13.964669 7f0584fc4900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 5018 2015-04-28 01:20:13.986549 7f0584fc4900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. 2015-04-28 01:20:13.986595 7f0584fc4900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) 2015-04-28 01:20:13.988160 7f0584fc4900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work 2015-04-28 01:20:13.988177 7f0584fc4900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-04-28 01:20:13.989013 7f0584fc4900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2015-04-28 01:20:13.989173 7f0584fc4900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.12-031812-generic >= 3.5 2015-04-28 01:20:13.998993 7f0584fc4900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f0584fc4900 time 2015-04-28 01:20:13.994123 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f0582107ec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -48> 2015-04-28 01:20:13.960066 7f0584fc4900 5 asok(0x4ed4000) register_command perfcounters_dump hook 0x4e74050 -47> 2015-04-28 01:20:13.960142 7f0584fc4900 5 asok(0x4ed4000) register_command 1 hook 0x4e74050 -46> 2015-04-28 01:20:13.960175 7f0584fc4900 5 asok(0x4ed4000) register_command perf dump hook 0x4e74050 -45> 2015-04-28 01:20:13.960191 7f0584fc4900 5 asok(0x4ed4000) register_command perfcounters_schema hook 0x4e74050 -44> 2015-04-28 01:20:13.960200 7f0584fc4900 5 asok(0x4ed4000) register_command 2 hook 0x4e74050 -43> 2015-04-28 01:20:13.960204 7f0584fc4900 5 asok(0x4ed4000) register_command perf schema hook 0x4e74050 -42> 2015-04-28 01:20:13.960210 7f0584fc4900 5 asok(0x4ed4000) register_command perf reset hook 0x4e74050 -41> 2015-04-28 01:20:13.960232 7f0584fc4900 5 asok(0x4ed4000) register_command config show hook 0x4e74050 -40> 2015-04-28 01:20:13.960241 7f0584fc4900 5 asok(0x4ed4000) register_command config set hook 0x4e74050 -39> 2015-04-28 01:20:13.960246 7f0584fc4900 5 asok(0x4ed4000) register_command config get hook 0x4e74050 -38> 2015-04-28 01:20:13.960250 7f0584fc4900 5 asok(0x4ed4000) register_command config diff hook 0x4e74050 -37> 2015-04-28 01:20:13.960254 7f0584fc4900 5 asok(0x4ed4000) register_command log flush hook 0x4e74050 -36> 2015-04-28 01:20:13.960263 7f0584fc4900 5 asok(0x4ed4000) register_command log dump hook 0x4e74050 -35> 2015-04-28 01:20:13.960269 7f0584fc4900 5 asok(0x4ed4000) register_command log reopen hook 0x4e74050 -34> 2015-04-28 01:20:13.964669 7f0584fc4900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 5018 -33> 2015-04-28 01:20:13.966217 7f0584fc4900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -32> 2015-04-28 01:20:13.966239 7f0584fc4900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6800/5018 need_addr=0 -31> 2015-04-28 01:20:13.966264 7f0584fc4900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -30> 2015-04-28 01:20:13.966270 7f0584fc4900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6800/5018 need_addr=0 -29> 2015-04-28 01:20:13.966286 7f0584fc4900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -28> 2015-04-28 01:20:13.966297 7f0584fc4900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6801/5018 need_addr=0 -27> 2015-04-28 01:20:13.966315 7f0584fc4900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -26> 2015-04-28 01:20:13.966324 7f0584fc4900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6801/5018 need_addr=0 -25> 2015-04-28 01:20:13.968763 7f0584fc4900 5 asok(0x4ed4000) init /var/run/ceph/ceph-osd.23.asok -24> 2015-04-28 01:20:13.968787 7f0584fc4900 5 asok(0x4ed4000) bind_and_listen /var/run/ceph/ceph-osd.23.asok -23> 2015-04-28 01:20:13.968967 7f0584fc4900 5 asok(0x4ed4000) register_command 0 hook 0x4e700a8 -22> 2015-04-28 01:20:13.968982 7f0584fc4900 5 asok(0x4ed4000) register_command version hook 0x4e700a8 -21> 2015-04-28 01:20:13.968988 7f0584fc4900 5 asok(0x4ed4000) register_command git_version hook 0x4e700a8 -20> 2015-04-28 01:20:13.968998 7f0584fc4900 5 asok(0x4ed4000) register_command help hook 0x4e74150 -19> 2015-04-28 01:20:13.969005 7f0584fc4900 5 asok(0x4ed4000) register_command get_command_descriptions hook 0x4e74140 -18> 2015-04-28 01:20:13.969074 7f0584fc4900 10 monclient(hunting): build_initial_monmap -17> 2015-04-28 01:20:13.969068 7f057f0cf700 5 asok(0x4ed4000) entry start -16> 2015-04-28 01:20:13.985872 7f0584fc4900 5 adding auth protocol: cephx -15> 2015-04-28 01:20:13.985888 7f0584fc4900 5 adding auth protocol: cephx -14> 2015-04-28 01:20:13.986052 7f0584fc4900 5 asok(0x4ed4000) register_command objecter_requests hook 0x4e74180 -13> 2015-04-28 01:20:13.986162 7f0584fc4900 1 -- 192.168.30.73:6800/5018 messenger.start -12> 2015-04-28 01:20:13.986220 7f0584fc4900 1 -- :/0 messenger.start -11> 2015-04-28 01:20:13.986259 7f0584fc4900 1 -- 192.168.30.73:6801/5018 messenger.start -10> 2015-04-28 01:20:13.986308 7f0584fc4900 1 -- 192.168.32.73:6801/5018 messenger.start -9> 2015-04-28 01:20:13.986347 7f0584fc4900 1 -- 192.168.32.73:6800/5018 messenger.start -8> 2015-04-28 01:20:13.986387 7f0584fc4900 1 -- :/0 messenger.start -7> 2015-04-28 01:20:13.986483 7f0584fc4900 2 osd.23 0 mounting /var/lib/ceph/osd/ceph-23 /var/lib/ceph/osd/ceph-23/journal -6> 2015-04-28 01:20:13.986549 7f0584fc4900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. -5> 2015-04-28 01:20:13.986595 7f0584fc4900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) -4> 2015-04-28 01:20:13.988160 7f0584fc4900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work -3> 2015-04-28 01:20:13.988177 7f0584fc4900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option -2> 2015-04-28 01:20:13.989013 7f0584fc4900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) -1> 2015-04-28 01:20:13.989173 7f0584fc4900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.12-031812-generic >= 3.5 0> 2015-04-28 01:20:13.998993 7f0584fc4900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f0584fc4900 time 2015-04-28 01:20:13.994123 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f0582107ec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 01:20:14.005767 7f0584fc4900 -1 *** Caught signal (Aborted) ** in thread 7f0584fc4900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f0583c7d340] 3: (gsignal()+0x39) [0x7f058211ccc9] 4: (abort()+0x148) [0x7f05821200d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f0582a276b5] 6: (()+0x5e836) [0x7f0582a25836] 7: (()+0x5e863) [0x7f0582a25863] 8: (()+0x5eaa2) [0x7f0582a25aa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f0582107ec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2015-04-28 01:20:14.005767 7f0584fc4900 -1 *** Caught signal (Aborted) ** in thread 7f0584fc4900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f0583c7d340] 3: (gsignal()+0x39) [0x7f058211ccc9] 4: (abort()+0x148) [0x7f05821200d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f0582a276b5] 6: (()+0x5e836) [0x7f0582a25836] 7: (()+0x5e863) [0x7f0582a25863] 8: (()+0x5eaa2) [0x7f0582a25aa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f0582107ec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 01:20:14.416460 7f8c1cfcc900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 5296 2015-04-28 01:20:14.438506 7f8c1cfcc900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. 2015-04-28 01:20:14.438556 7f8c1cfcc900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) 2015-04-28 01:20:14.440086 7f8c1cfcc900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work 2015-04-28 01:20:14.440106 7f8c1cfcc900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-04-28 01:20:14.441139 7f8c1cfcc900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2015-04-28 01:20:14.441279 7f8c1cfcc900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.12-031812-generic >= 3.5 2015-04-28 01:20:14.450947 7f8c1cfcc900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f8c1cfcc900 time 2015-04-28 01:20:14.446102 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f8c1a10fec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -48> 2015-04-28 01:20:14.412131 7f8c1cfcc900 5 asok(0x56ec000) register_command perfcounters_dump hook 0x568c050 -47> 2015-04-28 01:20:14.412210 7f8c1cfcc900 5 asok(0x56ec000) register_command 1 hook 0x568c050 -46> 2015-04-28 01:20:14.412222 7f8c1cfcc900 5 asok(0x56ec000) register_command perf dump hook 0x568c050 -45> 2015-04-28 01:20:14.412233 7f8c1cfcc900 5 asok(0x56ec000) register_command perfcounters_schema hook 0x568c050 -44> 2015-04-28 01:20:14.412242 7f8c1cfcc900 5 asok(0x56ec000) register_command 2 hook 0x568c050 -43> 2015-04-28 01:20:14.412246 7f8c1cfcc900 5 asok(0x56ec000) register_command perf schema hook 0x568c050 -42> 2015-04-28 01:20:14.412256 7f8c1cfcc900 5 asok(0x56ec000) register_command perf reset hook 0x568c050 -41> 2015-04-28 01:20:14.412279 7f8c1cfcc900 5 asok(0x56ec000) register_command config show hook 0x568c050 -40> 2015-04-28 01:20:14.412290 7f8c1cfcc900 5 asok(0x56ec000) register_command config set hook 0x568c050 -39> 2015-04-28 01:20:14.412302 7f8c1cfcc900 5 asok(0x56ec000) register_command config get hook 0x568c050 -38> 2015-04-28 01:20:14.412306 7f8c1cfcc900 5 asok(0x56ec000) register_command config diff hook 0x568c050 -37> 2015-04-28 01:20:14.412315 7f8c1cfcc900 5 asok(0x56ec000) register_command log flush hook 0x568c050 -36> 2015-04-28 01:20:14.412320 7f8c1cfcc900 5 asok(0x56ec000) register_command log dump hook 0x568c050 -35> 2015-04-28 01:20:14.412329 7f8c1cfcc900 5 asok(0x56ec000) register_command log reopen hook 0x568c050 -34> 2015-04-28 01:20:14.416460 7f8c1cfcc900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 5296 -33> 2015-04-28 01:20:14.417991 7f8c1cfcc900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -32> 2015-04-28 01:20:14.418011 7f8c1cfcc900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6800/5296 need_addr=0 -31> 2015-04-28 01:20:14.418036 7f8c1cfcc900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -30> 2015-04-28 01:20:14.418047 7f8c1cfcc900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6800/5296 need_addr=0 -29> 2015-04-28 01:20:14.418064 7f8c1cfcc900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -28> 2015-04-28 01:20:14.418074 7f8c1cfcc900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6801/5296 need_addr=0 -27> 2015-04-28 01:20:14.418091 7f8c1cfcc900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -26> 2015-04-28 01:20:14.418101 7f8c1cfcc900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6801/5296 need_addr=0 -25> 2015-04-28 01:20:14.420534 7f8c1cfcc900 5 asok(0x56ec000) init /var/run/ceph/ceph-osd.23.asok -24> 2015-04-28 01:20:14.420556 7f8c1cfcc900 5 asok(0x56ec000) bind_and_listen /var/run/ceph/ceph-osd.23.asok -23> 2015-04-28 01:20:14.420739 7f8c1cfcc900 5 asok(0x56ec000) register_command 0 hook 0x56880a8 -22> 2015-04-28 01:20:14.420752 7f8c1cfcc900 5 asok(0x56ec000) register_command version hook 0x56880a8 -21> 2015-04-28 01:20:14.420759 7f8c1cfcc900 5 asok(0x56ec000) register_command git_version hook 0x56880a8 -20> 2015-04-28 01:20:14.420770 7f8c1cfcc900 5 asok(0x56ec000) register_command help hook 0x568c150 -19> 2015-04-28 01:20:14.420780 7f8c1cfcc900 5 asok(0x56ec000) register_command get_command_descriptions hook 0x568c140 -18> 2015-04-28 01:20:14.420854 7f8c1cfcc900 10 monclient(hunting): build_initial_monmap -17> 2015-04-28 01:20:14.420886 7f8c170d7700 5 asok(0x56ec000) entry start -16> 2015-04-28 01:20:14.437803 7f8c1cfcc900 5 adding auth protocol: cephx -15> 2015-04-28 01:20:14.437822 7f8c1cfcc900 5 adding auth protocol: cephx -14> 2015-04-28 01:20:14.437998 7f8c1cfcc900 5 asok(0x56ec000) register_command objecter_requests hook 0x568c180 -13> 2015-04-28 01:20:14.438117 7f8c1cfcc900 1 -- 192.168.30.73:6800/5296 messenger.start -12> 2015-04-28 01:20:14.438177 7f8c1cfcc900 1 -- :/0 messenger.start -11> 2015-04-28 01:20:14.438219 7f8c1cfcc900 1 -- 192.168.30.73:6801/5296 messenger.start -10> 2015-04-28 01:20:14.438262 7f8c1cfcc900 1 -- 192.168.32.73:6801/5296 messenger.start -9> 2015-04-28 01:20:14.438304 7f8c1cfcc900 1 -- 192.168.32.73:6800/5296 messenger.start -8> 2015-04-28 01:20:14.438343 7f8c1cfcc900 1 -- :/0 messenger.start -7> 2015-04-28 01:20:14.438436 7f8c1cfcc900 2 osd.23 0 mounting /var/lib/ceph/osd/ceph-23 /var/lib/ceph/osd/ceph-23/journal -6> 2015-04-28 01:20:14.438506 7f8c1cfcc900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. -5> 2015-04-28 01:20:14.438556 7f8c1cfcc900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) -4> 2015-04-28 01:20:14.440086 7f8c1cfcc900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work -3> 2015-04-28 01:20:14.440106 7f8c1cfcc900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option -2> 2015-04-28 01:20:14.441139 7f8c1cfcc900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) -1> 2015-04-28 01:20:14.441279 7f8c1cfcc900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.12-031812-generic >= 3.5 0> 2015-04-28 01:20:14.450947 7f8c1cfcc900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f8c1cfcc900 time 2015-04-28 01:20:14.446102 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f8c1a10fec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 01:20:14.457761 7f8c1cfcc900 -1 *** Caught signal (Aborted) ** in thread 7f8c1cfcc900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f8c1bc85340] 3: (gsignal()+0x39) [0x7f8c1a124cc9] 4: (abort()+0x148) [0x7f8c1a1280d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f8c1aa2f6b5] 6: (()+0x5e836) [0x7f8c1aa2d836] 7: (()+0x5e863) [0x7f8c1aa2d863] 8: (()+0x5eaa2) [0x7f8c1aa2daa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f8c1a10fec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2015-04-28 01:20:14.457761 7f8c1cfcc900 -1 *** Caught signal (Aborted) ** in thread 7f8c1cfcc900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f8c1bc85340] 3: (gsignal()+0x39) [0x7f8c1a124cc9] 4: (abort()+0x148) [0x7f8c1a1280d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f8c1aa2f6b5] 6: (()+0x5e836) [0x7f8c1aa2d836] 7: (()+0x5e863) [0x7f8c1aa2d863] 8: (()+0x5eaa2) [0x7f8c1aa2daa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f8c1a10fec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 01:20:14.895357 7fc0d192e900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 5537 2015-04-28 01:20:14.917094 7fc0d192e900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. 2015-04-28 01:20:14.917141 7fc0d192e900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) 2015-04-28 01:20:14.918724 7fc0d192e900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work 2015-04-28 01:20:14.918744 7fc0d192e900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-04-28 01:20:14.919684 7fc0d192e900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2015-04-28 01:20:14.919848 7fc0d192e900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.12-031812-generic >= 3.5 2015-04-28 01:20:14.929754 7fc0d192e900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7fc0d192e900 time 2015-04-28 01:20:14.924873 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7fc0cea71ec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -48> 2015-04-28 01:20:14.890999 7fc0d192e900 5 asok(0x4fea000) register_command perfcounters_dump hook 0x4f8a050 -47> 2015-04-28 01:20:14.891066 7fc0d192e900 5 asok(0x4fea000) register_command 1 hook 0x4f8a050 -46> 2015-04-28 01:20:14.891082 7fc0d192e900 5 asok(0x4fea000) register_command perf dump hook 0x4f8a050 -45> 2015-04-28 01:20:14.891095 7fc0d192e900 5 asok(0x4fea000) register_command perfcounters_schema hook 0x4f8a050 -44> 2015-04-28 01:20:14.891100 7fc0d192e900 5 asok(0x4fea000) register_command 2 hook 0x4f8a050 -43> 2015-04-28 01:20:14.891105 7fc0d192e900 5 asok(0x4fea000) register_command perf schema hook 0x4f8a050 -42> 2015-04-28 01:20:14.891111 7fc0d192e900 5 asok(0x4fea000) register_command perf reset hook 0x4f8a050 -41> 2015-04-28 01:20:14.891129 7fc0d192e900 5 asok(0x4fea000) register_command config show hook 0x4f8a050 -40> 2015-04-28 01:20:14.891141 7fc0d192e900 5 asok(0x4fea000) register_command config set hook 0x4f8a050 -39> 2015-04-28 01:20:14.891149 7fc0d192e900 5 asok(0x4fea000) register_command config get hook 0x4f8a050 -38> 2015-04-28 01:20:14.891157 7fc0d192e900 5 asok(0x4fea000) register_command config diff hook 0x4f8a050 -37> 2015-04-28 01:20:14.891165 7fc0d192e900 5 asok(0x4fea000) register_command log flush hook 0x4f8a050 -36> 2015-04-28 01:20:14.891174 7fc0d192e900 5 asok(0x4fea000) register_command log dump hook 0x4f8a050 -35> 2015-04-28 01:20:14.891183 7fc0d192e900 5 asok(0x4fea000) register_command log reopen hook 0x4f8a050 -34> 2015-04-28 01:20:14.895357 7fc0d192e900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 5537 -33> 2015-04-28 01:20:14.896900 7fc0d192e900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -32> 2015-04-28 01:20:14.896921 7fc0d192e900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6800/5537 need_addr=0 -31> 2015-04-28 01:20:14.896946 7fc0d192e900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -30> 2015-04-28 01:20:14.896953 7fc0d192e900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6800/5537 need_addr=0 -29> 2015-04-28 01:20:14.896970 7fc0d192e900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -28> 2015-04-28 01:20:14.896980 7fc0d192e900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6801/5537 need_addr=0 -27> 2015-04-28 01:20:14.896997 7fc0d192e900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -26> 2015-04-28 01:20:14.897007 7fc0d192e900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6801/5537 need_addr=0 -25> 2015-04-28 01:20:14.899391 7fc0d192e900 5 asok(0x4fea000) init /var/run/ceph/ceph-osd.23.asok -24> 2015-04-28 01:20:14.899413 7fc0d192e900 5 asok(0x4fea000) bind_and_listen /var/run/ceph/ceph-osd.23.asok -23> 2015-04-28 01:20:14.899580 7fc0d192e900 5 asok(0x4fea000) register_command 0 hook 0x4f860a8 -22> 2015-04-28 01:20:14.899594 7fc0d192e900 5 asok(0x4fea000) register_command version hook 0x4f860a8 -21> 2015-04-28 01:20:14.899600 7fc0d192e900 5 asok(0x4fea000) register_command git_version hook 0x4f860a8 -20> 2015-04-28 01:20:14.899607 7fc0d192e900 5 asok(0x4fea000) register_command help hook 0x4f8a150 -19> 2015-04-28 01:20:14.899613 7fc0d192e900 5 asok(0x4fea000) register_command get_command_descriptions hook 0x4f8a140 -18> 2015-04-28 01:20:14.899680 7fc0d192e900 10 monclient(hunting): build_initial_monmap -17> 2015-04-28 01:20:14.899709 7fc0cba39700 5 asok(0x4fea000) entry start -16> 2015-04-28 01:20:14.916396 7fc0d192e900 5 adding auth protocol: cephx -15> 2015-04-28 01:20:14.916413 7fc0d192e900 5 adding auth protocol: cephx -14> 2015-04-28 01:20:14.916573 7fc0d192e900 5 asok(0x4fea000) register_command objecter_requests hook 0x4f8a180 -13> 2015-04-28 01:20:14.916684 7fc0d192e900 1 -- 192.168.30.73:6800/5537 messenger.start -12> 2015-04-28 01:20:14.916742 7fc0d192e900 1 -- :/0 messenger.start -11> 2015-04-28 01:20:14.916790 7fc0d192e900 1 -- 192.168.30.73:6801/5537 messenger.start -10> 2015-04-28 01:20:14.916834 7fc0d192e900 1 -- 192.168.32.73:6801/5537 messenger.start -9> 2015-04-28 01:20:14.916873 7fc0d192e900 1 -- 192.168.32.73:6800/5537 messenger.start -8> 2015-04-28 01:20:14.916914 7fc0d192e900 1 -- :/0 messenger.start -7> 2015-04-28 01:20:14.917022 7fc0d192e900 2 osd.23 0 mounting /var/lib/ceph/osd/ceph-23 /var/lib/ceph/osd/ceph-23/journal -6> 2015-04-28 01:20:14.917094 7fc0d192e900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. -5> 2015-04-28 01:20:14.917141 7fc0d192e900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) -4> 2015-04-28 01:20:14.918724 7fc0d192e900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work -3> 2015-04-28 01:20:14.918744 7fc0d192e900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option -2> 2015-04-28 01:20:14.919684 7fc0d192e900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) -1> 2015-04-28 01:20:14.919848 7fc0d192e900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.12-031812-generic >= 3.5 0> 2015-04-28 01:20:14.929754 7fc0d192e900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7fc0d192e900 time 2015-04-28 01:20:14.924873 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7fc0cea71ec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 01:20:14.936638 7fc0d192e900 -1 *** Caught signal (Aborted) ** in thread 7fc0d192e900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7fc0d05e7340] 3: (gsignal()+0x39) [0x7fc0cea86cc9] 4: (abort()+0x148) [0x7fc0cea8a0d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7fc0cf3916b5] 6: (()+0x5e836) [0x7fc0cf38f836] 7: (()+0x5e863) [0x7fc0cf38f863] 8: (()+0x5eaa2) [0x7fc0cf38faa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7fc0cea71ec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2015-04-28 01:20:14.936638 7fc0d192e900 -1 *** Caught signal (Aborted) ** in thread 7fc0d192e900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7fc0d05e7340] 3: (gsignal()+0x39) [0x7fc0cea86cc9] 4: (abort()+0x148) [0x7fc0cea8a0d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7fc0cf3916b5] 6: (()+0x5e836) [0x7fc0cf38f836] 7: (()+0x5e863) [0x7fc0cf38f863] 8: (()+0x5eaa2) [0x7fc0cf38faa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7fc0cea71ec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 01:20:15.356344 7fea4aaa2900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 5751 2015-04-28 01:20:15.377884 7fea4aaa2900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. 2015-04-28 01:20:15.377930 7fea4aaa2900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) 2015-04-28 01:20:15.379565 7fea4aaa2900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work 2015-04-28 01:20:15.379585 7fea4aaa2900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-04-28 01:20:15.380657 7fea4aaa2900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2015-04-28 01:20:15.380806 7fea4aaa2900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.12-031812-generic >= 3.5 2015-04-28 01:20:15.390638 7fea4aaa2900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7fea4aaa2900 time 2015-04-28 01:20:15.385784 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7fea47be5ec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -48> 2015-04-28 01:20:15.352204 7fea4aaa2900 5 asok(0x4782000) register_command perfcounters_dump hook 0x4722050 -47> 2015-04-28 01:20:15.352259 7fea4aaa2900 5 asok(0x4782000) register_command 1 hook 0x4722050 -46> 2015-04-28 01:20:15.352274 7fea4aaa2900 5 asok(0x4782000) register_command perf dump hook 0x4722050 -45> 2015-04-28 01:20:15.352286 7fea4aaa2900 5 asok(0x4782000) register_command perfcounters_schema hook 0x4722050 -44> 2015-04-28 01:20:15.352295 7fea4aaa2900 5 asok(0x4782000) register_command 2 hook 0x4722050 -43> 2015-04-28 01:20:15.352299 7fea4aaa2900 5 asok(0x4782000) register_command perf schema hook 0x4722050 -42> 2015-04-28 01:20:15.352307 7fea4aaa2900 5 asok(0x4782000) register_command perf reset hook 0x4722050 -41> 2015-04-28 01:20:15.352321 7fea4aaa2900 5 asok(0x4782000) register_command config show hook 0x4722050 -40> 2015-04-28 01:20:15.352329 7fea4aaa2900 5 asok(0x4782000) register_command config set hook 0x4722050 -39> 2015-04-28 01:20:15.352338 7fea4aaa2900 5 asok(0x4782000) register_command config get hook 0x4722050 -38> 2015-04-28 01:20:15.352343 7fea4aaa2900 5 asok(0x4782000) register_command config diff hook 0x4722050 -37> 2015-04-28 01:20:15.352348 7fea4aaa2900 5 asok(0x4782000) register_command log flush hook 0x4722050 -36> 2015-04-28 01:20:15.352357 7fea4aaa2900 5 asok(0x4782000) register_command log dump hook 0x4722050 -35> 2015-04-28 01:20:15.352362 7fea4aaa2900 5 asok(0x4782000) register_command log reopen hook 0x4722050 -34> 2015-04-28 01:20:15.356344 7fea4aaa2900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 5751 -33> 2015-04-28 01:20:15.357820 7fea4aaa2900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -32> 2015-04-28 01:20:15.357841 7fea4aaa2900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6800/5751 need_addr=0 -31> 2015-04-28 01:20:15.357867 7fea4aaa2900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -30> 2015-04-28 01:20:15.357873 7fea4aaa2900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6800/5751 need_addr=0 -29> 2015-04-28 01:20:15.357890 7fea4aaa2900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -28> 2015-04-28 01:20:15.357902 7fea4aaa2900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6801/5751 need_addr=0 -27> 2015-04-28 01:20:15.357920 7fea4aaa2900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -26> 2015-04-28 01:20:15.357930 7fea4aaa2900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6801/5751 need_addr=0 -25> 2015-04-28 01:20:15.360260 7fea4aaa2900 5 asok(0x4782000) init /var/run/ceph/ceph-osd.23.asok -24> 2015-04-28 01:20:15.360282 7fea4aaa2900 5 asok(0x4782000) bind_and_listen /var/run/ceph/ceph-osd.23.asok -23> 2015-04-28 01:20:15.360415 7fea4aaa2900 5 asok(0x4782000) register_command 0 hook 0x471e0a8 -22> 2015-04-28 01:20:15.360429 7fea4aaa2900 5 asok(0x4782000) register_command version hook 0x471e0a8 -21> 2015-04-28 01:20:15.360435 7fea4aaa2900 5 asok(0x4782000) register_command git_version hook 0x471e0a8 -20> 2015-04-28 01:20:15.360441 7fea4aaa2900 5 asok(0x4782000) register_command help hook 0x4722150 -19> 2015-04-28 01:20:15.360446 7fea4aaa2900 5 asok(0x4782000) register_command get_command_descriptions hook 0x4722140 -18> 2015-04-28 01:20:15.360516 7fea4aaa2900 10 monclient(hunting): build_initial_monmap -17> 2015-04-28 01:20:15.360545 7fea44bad700 5 asok(0x4782000) entry start -16> 2015-04-28 01:20:15.377206 7fea4aaa2900 5 adding auth protocol: cephx -15> 2015-04-28 01:20:15.377221 7fea4aaa2900 5 adding auth protocol: cephx -14> 2015-04-28 01:20:15.377387 7fea4aaa2900 5 asok(0x4782000) register_command objecter_requests hook 0x4722180 -13> 2015-04-28 01:20:15.377496 7fea4aaa2900 1 -- 192.168.30.73:6800/5751 messenger.start -12> 2015-04-28 01:20:15.377557 7fea4aaa2900 1 -- :/0 messenger.start -11> 2015-04-28 01:20:15.377594 7fea4aaa2900 1 -- 192.168.30.73:6801/5751 messenger.start -10> 2015-04-28 01:20:15.377643 7fea4aaa2900 1 -- 192.168.32.73:6801/5751 messenger.start -9> 2015-04-28 01:20:15.377680 7fea4aaa2900 1 -- 192.168.32.73:6800/5751 messenger.start -8> 2015-04-28 01:20:15.377719 7fea4aaa2900 1 -- :/0 messenger.start -7> 2015-04-28 01:20:15.377816 7fea4aaa2900 2 osd.23 0 mounting /var/lib/ceph/osd/ceph-23 /var/lib/ceph/osd/ceph-23/journal -6> 2015-04-28 01:20:15.377884 7fea4aaa2900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. -5> 2015-04-28 01:20:15.377930 7fea4aaa2900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) -4> 2015-04-28 01:20:15.379565 7fea4aaa2900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work -3> 2015-04-28 01:20:15.379585 7fea4aaa2900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option -2> 2015-04-28 01:20:15.380657 7fea4aaa2900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) -1> 2015-04-28 01:20:15.380806 7fea4aaa2900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.12-031812-generic >= 3.5 0> 2015-04-28 01:20:15.390638 7fea4aaa2900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7fea4aaa2900 time 2015-04-28 01:20:15.385784 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7fea47be5ec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 01:20:15.397467 7fea4aaa2900 -1 *** Caught signal (Aborted) ** in thread 7fea4aaa2900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7fea4975b340] 3: (gsignal()+0x39) [0x7fea47bfacc9] 4: (abort()+0x148) [0x7fea47bfe0d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7fea485056b5] 6: (()+0x5e836) [0x7fea48503836] 7: (()+0x5e863) [0x7fea48503863] 8: (()+0x5eaa2) [0x7fea48503aa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7fea47be5ec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2015-04-28 01:20:15.397467 7fea4aaa2900 -1 *** Caught signal (Aborted) ** in thread 7fea4aaa2900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7fea4975b340] 3: (gsignal()+0x39) [0x7fea47bfacc9] 4: (abort()+0x148) [0x7fea47bfe0d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7fea485056b5] 6: (()+0x5e836) [0x7fea48503836] 7: (()+0x5e863) [0x7fea48503863] 8: (()+0x5eaa2) [0x7fea48503aa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7fea47be5ec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 01:22:04.409349 7f7db081b900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 12697 2015-04-28 01:22:04.431991 7f7db081b900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. 2015-04-28 01:22:04.432032 7f7db081b900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) 2015-04-28 01:22:04.433579 7f7db081b900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work 2015-04-28 01:22:04.433599 7f7db081b900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-04-28 01:22:04.434231 7f7db081b900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2015-04-28 01:22:04.434367 7f7db081b900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.12-031812-generic >= 3.5 2015-04-28 01:22:04.443809 7f7db081b900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f7db081b900 time 2015-04-28 01:22:04.438955 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f7dad95eec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -48> 2015-04-28 01:22:04.404840 7f7db081b900 5 asok(0x38c6000) register_command perfcounters_dump hook 0x3866050 -47> 2015-04-28 01:22:04.404916 7f7db081b900 5 asok(0x38c6000) register_command 1 hook 0x3866050 -46> 2015-04-28 01:22:04.404926 7f7db081b900 5 asok(0x38c6000) register_command perf dump hook 0x3866050 -45> 2015-04-28 01:22:04.404941 7f7db081b900 5 asok(0x38c6000) register_command perfcounters_schema hook 0x3866050 -44> 2015-04-28 01:22:04.404947 7f7db081b900 5 asok(0x38c6000) register_command 2 hook 0x3866050 -43> 2015-04-28 01:22:04.404951 7f7db081b900 5 asok(0x38c6000) register_command perf schema hook 0x3866050 -42> 2015-04-28 01:22:04.404957 7f7db081b900 5 asok(0x38c6000) register_command perf reset hook 0x3866050 -41> 2015-04-28 01:22:04.404981 7f7db081b900 5 asok(0x38c6000) register_command config show hook 0x3866050 -40> 2015-04-28 01:22:04.404991 7f7db081b900 5 asok(0x38c6000) register_command config set hook 0x3866050 -39> 2015-04-28 01:22:04.404997 7f7db081b900 5 asok(0x38c6000) register_command config get hook 0x3866050 -38> 2015-04-28 01:22:04.405001 7f7db081b900 5 asok(0x38c6000) register_command config diff hook 0x3866050 -37> 2015-04-28 01:22:04.405006 7f7db081b900 5 asok(0x38c6000) register_command log flush hook 0x3866050 -36> 2015-04-28 01:22:04.405012 7f7db081b900 5 asok(0x38c6000) register_command log dump hook 0x3866050 -35> 2015-04-28 01:22:04.405026 7f7db081b900 5 asok(0x38c6000) register_command log reopen hook 0x3866050 -34> 2015-04-28 01:22:04.409349 7f7db081b900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 12697 -33> 2015-04-28 01:22:04.411087 7f7db081b900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -32> 2015-04-28 01:22:04.411111 7f7db081b900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6802/12697 need_addr=0 -31> 2015-04-28 01:22:04.411140 7f7db081b900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -30> 2015-04-28 01:22:04.411150 7f7db081b900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6802/12697 need_addr=0 -29> 2015-04-28 01:22:04.411169 7f7db081b900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -28> 2015-04-28 01:22:04.411178 7f7db081b900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6803/12697 need_addr=0 -27> 2015-04-28 01:22:04.411196 7f7db081b900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -26> 2015-04-28 01:22:04.411207 7f7db081b900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6803/12697 need_addr=0 -25> 2015-04-28 01:22:04.413641 7f7db081b900 5 asok(0x38c6000) init /var/run/ceph/ceph-osd.23.asok -24> 2015-04-28 01:22:04.413665 7f7db081b900 5 asok(0x38c6000) bind_and_listen /var/run/ceph/ceph-osd.23.asok -23> 2015-04-28 01:22:04.413844 7f7db081b900 5 asok(0x38c6000) register_command 0 hook 0x38620a8 -22> 2015-04-28 01:22:04.413862 7f7db081b900 5 asok(0x38c6000) register_command version hook 0x38620a8 -21> 2015-04-28 01:22:04.413872 7f7db081b900 5 asok(0x38c6000) register_command git_version hook 0x38620a8 -20> 2015-04-28 01:22:04.413884 7f7db081b900 5 asok(0x38c6000) register_command help hook 0x3866150 -19> 2015-04-28 01:22:04.413893 7f7db081b900 5 asok(0x38c6000) register_command get_command_descriptions hook 0x3866140 -18> 2015-04-28 01:22:04.413955 7f7daa926700 5 asok(0x38c6000) entry start -17> 2015-04-28 01:22:04.413964 7f7db081b900 10 monclient(hunting): build_initial_monmap -16> 2015-04-28 01:22:04.431244 7f7db081b900 5 adding auth protocol: cephx -15> 2015-04-28 01:22:04.431261 7f7db081b900 5 adding auth protocol: cephx -14> 2015-04-28 01:22:04.431436 7f7db081b900 5 asok(0x38c6000) register_command objecter_requests hook 0x3866180 -13> 2015-04-28 01:22:04.431566 7f7db081b900 1 -- 192.168.30.73:6802/12697 messenger.start -12> 2015-04-28 01:22:04.431629 7f7db081b900 1 -- :/0 messenger.start -11> 2015-04-28 01:22:04.431671 7f7db081b900 1 -- 192.168.30.73:6803/12697 messenger.start -10> 2015-04-28 01:22:04.431712 7f7db081b900 1 -- 192.168.32.73:6803/12697 messenger.start -9> 2015-04-28 01:22:04.431752 7f7db081b900 1 -- 192.168.32.73:6802/12697 messenger.start -8> 2015-04-28 01:22:04.431805 7f7db081b900 1 -- :/0 messenger.start -7> 2015-04-28 01:22:04.431913 7f7db081b900 2 osd.23 0 mounting /var/lib/ceph/osd/ceph-23 /var/lib/ceph/osd/ceph-23/journal -6> 2015-04-28 01:22:04.431991 7f7db081b900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. -5> 2015-04-28 01:22:04.432032 7f7db081b900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) -4> 2015-04-28 01:22:04.433579 7f7db081b900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work -3> 2015-04-28 01:22:04.433599 7f7db081b900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option -2> 2015-04-28 01:22:04.434231 7f7db081b900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) -1> 2015-04-28 01:22:04.434367 7f7db081b900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.12-031812-generic >= 3.5 0> 2015-04-28 01:22:04.443809 7f7db081b900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f7db081b900 time 2015-04-28 01:22:04.438955 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f7dad95eec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 01:22:04.450742 7f7db081b900 -1 *** Caught signal (Aborted) ** in thread 7f7db081b900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f7daf4d4340] 3: (gsignal()+0x39) [0x7f7dad973cc9] 4: (abort()+0x148) [0x7f7dad9770d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f7dae27e6b5] 6: (()+0x5e836) [0x7f7dae27c836] 7: (()+0x5e863) [0x7f7dae27c863] 8: (()+0x5eaa2) [0x7f7dae27caa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f7dad95eec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2015-04-28 01:22:04.450742 7f7db081b900 -1 *** Caught signal (Aborted) ** in thread 7f7db081b900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f7daf4d4340] 3: (gsignal()+0x39) [0x7f7dad973cc9] 4: (abort()+0x148) [0x7f7dad9770d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f7dae27e6b5] 6: (()+0x5e836) [0x7f7dae27c836] 7: (()+0x5e863) [0x7f7dae27c863] 8: (()+0x5eaa2) [0x7f7dae27caa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f7dad95eec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 01:22:08.262126 7effdd791900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 14173 2015-04-28 01:22:08.283810 7effdd791900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. 2015-04-28 01:22:08.283855 7effdd791900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) 2015-04-28 01:22:08.285349 7effdd791900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work 2015-04-28 01:22:08.285373 7effdd791900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-04-28 01:22:08.286338 7effdd791900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2015-04-28 01:22:08.286487 7effdd791900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.12-031812-generic >= 3.5 2015-04-28 01:22:08.309646 7effdd791900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7effdd791900 time 2015-04-28 01:22:08.304613 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7effda8d4ec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -48> 2015-04-28 01:22:08.257622 7effdd791900 5 asok(0x4b62000) register_command perfcounters_dump hook 0x4b02050 -47> 2015-04-28 01:22:08.257786 7effdd791900 5 asok(0x4b62000) register_command 1 hook 0x4b02050 -46> 2015-04-28 01:22:08.257830 7effdd791900 5 asok(0x4b62000) register_command perf dump hook 0x4b02050 -45> 2015-04-28 01:22:08.257853 7effdd791900 5 asok(0x4b62000) register_command perfcounters_schema hook 0x4b02050 -44> 2015-04-28 01:22:08.257863 7effdd791900 5 asok(0x4b62000) register_command 2 hook 0x4b02050 -43> 2015-04-28 01:22:08.257879 7effdd791900 5 asok(0x4b62000) register_command perf schema hook 0x4b02050 -42> 2015-04-28 01:22:08.257891 7effdd791900 5 asok(0x4b62000) register_command perf reset hook 0x4b02050 -41> 2015-04-28 01:22:08.257922 7effdd791900 5 asok(0x4b62000) register_command config show hook 0x4b02050 -40> 2015-04-28 01:22:08.257932 7effdd791900 5 asok(0x4b62000) register_command config set hook 0x4b02050 -39> 2015-04-28 01:22:08.257938 7effdd791900 5 asok(0x4b62000) register_command config get hook 0x4b02050 -38> 2015-04-28 01:22:08.257942 7effdd791900 5 asok(0x4b62000) register_command config diff hook 0x4b02050 -37> 2015-04-28 01:22:08.257947 7effdd791900 5 asok(0x4b62000) register_command log flush hook 0x4b02050 -36> 2015-04-28 01:22:08.257952 7effdd791900 5 asok(0x4b62000) register_command log dump hook 0x4b02050 -35> 2015-04-28 01:22:08.257957 7effdd791900 5 asok(0x4b62000) register_command log reopen hook 0x4b02050 -34> 2015-04-28 01:22:08.262126 7effdd791900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 14173 -33> 2015-04-28 01:22:08.263602 7effdd791900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -32> 2015-04-28 01:22:08.263622 7effdd791900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6802/14173 need_addr=0 -31> 2015-04-28 01:22:08.263652 7effdd791900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -30> 2015-04-28 01:22:08.263662 7effdd791900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6802/14173 need_addr=0 -29> 2015-04-28 01:22:08.263684 7effdd791900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -28> 2015-04-28 01:22:08.263694 7effdd791900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6803/14173 need_addr=0 -27> 2015-04-28 01:22:08.263715 7effdd791900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -26> 2015-04-28 01:22:08.263726 7effdd791900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6803/14173 need_addr=0 -25> 2015-04-28 01:22:08.266196 7effdd791900 5 asok(0x4b62000) init /var/run/ceph/ceph-osd.23.asok -24> 2015-04-28 01:22:08.266219 7effdd791900 5 asok(0x4b62000) bind_and_listen /var/run/ceph/ceph-osd.23.asok -23> 2015-04-28 01:22:08.266389 7effdd791900 5 asok(0x4b62000) register_command 0 hook 0x4afe0a8 -22> 2015-04-28 01:22:08.266404 7effdd791900 5 asok(0x4b62000) register_command version hook 0x4afe0a8 -21> 2015-04-28 01:22:08.266413 7effdd791900 5 asok(0x4b62000) register_command git_version hook 0x4afe0a8 -20> 2015-04-28 01:22:08.266425 7effdd791900 5 asok(0x4b62000) register_command help hook 0x4b02150 -19> 2015-04-28 01:22:08.266434 7effdd791900 5 asok(0x4b62000) register_command get_command_descriptions hook 0x4b02140 -18> 2015-04-28 01:22:08.266497 7effd789c700 5 asok(0x4b62000) entry start -17> 2015-04-28 01:22:08.266505 7effdd791900 10 monclient(hunting): build_initial_monmap -16> 2015-04-28 01:22:08.283154 7effdd791900 5 adding auth protocol: cephx -15> 2015-04-28 01:22:08.283173 7effdd791900 5 adding auth protocol: cephx -14> 2015-04-28 01:22:08.283332 7effdd791900 5 asok(0x4b62000) register_command objecter_requests hook 0x4b02180 -13> 2015-04-28 01:22:08.283443 7effdd791900 1 -- 192.168.30.73:6802/14173 messenger.start -12> 2015-04-28 01:22:08.283504 7effdd791900 1 -- :/0 messenger.start -11> 2015-04-28 01:22:08.283544 7effdd791900 1 -- 192.168.30.73:6803/14173 messenger.start -10> 2015-04-28 01:22:08.283581 7effdd791900 1 -- 192.168.32.73:6803/14173 messenger.start -9> 2015-04-28 01:22:08.283619 7effdd791900 1 -- 192.168.32.73:6802/14173 messenger.start -8> 2015-04-28 01:22:08.283655 7effdd791900 1 -- :/0 messenger.start -7> 2015-04-28 01:22:08.283743 7effdd791900 2 osd.23 0 mounting /var/lib/ceph/osd/ceph-23 /var/lib/ceph/osd/ceph-23/journal -6> 2015-04-28 01:22:08.283810 7effdd791900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. -5> 2015-04-28 01:22:08.283855 7effdd791900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) -4> 2015-04-28 01:22:08.285349 7effdd791900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work -3> 2015-04-28 01:22:08.285373 7effdd791900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option -2> 2015-04-28 01:22:08.286338 7effdd791900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) -1> 2015-04-28 01:22:08.286487 7effdd791900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.12-031812-generic >= 3.5 0> 2015-04-28 01:22:08.309646 7effdd791900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7effdd791900 time 2015-04-28 01:22:08.304613 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7effda8d4ec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 01:22:08.316608 7effdd791900 -1 *** Caught signal (Aborted) ** in thread 7effdd791900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7effdc44a340] 3: (gsignal()+0x39) [0x7effda8e9cc9] 4: (abort()+0x148) [0x7effda8ed0d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7effdb1f46b5] 6: (()+0x5e836) [0x7effdb1f2836] 7: (()+0x5e863) [0x7effdb1f2863] 8: (()+0x5eaa2) [0x7effdb1f2aa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7effda8d4ec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2015-04-28 01:22:08.316608 7effdd791900 -1 *** Caught signal (Aborted) ** in thread 7effdd791900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7effdc44a340] 3: (gsignal()+0x39) [0x7effda8e9cc9] 4: (abort()+0x148) [0x7effda8ed0d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7effdb1f46b5] 6: (()+0x5e836) [0x7effdb1f2836] 7: (()+0x5e863) [0x7effdb1f2863] 8: (()+0x5eaa2) [0x7effdb1f2aa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7effda8d4ec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 01:22:09.089052 7f05ab9f2900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 14445 2015-04-28 01:22:09.110695 7f05ab9f2900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. 2015-04-28 01:22:09.110739 7f05ab9f2900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) 2015-04-28 01:22:09.112259 7f05ab9f2900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work 2015-04-28 01:22:09.112280 7f05ab9f2900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-04-28 01:22:09.113132 7f05ab9f2900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2015-04-28 01:22:09.113275 7f05ab9f2900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.12-031812-generic >= 3.5 2015-04-28 01:22:09.147935 7f05ab9f2900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f05ab9f2900 time 2015-04-28 01:22:09.143055 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f05a8b35ec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -48> 2015-04-28 01:22:09.084836 7f05ab9f2900 5 asok(0x394c000) register_command perfcounters_dump hook 0x38ec050 -47> 2015-04-28 01:22:09.084901 7f05ab9f2900 5 asok(0x394c000) register_command 1 hook 0x38ec050 -46> 2015-04-28 01:22:09.084908 7f05ab9f2900 5 asok(0x394c000) register_command perf dump hook 0x38ec050 -45> 2015-04-28 01:22:09.084920 7f05ab9f2900 5 asok(0x394c000) register_command perfcounters_schema hook 0x38ec050 -44> 2015-04-28 01:22:09.084925 7f05ab9f2900 5 asok(0x394c000) register_command 2 hook 0x38ec050 -43> 2015-04-28 01:22:09.084929 7f05ab9f2900 5 asok(0x394c000) register_command perf schema hook 0x38ec050 -42> 2015-04-28 01:22:09.084934 7f05ab9f2900 5 asok(0x394c000) register_command perf reset hook 0x38ec050 -41> 2015-04-28 01:22:09.084953 7f05ab9f2900 5 asok(0x394c000) register_command config show hook 0x38ec050 -40> 2015-04-28 01:22:09.084959 7f05ab9f2900 5 asok(0x394c000) register_command config set hook 0x38ec050 -39> 2015-04-28 01:22:09.084964 7f05ab9f2900 5 asok(0x394c000) register_command config get hook 0x38ec050 -38> 2015-04-28 01:22:09.084968 7f05ab9f2900 5 asok(0x394c000) register_command config diff hook 0x38ec050 -37> 2015-04-28 01:22:09.084972 7f05ab9f2900 5 asok(0x394c000) register_command log flush hook 0x38ec050 -36> 2015-04-28 01:22:09.084977 7f05ab9f2900 5 asok(0x394c000) register_command log dump hook 0x38ec050 -35> 2015-04-28 01:22:09.084982 7f05ab9f2900 5 asok(0x394c000) register_command log reopen hook 0x38ec050 -34> 2015-04-28 01:22:09.089052 7f05ab9f2900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 14445 -33> 2015-04-28 01:22:09.090506 7f05ab9f2900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -32> 2015-04-28 01:22:09.090527 7f05ab9f2900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6804/14445 need_addr=0 -31> 2015-04-28 01:22:09.090556 7f05ab9f2900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -30> 2015-04-28 01:22:09.090566 7f05ab9f2900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6804/14445 need_addr=0 -29> 2015-04-28 01:22:09.090586 7f05ab9f2900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -28> 2015-04-28 01:22:09.090596 7f05ab9f2900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6805/14445 need_addr=0 -27> 2015-04-28 01:22:09.090617 7f05ab9f2900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -26> 2015-04-28 01:22:09.090627 7f05ab9f2900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6805/14445 need_addr=0 -25> 2015-04-28 01:22:09.092995 7f05ab9f2900 5 asok(0x394c000) init /var/run/ceph/ceph-osd.23.asok -24> 2015-04-28 01:22:09.093017 7f05ab9f2900 5 asok(0x394c000) bind_and_listen /var/run/ceph/ceph-osd.23.asok -23> 2015-04-28 01:22:09.093198 7f05ab9f2900 5 asok(0x394c000) register_command 0 hook 0x38e80a8 -22> 2015-04-28 01:22:09.093211 7f05ab9f2900 5 asok(0x394c000) register_command version hook 0x38e80a8 -21> 2015-04-28 01:22:09.093217 7f05ab9f2900 5 asok(0x394c000) register_command git_version hook 0x38e80a8 -20> 2015-04-28 01:22:09.093228 7f05ab9f2900 5 asok(0x394c000) register_command help hook 0x38ec150 -19> 2015-04-28 01:22:09.093234 7f05ab9f2900 5 asok(0x394c000) register_command get_command_descriptions hook 0x38ec140 -18> 2015-04-28 01:22:09.093306 7f05ab9f2900 10 monclient(hunting): build_initial_monmap -17> 2015-04-28 01:22:09.093337 7f05a5afd700 5 asok(0x394c000) entry start -16> 2015-04-28 01:22:09.110017 7f05ab9f2900 5 adding auth protocol: cephx -15> 2015-04-28 01:22:09.110035 7f05ab9f2900 5 adding auth protocol: cephx -14> 2015-04-28 01:22:09.110205 7f05ab9f2900 5 asok(0x394c000) register_command objecter_requests hook 0x38ec180 -13> 2015-04-28 01:22:09.110318 7f05ab9f2900 1 -- 192.168.30.73:6804/14445 messenger.start -12> 2015-04-28 01:22:09.110377 7f05ab9f2900 1 -- :/0 messenger.start -11> 2015-04-28 01:22:09.110416 7f05ab9f2900 1 -- 192.168.30.73:6805/14445 messenger.start -10> 2015-04-28 01:22:09.110456 7f05ab9f2900 1 -- 192.168.32.73:6805/14445 messenger.start -9> 2015-04-28 01:22:09.110496 7f05ab9f2900 1 -- 192.168.32.73:6804/14445 messenger.start -8> 2015-04-28 01:22:09.110533 7f05ab9f2900 1 -- :/0 messenger.start -7> 2015-04-28 01:22:09.110629 7f05ab9f2900 2 osd.23 0 mounting /var/lib/ceph/osd/ceph-23 /var/lib/ceph/osd/ceph-23/journal -6> 2015-04-28 01:22:09.110695 7f05ab9f2900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. -5> 2015-04-28 01:22:09.110739 7f05ab9f2900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) -4> 2015-04-28 01:22:09.112259 7f05ab9f2900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work -3> 2015-04-28 01:22:09.112280 7f05ab9f2900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option -2> 2015-04-28 01:22:09.113132 7f05ab9f2900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) -1> 2015-04-28 01:22:09.113275 7f05ab9f2900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.12-031812-generic >= 3.5 0> 2015-04-28 01:22:09.147935 7f05ab9f2900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f05ab9f2900 time 2015-04-28 01:22:09.143055 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f05a8b35ec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 01:22:09.154854 7f05ab9f2900 -1 *** Caught signal (Aborted) ** in thread 7f05ab9f2900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f05aa6ab340] 3: (gsignal()+0x39) [0x7f05a8b4acc9] 4: (abort()+0x148) [0x7f05a8b4e0d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f05a94556b5] 6: (()+0x5e836) [0x7f05a9453836] 7: (()+0x5e863) [0x7f05a9453863] 8: (()+0x5eaa2) [0x7f05a9453aa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f05a8b35ec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2015-04-28 01:22:09.154854 7f05ab9f2900 -1 *** Caught signal (Aborted) ** in thread 7f05ab9f2900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f05aa6ab340] 3: (gsignal()+0x39) [0x7f05a8b4acc9] 4: (abort()+0x148) [0x7f05a8b4e0d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f05a94556b5] 6: (()+0x5e836) [0x7f05a9453836] 7: (()+0x5e863) [0x7f05a9453863] 8: (()+0x5eaa2) [0x7f05a9453aa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f05a8b35ec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 01:22:11.183197 7f0142642900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 15086 2015-04-28 01:22:11.205257 7f0142642900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. 2015-04-28 01:22:11.205298 7f0142642900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) 2015-04-28 01:22:11.206795 7f0142642900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work 2015-04-28 01:22:11.206815 7f0142642900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-04-28 01:22:11.207712 7f0142642900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2015-04-28 01:22:11.207855 7f0142642900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.12-031812-generic >= 3.5 2015-04-28 01:22:11.216999 7f0142642900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f0142642900 time 2015-04-28 01:22:11.212163 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f013f785ec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -48> 2015-04-28 01:22:11.178716 7f0142642900 5 asok(0x4cb8000) register_command perfcounters_dump hook 0x4c58050 -47> 2015-04-28 01:22:11.178780 7f0142642900 5 asok(0x4cb8000) register_command 1 hook 0x4c58050 -46> 2015-04-28 01:22:11.178814 7f0142642900 5 asok(0x4cb8000) register_command perf dump hook 0x4c58050 -45> 2015-04-28 01:22:11.178839 7f0142642900 5 asok(0x4cb8000) register_command perfcounters_schema hook 0x4c58050 -44> 2015-04-28 01:22:11.178854 7f0142642900 5 asok(0x4cb8000) register_command 2 hook 0x4c58050 -43> 2015-04-28 01:22:11.178865 7f0142642900 5 asok(0x4cb8000) register_command perf schema hook 0x4c58050 -42> 2015-04-28 01:22:11.178875 7f0142642900 5 asok(0x4cb8000) register_command perf reset hook 0x4c58050 -41> 2015-04-28 01:22:11.178904 7f0142642900 5 asok(0x4cb8000) register_command config show hook 0x4c58050 -40> 2015-04-28 01:22:11.178914 7f0142642900 5 asok(0x4cb8000) register_command config set hook 0x4c58050 -39> 2015-04-28 01:22:11.178922 7f0142642900 5 asok(0x4cb8000) register_command config get hook 0x4c58050 -38> 2015-04-28 01:22:11.178931 7f0142642900 5 asok(0x4cb8000) register_command config diff hook 0x4c58050 -37> 2015-04-28 01:22:11.178939 7f0142642900 5 asok(0x4cb8000) register_command log flush hook 0x4c58050 -36> 2015-04-28 01:22:11.178948 7f0142642900 5 asok(0x4cb8000) register_command log dump hook 0x4c58050 -35> 2015-04-28 01:22:11.178956 7f0142642900 5 asok(0x4cb8000) register_command log reopen hook 0x4c58050 -34> 2015-04-28 01:22:11.183197 7f0142642900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 15086 -33> 2015-04-28 01:22:11.184944 7f0142642900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -32> 2015-04-28 01:22:11.184977 7f0142642900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6806/15086 need_addr=0 -31> 2015-04-28 01:22:11.185020 7f0142642900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -30> 2015-04-28 01:22:11.185033 7f0142642900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6806/15086 need_addr=0 -29> 2015-04-28 01:22:11.185069 7f0142642900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -28> 2015-04-28 01:22:11.185090 7f0142642900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6807/15086 need_addr=0 -27> 2015-04-28 01:22:11.185128 7f0142642900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -26> 2015-04-28 01:22:11.185141 7f0142642900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6807/15086 need_addr=0 -25> 2015-04-28 01:22:11.187666 7f0142642900 5 asok(0x4cb8000) init /var/run/ceph/ceph-osd.23.asok -24> 2015-04-28 01:22:11.187687 7f0142642900 5 asok(0x4cb8000) bind_and_listen /var/run/ceph/ceph-osd.23.asok -23> 2015-04-28 01:22:11.187851 7f0142642900 5 asok(0x4cb8000) register_command 0 hook 0x4c540a8 -22> 2015-04-28 01:22:11.187865 7f0142642900 5 asok(0x4cb8000) register_command version hook 0x4c540a8 -21> 2015-04-28 01:22:11.187874 7f0142642900 5 asok(0x4cb8000) register_command git_version hook 0x4c540a8 -20> 2015-04-28 01:22:11.187884 7f0142642900 5 asok(0x4cb8000) register_command help hook 0x4c58150 -19> 2015-04-28 01:22:11.187893 7f0142642900 5 asok(0x4cb8000) register_command get_command_descriptions hook 0x4c58140 -18> 2015-04-28 01:22:11.187978 7f0142642900 10 monclient(hunting): build_initial_monmap -17> 2015-04-28 01:22:11.187976 7f013c74d700 5 asok(0x4cb8000) entry start -16> 2015-04-28 01:22:11.204425 7f0142642900 5 adding auth protocol: cephx -15> 2015-04-28 01:22:11.204443 7f0142642900 5 adding auth protocol: cephx -14> 2015-04-28 01:22:11.204598 7f0142642900 5 asok(0x4cb8000) register_command objecter_requests hook 0x4c58180 -13> 2015-04-28 01:22:11.204738 7f0142642900 1 -- 192.168.30.73:6806/15086 messenger.start -12> 2015-04-28 01:22:11.204828 7f0142642900 1 -- :/0 messenger.start -11> 2015-04-28 01:22:11.204886 7f0142642900 1 -- 192.168.30.73:6807/15086 messenger.start -10> 2015-04-28 01:22:11.204947 7f0142642900 1 -- 192.168.32.73:6807/15086 messenger.start -9> 2015-04-28 01:22:11.205013 7f0142642900 1 -- 192.168.32.73:6806/15086 messenger.start -8> 2015-04-28 01:22:11.205061 7f0142642900 1 -- :/0 messenger.start -7> 2015-04-28 01:22:11.205183 7f0142642900 2 osd.23 0 mounting /var/lib/ceph/osd/ceph-23 /var/lib/ceph/osd/ceph-23/journal -6> 2015-04-28 01:22:11.205257 7f0142642900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. -5> 2015-04-28 01:22:11.205298 7f0142642900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) -4> 2015-04-28 01:22:11.206795 7f0142642900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work -3> 2015-04-28 01:22:11.206815 7f0142642900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option -2> 2015-04-28 01:22:11.207712 7f0142642900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) -1> 2015-04-28 01:22:11.207855 7f0142642900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.12-031812-generic >= 3.5 0> 2015-04-28 01:22:11.216999 7f0142642900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f0142642900 time 2015-04-28 01:22:11.212163 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f013f785ec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 01:22:11.223753 7f0142642900 -1 *** Caught signal (Aborted) ** in thread 7f0142642900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f01412fb340] 3: (gsignal()+0x39) [0x7f013f79acc9] 4: (abort()+0x148) [0x7f013f79e0d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f01400a56b5] 6: (()+0x5e836) [0x7f01400a3836] 7: (()+0x5e863) [0x7f01400a3863] 8: (()+0x5eaa2) [0x7f01400a3aa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f013f785ec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2015-04-28 01:22:11.223753 7f0142642900 -1 *** Caught signal (Aborted) ** in thread 7f0142642900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f01412fb340] 3: (gsignal()+0x39) [0x7f013f79acc9] 4: (abort()+0x148) [0x7f013f79e0d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f01400a56b5] 6: (()+0x5e836) [0x7f01400a3836] 7: (()+0x5e863) [0x7f01400a3863] 8: (()+0x5eaa2) [0x7f01400a3aa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f013f785ec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 01:22:11.919216 7efd96853900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 15368 2015-04-28 01:22:11.941148 7efd96853900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. 2015-04-28 01:22:11.941194 7efd96853900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) 2015-04-28 01:22:11.942752 7efd96853900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work 2015-04-28 01:22:11.942771 7efd96853900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-04-28 01:22:11.943754 7efd96853900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2015-04-28 01:22:11.943927 7efd96853900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.12-031812-generic >= 3.5 2015-04-28 01:22:11.953820 7efd96853900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7efd96853900 time 2015-04-28 01:22:11.948736 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7efd93996ec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -48> 2015-04-28 01:22:11.914940 7efd96853900 5 asok(0x567a000) register_command perfcounters_dump hook 0x561a050 -47> 2015-04-28 01:22:11.915008 7efd96853900 5 asok(0x567a000) register_command 1 hook 0x561a050 -46> 2015-04-28 01:22:11.915025 7efd96853900 5 asok(0x567a000) register_command perf dump hook 0x561a050 -45> 2015-04-28 01:22:11.915037 7efd96853900 5 asok(0x567a000) register_command perfcounters_schema hook 0x561a050 -44> 2015-04-28 01:22:11.915050 7efd96853900 5 asok(0x567a000) register_command 2 hook 0x561a050 -43> 2015-04-28 01:22:11.915056 7efd96853900 5 asok(0x567a000) register_command perf schema hook 0x561a050 -42> 2015-04-28 01:22:11.915065 7efd96853900 5 asok(0x567a000) register_command perf reset hook 0x561a050 -41> 2015-04-28 01:22:11.915086 7efd96853900 5 asok(0x567a000) register_command config show hook 0x561a050 -40> 2015-04-28 01:22:11.915096 7efd96853900 5 asok(0x567a000) register_command config set hook 0x561a050 -39> 2015-04-28 01:22:11.915102 7efd96853900 5 asok(0x567a000) register_command config get hook 0x561a050 -38> 2015-04-28 01:22:11.915110 7efd96853900 5 asok(0x567a000) register_command config diff hook 0x561a050 -37> 2015-04-28 01:22:11.915116 7efd96853900 5 asok(0x567a000) register_command log flush hook 0x561a050 -36> 2015-04-28 01:22:11.915123 7efd96853900 5 asok(0x567a000) register_command log dump hook 0x561a050 -35> 2015-04-28 01:22:11.915132 7efd96853900 5 asok(0x567a000) register_command log reopen hook 0x561a050 -34> 2015-04-28 01:22:11.919216 7efd96853900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 15368 -33> 2015-04-28 01:22:11.920773 7efd96853900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -32> 2015-04-28 01:22:11.920796 7efd96853900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6800/15368 need_addr=0 -31> 2015-04-28 01:22:11.920825 7efd96853900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -30> 2015-04-28 01:22:11.920841 7efd96853900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6800/15368 need_addr=0 -29> 2015-04-28 01:22:11.920860 7efd96853900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -28> 2015-04-28 01:22:11.920869 7efd96853900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6801/15368 need_addr=0 -27> 2015-04-28 01:22:11.920886 7efd96853900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -26> 2015-04-28 01:22:11.920895 7efd96853900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6801/15368 need_addr=0 -25> 2015-04-28 01:22:11.923328 7efd96853900 5 asok(0x567a000) init /var/run/ceph/ceph-osd.23.asok -24> 2015-04-28 01:22:11.923352 7efd96853900 5 asok(0x567a000) bind_and_listen /var/run/ceph/ceph-osd.23.asok -23> 2015-04-28 01:22:11.923532 7efd96853900 5 asok(0x567a000) register_command 0 hook 0x56160a8 -22> 2015-04-28 01:22:11.923547 7efd96853900 5 asok(0x567a000) register_command version hook 0x56160a8 -21> 2015-04-28 01:22:11.923557 7efd96853900 5 asok(0x567a000) register_command git_version hook 0x56160a8 -20> 2015-04-28 01:22:11.923568 7efd96853900 5 asok(0x567a000) register_command help hook 0x561a150 -19> 2015-04-28 01:22:11.923577 7efd96853900 5 asok(0x567a000) register_command get_command_descriptions hook 0x561a140 -18> 2015-04-28 01:22:11.923657 7efd96853900 10 monclient(hunting): build_initial_monmap -17> 2015-04-28 01:22:11.923658 7efd9095e700 5 asok(0x567a000) entry start -16> 2015-04-28 01:22:11.940473 7efd96853900 5 adding auth protocol: cephx -15> 2015-04-28 01:22:11.940488 7efd96853900 5 adding auth protocol: cephx -14> 2015-04-28 01:22:11.940654 7efd96853900 5 asok(0x567a000) register_command objecter_requests hook 0x561a180 -13> 2015-04-28 01:22:11.940765 7efd96853900 1 -- 192.168.30.73:6800/15368 messenger.start -12> 2015-04-28 01:22:11.940823 7efd96853900 1 -- :/0 messenger.start -11> 2015-04-28 01:22:11.940862 7efd96853900 1 -- 192.168.30.73:6801/15368 messenger.start -10> 2015-04-28 01:22:11.940901 7efd96853900 1 -- 192.168.32.73:6801/15368 messenger.start -9> 2015-04-28 01:22:11.940940 7efd96853900 1 -- 192.168.32.73:6800/15368 messenger.start -8> 2015-04-28 01:22:11.940983 7efd96853900 1 -- :/0 messenger.start -7> 2015-04-28 01:22:11.941081 7efd96853900 2 osd.23 0 mounting /var/lib/ceph/osd/ceph-23 /var/lib/ceph/osd/ceph-23/journal -6> 2015-04-28 01:22:11.941148 7efd96853900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. -5> 2015-04-28 01:22:11.941194 7efd96853900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) -4> 2015-04-28 01:22:11.942752 7efd96853900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work -3> 2015-04-28 01:22:11.942771 7efd96853900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option -2> 2015-04-28 01:22:11.943754 7efd96853900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) -1> 2015-04-28 01:22:11.943927 7efd96853900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.12-031812-generic >= 3.5 0> 2015-04-28 01:22:11.953820 7efd96853900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7efd96853900 time 2015-04-28 01:22:11.948736 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7efd93996ec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 01:22:11.960938 7efd96853900 -1 *** Caught signal (Aborted) ** in thread 7efd96853900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7efd9550c340] 3: (gsignal()+0x39) [0x7efd939abcc9] 4: (abort()+0x148) [0x7efd939af0d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7efd942b66b5] 6: (()+0x5e836) [0x7efd942b4836] 7: (()+0x5e863) [0x7efd942b4863] 8: (()+0x5eaa2) [0x7efd942b4aa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7efd93996ec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2015-04-28 01:22:11.960938 7efd96853900 -1 *** Caught signal (Aborted) ** in thread 7efd96853900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7efd9550c340] 3: (gsignal()+0x39) [0x7efd939abcc9] 4: (abort()+0x148) [0x7efd939af0d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7efd942b66b5] 6: (()+0x5e836) [0x7efd942b4836] 7: (()+0x5e863) [0x7efd942b4863] 8: (()+0x5eaa2) [0x7efd942b4aa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7efd93996ec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 01:22:12.678341 7f8a76639900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 15634 2015-04-28 01:22:12.700026 7f8a76639900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. 2015-04-28 01:22:12.700073 7f8a76639900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) 2015-04-28 01:22:12.701626 7f8a76639900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work 2015-04-28 01:22:12.701648 7f8a76639900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-04-28 01:22:12.702479 7f8a76639900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2015-04-28 01:22:12.702633 7f8a76639900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.12-031812-generic >= 3.5 2015-04-28 01:22:12.711935 7f8a76639900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f8a76639900 time 2015-04-28 01:22:12.707094 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f8a7377cec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -48> 2015-04-28 01:22:12.674052 7f8a76639900 5 asok(0x54e6000) register_command perfcounters_dump hook 0x5486050 -47> 2015-04-28 01:22:12.674127 7f8a76639900 5 asok(0x54e6000) register_command 1 hook 0x5486050 -46> 2015-04-28 01:22:12.674135 7f8a76639900 5 asok(0x54e6000) register_command perf dump hook 0x5486050 -45> 2015-04-28 01:22:12.674171 7f8a76639900 5 asok(0x54e6000) register_command perfcounters_schema hook 0x5486050 -44> 2015-04-28 01:22:12.674181 7f8a76639900 5 asok(0x54e6000) register_command 2 hook 0x5486050 -43> 2015-04-28 01:22:12.674186 7f8a76639900 5 asok(0x54e6000) register_command perf schema hook 0x5486050 -42> 2015-04-28 01:22:12.674196 7f8a76639900 5 asok(0x54e6000) register_command perf reset hook 0x5486050 -41> 2015-04-28 01:22:12.674229 7f8a76639900 5 asok(0x54e6000) register_command config show hook 0x5486050 -40> 2015-04-28 01:22:12.674238 7f8a76639900 5 asok(0x54e6000) register_command config set hook 0x5486050 -39> 2015-04-28 01:22:12.674243 7f8a76639900 5 asok(0x54e6000) register_command config get hook 0x5486050 -38> 2015-04-28 01:22:12.674247 7f8a76639900 5 asok(0x54e6000) register_command config diff hook 0x5486050 -37> 2015-04-28 01:22:12.674252 7f8a76639900 5 asok(0x54e6000) register_command log flush hook 0x5486050 -36> 2015-04-28 01:22:12.674256 7f8a76639900 5 asok(0x54e6000) register_command log dump hook 0x5486050 -35> 2015-04-28 01:22:12.674261 7f8a76639900 5 asok(0x54e6000) register_command log reopen hook 0x5486050 -34> 2015-04-28 01:22:12.678341 7f8a76639900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 15634 -33> 2015-04-28 01:22:12.679846 7f8a76639900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -32> 2015-04-28 01:22:12.679867 7f8a76639900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6800/15634 need_addr=0 -31> 2015-04-28 01:22:12.679893 7f8a76639900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -30> 2015-04-28 01:22:12.679899 7f8a76639900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6800/15634 need_addr=0 -29> 2015-04-28 01:22:12.679935 7f8a76639900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -28> 2015-04-28 01:22:12.679955 7f8a76639900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6801/15634 need_addr=0 -27> 2015-04-28 01:22:12.679973 7f8a76639900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -26> 2015-04-28 01:22:12.679984 7f8a76639900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6801/15634 need_addr=0 -25> 2015-04-28 01:22:12.682337 7f8a76639900 5 asok(0x54e6000) init /var/run/ceph/ceph-osd.23.asok -24> 2015-04-28 01:22:12.682359 7f8a76639900 5 asok(0x54e6000) bind_and_listen /var/run/ceph/ceph-osd.23.asok -23> 2015-04-28 01:22:12.682532 7f8a76639900 5 asok(0x54e6000) register_command 0 hook 0x54820a8 -22> 2015-04-28 01:22:12.682546 7f8a76639900 5 asok(0x54e6000) register_command version hook 0x54820a8 -21> 2015-04-28 01:22:12.682556 7f8a76639900 5 asok(0x54e6000) register_command git_version hook 0x54820a8 -20> 2015-04-28 01:22:12.682567 7f8a76639900 5 asok(0x54e6000) register_command help hook 0x5486150 -19> 2015-04-28 01:22:12.682576 7f8a76639900 5 asok(0x54e6000) register_command get_command_descriptions hook 0x5486140 -18> 2015-04-28 01:22:12.682648 7f8a76639900 10 monclient(hunting): build_initial_monmap -17> 2015-04-28 01:22:12.682676 7f8a70744700 5 asok(0x54e6000) entry start -16> 2015-04-28 01:22:12.699354 7f8a76639900 5 adding auth protocol: cephx -15> 2015-04-28 01:22:12.699372 7f8a76639900 5 adding auth protocol: cephx -14> 2015-04-28 01:22:12.699526 7f8a76639900 5 asok(0x54e6000) register_command objecter_requests hook 0x5486180 -13> 2015-04-28 01:22:12.699634 7f8a76639900 1 -- 192.168.30.73:6800/15634 messenger.start -12> 2015-04-28 01:22:12.699691 7f8a76639900 1 -- :/0 messenger.start -11> 2015-04-28 01:22:12.699729 7f8a76639900 1 -- 192.168.30.73:6801/15634 messenger.start -10> 2015-04-28 01:22:12.699767 7f8a76639900 1 -- 192.168.32.73:6801/15634 messenger.start -9> 2015-04-28 01:22:12.699805 7f8a76639900 1 -- 192.168.32.73:6800/15634 messenger.start -8> 2015-04-28 01:22:12.699842 7f8a76639900 1 -- :/0 messenger.start -7> 2015-04-28 01:22:12.699957 7f8a76639900 2 osd.23 0 mounting /var/lib/ceph/osd/ceph-23 /var/lib/ceph/osd/ceph-23/journal -6> 2015-04-28 01:22:12.700026 7f8a76639900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. -5> 2015-04-28 01:22:12.700073 7f8a76639900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) -4> 2015-04-28 01:22:12.701626 7f8a76639900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work -3> 2015-04-28 01:22:12.701648 7f8a76639900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option -2> 2015-04-28 01:22:12.702479 7f8a76639900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) -1> 2015-04-28 01:22:12.702633 7f8a76639900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.12-031812-generic >= 3.5 0> 2015-04-28 01:22:12.711935 7f8a76639900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f8a76639900 time 2015-04-28 01:22:12.707094 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f8a7377cec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 01:22:12.718752 7f8a76639900 -1 *** Caught signal (Aborted) ** in thread 7f8a76639900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f8a752f2340] 3: (gsignal()+0x39) [0x7f8a73791cc9] 4: (abort()+0x148) [0x7f8a737950d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f8a7409c6b5] 6: (()+0x5e836) [0x7f8a7409a836] 7: (()+0x5e863) [0x7f8a7409a863] 8: (()+0x5eaa2) [0x7f8a7409aaa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f8a7377cec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2015-04-28 01:22:12.718752 7f8a76639900 -1 *** Caught signal (Aborted) ** in thread 7f8a76639900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f8a752f2340] 3: (gsignal()+0x39) [0x7f8a73791cc9] 4: (abort()+0x148) [0x7f8a737950d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f8a7409c6b5] 6: (()+0x5e836) [0x7f8a7409a836] 7: (()+0x5e863) [0x7f8a7409a863] 8: (()+0x5eaa2) [0x7f8a7409aaa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f8a7377cec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 01:24:57.414170 7fd0c1872900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 22568 2015-04-28 01:24:57.427195 7fd0c1872900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. 2015-04-28 01:24:57.427232 7fd0c1872900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) 2015-04-28 01:24:57.428223 7fd0c1872900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work 2015-04-28 01:24:57.428236 7fd0c1872900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-04-28 01:24:57.428586 7fd0c1872900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2015-04-28 01:24:57.428660 7fd0c1872900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.12-031812-generic >= 3.5 2015-04-28 01:24:57.434194 7fd0c1872900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7fd0c1872900 time 2015-04-28 01:24:57.431602 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7fd0be9b5ec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -48> 2015-04-28 01:24:57.411947 7fd0c1872900 5 asok(0x5724000) register_command perfcounters_dump hook 0x56c4050 -47> 2015-04-28 01:24:57.411987 7fd0c1872900 5 asok(0x5724000) register_command 1 hook 0x56c4050 -46> 2015-04-28 01:24:57.411993 7fd0c1872900 5 asok(0x5724000) register_command perf dump hook 0x56c4050 -45> 2015-04-28 01:24:57.412001 7fd0c1872900 5 asok(0x5724000) register_command perfcounters_schema hook 0x56c4050 -44> 2015-04-28 01:24:57.412006 7fd0c1872900 5 asok(0x5724000) register_command 2 hook 0x56c4050 -43> 2015-04-28 01:24:57.412010 7fd0c1872900 5 asok(0x5724000) register_command perf schema hook 0x56c4050 -42> 2015-04-28 01:24:57.412014 7fd0c1872900 5 asok(0x5724000) register_command perf reset hook 0x56c4050 -41> 2015-04-28 01:24:57.412031 7fd0c1872900 5 asok(0x5724000) register_command config show hook 0x56c4050 -40> 2015-04-28 01:24:57.412037 7fd0c1872900 5 asok(0x5724000) register_command config set hook 0x56c4050 -39> 2015-04-28 01:24:57.412041 7fd0c1872900 5 asok(0x5724000) register_command config get hook 0x56c4050 -38> 2015-04-28 01:24:57.412046 7fd0c1872900 5 asok(0x5724000) register_command config diff hook 0x56c4050 -37> 2015-04-28 01:24:57.412050 7fd0c1872900 5 asok(0x5724000) register_command log flush hook 0x56c4050 -36> 2015-04-28 01:24:57.412055 7fd0c1872900 5 asok(0x5724000) register_command log dump hook 0x56c4050 -35> 2015-04-28 01:24:57.412059 7fd0c1872900 5 asok(0x5724000) register_command log reopen hook 0x56c4050 -34> 2015-04-28 01:24:57.414170 7fd0c1872900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 22568 -33> 2015-04-28 01:24:57.415022 7fd0c1872900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -32> 2015-04-28 01:24:57.415034 7fd0c1872900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6802/22568 need_addr=0 -31> 2015-04-28 01:24:57.415050 7fd0c1872900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -30> 2015-04-28 01:24:57.415055 7fd0c1872900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6802/22568 need_addr=0 -29> 2015-04-28 01:24:57.415066 7fd0c1872900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -28> 2015-04-28 01:24:57.415071 7fd0c1872900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6803/22568 need_addr=0 -27> 2015-04-28 01:24:57.415082 7fd0c1872900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -26> 2015-04-28 01:24:57.415087 7fd0c1872900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6803/22568 need_addr=0 -25> 2015-04-28 01:24:57.416411 7fd0c1872900 5 asok(0x5724000) init /var/run/ceph/ceph-osd.23.asok -24> 2015-04-28 01:24:57.416424 7fd0c1872900 5 asok(0x5724000) bind_and_listen /var/run/ceph/ceph-osd.23.asok -23> 2015-04-28 01:24:57.416521 7fd0c1872900 5 asok(0x5724000) register_command 0 hook 0x56c00a8 -22> 2015-04-28 01:24:57.416528 7fd0c1872900 5 asok(0x5724000) register_command version hook 0x56c00a8 -21> 2015-04-28 01:24:57.416533 7fd0c1872900 5 asok(0x5724000) register_command git_version hook 0x56c00a8 -20> 2015-04-28 01:24:57.416539 7fd0c1872900 5 asok(0x5724000) register_command help hook 0x56c4150 -19> 2015-04-28 01:24:57.416543 7fd0c1872900 5 asok(0x5724000) register_command get_command_descriptions hook 0x56c4140 -18> 2015-04-28 01:24:57.416587 7fd0c1872900 10 monclient(hunting): build_initial_monmap -17> 2015-04-28 01:24:57.416600 7fd0bb97d700 5 asok(0x5724000) entry start -16> 2015-04-28 01:24:57.425878 7fd0c1872900 5 adding auth protocol: cephx -15> 2015-04-28 01:24:57.425890 7fd0c1872900 5 adding auth protocol: cephx -14> 2015-04-28 01:24:57.425995 7fd0c1872900 5 asok(0x5724000) register_command objecter_requests hook 0x56c4180 -13> 2015-04-28 01:24:57.426087 7fd0c1872900 1 -- 192.168.30.73:6802/22568 messenger.start -12> 2015-04-28 01:24:57.426123 7fd0c1872900 1 -- :/0 messenger.start -11> 2015-04-28 01:24:57.426148 7fd0c1872900 1 -- 192.168.30.73:6803/22568 messenger.start -10> 2015-04-28 01:24:57.426173 7fd0c1872900 1 -- 192.168.32.73:6803/22568 messenger.start -9> 2015-04-28 01:24:57.426198 7fd0c1872900 1 -- 192.168.32.73:6802/22568 messenger.start -8> 2015-04-28 01:24:57.426221 7fd0c1872900 1 -- :/0 messenger.start -7> 2015-04-28 01:24:57.427145 7fd0c1872900 2 osd.23 0 mounting /var/lib/ceph/osd/ceph-23 /var/lib/ceph/osd/ceph-23/journal -6> 2015-04-28 01:24:57.427195 7fd0c1872900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. -5> 2015-04-28 01:24:57.427232 7fd0c1872900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) -4> 2015-04-28 01:24:57.428223 7fd0c1872900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work -3> 2015-04-28 01:24:57.428236 7fd0c1872900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option -2> 2015-04-28 01:24:57.428586 7fd0c1872900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) -1> 2015-04-28 01:24:57.428660 7fd0c1872900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.12-031812-generic >= 3.5 0> 2015-04-28 01:24:57.434194 7fd0c1872900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7fd0c1872900 time 2015-04-28 01:24:57.431602 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7fd0be9b5ec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 01:24:57.437860 7fd0c1872900 -1 *** Caught signal (Aborted) ** in thread 7fd0c1872900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7fd0c052b340] 3: (gsignal()+0x39) [0x7fd0be9cacc9] 4: (abort()+0x148) [0x7fd0be9ce0d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7fd0bf2d56b5] 6: (()+0x5e836) [0x7fd0bf2d3836] 7: (()+0x5e863) [0x7fd0bf2d3863] 8: (()+0x5eaa2) [0x7fd0bf2d3aa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7fd0be9b5ec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2015-04-28 01:24:57.437860 7fd0c1872900 -1 *** Caught signal (Aborted) ** in thread 7fd0c1872900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7fd0c052b340] 3: (gsignal()+0x39) [0x7fd0be9cacc9] 4: (abort()+0x148) [0x7fd0be9ce0d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7fd0bf2d56b5] 6: (()+0x5e836) [0x7fd0bf2d3836] 7: (()+0x5e863) [0x7fd0bf2d3863] 8: (()+0x5eaa2) [0x7fd0bf2d3aa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7fd0be9b5ec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 01:24:57.823767 7f5b288bc900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 23148 2015-04-28 01:24:57.835823 7f5b288bc900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. 2015-04-28 01:24:57.835850 7f5b288bc900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) 2015-04-28 01:24:57.836834 7f5b288bc900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work 2015-04-28 01:24:57.836844 7f5b288bc900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-04-28 01:24:57.837302 7f5b288bc900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2015-04-28 01:24:57.837375 7f5b288bc900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.12-031812-generic >= 3.5 2015-04-28 01:24:57.842378 7f5b288bc900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f5b288bc900 time 2015-04-28 01:24:57.839828 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f5b259ffec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -48> 2015-04-28 01:24:57.821546 7f5b288bc900 5 asok(0x3ff2000) register_command perfcounters_dump hook 0x3f92050 -47> 2015-04-28 01:24:57.821586 7f5b288bc900 5 asok(0x3ff2000) register_command 1 hook 0x3f92050 -46> 2015-04-28 01:24:57.821592 7f5b288bc900 5 asok(0x3ff2000) register_command perf dump hook 0x3f92050 -45> 2015-04-28 01:24:57.821599 7f5b288bc900 5 asok(0x3ff2000) register_command perfcounters_schema hook 0x3f92050 -44> 2015-04-28 01:24:57.821604 7f5b288bc900 5 asok(0x3ff2000) register_command 2 hook 0x3f92050 -43> 2015-04-28 01:24:57.821607 7f5b288bc900 5 asok(0x3ff2000) register_command perf schema hook 0x3f92050 -42> 2015-04-28 01:24:57.821614 7f5b288bc900 5 asok(0x3ff2000) register_command perf reset hook 0x3f92050 -41> 2015-04-28 01:24:57.821629 7f5b288bc900 5 asok(0x3ff2000) register_command config show hook 0x3f92050 -40> 2015-04-28 01:24:57.821635 7f5b288bc900 5 asok(0x3ff2000) register_command config set hook 0x3f92050 -39> 2015-04-28 01:24:57.821639 7f5b288bc900 5 asok(0x3ff2000) register_command config get hook 0x3f92050 -38> 2015-04-28 01:24:57.821644 7f5b288bc900 5 asok(0x3ff2000) register_command config diff hook 0x3f92050 -37> 2015-04-28 01:24:57.821648 7f5b288bc900 5 asok(0x3ff2000) register_command log flush hook 0x3f92050 -36> 2015-04-28 01:24:57.821653 7f5b288bc900 5 asok(0x3ff2000) register_command log dump hook 0x3f92050 -35> 2015-04-28 01:24:57.821657 7f5b288bc900 5 asok(0x3ff2000) register_command log reopen hook 0x3f92050 -34> 2015-04-28 01:24:57.823767 7f5b288bc900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 23148 -33> 2015-04-28 01:24:57.824707 7f5b288bc900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -32> 2015-04-28 01:24:57.824719 7f5b288bc900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6802/23148 need_addr=0 -31> 2015-04-28 01:24:57.824733 7f5b288bc900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -30> 2015-04-28 01:24:57.824739 7f5b288bc900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6802/23148 need_addr=0 -29> 2015-04-28 01:24:57.824749 7f5b288bc900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -28> 2015-04-28 01:24:57.824755 7f5b288bc900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6803/23148 need_addr=0 -27> 2015-04-28 01:24:57.824765 7f5b288bc900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -26> 2015-04-28 01:24:57.824771 7f5b288bc900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6803/23148 need_addr=0 -25> 2015-04-28 01:24:57.826098 7f5b288bc900 5 asok(0x3ff2000) init /var/run/ceph/ceph-osd.23.asok -24> 2015-04-28 01:24:57.826112 7f5b288bc900 5 asok(0x3ff2000) bind_and_listen /var/run/ceph/ceph-osd.23.asok -23> 2015-04-28 01:24:57.826216 7f5b288bc900 5 asok(0x3ff2000) register_command 0 hook 0x3f8e0a8 -22> 2015-04-28 01:24:57.826227 7f5b288bc900 5 asok(0x3ff2000) register_command version hook 0x3f8e0a8 -21> 2015-04-28 01:24:57.826231 7f5b288bc900 5 asok(0x3ff2000) register_command git_version hook 0x3f8e0a8 -20> 2015-04-28 01:24:57.826236 7f5b288bc900 5 asok(0x3ff2000) register_command help hook 0x3f92150 -19> 2015-04-28 01:24:57.826242 7f5b288bc900 5 asok(0x3ff2000) register_command get_command_descriptions hook 0x3f92140 -18> 2015-04-28 01:24:57.826281 7f5b288bc900 10 monclient(hunting): build_initial_monmap -17> 2015-04-28 01:24:57.826309 7f5b229c7700 5 asok(0x3ff2000) entry start -16> 2015-04-28 01:24:57.835421 7f5b288bc900 5 adding auth protocol: cephx -15> 2015-04-28 01:24:57.835430 7f5b288bc900 5 adding auth protocol: cephx -14> 2015-04-28 01:24:57.835524 7f5b288bc900 5 asok(0x3ff2000) register_command objecter_requests hook 0x3f92180 -13> 2015-04-28 01:24:57.835590 7f5b288bc900 1 -- 192.168.30.73:6802/23148 messenger.start -12> 2015-04-28 01:24:57.835622 7f5b288bc900 1 -- :/0 messenger.start -11> 2015-04-28 01:24:57.835644 7f5b288bc900 1 -- 192.168.30.73:6803/23148 messenger.start -10> 2015-04-28 01:24:57.835668 7f5b288bc900 1 -- 192.168.32.73:6803/23148 messenger.start -9> 2015-04-28 01:24:57.835694 7f5b288bc900 1 -- 192.168.32.73:6802/23148 messenger.start -8> 2015-04-28 01:24:57.835723 7f5b288bc900 1 -- :/0 messenger.start -7> 2015-04-28 01:24:57.835783 7f5b288bc900 2 osd.23 0 mounting /var/lib/ceph/osd/ceph-23 /var/lib/ceph/osd/ceph-23/journal -6> 2015-04-28 01:24:57.835823 7f5b288bc900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. -5> 2015-04-28 01:24:57.835850 7f5b288bc900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) -4> 2015-04-28 01:24:57.836834 7f5b288bc900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work -3> 2015-04-28 01:24:57.836844 7f5b288bc900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option -2> 2015-04-28 01:24:57.837302 7f5b288bc900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) -1> 2015-04-28 01:24:57.837375 7f5b288bc900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.12-031812-generic >= 3.5 0> 2015-04-28 01:24:57.842378 7f5b288bc900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f5b288bc900 time 2015-04-28 01:24:57.839828 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f5b259ffec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 01:24:57.845949 7f5b288bc900 -1 *** Caught signal (Aborted) ** in thread 7f5b288bc900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f5b27575340] 3: (gsignal()+0x39) [0x7f5b25a14cc9] 4: (abort()+0x148) [0x7f5b25a180d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f5b2631f6b5] 6: (()+0x5e836) [0x7f5b2631d836] 7: (()+0x5e863) [0x7f5b2631d863] 8: (()+0x5eaa2) [0x7f5b2631daa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f5b259ffec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2015-04-28 01:24:57.845949 7f5b288bc900 -1 *** Caught signal (Aborted) ** in thread 7f5b288bc900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f5b27575340] 3: (gsignal()+0x39) [0x7f5b25a14cc9] 4: (abort()+0x148) [0x7f5b25a180d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f5b2631f6b5] 6: (()+0x5e836) [0x7f5b2631d836] 7: (()+0x5e863) [0x7f5b2631d863] 8: (()+0x5eaa2) [0x7f5b2631daa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f5b259ffec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 01:24:58.245367 7f7d3b15e900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 23725 2015-04-28 01:24:58.257295 7f7d3b15e900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. 2015-04-28 01:24:58.257333 7f7d3b15e900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) 2015-04-28 01:24:58.258226 7f7d3b15e900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work 2015-04-28 01:24:58.258246 7f7d3b15e900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-04-28 01:24:58.258854 7f7d3b15e900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2015-04-28 01:24:58.258935 7f7d3b15e900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.12-031812-generic >= 3.5 2015-04-28 01:24:58.264037 7f7d3b15e900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f7d3b15e900 time 2015-04-28 01:24:58.261481 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f7d382a1ec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -48> 2015-04-28 01:24:58.243068 7f7d3b15e900 5 asok(0x4896000) register_command perfcounters_dump hook 0x4836050 -47> 2015-04-28 01:24:58.243110 7f7d3b15e900 5 asok(0x4896000) register_command 1 hook 0x4836050 -46> 2015-04-28 01:24:58.243115 7f7d3b15e900 5 asok(0x4896000) register_command perf dump hook 0x4836050 -45> 2015-04-28 01:24:58.243125 7f7d3b15e900 5 asok(0x4896000) register_command perfcounters_schema hook 0x4836050 -44> 2015-04-28 01:24:58.243131 7f7d3b15e900 5 asok(0x4896000) register_command 2 hook 0x4836050 -43> 2015-04-28 01:24:58.243134 7f7d3b15e900 5 asok(0x4896000) register_command perf schema hook 0x4836050 -42> 2015-04-28 01:24:58.243137 7f7d3b15e900 5 asok(0x4896000) register_command perf reset hook 0x4836050 -41> 2015-04-28 01:24:58.243150 7f7d3b15e900 5 asok(0x4896000) register_command config show hook 0x4836050 -40> 2015-04-28 01:24:58.243156 7f7d3b15e900 5 asok(0x4896000) register_command config set hook 0x4836050 -39> 2015-04-28 01:24:58.243158 7f7d3b15e900 5 asok(0x4896000) register_command config get hook 0x4836050 -38> 2015-04-28 01:24:58.243161 7f7d3b15e900 5 asok(0x4896000) register_command config diff hook 0x4836050 -37> 2015-04-28 01:24:58.243164 7f7d3b15e900 5 asok(0x4896000) register_command log flush hook 0x4836050 -36> 2015-04-28 01:24:58.243169 7f7d3b15e900 5 asok(0x4896000) register_command log dump hook 0x4836050 -35> 2015-04-28 01:24:58.243172 7f7d3b15e900 5 asok(0x4896000) register_command log reopen hook 0x4836050 -34> 2015-04-28 01:24:58.245367 7f7d3b15e900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 23725 -33> 2015-04-28 01:24:58.246234 7f7d3b15e900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -32> 2015-04-28 01:24:58.246245 7f7d3b15e900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6802/23725 need_addr=0 -31> 2015-04-28 01:24:58.246260 7f7d3b15e900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -30> 2015-04-28 01:24:58.246263 7f7d3b15e900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6802/23725 need_addr=0 -29> 2015-04-28 01:24:58.246273 7f7d3b15e900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -28> 2015-04-28 01:24:58.246276 7f7d3b15e900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6803/23725 need_addr=0 -27> 2015-04-28 01:24:58.246287 7f7d3b15e900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -26> 2015-04-28 01:24:58.246293 7f7d3b15e900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6803/23725 need_addr=0 -25> 2015-04-28 01:24:58.247565 7f7d3b15e900 5 asok(0x4896000) init /var/run/ceph/ceph-osd.23.asok -24> 2015-04-28 01:24:58.247576 7f7d3b15e900 5 asok(0x4896000) bind_and_listen /var/run/ceph/ceph-osd.23.asok -23> 2015-04-28 01:24:58.247670 7f7d3b15e900 5 asok(0x4896000) register_command 0 hook 0x48320a8 -22> 2015-04-28 01:24:58.247677 7f7d3b15e900 5 asok(0x4896000) register_command version hook 0x48320a8 -21> 2015-04-28 01:24:58.247680 7f7d3b15e900 5 asok(0x4896000) register_command git_version hook 0x48320a8 -20> 2015-04-28 01:24:58.247684 7f7d3b15e900 5 asok(0x4896000) register_command help hook 0x4836150 -19> 2015-04-28 01:24:58.247687 7f7d3b15e900 5 asok(0x4896000) register_command get_command_descriptions hook 0x4836140 -18> 2015-04-28 01:24:58.247726 7f7d3b15e900 10 monclient(hunting): build_initial_monmap -17> 2015-04-28 01:24:58.247723 7f7d35269700 5 asok(0x4896000) entry start -16> 2015-04-28 01:24:58.256887 7f7d3b15e900 5 adding auth protocol: cephx -15> 2015-04-28 01:24:58.256896 7f7d3b15e900 5 adding auth protocol: cephx -14> 2015-04-28 01:24:58.256993 7f7d3b15e900 5 asok(0x4896000) register_command objecter_requests hook 0x4836180 -13> 2015-04-28 01:24:58.257065 7f7d3b15e900 1 -- 192.168.30.73:6802/23725 messenger.start -12> 2015-04-28 01:24:58.257100 7f7d3b15e900 1 -- :/0 messenger.start -11> 2015-04-28 01:24:58.257125 7f7d3b15e900 1 -- 192.168.30.73:6803/23725 messenger.start -10> 2015-04-28 01:24:58.257151 7f7d3b15e900 1 -- 192.168.32.73:6803/23725 messenger.start -9> 2015-04-28 01:24:58.257175 7f7d3b15e900 1 -- 192.168.32.73:6802/23725 messenger.start -8> 2015-04-28 01:24:58.257199 7f7d3b15e900 1 -- :/0 messenger.start -7> 2015-04-28 01:24:58.257257 7f7d3b15e900 2 osd.23 0 mounting /var/lib/ceph/osd/ceph-23 /var/lib/ceph/osd/ceph-23/journal -6> 2015-04-28 01:24:58.257295 7f7d3b15e900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. -5> 2015-04-28 01:24:58.257333 7f7d3b15e900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) -4> 2015-04-28 01:24:58.258226 7f7d3b15e900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work -3> 2015-04-28 01:24:58.258246 7f7d3b15e900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option -2> 2015-04-28 01:24:58.258854 7f7d3b15e900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) -1> 2015-04-28 01:24:58.258935 7f7d3b15e900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.12-031812-generic >= 3.5 0> 2015-04-28 01:24:58.264037 7f7d3b15e900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f7d3b15e900 time 2015-04-28 01:24:58.261481 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f7d382a1ec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 01:24:58.267672 7f7d3b15e900 -1 *** Caught signal (Aborted) ** in thread 7f7d3b15e900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f7d39e17340] 3: (gsignal()+0x39) [0x7f7d382b6cc9] 4: (abort()+0x148) [0x7f7d382ba0d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f7d38bc16b5] 6: (()+0x5e836) [0x7f7d38bbf836] 7: (()+0x5e863) [0x7f7d38bbf863] 8: (()+0x5eaa2) [0x7f7d38bbfaa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f7d382a1ec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2015-04-28 01:24:58.267672 7f7d3b15e900 -1 *** Caught signal (Aborted) ** in thread 7f7d3b15e900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f7d39e17340] 3: (gsignal()+0x39) [0x7f7d382b6cc9] 4: (abort()+0x148) [0x7f7d382ba0d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f7d38bc16b5] 6: (()+0x5e836) [0x7f7d38bbf836] 7: (()+0x5e863) [0x7f7d38bbf863] 8: (()+0x5eaa2) [0x7f7d38bbfaa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f7d382a1ec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 01:24:58.660741 7f8473d3a900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 24304 2015-04-28 01:24:58.672863 7f8473d3a900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. 2015-04-28 01:24:58.672893 7f8473d3a900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) 2015-04-28 01:24:58.673873 7f8473d3a900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work 2015-04-28 01:24:58.673884 7f8473d3a900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-04-28 01:24:58.674347 7f8473d3a900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2015-04-28 01:24:58.674419 7f8473d3a900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.12-031812-generic >= 3.5 2015-04-28 01:24:58.679464 7f8473d3a900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f8473d3a900 time 2015-04-28 01:24:58.676917 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f8470e7dec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -48> 2015-04-28 01:24:58.658515 7f8473d3a900 5 asok(0x3b20000) register_command perfcounters_dump hook 0x3ac0050 -47> 2015-04-28 01:24:58.658558 7f8473d3a900 5 asok(0x3b20000) register_command 1 hook 0x3ac0050 -46> 2015-04-28 01:24:58.658575 7f8473d3a900 5 asok(0x3b20000) register_command perf dump hook 0x3ac0050 -45> 2015-04-28 01:24:58.658584 7f8473d3a900 5 asok(0x3b20000) register_command perfcounters_schema hook 0x3ac0050 -44> 2015-04-28 01:24:58.658589 7f8473d3a900 5 asok(0x3b20000) register_command 2 hook 0x3ac0050 -43> 2015-04-28 01:24:58.658592 7f8473d3a900 5 asok(0x3b20000) register_command perf schema hook 0x3ac0050 -42> 2015-04-28 01:24:58.658596 7f8473d3a900 5 asok(0x3b20000) register_command perf reset hook 0x3ac0050 -41> 2015-04-28 01:24:58.658609 7f8473d3a900 5 asok(0x3b20000) register_command config show hook 0x3ac0050 -40> 2015-04-28 01:24:58.658615 7f8473d3a900 5 asok(0x3b20000) register_command config set hook 0x3ac0050 -39> 2015-04-28 01:24:58.658617 7f8473d3a900 5 asok(0x3b20000) register_command config get hook 0x3ac0050 -38> 2015-04-28 01:24:58.658620 7f8473d3a900 5 asok(0x3b20000) register_command config diff hook 0x3ac0050 -37> 2015-04-28 01:24:58.658622 7f8473d3a900 5 asok(0x3b20000) register_command log flush hook 0x3ac0050 -36> 2015-04-28 01:24:58.658624 7f8473d3a900 5 asok(0x3b20000) register_command log dump hook 0x3ac0050 -35> 2015-04-28 01:24:58.658639 7f8473d3a900 5 asok(0x3b20000) register_command log reopen hook 0x3ac0050 -34> 2015-04-28 01:24:58.660741 7f8473d3a900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 24304 -33> 2015-04-28 01:24:58.661639 7f8473d3a900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -32> 2015-04-28 01:24:58.661650 7f8473d3a900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6802/24304 need_addr=0 -31> 2015-04-28 01:24:58.661666 7f8473d3a900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -30> 2015-04-28 01:24:58.661671 7f8473d3a900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6802/24304 need_addr=0 -29> 2015-04-28 01:24:58.661681 7f8473d3a900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -28> 2015-04-28 01:24:58.661686 7f8473d3a900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6803/24304 need_addr=0 -27> 2015-04-28 01:24:58.661696 7f8473d3a900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -26> 2015-04-28 01:24:58.661701 7f8473d3a900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6803/24304 need_addr=0 -25> 2015-04-28 01:24:58.662927 7f8473d3a900 5 asok(0x3b20000) init /var/run/ceph/ceph-osd.23.asok -24> 2015-04-28 01:24:58.662941 7f8473d3a900 5 asok(0x3b20000) bind_and_listen /var/run/ceph/ceph-osd.23.asok -23> 2015-04-28 01:24:58.663053 7f8473d3a900 5 asok(0x3b20000) register_command 0 hook 0x3abc0a8 -22> 2015-04-28 01:24:58.663061 7f8473d3a900 5 asok(0x3b20000) register_command version hook 0x3abc0a8 -21> 2015-04-28 01:24:58.663064 7f8473d3a900 5 asok(0x3b20000) register_command git_version hook 0x3abc0a8 -20> 2015-04-28 01:24:58.663067 7f8473d3a900 5 asok(0x3b20000) register_command help hook 0x3ac0150 -19> 2015-04-28 01:24:58.663070 7f8473d3a900 5 asok(0x3b20000) register_command get_command_descriptions hook 0x3ac0140 -18> 2015-04-28 01:24:58.663116 7f8473d3a900 10 monclient(hunting): build_initial_monmap -17> 2015-04-28 01:24:58.663112 7f846de45700 5 asok(0x3b20000) entry start -16> 2015-04-28 01:24:58.672447 7f8473d3a900 5 adding auth protocol: cephx -15> 2015-04-28 01:24:58.672459 7f8473d3a900 5 adding auth protocol: cephx -14> 2015-04-28 01:24:58.672573 7f8473d3a900 5 asok(0x3b20000) register_command objecter_requests hook 0x3ac0180 -13> 2015-04-28 01:24:58.672647 7f8473d3a900 1 -- 192.168.30.73:6802/24304 messenger.start -12> 2015-04-28 01:24:58.672679 7f8473d3a900 1 -- :/0 messenger.start -11> 2015-04-28 01:24:58.672701 7f8473d3a900 1 -- 192.168.30.73:6803/24304 messenger.start -10> 2015-04-28 01:24:58.672724 7f8473d3a900 1 -- 192.168.32.73:6803/24304 messenger.start -9> 2015-04-28 01:24:58.672747 7f8473d3a900 1 -- 192.168.32.73:6802/24304 messenger.start -8> 2015-04-28 01:24:58.672769 7f8473d3a900 1 -- :/0 messenger.start -7> 2015-04-28 01:24:58.672822 7f8473d3a900 2 osd.23 0 mounting /var/lib/ceph/osd/ceph-23 /var/lib/ceph/osd/ceph-23/journal -6> 2015-04-28 01:24:58.672863 7f8473d3a900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. -5> 2015-04-28 01:24:58.672893 7f8473d3a900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) -4> 2015-04-28 01:24:58.673873 7f8473d3a900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work -3> 2015-04-28 01:24:58.673884 7f8473d3a900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option -2> 2015-04-28 01:24:58.674347 7f8473d3a900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) -1> 2015-04-28 01:24:58.674419 7f8473d3a900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.12-031812-generic >= 3.5 0> 2015-04-28 01:24:58.679464 7f8473d3a900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f8473d3a900 time 2015-04-28 01:24:58.676917 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f8470e7dec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 01:24:58.683012 7f8473d3a900 -1 *** Caught signal (Aborted) ** in thread 7f8473d3a900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f84729f3340] 3: (gsignal()+0x39) [0x7f8470e92cc9] 4: (abort()+0x148) [0x7f8470e960d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f847179d6b5] 6: (()+0x5e836) [0x7f847179b836] 7: (()+0x5e863) [0x7f847179b863] 8: (()+0x5eaa2) [0x7f847179baa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f8470e7dec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2015-04-28 01:24:58.683012 7f8473d3a900 -1 *** Caught signal (Aborted) ** in thread 7f8473d3a900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f84729f3340] 3: (gsignal()+0x39) [0x7f8470e92cc9] 4: (abort()+0x148) [0x7f8470e960d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f847179d6b5] 6: (()+0x5e836) [0x7f847179b836] 7: (()+0x5e863) [0x7f847179b863] 8: (()+0x5eaa2) [0x7f847179baa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f8470e7dec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 01:24:59.090765 7fe745709900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 24927 2015-04-28 01:24:59.102951 7fe745709900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. 2015-04-28 01:24:59.102990 7fe745709900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) 2015-04-28 01:24:59.103971 7fe745709900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work 2015-04-28 01:24:59.103983 7fe745709900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-04-28 01:24:59.104630 7fe745709900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2015-04-28 01:24:59.104711 7fe745709900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.12-031812-generic >= 3.5 2015-04-28 01:24:59.109926 7fe745709900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7fe745709900 time 2015-04-28 01:24:59.107433 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7fe74284cec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -48> 2015-04-28 01:24:59.088553 7fe745709900 5 asok(0x4a4c000) register_command perfcounters_dump hook 0x49ec050 -47> 2015-04-28 01:24:59.088609 7fe745709900 5 asok(0x4a4c000) register_command 1 hook 0x49ec050 -46> 2015-04-28 01:24:59.088614 7fe745709900 5 asok(0x4a4c000) register_command perf dump hook 0x49ec050 -45> 2015-04-28 01:24:59.088621 7fe745709900 5 asok(0x4a4c000) register_command perfcounters_schema hook 0x49ec050 -44> 2015-04-28 01:24:59.088624 7fe745709900 5 asok(0x4a4c000) register_command 2 hook 0x49ec050 -43> 2015-04-28 01:24:59.088626 7fe745709900 5 asok(0x4a4c000) register_command perf schema hook 0x49ec050 -42> 2015-04-28 01:24:59.088643 7fe745709900 5 asok(0x4a4c000) register_command perf reset hook 0x49ec050 -41> 2015-04-28 01:24:59.088658 7fe745709900 5 asok(0x4a4c000) register_command config show hook 0x49ec050 -40> 2015-04-28 01:24:59.088664 7fe745709900 5 asok(0x4a4c000) register_command config set hook 0x49ec050 -39> 2015-04-28 01:24:59.088667 7fe745709900 5 asok(0x4a4c000) register_command config get hook 0x49ec050 -38> 2015-04-28 01:24:59.088669 7fe745709900 5 asok(0x4a4c000) register_command config diff hook 0x49ec050 -37> 2015-04-28 01:24:59.088671 7fe745709900 5 asok(0x4a4c000) register_command log flush hook 0x49ec050 -36> 2015-04-28 01:24:59.088674 7fe745709900 5 asok(0x4a4c000) register_command log dump hook 0x49ec050 -35> 2015-04-28 01:24:59.088676 7fe745709900 5 asok(0x4a4c000) register_command log reopen hook 0x49ec050 -34> 2015-04-28 01:24:59.090765 7fe745709900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 24927 -33> 2015-04-28 01:24:59.091675 7fe745709900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -32> 2015-04-28 01:24:59.091686 7fe745709900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6802/24927 need_addr=0 -31> 2015-04-28 01:24:59.091714 7fe745709900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -30> 2015-04-28 01:24:59.091718 7fe745709900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6802/24927 need_addr=0 -29> 2015-04-28 01:24:59.091728 7fe745709900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -28> 2015-04-28 01:24:59.091740 7fe745709900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6803/24927 need_addr=0 -27> 2015-04-28 01:24:59.091752 7fe745709900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -26> 2015-04-28 01:24:59.091757 7fe745709900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6803/24927 need_addr=0 -25> 2015-04-28 01:24:59.093123 7fe745709900 5 asok(0x4a4c000) init /var/run/ceph/ceph-osd.23.asok -24> 2015-04-28 01:24:59.093149 7fe745709900 5 asok(0x4a4c000) bind_and_listen /var/run/ceph/ceph-osd.23.asok -23> 2015-04-28 01:24:59.093262 7fe745709900 5 asok(0x4a4c000) register_command 0 hook 0x49e80a8 -22> 2015-04-28 01:24:59.093270 7fe745709900 5 asok(0x4a4c000) register_command version hook 0x49e80a8 -21> 2015-04-28 01:24:59.093285 7fe745709900 5 asok(0x4a4c000) register_command git_version hook 0x49e80a8 -20> 2015-04-28 01:24:59.093289 7fe745709900 5 asok(0x4a4c000) register_command help hook 0x49ec150 -19> 2015-04-28 01:24:59.093292 7fe745709900 5 asok(0x4a4c000) register_command get_command_descriptions hook 0x49ec140 -18> 2015-04-28 01:24:59.093332 7fe745709900 10 monclient(hunting): build_initial_monmap -17> 2015-04-28 01:24:59.093329 7fe73f814700 5 asok(0x4a4c000) entry start -16> 2015-04-28 01:24:59.102416 7fe745709900 5 adding auth protocol: cephx -15> 2015-04-28 01:24:59.102432 7fe745709900 5 adding auth protocol: cephx -14> 2015-04-28 01:24:59.102565 7fe745709900 5 asok(0x4a4c000) register_command objecter_requests hook 0x49ec180 -13> 2015-04-28 01:24:59.102671 7fe745709900 1 -- 192.168.30.73:6802/24927 messenger.start -12> 2015-04-28 01:24:59.102718 7fe745709900 1 -- :/0 messenger.start -11> 2015-04-28 01:24:59.102742 7fe745709900 1 -- 192.168.30.73:6803/24927 messenger.start -10> 2015-04-28 01:24:59.102767 7fe745709900 1 -- 192.168.32.73:6803/24927 messenger.start -9> 2015-04-28 01:24:59.102804 7fe745709900 1 -- 192.168.32.73:6802/24927 messenger.start -8> 2015-04-28 01:24:59.102826 7fe745709900 1 -- :/0 messenger.start -7> 2015-04-28 01:24:59.102896 7fe745709900 2 osd.23 0 mounting /var/lib/ceph/osd/ceph-23 /var/lib/ceph/osd/ceph-23/journal -6> 2015-04-28 01:24:59.102951 7fe745709900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. -5> 2015-04-28 01:24:59.102990 7fe745709900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) -4> 2015-04-28 01:24:59.103971 7fe745709900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work -3> 2015-04-28 01:24:59.103983 7fe745709900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option -2> 2015-04-28 01:24:59.104630 7fe745709900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) -1> 2015-04-28 01:24:59.104711 7fe745709900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.12-031812-generic >= 3.5 0> 2015-04-28 01:24:59.109926 7fe745709900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7fe745709900 time 2015-04-28 01:24:59.107433 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7fe74284cec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 01:24:59.113398 7fe745709900 -1 *** Caught signal (Aborted) ** in thread 7fe745709900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7fe7443c2340] 3: (gsignal()+0x39) [0x7fe742861cc9] 4: (abort()+0x148) [0x7fe7428650d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7fe74316c6b5] 6: (()+0x5e836) [0x7fe74316a836] 7: (()+0x5e863) [0x7fe74316a863] 8: (()+0x5eaa2) [0x7fe74316aaa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7fe74284cec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2015-04-28 01:24:59.113398 7fe745709900 -1 *** Caught signal (Aborted) ** in thread 7fe745709900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7fe7443c2340] 3: (gsignal()+0x39) [0x7fe742861cc9] 4: (abort()+0x148) [0x7fe7428650d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7fe74316c6b5] 6: (()+0x5e836) [0x7fe74316a836] 7: (()+0x5e863) [0x7fe74316a863] 8: (()+0x5eaa2) [0x7fe74316aaa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7fe74284cec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 01:24:59.562185 7fc7f8ee2900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 25407 2015-04-28 01:24:59.573824 7fc7f8ee2900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. 2015-04-28 01:24:59.573848 7fc7f8ee2900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) 2015-04-28 01:24:59.574748 7fc7f8ee2900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work 2015-04-28 01:24:59.574759 7fc7f8ee2900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-04-28 01:24:59.575432 7fc7f8ee2900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2015-04-28 01:24:59.575509 7fc7f8ee2900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.12-031812-generic >= 3.5 2015-04-28 01:24:59.580456 7fc7f8ee2900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7fc7f8ee2900 time 2015-04-28 01:24:59.577972 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7fc7f6025ec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -48> 2015-04-28 01:24:59.560007 7fc7f8ee2900 5 asok(0x47de000) register_command perfcounters_dump hook 0x477e050 -47> 2015-04-28 01:24:59.560037 7fc7f8ee2900 5 asok(0x47de000) register_command 1 hook 0x477e050 -46> 2015-04-28 01:24:59.560041 7fc7f8ee2900 5 asok(0x47de000) register_command perf dump hook 0x477e050 -45> 2015-04-28 01:24:59.560047 7fc7f8ee2900 5 asok(0x47de000) register_command perfcounters_schema hook 0x477e050 -44> 2015-04-28 01:24:59.560050 7fc7f8ee2900 5 asok(0x47de000) register_command 2 hook 0x477e050 -43> 2015-04-28 01:24:59.560052 7fc7f8ee2900 5 asok(0x47de000) register_command perf schema hook 0x477e050 -42> 2015-04-28 01:24:59.560055 7fc7f8ee2900 5 asok(0x47de000) register_command perf reset hook 0x477e050 -41> 2015-04-28 01:24:59.560061 7fc7f8ee2900 5 asok(0x47de000) register_command config show hook 0x477e050 -40> 2015-04-28 01:24:59.560065 7fc7f8ee2900 5 asok(0x47de000) register_command config set hook 0x477e050 -39> 2015-04-28 01:24:59.560067 7fc7f8ee2900 5 asok(0x47de000) register_command config get hook 0x477e050 -38> 2015-04-28 01:24:59.560070 7fc7f8ee2900 5 asok(0x47de000) register_command config diff hook 0x477e050 -37> 2015-04-28 01:24:59.560072 7fc7f8ee2900 5 asok(0x47de000) register_command log flush hook 0x477e050 -36> 2015-04-28 01:24:59.560075 7fc7f8ee2900 5 asok(0x47de000) register_command log dump hook 0x477e050 -35> 2015-04-28 01:24:59.560077 7fc7f8ee2900 5 asok(0x47de000) register_command log reopen hook 0x477e050 -34> 2015-04-28 01:24:59.562185 7fc7f8ee2900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 25407 -33> 2015-04-28 01:24:59.563031 7fc7f8ee2900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -32> 2015-04-28 01:24:59.563042 7fc7f8ee2900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6802/25407 need_addr=0 -31> 2015-04-28 01:24:59.563058 7fc7f8ee2900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -30> 2015-04-28 01:24:59.563063 7fc7f8ee2900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6802/25407 need_addr=0 -29> 2015-04-28 01:24:59.563075 7fc7f8ee2900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -28> 2015-04-28 01:24:59.563080 7fc7f8ee2900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6803/25407 need_addr=0 -27> 2015-04-28 01:24:59.563092 7fc7f8ee2900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -26> 2015-04-28 01:24:59.563097 7fc7f8ee2900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6803/25407 need_addr=0 -25> 2015-04-28 01:24:59.564420 7fc7f8ee2900 5 asok(0x47de000) init /var/run/ceph/ceph-osd.23.asok -24> 2015-04-28 01:24:59.564431 7fc7f8ee2900 5 asok(0x47de000) bind_and_listen /var/run/ceph/ceph-osd.23.asok -23> 2015-04-28 01:24:59.564529 7fc7f8ee2900 5 asok(0x47de000) register_command 0 hook 0x477a0a8 -22> 2015-04-28 01:24:59.564538 7fc7f8ee2900 5 asok(0x47de000) register_command version hook 0x477a0a8 -21> 2015-04-28 01:24:59.564543 7fc7f8ee2900 5 asok(0x47de000) register_command git_version hook 0x477a0a8 -20> 2015-04-28 01:24:59.564548 7fc7f8ee2900 5 asok(0x47de000) register_command help hook 0x477e150 -19> 2015-04-28 01:24:59.564553 7fc7f8ee2900 5 asok(0x47de000) register_command get_command_descriptions hook 0x477e140 -18> 2015-04-28 01:24:59.564595 7fc7f8ee2900 10 monclient(hunting): build_initial_monmap -17> 2015-04-28 01:24:59.564623 7fc7f2fed700 5 asok(0x47de000) entry start -16> 2015-04-28 01:24:59.573454 7fc7f8ee2900 5 adding auth protocol: cephx -15> 2015-04-28 01:24:59.573464 7fc7f8ee2900 5 adding auth protocol: cephx -14> 2015-04-28 01:24:59.573557 7fc7f8ee2900 5 asok(0x47de000) register_command objecter_requests hook 0x477e180 -13> 2015-04-28 01:24:59.573620 7fc7f8ee2900 1 -- 192.168.30.73:6802/25407 messenger.start -12> 2015-04-28 01:24:59.573650 7fc7f8ee2900 1 -- :/0 messenger.start -11> 2015-04-28 01:24:59.573672 7fc7f8ee2900 1 -- 192.168.30.73:6803/25407 messenger.start -10> 2015-04-28 01:24:59.573694 7fc7f8ee2900 1 -- 192.168.32.73:6803/25407 messenger.start -9> 2015-04-28 01:24:59.573715 7fc7f8ee2900 1 -- 192.168.32.73:6802/25407 messenger.start -8> 2015-04-28 01:24:59.573735 7fc7f8ee2900 1 -- :/0 messenger.start -7> 2015-04-28 01:24:59.573786 7fc7f8ee2900 2 osd.23 0 mounting /var/lib/ceph/osd/ceph-23 /var/lib/ceph/osd/ceph-23/journal -6> 2015-04-28 01:24:59.573824 7fc7f8ee2900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. -5> 2015-04-28 01:24:59.573848 7fc7f8ee2900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) -4> 2015-04-28 01:24:59.574748 7fc7f8ee2900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work -3> 2015-04-28 01:24:59.574759 7fc7f8ee2900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option -2> 2015-04-28 01:24:59.575432 7fc7f8ee2900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) -1> 2015-04-28 01:24:59.575509 7fc7f8ee2900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.12-031812-generic >= 3.5 0> 2015-04-28 01:24:59.580456 7fc7f8ee2900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7fc7f8ee2900 time 2015-04-28 01:24:59.577972 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7fc7f6025ec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 01:24:59.584130 7fc7f8ee2900 -1 *** Caught signal (Aborted) ** in thread 7fc7f8ee2900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7fc7f7b9b340] 3: (gsignal()+0x39) [0x7fc7f603acc9] 4: (abort()+0x148) [0x7fc7f603e0d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7fc7f69456b5] 6: (()+0x5e836) [0x7fc7f6943836] 7: (()+0x5e863) [0x7fc7f6943863] 8: (()+0x5eaa2) [0x7fc7f6943aa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7fc7f6025ec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2015-04-28 01:24:59.584130 7fc7f8ee2900 -1 *** Caught signal (Aborted) ** in thread 7fc7f8ee2900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7fc7f7b9b340] 3: (gsignal()+0x39) [0x7fc7f603acc9] 4: (abort()+0x148) [0x7fc7f603e0d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7fc7f69456b5] 6: (()+0x5e836) [0x7fc7f6943836] 7: (()+0x5e863) [0x7fc7f6943863] 8: (()+0x5eaa2) [0x7fc7f6943aa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7fc7f6025ec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 01:36:52.291182 7f42b0e42900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 51922 2015-04-28 01:36:52.302926 7f42b0e42900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. 2015-04-28 01:36:52.302953 7f42b0e42900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) 2015-04-28 01:36:52.303926 7f42b0e42900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work 2015-04-28 01:36:52.303937 7f42b0e42900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-04-28 01:36:52.304256 7f42b0e42900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2015-04-28 01:36:52.304338 7f42b0e42900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.12-031812-generic >= 3.5 2015-04-28 01:36:52.309209 7f42b0e42900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f42b0e42900 time 2015-04-28 01:36:52.306658 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f42adf85ec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -48> 2015-04-28 01:36:52.288999 7f42b0e42900 5 asok(0x3b98000) register_command perfcounters_dump hook 0x3b38050 -47> 2015-04-28 01:36:52.289032 7f42b0e42900 5 asok(0x3b98000) register_command 1 hook 0x3b38050 -46> 2015-04-28 01:36:52.289038 7f42b0e42900 5 asok(0x3b98000) register_command perf dump hook 0x3b38050 -45> 2015-04-28 01:36:52.289046 7f42b0e42900 5 asok(0x3b98000) register_command perfcounters_schema hook 0x3b38050 -44> 2015-04-28 01:36:52.289051 7f42b0e42900 5 asok(0x3b98000) register_command 2 hook 0x3b38050 -43> 2015-04-28 01:36:52.289053 7f42b0e42900 5 asok(0x3b98000) register_command perf schema hook 0x3b38050 -42> 2015-04-28 01:36:52.289058 7f42b0e42900 5 asok(0x3b98000) register_command perf reset hook 0x3b38050 -41> 2015-04-28 01:36:52.289067 7f42b0e42900 5 asok(0x3b98000) register_command config show hook 0x3b38050 -40> 2015-04-28 01:36:52.289072 7f42b0e42900 5 asok(0x3b98000) register_command config set hook 0x3b38050 -39> 2015-04-28 01:36:52.289076 7f42b0e42900 5 asok(0x3b98000) register_command config get hook 0x3b38050 -38> 2015-04-28 01:36:52.289079 7f42b0e42900 5 asok(0x3b98000) register_command config diff hook 0x3b38050 -37> 2015-04-28 01:36:52.289083 7f42b0e42900 5 asok(0x3b98000) register_command log flush hook 0x3b38050 -36> 2015-04-28 01:36:52.289086 7f42b0e42900 5 asok(0x3b98000) register_command log dump hook 0x3b38050 -35> 2015-04-28 01:36:52.289090 7f42b0e42900 5 asok(0x3b98000) register_command log reopen hook 0x3b38050 -34> 2015-04-28 01:36:52.291182 7f42b0e42900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 51922 -33> 2015-04-28 01:36:52.291961 7f42b0e42900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -32> 2015-04-28 01:36:52.291973 7f42b0e42900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6806/51922 need_addr=0 -31> 2015-04-28 01:36:52.291997 7f42b0e42900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -30> 2015-04-28 01:36:52.292003 7f42b0e42900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6806/51922 need_addr=0 -29> 2015-04-28 01:36:52.292016 7f42b0e42900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -28> 2015-04-28 01:36:52.292020 7f42b0e42900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6807/51922 need_addr=0 -27> 2015-04-28 01:36:52.292033 7f42b0e42900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -26> 2015-04-28 01:36:52.292038 7f42b0e42900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6807/51922 need_addr=0 -25> 2015-04-28 01:36:52.293288 7f42b0e42900 5 asok(0x3b98000) init /var/run/ceph/ceph-osd.23.asok -24> 2015-04-28 01:36:52.293300 7f42b0e42900 5 asok(0x3b98000) bind_and_listen /var/run/ceph/ceph-osd.23.asok -23> 2015-04-28 01:36:52.293385 7f42b0e42900 5 asok(0x3b98000) register_command 0 hook 0x3b340a8 -22> 2015-04-28 01:36:52.293393 7f42b0e42900 5 asok(0x3b98000) register_command version hook 0x3b340a8 -21> 2015-04-28 01:36:52.293397 7f42b0e42900 5 asok(0x3b98000) register_command git_version hook 0x3b340a8 -20> 2015-04-28 01:36:52.293403 7f42b0e42900 5 asok(0x3b98000) register_command help hook 0x3b38150 -19> 2015-04-28 01:36:52.293406 7f42b0e42900 5 asok(0x3b98000) register_command get_command_descriptions hook 0x3b38140 -18> 2015-04-28 01:36:52.293438 7f42aaf4d700 5 asok(0x3b98000) entry start -17> 2015-04-28 01:36:52.293443 7f42b0e42900 10 monclient(hunting): build_initial_monmap -16> 2015-04-28 01:36:52.302497 7f42b0e42900 5 adding auth protocol: cephx -15> 2015-04-28 01:36:52.302520 7f42b0e42900 5 adding auth protocol: cephx -14> 2015-04-28 01:36:52.302630 7f42b0e42900 5 asok(0x3b98000) register_command objecter_requests hook 0x3b38180 -13> 2015-04-28 01:36:52.302707 7f42b0e42900 1 -- 192.168.30.73:6806/51922 messenger.start -12> 2015-04-28 01:36:52.302739 7f42b0e42900 1 -- :/0 messenger.start -11> 2015-04-28 01:36:52.302761 7f42b0e42900 1 -- 192.168.30.73:6807/51922 messenger.start -10> 2015-04-28 01:36:52.302783 7f42b0e42900 1 -- 192.168.32.73:6807/51922 messenger.start -9> 2015-04-28 01:36:52.302807 7f42b0e42900 1 -- 192.168.32.73:6806/51922 messenger.start -8> 2015-04-28 01:36:52.302826 7f42b0e42900 1 -- :/0 messenger.start -7> 2015-04-28 01:36:52.302882 7f42b0e42900 2 osd.23 0 mounting /var/lib/ceph/osd/ceph-23 /var/lib/ceph/osd/ceph-23/journal -6> 2015-04-28 01:36:52.302926 7f42b0e42900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. -5> 2015-04-28 01:36:52.302953 7f42b0e42900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) -4> 2015-04-28 01:36:52.303926 7f42b0e42900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work -3> 2015-04-28 01:36:52.303937 7f42b0e42900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option -2> 2015-04-28 01:36:52.304256 7f42b0e42900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) -1> 2015-04-28 01:36:52.304338 7f42b0e42900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.12-031812-generic >= 3.5 0> 2015-04-28 01:36:52.309209 7f42b0e42900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f42b0e42900 time 2015-04-28 01:36:52.306658 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f42adf85ec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 01:36:52.312850 7f42b0e42900 -1 *** Caught signal (Aborted) ** in thread 7f42b0e42900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f42afafb340] 3: (gsignal()+0x39) [0x7f42adf9acc9] 4: (abort()+0x148) [0x7f42adf9e0d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f42ae8a56b5] 6: (()+0x5e836) [0x7f42ae8a3836] 7: (()+0x5e863) [0x7f42ae8a3863] 8: (()+0x5eaa2) [0x7f42ae8a3aa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f42adf85ec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2015-04-28 01:36:52.312850 7f42b0e42900 -1 *** Caught signal (Aborted) ** in thread 7f42b0e42900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f42afafb340] 3: (gsignal()+0x39) [0x7f42adf9acc9] 4: (abort()+0x148) [0x7f42adf9e0d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f42ae8a56b5] 6: (()+0x5e836) [0x7f42ae8a3836] 7: (()+0x5e863) [0x7f42ae8a3863] 8: (()+0x5eaa2) [0x7f42ae8a3aa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f42adf85ec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 01:36:52.708001 7f9e11a3e900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 52552 2015-04-28 01:36:52.720205 7f9e11a3e900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. 2015-04-28 01:36:52.720234 7f9e11a3e900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) 2015-04-28 01:36:52.721188 7f9e11a3e900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work 2015-04-28 01:36:52.721197 7f9e11a3e900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-04-28 01:36:52.721639 7f9e11a3e900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2015-04-28 01:36:52.721697 7f9e11a3e900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.12-031812-generic >= 3.5 2015-04-28 01:36:52.726536 7f9e11a3e900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f9e11a3e900 time 2015-04-28 01:36:52.723997 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f9e0eb81ec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -48> 2015-04-28 01:36:52.705162 7f9e11a3e900 5 asok(0x3f10000) register_command perfcounters_dump hook 0x3eb0050 -47> 2015-04-28 01:36:52.705197 7f9e11a3e900 5 asok(0x3f10000) register_command 1 hook 0x3eb0050 -46> 2015-04-28 01:36:52.705201 7f9e11a3e900 5 asok(0x3f10000) register_command perf dump hook 0x3eb0050 -45> 2015-04-28 01:36:52.705207 7f9e11a3e900 5 asok(0x3f10000) register_command perfcounters_schema hook 0x3eb0050 -44> 2015-04-28 01:36:52.705210 7f9e11a3e900 5 asok(0x3f10000) register_command 2 hook 0x3eb0050 -43> 2015-04-28 01:36:52.705212 7f9e11a3e900 5 asok(0x3f10000) register_command perf schema hook 0x3eb0050 -42> 2015-04-28 01:36:52.705215 7f9e11a3e900 5 asok(0x3f10000) register_command perf reset hook 0x3eb0050 -41> 2015-04-28 01:36:52.705227 7f9e11a3e900 5 asok(0x3f10000) register_command config show hook 0x3eb0050 -40> 2015-04-28 01:36:52.705240 7f9e11a3e900 5 asok(0x3f10000) register_command config set hook 0x3eb0050 -39> 2015-04-28 01:36:52.705245 7f9e11a3e900 5 asok(0x3f10000) register_command config get hook 0x3eb0050 -38> 2015-04-28 01:36:52.705248 7f9e11a3e900 5 asok(0x3f10000) register_command config diff hook 0x3eb0050 -37> 2015-04-28 01:36:52.705252 7f9e11a3e900 5 asok(0x3f10000) register_command log flush hook 0x3eb0050 -36> 2015-04-28 01:36:52.705255 7f9e11a3e900 5 asok(0x3f10000) register_command log dump hook 0x3eb0050 -35> 2015-04-28 01:36:52.705259 7f9e11a3e900 5 asok(0x3f10000) register_command log reopen hook 0x3eb0050 -34> 2015-04-28 01:36:52.708001 7f9e11a3e900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 52552 -33> 2015-04-28 01:36:52.709007 7f9e11a3e900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -32> 2015-04-28 01:36:52.709019 7f9e11a3e900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6804/52552 need_addr=0 -31> 2015-04-28 01:36:52.709035 7f9e11a3e900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -30> 2015-04-28 01:36:52.709040 7f9e11a3e900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6804/52552 need_addr=0 -29> 2015-04-28 01:36:52.709052 7f9e11a3e900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -28> 2015-04-28 01:36:52.709057 7f9e11a3e900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6805/52552 need_addr=0 -27> 2015-04-28 01:36:52.709068 7f9e11a3e900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -26> 2015-04-28 01:36:52.709073 7f9e11a3e900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6805/52552 need_addr=0 -25> 2015-04-28 01:36:52.710396 7f9e11a3e900 5 asok(0x3f10000) init /var/run/ceph/ceph-osd.23.asok -24> 2015-04-28 01:36:52.710408 7f9e11a3e900 5 asok(0x3f10000) bind_and_listen /var/run/ceph/ceph-osd.23.asok -23> 2015-04-28 01:36:52.710506 7f9e11a3e900 5 asok(0x3f10000) register_command 0 hook 0x3eac0a8 -22> 2015-04-28 01:36:52.710514 7f9e11a3e900 5 asok(0x3f10000) register_command version hook 0x3eac0a8 -21> 2015-04-28 01:36:52.710517 7f9e11a3e900 5 asok(0x3f10000) register_command git_version hook 0x3eac0a8 -20> 2015-04-28 01:36:52.710523 7f9e11a3e900 5 asok(0x3f10000) register_command help hook 0x3eb0150 -19> 2015-04-28 01:36:52.710527 7f9e11a3e900 5 asok(0x3f10000) register_command get_command_descriptions hook 0x3eb0140 -18> 2015-04-28 01:36:52.710569 7f9e11a3e900 10 monclient(hunting): build_initial_monmap -17> 2015-04-28 01:36:52.710569 7f9e0bb49700 5 asok(0x3f10000) entry start -16> 2015-04-28 01:36:52.719771 7f9e11a3e900 5 adding auth protocol: cephx -15> 2015-04-28 01:36:52.719781 7f9e11a3e900 5 adding auth protocol: cephx -14> 2015-04-28 01:36:52.719887 7f9e11a3e900 5 asok(0x3f10000) register_command objecter_requests hook 0x3eb0180 -13> 2015-04-28 01:36:52.719979 7f9e11a3e900 1 -- 192.168.30.73:6804/52552 messenger.start -12> 2015-04-28 01:36:52.720018 7f9e11a3e900 1 -- :/0 messenger.start -11> 2015-04-28 01:36:52.720041 7f9e11a3e900 1 -- 192.168.30.73:6805/52552 messenger.start -10> 2015-04-28 01:36:52.720065 7f9e11a3e900 1 -- 192.168.32.73:6805/52552 messenger.start -9> 2015-04-28 01:36:52.720088 7f9e11a3e900 1 -- 192.168.32.73:6804/52552 messenger.start -8> 2015-04-28 01:36:52.720109 7f9e11a3e900 1 -- :/0 messenger.start -7> 2015-04-28 01:36:52.720162 7f9e11a3e900 2 osd.23 0 mounting /var/lib/ceph/osd/ceph-23 /var/lib/ceph/osd/ceph-23/journal -6> 2015-04-28 01:36:52.720205 7f9e11a3e900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. -5> 2015-04-28 01:36:52.720234 7f9e11a3e900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) -4> 2015-04-28 01:36:52.721188 7f9e11a3e900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work -3> 2015-04-28 01:36:52.721197 7f9e11a3e900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option -2> 2015-04-28 01:36:52.721639 7f9e11a3e900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) -1> 2015-04-28 01:36:52.721697 7f9e11a3e900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.12-031812-generic >= 3.5 0> 2015-04-28 01:36:52.726536 7f9e11a3e900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f9e11a3e900 time 2015-04-28 01:36:52.723997 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f9e0eb81ec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 01:36:52.730193 7f9e11a3e900 -1 *** Caught signal (Aborted) ** in thread 7f9e11a3e900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f9e106f7340] 3: (gsignal()+0x39) [0x7f9e0eb96cc9] 4: (abort()+0x148) [0x7f9e0eb9a0d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f9e0f4a16b5] 6: (()+0x5e836) [0x7f9e0f49f836] 7: (()+0x5e863) [0x7f9e0f49f863] 8: (()+0x5eaa2) [0x7f9e0f49faa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f9e0eb81ec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2015-04-28 01:36:52.730193 7f9e11a3e900 -1 *** Caught signal (Aborted) ** in thread 7f9e11a3e900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f9e106f7340] 3: (gsignal()+0x39) [0x7f9e0eb96cc9] 4: (abort()+0x148) [0x7f9e0eb9a0d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f9e0f4a16b5] 6: (()+0x5e836) [0x7f9e0f49f836] 7: (()+0x5e863) [0x7f9e0f49f863] 8: (()+0x5eaa2) [0x7f9e0f49faa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f9e0eb81ec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 01:36:53.115092 7f0e2d089900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 53105 2015-04-28 01:36:53.126983 7f0e2d089900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. 2015-04-28 01:36:53.127009 7f0e2d089900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) 2015-04-28 01:36:53.128000 7f0e2d089900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work 2015-04-28 01:36:53.128011 7f0e2d089900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-04-28 01:36:53.128537 7f0e2d089900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2015-04-28 01:36:53.128604 7f0e2d089900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.12-031812-generic >= 3.5 2015-04-28 01:36:53.133666 7f0e2d089900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f0e2d089900 time 2015-04-28 01:36:53.131063 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f0e2a1ccec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -48> 2015-04-28 01:36:53.112897 7f0e2d089900 5 asok(0x49a2000) register_command perfcounters_dump hook 0x4942050 -47> 2015-04-28 01:36:53.112926 7f0e2d089900 5 asok(0x49a2000) register_command 1 hook 0x4942050 -46> 2015-04-28 01:36:53.112932 7f0e2d089900 5 asok(0x49a2000) register_command perf dump hook 0x4942050 -45> 2015-04-28 01:36:53.112940 7f0e2d089900 5 asok(0x49a2000) register_command perfcounters_schema hook 0x4942050 -44> 2015-04-28 01:36:53.112945 7f0e2d089900 5 asok(0x49a2000) register_command 2 hook 0x4942050 -43> 2015-04-28 01:36:53.112948 7f0e2d089900 5 asok(0x49a2000) register_command perf schema hook 0x4942050 -42> 2015-04-28 01:36:53.112953 7f0e2d089900 5 asok(0x49a2000) register_command perf reset hook 0x4942050 -41> 2015-04-28 01:36:53.112959 7f0e2d089900 5 asok(0x49a2000) register_command config show hook 0x4942050 -40> 2015-04-28 01:36:53.112964 7f0e2d089900 5 asok(0x49a2000) register_command config set hook 0x4942050 -39> 2015-04-28 01:36:53.112968 7f0e2d089900 5 asok(0x49a2000) register_command config get hook 0x4942050 -38> 2015-04-28 01:36:53.112972 7f0e2d089900 5 asok(0x49a2000) register_command config diff hook 0x4942050 -37> 2015-04-28 01:36:53.112977 7f0e2d089900 5 asok(0x49a2000) register_command log flush hook 0x4942050 -36> 2015-04-28 01:36:53.112981 7f0e2d089900 5 asok(0x49a2000) register_command log dump hook 0x4942050 -35> 2015-04-28 01:36:53.112985 7f0e2d089900 5 asok(0x49a2000) register_command log reopen hook 0x4942050 -34> 2015-04-28 01:36:53.115092 7f0e2d089900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 53105 -33> 2015-04-28 01:36:53.115880 7f0e2d089900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -32> 2015-04-28 01:36:53.115890 7f0e2d089900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6802/53105 need_addr=0 -31> 2015-04-28 01:36:53.115917 7f0e2d089900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -30> 2015-04-28 01:36:53.115923 7f0e2d089900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6802/53105 need_addr=0 -29> 2015-04-28 01:36:53.115935 7f0e2d089900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -28> 2015-04-28 01:36:53.115940 7f0e2d089900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6803/53105 need_addr=0 -27> 2015-04-28 01:36:53.115950 7f0e2d089900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -26> 2015-04-28 01:36:53.115955 7f0e2d089900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6803/53105 need_addr=0 -25> 2015-04-28 01:36:53.117159 7f0e2d089900 5 asok(0x49a2000) init /var/run/ceph/ceph-osd.23.asok -24> 2015-04-28 01:36:53.117171 7f0e2d089900 5 asok(0x49a2000) bind_and_listen /var/run/ceph/ceph-osd.23.asok -23> 2015-04-28 01:36:53.117260 7f0e2d089900 5 asok(0x49a2000) register_command 0 hook 0x493e0a8 -22> 2015-04-28 01:36:53.117268 7f0e2d089900 5 asok(0x49a2000) register_command version hook 0x493e0a8 -21> 2015-04-28 01:36:53.117271 7f0e2d089900 5 asok(0x49a2000) register_command git_version hook 0x493e0a8 -20> 2015-04-28 01:36:53.117276 7f0e2d089900 5 asok(0x49a2000) register_command help hook 0x4942150 -19> 2015-04-28 01:36:53.117280 7f0e2d089900 5 asok(0x49a2000) register_command get_command_descriptions hook 0x4942140 -18> 2015-04-28 01:36:53.117321 7f0e2d089900 10 monclient(hunting): build_initial_monmap -17> 2015-04-28 01:36:53.117317 7f0e27194700 5 asok(0x49a2000) entry start -16> 2015-04-28 01:36:53.126619 7f0e2d089900 5 adding auth protocol: cephx -15> 2015-04-28 01:36:53.126628 7f0e2d089900 5 adding auth protocol: cephx -14> 2015-04-28 01:36:53.126719 7f0e2d089900 5 asok(0x49a2000) register_command objecter_requests hook 0x4942180 -13> 2015-04-28 01:36:53.126778 7f0e2d089900 1 -- 192.168.30.73:6802/53105 messenger.start -12> 2015-04-28 01:36:53.126808 7f0e2d089900 1 -- :/0 messenger.start -11> 2015-04-28 01:36:53.126829 7f0e2d089900 1 -- 192.168.30.73:6803/53105 messenger.start -10> 2015-04-28 01:36:53.126851 7f0e2d089900 1 -- 192.168.32.73:6803/53105 messenger.start -9> 2015-04-28 01:36:53.126871 7f0e2d089900 1 -- 192.168.32.73:6802/53105 messenger.start -8> 2015-04-28 01:36:53.126891 7f0e2d089900 1 -- :/0 messenger.start -7> 2015-04-28 01:36:53.126945 7f0e2d089900 2 osd.23 0 mounting /var/lib/ceph/osd/ceph-23 /var/lib/ceph/osd/ceph-23/journal -6> 2015-04-28 01:36:53.126983 7f0e2d089900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. -5> 2015-04-28 01:36:53.127009 7f0e2d089900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) -4> 2015-04-28 01:36:53.128000 7f0e2d089900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work -3> 2015-04-28 01:36:53.128011 7f0e2d089900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option -2> 2015-04-28 01:36:53.128537 7f0e2d089900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) -1> 2015-04-28 01:36:53.128604 7f0e2d089900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.12-031812-generic >= 3.5 0> 2015-04-28 01:36:53.133666 7f0e2d089900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f0e2d089900 time 2015-04-28 01:36:53.131063 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f0e2a1ccec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 01:36:53.137315 7f0e2d089900 -1 *** Caught signal (Aborted) ** in thread 7f0e2d089900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f0e2bd42340] 3: (gsignal()+0x39) [0x7f0e2a1e1cc9] 4: (abort()+0x148) [0x7f0e2a1e50d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f0e2aaec6b5] 6: (()+0x5e836) [0x7f0e2aaea836] 7: (()+0x5e863) [0x7f0e2aaea863] 8: (()+0x5eaa2) [0x7f0e2aaeaaa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f0e2a1ccec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2015-04-28 01:36:53.137315 7f0e2d089900 -1 *** Caught signal (Aborted) ** in thread 7f0e2d089900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f0e2bd42340] 3: (gsignal()+0x39) [0x7f0e2a1e1cc9] 4: (abort()+0x148) [0x7f0e2a1e50d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f0e2aaec6b5] 6: (()+0x5e836) [0x7f0e2aaea836] 7: (()+0x5e863) [0x7f0e2aaea863] 8: (()+0x5eaa2) [0x7f0e2aaeaaa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f0e2a1ccec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 01:36:53.583112 7f50f4c36900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 53709 2015-04-28 01:36:53.595382 7f50f4c36900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. 2015-04-28 01:36:53.595418 7f50f4c36900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) 2015-04-28 01:36:53.596391 7f50f4c36900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work 2015-04-28 01:36:53.596404 7f50f4c36900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-04-28 01:36:53.596935 7f50f4c36900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2015-04-28 01:36:53.597003 7f50f4c36900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.12-031812-generic >= 3.5 2015-04-28 01:36:53.603293 7f50f4c36900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f50f4c36900 time 2015-04-28 01:36:53.600289 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f50f1d79ec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -48> 2015-04-28 01:36:53.580681 7f50f4c36900 5 asok(0x3e22000) register_command perfcounters_dump hook 0x3dc2050 -47> 2015-04-28 01:36:53.580725 7f50f4c36900 5 asok(0x3e22000) register_command 1 hook 0x3dc2050 -46> 2015-04-28 01:36:53.580735 7f50f4c36900 5 asok(0x3e22000) register_command perf dump hook 0x3dc2050 -45> 2015-04-28 01:36:53.580747 7f50f4c36900 5 asok(0x3e22000) register_command perfcounters_schema hook 0x3dc2050 -44> 2015-04-28 01:36:53.580752 7f50f4c36900 5 asok(0x3e22000) register_command 2 hook 0x3dc2050 -43> 2015-04-28 01:36:53.580754 7f50f4c36900 5 asok(0x3e22000) register_command perf schema hook 0x3dc2050 -42> 2015-04-28 01:36:53.580762 7f50f4c36900 5 asok(0x3e22000) register_command perf reset hook 0x3dc2050 -41> 2015-04-28 01:36:53.580777 7f50f4c36900 5 asok(0x3e22000) register_command config show hook 0x3dc2050 -40> 2015-04-28 01:36:53.580782 7f50f4c36900 5 asok(0x3e22000) register_command config set hook 0x3dc2050 -39> 2015-04-28 01:36:53.580784 7f50f4c36900 5 asok(0x3e22000) register_command config get hook 0x3dc2050 -38> 2015-04-28 01:36:53.580789 7f50f4c36900 5 asok(0x3e22000) register_command config diff hook 0x3dc2050 -37> 2015-04-28 01:36:53.580794 7f50f4c36900 5 asok(0x3e22000) register_command log flush hook 0x3dc2050 -36> 2015-04-28 01:36:53.580797 7f50f4c36900 5 asok(0x3e22000) register_command log dump hook 0x3dc2050 -35> 2015-04-28 01:36:53.580802 7f50f4c36900 5 asok(0x3e22000) register_command log reopen hook 0x3dc2050 -34> 2015-04-28 01:36:53.583112 7f50f4c36900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 53709 -33> 2015-04-28 01:36:53.583992 7f50f4c36900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -32> 2015-04-28 01:36:53.584005 7f50f4c36900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6802/53709 need_addr=0 -31> 2015-04-28 01:36:53.584029 7f50f4c36900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -30> 2015-04-28 01:36:53.584035 7f50f4c36900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6802/53709 need_addr=0 -29> 2015-04-28 01:36:53.584046 7f50f4c36900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -28> 2015-04-28 01:36:53.584052 7f50f4c36900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6803/53709 need_addr=0 -27> 2015-04-28 01:36:53.584069 7f50f4c36900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -26> 2015-04-28 01:36:53.584075 7f50f4c36900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6803/53709 need_addr=0 -25> 2015-04-28 01:36:53.585437 7f50f4c36900 5 asok(0x3e22000) init /var/run/ceph/ceph-osd.23.asok -24> 2015-04-28 01:36:53.585451 7f50f4c36900 5 asok(0x3e22000) bind_and_listen /var/run/ceph/ceph-osd.23.asok -23> 2015-04-28 01:36:53.585562 7f50f4c36900 5 asok(0x3e22000) register_command 0 hook 0x3dbe0a8 -22> 2015-04-28 01:36:53.585571 7f50f4c36900 5 asok(0x3e22000) register_command version hook 0x3dbe0a8 -21> 2015-04-28 01:36:53.585577 7f50f4c36900 5 asok(0x3e22000) register_command git_version hook 0x3dbe0a8 -20> 2015-04-28 01:36:53.585583 7f50f4c36900 5 asok(0x3e22000) register_command help hook 0x3dc2150 -19> 2015-04-28 01:36:53.585588 7f50f4c36900 5 asok(0x3e22000) register_command get_command_descriptions hook 0x3dc2140 -18> 2015-04-28 01:36:53.585624 7f50f4c36900 10 monclient(hunting): build_initial_monmap -17> 2015-04-28 01:36:53.588293 7f50eed41700 5 asok(0x3e22000) entry start -16> 2015-04-28 01:36:53.594977 7f50f4c36900 5 adding auth protocol: cephx -15> 2015-04-28 01:36:53.594988 7f50f4c36900 5 adding auth protocol: cephx -14> 2015-04-28 01:36:53.595092 7f50f4c36900 5 asok(0x3e22000) register_command objecter_requests hook 0x3dc2180 -13> 2015-04-28 01:36:53.595161 7f50f4c36900 1 -- 192.168.30.73:6802/53709 messenger.start -12> 2015-04-28 01:36:53.595193 7f50f4c36900 1 -- :/0 messenger.start -11> 2015-04-28 01:36:53.595215 7f50f4c36900 1 -- 192.168.30.73:6803/53709 messenger.start -10> 2015-04-28 01:36:53.595237 7f50f4c36900 1 -- 192.168.32.73:6803/53709 messenger.start -9> 2015-04-28 01:36:53.595258 7f50f4c36900 1 -- 192.168.32.73:6802/53709 messenger.start -8> 2015-04-28 01:36:53.595279 7f50f4c36900 1 -- :/0 messenger.start -7> 2015-04-28 01:36:53.595331 7f50f4c36900 2 osd.23 0 mounting /var/lib/ceph/osd/ceph-23 /var/lib/ceph/osd/ceph-23/journal -6> 2015-04-28 01:36:53.595382 7f50f4c36900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. -5> 2015-04-28 01:36:53.595418 7f50f4c36900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) -4> 2015-04-28 01:36:53.596391 7f50f4c36900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work -3> 2015-04-28 01:36:53.596404 7f50f4c36900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option -2> 2015-04-28 01:36:53.596935 7f50f4c36900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) -1> 2015-04-28 01:36:53.597003 7f50f4c36900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.12-031812-generic >= 3.5 0> 2015-04-28 01:36:53.603293 7f50f4c36900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f50f4c36900 time 2015-04-28 01:36:53.600289 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f50f1d79ec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 01:36:53.607187 7f50f4c36900 -1 *** Caught signal (Aborted) ** in thread 7f50f4c36900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f50f38ef340] 3: (gsignal()+0x39) [0x7f50f1d8ecc9] 4: (abort()+0x148) [0x7f50f1d920d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f50f26996b5] 6: (()+0x5e836) [0x7f50f2697836] 7: (()+0x5e863) [0x7f50f2697863] 8: (()+0x5eaa2) [0x7f50f2697aa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f50f1d79ec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2015-04-28 01:36:53.607187 7f50f4c36900 -1 *** Caught signal (Aborted) ** in thread 7f50f4c36900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f50f38ef340] 3: (gsignal()+0x39) [0x7f50f1d8ecc9] 4: (abort()+0x148) [0x7f50f1d920d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f50f26996b5] 6: (()+0x5e836) [0x7f50f2697836] 7: (()+0x5e863) [0x7f50f2697863] 8: (()+0x5eaa2) [0x7f50f2697aa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f50f1d79ec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 01:36:54.037220 7f9ccaf7c900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 54355 2015-04-28 01:36:54.048540 7f9ccaf7c900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. 2015-04-28 01:36:54.048563 7f9ccaf7c900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) 2015-04-28 01:36:54.049843 7f9ccaf7c900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work 2015-04-28 01:36:54.049855 7f9ccaf7c900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-04-28 01:36:54.051016 7f9ccaf7c900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2015-04-28 01:36:54.051082 7f9ccaf7c900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.12-031812-generic >= 3.5 2015-04-28 01:36:54.056260 7f9ccaf7c900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f9ccaf7c900 time 2015-04-28 01:36:54.053608 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f9cc80bfec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -48> 2015-04-28 01:36:54.034920 7f9ccaf7c900 5 asok(0x3d98000) register_command perfcounters_dump hook 0x3d38050 -47> 2015-04-28 01:36:54.034952 7f9ccaf7c900 5 asok(0x3d98000) register_command 1 hook 0x3d38050 -46> 2015-04-28 01:36:54.034956 7f9ccaf7c900 5 asok(0x3d98000) register_command perf dump hook 0x3d38050 -45> 2015-04-28 01:36:54.034962 7f9ccaf7c900 5 asok(0x3d98000) register_command perfcounters_schema hook 0x3d38050 -44> 2015-04-28 01:36:54.035020 7f9ccaf7c900 5 asok(0x3d98000) register_command 2 hook 0x3d38050 -43> 2015-04-28 01:36:54.035031 7f9ccaf7c900 5 asok(0x3d98000) register_command perf schema hook 0x3d38050 -42> 2015-04-28 01:36:54.035036 7f9ccaf7c900 5 asok(0x3d98000) register_command perf reset hook 0x3d38050 -41> 2015-04-28 01:36:54.035043 7f9ccaf7c900 5 asok(0x3d98000) register_command config show hook 0x3d38050 -40> 2015-04-28 01:36:54.035048 7f9ccaf7c900 5 asok(0x3d98000) register_command config set hook 0x3d38050 -39> 2015-04-28 01:36:54.035051 7f9ccaf7c900 5 asok(0x3d98000) register_command config get hook 0x3d38050 -38> 2015-04-28 01:36:54.035055 7f9ccaf7c900 5 asok(0x3d98000) register_command config diff hook 0x3d38050 -37> 2015-04-28 01:36:54.035058 7f9ccaf7c900 5 asok(0x3d98000) register_command log flush hook 0x3d38050 -36> 2015-04-28 01:36:54.035063 7f9ccaf7c900 5 asok(0x3d98000) register_command log dump hook 0x3d38050 -35> 2015-04-28 01:36:54.035067 7f9ccaf7c900 5 asok(0x3d98000) register_command log reopen hook 0x3d38050 -34> 2015-04-28 01:36:54.037220 7f9ccaf7c900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 54355 -33> 2015-04-28 01:36:54.037918 7f9ccaf7c900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -32> 2015-04-28 01:36:54.037929 7f9ccaf7c900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6806/54355 need_addr=0 -31> 2015-04-28 01:36:54.037954 7f9ccaf7c900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -30> 2015-04-28 01:36:54.037961 7f9ccaf7c900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6806/54355 need_addr=0 -29> 2015-04-28 01:36:54.037978 7f9ccaf7c900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -28> 2015-04-28 01:36:54.037984 7f9ccaf7c900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6807/54355 need_addr=0 -27> 2015-04-28 01:36:54.038002 7f9ccaf7c900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -26> 2015-04-28 01:36:54.038009 7f9ccaf7c900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6807/54355 need_addr=0 -25> 2015-04-28 01:36:54.039161 7f9ccaf7c900 5 asok(0x3d98000) init /var/run/ceph/ceph-osd.23.asok -24> 2015-04-28 01:36:54.039175 7f9ccaf7c900 5 asok(0x3d98000) bind_and_listen /var/run/ceph/ceph-osd.23.asok -23> 2015-04-28 01:36:54.039257 7f9ccaf7c900 5 asok(0x3d98000) register_command 0 hook 0x3d340a8 -22> 2015-04-28 01:36:54.039265 7f9ccaf7c900 5 asok(0x3d98000) register_command version hook 0x3d340a8 -21> 2015-04-28 01:36:54.039284 7f9ccaf7c900 5 asok(0x3d98000) register_command git_version hook 0x3d340a8 -20> 2015-04-28 01:36:54.039293 7f9ccaf7c900 5 asok(0x3d98000) register_command help hook 0x3d38150 -19> 2015-04-28 01:36:54.039300 7f9ccaf7c900 5 asok(0x3d98000) register_command get_command_descriptions hook 0x3d38140 -18> 2015-04-28 01:36:54.039342 7f9ccaf7c900 10 monclient(hunting): build_initial_monmap -17> 2015-04-28 01:36:54.039342 7f9cc5087700 5 asok(0x3d98000) entry start -16> 2015-04-28 01:36:54.048152 7f9ccaf7c900 5 adding auth protocol: cephx -15> 2015-04-28 01:36:54.048163 7f9ccaf7c900 5 adding auth protocol: cephx -14> 2015-04-28 01:36:54.048249 7f9ccaf7c900 5 asok(0x3d98000) register_command objecter_requests hook 0x3d38180 -13> 2015-04-28 01:36:54.048312 7f9ccaf7c900 1 -- 192.168.30.73:6806/54355 messenger.start -12> 2015-04-28 01:36:54.048346 7f9ccaf7c900 1 -- :/0 messenger.start -11> 2015-04-28 01:36:54.048370 7f9ccaf7c900 1 -- 192.168.30.73:6807/54355 messenger.start -10> 2015-04-28 01:36:54.048393 7f9ccaf7c900 1 -- 192.168.32.73:6807/54355 messenger.start -9> 2015-04-28 01:36:54.048417 7f9ccaf7c900 1 -- 192.168.32.73:6806/54355 messenger.start -8> 2015-04-28 01:36:54.048442 7f9ccaf7c900 1 -- :/0 messenger.start -7> 2015-04-28 01:36:54.048503 7f9ccaf7c900 2 osd.23 0 mounting /var/lib/ceph/osd/ceph-23 /var/lib/ceph/osd/ceph-23/journal -6> 2015-04-28 01:36:54.048540 7f9ccaf7c900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. -5> 2015-04-28 01:36:54.048563 7f9ccaf7c900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) -4> 2015-04-28 01:36:54.049843 7f9ccaf7c900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work -3> 2015-04-28 01:36:54.049855 7f9ccaf7c900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option -2> 2015-04-28 01:36:54.051016 7f9ccaf7c900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) -1> 2015-04-28 01:36:54.051082 7f9ccaf7c900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.12-031812-generic >= 3.5 0> 2015-04-28 01:36:54.056260 7f9ccaf7c900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f9ccaf7c900 time 2015-04-28 01:36:54.053608 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f9cc80bfec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 01:36:54.060333 7f9ccaf7c900 -1 *** Caught signal (Aborted) ** in thread 7f9ccaf7c900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f9cc9c35340] 3: (gsignal()+0x39) [0x7f9cc80d4cc9] 4: (abort()+0x148) [0x7f9cc80d80d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f9cc89df6b5] 6: (()+0x5e836) [0x7f9cc89dd836] 7: (()+0x5e863) [0x7f9cc89dd863] 8: (()+0x5eaa2) [0x7f9cc89ddaa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f9cc80bfec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2015-04-28 01:36:54.060333 7f9ccaf7c900 -1 *** Caught signal (Aborted) ** in thread 7f9ccaf7c900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f9cc9c35340] 3: (gsignal()+0x39) [0x7f9cc80d4cc9] 4: (abort()+0x148) [0x7f9cc80d80d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f9cc89df6b5] 6: (()+0x5e836) [0x7f9cc89dd836] 7: (()+0x5e863) [0x7f9cc89dd863] 8: (()+0x5eaa2) [0x7f9cc89ddaa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f9cc80bfec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 01:36:54.448016 7fc0a0799900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 54811 2015-04-28 01:36:54.459462 7fc0a0799900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. 2015-04-28 01:36:54.459504 7fc0a0799900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) 2015-04-28 01:36:54.460473 7fc0a0799900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work 2015-04-28 01:36:54.460482 7fc0a0799900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-04-28 01:36:54.460998 7fc0a0799900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2015-04-28 01:36:54.461078 7fc0a0799900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.12-031812-generic >= 3.5 2015-04-28 01:36:54.465961 7fc0a0799900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7fc0a0799900 time 2015-04-28 01:36:54.463504 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7fc09d8dcec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -48> 2015-04-28 01:36:54.445903 7fc0a0799900 5 asok(0x39c4000) register_command perfcounters_dump hook 0x3964050 -47> 2015-04-28 01:36:54.445927 7fc0a0799900 5 asok(0x39c4000) register_command 1 hook 0x3964050 -46> 2015-04-28 01:36:54.445932 7fc0a0799900 5 asok(0x39c4000) register_command perf dump hook 0x3964050 -45> 2015-04-28 01:36:54.445941 7fc0a0799900 5 asok(0x39c4000) register_command perfcounters_schema hook 0x3964050 -44> 2015-04-28 01:36:54.445945 7fc0a0799900 5 asok(0x39c4000) register_command 2 hook 0x3964050 -43> 2015-04-28 01:36:54.445947 7fc0a0799900 5 asok(0x39c4000) register_command perf schema hook 0x3964050 -42> 2015-04-28 01:36:54.445951 7fc0a0799900 5 asok(0x39c4000) register_command perf reset hook 0x3964050 -41> 2015-04-28 01:36:54.445957 7fc0a0799900 5 asok(0x39c4000) register_command config show hook 0x3964050 -40> 2015-04-28 01:36:54.445961 7fc0a0799900 5 asok(0x39c4000) register_command config set hook 0x3964050 -39> 2015-04-28 01:36:54.445966 7fc0a0799900 5 asok(0x39c4000) register_command config get hook 0x3964050 -38> 2015-04-28 01:36:54.445969 7fc0a0799900 5 asok(0x39c4000) register_command config diff hook 0x3964050 -37> 2015-04-28 01:36:54.445972 7fc0a0799900 5 asok(0x39c4000) register_command log flush hook 0x3964050 -36> 2015-04-28 01:36:54.445975 7fc0a0799900 5 asok(0x39c4000) register_command log dump hook 0x3964050 -35> 2015-04-28 01:36:54.445979 7fc0a0799900 5 asok(0x39c4000) register_command log reopen hook 0x3964050 -34> 2015-04-28 01:36:54.448016 7fc0a0799900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 54811 -33> 2015-04-28 01:36:54.448752 7fc0a0799900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -32> 2015-04-28 01:36:54.448762 7fc0a0799900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6806/54811 need_addr=0 -31> 2015-04-28 01:36:54.448778 7fc0a0799900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -30> 2015-04-28 01:36:54.448783 7fc0a0799900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6806/54811 need_addr=0 -29> 2015-04-28 01:36:54.448795 7fc0a0799900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -28> 2015-04-28 01:36:54.448799 7fc0a0799900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6807/54811 need_addr=0 -27> 2015-04-28 01:36:54.448812 7fc0a0799900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -26> 2015-04-28 01:36:54.448817 7fc0a0799900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6807/54811 need_addr=0 -25> 2015-04-28 01:36:54.450090 7fc0a0799900 5 asok(0x39c4000) init /var/run/ceph/ceph-osd.23.asok -24> 2015-04-28 01:36:54.450102 7fc0a0799900 5 asok(0x39c4000) bind_and_listen /var/run/ceph/ceph-osd.23.asok -23> 2015-04-28 01:36:54.450188 7fc0a0799900 5 asok(0x39c4000) register_command 0 hook 0x39600a8 -22> 2015-04-28 01:36:54.450195 7fc0a0799900 5 asok(0x39c4000) register_command version hook 0x39600a8 -21> 2015-04-28 01:36:54.450198 7fc0a0799900 5 asok(0x39c4000) register_command git_version hook 0x39600a8 -20> 2015-04-28 01:36:54.450201 7fc0a0799900 5 asok(0x39c4000) register_command help hook 0x3964150 -19> 2015-04-28 01:36:54.450204 7fc0a0799900 5 asok(0x39c4000) register_command get_command_descriptions hook 0x3964140 -18> 2015-04-28 01:36:54.450237 7fc0a0799900 10 monclient(hunting): build_initial_monmap -17> 2015-04-28 01:36:54.450235 7fc09a8a4700 5 asok(0x39c4000) entry start -16> 2015-04-28 01:36:54.459021 7fc0a0799900 5 adding auth protocol: cephx -15> 2015-04-28 01:36:54.459030 7fc0a0799900 5 adding auth protocol: cephx -14> 2015-04-28 01:36:54.459114 7fc0a0799900 5 asok(0x39c4000) register_command objecter_requests hook 0x3964180 -13> 2015-04-28 01:36:54.459173 7fc0a0799900 1 -- 192.168.30.73:6806/54811 messenger.start -12> 2015-04-28 01:36:54.459215 7fc0a0799900 1 -- :/0 messenger.start -11> 2015-04-28 01:36:54.459243 7fc0a0799900 1 -- 192.168.30.73:6807/54811 messenger.start -10> 2015-04-28 01:36:54.459279 7fc0a0799900 1 -- 192.168.32.73:6807/54811 messenger.start -9> 2015-04-28 01:36:54.459316 7fc0a0799900 1 -- 192.168.32.73:6806/54811 messenger.start -8> 2015-04-28 01:36:54.459357 7fc0a0799900 1 -- :/0 messenger.start -7> 2015-04-28 01:36:54.459422 7fc0a0799900 2 osd.23 0 mounting /var/lib/ceph/osd/ceph-23 /var/lib/ceph/osd/ceph-23/journal -6> 2015-04-28 01:36:54.459462 7fc0a0799900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. -5> 2015-04-28 01:36:54.459504 7fc0a0799900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) -4> 2015-04-28 01:36:54.460473 7fc0a0799900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work -3> 2015-04-28 01:36:54.460482 7fc0a0799900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option -2> 2015-04-28 01:36:54.460998 7fc0a0799900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) -1> 2015-04-28 01:36:54.461078 7fc0a0799900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.12-031812-generic >= 3.5 0> 2015-04-28 01:36:54.465961 7fc0a0799900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7fc0a0799900 time 2015-04-28 01:36:54.463504 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7fc09d8dcec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 01:36:54.469687 7fc0a0799900 -1 *** Caught signal (Aborted) ** in thread 7fc0a0799900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7fc09f452340] 3: (gsignal()+0x39) [0x7fc09d8f1cc9] 4: (abort()+0x148) [0x7fc09d8f50d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7fc09e1fc6b5] 6: (()+0x5e836) [0x7fc09e1fa836] 7: (()+0x5e863) [0x7fc09e1fa863] 8: (()+0x5eaa2) [0x7fc09e1faaa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7fc09d8dcec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2015-04-28 01:36:54.469687 7fc0a0799900 -1 *** Caught signal (Aborted) ** in thread 7fc0a0799900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7fc09f452340] 3: (gsignal()+0x39) [0x7fc09d8f1cc9] 4: (abort()+0x148) [0x7fc09d8f50d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7fc09e1fc6b5] 6: (()+0x5e836) [0x7fc09e1fa836] 7: (()+0x5e863) [0x7fc09e1fa863] 8: (()+0x5eaa2) [0x7fc09e1faaa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7fc09d8dcec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 01:46:26.926983 7f3d7944c900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 76213 2015-04-28 01:46:26.939042 7f3d7944c900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. 2015-04-28 01:46:26.939068 7f3d7944c900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) 2015-04-28 01:46:26.940060 7f3d7944c900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work 2015-04-28 01:46:26.940070 7f3d7944c900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-04-28 01:46:26.940477 7f3d7944c900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2015-04-28 01:46:26.940540 7f3d7944c900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.12-031812-generic >= 3.5 2015-04-28 01:46:26.945526 7f3d7944c900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f3d7944c900 time 2015-04-28 01:46:26.942990 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f3d7658fec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -48> 2015-04-28 01:46:26.924738 7f3d7944c900 5 asok(0x39fe000) register_command perfcounters_dump hook 0x399e050 -47> 2015-04-28 01:46:26.924777 7f3d7944c900 5 asok(0x39fe000) register_command 1 hook 0x399e050 -46> 2015-04-28 01:46:26.924784 7f3d7944c900 5 asok(0x39fe000) register_command perf dump hook 0x399e050 -45> 2015-04-28 01:46:26.924792 7f3d7944c900 5 asok(0x39fe000) register_command perfcounters_schema hook 0x399e050 -44> 2015-04-28 01:46:26.924796 7f3d7944c900 5 asok(0x39fe000) register_command 2 hook 0x399e050 -43> 2015-04-28 01:46:26.924799 7f3d7944c900 5 asok(0x39fe000) register_command perf schema hook 0x399e050 -42> 2015-04-28 01:46:26.924804 7f3d7944c900 5 asok(0x39fe000) register_command perf reset hook 0x399e050 -41> 2015-04-28 01:46:26.924815 7f3d7944c900 5 asok(0x39fe000) register_command config show hook 0x399e050 -40> 2015-04-28 01:46:26.924820 7f3d7944c900 5 asok(0x39fe000) register_command config set hook 0x399e050 -39> 2015-04-28 01:46:26.924825 7f3d7944c900 5 asok(0x39fe000) register_command config get hook 0x399e050 -38> 2015-04-28 01:46:26.924830 7f3d7944c900 5 asok(0x39fe000) register_command config diff hook 0x399e050 -37> 2015-04-28 01:46:26.924834 7f3d7944c900 5 asok(0x39fe000) register_command log flush hook 0x399e050 -36> 2015-04-28 01:46:26.924838 7f3d7944c900 5 asok(0x39fe000) register_command log dump hook 0x399e050 -35> 2015-04-28 01:46:26.924843 7f3d7944c900 5 asok(0x39fe000) register_command log reopen hook 0x399e050 -34> 2015-04-28 01:46:26.926983 7f3d7944c900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 76213 -33> 2015-04-28 01:46:26.927825 7f3d7944c900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -32> 2015-04-28 01:46:26.927836 7f3d7944c900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6800/76213 need_addr=0 -31> 2015-04-28 01:46:26.927851 7f3d7944c900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -30> 2015-04-28 01:46:26.927857 7f3d7944c900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6800/76213 need_addr=0 -29> 2015-04-28 01:46:26.927868 7f3d7944c900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -28> 2015-04-28 01:46:26.927873 7f3d7944c900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6801/76213 need_addr=0 -27> 2015-04-28 01:46:26.927885 7f3d7944c900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -26> 2015-04-28 01:46:26.927890 7f3d7944c900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6801/76213 need_addr=0 -25> 2015-04-28 01:46:26.929225 7f3d7944c900 5 asok(0x39fe000) init /var/run/ceph/ceph-osd.23.asok -24> 2015-04-28 01:46:26.929238 7f3d7944c900 5 asok(0x39fe000) bind_and_listen /var/run/ceph/ceph-osd.23.asok -23> 2015-04-28 01:46:26.929333 7f3d7944c900 5 asok(0x39fe000) register_command 0 hook 0x399a0a8 -22> 2015-04-28 01:46:26.929341 7f3d7944c900 5 asok(0x39fe000) register_command version hook 0x399a0a8 -21> 2015-04-28 01:46:26.929346 7f3d7944c900 5 asok(0x39fe000) register_command git_version hook 0x399a0a8 -20> 2015-04-28 01:46:26.929351 7f3d7944c900 5 asok(0x39fe000) register_command help hook 0x399e150 -19> 2015-04-28 01:46:26.929356 7f3d7944c900 5 asok(0x39fe000) register_command get_command_descriptions hook 0x399e140 -18> 2015-04-28 01:46:26.929395 7f3d7944c900 10 monclient(hunting): build_initial_monmap -17> 2015-04-28 01:46:26.929393 7f3d73557700 5 asok(0x39fe000) entry start -16> 2015-04-28 01:46:26.938622 7f3d7944c900 5 adding auth protocol: cephx -15> 2015-04-28 01:46:26.938634 7f3d7944c900 5 adding auth protocol: cephx -14> 2015-04-28 01:46:26.938734 7f3d7944c900 5 asok(0x39fe000) register_command objecter_requests hook 0x399e180 -13> 2015-04-28 01:46:26.938806 7f3d7944c900 1 -- 192.168.30.73:6800/76213 messenger.start -12> 2015-04-28 01:46:26.938842 7f3d7944c900 1 -- :/0 messenger.start -11> 2015-04-28 01:46:26.938866 7f3d7944c900 1 -- 192.168.30.73:6801/76213 messenger.start -10> 2015-04-28 01:46:26.938891 7f3d7944c900 1 -- 192.168.32.73:6801/76213 messenger.start -9> 2015-04-28 01:46:26.938916 7f3d7944c900 1 -- 192.168.32.73:6800/76213 messenger.start -8> 2015-04-28 01:46:26.938940 7f3d7944c900 1 -- :/0 messenger.start -7> 2015-04-28 01:46:26.939001 7f3d7944c900 2 osd.23 0 mounting /var/lib/ceph/osd/ceph-23 /var/lib/ceph/osd/ceph-23/journal -6> 2015-04-28 01:46:26.939042 7f3d7944c900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. -5> 2015-04-28 01:46:26.939068 7f3d7944c900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) -4> 2015-04-28 01:46:26.940060 7f3d7944c900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work -3> 2015-04-28 01:46:26.940070 7f3d7944c900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option -2> 2015-04-28 01:46:26.940477 7f3d7944c900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) -1> 2015-04-28 01:46:26.940540 7f3d7944c900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.12-031812-generic >= 3.5 0> 2015-04-28 01:46:26.945526 7f3d7944c900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f3d7944c900 time 2015-04-28 01:46:26.942990 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f3d7658fec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 01:46:26.949108 7f3d7944c900 -1 *** Caught signal (Aborted) ** in thread 7f3d7944c900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f3d78105340] 3: (gsignal()+0x39) [0x7f3d765a4cc9] 4: (abort()+0x148) [0x7f3d765a80d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f3d76eaf6b5] 6: (()+0x5e836) [0x7f3d76ead836] 7: (()+0x5e863) [0x7f3d76ead863] 8: (()+0x5eaa2) [0x7f3d76eadaa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f3d7658fec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2015-04-28 01:46:26.949108 7f3d7944c900 -1 *** Caught signal (Aborted) ** in thread 7f3d7944c900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f3d78105340] 3: (gsignal()+0x39) [0x7f3d765a4cc9] 4: (abort()+0x148) [0x7f3d765a80d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f3d76eaf6b5] 6: (()+0x5e836) [0x7f3d76ead836] 7: (()+0x5e863) [0x7f3d76ead863] 8: (()+0x5eaa2) [0x7f3d76eadaa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f3d7658fec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 01:46:27.365946 7f4372221900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 76827 2015-04-28 01:46:27.378183 7f4372221900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. 2015-04-28 01:46:27.378211 7f4372221900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) 2015-04-28 01:46:27.379155 7f4372221900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work 2015-04-28 01:46:27.379164 7f4372221900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-04-28 01:46:27.379665 7f4372221900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2015-04-28 01:46:27.379731 7f4372221900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.12-031812-generic >= 3.5 2015-04-28 01:46:27.384999 7f4372221900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f4372221900 time 2015-04-28 01:46:27.382428 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f436f364ec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -48> 2015-04-28 01:46:27.363721 7f4372221900 5 asok(0x5506000) register_command perfcounters_dump hook 0x54a6050 -47> 2015-04-28 01:46:27.363757 7f4372221900 5 asok(0x5506000) register_command 1 hook 0x54a6050 -46> 2015-04-28 01:46:27.363763 7f4372221900 5 asok(0x5506000) register_command perf dump hook 0x54a6050 -45> 2015-04-28 01:46:27.363771 7f4372221900 5 asok(0x5506000) register_command perfcounters_schema hook 0x54a6050 -44> 2015-04-28 01:46:27.363776 7f4372221900 5 asok(0x5506000) register_command 2 hook 0x54a6050 -43> 2015-04-28 01:46:27.363779 7f4372221900 5 asok(0x5506000) register_command perf schema hook 0x54a6050 -42> 2015-04-28 01:46:27.363784 7f4372221900 5 asok(0x5506000) register_command perf reset hook 0x54a6050 -41> 2015-04-28 01:46:27.363799 7f4372221900 5 asok(0x5506000) register_command config show hook 0x54a6050 -40> 2015-04-28 01:46:27.363804 7f4372221900 5 asok(0x5506000) register_command config set hook 0x54a6050 -39> 2015-04-28 01:46:27.363809 7f4372221900 5 asok(0x5506000) register_command config get hook 0x54a6050 -38> 2015-04-28 01:46:27.363814 7f4372221900 5 asok(0x5506000) register_command config diff hook 0x54a6050 -37> 2015-04-28 01:46:27.363818 7f4372221900 5 asok(0x5506000) register_command log flush hook 0x54a6050 -36> 2015-04-28 01:46:27.363822 7f4372221900 5 asok(0x5506000) register_command log dump hook 0x54a6050 -35> 2015-04-28 01:46:27.363825 7f4372221900 5 asok(0x5506000) register_command log reopen hook 0x54a6050 -34> 2015-04-28 01:46:27.365946 7f4372221900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 76827 -33> 2015-04-28 01:46:27.366856 7f4372221900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -32> 2015-04-28 01:46:27.366868 7f4372221900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6800/76827 need_addr=0 -31> 2015-04-28 01:46:27.366886 7f4372221900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -30> 2015-04-28 01:46:27.366892 7f4372221900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6800/76827 need_addr=0 -29> 2015-04-28 01:46:27.366903 7f4372221900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -28> 2015-04-28 01:46:27.366909 7f4372221900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6801/76827 need_addr=0 -27> 2015-04-28 01:46:27.366920 7f4372221900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -26> 2015-04-28 01:46:27.366926 7f4372221900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6801/76827 need_addr=0 -25> 2015-04-28 01:46:27.368317 7f4372221900 5 asok(0x5506000) init /var/run/ceph/ceph-osd.23.asok -24> 2015-04-28 01:46:27.368330 7f4372221900 5 asok(0x5506000) bind_and_listen /var/run/ceph/ceph-osd.23.asok -23> 2015-04-28 01:46:27.368428 7f4372221900 5 asok(0x5506000) register_command 0 hook 0x54a20a8 -22> 2015-04-28 01:46:27.368436 7f4372221900 5 asok(0x5506000) register_command version hook 0x54a20a8 -21> 2015-04-28 01:46:27.368440 7f4372221900 5 asok(0x5506000) register_command git_version hook 0x54a20a8 -20> 2015-04-28 01:46:27.368443 7f4372221900 5 asok(0x5506000) register_command help hook 0x54a6150 -19> 2015-04-28 01:46:27.368446 7f4372221900 5 asok(0x5506000) register_command get_command_descriptions hook 0x54a6140 -18> 2015-04-28 01:46:27.368483 7f4372221900 10 monclient(hunting): build_initial_monmap -17> 2015-04-28 01:46:27.368493 7f436c32c700 5 asok(0x5506000) entry start -16> 2015-04-28 01:46:27.377773 7f4372221900 5 adding auth protocol: cephx -15> 2015-04-28 01:46:27.377786 7f4372221900 5 adding auth protocol: cephx -14> 2015-04-28 01:46:27.377890 7f4372221900 5 asok(0x5506000) register_command objecter_requests hook 0x54a6180 -13> 2015-04-28 01:46:27.377962 7f4372221900 1 -- 192.168.30.73:6800/76827 messenger.start -12> 2015-04-28 01:46:27.377998 7f4372221900 1 -- :/0 messenger.start -11> 2015-04-28 01:46:27.378023 7f4372221900 1 -- 192.168.30.73:6801/76827 messenger.start -10> 2015-04-28 01:46:27.378045 7f4372221900 1 -- 192.168.32.73:6801/76827 messenger.start -9> 2015-04-28 01:46:27.378067 7f4372221900 1 -- 192.168.32.73:6800/76827 messenger.start -8> 2015-04-28 01:46:27.378089 7f4372221900 1 -- :/0 messenger.start -7> 2015-04-28 01:46:27.378139 7f4372221900 2 osd.23 0 mounting /var/lib/ceph/osd/ceph-23 /var/lib/ceph/osd/ceph-23/journal -6> 2015-04-28 01:46:27.378183 7f4372221900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. -5> 2015-04-28 01:46:27.378211 7f4372221900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) -4> 2015-04-28 01:46:27.379155 7f4372221900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work -3> 2015-04-28 01:46:27.379164 7f4372221900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option -2> 2015-04-28 01:46:27.379665 7f4372221900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) -1> 2015-04-28 01:46:27.379731 7f4372221900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.12-031812-generic >= 3.5 0> 2015-04-28 01:46:27.384999 7f4372221900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f4372221900 time 2015-04-28 01:46:27.382428 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f436f364ec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 01:46:27.388606 7f4372221900 -1 *** Caught signal (Aborted) ** in thread 7f4372221900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f4370eda340] 3: (gsignal()+0x39) [0x7f436f379cc9] 4: (abort()+0x148) [0x7f436f37d0d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f436fc846b5] 6: (()+0x5e836) [0x7f436fc82836] 7: (()+0x5e863) [0x7f436fc82863] 8: (()+0x5eaa2) [0x7f436fc82aa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f436f364ec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2015-04-28 01:46:27.388606 7f4372221900 -1 *** Caught signal (Aborted) ** in thread 7f4372221900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f4370eda340] 3: (gsignal()+0x39) [0x7f436f379cc9] 4: (abort()+0x148) [0x7f436f37d0d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f436fc846b5] 6: (()+0x5e836) [0x7f436fc82836] 7: (()+0x5e863) [0x7f436fc82863] 8: (()+0x5eaa2) [0x7f436fc82aa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f436f364ec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 01:46:27.778721 7f7772772900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 77381 2015-04-28 01:46:27.790877 7f7772772900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. 2015-04-28 01:46:27.790900 7f7772772900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) 2015-04-28 01:46:27.791803 7f7772772900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work 2015-04-28 01:46:27.791814 7f7772772900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-04-28 01:46:27.792331 7f7772772900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2015-04-28 01:46:27.792410 7f7772772900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.12-031812-generic >= 3.5 2015-04-28 01:46:27.797665 7f7772772900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f7772772900 time 2015-04-28 01:46:27.794923 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f776f8b5ec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -48> 2015-04-28 01:46:27.776489 7f7772772900 5 asok(0x5810000) register_command perfcounters_dump hook 0x57b0050 -47> 2015-04-28 01:46:27.776531 7f7772772900 5 asok(0x5810000) register_command 1 hook 0x57b0050 -46> 2015-04-28 01:46:27.776536 7f7772772900 5 asok(0x5810000) register_command perf dump hook 0x57b0050 -45> 2015-04-28 01:46:27.776542 7f7772772900 5 asok(0x5810000) register_command perfcounters_schema hook 0x57b0050 -44> 2015-04-28 01:46:27.776557 7f7772772900 5 asok(0x5810000) register_command 2 hook 0x57b0050 -43> 2015-04-28 01:46:27.776563 7f7772772900 5 asok(0x5810000) register_command perf schema hook 0x57b0050 -42> 2015-04-28 01:46:27.776567 7f7772772900 5 asok(0x5810000) register_command perf reset hook 0x57b0050 -41> 2015-04-28 01:46:27.776580 7f7772772900 5 asok(0x5810000) register_command config show hook 0x57b0050 -40> 2015-04-28 01:46:27.776585 7f7772772900 5 asok(0x5810000) register_command config set hook 0x57b0050 -39> 2015-04-28 01:46:27.776588 7f7772772900 5 asok(0x5810000) register_command config get hook 0x57b0050 -38> 2015-04-28 01:46:27.776592 7f7772772900 5 asok(0x5810000) register_command config diff hook 0x57b0050 -37> 2015-04-28 01:46:27.776595 7f7772772900 5 asok(0x5810000) register_command log flush hook 0x57b0050 -36> 2015-04-28 01:46:27.776599 7f7772772900 5 asok(0x5810000) register_command log dump hook 0x57b0050 -35> 2015-04-28 01:46:27.776602 7f7772772900 5 asok(0x5810000) register_command log reopen hook 0x57b0050 -34> 2015-04-28 01:46:27.778721 7f7772772900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 77381 -33> 2015-04-28 01:46:27.779581 7f7772772900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -32> 2015-04-28 01:46:27.779593 7f7772772900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6800/77381 need_addr=0 -31> 2015-04-28 01:46:27.779609 7f7772772900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -30> 2015-04-28 01:46:27.779614 7f7772772900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6800/77381 need_addr=0 -29> 2015-04-28 01:46:27.779627 7f7772772900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -28> 2015-04-28 01:46:27.779633 7f7772772900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6801/77381 need_addr=0 -27> 2015-04-28 01:46:27.779645 7f7772772900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -26> 2015-04-28 01:46:27.779650 7f7772772900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6801/77381 need_addr=0 -25> 2015-04-28 01:46:27.781007 7f7772772900 5 asok(0x5810000) init /var/run/ceph/ceph-osd.23.asok -24> 2015-04-28 01:46:27.781020 7f7772772900 5 asok(0x5810000) bind_and_listen /var/run/ceph/ceph-osd.23.asok -23> 2015-04-28 01:46:27.781126 7f7772772900 5 asok(0x5810000) register_command 0 hook 0x57ac0a8 -22> 2015-04-28 01:46:27.781135 7f7772772900 5 asok(0x5810000) register_command version hook 0x57ac0a8 -21> 2015-04-28 01:46:27.781140 7f7772772900 5 asok(0x5810000) register_command git_version hook 0x57ac0a8 -20> 2015-04-28 01:46:27.781144 7f7772772900 5 asok(0x5810000) register_command help hook 0x57b0150 -19> 2015-04-28 01:46:27.781147 7f7772772900 5 asok(0x5810000) register_command get_command_descriptions hook 0x57b0140 -18> 2015-04-28 01:46:27.781185 7f7772772900 10 monclient(hunting): build_initial_monmap -17> 2015-04-28 01:46:27.781197 7f776c87d700 5 asok(0x5810000) entry start -16> 2015-04-28 01:46:27.790500 7f7772772900 5 adding auth protocol: cephx -15> 2015-04-28 01:46:27.790512 7f7772772900 5 adding auth protocol: cephx -14> 2015-04-28 01:46:27.790611 7f7772772900 5 asok(0x5810000) register_command objecter_requests hook 0x57b0180 -13> 2015-04-28 01:46:27.790676 7f7772772900 1 -- 192.168.30.73:6800/77381 messenger.start -12> 2015-04-28 01:46:27.790711 7f7772772900 1 -- :/0 messenger.start -11> 2015-04-28 01:46:27.790732 7f7772772900 1 -- 192.168.30.73:6801/77381 messenger.start -10> 2015-04-28 01:46:27.790752 7f7772772900 1 -- 192.168.32.73:6801/77381 messenger.start -9> 2015-04-28 01:46:27.790771 7f7772772900 1 -- 192.168.32.73:6800/77381 messenger.start -8> 2015-04-28 01:46:27.790791 7f7772772900 1 -- :/0 messenger.start -7> 2015-04-28 01:46:27.790837 7f7772772900 2 osd.23 0 mounting /var/lib/ceph/osd/ceph-23 /var/lib/ceph/osd/ceph-23/journal -6> 2015-04-28 01:46:27.790877 7f7772772900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. -5> 2015-04-28 01:46:27.790900 7f7772772900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) -4> 2015-04-28 01:46:27.791803 7f7772772900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work -3> 2015-04-28 01:46:27.791814 7f7772772900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option -2> 2015-04-28 01:46:27.792331 7f7772772900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) -1> 2015-04-28 01:46:27.792410 7f7772772900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.12-031812-generic >= 3.5 0> 2015-04-28 01:46:27.797665 7f7772772900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f7772772900 time 2015-04-28 01:46:27.794923 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f776f8b5ec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 01:46:27.801327 7f7772772900 -1 *** Caught signal (Aborted) ** in thread 7f7772772900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f777142b340] 3: (gsignal()+0x39) [0x7f776f8cacc9] 4: (abort()+0x148) [0x7f776f8ce0d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f77701d56b5] 6: (()+0x5e836) [0x7f77701d3836] 7: (()+0x5e863) [0x7f77701d3863] 8: (()+0x5eaa2) [0x7f77701d3aa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f776f8b5ec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2015-04-28 01:46:27.801327 7f7772772900 -1 *** Caught signal (Aborted) ** in thread 7f7772772900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f777142b340] 3: (gsignal()+0x39) [0x7f776f8cacc9] 4: (abort()+0x148) [0x7f776f8ce0d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f77701d56b5] 6: (()+0x5e836) [0x7f77701d3836] 7: (()+0x5e863) [0x7f77701d3863] 8: (()+0x5eaa2) [0x7f77701d3aa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f776f8b5ec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 01:46:28.202777 7f9d38601900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 77967 2015-04-28 01:46:28.214856 7f9d38601900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. 2015-04-28 01:46:28.214896 7f9d38601900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) 2015-04-28 01:46:28.215896 7f9d38601900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work 2015-04-28 01:46:28.215922 7f9d38601900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-04-28 01:46:28.216425 7f9d38601900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2015-04-28 01:46:28.216523 7f9d38601900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.12-031812-generic >= 3.5 2015-04-28 01:46:28.221781 7f9d38601900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f9d38601900 time 2015-04-28 01:46:28.219291 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f9d35744ec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -48> 2015-04-28 01:46:28.200556 7f9d38601900 5 asok(0x4b82000) register_command perfcounters_dump hook 0x4b22050 -47> 2015-04-28 01:46:28.200594 7f9d38601900 5 asok(0x4b82000) register_command 1 hook 0x4b22050 -46> 2015-04-28 01:46:28.200599 7f9d38601900 5 asok(0x4b82000) register_command perf dump hook 0x4b22050 -45> 2015-04-28 01:46:28.200605 7f9d38601900 5 asok(0x4b82000) register_command perfcounters_schema hook 0x4b22050 -44> 2015-04-28 01:46:28.200608 7f9d38601900 5 asok(0x4b82000) register_command 2 hook 0x4b22050 -43> 2015-04-28 01:46:28.200610 7f9d38601900 5 asok(0x4b82000) register_command perf schema hook 0x4b22050 -42> 2015-04-28 01:46:28.200614 7f9d38601900 5 asok(0x4b82000) register_command perf reset hook 0x4b22050 -41> 2015-04-28 01:46:28.200621 7f9d38601900 5 asok(0x4b82000) register_command config show hook 0x4b22050 -40> 2015-04-28 01:46:28.200638 7f9d38601900 5 asok(0x4b82000) register_command config set hook 0x4b22050 -39> 2015-04-28 01:46:28.200642 7f9d38601900 5 asok(0x4b82000) register_command config get hook 0x4b22050 -38> 2015-04-28 01:46:28.200645 7f9d38601900 5 asok(0x4b82000) register_command config diff hook 0x4b22050 -37> 2015-04-28 01:46:28.200649 7f9d38601900 5 asok(0x4b82000) register_command log flush hook 0x4b22050 -36> 2015-04-28 01:46:28.200651 7f9d38601900 5 asok(0x4b82000) register_command log dump hook 0x4b22050 -35> 2015-04-28 01:46:28.200654 7f9d38601900 5 asok(0x4b82000) register_command log reopen hook 0x4b22050 -34> 2015-04-28 01:46:28.202777 7f9d38601900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 77967 -33> 2015-04-28 01:46:28.203684 7f9d38601900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -32> 2015-04-28 01:46:28.203695 7f9d38601900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6800/77967 need_addr=0 -31> 2015-04-28 01:46:28.203723 7f9d38601900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -30> 2015-04-28 01:46:28.203727 7f9d38601900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6800/77967 need_addr=0 -29> 2015-04-28 01:46:28.203736 7f9d38601900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -28> 2015-04-28 01:46:28.203747 7f9d38601900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6801/77967 need_addr=0 -27> 2015-04-28 01:46:28.203757 7f9d38601900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -26> 2015-04-28 01:46:28.203762 7f9d38601900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6801/77967 need_addr=0 -25> 2015-04-28 01:46:28.205077 7f9d38601900 5 asok(0x4b82000) init /var/run/ceph/ceph-osd.23.asok -24> 2015-04-28 01:46:28.205103 7f9d38601900 5 asok(0x4b82000) bind_and_listen /var/run/ceph/ceph-osd.23.asok -23> 2015-04-28 01:46:28.205198 7f9d38601900 5 asok(0x4b82000) register_command 0 hook 0x4b1e0a8 -22> 2015-04-28 01:46:28.205207 7f9d38601900 5 asok(0x4b82000) register_command version hook 0x4b1e0a8 -21> 2015-04-28 01:46:28.205210 7f9d38601900 5 asok(0x4b82000) register_command git_version hook 0x4b1e0a8 -20> 2015-04-28 01:46:28.205225 7f9d38601900 5 asok(0x4b82000) register_command help hook 0x4b22150 -19> 2015-04-28 01:46:28.205228 7f9d38601900 5 asok(0x4b82000) register_command get_command_descriptions hook 0x4b22140 -18> 2015-04-28 01:46:28.205268 7f9d38601900 10 monclient(hunting): build_initial_monmap -17> 2015-04-28 01:46:28.205296 7f9d3270c700 5 asok(0x4b82000) entry start -16> 2015-04-28 01:46:28.214293 7f9d38601900 5 adding auth protocol: cephx -15> 2015-04-28 01:46:28.214310 7f9d38601900 5 adding auth protocol: cephx -14> 2015-04-28 01:46:28.214462 7f9d38601900 5 asok(0x4b82000) register_command objecter_requests hook 0x4b22180 -13> 2015-04-28 01:46:28.214574 7f9d38601900 1 -- 192.168.30.73:6800/77967 messenger.start -12> 2015-04-28 01:46:28.214624 7f9d38601900 1 -- :/0 messenger.start -11> 2015-04-28 01:46:28.214650 7f9d38601900 1 -- 192.168.30.73:6801/77967 messenger.start -10> 2015-04-28 01:46:28.214682 7f9d38601900 1 -- 192.168.32.73:6801/77967 messenger.start -9> 2015-04-28 01:46:28.214706 7f9d38601900 1 -- 192.168.32.73:6800/77967 messenger.start -8> 2015-04-28 01:46:28.214731 7f9d38601900 1 -- :/0 messenger.start -7> 2015-04-28 01:46:28.214808 7f9d38601900 2 osd.23 0 mounting /var/lib/ceph/osd/ceph-23 /var/lib/ceph/osd/ceph-23/journal -6> 2015-04-28 01:46:28.214856 7f9d38601900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. -5> 2015-04-28 01:46:28.214896 7f9d38601900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) -4> 2015-04-28 01:46:28.215896 7f9d38601900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work -3> 2015-04-28 01:46:28.215922 7f9d38601900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option -2> 2015-04-28 01:46:28.216425 7f9d38601900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) -1> 2015-04-28 01:46:28.216523 7f9d38601900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.12-031812-generic >= 3.5 0> 2015-04-28 01:46:28.221781 7f9d38601900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f9d38601900 time 2015-04-28 01:46:28.219291 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f9d35744ec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 01:46:28.225354 7f9d38601900 -1 *** Caught signal (Aborted) ** in thread 7f9d38601900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f9d372ba340] 3: (gsignal()+0x39) [0x7f9d35759cc9] 4: (abort()+0x148) [0x7f9d3575d0d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f9d360646b5] 6: (()+0x5e836) [0x7f9d36062836] 7: (()+0x5e863) [0x7f9d36062863] 8: (()+0x5eaa2) [0x7f9d36062aa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f9d35744ec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2015-04-28 01:46:28.225354 7f9d38601900 -1 *** Caught signal (Aborted) ** in thread 7f9d38601900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f9d372ba340] 3: (gsignal()+0x39) [0x7f9d35759cc9] 4: (abort()+0x148) [0x7f9d3575d0d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f9d360646b5] 6: (()+0x5e836) [0x7f9d36062836] 7: (()+0x5e863) [0x7f9d36062863] 8: (()+0x5eaa2) [0x7f9d36062aa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f9d35744ec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 01:46:28.626367 7f51610a4900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 78592 2015-04-28 01:46:28.638984 7f51610a4900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. 2015-04-28 01:46:28.639024 7f51610a4900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) 2015-04-28 01:46:28.640079 7f51610a4900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work 2015-04-28 01:46:28.640091 7f51610a4900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-04-28 01:46:28.640988 7f51610a4900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2015-04-28 01:46:28.641075 7f51610a4900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.12-031812-generic >= 3.5 2015-04-28 01:46:28.646675 7f51610a4900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f51610a4900 time 2015-04-28 01:46:28.644014 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f515e1e7ec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -48> 2015-04-28 01:46:28.623931 7f51610a4900 5 asok(0x4dfc000) register_command perfcounters_dump hook 0x4d9c050 -47> 2015-04-28 01:46:28.623977 7f51610a4900 5 asok(0x4dfc000) register_command 1 hook 0x4d9c050 -46> 2015-04-28 01:46:28.623985 7f51610a4900 5 asok(0x4dfc000) register_command perf dump hook 0x4d9c050 -45> 2015-04-28 01:46:28.623997 7f51610a4900 5 asok(0x4dfc000) register_command perfcounters_schema hook 0x4d9c050 -44> 2015-04-28 01:46:28.624003 7f51610a4900 5 asok(0x4dfc000) register_command 2 hook 0x4d9c050 -43> 2015-04-28 01:46:28.624009 7f51610a4900 5 asok(0x4dfc000) register_command perf schema hook 0x4d9c050 -42> 2015-04-28 01:46:28.624016 7f51610a4900 5 asok(0x4dfc000) register_command perf reset hook 0x4d9c050 -41> 2015-04-28 01:46:28.624033 7f51610a4900 5 asok(0x4dfc000) register_command config show hook 0x4d9c050 -40> 2015-04-28 01:46:28.624040 7f51610a4900 5 asok(0x4dfc000) register_command config set hook 0x4d9c050 -39> 2015-04-28 01:46:28.624047 7f51610a4900 5 asok(0x4dfc000) register_command config get hook 0x4d9c050 -38> 2015-04-28 01:46:28.624053 7f51610a4900 5 asok(0x4dfc000) register_command config diff hook 0x4d9c050 -37> 2015-04-28 01:46:28.624059 7f51610a4900 5 asok(0x4dfc000) register_command log flush hook 0x4d9c050 -36> 2015-04-28 01:46:28.624065 7f51610a4900 5 asok(0x4dfc000) register_command log dump hook 0x4d9c050 -35> 2015-04-28 01:46:28.624072 7f51610a4900 5 asok(0x4dfc000) register_command log reopen hook 0x4d9c050 -34> 2015-04-28 01:46:28.626367 7f51610a4900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 78592 -33> 2015-04-28 01:46:28.627382 7f51610a4900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -32> 2015-04-28 01:46:28.627395 7f51610a4900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6800/78592 need_addr=0 -31> 2015-04-28 01:46:28.627419 7f51610a4900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -30> 2015-04-28 01:46:28.627426 7f51610a4900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6800/78592 need_addr=0 -29> 2015-04-28 01:46:28.627441 7f51610a4900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -28> 2015-04-28 01:46:28.627447 7f51610a4900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6801/78592 need_addr=0 -27> 2015-04-28 01:46:28.627461 7f51610a4900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -26> 2015-04-28 01:46:28.627468 7f51610a4900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6801/78592 need_addr=0 -25> 2015-04-28 01:46:28.628861 7f51610a4900 5 asok(0x4dfc000) init /var/run/ceph/ceph-osd.23.asok -24> 2015-04-28 01:46:28.628877 7f51610a4900 5 asok(0x4dfc000) bind_and_listen /var/run/ceph/ceph-osd.23.asok -23> 2015-04-28 01:46:28.628989 7f51610a4900 5 asok(0x4dfc000) register_command 0 hook 0x4d980a8 -22> 2015-04-28 01:46:28.628999 7f51610a4900 5 asok(0x4dfc000) register_command version hook 0x4d980a8 -21> 2015-04-28 01:46:28.629006 7f51610a4900 5 asok(0x4dfc000) register_command git_version hook 0x4d980a8 -20> 2015-04-28 01:46:28.629014 7f51610a4900 5 asok(0x4dfc000) register_command help hook 0x4d9c150 -19> 2015-04-28 01:46:28.629021 7f51610a4900 5 asok(0x4dfc000) register_command get_command_descriptions hook 0x4d9c140 -18> 2015-04-28 01:46:28.629068 7f51610a4900 10 monclient(hunting): build_initial_monmap -17> 2015-04-28 01:46:28.631927 7f515b1af700 5 asok(0x4dfc000) entry start -16> 2015-04-28 01:46:28.638456 7f51610a4900 5 adding auth protocol: cephx -15> 2015-04-28 01:46:28.638474 7f51610a4900 5 adding auth protocol: cephx -14> 2015-04-28 01:46:28.638592 7f51610a4900 5 asok(0x4dfc000) register_command objecter_requests hook 0x4d9c180 -13> 2015-04-28 01:46:28.638691 7f51610a4900 1 -- 192.168.30.73:6800/78592 messenger.start -12> 2015-04-28 01:46:28.638734 7f51610a4900 1 -- :/0 messenger.start -11> 2015-04-28 01:46:28.638766 7f51610a4900 1 -- 192.168.30.73:6801/78592 messenger.start -10> 2015-04-28 01:46:28.638802 7f51610a4900 1 -- 192.168.32.73:6801/78592 messenger.start -9> 2015-04-28 01:46:28.638835 7f51610a4900 1 -- 192.168.32.73:6800/78592 messenger.start -8> 2015-04-28 01:46:28.638865 7f51610a4900 1 -- :/0 messenger.start -7> 2015-04-28 01:46:28.638933 7f51610a4900 2 osd.23 0 mounting /var/lib/ceph/osd/ceph-23 /var/lib/ceph/osd/ceph-23/journal -6> 2015-04-28 01:46:28.638984 7f51610a4900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. -5> 2015-04-28 01:46:28.639024 7f51610a4900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) -4> 2015-04-28 01:46:28.640079 7f51610a4900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work -3> 2015-04-28 01:46:28.640091 7f51610a4900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option -2> 2015-04-28 01:46:28.640988 7f51610a4900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) -1> 2015-04-28 01:46:28.641075 7f51610a4900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.12-031812-generic >= 3.5 0> 2015-04-28 01:46:28.646675 7f51610a4900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f51610a4900 time 2015-04-28 01:46:28.644014 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f515e1e7ec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 01:46:28.650655 7f51610a4900 -1 *** Caught signal (Aborted) ** in thread 7f51610a4900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f515fd5d340] 3: (gsignal()+0x39) [0x7f515e1fccc9] 4: (abort()+0x148) [0x7f515e2000d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f515eb076b5] 6: (()+0x5e836) [0x7f515eb05836] 7: (()+0x5e863) [0x7f515eb05863] 8: (()+0x5eaa2) [0x7f515eb05aa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f515e1e7ec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2015-04-28 01:46:28.650655 7f51610a4900 -1 *** Caught signal (Aborted) ** in thread 7f51610a4900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f515fd5d340] 3: (gsignal()+0x39) [0x7f515e1fccc9] 4: (abort()+0x148) [0x7f515e2000d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f515eb076b5] 6: (()+0x5e836) [0x7f515eb05836] 7: (()+0x5e863) [0x7f515eb05863] 8: (()+0x5eaa2) [0x7f515eb05aa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f515e1e7ec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 01:46:29.036992 7f421b9a9900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 79108 2015-04-28 01:46:29.049097 7f421b9a9900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. 2015-04-28 01:46:29.049126 7f421b9a9900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) 2015-04-28 01:46:29.050078 7f421b9a9900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work 2015-04-28 01:46:29.050089 7f421b9a9900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-04-28 01:46:29.050661 7f421b9a9900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2015-04-28 01:46:29.050740 7f421b9a9900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.12-031812-generic >= 3.5 2015-04-28 01:46:29.055591 7f421b9a9900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f421b9a9900 time 2015-04-28 01:46:29.053166 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f4218aecec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -48> 2015-04-28 01:46:29.034690 7f421b9a9900 5 asok(0x4b16000) register_command perfcounters_dump hook 0x4ab6050 -47> 2015-04-28 01:46:29.034731 7f421b9a9900 5 asok(0x4b16000) register_command 1 hook 0x4ab6050 -46> 2015-04-28 01:46:29.034737 7f421b9a9900 5 asok(0x4b16000) register_command perf dump hook 0x4ab6050 -45> 2015-04-28 01:46:29.034746 7f421b9a9900 5 asok(0x4b16000) register_command perfcounters_schema hook 0x4ab6050 -44> 2015-04-28 01:46:29.034750 7f421b9a9900 5 asok(0x4b16000) register_command 2 hook 0x4ab6050 -43> 2015-04-28 01:46:29.034753 7f421b9a9900 5 asok(0x4b16000) register_command perf schema hook 0x4ab6050 -42> 2015-04-28 01:46:29.034758 7f421b9a9900 5 asok(0x4b16000) register_command perf reset hook 0x4ab6050 -41> 2015-04-28 01:46:29.034773 7f421b9a9900 5 asok(0x4b16000) register_command config show hook 0x4ab6050 -40> 2015-04-28 01:46:29.034779 7f421b9a9900 5 asok(0x4b16000) register_command config set hook 0x4ab6050 -39> 2015-04-28 01:46:29.034783 7f421b9a9900 5 asok(0x4b16000) register_command config get hook 0x4ab6050 -38> 2015-04-28 01:46:29.034787 7f421b9a9900 5 asok(0x4b16000) register_command config diff hook 0x4ab6050 -37> 2015-04-28 01:46:29.034792 7f421b9a9900 5 asok(0x4b16000) register_command log flush hook 0x4ab6050 -36> 2015-04-28 01:46:29.034796 7f421b9a9900 5 asok(0x4b16000) register_command log dump hook 0x4ab6050 -35> 2015-04-28 01:46:29.034801 7f421b9a9900 5 asok(0x4b16000) register_command log reopen hook 0x4ab6050 -34> 2015-04-28 01:46:29.036992 7f421b9a9900 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process ceph-osd, pid 79108 -33> 2015-04-28 01:46:29.037857 7f421b9a9900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -32> 2015-04-28 01:46:29.037868 7f421b9a9900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6800/79108 need_addr=0 -31> 2015-04-28 01:46:29.037885 7f421b9a9900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -30> 2015-04-28 01:46:29.037890 7f421b9a9900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6800/79108 need_addr=0 -29> 2015-04-28 01:46:29.037900 7f421b9a9900 1 -- 192.168.32.73:0/0 learned my addr 192.168.32.73:0/0 -28> 2015-04-28 01:46:29.037905 7f421b9a9900 1 accepter.accepter.bind my_inst.addr is 192.168.32.73:6801/79108 need_addr=0 -27> 2015-04-28 01:46:29.037914 7f421b9a9900 1 -- 192.168.30.73:0/0 learned my addr 192.168.30.73:0/0 -26> 2015-04-28 01:46:29.037919 7f421b9a9900 1 accepter.accepter.bind my_inst.addr is 192.168.30.73:6801/79108 need_addr=0 -25> 2015-04-28 01:46:29.039278 7f421b9a9900 5 asok(0x4b16000) init /var/run/ceph/ceph-osd.23.asok -24> 2015-04-28 01:46:29.039291 7f421b9a9900 5 asok(0x4b16000) bind_and_listen /var/run/ceph/ceph-osd.23.asok -23> 2015-04-28 01:46:29.039390 7f421b9a9900 5 asok(0x4b16000) register_command 0 hook 0x4ab20a8 -22> 2015-04-28 01:46:29.039398 7f421b9a9900 5 asok(0x4b16000) register_command version hook 0x4ab20a8 -21> 2015-04-28 01:46:29.039403 7f421b9a9900 5 asok(0x4b16000) register_command git_version hook 0x4ab20a8 -20> 2015-04-28 01:46:29.039409 7f421b9a9900 5 asok(0x4b16000) register_command help hook 0x4ab6150 -19> 2015-04-28 01:46:29.039413 7f421b9a9900 5 asok(0x4b16000) register_command get_command_descriptions hook 0x4ab6140 -18> 2015-04-28 01:46:29.039452 7f421b9a9900 10 monclient(hunting): build_initial_monmap -17> 2015-04-28 01:46:29.039476 7f4215ab4700 5 asok(0x4b16000) entry start -16> 2015-04-28 01:46:29.048680 7f421b9a9900 5 adding auth protocol: cephx -15> 2015-04-28 01:46:29.048692 7f421b9a9900 5 adding auth protocol: cephx -14> 2015-04-28 01:46:29.048797 7f421b9a9900 5 asok(0x4b16000) register_command objecter_requests hook 0x4ab6180 -13> 2015-04-28 01:46:29.048871 7f421b9a9900 1 -- 192.168.30.73:6800/79108 messenger.start -12> 2015-04-28 01:46:29.048910 7f421b9a9900 1 -- :/0 messenger.start -11> 2015-04-28 01:46:29.048932 7f421b9a9900 1 -- 192.168.30.73:6801/79108 messenger.start -10> 2015-04-28 01:46:29.048955 7f421b9a9900 1 -- 192.168.32.73:6801/79108 messenger.start -9> 2015-04-28 01:46:29.048977 7f421b9a9900 1 -- 192.168.32.73:6800/79108 messenger.start -8> 2015-04-28 01:46:29.048998 7f421b9a9900 1 -- :/0 messenger.start -7> 2015-04-28 01:46:29.049055 7f421b9a9900 2 osd.23 0 mounting /var/lib/ceph/osd/ceph-23 /var/lib/ceph/osd/ceph-23/journal -6> 2015-04-28 01:46:29.049097 7f421b9a9900 -1 filestore(/var/lib/ceph/osd/ceph-23) FileStore::mount : stale version stamp detected: 3. Proceeding, do_update is set, performing disk format upgrade. -5> 2015-04-28 01:46:29.049126 7f421b9a9900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) -4> 2015-04-28 01:46:29.050078 7f421b9a9900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work -3> 2015-04-28 01:46:29.050089 7f421b9a9900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option -2> 2015-04-28 01:46:29.050661 7f421b9a9900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) -1> 2015-04-28 01:46:29.050740 7f421b9a9900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is supported and kernel 3.18.12-031812-generic >= 3.5 0> 2015-04-28 01:46:29.055591 7f421b9a9900 -1 common/Throttle.cc: In function 'Throttle::Throttle(CephContext*, std::string, int64_t, bool)' thread 7f421b9a9900 time 2015-04-28 01:46:29.053166 common/Throttle.cc: 38: FAILED assert(m >= 0) ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc271b] 2: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 3: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 4: (FileStore::open_journal()+0xd9) [0x90a3c9] 5: (FileStore::mount()+0x2824) [0x90f9f4] 6: (OSD::init()+0x259) [0x6c47b9] 7: (main()+0x2860) [0x651fc0] 8: (__libc_start_main()+0xf5) [0x7f4218aecec5] 9: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 01:46:29.059070 7f421b9a9900 -1 *** Caught signal (Aborted) ** in thread 7f421b9a9900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f421a662340] 3: (gsignal()+0x39) [0x7f4218b01cc9] 4: (abort()+0x148) [0x7f4218b050d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f421940c6b5] 6: (()+0x5e836) [0x7f421940a836] 7: (()+0x5e863) [0x7f421940a863] 8: (()+0x5eaa2) [0x7f421940aaa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f4218aecec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2015-04-28 01:46:29.059070 7f421b9a9900 -1 *** Caught signal (Aborted) ** in thread 7f421b9a9900 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) 1: /usr/bin/ceph-osd() [0xacaf4a] 2: (()+0x10340) [0x7f421a662340] 3: (gsignal()+0x39) [0x7f4218b01cc9] 4: (abort()+0x148) [0x7f4218b050d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f421940c6b5] 6: (()+0x5e836) [0x7f421940a836] 7: (()+0x5e863) [0x7f421940a863] 8: (()+0x5eaa2) [0x7f421940aaa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0xbc2908] 10: (Throttle::Throttle(CephContext*, std::string, long, bool)+0x290) [0xba7670] 11: (FileJournal::FileJournal(uuid_d, Finisher*, Cond*, char const*, bool, bool, bool)+0x6ad) [0x9312dd] 12: (FileStore::open_journal()+0xd9) [0x90a3c9] 13: (FileStore::mount()+0x2824) [0x90f9f4] 14: (OSD::init()+0x259) [0x6c47b9] 15: (main()+0x2860) [0x651fc0] 16: (__libc_start_main()+0xf5) [0x7f4218aecec5] 17: /usr/bin/ceph-osd() [0x66aff7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.23.log --- end dump of recent events --- 2015-04-28 02:01:10.527075 7f1c5706a900 0 ceph version 0.87.2 (87a7cec9ab11c677de2ab23a7668a77d2f5b955e), process ceph-osd, pid 112284 2015-04-28 02:01:10.529626 7f1c5706a900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) 2015-04-28 02:01:10.529635 7f1c5706a900 1 filestore(/var/lib/ceph/osd/ceph-23) disabling 'filestore replica fadvise' due to known issues with fadvise(DONTNEED) on xfs 2015-04-28 02:01:10.530794 7f1c5706a900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work 2015-04-28 02:01:10.530805 7f1c5706a900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-04-28 02:01:10.531166 7f1c5706a900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2015-04-28 02:01:10.531220 7f1c5706a900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is disabled by conf 2015-04-28 02:01:10.533784 7f1c5706a900 0 filestore(/var/lib/ceph/osd/ceph-23) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled 2015-04-28 02:01:11.445569 7f1c5706a900 1 journal _open /var/lib/ceph/osd/ceph-23/journal fd 20: 59055800320 bytes, block size 4096 bytes, directio = 1, aio = 1 2015-04-28 02:01:11.448867 7f1c5706a900 1 journal _open /var/lib/ceph/osd/ceph-23/journal fd 20: 59055800320 bytes, block size 4096 bytes, directio = 1, aio = 1 2015-04-28 02:01:11.450409 7f1c5706a900 1 journal close /var/lib/ceph/osd/ceph-23/journal 2015-04-28 02:01:11.466185 7f1c5706a900 0 filestore(/var/lib/ceph/osd/ceph-23) backend xfs (magic 0x58465342) 2015-04-28 02:01:11.467550 7f1c5706a900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is supported and appears to work 2015-04-28 02:01:11.467563 7f1c5706a900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-04-28 02:01:11.468184 7f1c5706a900 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2015-04-28 02:01:11.468247 7f1c5706a900 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-23) detect_feature: extsize is disabled by conf 2015-04-28 02:01:11.471446 7f1c5706a900 0 filestore(/var/lib/ceph/osd/ceph-23) mount: WRITEAHEAD journal mode explicitly enabled in conf 2015-04-28 02:01:11.476949 7f1c5706a900 1 journal _open /var/lib/ceph/osd/ceph-23/journal fd 20: 59055800320 bytes, block size 4096 bytes, directio = 1, aio = 1 2015-04-28 02:01:11.480013 7f1c5706a900 1 journal _open /var/lib/ceph/osd/ceph-23/journal fd 20: 59055800320 bytes, block size 4096 bytes, directio = 1, aio = 1 2015-04-28 02:01:11.539517 7f1c5706a900 0 cls/hello/cls_hello.cc:271: loading cls_hello 2015-04-28 02:01:11.592068 7f1c5706a900 0 osd.23 2948 crush map has features 2200130813952, adjusting msgr requires for clients 2015-04-28 02:01:11.592083 7f1c5706a900 0 osd.23 2948 crush map has features 2200130813952 was 8705, adjusting msgr requires for mons 2015-04-28 02:01:11.592092 7f1c5706a900 0 osd.23 2948 crush map has features 2200130813952, adjusting msgr requires for osds 2015-04-28 02:01:11.592109 7f1c5706a900 0 osd.23 2948 load_pgs 2015-04-28 02:01:18.572347 7f1c5706a900 0 osd.23 2948 load_pgs opened 339 pgs 2015-04-28 02:01:18.575184 7f1c5706a900 -1 osd.23 2948 set_disk_tp_priority(22) Invalid argument: osd_disk_thread_ioprio_class is but only the following values are allowed: idle, be or rt 2015-04-28 02:01:18.580288 7f1c3b831700 0 osd.23 2948 ignoring osdmap until we have initialized 2015-04-28 02:01:18.580368 7f1c3b831700 0 osd.23 2948 ignoring osdmap until we have initialized 2015-04-28 02:01:18.583364 7f1c3b831700 0 osd.23 2948 ignoring osdmap until we have initialized 2015-04-28 02:01:18.591359 7f1c5706a900 0 osd.23 2948 done with init, starting boot process 2015-04-28 02:01:37.799066 7f1c2d815700 0 log_channel(default) log [INF] : 2.300 restarting backfill on osd.14 from (0'0,0'0] MAX to 2939'31751 2015-04-28 02:01:37.833799 7f1c2900c700 0 log_channel(default) log [INF] : 4.a restarting backfill on osd.8 from (0'0,0'0] MAX to 2948'28857 2015-04-28 02:01:37.873911 7f1c2f819700 0 log_channel(default) log [INF] : 4.5e restarting backfill on osd.9 from (0'0,0'0] MAX to 2948'33883 2015-04-28 02:01:37.881545 7f1c2b010700 0 log_channel(default) log [INF] : 3.25c restarting backfill on osd.15 from (0'0,0'0] MAX to 4031'182303 2015-04-28 02:01:39.084400 7f1c2b811700 0 log_channel(default) log [INF] : 4.30a restarting backfill on osd.14 from (0'0,0'0] MAX to 2948'29092 2015-04-28 02:01:39.084678 7f1c2980d700 0 log_channel(default) log [INF] : 3.3e restarting backfill on osd.8 from (0'0,0'0] MAX to 4085'229254 2015-04-28 02:01:39.085160 7f1c3101c700 0 log_channel(default) log [INF] : 4.29 restarting backfill on osd.10 from (0'0,0'0] MAX to 2948'39507 2015-04-28 02:01:39.085560 7f1c2d815700 0 log_channel(default) log [INF] : 4.30b restarting backfill on osd.13 from (0'0,0'0] MAX to 2948'36554 2015-04-28 02:01:39.089433 7f1c2f819700 0 log_channel(default) log [INF] : 3.67 restarting backfill on osd.13 from (0'0,0'0] MAX to 4126'373253 2015-04-28 02:01:39.098305 7f1c2f018700 0 log_channel(default) log [INF] : 4.e0 restarting backfill on osd.15 from (0'0,0'0] MAX to 2948'46062 2015-04-28 02:01:39.098859 7f1c2880b700 0 log_channel(default) log [INF] : 4.1a4 restarting backfill on osd.8 from (0'0,0'0] MAX to 2948'80598 2015-04-28 02:01:39.133820 7f1c3201e700 0 log_channel(default) log [INF] : 4.20 restarting backfill on osd.9 from (0'0,0'0] MAX to 2948'41173 2015-04-28 02:01:39.143109 7f1c2980d700 0 log_channel(default) log [INF] : 3.275 restarting backfill on osd.13 from (0'0,0'0] MAX to 4026'527541 2015-04-28 02:01:39.143394 7f1c2900c700 0 log_channel(default) log [INF] : 2.199 restarting backfill on osd.14 from (0'0,0'0] MAX to 2939'24377 2015-04-28 02:01:40.093593 7f1c2f819700 0 log_channel(default) log [INF] : 3.29d restarting backfill on osd.13 from (0'0,0'0] MAX to 4132'114304 2015-04-28 02:01:40.105105 7f1c2c813700 0 log_channel(default) log [INF] : 4.2cf restarting backfill on osd.14 from (0'0,0'0] MAX to 2948'41567 2015-04-28 02:01:40.105475 7f1c2b010700 0 log_channel(default) log [INF] : 4.31f restarting backfill on osd.10 from (0'0,0'0] MAX to 2948'36672 2015-04-28 02:01:40.107535 7f1c2c012700 0 log_channel(default) log [INF] : 2.1d restarting backfill on osd.10 from (0'0,0'0] MAX to 2939'5776 2015-04-28 02:01:40.108459 7f1c3101c700 0 log_channel(default) log [INF] : 2.2af restarting backfill on osd.13 from (0'0,0'0] MAX to 2939'5845 2015-04-28 02:01:40.108766 7f1c2980d700 0 log_channel(default) log [INF] : 3.235 restarting backfill on osd.15 from (0'0,0'0] MAX to 4132'682673 2015-04-28 02:01:40.108814 7f1c3081b700 0 log_channel(default) log [INF] : 3.15 restarting backfill on osd.8 from (0'0,0'0] MAX to 4110'221929 2015-04-28 02:01:40.109200 7f1c3201e700 0 log_channel(default) log [INF] : 3.fa restarting backfill on osd.8 from (0'0,0'0] MAX to 4074'241700 2015-04-28 02:01:40.110005 7f1c2e817700 0 log_channel(default) log [INF] : 3.257 restarting backfill on osd.12 from (0'0,0'0] MAX to 4132'2354351 2015-04-28 02:01:40.124993 7f1c2d815700 0 log_channel(default) log [INF] : 4.132 restarting backfill on osd.10 from (0'0,0'0] MAX to 2948'73197 2015-04-28 02:01:40.151539 7f1c3181d700 0 log_channel(default) log [INF] : 3.21b restarting backfill on osd.10 from (0'0,0'0] MAX to 4061'299701 2015-04-28 02:01:59.387294 7f1c3eeb5700 0 -- 192.168.30.73:6815/112284 >> 192.168.30.222:0/4055426025 pipe(0x2e636680 sd=125 :6815 s=0 pgs=0 cs=0 l=0 c=0x2e8e2260).accept peer addr is really 192.168.30.222:0/4055426025 (socket is 192.168.30.222:34552/0) 2015-04-28 02:05:17.548286 7f1c0f4c7700 0 -- 192.168.30.73:6815/112284 >> 192.168.30.40:0/416961250 pipe(0x300402c0 sd=150 :6815 s=0 pgs=0 cs=0 l=0 c=0x2cb538c0).accept peer addr is really 192.168.30.40:0/416961250 (socket is 192.168.30.40:48720/0)