*** Caught signal (Segmentation fault) ** in thread 7f8f49a927c0 ceph version 0.80.7 (6c0127fcb58008793d3c8b62d925bc91963672a3) 1: rados() [0x43eedf] 2: (()+0x10340) [0x7f8f48738340] 3: (librados::IoCtxImpl::snap_lookup(char const*, unsigned long*)+0x17) [0x7f8f48aff127] 4: (main()+0x1385) [0x411e75] 5: (__libc_start_main()+0xf5) [0x7f8f4795fec5] 6: rados() [0x41c6f7] 2014-10-29 12:03:22.761653 7f8f49a927c0 -1 *** Caught signal (Segmentation fault) ** in thread 7f8f49a927c0 ceph version 0.80.7 (6c0127fcb58008793d3c8b62d925bc91963672a3) 1: rados() [0x43eedf] 2: (()+0x10340) [0x7f8f48738340] 3: (librados::IoCtxImpl::snap_lookup(char const*, unsigned long*)+0x17) [0x7f8f48aff127] 4: (main()+0x1385) [0x411e75] 5: (__libc_start_main()+0xf5) [0x7f8f4795fec5] 6: rados() [0x41c6f7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -59> 2014-10-29 12:03:22.753162 7f8f49a927c0 5 asok(0xfc8420) register_command perfcounters_dump hook 0xfc9f80 -58> 2014-10-29 12:03:22.753194 7f8f49a927c0 5 asok(0xfc8420) register_command 1 hook 0xfc9f80 -57> 2014-10-29 12:03:22.753198 7f8f49a927c0 5 asok(0xfc8420) register_command perf dump hook 0xfc9f80 -56> 2014-10-29 12:03:22.753205 7f8f49a927c0 5 asok(0xfc8420) register_command perfcounters_schema hook 0xfc9f80 -55> 2014-10-29 12:03:22.753213 7f8f49a927c0 5 asok(0xfc8420) register_command 2 hook 0xfc9f80 -54> 2014-10-29 12:03:22.753218 7f8f49a927c0 5 asok(0xfc8420) register_command perf schema hook 0xfc9f80 -53> 2014-10-29 12:03:22.753223 7f8f49a927c0 5 asok(0xfc8420) register_command config show hook 0xfc9f80 -52> 2014-10-29 12:03:22.753245 7f8f49a927c0 5 asok(0xfc8420) register_command config set hook 0xfc9f80 -51> 2014-10-29 12:03:22.753248 7f8f49a927c0 5 asok(0xfc8420) register_command config get hook 0xfc9f80 -50> 2014-10-29 12:03:22.753254 7f8f49a927c0 5 asok(0xfc8420) register_command log flush hook 0xfc9f80 -49> 2014-10-29 12:03:22.753256 7f8f49a927c0 5 asok(0xfc8420) register_command log dump hook 0xfc9f80 -48> 2014-10-29 12:03:22.753261 7f8f49a927c0 5 asok(0xfc8420) register_command log reopen hook 0xfc9f80 -47> 2014-10-29 12:03:22.756957 7f8f49a927c0 10 monclient(hunting): build_initial_monmap -46> 2014-10-29 12:03:22.757031 7f8f49a927c0 1 librados: starting msgr at :/0 -45> 2014-10-29 12:03:22.757042 7f8f49a927c0 1 librados: starting objecter -44> 2014-10-29 12:03:22.757072 7f8f49a927c0 1 -- :/0 messenger.start -43> 2014-10-29 12:03:22.757106 7f8f49a927c0 1 librados: setting wanted keys -42> 2014-10-29 12:03:22.757110 7f8f49a927c0 1 librados: calling monclient init -41> 2014-10-29 12:03:22.757112 7f8f49a927c0 10 monclient(hunting): init -40> 2014-10-29 12:03:22.757116 7f8f49a927c0 5 adding auth protocol: cephx -39> 2014-10-29 12:03:22.757119 7f8f49a927c0 10 monclient(hunting): auth_supported 2 method cephx -38> 2014-10-29 12:03:22.757305 7f8f49a927c0 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.client.admin.keyring -37> 2014-10-29 12:03:22.757380 7f8f49a927c0 10 monclient(hunting): _reopen_session rank -1 name -36> 2014-10-29 12:03:22.757434 7f8f49a927c0 10 monclient(hunting): picked mon.node02 con 0xfea490 addr 10.102.4.12:6789/0 -35> 2014-10-29 12:03:22.757464 7f8f49a927c0 10 monclient(hunting): _send_mon_message to mon.node02 at 10.102.4.12:6789/0 -34> 2014-10-29 12:03:22.757469 7f8f49a927c0 1 -- :/1025898 --> 10.102.4.12:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- ?+0 0xfea850 con 0xfea490 -33> 2014-10-29 12:03:22.757479 7f8f49a927c0 10 monclient(hunting): renew_subs -32> 2014-10-29 12:03:22.757482 7f8f49a927c0 10 monclient(hunting): authenticate will time out at 2014-10-29 12:08:22.757482 -31> 2014-10-29 12:03:22.758184 7f8f49a8a700 1 -- 10.102.4.8:0/1025898 learned my addr 10.102.4.8:0/1025898 -30> 2014-10-29 12:03:22.759180 7f8f44bc5700 1 -- 10.102.4.8:0/1025898 <== mon.1 10.102.4.12:6789/0 1 ==== mon_map v1 ==== 488+0+0 (2841544696 0 0) 0x7f8f34000bd0 con 0xfea490 -29> 2014-10-29 12:03:22.759224 7f8f44bc5700 10 monclient(hunting): handle_monmap mon_map v1 -28> 2014-10-29 12:03:22.759236 7f8f44bc5700 10 monclient(hunting): got monmap 3, mon.node02 is now rank 1 -27> 2014-10-29 12:03:22.759239 7f8f44bc5700 10 monclient(hunting): dump: epoch 3 fsid 16f21728-8aeb-4e61-b7d9-b395ed0d2461 last_changed 2014-01-31 17:09:21.797791 created 0.000000 0: 10.102.4.11:6789/0 mon.node01 1: 10.102.4.12:6789/0 mon.node02 2: 10.102.4.13:6789/0 mon.node03 -26> 2014-10-29 12:03:22.759268 7f8f44bc5700 1 -- 10.102.4.8:0/1025898 <== mon.1 10.102.4.12:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (224192321 0 0) 0x7f8f340010c0 con 0xfea490 -25> 2014-10-29 12:03:22.759299 7f8f44bc5700 10 monclient(hunting): my global_id is 92212660 -24> 2014-10-29 12:03:22.759440 7f8f44bc5700 10 monclient(hunting): _send_mon_message to mon.node02 at 10.102.4.12:6789/0 -23> 2014-10-29 12:03:22.759449 7f8f44bc5700 1 -- 10.102.4.8:0/1025898 --> 10.102.4.12:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0 0x7f8f380015b0 con 0xfea490 -22> 2014-10-29 12:03:22.760192 7f8f44bc5700 1 -- 10.102.4.8:0/1025898 <== mon.1 10.102.4.12:6789/0 3 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (2473196904 0 0) 0x7f8f340010c0 con 0xfea490 -21> 2014-10-29 12:03:22.760294 7f8f44bc5700 10 monclient(hunting): _send_mon_message to mon.node02 at 10.102.4.12:6789/0 -20> 2014-10-29 12:03:22.760303 7f8f44bc5700 1 -- 10.102.4.8:0/1025898 --> 10.102.4.12:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- ?+0 0x7f8f38003760 con 0xfea490 -19> 2014-10-29 12:03:22.761069 7f8f44bc5700 1 -- 10.102.4.8:0/1025898 <== mon.1 10.102.4.12:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 393+0+0 (2402764092 0 0) 0x7f8f340008c0 con 0xfea490 -18> 2014-10-29 12:03:22.761136 7f8f44bc5700 1 monclient(hunting): found mon.node02 -17> 2014-10-29 12:03:22.761141 7f8f44bc5700 10 monclient: _send_mon_message to mon.node02 at 10.102.4.12:6789/0 -16> 2014-10-29 12:03:22.761144 7f8f44bc5700 1 -- 10.102.4.8:0/1025898 --> 10.102.4.12:6789/0 -- mon_subscribe({monmap=0+}) v2 -- ?+0 0xfeab30 con 0xfea490 -15> 2014-10-29 12:03:22.761203 7f8f49a927c0 5 monclient: authenticate success, global_id 92212660 -14> 2014-10-29 12:03:22.761232 7f8f49a927c0 5 asok(0xfc8420) register_command objecter_requests hook 0xfead40 -13> 2014-10-29 12:03:22.761293 7f8f49a927c0 10 monclient: renew_subs -12> 2014-10-29 12:03:22.761299 7f8f49a927c0 10 monclient: _send_mon_message to mon.node02 at 10.102.4.12:6789/0 -11> 2014-10-29 12:03:22.761305 7f8f49a927c0 1 -- 10.102.4.8:0/1025898 --> 10.102.4.12:6789/0 -- mon_subscribe({monmap=4+,osdmap=0}) v2 -- ?+0 0xfed050 con 0xfea490 -10> 2014-10-29 12:03:22.761316 7f8f49a927c0 10 monclient: renew_subs -9> 2014-10-29 12:03:22.761320 7f8f49a927c0 10 monclient: _send_mon_message to mon.node02 at 10.102.4.12:6789/0 -8> 2014-10-29 12:03:22.761322 7f8f49a927c0 1 -- 10.102.4.8:0/1025898 --> 10.102.4.12:6789/0 -- mon_subscribe({monmap=4+,osdmap=0}) v2 -- ?+0 0xfee100 con 0xfea490 -7> 2014-10-29 12:03:22.761362 7f8f49a927c0 1 librados: init done -6> 2014-10-29 12:03:22.761598 7f8f44bc5700 1 -- 10.102.4.8:0/1025898 <== mon.1 10.102.4.12:6789/0 5 ==== mon_map v1 ==== 488+0+0 (2841544696 0 0) 0x7f8f340013b0 con 0xfea490 -5> 2014-10-29 12:03:22.761617 7f8f44bc5700 10 monclient: handle_monmap mon_map v1 -4> 2014-10-29 12:03:22.761629 7f8f44bc5700 10 monclient: got monmap 3, mon.node02 is now rank 1 -3> 2014-10-29 12:03:22.761633 7f8f44bc5700 10 monclient: dump: epoch 3 fsid 16f21728-8aeb-4e61-b7d9-b395ed0d2461 last_changed 2014-01-31 17:09:21.797791 created 0.000000 0: 10.102.4.11:6789/0 mon.node01 1: 10.102.4.12:6789/0 mon.node02 2: 10.102.4.13:6789/0 mon.node03 -2> 2014-10-29 12:03:22.761651 7f8f44bc5700 1 -- 10.102.4.8:0/1025898 <== mon.1 10.102.4.12:6789/0 6 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (1490117661 0 0) 0x7f8f34001650 con 0xfea490 -1> 2014-10-29 12:03:22.761653 7f8f49a927c0 -1 *** Caught signal (Segmentation fault) ** in thread 7f8f49a927c0 ceph version 0.80.7 (6c0127fcb58008793d3c8b62d925bc91963672a3) 1: rados() [0x43eedf] 2: (()+0x10340) [0x7f8f48738340] 3: (librados::IoCtxImpl::snap_lookup(char const*, unsigned long*)+0x17) [0x7f8f48aff127] 4: (main()+0x1385) [0x411e75] 5: (__libc_start_main()+0xf5) [0x7f8f4795fec5] 6: rados() [0x41c6f7] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 0> 2014-10-29 12:03:22.761664 7f8f44bc5700 10 monclient: handle_subscribe_ack sent 2014-10-29 12:03:22.757480 renew after 2014-10-29 12:05:52.757480 --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle -2/-2 (syslog threshold) 99/99 (stderr threshold) max_recent 500 max_new 1000 log_file --- end dump of recent events ---