Project

General

Profile

Bug #2173 » mds.1.log

soft crack, 03/14/2012 04:14 AM

 
2012-03-14 19:11:32.796799 7fec60f517a0 ceph version 0.43 (commit:9fa8781c0147d66fcef7c2dd0e09cd3c69747d37), process ceph-mds, pid 21575
2012-03-14 19:11:32.796911 7fec60f517a0 -- 0.0.0.0:6803/21575 accepter.bind ms_addr is 0.0.0.0:6803/21575 need_addr=1
2012-03-14 19:11:32.797492 7fec60f517a0 monclient(hunting): build_initial_monmap
2012-03-14 19:11:32.797578 7fec60f517a0 -- 0.0.0.0:6803/21575 messenger.start
2012-03-14 19:11:32.797589 7fec60f517a0 -- 0.0.0.0:6803/21575 accepter.start
2012-03-14 19:11:32.797748 7fec60f517a0 mds.-1.0 168 MDSCacheObject
2012-03-14 19:11:32.797760 7fec60f517a0 mds.-1.0 2144 CInode
2012-03-14 19:11:32.797765 7fec60f517a0 mds.-1.0 16 elist<>::item *7=112
2012-03-14 19:11:32.797770 7fec60f517a0 mds.-1.0 360 inode_t
2012-03-14 19:11:32.797775 7fec60f517a0 mds.-1.0 56 nest_info_t
2012-03-14 19:11:32.797780 7fec60f517a0 mds.-1.0 32 frag_info_t
2012-03-14 19:11:32.797785 7fec60f517a0 mds.-1.0 40 SimpleLock *5=200
2012-03-14 19:11:32.797789 7fec60f517a0 mds.-1.0 48 ScatterLock *3=144
2012-03-14 19:11:32.797794 7fec60f517a0 mds.-1.0 472 CDentry
2012-03-14 19:11:32.797799 7fec60f517a0 mds.-1.0 16 elist<>::item
2012-03-14 19:11:32.797804 7fec60f517a0 mds.-1.0 40 SimpleLock
2012-03-14 19:11:32.797808 7fec60f517a0 mds.-1.0 1016 CDir
2012-03-14 19:11:32.797813 7fec60f517a0 mds.-1.0 16 elist<>::item *2=32
2012-03-14 19:11:32.797818 7fec60f517a0 mds.-1.0 192 fnode_t
2012-03-14 19:11:32.797823 7fec60f517a0 mds.-1.0 56 nest_info_t *2
2012-03-14 19:11:32.797827 7fec60f517a0 mds.-1.0 32 frag_info_t *2
2012-03-14 19:11:32.797833 7fec60f517a0 mds.-1.0 168 Capability
2012-03-14 19:11:32.797838 7fec60f517a0 mds.-1.0 32 xlist<>::item *2=64
2012-03-14 19:11:32.797865 7fec60f517a0 monclient(hunting): init auth_supported none
2012-03-14 19:11:32.797980 7fec60f517a0 monclient(hunting): auth_supported 1
2012-03-14 19:11:32.797996 7fec60f517a0 monclient(hunting): _reopen_session
2012-03-14 19:11:32.798043 7fec60f517a0 monclient(hunting): _pick_new_mon picked mon.1 con 0xdb9780 addr 192.168.10.201:6789/0
2012-03-14 19:11:32.798066 7fec60f517a0 monclient(hunting): _send_mon_message to mon.1 at 192.168.10.201:6789/0
2012-03-14 19:11:32.798082 7fec60f517a0 -- 0.0.0.0:6803/21575 --> 192.168.10.201:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- ?+0 0xdb0400 con 0xdb9780
2012-03-14 19:11:32.798089 7fec60f517a0 monclient(hunting): renew_subs
2012-03-14 19:11:32.798254 7fec60f4d700 -- 192.168.10.201:6803/21575 learned my addr 192.168.10.201:6803/21575
2012-03-14 19:11:32.798287 7fec60f4d700 mds.-1.0 MDS::ms_get_authorizer type=mon
2012-03-14 19:11:32.798512 7fec5c85e700 mds.-1.0 ms_handle_connect on 192.168.10.201:6789/0
2012-03-14 19:11:32.798930 7fec5c85e700 -- 192.168.10.201:6803/21575 <== mon.0 192.168.10.201:6789/0 1 ==== mon_map v1 ==== 187+0+0 (757208574 0 0) 0xdb0600 con 0xdb9780
2012-03-14 19:11:32.798946 7fec5c85e700 monclient(hunting): handle_monmap mon_map v1
2012-03-14 19:11:32.798964 7fec5c85e700 monclient(hunting): got monmap 3, mon.1 is now rank 0
2012-03-14 19:11:32.798999 7fec5c85e700 monclient(hunting): dump:
epoch 3
fsid a743a194-fa91-48fb-8778-e294483273d9
last_changed 2012-03-14 16:34:00.587587
created 2012-03-01 18:05:44.531734
0: 192.168.10.201:6789/0 mon.1

2012-03-14 19:11:32.799006 7fec5c85e700 monclient(hunting): found mon.1
2012-03-14 19:11:32.799030 7fec5c85e700 -- 192.168.10.201:6803/21575 <== mon.0 192.168.10.201:6789/0 2 ==== auth_reply(proto 1 0 Success) v1 ==== 24+0+0 (601130924 0 0) 0xdb0a00 con 0xdb9780
2012-03-14 19:11:32.799038 7fec5c85e700 monclient: my global_id is 6086
2012-03-14 19:11:32.799046 7fec5c85e700 monclient: _send_mon_message to mon.1 at 192.168.10.201:6789/0
2012-03-14 19:11:32.799059 7fec5c85e700 -- 192.168.10.201:6803/21575 --> 192.168.10.201:6789/0 -- mon_subscribe({monmap=0+}) v2 -- ?+0 0xdc4380 con 0xdb9780
2012-03-14 19:11:32.799076 7fec5c85e700 monclient: _check_auth_rotating renewing rotating keys (they expired before 2012-03-14 19:11:02.799070)
2012-03-14 19:11:32.799215 7fec60f517a0 monclient: authenticate success, global_id 6086
2012-03-14 19:11:32.799256 7fec60f517a0 mds.-1.0 beacon_send up:boot seq 1 (currently up:boot)
2012-03-14 19:11:32.799281 7fec60f517a0 monclient: _send_mon_message to mon.1 at 192.168.10.201:6789/0
2012-03-14 19:11:32.799291 7fec60f517a0 -- 192.168.10.201:6803/21575 --> 192.168.10.201:6789/0 -- mdsbeacon(6086/1 up:boot seq 1 v0) v2 -- ?+0 0xdb8c80 con 0xdb9780
2012-03-14 19:11:32.799315 7fec5c85e700 -- 192.168.10.201:6803/21575 <== mon.0 192.168.10.201:6789/0 3 ==== mon_map v1 ==== 187+0+0 (757208574 0 0) 0xdb0800 con 0xdb9780
2012-03-14 19:11:32.799346 7fec60f517a0 monclient: renew_subs
2012-03-14 19:11:32.799364 7fec60f517a0 monclient: _send_mon_message to mon.1 at 192.168.10.201:6789/0
2012-03-14 19:11:32.799381 7fec60f517a0 -- 192.168.10.201:6803/21575 --> 192.168.10.201:6789/0 -- mon_subscribe({monmap=4+,osdmap=0}) v2 -- ?+0 0xdc41c0 con 0xdb9780
2012-03-14 19:11:32.799396 7fec60f517a0 monclient: renew_subs
2012-03-14 19:11:32.799407 7fec60f517a0 monclient: _send_mon_message to mon.1 at 192.168.10.201:6789/0
2012-03-14 19:11:32.799416 7fec60f517a0 -- 192.168.10.201:6803/21575 --> 192.168.10.201:6789/0 -- mon_subscribe({mdsmap=0+,monmap=4+,osdmap=0}) v2 -- ?+0 0xdc4a80 con 0xdb9780
2012-03-14 19:11:32.799434 7fec60f517a0 mds.-1.0 create_logger
2012-03-14 19:11:32.799491 7fec5c85e700 monclient: handle_monmap mon_map v1
2012-03-14 19:11:32.799508 7fec5c85e700 monclient: got monmap 3, mon.1 is now rank 0
2012-03-14 19:11:32.799522 7fec5c85e700 monclient: dump:
epoch 3
fsid a743a194-fa91-48fb-8778-e294483273d9
last_changed 2012-03-14 16:34:00.587587
created 2012-03-01 18:05:44.531734
0: 192.168.10.201:6789/0 mon.1

2012-03-14 19:11:32.799536 7fec5c85e700 -- 192.168.10.201:6803/21575 <== mon.0 192.168.10.201:6789/0 4 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (718078579 0 0) 0xdc4540 con 0xdb9780
2012-03-14 19:11:32.799548 7fec5c85e700 monclient: handle_subscribe_ack sent 2012-03-14 19:11:32.798093 renew after 2012-03-14 19:14:02.798093
2012-03-14 19:11:32.906058 7fec5c85e700 -- 192.168.10.201:6803/21575 <== mon.0 192.168.10.201:6789/0 5 ==== osd_map(3074..3074 src has 2574..3074) v3 ==== 4479+0+0 (2117585099 0 0) 0xdc5400 con 0xdb9780
2012-03-14 19:11:32.906212 7fec5c85e700 -- 192.168.10.201:6803/21575 <== mon.0 192.168.10.201:6789/0 6 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (718078579 0 0) 0xdc41c0 con 0xdb9780
2012-03-14 19:11:32.906224 7fec5c85e700 monclient: handle_subscribe_ack sent 0.000000, ignoring
2012-03-14 19:11:32.906234 7fec5c85e700 -- 192.168.10.201:6803/21575 <== mon.0 192.168.10.201:6789/0 7 ==== mdsmap(e 27317) v1 ==== 1129+0+0 (703453368 0 0) 0xdc5200 con 0xdb9780
2012-03-14 19:11:32.906243 7fec5c85e700 mds.-1.0 handle_mds_map epoch 27317 from mon.0
2012-03-14 19:11:32.906269 7fec5c85e700 mds.-1.0 my compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object}
2012-03-14 19:11:32.906278 7fec5c85e700 mds.-1.0 mdsmap compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object}
2012-03-14 19:11:32.906288 7fec5c85e700 mds.-1.0 map says i am 192.168.10.201:6803/21575 mds.-1.0 state up:standby
2012-03-14 19:11:32.906293 7fec5c85e700 mds.-1.0 handle_mds_map standby
2012-03-14 19:11:32.906306 7fec5c85e700 -- 192.168.10.201:6803/21575 <== mon.0 192.168.10.201:6789/0 8 ==== osd_map(3074..3074 src has 2574..3074) v3 ==== 4479+0+0 (2117585099 0 0) 0xdc5000 con 0xdb9780
2012-03-14 19:11:32.906320 7fec5c85e700 -- 192.168.10.201:6803/21575 <== mon.0 192.168.10.201:6789/0 9 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (718078579 0 0) 0xdc4380 con 0xdb9780
2012-03-14 19:11:32.906326 7fec5c85e700 monclient: handle_subscribe_ack sent 0.000000, ignoring
2012-03-14 19:11:35.798155 7fec5c05d700 monclient: tick
2012-03-14 19:11:35.798198 7fec5c05d700 monclient: _check_auth_rotating renewing rotating keys (they expired before 2012-03-14 19:11:05.798187)
2012-03-14 19:11:36.736752 7fec5c85e700 -- 192.168.10.201:6803/21575 <== mon.0 192.168.10.201:6789/0 10 ==== mdsmap(e 27318) v1 ==== 931+0+0 (46806493 0 0) 0xdc5600 con 0xdb9780
2012-03-14 19:11:36.736828 7fec5c85e700 mds.-1.0 handle_mds_map epoch 27318 from mon.0
2012-03-14 19:11:36.736882 7fec5c85e700 mds.-1.0 my compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object}
2012-03-14 19:11:36.736900 7fec5c85e700 mds.-1.0 mdsmap compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object}
2012-03-14 19:11:36.736917 7fec5c85e700 mds.0.75 map says i am 192.168.10.201:6803/21575 mds.0.75 state up:replay
2012-03-14 19:11:36.736931 7fec5c85e700 mds.0.75 peer mds gid 6085 removed from map
2012-03-14 19:11:36.736941 7fec5c85e700 -- 192.168.10.201:6803/21575 mark_down 192.168.10.201:6803/21382 -- pipe dne
2012-03-14 19:11:36.736949 7fec5c85e700 mds.0.75 handle_mds_map i am now mds.0.75
2012-03-14 19:11:36.736956 7fec5c85e700 mds.0.75 handle_mds_map state change up:standby --> up:replay
2012-03-14 19:11:36.736961 7fec5c85e700 mds.0.75 replay_start
2012-03-14 19:11:36.736973 7fec5c85e700 mds.0.cache set_recovery_set
2012-03-14 19:11:36.736979 7fec5c85e700 mds.0.75 recovery set is
2012-03-14 19:11:36.736984 7fec5c85e700 mds.0.75 need osdmap epoch 3075, have 3074
2012-03-14 19:11:36.736990 7fec5c85e700 mds.0.75 waiting for osdmap 3075 (which blacklists prior instance)
2012-03-14 19:11:36.736998 7fec5c85e700 monclient: renew_subs
2012-03-14 19:11:36.737009 7fec5c85e700 monclient: _send_mon_message to mon.1 at 192.168.10.201:6789/0
2012-03-14 19:11:36.737020 7fec5c85e700 -- 192.168.10.201:6803/21575 --> 192.168.10.201:6789/0 -- mon_subscribe({mdsmap=27319+,monmap=4+,osdmap=3075}) v2 -- ?+0 0xdc4380 con 0xdb9780
2012-03-14 19:11:36.737079 7fec5c85e700 mds.0.cache handle_mds_failure mds.0
2012-03-14 19:11:36.737087 7fec5c85e700 mds.0.cache handle_mds_failure mds.0 : recovery peers are
2012-03-14 19:11:36.737093 7fec5c85e700 mds.0.cache wants_resolve
2012-03-14 19:11:36.737098 7fec5c85e700 mds.0.cache got_resolve
2012-03-14 19:11:36.737103 7fec5c85e700 mds.0.cache rejoin_sent
2012-03-14 19:11:36.737108 7fec5c85e700 mds.0.cache rejoin_gather
2012-03-14 19:11:36.737113 7fec5c85e700 mds.0.cache rejoin_ack_gather
2012-03-14 19:11:36.737125 7fec5c85e700 mds.0.migrator handle_mds_failure_or_stop mds.0
2012-03-14 19:11:36.737141 7fec5c85e700 mds.0.cache show_subtrees - no subtrees
2012-03-14 19:11:36.764337 7fec5c85e700 -- 192.168.10.201:6803/21575 <== mon.0 192.168.10.201:6789/0 11 ==== osd_map(3075..3075 src has 2575..3075) v3 ==== 300+0+0 (2596067037 0 0) 0xdc5400 con 0xdb9780
2012-03-14 19:11:36.764421 7fec5c85e700 mds.0.75 boot_start 1: opening inotable
2012-03-14 19:11:36.764441 7fec5c85e700 mds.0.inotable: load
2012-03-14 19:11:36.764519 7fec5c85e700 -- 192.168.10.201:6803/21575 --> 192.168.10.201:6800/20518 -- osd_op(mds.0.75:1 mds0_inotable [read 0~0] 1.b852b893) v4 -- ?+0 0xd9f240 con 0xdc8280
2012-03-14 19:11:36.764532 7fec5c85e700 mds.0.75 boot_start 1: opening sessionmap
2012-03-14 19:11:36.764542 7fec5c85e700 mds.0.sessionmap load
2012-03-14 19:11:36.764561 7fec5c85e700 -- 192.168.10.201:6803/21575 --> 192.168.10.201:6800/20518 -- osd_op(mds.0.75:2 mds0_sessionmap [read 0~0] 1.3270c60b) v4 -- ?+0 0xd9f6c0 con 0xdc8280
2012-03-14 19:11:36.764570 7fec5c85e700 mds.0.75 boot_start 1: opening anchor table
2012-03-14 19:11:36.764578 7fec5c85e700 mds.0.anchortable: load
2012-03-14 19:11:36.764594 7fec5c85e700 -- 192.168.10.201:6803/21575 --> 192.168.10.201:6800/20518 -- osd_op(mds.0.75:3 mds_anchortable [read 0~0] 1.a977f6a7) v4 -- ?+0 0xd9f480 con 0xdc8280
2012-03-14 19:11:36.764602 7fec5c85e700 mds.0.75 boot_start 1: opening snap table
2012-03-14 19:11:36.764608 7fec5c85e700 mds.0.snaptable: load
2012-03-14 19:11:36.764623 7fec5c85e700 -- 192.168.10.201:6803/21575 --> 192.168.10.201:6800/20518 -- osd_op(mds.0.75:4 mds_snaptable [read 0~0] 1.d90270ad) v4 -- ?+0 0xd9fd80 con 0xdc8280
2012-03-14 19:11:36.764630 7fec5c85e700 mds.0.75 boot_start 1: opening mds log
2012-03-14 19:11:36.764636 7fec5c85e700 mds.0.log open discovering log bounds
2012-03-14 19:11:36.764698 7fec5c85e700 -- 192.168.10.201:6803/21575 --> 192.168.10.205:6800/6301 -- osd_op(mds.0.75:5 200.00000000 [read 0~0] 1.844f3494) v4 -- ?+0 0xd9fb40 con 0xdc83c0
2012-03-14 19:11:36.764731 7fec5c85e700 -- 192.168.10.201:6803/21575 <== mon.0 192.168.10.201:6789/0 12 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (718078579 0 0) 0xdd3380 con 0xdb9780
2012-03-14 19:11:36.764744 7fec5c85e700 monclient: handle_subscribe_ack sent 2012-03-14 19:11:36.737002 renew after 2012-03-14 19:14:06.737002
2012-03-14 19:11:36.764779 7fec5a759700 mds.0.75 MDS::ms_get_authorizer type=osd
2012-03-14 19:11:36.765007 7fec5c85e700 mds.0.75 ms_handle_connect on 192.168.10.201:6800/20518
2012-03-14 19:11:36.765122 7fec5a658700 mds.0.75 MDS::ms_get_authorizer type=osd
2012-03-14 19:11:36.765414 7fec5c85e700 mds.0.75 ms_handle_connect on 192.168.10.205:6800/6301
2012-03-14 19:11:36.766249 7fec5c85e700 -- 192.168.10.201:6803/21575 <== osd.0 192.168.10.205:6800/6301 1 ==== osd_op_reply(5 200.00000000 [read 0~84] = 0) v4 ==== 111+0+84 (3973618673 0 726835425) 0xdc5800 con 0xdc83c0
2012-03-14 19:11:36.766348 7fec5c85e700 -- 192.168.10.201:6803/21575 --> 192.168.10.201:6800/20518 -- osd_op(mds.0.75:6 200.00000037 [stat 0~0] 1.8293696a rwordered) v4 -- ?+0 0xd9f900 con 0xdc8280
2012-03-14 19:11:36.766372 7fec5c85e700 -- 192.168.10.201:6803/21575 --> 192.168.10.201:6800/20518 -- osd_op(mds.0.75:7 200.00000038 [stat 0~0] 1.63762215 rwordered) v4 -- ?+0 0xd9f6c0 con 0xdc8280
2012-03-14 19:11:36.799480 7fec5af5a700 mds.0.75 beacon_send up:replay seq 2 (currently up:replay)
2012-03-14 19:11:36.799635 7fec5af5a700 monclient: _send_mon_message to mon.1 at 192.168.10.201:6789/0
2012-03-14 19:11:36.799650 7fec5af5a700 -- 192.168.10.201:6803/21575 --> 192.168.10.201:6789/0 -- mdsbeacon(6086/1 up:replay seq 2 v27318) v2 -- ?+0 0xdd4780 con 0xdb9780
2012-03-14 19:11:36.800440 7fec5c85e700 -- 192.168.10.201:6803/21575 <== mon.0 192.168.10.201:6789/0 13 ==== mdsbeacon(6086/1 up:replay seq 2 v27318) v2 ==== 103+0+0 (1486747229 0 0) 0xdb8c80 con 0xdb9780
2012-03-14 19:11:36.800465 7fec5c85e700 mds.0.75 handle_mds_beacon up:replay seq 2 rtt 0.000845
2012-03-14 19:11:36.805298 7fec5c85e700 -- 192.168.10.201:6803/21575 <== osd.1 192.168.10.201:6800/20518 1 ==== osd_op_reply(1 mds0_inotable [read 0~61] = 0) v4 ==== 112+0+61 (2516225183 0 573390385) 0xdc5000 con 0xdc8280
2012-03-14 19:11:36.805323 7fec5c85e700 mds.0.inotable: load_2 got 61 bytes
2012-03-14 19:11:36.805336 7fec5c85e700 mds.0.inotable: load_2 loaded v54
2012-03-14 19:11:36.805368 7fec5c85e700 -- 192.168.10.201:6803/21575 <== osd.1 192.168.10.201:6800/20518 2 ==== osd_op_reply(2 mds0_sessionmap [read 0~0] = -2 (No such file or directory)) v4 ==== 114+0+0 (2111454805 0 0) 0xdc5400 con 0xdc8280
2012-03-14 19:11:36.805380 7fec5c85e700 mds.0.sessionmap dump
*** Caught signal (Aborted) **
in thread 7fec5c85e700
ceph version 0.43 (commit:9fa8781c0147d66fcef7c2dd0e09cd3c69747d37)
1: /usr/bin/ceph-mds() [0x7a2206]
2: (()+0x10060) [0x7fec60b32060]
3: (gsignal()+0x35) [0x7fec5f2b33a5]
4: (abort()+0x17b) [0x7fec5f2b6b0b]
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7fec5fb71d7d]
6: (()+0xb9f26) [0x7fec5fb6ff26]
7: (()+0xb9f53) [0x7fec5fb6ff53]
8: (()+0xba04e) [0x7fec5fb7004e]
9: (ceph::buffer::list::iterator::copy(unsigned int, char*)+0x127) [0x73b117]
10: (SessionMap::decode(ceph::buffer::list::iterator&)+0x50) [0x66f760]
11: (SessionMap::_load_finish(int, ceph::buffer::list&)+0x61) [0x6705c1]
12: (Objecter::handle_osd_op_reply(MOSDOpReply*)+0x119b) [0x68853b]
13: (MDS::handle_core_message(Message*)+0x987) [0x4c6147]
14: (MDS::_dispatch(Message*)+0x2f) [0x4c62df]
15: (MDS::ms_dispatch(Message*)+0x1da) [0x4c7b2a]
16: (SimpleMessenger::dispatch_entry()+0x783) [0x722993]
17: (SimpleMessenger::DispatchThread::entry()+0x1c) [0x4a545c]
18: (()+0x7efc) [0x7fec60b29efc]
19: (clone()+0x6d) [0x7fec5f35e89d]
(1-1/4)