<6>[ 312.408003] libceph: loaded (mon/osd proto 15/24, osdmap 5/6 5/6) <6>[ 312.444268] rbd: loaded rbd (rados block device) <6>[ 312.760938] libceph: client4132 fsid 155e9d3f-16e7-4f5e-9e56-a90071f0ea16 <6>[ 312.769991] libceph: mon1 10.214.133.30:6789 session established <6>[ 312.808073] rbd1: unknown partition table <6>[ 312.812642] rbd: rbd1: added with size 0x280000000 <6>[ 317.574137] kjournald starting. Commit interval 5 seconds <6>[ 317.601895] EXT3-fs (rbd1): using internal journal <6>[ 317.606739] EXT3-fs (rbd1): mounted filesystem with ordered data mode <7>[ 1032.279787] libceph: osds timeout <7>[ 1032.279793] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1042.981191] libceph: monc delayed_work <7>[ 1042.981198] libceph: con_keepalive ffff8802214c4410 <7>[ 1042.981204] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 1042.981209] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 1042.981212] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 1042.981215] libceph: __send_subscribe sub_sent=0 exp=0 want_osd=0 <7>[ 1042.981218] libceph: __schedule_delayed after 2000 <7>[ 1042.981225] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1042.981227] libceph: try_read tag 1 in_base_pos 0 <7>[ 1042.981235] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 1042.981237] libceph: try_write start ffff8802214c4410 state 5 <7>[ 1042.981240] libceph: try_write out_kvec_bytes 0 <7>[ 1042.981242] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 1042.981245] libceph: try_write out_kvec_bytes 1 <7>[ 1042.981247] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 1042.981256] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1042.981259] libceph: try_write nothing else to write. <7>[ 1042.981261] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 1047.253774] libceph: osds timeout <7>[ 1047.253780] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1062.227753] libceph: osds timeout <7>[ 1062.227758] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1062.986435] libceph: monc delayed_work <7>[ 1062.986442] libceph: con_keepalive ffff8802214c4410 <7>[ 1062.986448] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 1062.986452] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 1062.986455] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 1062.986459] libceph: __send_subscribe sub_sent=0 exp=1 want_osd=0 <7>[ 1062.986464] libceph: ----- ffff88020d5b5400 to mon1 15=mon_subscribe len 31+0+0 ----- <7>[ 1062.986468] libceph: __schedule_delayed after 1000 <7>[ 1062.986474] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1062.986477] libceph: try_read tag 1 in_base_pos 0 <7>[ 1062.986484] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 1062.986486] libceph: try_write start ffff8802214c4410 state 5 <7>[ 1062.986489] libceph: try_write out_kvec_bytes 0 <7>[ 1062.986492] libceph: prepare_write_message ffff88020d5b5400 seq 9 type 15 len 31+0+0 0 pgs <7>[ 1062.986498] libceph: prepare_write_message front_crc 2650579135 middle_crc 0 <7>[ 1062.986501] libceph: prepare_write_message_footer ffff8802214c4410 <7>[ 1062.986503] libceph: try_write out_kvec_bytes 98 <7>[ 1062.986506] libceph: write_partial_kvec ffff8802214c4410 98 left <7>[ 1062.986524] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1062.986527] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 1062.986530] libceph: try_write out_kvec_bytes 1 <7>[ 1062.986532] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 1062.986538] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1062.986540] libceph: try_write nothing else to write. <7>[ 1062.986543] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 1062.987030] libceph: ceph_sock_data_ready on ffff8802214c4410 state = 5, queueing work <7>[ 1062.987041] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 1062.987059] libceph: ceph_sock_data_ready on ffff8802214c4410 state = 5, queueing work <7>[ 1062.987062] libceph: queue_con_delay ffff8802214c4410 - already queued <7>[ 1062.987079] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1062.987082] libceph: try_read tag 1 in_base_pos 0 <7>[ 1062.987089] libceph: try_read got tag 8 <7>[ 1062.987092] libceph: prepare_read_ack ffff8802214c4410 <7>[ 1062.987097] libceph: got ack for seq 9 type 15 at ffff88020d5b5400 <7>[ 1062.987100] libceph: prepare_read_tag ffff8802214c4410 <7>[ 1062.987102] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1062.987105] libceph: try_read tag 1 in_base_pos 0 <7>[ 1062.987109] libceph: try_read got tag 7 <7>[ 1062.987111] libceph: prepare_read_message ffff8802214c4410 <7>[ 1062.987114] libceph: read_partial_message con ffff8802214c4410 msg (null) <7>[ 1062.987120] libceph: got hdr type 4 front 481 data 0 <7>[ 1062.987124] libceph: ceph_msg_new ffff88020d35c000 front 481 <7>[ 1062.987135] libceph: read_partial_message got msg ffff88020d35c000 481 (2132872380) + 0 (0) + 0 (0) <7>[ 1062.987140] libceph: ===== ffff88020d35c000 16 from mon1 4=mon_map len 481+0 (2132872380 0 0) ===== <7>[ 1062.987144] libceph: have_debugfs_info fsid 1 globalid 4132 <7>[ 1062.987146] libceph: handle_monmap <7>[ 1062.987149] libceph: monmap_decode ffff88020d750c04 ffff88020d750de1 len 477 <7>[ 1062.987152] libceph: monmap_decode epoch 1, num_mon 3 <7>[ 1062.987156] libceph: monmap_decode mon0 is 10.214.133.28:6789 <7>[ 1062.987159] libceph: monmap_decode mon1 is 10.214.133.30:6789 <7>[ 1062.987162] libceph: monmap_decode mon2 is 10.214.133.28:6790 <7>[ 1062.987166] libceph: ceph_msg_put last one on ffff88020d35c000 <7>[ 1062.987169] libceph: msg_kfree ffff88020d35c000 <7>[ 1062.987172] libceph: prepare_read_tag ffff8802214c4410 <7>[ 1062.987175] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1062.987178] libceph: try_read tag 1 in_base_pos 0 <7>[ 1062.987182] libceph: try_read got tag 7 <7>[ 1062.987185] libceph: prepare_read_message ffff8802214c4410 <7>[ 1062.987187] libceph: read_partial_message con ffff8802214c4410 msg (null) <7>[ 1062.987192] libceph: got hdr type 16 front 20 data 0 <7>[ 1062.987211] libceph: read_partial_message got msg ffff88020d5b4500 20 (473278946) + 0 (0) + 0 (0) <7>[ 1062.987216] libceph: ===== ffff88020d5b4500 17 from mon1 16=mon_subscribe_ack len 20+0 (473278946 0 0) ===== <7>[ 1062.987220] libceph: handle_subscribe_ack after 300 seconds <7>[ 1062.987223] libceph: prepare_read_tag ffff8802214c4410 <7>[ 1062.987226] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1062.987228] libceph: try_read tag 1 in_base_pos 0 <7>[ 1062.987233] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 1062.987235] libceph: try_write start ffff8802214c4410 state 5 <7>[ 1062.987238] libceph: try_write out_kvec_bytes 0 <7>[ 1062.987241] libceph: prepare_write_ack ffff8802214c4410 15 -> 17 <7>[ 1062.987243] libceph: try_write out_kvec_bytes 9 <7>[ 1062.987246] libceph: write_partial_kvec ffff8802214c4410 9 left <7>[ 1062.987253] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1062.987256] libceph: try_write nothing else to write. <7>[ 1062.987258] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 1072.989061] libceph: monc delayed_work <7>[ 1072.989068] libceph: con_keepalive ffff8802214c4410 <7>[ 1072.989075] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 1072.989079] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 1072.989082] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 1072.989085] libceph: __send_subscribe sub_sent=0 exp=0 want_osd=0 <7>[ 1072.989088] libceph: __schedule_delayed after 2000 <7>[ 1072.989095] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1072.989097] libceph: try_read tag 1 in_base_pos 0 <7>[ 1072.989104] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 1072.989107] libceph: try_write start ffff8802214c4410 state 5 <7>[ 1072.989109] libceph: try_write out_kvec_bytes 0 <7>[ 1072.989112] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 1072.989114] libceph: try_write out_kvec_bytes 1 <7>[ 1072.989116] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 1072.989125] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1072.989128] libceph: try_write nothing else to write. <7>[ 1072.989130] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 1077.201744] libceph: osds timeout <7>[ 1077.201749] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1092.175742] libceph: osds timeout <7>[ 1092.175747] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1093.014283] libceph: monc delayed_work <7>[ 1093.014290] libceph: con_keepalive ffff8802214c4410 <7>[ 1093.014296] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 1093.014300] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 1093.014303] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 1093.014306] libceph: __send_subscribe sub_sent=0 exp=0 want_osd=0 <7>[ 1093.014309] libceph: __schedule_delayed after 2000 <7>[ 1093.014316] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1093.014318] libceph: try_read tag 1 in_base_pos 0 <7>[ 1093.014325] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 1093.014328] libceph: try_write start ffff8802214c4410 state 5 <7>[ 1093.014330] libceph: try_write out_kvec_bytes 0 <7>[ 1093.014333] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 1093.014335] libceph: try_write out_kvec_bytes 1 <7>[ 1093.014337] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 1093.014346] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1093.014349] libceph: try_write nothing else to write. <7>[ 1093.014351] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 1107.149730] libceph: osds timeout <7>[ 1107.149735] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1113.019538] libceph: monc delayed_work <7>[ 1113.019544] libceph: con_keepalive ffff8802214c4410 <7>[ 1113.019550] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 1113.019555] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 1113.019558] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 1113.019561] libceph: __send_subscribe sub_sent=0 exp=0 want_osd=0 <7>[ 1113.019563] libceph: __schedule_delayed after 2000 <7>[ 1113.019570] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1113.019573] libceph: try_read tag 1 in_base_pos 0 <7>[ 1113.019579] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 1113.019582] libceph: try_write start ffff8802214c4410 state 5 <7>[ 1113.019584] libceph: try_write out_kvec_bytes 0 <7>[ 1113.019587] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 1113.019589] libceph: try_write out_kvec_bytes 1 <7>[ 1113.019592] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 1113.019601] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1113.019603] libceph: try_write nothing else to write. <7>[ 1113.019606] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 1122.123724] libceph: osds timeout <7>[ 1122.123730] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1133.024774] libceph: monc delayed_work <7>[ 1133.024780] libceph: con_keepalive ffff8802214c4410 <7>[ 1133.024786] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 1133.024791] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 1133.024794] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 1133.024797] libceph: __send_subscribe sub_sent=0 exp=0 want_osd=0 <7>[ 1133.024799] libceph: __schedule_delayed after 2000 <7>[ 1133.024806] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1133.024809] libceph: try_read tag 1 in_base_pos 0 <7>[ 1133.024816] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 1133.024819] libceph: try_write start ffff8802214c4410 state 5 <7>[ 1133.024821] libceph: try_write out_kvec_bytes 0 <7>[ 1133.024823] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 1133.024826] libceph: try_write out_kvec_bytes 1 <7>[ 1133.024828] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 1133.024837] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1133.024839] libceph: try_write nothing else to write. <7>[ 1133.024842] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 1137.097712] libceph: osds timeout <7>[ 1137.097718] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1152.071702] libceph: osds timeout <7>[ 1152.071708] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1153.030043] libceph: monc delayed_work <7>[ 1153.030049] libceph: con_keepalive ffff8802214c4410 <7>[ 1153.030055] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 1153.030060] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 1153.030063] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 1153.030066] libceph: __send_subscribe sub_sent=0 exp=0 want_osd=0 <7>[ 1153.030068] libceph: __schedule_delayed after 2000 <7>[ 1153.030075] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1153.030078] libceph: try_read tag 1 in_base_pos 0 <7>[ 1153.030084] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 1153.030087] libceph: try_write start ffff8802214c4410 state 5 <7>[ 1153.030090] libceph: try_write out_kvec_bytes 0 <7>[ 1153.030092] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 1153.030094] libceph: try_write out_kvec_bytes 1 <7>[ 1153.030097] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 1153.030106] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1153.030109] libceph: try_write nothing else to write. <7>[ 1153.030111] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 1167.045694] libceph: osds timeout <7>[ 1167.045700] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1173.035292] libceph: monc delayed_work <7>[ 1173.035299] libceph: con_keepalive ffff8802214c4410 <7>[ 1173.035305] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 1173.035309] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 1173.035313] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 1173.035315] libceph: __send_subscribe sub_sent=0 exp=0 want_osd=0 <7>[ 1173.035318] libceph: __schedule_delayed after 2000 <7>[ 1173.035325] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1173.035328] libceph: try_read tag 1 in_base_pos 0 <7>[ 1173.035334] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 1173.035337] libceph: try_write start ffff8802214c4410 state 5 <7>[ 1173.035339] libceph: try_write out_kvec_bytes 0 <7>[ 1173.035341] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 1173.035344] libceph: try_write out_kvec_bytes 1 <7>[ 1173.035346] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 1173.035355] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1173.035358] libceph: try_write nothing else to write. <7>[ 1173.035360] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 1182.019682] libceph: osds timeout <7>[ 1182.019688] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1193.040549] libceph: monc delayed_work <7>[ 1193.040555] libceph: con_keepalive ffff8802214c4410 <7>[ 1193.040560] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 1193.040565] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 1193.040568] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 1193.040571] libceph: __send_subscribe sub_sent=0 exp=0 want_osd=0 <7>[ 1193.040573] libceph: __schedule_delayed after 2000 <7>[ 1193.040579] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1193.040582] libceph: try_read tag 1 in_base_pos 0 <7>[ 1193.040588] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 1193.040591] libceph: try_write start ffff8802214c4410 state 5 <7>[ 1193.040593] libceph: try_write out_kvec_bytes 0 <7>[ 1193.040596] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 1193.040598] libceph: try_write out_kvec_bytes 1 <7>[ 1193.040601] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 1193.040610] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1193.040612] libceph: try_write nothing else to write. <7>[ 1193.040615] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 1196.993673] libceph: osds timeout <7>[ 1196.993679] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1211.967665] libceph: osds timeout <7>[ 1211.967671] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1213.045806] libceph: monc delayed_work <7>[ 1213.045812] libceph: con_keepalive ffff8802214c4410 <7>[ 1213.045818] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 1213.045822] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 1213.045826] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 1213.045828] libceph: __send_subscribe sub_sent=0 exp=1 want_osd=0 <7>[ 1213.045834] libceph: ----- ffff88020d5b5400 to mon1 15=mon_subscribe len 31+0+0 ----- <7>[ 1213.045837] libceph: __schedule_delayed after 1000 <7>[ 1213.045843] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1213.045846] libceph: try_read tag 1 in_base_pos 0 <7>[ 1213.045852] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 1213.045855] libceph: try_write start ffff8802214c4410 state 5 <7>[ 1213.045857] libceph: try_write out_kvec_bytes 0 <7>[ 1213.045861] libceph: prepare_write_message ffff88020d5b5400 seq 10 type 15 len 31+0+0 0 pgs <7>[ 1213.045865] libceph: prepare_write_message front_crc 2650579135 middle_crc 0 <7>[ 1213.045868] libceph: prepare_write_message_footer ffff8802214c4410 <7>[ 1213.045870] libceph: try_write out_kvec_bytes 98 <7>[ 1213.045873] libceph: write_partial_kvec ffff8802214c4410 98 left <7>[ 1213.045891] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1213.045894] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 1213.045896] libceph: try_write out_kvec_bytes 1 <7>[ 1213.045899] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 1213.045905] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1213.045907] libceph: try_write nothing else to write. <7>[ 1213.045910] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 1213.046351] libceph: ceph_sock_data_ready on ffff8802214c4410 state = 5, queueing work <7>[ 1213.046362] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 1213.046377] libceph: ceph_sock_data_ready on ffff8802214c4410 state = 5, queueing work <7>[ 1213.046380] libceph: queue_con_delay ffff8802214c4410 - already queued <7>[ 1213.046396] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1213.046399] libceph: try_read tag 1 in_base_pos 0 <7>[ 1213.046406] libceph: try_read got tag 8 <7>[ 1213.046409] libceph: prepare_read_ack ffff8802214c4410 <7>[ 1213.046414] libceph: got ack for seq 10 type 15 at ffff88020d5b5400 <7>[ 1213.046416] libceph: prepare_read_tag ffff8802214c4410 <7>[ 1213.046419] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1213.046421] libceph: try_read tag 1 in_base_pos 0 <7>[ 1213.046425] libceph: try_read got tag 7 <7>[ 1213.046428] libceph: prepare_read_message ffff8802214c4410 <7>[ 1213.046430] libceph: read_partial_message con ffff8802214c4410 msg (null) <7>[ 1213.046436] libceph: got hdr type 4 front 481 data 0 <7>[ 1213.046440] libceph: ceph_msg_new ffff88020d35db00 front 481 <7>[ 1213.046450] libceph: read_partial_message got msg ffff88020d35db00 481 (2132872380) + 0 (0) + 0 (0) <7>[ 1213.046455] libceph: ===== ffff88020d35db00 18 from mon1 4=mon_map len 481+0 (2132872380 0 0) ===== <7>[ 1213.046458] libceph: have_debugfs_info fsid 1 globalid 4132 <7>[ 1213.046461] libceph: handle_monmap <7>[ 1213.046463] libceph: monmap_decode ffff88020d750a04 ffff88020d750be1 len 477 <7>[ 1213.046466] libceph: monmap_decode epoch 1, num_mon 3 <7>[ 1213.046470] libceph: monmap_decode mon0 is 10.214.133.28:6789 <7>[ 1213.046473] libceph: monmap_decode mon1 is 10.214.133.30:6789 <7>[ 1213.046476] libceph: monmap_decode mon2 is 10.214.133.28:6790 <7>[ 1213.046480] libceph: ceph_msg_put last one on ffff88020d35db00 <7>[ 1213.046483] libceph: msg_kfree ffff88020d35db00 <7>[ 1213.046486] libceph: prepare_read_tag ffff8802214c4410 <7>[ 1213.046489] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1213.046491] libceph: try_read tag 1 in_base_pos 0 <7>[ 1213.046496] libceph: try_read got tag 7 <7>[ 1213.046498] libceph: prepare_read_message ffff8802214c4410 <7>[ 1213.046501] libceph: read_partial_message con ffff8802214c4410 msg (null) <7>[ 1213.046506] libceph: got hdr type 16 front 20 data 0 <7>[ 1213.046524] libceph: read_partial_message got msg ffff88020d5b4500 20 (473278946) + 0 (0) + 0 (0) <7>[ 1213.046529] libceph: ===== ffff88020d5b4500 19 from mon1 16=mon_subscribe_ack len 20+0 (473278946 0 0) ===== <7>[ 1213.046532] libceph: handle_subscribe_ack after 300 seconds <7>[ 1213.046536] libceph: prepare_read_tag ffff8802214c4410 <7>[ 1213.046538] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1213.046541] libceph: try_read tag 1 in_base_pos 0 <7>[ 1213.046546] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 1213.046548] libceph: try_write start ffff8802214c4410 state 5 <7>[ 1213.046551] libceph: try_write out_kvec_bytes 0 <7>[ 1213.046554] libceph: prepare_write_ack ffff8802214c4410 17 -> 19 <7>[ 1213.046556] libceph: try_write out_kvec_bytes 9 <7>[ 1213.046559] libceph: write_partial_kvec ffff8802214c4410 9 left <7>[ 1213.046566] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1213.046568] libceph: try_write nothing else to write. <7>[ 1213.046571] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 1215.342339] libceph: ceph_sock_state_change ffff88020c7c4030 state = 5 sk_state = 8 <7>[ 1215.342344] libceph: ceph_sock_state_change TCP_CLOSE_WAIT <7>[ 1215.342348] libceph: con_sock_state_closing con ffff88020c7c4030 sock 3 -> 4 <7>[ 1215.342351] libceph: get_osd ffff88020c7c4000 1 -> 2 <7>[ 1215.342360] libceph: queue_con_delay ffff88020c7c4030 0 <4>[ 1215.342380] libceph: osd4 10.214.133.30:6803 socket closed (con state OPEN) <7>[ 1215.349349] libceph: fault ffff88020c7c4030 state 5 to peer 10.214.133.30:6803 <7>[ 1215.349352] libceph: con_close_socket on ffff88020c7c4030 sock ffff88020cfc0800 <7>[ 1215.349369] libceph: ceph_sock_state_change ffff88020c7c4030 state = 5 sk_state = 9 <7>[ 1215.349395] libceph: con_sock_state_closed con ffff88020c7c4030 sock 4 -> 1 <7>[ 1215.349398] libceph: fault on LOSSYTX channel, marking CLOSED <7>[ 1215.349401] libceph: osd_reset osd4 <7>[ 1215.349405] libceph: __kick_osd_requests osd4 <7>[ 1215.349407] libceph: __reset_osd ffff88020c7c4000 osd4 <7>[ 1215.349420] libceph: con_close ffff88020c7c4030 peer 10.214.133.30:6803 <7>[ 1215.349423] libceph: reset_connection ffff88020c7c4030 <7>[ 1215.349426] libceph: con_close_socket on ffff88020c7c4030 sock (null) <7>[ 1215.349429] libceph: con_sock_state_closed con ffff88020c7c4030 sock 1 -> 1 <7>[ 1215.349434] libceph: con_open ffff88020c7c4030 10.214.133.30:6803 <7>[ 1215.349438] libceph: get_osd ffff88020c7c4000 2 -> 3 <7>[ 1215.349444] libceph: queue_con_delay ffff88020c7c4030 0 <7>[ 1215.349447] libceph: __register_request ffff880224d03000 tid 950 <7>[ 1215.349450] libceph: first request, scheduling timeout <7>[ 1215.349456] libceph: __unregister_linger_request ffff880224d03000 <7>[ 1215.349459] libceph: requeued lingering ffff880224d03000 tid 950 osd4 <7>[ 1215.349462] libceph: send_queued <7>[ 1215.349465] libceph: send_request ffff880224d03000 tid 950 to osd4 flags 36 <7>[ 1215.349469] libceph: get_osd ffff88020c7c4000 3 -> 4 <7>[ 1215.349473] libceph: ----- ffff88020b761d00 to osd4 42=osd_op len 130+0+0 ----- <7>[ 1215.349477] libceph: get_osd ffff88020c7c4000 4 -> 5 <7>[ 1215.349480] libceph: queue_con_delay ffff88020c7c4030 - already queued <7>[ 1215.349482] libceph: put_osd ffff88020c7c4000 5 -> 4 <7>[ 1215.349486] libceph: put_osd ffff88020c7c4000 4 -> 3 <7>[ 1215.349490] libceph: con_work OPENING <7>[ 1215.349492] libceph: try_read start on ffff88020c7c4030 state 2 <7>[ 1215.349495] libceph: try_write start ffff88020c7c4030 state 2 <7>[ 1215.349498] libceph: try_write out_kvec_bytes 0 <7>[ 1215.349500] libceph: prepare_read_banner ffff88020c7c4030 <7>[ 1215.349529] libceph: try_write initiating connect on ffff88020c7c4030 new state 3 <7>[ 1215.349543] libceph: connect 10.214.133.30:6803 <7>[ 1215.349546] libceph: con_sock_state_connecting con ffff88020c7c4030 sock 1 -> 2 <7>[ 1215.349570] libceph: connect 10.214.133.30:6803 EINPROGRESS sk_state = 2 <7>[ 1215.349574] libceph: write_partial_kvec ffff88020c7c4030 145 left <7>[ 1215.349579] libceph: write_partial_kvec ffff88020c7c4030 145 left in 2 kvecs ret = 0 <7>[ 1215.349582] libceph: try_write done on ffff88020c7c4030 ret 0 <7>[ 1215.349585] libceph: put_osd ffff88020c7c4000 3 -> 2 <7>[ 1215.349738] libceph: ceph_sock_state_change ffff88020c7c4030 state = 3 sk_state = 1 <7>[ 1215.349743] libceph: ceph_sock_state_change TCP_ESTABLISHED <7>[ 1215.349746] libceph: con_sock_state_connected con ffff88020c7c4030 sock 2 -> 3 <7>[ 1215.349750] libceph: get_osd ffff88020c7c4000 2 -> 3 <7>[ 1215.349758] libceph: queue_con_delay ffff88020c7c4030 0 <7>[ 1215.349788] libceph: try_read start on ffff88020c7c4030 state 3 <7>[ 1215.349791] libceph: try_read tag 1 in_base_pos 0 <7>[ 1215.349793] libceph: try_read connecting <7>[ 1215.349796] libceph: read_partial_banner ffff88020c7c4030 at 0 <7>[ 1215.349802] libceph: try_read done on ffff88020c7c4030 ret 0 <7>[ 1215.349805] libceph: try_write start ffff88020c7c4030 state 3 <7>[ 1215.349807] libceph: try_write out_kvec_bytes 145 <7>[ 1215.349810] libceph: write_partial_kvec ffff88020c7c4030 145 left <7>[ 1215.349825] libceph: write_partial_kvec ffff88020c7c4030 0 left in 0 kvecs ret = 1 <7>[ 1215.349828] libceph: try_write nothing else to write. <7>[ 1215.349830] libceph: try_write done on ffff88020c7c4030 ret 0 <7>[ 1215.349834] libceph: put_osd ffff88020c7c4000 3 -> 2 <7>[ 1215.350107] libceph: ceph_sock_data_ready on ffff88020c7c4030 state = 3, queueing work <7>[ 1215.350111] libceph: get_osd ffff88020c7c4000 2 -> 3 <7>[ 1215.350118] libceph: queue_con_delay ffff88020c7c4030 0 <7>[ 1215.350139] libceph: ceph_sock_data_ready on ffff88020c7c4030 state = 3, queueing work <7>[ 1215.350142] libceph: get_osd ffff88020c7c4000 3 -> 4 <7>[ 1215.350145] libceph: queue_con_delay ffff88020c7c4030 - already queued <7>[ 1215.350147] libceph: put_osd ffff88020c7c4000 4 -> 3 <7>[ 1215.350158] libceph: try_read start on ffff88020c7c4030 state 3 <7>[ 1215.350161] libceph: try_read tag 1 in_base_pos 0 <7>[ 1215.350164] libceph: try_read connecting <7>[ 1215.350166] libceph: read_partial_banner ffff88020c7c4030 at 0 <7>[ 1215.350178] libceph: process_banner on ffff88020c7c4030 <7>[ 1215.350183] libceph: prepare_write_connect ffff88020c7c4030 cseq=0 gseq=8 proto=24 <7>[ 1215.350187] libceph: prepare_read_connect ffff88020c7c4030 <7>[ 1215.350190] libceph: try_read done on ffff88020c7c4030 ret 0 <7>[ 1215.350192] libceph: try_write start ffff88020c7c4030 state 4 <7>[ 1215.350195] libceph: try_write out_kvec_bytes 255 <7>[ 1215.350198] libceph: write_partial_kvec ffff88020c7c4030 255 left <7>[ 1215.350213] libceph: write_partial_kvec ffff88020c7c4030 0 left in 0 kvecs ret = 1 <7>[ 1215.350215] libceph: try_write nothing else to write. <7>[ 1215.350218] libceph: try_write done on ffff88020c7c4030 ret 0 <7>[ 1215.350221] libceph: put_osd ffff88020c7c4000 3 -> 2 <7>[ 1215.350533] libceph: ceph_sock_data_ready on ffff88020c7c4030 state = 4, queueing work <7>[ 1215.350537] libceph: get_osd ffff88020c7c4000 2 -> 3 <7>[ 1215.350544] libceph: queue_con_delay ffff88020c7c4030 0 <7>[ 1215.350559] libceph: ceph_sock_data_ready on ffff88020c7c4030 state = 4, queueing work <7>[ 1215.350563] libceph: get_osd ffff88020c7c4000 3 -> 4 <7>[ 1215.350565] libceph: queue_con_delay ffff88020c7c4030 - already queued <7>[ 1215.350568] libceph: put_osd ffff88020c7c4000 4 -> 3 <7>[ 1215.350579] libceph: try_read start on ffff88020c7c4030 state 4 <7>[ 1215.350582] libceph: try_read tag 1 in_base_pos 0 <7>[ 1215.350584] libceph: try_read negotiating <7>[ 1215.350587] libceph: read_partial_connect ffff88020c7c4030 at 0 <7>[ 1215.350604] libceph: read_partial_connect ffff88020c7c4030 tag 1, con_seq = 1, g_seq = 6 <7>[ 1215.350607] libceph: process_connect on ffff88020c7c4030 tag 1 <7>[ 1215.350610] libceph: process_connect got READY gseq 6 cseq 1 (1) <7>[ 1215.350612] libceph: prepare_read_tag ffff88020c7c4030 <7>[ 1215.350615] libceph: try_read start on ffff88020c7c4030 state 5 <7>[ 1215.350617] libceph: try_read tag 1 in_base_pos 0 <7>[ 1215.350622] libceph: try_read done on ffff88020c7c4030 ret 0 <7>[ 1215.350625] libceph: try_write start ffff88020c7c4030 state 5 <7>[ 1215.350627] libceph: try_write out_kvec_bytes 0 <7>[ 1215.350631] libceph: prepare_write_message ffff88020b761d00 seq 1 type 42 len 130+0+0 0 pgs <7>[ 1215.350635] libceph: prepare_write_message front_crc 2251967416 middle_crc 0 <7>[ 1215.350637] libceph: prepare_write_message_footer ffff88020c7c4030 <7>[ 1215.350640] libceph: try_write out_kvec_bytes 197 <7>[ 1215.350642] libceph: write_partial_kvec ffff88020c7c4030 197 left <7>[ 1215.350655] libceph: write_partial_kvec ffff88020c7c4030 0 left in 0 kvecs ret = 1 <7>[ 1215.350657] libceph: try_write nothing else to write. <7>[ 1215.350660] libceph: try_write done on ffff88020c7c4030 ret 0 <7>[ 1215.350663] libceph: put_osd ffff88020c7c4000 3 -> 2 <7>[ 1215.351374] libceph: ceph_sock_data_ready on ffff88020c7c4030 state = 5, queueing work <7>[ 1215.351378] libceph: get_osd ffff88020c7c4000 2 -> 3 <7>[ 1215.351385] libceph: queue_con_delay ffff88020c7c4030 0 <7>[ 1215.351397] libceph: try_read start on ffff88020c7c4030 state 5 <7>[ 1215.351400] libceph: try_read tag 1 in_base_pos 0 <7>[ 1215.351407] libceph: try_read got tag 8 <7>[ 1215.351410] libceph: prepare_read_ack ffff88020c7c4030 <7>[ 1215.351416] libceph: got ack for seq 1 type 42 at ffff88020b761d00 <7>[ 1215.351419] libceph: put_osd ffff88020c7c4000 3 -> 2 <7>[ 1215.351421] libceph: prepare_read_tag ffff88020c7c4030 <7>[ 1215.351424] libceph: try_read start on ffff88020c7c4030 state 5 <7>[ 1215.351426] libceph: try_read tag 1 in_base_pos 0 <7>[ 1215.351431] libceph: try_read got tag 7 <7>[ 1215.351433] libceph: prepare_read_message ffff88020c7c4030 <7>[ 1215.351436] libceph: read_partial_message con ffff88020c7c4030 msg (null) <7>[ 1215.351441] libceph: got hdr type 43 front 108 data 0 <7>[ 1215.351446] libceph: get_osd ffff88020c7c4000 2 -> 3 <7>[ 1215.351448] libceph: get_reply tid 950 ffff88020b761a00 <7>[ 1215.351452] libceph: get_osd ffff88020c7c4000 3 -> 4 <7>[ 1215.351462] libceph: read_partial_message got msg ffff88020b761a00 108 (2795223248) + 0 (0) + 0 (0) <7>[ 1215.351465] libceph: put_osd ffff88020c7c4000 4 -> 3 <7>[ 1215.351470] libceph: ===== ffff88020b761a00 1 from osd4 43=osd_opreply len 108+0 (2795223248 0 0) ===== <7>[ 1215.351473] libceph: handle_reply ffff88020b761a00 tid 950 result 0 <7>[ 1215.351476] libceph: dropping con_filling_msg ref ffff88020c7c4030 <7>[ 1215.351479] libceph: put_osd ffff88020c7c4000 3 -> 2 <7>[ 1215.351481] libceph: handle_reply tid 950 flags 37 <7>[ 1215.351484] libceph: __register_linger_request ffff880224d03000 <7>[ 1215.351487] libceph: __unregister_request ffff880224d03000 tid 950 <7>[ 1215.351490] libceph: no requests, canceling timeout <7>[ 1215.351495] libceph: req=ffff880224d03000 req->r_linger=1 <7>[ 1215.351498] libceph: prepare_read_tag ffff88020c7c4030 <7>[ 1215.351500] libceph: try_read start on ffff88020c7c4030 state 5 <7>[ 1215.351503] libceph: try_read tag 1 in_base_pos 0 <7>[ 1215.351508] libceph: try_read done on ffff88020c7c4030 ret 0 <7>[ 1215.351510] libceph: try_write start ffff88020c7c4030 state 5 <7>[ 1215.351513] libceph: try_write out_kvec_bytes 0 <7>[ 1215.351516] libceph: prepare_write_ack ffff88020c7c4030 0 -> 1 <7>[ 1215.351518] libceph: try_write out_kvec_bytes 9 <7>[ 1215.351521] libceph: write_partial_kvec ffff88020c7c4030 9 left <7>[ 1215.351528] libceph: write_partial_kvec ffff88020c7c4030 0 left in 0 kvecs ret = 1 <7>[ 1215.351530] libceph: try_write nothing else to write. <7>[ 1215.351533] libceph: try_write done on ffff88020c7c4030 ret 0 <7>[ 1215.351536] libceph: put_osd ffff88020c7c4000 2 -> 1 <7>[ 1223.048421] libceph: monc delayed_work <7>[ 1223.048428] libceph: con_keepalive ffff8802214c4410 <7>[ 1223.048434] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 1223.048439] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 1223.048442] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 1223.048445] libceph: __send_subscribe sub_sent=0 exp=0 want_osd=0 <7>[ 1223.048447] libceph: __schedule_delayed after 2000 <7>[ 1223.048454] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1223.048457] libceph: try_read tag 1 in_base_pos 0 <7>[ 1223.048463] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 1223.048466] libceph: try_write start ffff8802214c4410 state 5 <7>[ 1223.048468] libceph: try_write out_kvec_bytes 0 <7>[ 1223.048471] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 1223.048473] libceph: try_write out_kvec_bytes 1 <7>[ 1223.048476] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 1223.048485] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1223.048487] libceph: try_write nothing else to write. <7>[ 1223.048490] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 1226.941655] libceph: osds timeout <7>[ 1226.941661] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1241.915642] libceph: osds timeout <7>[ 1241.915648] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1243.073634] libceph: monc delayed_work <7>[ 1243.073640] libceph: con_keepalive ffff8802214c4410 <7>[ 1243.073646] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 1243.073651] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 1243.073654] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 1243.073657] libceph: __send_subscribe sub_sent=0 exp=0 want_osd=0 <7>[ 1243.073659] libceph: __schedule_delayed after 2000 <7>[ 1243.073666] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1243.073669] libceph: try_read tag 1 in_base_pos 0 <7>[ 1243.073676] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 1243.073678] libceph: try_write start ffff8802214c4410 state 5 <7>[ 1243.073681] libceph: try_write out_kvec_bytes 0 <7>[ 1243.073683] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 1243.073685] libceph: try_write out_kvec_bytes 1 <7>[ 1243.073688] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 1243.073697] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1243.073699] libceph: try_write nothing else to write. <7>[ 1243.073702] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 1256.889639] libceph: osds timeout <7>[ 1256.889645] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1263.078887] libceph: monc delayed_work <7>[ 1263.078893] libceph: con_keepalive ffff8802214c4410 <7>[ 1263.078900] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 1263.078904] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 1263.078907] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 1263.078910] libceph: __send_subscribe sub_sent=0 exp=0 want_osd=0 <7>[ 1263.078913] libceph: __schedule_delayed after 2000 <7>[ 1263.078919] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1263.078922] libceph: try_read tag 1 in_base_pos 0 <7>[ 1263.078929] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 1263.078932] libceph: try_write start ffff8802214c4410 state 5 <7>[ 1263.078934] libceph: try_write out_kvec_bytes 0 <7>[ 1263.078936] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 1263.078939] libceph: try_write out_kvec_bytes 1 <7>[ 1263.078941] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 1263.078950] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1263.078953] libceph: try_write nothing else to write. <7>[ 1263.078955] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 1271.863630] libceph: osds timeout <7>[ 1271.863635] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1283.084116] libceph: monc delayed_work <7>[ 1283.084122] libceph: con_keepalive ffff8802214c4410 <7>[ 1283.084129] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 1283.084133] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 1283.084136] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 1283.084139] libceph: __send_subscribe sub_sent=0 exp=0 want_osd=0 <7>[ 1283.084142] libceph: __schedule_delayed after 2000 <7>[ 1283.084148] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1283.084151] libceph: try_read tag 1 in_base_pos 0 <7>[ 1283.084158] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 1283.084161] libceph: try_write start ffff8802214c4410 state 5 <7>[ 1283.084163] libceph: try_write out_kvec_bytes 0 <7>[ 1283.084165] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 1283.084168] libceph: try_write out_kvec_bytes 1 <7>[ 1283.084170] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 1283.084179] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1283.084182] libceph: try_write nothing else to write. <7>[ 1283.084184] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 1286.837611] libceph: osds timeout <7>[ 1286.837617] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1301.811599] libceph: osds timeout <7>[ 1301.811605] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1303.089394] libceph: monc delayed_work <7>[ 1303.089400] libceph: con_keepalive ffff8802214c4410 <7>[ 1303.089406] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 1303.089410] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 1303.089414] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 1303.089416] libceph: __send_subscribe sub_sent=0 exp=0 want_osd=0 <7>[ 1303.089419] libceph: __schedule_delayed after 2000 <7>[ 1303.089425] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1303.089428] libceph: try_read tag 1 in_base_pos 0 <7>[ 1303.089434] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 1303.089437] libceph: try_write start ffff8802214c4410 state 5 <7>[ 1303.089440] libceph: try_write out_kvec_bytes 0 <7>[ 1303.089442] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 1303.089444] libceph: try_write out_kvec_bytes 1 <7>[ 1303.089447] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 1303.089457] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1303.089459] libceph: try_write nothing else to write. <7>[ 1303.089462] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 1316.785588] libceph: osds timeout <7>[ 1316.785594] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1323.094636] libceph: monc delayed_work <7>[ 1323.094642] libceph: con_keepalive ffff8802214c4410 <7>[ 1323.094648] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 1323.094653] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 1323.094656] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 1323.094659] libceph: __send_subscribe sub_sent=0 exp=0 want_osd=0 <7>[ 1323.094661] libceph: __schedule_delayed after 2000 <7>[ 1323.094668] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1323.094671] libceph: try_read tag 1 in_base_pos 0 <7>[ 1323.094678] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 1323.094680] libceph: try_write start ffff8802214c4410 state 5 <7>[ 1323.094683] libceph: try_write out_kvec_bytes 0 <7>[ 1323.094685] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 1323.094687] libceph: try_write out_kvec_bytes 1 <7>[ 1323.094690] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 1323.094699] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1323.094701] libceph: try_write nothing else to write. <7>[ 1323.094704] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 1331.759576] libceph: osds timeout <7>[ 1331.759582] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1343.099883] libceph: monc delayed_work <7>[ 1343.099889] libceph: con_keepalive ffff8802214c4410 <7>[ 1343.099895] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 1343.099900] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 1343.099903] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 1343.099906] libceph: __send_subscribe sub_sent=0 exp=0 want_osd=0 <7>[ 1343.099908] libceph: __schedule_delayed after 2000 <7>[ 1343.099915] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1343.099918] libceph: try_read tag 1 in_base_pos 0 <7>[ 1343.099924] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 1343.099927] libceph: try_write start ffff8802214c4410 state 5 <7>[ 1343.099929] libceph: try_write out_kvec_bytes 0 <7>[ 1343.099932] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 1343.099934] libceph: try_write out_kvec_bytes 1 <7>[ 1343.099937] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 1343.099946] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1343.099948] libceph: try_write nothing else to write. <7>[ 1343.099951] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 1346.733568] libceph: osds timeout <7>[ 1346.733574] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1361.707556] libceph: osds timeout <7>[ 1361.707562] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1363.105134] libceph: monc delayed_work <7>[ 1363.105141] libceph: con_keepalive ffff8802214c4410 <7>[ 1363.105147] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 1363.105151] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 1363.105154] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 1363.105158] libceph: __send_subscribe sub_sent=0 exp=1 want_osd=0 <7>[ 1363.105163] libceph: ----- ffff88020d5b5400 to mon1 15=mon_subscribe len 31+0+0 ----- <7>[ 1363.105166] libceph: __schedule_delayed after 1000 <7>[ 1363.105173] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1363.105176] libceph: try_read tag 1 in_base_pos 0 <7>[ 1363.105182] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 1363.105185] libceph: try_write start ffff8802214c4410 state 5 <7>[ 1363.105187] libceph: try_write out_kvec_bytes 0 <7>[ 1363.105191] libceph: prepare_write_message ffff88020d5b5400 seq 11 type 15 len 31+0+0 0 pgs <7>[ 1363.105196] libceph: prepare_write_message front_crc 2650579135 middle_crc 0 <7>[ 1363.105198] libceph: prepare_write_message_footer ffff8802214c4410 <7>[ 1363.105201] libceph: try_write out_kvec_bytes 98 <7>[ 1363.105203] libceph: write_partial_kvec ffff8802214c4410 98 left <7>[ 1363.105222] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1363.105225] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 1363.105228] libceph: try_write out_kvec_bytes 1 <7>[ 1363.105230] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 1363.105236] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1363.105238] libceph: try_write nothing else to write. <7>[ 1363.105241] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 1363.105668] libceph: ceph_sock_data_ready on ffff8802214c4410 state = 5, queueing work <7>[ 1363.105679] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 1363.105696] libceph: ceph_sock_data_ready on ffff8802214c4410 state = 5, queueing work <7>[ 1363.105700] libceph: queue_con_delay ffff8802214c4410 - already queued <7>[ 1363.105716] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1363.105719] libceph: try_read tag 1 in_base_pos 0 <7>[ 1363.105726] libceph: try_read got tag 8 <7>[ 1363.105729] libceph: prepare_read_ack ffff8802214c4410 <7>[ 1363.105734] libceph: got ack for seq 11 type 15 at ffff88020d5b5400 <7>[ 1363.105736] libceph: prepare_read_tag ffff8802214c4410 <7>[ 1363.105739] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1363.105741] libceph: try_read tag 1 in_base_pos 0 <7>[ 1363.105745] libceph: try_read got tag 7 <7>[ 1363.105748] libceph: prepare_read_message ffff8802214c4410 <7>[ 1363.105751] libceph: read_partial_message con ffff8802214c4410 msg (null) <7>[ 1363.105756] libceph: got hdr type 4 front 481 data 0 <7>[ 1363.105761] libceph: ceph_msg_new ffff88020d35c600 front 481 <7>[ 1363.105771] libceph: read_partial_message got msg ffff88020d35c600 481 (2132872380) + 0 (0) + 0 (0) <7>[ 1363.105776] libceph: ===== ffff88020d35c600 20 from mon1 4=mon_map len 481+0 (2132872380 0 0) ===== <7>[ 1363.105779] libceph: have_debugfs_info fsid 1 globalid 4132 <7>[ 1363.105782] libceph: handle_monmap <7>[ 1363.105785] libceph: monmap_decode ffff88020d753c04 ffff88020d753de1 len 477 <7>[ 1363.105787] libceph: monmap_decode epoch 1, num_mon 3 <7>[ 1363.105791] libceph: monmap_decode mon0 is 10.214.133.28:6789 <7>[ 1363.105794] libceph: monmap_decode mon1 is 10.214.133.30:6789 <7>[ 1363.105797] libceph: monmap_decode mon2 is 10.214.133.28:6790 <7>[ 1363.105801] libceph: ceph_msg_put last one on ffff88020d35c600 <7>[ 1363.105804] libceph: msg_kfree ffff88020d35c600 <7>[ 1363.105807] libceph: prepare_read_tag ffff8802214c4410 <7>[ 1363.105810] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1363.105812] libceph: try_read tag 1 in_base_pos 0 <7>[ 1363.105817] libceph: try_read got tag 7 <7>[ 1363.105819] libceph: prepare_read_message ffff8802214c4410 <7>[ 1363.105822] libceph: read_partial_message con ffff8802214c4410 msg (null) <7>[ 1363.105827] libceph: got hdr type 16 front 20 data 0 <7>[ 1363.105846] libceph: read_partial_message got msg ffff88020d5b4500 20 (473278946) + 0 (0) + 0 (0) <7>[ 1363.105851] libceph: ===== ffff88020d5b4500 21 from mon1 16=mon_subscribe_ack len 20+0 (473278946 0 0) ===== <7>[ 1363.105854] libceph: handle_subscribe_ack after 300 seconds <7>[ 1363.105858] libceph: prepare_read_tag ffff8802214c4410 <7>[ 1363.105860] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1363.105863] libceph: try_read tag 1 in_base_pos 0 <7>[ 1363.105868] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 1363.105870] libceph: try_write start ffff8802214c4410 state 5 <7>[ 1363.105872] libceph: try_write out_kvec_bytes 0 <7>[ 1363.105876] libceph: prepare_write_ack ffff8802214c4410 19 -> 21 <7>[ 1363.105878] libceph: try_write out_kvec_bytes 9 <7>[ 1363.105881] libceph: write_partial_kvec ffff8802214c4410 9 left <7>[ 1363.105889] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1363.105891] libceph: try_write nothing else to write. <7>[ 1363.105894] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 1373.107762] libceph: monc delayed_work <7>[ 1373.107768] libceph: con_keepalive ffff8802214c4410 <7>[ 1373.107774] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 1373.107778] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 1373.107781] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 1373.107784] libceph: __send_subscribe sub_sent=0 exp=0 want_osd=0 <7>[ 1373.107787] libceph: __schedule_delayed after 2000 <7>[ 1373.107793] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1373.107796] libceph: try_read tag 1 in_base_pos 0 <7>[ 1373.107802] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 1373.107805] libceph: try_write start ffff8802214c4410 state 5 <7>[ 1373.107807] libceph: try_write out_kvec_bytes 0 <7>[ 1373.107810] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 1373.107812] libceph: try_write out_kvec_bytes 1 <7>[ 1373.107815] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 1373.107824] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1373.107826] libceph: try_write nothing else to write. <7>[ 1373.107829] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 1376.681542] libceph: osds timeout <7>[ 1376.681548] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1391.655531] libceph: osds timeout <7>[ 1391.655536] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1393.132975] libceph: monc delayed_work <7>[ 1393.132981] libceph: con_keepalive ffff8802214c4410 <7>[ 1393.132987] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 1393.132991] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 1393.132994] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 1393.132997] libceph: __send_subscribe sub_sent=0 exp=0 want_osd=0 <7>[ 1393.133000] libceph: __schedule_delayed after 2000 <7>[ 1393.133006] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1393.133008] libceph: try_read tag 1 in_base_pos 0 <7>[ 1393.133015] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 1393.133017] libceph: try_write start ffff8802214c4410 state 5 <7>[ 1393.133020] libceph: try_write out_kvec_bytes 0 <7>[ 1393.133022] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 1393.133024] libceph: try_write out_kvec_bytes 1 <7>[ 1393.133027] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 1393.133036] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1393.133039] libceph: try_write nothing else to write. <7>[ 1393.133041] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 1406.629518] libceph: osds timeout <7>[ 1406.629523] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1413.138223] libceph: monc delayed_work <7>[ 1413.138229] libceph: con_keepalive ffff8802214c4410 <7>[ 1413.138235] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 1413.138239] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 1413.138242] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 1413.138245] libceph: __send_subscribe sub_sent=0 exp=0 want_osd=0 <7>[ 1413.138248] libceph: __schedule_delayed after 2000 <7>[ 1413.138254] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1413.138256] libceph: try_read tag 1 in_base_pos 0 <7>[ 1413.138263] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 1413.138265] libceph: try_write start ffff8802214c4410 state 5 <7>[ 1413.138268] libceph: try_write out_kvec_bytes 0 <7>[ 1413.138270] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 1413.138273] libceph: try_write out_kvec_bytes 1 <7>[ 1413.138276] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 1413.138284] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1413.138287] libceph: try_write nothing else to write. <7>[ 1413.138289] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 1421.603508] libceph: osds timeout <7>[ 1421.603514] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1433.143455] libceph: monc delayed_work <7>[ 1433.143461] libceph: con_keepalive ffff8802214c4410 <7>[ 1433.143467] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 1433.143471] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 1433.143475] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 1433.143478] libceph: __send_subscribe sub_sent=0 exp=0 want_osd=0 <7>[ 1433.143481] libceph: __schedule_delayed after 2000 <7>[ 1433.143487] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1433.143490] libceph: try_read tag 1 in_base_pos 0 <7>[ 1433.143497] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 1433.143499] libceph: try_write start ffff8802214c4410 state 5 <7>[ 1433.143502] libceph: try_write out_kvec_bytes 0 <7>[ 1433.143504] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 1433.143506] libceph: try_write out_kvec_bytes 1 <7>[ 1433.143509] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 1433.143518] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1433.143521] libceph: try_write nothing else to write. <7>[ 1433.143523] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 1436.577494] libceph: osds timeout <7>[ 1436.577500] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1451.551482] libceph: osds timeout <7>[ 1451.551488] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1453.148708] libceph: monc delayed_work <7>[ 1453.148714] libceph: con_keepalive ffff8802214c4410 <7>[ 1453.148721] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 1453.148725] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 1453.148728] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 1453.148732] libceph: __send_subscribe sub_sent=0 exp=0 want_osd=0 <7>[ 1453.148734] libceph: __schedule_delayed after 2000 <7>[ 1453.148741] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1453.148744] libceph: try_read tag 1 in_base_pos 0 <7>[ 1453.148750] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 1453.148753] libceph: try_write start ffff8802214c4410 state 5 <7>[ 1453.148755] libceph: try_write out_kvec_bytes 0 <7>[ 1453.148758] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 1453.148760] libceph: try_write out_kvec_bytes 1 <7>[ 1453.148763] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 1453.148771] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1453.148774] libceph: try_write nothing else to write. <7>[ 1453.148776] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 1466.525471] libceph: osds timeout <7>[ 1466.525478] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1473.153963] libceph: monc delayed_work <7>[ 1473.153969] libceph: con_keepalive ffff8802214c4410 <7>[ 1473.153977] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 1473.153981] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 1473.153984] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 1473.153988] libceph: __send_subscribe sub_sent=0 exp=0 want_osd=0 <7>[ 1473.153991] libceph: __schedule_delayed after 2000 <7>[ 1473.153999] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1473.154001] libceph: try_read tag 1 in_base_pos 0 <7>[ 1473.154009] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 1473.154012] libceph: try_write start ffff8802214c4410 state 5 <7>[ 1473.154014] libceph: try_write out_kvec_bytes 0 <7>[ 1473.154017] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 1473.154019] libceph: try_write out_kvec_bytes 1 <7>[ 1473.154022] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 1473.154031] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1473.154034] libceph: try_write nothing else to write. <7>[ 1473.154036] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 1481.499461] libceph: osds timeout <7>[ 1481.499467] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1493.159218] libceph: monc delayed_work <7>[ 1493.159224] libceph: con_keepalive ffff8802214c4410 <7>[ 1493.159230] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 1493.159234] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 1493.159238] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 1493.159240] libceph: __send_subscribe sub_sent=0 exp=0 want_osd=0 <7>[ 1493.159243] libceph: __schedule_delayed after 2000 <7>[ 1493.159249] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1493.159252] libceph: try_read tag 1 in_base_pos 0 <7>[ 1493.159258] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 1493.159261] libceph: try_write start ffff8802214c4410 state 5 <7>[ 1493.159263] libceph: try_write out_kvec_bytes 0 <7>[ 1493.159266] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 1493.159268] libceph: try_write out_kvec_bytes 1 <7>[ 1493.159271] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 1493.159280] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1493.159283] libceph: try_write nothing else to write. <7>[ 1493.159285] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 1496.473444] libceph: osds timeout <7>[ 1496.473450] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1511.447432] libceph: osds timeout <7>[ 1511.447437] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1513.164455] libceph: monc delayed_work <7>[ 1513.164461] libceph: con_keepalive ffff8802214c4410 <7>[ 1513.164468] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 1513.164472] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 1513.164475] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 1513.164478] libceph: __send_subscribe sub_sent=0 exp=1 want_osd=0 <7>[ 1513.164484] libceph: ----- ffff88020d5b5400 to mon1 15=mon_subscribe len 31+0+0 ----- <7>[ 1513.164487] libceph: __schedule_delayed after 1000 <7>[ 1513.164494] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1513.164497] libceph: try_read tag 1 in_base_pos 0 <7>[ 1513.164504] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 1513.164507] libceph: try_write start ffff8802214c4410 state 5 <7>[ 1513.164509] libceph: try_write out_kvec_bytes 0 <7>[ 1513.164513] libceph: prepare_write_message ffff88020d5b5400 seq 12 type 15 len 31+0+0 0 pgs <7>[ 1513.164518] libceph: prepare_write_message front_crc 2650579135 middle_crc 0 <7>[ 1513.164521] libceph: prepare_write_message_footer ffff8802214c4410 <7>[ 1513.164523] libceph: try_write out_kvec_bytes 98 <7>[ 1513.164526] libceph: write_partial_kvec ffff8802214c4410 98 left <7>[ 1513.164545] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1513.164548] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 1513.164550] libceph: try_write out_kvec_bytes 1 <7>[ 1513.164553] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 1513.164558] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1513.164561] libceph: try_write nothing else to write. <7>[ 1513.164564] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 1513.165063] libceph: ceph_sock_data_ready on ffff8802214c4410 state = 5, queueing work <7>[ 1513.165074] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 1513.165091] libceph: ceph_sock_data_ready on ffff8802214c4410 state = 5, queueing work <7>[ 1513.165094] libceph: queue_con_delay ffff8802214c4410 - already queued <7>[ 1513.165111] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1513.165114] libceph: try_read tag 1 in_base_pos 0 <7>[ 1513.165121] libceph: try_read got tag 8 <7>[ 1513.165124] libceph: prepare_read_ack ffff8802214c4410 <7>[ 1513.165129] libceph: got ack for seq 12 type 15 at ffff88020d5b5400 <7>[ 1513.165132] libceph: prepare_read_tag ffff8802214c4410 <7>[ 1513.165134] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1513.165136] libceph: try_read tag 1 in_base_pos 0 <7>[ 1513.165141] libceph: try_read got tag 7 <7>[ 1513.165143] libceph: prepare_read_message ffff8802214c4410 <7>[ 1513.165146] libceph: read_partial_message con ffff8802214c4410 msg (null) <7>[ 1513.165152] libceph: got hdr type 4 front 481 data 0 <7>[ 1513.165156] libceph: ceph_msg_new ffff88020d35d000 front 481 <7>[ 1513.165167] libceph: read_partial_message got msg ffff88020d35d000 481 (2132872380) + 0 (0) + 0 (0) <7>[ 1513.165172] libceph: ===== ffff88020d35d000 22 from mon1 4=mon_map len 481+0 (2132872380 0 0) ===== <7>[ 1513.165176] libceph: have_debugfs_info fsid 1 globalid 4132 <7>[ 1513.165178] libceph: handle_monmap <7>[ 1513.165181] libceph: monmap_decode ffff88020d752204 ffff88020d7523e1 len 477 <7>[ 1513.165184] libceph: monmap_decode epoch 1, num_mon 3 <7>[ 1513.165188] libceph: monmap_decode mon0 is 10.214.133.28:6789 <7>[ 1513.165191] libceph: monmap_decode mon1 is 10.214.133.30:6789 <7>[ 1513.165194] libceph: monmap_decode mon2 is 10.214.133.28:6790 <7>[ 1513.165198] libceph: ceph_msg_put last one on ffff88020d35d000 <7>[ 1513.165201] libceph: msg_kfree ffff88020d35d000 <7>[ 1513.165205] libceph: prepare_read_tag ffff8802214c4410 <7>[ 1513.165207] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1513.165210] libceph: try_read tag 1 in_base_pos 0 <7>[ 1513.165214] libceph: try_read got tag 7 <7>[ 1513.165216] libceph: prepare_read_message ffff8802214c4410 <7>[ 1513.165219] libceph: read_partial_message con ffff8802214c4410 msg (null) <7>[ 1513.165224] libceph: got hdr type 16 front 20 data 0 <7>[ 1513.165243] libceph: read_partial_message got msg ffff88020d5b4500 20 (473278946) + 0 (0) + 0 (0) <7>[ 1513.165248] libceph: ===== ffff88020d5b4500 23 from mon1 16=mon_subscribe_ack len 20+0 (473278946 0 0) ===== <7>[ 1513.165252] libceph: handle_subscribe_ack after 300 seconds <7>[ 1513.165255] libceph: prepare_read_tag ffff8802214c4410 <7>[ 1513.165258] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1513.165260] libceph: try_read tag 1 in_base_pos 0 <7>[ 1513.165265] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 1513.165268] libceph: try_write start ffff8802214c4410 state 5 <7>[ 1513.165270] libceph: try_write out_kvec_bytes 0 <7>[ 1513.165273] libceph: prepare_write_ack ffff8802214c4410 21 -> 23 <7>[ 1513.165276] libceph: try_write out_kvec_bytes 9 <7>[ 1513.165278] libceph: write_partial_kvec ffff8802214c4410 9 left <7>[ 1513.165286] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1513.165288] libceph: try_write nothing else to write. <7>[ 1513.165290] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 1523.167079] libceph: monc delayed_work <7>[ 1523.167085] libceph: con_keepalive ffff8802214c4410 <7>[ 1523.167091] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 1523.167096] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 1523.167099] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 1523.167102] libceph: __send_subscribe sub_sent=0 exp=0 want_osd=0 <7>[ 1523.167105] libceph: __schedule_delayed after 2000 <7>[ 1523.167112] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1523.167114] libceph: try_read tag 1 in_base_pos 0 <7>[ 1523.167121] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 1523.167123] libceph: try_write start ffff8802214c4410 state 5 <7>[ 1523.167126] libceph: try_write out_kvec_bytes 0 <7>[ 1523.167128] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 1523.167130] libceph: try_write out_kvec_bytes 1 <7>[ 1523.167133] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 1523.167142] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1523.167144] libceph: try_write nothing else to write. <7>[ 1523.167147] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 1526.421415] libceph: osds timeout <7>[ 1526.421421] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1541.395410] libceph: osds timeout <7>[ 1541.395416] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1543.192292] libceph: monc delayed_work <7>[ 1543.192299] libceph: con_keepalive ffff8802214c4410 <7>[ 1543.192305] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 1543.192309] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 1543.192312] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 1543.192316] libceph: __send_subscribe sub_sent=0 exp=0 want_osd=0 <7>[ 1543.192318] libceph: __schedule_delayed after 2000 <7>[ 1543.192325] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1543.192328] libceph: try_read tag 1 in_base_pos 0 <7>[ 1543.192334] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 1543.192337] libceph: try_write start ffff8802214c4410 state 5 <7>[ 1543.192340] libceph: try_write out_kvec_bytes 0 <7>[ 1543.192342] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 1543.192344] libceph: try_write out_kvec_bytes 1 <7>[ 1543.192347] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 1543.192356] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1543.192358] libceph: try_write nothing else to write. <7>[ 1543.192361] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 1556.369396] libceph: osds timeout <7>[ 1556.369402] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1563.197542] libceph: monc delayed_work <7>[ 1563.197548] libceph: con_keepalive ffff8802214c4410 <7>[ 1563.197554] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 1563.197558] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 1563.197562] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 1563.197565] libceph: __send_subscribe sub_sent=0 exp=0 want_osd=0 <7>[ 1563.197567] libceph: __schedule_delayed after 2000 <7>[ 1563.197574] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1563.197577] libceph: try_read tag 1 in_base_pos 0 <7>[ 1563.197583] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 1563.197586] libceph: try_write start ffff8802214c4410 state 5 <7>[ 1563.197588] libceph: try_write out_kvec_bytes 0 <7>[ 1563.197591] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 1563.197593] libceph: try_write out_kvec_bytes 1 <7>[ 1563.197596] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 1563.197605] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1563.197607] libceph: try_write nothing else to write. <7>[ 1563.197610] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 1571.343383] libceph: osds timeout <7>[ 1571.343389] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1583.202779] libceph: monc delayed_work <7>[ 1583.202786] libceph: con_keepalive ffff8802214c4410 <7>[ 1583.202792] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 1583.202796] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 1583.202799] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 1583.202802] libceph: __send_subscribe sub_sent=0 exp=0 want_osd=0 <7>[ 1583.202805] libceph: __schedule_delayed after 2000 <7>[ 1583.202812] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1583.202814] libceph: try_read tag 1 in_base_pos 0 <7>[ 1583.202821] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 1583.202824] libceph: try_write start ffff8802214c4410 state 5 <7>[ 1583.202826] libceph: try_write out_kvec_bytes 0 <7>[ 1583.202829] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 1583.202831] libceph: try_write out_kvec_bytes 1 <7>[ 1583.202833] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 1583.202842] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1583.202845] libceph: try_write nothing else to write. <7>[ 1583.202847] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 1586.317378] libceph: osds timeout <7>[ 1586.317384] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1601.291358] libceph: osds timeout <7>[ 1601.291364] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1603.208033] libceph: monc delayed_work <7>[ 1603.208039] libceph: con_keepalive ffff8802214c4410 <7>[ 1603.208046] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 1603.208050] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 1603.208053] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 1603.208056] libceph: __send_subscribe sub_sent=0 exp=0 want_osd=0 <7>[ 1603.208059] libceph: __schedule_delayed after 2000 <7>[ 1603.208066] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1603.208068] libceph: try_read tag 1 in_base_pos 0 <7>[ 1603.208075] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 1603.208078] libceph: try_write start ffff8802214c4410 state 5 <7>[ 1603.208080] libceph: try_write out_kvec_bytes 0 <7>[ 1603.208082] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 1603.208085] libceph: try_write out_kvec_bytes 1 <7>[ 1603.208088] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 1603.208096] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1603.208099] libceph: try_write nothing else to write. <7>[ 1603.208101] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 1616.265352] libceph: osds timeout <7>[ 1616.265357] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1623.213283] libceph: monc delayed_work <7>[ 1623.213289] libceph: con_keepalive ffff8802214c4410 <7>[ 1623.213295] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 1623.213300] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 1623.213303] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 1623.213306] libceph: __send_subscribe sub_sent=0 exp=0 want_osd=0 <7>[ 1623.213309] libceph: __schedule_delayed after 2000 <7>[ 1623.213316] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1623.213319] libceph: try_read tag 1 in_base_pos 0 <7>[ 1623.213325] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 1623.213328] libceph: try_write start ffff8802214c4410 state 5 <7>[ 1623.213331] libceph: try_write out_kvec_bytes 0 <7>[ 1623.213333] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 1623.213335] libceph: try_write out_kvec_bytes 1 <7>[ 1623.213338] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 1623.213347] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1623.213349] libceph: try_write nothing else to write. <7>[ 1623.213352] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 1631.239345] libceph: osds timeout <7>[ 1631.239351] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1643.218539] libceph: monc delayed_work <7>[ 1643.218545] libceph: con_keepalive ffff8802214c4410 <7>[ 1643.218551] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 1643.218556] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 1643.218559] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 1643.218562] libceph: __send_subscribe sub_sent=0 exp=0 want_osd=0 <7>[ 1643.218565] libceph: __schedule_delayed after 2000 <7>[ 1643.218572] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1643.218574] libceph: try_read tag 1 in_base_pos 0 <7>[ 1643.218581] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 1643.218584] libceph: try_write start ffff8802214c4410 state 5 <7>[ 1643.218586] libceph: try_write out_kvec_bytes 0 <7>[ 1643.218589] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 1643.218591] libceph: try_write out_kvec_bytes 1 <7>[ 1643.218594] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 1643.218603] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1643.218606] libceph: try_write nothing else to write. <7>[ 1643.218608] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 1646.213346] libceph: osds timeout <7>[ 1646.213352] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1661.187330] libceph: osds timeout <7>[ 1661.187336] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1663.223794] libceph: monc delayed_work <7>[ 1663.223800] libceph: con_keepalive ffff8802214c4410 <7>[ 1663.223807] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 1663.223811] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 1663.223814] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 1663.223817] libceph: __send_subscribe sub_sent=0 exp=1 want_osd=0 <7>[ 1663.223823] libceph: ----- ffff88020d5b5400 to mon1 15=mon_subscribe len 31+0+0 ----- <7>[ 1663.223826] libceph: __schedule_delayed after 1000 <7>[ 1663.223833] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1663.223836] libceph: try_read tag 1 in_base_pos 0 <7>[ 1663.223842] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 1663.223845] libceph: try_write start ffff8802214c4410 state 5 <7>[ 1663.223847] libceph: try_write out_kvec_bytes 0 <7>[ 1663.223851] libceph: prepare_write_message ffff88020d5b5400 seq 13 type 15 len 31+0+0 0 pgs <7>[ 1663.223855] libceph: prepare_write_message front_crc 2650579135 middle_crc 0 <7>[ 1663.223858] libceph: prepare_write_message_footer ffff8802214c4410 <7>[ 1663.223860] libceph: try_write out_kvec_bytes 98 <7>[ 1663.223863] libceph: write_partial_kvec ffff8802214c4410 98 left <7>[ 1663.223882] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1663.223885] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 1663.223887] libceph: try_write out_kvec_bytes 1 <7>[ 1663.223890] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 1663.223895] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1663.223898] libceph: try_write nothing else to write. <7>[ 1663.223900] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 1663.224371] libceph: ceph_sock_data_ready on ffff8802214c4410 state = 5, queueing work <7>[ 1663.224382] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 1663.224398] libceph: ceph_sock_data_ready on ffff8802214c4410 state = 5, queueing work <7>[ 1663.224402] libceph: queue_con_delay ffff8802214c4410 - already queued <7>[ 1663.224418] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1663.224422] libceph: try_read tag 1 in_base_pos 0 <7>[ 1663.224428] libceph: try_read got tag 8 <7>[ 1663.224431] libceph: prepare_read_ack ffff8802214c4410 <7>[ 1663.224436] libceph: got ack for seq 13 type 15 at ffff88020d5b5400 <7>[ 1663.224438] libceph: prepare_read_tag ffff8802214c4410 <7>[ 1663.224441] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1663.224443] libceph: try_read tag 1 in_base_pos 0 <7>[ 1663.224448] libceph: try_read got tag 7 <7>[ 1663.224450] libceph: prepare_read_message ffff8802214c4410 <7>[ 1663.224453] libceph: read_partial_message con ffff8802214c4410 msg (null) <7>[ 1663.224458] libceph: got hdr type 4 front 481 data 0 <7>[ 1663.224462] libceph: ceph_msg_new ffff88020d35d600 front 481 <7>[ 1663.224473] libceph: read_partial_message got msg ffff88020d35d600 481 (2132872380) + 0 (0) + 0 (0) <7>[ 1663.224478] libceph: ===== ffff88020d35d600 24 from mon1 4=mon_map len 481+0 (2132872380 0 0) ===== <7>[ 1663.224482] libceph: have_debugfs_info fsid 1 globalid 4132 <7>[ 1663.224484] libceph: handle_monmap <7>[ 1663.224487] libceph: monmap_decode ffff88020d752c04 ffff88020d752de1 len 477 <7>[ 1663.224490] libceph: monmap_decode epoch 1, num_mon 3 <7>[ 1663.224493] libceph: monmap_decode mon0 is 10.214.133.28:6789 <7>[ 1663.224496] libceph: monmap_decode mon1 is 10.214.133.30:6789 <7>[ 1663.224499] libceph: monmap_decode mon2 is 10.214.133.28:6790 <7>[ 1663.224504] libceph: ceph_msg_put last one on ffff88020d35d600 <7>[ 1663.224506] libceph: msg_kfree ffff88020d35d600 <7>[ 1663.224510] libceph: prepare_read_tag ffff8802214c4410 <7>[ 1663.224512] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1663.224515] libceph: try_read tag 1 in_base_pos 0 <7>[ 1663.224519] libceph: try_read got tag 7 <7>[ 1663.224522] libceph: prepare_read_message ffff8802214c4410 <7>[ 1663.224524] libceph: read_partial_message con ffff8802214c4410 msg (null) <7>[ 1663.224529] libceph: got hdr type 16 front 20 data 0 <7>[ 1663.224549] libceph: read_partial_message got msg ffff88020d5b4500 20 (473278946) + 0 (0) + 0 (0) <7>[ 1663.224554] libceph: ===== ffff88020d5b4500 25 from mon1 16=mon_subscribe_ack len 20+0 (473278946 0 0) ===== <7>[ 1663.224557] libceph: handle_subscribe_ack after 300 seconds <7>[ 1663.224560] libceph: prepare_read_tag ffff8802214c4410 <7>[ 1663.224563] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1663.224565] libceph: try_read tag 1 in_base_pos 0 <7>[ 1663.224570] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 1663.224573] libceph: try_write start ffff8802214c4410 state 5 <7>[ 1663.224575] libceph: try_write out_kvec_bytes 0 <7>[ 1663.224578] libceph: prepare_write_ack ffff8802214c4410 23 -> 25 <7>[ 1663.224581] libceph: try_write out_kvec_bytes 9 <7>[ 1663.224583] libceph: write_partial_kvec ffff8802214c4410 9 left <7>[ 1663.224590] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1663.224593] libceph: try_write nothing else to write. <7>[ 1663.224595] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 1673.226432] libceph: monc delayed_work <7>[ 1673.226438] libceph: con_keepalive ffff8802214c4410 <7>[ 1673.226444] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 1673.226449] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 1673.226452] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 1673.226455] libceph: __send_subscribe sub_sent=0 exp=0 want_osd=0 <7>[ 1673.226457] libceph: __schedule_delayed after 2000 <7>[ 1673.226464] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1673.226466] libceph: try_read tag 1 in_base_pos 0 <7>[ 1673.226473] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 1673.226476] libceph: try_write start ffff8802214c4410 state 5 <7>[ 1673.226478] libceph: try_write out_kvec_bytes 0 <7>[ 1673.226481] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 1673.226483] libceph: try_write out_kvec_bytes 1 <7>[ 1673.226486] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 1673.226494] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1673.226497] libceph: try_write nothing else to write. <7>[ 1673.226499] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 1676.161320] libceph: osds timeout <7>[ 1676.161326] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1691.135319] libceph: osds timeout <7>[ 1691.135325] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1693.251638] libceph: monc delayed_work <7>[ 1693.251645] libceph: con_keepalive ffff8802214c4410 <7>[ 1693.251651] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 1693.251655] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 1693.251658] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 1693.251662] libceph: __send_subscribe sub_sent=0 exp=0 want_osd=0 <7>[ 1693.251664] libceph: __schedule_delayed after 2000 <7>[ 1693.251671] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1693.251674] libceph: try_read tag 1 in_base_pos 0 <7>[ 1693.251680] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 1693.251683] libceph: try_write start ffff8802214c4410 state 5 <7>[ 1693.251685] libceph: try_write out_kvec_bytes 0 <7>[ 1693.251688] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 1693.251690] libceph: try_write out_kvec_bytes 1 <7>[ 1693.251693] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 1693.251701] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1693.251704] libceph: try_write nothing else to write. <7>[ 1693.251706] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 1706.109299] libceph: osds timeout <7>[ 1706.109305] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1713.256894] libceph: monc delayed_work <7>[ 1713.256900] libceph: con_keepalive ffff8802214c4410 <7>[ 1713.256907] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 1713.256911] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 1713.256914] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 1713.256917] libceph: __send_subscribe sub_sent=0 exp=0 want_osd=0 <7>[ 1713.256920] libceph: __schedule_delayed after 2000 <7>[ 1713.256927] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1713.256929] libceph: try_read tag 1 in_base_pos 0 <7>[ 1713.256936] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 1713.256939] libceph: try_write start ffff8802214c4410 state 5 <7>[ 1713.256941] libceph: try_write out_kvec_bytes 0 <7>[ 1713.256944] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 1713.256946] libceph: try_write out_kvec_bytes 1 <7>[ 1713.256948] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 1713.256957] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1713.256960] libceph: try_write nothing else to write. <7>[ 1713.256962] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 1716.612254] libceph: parse_ips on '10.214.133.28:6789,10.214.133.30:6789,10.214.133.28:6790' <7>[ 1716.612259] libceph: parse_ips got 10.214.133.28:6789 <7>[ 1716.612263] libceph: parse_ips got 10.214.133.30:6789 <7>[ 1716.612266] libceph: parse_ips got 10.214.133.28:6790 <7>[ 1716.612270] libceph: got string token 6 val admin <7>[ 1716.612274] libceph: got string token 8 val client.admin <7>[ 1716.612281] rbd: rbd_client_create <7>[ 1716.612292] libceph: ceph_messenger_init ffff880224d20110 <7>[ 1716.612295] libceph: init <7>[ 1716.612299] libceph: auth_init name 'admin' <7>[ 1716.612302] libceph: auth_init name admin <7>[ 1716.612305] libceph: ceph_msg_new ffff88020d5b1200 front 20 <7>[ 1716.612308] libceph: ceph_msg_new ffff88020d5b1100 front 96 <7>[ 1716.612311] libceph: ceph_msg_new ffff88020d5b1800 front 4096 <7>[ 1716.612314] libceph: ceph_msg_new ffff88020d5b0300 front 4096 <7>[ 1716.612316] libceph: con_init ffff880224d20410 <7>[ 1716.612319] libceph: con_sock_state_init con ffff880224d20410 sock 0 -> 1 <7>[ 1716.612325] libceph: init <7>[ 1716.612337] libceph: msgpool osd_op init <7>[ 1716.612341] libceph: ceph_msg_new ffff88020d5b0400 front 4096 <7>[ 1716.612344] libceph: msgpool_alloc (null) ffff88020d5b0400 <7>[ 1716.612348] libceph: ceph_msg_new ffff88020d5b0f00 front 4096 <7>[ 1716.612350] libceph: msgpool_alloc (null) ffff88020d5b0f00 <7>[ 1716.612353] libceph: ceph_msg_new ffff88020d5b0e00 front 4096 <7>[ 1716.612356] libceph: msgpool_alloc (null) ffff88020d5b0e00 <7>[ 1716.612358] libceph: ceph_msg_new ffff88020d5b1900 front 4096 <7>[ 1716.612361] libceph: msgpool_alloc (null) ffff88020d5b1900 <7>[ 1716.612364] libceph: ceph_msg_new ffff88020d5b1a00 front 4096 <7>[ 1716.612367] libceph: msgpool_alloc (null) ffff88020d5b1a00 <7>[ 1716.612370] libceph: ceph_msg_new ffff88020d5b1f00 front 4096 <7>[ 1716.612373] libceph: msgpool_alloc (null) ffff88020d5b1f00 <7>[ 1716.612376] libceph: ceph_msg_new ffff88020d5b1700 front 4096 <7>[ 1716.612378] libceph: msgpool_alloc (null) ffff88020d5b1700 <7>[ 1716.612381] libceph: ceph_msg_new ffff88020d5b1b00 front 4096 <7>[ 1716.612383] libceph: msgpool_alloc (null) ffff88020d5b1b00 <7>[ 1716.612386] libceph: ceph_msg_new ffff88020d5b0600 front 4096 <7>[ 1716.612389] libceph: msgpool_alloc (null) ffff88020d5b0600 <7>[ 1716.612392] libceph: ceph_msg_new ffff88020d5b0800 front 4096 <7>[ 1716.612394] libceph: msgpool_alloc (null) ffff88020d5b0800 <7>[ 1716.612397] libceph: msgpool osd_op_reply init <7>[ 1716.612401] libceph: ceph_msg_new ffff88020d5b1400 front 512 <7>[ 1716.612403] libceph: msgpool_alloc (null) ffff88020d5b1400 <7>[ 1716.612406] libceph: ceph_msg_new ffff88020d5b1300 front 512 <7>[ 1716.612409] libceph: msgpool_alloc (null) ffff88020d5b1300 <7>[ 1716.612412] libceph: ceph_msg_new ffff88020d5b1600 front 512 <7>[ 1716.612414] libceph: msgpool_alloc (null) ffff88020d5b1600 <7>[ 1716.612418] libceph: ceph_msg_new ffff88020d5b0700 front 512 <7>[ 1716.612420] libceph: msgpool_alloc (null) ffff88020d5b0700 <7>[ 1716.612423] libceph: ceph_msg_new ffff88020d5b0b00 front 512 <7>[ 1716.612425] libceph: msgpool_alloc (null) ffff88020d5b0b00 <7>[ 1716.612428] libceph: ceph_msg_new ffff88020d5b0900 front 512 <7>[ 1716.612431] libceph: msgpool_alloc (null) ffff88020d5b0900 <7>[ 1716.612434] libceph: ceph_msg_new ffff88020d5b0000 front 512 <7>[ 1716.612436] libceph: msgpool_alloc (null) ffff88020d5b0000 <7>[ 1716.612440] libceph: ceph_msg_new ffff88020d5b1e00 front 512 <7>[ 1716.612442] libceph: msgpool_alloc (null) ffff88020d5b1e00 <7>[ 1716.612445] libceph: ceph_msg_new ffff88020d5b0a00 front 512 <7>[ 1716.612447] libceph: msgpool_alloc (null) ffff88020d5b0a00 <7>[ 1716.612450] libceph: ceph_msg_new ffff88020d5b1c00 front 512 <7>[ 1716.612453] libceph: msgpool_alloc (null) ffff88020d5b1c00 <7>[ 1716.612565] libceph: open_session start <7>[ 1716.612580] libceph: open_session num=3 r=114 -> mon0 <7>[ 1716.612583] libceph: open_session mon0 opening <7>[ 1716.612588] libceph: con_open ffff880224d20410 10.214.133.28:6789 <7>[ 1716.612596] libceph: queue_con_delay ffff880224d20410 0 <7>[ 1716.612599] libceph: auth_build_hello <7>[ 1716.612607] libceph: con_work OPENING <7>[ 1716.612609] libceph: try_read start on ffff880224d20410 state 2 <7>[ 1716.612612] libceph: try_write start ffff880224d20410 state 2 <7>[ 1716.612615] libceph: try_write out_kvec_bytes 0 <7>[ 1716.612618] libceph: prepare_read_banner ffff880224d20410 <7>[ 1716.612620] libceph: try_write initiating connect on ffff880224d20410 new state 3 <7>[ 1716.612634] libceph: connect 10.214.133.28:6789 <7>[ 1716.612637] libceph: con_sock_state_connecting con ffff880224d20410 sock 1 -> 2 <7>[ 1716.612696] libceph: connect 10.214.133.28:6789 EINPROGRESS sk_state = 2 <7>[ 1716.612699] libceph: write_partial_kvec ffff880224d20410 145 left <7>[ 1716.612705] libceph: write_partial_kvec ffff880224d20410 145 left in 2 kvecs ret = 0 <7>[ 1716.612708] libceph: try_write done on ffff880224d20410 ret 0 <7>[ 1716.612718] libceph: ----- ffff88020d5b0300 to mon0 17=auth len 60+0+0 ----- <7>[ 1716.612721] libceph: __schedule_delayed after 1000 <7>[ 1716.612726] libceph: mount waiting for mon_map <7>[ 1716.613080] libceph: ceph_sock_state_change ffff880224d20410 state = 3 sk_state = 1 <7>[ 1716.613084] libceph: ceph_sock_state_change TCP_ESTABLISHED <7>[ 1716.613088] libceph: con_sock_state_connected con ffff880224d20410 sock 2 -> 3 <7>[ 1716.613097] libceph: queue_con_delay ffff880224d20410 0 <7>[ 1716.613124] libceph: try_read start on ffff880224d20410 state 3 <7>[ 1716.613127] libceph: try_read tag 1 in_base_pos 0 <7>[ 1716.613129] libceph: try_read connecting <7>[ 1716.613132] libceph: read_partial_banner ffff880224d20410 at 0 <7>[ 1716.613138] libceph: try_read done on ffff880224d20410 ret 0 <7>[ 1716.613141] libceph: try_write start ffff880224d20410 state 3 <7>[ 1716.613144] libceph: try_write out_kvec_bytes 145 <7>[ 1716.613147] libceph: write_partial_kvec ffff880224d20410 145 left <7>[ 1716.613162] libceph: write_partial_kvec ffff880224d20410 0 left in 0 kvecs ret = 1 <7>[ 1716.613165] libceph: try_write nothing else to write. <7>[ 1716.613167] libceph: try_write done on ffff880224d20410 ret 0 <7>[ 1716.613425] libceph: ceph_sock_data_ready on ffff880224d20410 state = 3, queueing work <7>[ 1716.613434] libceph: queue_con_delay ffff880224d20410 0 <7>[ 1716.613454] libceph: ceph_sock_data_ready on ffff880224d20410 state = 3, queueing work <7>[ 1716.613458] libceph: queue_con_delay ffff880224d20410 - already queued <7>[ 1716.613469] libceph: try_read start on ffff880224d20410 state 3 <7>[ 1716.613472] libceph: try_read tag 1 in_base_pos 0 <7>[ 1716.613474] libceph: try_read connecting <7>[ 1716.613477] libceph: read_partial_banner ffff880224d20410 at 0 <7>[ 1716.613489] libceph: process_banner on ffff880224d20410 <7>[ 1716.613495] libceph: prepare_write_connect ffff880224d20410 cseq=0 gseq=1 proto=15 <7>[ 1716.613498] libceph: prepare_read_connect ffff880224d20410 <7>[ 1716.613500] libceph: try_read done on ffff880224d20410 ret 0 <7>[ 1716.613503] libceph: try_write start ffff880224d20410 state 4 <7>[ 1716.613505] libceph: try_write out_kvec_bytes 33 <7>[ 1716.613508] libceph: write_partial_kvec ffff880224d20410 33 left <7>[ 1716.613523] libceph: write_partial_kvec ffff880224d20410 0 left in 0 kvecs ret = 1 <7>[ 1716.613526] libceph: try_write nothing else to write. <7>[ 1716.613528] libceph: try_write done on ffff880224d20410 ret 0 <7>[ 1716.613711] libceph: ceph_sock_data_ready on ffff880224d20410 state = 4, queueing work <7>[ 1716.613719] libceph: queue_con_delay ffff880224d20410 0 <7>[ 1716.613731] libceph: try_read start on ffff880224d20410 state 4 <7>[ 1716.613734] libceph: try_read tag 1 in_base_pos 0 <7>[ 1716.613737] libceph: try_read negotiating <7>[ 1716.613740] libceph: read_partial_connect ffff880224d20410 at 0 <7>[ 1716.613748] libceph: read_partial_connect ffff880224d20410 tag 1, con_seq = 1, g_seq = 14 <7>[ 1716.613751] libceph: process_connect on ffff880224d20410 tag 1 <7>[ 1716.613754] libceph: process_connect got READY gseq 14 cseq 1 (1) <7>[ 1716.613757] libceph: prepare_read_tag ffff880224d20410 <7>[ 1716.613759] libceph: try_read start on ffff880224d20410 state 5 <7>[ 1716.613762] libceph: try_read tag 1 in_base_pos 0 <7>[ 1716.613766] libceph: try_read done on ffff880224d20410 ret 0 <7>[ 1716.613769] libceph: try_write start ffff880224d20410 state 5 <7>[ 1716.613771] libceph: try_write out_kvec_bytes 0 <7>[ 1716.613775] libceph: prepare_write_message ffff88020d5b0300 seq 1 type 17 len 60+0+0 0 pgs <7>[ 1716.613780] libceph: prepare_write_message front_crc 3679117932 middle_crc 0 <7>[ 1716.613783] libceph: prepare_write_message_footer ffff880224d20410 <7>[ 1716.613785] libceph: try_write out_kvec_bytes 127 <7>[ 1716.613788] libceph: write_partial_kvec ffff880224d20410 127 left <7>[ 1716.613803] libceph: write_partial_kvec ffff880224d20410 0 left in 0 kvecs ret = 1 <7>[ 1716.613805] libceph: try_write nothing else to write. <7>[ 1716.613808] libceph: try_write done on ffff880224d20410 ret 0 <7>[ 1716.614536] libceph: ceph_sock_data_ready on ffff880224d20410 state = 5, queueing work <7>[ 1716.614545] libceph: queue_con_delay ffff880224d20410 0 <7>[ 1716.614560] libceph: ceph_sock_data_ready on ffff880224d20410 state = 5, queueing work <7>[ 1716.614564] libceph: queue_con_delay ffff880224d20410 - already queued <7>[ 1716.614582] libceph: try_read start on ffff880224d20410 state 5 <7>[ 1716.614585] libceph: try_read tag 1 in_base_pos 0 <7>[ 1716.614592] libceph: try_read got tag 8 <7>[ 1716.614594] libceph: prepare_read_ack ffff880224d20410 <7>[ 1716.614599] libceph: got ack for seq 1 type 17 at ffff88020d5b0300 <7>[ 1716.614602] libceph: prepare_read_tag ffff880224d20410 <7>[ 1716.614604] libceph: try_read start on ffff880224d20410 state 5 <7>[ 1716.614607] libceph: try_read tag 1 in_base_pos 0 <7>[ 1716.614611] libceph: try_read got tag 7 <7>[ 1716.614613] libceph: prepare_read_message ffff880224d20410 <7>[ 1716.614616] libceph: read_partial_message con ffff880224d20410 msg (null) <7>[ 1716.614622] libceph: got hdr type 4 front 481 data 0 <7>[ 1716.614628] libceph: ceph_msg_new ffff88020d35db00 front 481 <7>[ 1716.614644] libceph: read_partial_message got msg ffff88020d35db00 481 (2132872380) + 0 (0) + 0 (0) <7>[ 1716.614649] libceph: ===== ffff88020d35db00 1 from mon0 4=mon_map len 481+0 (2132872380 0 0) ===== <7>[ 1716.614652] libceph: have_debugfs_info fsid 0 globalid 0 <7>[ 1716.614655] libceph: handle_monmap <7>[ 1716.614657] libceph: monmap_decode ffff88020d752204 ffff88020d7523e1 len 477 <7>[ 1716.614660] libceph: monmap_decode epoch 1, num_mon 3 <7>[ 1716.614664] libceph: monmap_decode mon0 is 10.214.133.28:6789 <7>[ 1716.614667] libceph: monmap_decode mon1 is 10.214.133.30:6789 <7>[ 1716.614670] libceph: monmap_decode mon2 is 10.214.133.28:6790 <7>[ 1716.614673] libceph: have_debugfs_info fsid 1 globalid 0 <7>[ 1716.614681] libceph: ceph_msg_put last one on ffff88020d35db00 <7>[ 1716.614683] libceph: msg_kfree ffff88020d35db00 <7>[ 1716.614687] libceph: prepare_read_tag ffff880224d20410 <7>[ 1716.614690] libceph: try_read start on ffff880224d20410 state 5 <7>[ 1716.614692] libceph: try_read tag 1 in_base_pos 0 <7>[ 1716.614697] libceph: try_read got tag 7 <7>[ 1716.614700] libceph: prepare_read_message ffff880224d20410 <7>[ 1716.614702] libceph: read_partial_message con ffff880224d20410 msg (null) <7>[ 1716.614707] libceph: got hdr type 18 front 33 data 0 <7>[ 1716.614717] libceph: read_partial_message got msg ffff88020d5b1800 33 (1721559937) + 0 (0) + 0 (0) <7>[ 1716.614722] libceph: ===== ffff88020d5b1800 2 from mon0 18=auth_reply len 33+0 (1721559937 0 0) ===== <7>[ 1716.614725] libceph: have_debugfs_info fsid 1 globalid 0 <7>[ 1716.614728] libceph: handle_auth_reply ffff880224d23000 ffff880224d23021 <7>[ 1716.614731] libceph: result 0 '' gid 4116 len 9 <7>[ 1716.614734] libceph: set global_id 0 -> 4116 <7>[ 1716.614737] libceph: ceph_x_init ffff880212d20f80 <7>[ 1716.614740] libceph: handle_reply got server challenge 76d9b668305142d8 <7>[ 1716.614744] libceph: build_request want 27 have 0 need 27 <7>[ 1716.614747] libceph: get_auth_session_key <7>[ 1716.614775] libceph: server_challenge 76d9b668305142d8 client_challenge 661815bbb8185a82 key d906f0373d0fde05 <7>[ 1716.614778] libceph: built request 32 bytes <7>[ 1716.614783] libceph: ----- ffff88020d5b0300 to mon0 17=auth len 58+0+0 ----- <7>[ 1716.614787] libceph: queue_con_delay ffff880224d20410 0 <7>[ 1716.614790] libceph: have_debugfs_info fsid 1 globalid 4116 <6>[ 1716.614793] libceph: client4116 fsid 155e9d3f-16e7-4f5e-9e56-a90071f0ea16 <7>[ 1716.621624] libceph: ceph_debugfs_client_init ffff880224d20000 155e9d3f-16e7-4f5e-9e56-a90071f0ea16.client4116 <7>[ 1716.621691] libceph: prepare_read_tag ffff880224d20410 <7>[ 1716.621696] libceph: try_read start on ffff880224d20410 state 5 <7>[ 1716.621701] libceph: try_read tag 1 in_base_pos 0 <7>[ 1716.621710] libceph: try_read done on ffff880224d20410 ret 0 <7>[ 1716.621715] libceph: try_write start ffff880224d20410 state 5 <7>[ 1716.621719] libceph: try_write out_kvec_bytes 0 <7>[ 1716.621725] libceph: prepare_write_message ffff88020d5b0300 seq 2 type 17 len 58+0+0 0 pgs <7>[ 1716.621731] libceph: prepare_write_message front_crc 2978719476 middle_crc 0 <7>[ 1716.621736] libceph: prepare_write_message_footer ffff880224d20410 <7>[ 1716.621740] libceph: try_write out_kvec_bytes 134 <7>[ 1716.621745] libceph: write_partial_kvec ffff880224d20410 134 left <7>[ 1716.621772] libceph: write_partial_kvec ffff880224d20410 0 left in 0 kvecs ret = 1 <7>[ 1716.621776] libceph: try_write nothing else to write. <7>[ 1716.621781] libceph: try_write done on ffff880224d20410 ret 0 <7>[ 1716.621789] libceph: try_read start on ffff880224d20410 state 5 <7>[ 1716.621793] libceph: try_read tag 1 in_base_pos 0 <7>[ 1716.621801] libceph: try_read done on ffff880224d20410 ret 0 <7>[ 1716.621806] libceph: try_write start ffff880224d20410 state 5 <7>[ 1716.621810] libceph: try_write out_kvec_bytes 0 <7>[ 1716.621813] libceph: try_write nothing else to write. <7>[ 1716.621818] libceph: try_write done on ffff880224d20410 ret 0 <7>[ 1716.622478] libceph: ceph_sock_data_ready on ffff880224d20410 state = 5, queueing work <7>[ 1716.622489] libceph: queue_con_delay ffff880224d20410 0 <7>[ 1716.622506] libceph: try_read start on ffff880224d20410 state 5 <7>[ 1716.622511] libceph: try_read tag 1 in_base_pos 0 <7>[ 1716.622520] libceph: try_read got tag 8 <7>[ 1716.622524] libceph: prepare_read_ack ffff880224d20410 <7>[ 1716.622533] libceph: got ack for seq 2 type 17 at ffff88020d5b0300 <7>[ 1716.622537] libceph: prepare_read_tag ffff880224d20410 <7>[ 1716.622542] libceph: try_read start on ffff880224d20410 state 5 <7>[ 1716.622546] libceph: try_read tag 1 in_base_pos 0 <7>[ 1716.622554] libceph: try_read got tag 7 <7>[ 1716.622558] libceph: prepare_read_message ffff880224d20410 <7>[ 1716.622562] libceph: read_partial_message con ffff880224d20410 msg (null) <7>[ 1716.622572] libceph: got hdr type 18 front 206 data 0 <7>[ 1716.622590] libceph: read_partial_message got msg ffff88020d5b1800 206 (238205801) + 0 (0) + 0 (0) <7>[ 1716.622598] libceph: ===== ffff88020d5b1800 3 from mon0 18=auth_reply len 206+0 (238205801 0 0) ===== <7>[ 1716.622604] libceph: have_debugfs_info fsid 1 globalid 4116 <7>[ 1716.622609] libceph: ceph_x_is_authenticated want=39 need=39 have=0 <7>[ 1716.622614] libceph: handle_auth_reply ffff880224d23000 ffff880224d230ce <7>[ 1716.622619] libceph: result 0 '' gid 4116 len 182 <7>[ 1716.622624] libceph: handle_reply op 256 result 0 <7>[ 1716.622629] libceph: 1 tickets <7>[ 1716.622633] libceph: ticket type 32 auth <7>[ 1716.622638] libceph: ceph_x_decrypt len 48 <7>[ 1716.622657] libceph: decrypted 37 bytes <7>[ 1716.622662] libceph: expires=1364480786 renew_after=1364469986 <7>[ 1716.622667] libceph: ticket blob is 109 bytes <7>[ 1716.622671] libceph: decode_buffer len 96 <7>[ 1716.622676] libceph: buffer_new ffff88020b2c67c0 <7>[ 1716.622681] libceph: got ticket service 32 (auth) secret_id 2 len 96 <7>[ 1716.622688] libceph: build_request want 27 have 20 need 7 <7>[ 1716.622693] libceph: build_authorizer for auth ffff88020d35c638 <7>[ 1716.622697] libceph: need len 160 <7>[ 1716.622702] libceph: buffer_new ffff88020b2c65c0 <7>[ 1716.622707] libceph: th ffff88021fe71960 secret_id 2 2 <7>[ 1716.622737] libceph: built authorizer nonce 9cb4f47007db12a6 len 158 <7>[ 1716.622741] libceph: built request 165 bytes <7>[ 1716.622749] libceph: ----- ffff88020d5b0300 to mon0 17=auth len 191+0+0 ----- <7>[ 1716.622757] libceph: queue_con_delay ffff880224d20410 0 <7>[ 1716.622762] libceph: prepare_read_tag ffff880224d20410 <7>[ 1716.622767] libceph: try_read start on ffff880224d20410 state 5 <7>[ 1716.622771] libceph: try_read tag 1 in_base_pos 0 <7>[ 1716.622780] libceph: try_read done on ffff880224d20410 ret 0 <7>[ 1716.622784] libceph: try_write start ffff880224d20410 state 5 <7>[ 1716.622788] libceph: try_write out_kvec_bytes 0 <7>[ 1716.622795] libceph: prepare_write_message ffff88020d5b0300 seq 3 type 17 len 191+0+0 0 pgs <7>[ 1716.622801] libceph: prepare_write_message front_crc 3265867497 middle_crc 0 <7>[ 1716.622806] libceph: prepare_write_message_footer ffff880224d20410 <7>[ 1716.622810] libceph: try_write out_kvec_bytes 267 <7>[ 1716.622815] libceph: write_partial_kvec ffff880224d20410 267 left <7>[ 1716.622837] libceph: write_partial_kvec ffff880224d20410 0 left in 0 kvecs ret = 1 <7>[ 1716.622841] libceph: try_write nothing else to write. <7>[ 1716.622846] libceph: try_write done on ffff880224d20410 ret 0 <7>[ 1716.622855] libceph: try_read start on ffff880224d20410 state 5 <7>[ 1716.622859] libceph: try_read tag 1 in_base_pos 0 <7>[ 1716.622867] libceph: try_read done on ffff880224d20410 ret 0 <7>[ 1716.622871] libceph: try_write start ffff880224d20410 state 5 <7>[ 1716.622875] libceph: try_write out_kvec_bytes 0 <7>[ 1716.622879] libceph: try_write nothing else to write. <7>[ 1716.622884] libceph: try_write done on ffff880224d20410 ret 0 <7>[ 1716.623529] libceph: ceph_sock_data_ready on ffff880224d20410 state = 5, queueing work <7>[ 1716.623540] libceph: queue_con_delay ffff880224d20410 0 <7>[ 1716.623555] libceph: try_read start on ffff880224d20410 state 5 <7>[ 1716.623560] libceph: try_read tag 1 in_base_pos 0 <7>[ 1716.623569] libceph: try_read got tag 8 <7>[ 1716.623573] libceph: prepare_read_ack ffff880224d20410 <7>[ 1716.623581] libceph: got ack for seq 3 type 17 at ffff88020d5b0300 <7>[ 1716.623585] libceph: prepare_read_tag ffff880224d20410 <7>[ 1716.623589] libceph: try_read start on ffff880224d20410 state 5 <7>[ 1716.623593] libceph: try_read tag 1 in_base_pos 0 <7>[ 1716.623600] libceph: try_read got tag 7 <7>[ 1716.623605] libceph: prepare_read_message ffff880224d20410 <7>[ 1716.623609] libceph: read_partial_message con ffff880224d20410 msg (null) <7>[ 1716.623618] libceph: got hdr type 18 front 580 data 0 <7>[ 1716.623636] libceph: read_partial_message got msg ffff88020d5b1800 580 (2173735363) + 0 (0) + 0 (0) <7>[ 1716.623643] libceph: ===== ffff88020d5b1800 4 from mon0 18=auth_reply len 580+0 (2173735363 0 0) ===== <7>[ 1716.623649] libceph: have_debugfs_info fsid 1 globalid 4116 <7>[ 1716.623654] libceph: ceph_x_is_authenticated want=39 need=7 have=32 <7>[ 1716.623659] libceph: handle_auth_reply ffff880224d23000 ffff880224d23244 <7>[ 1716.623664] libceph: result 0 '' gid 4116 len 556 <7>[ 1716.623668] libceph: handle_reply op 512 result 0 <7>[ 1716.623673] libceph: 3 tickets <7>[ 1716.623677] libceph: ticket type 1 mon <7>[ 1716.623681] libceph: ceph_x_decrypt len 48 <7>[ 1716.623698] libceph: decrypted 37 bytes <7>[ 1716.623703] libceph: expires=1364441186 renew_after=1364440286 <7>[ 1716.623708] libceph: ticket blob is 109 bytes <7>[ 1716.623712] libceph: decode_buffer len 96 <7>[ 1716.623716] libceph: buffer_new ffff88020b2c6e80 <7>[ 1716.623721] libceph: got ticket service 1 (mon) secret_id 2 len 96 <7>[ 1716.623725] libceph: ticket type 2 mds <7>[ 1716.623729] libceph: ceph_x_decrypt len 48 <7>[ 1716.623742] libceph: decrypted 37 bytes <7>[ 1716.623747] libceph: expires=1364441186 renew_after=1364440286 <7>[ 1716.623752] libceph: ticket blob is 125 bytes <7>[ 1716.623756] libceph: decode_buffer len 112 <7>[ 1716.623760] libceph: buffer_new ffff88020b2c6940 <7>[ 1716.623765] libceph: got ticket service 2 (mds) secret_id 2 len 112 <7>[ 1716.623769] libceph: ticket type 4 osd <7>[ 1716.623773] libceph: ceph_x_decrypt len 48 <7>[ 1716.623786] libceph: decrypted 37 bytes <7>[ 1716.623791] libceph: expires=1364441186 renew_after=1364440286 <7>[ 1716.623795] libceph: ticket blob is 125 bytes <7>[ 1716.623799] libceph: decode_buffer len 112 <7>[ 1716.623803] libceph: buffer_new ffff88020b2c6100 <7>[ 1716.623808] libceph: got ticket service 4 (osd) secret_id 2 len 112 <7>[ 1716.623814] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 1716.623818] libceph: authenticated, starting session <7>[ 1716.623823] libceph: __send_subscribe sub_sent=0 exp=1 want_osd=1 <7>[ 1716.623827] libceph: __send_subscribe to 'osdmap' 0 <7>[ 1716.623834] libceph: ----- ffff88020d5b1100 to mon0 15=mon_subscribe len 58+0+0 ----- <7>[ 1716.623841] libceph: queue_con_delay ffff880224d20410 0 <7>[ 1716.623847] libceph: prepare_read_tag ffff880224d20410 <7>[ 1716.623852] libceph: try_read start on ffff880224d20410 state 5 <7>[ 1716.623856] libceph: try_read tag 1 in_base_pos 0 <7>[ 1716.623863] libceph: try_read done on ffff880224d20410 ret 0 <7>[ 1716.623867] libceph: try_write start ffff880224d20410 state 5 <7>[ 1716.623871] libceph: try_write out_kvec_bytes 0 <7>[ 1716.623876] libceph: prepare_write_message ffff88020d5b1100 seq 4 type 15 len 58+0+0 0 pgs <7>[ 1716.623881] libceph: prepare_write_message front_crc 995354139 middle_crc 0 <7>[ 1716.623885] libceph: prepare_write_message_footer ffff880224d20410 <7>[ 1716.623889] libceph: try_write out_kvec_bytes 134 <7>[ 1716.623893] libceph: write_partial_kvec ffff880224d20410 134 left <7>[ 1716.623917] libceph: write_partial_kvec ffff880224d20410 0 left in 0 kvecs ret = 1 <7>[ 1716.623921] libceph: try_write nothing else to write. <7>[ 1716.623926] libceph: try_write done on ffff880224d20410 ret 0 <7>[ 1716.623934] libceph: try_read start on ffff880224d20410 state 5 <7>[ 1716.623939] libceph: try_read tag 1 in_base_pos 0 <7>[ 1716.623946] libceph: try_read done on ffff880224d20410 ret 0 <7>[ 1716.623951] libceph: try_write start ffff880224d20410 state 5 <7>[ 1716.623955] libceph: try_write out_kvec_bytes 0 <7>[ 1716.623959] libceph: try_write nothing else to write. <7>[ 1716.623963] libceph: try_write done on ffff880224d20410 ret 0 <7>[ 1716.624398] libceph: ceph_sock_data_ready on ffff880224d20410 state = 5, queueing work <7>[ 1716.624409] libceph: queue_con_delay ffff880224d20410 0 <7>[ 1716.624424] libceph: try_read start on ffff880224d20410 state 5 <7>[ 1716.624428] libceph: try_read tag 1 in_base_pos 0 <7>[ 1716.624437] libceph: try_read got tag 8 <7>[ 1716.624441] libceph: prepare_read_ack ffff880224d20410 <7>[ 1716.624449] libceph: got ack for seq 4 type 15 at ffff88020d5b1100 <7>[ 1716.624454] libceph: prepare_read_tag ffff880224d20410 <7>[ 1716.624458] libceph: try_read start on ffff880224d20410 state 5 <7>[ 1716.624463] libceph: try_read tag 1 in_base_pos 0 <7>[ 1716.624470] libceph: try_read got tag 7 <7>[ 1716.624474] libceph: prepare_read_message ffff880224d20410 <7>[ 1716.624478] libceph: read_partial_message con ffff880224d20410 msg (null) <7>[ 1716.624487] libceph: got hdr type 4 front 481 data 0 <7>[ 1716.624493] libceph: ceph_msg_new ffff88020d35ca00 front 481 <7>[ 1716.624510] libceph: read_partial_message got msg ffff88020d35ca00 481 (2132872380) + 0 (0) + 0 (0) <7>[ 1716.624517] libceph: ===== ffff88020d35ca00 5 from mon0 4=mon_map len 481+0 (2132872380 0 0) ===== <7>[ 1716.624553] libceph: ceph_sock_data_ready on ffff880224d20410 state = 5, queueing work <7>[ 1716.624560] libceph: queue_con_delay ffff880224d20410 0 <7>[ 1716.624568] libceph: have_debugfs_info fsid 1 globalid 4116 <7>[ 1716.624572] libceph: handle_monmap <7>[ 1716.624577] libceph: monmap_decode ffff88020d752204 ffff88020d7523e1 len 477 <7>[ 1716.624581] libceph: monmap_decode epoch 1, num_mon 3 <7>[ 1716.624587] libceph: monmap_decode mon0 is 10.214.133.28:6789 <7>[ 1716.624592] libceph: monmap_decode mon1 is 10.214.133.30:6789 <7>[ 1716.624597] libceph: monmap_decode mon2 is 10.214.133.28:6790 <7>[ 1716.624608] libceph: ceph_msg_put last one on ffff88020d35ca00 <7>[ 1716.624613] libceph: msg_kfree ffff88020d35ca00 <7>[ 1716.624619] libceph: prepare_read_tag ffff880224d20410 <7>[ 1716.624623] libceph: try_read start on ffff880224d20410 state 5 <7>[ 1716.624628] libceph: try_read tag 1 in_base_pos 0 <7>[ 1716.624636] libceph: try_read got tag 7 <7>[ 1716.624640] libceph: prepare_read_message ffff880224d20410 <7>[ 1716.624645] libceph: read_partial_message con ffff880224d20410 msg (null) <7>[ 1716.624654] libceph: got hdr type 41 front 1784 data 0 <7>[ 1716.624660] libceph: ceph_msg_new ffff88020d35ca00 front 1784 <7>[ 1716.624682] libceph: read_partial_message got msg ffff88020d35ca00 1784 (3640922378) + 0 (0) + 0 (0) <7>[ 1716.624690] libceph: ===== ffff88020d35ca00 6 from mon0 41=osd_map len 1784+0 (3640922378 0 0) ===== <7>[ 1716.624695] libceph: handle_map have 0 <7>[ 1716.624700] libceph: 0 inc maps <7>[ 1716.624705] libceph: 1 full maps <7>[ 1716.624709] libceph: taking full map 5 len 1752 <7>[ 1716.624729] libceph: ceph_sock_data_ready on ffff880224d20410 state = 5, queueing work <7>[ 1716.624734] libceph: queue_con_delay ffff880224d20410 - already queued <7>[ 1716.624743] libceph: osdmap_decode ffff8802148c6020 to ffff8802148c66f8 len 1752 <7>[ 1716.624750] libceph: 3 pool names <7>[ 1716.624754] libceph: pool 0 len 4 <7>[ 1716.624759] libceph: name is data <7>[ 1716.624763] libceph: pool 1 len 8 <7>[ 1716.624767] libceph: name is metadata <7>[ 1716.624771] libceph: pool 2 len 3 <7>[ 1716.624775] libceph: name is rbd <7>[ 1716.624781] libceph: osdmap_decode max_osd = 6 <7>[ 1716.624786] libceph: osdmap_decode crush len 618 from off 0x46e <7>[ 1716.624791] libceph: crush_decode ffff8802148c648e to ffff8802148c66f8 len 618 <7>[ 1716.624797] libceph: crush_decode bucket 0 off 14 ffff8802148c64a2 to ffff8802148c66f8 <7>[ 1716.624803] libceph: crush_decode bucket size 1 off 24 ffff8802148c64b2 to ffff8802148c66f8 <7>[ 1716.624808] libceph: crush_decode_straw_bucket ffff8802148c64b6 to ffff8802148c66f8 <7>[ 1716.624814] libceph: crush_decode bucket 1 off 34 ffff8802148c64c2 to ffff8802148c66f8 <7>[ 1716.624819] libceph: crush_decode bucket size 6 off 44 ffff8802148c64d2 to ffff8802148c66f8 <7>[ 1716.624824] libceph: crush_decode_straw_bucket ffff8802148c64ea to ffff8802148c66f8 <7>[ 1716.624830] libceph: crush_decode bucket 2 off 90 ffff8802148c651e to ffff8802148c66f8 <7>[ 1716.624835] libceph: crush_decode bucket size 1 off a0 ffff8802148c652e to ffff8802148c66f8 <7>[ 1716.624840] libceph: crush_decode_straw_bucket ffff8802148c6532 to ffff8802148c66f8 <7>[ 1716.624845] libceph: rule vec is ffff88020d341dc0 <7>[ 1716.624850] libceph: crush_decode rule 0 off c4 ffff8802148c6552 to ffff8802148c66f8 <7>[ 1716.624854] libceph: rule 0 is at ffff88020b2c6580 <7>[ 1716.624859] libceph: crush_decode rule 1 off f4 ffff8802148c6582 to ffff8802148c66f8 <7>[ 1716.624864] libceph: rule 1 is at ffff88020b2c6700 <7>[ 1716.624869] libceph: crush_decode rule 2 off 124 ffff8802148c65b2 to ffff8802148c66f8 <7>[ 1716.624873] libceph: rule 2 is at ffff88020b2c6000 <7>[ 1716.624877] libceph: crush decode tunable choose_local_tries = 2 <7>[ 1716.624882] libceph: crush decode tunable choose_local_fallback_tries = <7>[ 1716.624886] libceph: crush decode tunable choose_total_tries = 19 <7>[ 1716.624889] libceph: crush_decode success <7>[ 1716.624893] libceph: osdmap_decode done ffff8802148c66f8 ffff8802148c66f8 <7>[ 1716.624898] libceph: kick_requests <7>[ 1716.624907] libceph: send_queued <7>[ 1716.624918] libceph: ceph_msg_put last one on ffff88020d35ca00 <7>[ 1716.624922] libceph: msg_kfree ffff88020d35ca00 <7>[ 1716.624930] libceph: prepare_read_tag ffff880224d20410 <7>[ 1716.624934] libceph: try_read start on ffff880224d20410 state 5 <7>[ 1716.624939] libceph: try_read tag 1 in_base_pos 0 <7>[ 1716.624948] libceph: try_read got tag 7 <7>[ 1716.624952] libceph: prepare_read_message ffff880224d20410 <7>[ 1716.624956] libceph: read_partial_message con ffff880224d20410 msg (null) <7>[ 1716.624965] libceph: got hdr type 16 front 20 data 0 <7>[ 1716.624982] libceph: read_partial_message got msg ffff88020d5b1200 20 (473278946) + 0 (0) + 0 (0) <7>[ 1716.624985] rbd: rbd_client_create created ffff88020beecea0 <7>[ 1716.624994] libceph: ===== ffff88020d5b1200 7 from mon0 16=mon_subscribe_ack len 20+0 (473278946 0 0) ===== <7>[ 1716.624996] rbd: rbd id object name is rbd_id.afs254-vicepa <7>[ 1716.625003] rbd: rbd_do_request object_name=rbd_id.afs254-vicepa ofs=0 len=64 coll= (null)[0] <6>[ 1716.625008] libceph: mon0 10.214.133.28:6789 session established <7>[ 1716.625012] libceph: ceph_msg_new ffff88020d5b1000 front 512 <7>[ 1716.625014] libceph: ceph_msg_new ffff88020d5b0200 front 208 <7>[ 1716.625017] libceph: mapping 0~64 osize 1073741824 fl_su 1073741824 <7>[ 1716.625018] libceph: osize 1073741824 / su 1073741824 = su_per_object 1 <7>[ 1716.625020] libceph: off 0 / su 1073741824 = bl 0 <7>[ 1716.625021] libceph: objset 0 * sc 1 = ono 0 <7>[ 1716.625022] libceph: obj extent 0~64 <7>[ 1716.625024] libceph: calc_layout bno=0 0~64 (1 pages) <7>[ 1716.625030] libceph: __register_request ffff88020b496c00 tid 1 <7>[ 1716.625031] libceph: first request, scheduling timeout <7>[ 1716.625036] libceph: map_request ffff88020b496c00 tid 1 <7>[ 1716.625038] libceph: calc_object_layout 'rbd_id.afs254-vicepa' pgid 2.eb195e3b <7>[ 1716.625046] libceph: map_request tid 1 pgid 2.5e3b osd0 (was osd-1) <7>[ 1716.625047] libceph: con_init ffff880221ea3830 <7>[ 1716.625049] libceph: con_sock_state_init con ffff880221ea3830 sock 0 -> 1 <7>[ 1716.625053] libceph: map_request osd ffff880221ea3800 is osd0 <7>[ 1716.625055] libceph: __insert_osd ffff880221ea3800 osd0 <7>[ 1716.625059] libceph: con_open ffff880221ea3830 10.214.133.28:6800 <7>[ 1716.625061] libceph: get_osd ffff880221ea3800 1 -> 2 <7>[ 1716.625068] libceph: queue_con_delay ffff880221ea3830 0 <7>[ 1716.625069] libceph: __remove_osd_from_lru ffff880221ea3800 <7>[ 1716.625071] libceph: send_request ffff88020b496c00 tid 1 to osd0 flags 16 <7>[ 1716.625077] libceph: con_work OPENING <7>[ 1716.625079] libceph: try_read start on ffff880221ea3830 state 2 <7>[ 1716.625080] libceph: try_write start ffff880221ea3830 state 2 <7>[ 1716.625081] libceph: try_write out_kvec_bytes 0 <7>[ 1716.625083] libceph: prepare_read_banner ffff880221ea3830 <7>[ 1716.625084] libceph: try_write initiating connect on ffff880221ea3830 new state 3 <7>[ 1716.625096] libceph: connect 10.214.133.28:6800 <7>[ 1716.625097] libceph: con_sock_state_connecting con ffff880221ea3830 sock 1 -> 2 <7>[ 1716.625122] libceph: connect 10.214.133.28:6800 EINPROGRESS sk_state = 2 <7>[ 1716.625124] libceph: write_partial_kvec ffff880221ea3830 145 left <7>[ 1716.625128] libceph: write_partial_kvec ffff880221ea3830 145 left in 2 kvecs ret = 0 <7>[ 1716.625129] libceph: try_write done on ffff880221ea3830 ret 0 <7>[ 1716.625131] libceph: put_osd ffff880221ea3800 2 -> 1 <7>[ 1716.625137] libceph: get_osd ffff880221ea3800 1 -> 2 <7>[ 1716.625140] libceph: ----- ffff88020d5b0200 to osd0 42=osd_op len 128+0+9 ----- <7>[ 1716.625288] libceph: ceph_sock_state_change ffff880221ea3830 state = 3 sk_state = 1 <7>[ 1716.625290] libceph: ceph_sock_state_change TCP_ESTABLISHED <7>[ 1716.625292] libceph: con_sock_state_connected con ffff880221ea3830 sock 2 -> 3 <7>[ 1716.625294] libceph: get_osd ffff880221ea3800 2 -> 3 <7>[ 1716.625301] libceph: queue_con_delay ffff880221ea3830 0 <7>[ 1716.625327] libceph: try_read start on ffff880221ea3830 state 3 <7>[ 1716.625329] libceph: try_read tag 1 in_base_pos 0 <7>[ 1716.625330] libceph: try_read connecting <7>[ 1716.625331] libceph: read_partial_banner ffff880221ea3830 at 0 <7>[ 1716.625336] libceph: try_read done on ffff880221ea3830 ret 0 <7>[ 1716.625337] libceph: try_write start ffff880221ea3830 state 3 <7>[ 1716.625339] libceph: try_write out_kvec_bytes 145 <7>[ 1716.625340] libceph: write_partial_kvec ffff880221ea3830 145 left <7>[ 1716.625356] libceph: write_partial_kvec ffff880221ea3830 0 left in 0 kvecs ret = 1 <7>[ 1716.625357] libceph: try_write nothing else to write. <7>[ 1716.625358] libceph: try_write done on ffff880221ea3830 ret 0 <7>[ 1716.625360] libceph: put_osd ffff880221ea3800 3 -> 2 <7>[ 1716.625685] libceph: ceph_sock_data_ready on ffff880221ea3830 state = 3, queueing work <7>[ 1716.625687] libceph: get_osd ffff880221ea3800 2 -> 3 <7>[ 1716.625694] libceph: queue_con_delay ffff880221ea3830 0 <7>[ 1716.625712] libceph: ceph_sock_data_ready on ffff880221ea3830 state = 3, queueing work <7>[ 1716.625714] libceph: get_osd ffff880221ea3800 3 -> 4 <7>[ 1716.625715] libceph: queue_con_delay ffff880221ea3830 - already queued <7>[ 1716.625717] libceph: put_osd ffff880221ea3800 4 -> 3 <7>[ 1716.625730] libceph: try_read start on ffff880221ea3830 state 3 <7>[ 1716.625732] libceph: try_read tag 1 in_base_pos 0 <7>[ 1716.625733] libceph: try_read connecting <7>[ 1716.625734] libceph: read_partial_banner ffff880221ea3830 at 0 <7>[ 1716.625746] libceph: process_banner on ffff880221ea3830 <7>[ 1716.625749] libceph: prepare_write_connect ffff880221ea3830 cseq=0 gseq=2 proto=24 <7>[ 1716.625752] libceph: build_authorizer for osd ffff880221f6ca80 <7>[ 1716.625753] libceph: need len 176 <7>[ 1716.625754] libceph: buffer_new ffff880225f07cc0 <7>[ 1716.625756] libceph: th ffff88021fe713c0 secret_id 2 2 <7>[ 1716.625778] libceph: built authorizer nonce 809ed3c3ce7b59c9 len 174 <7>[ 1716.625780] libceph: prepare_read_connect ffff880221ea3830 <7>[ 1716.625781] libceph: try_read done on ffff880221ea3830 ret 0 <7>[ 1716.625783] libceph: try_write start ffff880221ea3830 state 4 <7>[ 1716.625784] libceph: try_write out_kvec_bytes 207 <7>[ 1716.625785] libceph: write_partial_kvec ffff880221ea3830 207 left <7>[ 1716.625807] libceph: write_partial_kvec ffff880221ea3830 0 left in 0 kvecs ret = 1 <7>[ 1716.625808] libceph: try_write nothing else to write. <7>[ 1716.625809] libceph: try_write done on ffff880221ea3830 ret 0 <7>[ 1716.625811] libceph: put_osd ffff880221ea3800 3 -> 2 <7>[ 1716.626099] libceph: ceph_sock_data_ready on ffff880221ea3830 state = 4, queueing work <7>[ 1716.626102] libceph: get_osd ffff880221ea3800 2 -> 3 <7>[ 1716.626109] libceph: queue_con_delay ffff880221ea3830 0 <7>[ 1716.626123] libceph: ceph_sock_data_ready on ffff880221ea3830 state = 4, queueing work <7>[ 1716.626125] libceph: get_osd ffff880221ea3800 3 -> 4 <7>[ 1716.626126] libceph: queue_con_delay ffff880221ea3830 - already queued <7>[ 1716.626128] libceph: put_osd ffff880221ea3800 4 -> 3 <7>[ 1716.626141] libceph: try_read start on ffff880221ea3830 state 4 <7>[ 1716.626143] libceph: try_read tag 1 in_base_pos 0 <7>[ 1716.626144] libceph: try_read negotiating <7>[ 1716.626145] libceph: read_partial_connect ffff880221ea3830 at 0 <7>[ 1716.626163] libceph: read_partial_connect ffff880221ea3830 tag 1, con_seq = 1, g_seq = 4 <7>[ 1716.626165] libceph: process_connect on ffff880221ea3830 tag 1 <7>[ 1716.626166] libceph: process_connect got READY gseq 4 cseq 1 (1) <7>[ 1716.626168] libceph: prepare_read_tag ffff880221ea3830 <7>[ 1716.626169] libceph: try_read start on ffff880221ea3830 state 5 <7>[ 1716.626171] libceph: try_read tag 1 in_base_pos 0 <7>[ 1716.626174] libceph: try_read done on ffff880221ea3830 ret 0 <7>[ 1716.626175] libceph: try_write start ffff880221ea3830 state 5 <7>[ 1716.626176] libceph: try_write out_kvec_bytes 0 <7>[ 1716.626179] libceph: prepare_write_message ffff88020d5b0200 seq 1 type 42 len 128+0+9 1 pgs <7>[ 1716.626182] libceph: prepare_write_message front_crc 1005476480 middle_crc 0 <7>[ 1716.626183] libceph: try_write out_kvec_bytes 182 <7>[ 1716.626185] libceph: write_partial_kvec ffff880221ea3830 182 left <7>[ 1716.626191] libceph: write_partial_kvec ffff880221ea3830 0 left in 0 kvecs ret = 1 <7>[ 1716.626193] libceph: write_partial_msg_pages ffff880221ea3830 msg ffff88020d5b0200 page 0/1 offset 0 <7>[ 1716.626197] libceph: write_partial_msg_pages ffff880221ea3830 msg ffff88020d5b0200 done <7>[ 1716.626199] libceph: prepare_write_message_footer ffff880221ea3830 <7>[ 1716.626200] libceph: write_partial_kvec ffff880221ea3830 13 left <7>[ 1716.626210] libceph: write_partial_kvec ffff880221ea3830 0 left in 0 kvecs ret = 1 <7>[ 1716.626211] libceph: try_write nothing else to write. <7>[ 1716.626212] libceph: try_write done on ffff880221ea3830 ret 0 <7>[ 1716.626214] libceph: put_osd ffff880221ea3800 3 -> 2 <7>[ 1716.627292] libceph: ceph_sock_data_ready on ffff880221ea3830 state = 5, queueing work <7>[ 1716.627294] libceph: get_osd ffff880221ea3800 2 -> 3 <7>[ 1716.627301] libceph: queue_con_delay ffff880221ea3830 0 <7>[ 1716.627315] libceph: try_read start on ffff880221ea3830 state 5 <7>[ 1716.627317] libceph: try_read tag 1 in_base_pos 0 <7>[ 1716.627322] libceph: try_read got tag 8 <7>[ 1716.627323] libceph: prepare_read_ack ffff880221ea3830 <7>[ 1716.627328] libceph: got ack for seq 1 type 42 at ffff88020d5b0200 <7>[ 1716.627329] libceph: put_osd ffff880221ea3800 3 -> 2 <7>[ 1716.627331] libceph: prepare_read_tag ffff880221ea3830 <7>[ 1716.627332] libceph: try_read start on ffff880221ea3830 state 5 <7>[ 1716.627333] libceph: try_read tag 1 in_base_pos 0 <7>[ 1716.627336] libceph: try_read got tag 7 <7>[ 1716.627337] libceph: prepare_read_message ffff880221ea3830 <7>[ 1716.627339] libceph: read_partial_message con ffff880221ea3830 msg (null) <7>[ 1716.627343] libceph: got hdr type 43 front 106 data 0 <7>[ 1716.627346] libceph: get_osd ffff880221ea3800 2 -> 3 <7>[ 1716.627348] libceph: get_reply tid 1 ffff88020d5b1000 <7>[ 1716.627350] libceph: get_osd ffff880221ea3800 3 -> 4 <7>[ 1716.627359] libceph: read_partial_message got msg ffff88020d5b1000 106 (2886744250) + 0 (0) + 0 (0) <7>[ 1716.627360] libceph: put_osd ffff880221ea3800 4 -> 3 <7>[ 1716.627364] libceph: ===== ffff88020d5b1000 1 from osd0 43=osd_opreply len 106+0 (2886744250 0 0) ===== <7>[ 1716.627366] libceph: handle_reply ffff88020d5b1000 tid 1 result -2 <7>[ 1716.627368] libceph: dropping con_filling_msg ref ffff880221ea3830 <7>[ 1716.627369] libceph: put_osd ffff880221ea3800 3 -> 2 <7>[ 1716.627370] libceph: handle_reply result -2 bytes 0 <7>[ 1716.627372] libceph: handle_reply tid 1 flags 16 <7>[ 1716.627373] libceph: __unregister_request ffff88020b496c00 tid 1 <7>[ 1716.627375] libceph: moving osd to ffff880221ea3800 lru <7>[ 1716.627376] libceph: __move_osd_to_lru ffff880221ea3800 <7>[ 1716.627377] libceph: no requests, canceling timeout <7>[ 1716.627385] libceph: req=ffff88020b496c00 req->r_linger=0 <7>[ 1716.627387] libceph: prepare_read_tag ffff880221ea3830 <7>[ 1716.627388] libceph: try_read start on ffff880221ea3830 state 5 <7>[ 1716.627389] libceph: try_read tag 1 in_base_pos 0 <7>[ 1716.627393] libceph: try_read done on ffff880221ea3830 ret 0 <7>[ 1716.627394] libceph: try_write start ffff880221ea3830 state 5 <7>[ 1716.627395] libceph: try_write out_kvec_bytes 0 <7>[ 1716.627397] libceph: prepare_write_ack ffff880221ea3830 0 -> 1 <7>[ 1716.627398] libceph: try_write out_kvec_bytes 9 <7>[ 1716.627399] libceph: write_partial_kvec ffff880221ea3830 9 left <7>[ 1716.627405] libceph: write_partial_kvec ffff880221ea3830 0 left in 0 kvecs ret = 1 <7>[ 1716.627406] libceph: try_write nothing else to write. <7>[ 1716.627408] libceph: try_write done on ffff880221ea3830 ret 0 <7>[ 1716.627409] libceph: put_osd ffff880221ea3800 2 -> 1 <7>[ 1716.627421] libceph: wait_request tid 1 result -2 <7>[ 1716.627423] rbd: reassert_ver=0 <7>[ 1716.627424] libceph: ceph_msg_put last one on ffff88020d5b0200 <7>[ 1716.627425] libceph: msg_kfree ffff88020d5b0200 <7>[ 1716.627427] libceph: ceph_msg_put last one on ffff88020d5b1000 <7>[ 1716.627428] libceph: msg_kfree ffff88020d5b1000 <7>[ 1716.627433] rbd: cls_exec returned -2 <7>[ 1716.627435] rbd: rbd_dev_image_id: rbd_req_sync_exec returned -2 <7>[ 1716.627441] rbd: rbd_do_request object_name=afs254-vicepa.rbd ofs=0 len=112 coll= (null)[0] <7>[ 1716.627446] libceph: ceph_msg_new ffff88020d5b1000 front 512 <7>[ 1716.627447] libceph: ceph_msg_new ffff88020d5b0200 front 208 <7>[ 1716.627450] libceph: mapping 0~112 osize 1073741824 fl_su 1073741824 <7>[ 1716.627451] libceph: osize 1073741824 / su 1073741824 = su_per_object 1 <7>[ 1716.627452] libceph: off 0 / su 1073741824 = bl 0 <7>[ 1716.627454] libceph: objset 0 * sc 1 = ono 0 <7>[ 1716.627455] libceph: obj extent 0~112 <7>[ 1716.627456] libceph: calc_layout bno=0 0~112 (1 pages) <7>[ 1716.627459] libceph: __register_request ffff88020b496c00 tid 2 <7>[ 1716.627461] libceph: first request, scheduling timeout <7>[ 1716.627465] libceph: map_request ffff88020b496c00 tid 2 <7>[ 1716.627467] libceph: calc_object_layout 'afs254-vicepa.rbd' pgid 2.e5a4a936 <7>[ 1716.627473] libceph: map_request tid 2 pgid 2.a936 osd3 (was osd-1) <7>[ 1716.627475] libceph: con_init ffff880221ea4030 <7>[ 1716.627477] libceph: con_sock_state_init con ffff880221ea4030 sock 0 -> 1 <7>[ 1716.627481] libceph: map_request osd ffff880221ea4000 is osd3 <7>[ 1716.627483] libceph: __insert_osd ffff880221ea4000 osd3 <7>[ 1716.627486] libceph: con_open ffff880221ea4030 10.214.133.30:6800 <7>[ 1716.627488] libceph: get_osd ffff880221ea4000 1 -> 2 <7>[ 1716.627494] libceph: queue_con_delay ffff880221ea4030 0 <7>[ 1716.627495] libceph: __remove_osd_from_lru ffff880221ea4000 <7>[ 1716.627497] libceph: send_request ffff88020b496c00 tid 2 to osd3 flags 16 <7>[ 1716.627503] libceph: con_work OPENING <7>[ 1716.627504] libceph: try_read start on ffff880221ea4030 state 2 <7>[ 1716.627505] libceph: try_write start ffff880221ea4030 state 2 <7>[ 1716.627506] libceph: try_write out_kvec_bytes 0 <7>[ 1716.627508] libceph: prepare_read_banner ffff880221ea4030 <7>[ 1716.627509] libceph: try_write initiating connect on ffff880221ea4030 new state 3 <7>[ 1716.627520] libceph: connect 10.214.133.30:6800 <7>[ 1716.627522] libceph: con_sock_state_connecting con ffff880221ea4030 sock 1 -> 2 <7>[ 1716.627543] libceph: connect 10.214.133.30:6800 EINPROGRESS sk_state = 2 <7>[ 1716.627544] libceph: write_partial_kvec ffff880221ea4030 145 left <7>[ 1716.627548] libceph: write_partial_kvec ffff880221ea4030 145 left in 2 kvecs ret = 0 <7>[ 1716.627550] libceph: try_write done on ffff880221ea4030 ret 0 <7>[ 1716.627551] libceph: put_osd ffff880221ea4000 2 -> 1 <7>[ 1716.627557] libceph: get_osd ffff880221ea4000 1 -> 2 <7>[ 1716.627560] libceph: ----- ffff88020d5b0200 to osd3 42=osd_op len 125+0+0 ----- <7>[ 1716.627733] libceph: ceph_sock_state_change ffff880221ea4030 state = 3 sk_state = 1 <7>[ 1716.627734] libceph: ceph_sock_state_change TCP_ESTABLISHED <7>[ 1716.627736] libceph: con_sock_state_connected con ffff880221ea4030 sock 2 -> 3 <7>[ 1716.627737] libceph: get_osd ffff880221ea4000 2 -> 3 <7>[ 1716.627744] libceph: queue_con_delay ffff880221ea4030 0 <7>[ 1716.627764] libceph: try_read start on ffff880221ea4030 state 3 <7>[ 1716.627766] libceph: try_read tag 1 in_base_pos 0 <7>[ 1716.627767] libceph: try_read connecting <7>[ 1716.627768] libceph: read_partial_banner ffff880221ea4030 at 0 <7>[ 1716.627773] libceph: try_read done on ffff880221ea4030 ret 0 <7>[ 1716.627774] libceph: try_write start ffff880221ea4030 state 3 <7>[ 1716.627775] libceph: try_write out_kvec_bytes 145 <7>[ 1716.627777] libceph: write_partial_kvec ffff880221ea4030 145 left <7>[ 1716.627791] libceph: write_partial_kvec ffff880221ea4030 0 left in 0 kvecs ret = 1 <7>[ 1716.627792] libceph: try_write nothing else to write. <7>[ 1716.627794] libceph: try_write done on ffff880221ea4030 ret 0 <7>[ 1716.627796] libceph: put_osd ffff880221ea4000 3 -> 2 <7>[ 1716.628089] libceph: ceph_sock_data_ready on ffff880221ea4030 state = 3, queueing work <7>[ 1716.628091] libceph: get_osd ffff880221ea4000 2 -> 3 <7>[ 1716.628097] libceph: queue_con_delay ffff880221ea4030 0 <7>[ 1716.628116] libceph: ceph_sock_data_ready on ffff880221ea4030 state = 3, queueing work <7>[ 1716.628117] libceph: get_osd ffff880221ea4000 3 -> 4 <7>[ 1716.628119] libceph: queue_con_delay ffff880221ea4030 - already queued <7>[ 1716.628120] libceph: put_osd ffff880221ea4000 4 -> 3 <7>[ 1716.628130] libceph: try_read start on ffff880221ea4030 state 3 <7>[ 1716.628131] libceph: try_read tag 1 in_base_pos 0 <7>[ 1716.628132] libceph: try_read connecting <7>[ 1716.628134] libceph: read_partial_banner ffff880221ea4030 at 0 <7>[ 1716.628145] libceph: process_banner on ffff880221ea4030 <7>[ 1716.628148] libceph: prepare_write_connect ffff880221ea4030 cseq=0 gseq=3 proto=24 <7>[ 1716.628150] libceph: build_authorizer for osd ffff8801de5bfc80 <7>[ 1716.628151] libceph: need len 176 <7>[ 1716.628153] libceph: buffer_new ffff880212d20ec0 <7>[ 1716.628154] libceph: th ffff88021fe713c0 secret_id 2 2 <7>[ 1716.628175] libceph: built authorizer nonce e781563dde079ca3 len 174 <7>[ 1716.628177] libceph: prepare_read_connect ffff880221ea4030 <7>[ 1716.628179] libceph: try_read done on ffff880221ea4030 ret 0 <7>[ 1716.628180] libceph: try_write start ffff880221ea4030 state 4 <7>[ 1716.628181] libceph: try_write out_kvec_bytes 207 <7>[ 1716.628182] libceph: write_partial_kvec ffff880221ea4030 207 left <7>[ 1716.628197] libceph: write_partial_kvec ffff880221ea4030 0 left in 0 kvecs ret = 1 <7>[ 1716.628198] libceph: try_write nothing else to write. <7>[ 1716.628199] libceph: try_write done on ffff880221ea4030 ret 0 <7>[ 1716.628201] libceph: put_osd ffff880221ea4000 3 -> 2 <7>[ 1716.628444] libceph: ceph_sock_data_ready on ffff880221ea4030 state = 4, queueing work <7>[ 1716.628446] libceph: get_osd ffff880221ea4000 2 -> 3 <7>[ 1716.628452] libceph: queue_con_delay ffff880221ea4030 0 <7>[ 1716.628465] libceph: ceph_sock_data_ready on ffff880221ea4030 state = 4, queueing work <7>[ 1716.628466] libceph: get_osd ffff880221ea4000 3 -> 4 <7>[ 1716.628468] libceph: queue_con_delay ffff880221ea4030 - already queued <7>[ 1716.628469] libceph: put_osd ffff880221ea4000 4 -> 3 <7>[ 1716.628479] libceph: try_read start on ffff880221ea4030 state 4 <7>[ 1716.628481] libceph: try_read tag 1 in_base_pos 0 <7>[ 1716.628482] libceph: try_read negotiating <7>[ 1716.628483] libceph: read_partial_connect ffff880221ea4030 at 0 <7>[ 1716.628499] libceph: read_partial_connect ffff880221ea4030 tag 1, con_seq = 1, g_seq = 7 <7>[ 1716.628501] libceph: process_connect on ffff880221ea4030 tag 1 <7>[ 1716.628502] libceph: process_connect got READY gseq 7 cseq 1 (1) <7>[ 1716.628504] libceph: prepare_read_tag ffff880221ea4030 <7>[ 1716.628505] libceph: try_read start on ffff880221ea4030 state 5 <7>[ 1716.628506] libceph: try_read tag 1 in_base_pos 0 <7>[ 1716.628510] libceph: try_read done on ffff880221ea4030 ret 0 <7>[ 1716.628511] libceph: try_write start ffff880221ea4030 state 5 <7>[ 1716.628512] libceph: try_write out_kvec_bytes 0 <7>[ 1716.628515] libceph: prepare_write_message ffff88020d5b0200 seq 1 type 42 len 125+0+0 1 pgs <7>[ 1716.628517] libceph: prepare_write_message front_crc 548947366 middle_crc 0 <7>[ 1716.628519] libceph: prepare_write_message_footer ffff880221ea4030 <7>[ 1716.628520] libceph: try_write out_kvec_bytes 192 <7>[ 1716.628521] libceph: write_partial_kvec ffff880221ea4030 192 left <7>[ 1716.628532] libceph: write_partial_kvec ffff880221ea4030 0 left in 0 kvecs ret = 1 <7>[ 1716.628533] libceph: try_write nothing else to write. <7>[ 1716.628534] libceph: try_write done on ffff880221ea4030 ret 0 <7>[ 1716.628536] libceph: put_osd ffff880221ea4000 3 -> 2 <7>[ 1716.629208] libceph: ceph_sock_data_ready on ffff880221ea4030 state = 5, queueing work <7>[ 1716.629210] libceph: get_osd ffff880221ea4000 2 -> 3 <7>[ 1716.629216] libceph: queue_con_delay ffff880221ea4030 0 <7>[ 1716.629226] libceph: try_read start on ffff880221ea4030 state 5 <7>[ 1716.629228] libceph: try_read tag 1 in_base_pos 0 <7>[ 1716.629234] libceph: try_read got tag 8 <7>[ 1716.629235] libceph: prepare_read_ack ffff880221ea4030 <7>[ 1716.629239] libceph: got ack for seq 1 type 42 at ffff88020d5b0200 <7>[ 1716.629241] libceph: put_osd ffff880221ea4000 3 -> 2 <7>[ 1716.629242] libceph: prepare_read_tag ffff880221ea4030 <7>[ 1716.629243] libceph: try_read start on ffff880221ea4030 state 5 <7>[ 1716.629245] libceph: try_read tag 1 in_base_pos 0 <7>[ 1716.629248] libceph: try_read got tag 7 <7>[ 1716.629249] libceph: prepare_read_message ffff880221ea4030 <7>[ 1716.629250] libceph: read_partial_message con ffff880221ea4030 msg (null) <7>[ 1716.629254] libceph: got hdr type 43 front 103 data 0 <7>[ 1716.629257] libceph: get_osd ffff880221ea4000 2 -> 3 <7>[ 1716.629258] libceph: get_reply tid 2 ffff88020d5b1000 <7>[ 1716.629261] libceph: get_osd ffff880221ea4000 3 -> 4 <7>[ 1716.629269] libceph: read_partial_message got msg ffff88020d5b1000 103 (1679879841) + 0 (0) + 0 (0) <7>[ 1716.629270] libceph: put_osd ffff880221ea4000 4 -> 3 <7>[ 1716.629274] libceph: ===== ffff88020d5b1000 1 from osd3 43=osd_opreply len 103+0 (1679879841 0 0) ===== <7>[ 1716.629276] libceph: handle_reply ffff88020d5b1000 tid 2 result -2 <7>[ 1716.629277] libceph: dropping con_filling_msg ref ffff880221ea4030 <7>[ 1716.629279] libceph: put_osd ffff880221ea4000 3 -> 2 <7>[ 1716.629280] libceph: handle_reply result -2 bytes 0 <7>[ 1716.629281] libceph: handle_reply tid 2 flags 16 <7>[ 1716.629283] libceph: __unregister_request ffff88020b496c00 tid 2 <7>[ 1716.629284] libceph: moving osd to ffff880221ea4000 lru <7>[ 1716.629285] libceph: __move_osd_to_lru ffff880221ea4000 <7>[ 1716.629286] libceph: no requests, canceling timeout <7>[ 1716.629295] libceph: req=ffff88020b496c00 req->r_linger=0 <7>[ 1716.629296] libceph: prepare_read_tag ffff880221ea4030 <7>[ 1716.629298] libceph: try_read start on ffff880221ea4030 state 5 <7>[ 1716.629299] libceph: try_read tag 1 in_base_pos 0 <7>[ 1716.629303] libceph: try_read done on ffff880221ea4030 ret 0 <7>[ 1716.629304] libceph: try_write start ffff880221ea4030 state 5 <7>[ 1716.629305] libceph: try_write out_kvec_bytes 0 <7>[ 1716.629307] libceph: prepare_write_ack ffff880221ea4030 0 -> 1 <7>[ 1716.629308] libceph: try_write out_kvec_bytes 9 <7>[ 1716.629309] libceph: write_partial_kvec ffff880221ea4030 9 left <7>[ 1716.629315] libceph: write_partial_kvec ffff880221ea4030 0 left in 0 kvecs ret = 1 <7>[ 1716.629316] libceph: try_write nothing else to write. <7>[ 1716.629317] libceph: try_write done on ffff880221ea4030 ret 0 <7>[ 1716.629319] libceph: put_osd ffff880221ea4000 2 -> 1 <7>[ 1716.629365] libceph: wait_request tid 2 result -2 <7>[ 1716.629366] rbd: reassert_ver=0 <7>[ 1716.629368] libceph: ceph_msg_put last one on ffff88020d5b0200 <7>[ 1716.629369] libceph: msg_kfree ffff88020d5b0200 <7>[ 1716.629371] libceph: ceph_msg_put last one on ffff88020d5b1000 <7>[ 1716.629373] libceph: msg_kfree ffff88020d5b1000 <7>[ 1716.629378] rbd: probe failed, returning -2 <7>[ 1716.629379] rbd: rbd_release_client ffff88020beecea0 <7>[ 1716.629381] libceph: destroy_client ffff880224d20000 <7>[ 1716.629734] libceph: osdmap_destroy ffff88020d2bb880 <7>[ 1716.629745] libceph: remove_all_osds ffff880224d20960 <7>[ 1716.629748] libceph: __remove_osd ffff880221ea3800 <7>[ 1716.629758] libceph: con_close ffff880221ea3830 peer 10.214.133.28:6800 <7>[ 1716.629759] libceph: reset_connection ffff880221ea3830 <7>[ 1716.629761] libceph: con_close_socket on ffff880221ea3830 sock ffff88020cfe4400 <7>[ 1716.629793] libceph: ceph_sock_state_change ffff880221ea3830 state = 1 sk_state = 4 <7>[ 1716.629814] libceph: con_sock_state_closed con ffff880221ea3830 sock 3 -> 1 <7>[ 1716.629817] libceph: put_osd ffff880221ea3800 1 -> 0 <7>[ 1716.629818] libceph: buffer_release ffff880225f07cc0 <7>[ 1716.629821] libceph: __remove_osd ffff880221ea4000 <7>[ 1716.629824] libceph: con_close ffff880221ea4030 peer 10.214.133.30:6800 <7>[ 1716.629825] libceph: reset_connection ffff880221ea4030 <7>[ 1716.629827] libceph: con_close_socket on ffff880221ea4030 sock ffff88020cfe4800 <7>[ 1716.629835] libceph: ceph_sock_state_change ffff880221ea4030 state = 1 sk_state = 4 <7>[ 1716.629844] libceph: con_sock_state_closed con ffff880221ea4030 sock 3 -> 1 <7>[ 1716.629846] libceph: put_osd ffff880221ea4000 1 -> 0 <7>[ 1716.629847] libceph: buffer_release ffff880212d20ec0 <7>[ 1716.629854] libceph: msgpool osd_op destroy <7>[ 1716.629855] libceph: msgpool_release osd_op ffff88020d5b0800 <7>[ 1716.629856] libceph: ceph_msg_put last one on ffff88020d5b0800 <7>[ 1716.629857] libceph: msg_kfree ffff88020d5b0800 <7>[ 1716.629859] libceph: msgpool_release osd_op ffff88020d5b0600 <7>[ 1716.629860] libceph: ceph_msg_put last one on ffff88020d5b0600 <7>[ 1716.629861] libceph: msg_kfree ffff88020d5b0600 <7>[ 1716.629863] libceph: msgpool_release osd_op ffff88020d5b1b00 <7>[ 1716.629864] libceph: ceph_msg_put last one on ffff88020d5b1b00 <7>[ 1716.629865] libceph: msg_kfree ffff88020d5b1b00 <7>[ 1716.629867] libceph: msgpool_release osd_op ffff88020d5b1700 <7>[ 1716.629868] libceph: ceph_msg_put last one on ffff88020d5b1700 <7>[ 1716.629869] libceph: msg_kfree ffff88020d5b1700 <7>[ 1716.629871] libceph: msgpool_release osd_op ffff88020d5b1f00 <7>[ 1716.629872] libceph: ceph_msg_put last one on ffff88020d5b1f00 <7>[ 1716.629873] libceph: msg_kfree ffff88020d5b1f00 <7>[ 1716.629875] libceph: msgpool_release osd_op ffff88020d5b1a00 <7>[ 1716.629877] libceph: ceph_msg_put last one on ffff88020d5b1a00 <7>[ 1716.629878] libceph: msg_kfree ffff88020d5b1a00 <7>[ 1716.629881] libceph: msgpool_release osd_op ffff88020d5b1900 <7>[ 1716.629882] libceph: ceph_msg_put last one on ffff88020d5b1900 <7>[ 1716.629884] libceph: msg_kfree ffff88020d5b1900 <7>[ 1716.629901] libceph: msgpool_release osd_op ffff88020d5b0e00 <7>[ 1716.629902] libceph: ceph_msg_put last one on ffff88020d5b0e00 <7>[ 1716.629903] libceph: msg_kfree ffff88020d5b0e00 <7>[ 1716.629905] libceph: msgpool_release osd_op ffff88020d5b0f00 <7>[ 1716.629906] libceph: ceph_msg_put last one on ffff88020d5b0f00 <7>[ 1716.629907] libceph: msg_kfree ffff88020d5b0f00 <7>[ 1716.629908] libceph: msgpool_release osd_op ffff88020d5b0400 <7>[ 1716.629909] libceph: ceph_msg_put last one on ffff88020d5b0400 <7>[ 1716.629910] libceph: msg_kfree ffff88020d5b0400 <7>[ 1716.629913] libceph: msgpool osd_op_reply destroy <7>[ 1716.629914] libceph: msgpool_release osd_op_reply ffff88020d5b1c00 <7>[ 1716.629915] libceph: ceph_msg_put last one on ffff88020d5b1c00 <7>[ 1716.629916] libceph: msg_kfree ffff88020d5b1c00 <7>[ 1716.629923] libceph: msgpool_release osd_op_reply ffff88020d5b0a00 <7>[ 1716.629925] libceph: ceph_msg_put last one on ffff88020d5b0a00 <7>[ 1716.629926] libceph: msg_kfree ffff88020d5b0a00 <7>[ 1716.629927] libceph: msgpool_release osd_op_reply ffff88020d5b1e00 <7>[ 1716.629928] libceph: ceph_msg_put last one on ffff88020d5b1e00 <7>[ 1716.629930] libceph: msg_kfree ffff88020d5b1e00 <7>[ 1716.629931] libceph: msgpool_release osd_op_reply ffff88020d5b0000 <7>[ 1716.629932] libceph: ceph_msg_put last one on ffff88020d5b0000 <7>[ 1716.629933] libceph: msg_kfree ffff88020d5b0000 <7>[ 1716.629935] libceph: msgpool_release osd_op_reply ffff88020d5b0900 <7>[ 1716.629936] libceph: ceph_msg_put last one on ffff88020d5b0900 <7>[ 1716.629937] libceph: msg_kfree ffff88020d5b0900 <7>[ 1716.629939] libceph: msgpool_release osd_op_reply ffff88020d5b0b00 <7>[ 1716.629940] libceph: ceph_msg_put last one on ffff88020d5b0b00 <7>[ 1716.629941] libceph: msg_kfree ffff88020d5b0b00 <7>[ 1716.629943] libceph: msgpool_release osd_op_reply ffff88020d5b0700 <7>[ 1716.629944] libceph: ceph_msg_put last one on ffff88020d5b0700 <7>[ 1716.629945] libceph: msg_kfree ffff88020d5b0700 <7>[ 1716.629946] libceph: msgpool_release osd_op_reply ffff88020d5b1600 <7>[ 1716.629947] libceph: ceph_msg_put last one on ffff88020d5b1600 <7>[ 1716.629948] libceph: msg_kfree ffff88020d5b1600 <7>[ 1716.629950] libceph: msgpool_release osd_op_reply ffff88020d5b1300 <7>[ 1716.629951] libceph: ceph_msg_put last one on ffff88020d5b1300 <7>[ 1716.629952] libceph: msg_kfree ffff88020d5b1300 <7>[ 1716.629954] libceph: msgpool_release osd_op_reply ffff88020d5b1400 <7>[ 1716.629955] libceph: ceph_msg_put last one on ffff88020d5b1400 <7>[ 1716.629956] libceph: msg_kfree ffff88020d5b1400 <7>[ 1716.629958] libceph: stop <7>[ 1716.631767] libceph: handle_subscribe_ack after 300 seconds <7>[ 1716.631783] libceph: prepare_read_tag ffff880224d20410 <7>[ 1716.631787] libceph: __close_session closing mon0 <7>[ 1716.631790] libceph: ceph_msg_revoke_incoming msg ffff88020d5b1800 null con <7>[ 1716.631792] libceph: ceph_msg_revoke_incoming msg ffff88020d5b1200 null con <7>[ 1716.631803] libceph: try_read start on ffff880224d20410 state 5 <7>[ 1716.631807] libceph: try_read tag 1 in_base_pos 0 <7>[ 1716.631817] libceph: try_read done on ffff880224d20410 ret 0 <7>[ 1716.631822] libceph: try_write start ffff880224d20410 state 5 <7>[ 1716.631826] libceph: try_write out_kvec_bytes 0 <7>[ 1716.631831] libceph: prepare_write_ack ffff880224d20410 4 -> 7 <7>[ 1716.631835] libceph: try_write out_kvec_bytes 9 <7>[ 1716.631839] libceph: write_partial_kvec ffff880224d20410 9 left <7>[ 1716.631852] libceph: write_partial_kvec ffff880224d20410 0 left in 0 kvecs ret = 1 <7>[ 1716.631856] libceph: try_write nothing else to write. <7>[ 1716.631861] libceph: try_write done on ffff880224d20410 ret 0 <7>[ 1716.631873] libceph: try_read start on ffff880224d20410 state 5 <7>[ 1716.631878] libceph: try_read tag 1 in_base_pos 0 <7>[ 1716.631886] libceph: try_read done on ffff880224d20410 ret 0 <7>[ 1716.631896] libceph: try_write start ffff880224d20410 state 5 <7>[ 1716.631903] libceph: try_write out_kvec_bytes 0 <7>[ 1716.631912] libceph: try_write nothing else to write. <7>[ 1716.631920] libceph: try_write done on ffff880224d20410 ret 0 <7>[ 1716.631934] libceph: con_close ffff880224d20410 peer 10.214.133.28:6789 <7>[ 1716.631941] libceph: reset_connection ffff880224d20410 <7>[ 1716.631948] libceph: con_close_socket on ffff880224d20410 sock ffff88020cfe3000 <7>[ 1716.631981] libceph: ceph_sock_state_change ffff880224d20410 state = 1 sk_state = 4 <7>[ 1716.632006] libceph: con_sock_state_closed con ffff880224d20410 sock 3 -> 1 <7>[ 1716.632010] libceph: auth_reset ffff880212d20f80 <7>[ 1716.632012] libceph: reset <7>[ 1716.632060] libceph: auth_destroy ffff880212d20f80 <7>[ 1716.632068] libceph: ceph_x_destroy ffff880212d20f80 <7>[ 1716.632071] libceph: remove_ticket_handler ffff88021fe71e40 1 <7>[ 1716.632074] libceph: buffer_release ffff88020b2c6e80 <7>[ 1716.632077] libceph: remove_ticket_handler ffff88021fe71420 2 <7>[ 1716.632080] libceph: buffer_release ffff88020b2c6940 <7>[ 1716.632083] libceph: remove_ticket_handler ffff88021fe713c0 4 <7>[ 1716.632086] libceph: buffer_release ffff88020b2c6100 <7>[ 1716.632089] libceph: remove_ticket_handler ffff88021fe71960 32 <7>[ 1716.632092] libceph: buffer_release ffff88020b2c67c0 <7>[ 1716.632095] libceph: buffer_release ffff88020b2c65c0 <7>[ 1716.632098] libceph: ceph_msg_put last one on ffff88020d5b0300 <7>[ 1716.632100] libceph: msg_kfree ffff88020d5b0300 <7>[ 1716.632103] libceph: ceph_msg_put last one on ffff88020d5b1800 <7>[ 1716.632106] libceph: msg_kfree ffff88020d5b1800 <7>[ 1716.632109] libceph: ceph_msg_put last one on ffff88020d5b1100 <7>[ 1716.632111] libceph: msg_kfree ffff88020d5b1100 <7>[ 1716.632114] libceph: ceph_msg_put last one on ffff88020d5b1200 <7>[ 1716.632117] libceph: msg_kfree ffff88020d5b1200 <7>[ 1716.632120] libceph: ceph_debugfs_client_cleanup ffff880224d20000 <7>[ 1716.632184] libceph: destroy_options ffff88020d5b1500 <7>[ 1716.632192] libceph: destroy_client ffff880224d20000 done <7>[ 1716.632199] rbd: Error adding device 10.214.133.28:6789,10.214.133.30:6789,10.214.133.28:6790 name=admin,key=client.admin rba <7>[ 1721.083296] libceph: osds timeout <7>[ 1721.083301] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1733.262155] libceph: monc delayed_work <7>[ 1733.262162] libceph: con_keepalive ffff8802214c4410 <7>[ 1733.262168] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 1733.262173] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 1733.262176] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 1733.262179] libceph: __send_subscribe sub_sent=0 exp=0 want_osd=0 <7>[ 1733.262182] libceph: __schedule_delayed after 2000 <7>[ 1733.262188] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1733.262191] libceph: try_read tag 1 in_base_pos 0 <7>[ 1733.262198] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 1733.262201] libceph: try_write start ffff8802214c4410 state 5 <7>[ 1733.262203] libceph: try_write out_kvec_bytes 0 <7>[ 1733.262206] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 1733.262208] libceph: try_write out_kvec_bytes 1 <7>[ 1733.262211] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 1733.262220] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1733.262223] libceph: try_write nothing else to write. <7>[ 1733.262225] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 1736.057289] libceph: osds timeout <7>[ 1736.057295] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1751.031281] libceph: osds timeout <7>[ 1751.031286] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1753.267398] libceph: monc delayed_work <7>[ 1753.267404] libceph: con_keepalive ffff8802214c4410 <7>[ 1753.267411] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 1753.267415] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 1753.267418] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 1753.267421] libceph: __send_subscribe sub_sent=0 exp=0 want_osd=0 <7>[ 1753.267424] libceph: __schedule_delayed after 2000 <7>[ 1753.267431] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1753.267434] libceph: try_read tag 1 in_base_pos 0 <7>[ 1753.267440] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 1753.267443] libceph: try_write start ffff8802214c4410 state 5 <7>[ 1753.267445] libceph: try_write out_kvec_bytes 0 <7>[ 1753.267448] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 1753.267450] libceph: try_write out_kvec_bytes 1 <7>[ 1753.267453] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 1753.267461] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1753.267464] libceph: try_write nothing else to write. <7>[ 1753.267466] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 1766.005269] libceph: osds timeout <7>[ 1766.005275] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1773.272657] libceph: monc delayed_work <7>[ 1773.272663] libceph: con_keepalive ffff8802214c4410 <7>[ 1773.272669] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 1773.272673] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 1773.272676] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 1773.272679] libceph: __send_subscribe sub_sent=0 exp=0 want_osd=0 <7>[ 1773.272682] libceph: __schedule_delayed after 2000 <7>[ 1773.272688] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1773.272690] libceph: try_read tag 1 in_base_pos 0 <7>[ 1773.272697] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 1773.272699] libceph: try_write start ffff8802214c4410 state 5 <7>[ 1773.272702] libceph: try_write out_kvec_bytes 0 <7>[ 1773.272704] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 1773.272706] libceph: try_write out_kvec_bytes 1 <7>[ 1773.272709] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 1773.272719] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1773.272721] libceph: try_write nothing else to write. <7>[ 1773.272724] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 1780.979258] libceph: osds timeout <7>[ 1780.979264] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1793.277888] libceph: monc delayed_work <7>[ 1793.277894] libceph: con_keepalive ffff8802214c4410 <7>[ 1793.277900] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 1793.277904] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 1793.277907] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 1793.277910] libceph: __send_subscribe sub_sent=0 exp=0 want_osd=0 <7>[ 1793.277913] libceph: __schedule_delayed after 2000 <7>[ 1793.277919] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1793.277921] libceph: try_read tag 1 in_base_pos 0 <7>[ 1793.277928] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 1793.277931] libceph: try_write start ffff8802214c4410 state 5 <7>[ 1793.277933] libceph: try_write out_kvec_bytes 0 <7>[ 1793.277935] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 1793.277938] libceph: try_write out_kvec_bytes 1 <7>[ 1793.277941] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 1793.277949] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1793.277952] libceph: try_write nothing else to write. <7>[ 1793.277954] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 1795.953246] libceph: osds timeout <7>[ 1795.953252] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1810.927240] libceph: osds timeout <7>[ 1810.927245] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1813.283146] libceph: monc delayed_work <7>[ 1813.283152] libceph: con_keepalive ffff8802214c4410 <7>[ 1813.283158] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 1813.283163] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 1813.283166] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 1813.283169] libceph: __send_subscribe sub_sent=0 exp=1 want_osd=0 <7>[ 1813.283175] libceph: ----- ffff88020d5b5400 to mon1 15=mon_subscribe len 31+0+0 ----- <7>[ 1813.283178] libceph: __schedule_delayed after 1000 <7>[ 1813.283185] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1813.283187] libceph: try_read tag 1 in_base_pos 0 <7>[ 1813.283194] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 1813.283196] libceph: try_write start ffff8802214c4410 state 5 <7>[ 1813.283199] libceph: try_write out_kvec_bytes 0 <7>[ 1813.283202] libceph: prepare_write_message ffff88020d5b5400 seq 14 type 15 len 31+0+0 0 pgs <7>[ 1813.283207] libceph: prepare_write_message front_crc 2650579135 middle_crc 0 <7>[ 1813.283210] libceph: prepare_write_message_footer ffff8802214c4410 <7>[ 1813.283212] libceph: try_write out_kvec_bytes 98 <7>[ 1813.283215] libceph: write_partial_kvec ffff8802214c4410 98 left <7>[ 1813.283234] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1813.283237] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 1813.283239] libceph: try_write out_kvec_bytes 1 <7>[ 1813.283242] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 1813.283248] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1813.283250] libceph: try_write nothing else to write. <7>[ 1813.283253] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 1813.283698] libceph: ceph_sock_data_ready on ffff8802214c4410 state = 5, queueing work <7>[ 1813.283709] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 1813.283725] libceph: ceph_sock_data_ready on ffff8802214c4410 state = 5, queueing work <7>[ 1813.283729] libceph: queue_con_delay ffff8802214c4410 - already queued <7>[ 1813.283746] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1813.283749] libceph: try_read tag 1 in_base_pos 0 <7>[ 1813.283755] libceph: try_read got tag 8 <7>[ 1813.283758] libceph: prepare_read_ack ffff8802214c4410 <7>[ 1813.283763] libceph: got ack for seq 14 type 15 at ffff88020d5b5400 <7>[ 1813.283766] libceph: prepare_read_tag ffff8802214c4410 <7>[ 1813.283768] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1813.283771] libceph: try_read tag 1 in_base_pos 0 <7>[ 1813.283775] libceph: try_read got tag 7 <7>[ 1813.283778] libceph: prepare_read_message ffff8802214c4410 <7>[ 1813.283781] libceph: read_partial_message con ffff8802214c4410 msg (null) <7>[ 1813.283786] libceph: got hdr type 4 front 481 data 0 <7>[ 1813.283791] libceph: ceph_msg_new ffff88020d35c400 front 481 <7>[ 1813.283801] libceph: read_partial_message got msg ffff88020d35c400 481 (2132872380) + 0 (0) + 0 (0) <7>[ 1813.283806] libceph: ===== ffff88020d35c400 26 from mon1 4=mon_map len 481+0 (2132872380 0 0) ===== <7>[ 1813.283810] libceph: have_debugfs_info fsid 1 globalid 4132 <7>[ 1813.283812] libceph: handle_monmap <7>[ 1813.283815] libceph: monmap_decode ffff88020d752c04 ffff88020d752de1 len 477 <7>[ 1813.283818] libceph: monmap_decode epoch 1, num_mon 3 <7>[ 1813.283822] libceph: monmap_decode mon0 is 10.214.133.28:6789 <7>[ 1813.283825] libceph: monmap_decode mon1 is 10.214.133.30:6789 <7>[ 1813.283828] libceph: monmap_decode mon2 is 10.214.133.28:6790 <7>[ 1813.283832] libceph: ceph_msg_put last one on ffff88020d35c400 <7>[ 1813.283835] libceph: msg_kfree ffff88020d35c400 <7>[ 1813.283838] libceph: prepare_read_tag ffff8802214c4410 <7>[ 1813.283841] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1813.283843] libceph: try_read tag 1 in_base_pos 0 <7>[ 1813.283848] libceph: try_read got tag 7 <7>[ 1813.283850] libceph: prepare_read_message ffff8802214c4410 <7>[ 1813.283853] libceph: read_partial_message con ffff8802214c4410 msg (null) <7>[ 1813.283857] libceph: got hdr type 16 front 20 data 0 <7>[ 1813.283877] libceph: read_partial_message got msg ffff88020d5b4500 20 (473278946) + 0 (0) + 0 (0) <7>[ 1813.283882] libceph: ===== ffff88020d5b4500 27 from mon1 16=mon_subscribe_ack len 20+0 (473278946 0 0) ===== <7>[ 1813.283885] libceph: handle_subscribe_ack after 300 seconds <7>[ 1813.283888] libceph: prepare_read_tag ffff8802214c4410 <7>[ 1813.283891] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1813.283893] libceph: try_read tag 1 in_base_pos 0 <7>[ 1813.283898] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 1813.283901] libceph: try_write start ffff8802214c4410 state 5 <7>[ 1813.283903] libceph: try_write out_kvec_bytes 0 <7>[ 1813.283906] libceph: prepare_write_ack ffff8802214c4410 25 -> 27 <7>[ 1813.283909] libceph: try_write out_kvec_bytes 9 <7>[ 1813.283912] libceph: write_partial_kvec ffff8802214c4410 9 left <7>[ 1813.283919] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1813.283921] libceph: try_write nothing else to write. <7>[ 1813.283924] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 1823.285777] libceph: monc delayed_work <7>[ 1823.285783] libceph: con_keepalive ffff8802214c4410 <7>[ 1823.285789] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 1823.285793] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 1823.285796] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 1823.285799] libceph: __send_subscribe sub_sent=0 exp=0 want_osd=0 <7>[ 1823.285802] libceph: __schedule_delayed after 2000 <7>[ 1823.285808] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1823.285811] libceph: try_read tag 1 in_base_pos 0 <7>[ 1823.285818] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 1823.285820] libceph: try_write start ffff8802214c4410 state 5 <7>[ 1823.285823] libceph: try_write out_kvec_bytes 0 <7>[ 1823.285825] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 1823.285827] libceph: try_write out_kvec_bytes 1 <7>[ 1823.285830] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 1823.285839] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1823.285841] libceph: try_write nothing else to write. <7>[ 1823.285844] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 1825.901229] libceph: osds timeout <7>[ 1825.901235] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1840.875219] libceph: osds timeout <7>[ 1840.875225] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1843.310997] libceph: monc delayed_work <7>[ 1843.311002] libceph: con_keepalive ffff8802214c4410 <7>[ 1843.311008] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 1843.311013] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 1843.311016] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 1843.311019] libceph: __send_subscribe sub_sent=0 exp=0 want_osd=0 <7>[ 1843.311021] libceph: __schedule_delayed after 2000 <7>[ 1843.311027] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1843.311030] libceph: try_read tag 1 in_base_pos 0 <7>[ 1843.311036] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 1843.311039] libceph: try_write start ffff8802214c4410 state 5 <7>[ 1843.311041] libceph: try_write out_kvec_bytes 0 <7>[ 1843.311044] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 1843.311046] libceph: try_write out_kvec_bytes 1 <7>[ 1843.311049] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 1843.311058] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1843.311060] libceph: try_write nothing else to write. <7>[ 1843.311063] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 1855.849207] libceph: osds timeout <7>[ 1855.849213] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1863.316247] libceph: monc delayed_work <7>[ 1863.316253] libceph: con_keepalive ffff8802214c4410 <7>[ 1863.316259] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 1863.316263] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 1863.316266] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 1863.316269] libceph: __send_subscribe sub_sent=0 exp=0 want_osd=0 <7>[ 1863.316272] libceph: __schedule_delayed after 2000 <7>[ 1863.316278] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1863.316280] libceph: try_read tag 1 in_base_pos 0 <7>[ 1863.316287] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 1863.316289] libceph: try_write start ffff8802214c4410 state 5 <7>[ 1863.316291] libceph: try_write out_kvec_bytes 0 <7>[ 1863.316294] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 1863.316296] libceph: try_write out_kvec_bytes 1 <7>[ 1863.316299] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 1863.316308] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1863.316310] libceph: try_write nothing else to write. <7>[ 1863.316313] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 1870.823198] libceph: osds timeout <7>[ 1870.823203] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1883.321494] libceph: monc delayed_work <7>[ 1883.321501] libceph: con_keepalive ffff8802214c4410 <7>[ 1883.321507] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 1883.321511] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 1883.321514] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 1883.321518] libceph: __send_subscribe sub_sent=0 exp=0 want_osd=0 <7>[ 1883.321520] libceph: __schedule_delayed after 2000 <7>[ 1883.321527] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1883.321530] libceph: try_read tag 1 in_base_pos 0 <7>[ 1883.321536] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 1883.321539] libceph: try_write start ffff8802214c4410 state 5 <7>[ 1883.321541] libceph: try_write out_kvec_bytes 0 <7>[ 1883.321544] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 1883.321546] libceph: try_write out_kvec_bytes 1 <7>[ 1883.321549] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 1883.321558] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1883.321561] libceph: try_write nothing else to write. <7>[ 1883.321563] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 1885.797188] libceph: osds timeout <7>[ 1885.797194] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1900.771178] libceph: osds timeout <7>[ 1900.771183] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1903.326747] libceph: monc delayed_work <7>[ 1903.326753] libceph: con_keepalive ffff8802214c4410 <7>[ 1903.326759] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 1903.326763] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 1903.326766] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 1903.326769] libceph: __send_subscribe sub_sent=0 exp=0 want_osd=0 <7>[ 1903.326772] libceph: __schedule_delayed after 2000 <7>[ 1903.326778] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1903.326781] libceph: try_read tag 1 in_base_pos 0 <7>[ 1903.326787] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 1903.326790] libceph: try_write start ffff8802214c4410 state 5 <7>[ 1903.326792] libceph: try_write out_kvec_bytes 0 <7>[ 1903.326794] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 1903.326797] libceph: try_write out_kvec_bytes 1 <7>[ 1903.326800] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 1903.326809] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1903.326811] libceph: try_write nothing else to write. <7>[ 1903.326814] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 1915.745164] libceph: osds timeout <7>[ 1915.745170] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1923.331998] libceph: monc delayed_work <7>[ 1923.332004] libceph: con_keepalive ffff8802214c4410 <7>[ 1923.332010] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 1923.332015] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 1923.332018] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 1923.332021] libceph: __send_subscribe sub_sent=0 exp=0 want_osd=0 <7>[ 1923.332023] libceph: __schedule_delayed after 2000 <7>[ 1923.332030] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1923.332032] libceph: try_read tag 1 in_base_pos 0 <7>[ 1923.332039] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 1923.332042] libceph: try_write start ffff8802214c4410 state 5 <7>[ 1923.332044] libceph: try_write out_kvec_bytes 0 <7>[ 1923.332047] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 1923.332049] libceph: try_write out_kvec_bytes 1 <7>[ 1923.332052] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 1923.332061] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1923.332064] libceph: try_write nothing else to write. <7>[ 1923.332066] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 1930.719153] libceph: osds timeout <7>[ 1930.719159] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1943.337248] libceph: monc delayed_work <7>[ 1943.337254] libceph: con_keepalive ffff8802214c4410 <7>[ 1943.337260] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 1943.337264] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 1943.337268] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 1943.337270] libceph: __send_subscribe sub_sent=0 exp=0 want_osd=0 <7>[ 1943.337273] libceph: __schedule_delayed after 2000 <7>[ 1943.337279] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1943.337282] libceph: try_read tag 1 in_base_pos 0 <7>[ 1943.337288] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 1943.337291] libceph: try_write start ffff8802214c4410 state 5 <7>[ 1943.337293] libceph: try_write out_kvec_bytes 0 <7>[ 1943.337296] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 1943.337298] libceph: try_write out_kvec_bytes 1 <7>[ 1943.337301] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 1943.337310] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1943.337312] libceph: try_write nothing else to write. <7>[ 1943.337315] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 1945.693140] libceph: osds timeout <7>[ 1945.693146] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1960.667133] libceph: osds timeout <7>[ 1960.667139] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1963.342498] libceph: monc delayed_work <7>[ 1963.342504] libceph: con_keepalive ffff8802214c4410 <7>[ 1963.342510] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 1963.342514] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 1963.342518] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 1963.342520] libceph: __send_subscribe sub_sent=0 exp=1 want_osd=0 <7>[ 1963.342526] libceph: ----- ffff88020d5b5400 to mon1 15=mon_subscribe len 31+0+0 ----- <7>[ 1963.342529] libceph: __schedule_delayed after 1000 <7>[ 1963.342535] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1963.342538] libceph: try_read tag 1 in_base_pos 0 <7>[ 1963.342545] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 1963.342547] libceph: try_write start ffff8802214c4410 state 5 <7>[ 1963.342550] libceph: try_write out_kvec_bytes 0 <7>[ 1963.342553] libceph: prepare_write_message ffff88020d5b5400 seq 15 type 15 len 31+0+0 0 pgs <7>[ 1963.342558] libceph: prepare_write_message front_crc 2650579135 middle_crc 0 <7>[ 1963.342561] libceph: prepare_write_message_footer ffff8802214c4410 <7>[ 1963.342563] libceph: try_write out_kvec_bytes 98 <7>[ 1963.342566] libceph: write_partial_kvec ffff8802214c4410 98 left <7>[ 1963.342584] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1963.342587] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 1963.342589] libceph: try_write out_kvec_bytes 1 <7>[ 1963.342592] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 1963.342598] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1963.342600] libceph: try_write nothing else to write. <7>[ 1963.342603] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 1963.343050] libceph: ceph_sock_data_ready on ffff8802214c4410 state = 5, queueing work <7>[ 1963.343061] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 1963.343077] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1963.343081] libceph: try_read tag 1 in_base_pos 0 <7>[ 1963.343087] libceph: try_read got tag 8 <7>[ 1963.343090] libceph: prepare_read_ack ffff8802214c4410 <7>[ 1963.343095] libceph: got ack for seq 15 type 15 at ffff88020d5b5400 <7>[ 1963.343098] libceph: prepare_read_tag ffff8802214c4410 <7>[ 1963.343100] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1963.343103] libceph: try_read tag 1 in_base_pos 0 <7>[ 1963.343107] libceph: try_read got tag 7 <7>[ 1963.343109] libceph: prepare_read_message ffff8802214c4410 <7>[ 1963.343112] libceph: read_partial_message con ffff8802214c4410 msg (null) <7>[ 1963.343118] libceph: got hdr type 4 front 481 data 0 <7>[ 1963.343122] libceph: ceph_msg_new ffff88020d35d100 front 481 <7>[ 1963.343133] libceph: read_partial_message got msg ffff88020d35d100 481 (2132872380) + 0 (0) + 0 (0) <7>[ 1963.343138] libceph: ===== ffff88020d35d100 28 from mon1 4=mon_map len 481+0 (2132872380 0 0) ===== <7>[ 1963.343154] libceph: ceph_sock_data_ready on ffff8802214c4410 state = 5, queueing work <7>[ 1963.343158] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 1963.343164] libceph: have_debugfs_info fsid 1 globalid 4132 <7>[ 1963.343166] libceph: handle_monmap <7>[ 1963.343169] libceph: monmap_decode ffff88020d752c04 ffff88020d752de1 len 477 <7>[ 1963.343172] libceph: monmap_decode epoch 1, num_mon 3 <7>[ 1963.343176] libceph: monmap_decode mon0 is 10.214.133.28:6789 <7>[ 1963.343179] libceph: monmap_decode mon1 is 10.214.133.30:6789 <7>[ 1963.343182] libceph: monmap_decode mon2 is 10.214.133.28:6790 <7>[ 1963.343186] libceph: ceph_msg_put last one on ffff88020d35d100 <7>[ 1963.343188] libceph: msg_kfree ffff88020d35d100 <7>[ 1963.343192] libceph: prepare_read_tag ffff8802214c4410 <7>[ 1963.343195] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1963.343197] libceph: try_read tag 1 in_base_pos 0 <7>[ 1963.343202] libceph: try_read got tag 7 <7>[ 1963.343205] libceph: prepare_read_message ffff8802214c4410 <7>[ 1963.343207] libceph: read_partial_message con ffff8802214c4410 msg (null) <7>[ 1963.343212] libceph: got hdr type 16 front 20 data 0 <7>[ 1963.343231] libceph: read_partial_message got msg ffff88020d5b4500 20 (473278946) + 0 (0) + 0 (0) <7>[ 1963.343236] libceph: ===== ffff88020d5b4500 29 from mon1 16=mon_subscribe_ack len 20+0 (473278946 0 0) ===== <7>[ 1963.343239] libceph: handle_subscribe_ack after 300 seconds <7>[ 1963.343242] libceph: prepare_read_tag ffff8802214c4410 <7>[ 1963.343245] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1963.343247] libceph: try_read tag 1 in_base_pos 0 <7>[ 1963.343252] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 1963.343254] libceph: try_write start ffff8802214c4410 state 5 <7>[ 1963.343257] libceph: try_write out_kvec_bytes 0 <7>[ 1963.343260] libceph: prepare_write_ack ffff8802214c4410 27 -> 29 <7>[ 1963.343262] libceph: try_write out_kvec_bytes 9 <7>[ 1963.343265] libceph: write_partial_kvec ffff8802214c4410 9 left <7>[ 1963.343273] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1963.343275] libceph: try_write nothing else to write. <7>[ 1963.343278] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 1963.343282] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1963.343285] libceph: try_read tag 1 in_base_pos 0 <7>[ 1963.343289] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 1963.343292] libceph: try_write start ffff8802214c4410 state 5 <7>[ 1963.343294] libceph: try_write out_kvec_bytes 0 <7>[ 1963.343296] libceph: try_write nothing else to write. <7>[ 1963.343299] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 1973.345116] libceph: monc delayed_work <7>[ 1973.345123] libceph: con_keepalive ffff8802214c4410 <7>[ 1973.345129] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 1973.345133] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 1973.345137] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 1973.345140] libceph: __send_subscribe sub_sent=0 exp=0 want_osd=0 <7>[ 1973.345142] libceph: __schedule_delayed after 2000 <7>[ 1973.345149] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1973.345152] libceph: try_read tag 1 in_base_pos 0 <7>[ 1973.345159] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 1973.345161] libceph: try_write start ffff8802214c4410 state 5 <7>[ 1973.345164] libceph: try_write out_kvec_bytes 0 <7>[ 1973.345166] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 1973.345168] libceph: try_write out_kvec_bytes 1 <7>[ 1973.345171] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 1973.345180] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1973.345183] libceph: try_write nothing else to write. <7>[ 1973.345185] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 1975.641119] libceph: osds timeout <7>[ 1975.641125] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1990.615108] libceph: osds timeout <7>[ 1990.615113] libceph: __remove_old_osds ffff8802214c4960 <7>[ 1993.370338] libceph: monc delayed_work <7>[ 1993.370344] libceph: con_keepalive ffff8802214c4410 <7>[ 1993.370350] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 1993.370354] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 1993.370357] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 1993.370360] libceph: __send_subscribe sub_sent=0 exp=0 want_osd=0 <7>[ 1993.370363] libceph: __schedule_delayed after 2000 <7>[ 1993.370369] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 1993.370371] libceph: try_read tag 1 in_base_pos 0 <7>[ 1993.370378] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 1993.370380] libceph: try_write start ffff8802214c4410 state 5 <7>[ 1993.370383] libceph: try_write out_kvec_bytes 0 <7>[ 1993.370385] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 1993.370387] libceph: try_write out_kvec_bytes 1 <7>[ 1993.370390] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 1993.370399] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 1993.370402] libceph: try_write nothing else to write. <7>[ 1993.370404] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 2005.589096] libceph: osds timeout <7>[ 2005.589102] libceph: __remove_old_osds ffff8802214c4960 <7>[ 2013.375590] libceph: monc delayed_work <7>[ 2013.375596] libceph: con_keepalive ffff8802214c4410 <7>[ 2013.375602] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 2013.375606] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 2013.375610] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 2013.375613] libceph: __send_subscribe sub_sent=0 exp=0 want_osd=0 <7>[ 2013.375615] libceph: __schedule_delayed after 2000 <7>[ 2013.375621] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 2013.375624] libceph: try_read tag 1 in_base_pos 0 <7>[ 2013.375630] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 2013.375633] libceph: try_write start ffff8802214c4410 state 5 <7>[ 2013.375635] libceph: try_write out_kvec_bytes 0 <7>[ 2013.375638] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 2013.375640] libceph: try_write out_kvec_bytes 1 <7>[ 2013.375643] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 2013.375652] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 2013.375654] libceph: try_write nothing else to write. <7>[ 2013.375657] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 2020.563088] libceph: osds timeout <7>[ 2020.563093] libceph: __remove_old_osds ffff8802214c4960 <7>[ 2033.380838] libceph: monc delayed_work <7>[ 2033.380844] libceph: con_keepalive ffff8802214c4410 <7>[ 2033.380850] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 2033.380854] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 2033.380858] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 2033.380860] libceph: __send_subscribe sub_sent=0 exp=0 want_osd=0 <7>[ 2033.380863] libceph: __schedule_delayed after 2000 <7>[ 2033.380869] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 2033.380872] libceph: try_read tag 1 in_base_pos 0 <7>[ 2033.380878] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 2033.380881] libceph: try_write start ffff8802214c4410 state 5 <7>[ 2033.380883] libceph: try_write out_kvec_bytes 0 <7>[ 2033.380886] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 2033.380888] libceph: try_write out_kvec_bytes 1 <7>[ 2033.380891] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 2033.380900] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 2033.380902] libceph: try_write nothing else to write. <7>[ 2033.380905] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 2035.537073] libceph: osds timeout <7>[ 2035.537079] libceph: __remove_old_osds ffff8802214c4960 <7>[ 2050.511061] libceph: osds timeout <7>[ 2050.511067] libceph: __remove_old_osds ffff8802214c4960 <7>[ 2053.386068] libceph: monc delayed_work <7>[ 2053.386074] libceph: con_keepalive ffff8802214c4410 <7>[ 2053.386081] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 2053.386085] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 2053.386088] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 2053.386092] libceph: __send_subscribe sub_sent=0 exp=0 want_osd=0 <7>[ 2053.386094] libceph: __schedule_delayed after 2000 <7>[ 2053.386101] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 2053.386104] libceph: try_read tag 1 in_base_pos 0 <7>[ 2053.386110] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 2053.386113] libceph: try_write start ffff8802214c4410 state 5 <7>[ 2053.386115] libceph: try_write out_kvec_bytes 0 <7>[ 2053.386118] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 2053.386120] libceph: try_write out_kvec_bytes 1 <7>[ 2053.386123] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 2053.386131] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 2053.386134] libceph: try_write nothing else to write. <7>[ 2053.386136] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 2065.485058] libceph: osds timeout <7>[ 2065.485063] libceph: __remove_old_osds ffff8802214c4960 <7>[ 2073.391326] libceph: monc delayed_work <7>[ 2073.391331] libceph: con_keepalive ffff8802214c4410 <7>[ 2073.391337] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 2073.391342] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 2073.391345] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 2073.391348] libceph: __send_subscribe sub_sent=0 exp=0 want_osd=0 <7>[ 2073.391350] libceph: __schedule_delayed after 2000 <7>[ 2073.391356] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 2073.391359] libceph: try_read tag 1 in_base_pos 0 <7>[ 2073.391365] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 2073.391368] libceph: try_write start ffff8802214c4410 state 5 <7>[ 2073.391370] libceph: try_write out_kvec_bytes 0 <7>[ 2073.391373] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 2073.391375] libceph: try_write out_kvec_bytes 1 <7>[ 2073.391378] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 2073.391387] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 2073.391389] libceph: try_write nothing else to write. <7>[ 2073.391392] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 2080.459046] libceph: osds timeout <7>[ 2080.459051] libceph: __remove_old_osds ffff8802214c4960 <7>[ 2093.396573] libceph: monc delayed_work <7>[ 2093.396579] libceph: con_keepalive ffff8802214c4410 <7>[ 2093.396585] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 2093.396589] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 2093.396592] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 2093.396595] libceph: __send_subscribe sub_sent=0 exp=0 want_osd=0 <7>[ 2093.396598] libceph: __schedule_delayed after 2000 <7>[ 2093.396604] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 2093.396606] libceph: try_read tag 1 in_base_pos 0 <7>[ 2093.396613] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 2093.396616] libceph: try_write start ffff8802214c4410 state 5 <7>[ 2093.396618] libceph: try_write out_kvec_bytes 0 <7>[ 2093.396620] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 2093.396623] libceph: try_write out_kvec_bytes 1 <7>[ 2093.396625] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 2093.396635] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 2093.396637] libceph: try_write nothing else to write. <7>[ 2093.396640] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 2095.433029] libceph: osds timeout <7>[ 2095.433034] libceph: __remove_old_osds ffff8802214c4960 <7>[ 2110.407013] libceph: osds timeout <7>[ 2110.407019] libceph: __remove_old_osds ffff8802214c4960 <7>[ 2113.401823] libceph: monc delayed_work <7>[ 2113.401829] libceph: con_keepalive ffff8802214c4410 <7>[ 2113.401835] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 2113.401839] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 2113.401842] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 2113.401845] libceph: __send_subscribe sub_sent=0 exp=1 want_osd=0 <7>[ 2113.401850] libceph: ----- ffff88020d5b5400 to mon1 15=mon_subscribe len 31+0+0 ----- <7>[ 2113.401853] libceph: __schedule_delayed after 1000 <7>[ 2113.401859] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 2113.401862] libceph: try_read tag 1 in_base_pos 0 <7>[ 2113.401869] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 2113.401871] libceph: try_write start ffff8802214c4410 state 5 <7>[ 2113.401873] libceph: try_write out_kvec_bytes 0 <7>[ 2113.401877] libceph: prepare_write_message ffff88020d5b5400 seq 16 type 15 len 31+0+0 0 pgs <7>[ 2113.401881] libceph: prepare_write_message front_crc 2650579135 middle_crc 0 <7>[ 2113.401884] libceph: prepare_write_message_footer ffff8802214c4410 <7>[ 2113.401886] libceph: try_write out_kvec_bytes 98 <7>[ 2113.401889] libceph: write_partial_kvec ffff8802214c4410 98 left <7>[ 2113.401907] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 2113.401910] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 2113.401913] libceph: try_write out_kvec_bytes 1 <7>[ 2113.401915] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 2113.401921] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 2113.401923] libceph: try_write nothing else to write. <7>[ 2113.401926] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 2113.402500] libceph: ceph_sock_data_ready on ffff8802214c4410 state = 5, queueing work <7>[ 2113.402511] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 2113.402528] libceph: ceph_sock_data_ready on ffff8802214c4410 state = 5, queueing work <7>[ 2113.402532] libceph: queue_con_delay ffff8802214c4410 - already queued <7>[ 2113.402548] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 2113.402551] libceph: try_read tag 1 in_base_pos 0 <7>[ 2113.402558] libceph: try_read got tag 8 <7>[ 2113.402560] libceph: prepare_read_ack ffff8802214c4410 <7>[ 2113.402565] libceph: got ack for seq 16 type 15 at ffff88020d5b5400 <7>[ 2113.402568] libceph: prepare_read_tag ffff8802214c4410 <7>[ 2113.402571] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 2113.402573] libceph: try_read tag 1 in_base_pos 0 <7>[ 2113.402577] libceph: try_read got tag 7 <7>[ 2113.402580] libceph: prepare_read_message ffff8802214c4410 <7>[ 2113.402582] libceph: read_partial_message con ffff8802214c4410 msg (null) <7>[ 2113.402588] libceph: got hdr type 4 front 481 data 0 <7>[ 2113.402592] libceph: ceph_msg_new ffff88020d35db00 front 481 <7>[ 2113.402601] libceph: read_partial_message got msg ffff88020d35db00 481 (2132872380) + 0 (0) + 0 (0) <7>[ 2113.402606] libceph: ===== ffff88020d35db00 30 from mon1 4=mon_map len 481+0 (2132872380 0 0) ===== <7>[ 2113.402610] libceph: have_debugfs_info fsid 1 globalid 4132 <7>[ 2113.402612] libceph: handle_monmap <7>[ 2113.402615] libceph: monmap_decode ffff88020d752c04 ffff88020d752de1 len 477 <7>[ 2113.402618] libceph: monmap_decode epoch 1, num_mon 3 <7>[ 2113.402622] libceph: monmap_decode mon0 is 10.214.133.28:6789 <7>[ 2113.402625] libceph: monmap_decode mon1 is 10.214.133.30:6789 <7>[ 2113.402628] libceph: monmap_decode mon2 is 10.214.133.28:6790 <7>[ 2113.402632] libceph: ceph_msg_put last one on ffff88020d35db00 <7>[ 2113.402634] libceph: msg_kfree ffff88020d35db00 <7>[ 2113.402638] libceph: prepare_read_tag ffff8802214c4410 <7>[ 2113.402640] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 2113.402643] libceph: try_read tag 1 in_base_pos 0 <7>[ 2113.402647] libceph: try_read got tag 7 <7>[ 2113.402650] libceph: prepare_read_message ffff8802214c4410 <7>[ 2113.402652] libceph: read_partial_message con ffff8802214c4410 msg (null) <7>[ 2113.402657] libceph: got hdr type 16 front 20 data 0 <7>[ 2113.402676] libceph: read_partial_message got msg ffff88020d5b4500 20 (473278946) + 0 (0) + 0 (0) <7>[ 2113.402681] libceph: ===== ffff88020d5b4500 31 from mon1 16=mon_subscribe_ack len 20+0 (473278946 0 0) ===== <7>[ 2113.402684] libceph: handle_subscribe_ack after 300 seconds <7>[ 2113.402687] libceph: prepare_read_tag ffff8802214c4410 <7>[ 2113.402690] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 2113.402692] libceph: try_read tag 1 in_base_pos 0 <7>[ 2113.402697] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 2113.402700] libceph: try_write start ffff8802214c4410 state 5 <7>[ 2113.402702] libceph: try_write out_kvec_bytes 0 <7>[ 2113.402705] libceph: prepare_write_ack ffff8802214c4410 29 -> 31 <7>[ 2113.402708] libceph: try_write out_kvec_bytes 9 <7>[ 2113.402710] libceph: write_partial_kvec ffff8802214c4410 9 left <7>[ 2113.402718] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 2113.402720] libceph: try_write nothing else to write. <7>[ 2113.402723] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 2114.090952] libceph: ceph_sock_state_change ffff88020c7c4030 state = 5 sk_state = 8 <7>[ 2114.090956] libceph: ceph_sock_state_change TCP_CLOSE_WAIT <7>[ 2114.090960] libceph: con_sock_state_closing con ffff88020c7c4030 sock 3 -> 4 <7>[ 2114.090964] libceph: get_osd ffff88020c7c4000 1 -> 2 <7>[ 2114.090972] libceph: queue_con_delay ffff88020c7c4030 0 <4>[ 2114.090992] libceph: osd4 10.214.133.30:6803 socket closed (con state OPEN) <7>[ 2114.097962] libceph: fault ffff88020c7c4030 state 5 to peer 10.214.133.30:6803 <7>[ 2114.097965] libceph: con_close_socket on ffff88020c7c4030 sock ffff88020cfe1800 <7>[ 2114.097982] libceph: ceph_sock_state_change ffff88020c7c4030 state = 5 sk_state = 9 <7>[ 2114.097996] libceph: con_sock_state_closed con ffff88020c7c4030 sock 4 -> 1 <7>[ 2114.097999] libceph: fault on LOSSYTX channel, marking CLOSED <7>[ 2114.098002] libceph: osd_reset osd4 <7>[ 2114.098006] libceph: __kick_osd_requests osd4 <7>[ 2114.098009] libceph: __reset_osd ffff88020c7c4000 osd4 <7>[ 2114.098014] libceph: con_close ffff88020c7c4030 peer 10.214.133.30:6803 <7>[ 2114.098016] libceph: reset_connection ffff88020c7c4030 <7>[ 2114.098020] libceph: con_close_socket on ffff88020c7c4030 sock (null) <7>[ 2114.098023] libceph: con_sock_state_closed con ffff88020c7c4030 sock 1 -> 1 <7>[ 2114.098027] libceph: con_open ffff88020c7c4030 10.214.133.30:6803 <7>[ 2114.098031] libceph: get_osd ffff88020c7c4000 2 -> 3 <7>[ 2114.098034] libceph: queue_con_delay ffff88020c7c4030 0 <7>[ 2114.098038] libceph: __register_request ffff880224d03000 tid 951 <7>[ 2114.098040] libceph: first request, scheduling timeout <7>[ 2114.098045] libceph: __unregister_linger_request ffff880224d03000 <7>[ 2114.098048] libceph: requeued lingering ffff880224d03000 tid 951 osd4 <7>[ 2114.098051] libceph: send_queued <7>[ 2114.098054] libceph: send_request ffff880224d03000 tid 951 to osd4 flags 36 <7>[ 2114.098058] libceph: get_osd ffff88020c7c4000 3 -> 4 <7>[ 2114.098062] libceph: ----- ffff88020b761d00 to osd4 42=osd_op len 130+0+0 ----- <7>[ 2114.098072] libceph: get_osd ffff88020c7c4000 4 -> 5 <7>[ 2114.098075] libceph: queue_con_delay ffff88020c7c4030 - already queued <7>[ 2114.098078] libceph: put_osd ffff88020c7c4000 5 -> 4 <7>[ 2114.098081] libceph: put_osd ffff88020c7c4000 4 -> 3 <7>[ 2114.098085] libceph: con_work OPENING <7>[ 2114.098088] libceph: try_read start on ffff88020c7c4030 state 2 <7>[ 2114.098091] libceph: try_write start ffff88020c7c4030 state 2 <7>[ 2114.098093] libceph: try_write out_kvec_bytes 0 <7>[ 2114.098096] libceph: prepare_read_banner ffff88020c7c4030 <7>[ 2114.098099] libceph: try_write initiating connect on ffff88020c7c4030 new state 3 <7>[ 2114.098112] libceph: connect 10.214.133.30:6803 <7>[ 2114.098115] libceph: con_sock_state_connecting con ffff88020c7c4030 sock 1 -> 2 <7>[ 2114.098160] libceph: connect 10.214.133.30:6803 EINPROGRESS sk_state = 2 <7>[ 2114.098164] libceph: write_partial_kvec ffff88020c7c4030 145 left <7>[ 2114.098169] libceph: write_partial_kvec ffff88020c7c4030 145 left in 2 kvecs ret = 0 <7>[ 2114.098172] libceph: try_write done on ffff88020c7c4030 ret 0 <7>[ 2114.098175] libceph: put_osd ffff88020c7c4000 3 -> 2 <7>[ 2114.098298] libceph: ceph_sock_state_change ffff88020c7c4030 state = 3 sk_state = 1 <7>[ 2114.098301] libceph: ceph_sock_state_change TCP_ESTABLISHED <7>[ 2114.098304] libceph: con_sock_state_connected con ffff88020c7c4030 sock 2 -> 3 <7>[ 2114.098307] libceph: get_osd ffff88020c7c4000 2 -> 3 <7>[ 2114.098313] libceph: queue_con_delay ffff88020c7c4030 0 <7>[ 2114.098334] libceph: try_read start on ffff88020c7c4030 state 3 <7>[ 2114.098337] libceph: try_read tag 1 in_base_pos 0 <7>[ 2114.098340] libceph: try_read connecting <7>[ 2114.098342] libceph: read_partial_banner ffff88020c7c4030 at 0 <7>[ 2114.098348] libceph: try_read done on ffff88020c7c4030 ret 0 <7>[ 2114.098351] libceph: try_write start ffff88020c7c4030 state 3 <7>[ 2114.098353] libceph: try_write out_kvec_bytes 145 <7>[ 2114.098356] libceph: write_partial_kvec ffff88020c7c4030 145 left <7>[ 2114.098371] libceph: write_partial_kvec ffff88020c7c4030 0 left in 0 kvecs ret = 1 <7>[ 2114.098373] libceph: try_write nothing else to write. <7>[ 2114.098376] libceph: try_write done on ffff88020c7c4030 ret 0 <7>[ 2114.098379] libceph: put_osd ffff88020c7c4000 3 -> 2 <7>[ 2114.098654] libceph: ceph_sock_data_ready on ffff88020c7c4030 state = 3, queueing work <7>[ 2114.098658] libceph: get_osd ffff88020c7c4000 2 -> 3 <7>[ 2114.098663] libceph: queue_con_delay ffff88020c7c4030 0 <7>[ 2114.098678] libceph: ceph_sock_data_ready on ffff88020c7c4030 state = 3, queueing work <7>[ 2114.098681] libceph: get_osd ffff88020c7c4000 3 -> 4 <7>[ 2114.098683] libceph: queue_con_delay ffff88020c7c4030 - already queued <7>[ 2114.098686] libceph: put_osd ffff88020c7c4000 4 -> 3 <7>[ 2114.098695] libceph: try_read start on ffff88020c7c4030 state 3 <7>[ 2114.098697] libceph: try_read tag 1 in_base_pos 0 <7>[ 2114.098700] libceph: try_read connecting <7>[ 2114.098702] libceph: read_partial_banner ffff88020c7c4030 at 0 <7>[ 2114.098714] libceph: process_banner on ffff88020c7c4030 <7>[ 2114.098718] libceph: prepare_write_connect ffff88020c7c4030 cseq=0 gseq=9 proto=24 <7>[ 2114.098722] libceph: prepare_read_connect ffff88020c7c4030 <7>[ 2114.098725] libceph: try_read done on ffff88020c7c4030 ret 0 <7>[ 2114.098728] libceph: try_write start ffff88020c7c4030 state 4 <7>[ 2114.098730] libceph: try_write out_kvec_bytes 255 <7>[ 2114.098733] libceph: write_partial_kvec ffff88020c7c4030 255 left <7>[ 2114.098751] libceph: write_partial_kvec ffff88020c7c4030 0 left in 0 kvecs ret = 1 <7>[ 2114.098753] libceph: try_write nothing else to write. <7>[ 2114.098756] libceph: try_write done on ffff88020c7c4030 ret 0 <7>[ 2114.098759] libceph: put_osd ffff88020c7c4000 3 -> 2 <7>[ 2114.098982] libceph: ceph_sock_data_ready on ffff88020c7c4030 state = 4, queueing work <7>[ 2114.098987] libceph: get_osd ffff88020c7c4000 2 -> 3 <7>[ 2114.098994] libceph: queue_con_delay ffff88020c7c4030 0 <7>[ 2114.099009] libceph: ceph_sock_data_ready on ffff88020c7c4030 state = 4, queueing work <7>[ 2114.099013] libceph: get_osd ffff88020c7c4000 3 -> 4 <7>[ 2114.099015] libceph: queue_con_delay ffff88020c7c4030 - already queued <7>[ 2114.099018] libceph: put_osd ffff88020c7c4000 4 -> 3 <7>[ 2114.099029] libceph: try_read start on ffff88020c7c4030 state 4 <7>[ 2114.099032] libceph: try_read tag 1 in_base_pos 0 <7>[ 2114.099034] libceph: try_read negotiating <7>[ 2114.099037] libceph: read_partial_connect ffff88020c7c4030 at 0 <7>[ 2114.099055] libceph: read_partial_connect ffff88020c7c4030 tag 1, con_seq = 1, g_seq = 8 <7>[ 2114.099058] libceph: process_connect on ffff88020c7c4030 tag 1 <7>[ 2114.099061] libceph: process_connect got READY gseq 8 cseq 1 (1) <7>[ 2114.099064] libceph: prepare_read_tag ffff88020c7c4030 <7>[ 2114.099066] libceph: try_read start on ffff88020c7c4030 state 5 <7>[ 2114.099069] libceph: try_read tag 1 in_base_pos 0 <7>[ 2114.099073] libceph: try_read done on ffff88020c7c4030 ret 0 <7>[ 2114.099076] libceph: try_write start ffff88020c7c4030 state 5 <7>[ 2114.099078] libceph: try_write out_kvec_bytes 0 <7>[ 2114.099082] libceph: prepare_write_message ffff88020b761d00 seq 1 type 42 len 130+0+0 0 pgs <7>[ 2114.099086] libceph: prepare_write_message front_crc 2251967416 middle_crc 0 <7>[ 2114.099089] libceph: prepare_write_message_footer ffff88020c7c4030 <7>[ 2114.099091] libceph: try_write out_kvec_bytes 197 <7>[ 2114.099094] libceph: write_partial_kvec ffff88020c7c4030 197 left <7>[ 2114.099111] libceph: write_partial_kvec ffff88020c7c4030 0 left in 0 kvecs ret = 1 <7>[ 2114.099115] libceph: try_write nothing else to write. <7>[ 2114.099119] libceph: try_write done on ffff88020c7c4030 ret 0 <7>[ 2114.099123] libceph: put_osd ffff88020c7c4000 3 -> 2 <7>[ 2114.099780] libceph: ceph_sock_data_ready on ffff88020c7c4030 state = 5, queueing work <7>[ 2114.099784] libceph: get_osd ffff88020c7c4000 2 -> 3 <7>[ 2114.099792] libceph: queue_con_delay ffff88020c7c4030 0 <7>[ 2114.099805] libceph: try_read start on ffff88020c7c4030 state 5 <7>[ 2114.099808] libceph: try_read tag 1 in_base_pos 0 <7>[ 2114.099815] libceph: try_read got tag 8 <7>[ 2114.099817] libceph: prepare_read_ack ffff88020c7c4030 <7>[ 2114.099823] libceph: got ack for seq 1 type 42 at ffff88020b761d00 <7>[ 2114.099826] libceph: put_osd ffff88020c7c4000 3 -> 2 <7>[ 2114.099828] libceph: prepare_read_tag ffff88020c7c4030 <7>[ 2114.099831] libceph: try_read start on ffff88020c7c4030 state 5 <7>[ 2114.099833] libceph: try_read tag 1 in_base_pos 0 <7>[ 2114.099838] libceph: try_read got tag 7 <7>[ 2114.099840] libceph: prepare_read_message ffff88020c7c4030 <7>[ 2114.099843] libceph: read_partial_message con ffff88020c7c4030 msg (null) <7>[ 2114.099848] libceph: got hdr type 43 front 108 data 0 <7>[ 2114.099853] libceph: get_osd ffff88020c7c4000 2 -> 3 <7>[ 2114.099855] libceph: get_reply tid 951 ffff88020b761a00 <7>[ 2114.099859] libceph: get_osd ffff88020c7c4000 3 -> 4 <7>[ 2114.099869] libceph: read_partial_message got msg ffff88020b761a00 108 (2795223248) + 0 (0) + 0 (0) <7>[ 2114.099872] libceph: put_osd ffff88020c7c4000 4 -> 3 <7>[ 2114.099877] libceph: ===== ffff88020b761a00 1 from osd4 43=osd_opreply len 108+0 (2795223248 0 0) ===== <7>[ 2114.099880] libceph: handle_reply ffff88020b761a00 tid 951 result 0 <7>[ 2114.099883] libceph: dropping con_filling_msg ref ffff88020c7c4030 <7>[ 2114.099886] libceph: put_osd ffff88020c7c4000 3 -> 2 <7>[ 2114.099889] libceph: handle_reply tid 951 flags 37 <7>[ 2114.099891] libceph: __register_linger_request ffff880224d03000 <7>[ 2114.099894] libceph: __unregister_request ffff880224d03000 tid 951 <7>[ 2114.099897] libceph: no requests, canceling timeout <7>[ 2114.099902] libceph: req=ffff880224d03000 req->r_linger=1 <7>[ 2114.099905] libceph: prepare_read_tag ffff88020c7c4030 <7>[ 2114.099907] libceph: try_read start on ffff88020c7c4030 state 5 <7>[ 2114.099910] libceph: try_read tag 1 in_base_pos 0 <7>[ 2114.099915] libceph: try_read done on ffff88020c7c4030 ret 0 <7>[ 2114.099917] libceph: try_write start ffff88020c7c4030 state 5 <7>[ 2114.099920] libceph: try_write out_kvec_bytes 0 <7>[ 2114.099923] libceph: prepare_write_ack ffff88020c7c4030 0 -> 1 <7>[ 2114.099925] libceph: try_write out_kvec_bytes 9 <7>[ 2114.099927] libceph: write_partial_kvec ffff88020c7c4030 9 left <7>[ 2114.099935] libceph: write_partial_kvec ffff88020c7c4030 0 left in 0 kvecs ret = 1 <7>[ 2114.099938] libceph: try_write nothing else to write. <7>[ 2114.099940] libceph: try_write done on ffff88020c7c4030 ret 0 <7>[ 2114.099943] libceph: put_osd ffff88020c7c4000 2 -> 1 <7>[ 2123.404441] libceph: monc delayed_work <7>[ 2123.404447] libceph: con_keepalive ffff8802214c4410 <7>[ 2123.404454] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 2123.404458] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 2123.404462] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 2123.404465] libceph: __send_subscribe sub_sent=0 exp=0 want_osd=0 <7>[ 2123.404468] libceph: __schedule_delayed after 2000 <7>[ 2123.404477] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 2123.404480] libceph: try_read tag 1 in_base_pos 0 <7>[ 2123.404487] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 2123.404489] libceph: try_write start ffff8802214c4410 state 5 <7>[ 2123.404492] libceph: try_write out_kvec_bytes 0 <7>[ 2123.404494] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 2123.404497] libceph: try_write out_kvec_bytes 1 <7>[ 2123.404499] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 2123.404508] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 2123.404511] libceph: try_write nothing else to write. <7>[ 2123.404513] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 2125.381002] libceph: osds timeout <7>[ 2125.381008] libceph: __remove_old_osds ffff8802214c4960 <7>[ 2140.354993] libceph: osds timeout <7>[ 2140.354999] libceph: __remove_old_osds ffff8802214c4960 <7>[ 2143.429670] libceph: monc delayed_work <7>[ 2143.429676] libceph: con_keepalive ffff8802214c4410 <7>[ 2143.429681] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 2143.429686] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 2143.429689] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 2143.429692] libceph: __send_subscribe sub_sent=0 exp=0 want_osd=0 <7>[ 2143.429694] libceph: __schedule_delayed after 2000 <7>[ 2143.429700] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 2143.429703] libceph: try_read tag 1 in_base_pos 0 <7>[ 2143.429709] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 2143.429712] libceph: try_write start ffff8802214c4410 state 5 <7>[ 2143.429714] libceph: try_write out_kvec_bytes 0 <7>[ 2143.429717] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 2143.429719] libceph: try_write out_kvec_bytes 1 <7>[ 2143.429722] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 2143.429731] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 2143.429733] libceph: try_write nothing else to write. <7>[ 2143.429736] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 2155.328980] libceph: osds timeout <7>[ 2155.328986] libceph: __remove_old_osds ffff8802214c4960 <7>[ 2163.434901] libceph: monc delayed_work <7>[ 2163.434907] libceph: con_keepalive ffff8802214c4410 <7>[ 2163.434913] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 2163.434917] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 2163.434921] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 2163.434923] libceph: __send_subscribe sub_sent=0 exp=0 want_osd=0 <7>[ 2163.434926] libceph: __schedule_delayed after 2000 <7>[ 2163.434933] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 2163.434936] libceph: try_read tag 1 in_base_pos 0 <7>[ 2163.434942] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 2163.434945] libceph: try_write start ffff8802214c4410 state 5 <7>[ 2163.434948] libceph: try_write out_kvec_bytes 0 <7>[ 2163.434950] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 2163.434952] libceph: try_write out_kvec_bytes 1 <7>[ 2163.434955] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 2163.434964] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 2163.434966] libceph: try_write nothing else to write. <7>[ 2163.434969] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 2170.302969] libceph: osds timeout <7>[ 2170.302974] libceph: __remove_old_osds ffff8802214c4960 <7>[ 2183.440150] libceph: monc delayed_work <7>[ 2183.440156] libceph: con_keepalive ffff8802214c4410 <7>[ 2183.440162] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 2183.440166] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 2183.440170] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 2183.440173] libceph: __send_subscribe sub_sent=0 exp=0 want_osd=0 <7>[ 2183.440175] libceph: __schedule_delayed after 2000 <7>[ 2183.440182] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 2183.440185] libceph: try_read tag 1 in_base_pos 0 <7>[ 2183.440191] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 2183.440194] libceph: try_write start ffff8802214c4410 state 5 <7>[ 2183.440196] libceph: try_write out_kvec_bytes 0 <7>[ 2183.440199] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 2183.440201] libceph: try_write out_kvec_bytes 1 <7>[ 2183.440204] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 2183.440213] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 2183.440215] libceph: try_write nothing else to write. <7>[ 2183.440218] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 2185.276954] libceph: osds timeout <7>[ 2185.276959] libceph: __remove_old_osds ffff8802214c4960 <7>[ 2200.250944] libceph: osds timeout <7>[ 2200.250950] libceph: __remove_old_osds ffff8802214c4960 <7>[ 2203.445403] libceph: monc delayed_work <7>[ 2203.445409] libceph: con_keepalive ffff8802214c4410 <7>[ 2203.445415] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 2203.445419] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 2203.445423] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 2203.445426] libceph: __send_subscribe sub_sent=0 exp=0 want_osd=0 <7>[ 2203.445429] libceph: __schedule_delayed after 2000 <7>[ 2203.445435] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 2203.445438] libceph: try_read tag 1 in_base_pos 0 <7>[ 2203.445445] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 2203.445447] libceph: try_write start ffff8802214c4410 state 5 <7>[ 2203.445450] libceph: try_write out_kvec_bytes 0 <7>[ 2203.445452] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 2203.445455] libceph: try_write out_kvec_bytes 1 <7>[ 2203.445457] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 2203.445466] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 2203.445469] libceph: try_write nothing else to write. <7>[ 2203.445471] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 2215.224931] libceph: osds timeout <7>[ 2215.224937] libceph: __remove_old_osds ffff8802214c4960 <7>[ 2223.450649] libceph: monc delayed_work <7>[ 2223.450655] libceph: con_keepalive ffff8802214c4410 <7>[ 2223.450661] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 2223.450666] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 2223.450669] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 2223.450672] libceph: __send_subscribe sub_sent=0 exp=0 want_osd=0 <7>[ 2223.450675] libceph: __schedule_delayed after 2000 <7>[ 2223.450681] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 2223.450683] libceph: try_read tag 1 in_base_pos 0 <7>[ 2223.450690] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 2223.450693] libceph: try_write start ffff8802214c4410 state 5 <7>[ 2223.450695] libceph: try_write out_kvec_bytes 0 <7>[ 2223.450697] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 2223.450700] libceph: try_write out_kvec_bytes 1 <7>[ 2223.450703] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 2223.450712] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 2223.450714] libceph: try_write nothing else to write. <7>[ 2223.450717] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 2230.198919] libceph: osds timeout <7>[ 2230.198925] libceph: __remove_old_osds ffff8802214c4960 <7>[ 2240.684891] libceph: parse_options ffff88020b413a00 options 'name=afs' dev_name '10.214.133.28:6789,10.214.133.30:6789,10.21' <7>[ 2240.684897] libceph: parse_ips on '10.214.133.28:6789,10.214.133.30:6789,10.214.133.28:6790' <7>[ 2240.684902] libceph: parse_ips got 10.214.133.28:6789 <7>[ 2240.684906] libceph: parse_ips got 10.214.133.30:6789 <7>[ 2240.684909] libceph: parse_ips got 10.214.133.28:6790 <7>[ 2240.684913] libceph: got string token 6 val afs <7>[ 2240.684918] rbd: rbd_client_create <7>[ 2240.684930] libceph: ceph_messenger_init ffff88020d36c110 <7>[ 2240.684933] libceph: init <7>[ 2240.684937] libceph: auth_init name 'afs' <7>[ 2240.684940] libceph: auth_init name afs <7>[ 2240.684943] libceph: ceph_msg_new ffff88020b413f00 front 20 <7>[ 2240.684946] libceph: ceph_msg_new ffff88020b413e00 front 96 <7>[ 2240.684949] libceph: ceph_msg_new ffff88020b413d00 front 4096 <7>[ 2240.684952] libceph: ceph_msg_new ffff88020b413c00 front 4096 <7>[ 2240.684955] libceph: con_init ffff88020d36c410 <7>[ 2240.684958] libceph: con_sock_state_init con ffff88020d36c410 sock 0 -> 1 <7>[ 2240.684964] libceph: init <7>[ 2240.684976] libceph: msgpool osd_op init <7>[ 2240.684981] libceph: ceph_msg_new ffff88020b413b00 front 4096 <7>[ 2240.684983] libceph: msgpool_alloc (null) ffff88020b413b00 <7>[ 2240.684987] libceph: ceph_msg_new ffff88020b412f00 front 4096 <7>[ 2240.684990] libceph: msgpool_alloc (null) ffff88020b412f00 <7>[ 2240.684993] libceph: ceph_msg_new ffff88020b412c00 front 4096 <7>[ 2240.684995] libceph: msgpool_alloc (null) ffff88020b412c00 <7>[ 2240.684998] libceph: ceph_msg_new ffff88020b412a00 front 4096 <7>[ 2240.685001] libceph: msgpool_alloc (null) ffff88020b412a00 <7>[ 2240.685004] libceph: ceph_msg_new ffff88020b412800 front 4096 <7>[ 2240.685006] libceph: msgpool_alloc (null) ffff88020b412800 <7>[ 2240.685010] libceph: ceph_msg_new ffff88020b412500 front 4096 <7>[ 2240.685012] libceph: msgpool_alloc (null) ffff88020b412500 <7>[ 2240.685015] libceph: ceph_msg_new ffff88020b412b00 front 4096 <7>[ 2240.685018] libceph: msgpool_alloc (null) ffff88020b412b00 <7>[ 2240.685020] libceph: ceph_msg_new ffff88020b412400 front 4096 <7>[ 2240.685023] libceph: msgpool_alloc (null) ffff88020b412400 <7>[ 2240.685026] libceph: ceph_msg_new ffff88020b412d00 front 4096 <7>[ 2240.685028] libceph: msgpool_alloc (null) ffff88020b412d00 <7>[ 2240.685031] libceph: ceph_msg_new ffff88020b412900 front 4096 <7>[ 2240.685034] libceph: msgpool_alloc (null) ffff88020b412900 <7>[ 2240.685037] libceph: msgpool osd_op_reply init <7>[ 2240.685040] libceph: ceph_msg_new ffff88020b412100 front 512 <7>[ 2240.685043] libceph: msgpool_alloc (null) ffff88020b412100 <7>[ 2240.685046] libceph: ceph_msg_new ffff88020b412600 front 512 <7>[ 2240.685048] libceph: msgpool_alloc (null) ffff88020b412600 <7>[ 2240.685051] libceph: ceph_msg_new ffff88020b412200 front 512 <7>[ 2240.685053] libceph: msgpool_alloc (null) ffff88020b412200 <7>[ 2240.685056] libceph: ceph_msg_new ffff88020b412300 front 512 <7>[ 2240.685059] libceph: msgpool_alloc (null) ffff88020b412300 <7>[ 2240.685062] libceph: ceph_msg_new ffff88020b412000 front 512 <7>[ 2240.685064] libceph: msgpool_alloc (null) ffff88020b412000 <7>[ 2240.685067] libceph: ceph_msg_new ffff88020b412700 front 512 <7>[ 2240.685070] libceph: msgpool_alloc (null) ffff88020b412700 <7>[ 2240.685073] libceph: ceph_msg_new ffff88020b413900 front 512 <7>[ 2240.685075] libceph: msgpool_alloc (null) ffff88020b413900 <7>[ 2240.685078] libceph: ceph_msg_new ffff88020b413700 front 512 <7>[ 2240.685081] libceph: msgpool_alloc (null) ffff88020b413700 <7>[ 2240.685084] libceph: ceph_msg_new ffff88020b413200 front 512 <7>[ 2240.685086] libceph: msgpool_alloc (null) ffff88020b413200 <7>[ 2240.685089] libceph: ceph_msg_new ffff88020b413400 front 512 <7>[ 2240.685092] libceph: msgpool_alloc (null) ffff88020b413400 <7>[ 2240.685212] libceph: open_session start <7>[ 2240.685222] libceph: open_session num=3 r=-94 -> mon0 <7>[ 2240.685225] libceph: open_session mon0 opening <7>[ 2240.685229] libceph: con_open ffff88020d36c410 10.214.133.28:6789 <7>[ 2240.685237] libceph: queue_con_delay ffff88020d36c410 0 <7>[ 2240.685240] libceph: auth_build_hello <7>[ 2240.685248] libceph: con_work OPENING <7>[ 2240.685250] libceph: try_read start on ffff88020d36c410 state 2 <7>[ 2240.685253] libceph: try_write start ffff88020d36c410 state 2 <7>[ 2240.685256] libceph: try_write out_kvec_bytes 0 <7>[ 2240.685258] libceph: prepare_read_banner ffff88020d36c410 <7>[ 2240.685261] libceph: try_write initiating connect on ffff88020d36c410 new state 3 <7>[ 2240.685275] libceph: connect 10.214.133.28:6789 <7>[ 2240.685279] libceph: con_sock_state_connecting con ffff88020d36c410 sock 1 -> 2 <7>[ 2240.685318] libceph: connect 10.214.133.28:6789 EINPROGRESS sk_state = 2 <7>[ 2240.685322] libceph: write_partial_kvec ffff88020d36c410 145 left <7>[ 2240.685327] libceph: write_partial_kvec ffff88020d36c410 145 left in 2 kvecs ret = 0 <7>[ 2240.685330] libceph: try_write done on ffff88020d36c410 ret 0 <7>[ 2240.685340] libceph: ----- ffff88020b413c00 to mon0 17=auth len 58+0+0 ----- <7>[ 2240.685343] libceph: __schedule_delayed after 1000 <7>[ 2240.685348] libceph: mount waiting for mon_map <7>[ 2240.685613] libceph: ceph_sock_state_change ffff88020d36c410 state = 3 sk_state = 1 <7>[ 2240.685618] libceph: ceph_sock_state_change TCP_ESTABLISHED <7>[ 2240.685622] libceph: con_sock_state_connected con ffff88020d36c410 sock 2 -> 3 <7>[ 2240.685630] libceph: queue_con_delay ffff88020d36c410 0 <7>[ 2240.685654] libceph: try_read start on ffff88020d36c410 state 3 <7>[ 2240.685657] libceph: try_read tag 1 in_base_pos 0 <7>[ 2240.685660] libceph: try_read connecting <7>[ 2240.685662] libceph: read_partial_banner ffff88020d36c410 at 0 <7>[ 2240.685669] libceph: try_read done on ffff88020d36c410 ret 0 <7>[ 2240.685671] libceph: try_write start ffff88020d36c410 state 3 <7>[ 2240.685674] libceph: try_write out_kvec_bytes 145 <7>[ 2240.685677] libceph: write_partial_kvec ffff88020d36c410 145 left <7>[ 2240.685693] libceph: write_partial_kvec ffff88020d36c410 0 left in 0 kvecs ret = 1 <7>[ 2240.685695] libceph: try_write nothing else to write. <7>[ 2240.685698] libceph: try_write done on ffff88020d36c410 ret 0 <7>[ 2240.685990] libceph: ceph_sock_data_ready on ffff88020d36c410 state = 3, queueing work <7>[ 2240.685998] libceph: queue_con_delay ffff88020d36c410 0 <7>[ 2240.686018] libceph: ceph_sock_data_ready on ffff88020d36c410 state = 3, queueing work <7>[ 2240.686022] libceph: queue_con_delay ffff88020d36c410 - already queued <7>[ 2240.686032] libceph: try_read start on ffff88020d36c410 state 3 <7>[ 2240.686035] libceph: try_read tag 1 in_base_pos 0 <7>[ 2240.686038] libceph: try_read connecting <7>[ 2240.686040] libceph: read_partial_banner ffff88020d36c410 at 0 <7>[ 2240.686053] libceph: process_banner on ffff88020d36c410 <7>[ 2240.686058] libceph: prepare_write_connect ffff88020d36c410 cseq=0 gseq=1 proto=15 <7>[ 2240.686061] libceph: prepare_read_connect ffff88020d36c410 <7>[ 2240.686063] libceph: try_read done on ffff88020d36c410 ret 0 <7>[ 2240.686066] libceph: try_write start ffff88020d36c410 state 4 <7>[ 2240.686068] libceph: try_write out_kvec_bytes 33 <7>[ 2240.686071] libceph: write_partial_kvec ffff88020d36c410 33 left <7>[ 2240.686086] libceph: write_partial_kvec ffff88020d36c410 0 left in 0 kvecs ret = 1 <7>[ 2240.686088] libceph: try_write nothing else to write. <7>[ 2240.686091] libceph: try_write done on ffff88020d36c410 ret 0 <7>[ 2240.686303] libceph: ceph_sock_data_ready on ffff88020d36c410 state = 4, queueing work <7>[ 2240.686311] libceph: queue_con_delay ffff88020d36c410 0 <7>[ 2240.686322] libceph: try_read start on ffff88020d36c410 state 4 <7>[ 2240.686325] libceph: try_read tag 1 in_base_pos 0 <7>[ 2240.686328] libceph: try_read negotiating <7>[ 2240.686331] libceph: read_partial_connect ffff88020d36c410 at 0 <7>[ 2240.686339] libceph: read_partial_connect ffff88020d36c410 tag 1, con_seq = 1, g_seq = 15 <7>[ 2240.686342] libceph: process_connect on ffff88020d36c410 tag 1 <7>[ 2240.686345] libceph: process_connect got READY gseq 15 cseq 1 (1) <7>[ 2240.686348] libceph: prepare_read_tag ffff88020d36c410 <7>[ 2240.686351] libceph: try_read start on ffff88020d36c410 state 5 <7>[ 2240.686353] libceph: try_read tag 1 in_base_pos 0 <7>[ 2240.686358] libceph: try_read done on ffff88020d36c410 ret 0 <7>[ 2240.686361] libceph: try_write start ffff88020d36c410 state 5 <7>[ 2240.686363] libceph: try_write out_kvec_bytes 0 <7>[ 2240.686367] libceph: prepare_write_message ffff88020b413c00 seq 1 type 17 len 58+0+0 0 pgs <7>[ 2240.686372] libceph: prepare_write_message front_crc 1554419673 middle_crc 0 <7>[ 2240.686375] libceph: prepare_write_message_footer ffff88020d36c410 <7>[ 2240.686377] libceph: try_write out_kvec_bytes 125 <7>[ 2240.686380] libceph: write_partial_kvec ffff88020d36c410 125 left <7>[ 2240.686395] libceph: write_partial_kvec ffff88020d36c410 0 left in 0 kvecs ret = 1 <7>[ 2240.686398] libceph: try_write nothing else to write. <7>[ 2240.686400] libceph: try_write done on ffff88020d36c410 ret 0 <7>[ 2240.686964] libceph: ceph_sock_data_ready on ffff88020d36c410 state = 5, queueing work <7>[ 2240.686972] libceph: queue_con_delay ffff88020d36c410 0 <7>[ 2240.686988] libceph: ceph_sock_data_ready on ffff88020d36c410 state = 5, queueing work <7>[ 2240.686991] libceph: queue_con_delay ffff88020d36c410 - already queued <7>[ 2240.687009] libceph: try_read start on ffff88020d36c410 state 5 <7>[ 2240.687013] libceph: try_read tag 1 in_base_pos 0 <7>[ 2240.687019] libceph: try_read got tag 8 <7>[ 2240.687021] libceph: prepare_read_ack ffff88020d36c410 <7>[ 2240.687027] libceph: got ack for seq 1 type 17 at ffff88020b413c00 <7>[ 2240.687029] libceph: prepare_read_tag ffff88020d36c410 <7>[ 2240.687032] libceph: try_read start on ffff88020d36c410 state 5 <7>[ 2240.687035] libceph: try_read tag 1 in_base_pos 0 <7>[ 2240.687039] libceph: try_read got tag 7 <7>[ 2240.687041] libceph: prepare_read_message ffff88020d36c410 <7>[ 2240.687044] libceph: read_partial_message con ffff88020d36c410 msg (null) <7>[ 2240.687049] libceph: got hdr type 4 front 481 data 0 <7>[ 2240.687053] libceph: ceph_msg_new ffff88020be9c100 front 481 <7>[ 2240.687064] libceph: read_partial_message got msg ffff88020be9c100 481 (2132872380) + 0 (0) + 0 (0) <7>[ 2240.687069] libceph: ===== ffff88020be9c100 1 from mon0 4=mon_map len 481+0 (2132872380 0 0) ===== <7>[ 2240.687073] libceph: have_debugfs_info fsid 0 globalid 0 <7>[ 2240.687075] libceph: handle_monmap <7>[ 2240.687078] libceph: monmap_decode ffff88020bd2f804 ffff88020bd2f9e1 len 477 <7>[ 2240.687081] libceph: monmap_decode epoch 1, num_mon 3 <7>[ 2240.687090] libceph: monmap_decode mon0 is 10.214.133.28:6789 <7>[ 2240.687094] libceph: monmap_decode mon1 is 10.214.133.30:6789 <7>[ 2240.687097] libceph: monmap_decode mon2 is 10.214.133.28:6790 <7>[ 2240.687100] libceph: have_debugfs_info fsid 1 globalid 0 <7>[ 2240.687107] libceph: ceph_msg_put last one on ffff88020be9c100 <7>[ 2240.687110] libceph: msg_kfree ffff88020be9c100 <7>[ 2240.687114] libceph: prepare_read_tag ffff88020d36c410 <7>[ 2240.687117] libceph: try_read start on ffff88020d36c410 state 5 <7>[ 2240.687119] libceph: try_read tag 1 in_base_pos 0 <7>[ 2240.687124] libceph: try_read got tag 7 <7>[ 2240.687126] libceph: prepare_read_message ffff88020d36c410 <7>[ 2240.687129] libceph: read_partial_message con ffff88020d36c410 msg (null) <7>[ 2240.687134] libceph: got hdr type 18 front 33 data 0 <7>[ 2240.687144] libceph: read_partial_message got msg ffff88020b413d00 33 (1240062526) + 0 (0) + 0 (0) <7>[ 2240.687149] libceph: ===== ffff88020b413d00 2 from mon0 18=auth_reply len 33+0 (1240062526 0 0) ===== <7>[ 2240.687152] libceph: have_debugfs_info fsid 1 globalid 0 <7>[ 2240.687155] libceph: handle_auth_reply ffff88020d369000 ffff88020d369021 <7>[ 2240.687158] libceph: result 0 '' gid 4119 len 9 <7>[ 2240.687161] libceph: set global_id 0 -> 4119 <7>[ 2240.687163] libceph: ceph_x_init ffff88020b2420c0 <3>[ 2240.687166] libceph: no secret set (for auth_x protocol) <3>[ 2240.692591] libceph: error -22 on auth protocol 2 init <7>[ 2240.697890] libceph: have_debugfs_info fsid 1 globalid 4119 <6>[ 2240.697893] libceph: client4119 fsid 155e9d3f-16e7-4f5e-9e56-a90071f0ea16 <7>[ 2240.697928] libceph: destroy_client ffff88020d36c000 <7>[ 2240.698117] libceph: remove_all_osds ffff88020d36c960 <7>[ 2240.698134] libceph: msgpool osd_op destroy <7>[ 2240.698136] libceph: msgpool_release osd_op ffff88020b412900 <7>[ 2240.698137] libceph: ceph_msg_put last one on ffff88020b412900 <7>[ 2240.698138] libceph: msg_kfree ffff88020b412900 <7>[ 2240.698140] libceph: msgpool_release osd_op ffff88020b412d00 <7>[ 2240.698141] libceph: ceph_msg_put last one on ffff88020b412d00 <7>[ 2240.698143] libceph: msg_kfree ffff88020b412d00 <7>[ 2240.698144] libceph: msgpool_release osd_op ffff88020b412400 <7>[ 2240.698146] libceph: ceph_msg_put last one on ffff88020b412400 <7>[ 2240.698147] libceph: msg_kfree ffff88020b412400 <7>[ 2240.698148] libceph: msgpool_release osd_op ffff88020b412b00 <7>[ 2240.698150] libceph: ceph_msg_put last one on ffff88020b412b00 <7>[ 2240.698151] libceph: msg_kfree ffff88020b412b00 <7>[ 2240.698153] libceph: msgpool_release osd_op ffff88020b412500 <7>[ 2240.698154] libceph: ceph_msg_put last one on ffff88020b412500 <7>[ 2240.698155] libceph: msg_kfree ffff88020b412500 <7>[ 2240.698157] libceph: msgpool_release osd_op ffff88020b412800 <7>[ 2240.698158] libceph: ceph_msg_put last one on ffff88020b412800 <7>[ 2240.698159] libceph: msg_kfree ffff88020b412800 <7>[ 2240.698161] libceph: msgpool_release osd_op ffff88020b412a00 <7>[ 2240.698162] libceph: ceph_msg_put last one on ffff88020b412a00 <7>[ 2240.698163] libceph: msg_kfree ffff88020b412a00 <7>[ 2240.698165] libceph: msgpool_release osd_op ffff88020b412c00 <7>[ 2240.698166] libceph: ceph_msg_put last one on ffff88020b412c00 <7>[ 2240.698167] libceph: msg_kfree ffff88020b412c00 <7>[ 2240.698169] libceph: msgpool_release osd_op ffff88020b412f00 <7>[ 2240.698170] libceph: ceph_msg_put last one on ffff88020b412f00 <7>[ 2240.698171] libceph: msg_kfree ffff88020b412f00 <7>[ 2240.698173] libceph: msgpool_release osd_op ffff88020b413b00 <7>[ 2240.698174] libceph: ceph_msg_put last one on ffff88020b413b00 <7>[ 2240.698175] libceph: msg_kfree ffff88020b413b00 <7>[ 2240.698178] libceph: msgpool osd_op_reply destroy <7>[ 2240.698179] libceph: msgpool_release osd_op_reply ffff88020b413400 <7>[ 2240.698180] libceph: ceph_msg_put last one on ffff88020b413400 <7>[ 2240.698181] libceph: msg_kfree ffff88020b413400 <7>[ 2240.698183] libceph: msgpool_release osd_op_reply ffff88020b413200 <7>[ 2240.698184] libceph: ceph_msg_put last one on ffff88020b413200 <7>[ 2240.698185] libceph: msg_kfree ffff88020b413200 <7>[ 2240.698187] libceph: msgpool_release osd_op_reply ffff88020b413700 <7>[ 2240.698188] libceph: ceph_msg_put last one on ffff88020b413700 <7>[ 2240.698189] libceph: msg_kfree ffff88020b413700 <7>[ 2240.698191] libceph: msgpool_release osd_op_reply ffff88020b413900 <7>[ 2240.698192] libceph: ceph_msg_put last one on ffff88020b413900 <7>[ 2240.698193] libceph: msg_kfree ffff88020b413900 <7>[ 2240.698195] libceph: msgpool_release osd_op_reply ffff88020b412700 <7>[ 2240.698196] libceph: ceph_msg_put last one on ffff88020b412700 <7>[ 2240.698197] libceph: msg_kfree ffff88020b412700 <7>[ 2240.698199] libceph: msgpool_release osd_op_reply ffff88020b412000 <7>[ 2240.698200] libceph: ceph_msg_put last one on ffff88020b412000 <7>[ 2240.698201] libceph: msg_kfree ffff88020b412000 <7>[ 2240.698203] libceph: msgpool_release osd_op_reply ffff88020b412300 <7>[ 2240.698204] libceph: ceph_msg_put last one on ffff88020b412300 <7>[ 2240.698205] libceph: msg_kfree ffff88020b412300 <7>[ 2240.698207] libceph: msgpool_release osd_op_reply ffff88020b412200 <7>[ 2240.698208] libceph: ceph_msg_put last one on ffff88020b412200 <7>[ 2240.698209] libceph: msg_kfree ffff88020b412200 <7>[ 2240.698211] libceph: msgpool_release osd_op_reply ffff88020b412600 <7>[ 2240.698212] libceph: ceph_msg_put last one on ffff88020b412600 <7>[ 2240.698213] libceph: msg_kfree ffff88020b412600 <7>[ 2240.698214] libceph: msgpool_release osd_op_reply ffff88020b412100 <7>[ 2240.698216] libceph: ceph_msg_put last one on ffff88020b412100 <7>[ 2240.698217] libceph: msg_kfree ffff88020b412100 <7>[ 2240.698219] libceph: stop <7>[ 2240.704809] libceph: ceph_debugfs_client_init ffff88020d36c000 155e9d3f-16e7-4f5e-9e56-a90071f0ea16.client4119 <7>[ 2240.704849] libceph: prepare_read_tag ffff88020d36c410 <7>[ 2240.704856] libceph: try_read start on ffff88020d36c410 state 5 <7>[ 2240.704862] libceph: try_read tag 1 in_base_pos 0 <7>[ 2240.704872] libceph: __close_session closing mon0 <7>[ 2240.704873] libceph: try_read done on ffff88020d36c410 ret 0 <7>[ 2240.704876] libceph: try_write start ffff88020d36c410 state 5 <7>[ 2240.704884] libceph: try_write out_kvec_bytes 0 <7>[ 2240.704889] libceph: ceph_msg_revoke_incoming msg ffff88020b413d00 null con <7>[ 2240.704892] libceph: ceph_msg_revoke_incoming msg ffff88020b413f00 null con <7>[ 2240.704899] libceph: prepare_write_ack ffff88020d36c410 0 -> 2 <7>[ 2240.704905] libceph: try_write out_kvec_bytes 9 <7>[ 2240.704911] libceph: write_partial_kvec ffff88020d36c410 9 left <7>[ 2240.704922] libceph: write_partial_kvec ffff88020d36c410 0 left in 0 kvecs ret = 1 <7>[ 2240.704928] libceph: try_write nothing else to write. <7>[ 2240.704935] libceph: try_write done on ffff88020d36c410 ret 0 <7>[ 2240.705009] libceph: con_close ffff88020d36c410 peer 10.214.133.28:6789 <7>[ 2240.705014] libceph: reset_connection ffff88020d36c410 <7>[ 2240.705017] libceph: con_close_socket on ffff88020d36c410 sock ffff88020cfd3c00 <7>[ 2240.705054] libceph: ceph_sock_state_change ffff88020d36c410 state = 1 sk_state = 4 <7>[ 2240.705079] libceph: con_sock_state_closed con ffff88020d36c410 sock 3 -> 1 <7>[ 2240.705085] libceph: auth_reset ffff88020b2420c0 <7>[ 2240.705124] libceph: auth_destroy ffff88020b2420c0 <7>[ 2240.705128] libceph: ceph_msg_put last one on ffff88020b413c00 <7>[ 2240.705130] libceph: msg_kfree ffff88020b413c00 <7>[ 2240.705133] libceph: ceph_msg_put last one on ffff88020b413d00 <7>[ 2240.705136] libceph: msg_kfree ffff88020b413d00 <7>[ 2240.705139] libceph: ceph_msg_put last one on ffff88020b413e00 <7>[ 2240.705141] libceph: msg_kfree ffff88020b413e00 <7>[ 2240.705144] libceph: ceph_msg_put last one on ffff88020b413f00 <7>[ 2240.705147] libceph: msg_kfree ffff88020b413f00 <7>[ 2240.705150] libceph: ceph_debugfs_client_cleanup ffff88020d36c000 <7>[ 2240.705228] libceph: destroy_options ffff88020b413a00 <7>[ 2240.705233] libceph: destroy_client ffff88020d36c000 done <7>[ 2240.705236] libceph: destroy_options ffff88020b413a00 <7>[ 2240.705241] rbd: Error adding device 10.214.133.28:6789,10.214.133.30:6789,10.214.133.28:6790 name=afs afs afs254-vicepa <7>[ 2243.455897] libceph: monc delayed_work <7>[ 2243.455903] libceph: con_keepalive ffff8802214c4410 <7>[ 2243.455909] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 2243.455914] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 2243.455917] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 2243.455920] libceph: __send_subscribe sub_sent=0 exp=0 want_osd=0 <7>[ 2243.455923] libceph: __schedule_delayed after 2000 <7>[ 2243.455929] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 2243.455932] libceph: try_read tag 1 in_base_pos 0 <7>[ 2243.455938] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 2243.455941] libceph: try_write start ffff8802214c4410 state 5 <7>[ 2243.455943] libceph: try_write out_kvec_bytes 0 <7>[ 2243.455946] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 2243.455948] libceph: try_write out_kvec_bytes 1 <7>[ 2243.455951] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 2243.455960] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 2243.455962] libceph: try_write nothing else to write. <7>[ 2243.455965] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 2245.172907] libceph: osds timeout <7>[ 2245.172913] libceph: __remove_old_osds ffff8802214c4960 <7>[ 2260.146890] libceph: osds timeout <7>[ 2260.146896] libceph: __remove_old_osds ffff8802214c4960 <7>[ 2263.461140] libceph: monc delayed_work <7>[ 2263.461146] libceph: con_keepalive ffff8802214c4410 <7>[ 2263.461152] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 2263.461156] libceph: ceph_x_should_authenticate want=39 need=0 have=39 <7>[ 2263.461160] libceph: ceph_x_is_authenticated want=39 need=0 have=39 <7>[ 2263.461163] libceph: __send_subscribe sub_sent=0 exp=1 want_osd=0 <7>[ 2263.461168] libceph: ----- ffff88020d5b5400 to mon1 15=mon_subscribe len 31+0+0 ----- <7>[ 2263.461171] libceph: __schedule_delayed after 1000 <7>[ 2263.461178] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 2263.461181] libceph: try_read tag 1 in_base_pos 0 <7>[ 2263.461187] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 2263.461190] libceph: try_write start ffff8802214c4410 state 5 <7>[ 2263.461192] libceph: try_write out_kvec_bytes 0 <7>[ 2263.461196] libceph: prepare_write_message ffff88020d5b5400 seq 17 type 15 len 31+0+0 0 pgs <7>[ 2263.461200] libceph: prepare_write_message front_crc 2650579135 middle_crc 0 <7>[ 2263.461202] libceph: prepare_write_message_footer ffff8802214c4410 <7>[ 2263.461205] libceph: try_write out_kvec_bytes 98 <7>[ 2263.461208] libceph: write_partial_kvec ffff8802214c4410 98 left <7>[ 2263.461226] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 2263.461229] libceph: prepare_write_keepalive ffff8802214c4410 <7>[ 2263.461231] libceph: try_write out_kvec_bytes 1 <7>[ 2263.461234] libceph: write_partial_kvec ffff8802214c4410 1 left <7>[ 2263.461240] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 2263.461243] libceph: try_write nothing else to write. <7>[ 2263.461246] libceph: try_write done on ffff8802214c4410 ret 0 <7>[ 2263.461692] libceph: ceph_sock_data_ready on ffff8802214c4410 state = 5, queueing work <7>[ 2263.461702] libceph: queue_con_delay ffff8802214c4410 0 <7>[ 2263.461717] libceph: ceph_sock_data_ready on ffff8802214c4410 state = 5, queueing work <7>[ 2263.461721] libceph: queue_con_delay ffff8802214c4410 - already queued <7>[ 2263.461737] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 2263.461740] libceph: try_read tag 1 in_base_pos 0 <7>[ 2263.461747] libceph: try_read got tag 8 <7>[ 2263.461749] libceph: prepare_read_ack ffff8802214c4410 <7>[ 2263.461754] libceph: got ack for seq 17 type 15 at ffff88020d5b5400 <7>[ 2263.461757] libceph: prepare_read_tag ffff8802214c4410 <7>[ 2263.461759] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 2263.461762] libceph: try_read tag 1 in_base_pos 0 <7>[ 2263.461766] libceph: try_read got tag 7 <7>[ 2263.461769] libceph: prepare_read_message ffff8802214c4410 <7>[ 2263.461771] libceph: read_partial_message con ffff8802214c4410 msg (null) <7>[ 2263.461777] libceph: got hdr type 4 front 481 data 0 <7>[ 2263.461781] libceph: ceph_msg_new ffff88020d35cf00 front 481 <7>[ 2263.461790] libceph: read_partial_message got msg ffff88020d35cf00 481 (2132872380) + 0 (0) + 0 (0) <7>[ 2263.461795] libceph: ===== ffff88020d35cf00 32 from mon1 4=mon_map len 481+0 (2132872380 0 0) ===== <7>[ 2263.461799] libceph: have_debugfs_info fsid 1 globalid 4132 <7>[ 2263.461801] libceph: handle_monmap <7>[ 2263.461804] libceph: monmap_decode ffff88020d753404 ffff88020d7535e1 len 477 <7>[ 2263.461807] libceph: monmap_decode epoch 1, num_mon 3 <7>[ 2263.461810] libceph: monmap_decode mon0 is 10.214.133.28:6789 <7>[ 2263.461813] libceph: monmap_decode mon1 is 10.214.133.30:6789 <7>[ 2263.461816] libceph: monmap_decode mon2 is 10.214.133.28:6790 <7>[ 2263.461820] libceph: ceph_msg_put last one on ffff88020d35cf00 <7>[ 2263.461823] libceph: msg_kfree ffff88020d35cf00 <7>[ 2263.461827] libceph: prepare_read_tag ffff8802214c4410 <7>[ 2263.461829] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 2263.461832] libceph: try_read tag 1 in_base_pos 0 <7>[ 2263.461836] libceph: try_read got tag 7 <7>[ 2263.461839] libceph: prepare_read_message ffff8802214c4410 <7>[ 2263.461842] libceph: read_partial_message con ffff8802214c4410 msg (null) <7>[ 2263.461846] libceph: got hdr type 16 front 20 data 0 <7>[ 2263.461866] libceph: read_partial_message got msg ffff88020d5b4500 20 (473278946) + 0 (0) + 0 (0) <7>[ 2263.461871] libceph: ===== ffff88020d5b4500 33 from mon1 16=mon_subscribe_ack len 20+0 (473278946 0 0) ===== <7>[ 2263.461874] libceph: handle_subscribe_ack after 300 seconds <7>[ 2263.461877] libceph: prepare_read_tag ffff8802214c4410 <7>[ 2263.461880] libceph: try_read start on ffff8802214c4410 state 5 <7>[ 2263.461883] libceph: try_read tag 1 in_base_pos 0 <7>[ 2263.461887] libceph: try_read done on ffff8802214c4410 ret 0 <7>[ 2263.461890] libceph: try_write start ffff8802214c4410 state 5 <7>[ 2263.461892] libceph: try_write out_kvec_bytes 0 <7>[ 2263.461895] libceph: prepare_write_ack ffff8802214c4410 31 -> 33 <7>[ 2263.461898] libceph: try_write out_kvec_bytes 9 <7>[ 2263.461901] libceph: write_partial_kvec ffff8802214c4410 9 left <7>[ 2263.461908] libceph: write_partial_kvec ffff8802214c4410 0 left in 0 kvecs ret = 1 <7>[ 2263.461910] libceph: try_write nothing else to write. <7>[ 2263.461913] libceph: try_write done on ffff8802214c4410 ret 0 <1>[ 2265.138269] BUG: unable to handle kernel NULL pointer dereference at 00000000000001a3 <1>[ 2265.146144] IP: [] __lock_acquire+0x1be/0x1d10 <4>[ 2265.152171] PGD 0 <4>[ 2265.154204] Oops: 0002 [#1] SMP