Ceph

2012-05-23 12:47:00.286345 7fd8c1557700  0 log [WRN] : slow request 30.981818 seconds old, received at 2012-05-23 12:46:29.304466: osd_sub_op_reply(osd.0.0:91638 0.1e2 207a6fe2/10000001bf9.0000009d/head [push] ack, result = 0) v1 currently queued for pg

2012-05-23 12:47:00.286349 7fd8c1557700  0 log [WRN] : slow request 30.318584 seconds old, received at 2012-05-23 12:46:29.967700: osd_sub_op_reply(osd.0.0:91639 0.1e2 43d86fe2/10000001c12.00000063/head [push] ack, result = 0) v1 currently queued for pg

2012-05-23 12:47:02.422288 7fd8c4d5e700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fd8b6440700' had timed out after 30

2012-05-23 12:47:02.422313 7fd8c4d5e700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fd8b6c41700' had timed out after 30

2012-05-23 12:47:07.422379 7fd8c4d5e700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fd8b6440700' had timed out after 30

2012-05-23 12:47:07.422406 7fd8c4d5e700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fd8b6c41700' had timed out after 30

2012-05-23 12:47:12.422476 7fd8c4d5e700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fd8b6440700' had timed out after 30

2012-05-23 12:47:12.422498 7fd8c4d5e700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fd8b6c41700' had timed out after 30

2012-05-23 12:47:17.422577 7fd8c4d5e700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fd8b6440700' had timed out after 30

2012-05-23 12:47:17.422601 7fd8c4d5e700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fd8b6c41700' had timed out after 30

2012-05-23 12:47:22.422683 7fd8c4d5e700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fd8b6440700' had timed out after 30

2012-05-23 12:47:22.422711 7fd8c4d5e700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fd8b6c41700' had timed out after 30

2012-05-23 12:47:27.422777 7fd8c4d5e700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fd8b6440700' had timed out after 30

2012-05-23 12:47:27.422802 7fd8c4d5e700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fd8b6c41700' had timed out after 30

2012-05-23 12:47:28.289740 7fd8c1557700  0 log [WRN] : 6 slow requests, 6 included below; oldest blocked for > 60.363032 secs

2012-05-23 12:47:28.289752 7fd8c1557700  0 log [WRN] : slow request 60.363032 seconds old, received at 2012-05-23 12:46:27.926658: osd_sub_op(osd.9.0:69857 0.1 a9996001/10000002ef7.0000006d/head [push] v 49'3950 snapset=0=[]:[] snapc=0=[]) v6 currently started

2012-05-23 12:47:28.289756 7fd8c1557700  0 log [WRN] : slow request 60.294207 seconds old, received at 2012-05-23 12:46:27.995483: osd_sub_op(osd.11.0:94345 0.15a 1184015a/10000001fa0.0000005f/head [push] v 49'2290 snapset=0=[]:[] snapc=0=[]) v6 currently started

2012-05-23 12:47:28.289760 7fd8c1557700  0 log [WRN] : slow request 60.276494 seconds old, received at 2012-05-23 12:46:28.013196: osd_sub_op(osd.9.0:69858 0.1 b5b96001/100000008dc.0000002d/head [push] v 11'440 snapset=0=[]:[] snapc=0=[]) v6 currently queued for pg

2012-05-23 12:47:28.289762 7fd8c1557700  0 log [WRN] : slow request 60.272563 seconds old, received at 2012-05-23 12:46:28.017127: osd_sub_op(osd.10.0:70596 0.8a de4da08a/10000001d66.00000022/head [push] v 49'2069 snapset=0=[]:[] snapc=0=[]) v6 currently queued for pg

2012-05-23 12:47:28.289765 7fd8c1557700  0 log [WRN] : slow request 60.046962 seconds old, received at 2012-05-23 12:46:28.242728: osd_sub_op(osd.10.0:70597 0.8a 298da08a/1000000265a.000003f3/head [push] v 49'3147 snapset=0=[]:[] snapc=0=[]) v6 currently queued for pg

2012-05-23 12:47:29.289926 7fd8c1557700  0 log [WRN] : 11 slow requests, 6 included below; oldest blocked for > 61.363189 secs

2012-05-23 12:47:29.289940 7fd8c1557700  0 log [WRN] : slow request 61.032098 seconds old, received at 2012-05-23 12:46:28.257749: osd_sub_op(osd.10.0:70598 0.8a cc9da08a/10000000066.000000db/head [push] v 11'26 snapset=0=[]:[] snapc=0=[]) v6 currently queued for pg

2012-05-23 12:47:29.289950 7fd8c1557700  0 log [WRN] : slow request 60.988138 seconds old, received at 2012-05-23 12:46:28.301709: osd_sub_op_reply(osd.0.0:91635 0.1e2 56286fe2/1000000224d.0000006f/head [push] ack, result = 0) v1 currently queued for pg

2012-05-23 12:47:29.289955 7fd8c1557700  0 log [WRN] : slow request 60.988128 seconds old, received at 2012-05-23 12:46:28.301719: osd_sub_op(osd.10.0:70599 0.8a 63cda08a/100000018c6.000000c3/head [push] v 49'1617 snapset=0=[]:[] snapc=0=[]) v6 currently queued for pg

2012-05-23 12:47:29.289960 7fd8c1557700  0 log [WRN] : slow request 60.963591 seconds old, received at 2012-05-23 12:46:28.326256: osd_sub_op(osd.10.0:70600 0.8a a54ea08a/10000001376.0000001a/head [push] v 49'1130 snapset=0=[]:[] snapc=0=[]) v6 currently queued for pg

2012-05-23 12:47:29.289964 7fd8c1557700  0 log [WRN] : slow request 60.948793 seconds old, received at 2012-05-23 12:46:28.341054: pg_backfill(progress 0.8a e 1092/1092 lb de4da08a/10000001d66.00000022/head) v1 currently queued for pg

2012-05-23 12:47:30.290041 7fd8c1557700  0 log [WRN] : 14 slow requests, 4 included below; oldest blocked for > 62.363328 secs

2012-05-23 12:47:30.290051 7fd8c1557700  0 log [WRN] : slow request 61.948913 seconds old, received at 2012-05-23 12:46:28.341073: osd_sub_op_reply(osd.0.0:91636 0.1e2 28a96fe2/10000002c7a.000000e8/head [push] ack, result = 0) v1 currently queued for pg

2012-05-23 12:47:30.290057 7fd8c1557700  0 log [WRN] : slow request 61.632310 seconds old, received at 2012-05-23 12:46:28.657676: osd_sub_op_reply(osd.0.0:91637 0.1e2 58ca6fe2/10000002eba.00000064/head [push] ack, result = 0) v1 currently queued for pg

2012-05-23 12:47:30.290062 7fd8c1557700  0 log [WRN] : slow request 60.985520 seconds old, received at 2012-05-23 12:46:29.304466: osd_sub_op_reply(osd.0.0:91638 0.1e2 207a6fe2/10000001bf9.0000009d/head [push] ack, result = 0) v1 currently queued for pg

2012-05-23 12:47:30.290065 7fd8c1557700  0 log [WRN] : slow request 60.322286 seconds old, received at 2012-05-23 12:46:29.967700: osd_sub_op_reply(osd.0.0:91639 0.1e2 43d86fe2/10000001c12.00000063/head [push] ack, result = 0) v1 currently queued for pg

2012-05-23 12:47:32.423161 7fd8c4d5e700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fd8b6440700' had timed out after 30

2012-05-23 12:47:32.423192 7fd8c4d5e700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fd8b6c41700' had timed out after 30

2012-05-23 12:47:37.423267 7fd8c4d5e700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fd8b6440700' had timed out after 30

2012-05-23 12:47:37.423289 7fd8c4d5e700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fd8b6c41700' had timed out after 30

2012-05-23 12:47:42.423351 7fd8c4d5e700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fd8b6440700' had timed out after 30

2012-05-23 12:47:42.423366 7fd8c4d5e700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fd8b6c41700' had timed out after 30

2012-05-23 12:47:47.423438 7fd8c4d5e700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fd8b6440700' had timed out after 30

2012-05-23 12:47:47.423461 7fd8c4d5e700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fd8b6c41700' had timed out after 30

2012-05-23 12:47:52.423532 7fd8c4d5e700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fd8b6440700' had timed out after 30

2012-05-23 12:47:52.423551 7fd8c4d5e700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fd8b6c41700' had timed out after 30

2012-05-23 12:47:57.423622 7fd8c4d5e700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fd8b6440700' had timed out after 30

2012-05-23 12:47:57.423655 7fd8c4d5e700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fd8b6c41700' had timed out after 30

2012-05-23 12:48:02.423720 7fd8c4d5e700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fd8b6440700' had timed out after 30

2012-05-23 12:48:02.423737 7fd8c4d5e700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fd8b6c41700' had timed out after 30

2012-05-23 12:48:07.423804 7fd8c4d5e700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fd8b6440700' had timed out after 30

2012-05-23 12:48:07.423824 7fd8c4d5e700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fd8b6c41700' had timed out after 30

2012-05-23 12:48:12.423894 7fd8c4d5e700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fd8b6440700' had timed out after 30

2012-05-23 12:48:12.423917 7fd8c4d5e700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fd8b6c41700' had timed out after 30

2012-05-23 12:48:17.423994 7fd8c4d5e700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fd8b6440700' had timed out after 30

2012-05-23 12:48:17.424050 7fd8c4d5e700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fd8b6c41700' had timed out after 30

2012-05-23 12:48:22.424121 7fd8c4d5e700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fd8b6440700' had timed out after 30

2012-05-23 12:48:22.424147 7fd8c4d5e700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fd8b6c41700' had timed out after 30

2012-05-23 12:48:27.424229 7fd8c4d5e700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fd8b6440700' had timed out after 30

2012-05-23 12:48:27.424262 7fd8c4d5e700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fd8b6c41700' had timed out after 30

2012-05-23 12:48:28.297739 7fd8c1557700  0 log [WRN] : 6 slow requests, 6 included below; oldest blocked for > 120.370998 secs

2012-05-23 12:48:28.297754 7fd8c1557700  0 log [WRN] : slow request 120.370998 seconds old, received at 2012-05-23 12:46:27.926658: osd_sub_op(osd.9.0:69857 0.1 a9996001/10000002ef7.0000006d/head [push] v 49'3950 snapset=0=[]:[] snapc=0=[]) v6 currently started

2012-05-23 12:48:28.297760 7fd8c1557700  0 log [WRN] : slow request 120.302173 seconds old, received at 2012-05-23 12:46:27.995483: osd_sub_op(osd.11.0:94345 0.15a 1184015a/10000001fa0.0000005f/head [push] v 49'2290 snapset=0=[]:[] snapc=0=[]) v6 currently started

2012-05-23 12:48:28.297766 7fd8c1557700  0 log [WRN] : slow request 120.284460 seconds old, received at 2012-05-23 12:46:28.013196: osd_sub_op(osd.9.0:69858 0.1 b5b96001/100000008dc.0000002d/head [push] v 11'440 snapset=0=[]:[] snapc=0=[]) v6 currently queued for pg

2012-05-23 12:48:28.297770 7fd8c1557700  0 log [WRN] : slow request 120.280529 seconds old, received at 2012-05-23 12:46:28.017127: osd_sub_op(osd.10.0:70596 0.8a de4da08a/10000001d66.00000022/head [push] v 49'2069 snapset=0=[]:[] snapc=0=[]) v6 currently queued for pg

2012-05-23 12:48:28.297773 7fd8c1557700  0 log [WRN] : slow request 120.054928 seconds old, received at 2012-05-23 12:46:28.242728: osd_sub_op(osd.10.0:70597 0.8a 298da08a/1000000265a.000003f3/head [push] v 49'3147 snapset=0=[]:[] snapc=0=[]) v6 currently queued for pg

2012-05-23 12:48:29.297862 7fd8c1557700  0 log [WRN] : 11 slow requests, 6 included below; oldest blocked for > 121.371126 secs

2012-05-23 12:48:29.297877 7fd8c1557700  0 log [WRN] : slow request 121.040035 seconds old, received at 2012-05-23 12:46:28.257749: osd_sub_op(osd.10.0:70598 0.8a cc9da08a/10000000066.000000db/head [push] v 11'26 snapset=0=[]:[] snapc=0=[]) v6 currently queued for pg

2012-05-23 12:48:29.297882 7fd8c1557700  0 log [WRN] : slow request 120.996075 seconds old, received at 2012-05-23 12:46:28.301709: osd_sub_op_reply(osd.0.0:91635 0.1e2 56286fe2/1000000224d.0000006f/head [push] ack, result = 0) v1 currently queued for pg

2012-05-23 12:48:29.297886 7fd8c1557700  0 log [WRN] : slow request 120.996065 seconds old, received at 2012-05-23 12:46:28.301719: osd_sub_op(osd.10.0:70599 0.8a 63cda08a/100000018c6.000000c3/head [push] v 49'1617 snapset=0=[]:[] snapc=0=[]) v6 currently queued for pg

2012-05-23 12:48:29.297891 7fd8c1557700  0 log [WRN] : slow request 120.971528 seconds old, received at 2012-05-23 12:46:28.326256: osd_sub_op(osd.10.0:70600 0.8a a54ea08a/10000001376.0000001a/head [push] v 49'1130 snapset=0=[]:[] snapc=0=[]) v6 currently queued for pg

2012-05-23 12:48:29.297894 7fd8c1557700  0 log [WRN] : slow request 120.956730 seconds old, received at 2012-05-23 12:46:28.341054: pg_backfill(progress 0.8a e 1092/1092 lb de4da08a/10000001d66.00000022/head) v1 currently queued for pg

2012-05-23 12:48:30.297977 7fd8c1557700  0 log [WRN] : 14 slow requests, 4 included below; oldest blocked for > 122.371253 secs

2012-05-23 12:48:30.297993 7fd8c1557700  0 log [WRN] : slow request 121.956838 seconds old, received at 2012-05-23 12:46:28.341073: osd_sub_op_reply(osd.0.0:91636 0.1e2 28a96fe2/10000002c7a.000000e8/head [push] ack, result = 0) v1 currently queued for pg

2012-05-23 12:48:30.297998 7fd8c1557700  0 log [WRN] : slow request 121.640235 seconds old, received at 2012-05-23 12:46:28.657676: osd_sub_op_reply(osd.0.0:91637 0.1e2 58ca6fe2/10000002eba.00000064/head [push] ack, result = 0) v1 currently queued for pg

2012-05-23 12:48:30.298024 7fd8c1557700  0 log [WRN] : slow request 120.993445 seconds old, received at 2012-05-23 12:46:29.304466: osd_sub_op_reply(osd.0.0:91638 0.1e2 207a6fe2/10000001bf9.0000009d/head [push] ack, result = 0) v1 currently queued for pg

2012-05-23 12:48:30.298031 7fd8c1557700  0 log [WRN] : slow request 120.330211 seconds old, received at 2012-05-23 12:46:29.967700: osd_sub_op_reply(osd.0.0:91639 0.1e2 43d86fe2/10000001c12.00000063/head [push] ack, result = 0) v1 currently queued for pg

2012-05-23 12:48:32.424344 7fd8c4d5e700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fd8b6440700' had timed out after 30

2012-05-23 12:48:32.424371 7fd8c4d5e700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fd8b6c41700' had timed out after 30

2012-05-23 12:48:37.424442 7fd8c4d5e700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fd8b6440700' had timed out after 30

2012-05-23 12:48:37.424464 7fd8c4d5e700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fd8b6c41700' had timed out after 30

2012-05-23 12:48:42.424542 7fd8c4d5e700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fd8b6440700' had timed out after 30

2012-05-23 12:48:42.424569 7fd8c4d5e700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fd8b6c41700' had timed out after 30

2012-05-23 12:48:47.424671 7fd8c4d5e700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fd8b6440700' had timed out after 30

2012-05-23 12:48:47.424693 7fd8c4d5e700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fd8b6c41700' had timed out after 30

2012-05-23 12:48:50.313742 7fd8b6440700  1 heartbeat_map reset_timeout 'OSD::op_tp thread 0x7fd8b6440700' had timed out after 30

2012-05-23 12:48:50.313850 7fd8b6c41700  1 heartbeat_map reset_timeout 'OSD::op_tp thread 0x7fd8b6c41700' had timed out after 30

2012-05-23 12:50:28.318131 7fd8c1557700  0 log [WRN] : 6 slow requests, 6 included below; oldest blocked for > 240.304873 secs

2012-05-23 12:50:28.318139 7fd8c1557700  0 log [WRN] : slow request 240.304873 seconds old, received at 2012-05-23 12:46:28.013196: osd_sub_op(osd.9.0:69858 0.1 b5b96001/100000008dc.0000002d/head [push] v 11'440 snapset=0=[]:[] snapc=0=[]) v6 currently queued for pg

2012-05-23 12:50:28.318144 7fd8c1557700  0 log [WRN] : slow request 240.300942 seconds old, received at 2012-05-23 12:46:28.017127: osd_sub_op(osd.10.0:70596 0.8a de4da08a/10000001d66.00000022/head [push] v 49'2069 snapset=0=[]:[] snapc=0=[]) v6 currently queued for pg

2012-05-23 12:50:28.318147 7fd8c1557700  0 log [WRN] : slow request 240.075341 seconds old, received at 2012-05-23 12:46:28.242728: osd_sub_op(osd.10.0:70597 0.8a 298da08a/1000000265a.000003f3/head [push] v 49'3147 snapset=0=[]:[] snapc=0=[]) v6 currently queued for pg

2012-05-23 12:50:28.318149 7fd8c1557700  0 log [WRN] : slow request 240.060320 seconds old, received at 2012-05-23 12:46:28.257749: osd_sub_op(osd.10.0:70598 0.8a cc9da08a/10000000066.000000db/head [push] v 11'26 snapset=0=[]:[] snapc=0=[]) v6 currently queued for pg

2012-05-23 12:50:28.318151 7fd8c1557700  0 log [WRN] : slow request 240.016360 seconds old, received at 2012-05-23 12:46:28.301709: osd_sub_op_reply(osd.0.0:91635 0.1e2 56286fe2/1000000224d.0000006f/head [push] ack, result = 0) v1 currently queued for pg

2012-05-23 12:50:29.318959 7fd8c1557700  0 log [WRN] : 11 slow requests, 6 included below; oldest blocked for > 241.305663 secs

2012-05-23 12:50:29.318983 7fd8c1557700  0 log [WRN] : slow request 241.017140 seconds old, received at 2012-05-23 12:46:28.301719: osd_sub_op(osd.10.0:70599 0.8a 63cda08a/100000018c6.000000c3/head [push] v 49'1617 snapset=0=[]:[] snapc=0=[]) v6 currently queued for pg

2012-05-23 12:50:29.318988 7fd8c1557700  0 log [WRN] : slow request 240.992603 seconds old, received at 2012-05-23 12:46:28.326256: osd_sub_op(osd.10.0:70600 0.8a a54ea08a/10000001376.0000001a/head [push] v 49'1130 snapset=0=[]:[] snapc=0=[]) v6 currently queued for pg

2012-05-23 12:50:29.318993 7fd8c1557700  0 log [WRN] : slow request 240.977805 seconds old, received at 2012-05-23 12:46:28.341054: pg_backfill(progress 0.8a e 1092/1092 lb de4da08a/10000001d66.00000022/head) v1 currently queued for pg

2012-05-23 12:50:29.318998 7fd8c1557700  0 log [WRN] : slow request 240.977786 seconds old, received at 2012-05-23 12:46:28.341073: osd_sub_op_reply(osd.0.0:91636 0.1e2 28a96fe2/10000002c7a.000000e8/head [push] ack, result = 0) v1 currently queued for pg

2012-05-23 12:50:29.319014 7fd8c1557700  0 log [WRN] : slow request 240.661183 seconds old, received at 2012-05-23 12:46:28.657676: osd_sub_op_reply(osd.0.0:91637 0.1e2 58ca6fe2/10000002eba.00000064/head [push] ack, result = 0) v1 currently queued for pg

2012-05-23 12:50:30.319033 7fd8c1557700  0 log [WRN] : 12 slow requests, 2 included below; oldest blocked for > 242.305774 secs

2012-05-23 12:50:30.319045 7fd8c1557700  0 log [WRN] : slow request 241.014504 seconds old, received at 2012-05-23 12:46:29.304466: osd_sub_op_reply(osd.0.0:91638 0.1e2 207a6fe2/10000001bf9.0000009d/head [push] ack, result = 0) v1 currently queued for pg

2012-05-23 12:50:30.319049 7fd8c1557700  0 log [WRN] : slow request 240.351270 seconds old, received at 2012-05-23 12:46:29.967700: osd_sub_op_reply(osd.0.0:91639 0.1e2 43d86fe2/10000001c12.00000063/head [push] ack, result = 0) v1 currently queued for pg

2012-05-23 12:53:38.948023 7fd8b0807700  0 -- 192.168.1.101:6804/30866 >> 192.168.1.101:6805/30955 pipe(0x5f3c000 sd=61 pgs=63 cs=1 l=0).fault with nothing to send, going to standby

2012-05-23 12:53:40.590523 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:53:40.590520) v2 because of no pipe on con 0x6783680

2012-05-23 12:53:42.890827 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:53:42.890824) v2 because of no pipe on con 0x6783680

2012-05-23 12:53:44.591155 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:53:44.591152) v2 because of no pipe on con 0x6783680

2012-05-23 12:53:48.091486 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:53:48.091482) v2 because of no pipe on con 0x6783680

2012-05-23 12:53:49.191830 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:53:49.191826) v2 because of no pipe on con 0x6783680

2012-05-23 12:53:50.892130 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:53:50.892127) v2 because of no pipe on con 0x6783680

2012-05-23 12:53:52.592419 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:53:52.592416) v2 because of no pipe on con 0x6783680

2012-05-23 12:53:55.349112 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:53:35.349111)

2012-05-23 12:53:56.349338 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:53:36.349337)

2012-05-23 12:53:57.349560 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:53:37.349560)

2012-05-23 12:53:57.892724 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:53:57.892721) v2 because of no pipe on con 0x6783680

2012-05-23 12:53:57.892787 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:53:37.892786)

2012-05-23 12:53:58.349720 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:53:38.349719)

2012-05-23 12:53:59.349867 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:53:39.349866)

2012-05-23 12:54:00.193037 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:54:00.193034) v2 because of no pipe on con 0x6783680

2012-05-23 12:54:00.193095 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:53:40.193095)

2012-05-23 12:54:00.349998 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:53:40.349997)

2012-05-23 12:54:01.350138 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:53:41.350137)

2012-05-23 12:54:02.350288 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:53:42.350287)

2012-05-23 12:54:03.350438 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:53:43.350437)

2012-05-23 12:54:04.293337 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:54:04.293334) v2 because of no pipe on con 0x6783680

2012-05-23 12:54:04.293398 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:53:44.293397)

2012-05-23 12:54:04.350607 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:53:44.350606)

2012-05-23 12:54:05.350751 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:53:45.350750)

2012-05-23 12:54:05.993647 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:54:05.993644) v2 because of no pipe on con 0x6783680

2012-05-23 12:54:05.993720 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:53:45.993719)

2012-05-23 12:54:06.350937 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:53:46.350937)

2012-05-23 12:54:07.351076 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:53:47.351075)

2012-05-23 12:54:07.693989 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:54:07.693985) v2 because of no pipe on con 0x6783680

2012-05-23 12:54:07.694099 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:53:47.694098)

2012-05-23 12:54:08.354101 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:53:48.354100)

2012-05-23 12:54:09.354384 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:53:49.354382)

2012-05-23 12:54:10.354529 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:53:50.354528)

2012-05-23 12:54:11.354677 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:53:51.354677)

2012-05-23 12:54:12.354809 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:53:52.354808)

2012-05-23 12:54:12.394496 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:54:12.394489) v2 because of no pipe on con 0x6783680

2012-05-23 12:54:12.394626 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:53:52.394626)

2012-05-23 12:54:13.354964 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:53:53.354964)

2012-05-23 12:54:14.355088 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:53:54.355087)

2012-05-23 12:54:15.355228 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:53:55.355227)

2012-05-23 12:54:15.894945 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:54:15.894941) v2 because of no pipe on con 0x6783680

2012-05-23 12:54:15.895019 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:53:55.895019)

2012-05-23 12:54:16.355368 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:53:56.355367)

2012-05-23 12:54:17.355501 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:53:57.355501)

2012-05-23 12:54:18.355655 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:53:58.355654)

2012-05-23 12:54:19.355815 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:53:59.355814)

2012-05-23 12:54:19.395265 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:54:19.395261) v2 because of no pipe on con 0x6783680

2012-05-23 12:54:19.395338 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:53:59.395338)

2012-05-23 12:54:20.355975 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:00.355974)

2012-05-23 12:54:21.356114 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:01.356114)

2012-05-23 12:54:22.356275 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:02.356266)

2012-05-23 12:54:22.895570 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:54:22.895566) v2 because of no pipe on con 0x6783680

2012-05-23 12:54:22.895632 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:02.895631)

2012-05-23 12:54:23.356432 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:03.356431)

2012-05-23 12:54:24.356548 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:04.356547)

2012-05-23 12:54:25.356919 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:05.356918)

2012-05-23 12:54:26.357096 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:06.357095)

2012-05-23 12:54:26.995964 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:54:26.995958) v2 because of no pipe on con 0x6783680

2012-05-23 12:54:26.996058 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:06.996057)

2012-05-23 12:54:27.358058 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:07.358057)

2012-05-23 12:54:27.496298 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:54:27.496295) v2 because of no pipe on con 0x6783680

2012-05-23 12:54:27.496369 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:07.496368)

2012-05-23 12:54:27.996615 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:54:27.996612) v2 because of no pipe on con 0x6783680

2012-05-23 12:54:27.996679 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:07.996679)

2012-05-23 12:54:28.358197 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:08.358196)

2012-05-23 12:54:28.358262 7fd8c1557700  0 log [WRN] : 6 slow requests, 6 included below; oldest blocked for > 480.345029 secs

2012-05-23 12:54:28.358266 7fd8c1557700  0 log [WRN] : slow request 480.345029 seconds old, received at 2012-05-23 12:46:28.013196: osd_sub_op(osd.9.0:69858 0.1 b5b96001/100000008dc.0000002d/head [push] v 11'440 snapset=0=[]:[] snapc=0=[]) v6 currently queued for pg

2012-05-23 12:54:28.358269 7fd8c1557700  0 log [WRN] : slow request 480.341098 seconds old, received at 2012-05-23 12:46:28.017127: osd_sub_op(osd.10.0:70596 0.8a de4da08a/10000001d66.00000022/head [push] v 49'2069 snapset=0=[]:[] snapc=0=[]) v6 currently queued for pg

2012-05-23 12:54:28.358271 7fd8c1557700  0 log [WRN] : slow request 480.115497 seconds old, received at 2012-05-23 12:46:28.242728: osd_sub_op(osd.10.0:70597 0.8a 298da08a/1000000265a.000003f3/head [push] v 49'3147 snapset=0=[]:[] snapc=0=[]) v6 currently queued for pg

2012-05-23 12:54:28.358273 7fd8c1557700  0 log [WRN] : slow request 480.100476 seconds old, received at 2012-05-23 12:46:28.257749: osd_sub_op(osd.10.0:70598 0.8a cc9da08a/10000000066.000000db/head [push] v 11'26 snapset=0=[]:[] snapc=0=[]) v6 currently queued for pg

2012-05-23 12:54:28.358275 7fd8c1557700  0 log [WRN] : slow request 480.056516 seconds old, received at 2012-05-23 12:46:28.301709: osd_sub_op_reply(osd.0.0:91635 0.1e2 56286fe2/1000000224d.0000006f/head [push] ack, result = 0) v1 currently queued for pg

2012-05-23 12:54:29.358318 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:09.358318)

2012-05-23 12:54:29.358387 7fd8c1557700  0 log [WRN] : 11 slow requests, 6 included below; oldest blocked for > 481.345149 secs

2012-05-23 12:54:29.358392 7fd8c1557700  0 log [WRN] : slow request 481.056626 seconds old, received at 2012-05-23 12:46:28.301719: osd_sub_op(osd.10.0:70599 0.8a 63cda08a/100000018c6.000000c3/head [push] v 49'1617 snapset=0=[]:[] snapc=0=[]) v6 currently queued for pg

2012-05-23 12:54:29.358395 7fd8c1557700  0 log [WRN] : slow request 481.032089 seconds old, received at 2012-05-23 12:46:28.326256: osd_sub_op(osd.10.0:70600 0.8a a54ea08a/10000001376.0000001a/head [push] v 49'1130 snapset=0=[]:[] snapc=0=[]) v6 currently queued for pg

2012-05-23 12:54:29.358397 7fd8c1557700  0 log [WRN] : slow request 481.017291 seconds old, received at 2012-05-23 12:46:28.341054: pg_backfill(progress 0.8a e 1092/1092 lb de4da08a/10000001d66.00000022/head) v1 currently queued for pg

2012-05-23 12:54:29.358399 7fd8c1557700  0 log [WRN] : slow request 481.017272 seconds old, received at 2012-05-23 12:46:28.341073: osd_sub_op_reply(osd.0.0:91636 0.1e2 28a96fe2/10000002c7a.000000e8/head [push] ack, result = 0) v1 currently queued for pg

2012-05-23 12:54:29.358401 7fd8c1557700  0 log [WRN] : slow request 480.700669 seconds old, received at 2012-05-23 12:46:28.657676: osd_sub_op_reply(osd.0.0:91637 0.1e2 58ca6fe2/10000002eba.00000064/head [push] ack, result = 0) v1 currently queued for pg

2012-05-23 12:54:30.358474 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:10.358473)

2012-05-23 12:54:30.358523 7fd8c1557700  0 log [WRN] : 12 slow requests, 2 included below; oldest blocked for > 482.345306 secs

2012-05-23 12:54:30.358531 7fd8c1557700  0 log [WRN] : slow request 481.054036 seconds old, received at 2012-05-23 12:46:29.304466: osd_sub_op_reply(osd.0.0:91638 0.1e2 207a6fe2/10000001bf9.0000009d/head [push] ack, result = 0) v1 currently queued for pg

2012-05-23 12:54:30.358534 7fd8c1557700  0 log [WRN] : slow request 480.390802 seconds old, received at 2012-05-23 12:46:29.967700: osd_sub_op_reply(osd.0.0:91639 0.1e2 43d86fe2/10000001c12.00000063/head [push] ack, result = 0) v1 currently queued for pg

2012-05-23 12:54:31.358665 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:11.358664)

2012-05-23 12:54:32.358829 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:12.358828)

2012-05-23 12:54:33.358989 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:13.358989)

2012-05-23 12:54:33.896965 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:54:33.896961) v2 because of no pipe on con 0x6783680

2012-05-23 12:54:33.897041 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:13.897040)

2012-05-23 12:54:34.359126 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:14.359126)

2012-05-23 12:54:35.359266 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:15.359265)

2012-05-23 12:54:35.597279 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:54:35.597276) v2 because of no pipe on con 0x6783680

2012-05-23 12:54:35.597338 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:15.597338)

2012-05-23 12:54:36.359410 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:16.359410)

2012-05-23 12:54:37.359566 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:17.359566)

2012-05-23 12:54:38.359714 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:18.359713)

2012-05-23 12:54:39.359862 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:19.359861)

2012-05-23 12:54:39.697595 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:54:39.697593) v2 because of no pipe on con 0x6783680

2012-05-23 12:54:39.697658 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:19.697658)

2012-05-23 12:54:40.360032 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:20.360031)

2012-05-23 12:54:41.360183 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:21.360182)

2012-05-23 12:54:42.360358 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:22.360358)

2012-05-23 12:54:43.360512 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:23.360511)

2012-05-23 12:54:44.360662 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:24.360661)

2012-05-23 12:54:44.997907 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:54:44.997903) v2 because of no pipe on con 0x6783680

2012-05-23 12:54:44.997970 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:24.997969)

2012-05-23 12:54:45.360828 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:25.360827)

2012-05-23 12:54:46.360968 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:26.360967)

2012-05-23 12:54:47.298197 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:54:47.298194) v2 because of no pipe on con 0x6783680

2012-05-23 12:54:47.298255 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:27.298254)

2012-05-23 12:54:47.361125 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:27.361124)

2012-05-23 12:54:48.361298 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:28.361297)

2012-05-23 12:54:49.361444 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:29.361443)

2012-05-23 12:54:50.361593 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:30.361593)

2012-05-23 12:54:50.798495 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:54:50.798491) v2 because of no pipe on con 0x6783680

2012-05-23 12:54:50.798570 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:30.798562)

2012-05-23 12:54:51.361752 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:31.361752)

2012-05-23 12:54:52.361909 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:32.361908)

2012-05-23 12:54:53.362082 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:33.362081)

2012-05-23 12:54:54.362220 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:34.362220)

2012-05-23 12:54:55.363634 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:35.363633)

2012-05-23 12:54:56.098871 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:54:56.098866) v2 because of no pipe on con 0x6783680

2012-05-23 12:54:56.098956 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:36.098956)

2012-05-23 12:54:56.363784 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:36.363783)

2012-05-23 12:54:57.363938 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:37.363937)

2012-05-23 12:54:58.364091 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:38.364091)

2012-05-23 12:54:59.364239 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:39.364238)

2012-05-23 12:55:00.199200 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:55:00.199196) v2 because of no pipe on con 0x6783680

2012-05-23 12:55:00.199268 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:40.199267)

2012-05-23 12:55:00.364383 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:40.364382)

2012-05-23 12:55:01.364539 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:41.364538)

2012-05-23 12:55:02.364714 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:42.364713)

2012-05-23 12:55:03.364879 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:43.364878)

2012-05-23 12:55:04.365395 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:44.365394)

2012-05-23 12:55:04.899541 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:55:04.899538) v2 because of no pipe on con 0x6783680

2012-05-23 12:55:04.899606 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:44.899605)

2012-05-23 12:55:05.365544 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:45.365543)

2012-05-23 12:55:06.365676 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:46.365676)

2012-05-23 12:55:07.365806 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:47.365806)

2012-05-23 12:55:08.365939 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:48.365939)

2012-05-23 12:55:09.366421 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:49.366421)

2012-05-23 12:55:09.600524 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:55:09.600519) v2 because of no pipe on con 0x6783680

2012-05-23 12:55:09.600631 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:49.600631)

2012-05-23 12:55:10.100873 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:55:10.100870) v2 because of no pipe on con 0x6783680

2012-05-23 12:55:10.100934 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:50.100934)

2012-05-23 12:55:10.366580 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:50.366580)

2012-05-23 12:55:11.366714 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:51.366713)

2012-05-23 12:55:12.366887 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:52.366886)

2012-05-23 12:55:13.367058 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:53.367058)

2012-05-23 12:55:14.367191 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:54.367190)

2012-05-23 12:55:15.367309 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:55.367308)

2012-05-23 12:55:15.401184 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:55:15.401181) v2 because of no pipe on con 0x6783680

2012-05-23 12:55:15.401246 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:55.401245)

2012-05-23 12:55:16.367710 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:56.367709)

2012-05-23 12:55:17.367851 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:57.367850)

2012-05-23 12:55:17.701496 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:55:17.701492) v2 because of no pipe on con 0x6783680

2012-05-23 12:55:17.701567 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:57.701567)

2012-05-23 12:55:18.368013 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:58.368012)

2012-05-23 12:55:18.801819 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:55:18.801816) v2 because of no pipe on con 0x6783680

2012-05-23 12:55:18.801881 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:58.801881)

2012-05-23 12:55:19.368147 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:54:59.368147)

2012-05-23 12:55:20.368310 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:00.368309)

2012-05-23 12:55:21.102125 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:55:21.102122) v2 because of no pipe on con 0x6783680

2012-05-23 12:55:21.102191 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:01.102191)

2012-05-23 12:55:21.368466 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:01.368465)

2012-05-23 12:55:22.368627 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:02.368627)

2012-05-23 12:55:23.368799 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:03.368798)

2012-05-23 12:55:24.368962 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:04.368962)

2012-05-23 12:55:25.369117 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:05.369116)

2012-05-23 12:55:25.802460 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:55:25.802456) v2 because of no pipe on con 0x6783680

2012-05-23 12:55:25.802523 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:05.802523)

2012-05-23 12:55:26.369267 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:06.369266)

2012-05-23 12:55:26.902768 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:55:26.902765) v2 because of no pipe on con 0x6783680

2012-05-23 12:55:26.902831 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:06.902830)

2012-05-23 12:55:27.369448 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:07.369447)

2012-05-23 12:55:28.369597 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:08.369596)

2012-05-23 12:55:29.369767 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:09.369766)

2012-05-23 12:55:30.369923 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:10.369922)

2012-05-23 12:55:31.003062 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:55:31.003059) v2 because of no pipe on con 0x6783680

2012-05-23 12:55:31.003121 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:11.003120)

2012-05-23 12:55:31.370083 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:11.370083)

2012-05-23 12:55:32.370238 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:12.370230)

2012-05-23 12:55:33.371094 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:13.371093)

2012-05-23 12:55:33.903396 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:55:33.903393) v2 because of no pipe on con 0x6783680

2012-05-23 12:55:33.903840 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:13.903839)

2012-05-23 12:55:34.371269 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:14.371268)

2012-05-23 12:55:34.404129 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:55:34.404126) v2 because of no pipe on con 0x6783680

2012-05-23 12:55:34.404204 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:14.404204)

2012-05-23 12:55:35.371412 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:15.371411)

2012-05-23 12:55:36.371547 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:16.371546)

2012-05-23 12:55:37.371684 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:17.371684)

2012-05-23 12:55:38.371837 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:18.371837)

2012-05-23 12:55:39.371995 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:19.371987)

2012-05-23 12:55:39.704569 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:55:39.704566) v2 because of no pipe on con 0x6783680

2012-05-23 12:55:39.704765 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:19.704765)

2012-05-23 12:55:40.372161 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:20.372160)

2012-05-23 12:55:41.372304 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:21.372303)

2012-05-23 12:55:41.405079 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:55:41.405075) v2 because of no pipe on con 0x6783680

2012-05-23 12:55:41.405148 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:21.405147)

2012-05-23 12:55:42.372464 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:22.372463)

2012-05-23 12:55:43.372599 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:23.372599)

2012-05-23 12:55:44.372782 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:24.372781)

2012-05-23 12:55:44.905433 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:55:44.905430) v2 because of no pipe on con 0x6783680

2012-05-23 12:55:44.905504 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:24.905503)

2012-05-23 12:55:45.372924 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:25.372923)

2012-05-23 12:55:46.373125 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:26.373125)

2012-05-23 12:55:47.373273 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:27.373272)

2012-05-23 12:55:48.373447 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:28.373446)

2012-05-23 12:55:48.405931 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:55:48.405925) v2 because of no pipe on con 0x6783680

2012-05-23 12:55:48.406072 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:28.406071)

2012-05-23 12:55:49.373611 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:29.373610)

2012-05-23 12:55:50.373883 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:30.373883)

2012-05-23 12:55:51.374035 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:31.374034)

2012-05-23 12:55:52.374206 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:32.374206)

2012-05-23 12:55:53.106611 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:55:53.106607) v2 because of no pipe on con 0x6783680

2012-05-23 12:55:53.106720 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:33.106719)

2012-05-23 12:55:53.374364 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:33.374363)

2012-05-23 12:55:54.374511 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:34.374510)

2012-05-23 12:55:54.807112 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:55:54.807107) v2 because of no pipe on con 0x6783680

2012-05-23 12:55:54.807212 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:34.807211)

2012-05-23 12:55:55.374665 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:35.374665)

2012-05-23 12:55:56.374810 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:36.374809)

2012-05-23 12:55:57.374949 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:37.374949)

2012-05-23 12:55:58.375125 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:38.375124)

2012-05-23 12:55:59.375255 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:39.375254)

2012-05-23 12:56:00.107509 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:56:00.107506) v2 because of no pipe on con 0x6783680

2012-05-23 12:56:00.107613 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:40.107612)

2012-05-23 12:56:00.375409 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:40.375408)

2012-05-23 12:56:01.375546 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:41.375545)

2012-05-23 12:56:02.375700 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:42.375699)

2012-05-23 12:56:03.375851 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:43.375850)

2012-05-23 12:56:04.376015 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:44.376014)

2012-05-23 12:56:05.376178 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:45.376178)

2012-05-23 12:56:06.007875 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:56:06.007872) v2 because of no pipe on con 0x6783680

2012-05-23 12:56:06.007982 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:46.007982)

2012-05-23 12:56:06.376341 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:46.376341)

2012-05-23 12:56:07.376488 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:47.376487)

2012-05-23 12:56:07.708223 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:56:07.708220) v2 because of no pipe on con 0x6783680

2012-05-23 12:56:07.708290 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:47.708290)

2012-05-23 12:56:08.378835 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:48.378833)

2012-05-23 12:56:09.379034 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:49.379033)

2012-05-23 12:56:10.008571 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:56:10.008568) v2 because of no pipe on con 0x6783680

2012-05-23 12:56:10.008631 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:50.008630)

2012-05-23 12:56:10.379168 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:50.379167)

2012-05-23 12:56:11.108907 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:56:11.108904) v2 because of no pipe on con 0x6783680

2012-05-23 12:56:11.108983 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:51.108983)

2012-05-23 12:56:11.379321 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:51.379320)

2012-05-23 12:56:12.209267 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:56:12.209264) v2 because of no pipe on con 0x6783680

2012-05-23 12:56:12.209345 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:52.209344)

2012-05-23 12:56:12.379467 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:52.379466)

2012-05-23 12:56:13.379624 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:53.379623)

2012-05-23 12:56:14.379768 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:54.379767)

2012-05-23 12:56:15.379938 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:55.379937)

2012-05-23 12:56:16.309614 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:56:16.309611) v2 because of no pipe on con 0x6783680

2012-05-23 12:56:16.309693 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:56.309692)

2012-05-23 12:56:16.380080 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:56.380079)

2012-05-23 12:56:17.380224 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:57.380223)

2012-05-23 12:56:18.380377 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:58.380376)

2012-05-23 12:56:19.380503 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:55:59.380502)

2012-05-23 12:56:20.380630 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:00.380629)

2012-05-23 12:56:21.380770 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:01.380770)

2012-05-23 12:56:22.209938 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:56:22.209936) v2 because of no pipe on con 0x6783680

2012-05-23 12:56:22.210009 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:02.210009)

2012-05-23 12:56:22.380930 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:02.380929)

2012-05-23 12:56:23.310262 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:56:23.310257) v2 because of no pipe on con 0x6783680

2012-05-23 12:56:23.310345 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:03.310345)

2012-05-23 12:56:23.381098 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:03.381097)

2012-05-23 12:56:24.381242 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:04.381241)

2012-05-23 12:56:25.381384 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:05.381383)

2012-05-23 12:56:26.210608 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:56:26.210605) v2 because of no pipe on con 0x6783680

2012-05-23 12:56:26.210679 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:06.210678)

2012-05-23 12:56:26.381538 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:06.381537)

2012-05-23 12:56:27.381696 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:07.381695)

2012-05-23 12:56:28.381871 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:08.381870)

2012-05-23 12:56:29.382032 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:09.382032)

2012-05-23 12:56:30.310935 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:56:30.310932) v2 because of no pipe on con 0x6783680

2012-05-23 12:56:30.311025 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:10.311024)

2012-05-23 12:56:30.382176 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:10.382175)

2012-05-23 12:56:31.382330 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:11.382329)

2012-05-23 12:56:32.382468 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:12.382467)

2012-05-23 12:56:33.382636 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:13.382627)

2012-05-23 12:56:34.384134 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:14.384133)

2012-05-23 12:56:35.011270 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:56:35.011268) v2 because of no pipe on con 0x6783680

2012-05-23 12:56:35.011332 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:15.011332)

2012-05-23 12:56:35.384324 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:15.384323)

2012-05-23 12:56:36.384486 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:16.384486)

2012-05-23 12:56:37.384795 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:17.384794)

2012-05-23 12:56:38.384954 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:18.384953)

2012-05-23 12:56:39.385110 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:19.385109)

2012-05-23 12:56:40.311579 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:56:40.311575) v2 because of no pipe on con 0x6783680

2012-05-23 12:56:40.311644 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:20.311643)

2012-05-23 12:56:40.385394 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:20.385393)

2012-05-23 12:56:41.386261 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:21.386260)

2012-05-23 12:56:42.386423 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:22.386422)

2012-05-23 12:56:43.386582 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:23.386581)

2012-05-23 12:56:44.386767 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:24.386766)

2012-05-23 12:56:45.011898 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:56:45.011895) v2 because of no pipe on con 0x6783680

2012-05-23 12:56:45.011961 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:25.011960)

2012-05-23 12:56:45.386973 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:25.386972)

2012-05-23 12:56:46.387181 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:26.387180)

2012-05-23 12:56:47.387370 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:27.387369)

2012-05-23 12:56:48.387529 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:28.387528)

2012-05-23 12:56:49.387654 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:29.387653)

2012-05-23 12:56:49.712193 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:56:49.712190) v2 because of no pipe on con 0x6783680

2012-05-23 12:56:49.712259 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:29.712258)

2012-05-23 12:56:50.387805 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:30.387804)

2012-05-23 12:56:50.812514 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:56:50.812511) v2 because of no pipe on con 0x6783680

2012-05-23 12:56:50.812577 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:30.812577)

2012-05-23 12:56:51.312953 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:56:51.312949) v2 because of no pipe on con 0x6783680

2012-05-23 12:56:51.313104 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:31.313104)

2012-05-23 12:56:51.387960 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:31.387959)

2012-05-23 12:56:52.388116 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:32.388115)

2012-05-23 12:56:53.388277 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:33.388276)

2012-05-23 12:56:54.388399 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:34.388399)

2012-05-23 12:56:55.388558 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:35.388557)

2012-05-23 12:56:55.413336 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:56:55.413334) v2 because of no pipe on con 0x6783680

2012-05-23 12:56:55.413400 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:35.413399)

2012-05-23 12:56:56.388734 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:36.388733)

2012-05-23 12:56:57.388898 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:37.388897)

2012-05-23 12:56:57.713665 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:56:57.713661) v2 because of no pipe on con 0x6783680

2012-05-23 12:56:57.713743 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:37.713742)

2012-05-23 12:56:58.389071 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:38.389071)

2012-05-23 12:56:59.389233 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:39.389233)

2012-05-23 12:57:00.389410 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:40.389409)

2012-05-23 12:57:01.389556 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:41.389556)

2012-05-23 12:57:02.389690 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:42.389689)

2012-05-23 12:57:02.413981 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:57:02.413978) v2 because of no pipe on con 0x6783680

2012-05-23 12:57:02.414056 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:42.414055)

2012-05-23 12:57:03.389842 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:43.389841)

2012-05-23 12:57:04.389994 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:44.389993)

2012-05-23 12:57:05.390156 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:45.390155)

2012-05-23 12:57:05.914406 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:57:05.914404) v2 because of no pipe on con 0x6783680

2012-05-23 12:57:05.914469 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:45.914468)

2012-05-23 12:57:06.390315 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:46.390314)

2012-05-23 12:57:07.390464 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:47.390464)

2012-05-23 12:57:08.390628 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:48.390627)

2012-05-23 12:57:09.390787 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:49.390786)

2012-05-23 12:57:10.390943 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:50.390934)

2012-05-23 12:57:11.391048 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:51.391047)

2012-05-23 12:57:11.814708 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:57:11.814705) v2 because of no pipe on con 0x6783680

2012-05-23 12:57:11.814774 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:51.814774)

2012-05-23 12:57:12.391197 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:52.391197)

2012-05-23 12:57:12.915048 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:57:12.915045) v2 because of no pipe on con 0x6783680

2012-05-23 12:57:12.915118 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:52.915117)

2012-05-23 12:57:13.391335 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:53.391335)

2012-05-23 12:57:14.391465 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:54.391464)

2012-05-23 12:57:15.391610 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:55.391609)

2012-05-23 12:57:16.391746 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:56.391746)

2012-05-23 12:57:17.391890 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:57.391889)

2012-05-23 12:57:18.392071 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:58.392070)

2012-05-23 12:57:18.815566 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:57:18.815560) v2 because of no pipe on con 0x6783680

2012-05-23 12:57:18.815708 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:58.815706)

2012-05-23 12:57:19.392216 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:56:59.392215)

2012-05-23 12:57:20.392364 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:00.392364)

2012-05-23 12:57:21.392535 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:01.392534)

2012-05-23 12:57:22.315960 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:57:22.315957) v2 because of no pipe on con 0x6783680

2012-05-23 12:57:22.316032 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:02.316031)

2012-05-23 12:57:22.392720 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:02.392719)

2012-05-23 12:57:23.392882 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:03.392881)

2012-05-23 12:57:24.393053 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:04.393052)

2012-05-23 12:57:25.393200 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:05.393199)

2012-05-23 12:57:26.393355 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:06.393355)

2012-05-23 12:57:27.393511 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:07.393503)

2012-05-23 12:57:27.616279 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:57:27.616276) v2 because of no pipe on con 0x6783680

2012-05-23 12:57:27.616340 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:07.616339)

2012-05-23 12:57:28.393670 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:08.393661)

2012-05-23 12:57:29.393828 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:09.393827)

2012-05-23 12:57:30.393983 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:10.393983)

2012-05-23 12:57:31.394150 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:11.394150)

2012-05-23 12:57:31.716629 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:57:31.716625) v2 because of no pipe on con 0x6783680

2012-05-23 12:57:31.716708 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:11.716707)

2012-05-23 12:57:32.394312 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:12.394311)

2012-05-23 12:57:33.394459 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:13.394458)

2012-05-23 12:57:34.394607 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:14.394606)

2012-05-23 12:57:35.394771 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:15.394771)

2012-05-23 12:57:36.394904 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:16.394903)

2012-05-23 12:57:37.395048 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:17.395047)

2012-05-23 12:57:37.616971 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:57:37.616967) v2 because of no pipe on con 0x6783680

2012-05-23 12:57:37.617049 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:17.617049)

2012-05-23 12:57:38.395205 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:18.395204)

2012-05-23 12:57:39.395413 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:19.395412)

2012-05-23 12:57:40.395660 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:20.395659)

2012-05-23 12:57:41.395827 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:21.395826)

2012-05-23 12:57:42.395999 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:22.395998)

2012-05-23 12:57:42.917291 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:57:42.917288) v2 because of no pipe on con 0x6783680

2012-05-23 12:57:42.917353 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:22.917353)

2012-05-23 12:57:43.396186 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:23.396186)

2012-05-23 12:57:44.396339 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:24.396339)

2012-05-23 12:57:45.396484 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:25.396483)

2012-05-23 12:57:45.817611 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:57:45.817608) v2 because of no pipe on con 0x6783680

2012-05-23 12:57:45.817679 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:25.817678)

2012-05-23 12:57:46.396632 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:26.396631)

2012-05-23 12:57:47.396779 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:27.396778)

2012-05-23 12:57:48.396937 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:28.396936)

2012-05-23 12:57:49.397084 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:29.397083)

2012-05-23 12:57:50.397230 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:30.397229)

2012-05-23 12:57:51.397376 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:31.397375)

2012-05-23 12:57:51.717917 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:57:51.717915) v2 because of no pipe on con 0x6783680

2012-05-23 12:57:51.717975 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:31.717975)

2012-05-23 12:57:52.397511 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:32.397510)

2012-05-23 12:57:52.818222 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:57:52.818218) v2 because of no pipe on con 0x6783680

2012-05-23 12:57:52.818291 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:32.818290)

2012-05-23 12:57:53.397659 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:33.397659)

2012-05-23 12:57:54.397817 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:34.397816)

2012-05-23 12:57:55.397962 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:35.397961)

2012-05-23 12:57:56.398102 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:36.398101)

2012-05-23 12:57:57.398250 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:37.398249)

2012-05-23 12:57:58.118559 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:57:58.118555) v2 because of no pipe on con 0x6783680

2012-05-23 12:57:58.118628 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:38.118628)

2012-05-23 12:57:58.398429 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:38.398428)

2012-05-23 12:57:59.398587 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:39.398586)

2012-05-23 12:57:59.818905 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:57:59.818903) v2 because of no pipe on con 0x6783680

2012-05-23 12:57:59.818968 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:39.818968)

2012-05-23 12:58:00.319202 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:58:00.319198) v2 because of no pipe on con 0x6783680

2012-05-23 12:58:00.319264 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:40.319264)

2012-05-23 12:58:00.398763 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:40.398762)

2012-05-23 12:58:01.398919 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:41.398919)

2012-05-23 12:58:02.399076 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:42.399075)

2012-05-23 12:58:03.399246 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:43.399246)

2012-05-23 12:58:04.399409 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:44.399408)

2012-05-23 12:58:05.019495 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:58:05.019492) v2 because of no pipe on con 0x6783680

2012-05-23 12:58:05.019565 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:45.019565)

2012-05-23 12:58:05.399563 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:45.399562)

2012-05-23 12:58:06.399727 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:46.399726)

2012-05-23 12:58:07.399865 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:47.399864)

2012-05-23 12:58:07.919782 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:58:07.919779) v2 because of no pipe on con 0x6783680

2012-05-23 12:58:07.919845 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:47.919844)

2012-05-23 12:58:08.400031 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:48.400031)

2012-05-23 12:58:09.400188 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:49.400187)

2012-05-23 12:58:10.400323 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:50.400323)

2012-05-23 12:58:11.400459 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:51.400458)

2012-05-23 12:58:11.420118 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:58:11.420115) v2 because of no pipe on con 0x6783680

2012-05-23 12:58:11.420179 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:51.420179)

2012-05-23 12:58:12.400645 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:52.400645)

2012-05-23 12:58:13.120729 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:58:13.120564) v2 because of no pipe on con 0x6783680

2012-05-23 12:58:13.120860 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:53.120859)

2012-05-23 12:58:13.400793 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:53.400792)

2012-05-23 12:58:14.400969 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:54.400968)

2012-05-23 12:58:15.401160 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:55.401159)

2012-05-23 12:58:16.401287 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:56.401286)

2012-05-23 12:58:17.401441 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:57.401440)

2012-05-23 12:58:18.401625 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:58.401624)

2012-05-23 12:58:18.421104 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:58:18.421101) v2 because of no pipe on con 0x6783680

2012-05-23 12:58:18.421173 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:58.421172)

2012-05-23 12:58:19.401777 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:57:59.401776)

2012-05-23 12:58:20.401953 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:00.401952)

2012-05-23 12:58:20.721500 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:58:20.721496) v2 because of no pipe on con 0x6783680

2012-05-23 12:58:20.721599 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:00.721599)

2012-05-23 12:58:21.402099 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:01.402098)

2012-05-23 12:58:21.823976 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:58:21.823973) v2 because of no pipe on con 0x6783680

2012-05-23 12:58:21.824083 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:01.824083)

2012-05-23 12:58:22.402297 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:02.402296)

2012-05-23 12:58:23.402614 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:03.402614)

2012-05-23 12:58:24.402784 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:04.402783)

2012-05-23 12:58:25.402939 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:05.402938)

2012-05-23 12:58:26.403131 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:06.403130)

2012-05-23 12:58:26.524332 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:58:26.524329) v2 because of no pipe on con 0x6783680

2012-05-23 12:58:26.524388 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:06.524388)

2012-05-23 12:58:27.403286 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:07.403285)

2012-05-23 12:58:28.403453 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:08.403452)

2012-05-23 12:58:29.403666 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:09.403665)

2012-05-23 12:58:29.424719 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:58:29.424716) v2 because of no pipe on con 0x6783680

2012-05-23 12:58:29.424813 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:09.424812)

2012-05-23 12:58:30.403803 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:10.403802)

2012-05-23 12:58:31.125096 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:58:31.125093) v2 because of no pipe on con 0x6783680

2012-05-23 12:58:31.125170 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:11.125169)

2012-05-23 12:58:31.403931 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:11.403931)

2012-05-23 12:58:32.404061 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:12.404060)

2012-05-23 12:58:33.404206 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:13.404205)

2012-05-23 12:58:34.025427 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:58:34.025424) v2 because of no pipe on con 0x6783680

2012-05-23 12:58:34.025490 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:14.025490)

2012-05-23 12:58:34.404343 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:14.404342)

2012-05-23 12:58:35.125753 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:58:35.125749) v2 because of no pipe on con 0x6783680

2012-05-23 12:58:35.125829 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:15.125828)

2012-05-23 12:58:35.404490 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:15.404489)

2012-05-23 12:58:36.404622 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:16.404621)

2012-05-23 12:58:37.404778 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:17.404778)

2012-05-23 12:58:38.404914 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:18.404913)

2012-05-23 12:58:39.226079 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:58:39.226077) v2 because of no pipe on con 0x6783680

2012-05-23 12:58:39.226142 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:19.226142)

2012-05-23 12:58:39.405132 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:19.405131)

2012-05-23 12:58:40.405299 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:20.405298)

2012-05-23 12:58:41.405462 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:21.405461)

2012-05-23 12:58:42.405603 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:22.405603)

2012-05-23 12:58:42.726377 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:58:42.726373) v2 because of no pipe on con 0x6783680

2012-05-23 12:58:42.726436 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:22.726435)

2012-05-23 12:58:43.226682 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:58:43.226679) v2 because of no pipe on con 0x6783680

2012-05-23 12:58:43.226740 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:23.226739)

2012-05-23 12:58:43.405746 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:23.405745)

2012-05-23 12:58:44.405911 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:24.405903)

2012-05-23 12:58:45.406046 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:25.406046)

2012-05-23 12:58:46.406193 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:26.406193)

2012-05-23 12:58:47.406336 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:27.406335)

2012-05-23 12:58:48.406496 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:28.406495)

2012-05-23 12:58:49.126977 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:58:49.126974) v2 because of no pipe on con 0x6783680

2012-05-23 12:58:49.127047 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:29.127047)

2012-05-23 12:58:49.406637 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:29.406637)

2012-05-23 12:58:50.406783 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:30.406782)

2012-05-23 12:58:51.406931 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:31.406930)

2012-05-23 12:58:52.027295 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:58:52.027291) v2 because of no pipe on con 0x6783680

2012-05-23 12:58:52.027359 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:32.027358)

2012-05-23 12:58:52.407092 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:32.407092)

2012-05-23 12:58:53.407216 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:33.407215)

2012-05-23 12:58:54.407368 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:34.407367)

2012-05-23 12:58:55.407549 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:35.407548)

2012-05-23 12:58:55.527607 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:58:55.527604) v2 because of no pipe on con 0x6783680

2012-05-23 12:58:55.527679 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:35.527678)

2012-05-23 12:58:56.407679 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:36.407678)

2012-05-23 12:58:57.407823 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:37.407822)

2012-05-23 12:58:58.408000 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:38.407999)

2012-05-23 12:58:59.408174 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:39.408173)

2012-05-23 12:59:00.408327 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:40.408326)

2012-05-23 12:59:00.827922 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:59:00.827919) v2 because of no pipe on con 0x6783680

2012-05-23 12:59:00.827983 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:40.827983)

2012-05-23 12:59:01.408489 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:41.408488)

2012-05-23 12:59:02.408654 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:42.408654)

2012-05-23 12:59:03.128202 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:59:03.128199) v2 because of no pipe on con 0x6783680

2012-05-23 12:59:03.128264 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:43.128264)

2012-05-23 12:59:03.408831 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:43.408830)

2012-05-23 12:59:04.409017 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:44.409016)

2012-05-23 12:59:05.409160 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:45.409160)

2012-05-23 12:59:06.412070 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:46.412069)

2012-05-23 12:59:07.228510 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:59:07.228507) v2 because of no pipe on con 0x6783680

2012-05-23 12:59:07.228575 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:47.228575)

2012-05-23 12:59:07.412225 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:47.412224)

2012-05-23 12:59:08.328829 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:59:08.328826) v2 because of no pipe on con 0x6783680

2012-05-23 12:59:08.328892 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:48.328891)

2012-05-23 12:59:08.412369 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:48.412368)

2012-05-23 12:59:09.412485 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:49.412484)

2012-05-23 12:59:10.412641 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:50.412640)

2012-05-23 12:59:10.629351 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:59:10.629346) v2 because of no pipe on con 0x6783680

2012-05-23 12:59:10.629442 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:50.629441)

2012-05-23 12:59:11.412822 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:51.412821)

2012-05-23 12:59:12.413016 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:52.413016)

2012-05-23 12:59:13.413188 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:53.413187)

2012-05-23 12:59:14.413335 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:54.413335)

2012-05-23 12:59:15.329698 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:59:15.329695) v2 because of no pipe on con 0x6783680

2012-05-23 12:59:15.329760 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:55.329759)

2012-05-23 12:59:15.413483 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:55.413482)

2012-05-23 12:59:16.413644 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:56.413643)

2012-05-23 12:59:17.029993 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:59:17.029990) v2 because of no pipe on con 0x6783680

2012-05-23 12:59:17.030071 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:57.030070)

2012-05-23 12:59:17.413807 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:57.413806)

2012-05-23 12:59:18.130331 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:59:18.130328) v2 because of no pipe on con 0x6783680

2012-05-23 12:59:18.130395 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:58.130395)

2012-05-23 12:59:18.413949 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:58.413948)

2012-05-23 12:59:19.230618 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:59:19.230615) v2 because of no pipe on con 0x6783680

2012-05-23 12:59:19.230691 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:59.230691)

2012-05-23 12:59:19.414106 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:58:59.414105)

2012-05-23 12:59:20.414256 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:00.414255)

2012-05-23 12:59:20.930922 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:59:20.930919) v2 because of no pipe on con 0x6783680

2012-05-23 12:59:20.930987 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:00.930986)

2012-05-23 12:59:21.414389 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:01.414388)

2012-05-23 12:59:22.414533 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:02.414532)

2012-05-23 12:59:23.414681 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:03.414680)

2012-05-23 12:59:24.414846 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:04.414845)

2012-05-23 12:59:25.414972 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:05.414971)

2012-05-23 12:59:26.231238 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:59:26.231234) v2 because of no pipe on con 0x6783680

2012-05-23 12:59:26.231298 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:06.231298)

2012-05-23 12:59:26.415145 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:06.415145)

2012-05-23 12:59:27.415271 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:07.415271)

2012-05-23 12:59:28.415402 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:08.415402)

2012-05-23 12:59:29.415570 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:09.415569)

2012-05-23 12:59:30.331537 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:59:30.331534) v2 because of no pipe on con 0x6783680

2012-05-23 12:59:30.331598 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:10.331597)

2012-05-23 12:59:30.415741 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:10.415740)

2012-05-23 12:59:31.415887 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:11.415885)

2012-05-23 12:59:32.416094 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:12.416093)

2012-05-23 12:59:33.416276 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:13.416275)

2012-05-23 12:59:34.416425 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:14.416424)

2012-05-23 12:59:35.416569 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:15.416569)

2012-05-23 12:59:36.231830 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:59:36.231826) v2 because of no pipe on con 0x6783680

2012-05-23 12:59:36.231893 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:16.231893)

2012-05-23 12:59:36.416714 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:16.416713)

2012-05-23 12:59:36.732235 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:59:36.732230) v2 because of no pipe on con 0x6783680

2012-05-23 12:59:36.732319 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:16.732318)

2012-05-23 12:59:37.416946 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:17.416945)

2012-05-23 12:59:38.417092 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:18.417091)

2012-05-23 12:59:39.417229 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:19.417228)

2012-05-23 12:59:40.417379 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:20.417379)

2012-05-23 12:59:41.417548 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:21.417547)

2012-05-23 12:59:42.417700 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:22.417700)

2012-05-23 12:59:42.632569 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:59:42.632566) v2 because of no pipe on con 0x6783680

2012-05-23 12:59:42.632630 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:22.632629)

2012-05-23 12:59:43.417836 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:23.417836)

2012-05-23 12:59:44.417983 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:24.417982)

2012-05-23 12:59:45.418145 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:25.418145)

2012-05-23 12:59:46.418309 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:26.418309)

2012-05-23 12:59:47.418441 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:27.418441)

2012-05-23 12:59:47.932879 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:59:47.932876) v2 because of no pipe on con 0x6783680

2012-05-23 12:59:47.932938 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:27.932938)

2012-05-23 12:59:48.418584 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:28.418584)

2012-05-23 12:59:49.418712 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:29.418712)

2012-05-23 12:59:49.633191 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:59:49.633187) v2 because of no pipe on con 0x6783680

2012-05-23 12:59:49.633255 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:29.633255)

2012-05-23 12:59:50.418871 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:30.418870)

2012-05-23 12:59:51.419012 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:31.419011)

2012-05-23 12:59:52.419169 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:32.419168)

2012-05-23 12:59:53.419321 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:33.419320)

2012-05-23 12:59:54.333640 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:59:54.333633) v2 because of no pipe on con 0x6783680

2012-05-23 12:59:54.333758 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:34.333758)

2012-05-23 12:59:54.419500 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:34.419499)

2012-05-23 12:59:55.419645 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:35.419644)

2012-05-23 12:59:56.419792 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:36.419792)

2012-05-23 12:59:57.234100 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 12:59:57.234095) v2 because of no pipe on con 0x6783680

2012-05-23 12:59:57.234184 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:37.234183)

2012-05-23 12:59:57.419957 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:37.419956)

2012-05-23 12:59:58.420100 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:38.420100)

2012-05-23 12:59:59.420244 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:39.420243)

2012-05-23 13:00:00.420378 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:40.420378)

2012-05-23 13:00:01.334425 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 13:00:01.334423) v2 because of no pipe on con 0x6783680

2012-05-23 13:00:01.334489 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:41.334489)

2012-05-23 13:00:01.420530 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:41.420529)

2012-05-23 13:00:02.420657 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:42.420656)

2012-05-23 13:00:03.420815 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:43.420814)

2012-05-23 13:00:03.634730 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 13:00:03.634727) v2 because of no pipe on con 0x6783680

2012-05-23 13:00:03.634788 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:43.634787)

2012-05-23 13:00:04.420938 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:44.420937)

2012-05-23 13:00:05.421094 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:45.421094)

2012-05-23 13:00:06.421220 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:46.421220)

2012-05-23 13:00:07.421365 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:47.421364)

2012-05-23 13:00:08.335031 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 13:00:08.335028) v2 because of no pipe on con 0x6783680

2012-05-23 13:00:08.335096 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:48.335095)

2012-05-23 13:00:08.421505 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:48.421504)

2012-05-23 13:00:09.421650 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:49.421649)

2012-05-23 13:00:09.435341 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 13:00:09.435338) v2 because of no pipe on con 0x6783680

2012-05-23 13:00:09.435404 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:49.435404)

2012-05-23 13:00:10.421795 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:50.421794)

2012-05-23 13:00:11.421954 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:51.421953)

2012-05-23 13:00:12.422109 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:52.422108)

2012-05-23 13:00:13.422232 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:53.422231)

2012-05-23 13:00:14.422371 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:54.422370)

2012-05-23 13:00:15.335649 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 13:00:15.335646) v2 because of no pipe on con 0x6783680

2012-05-23 13:00:15.335716 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:55.335715)

2012-05-23 13:00:15.422511 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:55.422510)

2012-05-23 13:00:16.422674 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:56.422673)

2012-05-23 13:00:17.422809 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:57.422808)

2012-05-23 13:00:18.422980 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:58.422980)

2012-05-23 13:00:19.423124 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 12:59:59.423123)

2012-05-23 13:00:20.035980 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 13:00:20.035977) v2 because of no pipe on con 0x6783680

2012-05-23 13:00:20.036056 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 13:00:00.036055)

2012-05-23 13:00:20.423294 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 13:00:00.423293)

2012-05-23 13:00:21.136307 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 13:00:21.136304) v2 because of no pipe on con 0x6783680

2012-05-23 13:00:21.136376 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 13:00:01.136376)

2012-05-23 13:00:21.423434 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 13:00:01.423433)

2012-05-23 13:00:22.423584 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 13:00:02.423583)

2012-05-23 13:00:23.423736 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 13:00:03.423736)

2012-05-23 13:00:24.423889 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 13:00:04.423888)

2012-05-23 13:00:24.636620 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 13:00:24.636617) v2 because of no pipe on con 0x6783680

2012-05-23 13:00:24.636683 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 13:00:04.636682)

2012-05-23 13:00:25.424068 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 13:00:05.424067)

2012-05-23 13:00:26.424201 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 13:00:06.424200)

2012-05-23 13:00:27.424343 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 13:00:07.424342)

2012-05-23 13:00:28.136928 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 13:00:28.136925) v2 because of no pipe on con 0x6783680

2012-05-23 13:00:28.136993 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 13:00:08.136992)

2012-05-23 13:00:28.424466 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 13:00:08.424465)

2012-05-23 13:00:29.237239 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 13:00:29.237236) v2 because of no pipe on con 0x6783680

2012-05-23 13:00:29.237310 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 13:00:09.237309)

2012-05-23 13:00:29.424605 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 13:00:09.424604)

2012-05-23 13:00:29.737553 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 13:00:29.737550) v2 because of no pipe on con 0x6783680

2012-05-23 13:00:29.737625 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 13:00:09.737624)

2012-05-23 13:00:30.424740 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 13:00:10.424740)

2012-05-23 13:00:31.424869 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 13:00:11.424868)

2012-05-23 13:00:32.037870 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 13:00:32.037867) v2 because of no pipe on con 0x6783680

2012-05-23 13:00:32.037927 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 13:00:12.037926)

2012-05-23 13:00:32.425038 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 13:00:12.425038)

2012-05-23 13:00:33.425185 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 13:00:13.425185)

2012-05-23 13:00:34.425305 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 13:00:14.425304)

2012-05-23 13:00:35.425480 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 13:00:15.425479)

2012-05-23 13:00:36.425641 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 13:00:16.425641)

2012-05-23 13:00:36.738167 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 13:00:36.738164) v2 because of no pipe on con 0x6783680

2012-05-23 13:00:36.738227 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 13:00:16.738226)

2012-05-23 13:00:37.425779 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 13:00:17.425778)

2012-05-23 13:00:38.425953 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 13:00:18.425951)

2012-05-23 13:00:39.426118 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 13:00:19.426117)

2012-05-23 13:00:40.426280 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 13:00:20.426280)

2012-05-23 13:00:40.838551 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 13:00:40.838549) v2 because of no pipe on con 0x6783680

2012-05-23 13:00:40.838616 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 13:00:20.838616)

2012-05-23 13:00:41.426415 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 13:00:21.426414)

2012-05-23 13:00:42.426542 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 13:00:22.426542)

2012-05-23 13:00:43.426680 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 13:00:23.426679)

2012-05-23 13:00:44.339132 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 13:00:44.339129) v2 because of no pipe on con 0x6783680

2012-05-23 13:00:44.339267 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 13:00:24.339266)

2012-05-23 13:00:44.426842 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 13:00:24.426842)

2012-05-23 13:00:45.426980 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 13:00:25.426980)

2012-05-23 13:00:46.427132 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 13:00:26.427131)

2012-05-23 13:00:47.427268 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 13:00:27.427267)

2012-05-23 13:00:48.427417 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 13:00:28.427416)

2012-05-23 13:00:48.439645 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 13:00:48.439641) v2 because of no pipe on con 0x6783680

2012-05-23 13:00:48.439811 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 13:00:28.439810)

2012-05-23 13:00:49.427544 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 13:00:29.427544)

2012-05-23 13:00:50.427689 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 13:00:30.427688)

2012-05-23 13:00:51.427821 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 13:00:31.427820)

2012-05-23 13:00:52.427992 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 13:00:32.427992)

2012-05-23 13:00:53.140099 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1092 stamp 2012-05-23 13:00:53.140096) v2 because of no pipe on con 0x6783680

2012-05-23 13:00:53.140164 7fd8b443c700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 13:00:33.140163)

2012-05-23 13:00:53.428181 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 13:00:33.428180)

2012-05-23 13:00:54.428350 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 13:00:34.428349)

2012-05-23 13:00:55.428506 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 13:00:35.428505)

2012-05-23 13:00:56.428665 7fd8c1557700 -1 osd.0 1092 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 13:00:36.428665)

2012-05-23 13:00:57.434364 7fd8c1557700 -1 osd.0 1093 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 13:00:37.434363)

2012-05-23 13:00:58.434723 7fd8c1557700 -1 osd.0 1094 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 13:00:38.434722)

2012-05-23 13:00:58.440401 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1094 stamp 2012-05-23 13:00:58.440398) v2 because of no pipe on con 0x6783680

2012-05-23 13:00:58.440475 7fd8b443c700 -1 osd.0 1094 heartbeat_check: no reply from osd.1 since 2012-05-23 12:53:34.690240 (cutoff 2012-05-23 13:00:38.440474)

2012-05-23 13:12:41.871688 7fd8c1557700  0 log [WRN] : 3 slow requests, 3 included below; oldest blocked for > 701.396163 secs

2012-05-23 13:12:41.871699 7fd8c1557700  0 log [WRN] : slow request 701.396163 seconds old, received at 2012-05-23 13:01:00.475475: osd_sub_op(osd.11.0:109553 0.15a 9f26015a/1000000306b.00000011/head [push] v 49'4038 snapset=0=[]:[] snapc=0=[]) v6 currently started

2012-05-23 13:12:41.871703 7fd8c1557700  0 log [WRN] : slow request 701.349010 seconds old, received at 2012-05-23 13:01:00.522628: osd_sub_op(osd.10.0:70617 0.8a 3be0b08a/100000025b8.0000011c/head [push] v 49'3074 snapset=0=[]:[] snapc=0=[]) v6 currently started

2012-05-23 13:12:41.871704 7fd8c1557700  0 log [WRN] : slow request 701.166964 seconds old, received at 2012-05-23 13:01:00.704674: osd_sub_op(osd.9.0:81338 0.1 16aa6001/10000000058.0000014e/head [push] v 11'18 snapset=0=[]:[] snapc=0=[]) v6 currently started

2012-05-23 13:13:06.125097 7fd8b3433700  0 -- 192.168.1.101:6804/30866 >> 192.168.1.102:6802/16500 pipe(0x5a88500 sd=38 pgs=244 cs=1 l=0).fault with nothing to send, going to standby

2012-05-23 13:13:06.904935 7fd8b1d1c700  0 -- 192.168.1.101:6804/30866 >> 192.168.1.102:6806/16586 pipe(0x5b90000 sd=46 pgs=178 cs=1 l=0).fault with nothing to send, going to standby

2012-05-23 13:13:08.513289 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1157 stamp 2012-05-23 13:13:08.513275) v2 because of no pipe on con 0x5c4e500

2012-05-23 13:13:08.513370 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1157 stamp 2012-05-23 13:13:08.513275) v2 because of no pipe on con 0x5c4ec80

2012-05-23 13:13:08.978182 7fd8b2120700  0 -- 192.168.1.101:6804/30866 >> 192.168.1.102:6809/16671 pipe(0x5b90c80 sd=47 pgs=225 cs=1 l=0).fault with nothing to send, going to standby

2012-05-23 13:13:09.121975 7fd8b9d48700  0 monclient: hunting for new mon

2012-05-23 13:13:09.295377 7fd8b2e2d700  0 -- 192.168.1.101:6804/30866 >> 192.168.1.103:6805/10049 pipe(0x5a89180 sd=42 pgs=42 cs=1 l=0).fault initiating reconnect

2012-05-23 13:13:09.295495 7fd8b2a29700  0 -- 192.168.1.101:6804/30866 >> 192.168.1.103:6805/10049 pipe(0x5a89180 sd=42 pgs=42 cs=2 l=0).fault first fault

2012-05-23 13:13:10.124101 7fd8b1716700  0 -- 192.168.1.101:6804/30866 >> 192.168.1.102:6812/16757 pipe(0x5b90a00 sd=48 pgs=209 cs=1 l=0).fault with nothing to send, going to standby

2012-05-23 13:13:10.532595 7fd8b302f700  0 -- 192.168.1.101:6804/30866 >> 192.168.1.103:6809/10134 pipe(0x5a89400 sd=66 pgs=74 cs=3 l=0).fault initiating reconnect

2012-05-23 13:13:11.413664 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1157 stamp 2012-05-23 13:13:11.413653) v2 because of no pipe on con 0x5c4e500

2012-05-23 13:13:11.413700 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1157 stamp 2012-05-23 13:13:11.413653) v2 because of no pipe on con 0x5c4ec80

2012-05-23 13:13:11.413709 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1157 stamp 2012-05-23 13:13:11.413653) v2 because of no pipe on con 0x5c4e780

2012-05-23 13:13:11.413715 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1157 stamp 2012-05-23 13:13:11.413653) v2 because of no pipe on con 0x5c4edc0

2012-05-23 13:13:11.413729 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1157 stamp 2012-05-23 13:13:11.413653) v2 because of no pipe on con 0x5c4e640

2012-05-23 13:13:11.413751 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1157 stamp 2012-05-23 13:13:11.413653) v2 because of no pipe on con 0x5c4e3c0

2012-05-23 13:13:11.456991 7fd8b3c3b700  0 -- 192.168.1.101:6804/30866 >> 192.168.1.103:6814/10291 pipe(0x5a88000 sd=34 pgs=50 cs=1 l=0).fault with nothing to send, going to standby

2012-05-23 13:13:12.580978 7fd8b3837700  0 -- 192.168.1.101:6804/30866 >> 192.168.1.103:6817/10376 pipe(0x5a88780 sd=36 pgs=72 cs=1 l=0).fault with nothing to send, going to standby

2012-05-23 13:13:14.913988 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1157 stamp 2012-05-23 13:13:14.913979) v2 because of no pipe on con 0x5c4e500

2012-05-23 13:13:14.914019 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1157 stamp 2012-05-23 13:13:14.913979) v2 because of no pipe on con 0x5c4ec80

2012-05-23 13:13:14.914024 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1157 stamp 2012-05-23 13:13:14.913979) v2 because of no pipe on con 0x5c4e780

2012-05-23 13:13:14.914028 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1157 stamp 2012-05-23 13:13:14.913979) v2 because of no pipe on con 0x5c4edc0

2012-05-23 13:13:14.914031 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1157 stamp 2012-05-23 13:13:14.913979) v2 because of no pipe on con 0x5c4e000

2012-05-23 13:13:14.914034 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1157 stamp 2012-05-23 13:13:14.913979) v2 because of no pipe on con 0x5c4eb40

2012-05-23 13:13:14.914037 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1157 stamp 2012-05-23 13:13:14.913979) v2 because of no pipe on con 0x5c4e640

2012-05-23 13:13:14.914040 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1157 stamp 2012-05-23 13:13:14.913979) v2 because of no pipe on con 0x5c4e3c0

2012-05-23 13:13:19.614281 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1157 stamp 2012-05-23 13:13:19.614271) v2 because of no pipe on con 0x5c4e500

2012-05-23 13:13:19.614311 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1157 stamp 2012-05-23 13:13:19.614271) v2 because of no pipe on con 0x5c4ec80

2012-05-23 13:13:19.614315 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1157 stamp 2012-05-23 13:13:19.614271) v2 because of no pipe on con 0x5c4e780

2012-05-23 13:13:19.614319 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1157 stamp 2012-05-23 13:13:19.614271) v2 because of no pipe on con 0x5c4edc0

2012-05-23 13:13:19.614322 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1157 stamp 2012-05-23 13:13:19.614271) v2 because of no pipe on con 0x5c4e000

2012-05-23 13:13:19.614330 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1157 stamp 2012-05-23 13:13:19.614271) v2 because of no pipe on con 0x5c4eb40

2012-05-23 13:13:19.614336 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1157 stamp 2012-05-23 13:13:19.614271) v2 because of no pipe on con 0x5c4e640

2012-05-23 13:13:19.614339 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1157 stamp 2012-05-23 13:13:19.614271) v2 because of no pipe on con 0x5c4e3c0

2012-05-23 13:13:21.914566 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1157 stamp 2012-05-23 13:13:21.914556) v2 because of no pipe on con 0x5c4e500

2012-05-23 13:13:21.914599 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1157 stamp 2012-05-23 13:13:21.914556) v2 because of no pipe on con 0x5c4ec80

2012-05-23 13:13:21.914606 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1157 stamp 2012-05-23 13:13:21.914556) v2 because of no pipe on con 0x5c4e780

2012-05-23 13:13:21.914611 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1157 stamp 2012-05-23 13:13:21.914556) v2 because of no pipe on con 0x5c4edc0

2012-05-23 13:13:21.914616 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1157 stamp 2012-05-23 13:13:21.914556) v2 because of no pipe on con 0x5c4e000

2012-05-23 13:13:21.914620 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1157 stamp 2012-05-23 13:13:21.914556) v2 because of no pipe on con 0x5c4eb40

2012-05-23 13:13:21.914624 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1157 stamp 2012-05-23 13:13:21.914556) v2 because of no pipe on con 0x5c4e640

2012-05-23 13:13:21.914628 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1157 stamp 2012-05-23 13:13:21.914556) v2 because of no pipe on con 0x5c4e3c0

2012-05-23 13:13:25.889689 7fd8c1557700 -1 osd.0 1157 heartbeat_check: no reply from osd.4 since 2012-05-23 13:13:05.011607 (cutoff 2012-05-23 13:13:05.889688)

2012-05-23 13:13:25.889712 7fd8c1557700 -1 osd.0 1157 heartbeat_check: no reply from osd.5 since 2012-05-23 13:13:05.011607 (cutoff 2012-05-23 13:13:05.889688)

2012-05-23 13:13:26.090486 7fd8b3736700  0 -- 192.168.1.101:6804/30866 >> 192.168.1.101:6802/30955 pipe(0x781d900 sd=37 pgs=0 cs=0 l=0).accept we reset (peer sent cseq 2), sending RESETSESSION

2012-05-23 13:13:27.814930 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e0 stamp 2012-05-23 13:13:27.814910) v2 because of no pipe on con 0x5c4e500

2012-05-23 13:13:27.814970 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e0 stamp 2012-05-23 13:13:27.814910) v2 because of no pipe on con 0x5c4ec80

2012-05-23 13:13:27.814978 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e0 stamp 2012-05-23 13:13:27.814910) v2 because of no pipe on con 0x5c4e780

2012-05-23 13:13:27.814984 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e0 stamp 2012-05-23 13:13:27.814910) v2 because of no pipe on con 0x5c4edc0

2012-05-23 13:13:27.814989 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e0 stamp 2012-05-23 13:13:27.814910) v2 because of no pipe on con 0x5c4e000

2012-05-23 13:13:27.814995 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e0 stamp 2012-05-23 13:13:27.814910) v2 because of no pipe on con 0x5c4eb40

2012-05-23 13:13:27.815000 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e0 stamp 2012-05-23 13:13:27.814910) v2 because of no pipe on con 0x5c4e3c0

2012-05-23 13:13:31.315248 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e0 stamp 2012-05-23 13:13:31.315240) v2 because of no pipe on con 0x5c4e500

2012-05-23 13:13:31.315271 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e0 stamp 2012-05-23 13:13:31.315240) v2 because of no pipe on con 0x5c4ec80

2012-05-23 13:13:31.315277 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e0 stamp 2012-05-23 13:13:31.315240) v2 because of no pipe on con 0x5c4e780

2012-05-23 13:13:31.315282 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e0 stamp 2012-05-23 13:13:31.315240) v2 because of no pipe on con 0x5c4edc0

2012-05-23 13:13:31.315287 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e0 stamp 2012-05-23 13:13:31.315240) v2 because of no pipe on con 0x5c4e000

2012-05-23 13:13:31.315291 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e0 stamp 2012-05-23 13:13:31.315240) v2 because of no pipe on con 0x5c4eb40

2012-05-23 13:13:31.315296 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e0 stamp 2012-05-23 13:13:31.315240) v2 because of no pipe on con 0x5c4e3c0

2012-05-23 13:13:33.015517 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e0 stamp 2012-05-23 13:13:33.015509) v2 because of no pipe on con 0x5c4e500

2012-05-23 13:13:33.015549 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e0 stamp 2012-05-23 13:13:33.015509) v2 because of no pipe on con 0x5c4ec80

2012-05-23 13:13:33.015558 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e0 stamp 2012-05-23 13:13:33.015509) v2 because of no pipe on con 0x5c4e780

2012-05-23 13:13:33.015565 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e0 stamp 2012-05-23 13:13:33.015509) v2 because of no pipe on con 0x5c4edc0

2012-05-23 13:13:33.015571 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e0 stamp 2012-05-23 13:13:33.015509) v2 because of no pipe on con 0x5c4e000

2012-05-23 13:13:33.015575 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e0 stamp 2012-05-23 13:13:33.015509) v2 because of no pipe on con 0x5c4eb40

2012-05-23 13:13:33.015578 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e0 stamp 2012-05-23 13:13:33.015509) v2 because of no pipe on con 0x5c4e3c0

2012-05-23 13:13:35.315822 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e0 stamp 2012-05-23 13:13:35.315813) v2 because of no pipe on con 0x5c4e500

2012-05-23 13:13:35.315848 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e0 stamp 2012-05-23 13:13:35.315813) v2 because of no pipe on con 0x5c4ec80

2012-05-23 13:13:35.315854 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e0 stamp 2012-05-23 13:13:35.315813) v2 because of no pipe on con 0x5c4e780

2012-05-23 13:13:35.315861 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e0 stamp 2012-05-23 13:13:35.315813) v2 because of no pipe on con 0x5c4edc0

2012-05-23 13:13:35.315866 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e0 stamp 2012-05-23 13:13:35.315813) v2 because of no pipe on con 0x5c4e000

2012-05-23 13:13:35.315872 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e0 stamp 2012-05-23 13:13:35.315813) v2 because of no pipe on con 0x5c4eb40

2012-05-23 13:13:35.315877 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e0 stamp 2012-05-23 13:13:35.315813) v2 because of no pipe on con 0x5c4e3c0

2012-05-23 13:13:36.416128 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e0 stamp 2012-05-23 13:13:36.416120) v2 because of no pipe on con 0x5c4e500

2012-05-23 13:13:36.416153 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e0 stamp 2012-05-23 13:13:36.416120) v2 because of no pipe on con 0x5c4ec80

2012-05-23 13:13:36.416161 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e0 stamp 2012-05-23 13:13:36.416120) v2 because of no pipe on con 0x5c4e780

2012-05-23 13:13:36.416167 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e0 stamp 2012-05-23 13:13:36.416120) v2 because of no pipe on con 0x5c4edc0

2012-05-23 13:13:36.416173 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e0 stamp 2012-05-23 13:13:36.416120) v2 because of no pipe on con 0x5c4e000

2012-05-23 13:13:36.416179 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e0 stamp 2012-05-23 13:13:36.416120) v2 because of no pipe on con 0x5c4eb40

2012-05-23 13:13:36.416183 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e0 stamp 2012-05-23 13:13:36.416120) v2 because of no pipe on con 0x5c4e3c0

2012-05-23 13:13:38.716422 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e0 stamp 2012-05-23 13:13:38.716413) v2 because of no pipe on con 0x5c4e500

2012-05-23 13:13:38.716457 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e0 stamp 2012-05-23 13:13:38.716413) v2 because of no pipe on con 0x5c4ec80

2012-05-23 13:13:38.716469 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e0 stamp 2012-05-23 13:13:38.716413) v2 because of no pipe on con 0x5c4e780

2012-05-23 13:13:38.716476 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e0 stamp 2012-05-23 13:13:38.716413) v2 because of no pipe on con 0x5c4edc0

2012-05-23 13:13:38.716483 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e0 stamp 2012-05-23 13:13:38.716413) v2 because of no pipe on con 0x5c4e000

2012-05-23 13:13:38.716487 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e0 stamp 2012-05-23 13:13:38.716413) v2 because of no pipe on con 0x5c4eb40

2012-05-23 13:13:38.716490 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e0 stamp 2012-05-23 13:13:38.716413) v2 because of no pipe on con 0x5c4e3c0

2012-05-23 13:13:39.816738 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e0 stamp 2012-05-23 13:13:39.816729) v2 because of no pipe on con 0x5c4e500

2012-05-23 13:13:39.816772 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e0 stamp 2012-05-23 13:13:39.816729) v2 because of no pipe on con 0x5c4ec80

2012-05-23 13:13:39.816827 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e0 stamp 2012-05-23 13:13:39.816729) v2 because of no pipe on con 0x5c4e780

2012-05-23 13:13:39.816836 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e0 stamp 2012-05-23 13:13:39.816729) v2 because of no pipe on con 0x5c4edc0

2012-05-23 13:13:39.816844 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e0 stamp 2012-05-23 13:13:39.816729) v2 because of no pipe on con 0x5c4e000

2012-05-23 13:13:39.816850 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e0 stamp 2012-05-23 13:13:39.816729) v2 because of no pipe on con 0x5c4eb40

2012-05-23 13:13:39.816853 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e0 stamp 2012-05-23 13:13:39.816729) v2 because of no pipe on con 0x5c4e3c0

2012-05-23 13:13:45.117094 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1159 stamp 2012-05-23 13:13:45.117085) v2 because of no pipe on con 0x5c4e500

2012-05-23 13:13:45.117121 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1159 stamp 2012-05-23 13:13:45.117085) v2 because of no pipe on con 0x5c4e780

2012-05-23 13:13:45.117126 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1159 stamp 2012-05-23 13:13:45.117085) v2 because of no pipe on con 0x5c4e000

2012-05-23 13:13:45.117131 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1159 stamp 2012-05-23 13:13:45.117085) v2 because of no pipe on con 0x5c4eb40

2012-05-23 13:13:45.117140 7fd8b443c700  0 -- 192.168.1.101:0/30866 send_message dropped message osd_ping(ping e1159 stamp 2012-05-23 13:13:45.117085) v2 because of no pipe on con 0x5c4e3c0

2012-05-23 13:13:45.117145 7fd8b443c700 -1 osd.0 1159 heartbeat_check: no reply from osd.4 since 2012-05-23 13:13:05.011607 (cutoff 2012-05-23 13:13:25.117144)

2012-05-23 13:13:45.117158 7fd8b443c700 -1 osd.0 1159 heartbeat_check: no reply from osd.6 since 2012-05-23 13:13:08.513275 (cutoff 2012-05-23 13:13:25.117144)

2012-05-23 13:13:45.117166 7fd8b443c700 -1 osd.0 1159 heartbeat_check: no reply from osd.8 since 2012-05-23 13:13:11.413653 (cutoff 2012-05-23 13:13:25.117144)

2012-05-23 13:13:45.117171 7fd8b443c700 -1 osd.0 1159 heartbeat_check: no reply from osd.9 since 2012-05-23 13:13:11.413653 (cutoff 2012-05-23 13:13:25.117144)

2012-05-23 13:13:45.117177 7fd8b443c700 -1 osd.0 1159 heartbeat_check: no reply from osd.11 since 2012-05-23 13:13:08.513275 (cutoff 2012-05-23 13:13:25.117144)

2012-05-23 13:13:49.735535 7fd8c1557700 -1 osd.0 1159 heartbeat_check: no reply from osd.4 since 2012-05-23 13:13:05.011607 (cutoff 2012-05-23 13:13:29.735534)

2012-05-23 13:13:49.735564 7fd8c1557700 -1 osd.0 1159 heartbeat_check: no reply from osd.6 since 2012-05-23 13:13:08.513275 (cutoff 2012-05-23 13:13:29.735534)

2012-05-23 13:13:49.735572 7fd8c1557700 -1 osd.0 1159 heartbeat_check: no reply from osd.8 since 2012-05-23 13:13:11.413653 (cutoff 2012-05-23 13:13:29.735534)

2012-05-23 13:13:49.735577 7fd8c1557700 -1 osd.0 1159 heartbeat_check: no reply from osd.9 since 2012-05-23 13:13:11.413653 (cutoff 2012-05-23 13:13:29.735534)

2012-05-23 13:13:49.735591 7fd8c1557700 -1 osd.0 1159 heartbeat_check: no reply from osd.11 since 2012-05-23 13:13:08.513275 (cutoff 2012-05-23 13:13:29.735534)

2012-05-23 13:13:53.689234 7fd8b9547700  0 osd.0 1174 from dead osd.1, dropping, sharing map

2012-05-23 13:13:55.375872 7fd8c1d58700 -1 osd.0 1178 *** Got signal Terminated ***

2012-05-23 13:13:55.375903 7fd8c1d58700 -1 osd.0 1178  pausing thread pools

2012-05-23 13:13:55.375907 7fd8c1d58700 -1 osd.0 1178  flushing io

2012-05-23 13:14:03.824082 7fbf2bd08780  0 ceph version 0.47.1 (commit:f5a9404445e2ed5ec2ee828aa53d73d4a002f7a5), process ceph-osd, pid 27494

2012-05-23 13:14:03.918586 7fbf2bd08780  0 filestore(/data/osd.0) mount FIEMAP ioctl is supported and appears to work

2012-05-23 13:14:03.918873 7fbf2bd08780  0 filestore(/data/osd.0) mount detected btrfs

2012-05-23 13:14:03.918891 7fbf2bd08780  0 filestore(/data/osd.0) mount btrfs CLONE_RANGE ioctl is supported

2012-05-23 13:14:04.747634 7fbf2bd08780  0 filestore(/data/osd.0) mount btrfs SNAP_CREATE is supported

2012-05-23 13:14:04.754368 7fbf2bd08780  0 filestore(/data/osd.0) mount btrfs SNAP_DESTROY is supported

2012-05-23 13:14:04.894286 7fbf2bd08780  0 filestore(/data/osd.0) mount btrfs START_SYNC is supported (transid 47500)

2012-05-23 13:14:05.092133 7fbf2bd08780  0 filestore(/data/osd.0) mount btrfs WAIT_SYNC is supported

2012-05-23 13:14:05.324604 7fbf2bd08780  0 filestore(/data/osd.0) mount btrfs SNAP_CREATE_V2 is supported

2012-05-23 13:14:05.329148 7fbf2bd08780  0 filestore(/data/osd.0) mount found snaps <884133,884152>

2012-05-23 13:14:06.754919 7fbf2bd08780  0 filestore(/data/osd.0) mount: enabling PARALLEL journal mode: btrfs, SNAP_CREATE_V2 detected and 'filestore btrfs snap' mode is enabled

2012-05-23 13:14:07.240913 7fbf2bd08780  0 journal  kernel version is 3.4.0

2012-05-23 13:14:07.240977 7fbf2bd08780  1 journal _open /dev/sda5 fd 22: 1019899904 bytes, block size 4096 bytes, directio = 1, aio = 0

2012-05-23 13:14:15.441703 7fe613e02780  0 ceph version 0.47.1 (commit:f5a9404445e2ed5ec2ee828aa53d73d4a002f7a5), process ceph-osd, pid 28673

2012-05-23 13:14:15.733081 7fe613e02780  0 filestore(/data/osd.0) mount FIEMAP ioctl is supported and appears to work

2012-05-23 13:14:15.733524 7fe613e02780  0 filestore(/data/osd.0) mount detected btrfs

2012-05-23 13:14:15.733549 7fe613e02780  0 filestore(/data/osd.0) mount btrfs CLONE_RANGE ioctl is supported

2012-05-23 13:14:16.741640 7fe613e02780  0 filestore(/data/osd.0) mount btrfs SNAP_CREATE is supported

2012-05-23 13:14:16.742251 7fe613e02780  0 filestore(/data/osd.0) mount btrfs SNAP_DESTROY is supported

2012-05-23 13:14:16.864784 7fe613e02780  0 filestore(/data/osd.0) mount btrfs START_SYNC is supported (transid 47504)

2012-05-23 13:14:17.070262 7fe613e02780  0 filestore(/data/osd.0) mount btrfs WAIT_SYNC is supported

2012-05-23 13:14:17.350699 7fe613e02780  0 filestore(/data/osd.0) mount btrfs SNAP_CREATE_V2 is supported

2012-05-23 13:14:17.354178 7fe613e02780  0 filestore(/data/osd.0) mount found snaps <884133,884152>

2012-05-23 13:14:18.480607 7fe613e02780  0 filestore(/data/osd.0) mount: enabling PARALLEL journal mode: btrfs, SNAP_CREATE_V2 detected and 'filestore btrfs snap' mode is enabled

2012-05-23 13:14:18.503060 7fe613e02780  0 journal  kernel version is 3.4.0

2012-05-23 13:14:18.503136 7fe613e02780  1 journal _open /dev/sda5 fd 25: 1019899904 bytes, block size 4096 bytes, directio = 1, aio = 0

2012-05-23 13:14:25.871173 7fe613e02780  0 journal  kernel version is 3.4.0

2012-05-23 13:14:25.871298 7fe613e02780  1 journal _open /dev/sda5 fd 25: 1019899904 bytes, block size 4096 bytes, directio = 1, aio = 0

2012-05-23 13:14:26.234186 7fe613e02780  1 journal close /dev/sda5

2012-05-23 13:14:26.405298 7fe613e02780  0 filestore(/data/osd.0) mount FIEMAP ioctl is supported and appears to work

2012-05-23 13:14:26.405731 7fe613e02780  0 filestore(/data/osd.0) mount detected btrfs

2012-05-23 13:14:26.405742 7fe613e02780  0 filestore(/data/osd.0) mount btrfs CLONE_RANGE ioctl is supported

2012-05-23 13:14:26.633024 7fe613e02780  0 filestore(/data/osd.0) mount btrfs SNAP_CREATE is supported

2012-05-23 13:14:26.634151 7fe613e02780  0 filestore(/data/osd.0) mount btrfs SNAP_DESTROY is supported

2012-05-23 13:14:26.734524 7fe613e02780  0 filestore(/data/osd.0) mount btrfs START_SYNC is supported (transid 47511)

2012-05-23 13:14:26.901185 7fe613e02780  0 filestore(/data/osd.0) mount btrfs WAIT_SYNC is supported

2012-05-23 13:14:27.162293 7fe613e02780  0 filestore(/data/osd.0) mount btrfs SNAP_CREATE_V2 is supported

2012-05-23 13:14:27.170097 7fe613e02780  0 filestore(/data/osd.0) mount found snaps <884152,884345>

2012-05-23 13:14:27.765579 7fe613e02780  0 filestore(/data/osd.0) mount: enabling PARALLEL journal mode: btrfs, SNAP_CREATE_V2 detected and 'filestore btrfs snap' mode is enabled

2012-05-23 13:14:27.778073 7fe613e02780  0 journal  kernel version is 3.4.0

2012-05-23 13:14:27.778173 7fe613e02780  1 journal _open /dev/sda5 fd 29: 1019899904 bytes, block size 4096 bytes, directio = 1, aio = 0

2012-05-23 13:14:28.538025 7fe613e02780  0 journal  kernel version is 3.4.0

2012-05-23 13:14:28.538112 7fe613e02780  1 journal _open /dev/sda5 fd 29: 1019899904 bytes, block size 4096 bytes, directio = 1, aio = 0

2012-05-23 13:28:35.456547 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

2012-05-23 13:28:35.456585 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

2012-05-23 13:28:40.456652 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

2012-05-23 13:28:40.456680 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

2012-05-23 13:28:45.456747 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

2012-05-23 13:28:45.456768 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

2012-05-23 13:28:50.456843 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

2012-05-23 13:28:50.456874 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

2012-05-23 13:28:55.456945 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

2012-05-23 13:28:55.456971 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

2012-05-23 13:29:00.457039 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

2012-05-23 13:29:00.457067 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

2012-05-23 13:29:05.458511 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

2012-05-23 13:29:05.458537 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

2012-05-23 13:29:10.458606 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

2012-05-23 13:29:10.458639 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

2012-05-23 13:29:15.458717 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

2012-05-23 13:29:15.458741 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

2012-05-23 13:29:20.458816 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

2012-05-23 13:29:20.458841 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

2012-05-23 13:29:25.458909 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

2012-05-23 13:29:25.458938 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

2012-05-23 13:29:30.459013 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

2012-05-23 13:29:30.459036 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

2012-05-23 13:29:35.459099 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

2012-05-23 13:29:35.459129 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

2012-05-23 13:29:40.459193 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

2012-05-23 13:29:40.459207 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

2012-05-23 13:29:45.459277 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

2012-05-23 13:29:45.459311 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

2012-05-23 13:29:50.459378 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

2012-05-23 13:29:50.459398 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

2012-05-23 13:29:55.459469 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

2012-05-23 13:29:55.459496 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

2012-05-23 13:30:00.459562 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

2012-05-23 13:30:00.459581 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

2012-05-23 13:30:05.459645 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

2012-05-23 13:30:05.459674 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

2012-05-23 13:30:10.459750 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

2012-05-23 13:30:10.459774 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

2012-05-23 13:30:15.459842 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

2012-05-23 13:30:15.459866 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

2012-05-23 13:30:20.459936 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

2012-05-23 13:30:20.459960 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

2012-05-23 13:30:25.460026 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

2012-05-23 13:30:25.460049 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

2012-05-23 13:30:30.460124 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

2012-05-23 13:30:30.460150 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

2012-05-23 13:30:35.460246 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

2012-05-23 13:30:35.460287 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

2012-05-23 13:30:40.460356 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

2012-05-23 13:30:40.460381 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

2012-05-23 13:30:45.460458 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

2012-05-23 13:30:45.460481 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

2012-05-23 13:30:50.460548 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

2012-05-23 13:30:50.460569 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

2012-05-23 13:30:55.460641 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

2012-05-23 13:30:55.460659 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

2012-05-23 13:31:00.460729 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

2012-05-23 13:31:00.460753 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

2012-05-23 13:31:05.460826 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

2012-05-23 13:31:05.460847 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

2012-05-23 13:31:10.460917 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

2012-05-23 13:31:10.460941 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

2012-05-23 13:31:15.461015 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

2012-05-23 13:31:15.461038 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

2012-05-23 13:31:20.461109 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

2012-05-23 13:31:20.461137 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

2012-05-23 13:31:25.461206 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

2012-05-23 13:31:25.461230 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

2012-05-23 13:31:30.461302 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

2012-05-23 13:31:30.461326 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

2012-05-23 13:31:35.461391 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

2012-05-23 13:31:35.461412 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

2012-05-23 13:31:40.461538 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

2012-05-23 13:31:40.461562 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

2012-05-23 13:31:45.461633 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

2012-05-23 13:31:45.461658 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

2012-05-23 13:31:50.461722 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

2012-05-23 13:31:50.461742 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

2012-05-23 13:31:55.461808 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

2012-05-23 13:31:55.461829 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

2012-05-23 13:32:00.461900 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

2012-05-23 13:32:00.461924 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

2012-05-23 13:32:05.462109 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

2012-05-23 13:32:05.462125 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

2012-05-23 13:32:10.462195 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

2012-05-23 13:32:10.462218 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

2012-05-23 13:32:15.462288 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

2012-05-23 13:32:15.462315 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

2012-05-23 13:32:20.462428 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

2012-05-23 13:32:20.462460 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

2012-05-23 13:32:25.462539 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

2012-05-23 13:32:25.462574 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

2012-05-23 13:32:30.462652 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

2012-05-23 13:32:30.462681 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

2012-05-23 13:32:35.462756 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

2012-05-23 13:32:35.462780 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

2012-05-23 13:32:40.462852 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

2012-05-23 13:32:40.462875 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

2012-05-23 13:32:45.462944 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

2012-05-23 13:32:45.462965 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

2012-05-23 13:32:50.463453 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

2012-05-23 13:32:50.463476 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

2012-05-23 13:32:55.463541 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

2012-05-23 13:32:55.463567 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

2012-05-23 13:33:00.463638 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

2012-05-23 13:33:00.463661 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

2012-05-23 13:33:05.463740 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

2012-05-23 13:33:05.463761 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had suicide timed out after 300

2012-05-23 13:33:08.958564 7fe61124d700 -1 common/HeartbeatMap.cc: In function 'bool ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d*, const char*, time_t)' thread 7fe61124d700 time 2012-05-23 13:33:06.572705

common/HeartbeatMap.cc: 78: FAILED assert(0 == "hit suicide timeout")

 ceph version 0.47.1 (commit:f5a9404445e2ed5ec2ee828aa53d73d4a002f7a5)

 1: (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d*, char const*, long)+0x460) [0x82a800]

 2: (ceph::HeartbeatMap::is_healthy()+0x87) [0x82ae27]

 3: (ceph::HeartbeatMap::check_touch_file()+0x23) [0x82b063]

 4: (CephContextServiceThread::entry()+0x54) [0x7ad054]

 5: (()+0x8ec6) [0x7fe6130a3ec6]

 6: (clone()+0x6d) [0x7fe611d5051d]

 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---

  -140> 2012-05-23 13:14:15.441703 7fe613e02780  0 ceph version 0.47.1 (commit:f5a9404445e2ed5ec2ee828aa53d73d4a002f7a5), process ceph-osd, pid 28673

  -139> 2012-05-23 13:14:15.733081 7fe613e02780  0 filestore(/data/osd.0) mount FIEMAP ioctl is supported and appears to work

  -138> 2012-05-23 13:14:15.733524 7fe613e02780  0 filestore(/data/osd.0) mount detected btrfs

  -137> 2012-05-23 13:14:15.733549 7fe613e02780  0 filestore(/data/osd.0) mount btrfs CLONE_RANGE ioctl is supported

  -136> 2012-05-23 13:14:16.741640 7fe613e02780  0 filestore(/data/osd.0) mount btrfs SNAP_CREATE is supported

  -135> 2012-05-23 13:14:16.742251 7fe613e02780  0 filestore(/data/osd.0) mount btrfs SNAP_DESTROY is supported

  -134> 2012-05-23 13:14:16.864784 7fe613e02780  0 filestore(/data/osd.0) mount btrfs START_SYNC is supported (transid 47504)

  -133> 2012-05-23 13:14:17.070262 7fe613e02780  0 filestore(/data/osd.0) mount btrfs WAIT_SYNC is supported

  -132> 2012-05-23 13:14:17.350699 7fe613e02780  0 filestore(/data/osd.0) mount btrfs SNAP_CREATE_V2 is supported

  -131> 2012-05-23 13:14:17.354178 7fe613e02780  0 filestore(/data/osd.0) mount found snaps <884133,884152>

  -130> 2012-05-23 13:14:18.480607 7fe613e02780  0 filestore(/data/osd.0) mount: enabling PARALLEL journal mode: btrfs, SNAP_CREATE_V2 detected and 'filestore btrfs snap' mode is enabled

  -129> 2012-05-23 13:14:18.503060 7fe613e02780  0 journal  kernel version is 3.4.0

  -128> 2012-05-23 13:14:18.503136 7fe613e02780  1 journal _open /dev/sda5 fd 25: 1019899904 bytes, block size 4096 bytes, directio = 1, aio = 0

  -127> 2012-05-23 13:14:25.871173 7fe613e02780  0 journal  kernel version is 3.4.0

  -126> 2012-05-23 13:14:25.871298 7fe613e02780  1 journal _open /dev/sda5 fd 25: 1019899904 bytes, block size 4096 bytes, directio = 1, aio = 0

  -125> 2012-05-23 13:14:26.234186 7fe613e02780  1 journal close /dev/sda5

  -124> 2012-05-23 13:14:26.405298 7fe613e02780  0 filestore(/data/osd.0) mount FIEMAP ioctl is supported and appears to work

  -123> 2012-05-23 13:14:26.405731 7fe613e02780  0 filestore(/data/osd.0) mount detected btrfs

  -122> 2012-05-23 13:14:26.405742 7fe613e02780  0 filestore(/data/osd.0) mount btrfs CLONE_RANGE ioctl is supported

  -121> 2012-05-23 13:14:26.633024 7fe613e02780  0 filestore(/data/osd.0) mount btrfs SNAP_CREATE is supported

  -120> 2012-05-23 13:14:26.634151 7fe613e02780  0 filestore(/data/osd.0) mount btrfs SNAP_DESTROY is supported

  -119> 2012-05-23 13:14:26.734524 7fe613e02780  0 filestore(/data/osd.0) mount btrfs START_SYNC is supported (transid 47511)

  -118> 2012-05-23 13:14:26.901185 7fe613e02780  0 filestore(/data/osd.0) mount btrfs WAIT_SYNC is supported

  -117> 2012-05-23 13:14:27.162293 7fe613e02780  0 filestore(/data/osd.0) mount btrfs SNAP_CREATE_V2 is supported

  -116> 2012-05-23 13:14:27.170097 7fe613e02780  0 filestore(/data/osd.0) mount found snaps <884152,884345>

  -115> 2012-05-23 13:14:27.765579 7fe613e02780  0 filestore(/data/osd.0) mount: enabling PARALLEL journal mode: btrfs, SNAP_CREATE_V2 detected and 'filestore btrfs snap' mode is enabled

  -114> 2012-05-23 13:14:27.778073 7fe613e02780  0 journal  kernel version is 3.4.0

  -113> 2012-05-23 13:14:27.778173 7fe613e02780  1 journal _open /dev/sda5 fd 29: 1019899904 bytes, block size 4096 bytes, directio = 1, aio = 0

  -112> 2012-05-23 13:14:28.538025 7fe613e02780  0 journal  kernel version is 3.4.0

  -111> 2012-05-23 13:14:28.538112 7fe613e02780  1 journal _open /dev/sda5 fd 29: 1019899904 bytes, block size 4096 bytes, directio = 1, aio = 0

  -110> 2012-05-23 13:28:35.456547 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

  -109> 2012-05-23 13:28:35.456585 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

  -108> 2012-05-23 13:28:40.456652 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

  -107> 2012-05-23 13:28:40.456680 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

  -106> 2012-05-23 13:28:45.456747 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

  -105> 2012-05-23 13:28:45.456768 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

  -104> 2012-05-23 13:28:50.456843 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

  -103> 2012-05-23 13:28:50.456874 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

  -102> 2012-05-23 13:28:55.456945 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

  -101> 2012-05-23 13:28:55.456971 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

  -100> 2012-05-23 13:29:00.457039 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

   -99> 2012-05-23 13:29:00.457067 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

   -98> 2012-05-23 13:29:05.458511 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

   -97> 2012-05-23 13:29:05.458537 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

   -96> 2012-05-23 13:29:10.458606 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

   -95> 2012-05-23 13:29:10.458639 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

   -94> 2012-05-23 13:29:15.458717 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

   -93> 2012-05-23 13:29:15.458741 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

   -92> 2012-05-23 13:29:20.458816 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

   -91> 2012-05-23 13:29:20.458841 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

   -90> 2012-05-23 13:29:25.458909 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

   -89> 2012-05-23 13:29:25.458938 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

   -88> 2012-05-23 13:29:30.459013 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

   -87> 2012-05-23 13:29:30.459036 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

   -86> 2012-05-23 13:29:35.459099 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

   -85> 2012-05-23 13:29:35.459129 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

   -84> 2012-05-23 13:29:40.459193 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

   -83> 2012-05-23 13:29:40.459207 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

   -82> 2012-05-23 13:29:45.459277 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

   -81> 2012-05-23 13:29:45.459311 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

   -80> 2012-05-23 13:29:50.459378 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

   -79> 2012-05-23 13:29:50.459398 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

   -78> 2012-05-23 13:29:55.459469 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

   -77> 2012-05-23 13:29:55.459496 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

   -76> 2012-05-23 13:30:00.459562 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

   -75> 2012-05-23 13:30:00.459581 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

   -74> 2012-05-23 13:30:05.459645 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

   -73> 2012-05-23 13:30:05.459674 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

   -72> 2012-05-23 13:30:10.459750 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

   -71> 2012-05-23 13:30:10.459774 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

   -70> 2012-05-23 13:30:15.459842 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

   -69> 2012-05-23 13:30:15.459866 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

   -68> 2012-05-23 13:30:20.459936 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

   -67> 2012-05-23 13:30:20.459960 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

   -66> 2012-05-23 13:30:25.460026 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

   -65> 2012-05-23 13:30:25.460049 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

   -64> 2012-05-23 13:30:30.460124 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

   -63> 2012-05-23 13:30:30.460150 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

   -62> 2012-05-23 13:30:35.460246 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

   -61> 2012-05-23 13:30:35.460287 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

   -60> 2012-05-23 13:30:40.460356 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

   -59> 2012-05-23 13:30:40.460381 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

   -58> 2012-05-23 13:30:45.460458 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

   -57> 2012-05-23 13:30:45.460481 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

   -56> 2012-05-23 13:30:50.460548 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

   -55> 2012-05-23 13:30:50.460569 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

   -54> 2012-05-23 13:30:55.460641 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

   -53> 2012-05-23 13:30:55.460659 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

   -52> 2012-05-23 13:31:00.460729 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

   -51> 2012-05-23 13:31:00.460753 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

   -50> 2012-05-23 13:31:05.460826 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

   -49> 2012-05-23 13:31:05.460847 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

   -48> 2012-05-23 13:31:10.460917 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

   -47> 2012-05-23 13:31:10.460941 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

   -46> 2012-05-23 13:31:15.461015 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

   -45> 2012-05-23 13:31:15.461038 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

   -44> 2012-05-23 13:31:20.461109 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

   -43> 2012-05-23 13:31:20.461137 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

   -42> 2012-05-23 13:31:25.461206 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

   -41> 2012-05-23 13:31:25.461230 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

   -40> 2012-05-23 13:31:30.461302 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

   -39> 2012-05-23 13:31:30.461326 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

   -38> 2012-05-23 13:31:35.461391 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

   -37> 2012-05-23 13:31:35.461412 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

   -36> 2012-05-23 13:31:40.461538 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

   -35> 2012-05-23 13:31:40.461562 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

   -34> 2012-05-23 13:31:45.461633 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

   -33> 2012-05-23 13:31:45.461658 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

   -32> 2012-05-23 13:31:50.461722 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

   -31> 2012-05-23 13:31:50.461742 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

   -30> 2012-05-23 13:31:55.461808 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

   -29> 2012-05-23 13:31:55.461829 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

   -28> 2012-05-23 13:32:00.461900 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

   -27> 2012-05-23 13:32:00.461924 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

   -26> 2012-05-23 13:32:05.462109 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

   -25> 2012-05-23 13:32:05.462125 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

   -24> 2012-05-23 13:32:10.462195 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

   -23> 2012-05-23 13:32:10.462218 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

   -22> 2012-05-23 13:32:15.462288 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

   -21> 2012-05-23 13:32:15.462315 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

   -20> 2012-05-23 13:32:20.462428 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

   -19> 2012-05-23 13:32:20.462460 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

   -18> 2012-05-23 13:32:25.462539 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

   -17> 2012-05-23 13:32:25.462574 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

   -16> 2012-05-23 13:32:30.462652 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

   -15> 2012-05-23 13:32:30.462681 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

   -14> 2012-05-23 13:32:35.462756 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

   -13> 2012-05-23 13:32:35.462780 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

   -12> 2012-05-23 13:32:40.462852 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

   -11> 2012-05-23 13:32:40.462875 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

   -10> 2012-05-23 13:32:45.462944 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

    -9> 2012-05-23 13:32:45.462965 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

    -8> 2012-05-23 13:32:50.463453 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

    -7> 2012-05-23 13:32:50.463476 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

    -6> 2012-05-23 13:32:55.463541 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

    -5> 2012-05-23 13:32:55.463567 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

    -4> 2012-05-23 13:33:00.463638 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

    -3> 2012-05-23 13:33:00.463661 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60292f700' had timed out after 30

    -2> 2012-05-23 13:33:05.463740 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had timed out after 30

    -1> 2012-05-23 13:33:05.463761 7fe61124d700  1 heartbeat_map is_healthy 'OSD::op_tp thread 0x7fe60212e700' had suicide timed out after 300

     0> 2012-05-23 13:33:08.958564 7fe61124d700 -1 common/HeartbeatMap.cc: In function 'bool ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d*, const char*, time_t)' thread 7fe61124d700 time 2012-05-23 13:33:06.572705

common/HeartbeatMap.cc: 78: FAILED assert(0 == "hit suicide timeout")

 ceph version 0.47.1 (commit:f5a9404445e2ed5ec2ee828aa53d73d4a002f7a5)

 1: (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d*, char const*, long)+0x460) [0x82a800]

 2: (ceph::HeartbeatMap::is_healthy()+0x87) [0x82ae27]

 3: (ceph::HeartbeatMap::check_touch_file()+0x23) [0x82b063]

 4: (CephContextServiceThread::entry()+0x54) [0x7ad054]

 5: (()+0x8ec6) [0x7fe6130a3ec6]

 6: (clone()+0x6d) [0x7fe611d5051d]

 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- end dump of recent events ---

2012-05-23 13:33:09.013782 7fe61124d700 -1 *** Caught signal (Aborted) **

 in thread 7fe61124d700

 ceph version 0.47.1 (commit:f5a9404445e2ed5ec2ee828aa53d73d4a002f7a5)

 1: /usr/bin/ceph-osd() [0x6f2847]

 2: (()+0x10bf0) [0x7fe6130abbf0]

 3: (gsignal()+0x35) [0x7fe611c9fa95]

 4: (abort()+0x17b) [0x7fe611ca0f0b]

 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7fe6125ccbed]

 6: (()+0xb9da6) [0x7fe6125cada6]

 7: (()+0xb9dd3) [0x7fe6125cadd3]

 8: (()+0xb9ece) [0x7fe6125caece]

 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26f) [0x7a406f]

 10: (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d*, char const*, long)+0x460) [0x82a800]

 11: (ceph::HeartbeatMap::is_healthy()+0x87) [0x82ae27]

 12: (ceph::HeartbeatMap::check_touch_file()+0x23) [0x82b063]

 13: (CephContextServiceThread::entry()+0x54) [0x7ad054]

 14: (()+0x8ec6) [0x7fe6130a3ec6]

 15: (clone()+0x6d) [0x7fe611d5051d]

 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---

     0> 2012-05-23 13:33:09.013782 7fe61124d700 -1 *** Caught signal (Aborted) **

 in thread 7fe61124d700

 ceph version 0.47.1 (commit:f5a9404445e2ed5ec2ee828aa53d73d4a002f7a5)

 1: /usr/bin/ceph-osd() [0x6f2847]

 2: (()+0x10bf0) [0x7fe6130abbf0]

 3: (gsignal()+0x35) [0x7fe611c9fa95]

 4: (abort()+0x17b) [0x7fe611ca0f0b]

 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7fe6125ccbed]

 6: (()+0xb9da6) [0x7fe6125cada6]

 7: (()+0xb9dd3) [0x7fe6125cadd3]

 8: (()+0xb9ece) [0x7fe6125caece]

 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26f) [0x7a406f]

 10: (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d*, char const*, long)+0x460) [0x82a800]

 11: (ceph::HeartbeatMap::is_healthy()+0x87) [0x82ae27]

 12: (ceph::HeartbeatMap::check_touch_file()+0x23) [0x82b063]

 13: (CephContextServiceThread::entry()+0x54) [0x7ad054]

 14: (()+0x8ec6) [0x7fe6130a3ec6]

 15: (clone()+0x6d) [0x7fe611d5051d]

 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- end dump of recent events ---

2012-05-23 13:46:30.825695 7fbd1f2f6780  0 ceph version 0.47.1 (commit:f5a9404445e2ed5ec2ee828aa53d73d4a002f7a5), process ceph-osd, pid 2598

2012-05-23 13:46:31.888872 7fbd1f2f6780  0 filestore(/data/osd.0) mount FIEMAP ioctl is supported and appears to work

2012-05-23 13:46:32.028260 7fbd1f2f6780  0 filestore(/data/osd.0) mount detected btrfs

2012-05-23 13:46:32.028292 7fbd1f2f6780  0 filestore(/data/osd.0) mount btrfs CLONE_RANGE ioctl is supported

2012-05-23 13:46:32.798640 7fbd1f2f6780  0 filestore(/data/osd.0) mount btrfs SNAP_CREATE is supported

2012-05-23 13:46:32.818091 7fbd1f2f6780  0 filestore(/data/osd.0) mount btrfs SNAP_DESTROY is supported

2012-05-23 13:46:32.818538 7fbd1f2f6780  0 filestore(/data/osd.0) mount btrfs START_SYNC is supported (transid 47642)

2012-05-23 13:46:32.933426 7fbd1f2f6780  0 filestore(/data/osd.0) mount btrfs WAIT_SYNC is supported

2012-05-23 13:46:33.749595 7fbd1f2f6780  0 filestore(/data/osd.0) mount btrfs SNAP_CREATE_V2 is supported

2012-05-23 13:46:33.754180 7fbd1f2f6780  0 filestore(/data/osd.0) mount found snaps <894661,895198>

2012-05-23 13:46:41.641907 7ff9dabbf780  0 ceph version 0.47.1 (commit:f5a9404445e2ed5ec2ee828aa53d73d4a002f7a5), process ceph-osd, pid 3739

2012-05-23 13:46:41.883662 7ff9dabbf780  0 filestore(/data/osd.0) mount FIEMAP ioctl is supported and appears to work

2012-05-23 13:46:41.884234 7ff9dabbf780  0 filestore(/data/osd.0) mount detected btrfs

2012-05-23 13:46:41.884258 7ff9dabbf780  0 filestore(/data/osd.0) mount btrfs CLONE_RANGE ioctl is supported

2012-05-23 13:46:43.723229 7ff9dabbf780  0 filestore(/data/osd.0) mount btrfs SNAP_CREATE is supported

2012-05-23 13:46:43.723838 7ff9dabbf780  0 filestore(/data/osd.0) mount btrfs SNAP_DESTROY is supported

2012-05-23 13:46:43.724157 7ff9dabbf780  0 filestore(/data/osd.0) mount btrfs START_SYNC is supported (transid 47646)

2012-05-23 13:46:43.832690 7ff9dabbf780  0 filestore(/data/osd.0) mount btrfs WAIT_SYNC is supported

2012-05-23 13:46:44.199892 7ff9dabbf780  0 filestore(/data/osd.0) mount btrfs SNAP_CREATE_V2 is supported

2012-05-23 13:46:44.912237 7ff9dabbf780  0 filestore(/data/osd.0) mount found snaps <894661,895198>

2012-05-23 13:46:52.997074 7f88088cb780  0 ceph version 0.47.1 (commit:f5a9404445e2ed5ec2ee828aa53d73d4a002f7a5), process ceph-osd, pid 4889

2012-05-23 13:46:53.116230 7f88088cb780  0 filestore(/data/osd.0) mount FIEMAP ioctl is supported and appears to work

2012-05-23 13:46:53.116537 7f88088cb780  0 filestore(/data/osd.0) mount detected btrfs

2012-05-23 13:46:53.116554 7f88088cb780  0 filestore(/data/osd.0) mount btrfs CLONE_RANGE ioctl is supported

2012-05-23 13:46:53.912130 7f88088cb780  0 filestore(/data/osd.0) mount btrfs SNAP_CREATE is supported

2012-05-23 13:46:53.912291 7f88088cb780  0 filestore(/data/osd.0) mount btrfs SNAP_DESTROY is supported

2012-05-23 13:46:53.962067 7f88088cb780  0 filestore(/data/osd.0) mount btrfs START_SYNC is supported (transid 47650)

2012-05-23 13:46:54.422149 7f88088cb780  0 filestore(/data/osd.0) mount btrfs WAIT_SYNC is supported

2012-05-23 13:46:54.591489 7f88088cb780  0 filestore(/data/osd.0) mount btrfs SNAP_CREATE_V2 is supported

2012-05-23 13:46:54.602115 7f88088cb780  0 filestore(/data/osd.0) mount found snaps <894661,895198>

2012-05-23 13:46:55.955985 7f88088cb780  0 filestore(/data/osd.0) mount: enabling PARALLEL journal mode: btrfs, SNAP_CREATE_V2 detected and 'filestore btrfs snap' mode is enabled

2012-05-23 13:46:58.289295 7f88088cb780  0 journal  kernel version is 3.4.0

2012-05-23 13:46:58.289387 7f88088cb780  1 journal _open /dev/sda5 fd 22: 1019899904 bytes, block size 4096 bytes, directio = 1, aio = 0

2012-05-23 13:47:29.904431 7fc0fb13f780  0 ceph version 0.47.1 (commit:f5a9404445e2ed5ec2ee828aa53d73d4a002f7a5), process ceph-osd, pid 6052

2012-05-23 13:47:30.094823 7fc0fb13f780  0 filestore(/data/osd.0) mount FIEMAP ioctl is supported and appears to work

2012-05-23 13:47:30.095310 7fc0fb13f780  0 filestore(/data/osd.0) mount detected btrfs

2012-05-23 13:47:30.095328 7fc0fb13f780  0 filestore(/data/osd.0) mount btrfs CLONE_RANGE ioctl is supported

2012-05-23 13:47:30.716607 7fc0fb13f780  0 filestore(/data/osd.0) mount btrfs SNAP_CREATE is supported

2012-05-23 13:47:30.717089 7fc0fb13f780  0 filestore(/data/osd.0) mount btrfs SNAP_DESTROY is supported

2012-05-23 13:47:30.718662 7fc0fb13f780  0 filestore(/data/osd.0) mount btrfs START_SYNC is supported (transid 47655)

2012-05-23 13:47:30.981313 7fc0fb13f780  0 filestore(/data/osd.0) mount btrfs WAIT_SYNC is supported

2012-05-23 13:47:31.067894 7fc0fb13f780  0 filestore(/data/osd.0) mount btrfs SNAP_CREATE_V2 is supported

2012-05-23 13:47:31.069186 7fc0fb13f780  0 filestore(/data/osd.0) mount found snaps <894661,895198>

2012-05-23 13:47:32.274316 7fc0fb13f780  0 filestore(/data/osd.0) mount: enabling PARALLEL journal mode: btrfs, SNAP_CREATE_V2 detected and 'filestore btrfs snap' mode is enabled

2012-05-23 13:47:32.325215 7fc0fb13f780  0 journal  kernel version is 3.4.0

2012-05-23 13:47:32.325285 7fc0fb13f780  1 journal _open /dev/sda5 fd 22: 1019899904 bytes, block size 4096 bytes, directio = 1, aio = 0