Bug #8098
closedceph v0.79-125 : Random osd's are flapping too frequently : OSD wrongly marked me down
0%
Description
Hello Developers
I have been facing weird problem with my ceph cluster
Problem : Randomly OSDs are flapping , i.e some osds are getting down and in few seconds they are coming up , and there always some osds are down. So this problem is not specific to few OSDS but to entire cluster OSDs.
- Disk , node , network and other infrastructure everything i have checked and is OK.
- originally cluster was running ceph version 0.79 on all the nodes
- I thought there might be some bugs in this version so i upgraded it to 0.79-125 and rebooted entire cluster after upgradation , but still problem persist.
- After that i again upgraded to 0.79-185 from master which is the latest version , but NO LUCK
- FYI before and after upgradation , i have restarted all ceph services as well as OS reboot , but NO LUCK
- Debug MS = 1 , set to OSDs
- Below is the cluster health command output in difference of few seconds , it clearly shows OSDS are getting down and coming up
[root@storage0115-ib ~]# [root@storage0115-ib ~]# date;ceph -s Mon Apr 14 19:12:06 EEST 2014 cluster e8768ef6-93a1-4e8c-acda-3ac995e35003 health HEALTH_WARN 94 pgs peering; 29 pgs stale; *1/165 in osds are down* monmap e3: 3 mons at {storage0101-ib=192.168.100.101:6789/0,storage0107-ib=192.168.100.107:6789/0,storage0115-ib=192.168.100.115:6789/0}, election epoch 50, quorum 0,1,2 storage0101-ib,storage0107-ib,storage0115-ib mdsmap e67: 1/1/1 up {0=storage0110=up:active}, 3 up:standby osdmap e19041: 165 osds: 164 up, 165 in pgmap v38433: 5120 pgs, 3 pools, 5733 GB data, 1628 kobjects 5290 GB used, 443 TB / 448 TB avail 29 stale+active+clean 94 peering 4997 active+clean [root@storage0115-ib ~]# [root@storage0115-ib ~]# [root@storage0115-ib ~]# date;ceph -s *Mon Apr 14 19:12:10 EEST 2014* cluster e8768ef6-93a1-4e8c-acda-3ac995e35003 health HEALTH_WARN 93 pgs peering; 272 pgs stale; *8/165 in osds are down* monmap e3: 3 mons at {storage0101-ib=192.168.100.101:6789/0,storage0107-ib=192.168.100.107:6789/0,storage0115-ib=192.168.100.115:6789/0}, election epoch 50, quorum 0,1,2 storage0101-ib,storage0107-ib,storage0115-ib mdsmap e67: 1/1/1 up {0=storage0110=up:active}, 3 up:standby osdmap e19044: 165 osds: 157 up, 165 in pgmap v38439: 5120 pgs, 3 pools, 5733 GB data, 1628 kobjects 5290 GB used, 443 TB / 448 TB avail 2 stale+peering 270 stale+active+clean 91 peering 4757 active+clean [root@storage0115-ib ~]# date;ceph -s *Mon Apr 14 19:12:15 EEST 2014* cluster e8768ef6-93a1-4e8c-acda-3ac995e35003 health HEALTH_WARN 94 pgs peering; 371 pgs stale; *2/165 in osds are down* monmap e3: 3 mons at {storage0101-ib=192.168.100.101:6789/0,storage0107-ib=192.168.100.107:6789/0,storage0115-ib=192.168.100.115:6789/0}, election epoch 50, quorum 0,1,2 storage0101-ib,storage0107-ib,storage0115-ib mdsmap e67: 1/1/1 up {0=storage0110=up:active}, 3 up:standby osdmap e19049: 165 osds: 163 up, 165 in pgmap v38444: 5120 pgs, 3 pools, 5733 GB data, 1628 kobjects 5290 GB used, 443 TB / 448 TB avail 2 stale+peering 369 stale+active+clean 92 peering 4657 active+clean [root@storage0115-ib ~]# date;ceph -s *Mon Apr 14 19:12:19 EEST 2014* cluster e8768ef6-93a1-4e8c-acda-3ac995e35003 health HEALTH_WARN 6 pgs degraded; 94 pgs peering; 489 pgs stale; 2 pgs stuck unclean; recovery 958/5002182 objects degraded (0.019%); *2/165 in osds are down* monmap e3: 3 mons at {storage0101-ib=192.168.100.101:6789/0,storage0107-ib=192.168.100.107:6789/0,storage0115-ib=192.168.100.115:6789/0}, election epoch 50, quorum 0,1,2 storage0101-ib,storage0107-ib,storage0115-ib mdsmap e67: 1/1/1 up {0=storage0110=up:active}, 3 up:standby osdmap e19053: 165 osds: 163 up, 165 in pgmap v38448: 5120 pgs, 3 pools, 5733 GB data, 1628 kobjects 5290 GB used, 443 TB / 448 TB avail 958/5002182 objects degraded (0.019%) 11 stale+peering 478 stale+active+clean 6 active+degraded 83 peering 4542 active+clean [root@storage0115-ib ~]# date;ceph -s *Mon Apr 14 19:12:26 EEST 2014* cluster e8768ef6-93a1-4e8c-acda-3ac995e35003 health HEALTH_WARN 5 pgs degraded; 91 pgs peering; 438 pgs stale; 1 pgs stuck unclean; recovery 886/5002182 objects degraded (0.018%); *1/165 in osds are dow*n monmap e3: 3 mons at {storage0101-ib=192.168.100.101:6789/0,storage0107-ib=192.168.100.107:6789/0,storage0115-ib=192.168.100.115:6789/0}, election epoch 50, quorum 0,1,2 storage0101-ib,storage0107-ib,storage0115-ib mdsmap e67: 1/1/1 up {0=storage0110=up:active}, 3 up:standby osdmap e19060: 165 osds: 164 up, 165 in pgmap v38458: 5120 pgs, 3 pools, 5733 GB data, 1628 kobjects 5290 GB used, 443 TB / 448 TB avail 886/5002182 objects degraded (0.018%) 12 stale+peering 426 stale+active+clean 5 active+degraded 79 peering 4598 active+clean [root@storage0115-ib ~]# date;ceph -s *Mon Apr 14 19:12:35 EEST 2014* cluster e8768ef6-93a1-4e8c-acda-3ac995e35003 health HEALTH_WARN 10 pgs degraded; 87 pgs peering; 366 pgs stale; 1 pgs stuck unclean; recovery 5683/5002182 objects degraded (0.114%) monmap e3: 3 mons at {storage0101-ib=192.168.100.101:6789/0,storage0107-ib=192.168.100.107:6789/0,storage0115-ib=192.168.100.115:6789/0}, election epoch 50, quorum 0,1,2 storage0101-ib,storage0107-ib,storage0115-ib mdsmap e67: 1/1/1 up {0=storage0110=up:active}, 3 up:standby *osdmap e19067: 165 osds: 165 up, 165 in* pgmap v38467: 5120 pgs, 3 pools, 5733 GB data, 1628 kobjects 5291 GB used, 443 TB / 448 TB avail 5683/5002182 objects degraded (0.114%) 21 stale+peering 345 stale+active+clean 10 active+degraded 66 peering 4678 active+clean [root@storage0115-ib ~]# date;ceph -s *Mon Apr 14 19:12:47 EEST 2014* cluster e8768ef6-93a1-4e8c-acda-3ac995e35003 health HEALTH_WARN 2 pgs degraded; 85 pgs peering; 169 pgs stale; 2 pgs stuck unclean; recovery 423/5002182 objects degraded (0.008%) monmap e3: 3 mons at {storage0101-ib=192.168.100.101:6789/0,storage0107-ib=192.168.100.107:6789/0,storage0115-ib=192.168.100.115:6789/0}, election epoch 50, quorum 0,1,2 storage0101-ib,storage0107-ib,storage0115-ib mdsmap e67: 1/1/1 up {0=storage0110=up:active}, 3 up:standby * osdmap e19076: 165 osds: 165 up, 165 in* pgmap v38482: 5120 pgs, 3 pools, 5733 GB data, 1628 kobjects 5291 GB used, 443 TB / 448 TB avail 423/5002182 objects degraded (0.008%) 6 stale+peering 163 stale+active+clean 2 active+degraded 79 peering 4870 active+clean [root@storage0115-ib ~]# date;ceph -s *Mon Apr 14 19:12:58 EEST 2014* cluster e8768ef6-93a1-4e8c-acda-3ac995e35003 health HEALTH_WARN 1 pgs degraded; 79 pgs peering; 314 pgs stale; recovery 339/5002182 objects degraded (0.007%); *1/165 in osds are down* monmap e3: 3 mons at {storage0101-ib=192.168.100.101:6789/0,storage0107-ib=192.168.100.107:6789/0,storage0115-ib=192.168.100.115:6789/0}, election epoch 50, quorum 0,1,2 storage0101-ib,storage0107-ib,storage0115-ib mdsmap e67: 1/1/1 up {0=storage0110=up:active}, 3 up:standby osdmap e19083: 165 osds: 164 up, 165 in pgmap v38492: 5120 pgs, 3 pools, 5733 GB data, 1628 kobjects 5291 GB used, 443 TB / 448 TB avail 339/5002182 objects degraded (0.007%) 7 stale+peering 307 stale+active+clean 1 active+degraded 72 peering 4733 active+clean [root@storage0115-ib ~]#
- As an example i have copied logs of osd.163 and osd.51.log
LOGS FROM osd.163
[root@storage0112-ib ceph]# tail -100f ceph-osd.163.log 2014-04-14 18:49:26.760144 7fe8b6542700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.106:6807/3008357 pipe(0x4512a80 sd=255 :6821 s=2 pgs=628 cs=1 l=0 c=0x4e586e0).fault with nothing to send, going to standby 2014-04-14 18:49:26.760262 7fe8a5735700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.102:6818/1008897 pipe(0x3e05780 sd=230 :6821 s=2 pgs=528 cs=1 l=0 c=0x4e44ba0).fault with nothing to send, going to standby 2014-04-14 18:49:26.760363 7fe89eac9700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.102:6817/1013999 pipe(0x4da7a80 sd=57 :55066 s=2 pgs=501 cs=1 l=0 c=0x3f13700).fault with nothing to send, going to standby 2014-04-14 18:49:26.760497 7fe8b9d71700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.111:6805/2019168 pipe(0x4105f00 sd=215 :57674 s=2 pgs=504 cs=1 l=0 c=0x6939080).fault with nothing to send, going to standby 2014-04-14 18:49:26.760571 7fe8b9f73700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.108:6806/3006246 pipe(0x4510000 sd=207 :52322 s=2 pgs=705 cs=1 l=0 c=0x6939760).fault with nothing to send, going to standby 2014-04-14 18:49:26.761037 7fe8bb68a700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.114:6800/1064947 pipe(0x4510280 sd=201 :56237 s=2 pgs=427 cs=1 l=0 c=0x693cfc0).fault with nothing to send, going to standby 2014-04-14 18:49:26.761041 7fe8c23f7700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.115:6804/1020908 pipe(0x4da6900 sd=132 :45679 s=2 pgs=424 cs=1 l=0 c=0x6abeb40).fault with nothing to send, going to standby 2014-04-14 18:49:26.761287 7fe8ba276700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.115:6800/4012452 pipe(0x4dfdc80 sd=244 :6821 s=2 pgs=714 cs=1 l=0 c=0x3f14a40).fault with nothing to send, going to standby 2014-04-14 18:49:26.761646 7fe8b04e2700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.106:6800/3011682 pipe(0x3f7c100 sd=69 :6821 s=2 pgs=709 cs=1 l=0 c=0x45b70c0).fault with nothing to send, going to standby 2014-04-14 18:49:26.762007 7fe8b996d700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.108:6804/5178 pipe(0x4da4600 sd=144 :55370 s=2 pgs=368 cs=1 l=0 c=0x6918dc0).fault with nothing to send, going to standby 2014-04-14 18:49:26.761908 7fe8bd7ab700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.110:6827/37510 pipe(0x4107580 sd=163 :55475 s=2 pgs=358 cs=1 l=0 c=0x691aaa0).fault with nothing to send, going to standby 2014-04-14 18:49:26.763135 7fe89d0af700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.115:6813/2017035 pipe(0x4107800 sd=463 :6821 s=2 pgs=449 cs=1 l=0 c=0x6abc620).fault with nothing to send, going to standby 2014-04-14 18:49:26.763279 7fe89cbaa700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.101:6809/4029366 pipe(0x51c2d00 sd=323 :6821 s=2 pgs=816 cs=1 l=0 c=0x460d6a0).fault with nothing to send, going to standby 2014-04-14 18:49:26.763461 7fe8aeecc700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.107:6822/2008022 pipe(0x3f7b200 sd=210 :6821 s=2 pgs=490 cs=1 l=0 c=0x4e253e0).fault with nothing to send, going to standby 2014-04-14 18:49:26.763513 7fe8bbb8f700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.111:6818/1011034 pipe(0x4da0000 sd=190 :41808 s=2 pgs=431 cs=1 l=0 c=0x693aec0).fault with nothing to send, going to standby 2014-04-14 18:49:26.762625 7fe8aa684700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.105:6813/9002701 pipe(0x4dfb980 sd=334 :6821 s=2 pgs=1523 cs=1 l=0 c=0x3f13180).fault with nothing to send, going to standby 2014-04-14 18:49:26.764078 7fe8ba87c700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.103:6808/4033047 pipe(0x4105780 sd=156 :42310 s=2 pgs=766 cs=1 l=0 c=0x691b9c0).fault with nothing to send, going to standby 2014-04-14 18:49:26.762769 7fe8c1aee700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.110:6814/1030992 pipe(0x4da6e00 sd=134 :46412 s=2 pgs=482 cs=1 l=0 c=0x4640f20).fault with nothing to send, going to standby 2014-04-14 18:49:26.762776 7fe8deb7c700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.101:6801/1030688 pipe(0x3f79400 sd=93 :38016 s=2 pgs=534 cs=1 l=0 c=0x45fdac0).fault with nothing to send, going to standby 2014-04-14 18:49:26.764153 7fe8aa583700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.103:6807/3026719 pipe(0x4103980 sd=92 :6821 s=2 pgs=743 cs=1 l=0 c=0x691dac0).fault with nothing to send, going to standby 2014-04-14 18:49:26.764210 7fe8b502d700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.115:6819/4009855 pipe(0x3e02080 sd=119 :6821 s=2 pgs=816 cs=1 l=0 c=0x3f144c0).fault with nothing to send, going to standby 2014-04-14 18:49:26.764983 7fe8b7d5a700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.113:6822/1064877 pipe(0x4101680 sd=228 :33958 s=2 pgs=439 cs=1 l=0 c=0x4e2bb20).fault with nothing to send, going to standby 2014-04-14 18:49:26.765238 7fe8ae9c7700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.106:6811/3004240 pipe(0x6ab0780 sd=159 :6821 s=2 pgs=671 cs=1 l=0 c=0x3906b40).fault with nothing to send, going to standby 2014-04-14 18:49:26.765345 7fe8ded7e700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.102:6824/15636 pipe(0x4da4100 sd=104 :54179 s=2 pgs=394 cs=1 l=0 c=0x3f10f20).fault with nothing to send, going to standby 2014-04-14 18:49:26.765489 7fe8b2f0d700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.101:6811/3023921 pipe(0x4dfe400 sd=150 :6821 s=2 pgs=787 cs=1 l=0 c=0x6939b80).fault with nothing to send, going to standby 2014-04-14 18:49:26.765740 7fe8c25f9700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.102:6810/1019423 pipe(0x4da0a00 sd=108 :34679 s=2 pgs=492 cs=1 l=0 c=0x4e5b5a0).fault with nothing to send, going to standby 2014-04-14 18:49:26.765835 7fe89e1c0700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.105:6803/5003300 pipe(0x4dfd000 sd=45 :6821 s=2 pgs=1007 cs=1 l=0 c=0x3f132e0).fault with nothing to send, going to standby 2014-04-14 18:49:26.765984 7fe8c11e5700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.104:6814/1060356 pipe(0x3f7b480 sd=29 :36609 s=2 pgs=462 cs=1 l=0 c=0x3f102c0).fault with nothing to send, going to standby 2014-04-14 18:49:26.765989 7fe8a6a48700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.106:6829/2002245 pipe(0x6ab0f00 sd=68 :6821 s=2 pgs=2426 cs=1 l=0 c=0x460dc20).fault with nothing to send, going to standby 2014-04-14 18:49:26.766110 7fe8bc69a700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.111:6816/1022206 pipe(0x4da1400 sd=179 :42290 s=2 pgs=406 cs=1 l=0 c=0x693f380).fault with nothing to send, going to standby 2014-04-14 18:49:26.766182 7fe8be2b6700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.109:6812/4053163 pipe(0x3e04600 sd=24 :6821 s=2 pgs=10633 cs=1 l=0 c=0x4e40b00).fault with nothing to send, going to standby 2014-04-14 18:49:26.766192 7fe8b9b6f700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.109:6802/1058316 pipe(0x4107a80 sd=218 :53442 s=2 pgs=460 cs=1 l=0 c=0x406f0c0).fault with nothing to send, going to standby 2014-04-14 18:49:26.766246 7fe8b9569700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.104:6815/3057518 pipe(0x4106e00 sd=165 :33062 s=2 pgs=726 cs=1 l=0 c=0x693a7e0).fault with nothing to send, going to standby 2014-04-14 18:49:26.766394 7fe8bb387700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.109:6817/4055614 pipe(0x4df8000 sd=240 :6821 s=2 pgs=882 cs=1 l=0 c=0x4e2b9c0).fault with nothing to send, going to standby 2014-04-14 18:49:26.766609 7fe8c20f4700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.108:6810/2000970 pipe(0x3e06900 sd=101 :50580 s=2 pgs=631 cs=1 l=0 c=0x3fb9e40).fault with nothing to send, going to standby 2014-04-14 18:49:26.766664 7fe8bf0c4700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.113:6812/2059303 pipe(0x4da5780 sd=147 :39565 s=2 pgs=532 cs=1 l=0 c=0x4e08dc0).fault with nothing to send, going to standby 2014-04-14 18:49:26.766713 7fe8bd1a5700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.109:6803/5061793 pipe(0x4102580 sd=164 :46721 s=2 pgs=915 cs=1 l=0 c=0x693e300).fault with nothing to send, going to standby 2014-04-14 18:49:26.766716 7fe898867700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.105:6821/8000988 pipe(0x4dfda00 sd=196 :6821 s=2 pgs=1243 cs=1 l=0 c=0x3e09a20).fault with nothing to send, going to standby 2014-04-14 18:49:26.766767 7fe8bb084700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.107:6804/14881 pipe(0x4102300 sd=172 :56835 s=2 pgs=371 cs=1 l=0 c=0x693ac00).fault with nothing to send, going to standby 2014-04-14 18:49:26.766825 7fe8b15f3700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.110:6825/4036166 pipe(0x4513480 sd=203 :6821 s=2 pgs=832 cs=1 l=0 c=0x3e0c4c0).fault with nothing to send, going to standby 2014-04-14 18:49:26.767086 7fe8bf4c8700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.104:6812/51547 pipe(0x4da6680 sd=113 :39159 s=2 pgs=423 cs=1 l=0 c=0x45ccd00).fault with nothing to send, going to standby 2014-04-14 18:49:26.767096 7fe8bdeb2700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.114:6802/2007089 pipe(0x4103480 sd=161 :59020 s=2 pgs=557 cs=1 l=0 c=0x693dd80).fault with nothing to send, going to standby 2014-04-14 18:49:26.767147 7fe8c1bef700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.104:6804/2061463 pipe(0x4106400 sd=251 :6821 s=2 pgs=596 cs=1 l=0 c=0x4e46720).fault with nothing to send, going to standby 2014-04-14 18:49:26.767361 7fe8c05d9700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.113:6810/2001582 pipe(0x4105a00 sd=148 :6821 s=2 pgs=614 cs=1 l=0 c=0x3e0f0c0).fault with nothing to send, going to standby 2014-04-14 18:49:26.767741 7fe8bddb1700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.109:6825/2059246 pipe(0x4100f00 sd=167 :41990 s=2 pgs=597 cs=1 l=0 c=0x693e040).fault with nothing to send, going to standby 2014-04-14 18:49:26.767849 7fe89ffde700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.110:6805/3038601 pipe(0x4dfc380 sd=245 :6821 s=2 pgs=948 cs=1 l=0 c=0x4e46ca0).fault with nothing to send, going to standby 2014-04-14 18:49:26.767912 7fe89e4c3700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.114:6805/1003429 pipe(0x4106680 sd=327 :6821 s=2 pgs=354 cs=1 l=0 c=0x4e2e300).fault with nothing to send, going to standby 2014-04-14 18:49:26.768000 7fe89fcdb700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.111:6829/2017859 pipe(0x4da5f00 sd=309 :6821 s=2 pgs=535 cs=1 l=0 c=0x4e2e5c0).fault with nothing to send, going to standby 2014-04-14 18:49:26.768207 7fe8c09dd700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.104:6808/48025 pipe(0x3f7ee00 sd=31 :41331 s=2 pgs=403 cs=1 l=0 c=0x3f170c0).fault with nothing to send, going to standby 2014-04-14 18:49:26.768259 7fe8bcfa3700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.113:6808/1060374 pipe(0x4da4d80 sd=175 :58755 s=2 pgs=420 cs=1 l=0 c=0x693d540).fault with nothing to send, going to standby 2014-04-14 18:49:26.768310 7fe8be4b8700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.109:6811/51869 pipe(0x4da4880 sd=90 :46873 s=2 pgs=332 cs=1 l=0 c=0x3f11080).fault with nothing to send, going to standby 2014-04-14 18:49:26.768569 7fe8c1df1700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.101:6800/1026435 pipe(0x6ab4380 sd=46 :6821 s=2 pgs=596 cs=1 l=0 c=0x3fbe720).fault with nothing to send, going to standby 2014-04-14 18:49:26.768584 7fe8a6f4d700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.106:6826/6092 pipe(0x3f79b80 sd=38 :55297 s=2 pgs=384 cs=1 l=0 c=0x4e26e00).fault with nothing to send, going to standby 2014-04-14 18:49:26.768629 7fe8a6b49700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.102:6802/6271 pipe(0x4da0280 sd=106 :41261 s=2 pgs=413 cs=1 l=0 c=0x4e59b80).fault with nothing to send, going to standby 2014-04-14 18:49:26.768721 7fe899d7c700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.103:6800/2034753 pipe(0x3f7b980 sd=40 :39840 s=2 pgs=613 cs=1 l=0 c=0x4e5fa60).fault with nothing to send, going to standby 2014-04-14 18:49:26.768976 7fe89f2d1700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.103:6817/5030905 pipe(0x4da7300 sd=306 :6821 s=2 pgs=994 cs=1 l=0 c=0x45b2100).fault with nothing to send, going to standby 2014-04-14 18:49:26.769071 7fe8bcb9f700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.108:6817/2007814 pipe(0x3e01680 sd=102 :51322 s=2 pgs=607 cs=1 l=0 c=0x455b9c0).fault with nothing to send, going to standby 2014-04-14 18:49:26.769233 7fe89a07f700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.102:6820/12949 pipe(0x4da1180 sd=110 :43907 s=2 pgs=516 cs=1 l=0 c=0x688bb20).fault with nothing to send, going to standby 2014-04-14 18:49:26.769755 7fe8c26fa700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.111:6808/2009889 pipe(0x56ff080 sd=30 :55441 s=2 pgs=494 cs=1 l=0 c=0x406fbc0).fault with nothing to send, going to standby 2014-04-14 18:49:26.769818 7fe8aeac8700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.111:6809/2020353 pipe(0x3e05f00 sd=467 :6821 s=2 pgs=551 cs=1 l=0 c=0x4e21fa0).fault with nothing to send, going to standby 2014-04-14 18:49:26.770385 7fe8c21f5700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.104:6819/2056631 pipe(0x3f7c600 sd=233 :6821 s=2 pgs=614 cs=1 l=0 c=0x4e23020).fault with nothing to send, going to standby 2014-04-14 18:49:26.770510 7fe8a1dfc700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.107:6802/3003023 pipe(0x3e06180 sd=71 :6821 s=2 pgs=654 cs=1 l=0 c=0x3f11340).fault with nothing to send, going to standby 2014-04-14 18:49:26.770857 7fe8bd9ad700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.115:6805/1021892 pipe(0x3e02a80 sd=168 :6821 s=2 pgs=467 cs=1 l=0 c=0x4e46880).fault with nothing to send, going to standby 2014-04-14 18:49:26.770930 7fe8bcca0700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.110:6806/3032536 pipe(0x3e03200 sd=213 :6821 s=2 pgs=720 cs=1 l=0 c=0x4e21e40).fault with nothing to send, going to standby 2014-04-14 18:49:26.771083 7fe8ae3c1700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.108:6807/4002228 pipe(0x6ab4b00 sd=166 :6821 s=2 pgs=780 cs=1 l=0 c=0x3df65c0).fault with nothing to send, going to standby 2014-04-14 18:49:26.771269 7fe8bb98d700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.106:6806/1009649 pipe(0x4515280 sd=197 :45527 s=2 pgs=518 cs=1 l=0 c=0x693dc20).fault with nothing to send, going to standby 2014-04-14 18:49:26.771480 7fe8a01e0700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.109:6804/3056966 pipe(0x4df8280 sd=249 :6821 s=2 pgs=662 cs=1 l=0 c=0x4e42100).fault with nothing to send, going to standby 2014-04-14 18:49:26.771649 7fe8b2705700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.108:6801/4011474 pipe(0x6ab3480 sd=122 :6821 s=2 pgs=783 cs=1 l=0 c=0x460b5a0).fault with nothing to send, going to standby 2014-04-14 18:49:26.772206 7fe8c2e01700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.108:6808/1012898 pipe(0x3e00280 sd=79 :50753 s=2 pgs=493 cs=1 l=0 c=0x455b180).fault with nothing to send, going to standby 2014-04-14 18:49:26.772497 7fe89dab9700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.115:6809/1011533 pipe(0x3e03c00 sd=51 :58020 s=2 pgs=394 cs=1 l=0 c=0x691f0c0).fault with nothing to send, going to standby 2014-04-14 18:49:26.772747 7fe8bb185700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.111:6825/2012002 pipe(0x3e02300 sd=193 :6821 s=2 pgs=520 cs=1 l=0 c=0x3df4a40).fault with nothing to send, going to standby 2014-04-14 18:49:26.773546 7fe89a786700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.111:6800/4014159 pipe(0x4513700 sd=32 :6821 s=2 pgs=759 cs=1 l=0 c=0x4e256a0).fault with nothing to send, going to standby 2014-04-14 18:49:26.773785 7fe8b1cfa700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.101:6816/4019830 pipe(0x3e01900 sd=267 :6821 s=2 pgs=820 cs=1 l=0 c=0x3f177a0).fault with nothing to send, going to standby 2014-04-14 18:49:26.773874 7fe89d3b2700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.114:6822/1000532 pipe(0x4103c00 sd=486 :6821 s=2 pgs=2158 cs=1 l=0 c=0x4dae720).fault with nothing to send, going to standby 2014-04-14 18:49:26.773938 7fe89caa9700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.110:6812/5034877 pipe(0x51c2080 sd=82 :6821 s=2 pgs=855 cs=1 l=0 c=0x46082c0).fault with nothing to send, going to standby 2014-04-14 18:49:26.774154 7fe8a9f7d700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.103:6810/3036094 pipe(0x4dfd280 sd=274 :6821 s=2 pgs=791 cs=1 l=0 c=0x3e0aec0).fault with nothing to send, going to standby 2014-04-14 18:49:26.774164 7fe8b7653700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.104:6821/4054057 pipe(0x4dfbe80 sd=264 :6821 s=2 pgs=860 cs=1 l=0 c=0x3e08580).fault with nothing to send, going to standby 2014-04-14 18:49:26.774233 7fe8b05e3700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.104:6807/2059338 pipe(0x3f78c80 sd=138 :6821 s=2 pgs=738 cs=1 l=0 c=0x3e08b00).fault with nothing to send, going to standby 2014-04-14 18:49:26.774448 7fe8b6e4b700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.103:6815/2039519 pipe(0x4100280 sd=236 :6821 s=2 pgs=676 cs=1 l=0 c=0x3e09340).fault with nothing to send, going to standby 2014-04-14 18:49:26.774499 7fe8bb589700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.115:6802/6008752 pipe(0x4107d00 sd=293 :6821 s=2 pgs=918 cs=1 l=0 c=0x3e0a680).fault with nothing to send, going to standby 2014-04-14 18:49:26.775163 7fe8ae1bf700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.103:6821/31885 pipe(0x3f7a080 sd=36 :52574 s=2 pgs=411 cs=1 l=0 c=0x3f139c0).fault with nothing to send, going to standby 2014-04-14 18:49:26.775260 7fe8ba074700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.114:6807/12470 pipe(0x4105500 sd=216 :41255 s=2 pgs=251 cs=1 l=0 c=0x6938420).fault with nothing to send, going to standby 2014-04-14 18:49:26.775417 7fe8b7c59700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.101:6821/4022662 pipe(0x4100000 sd=222 :34937 s=2 pgs=797 cs=1 l=0 c=0x4e2ee00).fault with nothing to send, going to standby 2014-04-14 18:49:26.776471 7fe8c27fb700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.115:6832/2015402 pipe(0x4da2300 sd=130 :51852 s=2 pgs=2249 cs=1 l=0 c=0x4e565c0).fault with nothing to send, going to standby 2014-04-14 18:49:26.776510 7fe8aa381700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.110:6818/1042305 pipe(0x3e00f00 sd=86 :32802 s=2 pgs=385 cs=1 l=0 c=0x455dee0).fault with nothing to send, going to standby 2014-04-14 18:49:26.777072 7fe8ae2c0700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.113:6802/3006545 pipe(0x6ab7800 sd=170 :6821 s=2 pgs=644 cs=1 l=0 c=0x4dacd00).fault with nothing to send, going to standby 2014-04-14 18:49:26.777399 7fe8c0ee2700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.113:6806/3063888 pipe(0x4da2580 sd=141 :38315 s=2 pgs=641 cs=1 l=0 c=0x455fa60).fault with nothing to send, going to standby 2014-04-14 18:49:26.777721 7fe8a04e3700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.114:6828/3009870 pipe(0x4dfc600 sd=239 :6821 s=2 pgs=672 cs=1 l=0 c=0x4e548e0).fault with nothing to send, going to standby 2014-04-14 18:49:26.777789 7fe8c0fe3700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.109:6809/3050592 pipe(0x3e05500 sd=107 :33458 s=2 pgs=633 cs=1 l=0 c=0x693d120).fault with nothing to send, going to standby 2014-04-14 18:49:26.778745 7fe8a5e3c700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.107:6800/2004265 pipe(0x4da5000 sd=124 :58863 s=2 pgs=573 cs=1 l=0 c=0x6ab98c0).fault with nothing to send, going to standby 2014-04-14 18:49:26.780184 7fe8bc599700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.109:6807/2063104 pipe(0x6ab0c80 sd=117 :6821 s=2 pgs=632 cs=1 l=0 c=0x4608580).fault with nothing to send, going to standby 2014-04-14 18:49:26.780294 7fe8bc498700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.107:6815/1010676 pipe(0x4da7580 sd=173 :49411 s=2 pgs=462 cs=1 l=0 c=0x693dac0).fault with nothing to send, going to standby 2014-04-14 18:49:26.781906 7fe8bf1c5700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.102:6809/4018133 pipe(0x4da3c00 sd=153 :53298 s=2 pgs=761 cs=1 l=0 c=0x3f169e0).fault with nothing to send, going to standby 2014-04-14 18:49:26.782983 7fe8b14f2700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.102:6814/4016928 pipe(0x4103700 sd=52 :6821 s=2 pgs=805 cs=1 l=0 c=0x3e0f640).fault with nothing to send, going to standby 2014-04-14 18:49:26.784876 7fe8b7f5c700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.103:6824/37247 pipe(0x4101180 sd=223 :59662 s=2 pgs=378 cs=1 l=0 c=0x4e2e1a0).fault with nothing to send, going to standby 2014-04-14 18:49:26.788091 7fe8bba8e700 0 -- 192.168.1.112:6821/2032817 >> 192.168.1.113:6800/4061795 pipe(0x51c3480 sd=169 :6821 s=2 pgs=693 cs=1 l=0 c=0x460cd00).fault with nothing to send, going to standby 2014-04-14 18:49:27.156407 7fe8d201b700 0 log [WRN] : map e17971 wrongly marked me down 2014-04-14 18:50:24.313612 7fe8b704d700 0 -- 192.168.1.112:6805/3032817 >> 192.168.1.114:6828/3009870 pipe(0x56fbe80 sd=218 :32785 s=2 pgs=730 cs=1 l=0 c=0x4e29e40).fault with nothing to send, going to standby 2014-04-14 18:50:44.167132 7fe8b4522700 0 -- 192.168.1.112:6805/3032817 >> 192.168.1.111:6815/3009103 pipe(0x4104380 sd=249 :34287 s=2 pgs=702 cs=1 l=0 c=0x4e5dd80).fault with nothing to send, going to standby 2014-04-14 18:50:46.189621 7fe8b08e6700 0 -- 192.168.1.112:6805/3032817 >> 192.168.1.105:6833/9062372 pipe(0x4103c00 sd=285 :0 s=1 pgs=0 cs=0 l=0 c=0x3e0ee00).fault 2014-04-14 18:51:49.643502 7fe8b522f700 0 -- 192.168.1.112:6805/3032817 >> 192.168.1.105:6833/9062372 pipe(0x8c24d80 sd=159 :6805 s=0 pgs=0 cs=0 l=0 c=0x8d3b180).accept connect_seq 0 vs existing 0 state wait 2014-04-14 18:51:51.611604 7fe8dde34700 -1 osd.163 18096 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:31.611601) 2014-04-14 18:51:52.501655 7fe8c6c09700 -1 osd.163 18097 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:32.501651) 2014-04-14 18:51:52.611763 7fe8dde34700 -1 osd.163 18097 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:32.611759) 2014-04-14 18:51:53.612074 7fe8dde34700 -1 osd.163 18098 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:33.612071) 2014-04-14 18:51:54.612282 7fe8dde34700 -1 osd.163 18099 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:34.612276) 2014-04-14 18:51:54.808065 7fe8c6c09700 -1 osd.163 18099 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:34.808062) 2014-04-14 18:51:55.612496 7fe8dde34700 -1 osd.163 18100 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:35.612494) 2014-04-14 18:51:56.612657 7fe8dde34700 -1 osd.163 18101 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:36.612650) 2014-04-14 18:51:57.613004 7fe8dde34700 -1 osd.163 18102 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:37.613001) 2014-04-14 18:51:58.613230 7fe8dde34700 -1 osd.163 18103 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:38.613224) 2014-04-14 18:51:59.613397 7fe8dde34700 -1 osd.163 18104 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:39.613391) 2014-04-14 18:52:00.117250 7fe8c6c09700 -1 osd.163 18104 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:40.117247) 2014-04-14 18:52:00.613558 7fe8dde34700 -1 osd.163 18105 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:40.613552) 2014-04-14 18:52:01.613696 7fe8dde34700 -1 osd.163 18106 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:41.613690) 2014-04-14 18:52:02.613824 7fe8dde34700 -1 osd.163 18107 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:42.613817) 2014-04-14 18:52:03.614241 7fe8dde34700 -1 osd.163 18108 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:43.614238) 2014-04-14 18:52:04.614427 7fe8dde34700 -1 osd.163 18109 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:44.614421) 2014-04-14 18:52:05.614585 7fe8dde34700 -1 osd.163 18110 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:45.614579) 2014-04-14 18:52:06.019678 7fe8c6c09700 -1 osd.163 18110 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:46.019675) 2014-04-14 18:52:06.614768 7fe8dde34700 -1 osd.163 18111 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:46.614762) 2014-04-14 18:52:07.121488 7fe8c6c09700 -1 osd.163 18111 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:47.121239) 2014-04-14 18:52:07.614932 7fe8dde34700 -1 osd.163 18112 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:47.614925) 2014-04-14 18:52:08.615090 7fe8dde34700 -1 osd.163 18113 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:48.615084) 2014-04-14 18:52:09.615448 7fe8dde34700 -1 osd.163 18114 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:49.615441) 2014-04-14 18:52:10.615609 7fe8dde34700 -1 osd.163 18115 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:50.615603) 2014-04-14 18:52:11.615901 7fe8dde34700 -1 osd.163 18116 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:51.615899) 2014-04-14 18:52:12.616056 7fe8dde34700 -1 osd.163 18117 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:52.616052) 2014-04-14 18:52:13.024155 7fe8c6c09700 -1 osd.163 18117 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:53.024151) 2014-04-14 18:52:13.616246 7fe8dde34700 -1 osd.163 18118 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:53.616240) 2014-04-14 18:52:14.616563 7fe8dde34700 -1 osd.163 18119 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:54.616559) 2014-04-14 18:52:15.616737 7fe8dde34700 -1 osd.163 18120 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:55.616731) 2014-04-14 18:52:16.526681 7fe8c6c09700 -1 osd.163 18120 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:56.526678) 2014-04-14 18:52:16.616888 7fe8dde34700 -1 osd.163 18120 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:56.616884) 2014-04-14 18:52:17.617048 7fe8dde34700 -1 osd.163 18121 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:57.617043) 2014-04-14 18:52:18.617217 7fe8dde34700 -1 osd.163 18122 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:58.617211) 2014-04-14 18:52:19.429228 7fe8c6c09700 -1 osd.163 18122 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:59.429224) 2014-04-14 18:52:19.617398 7fe8dde34700 -1 osd.163 18123 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:59.617392) 2014-04-14 18:52:20.617751 7fe8dde34700 -1 osd.163 18123 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:52:00.617745) 2014-04-14 18:52:21.617985 7fe8dde34700 -1 osd.163 18124 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:52:01.617983) 2014-04-14 18:52:21.731688 7fe8c6c09700 -1 osd.163 18125 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:52:01.731683) 2014-04-14 18:52:22.618289 7fe8dde34700 -1 osd.163 18125 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:52:02.618285) 2014-04-14 18:52:23.618432 7fe8dde34700 -1 osd.163 18126 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:52:03.618425) 2014-04-14 18:52:24.035341 7fe8c6c09700 -1 osd.163 18127 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:52:04.035339) 2014-04-14 18:52:24.618602 7fe8dde34700 -1 osd.163 18127 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:52:04.618596) 2014-04-14 18:52:25.618765 7fe8dde34700 -1 osd.163 18128 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:52:05.618759) 2014-04-14 18:52:25.741377 7fe8c6c09700 -1 osd.163 18128 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:52:05.741373) 2014-04-14 18:52:26.618917 7fe8dde34700 -1 osd.163 18129 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:52:06.618911) 2014-04-14 18:52:26.847912 7fe8c6c09700 -1 osd.163 18129 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:52:06.847908) 2014-04-14 18:52:27.354395 7fe8c6c09700 -1 osd.163 18129 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:52:07.354391) 2014-04-14 18:52:27.619116 7fe8dde34700 -1 osd.163 18129 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:52:07.619112) 2014-04-14 18:52:28.460652 7fe8c6c09700 -1 osd.163 18129 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:52:08.460644) 2014-04-14 18:52:28.619248 7fe8dde34700 -1 osd.163 18129 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:52:08.619244) 2014-04-14 18:52:29.619605 7fe8dde34700 -1 osd.163 18130 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:52:09.619603) 2014-04-14 18:52:30.619740 7fe8dde34700 -1 osd.163 18131 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:52:10.619734) 2014-04-14 18:52:30.767241 7fe8c6c09700 -1 osd.163 18131 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:52:10.767238) 2014-04-14 18:52:31.619909 7fe8dde34700 -1 osd.163 18131 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:52:11.619903) 2014-04-14 18:52:32.469702 7fe8c6c09700 -1 osd.163 18132 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:52:12.469698) 2014-04-14 18:52:32.620306 7fe8dde34700 -1 osd.163 18132 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:52:12.620303) 2014-04-14 18:52:33.620481 7fe8dde34700 -1 osd.163 18133 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:52:13.620474) 2014-04-14 18:52:34.172145 7fe8c6c09700 -1 osd.163 18134 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:52:14.172141) 2014-04-14 18:52:34.346377 7fe89fcdb700 0 -- 192.168.1.112:0/32817 >> 192.168.1.105:6831/10060022 pipe(0x4104380 sd=450 :0 s=1 pgs=0 cs=0 l=1 c=0x691a3c0).fault 2014-04-14 18:52:44.605343 7fe8a0ceb700 0 -- 192.168.1.112:6805/3032817 >> 192.168.1.105:6834/11000988 pipe(0x4514b00 sd=145 :0 s=1 pgs=0 cs=0 l=0 c=0x46091e0).fault 2014-04-14 18:53:12.627971 7fe8dde34700 -1 osd.163 18168 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:52:52.627965) 2014-04-14 18:53:13.103357 7fe8c6c09700 -1 osd.163 18169 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:52:53.103353) 2014-04-14 18:53:13.628154 7fe8dde34700 -1 osd.163 18169 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:52:53.628147) 2014-04-14 18:53:14.628330 7fe8dde34700 -1 osd.163 18169 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:52:54.628325) 2014-04-14 18:53:15.628470 7fe8dde34700 -1 osd.163 18170 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:52:55.628464) 2014-04-14 18:53:16.628937 7fe8dde34700 -1 osd.163 18171 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:52:56.628933) 2014-04-14 18:53:17.629365 7fe8dde34700 -1 osd.163 18172 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:52:57.629363) 2014-04-14 18:53:17.805755 7fe8c6c09700 -1 osd.163 18172 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:52:57.805751) 2014-04-14 18:53:18.629514 7fe8dde34700 -1 osd.163 18173 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:52:58.629508) 2014-04-14 18:53:19.629867 7fe8dde34700 -1 osd.163 18174 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:52:59.629864) 2014-04-14 18:53:20.630215 7fe8dde34700 -1 osd.163 18175 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:00.630211) 2014-04-14 18:53:21.630378 7fe8dde34700 -1 osd.163 18176 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:01.630371) 2014-04-14 18:53:22.630742 7fe8dde34700 -1 osd.163 18177 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:02.630739) 2014-04-14 18:53:23.108354 7fe8c6c09700 -1 osd.163 18177 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:03.108350) 2014-04-14 18:53:23.610856 7fe8c6c09700 -1 osd.163 18178 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:03.610852) 2014-04-14 18:53:23.630897 7fe8dde34700 -1 osd.163 18178 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:03.630894) 2014-04-14 18:53:24.631055 7fe8dde34700 -1 osd.163 18178 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:04.631049) 2014-04-14 18:53:25.313754 7fe8c6c09700 -1 osd.163 18178 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:05.313750) 2014-04-14 18:53:25.631223 7fe8dde34700 -1 osd.163 18178 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:05.631219) 2014-04-14 18:53:26.631373 7fe8dde34700 -1 osd.163 18178 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:06.631367) 2014-04-14 18:53:27.631607 7fe8dde34700 -1 osd.163 18179 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:07.631605) 2014-04-14 18:53:28.631818 7fe8dde34700 -1 osd.163 18180 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:08.631813) 2014-04-14 18:53:29.632280 7fe8dde34700 -1 osd.163 18181 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:09.632277) 2014-04-14 18:53:30.016239 7fe8c6c09700 -1 osd.163 18181 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:10.016176) 2014-04-14 18:53:30.632650 7fe8dde34700 -1 osd.163 18182 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:10.632646) 2014-04-14 18:53:31.632798 7fe8dde34700 -1 osd.163 18183 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:11.632791) 2014-04-14 18:53:32.632999 7fe8dde34700 -1 osd.163 18183 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:12.632992) 2014-04-14 18:53:33.633121 7fe8dde34700 -1 osd.163 18183 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:13.633114) 2014-04-14 18:53:34.633310 7fe8dde34700 -1 osd.163 18183 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:14.633303) 2014-04-14 18:53:35.318740 7fe8c6c09700 -1 osd.163 18183 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:15.318736) 2014-04-14 18:53:35.633462 7fe8dde34700 -1 osd.163 18183 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:15.633457) 2014-04-14 18:53:36.633849 7fe8dde34700 -1 osd.163 18184 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:16.633846) 2014-04-14 18:53:37.634082 7fe8dde34700 -1 osd.163 18185 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:17.634079) 2014-04-14 18:53:38.634372 7fe8dde34700 -1 osd.163 18186 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:18.634369) 2014-04-14 18:53:38.821204 7fe8c6c09700 -1 osd.163 18186 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:18.821176) 2014-04-14 18:53:39.323679 7fe8c6c09700 -1 osd.163 18187 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:19.323675) 2014-04-14 18:53:39.634749 7fe8dde34700 -1 osd.163 18187 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:19.634746) 2014-04-14 18:53:40.426168 7fe8c6c09700 -1 osd.163 18188 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:20.426165) 2014-04-14 18:53:40.634945 7fe8dde34700 -1 osd.163 18188 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:20.634941) 2014-04-14 18:53:40.928710 7fe8c6c09700 -1 osd.163 18188 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:20.928646) 2014-04-14 18:53:41.635148 7fe8dde34700 -1 osd.163 18188 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:21.635141) 2014-04-14 18:53:41.988075 7fe8a0bea700 0 -- 192.168.1.112:6805/3032817 >> 192.168.1.113:6834/5063888 pipe(0x6ab1b80 sd=219 :6805 s=2 pgs=923 cs=1 l=0 c=0x3f14d00).fault with nothing to send, going to standby 2014-04-14 18:53:42.635422 7fe8dde34700 -1 osd.163 18189 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:22.635418) 2014-04-14 18:53:43.635742 7fe8dde34700 -1 osd.163 18190 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:23.635739) 2014-04-14 18:53:43.831089 7fe8c6c09700 -1 osd.163 18190 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:23.831086) 2014-04-14 18:53:44.333505 7fe8c6c09700 -1 osd.163 18191 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:24.333501) 2014-04-14 18:53:44.635925 7fe8dde34700 -1 osd.163 18191 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:24.635920) 2014-04-14 18:53:45.636119 7fe8dde34700 -1 osd.163 18192 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:25.636113) 2014-04-14 18:53:46.036037 7fe8c6c09700 -1 osd.163 18192 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:26.036034) 2014-04-14 18:53:46.636275 7fe8dde34700 -1 osd.163 18193 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:26.636269) 2014-04-14 18:53:47.636592 7fe8dde34700 -1 osd.163 18194 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:27.636589) 2014-04-14 18:53:48.636716 7fe8dde34700 -1 osd.163 18195 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:28.636712) 2014-04-14 18:53:49.636918 7fe8dde34700 -1 osd.163 18196 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:29.636915) 2014-04-14 18:53:50.637123 7fe8dde34700 -1 osd.163 18197 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:30.637117) 2014-04-14 18:53:51.340347 7fe8c6c09700 -1 osd.163 18197 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:31.338605) 2014-04-14 18:53:51.637305 7fe8dde34700 -1 osd.163 18197 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:31.637300) 2014-04-14 18:53:52.637462 7fe8dde34700 -1 osd.163 18198 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:32.637456) 2014-04-14 18:53:53.637780 7fe8dde34700 -1 osd.163 18199 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:33.637777) 2014-04-14 18:53:54.624337 7fe8aeecc700 0 -- 192.168.1.112:0/32817 >> 192.168.1.105:6820/11060022 pipe(0x51c6400 sd=295 :0 s=1 pgs=0 cs=0 l=1 c=0x4e52d60).fault 2014-04-14 18:54:36.710730 7fe89cdac700 0 -- 192.168.1.112:6805/3032817 >> 192.168.1.109:6816/4056966 pipe(0x6ab4100 sd=240 :6805 s=2 pgs=939 cs=1 l=0 c=0x45b5d80).fault with nothing to send, going to standby 2014-04-14 18:54:48.825027 7fe8a01e0700 0 -- 192.168.1.112:6805/3032817 >> 192.168.1.105:6811/13002701 pipe(0x4dfd000 sd=32 :6805 s=2 pgs=2082 cs=1 l=0 c=0x45b11e0).fault with nothing to send, going to standby 2014-04-14 18:55:31.909562 7fe8b4d2a700 0 -- 192.168.1.112:6805/3032817 >> 192.168.1.106:6824/4004240 pipe(0x56ff800 sd=390 :6805 s=2 pgs=974 cs=1 l=0 c=0x4e20c60).fault with nothing to send, going to standby 2014-04-14 18:55:31.912494 7fe8c0ee2700 0 -- 192.168.1.112:6805/3032817 >> 192.168.1.110:6818/1042305 pipe(0x6ab5500 sd=107 :35463 s=2 pgs=454 cs=1 l=0 c=0x406f7a0).fault with nothing to send, going to standby 2014-04-14 18:55:42.055199 7fe8a5d3b700 0 -- 192.168.1.112:6805/3032817 >> 192.168.1.105:6820/13062372 pipe(0x4da6400 sd=158 :6805 s=2 pgs=2180 cs=1 l=0 c=0x455b700).fault with nothing to send, going to standby 2014-04-14 18:55:42.055903 7fe89d3b2700 0 -- 192.168.1.112:6805/3032817 >> 192.168.1.109:6807/5056966 pipe(0x56fa800 sd=311 :6805 s=2 pgs=1062 cs=1 l=0 c=0x694eca0).fault with nothing to send, going to standby 2014-04-14 18:55:42.056550 7fe8ad1af700 0 -- 192.168.1.112:6805/3032817 >> 192.168.1.103:6806/1031885 pipe(0x3f7be80 sd=257 :6805 s=2 pgs=543 cs=1 l=0 c=0x3f11600).fault with nothing to send, going to standby 2014-04-14 18:55:42.056722 7fe8d201b700 0 log [WRN] : map e18290 wrongly marked me down 2014-04-14 18:55:42.056798 7fe8b8663700 0 -- 192.168.1.112:6805/3032817 >> 192.168.1.101:6825/6022662 pipe(0x4106b80 sd=95 :6805 s=2 pgs=1300 cs=1 l=0 c=0x694ac00).fault with nothing to send, going to standby 2014-04-14 18:55:42.056915 7fe89d0af700 0 -- 192.168.1.112:6805/3032817 >> 192.168.1.101:6810/6029366 pipe(0x4da7a80 sd=364 :6805 s=2 pgs=1229 cs=1 l=0 c=0x455aec0).fault with nothing to send, going to standby 2014-04-14 18:55:42.057300 7fe89c7a6700 0 -- 192.168.1.112:6805/3032817 >> 192.168.1.106:6808/4011682 pipe(0x6ab3c00 sd=409 :6805 s=2 pgs=1020 cs=1 l=0 c=0x45b4780).fault with nothing to send, going to standby 2014-04-14 18:55:42.058007 7fe89f8d7700 0 -- 192.168.1.112:6805/3032817 >> 192.168.1.108:6820/5002228 pipe(0x51c2d00 sd=360 :6805 s=2 pgs=987 cs=1 l=0 c=0x8360dc0).fault with nothing to send, going to standby 2014-04-14 18:55:42.057436 7fe8b3311700 0 -- 192.168.1.112:6805/3032817 >> 192.168.1.113:6812/3001582 pipe(0x3f7b700 sd=124 :6805 s=2 pgs=762 cs=1 l=0 c=0x4daaec0).fault with nothing to send, going to standby 2014-04-14 18:55:42.058040 7fe8a9371700 0 -- 192.168.1.112:6805/3032817 >> 192.168.1.105:6817/14002701 pipe(0x4dfdf00 sd=164 :6805 s=2 pgs=2223 cs=1 l=0 c=0x4e211e0).fault with nothing to send, going to standby 2014-04-14 18:55:42.058067 7fe8bbc90700 0 -- 192.168.1.112:6805/3032817 >> 192.168.1.109:6804/7055614 pipe(0x4da0a00 sd=273 :6805 s=2 pgs=1422 cs=1 l=0 c=0x4559e40).fault with nothing to send, going to standby 2014-04-14 18:55:42.058281 7fe89eecd700 0 -- 192.168.1.112:6805/3032817 >> 192.168.1.103:6822/5033047 pipe(0x56fa080 sd=22 :6805 s=2 pgs=932 cs=1 l=0 c=0x45f8dc0).fault with nothing to send, going to standby 2014-04-14 18:55:42.058942 7fe8a8f6d700 0 -- 192.168.1.112:6805/3032817 >> 192.168.1.102:6804/4009966 pipe(0x4df9b80 sd=346 :6805 s=2 pgs=935 cs=1 l=0 c=0x406ce60).fault with nothing to send, going to standby 2014-04-14 18:55:42.059094 7fe8be2b6700 0 -- 192.168.1.112:6805/3032817 >> 192.168.1.103:6815/2039519 pipe(0x6ab7300 sd=141 :57968 s=2 pgs=699 cs=1 l=0 c=0x6919b80).fault with nothing to send, going to standby 2014-04-14 18:56:03.878064 7fe8c0bdf700 0 -- 192.168.1.112:6801/4032817 >> 192.168.1.108:6806/3000970 pipe(0x4514100 sd=123 :60956 s=2 pgs=1010 cs=1 l=0 c=0x406dd80).fault with nothing to send, going to standby 2014-04-14 18:56:16.289779 7fe8c16ea700 0 -- 192.168.1.112:6801/4032817 >> 192.168.1.111:6831/4012002 pipe(0x56fd500 sd=110 :41131 s=2 pgs=1087 cs=1 l=0 c=0x45fd120).fault with nothing to send, going to standby 2014-04-14 18:56:32.789640 7fe8dea7b700 0 -- 192.168.1.112:6801/4032817 >> 192.168.1.105:6803/10003300 pipe(0x56fb700 sd=92 :35927 s=2 pgs=1801 cs=1 l=0 c=0x45ff220).fault with nothing to send, going to standby
LOGS FROM osd.51
[root@storage0105-ib ceph]# tail -100f ceph-osd.51.log 2014-04-14 18:58:42.107780 7f5ceb81e700 0 -- 192.168.1.105:6821/14060022 >> 192.168.1.106:6800/5004240 pipe(0x7719e00 sd=215 :37938 s=2 pgs=1179 cs=1 l=0 c=0x4c11fa0).fault with nothing to send, going to standby 2014-04-14 18:58:42.108366 7f5ceec4b700 0 -- 192.168.1.105:6821/14060022 >> 192.168.1.113:6826/7063888 pipe(0x4d12a80 sd=180 :47643 s=2 pgs=1385 cs=1 l=0 c=0x4b561a0).fault with nothing to send, going to standby 2014-04-14 18:58:42.108411 7f5cf0867700 0 -- 192.168.1.105:6821/14060022 >> 192.168.1.113:6802/5064877 pipe(0x4d11180 sd=163 :59031 s=2 pgs=1199 cs=1 l=0 c=0x5979fa0).fault with nothing to send, going to standby 2014-04-14 18:58:42.109075 7f5ce8f1f700 0 -- 192.168.1.105:6821/14060022 >> 192.168.1.106:6816/3002245 pipe(0x537e180 sd=249 :6821 s=2 pgs=2922 cs=1 l=0 c=0xa3dc360).fault with nothing to send, going to standby 2014-04-14 18:58:42.109948 7f5cf3291700 0 -- 192.168.1.105:6821/14060022 >> 192.168.1.107:6811/1009536 pipe(0x7bbdf00 sd=123 :56678 s=2 pgs=1846 cs=1 l=0 c=0x4fab2e0).fault with nothing to send, going to standby 2014-04-14 18:58:42.109997 7f5cf4aaa700 0 -- 192.168.1.105:6821/14060022 >> 192.168.1.104:6814/5049263 pipe(0x4e97580 sd=103 :36440 s=2 pgs=1128 cs=1 l=0 c=0x4fa8dc0).fault with nothing to send, going to standby 2014-04-14 18:58:42.110075 7f5cd40a8700 0 -- 192.168.1.105:6821/14060022 >> 192.168.1.114:6815/6007089 pipe(0x429c380 sd=55 :6821 s=2 pgs=1289 cs=1 l=0 c=0x597c780).fault with nothing to send, going to standby 2014-04-14 18:58:42.110765 7f5ce9424700 0 -- 192.168.1.105:6821/14060022 >> 192.168.1.112:6816/5026017 pipe(0x4d12080 sd=247 :6821 s=2 pgs=1198 cs=1 l=0 c=0x4faa100).fault with nothing to send, going to standby 2014-04-14 18:58:42.110797 7f5ceab11700 0 -- 192.168.1.105:6821/14060022 >> 192.168.1.108:6817/2007814 pipe(0x429d780 sd=232 :44225 s=2 pgs=928 cs=1 l=0 c=0x54a89a0).fault with nothing to send, going to standby 2014-04-14 18:58:42.111262 7f5cf3595700 0 -- 192.168.1.105:6821/14060022 >> 192.168.1.114:6802/6008433 pipe(0x7bbdc80 sd=122 :38880 s=2 pgs=1212 cs=1 l=0 c=0x4fae5c0).fault with nothing to send, going to standby 2014-04-14 18:58:42.111351 7f5cee847700 0 -- 192.168.1.105:6821/14060022 >> 192.168.1.111:6803/4009103 pipe(0x4d13480 sd=184 :43928 s=2 pgs=1011 cs=1 l=0 c=0x4b544c0).fault with nothing to send, going to standby 2014-04-14 18:58:42.113308 7f5cf58b7700 0 -- 192.168.1.105:6821/14060022 >> 192.168.1.102:6824/15636 pipe(0xaeec380 sd=87 :46747 s=2 pgs=694 cs=1 l=0 c=0x77bc620).fault with nothing to send, going to standby 2014-04-14 18:58:42.113372 7f5cf1675700 0 -- 192.168.1.105:6821/14060022 >> 192.168.1.115:6814/8009855 pipe(0x7bbd280 sd=152 :46908 s=2 pgs=1478 cs=1 l=0 c=0x6475540).fault with nothing to send, going to standby 2014-04-14 18:58:42.113410 7f5cf48a8700 0 -- 192.168.1.105:6821/14060022 >> 192.168.1.103:6800/2034753 pipe(0x4e95f00 sd=108 :52707 s=2 pgs=928 cs=1 l=0 c=0x4c170c0).fault with nothing to send, going to standby 2014-04-14 18:58:42.113775 7f5cef958700 0 -- 192.168.1.105:6821/14060022 >> 192.168.1.108:6802/3005178 pipe(0x771bc00 sd=173 :49030 s=2 pgs=967 cs=1 l=0 c=0xa3dca40).fault with nothing to send, going to standby 2014-04-14 18:58:42.114732 7f5cedd3c700 0 -- 192.168.1.105:6821/14060022 >> 192.168.1.112:6832/8034975 pipe(0x771f080 sd=314 :6821 s=2 pgs=1382 cs=1 l=0 c=0xb2cf640).fault with nothing to send, going to standby 2014-04-14 18:58:42.115240 7f5cf1f7e700 0 -- 192.168.1.105:6821/14060022 >> 192.168.1.101:6803/6017061 pipe(0x7bb9400 sd=144 :56930 s=2 pgs=3157 cs=1 l=0 c=0x4fae880).fault with nothing to send, going to standby 2014-04-14 18:58:43.138088 7f5ce9828700 0 -- 192.168.1.105:6819/15060022 >> 192.168.1.106:6816/3002245 pipe(0x537f800 sd=248 :6819 s=0 pgs=0 cs=0 l=0 c=0x77bad60).accept connect_seq 0 vs existing 0 state connecting 2014-04-14 18:58:43.138358 7f5ce9727700 0 -- 192.168.1.105:6819/15060022 >> 192.168.1.112:6801/3034109 pipe(0x537b200 sd=249 :6819 s=0 pgs=0 cs=0 l=0 c=0x77bf380).accept connect_seq 0 vs existing 0 state connecting 2014-04-14 18:58:43.138470 7f5ce9a2a700 0 -- 192.168.1.105:6819/15060022 >> 192.168.1.103:6801/5036094 pipe(0x4d39b80 sd=244 :6819 s=0 pgs=0 cs=0 l=0 c=0x4e8e5c0).accept connect_seq 0 vs existing 0 state connecting 2014-04-14 18:58:43.147678 7f5ce9020700 0 -- 192.168.1.105:6819/15060022 >> 192.168.1.112:6806/5032817 pipe(0x4e90a00 sd=255 :6819 s=0 pgs=0 cs=0 l=0 c=0x77b8420).accept connect_seq 0 vs existing 0 state connecting 2014-04-14 18:58:43.147827 7f5ce7deb700 0 -- 192.168.1.105:6819/15060022 >> 192.168.1.102:6823/11011704 pipe(0x4e97d00 sd=263 :6819 s=0 pgs=0 cs=0 l=0 c=0x77bd3e0).accept connect_seq 0 vs existing 0 state connecting 2014-04-14 18:58:43.148016 7f5ce7be9700 0 -- 192.168.1.105:6819/15060022 >> 192.168.1.101:6807/3018425 pipe(0x4e95000 sd=266 :6819 s=0 pgs=0 cs=0 l=0 c=0x77bd6a0).accept connect_seq 0 vs existing 0 state connecting 2014-04-14 18:58:43.148036 7f5ce7cea700 0 -- 192.168.1.105:6819/15060022 >> 192.168.1.104:6807/7061463 pipe(0x4e91900 sd=264 :6819 s=0 pgs=0 cs=0 l=0 c=0x77baaa0).accept connect_seq 0 vs existing 0 state connecting 2014-04-14 18:58:43.148081 7f5ce72e0700 0 -- 192.168.1.105:6819/15060022 >> 192.168.1.111:6803/4009103 pipe(0x4e97300 sd=274 :6819 s=0 pgs=0 cs=0 l=0 c=0x5a274e0).accept connect_seq 0 vs existing 0 state connecting 2014-04-14 18:58:43.148149 7f5ce8d1d700 0 -- 192.168.1.105:6819/15060022 >> 192.168.1.112:6829/1023135 pipe(0x4e95500 sd=260 :6819 s=0 pgs=0 cs=0 l=0 c=0x77b89a0).accept connect_seq 0 vs existing 0 state connecting 2014-04-14 18:58:43.148472 7f5ce6fdd700 0 -- 192.168.1.105:6819/15060022 >> 192.168.1.107:6829/3004265 pipe(0x4e91b80 sd=278 :6819 s=0 pgs=0 cs=0 l=0 c=0x4c16460).accept connect_seq 0 vs existing 0 state connecting 2014-04-14 18:58:43.149670 7f5ce7ae8700 0 -- 192.168.1.105:6819/15060022 >> 192.168.1.108:6811/3010125 pipe(0x4e94b00 sd=267 :6819 s=0 pgs=0 cs=0 l=0 c=0x59ee5c0).accept connect_seq 0 vs existing 0 state connecting 2014-04-14 18:58:43.158744 7f5ce9929700 0 -- 192.168.1.105:6819/15060022 >> 192.168.1.104:6817/7054057 pipe(0x4d3ee00 sd=246 :6819 s=0 pgs=0 cs=0 l=0 c=0x77bcba0).accept connect_seq 0 vs existing 0 state connecting 2014-04-14 18:59:02.414643 7f5d0ccab700 -1 osd.51 18440 heartbeat_check: no reply from osd.52 ever on either front or back, first ping sent 2014-04-14 18:58:42.164242 (cutoff 2014-04-14 18:58:42.414639) 2014-04-14 18:59:02.481970 7f5cf9bd0700 -1 osd.51 18440 heartbeat_check: no reply from osd.52 ever on either front or back, first ping sent 2014-04-14 18:58:42.164242 (cutoff 2014-04-14 18:58:42.481968) 2014-04-14 18:59:03.414829 7f5d0ccab700 -1 osd.51 18441 heartbeat_check: no reply from osd.52 ever on either front or back, first ping sent 2014-04-14 18:58:42.164242 (cutoff 2014-04-14 18:58:43.414827) 2014-04-14 18:59:0
Updated by karan singh about 10 years ago
I have been searching on internet and found there has been similar kind of bugs came up in past as well and fixes were created for them.
http://tracker.ceph.com/issues/5172
http://tracker.ceph.com/issues/5460
Looks like Version 0.79 also need some fixes.
Updated by Sage Weil about 10 years ago
This is usually caused by low memory leading to swapping. Can you verify the CPU and memory are not oversubscribed?
Updated by karan singh about 10 years ago
Thanks for your interest Sage , today morning the cluster looks normal and healthy. Unfortunately i do not have system performance logs at the time of problem , but now looks like overnight system memory got released , if we consider this a problem was due to CPU and memory.
Updated by Sage Weil about 10 years ago
- Status changed from New to Can't reproduce
Thanks for the follow-up. Please let us know if you can figure out how to reproduce the problem, or can gather more information when it happens again.