Project

General

Profile

Bug #8098

ceph v0.79-125 : Random osd's are flapping too frequently : OSD wrongly marked me down

Added by karan singh almost 10 years ago. Updated almost 10 years ago.

Status:
Can't reproduce
Priority:
Immediate
Assignee:
-
Category:
OSD
Target version:
% Done:

0%

Source:
other
Tags:
OSD flapping
Backport:
Regression:
No
Severity:
1 - critical
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Hello Developers

I have been facing weird problem with my ceph cluster

Problem : Randomly OSDs are flapping , i.e some osds are getting down and in few seconds they are coming up , and there always some osds are down. So this problem is not specific to few OSDS but to entire cluster OSDs.

  • Disk , node , network and other infrastructure everything i have checked and is OK.
  • originally cluster was running ceph version 0.79 on all the nodes
  • I thought there might be some bugs in this version so i upgraded it to 0.79-125 and rebooted entire cluster after upgradation , but still problem persist.
  • After that i again upgraded to 0.79-185 from master which is the latest version , but NO LUCK
  • FYI before and after upgradation , i have restarted all ceph services as well as OS reboot , but NO LUCK
  • Debug MS = 1 , set to OSDs
  • Below is the cluster health command output in difference of few seconds , it clearly shows OSDS are getting down and coming up

[root@storage0115-ib ~]#
[root@storage0115-ib ~]# date;ceph -s
Mon Apr 14 19:12:06 EEST 2014
    cluster e8768ef6-93a1-4e8c-acda-3ac995e35003
     health HEALTH_WARN 94 pgs peering; 29 pgs stale; *1/165 in osds are down*
     monmap e3: 3 mons at {storage0101-ib=192.168.100.101:6789/0,storage0107-ib=192.168.100.107:6789/0,storage0115-ib=192.168.100.115:6789/0}, election epoch 50, quorum 0,1,2 storage0101-ib,storage0107-ib,storage0115-ib
     mdsmap e67: 1/1/1 up {0=storage0110=up:active}, 3 up:standby
     osdmap e19041: 165 osds: 164 up, 165 in
      pgmap v38433: 5120 pgs, 3 pools, 5733 GB data, 1628 kobjects
            5290 GB used, 443 TB / 448 TB avail
                  29 stale+active+clean
                  94 peering
                4997 active+clean
[root@storage0115-ib ~]#
[root@storage0115-ib ~]#
[root@storage0115-ib ~]# date;ceph -s
*Mon Apr 14 19:12:10 EEST 2014*
    cluster e8768ef6-93a1-4e8c-acda-3ac995e35003
     health HEALTH_WARN 93 pgs peering; 272 pgs stale; *8/165 in osds are down*
     monmap e3: 3 mons at {storage0101-ib=192.168.100.101:6789/0,storage0107-ib=192.168.100.107:6789/0,storage0115-ib=192.168.100.115:6789/0}, election epoch 50, quorum 0,1,2 storage0101-ib,storage0107-ib,storage0115-ib
     mdsmap e67: 1/1/1 up {0=storage0110=up:active}, 3 up:standby
     osdmap e19044: 165 osds: 157 up, 165 in
      pgmap v38439: 5120 pgs, 3 pools, 5733 GB data, 1628 kobjects
            5290 GB used, 443 TB / 448 TB avail
                   2 stale+peering
                 270 stale+active+clean
                  91 peering
                4757 active+clean
[root@storage0115-ib ~]# date;ceph -s
*Mon Apr 14 19:12:15 EEST 2014*
    cluster e8768ef6-93a1-4e8c-acda-3ac995e35003
     health HEALTH_WARN 94 pgs peering; 371 pgs stale; *2/165 in osds are down*
     monmap e3: 3 mons at {storage0101-ib=192.168.100.101:6789/0,storage0107-ib=192.168.100.107:6789/0,storage0115-ib=192.168.100.115:6789/0}, election epoch 50, quorum 0,1,2 storage0101-ib,storage0107-ib,storage0115-ib
     mdsmap e67: 1/1/1 up {0=storage0110=up:active}, 3 up:standby
     osdmap e19049: 165 osds: 163 up, 165 in
      pgmap v38444: 5120 pgs, 3 pools, 5733 GB data, 1628 kobjects
            5290 GB used, 443 TB / 448 TB avail
                   2 stale+peering
                 369 stale+active+clean
                  92 peering
                4657 active+clean
[root@storage0115-ib ~]# date;ceph -s
*Mon Apr 14 19:12:19 EEST 2014*
    cluster e8768ef6-93a1-4e8c-acda-3ac995e35003
     health HEALTH_WARN 6 pgs degraded; 94 pgs peering; 489 pgs stale; 2 pgs stuck unclean; recovery 958/5002182 objects degraded (0.019%); *2/165 in osds are down*
     monmap e3: 3 mons at {storage0101-ib=192.168.100.101:6789/0,storage0107-ib=192.168.100.107:6789/0,storage0115-ib=192.168.100.115:6789/0}, election epoch 50, quorum 0,1,2 storage0101-ib,storage0107-ib,storage0115-ib
     mdsmap e67: 1/1/1 up {0=storage0110=up:active}, 3 up:standby
     osdmap e19053: 165 osds: 163 up, 165 in
      pgmap v38448: 5120 pgs, 3 pools, 5733 GB data, 1628 kobjects
            5290 GB used, 443 TB / 448 TB avail
            958/5002182 objects degraded (0.019%)
                  11 stale+peering
                 478 stale+active+clean
                   6 active+degraded
                  83 peering
                4542 active+clean
[root@storage0115-ib ~]# date;ceph -s
*Mon Apr 14 19:12:26 EEST 2014*
    cluster e8768ef6-93a1-4e8c-acda-3ac995e35003
     health HEALTH_WARN 5 pgs degraded; 91 pgs peering; 438 pgs stale; 1 pgs stuck unclean; recovery 886/5002182 objects degraded (0.018%); *1/165 in osds are dow*n
     monmap e3: 3 mons at {storage0101-ib=192.168.100.101:6789/0,storage0107-ib=192.168.100.107:6789/0,storage0115-ib=192.168.100.115:6789/0}, election epoch 50, quorum 0,1,2 storage0101-ib,storage0107-ib,storage0115-ib
     mdsmap e67: 1/1/1 up {0=storage0110=up:active}, 3 up:standby
     osdmap e19060: 165 osds: 164 up, 165 in
      pgmap v38458: 5120 pgs, 3 pools, 5733 GB data, 1628 kobjects
            5290 GB used, 443 TB / 448 TB avail
            886/5002182 objects degraded (0.018%)
                  12 stale+peering
                 426 stale+active+clean
                   5 active+degraded
                  79 peering
                4598 active+clean
[root@storage0115-ib ~]# date;ceph -s
*Mon Apr 14 19:12:35 EEST 2014*
    cluster e8768ef6-93a1-4e8c-acda-3ac995e35003
     health HEALTH_WARN 10 pgs degraded; 87 pgs peering; 366 pgs stale; 1 pgs stuck unclean; recovery 5683/5002182 objects degraded (0.114%)
     monmap e3: 3 mons at {storage0101-ib=192.168.100.101:6789/0,storage0107-ib=192.168.100.107:6789/0,storage0115-ib=192.168.100.115:6789/0}, election epoch 50, quorum 0,1,2 storage0101-ib,storage0107-ib,storage0115-ib
     mdsmap e67: 1/1/1 up {0=storage0110=up:active}, 3 up:standby
     *osdmap e19067: 165 osds: 165 up, 165 in*
      pgmap v38467: 5120 pgs, 3 pools, 5733 GB data, 1628 kobjects
            5291 GB used, 443 TB / 448 TB avail
            5683/5002182 objects degraded (0.114%)
                  21 stale+peering
                 345 stale+active+clean
                  10 active+degraded
                  66 peering
                4678 active+clean
[root@storage0115-ib ~]# date;ceph -s
*Mon Apr 14 19:12:47 EEST 2014*
    cluster e8768ef6-93a1-4e8c-acda-3ac995e35003
     health HEALTH_WARN 2 pgs degraded; 85 pgs peering; 169 pgs stale; 2 pgs stuck unclean; recovery 423/5002182 objects degraded (0.008%)
     monmap e3: 3 mons at {storage0101-ib=192.168.100.101:6789/0,storage0107-ib=192.168.100.107:6789/0,storage0115-ib=192.168.100.115:6789/0}, election epoch 50, quorum 0,1,2 storage0101-ib,storage0107-ib,storage0115-ib
     mdsmap e67: 1/1/1 up {0=storage0110=up:active}, 3 up:standby
    * osdmap e19076: 165 osds: 165 up, 165 in*
      pgmap v38482: 5120 pgs, 3 pools, 5733 GB data, 1628 kobjects
            5291 GB used, 443 TB / 448 TB avail
            423/5002182 objects degraded (0.008%)
                   6 stale+peering
                 163 stale+active+clean
                   2 active+degraded
                  79 peering
                4870 active+clean
[root@storage0115-ib ~]# date;ceph -s
*Mon Apr 14 19:12:58 EEST 2014*
    cluster e8768ef6-93a1-4e8c-acda-3ac995e35003
     health HEALTH_WARN 1 pgs degraded; 79 pgs peering; 314 pgs stale; recovery 339/5002182 objects degraded (0.007%); *1/165 in osds are down*
     monmap e3: 3 mons at {storage0101-ib=192.168.100.101:6789/0,storage0107-ib=192.168.100.107:6789/0,storage0115-ib=192.168.100.115:6789/0}, election epoch 50, quorum 0,1,2 storage0101-ib,storage0107-ib,storage0115-ib
     mdsmap e67: 1/1/1 up {0=storage0110=up:active}, 3 up:standby
     osdmap e19083: 165 osds: 164 up, 165 in
      pgmap v38492: 5120 pgs, 3 pools, 5733 GB data, 1628 kobjects
            5291 GB used, 443 TB / 448 TB avail
            339/5002182 objects degraded (0.007%)
                   7 stale+peering
                 307 stale+active+clean
                   1 active+degraded
                  72 peering
                4733 active+clean
[root@storage0115-ib ~]#

  • As an example i have copied logs of osd.163 and osd.51.log

LOGS FROM osd.163


[root@storage0112-ib ceph]# tail -100f ceph-osd.163.log
2014-04-14 18:49:26.760144 7fe8b6542700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.106:6807/3008357 pipe(0x4512a80 sd=255 :6821 s=2 pgs=628 cs=1 l=0 c=0x4e586e0).fault with nothing to send, going to standby
2014-04-14 18:49:26.760262 7fe8a5735700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.102:6818/1008897 pipe(0x3e05780 sd=230 :6821 s=2 pgs=528 cs=1 l=0 c=0x4e44ba0).fault with nothing to send, going to standby
2014-04-14 18:49:26.760363 7fe89eac9700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.102:6817/1013999 pipe(0x4da7a80 sd=57 :55066 s=2 pgs=501 cs=1 l=0 c=0x3f13700).fault with nothing to send, going to standby
2014-04-14 18:49:26.760497 7fe8b9d71700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.111:6805/2019168 pipe(0x4105f00 sd=215 :57674 s=2 pgs=504 cs=1 l=0 c=0x6939080).fault with nothing to send, going to standby
2014-04-14 18:49:26.760571 7fe8b9f73700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.108:6806/3006246 pipe(0x4510000 sd=207 :52322 s=2 pgs=705 cs=1 l=0 c=0x6939760).fault with nothing to send, going to standby
2014-04-14 18:49:26.761037 7fe8bb68a700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.114:6800/1064947 pipe(0x4510280 sd=201 :56237 s=2 pgs=427 cs=1 l=0 c=0x693cfc0).fault with nothing to send, going to standby
2014-04-14 18:49:26.761041 7fe8c23f7700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.115:6804/1020908 pipe(0x4da6900 sd=132 :45679 s=2 pgs=424 cs=1 l=0 c=0x6abeb40).fault with nothing to send, going to standby
2014-04-14 18:49:26.761287 7fe8ba276700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.115:6800/4012452 pipe(0x4dfdc80 sd=244 :6821 s=2 pgs=714 cs=1 l=0 c=0x3f14a40).fault with nothing to send, going to standby
2014-04-14 18:49:26.761646 7fe8b04e2700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.106:6800/3011682 pipe(0x3f7c100 sd=69 :6821 s=2 pgs=709 cs=1 l=0 c=0x45b70c0).fault with nothing to send, going to standby
2014-04-14 18:49:26.762007 7fe8b996d700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.108:6804/5178 pipe(0x4da4600 sd=144 :55370 s=2 pgs=368 cs=1 l=0 c=0x6918dc0).fault with nothing to send, going to standby
2014-04-14 18:49:26.761908 7fe8bd7ab700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.110:6827/37510 pipe(0x4107580 sd=163 :55475 s=2 pgs=358 cs=1 l=0 c=0x691aaa0).fault with nothing to send, going to standby
2014-04-14 18:49:26.763135 7fe89d0af700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.115:6813/2017035 pipe(0x4107800 sd=463 :6821 s=2 pgs=449 cs=1 l=0 c=0x6abc620).fault with nothing to send, going to standby
2014-04-14 18:49:26.763279 7fe89cbaa700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.101:6809/4029366 pipe(0x51c2d00 sd=323 :6821 s=2 pgs=816 cs=1 l=0 c=0x460d6a0).fault with nothing to send, going to standby
2014-04-14 18:49:26.763461 7fe8aeecc700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.107:6822/2008022 pipe(0x3f7b200 sd=210 :6821 s=2 pgs=490 cs=1 l=0 c=0x4e253e0).fault with nothing to send, going to standby
2014-04-14 18:49:26.763513 7fe8bbb8f700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.111:6818/1011034 pipe(0x4da0000 sd=190 :41808 s=2 pgs=431 cs=1 l=0 c=0x693aec0).fault with nothing to send, going to standby
2014-04-14 18:49:26.762625 7fe8aa684700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.105:6813/9002701 pipe(0x4dfb980 sd=334 :6821 s=2 pgs=1523 cs=1 l=0 c=0x3f13180).fault with nothing to send, going to standby
2014-04-14 18:49:26.764078 7fe8ba87c700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.103:6808/4033047 pipe(0x4105780 sd=156 :42310 s=2 pgs=766 cs=1 l=0 c=0x691b9c0).fault with nothing to send, going to standby
2014-04-14 18:49:26.762769 7fe8c1aee700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.110:6814/1030992 pipe(0x4da6e00 sd=134 :46412 s=2 pgs=482 cs=1 l=0 c=0x4640f20).fault with nothing to send, going to standby
2014-04-14 18:49:26.762776 7fe8deb7c700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.101:6801/1030688 pipe(0x3f79400 sd=93 :38016 s=2 pgs=534 cs=1 l=0 c=0x45fdac0).fault with nothing to send, going to standby
2014-04-14 18:49:26.764153 7fe8aa583700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.103:6807/3026719 pipe(0x4103980 sd=92 :6821 s=2 pgs=743 cs=1 l=0 c=0x691dac0).fault with nothing to send, going to standby
2014-04-14 18:49:26.764210 7fe8b502d700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.115:6819/4009855 pipe(0x3e02080 sd=119 :6821 s=2 pgs=816 cs=1 l=0 c=0x3f144c0).fault with nothing to send, going to standby
2014-04-14 18:49:26.764983 7fe8b7d5a700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.113:6822/1064877 pipe(0x4101680 sd=228 :33958 s=2 pgs=439 cs=1 l=0 c=0x4e2bb20).fault with nothing to send, going to standby
2014-04-14 18:49:26.765238 7fe8ae9c7700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.106:6811/3004240 pipe(0x6ab0780 sd=159 :6821 s=2 pgs=671 cs=1 l=0 c=0x3906b40).fault with nothing to send, going to standby
2014-04-14 18:49:26.765345 7fe8ded7e700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.102:6824/15636 pipe(0x4da4100 sd=104 :54179 s=2 pgs=394 cs=1 l=0 c=0x3f10f20).fault with nothing to send, going to standby
2014-04-14 18:49:26.765489 7fe8b2f0d700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.101:6811/3023921 pipe(0x4dfe400 sd=150 :6821 s=2 pgs=787 cs=1 l=0 c=0x6939b80).fault with nothing to send, going to standby
2014-04-14 18:49:26.765740 7fe8c25f9700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.102:6810/1019423 pipe(0x4da0a00 sd=108 :34679 s=2 pgs=492 cs=1 l=0 c=0x4e5b5a0).fault with nothing to send, going to standby
2014-04-14 18:49:26.765835 7fe89e1c0700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.105:6803/5003300 pipe(0x4dfd000 sd=45 :6821 s=2 pgs=1007 cs=1 l=0 c=0x3f132e0).fault with nothing to send, going to standby
2014-04-14 18:49:26.765984 7fe8c11e5700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.104:6814/1060356 pipe(0x3f7b480 sd=29 :36609 s=2 pgs=462 cs=1 l=0 c=0x3f102c0).fault with nothing to send, going to standby
2014-04-14 18:49:26.765989 7fe8a6a48700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.106:6829/2002245 pipe(0x6ab0f00 sd=68 :6821 s=2 pgs=2426 cs=1 l=0 c=0x460dc20).fault with nothing to send, going to standby
2014-04-14 18:49:26.766110 7fe8bc69a700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.111:6816/1022206 pipe(0x4da1400 sd=179 :42290 s=2 pgs=406 cs=1 l=0 c=0x693f380).fault with nothing to send, going to standby
2014-04-14 18:49:26.766182 7fe8be2b6700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.109:6812/4053163 pipe(0x3e04600 sd=24 :6821 s=2 pgs=10633 cs=1 l=0 c=0x4e40b00).fault with nothing to send, going to standby
2014-04-14 18:49:26.766192 7fe8b9b6f700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.109:6802/1058316 pipe(0x4107a80 sd=218 :53442 s=2 pgs=460 cs=1 l=0 c=0x406f0c0).fault with nothing to send, going to standby
2014-04-14 18:49:26.766246 7fe8b9569700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.104:6815/3057518 pipe(0x4106e00 sd=165 :33062 s=2 pgs=726 cs=1 l=0 c=0x693a7e0).fault with nothing to send, going to standby
2014-04-14 18:49:26.766394 7fe8bb387700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.109:6817/4055614 pipe(0x4df8000 sd=240 :6821 s=2 pgs=882 cs=1 l=0 c=0x4e2b9c0).fault with nothing to send, going to standby
2014-04-14 18:49:26.766609 7fe8c20f4700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.108:6810/2000970 pipe(0x3e06900 sd=101 :50580 s=2 pgs=631 cs=1 l=0 c=0x3fb9e40).fault with nothing to send, going to standby
2014-04-14 18:49:26.766664 7fe8bf0c4700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.113:6812/2059303 pipe(0x4da5780 sd=147 :39565 s=2 pgs=532 cs=1 l=0 c=0x4e08dc0).fault with nothing to send, going to standby
2014-04-14 18:49:26.766713 7fe8bd1a5700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.109:6803/5061793 pipe(0x4102580 sd=164 :46721 s=2 pgs=915 cs=1 l=0 c=0x693e300).fault with nothing to send, going to standby
2014-04-14 18:49:26.766716 7fe898867700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.105:6821/8000988 pipe(0x4dfda00 sd=196 :6821 s=2 pgs=1243 cs=1 l=0 c=0x3e09a20).fault with nothing to send, going to standby
2014-04-14 18:49:26.766767 7fe8bb084700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.107:6804/14881 pipe(0x4102300 sd=172 :56835 s=2 pgs=371 cs=1 l=0 c=0x693ac00).fault with nothing to send, going to standby
2014-04-14 18:49:26.766825 7fe8b15f3700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.110:6825/4036166 pipe(0x4513480 sd=203 :6821 s=2 pgs=832 cs=1 l=0 c=0x3e0c4c0).fault with nothing to send, going to standby
2014-04-14 18:49:26.767086 7fe8bf4c8700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.104:6812/51547 pipe(0x4da6680 sd=113 :39159 s=2 pgs=423 cs=1 l=0 c=0x45ccd00).fault with nothing to send, going to standby
2014-04-14 18:49:26.767096 7fe8bdeb2700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.114:6802/2007089 pipe(0x4103480 sd=161 :59020 s=2 pgs=557 cs=1 l=0 c=0x693dd80).fault with nothing to send, going to standby
2014-04-14 18:49:26.767147 7fe8c1bef700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.104:6804/2061463 pipe(0x4106400 sd=251 :6821 s=2 pgs=596 cs=1 l=0 c=0x4e46720).fault with nothing to send, going to standby
2014-04-14 18:49:26.767361 7fe8c05d9700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.113:6810/2001582 pipe(0x4105a00 sd=148 :6821 s=2 pgs=614 cs=1 l=0 c=0x3e0f0c0).fault with nothing to send, going to standby
2014-04-14 18:49:26.767741 7fe8bddb1700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.109:6825/2059246 pipe(0x4100f00 sd=167 :41990 s=2 pgs=597 cs=1 l=0 c=0x693e040).fault with nothing to send, going to standby
2014-04-14 18:49:26.767849 7fe89ffde700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.110:6805/3038601 pipe(0x4dfc380 sd=245 :6821 s=2 pgs=948 cs=1 l=0 c=0x4e46ca0).fault with nothing to send, going to standby
2014-04-14 18:49:26.767912 7fe89e4c3700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.114:6805/1003429 pipe(0x4106680 sd=327 :6821 s=2 pgs=354 cs=1 l=0 c=0x4e2e300).fault with nothing to send, going to standby
2014-04-14 18:49:26.768000 7fe89fcdb700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.111:6829/2017859 pipe(0x4da5f00 sd=309 :6821 s=2 pgs=535 cs=1 l=0 c=0x4e2e5c0).fault with nothing to send, going to standby
2014-04-14 18:49:26.768207 7fe8c09dd700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.104:6808/48025 pipe(0x3f7ee00 sd=31 :41331 s=2 pgs=403 cs=1 l=0 c=0x3f170c0).fault with nothing to send, going to standby
2014-04-14 18:49:26.768259 7fe8bcfa3700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.113:6808/1060374 pipe(0x4da4d80 sd=175 :58755 s=2 pgs=420 cs=1 l=0 c=0x693d540).fault with nothing to send, going to standby
2014-04-14 18:49:26.768310 7fe8be4b8700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.109:6811/51869 pipe(0x4da4880 sd=90 :46873 s=2 pgs=332 cs=1 l=0 c=0x3f11080).fault with nothing to send, going to standby
2014-04-14 18:49:26.768569 7fe8c1df1700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.101:6800/1026435 pipe(0x6ab4380 sd=46 :6821 s=2 pgs=596 cs=1 l=0 c=0x3fbe720).fault with nothing to send, going to standby
2014-04-14 18:49:26.768584 7fe8a6f4d700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.106:6826/6092 pipe(0x3f79b80 sd=38 :55297 s=2 pgs=384 cs=1 l=0 c=0x4e26e00).fault with nothing to send, going to standby
2014-04-14 18:49:26.768629 7fe8a6b49700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.102:6802/6271 pipe(0x4da0280 sd=106 :41261 s=2 pgs=413 cs=1 l=0 c=0x4e59b80).fault with nothing to send, going to standby
2014-04-14 18:49:26.768721 7fe899d7c700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.103:6800/2034753 pipe(0x3f7b980 sd=40 :39840 s=2 pgs=613 cs=1 l=0 c=0x4e5fa60).fault with nothing to send, going to standby
2014-04-14 18:49:26.768976 7fe89f2d1700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.103:6817/5030905 pipe(0x4da7300 sd=306 :6821 s=2 pgs=994 cs=1 l=0 c=0x45b2100).fault with nothing to send, going to standby
2014-04-14 18:49:26.769071 7fe8bcb9f700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.108:6817/2007814 pipe(0x3e01680 sd=102 :51322 s=2 pgs=607 cs=1 l=0 c=0x455b9c0).fault with nothing to send, going to standby
2014-04-14 18:49:26.769233 7fe89a07f700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.102:6820/12949 pipe(0x4da1180 sd=110 :43907 s=2 pgs=516 cs=1 l=0 c=0x688bb20).fault with nothing to send, going to standby
2014-04-14 18:49:26.769755 7fe8c26fa700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.111:6808/2009889 pipe(0x56ff080 sd=30 :55441 s=2 pgs=494 cs=1 l=0 c=0x406fbc0).fault with nothing to send, going to standby
2014-04-14 18:49:26.769818 7fe8aeac8700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.111:6809/2020353 pipe(0x3e05f00 sd=467 :6821 s=2 pgs=551 cs=1 l=0 c=0x4e21fa0).fault with nothing to send, going to standby
2014-04-14 18:49:26.770385 7fe8c21f5700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.104:6819/2056631 pipe(0x3f7c600 sd=233 :6821 s=2 pgs=614 cs=1 l=0 c=0x4e23020).fault with nothing to send, going to standby
2014-04-14 18:49:26.770510 7fe8a1dfc700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.107:6802/3003023 pipe(0x3e06180 sd=71 :6821 s=2 pgs=654 cs=1 l=0 c=0x3f11340).fault with nothing to send, going to standby
2014-04-14 18:49:26.770857 7fe8bd9ad700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.115:6805/1021892 pipe(0x3e02a80 sd=168 :6821 s=2 pgs=467 cs=1 l=0 c=0x4e46880).fault with nothing to send, going to standby
2014-04-14 18:49:26.770930 7fe8bcca0700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.110:6806/3032536 pipe(0x3e03200 sd=213 :6821 s=2 pgs=720 cs=1 l=0 c=0x4e21e40).fault with nothing to send, going to standby
2014-04-14 18:49:26.771083 7fe8ae3c1700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.108:6807/4002228 pipe(0x6ab4b00 sd=166 :6821 s=2 pgs=780 cs=1 l=0 c=0x3df65c0).fault with nothing to send, going to standby
2014-04-14 18:49:26.771269 7fe8bb98d700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.106:6806/1009649 pipe(0x4515280 sd=197 :45527 s=2 pgs=518 cs=1 l=0 c=0x693dc20).fault with nothing to send, going to standby
2014-04-14 18:49:26.771480 7fe8a01e0700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.109:6804/3056966 pipe(0x4df8280 sd=249 :6821 s=2 pgs=662 cs=1 l=0 c=0x4e42100).fault with nothing to send, going to standby
2014-04-14 18:49:26.771649 7fe8b2705700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.108:6801/4011474 pipe(0x6ab3480 sd=122 :6821 s=2 pgs=783 cs=1 l=0 c=0x460b5a0).fault with nothing to send, going to standby
2014-04-14 18:49:26.772206 7fe8c2e01700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.108:6808/1012898 pipe(0x3e00280 sd=79 :50753 s=2 pgs=493 cs=1 l=0 c=0x455b180).fault with nothing to send, going to standby
2014-04-14 18:49:26.772497 7fe89dab9700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.115:6809/1011533 pipe(0x3e03c00 sd=51 :58020 s=2 pgs=394 cs=1 l=0 c=0x691f0c0).fault with nothing to send, going to standby
2014-04-14 18:49:26.772747 7fe8bb185700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.111:6825/2012002 pipe(0x3e02300 sd=193 :6821 s=2 pgs=520 cs=1 l=0 c=0x3df4a40).fault with nothing to send, going to standby
2014-04-14 18:49:26.773546 7fe89a786700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.111:6800/4014159 pipe(0x4513700 sd=32 :6821 s=2 pgs=759 cs=1 l=0 c=0x4e256a0).fault with nothing to send, going to standby
2014-04-14 18:49:26.773785 7fe8b1cfa700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.101:6816/4019830 pipe(0x3e01900 sd=267 :6821 s=2 pgs=820 cs=1 l=0 c=0x3f177a0).fault with nothing to send, going to standby
2014-04-14 18:49:26.773874 7fe89d3b2700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.114:6822/1000532 pipe(0x4103c00 sd=486 :6821 s=2 pgs=2158 cs=1 l=0 c=0x4dae720).fault with nothing to send, going to standby
2014-04-14 18:49:26.773938 7fe89caa9700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.110:6812/5034877 pipe(0x51c2080 sd=82 :6821 s=2 pgs=855 cs=1 l=0 c=0x46082c0).fault with nothing to send, going to standby
2014-04-14 18:49:26.774154 7fe8a9f7d700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.103:6810/3036094 pipe(0x4dfd280 sd=274 :6821 s=2 pgs=791 cs=1 l=0 c=0x3e0aec0).fault with nothing to send, going to standby
2014-04-14 18:49:26.774164 7fe8b7653700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.104:6821/4054057 pipe(0x4dfbe80 sd=264 :6821 s=2 pgs=860 cs=1 l=0 c=0x3e08580).fault with nothing to send, going to standby
2014-04-14 18:49:26.774233 7fe8b05e3700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.104:6807/2059338 pipe(0x3f78c80 sd=138 :6821 s=2 pgs=738 cs=1 l=0 c=0x3e08b00).fault with nothing to send, going to standby
2014-04-14 18:49:26.774448 7fe8b6e4b700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.103:6815/2039519 pipe(0x4100280 sd=236 :6821 s=2 pgs=676 cs=1 l=0 c=0x3e09340).fault with nothing to send, going to standby
2014-04-14 18:49:26.774499 7fe8bb589700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.115:6802/6008752 pipe(0x4107d00 sd=293 :6821 s=2 pgs=918 cs=1 l=0 c=0x3e0a680).fault with nothing to send, going to standby
2014-04-14 18:49:26.775163 7fe8ae1bf700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.103:6821/31885 pipe(0x3f7a080 sd=36 :52574 s=2 pgs=411 cs=1 l=0 c=0x3f139c0).fault with nothing to send, going to standby
2014-04-14 18:49:26.775260 7fe8ba074700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.114:6807/12470 pipe(0x4105500 sd=216 :41255 s=2 pgs=251 cs=1 l=0 c=0x6938420).fault with nothing to send, going to standby
2014-04-14 18:49:26.775417 7fe8b7c59700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.101:6821/4022662 pipe(0x4100000 sd=222 :34937 s=2 pgs=797 cs=1 l=0 c=0x4e2ee00).fault with nothing to send, going to standby
2014-04-14 18:49:26.776471 7fe8c27fb700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.115:6832/2015402 pipe(0x4da2300 sd=130 :51852 s=2 pgs=2249 cs=1 l=0 c=0x4e565c0).fault with nothing to send, going to standby
2014-04-14 18:49:26.776510 7fe8aa381700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.110:6818/1042305 pipe(0x3e00f00 sd=86 :32802 s=2 pgs=385 cs=1 l=0 c=0x455dee0).fault with nothing to send, going to standby
2014-04-14 18:49:26.777072 7fe8ae2c0700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.113:6802/3006545 pipe(0x6ab7800 sd=170 :6821 s=2 pgs=644 cs=1 l=0 c=0x4dacd00).fault with nothing to send, going to standby
2014-04-14 18:49:26.777399 7fe8c0ee2700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.113:6806/3063888 pipe(0x4da2580 sd=141 :38315 s=2 pgs=641 cs=1 l=0 c=0x455fa60).fault with nothing to send, going to standby
2014-04-14 18:49:26.777721 7fe8a04e3700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.114:6828/3009870 pipe(0x4dfc600 sd=239 :6821 s=2 pgs=672 cs=1 l=0 c=0x4e548e0).fault with nothing to send, going to standby
2014-04-14 18:49:26.777789 7fe8c0fe3700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.109:6809/3050592 pipe(0x3e05500 sd=107 :33458 s=2 pgs=633 cs=1 l=0 c=0x693d120).fault with nothing to send, going to standby
2014-04-14 18:49:26.778745 7fe8a5e3c700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.107:6800/2004265 pipe(0x4da5000 sd=124 :58863 s=2 pgs=573 cs=1 l=0 c=0x6ab98c0).fault with nothing to send, going to standby
2014-04-14 18:49:26.780184 7fe8bc599700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.109:6807/2063104 pipe(0x6ab0c80 sd=117 :6821 s=2 pgs=632 cs=1 l=0 c=0x4608580).fault with nothing to send, going to standby
2014-04-14 18:49:26.780294 7fe8bc498700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.107:6815/1010676 pipe(0x4da7580 sd=173 :49411 s=2 pgs=462 cs=1 l=0 c=0x693dac0).fault with nothing to send, going to standby
2014-04-14 18:49:26.781906 7fe8bf1c5700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.102:6809/4018133 pipe(0x4da3c00 sd=153 :53298 s=2 pgs=761 cs=1 l=0 c=0x3f169e0).fault with nothing to send, going to standby
2014-04-14 18:49:26.782983 7fe8b14f2700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.102:6814/4016928 pipe(0x4103700 sd=52 :6821 s=2 pgs=805 cs=1 l=0 c=0x3e0f640).fault with nothing to send, going to standby
2014-04-14 18:49:26.784876 7fe8b7f5c700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.103:6824/37247 pipe(0x4101180 sd=223 :59662 s=2 pgs=378 cs=1 l=0 c=0x4e2e1a0).fault with nothing to send, going to standby
2014-04-14 18:49:26.788091 7fe8bba8e700  0 -- 192.168.1.112:6821/2032817 >> 192.168.1.113:6800/4061795 pipe(0x51c3480 sd=169 :6821 s=2 pgs=693 cs=1 l=0 c=0x460cd00).fault with nothing to send, going to standby
2014-04-14 18:49:27.156407 7fe8d201b700  0 log [WRN] : map e17971 wrongly marked me down
2014-04-14 18:50:24.313612 7fe8b704d700  0 -- 192.168.1.112:6805/3032817 >> 192.168.1.114:6828/3009870 pipe(0x56fbe80 sd=218 :32785 s=2 pgs=730 cs=1 l=0 c=0x4e29e40).fault with nothing to send, going to standby
2014-04-14 18:50:44.167132 7fe8b4522700  0 -- 192.168.1.112:6805/3032817 >> 192.168.1.111:6815/3009103 pipe(0x4104380 sd=249 :34287 s=2 pgs=702 cs=1 l=0 c=0x4e5dd80).fault with nothing to send, going to standby
2014-04-14 18:50:46.189621 7fe8b08e6700  0 -- 192.168.1.112:6805/3032817 >> 192.168.1.105:6833/9062372 pipe(0x4103c00 sd=285 :0 s=1 pgs=0 cs=0 l=0 c=0x3e0ee00).fault
2014-04-14 18:51:49.643502 7fe8b522f700  0 -- 192.168.1.112:6805/3032817 >> 192.168.1.105:6833/9062372 pipe(0x8c24d80 sd=159 :6805 s=0 pgs=0 cs=0 l=0 c=0x8d3b180).accept connect_seq 0 vs existing 0 state wait
2014-04-14 18:51:51.611604 7fe8dde34700 -1 osd.163 18096 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:31.611601)
2014-04-14 18:51:52.501655 7fe8c6c09700 -1 osd.163 18097 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:32.501651)
2014-04-14 18:51:52.611763 7fe8dde34700 -1 osd.163 18097 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:32.611759)
2014-04-14 18:51:53.612074 7fe8dde34700 -1 osd.163 18098 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:33.612071)
2014-04-14 18:51:54.612282 7fe8dde34700 -1 osd.163 18099 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:34.612276)
2014-04-14 18:51:54.808065 7fe8c6c09700 -1 osd.163 18099 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:34.808062)
2014-04-14 18:51:55.612496 7fe8dde34700 -1 osd.163 18100 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:35.612494)
2014-04-14 18:51:56.612657 7fe8dde34700 -1 osd.163 18101 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:36.612650)
2014-04-14 18:51:57.613004 7fe8dde34700 -1 osd.163 18102 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:37.613001)
2014-04-14 18:51:58.613230 7fe8dde34700 -1 osd.163 18103 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:38.613224)
2014-04-14 18:51:59.613397 7fe8dde34700 -1 osd.163 18104 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:39.613391)
2014-04-14 18:52:00.117250 7fe8c6c09700 -1 osd.163 18104 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:40.117247)
2014-04-14 18:52:00.613558 7fe8dde34700 -1 osd.163 18105 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:40.613552)
2014-04-14 18:52:01.613696 7fe8dde34700 -1 osd.163 18106 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:41.613690)
2014-04-14 18:52:02.613824 7fe8dde34700 -1 osd.163 18107 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:42.613817)
2014-04-14 18:52:03.614241 7fe8dde34700 -1 osd.163 18108 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:43.614238)
2014-04-14 18:52:04.614427 7fe8dde34700 -1 osd.163 18109 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:44.614421)
2014-04-14 18:52:05.614585 7fe8dde34700 -1 osd.163 18110 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:45.614579)
2014-04-14 18:52:06.019678 7fe8c6c09700 -1 osd.163 18110 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:46.019675)
2014-04-14 18:52:06.614768 7fe8dde34700 -1 osd.163 18111 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:46.614762)
2014-04-14 18:52:07.121488 7fe8c6c09700 -1 osd.163 18111 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:47.121239)
2014-04-14 18:52:07.614932 7fe8dde34700 -1 osd.163 18112 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:47.614925)
2014-04-14 18:52:08.615090 7fe8dde34700 -1 osd.163 18113 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:48.615084)
2014-04-14 18:52:09.615448 7fe8dde34700 -1 osd.163 18114 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:49.615441)
2014-04-14 18:52:10.615609 7fe8dde34700 -1 osd.163 18115 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:50.615603)
2014-04-14 18:52:11.615901 7fe8dde34700 -1 osd.163 18116 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:51.615899)
2014-04-14 18:52:12.616056 7fe8dde34700 -1 osd.163 18117 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:52.616052)
2014-04-14 18:52:13.024155 7fe8c6c09700 -1 osd.163 18117 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:53.024151)
2014-04-14 18:52:13.616246 7fe8dde34700 -1 osd.163 18118 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:53.616240)
2014-04-14 18:52:14.616563 7fe8dde34700 -1 osd.163 18119 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:54.616559)
2014-04-14 18:52:15.616737 7fe8dde34700 -1 osd.163 18120 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:55.616731)
2014-04-14 18:52:16.526681 7fe8c6c09700 -1 osd.163 18120 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:56.526678)
2014-04-14 18:52:16.616888 7fe8dde34700 -1 osd.163 18120 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:56.616884)
2014-04-14 18:52:17.617048 7fe8dde34700 -1 osd.163 18121 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:57.617043)
2014-04-14 18:52:18.617217 7fe8dde34700 -1 osd.163 18122 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:58.617211)
2014-04-14 18:52:19.429228 7fe8c6c09700 -1 osd.163 18122 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:59.429224)
2014-04-14 18:52:19.617398 7fe8dde34700 -1 osd.163 18123 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:51:59.617392)
2014-04-14 18:52:20.617751 7fe8dde34700 -1 osd.163 18123 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:52:00.617745)
2014-04-14 18:52:21.617985 7fe8dde34700 -1 osd.163 18124 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:52:01.617983)
2014-04-14 18:52:21.731688 7fe8c6c09700 -1 osd.163 18125 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:52:01.731683)
2014-04-14 18:52:22.618289 7fe8dde34700 -1 osd.163 18125 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:52:02.618285)
2014-04-14 18:52:23.618432 7fe8dde34700 -1 osd.163 18126 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:52:03.618425)
2014-04-14 18:52:24.035341 7fe8c6c09700 -1 osd.163 18127 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:52:04.035339)
2014-04-14 18:52:24.618602 7fe8dde34700 -1 osd.163 18127 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:52:04.618596)
2014-04-14 18:52:25.618765 7fe8dde34700 -1 osd.163 18128 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:52:05.618759)
2014-04-14 18:52:25.741377 7fe8c6c09700 -1 osd.163 18128 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:52:05.741373)
2014-04-14 18:52:26.618917 7fe8dde34700 -1 osd.163 18129 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:52:06.618911)
2014-04-14 18:52:26.847912 7fe8c6c09700 -1 osd.163 18129 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:52:06.847908)
2014-04-14 18:52:27.354395 7fe8c6c09700 -1 osd.163 18129 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:52:07.354391)
2014-04-14 18:52:27.619116 7fe8dde34700 -1 osd.163 18129 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:52:07.619112)
2014-04-14 18:52:28.460652 7fe8c6c09700 -1 osd.163 18129 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:52:08.460644)
2014-04-14 18:52:28.619248 7fe8dde34700 -1 osd.163 18129 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:52:08.619244)
2014-04-14 18:52:29.619605 7fe8dde34700 -1 osd.163 18130 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:52:09.619603)
2014-04-14 18:52:30.619740 7fe8dde34700 -1 osd.163 18131 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:52:10.619734)
2014-04-14 18:52:30.767241 7fe8c6c09700 -1 osd.163 18131 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:52:10.767238)
2014-04-14 18:52:31.619909 7fe8dde34700 -1 osd.163 18131 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:52:11.619903)
2014-04-14 18:52:32.469702 7fe8c6c09700 -1 osd.163 18132 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:52:12.469698)
2014-04-14 18:52:32.620306 7fe8dde34700 -1 osd.163 18132 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:52:12.620303)
2014-04-14 18:52:33.620481 7fe8dde34700 -1 osd.163 18133 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:52:13.620474)
2014-04-14 18:52:34.172145 7fe8c6c09700 -1 osd.163 18134 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:51:31.548773 (cutoff 2014-04-14 18:52:14.172141)
2014-04-14 18:52:34.346377 7fe89fcdb700  0 -- 192.168.1.112:0/32817 >> 192.168.1.105:6831/10060022 pipe(0x4104380 sd=450 :0 s=1 pgs=0 cs=0 l=1 c=0x691a3c0).fault
2014-04-14 18:52:44.605343 7fe8a0ceb700  0 -- 192.168.1.112:6805/3032817 >> 192.168.1.105:6834/11000988 pipe(0x4514b00 sd=145 :0 s=1 pgs=0 cs=0 l=0 c=0x46091e0).fault
2014-04-14 18:53:12.627971 7fe8dde34700 -1 osd.163 18168 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:52:52.627965)
2014-04-14 18:53:13.103357 7fe8c6c09700 -1 osd.163 18169 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:52:53.103353)
2014-04-14 18:53:13.628154 7fe8dde34700 -1 osd.163 18169 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:52:53.628147)
2014-04-14 18:53:14.628330 7fe8dde34700 -1 osd.163 18169 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:52:54.628325)
2014-04-14 18:53:15.628470 7fe8dde34700 -1 osd.163 18170 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:52:55.628464)
2014-04-14 18:53:16.628937 7fe8dde34700 -1 osd.163 18171 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:52:56.628933)
2014-04-14 18:53:17.629365 7fe8dde34700 -1 osd.163 18172 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:52:57.629363)
2014-04-14 18:53:17.805755 7fe8c6c09700 -1 osd.163 18172 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:52:57.805751)
2014-04-14 18:53:18.629514 7fe8dde34700 -1 osd.163 18173 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:52:58.629508)
2014-04-14 18:53:19.629867 7fe8dde34700 -1 osd.163 18174 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:52:59.629864)
2014-04-14 18:53:20.630215 7fe8dde34700 -1 osd.163 18175 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:00.630211)
2014-04-14 18:53:21.630378 7fe8dde34700 -1 osd.163 18176 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:01.630371)
2014-04-14 18:53:22.630742 7fe8dde34700 -1 osd.163 18177 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:02.630739)
2014-04-14 18:53:23.108354 7fe8c6c09700 -1 osd.163 18177 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:03.108350)
2014-04-14 18:53:23.610856 7fe8c6c09700 -1 osd.163 18178 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:03.610852)
2014-04-14 18:53:23.630897 7fe8dde34700 -1 osd.163 18178 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:03.630894)
2014-04-14 18:53:24.631055 7fe8dde34700 -1 osd.163 18178 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:04.631049)
2014-04-14 18:53:25.313754 7fe8c6c09700 -1 osd.163 18178 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:05.313750)
2014-04-14 18:53:25.631223 7fe8dde34700 -1 osd.163 18178 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:05.631219)
2014-04-14 18:53:26.631373 7fe8dde34700 -1 osd.163 18178 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:06.631367)
2014-04-14 18:53:27.631607 7fe8dde34700 -1 osd.163 18179 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:07.631605)
2014-04-14 18:53:28.631818 7fe8dde34700 -1 osd.163 18180 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:08.631813)
2014-04-14 18:53:29.632280 7fe8dde34700 -1 osd.163 18181 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:09.632277)
2014-04-14 18:53:30.016239 7fe8c6c09700 -1 osd.163 18181 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:10.016176)
2014-04-14 18:53:30.632650 7fe8dde34700 -1 osd.163 18182 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:10.632646)
2014-04-14 18:53:31.632798 7fe8dde34700 -1 osd.163 18183 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:11.632791)
2014-04-14 18:53:32.632999 7fe8dde34700 -1 osd.163 18183 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:12.632992)
2014-04-14 18:53:33.633121 7fe8dde34700 -1 osd.163 18183 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:13.633114)
2014-04-14 18:53:34.633310 7fe8dde34700 -1 osd.163 18183 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:14.633303)
2014-04-14 18:53:35.318740 7fe8c6c09700 -1 osd.163 18183 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:15.318736)
2014-04-14 18:53:35.633462 7fe8dde34700 -1 osd.163 18183 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:15.633457)
2014-04-14 18:53:36.633849 7fe8dde34700 -1 osd.163 18184 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:16.633846)
2014-04-14 18:53:37.634082 7fe8dde34700 -1 osd.163 18185 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:17.634079)
2014-04-14 18:53:38.634372 7fe8dde34700 -1 osd.163 18186 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:18.634369)
2014-04-14 18:53:38.821204 7fe8c6c09700 -1 osd.163 18186 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:18.821176)
2014-04-14 18:53:39.323679 7fe8c6c09700 -1 osd.163 18187 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:19.323675)
2014-04-14 18:53:39.634749 7fe8dde34700 -1 osd.163 18187 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:19.634746)
2014-04-14 18:53:40.426168 7fe8c6c09700 -1 osd.163 18188 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:20.426165)
2014-04-14 18:53:40.634945 7fe8dde34700 -1 osd.163 18188 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:20.634941)
2014-04-14 18:53:40.928710 7fe8c6c09700 -1 osd.163 18188 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:20.928646)
2014-04-14 18:53:41.635148 7fe8dde34700 -1 osd.163 18188 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:21.635141)
2014-04-14 18:53:41.988075 7fe8a0bea700  0 -- 192.168.1.112:6805/3032817 >> 192.168.1.113:6834/5063888 pipe(0x6ab1b80 sd=219 :6805 s=2 pgs=923 cs=1 l=0 c=0x3f14d00).fault with nothing to send, going to standby
2014-04-14 18:53:42.635422 7fe8dde34700 -1 osd.163 18189 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:22.635418)
2014-04-14 18:53:43.635742 7fe8dde34700 -1 osd.163 18190 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:23.635739)
2014-04-14 18:53:43.831089 7fe8c6c09700 -1 osd.163 18190 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:23.831086)
2014-04-14 18:53:44.333505 7fe8c6c09700 -1 osd.163 18191 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:24.333501)
2014-04-14 18:53:44.635925 7fe8dde34700 -1 osd.163 18191 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:24.635920)
2014-04-14 18:53:45.636119 7fe8dde34700 -1 osd.163 18192 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:25.636113)
2014-04-14 18:53:46.036037 7fe8c6c09700 -1 osd.163 18192 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:26.036034)
2014-04-14 18:53:46.636275 7fe8dde34700 -1 osd.163 18193 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:26.636269)
2014-04-14 18:53:47.636592 7fe8dde34700 -1 osd.163 18194 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:27.636589)
2014-04-14 18:53:48.636716 7fe8dde34700 -1 osd.163 18195 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:28.636712)
2014-04-14 18:53:49.636918 7fe8dde34700 -1 osd.163 18196 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:29.636915)
2014-04-14 18:53:50.637123 7fe8dde34700 -1 osd.163 18197 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:30.637117)
2014-04-14 18:53:51.340347 7fe8c6c09700 -1 osd.163 18197 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:31.338605)
2014-04-14 18:53:51.637305 7fe8dde34700 -1 osd.163 18197 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:31.637300)
2014-04-14 18:53:52.637462 7fe8dde34700 -1 osd.163 18198 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:32.637456)
2014-04-14 18:53:53.637780 7fe8dde34700 -1 osd.163 18199 heartbeat_check: no reply from osd.51 ever on either front or back, first ping sent 2014-04-14 18:52:52.184574 (cutoff 2014-04-14 18:53:33.637777)
2014-04-14 18:53:54.624337 7fe8aeecc700  0 -- 192.168.1.112:0/32817 >> 192.168.1.105:6820/11060022 pipe(0x51c6400 sd=295 :0 s=1 pgs=0 cs=0 l=1 c=0x4e52d60).fault
2014-04-14 18:54:36.710730 7fe89cdac700  0 -- 192.168.1.112:6805/3032817 >> 192.168.1.109:6816/4056966 pipe(0x6ab4100 sd=240 :6805 s=2 pgs=939 cs=1 l=0 c=0x45b5d80).fault with nothing to send, going to standby
2014-04-14 18:54:48.825027 7fe8a01e0700  0 -- 192.168.1.112:6805/3032817 >> 192.168.1.105:6811/13002701 pipe(0x4dfd000 sd=32 :6805 s=2 pgs=2082 cs=1 l=0 c=0x45b11e0).fault with nothing to send, going to standby
2014-04-14 18:55:31.909562 7fe8b4d2a700  0 -- 192.168.1.112:6805/3032817 >> 192.168.1.106:6824/4004240 pipe(0x56ff800 sd=390 :6805 s=2 pgs=974 cs=1 l=0 c=0x4e20c60).fault with nothing to send, going to standby
2014-04-14 18:55:31.912494 7fe8c0ee2700  0 -- 192.168.1.112:6805/3032817 >> 192.168.1.110:6818/1042305 pipe(0x6ab5500 sd=107 :35463 s=2 pgs=454 cs=1 l=0 c=0x406f7a0).fault with nothing to send, going to standby
2014-04-14 18:55:42.055199 7fe8a5d3b700  0 -- 192.168.1.112:6805/3032817 >> 192.168.1.105:6820/13062372 pipe(0x4da6400 sd=158 :6805 s=2 pgs=2180 cs=1 l=0 c=0x455b700).fault with nothing to send, going to standby
2014-04-14 18:55:42.055903 7fe89d3b2700  0 -- 192.168.1.112:6805/3032817 >> 192.168.1.109:6807/5056966 pipe(0x56fa800 sd=311 :6805 s=2 pgs=1062 cs=1 l=0 c=0x694eca0).fault with nothing to send, going to standby
2014-04-14 18:55:42.056550 7fe8ad1af700  0 -- 192.168.1.112:6805/3032817 >> 192.168.1.103:6806/1031885 pipe(0x3f7be80 sd=257 :6805 s=2 pgs=543 cs=1 l=0 c=0x3f11600).fault with nothing to send, going to standby
2014-04-14 18:55:42.056722 7fe8d201b700  0 log [WRN] : map e18290 wrongly marked me down
2014-04-14 18:55:42.056798 7fe8b8663700  0 -- 192.168.1.112:6805/3032817 >> 192.168.1.101:6825/6022662 pipe(0x4106b80 sd=95 :6805 s=2 pgs=1300 cs=1 l=0 c=0x694ac00).fault with nothing to send, going to standby
2014-04-14 18:55:42.056915 7fe89d0af700  0 -- 192.168.1.112:6805/3032817 >> 192.168.1.101:6810/6029366 pipe(0x4da7a80 sd=364 :6805 s=2 pgs=1229 cs=1 l=0 c=0x455aec0).fault with nothing to send, going to standby
2014-04-14 18:55:42.057300 7fe89c7a6700  0 -- 192.168.1.112:6805/3032817 >> 192.168.1.106:6808/4011682 pipe(0x6ab3c00 sd=409 :6805 s=2 pgs=1020 cs=1 l=0 c=0x45b4780).fault with nothing to send, going to standby
2014-04-14 18:55:42.058007 7fe89f8d7700  0 -- 192.168.1.112:6805/3032817 >> 192.168.1.108:6820/5002228 pipe(0x51c2d00 sd=360 :6805 s=2 pgs=987 cs=1 l=0 c=0x8360dc0).fault with nothing to send, going to standby
2014-04-14 18:55:42.057436 7fe8b3311700  0 -- 192.168.1.112:6805/3032817 >> 192.168.1.113:6812/3001582 pipe(0x3f7b700 sd=124 :6805 s=2 pgs=762 cs=1 l=0 c=0x4daaec0).fault with nothing to send, going to standby
2014-04-14 18:55:42.058040 7fe8a9371700  0 -- 192.168.1.112:6805/3032817 >> 192.168.1.105:6817/14002701 pipe(0x4dfdf00 sd=164 :6805 s=2 pgs=2223 cs=1 l=0 c=0x4e211e0).fault with nothing to send, going to standby
2014-04-14 18:55:42.058067 7fe8bbc90700  0 -- 192.168.1.112:6805/3032817 >> 192.168.1.109:6804/7055614 pipe(0x4da0a00 sd=273 :6805 s=2 pgs=1422 cs=1 l=0 c=0x4559e40).fault with nothing to send, going to standby
2014-04-14 18:55:42.058281 7fe89eecd700  0 -- 192.168.1.112:6805/3032817 >> 192.168.1.103:6822/5033047 pipe(0x56fa080 sd=22 :6805 s=2 pgs=932 cs=1 l=0 c=0x45f8dc0).fault with nothing to send, going to standby
2014-04-14 18:55:42.058942 7fe8a8f6d700  0 -- 192.168.1.112:6805/3032817 >> 192.168.1.102:6804/4009966 pipe(0x4df9b80 sd=346 :6805 s=2 pgs=935 cs=1 l=0 c=0x406ce60).fault with nothing to send, going to standby
2014-04-14 18:55:42.059094 7fe8be2b6700  0 -- 192.168.1.112:6805/3032817 >> 192.168.1.103:6815/2039519 pipe(0x6ab7300 sd=141 :57968 s=2 pgs=699 cs=1 l=0 c=0x6919b80).fault with nothing to send, going to standby
2014-04-14 18:56:03.878064 7fe8c0bdf700  0 -- 192.168.1.112:6801/4032817 >> 192.168.1.108:6806/3000970 pipe(0x4514100 sd=123 :60956 s=2 pgs=1010 cs=1 l=0 c=0x406dd80).fault with nothing to send, going to standby
2014-04-14 18:56:16.289779 7fe8c16ea700  0 -- 192.168.1.112:6801/4032817 >> 192.168.1.111:6831/4012002 pipe(0x56fd500 sd=110 :41131 s=2 pgs=1087 cs=1 l=0 c=0x45fd120).fault with nothing to send, going to standby
2014-04-14 18:56:32.789640 7fe8dea7b700  0 -- 192.168.1.112:6801/4032817 >> 192.168.1.105:6803/10003300 pipe(0x56fb700 sd=92 :35927 s=2 pgs=1801 cs=1 l=0 c=0x45ff220).fault with nothing to send, going to standby

LOGS FROM osd.51

[root@storage0105-ib ceph]# tail -100f ceph-osd.51.log
2014-04-14 18:58:42.107780 7f5ceb81e700  0 -- 192.168.1.105:6821/14060022 >> 192.168.1.106:6800/5004240 pipe(0x7719e00 sd=215 :37938 s=2 pgs=1179 cs=1 l=0 c=0x4c11fa0).fault with nothing to send, going to standby
2014-04-14 18:58:42.108366 7f5ceec4b700  0 -- 192.168.1.105:6821/14060022 >> 192.168.1.113:6826/7063888 pipe(0x4d12a80 sd=180 :47643 s=2 pgs=1385 cs=1 l=0 c=0x4b561a0).fault with nothing to send, going to standby
2014-04-14 18:58:42.108411 7f5cf0867700  0 -- 192.168.1.105:6821/14060022 >> 192.168.1.113:6802/5064877 pipe(0x4d11180 sd=163 :59031 s=2 pgs=1199 cs=1 l=0 c=0x5979fa0).fault with nothing to send, going to standby
2014-04-14 18:58:42.109075 7f5ce8f1f700  0 -- 192.168.1.105:6821/14060022 >> 192.168.1.106:6816/3002245 pipe(0x537e180 sd=249 :6821 s=2 pgs=2922 cs=1 l=0 c=0xa3dc360).fault with nothing to send, going to standby
2014-04-14 18:58:42.109948 7f5cf3291700  0 -- 192.168.1.105:6821/14060022 >> 192.168.1.107:6811/1009536 pipe(0x7bbdf00 sd=123 :56678 s=2 pgs=1846 cs=1 l=0 c=0x4fab2e0).fault with nothing to send, going to standby
2014-04-14 18:58:42.109997 7f5cf4aaa700  0 -- 192.168.1.105:6821/14060022 >> 192.168.1.104:6814/5049263 pipe(0x4e97580 sd=103 :36440 s=2 pgs=1128 cs=1 l=0 c=0x4fa8dc0).fault with nothing to send, going to standby
2014-04-14 18:58:42.110075 7f5cd40a8700  0 -- 192.168.1.105:6821/14060022 >> 192.168.1.114:6815/6007089 pipe(0x429c380 sd=55 :6821 s=2 pgs=1289 cs=1 l=0 c=0x597c780).fault with nothing to send, going to standby
2014-04-14 18:58:42.110765 7f5ce9424700  0 -- 192.168.1.105:6821/14060022 >> 192.168.1.112:6816/5026017 pipe(0x4d12080 sd=247 :6821 s=2 pgs=1198 cs=1 l=0 c=0x4faa100).fault with nothing to send, going to standby
2014-04-14 18:58:42.110797 7f5ceab11700  0 -- 192.168.1.105:6821/14060022 >> 192.168.1.108:6817/2007814 pipe(0x429d780 sd=232 :44225 s=2 pgs=928 cs=1 l=0 c=0x54a89a0).fault with nothing to send, going to standby
2014-04-14 18:58:42.111262 7f5cf3595700  0 -- 192.168.1.105:6821/14060022 >> 192.168.1.114:6802/6008433 pipe(0x7bbdc80 sd=122 :38880 s=2 pgs=1212 cs=1 l=0 c=0x4fae5c0).fault with nothing to send, going to standby
2014-04-14 18:58:42.111351 7f5cee847700  0 -- 192.168.1.105:6821/14060022 >> 192.168.1.111:6803/4009103 pipe(0x4d13480 sd=184 :43928 s=2 pgs=1011 cs=1 l=0 c=0x4b544c0).fault with nothing to send, going to standby
2014-04-14 18:58:42.113308 7f5cf58b7700  0 -- 192.168.1.105:6821/14060022 >> 192.168.1.102:6824/15636 pipe(0xaeec380 sd=87 :46747 s=2 pgs=694 cs=1 l=0 c=0x77bc620).fault with nothing to send, going to standby
2014-04-14 18:58:42.113372 7f5cf1675700  0 -- 192.168.1.105:6821/14060022 >> 192.168.1.115:6814/8009855 pipe(0x7bbd280 sd=152 :46908 s=2 pgs=1478 cs=1 l=0 c=0x6475540).fault with nothing to send, going to standby
2014-04-14 18:58:42.113410 7f5cf48a8700  0 -- 192.168.1.105:6821/14060022 >> 192.168.1.103:6800/2034753 pipe(0x4e95f00 sd=108 :52707 s=2 pgs=928 cs=1 l=0 c=0x4c170c0).fault with nothing to send, going to standby
2014-04-14 18:58:42.113775 7f5cef958700  0 -- 192.168.1.105:6821/14060022 >> 192.168.1.108:6802/3005178 pipe(0x771bc00 sd=173 :49030 s=2 pgs=967 cs=1 l=0 c=0xa3dca40).fault with nothing to send, going to standby
2014-04-14 18:58:42.114732 7f5cedd3c700  0 -- 192.168.1.105:6821/14060022 >> 192.168.1.112:6832/8034975 pipe(0x771f080 sd=314 :6821 s=2 pgs=1382 cs=1 l=0 c=0xb2cf640).fault with nothing to send, going to standby
2014-04-14 18:58:42.115240 7f5cf1f7e700  0 -- 192.168.1.105:6821/14060022 >> 192.168.1.101:6803/6017061 pipe(0x7bb9400 sd=144 :56930 s=2 pgs=3157 cs=1 l=0 c=0x4fae880).fault with nothing to send, going to standby
2014-04-14 18:58:43.138088 7f5ce9828700  0 -- 192.168.1.105:6819/15060022 >> 192.168.1.106:6816/3002245 pipe(0x537f800 sd=248 :6819 s=0 pgs=0 cs=0 l=0 c=0x77bad60).accept connect_seq 0 vs existing 0 state connecting
2014-04-14 18:58:43.138358 7f5ce9727700  0 -- 192.168.1.105:6819/15060022 >> 192.168.1.112:6801/3034109 pipe(0x537b200 sd=249 :6819 s=0 pgs=0 cs=0 l=0 c=0x77bf380).accept connect_seq 0 vs existing 0 state connecting
2014-04-14 18:58:43.138470 7f5ce9a2a700  0 -- 192.168.1.105:6819/15060022 >> 192.168.1.103:6801/5036094 pipe(0x4d39b80 sd=244 :6819 s=0 pgs=0 cs=0 l=0 c=0x4e8e5c0).accept connect_seq 0 vs existing 0 state connecting
2014-04-14 18:58:43.147678 7f5ce9020700  0 -- 192.168.1.105:6819/15060022 >> 192.168.1.112:6806/5032817 pipe(0x4e90a00 sd=255 :6819 s=0 pgs=0 cs=0 l=0 c=0x77b8420).accept connect_seq 0 vs existing 0 state connecting
2014-04-14 18:58:43.147827 7f5ce7deb700  0 -- 192.168.1.105:6819/15060022 >> 192.168.1.102:6823/11011704 pipe(0x4e97d00 sd=263 :6819 s=0 pgs=0 cs=0 l=0 c=0x77bd3e0).accept connect_seq 0 vs existing 0 state connecting
2014-04-14 18:58:43.148016 7f5ce7be9700  0 -- 192.168.1.105:6819/15060022 >> 192.168.1.101:6807/3018425 pipe(0x4e95000 sd=266 :6819 s=0 pgs=0 cs=0 l=0 c=0x77bd6a0).accept connect_seq 0 vs existing 0 state connecting
2014-04-14 18:58:43.148036 7f5ce7cea700  0 -- 192.168.1.105:6819/15060022 >> 192.168.1.104:6807/7061463 pipe(0x4e91900 sd=264 :6819 s=0 pgs=0 cs=0 l=0 c=0x77baaa0).accept connect_seq 0 vs existing 0 state connecting
2014-04-14 18:58:43.148081 7f5ce72e0700  0 -- 192.168.1.105:6819/15060022 >> 192.168.1.111:6803/4009103 pipe(0x4e97300 sd=274 :6819 s=0 pgs=0 cs=0 l=0 c=0x5a274e0).accept connect_seq 0 vs existing 0 state connecting
2014-04-14 18:58:43.148149 7f5ce8d1d700  0 -- 192.168.1.105:6819/15060022 >> 192.168.1.112:6829/1023135 pipe(0x4e95500 sd=260 :6819 s=0 pgs=0 cs=0 l=0 c=0x77b89a0).accept connect_seq 0 vs existing 0 state connecting
2014-04-14 18:58:43.148472 7f5ce6fdd700  0 -- 192.168.1.105:6819/15060022 >> 192.168.1.107:6829/3004265 pipe(0x4e91b80 sd=278 :6819 s=0 pgs=0 cs=0 l=0 c=0x4c16460).accept connect_seq 0 vs existing 0 state connecting
2014-04-14 18:58:43.149670 7f5ce7ae8700  0 -- 192.168.1.105:6819/15060022 >> 192.168.1.108:6811/3010125 pipe(0x4e94b00 sd=267 :6819 s=0 pgs=0 cs=0 l=0 c=0x59ee5c0).accept connect_seq 0 vs existing 0 state connecting
2014-04-14 18:58:43.158744 7f5ce9929700  0 -- 192.168.1.105:6819/15060022 >> 192.168.1.104:6817/7054057 pipe(0x4d3ee00 sd=246 :6819 s=0 pgs=0 cs=0 l=0 c=0x77bcba0).accept connect_seq 0 vs existing 0 state connecting
2014-04-14 18:59:02.414643 7f5d0ccab700 -1 osd.51 18440 heartbeat_check: no reply from osd.52 ever on either front or back, first ping sent 2014-04-14 18:58:42.164242 (cutoff 2014-04-14 18:58:42.414639)
2014-04-14 18:59:02.481970 7f5cf9bd0700 -1 osd.51 18440 heartbeat_check: no reply from osd.52 ever on either front or back, first ping sent 2014-04-14 18:58:42.164242 (cutoff 2014-04-14 18:58:42.481968)
2014-04-14 18:59:03.414829 7f5d0ccab700 -1 osd.51 18441 heartbeat_check: no reply from osd.52 ever on either front or back, first ping sent 2014-04-14 18:58:42.164242 (cutoff 2014-04-14 18:58:43.414827)
2014-04-14 18:59:0

History

#1 Updated by karan singh almost 10 years ago

I have been searching on internet and found there has been similar kind of bugs came up in past as well and fixes were created for them.

http://tracker.ceph.com/issues/5172

http://tracker.ceph.com/issues/5460

Looks like Version 0.79 also need some fixes.

#2 Updated by Sage Weil almost 10 years ago

This is usually caused by low memory leading to swapping. Can you verify the CPU and memory are not oversubscribed?

#3 Updated by karan singh almost 10 years ago

Thanks for your interest Sage , today morning the cluster looks normal and healthy. Unfortunately i do not have system performance logs at the time of problem , but now looks like overnight system memory got released , if we consider this a problem was due to CPU and memory.

#4 Updated by Sage Weil almost 10 years ago

  • Status changed from New to Can't reproduce

Thanks for the follow-up. Please let us know if you can figure out how to reproduce the problem, or can gather more information when it happens again.

Also available in: Atom PDF