Ceph

env
system:centos7.1.1503 x64
kernel:3.10.0-229.el7.x86_64
ceph version:0.94.10

error system log: /var/log/messages
May 22 22:17:33 wx-ceph-p70-219 bash: common/HeartbeatMap.cc: In function 'bool ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d*, const char*, time_t)' thread 7fb4dfceb700 time 2017-05-22 22:17:33.235759
May 22 22:17:33 wx-ceph-p70-219 bash: common/HeartbeatMap.cc: 79: FAILED assert(0 "hit suicide timeout")
May 22 22:17:33 wx-ceph-p70-219 bash: ceph version 0.94.10 (b1e0532418e4631af01acbc0cedd426f1905f4af)
May 22 22:17:33 wx-ceph-p70-219 bash: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0xbdf735]
May 22 22:17:33 wx-ceph-p70-219 bash: 2: (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d*, char const*, long)+0x2d9) [0xb12c89]
May 22 22:17:33 wx-ceph-p70-219 bash: 3: (ceph::HeartbeatMap::is_healthy()+0xde) [0xb1357e]
May 22 22:17:33 wx-ceph-p70-219 bash: 4: (ceph::HeartbeatMap::check_touch_file()+0x2c) [0xb13c9c]
May 22 22:17:33 wx-ceph-p70-219 bash: 5: (CephContextServiceThread::entry()+0x15b) [0xbefb7b]
May 22 22:17:33 wx-ceph-p70-219 bash: 6: (()+0x7df5) [0x7fb4e32ebdf5]
May 22 22:17:33 wx-ceph-p70-219 bash: 7: (clone()+0x6d) [0x7fb4e1dce1ad]
May 22 22:17:33 wx-ceph-p70-219 bash: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
May 22 22:17:33 wx-ceph-p70-219 bash: 2017-05-22 22:17:33.254203 7fb4dfceb700 -1 common/HeartbeatMap.cc: In function 'bool ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d*, const char*, time_t)' thread 7fb4dfceb700 time 2017-05-22 22:17:33.235759
May 22 22:17:33 wx-ceph-p70-219 bash: common/HeartbeatMap.cc: 79: FAILED assert(0 "hit suicide timeout")
May 22 22:17:33 wx-ceph-p70-219 bash: ceph version 0.94.10 (b1e0532418e4631af01acbc0cedd426f1905f4af)
May 22 22:17:33 wx-ceph-p70-219 bash: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0xbdf735]
May 22 22:17:33 wx-ceph-p70-219 bash: 2: (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d*, char const*, long)+0x2d9) [0xb12c89]
May 22 22:17:33 wx-ceph-p70-219 bash: 3: (ceph::HeartbeatMap::is_healthy()+0xde) [0xb1357e]
May 22 22:17:33 wx-ceph-p70-219 bash: 4: (ceph::HeartbeatMap::check_touch_file()+0x2c) [0xb13c9c]
May 22 22:17:33 wx-ceph-p70-219 bash: 5: (CephContextServiceThread::entry()+0x15b) [0xbefb7b]
May 22 22:17:33 wx-ceph-p70-219 bash: 6: (()+0x7df5) [0x7fb4e32ebdf5]
May 22 22:17:33 wx-ceph-p70-219 bash: 7: (clone()+0x6d) [0x7fb4e1dce1ad]
May 22 22:17:33 wx-ceph-p70-219 bash: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
May 22 22:17:33 wx-ceph-p70-219 bash: -5063> 2017-05-22 22:15:26.833345 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.37 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:15:06.833343)
May 22 22:17:33 wx-ceph-p70-219 bash: -5062> 2017-05-22 22:15:26.833371 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.40 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:15:06.833343)
May 22 22:17:33 wx-ceph-p70-219 bash: -4915> 2017-05-22 22:15:30.934816 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.37 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:15:10.934814)
May 22 22:17:33 wx-ceph-p70-219 bash: -4914> 2017-05-22 22:15:30.934833 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.40 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:15:10.934814)
May 22 22:17:33 wx-ceph-p70-219 bash: -4791> 2017-05-22 22:15:33.235743 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.37 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:15:13.235742)
May 22 22:17:33 wx-ceph-p70-219 bash: -4790> 2017-05-22 22:15:33.235753 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.40 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:15:13.235742)
May 22 22:17:33 wx-ceph-p70-219 bash: -4658> 2017-05-22 22:15:34.936892 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.37 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:15:14.936862)
May 22 22:17:33 wx-ceph-p70-219 bash: -4657> 2017-05-22 22:15:34.936916 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.40 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:15:14.936862)
May 22 22:17:33 wx-ceph-p70-219 bash: -4561> 2017-05-22 22:15:35.438020 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.37 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:15:15.438019)
May 22 22:17:33 wx-ceph-p70-219 bash: -4560> 2017-05-22 22:15:35.438038 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.40 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:15:15.438019)
May 22 22:17:33 wx-ceph-p70-219 bash: -4464> 2017-05-22 22:15:36.539053 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.37 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:15:16.539051)
May 22 22:17:33 wx-ceph-p70-219 bash: -4463> 2017-05-22 22:15:36.539067 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.40 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:15:16.539051)
May 22 22:17:33 wx-ceph-p70-219 bash: -4049> 2017-05-22 22:15:42.440001 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.37 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:15:22.439999)
May 22 22:17:33 wx-ceph-p70-219 bash: -4048> 2017-05-22 22:15:42.440017 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.40 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:15:22.439999)
May 22 22:17:33 wx-ceph-p70-219 bash: -3910> 2017-05-22 22:15:47.740966 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.37 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:15:27.740965)
May 22 22:17:33 wx-ceph-p70-219 bash: -3909> 2017-05-22 22:15:47.740976 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.40 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:15:27.740965)
May 22 22:17:33 wx-ceph-p70-219 bash: -3778> 2017-05-22 22:15:51.242019 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.37 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:15:31.241992)
May 22 22:17:33 wx-ceph-p70-219 bash: -3777> 2017-05-22 22:15:51.242035 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.40 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:15:31.241992)
May 22 22:17:33 wx-ceph-p70-219 bash: -3655> 2017-05-22 22:15:55.943972 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.37 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:15:35.943970)
May 22 22:17:33 wx-ceph-p70-219 bash: -3654> 2017-05-22 22:15:55.943996 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.40 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:15:35.943970)
May 22 22:17:33 wx-ceph-p70-219 bash: -3505> 2017-05-22 22:16:01.245043 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.37 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:15:41.245041)
May 22 22:17:33 wx-ceph-p70-219 bash: -3504> 2017-05-22 22:16:01.245058 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.40 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:15:41.245041)
May 22 22:17:33 wx-ceph-p70-219 bash: -3388> 2017-05-22 22:16:01.746178 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.37 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:15:41.746177)
May 22 22:17:33 wx-ceph-p70-219 bash: -3387> 2017-05-22 22:16:01.746193 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.40 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:15:41.746177)
May 22 22:17:33 wx-ceph-p70-219 bash: -3241> 2017-05-22 22:16:04.047567 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.37 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:15:44.047565)
May 22 22:17:33 wx-ceph-p70-219 bash: -3240> 2017-05-22 22:16:04.047587 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.40 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:15:44.047565)
May 22 22:17:33 wx-ceph-p70-219 bash: -3118> 2017-05-22 22:16:05.148620 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.37 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:15:45.148619)
May 22 22:17:33 wx-ceph-p70-219 bash: -3117> 2017-05-22 22:16:05.148634 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.40 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:15:45.148619)
May 22 22:17:33 wx-ceph-p70-219 bash: -2989> 2017-05-22 22:16:06.849781 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.37 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:15:46.849778)
May 22 22:17:33 wx-ceph-p70-219 bash: -2988> 2017-05-22 22:16:06.849802 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.40 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:15:46.849778)
May 22 22:17:33 wx-ceph-p70-219 bash: -2863> 2017-05-22 22:16:12.751198 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.37 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:15:52.751197)
May 22 22:17:33 wx-ceph-p70-219 bash: -2862> 2017-05-22 22:16:12.751212 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.40 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:15:52.751197)
May 22 22:17:33 wx-ceph-p70-219 bash: -2738> 2017-05-22 22:16:18.652599 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.37 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:15:58.652597)
May 22 22:17:33 wx-ceph-p70-219 bash: -2737> 2017-05-22 22:16:18.652617 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.40 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:15:58.652597)
May 22 22:17:33 wx-ceph-p70-219 bash: -2605> 2017-05-22 22:16:22.153819 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.37 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:16:02.153817)
May 22 22:17:33 wx-ceph-p70-219 bash: -2604> 2017-05-22 22:16:22.153836 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.40 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:16:02.153817)
May 22 22:17:33 wx-ceph-p70-219 bash: -2480> 2017-05-22 22:16:25.655265 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.37 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:16:05.655264)
May 22 22:17:33 wx-ceph-p70-219 bash: -2479> 2017-05-22 22:16:25.655281 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.40 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:16:05.655264)
May 22 22:17:33 wx-ceph-p70-219 bash: -2357> 2017-05-22 22:16:29.756546 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.37 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:16:09.756544)
May 22 22:17:33 wx-ceph-p70-219 bash: -2356> 2017-05-22 22:16:29.756560 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.40 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:16:09.756544)
May 22 22:17:33 wx-ceph-p70-219 bash: -2226> 2017-05-22 22:16:34.457624 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.37 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:16:14.457604)
May 22 22:17:33 wx-ceph-p70-219 bash: -2225> 2017-05-22 22:16:34.457644 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.40 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:16:14.457604)
May 22 22:17:33 wx-ceph-p70-219 bash: -2103> 2017-05-22 22:16:36.758890 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.37 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:16:16.758889)
May 22 22:17:33 wx-ceph-p70-219 bash: -2102> 2017-05-22 22:16:36.758911 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.40 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:16:16.758889)
May 22 22:17:33 wx-ceph-p70-219 bash: -1980> 2017-05-22 22:16:37.260005 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.37 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:16:17.260003)
May 22 22:17:33 wx-ceph-p70-219 bash: -1979> 2017-05-22 22:16:37.260020 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.40 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:16:17.260003)
May 22 22:17:33 wx-ceph-p70-219 bash: -1849> 2017-05-22 22:16:40.161327 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.37 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:16:20.161326)
May 22 22:17:33 wx-ceph-p70-219 bash: -1848> 2017-05-22 22:16:40.161346 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.40 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:16:20.161326)
May 22 22:17:33 wx-ceph-p70-219 bash: -1722> 2017-05-22 22:16:46.062620 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.37 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:16:26.062617)
May 22 22:17:33 wx-ceph-p70-219 bash: -1721> 2017-05-22 22:16:46.062636 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.40 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:16:26.062617)
May 22 22:17:33 wx-ceph-p70-219 bash: -1591> 2017-05-22 22:16:50.763903 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.37 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:16:30.763901)
May 22 22:17:33 wx-ceph-p70-219 bash: -1590> 2017-05-22 22:16:50.763920 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.40 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:16:30.763901)
May 22 22:17:33 wx-ceph-p70-219 bash: -1468> 2017-05-22 22:16:54.265066 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.37 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:16:34.265064)
May 22 22:17:33 wx-ceph-p70-219 bash: -1467> 2017-05-22 22:16:54.265078 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.40 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:16:34.265064)
May 22 22:17:33 wx-ceph-p70-219 bash: -1335> 2017-05-22 22:16:57.166358 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.37 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:16:37.166356)
May 22 22:17:33 wx-ceph-p70-219 bash: -1334> 2017-05-22 22:16:57.166373 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.40 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:16:37.166356)
May 22 22:17:33 wx-ceph-p70-219 bash: -1206> 2017-05-22 22:17:00.667559 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.37 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:16:40.667557)
May 22 22:17:33 wx-ceph-p70-219 bash: -1205> 2017-05-22 22:17:00.667575 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.40 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:16:40.667557)
May 22 22:17:33 wx-ceph-p70-219 bash: -1083> 2017-05-22 22:17:01.168635 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.37 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:16:41.168634)
May 22 22:17:33 wx-ceph-p70-219 bash: -1082> 2017-05-22 22:17:01.168647 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.40 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:16:41.168634)
May 22 22:17:33 wx-ceph-p70-219 bash: -960> 2017-05-22 22:17:05.269842 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.37 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:16:45.269840)
May 22 22:17:33 wx-ceph-p70-219 bash: -959> 2017-05-22 22:17:05.269866 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.40 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:16:45.269840)
May 22 22:17:33 wx-ceph-p70-219 bash: -837> 2017-05-22 22:17:09.970828 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.37 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:16:49.970826)
May 22 22:17:33 wx-ceph-p70-219 bash: -836> 2017-05-22 22:17:09.970843 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.40 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:16:49.970826)
May 22 22:17:33 wx-ceph-p70-219 bash: -706> 2017-05-22 22:17:11.072006 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.37 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:16:51.071999)
May 22 22:17:33 wx-ceph-p70-219 bash: -705> 2017-05-22 22:17:11.072022 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.40 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:16:51.071999)
May 22 22:17:33 wx-ceph-p70-219 bash: -579> 2017-05-22 22:17:15.773491 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.37 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:16:55.773489)
May 22 22:17:33 wx-ceph-p70-219 bash: -578> 2017-05-22 22:17:15.773505 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.40 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:16:55.773489)
May 22 22:17:33 wx-ceph-p70-219 bash: -456> 2017-05-22 22:17:19.874709 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.37 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:16:59.874707)
May 22 22:17:33 wx-ceph-p70-219 bash: -455> 2017-05-22 22:17:19.874721 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.40 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:16:59.874707)
May 22 22:17:33 wx-ceph-p70-219 bash: -325> 2017-05-22 22:17:23.975757 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.37 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:17:03.975755)
May 22 22:17:33 wx-ceph-p70-219 bash: -324> 2017-05-22 22:17:23.975771 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.40 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:17:03.975755)
May 22 22:17:33 wx-ceph-p70-219 bash: -202> 2017-05-22 22:17:25.676821 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.37 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:17:05.676820)
May 22 22:17:33 wx-ceph-p70-219 bash: -201> 2017-05-22 22:17:25.676833 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.40 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:17:05.676820)
May 22 22:17:33 wx-ceph-p70-219 bash: -73> 2017-05-22 22:17:31.577894 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.37 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:17:11.577893)
May 22 22:17:33 wx-ceph-p70-219 bash: -72> 2017-05-22 22:17:31.577907 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.40 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:17:11.577893)
May 22 22:17:33 wx-ceph-p70-219 bash: 0> 2017-05-22 22:17:33.254203 7fb4dfceb700 -1 common/HeartbeatMap.cc: In function 'bool ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d*, const char*, time_t)' thread 7fb4dfceb700 time 2017-05-22 22:17:33.235759
May 22 22:17:33 wx-ceph-p70-219 bash: common/HeartbeatMap.cc: 79: FAILED assert(0 == "hit suicide timeout")
May 22 22:17:33 wx-ceph-p70-219 bash: ceph version 0.94.10 (b1e0532418e4631af01acbc0cedd426f1905f4af)
May 22 22:17:33 wx-ceph-p70-219 bash: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0xbdf735]
May 22 22:17:33 wx-ceph-p70-219 bash: 2: (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d*, char const*, long)+0x2d9) [0xb12c89]
May 22 22:17:33 wx-ceph-p70-219 bash: 3: (ceph::HeartbeatMap::is_healthy()+0xde) [0xb1357e]
May 22 22:17:33 wx-ceph-p70-219 bash: 4: (ceph::HeartbeatMap::check_touch_file()+0x2c) [0xb13c9c]
May 22 22:17:33 wx-ceph-p70-219 bash: 5: (CephContextServiceThread::entry()+0x15b) [0xbefb7b]
May 22 22:17:33 wx-ceph-p70-219 bash: 6: (()+0x7df5) [0x7fb4e32ebdf5]
May 22 22:17:33 wx-ceph-p70-219 bash: 7: (clone()+0x6d) [0x7fb4e1dce1ad]
May 22 22:17:33 wx-ceph-p70-219 bash: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
May 22 22:17:33 wx-ceph-p70-219 bash: terminate called after throwing an instance of 'ceph::FailedAssertion'
May 22 22:17:33 wx-ceph-p70-219 bash: * Caught signal (Aborted) *
May 22 22:17:33 wx-ceph-p70-219 bash: in thread 7fb4dfceb700
May 22 22:17:33 wx-ceph-p70-219 bash: ceph version 0.94.10 (b1e0532418e4631af01acbc0cedd426f1905f4af)
May 22 22:17:33 wx-ceph-p70-219 bash: 1: /usr/bin/ceph-osd() [0xadc182]
May 22 22:17:33 wx-ceph-p70-219 bash: 2: (()+0xf130) [0x7fb4e32f3130]
May 22 22:17:33 wx-ceph-p70-219 bash: 3: (gsignal()+0x37) [0x7fb4e1d0d5d7]
May 22 22:17:33 wx-ceph-p70-219 bash: 4: (abort()+0x148) [0x7fb4e1d0ecc8]
May 22 22:17:33 wx-ceph-p70-219 bash: 5: (_gnu_cxx::_verbose_terminate_handler()+0x165) [0x7fb4e26119b5]
May 22 22:17:33 wx-ceph-p70-219 bash: 6: (()+0x5e926) [0x7fb4e260f926]
May 22 22:17:33 wx-ceph-p70-219 bash: 7: (()+0x5e953) [0x7fb4e260f953]
May 22 22:17:33 wx-ceph-p70-219 bash: 8: (()+0x5eb73) [0x7fb4e260fb73]
May 22 22:17:33 wx-ceph-p70-219 bash: 9: (ceph::__ceph_assert_fail(char const, char const*, int, char const*)+0x27a) [0xbdf92a]
May 22 22:17:33 wx-ceph-p70-219 bash: 10: (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d*, char const*, long)+0x2d9) [0xb12c89]
May 22 22:17:33 wx-ceph-p70-219 bash: 11: (ceph::HeartbeatMap::is_healthy()+0xde) [0xb1357e]
May 22 22:17:33 wx-ceph-p70-219 bash: 12: (ceph::HeartbeatMap::check_touch_file()+0x2c) [0xb13c9c]
May 22 22:17:33 wx-ceph-p70-219 bash: 13: (CephContextServiceThread::entry()+0x15b) [0xbefb7b]
May 22 22:17:33 wx-ceph-p70-219 bash: 14: (()+0x7df5) [0x7fb4e32ebdf5]
May 22 22:17:33 wx-ceph-p70-219 bash: 15: (clone()+0x6d) [0x7fb4e1dce1ad]
May 22 22:17:33 wx-ceph-p70-219 bash: 2017-05-22 22:17:33.325099 7fb4dfceb700 -1 Caught signal (Aborted)
May 22 22:17:33 wx-ceph-p70-219 bash: in thread 7fb4dfceb700
May 22 22:17:33 wx-ceph-p70-219 bash: ceph version 0.94.10 (b1e0532418e4631af01acbc0cedd426f1905f4af)
May 22 22:17:33 wx-ceph-p70-219 bash: 1: /usr/bin/ceph-osd() [0xadc182]
May 22 22:17:33 wx-ceph-p70-219 bash: 2: (()+0xf130) [0x7fb4e32f3130]
May 22 22:17:33 wx-ceph-p70-219 bash: 3: (gsignal()+0x37) [0x7fb4e1d0d5d7]
May 22 22:17:33 wx-ceph-p70-219 bash: 4: (abort()+0x148) [0x7fb4e1d0ecc8]
May 22 22:17:33 wx-ceph-p70-219 bash: 5: (_gnu_cxx::_verbose_terminate_handler()+0x165) [0x7fb4e26119b5]
May 22 22:17:33 wx-ceph-p70-219 bash: 6: (()+0x5e926) [0x7fb4e260f926]
May 22 22:17:33 wx-ceph-p70-219 bash: 7: (()+0x5e953) [0x7fb4e260f953]
May 22 22:17:33 wx-ceph-p70-219 bash: 8: (()+0x5eb73) [0x7fb4e260fb73]
May 22 22:17:33 wx-ceph-p70-219 bash: 9: (ceph::__ceph_assert_fail(char const, char const*, int, char const*)+0x27a) [0xbdf92a]
May 22 22:17:33 wx-ceph-p70-219 bash: 10: (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d*, char const*, long)+0x2d9) [0xb12c89]
May 22 22:17:33 wx-ceph-p70-219 bash: 11: (ceph::HeartbeatMap::is_healthy()+0xde) [0xb1357e]
May 22 22:17:33 wx-ceph-p70-219 bash: 12: (ceph::HeartbeatMap::check_touch_file()+0x2c) [0xb13c9c]
May 22 22:17:33 wx-ceph-p70-219 bash: 13: (CephContextServiceThread::entry()+0x15b) [0xbefb7b]
May 22 22:17:33 wx-ceph-p70-219 bash: 14: (()+0x7df5) [0x7fb4e32ebdf5]
May 22 22:17:33 wx-ceph-p70-219 bash: 15: (clone()+0x6d) [0x7fb4e1dce1ad]
May 22 22:17:33 wx-ceph-p70-219 bash: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
May 22 22:17:33 wx-ceph-p70-219 bash: 0> 2017-05-22 22:17:33.325099 7fb4dfceb700 -1 Caught signal (Aborted) *
May 22 22:17:33 wx-ceph-p70-219 bash: in thread 7fb4dfceb700
May 22 22:17:33 wx-ceph-p70-219 bash: ceph version 0.94.10 (b1e0532418e4631af01acbc0cedd426f1905f4af)
May 22 22:17:33 wx-ceph-p70-219 bash: 1: /usr/bin/ceph-osd() [0xadc182]
May 22 22:17:33 wx-ceph-p70-219 bash: 2: (()+0xf130) [0x7fb4e32f3130]
May 22 22:17:33 wx-ceph-p70-219 bash: 3: (gsignal()+0x37) [0x7fb4e1d0d5d7]
May 22 22:17:33 wx-ceph-p70-219 bash: 4: (abort()+0x148) [0x7fb4e1d0ecc8]
May 22 22:17:33 wx-ceph-p70-219 bash: 5: (_gnu_cxx::_verbose_terminate_handler()+0x165) [0x7fb4e26119b5]
May 22 22:17:33 wx-ceph-p70-219 bash: 6: (()+0x5e926) [0x7fb4e260f926]
May 22 22:17:33 wx-ceph-p70-219 bash: 7: (()+0x5e953) [0x7fb4e260f953]
May 22 22:17:33 wx-ceph-p70-219 bash: 8: (()+0x5eb73) [0x7fb4e260fb73]
May 22 22:17:33 wx-ceph-p70-219 bash: 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x27a) [0xbdf92a]
May 22 22:17:33 wx-ceph-p70-219 bash: 10: (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d*, char const*, long)+0x2d9) [0xb12c89]
May 22 22:17:33 wx-ceph-p70-219 bash: 11: (ceph::HeartbeatMap::is_healthy()+0xde) [0xb1357e]
May 22 22:17:33 wx-ceph-p70-219 bash: 12: (ceph::HeartbeatMap::check_touch_file()+0x2c) [0xb13c9c]
May 22 22:17:33 wx-ceph-p70-219 bash: 13: (CephContextServiceThread::entry()+0x15b) [0xbefb7b]
May 22 22:17:33 wx-ceph-p70-219 bash: 14: (()+0x7df5) [0x7fb4e32ebdf5]
May 22 22:17:33 wx-ceph-p70-219 bash: 15: (clone()+0x6d) [0x7fb4e1dce1ad]
May 22 22:17:33 wx-ceph-p70-219 bash: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
May 22 22:17:33 wx-ceph-p70-219 bash: /bin/bash: line 1: 16123 Aborted /usr/bin/ceph-osd -i 22 --pid-file /var/run/ceph/osd.22.pid -c /etc/ceph/ceph.conf --cluster ceph -f
May 22 22:17:33 wx-ceph-p70-219 systemd: ceph-osd.22.1494635575.114208085.service: main process exited, code=exited, status=134/n/a
May 22 22:20:01 wx-ceph-p70-219 systemd: Created slice user-0.slice.
May 22 22:20:01 wx-ceph-p70-219 systemd: Starting Session 12291 of user root.
May 22 22:20:01 wx-ceph-p70-219 systemd: Started Session 12291 of user root.
May 22 22:30:01 wx-ceph-p70-219 systemd: Created slice user-0.slice.
May 22 22:30:01 wx-ceph-p70-219 systemd: Starting Session 12292 of user root.
May 22 22:30:01 wx-ceph-p70-219 systemd: Started Session 12292 of user root.
May 22 22:35:15 wx-ceph-p70-219 sshd³⁶⁷³: error: Could not load host key: /etc/ssh/ssh_host_dsa_key
May 22 22:35:15 wx-ceph-p70-219 systemd: Created slice user-1002.slice.
May 22 22:35:15 wx-ceph-p70-219 systemd: Starting Session 12293 of user hjops.
May 22 22:35:15 wx-ceph-p70-219 systemd-logind: New session 12293 of user hjops.
May 22 22:35:15 wx-ceph-p70-219 systemd: Started Session 12293 of user hjops.
May 22 22:35:19 wx-ceph-p70-219 su: (to root) hjops on pts/0
May 22 22:35:39 wx-ceph-p70-219 systemd: Starting /bin/bash -c ulimit -n 1000000; /usr/bin/ceph-osd -i 22 --pid-file /var/run/ceph/osd.22.pid -c /etc/ceph/ceph.conf --cluster ceph -f...
May 22 22:35:39 wx-ceph-p70-219 systemd: Started /bin/bash -c ulimit -n 1000000; /usr/bin/ceph-osd -i 22 --pid-file /var/run/ceph/osd.22.pid -c /etc/ceph/ceph.conf --cluster ceph -f.
May 22 22:35:39 wx-ceph-p70-219 bash: starting osd.22 at :/0 osd_data /var/lib/ceph/osd/ceph-22 /var/lib/ceph/osd/ceph-22/journal
May 22 22:35:40 wx-ceph-p70-219 bash: SG_IO: bad/missing sense data, sb[]: 70 ce 05 ff 00 00 02 0a 03 00 00 00 20 00 00 80 39 36 00 00 00 00 00 00 00 00 00 00 00 00 00 00
May 22 22:35:40 wx-ceph-p70-219 bash: SG_IO: bad/missing sense data, sb[]: 70 ce 05 ff 00 00 02 0a 03 00 00 00 20 00 00 80 39 36 00 00 00 00 00 00 00 00 00 00 00 00 00 00
May 22 22:35:50 wx-ceph-p70-219 bash: 2017-05-22 22:35:50.678132 7f18ce660880 -1 osd.22 32024 log_to_monitors {default=true}

ceph osd log:/var/log/ceph/ceph-osd.22.log
2017-05-22 22:17:11.072006 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.37 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:16:51.071999)
2017-05-22 22:17:11.072022 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.40 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:16:51.071999)
2017-05-22 22:17:15.773491 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.37 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:16:55.773489)
2017-05-22 22:17:15.773505 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.40 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:16:55.773489)
2017-05-22 22:17:19.874709 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.37 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:16:59.874707)
2017-05-22 22:17:19.874721 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.40 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:16:59.874707)
2017-05-22 22:17:23.975757 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.37 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:17:03.975755)
2017-05-22 22:17:23.975771 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.40 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:17:03.975755)
2017-05-22 22:17:25.676821 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.37 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:17:05.676820)
2017-05-22 22:17:25.676833 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.40 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:17:05.676820)
2017-05-22 22:17:31.577894 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.37 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:17:11.577893)
2017-05-22 22:17:31.577907 7fb4b81cf700 -1 osd.22 32024 heartbeat_check: no reply from osd.40 since back 2017-05-22 22:15:06.325777 front 2017-05-22 22:15:06.325777 (cutoff 2017-05-22 22:17:11.577893)
2017-05-22 22:17:33.254203 7fb4dfceb700 -1 common/HeartbeatMap.cc: In function 'bool ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d*, const char*, time_t)' thread 7fb4dfceb700 time 2017-05-22 22:17:33.235759
common/HeartbeatMap.cc: 79: FAILED assert(0 == "hit suicide timeout")

ceph version 0.94.10 (b1e0532418e4631af01acbc0cedd426f1905f4af)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0xbdf735]
2: (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d*, char const*, long)+0x2d9) [0xb12c89]
3: (ceph::HeartbeatMap::is_healthy()+0xde) [0xb1357e]
4: (ceph::HeartbeatMap::check_touch_file()+0x2c) [0xb13c9c]
5: (CephContextServiceThread::entry()+0x15b) [0xbefb7b]
6: (()+0x7df5) [0x7fb4e32ebdf5]
7: (clone()+0x6d) [0x7fb4e1dce1ad]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
10000> 2017-05-22 22:14:21.446665 7fb4bd9da700 1 - 10.10.70.219:6802/16123 --> 10.10.70.209:0/766807174 -- osd_op_reply(69312 main.2777154.1__shadow_.AzM37k2yYFbJlj5FuKuYFyFRbcBMmXD_1 [read 0~4194304] v0'0 uv134 ondisk = 0) v6 -- ?+0 0x6412ec0 con 0x1a4fe2c0
9999> 2017-05-22 22:14:21.446707 7fb4bd9da700 5 - op tracker -- seq: 5580586, time: 2017-05-22 22:14:21.446707, event: done, op: osd_op(client.5219520.0:69312 main.2777154.1__shadow_.AzM37k2yYFbJlj5FuKuYFyFRbcBMmXD_1 [read 0~4194304] 18.881a5407 ack+read+known_if_redirected e32012)
9998> 2017-05-22 22:14:21.548140 7fb48f424700 1 - 10.10.70.219:6802/16123 <== client.5083848 10.10.70.183:0/2878669794 76 ==== osd_op(client.5083848.0:1010133 main.2777154.1_res/866657508789941754/e0efefbd8ddeb43455e664a7da9a5e3e.json [getxattrs,stat] 18.bcd0e23e ack+read+known_if_redirected e32012) v5 ==== 272+0+0 (4237566443 0 0) 0x21e37480 con 0x29389340
9997> 2017-05-22 22:14:21.548305 7fb4ba1d3700 5 - op tracker -- seq: 5580588, time: 2017-05-22 22:14:21.548305, event: reached_pg, op: osd_op(client.5083848.0:1010133 main.2777154.1_res/866657508789941754/e0efefbd8ddeb43455e664a7da9a5e3e.json [getxattrs,stat] 18.bcd0e23e ack+read+known_if_redirected e32012)
9996> 2017-05-22 22:14:21.569392 7fb4ba1d3700 1 - 10.10.70.219:6802/16123 --> 10.10.70.183:0/2878669794 -- osd_op_reply(1010133 main.2777154.1_res/866657508789941754/e0efefbd8ddeb43455e664a7da9a5e3e.json [getxattrs,stat] v0'0 uv0 ack = 2 ((2) No such file or directory)) v6 - ?+0 0x13864940 con 0x29389340
9995> 2017-05-22 22:14:21.570393 7fb4ba1d3700 5 - op tracker -- seq: 5580588, time: 2017-05-22 22:14:21.570393, event: done, op: osd_op(client.5083848.0:1010133 main.2777154.1_res/866657508789941754/e0efefbd8ddeb43455e664a7da9a5e3e.json [getxattrs,stat] 18.bcd0e23e ack+read+known_if_redirected e32012)
9994> 2017-05-22 22:14:21.586627 7fb48f424700 1 - 10.10.70.219:6802/16123 <== client.5083848 10.10.70.183:0/2878669794 77 ==== osd_op(client.5083848.0:1010138 main.2777154.1_res/866657508789941754/e0efefbd8ddeb43455e664a7da9a5e3e.json [create 0~0 [excl],setxattr user.rgw.idtag (20),writefull 0~800,setxattr user.rgw.manifest (465),setxattr user.rgw.acl (185),setxattr user.rgw.content_type (17),setxattr user.rgw.etag (33)] 18.bcd0e23e ondisk+write+known_if_redirected e32012) v5 ==== 462+0+1597 (1502149552 0 492074587) 0x14bb2f00 con 0x29389340
9993> 2017-05-22 22:14:21.586778 7fb4bc9d8700 5 - op tracker -- seq: 5580589, time: 2017-05-22 22:14:21.586777, event: reached_pg, op: osd_op(client.5083848.0:1010138 main.2777154.1_res/866657508789941754/e0efefbd8ddeb43455e664a7da9a5e3e.json [create 0~0 [excl],setx
attr user.rgw.idtag (20),writefull 0~800,setxattr user.rgw.manifest (465),setxattr user.rgw.acl (185),setxattr user.rgw.content_type (17),setxattr user.rgw.etag (33)] 18.bcd0e23e ondisk+write+known_if_redirected e32012)
9992> 2017-05-22 22:14:21.587334 7fb4bc9d8700 5 - op tracker -- seq: 5580589, time: 2017-05-22 22:14:21.587334, event: started, op: osd_op(client.5083848.0:1010138 main.2777154.1_res/866657508789941754/e0efefbd8ddeb43455e664a7da9a5e3e.json [create 0~0 [excl],setxattr user.rgw.idtag (20),writefull 0~800,setxattr user.rgw.manifest (465),setxattr user.rgw.acl (185),setxattr user.rgw.content_type (17),setxattr user.rgw.etag (33)] 18.bcd0e23e ondisk+write+known_if_redirected e32012)
9991> 2017-05-22 22:14:21.587844 7fb4bc9d8700 5 - op tracker -- seq: 5580589, time: 2017-05-22 22:14:21.587844, event: waiting for subops from 74, op: osd_op(client.5083848.0:1010138 main.2777154.1_res/866657508789941754/e0efefbd8ddeb43455e664a7da9a5e3e.json [create 0~0 [excl],setxattr user.rgw.idtag (20),writefull 0~800,setxattr user.rgw.manifest (465),setxattr user.rgw.acl (185),setxattr user.rgw.content_type (17),setxattr user.rgw.etag (33)] 18.bcd0e23e ondisk+write+known_if_redirected e32012)
9990> 2017-05-22 22:14:21.587919 7fb4bc9d8700 1 - 10.10.70.219:6801/21016123 --> 10.10.70.229:6831/4174358 -- osd_repop(client.5083848.0:1010138 18.23e 18/bcd0e23e/main.2777154.1_res/866657508789941754/e0efefbd8ddeb43455e664a7da9a5e3e.json/head v 32012'145335) v1 -- ?+2924 0x1f15cc00 con 0x18ce15a0
9989> 2017-05-22 22:14:21.588324 7fb4bc9d8700 5 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: false, divergent_priors: 0, writeout_from: 32012'145335, trimmed:
-9988> 2017-05-22 22:14:21.588403 7fb4bc9d8700 5 - op tracker -- seq: 5580589, time: 2017-05-22 22:14:21.588403, event: commit_queued_for_journal_write, op: osd_op(client.5083848.0:1010138 main.2777154.1_res/866657508789941754/e0efefbd8ddeb43455e664a7da9a5e3e.json [create 0~0 [excl],setxattr user.rgw.idtag (20),writefull 0~800,setxattr user.rgw.manifest (465),setxattr user.rgw.acl (185),setxattr user.rgw.content_type (17),setxattr user.rgw.etag (33)] 18.bcd0e23e ondisk+write+known_if_redirected e32012)
9987> 2017-05-22 22:14:21.588841 7fb4d7ff0700 5 - op tracker -- seq: 5580589, time: 2017-05-22 22:14:21.588841, event: write_thread_in_journal_buffer, op: osd_op(client.5083848.0:1010138 main.2777154.1_res/866657508789941754/e0efefbd8ddeb43455e664a7da9a5e3e.json [create 0~0 [excl],setxattr user.rgw.idtag (20),writefull 0~800,setxattr user.rgw.manifest (465),setxattr user.rgw.acl (185),setxattr user.rgw.content_type (17),setxattr user.rgw.etag (33)] 18.bcd0e23e ondisk+write+known_if_redirected e32012)
9986> 2017-05-22 22:14:21.593741 7fb4d77ef700 5 - op tracker -- seq: 5580589, time: 2017-05-22 22:14:21.593740, event: journaled_completion_queued, op: osd_op(client.5083848.0:1010138 main.2777154.1_res/866657508789941754/e0efefbd8ddeb43455e664a7da9a5e3e.json [create 0~0 [excl],setxattr user.rgw.idtag (20),writefull 0~800,setxattr user.rgw.manifest (465),setxattr user.rgw.acl (185),setxattr user.rgw.content_type (17),setxattr user.rgw.etag (33)] 18.bcd0e23e ondisk+write+known_if_redirected e32012)
9985> 2017-05-22 22:14:21.593810 7fb4d4fea700 5 - op tracker -- seq: 5580589, time: 2017-05-22 22:14:21.593809, event: op_commit, op: osd_op(client.5083848.0:1010138 main.2777154.1_res/866657508789941754/e0efefbd8ddeb43455e664a7da9a5e3e.json [create 0~0 [excl],setxattr user.rgw.idtag (20),writefull 0~800,setxattr user.rgw.manifest (465),setxattr user.rgw.acl (185),setxattr user.rgw.content_type (17),setxattr user.rgw.etag (33)] 18.bcd0e23e ondisk+write+known_if_redirected e32012)
9984> 2017-05-22 22:14:21.600844 7fb4d57eb700 5 - op tracker -- seq: 5580589, time: 2017-05-22 22:14:21.600844, event: op_applied, op: osd_op(client.5083848.0:1010138 main.2777154.1_res/866657508789941754/e0efefbd8ddeb43455e664a7da9a5e3e.json [create 0~0 [excl],setxattr user.rgw.idtag (20),writefull 0~800,setxattr user.rgw.manifest (465),setxattr user.rgw.acl (185),setxattr user.rgw.content_type (17),setxattr user.rgw.etag (33)] 18.bcd0e23e ondisk+write+known_if_redirected e32012)
9983> 2017-05-22 22:14:21.601620 7fb4c4bea700 1 - 10.10.70.219:6823/21016123 <== osd.40 10.10.70.220:0/12973 8 ==== osd_ping(ping e32012 stamp 2017-05-22 22:14:21.582011) v2 ==== 47+0+0 (1292924502 0 0) 0x67fa600 con 0xa63e3c0
9982> 2017-05-22 22:14:21.601669 7fb4c4bea700 1 - 10.10.70.219:6823/21016123 --> 10.10.70.220:0/12973 -- osd_ping(ping_reply e32012 stamp 2017-05-22 22:14:21.582011) v2 -- ?+0 0x1c135400 con 0xa63e3c0
9981> 2017-05-22 22:14:21.601676 7fb4c33e7700 1 - 10.10.70.219:6822/21016123 <== osd.40 10.10.70.220:0/12973 8 ==== osd_ping(ping e32012 stamp 2017-05-22 22:14:21.582011) v2 ==== 47+0+0 (1292924502 0 0) 0x67f8000 con 0x1b57ffa0
9980> 2017-05-22 22:14:21.601732 7fb4c33e7700 1 - 10.10.70.219:6822/21016123 --> 10.10.70.220:0/12973 -- osd_ping(ping_reply e32012 stamp 2017-05-22 22:14:21.582011) v2 -- ?+0 0x1a556a00 con 0x1b57ffa0
9979> 2017-05-22 22:14:21.667772 7fb4c4bea700 1 - 10.10.70.219:6823/21016123 <== osd.75 10.10.70.229:0/175341 13049 ==== osd_ping(ping e32012 stamp 2017-05-22 22:14:21.663538) v2 ==== 47+0+0 (494252363 0 0) 0x1edc0200 con 0x1aca5340
9978> 2017-05-22 22:14:21.667828 7fb4c4bea700 1 - 10.10.70.219:6823/21016123 --> 10.10.70.229:0/175341 -- osd_ping(ping_reply e32012 stamp 2017-05-22 22:14:21.663538) v2 -- ?+0 0x67fa600 con 0x1aca5340
9977> 2017-05-22 22:14:21.694546 7fb4c33e7700 1 - 10.10.70.219:6822/21016123 <== osd.55 10.10.70.225:0/699 14304 ==== osd_ping(ping e32012 stamp 2017-05-22 22:14:21.689423) v2 ==== 47+0+0 (2878231215 0 0) 0x121c6400 con 0x59527e0
9976> 2017-05-22 22:14:21.694554 7fb4c4bea700 1 - 10.10.70.219:6823/21016123 <== osd.55 10.10.70.225:0/699 14304 ==== osd_ping(ping e32012 stamp 2017-05-22 22:14:21.689423) v2 ==== 47+0+0 (2878231215 0 0) 0x1acbfc00 con 0x175dd020
9975> 2017-05-22 22:14:21.694601 7fb4c33e7700 1 - 10.10.70.219:6822/21016123 --> 10.10.70.225:0/699 -- osd_ping(ping_reply e32012 stamp 2017-05-22 22:14:21.689423) v2 -- ?+0 0x67f8000 con 0x59527e0
9974> 2017-05-22 22:14:21.694649 7fb4c4bea700 1 - 10.10.70.219:6823/21016123 --> 10.10.70.225:0/699 -- osd_ping(ping_reply e32012 stamp 2017-05-22 22:14:21.689423) v2 -- ?+0 0x1edc0200 con 0x175dd020
9973> 2017-05-22 22:14:21.734469 7fb4c4bea700 1 - 10.10.70.219:6823/21016123 <== osd.49 10.10.70.225:0/6294 18 ==== osd_ping(ping e32012 stamp 2017-05-22 22:14:21.277717) v2 ==== 47+0+0 (586331549 0 0) 0x264a2c00 con 0xa63f860
9972> 2017-05-22 22:14:21.734515 7fb4c4bea700 1 - 10.10.70.219:6823/21016123 --> 10.10.70.225:0/6294 -- osd_ping(ping_reply e32012 stamp 2017-05-22 22:14:21.277717) v2 -- ?+0 0x1acbfc00 con 0xa63f860
9971> 2017-05-22 22:14:21.734540 7fb4c33e7700 1 - 10.10.70.219:6822/21016123 <== osd.49 10.10.70.225:0/6294 18 ==== osd_ping(ping e32012 stamp 2017-05-22 22:14:21.277717) v2 ==== 47+0+0 (586331549 0 0) 0x1910ee00 con 0x1b57fe40
9970> 2017-05-22 22:14:21.734583 7fb4c33e7700 1 - 10.10.70.219:6822/21016123 --> 10.10.70.225:0/6294 -- osd_ping(ping_reply e32012 stamp 2017-05-22 22:14:21.277717) v2 -- ?+0 0x121c6400 con 0x1b57fe40
9969> 2017-05-22 22:14:21.788242 7fb4c4bea700 1 - 10.10.70.219:6823/21016123 <== osd.34 10.10.70.220:0/28056 1007 ==== osd_ping(ping e32012 stamp 2017-05-22 22:14:21.768628) v2 ==== 47+0+0 (3865759743 0 0) 0x18f12a00 con 0x1b46a580
9968> 2017-05-22 22:14:21.788304 7fb4c4bea700 1 - 10.10.70.219:6823/21016123 --> 10.10.70.220:0/28056 -- osd_ping(ping_reply e32012 stamp 2017-05-22 22:14:21.768628) v2 -- ?+0 0x264a2c00 con 0x1b46a580
9967> 2017-05-22 22:14:21.788322 7fb4c33e7700 1 - 10.10.70.219:6822/21016123 <== osd.34 10.10.70.220:0/28056 1007 ==== osd_ping(ping e32012 stamp 2017-05-22 22:14:21.768628) v2 ==== 47+0+0 (3865759743 0 0) 0x19515800 con 0x1058caa0
9966> 2017-05-22 22:14:21.788373 7fb4c33e7700 1 - 10.10.70.219:6822/21016123 --> 10.10.70.220:0/28056 -- osd_ping(ping_reply e32012 stamp 2017-05-22 22:14:21.768628) v2 -- ?+0 0x1910ee00 con 0x1058caa0
9965> 2017-05-22 22:14:21.800320 7fb4c4bea700 1 - 10.10.70.219:6823/21016123 <== osd.53 10.10.70.225:0/9357 639 ==== osd_ping(ping e32012 stamp 2017-05-22 22:14:21.571402) v2 ==== 47+0+0 (3025700214 0 0) 0x21578400 con 0x1fd73700
9964> 2017-05-22 22:14:21.800374 7fb4c4bea700 1 - 10.10.70.219:6823/21016123 --> 10.10.70.225:0/9357 -- osd_ping(ping_reply e32012 stamp 2017-05-22 22:14:21.571402) v2 -- ?+0 0x18f12a00 con 0x1fd73700
9963> 2017-05-22 22:14:21.803302 7fb4b2190700 1 - 10.10.70.219:6801/21016123 <== osd.74 10.10.70.229:6831/4174358 4580 ==== osd_repop_reply(client.5083848.0:1010138 18.23e ondisk, result = 0) v1 ==== 83+0+0 (264576200 0 0) 0x21a118c0 con 0x18ce15a0
9962> 2017-05-22 22:14:21.803420 7fb4ba1d3700 5 - op tracker -- seq: 5580590, time: 2017-05-22 22:14:21.803419, event: reached_pg, op: osd_repop_reply(client.5083848.0:1010138 18.23e ondisk, result = 0)
9961> 2017-05-22 22:14:21.803465 7fb4ba1d3700 5 - op tracker -- seq: 5580590, time: 2017-05-22 22:14:21.803465, event: started, op: osd_repop_reply(client.5083848.0:1010138 18.23e ondisk, result = 0)
9960> 2017-05-22 22:14:21.803482 7fb4ba1d3700 5 - op tracker -- seq: 5580589, time: 2017-05-22 22:14:21.803482, event: sub_op_commit_rec, op: osd_op(client.5083848.0:1010138 main.2777154.1_res/866657508789941754/e0efefbd8ddeb43455e664a7da9a5e3e.json [create 0~0 [excl],setxattr user.rgw.idtag (20),writefull 0~800,setxattr user.rgw.manifest (465),setxattr user.rgw.acl (185),setxattr user.rgw.content_type (17),setxattr user.rgw.etag (33)] 18.bcd0e23e ondisk+write+known_if_redirected e32012)
9959> 2017-05-22 22:14:21.803537 7fb4ba1d3700 1 - 10.10.70.219:6802/16123 --> 10.10.70.183:0/2878669794 -- osd_op_reply(1010138 main.2777154.1_res/866657508789941754/e0efefbd8ddeb43455e664a7da9a5e3e.json [create 0~0 [excl],setxattr user.rgw.idtag (20),writefull 0~800,setxattr user.rgw.manifest (465),setxattr user.rgw.acl (185),setxattr user.rgw.content_type (17),setxattr user.rgw.etag (33)] v32012'145335 uv145335 ondisk = 0) v6 -- ?+0 0xd8ad8c0 con 0x29389340
9958> 2017-05-22 22:14:21.803601 7fb4ba1d3700 5 - op tracker -- seq: 5580589, time: 2017-05-22 22:14:21.803601, event: commit_sent, op: osd_op(client.5083848.0:1010138 main.2777154.1_res/866657508789941754/e0efefbd8ddeb43455e664a7da9a5e3e.json [create 0~0 [excl],setxattr user.rgw.idtag (20),writefull 0~800,setxattr user.rgw.manifest (465),setxattr user.rgw.acl (185),setxattr user.rgw.content_type (17),setxattr user.rgw.etag (33)] 18.bcd0e23e ondisk+write+known_if_redirected e32012)
9957> 2017-05-22 22:14:21.803634 7fb4ba1d3700 5 - op tracker -- seq: 5580589, time: 2017-05-22 22:14:21.803634, event: done, op: osd_op(client.5083848.0:1010138 main.2777154.1_res/866657508789941754/e0efefbd8ddeb43455e664a7da9a5e3e.json [create 0~0 [excl],setxattr user.rgw.idtag (20),writefull 0~800,setxattr user.rgw.manifest (465),setxattr user.rgw.acl (185),setxattr user.rgw.content_type (17),setxattr user.rgw.etag (33)] 18.bcd0e23e ondisk+write+known_if_redirected e32012)
9956> 2017-05-22 22:14:21.803671 7fb4ba1d3700 5 - op tracker -- seq: 5580590, time: 2017-05-22 22:14:21.803671, event: done, op: osd_repop_reply(client.5083848.0:1010138 18.23e ondisk, result = 0)
9955> 2017-05-22 22:14:22.011885 7fb4da958700 5 osd.22 32012 tick
-9954> 2017-05-22 22:14:22.018502 7fb4c33e7700 1 - 10.10.70.219:6822/21016123 <== osd.53 10.10.70.225:0/9357 639 ==== osd_ping(ping e32012 stamp 2017-05-22 22:14:21.571402) v2 ==== 47+0+0 (3025700214 0 0) 0x160e2c00 con 0x20eb78c0
9953> 2017-05-22 22:14:22.018549 7fb4c33e7700 1 - 10.10.70.219:6822/21016123 --> 10.10.70.225:0/9357 -- osd_ping(ping_reply e32012 stamp 2017-05-22 22:14:21.571402) v2 -- ?+0 0x19515800 con 0x20eb78c0
9952> 2017-05-22 22:14:22.060160 7fb4c4bea700 1 - 10.10.70.219:6823/21016123 <== osd.76 10.10.70.229:0/176445 84 ==== osd_ping(ping e32012 stamp 2017-05-22 22:14:21.833439) v2 ==== 47+0+0 (3634172266 0 0) 0xbcb1600 con 0x1fd70b00
9951> 2017-05-22 22:14:22.060161 7fb4c33e7700 1 - 10.10.70.219:6822/21016123 <== osd.76 10.10.70.229:0/176445 84 ==== osd_ping(ping e32012 stamp 2017-05-22 22:14:21.833439) v2 ==== 47+0+0 (3634172266 0 0) 0xa59e000 con 0x1fd72680
9950> 2017-05-22 22:14:22.060211 7fb4c4bea700 1 - 10.10.70.219:6823/21016123 --> 10.10.70.229:0/176445 -- osd_ping(ping_reply e32012 stamp 2017-05-22 22:14:21.833439) v2 -- ?+0 0x21578400 con 0x1fd70b00
9949> 2017-05-22 22:14:22.060251 7fb4c33e7700 1 - 10.10.70.219:6822/21016123 --> 10.10.70.229:0/176445 -- osd_ping(ping_reply e32012 stamp 2017-05-22 22:14:21.833439) v2 -- ?+0 0x160e2c00 con 0x1fd72680
9948> 2017-05-22 22:14:22.113566 7fb4c33e7700 1 - 10.10.70.219:6822/21016123 <== osd.35 10.10.70.220:0/9276 37 ==== osd_ping(ping e32012 stamp 2017-05-22 22:14:22.093866) v2 ==== 47+0+0 (2865565059 0 0) 0x21577000 con 0x1b580520
9947> 2017-05-22 22:14:22.113624 7fb4c33e7700 1 - 10.10.70.219:6822/21016123 --> 10.10.70.220:0/9276 -- osd_ping(ping_reply e32012 stamp 2017-05-22 22:14:22.093866) v2 -- ?+0 0xa59e000 con 0x1b580520
9946> 2017-05-22 22:14:22.113662 7fb4c4bea700 1 - 10.10.70.219:6823/21016123 <== osd.35 10.10.70.220:0/9276 37 ==== osd_ping(ping e32012 stamp 2017-05-22 22:14:22.093866) v2 ==== 47+0+0 (2865565059 0 0) 0x11b8d000 con 0x1a500100
9945> 2017-05-22 22:14:22.113714 7fb4c4bea700 1 - 10.10.70.219:6823/21016123 --> 10.10.70.220:0/9276 -- osd_ping(ping_reply e32012 stamp 2017-05-22 22:14:22.093866) v2 -- ?+0 0xbcb1600 con 0x1a500100
9944> 2017-05-22 22:14:22.115108 7fb4c33e7700 1 - 10.10.70.219:6822/21016123 <== osd.75 10.10.70.229:0/175341 13049 ==== osd_ping(ping e32012 stamp 2017-05-22 22:14:21.663538) v2 ==== 47+0+0 (494252363 0 0) 0x13923a00 con 0x15a4be40
9943> 2017-05-22 22:14:22.115177 7fb4c33e7700 1 - 10.10.70.219:6822/21016123 --> 10.10.70.229:0/175341 -- osd_ping(ping_reply e32012 stamp 2017-05-22 22:14:21.663538) v2 -- ?+0 0x21577000 con 0x15a4be40
9942> 2017-05-22 22:14:22.116739 7fb49c364700 1 - 10.10.70.219:6802/16123 <== client.5219490 10.10.70.212:0/2625822716 8 ==== osd_op(client.5219490.0:66016 main.2777154.1__shadow_.0VAFovGr6EGGNm_Sgp-TJxs9_MaRreH_126 [read 0~4194304] 18.ee47b06 ack+read+known_if_redirected e32012) v5 ==== 218+0+0 (2564281907 0 0) 0x24ad3b80 con 0x1058bb80
9941> 2017-05-22 22:14:22.116869 7fb4bc9d8700 5 - op tracker -- seq: 5580591, time: 2017-05-22 22:14:22.116869, event: reached_pg, op: osd_op(client.5219490.0:66016 main.2777154.1__shadow_.0VAFovGr6EGGNm_Sgp-TJxs9_MaRreH_126 [read 0~4194304] 18.ee47b06 ack+read+known_if_redirected e32012)
9940> 2017-05-22 22:14:22.116941 7fb4bc9d8700 5 - op tracker -- seq: 5580591, time: 2017-05-22 22:14:22.116941, event: started, op: osd_op(client.5219490.0:66016 main.2777154.1__shadow_.0VAFovGr6EGGNm_Sgp-TJxs9_MaRreH_126 [read 0~4194304] 18.ee47b06 ack+read+known_if_redirected e32012)
9939> 2017-05-22 22:14:22.121795 7fb4bc9d8700 1 - 10.10.70.219:6802/16123 --> 10.10.70.212:0/2625822716 -- osd_op_reply(66016 main.2777154.1__shadow_.0VAFovGr6EGGNm_Sgp-TJxs9_MaRreH_126 [read 0~4194304] v0'0 uv128214 ondisk = 0) v6 -- ?+0 0x21945b80 con 0x1058bb80
9938> 2017-05-22 22:14:22.121834 7fb4bc9d8700 5 - op tracker -- seq: 5580591, time: 2017-05-22 22:14:22.121833, event: done, op: osd_op(client.5219490.0:66016 main.2777154.1__shadow_.0VAFovGr6EGGNm_Sgp-TJxs9_MaRreH_126 [read 0~4194304] 18.ee47b06 ack+read+known_if_redirected e32012)
9937> 2017-05-22 22:14:22.239313 7fb4c4bea700 1 - 10.10.70.219:6823/21016123 <== osd.78 10.10.70.229:0/178625 8535 ==== osd_ping(ping e32012 stamp 2017-05-22 22:14:22.236002) v2 ==== 47+0+0 (1583570461 0 0) 0x14b73200 con 0x18ce1700
9936> 2017-05-22 22:14:22.239397 7fb4c4bea700 1 - 10.10.70.219:6823/21016123 --> 10.10.70.229:0/178625 -- osd_ping(ping_reply e32012 stamp 2017-05-22 22:14:22.236002) v2 -- ?+0 0x11b8d000 con 0x18ce1700
9935> 2017-05-22 22:14:22.239395 7fb4c33e7700 1 - 10.10.70.219:6822/21016123 <== osd.78 10.10.70.229:0/178625 8535 ==== osd_ping(ping e32012 stamp 2017-05-22 22:14:22.236002) v2 ==== 47+0+0 (1583570461 0 0) 0x1bb6ca00 con 0x293889a0
9934> 2017-05-22 22:14:22.239537 7fb4c33e7700 1 - 10.10.70.219:6822/21016123 --> 10.10.70.229:0/178625 -- osd_ping(ping_reply e32012 stamp 2017-05-22 22:14:22.236002) v2 -- ?+0 0x13923a00 con 0x293889a0
9933> 2017-05-22 22:14:22.338411 7fb4c33e7700 1 - 10.10.70.219:6822/21016123 <== osd.76 10.10.70.229:0/176445 85 ==== osd_ping(ping e32012 stamp 2017-05-22 22:14:22.334230) v2 ==== 47+0+0 (351095243 0 0) 0xa59ec00 con 0x1fd72680
9932> 2017-05-22 22:14:22.338453 7fb4c4bea700 1 - 10.10.70.219:6823/21016123 <== osd.76 10.10.70.229:0/176445 85 ==== osd_ping(ping e32012 stamp 2017-05-22 22:14:22.334230) v2 ==== 47+0+0 (351095243 0 0) 0xbcaec00 con 0x1fd70b00
9931> 2017-05-22 22:14:22.338483 7fb4c33e7700 1 - 10.10.70.219:6822/21016123 --> 10.10.70.229:0/176445 -- osd_ping(ping_reply e32012 stamp 2017-05-22 22:14:22.334230) v2 -- ?+0 0x1bb6ca00 con 0x1fd72680
9930> 2017-05-22 22:14:22.338525 7fb4c4bea700 1 - 10.10.70.219:6823/21016123 --> 10.10.70.229:0/176445 -- osd_ping(ping_reply e32012 stamp 2017-05-22 22:14:22.334230) v2 -- ?+0 0x14b73200 con 0x1fd70b00
9929> 2017-05-22 22:14:22.395631 7fb49d190700 1 - 10.10.70.219:6802/16123 <== client.5219712 10.10.70.207:0/2792590543 20 ==== osd_op(client.5219712.0:53460 main.2777154.1__shadow_.0VAFovGr6EGGNm_Sgp-TJxs9_MaRreH_126 [read 0~4194304] 18.ee47b06 ack+read+known_if_redirected e32012) v5 ==== 218+0+0 (2564281907 0 0) 0x41bb480 con 0x1a501de0
9928> 2017-05-22 22:14:22.395748 7fb4ba1d3700 5 - op tracker -- seq: 5580592, time: 2017-05-22 22:14:22.395748, event: reached_pg, op: osd_op(client.5219712.0:53460 main.2777154.1__shadow_.0VAFovGr6EGGNm_Sgp-TJxs9_MaRreH_126 [read 0~4194304] 18.ee47b06 ack+read+known_if_redirected e32012)
9927> 2017-05-22 22:14:22.395829 7fb4ba1d3700 5 - op tracker -- seq: 5580592, time: 2017-05-22 22:14:22.395829, event: started, op: osd_op(client.5219712.0:53460 main.2777154.1__shadow_.0VAFovGr6EGGNm_Sgp-TJxs9_MaRreH_126 [read 0~4194304] 18.ee47b06 ack+read+known_if_redirected e32012)
9926> 2017-05-22 22:14:22.400890 7fb4ba1d3700 1 - 10.10.70.219:6802/16123 --> 10.10.70.207:0/2792590543 -- osd_op_reply(53460 main.2777154.1__shadow_.0VAFovGr6EGGNm_Sgp-TJxs9_MaRreH_126 [read 0~4194304] v0'0 uv128214 ondisk = 0) v6 -- ?+0 0x1ed8e3c0 con 0x1a501de0
9925> 2017-05-22 22:14:22.400940 7fb4ba1d3700 5 - op tracker -- seq: 5580592, time: 2017-05-22 22:14:22.400940, event: done, op: osd_op(client.5219712.0:53460 main.2777154.1__shadow_.0VAFovGr6EGGNm_Sgp-TJxs9_MaRreH_126 [read 0~4194304] 18.ee47b06 ack+read+known_if_redirected e32012)
9924> 2017-05-22 22:14:22.507410 7fb4a8cdf700 1 - 10.10.70.219:6801/21016123 <== osd.35 10.10.70.220:6830/36009276 38 ==== osd_repop(client.5022012.0:1934780 18.219 18/ec657619/main.2777154.1_res/866657508789941754/293fc909a0631395daeb538a4ee10acf.jpg/head v 32012'146043) v1 ==== 989+0+45407 (1961115933 0 2778441298) 0x29f70a00 con 0xa760f20
9923> 2017-05-22 22:14:22.507539 7fb4bb1d5700 5 - op tracker -- seq: 5580593, time: 2017-05-22 22:14:22.507539, event: reached_pg, op: osd_repop(client.5022012.0:1934780 18.219 18/ec657619/main.2777154.1_res/866657508789941754/293fc909a0631395daeb538a4ee10acf.jpg/head v 32012'146043)
9922> 2017-05-22 22:14:22.507576 7fb4bb1d5700 5 - op tracker -- seq: 5580593, time: 2017-05-22 22:14:22.507576, event: started, op: osd_repop(client.5022012.0:1934780 18.219 18/ec657619/main.2777154.1_res/866657508789941754/293fc909a0631395daeb538a4ee10acf.jpg/head v 32012'146043)
9921> 2017-05-22 22:14:22.507745 7fb4bb1d5700 5 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: false, divergent_priors: 0, writeout_from: 32012'146043, trimmed:
-9920> 2017-05-22 22:14:22.508305 7fb4bb1d5700 5 - op tracker -- seq: 5580593, time: 2017-05-22 22:14:22.508305, event: commit_queued_for_journal_write, op: osd_repop(client.5022012.0:1934780 18.219 18/ec657619/main.2777154.1_res/866657508789941754/293fc909a0631395daeb538a4ee10acf.jpg/head v 32012'146043)
9919> 2017-05-22 22:14:22.508380 7fb4d7ff0700 5 - op tracker -- seq: 5580593, time: 2017-05-22 22:14:22.508380, event: write_thread_in_journal_buffer, op: osd_repop(client.5022012.0:1934780 18.219 18/ec657619/main.2777154.1_res/866657508789941754/293fc909a0631395daeb538a4ee10acf.jpg/head v 32012'146043)
9918> 2017-05-22 22:14:22.517680 7fb4d77ef700 5 - op tracker -- seq: 5580593, time: 2017-05-22 22:14:22.517680, event: journaled_completion_queued, op: osd_repop(client.5022012.0:1934780 18.219 18/ec657619/main.2777154.1_res/866657508789941754/293fc909a0631395daeb538a4ee10acf.jpg/head v 32012'146043)
9917> 2017-05-22 22:14:22.517761 7fb4d4fea700 5 - op tracker -- seq: 5580593, time: 2017-05-22 22:14:22.517760, event: commit_sent, op: osd_repop(client.5022012.0:1934780 18.219 18/ec657619/main.2777154.1_res/866657508789941754/293fc909a0631395daeb538a4ee10acf.jpg/head v 32012'146043)
9916> 2017-05-22 22:14:22.517804 7fb4d4fea700 1 - 10.10.70.219:6801/21016123 --> 10.10.70.220:6830/36009276 -- osd_repop_reply(client.5022012.0:1934780 18.219 ondisk, result = 0) v1 -- ?+0 0x26031600 con 0xa760f20
9915> 2017-05-22 22:14:22.553149 7fb4d57eb700 5 - op tracker -- seq: 5580593, time: 2017-05-22 22:14:22.553149, event: sub_op_applied, op: osd_repop(client.5022012.0:1934780 18.219 18/ec657619/main.2777154.1_res/866657508789941754/293fc909a0631395daeb538a4ee10acf.jp
g/head v 32012'146043)
9914> 2017-05-22 22:14:22.553187 7fb4d57eb700 5 - op tracker -- seq: 5580593, time: 2017-05-22 22:14:22.553187, event: done, op: osd_repop(client.5022012.0:1934780 18.219 18/ec657619/main.2777154.1_res/866657508789941754/293fc909a0631395daeb538a4ee10acf.jpg/head v 32012'146043)
9913> 2017-05-22 22:14:22.677695 7fb4c33e7700 1 - 10.10.70.219:6822/21016123 <== osd.53 10.10.70.225:0/9357 640 ==== osd_ping(ping e32012 stamp 2017-05-22 22:14:22.672513) v2 ==== 47+0+0 (2004317213 0 0) 0x160e1200 con 0x20eb78c0
9912> 2017-05-22 22:14:22.677748 7fb4c33e7700 1 - 10.10.70.219:6822/21016123 --> 10.10.70.225:0/9357 -- osd_ping(ping_reply e32012 stamp 2017-05-22 22:14:22.672513) v2 -- ?+0 0xa59ec00 con 0x20eb78c0
9911> 2017-05-22 22:14:22.789130 7fb492159700 1 - 10.10.70.219:6801/21016123 <== osd.71 10.10.70.229:6801/171442 4389 ==== osd_repop(client.5022009.0:1847546 18.cf 18/5bc440cf/main.2777154.1__shadow_.shUfjBCdqtSMw0aNoO2r7lcapUeCCFH_4/head v 32012'147857) v1 ==== 955+0+262869 (298781668 0 3718812311) 0x1864cc00 con 0x136a12e0
9910> 2017-05-22 22:14:22.789256 7fb4bd9da700 5 - op tracker -- seq: 5580594, time: 2017-05-22 22:14:22.789256, event: reached_pg, op: osd_repop(client.5022009.0:1847546 18.cf 18/5bc440cf/main.2777154.1__shadow_.shUfjBCdqtSMw0aNoO2r7lcapUeCCFH_4/head v 32012'147857)
9909> 2017-05-22 22:14:22.789293 7fb4bd9da700 5 - op tracker -- seq: 5580594, time: 2017-05-22 22:14:22.789293, event: started, op: osd_repop(client.5022009.0:1847546 18.cf 18/5bc440cf/main.2777154.1__shadow_.shUfjBCdqtSMw0aNoO2r7lcapUeCCFH_4/head v 32012'147857)
9908> 2017-05-22 22:14:22.790145 7fb4bd9da700 5 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: false, divergent_priors: 0, writeout_from: 32012'147857, trimmed:
-9907> 2017-05-22 22:14:22.790455 7fb4bd9da700 5 - op tracker -- seq: 5580594, time: 2017-05-22 22:14:22.790454, event: commit_queued_for_journal_write, op: osd_repop(client.5022009.0:1847546 18.cf 18/5bc440cf/main.2777154.1__shadow_.shUfjBCdqtSMw0aNoO2r7lcapUeCCFH_4/head v 32012'147857)
9906> 2017-05-22 22:14:22.790558 7fb4d7ff0700 5 - op tracker -- seq: 5580594, time: 2017-05-22 22:14:22.790558, event: write_thread_in_journal_buffer, op: osd_repop(client.5022009.0:1847546 18.cf 18/5bc440cf/main.2777154.1__shadow_.shUfjBCdqtSMw0aNoO2r7lcapUeCCFH_4/head v 32012'147857)
9905> 2017-05-22 22:14:22.806383 7fb4d77ef700 5 - op tracker -- seq: 5580594, time: 2017-05-22 22:14:22.806383, event: journaled_completion_queued, op: osd_repop(client.5022009.0:1847546 18.cf 18/5bc440cf/main.2777154.1__shadow_.shUfjBCdqtSMw0aNoO2r7lcapUeCCFH_4/head v 32012'147857)
9904> 2017-05-22 22:14:22.806426 7fb4d4fea700 5 - op tracker -- seq: 5580594, time: 2017-05-22 22:14:22.806425, event: commit_sent, op: osd_repop(client.5022009.0:1847546 18.cf 18/5bc440cf/main.2777154.1__shadow_.shUfjBCdqtSMw0aNoO2r7lcapUeCCFH_4/head v 32012'147857)
9903> 2017-05-22 22:14:22.806481 7fb4d4fea700 1 - 10.10.70.219:6801/21016123 --> 10.10.70.229:6801/171442 -- osd_repop_reply(client.5022009.0:1847546 18.cf ondisk, result = 0) v1 -- ?+0 0x26030880 con 0x136a12e0
9902> 2017-05-22 22:14:22.807238 7fb4d57eb700 5 - op tracker -- seq: 5580594, time: 2017-05-22 22:14:22.807238, event: sub_op_applied, op: osd_repop(client.5022009.0:1847546 18.cf 18/5bc440cf/main.2777154.1__shadow_.shUfjBCdqtSMw0aNoO2r7lcapUeCCFH_4/head v 32012'147857)
9901> 2017-05-22 22:14:22.807332 7fb4d57eb700 5 - op tracker -- seq: 5580594, time: 2017-05-22 22:14:22.807332, event: done, op: osd_repop(client.5022009.0:1847546 18.cf 18/5bc440cf/main.2777154.1__shadow_.shUfjBCdqtSMw0aNoO2r7lcapUeCCFH_4/head v 32012'147857)
9900> 2017-05-22 22:14:22.810173 7fb4b81cf700 5 osd.22 32012 heartbeat: osd_stat(1587 GB used, 3999 GB avail, 5587 GB total, peers [21,23,33,34,36,37,40,49,52,53,55,56,71,74,75,76,77,78,79,80]/[] op hist [])
-9899> 2017-05-22 22:14:22.810204 7fb4b81cf700 1 - 10.10.70.219:0/16123 --> 10.10.70.197:6802/3003663 -- osd_ping(ping e32012 stamp 2017-05-22 22:14:22.810200) v2 -- ?+0 0x16865400 con 0x20eb8680
9898> 2017-05-22 22:14:22.810242 7fb4b81cf700 1 - 10.10.70.219:0/16123 --> 10.10.70.197:6803/3003663 -- osd_ping(ping e32012 stamp 2017-05-22 22:14:22.810200) v2 -- ?+0 0x67f9a00 con 0x20077b80