-34> 2015-12-02 19:18:51.419249 7fe26751a700 1 heartbeat_map reset_timeout 'FileStore::op_tp thread 0x7fe26751a700' had timed out after 60 -33> 2015-12-02 19:18:51.419265 7fe26751a700 1 heartbeat_map reset_timeout 'FileStore::op_tp thread 0x7fe26751a700' had suicide timed out after 180 -32> 2015-12-02 19:18:14.760127 7fe25638e700 1 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fe25638e700' had timed out after 4 -31> 2015-12-02 19:18:56.452129 7fe254b8b700 1 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fe254b8b700' had timed out after 4 -30> 2015-12-02 19:18:56.452157 7fe25538c700 1 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fe25538c700' had timed out after 4 -29> 2015-12-02 19:18:13.181031 7fe256b8f700 1 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fe256b8f700' had timed out after 4 -28> 2015-12-02 19:18:56.456469 7fe256b8f700 1 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fe256b8f700' had timed out after 15 -27> 2015-12-02 19:18:56.456896 7fe258b93700 1 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fe258b93700' had timed out after 4 -26> 2015-12-02 19:18:56.456906 7fe259394700 1 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fe259394700' had timed out after 4 -25> 2015-12-02 19:18:56.456910 7fe257b91700 1 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fe257b91700' had timed out after 4 -24> 2015-12-02 19:18:56.463232 7fe25638e700 1 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fe25638e700' had timed out after 15 -23> 2015-12-02 19:15:50.830196 7fe252b87700 5 osd.1 125 heartbeat: osd_stat(2323 MB used, 18146 MB avail, 20470 MB total, peers [0]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,7]) -22> 2015-12-02 19:19:16.731731 7fe252b87700 1 -- 10.0.0.12:0/3916 --> 10.0.0.11:6802/4004651 -- osd_ping(ping e125 stamp 2015-12-02 19:19:16.731727) v2 -- ?+0 0x99fec00 con 0xa0191e0 -21> 2015-12-02 19:19:16.731785 7fe252b87700 1 -- 10.0.0.12:0/3916 --> 10.0.0.11:6803/4004651 -- osd_ping(ping e125 stamp 2015-12-02 19:19:16.731727) v2 -- ?+0 0x99fd800 con 0xa019340 -20> 2015-12-02 19:19:16.731808 7fe252b87700 -1 osd.1 125 heartbeat_check: no reply from osd.0 since back 2015-12-02 19:06:06.939775 front 2015-12-02 19:06:06.939775 (cutoff 2015-12-02 19:18:56.731793) -19> 2015-12-02 19:19:16.732704 7fe26a64d700 -1 osd.1 125 heartbeat_check: no reply from osd.0 since back 2015-12-02 19:06:06.939775 front 2015-12-02 19:06:06.939775 (cutoff 2015-12-02 19:18:56.732703) -18> 2015-12-02 19:19:10.015399 7fe25b398700 10 monclient(hunting): picked mon.node0 con 0x8188dc0 addr 10.0.0.11:6789/0 -17> 2015-12-02 19:19:14.157890 7fe251283700 1 -- 10.0.0.12:6802/6003916 --> 10.0.0.11:6801/4004651 -- osd_repop(osd.1.0:6161 14.4f 6c8b444f/rbd_header.6e7074b0dc51/head//14 v 125'224) v1 -- ?+479 0x79a6000 con 0x7021b80 -16> 2015-12-02 19:19:28.475794 7fe258b93700 1 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fe258b93700' had timed out after 4 -15> 2015-12-02 19:19:28.475847 7fe257b91700 1 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fe257b91700' had timed out after 4 -14> 2015-12-02 19:19:28.475903 7fe254b8b700 1 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fe254b8b700' had timed out after 4 -13> 2015-12-02 19:19:28.476207 7fe25538c700 1 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fe25538c700' had timed out after 4 -12> 2015-12-02 19:19:28.476234 7fe25638e700 1 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fe25638e700' had timed out after 4 -11> 2015-12-02 19:19:29.023734 7fe256b8f700 1 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fe256b8f700' had timed out after 4 -10> 2015-12-02 19:19:29.026106 7fe25b398700 10 monclient(hunting): _send_mon_message to mon.node0 at 10.0.0.11:6789/0 -9> 2015-12-02 19:19:29.026119 7fe25b398700 1 -- 10.0.0.12:6801/3916 --> 10.0.0.11:6789/0 -- auth(proto 0 26 bytes epoch 1) v1 -- ?+0 0x4f26c00 con 0x8188dc0 -8> 2015-12-02 19:19:29.026153 7fe25b398700 10 monclient(hunting): renew_subs -7> 2015-12-02 19:19:28.475919 7fe259394700 1 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fe259394700' had timed out after 4 -6> 2015-12-02 19:19:29.338326 7fe252b87700 5 osd.1 125 heartbeat: osd_stat(2323 MB used, 18146 MB avail, 20470 MB total, peers [0]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,7]) -5> 2015-12-02 19:19:29.338413 7fe252b87700 1 -- 10.0.0.12:0/3916 --> 10.0.0.11:6802/4004651 -- osd_ping(ping e125 stamp 2015-12-02 19:19:29.338406) v2 -- ?+0 0xf163800 con 0xa0191e0 -4> 2015-12-02 19:19:29.338457 7fe26a64d700 0 log_channel(cluster) log [WRN] : 7 slow requests, 7 included below; oldest blocked for > 743.519176 secs -3> 2015-12-02 19:19:29.338489 7fe252b87700 1 -- 10.0.0.12:0/3916 --> 10.0.0.11:6803/4004651 -- osd_ping(ping e125 stamp 2015-12-02 19:19:29.338406) v2 -- ?+0 0xf160c00 con 0xa019340 -2> 2015-12-02 19:19:29.338538 7fe252b87700 -1 osd.1 125 heartbeat_check: no reply from osd.0 since back 2015-12-02 19:06:06.939775 front 2015-12-02 19:06:06.939775 (cutoff 2015-12-02 19:19:09.338536) -1> 2015-12-02 19:19:29.338596 7fe26a64d700 0 log_channel(cluster) log [WRN] : slow request 743.519176 seconds old, received at 2015-12-02 19:07:05.818637: osd_op(client.28261.0:26026 rbd_header.6e5c2ae8944a [watch ping cookie 140295290267568 gen 8] 14.64552175 ondisk+write+known_if_redirected e125) currently started 0> 2015-12-02 19:19:29.423829 7fe26f9d7700 -1 common/HeartbeatMap.cc: In function 'bool ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d*, const char*, time_t)' thread 7fe26f9d7700 time 2015-12-02 19:11:42.033541 common/HeartbeatMap.cc: 79: FAILED assert(0 == "hit suicide timeout") ceph version 0.94.2 (5fb85614ca8f354284c713a2f9c610860720bbf3) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0xbc5645] 2: (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d*, char const*, long)+0x2d9) [0xafb449] 3: (ceph::HeartbeatMap::is_healthy()+0xde) [0xafbd3e] 4: (ceph::HeartbeatMap::check_touch_file()+0x2c) [0xafc45c] 5: (CephContextServiceThread::entry()+0x15b) [0xbd587b] 6: (()+0x7df5) [0x7fe2727abdf5] 7: (clone()+0x6d) [0x7fe27128e1ad] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.1.log --- end dump of recent events --- 2015-12-02 19:19:29.424359 7fe26751a700 -1 common/HeartbeatMap.cc: In function 'bool ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d*, const char*, time_t)' thread 7fe26751a700 time 2015-12-02 19:18:51.419327 common/HeartbeatMap.cc: 79: FAILED assert(0 == "hit suicide timeout") ceph version 0.94.2 (5fb85614ca8f354284c713a2f9c610860720bbf3) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0xbc5645] 2: (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d*, char const*, long)+0x2d9) [0xafb449] 3: (ceph::HeartbeatMap::reset_timeout(ceph::heartbeat_handle_d*, long, long)+0x89) [0xafb769] 4: (FileStore::_do_transactions(std::list >&, unsigned long, ThreadPool::TPHandle*)+0x41) [0x915041] 5: (FileStore::_do_op(FileStore::OpSequencer*, ThreadPool::TPHandle&)+0x188) [0x915208] 6: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa76) [0xbb6366] 7: (ThreadPool::WorkThread::entry()+0x10) [0xbb73f0] 8: (()+0x7df5) [0x7fe2727abdf5] 9: (clone()+0x6d) [0x7fe27128e1ad] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2015-12-02 19:19:29.640804 7fe26a64d700 0 log_channel(cluster) log [WRN] : slow request 658.380192 seconds old, received at 2015-12-02 19:08:30.957621: osd_op(client.28261.0:26027 rbd_header.6e5c2ae8944a [watch ping cookie 140295290267568 gen 8] 14.64552175 ondisk+write+known_if_redirected e125) currently no flag points reached 2015-12-02 19:19:29.640825 7fe26a64d700 0 log_channel(cluster) log [WRN] : slow request 658.380040 seconds old, received at 2015-12-02 19:08:30.957773: osd_op(client.28261.0:26028 rbd_header.6e5c2ae8944a [watch ping cookie 140295290267568 gen 8] 14.64552175 ondisk+write+known_if_redirected e125) currently no flag points reached 2015-12-02 19:19:29.640829 7fe26a64d700 0 log_channel(cluster) log [WRN] : slow request 658.379999 seconds old, received at 2015-12-02 19:08:30.957814: osd_op(client.28261.0:26029 rbd_header.6e5c2ae8944a [watch ping cookie 140295290267568 gen 8] 14.64552175 ondisk+write+known_if_redirected e125) currently no flag points reached 2015-12-02 19:19:29.640832 7fe26a64d700 0 log_channel(cluster) log [WRN] : slow request 655.452309 seconds old, received at 2015-12-02 19:08:33.885504: osd_op(client.28261.0:26030 rbd_header.6e5c2ae8944a [watch ping cookie 140295290267568 gen 8] 14.64552175 ondisk+write+known_if_redirected e125) currently no flag points reached 2015-12-02 19:19:30.642416 7fe26a64d700 -1 osd.1 125 heartbeat_check: no reply from osd.0 since back 2015-12-02 19:06:06.939775 front 2015-12-02 19:06:06.939775 (cutoff 2015-12-02 19:19:10.642412) 2015-12-02 19:19:30.642637 7fe26a64d700 0 log_channel(cluster) log [WRN] : 7 slow requests, 7 included below; oldest blocked for > 744.823843 secs 2015-12-02 19:19:30.642649 7fe26a64d700 0 log_channel(cluster) log [WRN] : slow request 744.823843 seconds old, received at 2015-12-02 19:07:05.818637: osd_op(client.28261.0:26026 rbd_header.6e5c2ae8944a [watch ping cookie 140295290267568 gen 8] 14.64552175 ondisk+write+known_if_redirected e125) currently started 2015-12-02 19:19:30.642683 7fe26a64d700 0 log_channel(cluster) log [WRN] : slow request 659.684859 seconds old, received at 2015-12-02 19:08:30.957621: osd_op(client.28261.0:26027 rbd_header.6e5c2ae8944a [watch ping cookie 140295290267568 gen 8] 14.64552175 ondisk+write+known_if_redirected e125) currently no flag points reached 2015-12-02 19:19:30.642700 7fe26a64d700 0 log_channel(cluster) log [WRN] : slow request 659.684707 seconds old, received at 2015-12-02 19:08:30.957773: osd_op(client.28261.0:26028 rbd_header.6e5c2ae8944a [watch ping cookie 140295290267568 gen 8] 14.64552175 ondisk+write+known_if_redirected e125) currently no flag points reached 2015-12-02 19:19:30.642708 7fe26a64d700 0 log_channel(cluster) log [WRN] : slow request 659.684666 seconds old, received at 2015-12-02 19:08:30.957814: osd_op(client.28261.0:26029 rbd_header.6e5c2ae8944a [watch ping cookie 140295290267568 gen 8] 14.64552175 ondisk+write+known_if_redirected e125) currently no flag points reached 2015-12-02 19:19:30.642718 7fe26a64d700 0 log_channel(cluster) log [WRN] : slow request 656.756976 seconds old, received at 2015-12-02 19:08:33.885504: osd_op(client.28261.0:26030 rbd_header.6e5c2ae8944a [watch ping cookie 140295290267568 gen 8] 14.64552175 ondisk+write+known_if_redirected e125) currently no flag points reached --- begin dump of recent events --- -29> 2015-12-02 19:19:29.424359 7fe26751a700 -1 common/HeartbeatMap.cc: In function 'bool ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d*, const char*, time_t)' thread 7fe26751a700 time 2015-12-02 19:18:51.419327 common/HeartbeatMap.cc: 79: FAILED assert(0 == "hit suicide timeout") ceph version 0.94.2 (5fb85614ca8f354284c713a2f9c610860720bbf3) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0xbc5645] 2: (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d*, char const*, long)+0x2d9) [0xafb449] 3: (ceph::HeartbeatMap::reset_timeout(ceph::heartbeat_handle_d*, long, long)+0x89) [0xafb769] 4: (FileStore::_do_transactions(std::list >&, unsigned long, ThreadPool::TPHandle*)+0x41) [0x915041] 5: (FileStore::_do_op(FileStore::OpSequencer*, ThreadPool::TPHandle&)+0x188) [0x915208] 6: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa76) [0xbb6366] 7: (ThreadPool::WorkThread::entry()+0x10) [0xbb73f0] 8: (()+0x7df5) [0x7fe2727abdf5] 9: (clone()+0x6d) [0x7fe27128e1ad] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. -28> 2015-12-02 19:19:29.466286 7fe251283700 1 -- 10.0.0.12:6802/6003916 --> 10.0.0.11:6801/4004651 -- osd_repop(osd.1.0:6162 14.4f 6c8b444f/rbd_header.6e7074b0dc51/head//14 v 125'225) v1 -- ?+479 0x89c6c00 con 0x7021b80 -27> 2015-12-02 19:19:29.640804 7fe26a64d700 0 log_channel(cluster) log [WRN] : slow request 658.380192 seconds old, received at 2015-12-02 19:08:30.957621: osd_op(client.28261.0:26027 rbd_header.6e5c2ae8944a [watch ping cookie 140295290267568 gen 8] 14.64552175 ondisk+write+known_if_redirected e125) currently no flag points reached -26> 2015-12-02 19:19:29.640825 7fe26a64d700 0 log_channel(cluster) log [WRN] : slow request 658.380040 seconds old, received at 2015-12-02 19:08:30.957773: osd_op(client.28261.0:26028 rbd_header.6e5c2ae8944a [watch ping cookie 140295290267568 gen 8] 14.64552175 ondisk+write+known_if_redirected e125) currently no flag points reached -25> 2015-12-02 19:19:29.640829 7fe26a64d700 0 log_channel(cluster) log [WRN] : slow request 658.379999 seconds old, received at 2015-12-02 19:08:30.957814: osd_op(client.28261.0:26029 rbd_header.6e5c2ae8944a [watch ping cookie 140295290267568 gen 8] 14.64552175 ondisk+write+known_if_redirected e125) currently no flag points reached -24> 2015-12-02 19:19:29.640832 7fe26a64d700 0 log_channel(cluster) log [WRN] : slow request 655.452309 seconds old, received at 2015-12-02 19:08:33.885504: osd_op(client.28261.0:26030 rbd_header.6e5c2ae8944a [watch ping cookie 140295290267568 gen 8] 14.64552175 ondisk+write+known_if_redirected e125) currently no flag points reached -23> 2015-12-02 19:19:29.640873 7fe2633a8700 10 monclient(hunting): renew_subs -22> 2015-12-02 19:19:29.620593 7fe257390700 5 -- op tracker -- seq: 243956, time: 2015-12-02 19:19:29.620592, event: reached_pg, op: osd_op(client.28278.0:7860 rbd_header.6e7074b0dc51 [watch ping cookie 140379596790576 gen 1] 14.6c8b444f ondisk+write+known_if_redirected e125) -21> 2015-12-02 19:19:29.788994 7fe24e250700 1 -- 10.0.0.12:6801/3916 <== client.28261 10.0.0.12:0/1078132 44 ==== osd_op(client.28261.0:26032 rbd_header.6e5c2ae8944a [watch ping cookie 140295290267568 gen 8] 14.64552175 ondisk+write+known_if_redirected e125) v5 ==== 182+0+0 (154774817 0 0) 0x8694380 con 0x8270c60 -20> 2015-12-02 19:19:29.789035 7fe24e250700 5 -- op tracker -- seq: 243957, time: 2015-12-02 19:19:29.316731, event: header_read, op: osd_op(client.28261.0:26032 rbd_header.6e5c2ae8944a [watch ping cookie 140295290267568 gen 8] 14.64552175 ondisk+write+known_if_redirected e125) -19> 2015-12-02 19:19:29.789045 7fe24e250700 5 -- op tracker -- seq: 243957, time: 2015-12-02 19:19:29.316738, event: throttled, op: osd_op(client.28261.0:26032 rbd_header.6e5c2ae8944a [watch ping cookie 140295290267568 gen 8] 14.64552175 ondisk+write+known_if_redirected e125) -18> 2015-12-02 19:19:29.789050 7fe24e250700 5 -- op tracker -- seq: 243957, time: 2015-12-02 19:19:29.788928, event: all_read, op: osd_op(client.28261.0:26032 rbd_header.6e5c2ae8944a [watch ping cookie 140295290267568 gen 8] 14.64552175 ondisk+write+known_if_redirected e125) -17> 2015-12-02 19:19:29.789058 7fe24e250700 5 -- op tracker -- seq: 243957, time: 0.000000, event: dispatched, op: osd_op(client.28261.0:26032 rbd_header.6e5c2ae8944a [watch ping cookie 140295290267568 gen 8] 14.64552175 ondisk+write+known_if_redirected e125) -16> 2015-12-02 19:19:29.886413 7fe24e855700 1 -- 10.0.0.12:6801/3916 <== client.28278 10.0.0.12:0/1078698 44 ==== osd_op(client.28278.0:7861 rbd_header.6e7074b0dc51 [watch ping cookie 140379596790576 gen 1] 14.6c8b444f ondisk+write+known_if_redirected e125) v5 ==== 182+0+0 (283630935 0 0) 0x7b15780 con 0x8271ce0 -15> 2015-12-02 19:19:29.886544 7fe24e855700 5 -- op tracker -- seq: 243958, time: 2015-12-02 19:19:29.316616, event: header_read, op: osd_op(client.28278.0:7861 rbd_header.6e7074b0dc51 [watch ping cookie 140379596790576 gen 1] 14.6c8b444f ondisk+write+known_if_redirected e125) -14> 2015-12-02 19:19:29.886569 7fe24e855700 5 -- op tracker -- seq: 243958, time: 2015-12-02 19:19:29.316632, event: throttled, op: osd_op(client.28278.0:7861 rbd_header.6e7074b0dc51 [watch ping cookie 140379596790576 gen 1] 14.6c8b444f ondisk+write+known_if_redirected e125) -13> 2015-12-02 19:19:29.886581 7fe24e855700 5 -- op tracker -- seq: 243958, time: 2015-12-02 19:19:29.886348, event: all_read, op: osd_op(client.28278.0:7861 rbd_header.6e7074b0dc51 [watch ping cookie 140379596790576 gen 1] 14.6c8b444f ondisk+write+known_if_redirected e125) -12> 2015-12-02 19:19:29.886670 7fe24e855700 5 -- op tracker -- seq: 243958, time: 0.000000, event: dispatched, op: osd_op(client.28278.0:7861 rbd_header.6e7074b0dc51 [watch ping cookie 140379596790576 gen 1] 14.6c8b444f ondisk+write+known_if_redirected e125) -11> 2015-12-02 19:19:30.108618 7fe257390700 5 -- op tracker -- seq: 243956, time: 2015-12-02 19:19:30.108618, event: started, op: osd_op(client.28278.0:7860 rbd_header.6e7074b0dc51 [watch ping cookie 140379596790576 gen 1] 14.6c8b444f ondisk+write+known_if_redirected e125) -10> 2015-12-02 19:19:30.146859 7fe2633a8700 1 -- 10.0.0.12:6801/3916 <== client.28278 10.0.0.12:0/1078698 45 ==== ping magic: 0 v1 ==== 0+0+0 (0 0 0) 0x823bd40 con 0x8271ce0 -9> 2015-12-02 19:19:30.225214 7fe2633a8700 1 -- 10.0.0.12:6801/3916 <== client.28261 10.0.0.12:0/1078132 45 ==== ping magic: 0 v1 ==== 0+0+0 (0 0 0) 0x823ec80 con 0x8270c60 -8> 2015-12-02 19:19:30.642371 7fe26a64d700 5 osd.1 125 tick -7> 2015-12-02 19:19:30.642416 7fe26a64d700 -1 osd.1 125 heartbeat_check: no reply from osd.0 since back 2015-12-02 19:06:06.939775 front 2015-12-02 19:06:06.939775 (cutoff 2015-12-02 19:19:10.642412) -6> 2015-12-02 19:19:30.642637 7fe26a64d700 0 log_channel(cluster) log [WRN] : 7 slow requests, 7 included below; oldest blocked for > 744.823843 secs -5> 2015-12-02 19:19:30.642649 7fe26a64d700 0 log_channel(cluster) log [WRN] : slow request 744.823843 seconds old, received at 2015-12-02 19:07:05.818637: osd_op(client.28261.0:26026 rbd_header.6e5c2ae8944a [watch ping cookie 140295290267568 gen 8] 14.64552175 ondisk+write+known_if_redirected e125) currently started -4> 2015-12-02 19:19:30.642683 7fe26a64d700 0 log_channel(cluster) log [WRN] : slow request 659.684859 seconds old, received at 2015-12-02 19:08:30.957621: osd_op(client.28261.0:26027 rbd_header.6e5c2ae8944a [watch ping cookie 140295290267568 gen 8] 14.64552175 ondisk+write+known_if_redirected e125) currently no flag points reached -3> 2015-12-02 19:19:30.642700 7fe26a64d700 0 log_channel(cluster) log [WRN] : slow request 659.684707 seconds old, received at 2015-12-02 19:08:30.957773: osd_op(client.28261.0:26028 rbd_header.6e5c2ae8944a [watch ping cookie 140295290267568 gen 8] 14.64552175 ondisk+write+known_if_redirected e125) currently no flag points reached -2> 2015-12-02 19:19:30.642708 7fe26a64d700 0 log_channel(cluster) log [WRN] : slow request 659.684666 seconds old, received at 2015-12-02 19:08:30.957814: osd_op(client.28261.0:26029 rbd_header.6e5c2ae8944a [watch ping cookie 140295290267568 gen 8] 14.64552175 ondisk+write+known_if_redirected e125) currently no flag points reached -1> 2015-12-02 19:19:30.642718 7fe26a64d700 0 log_channel(cluster) log [WRN] : slow request 656.756976 seconds old, received at 2015-12-02 19:08:33.885504: osd_op(client.28261.0:26030 rbd_header.6e5c2ae8944a [watch ping cookie 140295290267568 gen 8] 14.64552175 ondisk+write+known_if_redirected e125) currently no flag points reached 0> 2015-12-02 19:19:31.470317 7fe2633a8700 10 monclient(hunting): renew_subs --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.1.log --- end dump of recent events ---