|
-34> 2015-12-02 19:18:51.419249 7fe26751a700 1 heartbeat_map reset_timeout 'FileStore::op_tp thread 0x7fe26751a700' had timed out after 60
|
|
-33> 2015-12-02 19:18:51.419265 7fe26751a700 1 heartbeat_map reset_timeout 'FileStore::op_tp thread 0x7fe26751a700' had suicide timed out after 180
|
|
-32> 2015-12-02 19:18:14.760127 7fe25638e700 1 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fe25638e700' had timed out after 4
|
|
-31> 2015-12-02 19:18:56.452129 7fe254b8b700 1 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fe254b8b700' had timed out after 4
|
|
-30> 2015-12-02 19:18:56.452157 7fe25538c700 1 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fe25538c700' had timed out after 4
|
|
-29> 2015-12-02 19:18:13.181031 7fe256b8f700 1 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fe256b8f700' had timed out after 4
|
|
-28> 2015-12-02 19:18:56.456469 7fe256b8f700 1 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fe256b8f700' had timed out after 15
|
|
-27> 2015-12-02 19:18:56.456896 7fe258b93700 1 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fe258b93700' had timed out after 4
|
|
-26> 2015-12-02 19:18:56.456906 7fe259394700 1 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fe259394700' had timed out after 4
|
|
-25> 2015-12-02 19:18:56.456910 7fe257b91700 1 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fe257b91700' had timed out after 4
|
|
-24> 2015-12-02 19:18:56.463232 7fe25638e700 1 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fe25638e700' had timed out after 15
|
|
-23> 2015-12-02 19:15:50.830196 7fe252b87700 5 osd.1 125 heartbeat: osd_stat(2323 MB used, 18146 MB avail, 20470 MB total, peers [0]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,7])
|
|
-22> 2015-12-02 19:19:16.731731 7fe252b87700 1 -- 10.0.0.12:0/3916 --> 10.0.0.11:6802/4004651 -- osd_ping(ping e125 stamp 2015-12-02 19:19:16.731727) v2 -- ?+0 0x99fec00 con 0xa0191e0
|
|
-21> 2015-12-02 19:19:16.731785 7fe252b87700 1 -- 10.0.0.12:0/3916 --> 10.0.0.11:6803/4004651 -- osd_ping(ping e125 stamp 2015-12-02 19:19:16.731727) v2 -- ?+0 0x99fd800 con 0xa019340
|
|
-20> 2015-12-02 19:19:16.731808 7fe252b87700 -1 osd.1 125 heartbeat_check: no reply from osd.0 since back 2015-12-02 19:06:06.939775 front 2015-12-02 19:06:06.939775 (cutoff 2015-12-02 19:18:56.731793)
|
|
-19> 2015-12-02 19:19:16.732704 7fe26a64d700 -1 osd.1 125 heartbeat_check: no reply from osd.0 since back 2015-12-02 19:06:06.939775 front 2015-12-02 19:06:06.939775 (cutoff 2015-12-02 19:18:56.732703)
|
|
-18> 2015-12-02 19:19:10.015399 7fe25b398700 10 monclient(hunting): picked mon.node0 con 0x8188dc0 addr 10.0.0.11:6789/0
|
|
-17> 2015-12-02 19:19:14.157890 7fe251283700 1 -- 10.0.0.12:6802/6003916 --> 10.0.0.11:6801/4004651 -- osd_repop(osd.1.0:6161 14.4f 6c8b444f/rbd_header.6e7074b0dc51/head//14 v 125'224) v1 -- ?+479 0x79a6000 con 0x7021b80
|
|
-16> 2015-12-02 19:19:28.475794 7fe258b93700 1 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fe258b93700' had timed out after 4
|
|
-15> 2015-12-02 19:19:28.475847 7fe257b91700 1 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fe257b91700' had timed out after 4
|
|
-14> 2015-12-02 19:19:28.475903 7fe254b8b700 1 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fe254b8b700' had timed out after 4
|
|
-13> 2015-12-02 19:19:28.476207 7fe25538c700 1 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fe25538c700' had timed out after 4
|
|
-12> 2015-12-02 19:19:28.476234 7fe25638e700 1 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fe25638e700' had timed out after 4
|
|
-11> 2015-12-02 19:19:29.023734 7fe256b8f700 1 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fe256b8f700' had timed out after 4
|
|
-10> 2015-12-02 19:19:29.026106 7fe25b398700 10 monclient(hunting): _send_mon_message to mon.node0 at 10.0.0.11:6789/0
|
|
-9> 2015-12-02 19:19:29.026119 7fe25b398700 1 -- 10.0.0.12:6801/3916 --> 10.0.0.11:6789/0 -- auth(proto 0 26 bytes epoch 1) v1 -- ?+0 0x4f26c00 con 0x8188dc0
|
|
-8> 2015-12-02 19:19:29.026153 7fe25b398700 10 monclient(hunting): renew_subs
|
|
-7> 2015-12-02 19:19:28.475919 7fe259394700 1 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7fe259394700' had timed out after 4
|
|
-6> 2015-12-02 19:19:29.338326 7fe252b87700 5 osd.1 125 heartbeat: osd_stat(2323 MB used, 18146 MB avail, 20470 MB total, peers [0]/[] op hist [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,7])
|
|
-5> 2015-12-02 19:19:29.338413 7fe252b87700 1 -- 10.0.0.12:0/3916 --> 10.0.0.11:6802/4004651 -- osd_ping(ping e125 stamp 2015-12-02 19:19:29.338406) v2 -- ?+0 0xf163800 con 0xa0191e0
|
|
-4> 2015-12-02 19:19:29.338457 7fe26a64d700 0 log_channel(cluster) log [WRN] : 7 slow requests, 7 included below; oldest blocked for > 743.519176 secs
|
|
-3> 2015-12-02 19:19:29.338489 7fe252b87700 1 -- 10.0.0.12:0/3916 --> 10.0.0.11:6803/4004651 -- osd_ping(ping e125 stamp 2015-12-02 19:19:29.338406) v2 -- ?+0 0xf160c00 con 0xa019340
|
|
-2> 2015-12-02 19:19:29.338538 7fe252b87700 -1 osd.1 125 heartbeat_check: no reply from osd.0 since back 2015-12-02 19:06:06.939775 front 2015-12-02 19:06:06.939775 (cutoff 2015-12-02 19:19:09.338536)
|
|
-1> 2015-12-02 19:19:29.338596 7fe26a64d700 0 log_channel(cluster) log [WRN] : slow request 743.519176 seconds old, received at 2015-12-02 19:07:05.818637: osd_op(client.28261.0:26026 rbd_header.6e5c2ae8944a [watch ping cookie 140295290267568 gen 8] 14.64552175 ondisk+write+known_if_redirected e125) currently started
|
|
0> 2015-12-02 19:19:29.423829 7fe26f9d7700 -1 common/HeartbeatMap.cc: In function 'bool ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d*, const char*, time_t)' thread 7fe26f9d7700 time 2015-12-02 19:11:42.033541
|
|
common/HeartbeatMap.cc: 79: FAILED assert(0 == "hit suicide timeout")
|
|
|
|
ceph version 0.94.2 (5fb85614ca8f354284c713a2f9c610860720bbf3)
|
|
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0xbc5645]
|
|
2: (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d*, char const*, long)+0x2d9) [0xafb449]
|
|
3: (ceph::HeartbeatMap::is_healthy()+0xde) [0xafbd3e]
|
|
4: (ceph::HeartbeatMap::check_touch_file()+0x2c) [0xafc45c]
|
|
5: (CephContextServiceThread::entry()+0x15b) [0xbd587b]
|
|
6: (()+0x7df5) [0x7fe2727abdf5]
|
|
7: (clone()+0x6d) [0x7fe27128e1ad]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- logging levels ---
|
|
0/ 5 none
|
|
0/ 1 lockdep
|
|
0/ 1 context
|
|
1/ 1 crush
|
|
1/ 5 mds
|
|
1/ 5 mds_balancer
|
|
1/ 5 mds_locker
|
|
1/ 5 mds_log
|
|
1/ 5 mds_log_expire
|
|
1/ 5 mds_migrator
|
|
0/ 1 buffer
|
|
0/ 1 timer
|
|
0/ 1 filer
|
|
0/ 1 striper
|
|
0/ 1 objecter
|
|
0/ 5 rados
|
|
0/ 5 rbd
|
|
0/ 5 rbd_replay
|
|
0/ 5 journaler
|
|
0/ 5 objectcacher
|
|
0/ 5 client
|
|
0/ 5 osd
|
|
0/ 5 optracker
|
|
0/ 5 objclass
|
|
1/ 3 filestore
|
|
1/ 3 keyvaluestore
|
|
1/ 3 journal
|
|
0/ 5 ms
|
|
1/ 5 mon
|
|
0/10 monc
|
|
1/ 5 paxos
|
|
0/ 5 tp
|
|
1/ 5 auth
|
|
1/ 5 crypto
|
|
1/ 1 finisher
|
|
1/ 5 heartbeatmap
|
|
1/ 5 perfcounter
|
|
1/ 5 rgw
|
|
1/10 civetweb
|
|
1/ 5 javaclient
|
|
1/ 5 asok
|
|
1/ 1 throttle
|
|
0/ 0 refs
|
|
1/ 5 xio
|
|
-2/-2 (syslog threshold)
|
|
-1/-1 (stderr threshold)
|
|
max_recent 10000
|
|
max_new 1000
|
|
log_file /var/log/ceph/ceph-osd.1.log
|
|
--- end dump of recent events ---
|
|
2015-12-02 19:19:29.424359 7fe26751a700 -1 common/HeartbeatMap.cc: In function 'bool ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d*, const char*, time_t)' thread 7fe26751a700 time 2015-12-02 19:18:51.419327
|
|
common/HeartbeatMap.cc: 79: FAILED assert(0 == "hit suicide timeout")
|
|
|
|
ceph version 0.94.2 (5fb85614ca8f354284c713a2f9c610860720bbf3)
|
|
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0xbc5645]
|
|
2: (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d*, char const*, long)+0x2d9) [0xafb449]
|
|
3: (ceph::HeartbeatMap::reset_timeout(ceph::heartbeat_handle_d*, long, long)+0x89) [0xafb769]
|
|
4: (FileStore::_do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long, ThreadPool::TPHandle*)+0x41) [0x915041]
|
|
5: (FileStore::_do_op(FileStore::OpSequencer*, ThreadPool::TPHandle&)+0x188) [0x915208]
|
|
6: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa76) [0xbb6366]
|
|
7: (ThreadPool::WorkThread::entry()+0x10) [0xbb73f0]
|
|
8: (()+0x7df5) [0x7fe2727abdf5]
|
|
9: (clone()+0x6d) [0x7fe27128e1ad]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
2015-12-02 19:19:29.640804 7fe26a64d700 0 log_channel(cluster) log [WRN] : slow request 658.380192 seconds old, received at 2015-12-02 19:08:30.957621: osd_op(client.28261.0:26027 rbd_header.6e5c2ae8944a [watch ping cookie 140295290267568 gen 8] 14.64552175 ondisk+write+known_if_redirected e125) currently no flag points reached
|
|
2015-12-02 19:19:29.640825 7fe26a64d700 0 log_channel(cluster) log [WRN] : slow request 658.380040 seconds old, received at 2015-12-02 19:08:30.957773: osd_op(client.28261.0:26028 rbd_header.6e5c2ae8944a [watch ping cookie 140295290267568 gen 8] 14.64552175 ondisk+write+known_if_redirected e125) currently no flag points reached
|
|
2015-12-02 19:19:29.640829 7fe26a64d700 0 log_channel(cluster) log [WRN] : slow request 658.379999 seconds old, received at 2015-12-02 19:08:30.957814: osd_op(client.28261.0:26029 rbd_header.6e5c2ae8944a [watch ping cookie 140295290267568 gen 8] 14.64552175 ondisk+write+known_if_redirected e125) currently no flag points reached
|
|
2015-12-02 19:19:29.640832 7fe26a64d700 0 log_channel(cluster) log [WRN] : slow request 655.452309 seconds old, received at 2015-12-02 19:08:33.885504: osd_op(client.28261.0:26030 rbd_header.6e5c2ae8944a [watch ping cookie 140295290267568 gen 8] 14.64552175 ondisk+write+known_if_redirected e125) currently no flag points reached
|
|
2015-12-02 19:19:30.642416 7fe26a64d700 -1 osd.1 125 heartbeat_check: no reply from osd.0 since back 2015-12-02 19:06:06.939775 front 2015-12-02 19:06:06.939775 (cutoff 2015-12-02 19:19:10.642412)
|
|
2015-12-02 19:19:30.642637 7fe26a64d700 0 log_channel(cluster) log [WRN] : 7 slow requests, 7 included below; oldest blocked for > 744.823843 secs
|
|
2015-12-02 19:19:30.642649 7fe26a64d700 0 log_channel(cluster) log [WRN] : slow request 744.823843 seconds old, received at 2015-12-02 19:07:05.818637: osd_op(client.28261.0:26026 rbd_header.6e5c2ae8944a [watch ping cookie 140295290267568 gen 8] 14.64552175 ondisk+write+known_if_redirected e125) currently started
|
|
2015-12-02 19:19:30.642683 7fe26a64d700 0 log_channel(cluster) log [WRN] : slow request 659.684859 seconds old, received at 2015-12-02 19:08:30.957621: osd_op(client.28261.0:26027 rbd_header.6e5c2ae8944a [watch ping cookie 140295290267568 gen 8] 14.64552175 ondisk+write+known_if_redirected e125) currently no flag points reached
|
|
2015-12-02 19:19:30.642700 7fe26a64d700 0 log_channel(cluster) log [WRN] : slow request 659.684707 seconds old, received at 2015-12-02 19:08:30.957773: osd_op(client.28261.0:26028 rbd_header.6e5c2ae8944a [watch ping cookie 140295290267568 gen 8] 14.64552175 ondisk+write+known_if_redirected e125) currently no flag points reached
|
|
2015-12-02 19:19:30.642708 7fe26a64d700 0 log_channel(cluster) log [WRN] : slow request 659.684666 seconds old, received at 2015-12-02 19:08:30.957814: osd_op(client.28261.0:26029 rbd_header.6e5c2ae8944a [watch ping cookie 140295290267568 gen 8] 14.64552175 ondisk+write+known_if_redirected e125) currently no flag points reached
|
|
2015-12-02 19:19:30.642718 7fe26a64d700 0 log_channel(cluster) log [WRN] : slow request 656.756976 seconds old, received at 2015-12-02 19:08:33.885504: osd_op(client.28261.0:26030 rbd_header.6e5c2ae8944a [watch ping cookie 140295290267568 gen 8] 14.64552175 ondisk+write+known_if_redirected e125) currently no flag points reached
|
|
--- begin dump of recent events ---
|
|
-29> 2015-12-02 19:19:29.424359 7fe26751a700 -1 common/HeartbeatMap.cc: In function 'bool ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d*, const char*, time_t)' thread 7fe26751a700 time 2015-12-02 19:18:51.419327
|
|
common/HeartbeatMap.cc: 79: FAILED assert(0 == "hit suicide timeout")
|
|
|
|
ceph version 0.94.2 (5fb85614ca8f354284c713a2f9c610860720bbf3)
|
|
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0xbc5645]
|
|
2: (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d*, char const*, long)+0x2d9) [0xafb449]
|
|
3: (ceph::HeartbeatMap::reset_timeout(ceph::heartbeat_handle_d*, long, long)+0x89) [0xafb769]
|
|
4: (FileStore::_do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long, ThreadPool::TPHandle*)+0x41) [0x915041]
|
|
5: (FileStore::_do_op(FileStore::OpSequencer*, ThreadPool::TPHandle&)+0x188) [0x915208]
|
|
6: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa76) [0xbb6366]
|
|
7: (ThreadPool::WorkThread::entry()+0x10) [0xbb73f0]
|
|
8: (()+0x7df5) [0x7fe2727abdf5]
|
|
9: (clone()+0x6d) [0x7fe27128e1ad]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
-28> 2015-12-02 19:19:29.466286 7fe251283700 1 -- 10.0.0.12:6802/6003916 --> 10.0.0.11:6801/4004651 -- osd_repop(osd.1.0:6162 14.4f 6c8b444f/rbd_header.6e7074b0dc51/head//14 v 125'225) v1 -- ?+479 0x89c6c00 con 0x7021b80
|
|
-27> 2015-12-02 19:19:29.640804 7fe26a64d700 0 log_channel(cluster) log [WRN] : slow request 658.380192 seconds old, received at 2015-12-02 19:08:30.957621: osd_op(client.28261.0:26027 rbd_header.6e5c2ae8944a [watch ping cookie 140295290267568 gen 8] 14.64552175 ondisk+write+known_if_redirected e125) currently no flag points reached
|
|
-26> 2015-12-02 19:19:29.640825 7fe26a64d700 0 log_channel(cluster) log [WRN] : slow request 658.380040 seconds old, received at 2015-12-02 19:08:30.957773: osd_op(client.28261.0:26028 rbd_header.6e5c2ae8944a [watch ping cookie 140295290267568 gen 8] 14.64552175 ondisk+write+known_if_redirected e125) currently no flag points reached
|
|
-25> 2015-12-02 19:19:29.640829 7fe26a64d700 0 log_channel(cluster) log [WRN] : slow request 658.379999 seconds old, received at 2015-12-02 19:08:30.957814: osd_op(client.28261.0:26029 rbd_header.6e5c2ae8944a [watch ping cookie 140295290267568 gen 8] 14.64552175 ondisk+write+known_if_redirected e125) currently no flag points reached
|
|
-24> 2015-12-02 19:19:29.640832 7fe26a64d700 0 log_channel(cluster) log [WRN] : slow request 655.452309 seconds old, received at 2015-12-02 19:08:33.885504: osd_op(client.28261.0:26030 rbd_header.6e5c2ae8944a [watch ping cookie 140295290267568 gen 8] 14.64552175 ondisk+write+known_if_redirected e125) currently no flag points reached
|
|
-23> 2015-12-02 19:19:29.640873 7fe2633a8700 10 monclient(hunting): renew_subs
|
|
-22> 2015-12-02 19:19:29.620593 7fe257390700 5 -- op tracker -- seq: 243956, time: 2015-12-02 19:19:29.620592, event: reached_pg, op: osd_op(client.28278.0:7860 rbd_header.6e7074b0dc51 [watch ping cookie 140379596790576 gen 1] 14.6c8b444f ondisk+write+known_if_redirected e125)
|
|
-21> 2015-12-02 19:19:29.788994 7fe24e250700 1 -- 10.0.0.12:6801/3916 <== client.28261 10.0.0.12:0/1078132 44 ==== osd_op(client.28261.0:26032 rbd_header.6e5c2ae8944a [watch ping cookie 140295290267568 gen 8] 14.64552175 ondisk+write+known_if_redirected e125) v5 ==== 182+0+0 (154774817 0 0) 0x8694380 con 0x8270c60
|
|
-20> 2015-12-02 19:19:29.789035 7fe24e250700 5 -- op tracker -- seq: 243957, time: 2015-12-02 19:19:29.316731, event: header_read, op: osd_op(client.28261.0:26032 rbd_header.6e5c2ae8944a [watch ping cookie 140295290267568 gen 8] 14.64552175 ondisk+write+known_if_redirected e125)
|
|
-19> 2015-12-02 19:19:29.789045 7fe24e250700 5 -- op tracker -- seq: 243957, time: 2015-12-02 19:19:29.316738, event: throttled, op: osd_op(client.28261.0:26032 rbd_header.6e5c2ae8944a [watch ping cookie 140295290267568 gen 8] 14.64552175 ondisk+write+known_if_redirected e125)
|
|
-18> 2015-12-02 19:19:29.789050 7fe24e250700 5 -- op tracker -- seq: 243957, time: 2015-12-02 19:19:29.788928, event: all_read, op: osd_op(client.28261.0:26032 rbd_header.6e5c2ae8944a [watch ping cookie 140295290267568 gen 8] 14.64552175 ondisk+write+known_if_redirected e125)
|
|
-17> 2015-12-02 19:19:29.789058 7fe24e250700 5 -- op tracker -- seq: 243957, time: 0.000000, event: dispatched, op: osd_op(client.28261.0:26032 rbd_header.6e5c2ae8944a [watch ping cookie 140295290267568 gen 8] 14.64552175 ondisk+write+known_if_redirected e125)
|
|
-16> 2015-12-02 19:19:29.886413 7fe24e855700 1 -- 10.0.0.12:6801/3916 <== client.28278 10.0.0.12:0/1078698 44 ==== osd_op(client.28278.0:7861 rbd_header.6e7074b0dc51 [watch ping cookie 140379596790576 gen 1] 14.6c8b444f ondisk+write+known_if_redirected e125) v5 ==== 182+0+0 (283630935 0 0) 0x7b15780 con 0x8271ce0
|
|
-15> 2015-12-02 19:19:29.886544 7fe24e855700 5 -- op tracker -- seq: 243958, time: 2015-12-02 19:19:29.316616, event: header_read, op: osd_op(client.28278.0:7861 rbd_header.6e7074b0dc51 [watch ping cookie 140379596790576 gen 1] 14.6c8b444f ondisk+write+known_if_redirected e125)
|
|
-14> 2015-12-02 19:19:29.886569 7fe24e855700 5 -- op tracker -- seq: 243958, time: 2015-12-02 19:19:29.316632, event: throttled, op: osd_op(client.28278.0:7861 rbd_header.6e7074b0dc51 [watch ping cookie 140379596790576 gen 1] 14.6c8b444f ondisk+write+known_if_redirected e125)
|
|
-13> 2015-12-02 19:19:29.886581 7fe24e855700 5 -- op tracker -- seq: 243958, time: 2015-12-02 19:19:29.886348, event: all_read, op: osd_op(client.28278.0:7861 rbd_header.6e7074b0dc51 [watch ping cookie 140379596790576 gen 1] 14.6c8b444f ondisk+write+known_if_redirected e125)
|
|
-12> 2015-12-02 19:19:29.886670 7fe24e855700 5 -- op tracker -- seq: 243958, time: 0.000000, event: dispatched, op: osd_op(client.28278.0:7861 rbd_header.6e7074b0dc51 [watch ping cookie 140379596790576 gen 1] 14.6c8b444f ondisk+write+known_if_redirected e125)
|
|
-11> 2015-12-02 19:19:30.108618 7fe257390700 5 -- op tracker -- seq: 243956, time: 2015-12-02 19:19:30.108618, event: started, op: osd_op(client.28278.0:7860 rbd_header.6e7074b0dc51 [watch ping cookie 140379596790576 gen 1] 14.6c8b444f ondisk+write+known_if_redirected e125)
|
|
-10> 2015-12-02 19:19:30.146859 7fe2633a8700 1 -- 10.0.0.12:6801/3916 <== client.28278 10.0.0.12:0/1078698 45 ==== ping magic: 0 v1 ==== 0+0+0 (0 0 0) 0x823bd40 con 0x8271ce0
|
|
-9> 2015-12-02 19:19:30.225214 7fe2633a8700 1 -- 10.0.0.12:6801/3916 <== client.28261 10.0.0.12:0/1078132 45 ==== ping magic: 0 v1 ==== 0+0+0 (0 0 0) 0x823ec80 con 0x8270c60
|
|
-8> 2015-12-02 19:19:30.642371 7fe26a64d700 5 osd.1 125 tick
|
|
-7> 2015-12-02 19:19:30.642416 7fe26a64d700 -1 osd.1 125 heartbeat_check: no reply from osd.0 since back 2015-12-02 19:06:06.939775 front 2015-12-02 19:06:06.939775 (cutoff 2015-12-02 19:19:10.642412)
|
|
-6> 2015-12-02 19:19:30.642637 7fe26a64d700 0 log_channel(cluster) log [WRN] : 7 slow requests, 7 included below; oldest blocked for > 744.823843 secs
|
|
-5> 2015-12-02 19:19:30.642649 7fe26a64d700 0 log_channel(cluster) log [WRN] : slow request 744.823843 seconds old, received at 2015-12-02 19:07:05.818637: osd_op(client.28261.0:26026 rbd_header.6e5c2ae8944a [watch ping cookie 140295290267568 gen 8] 14.64552175 ondisk+write+known_if_redirected e125) currently started
|
|
-4> 2015-12-02 19:19:30.642683 7fe26a64d700 0 log_channel(cluster) log [WRN] : slow request 659.684859 seconds old, received at 2015-12-02 19:08:30.957621: osd_op(client.28261.0:26027 rbd_header.6e5c2ae8944a [watch ping cookie 140295290267568 gen 8] 14.64552175 ondisk+write+known_if_redirected e125) currently no flag points reached
|
|
-3> 2015-12-02 19:19:30.642700 7fe26a64d700 0 log_channel(cluster) log [WRN] : slow request 659.684707 seconds old, received at 2015-12-02 19:08:30.957773: osd_op(client.28261.0:26028 rbd_header.6e5c2ae8944a [watch ping cookie 140295290267568 gen 8] 14.64552175 ondisk+write+known_if_redirected e125) currently no flag points reached
|
|
-2> 2015-12-02 19:19:30.642708 7fe26a64d700 0 log_channel(cluster) log [WRN] : slow request 659.684666 seconds old, received at 2015-12-02 19:08:30.957814: osd_op(client.28261.0:26029 rbd_header.6e5c2ae8944a [watch ping cookie 140295290267568 gen 8] 14.64552175 ondisk+write+known_if_redirected e125) currently no flag points reached
|
|
-1> 2015-12-02 19:19:30.642718 7fe26a64d700 0 log_channel(cluster) log [WRN] : slow request 656.756976 seconds old, received at 2015-12-02 19:08:33.885504: osd_op(client.28261.0:26030 rbd_header.6e5c2ae8944a [watch ping cookie 140295290267568 gen 8] 14.64552175 ondisk+write+known_if_redirected e125) currently no flag points reached
|
|
0> 2015-12-02 19:19:31.470317 7fe2633a8700 10 monclient(hunting): renew_subs
|
|
--- logging levels ---
|
|
0/ 5 none
|
|
0/ 1 lockdep
|
|
0/ 1 context
|
|
1/ 1 crush
|
|
1/ 5 mds
|
|
1/ 5 mds_balancer
|
|
1/ 5 mds_locker
|
|
1/ 5 mds_log
|
|
1/ 5 mds_log_expire
|
|
1/ 5 mds_migrator
|
|
0/ 1 buffer
|
|
0/ 1 timer
|
|
0/ 1 filer
|
|
0/ 1 striper
|
|
0/ 1 objecter
|
|
0/ 5 rados
|
|
0/ 5 rbd
|
|
0/ 5 rbd_replay
|
|
0/ 5 journaler
|
|
0/ 5 objectcacher
|
|
0/ 5 client
|
|
0/ 5 osd
|
|
0/ 5 optracker
|
|
0/ 5 objclass
|
|
1/ 3 filestore
|
|
1/ 3 keyvaluestore
|
|
1/ 3 journal
|
|
0/ 5 ms
|
|
1/ 5 mon
|
|
0/10 monc
|
|
1/ 5 paxos
|
|
0/ 5 tp
|
|
1/ 5 auth
|
|
1/ 5 crypto
|
|
1/ 1 finisher
|
|
1/ 5 heartbeatmap
|
|
1/ 5 perfcounter
|
|
1/ 5 rgw
|
|
1/10 civetweb
|
|
1/ 5 javaclient
|
|
1/ 5 asok
|
|
1/ 1 throttle
|
|
0/ 0 refs
|
|
1/ 5 xio
|
|
-2/-2 (syslog threshold)
|
|
-1/-1 (stderr threshold)
|
|
max_recent 10000
|
|
max_new 1000
|
|
log_file /var/log/ceph/ceph-osd.1.log
|
|
--- end dump of recent events ---
|