Ceph » rbd

   -13> 2019-05-10 19:48:47.259 7f4023448700  1 -- 172.21.15.183:0/3214476863 <== osd.3 v2:172.21.15.174:6802/34088 24 ==== osd_op_reply(66 rbd_header.2345ae263717 [notify cookie 94285965296256] v0'0 uv7 ondisk = 0) v8 ==== 167+0+8 (crc 0 0 0) 0x55c0a9245b80 con 0x55c0a8172000
   -12> 2019-05-10 19:48:47.260 7f4023448700  1 -- 172.21.15.183:0/3214476863 <== osd.3 v2:172.21.15.174:6802/34088 25 ==== watch-notify(notify_complete (2) cookie 94285965296256 notify 399431958590 ret 0) v3 ==== 42+0+38 (crc 0 0 0) 0x55c0a967ab40 con 0x55c0a8172000
   -11> 2019-05-10 19:48:47.260 7f4023448700  1 -- 172.21.15.183:0/3214476863 <== osd.3 v2:172.21.15.174:6802/34088 26 ==== osd_op_reply(67 rbd_header.2345ae263717 [notify-ack] v0'0 uv7 ondisk = 0) v8 ==== 167+0+0 (crc 0 0 0) 0x55c0a9245b80 con 0x55c0a8172000
   -10> 2019-05-10 19:48:47.260 7f3ff854d700  5 librbd::Watcher: 0x55c0a9244a00 block_notifies: blocked_count=1
    -9> 2019-05-10 19:48:47.260 7f3ff3d44700  5 librbd::io::ImageRequestWQ: 0x55c0a8c6b7a0 shut_down: shut_down: in_flight=0
    -8> 2019-05-10 19:48:47.260 7f3ff3d44700  1 -- 172.21.15.183:0/3214476863 --> [v2:172.21.15.174:6802/34088,v1:172.21.15.174:6803/34088] -- osd_op(unknown.0.0:68 1.4 1:26dfb1b6:::rbd_header.2345ae263717:head [watch unwatch cookie 94285959810816] snapc 0=[] ondisk+write+known_if_redirected e93) v8 -- 0x55c0a9339080 con 0x55c0a8172000
    -7> 2019-05-10 19:48:47.263 7f4023448700  1 -- 172.21.15.183:0/3214476863 <== osd.3 v2:172.21.15.174:6802/34088 27 ==== osd_op_reply(68 rbd_header.2345ae263717 [watch unwatch cookie 94285959810816] v93'8 uv7 ondisk = 0) v8 ==== 167+0+0 (crc 0 0 0) 0x55c0a9245b80 con 0x55c0a8172000
    -6> 2019-05-10 19:48:47.263 7f3ff3d44700  5 librbd::io::ObjectDispatcher: 0x55c0a93d6ab0 shut_down: 
    -5> 2019-05-10 19:48:47.263 7f3ff3d44700  5 librbd::io::ObjectDispatch: 0x55c0a8bc7740 shut_down: 
    -4> 2019-05-10 19:48:47.263 7f3ff3d44700  5 librbd::io::SimpleSchedulerObjectDispatch: 0x55c0a9799930 shut_down: 
    -3> 2019-05-10 19:48:47.263 7f3ff3d44700  5 librbd::cache::WriteAroundObjectDispatch: 0x55c0a9485680 shut_down: 
    -2> 2019-05-10 19:48:47.263 7f3ff3d44700  5 asok(0x55c0a4a24000) unregister_command rbd cache flush rbd/cimg
    -1> 2019-05-10 19:48:47.263 7f3ff3d44700  5 asok(0x55c0a4a24000) unregister_command rbd cache invalidate rbd/cimg
     0> 2019-05-10 19:48:47.265 7f3ff854d700 -1 *** Caught signal (Aborted) **
 in thread 7f3ff854d700 thread_name:dashboard

 ceph version 15.0.0-1187-g3cf15ba (3cf15babf04e7149f49d9537540367b53b2f0285) octopus (dev)
 1: (()+0xf6d0) [0x7f4026f816d0]
 2: (gsignal()+0x37) [0x7f4025b64277]
 3: (abort()+0x148) [0x7f4025b65968]
 4: (tcmalloc::Log(tcmalloc::LogMode, char const*, int, tcmalloc::LogItem, tcmalloc::LogItem, tcmalloc::LogItem, tcmalloc::LogItem)+0x1e6) [0x7f4028a85a66]
 5: (()+0x17644) [0x7f4028a7a644]
 6: (()+0xbcd1d) [0x7f400c07bd1d]
 7: (()+0xb393a) [0x7f400c07293a]
 8: (()+0xc8e0c) [0x7f400c087e0c]
 9: (rbd_close()+0x24) [0x7f400c040944]
 10: (()+0x2974d) [0x7f400c6c474d]
 11: (PyObject_Call()+0x43) [0x7f40290a7a63]
 12: (()+0x1c4ee) [0x7f400c6b74ee]
 13: (PyObject_Call()+0x43) [0x7f40290a7a63]
 14: (PyObject_CallFunctionObjArgs()+0xbc) [0x7f40290a835c]
 15: (PyEval_EvalFrameEx()+0x43d4) [0x7f402913e2d4]
 16: (PyEval_EvalCodeEx()+0x7ed) [0x7f402914303d]
 17: (()+0x70a6d) [0x7f40290cca6d]
 18: (PyObject_Call()+0x43) [0x7f40290a7a63]
 19: (PyEval_EvalFrameEx()+0x17fd) [0x7f402913b6fd]
 20: (PyEval_EvalCodeEx()+0x7ed) [0x7f402914303d]
 21: (()+0x70a6d) [0x7f40290cca6d]
 22: (PyObject_Call()+0x43) [0x7f40290a7a63]
 23: (PyEval_EvalFrameEx()+0x17fd) [0x7f402913b6fd]
 24: (PyEval_EvalCodeEx()+0x7ed) [0x7f402914303d]
 25: (PyEval_EvalFrameEx()+0x663c) [0x7f402914053c]
 26: (PyEval_EvalCodeEx()+0x7ed) [0x7f402914303d]
 27: (()+0x70a6d) [0x7f40290cca6d]
 28: (PyObject_Call()+0x43) [0x7f40290a7a63]
 29: (PyEval_EvalFrameEx()+0x17fd) [0x7f402913b6fd]
 30: (PyEval_EvalCodeEx()+0x7ed) [0x7f402914303d]
 31: (()+0x70a6d) [0x7f40290cca6d]
 32: (PyObject_Call()+0x43) [0x7f40290a7a63]
 33: (PyEval_EvalFrameEx()+0x17fd) [0x7f402913b6fd]
 34: (PyEval_EvalCodeEx()+0x7ed) [0x7f402914303d]
 35: (()+0x70a6d) [0x7f40290cca6d]
 36: (PyObject_Call()+0x43) [0x7f40290a7a63]
 37: (PyEval_EvalFrameEx()+0x17fd) [0x7f402913b6fd]
 38: (PyEval_EvalCodeEx()+0x7ed) [0x7f402914303d]
 39: (()+0x70a6d) [0x7f40290cca6d]
 40: (PyObject_Call()+0x43) [0x7f40290a7a63]
 41: (PyEval_EvalFrameEx()+0x17fd) [0x7f402913b6fd]
 42: (PyEval_EvalFrameEx()+0x67bd) [0x7f40291406bd]
 43: (PyEval_EvalFrameEx()+0x67bd) [0x7f40291406bd]
 44: (PyEval_EvalCodeEx()+0x7ed) [0x7f402914303d]
 45: (()+0x70978) [0x7f40290cc978]
 46: (PyObject_Call()+0x43) [0x7f40290a7a63]
 47: (()+0x5aa55) [0x7f40290b6a55]
 48: (PyObject_Call()+0x43) [0x7f40290a7a63]
 49: (PyEval_CallObjectWithKeywords()+0x47) [0x7f40291398f7]
 50: (()+0x115822) [0x7f4029171822]
 51: (()+0x7e25) [0x7f4026f79e25]
 52: (clone()+0x6d) [0x7f4025c2cbad]

/a/sage-2019-05-10_17:46:31-rados-wip-sage-testing-2019-05-10-0936-distro-basic-smithi/3946909