Project

General

Profile

Actions

Bug #55101

open

mon has slow op

Added by liqun zhang about 2 years ago. Updated about 2 years ago.

Status:
New
Priority:
Normal
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

there are 3 nodes in our cluster, 84 OSD each node.
after execute "systemctl restart ceph-osd.target" on node2, there is 1 slow op in mon.b

"ops": [
       {
"description": "osd_alive(want up_thru 1348 have 1348)",
"initiated_at": "2022-03-29T12:10:11.016387+0800",
"age": 9036.0939306100008,
"duration": 9036.0939519370004,
"type_data": {
"events": [ {
"time": "2022-03-29T12:10:11.016387+0800",
"event": "initiated"
}, {
"time": "2022-03-29T12:10:11.016391+0800",
"event": "throttled"
}, {
"time": "2022-03-29T12:10:11.016387+0800",
"event": "header_read"
}, {
"time": "2022-03-29T12:10:11.016404+0800",
"event": "all_read"
}, {
"time": "2022-03-29T12:10:11.016667+0800",
"event": "dispatched"
}, {
"time": "2022-03-29T12:10:11.016675+0800",
"event": "mon:_ms_dispatch"
}, {
"time": "2022-03-29T12:10:11.016677+0800",
"event": "mon:dispatch_op"
}, {
"time": "2022-03-29T12:10:11.016678+0800",
"event": "psvc:dispatch"
}, {
"time": "2022-03-29T12:10:11.016699+0800",
"event": "osdmap:preprocess_query"
}, {
"time": "2022-03-29T12:10:11.016702+0800",
"event": "osdmap:preprocess_alive"
}, {
"time": "2022-03-29T12:10:11.016717+0800",
"event": "osdmap:wait_for_writeable"
}, {
"time": "2022-03-29T12:10:11.016718+0800",
"event": "osdmap:wait_for_finished_proposal"
}, {
"time": "2022-03-29T12:10:11.721484+0800",
"event": "callback finished"
}, {
"time": "2022-03-29T12:10:11.721485+0800",
"event": "psvc:dispatch"
}, {
"time": "2022-03-29T12:10:11.721499+0800",
"event": "osdmap:preprocess_query"
}, {
"time": "2022-03-29T12:10:11.721502+0800",
"event": "osdmap:preprocess_alive"
}, {
"time": "2022-03-29T12:10:11.721510+0800",
"event": "osdmap:prepare_update"
}, {
"time": "2022-03-29T12:10:11.721512+0800",
"event": "osdmap:prepare_alive"
}, {
"time": "2022-03-29T12:10:11.721515+0800",
"event": "osdmap:wait_for_finished_proposal"
}, {
"time": "2022-03-29T12:10:12.548334+0800",
"event": "callback retry"
}, {
"time": "2022-03-29T12:10:12.548335+0800",
"event": "psvc:dispatch"
}, {
"time": "2022-03-29T12:10:12.548350+0800",
"event": "osdmap:wait_for_readable"
}, {
"time": "2022-03-29T12:10:12.548352+0800",
"event": "osdmap:wait_for_readable/paxos"
}, {
"time": "2022-03-29T12:10:12.548360+0800",
"event": "paxos:wait_for_readable"
}, {
"time": "2022-03-29T12:10:12.775644+0800",
"event": "callback finished"
}, {
"time": "2022-03-29T12:10:12.775645+0800",
"event": "psvc:dispatch"
}, {
"time": "2022-03-29T12:10:12.775669+0800",
"event": "osdmap:preprocess_query"
}, {
"time": "2022-03-29T12:10:12.775671+0800",
"event": "osdmap:preprocess_alive"
}, {
"time": "2022-03-29T12:10:12.775683+0800",
"event": "forward_request_leader"
}, {
"time": "2022-03-29T12:10:12.775758+0800",
"event": "forwarded"
}
],
"info": {
"seq": 9009263,
"src_is_mon": false,
"source": "osd.128 v1:142.137.11.142:7263/1961019",
"forwarded_to_leader": true
}
}

Files

55101.z01 (1000 KB) 55101.z01 liqun zhang, 04/07/2022 05:35 AM
55101.z03 (1000 KB) 55101.z03 liqun zhang, 04/07/2022 05:35 AM
55101.z02 (1000 KB) 55101.z02 liqun zhang, 04/07/2022 05:35 AM
55101.z04 (1000 KB) 55101.z04 liqun zhang, 04/07/2022 05:35 AM
55101.z05 (1000 KB) 55101.z05 liqun zhang, 04/07/2022 05:35 AM
55101.zip (269 KB) 55101.zip liqun zhang, 04/07/2022 05:35 AM
55101.z06 (1000 KB) 55101.z06 liqun zhang, 04/07/2022 05:36 AM
Actions #1

Updated by Radoslaw Zarzynski about 2 years ago

  • Status changed from New to Need More Info

Hello! We would need to take a look on the mon.b's log, preferably also on the one preceding the restart.

Actions #3

Updated by liqun zhang about 2 years ago

the attachment includes:
ceph-mon.a.log ceph-mon.b.log ceph-mon.c.log
b.ops (ceph daemon mon.b ops)
c.ops(ceph daemon mon.c ops)
ceph_s.out(ceph -s)

Actions #4

Updated by jianwei zhang about 2 years ago

ceph tag v15.2.13

Actions #5

Updated by Radoslaw Zarzynski about 2 years ago

  • Status changed from Need More Info to New
  • Assignee set to Siddharth Sharma
Actions

Also available in: Atom PDF