https://tracker.ceph.com/
https://tracker.ceph.com/favicon.ico
2018-06-15T14:41:42Z
Ceph
RADOS - Bug #24531: Mimic MONs have slow/long running ops
https://tracker.ceph.com/issues/24531?journal_id=115148
2018-06-15T14:41:42Z
Greg Farnum
gfarnum@redhat.com
<ul></ul><p>What's the output of "ceph versions" on this cluster?</p>
<p>We had issues in the lab with OSD failure reports not getting cleaned up properly from that op tracker, but I don't think that particular log has turned up and it's a bit confusing how that could have happened.</p>
RADOS - Bug #24531: Mimic MONs have slow/long running ops
https://tracker.ceph.com/issues/24531?journal_id=115149
2018-06-15T14:48:22Z
Wido den Hollander
wido@42on.com
<ul></ul><pre>
{
"mon": {
"ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable)": 3
},
"mgr": {
"ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable)": 3
},
"osd": {
"ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable)": 288
},
"mds": {
"ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable)": 3
},
"overall": {
"ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable)": 297
}
}
</pre>
<p>This cluster was installed today and we started to do some physical tests by pulling disks, pulling power cords, etc, etc.</p>
<p>Everything recovered just fine, but I saw these messages pop up in the logs and also in 'ceph health'.</p>
<p>Although the message is gone in the status the ops are still there.</p>
RADOS - Bug #24531: Mimic MONs have slow/long running ops
https://tracker.ceph.com/issues/24531?journal_id=115450
2018-06-20T21:36:00Z
Josh Durgin
<ul><li><strong>Assignee</strong> set to <i>Joao Eduardo Luis</i></li></ul><p>Joao, could you take a look at this?</p>
RADOS - Bug #24531: Mimic MONs have slow/long running ops
https://tracker.ceph.com/issues/24531?journal_id=126613
2018-12-28T15:25:40Z
Hector Martin
marcan@marcan.st
<ul></ul><p>I just hit this on a 13.2.1 single-host cluster with 1 mon and 8 OSDs. The log is basically identical to the one Wido reported. It seems osd.3 flapped and left behind two stuck ops with identical events.</p>
RADOS - Bug #24531: Mimic MONs have slow/long running ops
https://tracker.ceph.com/issues/24531?journal_id=126700
2019-01-02T22:17:31Z
Josh Durgin
<ul><li><strong>Duplicated by</strong> <i><a class="issue tracker-1 status-10 priority-4 priority-default closed" href="/issues/37768">Bug #37768</a>: mon gets stuck op for failing OSDs</i> added</li></ul>
RADOS - Bug #24531: Mimic MONs have slow/long running ops
https://tracker.ceph.com/issues/24531?journal_id=126702
2019-01-02T22:18:02Z
Josh Durgin
<ul><li><strong>Priority</strong> changed from <i>Normal</i> to <i>High</i></li></ul>
RADOS - Bug #24531: Mimic MONs have slow/long running ops
https://tracker.ceph.com/issues/24531?journal_id=127449
2019-01-15T09:14:31Z
Paul Emmerich
paul.emmerich@oocero.de
<ul></ul><p>I've seen this on a 13.2.2 cluster after restarting OSDs</p>
RADOS - Bug #24531: Mimic MONs have slow/long running ops
https://tracker.ceph.com/issues/24531?journal_id=128006
2019-01-23T21:04:33Z
Tobias Rehn
<ul></ul><p>I am also seeing this on latest mimic (13.2.4). So far it seems like its cosmetic and has no impact.</p>
<pre><code class="text syntaxhl"><span class="CodeRay">[root@mon01 ~]# ceph -s
cluster:
id: 7b62928c-3bb4-4196-a7c7-5da4fad63c53
health: HEALTH_WARN
4 slow ops, oldest one blocked for 103474 sec, daemons [mon.mon01,mon.mon02,mon.mon03] have slow ops.
</span></code></pre>
RADOS - Bug #24531: Mimic MONs have slow/long running ops
https://tracker.ceph.com/issues/24531?journal_id=128115
2019-01-25T14:42:32Z
Stig Telfer
<ul></ul><p>I see the same symptoms on a system running 13.2.2 - each monitor has a small number of slow ops, all initiated within a couple of minutes of one another about six weeks ago.</p>
RADOS - Bug #24531: Mimic MONs have slow/long running ops
https://tracker.ceph.com/issues/24531?journal_id=128158
2019-01-26T16:44:41Z
Paul Emmerich
paul.emmerich@oocero.de
<ul></ul><p>I've now encountered this on a total of 3 different clusters with 13.2.2 and 13.2.4</p>
RADOS - Bug #24531: Mimic MONs have slow/long running ops
https://tracker.ceph.com/issues/24531?journal_id=128513
2019-01-31T10:38:21Z
Марк Коренберг
socketpair@gmail.com
<ul></ul><p>Seems, the same:</p>
<pre>
mmarkk@mmwork:/mnt/cephfs/tg/.snap$ ceph -s
cluster:
id: 56ed206b-67cf-42a6-be65-9baf32334fc9
health: HEALTH_WARN
3 slow ops, oldest one blocked for 150263 sec, mon.node3 has slow ops
</pre>
<pre>
{
"ops": [
{
"description": "remove_snaps({52=[2,3]} v0)",
"initiated_at": "2019-01-29 21:48:01.206747",
"age": 150508.902028,
"duration": 150508.902042,
"type_data": {
"events": [
{
"time": "2019-01-29 21:48:01.206747",
"event": "initiated"
},
{
"time": "2019-01-29 21:48:01.206747",
"event": "header_read"
},
{
"time": "2019-01-29 21:48:01.206749",
"event": "throttled"
},
{
"time": "2019-01-29 21:48:01.206754",
"event": "all_read"
},
{
"time": "2019-01-29 21:48:01.206785",
"event": "dispatched"
},
{
"time": "2019-01-29 21:48:01.206789",
"event": "mon:_ms_dispatch"
},
{
"time": "2019-01-29 21:48:01.206790",
"event": "mon:dispatch_op"
},
{
"time": "2019-01-29 21:48:01.206791",
"event": "psvc:dispatch"
},
{
"time": "2019-01-29 21:48:01.206802",
"event": "osdmap:preprocess_query"
},
{
"time": "2019-01-29 21:48:01.206804",
"event": "osdmap:preprocess_remove_snaps"
},
{
"time": "2019-01-29 21:48:01.206810",
"event": "forward_request_leader"
},
{
"time": "2019-01-29 21:48:01.206839",
"event": "forwarded"
}
],
"info": {
"seq": 1364847,
"src_is_mon": false,
"source": "mds.0 10.80.20.103:6808/1421988905",
"forwarded_to_leader": true
}
}
},
{
"description": "remove_snaps({52=[4,5]} v0)",
"initiated_at": "2019-01-29 21:48:51.202066",
"age": 150458.906709,
"duration": 150458.908302,
"type_data": {
"events": [
{
"time": "2019-01-29 21:48:51.202066",
"event": "initiated"
},
{
"time": "2019-01-29 21:48:51.202066",
"event": "header_read"
},
{
"time": "2019-01-29 21:48:51.202068",
"event": "throttled"
},
{
"time": "2019-01-29 21:48:51.202072",
"event": "all_read"
},
{
"time": "2019-01-29 21:48:51.202102",
"event": "dispatched"
},
{
"time": "2019-01-29 21:48:51.202106",
"event": "mon:_ms_dispatch"
},
{
"time": "2019-01-29 21:48:51.202107",
"event": "mon:dispatch_op"
},
{
"time": "2019-01-29 21:48:51.202107",
"event": "psvc:dispatch"
},
{
"time": "2019-01-29 21:48:51.202117",
"event": "osdmap:preprocess_query"
},
{
"time": "2019-01-29 21:48:51.202118",
"event": "osdmap:preprocess_remove_snaps"
},
{
"time": "2019-01-29 21:48:51.202123",
"event": "forward_request_leader"
},
{
"time": "2019-01-29 21:48:51.202151",
"event": "forwarded"
}
],
"info": {
"seq": 1365062,
"src_is_mon": false,
"source": "mds.0 10.80.20.103:6808/1421988905",
"forwarded_to_leader": true
}
}
},
{
"description": "remove_snaps({52=[6,7]} v0)",
"initiated_at": "2019-01-29 21:58:01.208627",
"age": 149908.900147,
"duration": 149908.901793,
"type_data": {
"events": [
{
"time": "2019-01-29 21:58:01.208627",
"event": "initiated"
},
{
"time": "2019-01-29 21:58:01.208627",
"event": "header_read"
},
{
"time": "2019-01-29 21:58:01.208629",
"event": "throttled"
},
{
"time": "2019-01-29 21:58:01.208635",
"event": "all_read"
},
{
"time": "2019-01-29 21:58:01.208667",
"event": "dispatched"
},
{
"time": "2019-01-29 21:58:01.208671",
"event": "mon:_ms_dispatch"
},
{
"time": "2019-01-29 21:58:01.208672",
"event": "mon:dispatch_op"
},
{
"time": "2019-01-29 21:58:01.208672",
"event": "psvc:dispatch"
},
{
"time": "2019-01-29 21:58:01.208682",
"event": "osdmap:preprocess_query"
},
{
"time": "2019-01-29 21:58:01.208683",
"event": "osdmap:preprocess_remove_snaps"
},
{
"time": "2019-01-29 21:58:01.208688",
"event": "forward_request_leader"
},
{
"time": "2019-01-29 21:58:01.208736",
"event": "forwarded"
}
],
"info": {
"seq": 1367397,
"src_is_mon": false,
"source": "mds.0 10.80.20.103:6808/1421988905",
"forwarded_to_leader": true
}
}
}
],
"num_ops": 3
}
</pre>
RADOS - Bug #24531: Mimic MONs have slow/long running ops
https://tracker.ceph.com/issues/24531?journal_id=128514
2019-01-31T10:43:09Z
Марк Коренберг
socketpair@gmail.com
<ul></ul><p>I have restarted mon.node3 and now everything is OK.</p>
RADOS - Bug #24531: Mimic MONs have slow/long running ops
https://tracker.ceph.com/issues/24531?journal_id=134845
2019-04-17T06:58:30Z
Dan van der Ster
<ul></ul><p>We had this happen twice this week on a v13.2.5 cluster. (The cluster was recently upgraded from v12.2.11, where this never happened before.)</p>
<p>On our cluster it happens like this:</p>
<ul>
<li>some osd flapping glitch (many osds all go down and up for a few minutes)</li>
<li>osds resolve themselves, all coming back up, and PGs getting active correctly</li>
<li>a few thousand leftover slow requests on the mons that never resolve:<br /><pre>
health: HEALTH_WARN
5782 slow ops, oldest one blocked for 24106 sec, daemons [mon.cepherin-mon-084bea7b06,mon.cepherin-mon-7cb9b591e1,mon.cepherin1] have slow ops.
</pre></li>
</ul>
<ul>
<li>I've dumped the ops on the peon and leader mons.
<ul>
<li>Peon:<br /><pre>
"description": "osd_alive(want up_thru 313212 have 313212)",
... a few thousand times
"description": "osd_alive(want up_thru 313213 have 313213)",
"description": "osd_alive(want up_thru 313218 have 313218)",
"description": "osd_failure(failed timeout osd.1283 128.142.25.117:6908/2980276 for 10sec e313238 v313238)",
</pre></li>
</ul>
<ul>
<li>Leader:<br /><pre>
"description": "osd_failure(failed timeout osd.483 128.142.212.174:6878/3315556 for 21sec e313146 v313146)",
"description": "osd_failure(failed timeout osd.485 128.142.212.174:6825/3315534 for 21sec e313146 v313146)",
"description": "osd_failure(failed timeout osd.56 128.142.168.12:6876/337817 for 20sec e313159 v313159)",
"description": "osd_failure(failed timeout osd.527 128.142.210.141:6865/3134286 for 23sec e313176 v313176)",
"description": "osd_failure(failed timeout osd.264 128.142.162.136:6832/1715260 for 21sec e313159 v313159)",
"description": "osd_failure(failed timeout osd.5 128.142.168.12:6849/337797 for 22sec e313154 v313154)",
"description": "osd_failure(failed timeout osd.126 128.142.162.75:6885/4002283 for 24sec e313154 v313154)",
"description": "osd_failure(failed timeout osd.143 128.142.162.75:6887/4002279 for 50sec e313200 v313200)",
"description": "osd_failure(failed timeout osd.236 128.142.162.78:6812/2713226 for 29sec e313157 v313157)",
"description": "osd_failure(failed timeout osd.238 128.142.162.78:6943/2713280 for 29sec e313157 v313157)",
"description": "osd_failure(failed timeout osd.1248 128.142.25.108:6832/2865592 for 20sec e313140 v313140)",
"description": "osd_failure(failed timeout osd.1283 128.142.25.117:6908/2980276 for 10sec e313238 v313238)",
</pre></li>
<li>The leader ops are all stuck in:<br /><pre>
{
"time": "2019-04-16 23:49:42.506169",
"event": "osdmap:prepare_failure"
}
</pre></li>
</ul>
</li>
<li>workaround to remove these mon slow reqs is to restart the mons (in any order).</li>
</ul>
<p>I've posted all logs here: ceph-post-file: c2082cef-f867-4084-abb8-510e865a6c6e<br />Leader ops are here: ceph-post-file: e48ff1e2-0256-432b-894a-d84aa7230bf6<br />Peon ops are here: ceph-post-file: 28e55100-1c1b-4d37-b280-5651d91f28c8</p>
RADOS - Bug #24531: Mimic MONs have slow/long running ops
https://tracker.ceph.com/issues/24531?journal_id=137045
2019-05-21T14:20:03Z
jun gong
<ul></ul><p>same problem with Dan van der Ster,on a v13.2.5 cluster five hours ago.<br />I restart osd.0 when monitor logs show oldest slow ops(osd failure osd.0 ......),then monitor logs show oldest slow ops on osd.1<br />finally,i restart all 3 osds,then this problem resolved.</p>
RADOS - Bug #24531: Mimic MONs have slow/long running ops
https://tracker.ceph.com/issues/24531?journal_id=137069
2019-05-22T06:41:17Z
jun gong
<ul><li><strong>File</strong> <a href="/attachments/download/4205/mon.a.slowops">mon.a.slowops</a> added</li><li><strong>File</strong> <a href="/attachments/download/4203/mon.b.slowops">mon.b.slowops</a> added</li><li><strong>File</strong> <a href="/attachments/download/4204/mon.c.slowops">mon.c.slowops</a> added</li></ul><p>The attached file is three mon's dump_historic_slow_ops file.</p>
<p>I deploy v13.2.5 ceph by rook in kunnertes cluster,I restart all three osd one by one resolved this problem,so I can't found osd's slow ops.</p>
<p>The problem happend at 2019-05-21 09:18,and i restart three osd ordered between 10:00 to 10:30.</p>
RADOS - Bug #24531: Mimic MONs have slow/long running ops
https://tracker.ceph.com/issues/24531?journal_id=137070
2019-05-22T06:50:21Z
Dan van der Ster
<ul></ul><p>Joao sent this as a possible fix: <a class="external" href="https://github.com/ceph/ceph/pull/28177">https://github.com/ceph/ceph/pull/28177</a></p>
RADOS - Bug #24531: Mimic MONs have slow/long running ops
https://tracker.ceph.com/issues/24531?journal_id=142535
2019-08-06T16:04:17Z
Theo O
<ul></ul><p>I am encountering similar issues on a cluster with all daemons running<br /><pre><code class="text syntaxhl"><span class="CodeRay">ceph version 14.2.2 (4f8fa0a0024755aae7d95567c63f11d6862d55be) nautilus (stable)
</span></code></pre><br />with the following ops on the leader mon (with node IPs redacted)</p>
<pre><code class="text syntaxhl"><span class="CodeRay">{
"ops": [
{
"description": "osd_failure(failed timeout osd.23 [v2:osd-node1:6817/151142,v1:osd-node1:6818/151142] for 28sec e13553 v13553)",
"initiated_at": "2019-08-06 11:31:53.163578",
"age": 1308.2272771569999,
"duration": 1308.227311912,
"type_data": {
"events": [
{
"time": "2019-08-06 11:31:53.163578",
"event": "initiated"
},
{
"time": "2019-08-06 11:31:53.163578",
"event": "header_read"
},
{
"time": "2019-08-06 11:31:53.163577",
"event": "throttled"
},
{
"time": "2019-08-06 11:31:53.163580",
"event": "all_read"
},
{
"time": "2019-08-06 11:31:53.164160",
"event": "dispatched"
},
{
"time": "2019-08-06 11:31:53.164162",
"event": "mon:_ms_dispatch"
},
{
"time": "2019-08-06 11:31:53.164162",
"event": "mon:dispatch_op"
},
{
"time": "2019-08-06 11:31:53.164162",
"event": "psvc:dispatch"
},
{
"time": "2019-08-06 11:31:53.164168",
"event": "osdmap:preprocess_query"
},
{
"time": "2019-08-06 11:31:53.164169",
"event": "osdmap:preprocess_failure"
},
{
"time": "2019-08-06 11:31:53.164171",
"event": "osdmap:prepare_update"
},
{
"time": "2019-08-06 11:31:53.164172",
"event": "osdmap:prepare_failure"
}
],
"info": {
"seq": 1290829,
"src_is_mon": false,
"source": "osd.17 v2:osd-node2:6813/149653",
"forwarded_to_leader": false
}
}
},
{
"description": "osd_failure(failed timeout osd.24 [v2:osd-node1:6807/151331,v1:osd-node1:6809/151331] for 28sec e13553 v13553)",
"initiated_at": "2019-08-06 11:31:53.164483",
"age": 1308.2263727659999,
"duration": 1308.2265255320001,
"type_data": {
"events": [
{
"time": "2019-08-06 11:31:53.164483",
"event": "initiated"
},
{
"time": "2019-08-06 11:31:53.164483",
"event": "header_read"
},
{
"time": "2019-08-06 11:31:53.163582",
"event": "throttled"
},
{
"time": "2019-08-06 11:31:53.164500",
"event": "all_read"
},
{
"time": "2019-08-06 11:31:53.164512",
"event": "dispatched"
},
{
"time": "2019-08-06 11:31:53.164515",
"event": "mon:_ms_dispatch"
},
{
"time": "2019-08-06 11:31:53.164515",
"event": "mon:dispatch_op"
},
{
"time": "2019-08-06 11:31:53.164516",
"event": "psvc:dispatch"
},
{
"time": "2019-08-06 11:31:53.164527",
"event": "osdmap:preprocess_query"
},
{
"time": "2019-08-06 11:31:53.164528",
"event": "osdmap:preprocess_failure"
},
{
"time": "2019-08-06 11:31:53.164532",
"event": "osdmap:prepare_update"
},
{
"time": "2019-08-06 11:31:53.164533",
"event": "osdmap:prepare_failure"
}
],
"info": {
"seq": 1290831,
"src_is_mon": false,
"source": "osd.17 v2:osd-node2:6813/149653",
"forwarded_to_leader": false
}
}
}
],
"num_ops": 2
}
</span></code></pre>
RADOS - Bug #24531: Mimic MONs have slow/long running ops
https://tracker.ceph.com/issues/24531?journal_id=143636
2019-08-20T22:53:26Z
Josh Durgin
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Fix Under Review</i></li><li><strong>Pull request ID</strong> set to <i>28177</i></li></ul>
RADOS - Bug #24531: Mimic MONs have slow/long running ops
https://tracker.ceph.com/issues/24531?journal_id=143874
2019-08-21T21:47:25Z
Greg Farnum
gfarnum@redhat.com
<ul><li><strong>Status</strong> changed from <i>Fix Under Review</i> to <i>7</i></li></ul>
RADOS - Bug #24531: Mimic MONs have slow/long running ops
https://tracker.ceph.com/issues/24531?journal_id=143875
2019-08-21T21:48:23Z
Greg Farnum
gfarnum@redhat.com
<ul><li><strong>Backport</strong> set to <i>nautilus, mimic, luminous</i></li></ul>
RADOS - Bug #24531: Mimic MONs have slow/long running ops
https://tracker.ceph.com/issues/24531?journal_id=146100
2019-09-13T16:29:21Z
Neha Ojha
nojha@redhat.com
<ul><li><strong>Status</strong> changed from <i>7</i> to <i>Pending Backport</i></li></ul>
RADOS - Bug #24531: Mimic MONs have slow/long running ops
https://tracker.ceph.com/issues/24531?journal_id=146195
2019-09-16T07:21:15Z
Nathan Cutler
ncutler@suse.cz
<ul><li><strong>Copied to</strong> <i><a class="issue tracker-9 status-3 priority-4 priority-default closed" href="/issues/41862">Backport #41862</a>: nautilus: Mimic MONs have slow/long running ops</i> added</li></ul>
RADOS - Bug #24531: Mimic MONs have slow/long running ops
https://tracker.ceph.com/issues/24531?journal_id=146197
2019-09-16T07:21:22Z
Nathan Cutler
ncutler@suse.cz
<ul><li><strong>Copied to</strong> <i><a class="issue tracker-9 status-3 priority-4 priority-default closed" href="/issues/41863">Backport #41863</a>: mimic: Mimic MONs have slow/long running ops</i> added</li></ul>
RADOS - Bug #24531: Mimic MONs have slow/long running ops
https://tracker.ceph.com/issues/24531?journal_id=146199
2019-09-16T07:21:29Z
Nathan Cutler
ncutler@suse.cz
<ul><li><strong>Copied to</strong> <i><a class="issue tracker-9 status-3 priority-4 priority-default closed" href="/issues/41864">Backport #41864</a>: luminous: Mimic MONs have slow/long running ops</i> added</li></ul>
RADOS - Bug #24531: Mimic MONs have slow/long running ops
https://tracker.ceph.com/issues/24531?journal_id=150529
2019-11-05T12:25:38Z
Марк Коренберг
socketpair@gmail.com
<ul></ul><p>The same on Nautilus 14.2.3</p>
RADOS - Bug #24531: Mimic MONs have slow/long running ops
https://tracker.ceph.com/issues/24531?journal_id=150872
2019-11-07T21:11:38Z
Aaron Bassett
aaron.bassett@nantomics.com
<ul></ul><p>Seeing the same on 14.2.4</p>
RADOS - Bug #24531: Mimic MONs have slow/long running ops
https://tracker.ceph.com/issues/24531?journal_id=151661
2019-11-14T15:54:46Z
Nathan Cutler
ncutler@suse.cz
<ul><li><strong>Status</strong> changed from <i>Pending Backport</i> to <i>Resolved</i></li></ul><p>While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are in status "Resolved" or "Rejected".</p>
RADOS - Bug #24531: Mimic MONs have slow/long running ops
https://tracker.ceph.com/issues/24531?journal_id=157878
2020-02-05T22:19:49Z
Neha Ojha
nojha@redhat.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-1 status-10 priority-6 priority-high2 closed" href="/issues/43893">Bug #43893</a>: lingering osd_failure ops (due to failure_info holding references?)</i> added</li></ul>
RADOS - Bug #24531: Mimic MONs have slow/long running ops
https://tracker.ceph.com/issues/24531?journal_id=165279
2020-05-08T12:10:16Z
Aleksei Gutikov
aleksey.gutikov@synesis.ru
<ul></ul><p>Something similar, Ceph v14.2.0.</p>
<pre><code class="json syntaxhl"><span class="CodeRay">{
<span class="key"><span class="delimiter">"</span><span class="content">description</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">osd_pgtemp(e49509 {2.162=[378,160,56]} v49509)</span><span class="delimiter">"</span></span>,
<span class="key"><span class="delimiter">"</span><span class="content">initiated_at</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">2020-05-07 14:01:17.837714</span><span class="delimiter">"</span></span>,
<span class="key"><span class="delimiter">"</span><span class="content">age</span><span class="delimiter">"</span></span>: <span class="float">79318.903942007993</span>,
<span class="key"><span class="delimiter">"</span><span class="content">duration</span><span class="delimiter">"</span></span>: <span class="float">79318.903961598</span>,
<span class="key"><span class="delimiter">"</span><span class="content">type_data</span><span class="delimiter">"</span></span>: {
<span class="key"><span class="delimiter">"</span><span class="content">events</span><span class="delimiter">"</span></span>: [
{
<span class="key"><span class="delimiter">"</span><span class="content">time</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">2020-05-07 14:01:17.837714</span><span class="delimiter">"</span></span>,
<span class="key"><span class="delimiter">"</span><span class="content">event</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">initiated</span><span class="delimiter">"</span></span>
},
{
<span class="key"><span class="delimiter">"</span><span class="content">time</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">2020-05-07 14:01:17.837714</span><span class="delimiter">"</span></span>,
<span class="key"><span class="delimiter">"</span><span class="content">event</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">header_read</span><span class="delimiter">"</span></span>
},
{
<span class="key"><span class="delimiter">"</span><span class="content">time</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">2020-05-07 14:01:17.837716</span><span class="delimiter">"</span></span>,
<span class="key"><span class="delimiter">"</span><span class="content">event</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">throttled</span><span class="delimiter">"</span></span>
},
{
<span class="key"><span class="delimiter">"</span><span class="content">time</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">2020-05-07 14:01:17.837720</span><span class="delimiter">"</span></span>,
<span class="key"><span class="delimiter">"</span><span class="content">event</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">all_read</span><span class="delimiter">"</span></span>
},
{
<span class="key"><span class="delimiter">"</span><span class="content">time</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">2020-05-07 14:01:17.837738</span><span class="delimiter">"</span></span>,
<span class="key"><span class="delimiter">"</span><span class="content">event</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">dispatched</span><span class="delimiter">"</span></span>
},
{
<span class="key"><span class="delimiter">"</span><span class="content">time</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">2020-05-07 14:01:17.837740</span><span class="delimiter">"</span></span>,
<span class="key"><span class="delimiter">"</span><span class="content">event</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">mon:_ms_dispatch</span><span class="delimiter">"</span></span>
},
{
<span class="key"><span class="delimiter">"</span><span class="content">time</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">2020-05-07 14:01:17.837741</span><span class="delimiter">"</span></span>,
<span class="key"><span class="delimiter">"</span><span class="content">event</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">mon:dispatch_op</span><span class="delimiter">"</span></span>
},
{
<span class="key"><span class="delimiter">"</span><span class="content">time</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">2020-05-07 14:01:17.837741</span><span class="delimiter">"</span></span>,
<span class="key"><span class="delimiter">"</span><span class="content">event</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">psvc:dispatch</span><span class="delimiter">"</span></span>
},
{
<span class="key"><span class="delimiter">"</span><span class="content">time</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">2020-05-07 14:01:17.837742</span><span class="delimiter">"</span></span>,
<span class="key"><span class="delimiter">"</span><span class="content">event</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">osdmap:preprocess_query</span><span class="delimiter">"</span></span>
},
{
<span class="key"><span class="delimiter">"</span><span class="content">time</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">2020-05-07 14:01:17.837763</span><span class="delimiter">"</span></span>,
<span class="key"><span class="delimiter">"</span><span class="content">event</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">forward_request_leader</span><span class="delimiter">"</span></span>
},
{
<span class="key"><span class="delimiter">"</span><span class="content">time</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">2020-05-07 14:01:17.837778</span><span class="delimiter">"</span></span>,
<span class="key"><span class="delimiter">"</span><span class="content">event</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">forwarded</span><span class="delimiter">"</span></span>
}
],
<span class="key"><span class="delimiter">"</span><span class="content">info</span><span class="delimiter">"</span></span>: {
<span class="key"><span class="delimiter">"</span><span class="content">seq</span><span class="delimiter">"</span></span>: <span class="integer">882657</span>,
<span class="key"><span class="delimiter">"</span><span class="content">src_is_mon</span><span class="delimiter">"</span></span>: <span class="value">false</span>,
<span class="key"><span class="delimiter">"</span><span class="content">source</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">osd.378 v1:10.240.2.217:6861/288088</span><span class="delimiter">"</span></span>,
<span class="key"><span class="delimiter">"</span><span class="content">forwarded_to_leader</span><span class="delimiter">"</span></span>: <span class="value">true</span>
}
}
}
</span></code></pre>
<p>mon status output (for one that has slow ops):<br /><pre><code class="json syntaxhl"><span class="CodeRay">{
<span class="key"><span class="delimiter">"</span><span class="content">name</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">MSK-SR1-R2-CEPH-HOT1</span><span class="delimiter">"</span></span>,
<span class="key"><span class="delimiter">"</span><span class="content">rank</span><span class="delimiter">"</span></span>: <span class="integer">1</span>,
<span class="key"><span class="delimiter">"</span><span class="content">state</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">peon</span><span class="delimiter">"</span></span>,
<span class="key"><span class="delimiter">"</span><span class="content">election_epoch</span><span class="delimiter">"</span></span>: <span class="integer">62</span>,
<span class="key"><span class="delimiter">"</span><span class="content">quorum</span><span class="delimiter">"</span></span>: [
<span class="integer">0</span>,
<span class="integer">1</span>,
<span class="integer">2</span>
],
<span class="key"><span class="delimiter">"</span><span class="content">quorum_age</span><span class="delimiter">"</span></span>: <span class="integer">81311</span>,
<span class="key"><span class="delimiter">"</span><span class="content">features</span><span class="delimiter">"</span></span>: {
<span class="key"><span class="delimiter">"</span><span class="content">required_con</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">2449958747315912708</span><span class="delimiter">"</span></span>,
<span class="key"><span class="delimiter">"</span><span class="content">required_mon</span><span class="delimiter">"</span></span>: [
<span class="string"><span class="delimiter">"</span><span class="content">kraken</span><span class="delimiter">"</span></span>,
<span class="string"><span class="delimiter">"</span><span class="content">luminous</span><span class="delimiter">"</span></span>,
<span class="string"><span class="delimiter">"</span><span class="content">mimic</span><span class="delimiter">"</span></span>,
<span class="string"><span class="delimiter">"</span><span class="content">osdmap-prune</span><span class="delimiter">"</span></span>,
<span class="string"><span class="delimiter">"</span><span class="content">nautilus</span><span class="delimiter">"</span></span>
],
<span class="key"><span class="delimiter">"</span><span class="content">quorum_con</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">4611087854031667199</span><span class="delimiter">"</span></span>,
<span class="key"><span class="delimiter">"</span><span class="content">quorum_mon</span><span class="delimiter">"</span></span>: [
<span class="string"><span class="delimiter">"</span><span class="content">kraken</span><span class="delimiter">"</span></span>,
<span class="string"><span class="delimiter">"</span><span class="content">luminous</span><span class="delimiter">"</span></span>,
<span class="string"><span class="delimiter">"</span><span class="content">mimic</span><span class="delimiter">"</span></span>,
<span class="string"><span class="delimiter">"</span><span class="content">osdmap-prune</span><span class="delimiter">"</span></span>,
<span class="string"><span class="delimiter">"</span><span class="content">nautilus</span><span class="delimiter">"</span></span>
]
},
<span class="key"><span class="delimiter">"</span><span class="content">outside_quorum</span><span class="delimiter">"</span></span>: [],
<span class="key"><span class="delimiter">"</span><span class="content">extra_probe_peers</span><span class="delimiter">"</span></span>: [],
<span class="key"><span class="delimiter">"</span><span class="content">sync_provider</span><span class="delimiter">"</span></span>: [],
<span class="key"><span class="delimiter">"</span><span class="content">monmap</span><span class="delimiter">"</span></span>: {
<span class="key"><span class="delimiter">"</span><span class="content">epoch</span><span class="delimiter">"</span></span>: <span class="integer">2</span>,
<span class="key"><span class="delimiter">"</span><span class="content">fsid</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">ad99506a-05a5-11e8-975e-74d4351a7990</span><span class="delimiter">"</span></span>,
<span class="key"><span class="delimiter">"</span><span class="content">modified</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">2020-04-26 19:22:44.160085</span><span class="delimiter">"</span></span>,
<span class="key"><span class="delimiter">"</span><span class="content">created</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">2020-04-26 19:13:21.011855</span><span class="delimiter">"</span></span>,
<span class="key"><span class="delimiter">"</span><span class="content">min_mon_release</span><span class="delimiter">"</span></span>: <span class="integer">14</span>,
<span class="key"><span class="delimiter">"</span><span class="content">min_mon_release_name</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">nautilus</span><span class="delimiter">"</span></span>,
<span class="key"><span class="delimiter">"</span><span class="content">features</span><span class="delimiter">"</span></span>: {
<span class="key"><span class="delimiter">"</span><span class="content">persistent</span><span class="delimiter">"</span></span>: [
<span class="string"><span class="delimiter">"</span><span class="content">kraken</span><span class="delimiter">"</span></span>,
<span class="string"><span class="delimiter">"</span><span class="content">luminous</span><span class="delimiter">"</span></span>,
<span class="string"><span class="delimiter">"</span><span class="content">mimic</span><span class="delimiter">"</span></span>,
<span class="string"><span class="delimiter">"</span><span class="content">osdmap-prune</span><span class="delimiter">"</span></span>,
<span class="string"><span class="delimiter">"</span><span class="content">nautilus</span><span class="delimiter">"</span></span>
],
<span class="key"><span class="delimiter">"</span><span class="content">optional</span><span class="delimiter">"</span></span>: []
},
<span class="key"><span class="delimiter">"</span><span class="content">mons</span><span class="delimiter">"</span></span>: [
{
<span class="key"><span class="delimiter">"</span><span class="content">rank</span><span class="delimiter">"</span></span>: <span class="integer">0</span>,
<span class="key"><span class="delimiter">"</span><span class="content">name</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">MSK-SR1-R1-CEPH-HOT1</span><span class="delimiter">"</span></span>,
<span class="key"><span class="delimiter">"</span><span class="content">public_addrs</span><span class="delimiter">"</span></span>: {
<span class="key"><span class="delimiter">"</span><span class="content">addrvec</span><span class="delimiter">"</span></span>: [
{
<span class="key"><span class="delimiter">"</span><span class="content">type</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">v1</span><span class="delimiter">"</span></span>,
<span class="key"><span class="delimiter">"</span><span class="content">addr</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">10.240.2.201:6789</span><span class="delimiter">"</span></span>,
<span class="key"><span class="delimiter">"</span><span class="content">nonce</span><span class="delimiter">"</span></span>: <span class="integer">0</span>
},
{
<span class="key"><span class="delimiter">"</span><span class="content">type</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">v2</span><span class="delimiter">"</span></span>,
<span class="key"><span class="delimiter">"</span><span class="content">addr</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">10.240.2.201:3300</span><span class="delimiter">"</span></span>,
<span class="key"><span class="delimiter">"</span><span class="content">nonce</span><span class="delimiter">"</span></span>: <span class="integer">0</span>
}
]
},
<span class="key"><span class="delimiter">"</span><span class="content">addr</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">10.240.2.201:6789/0</span><span class="delimiter">"</span></span>,
<span class="key"><span class="delimiter">"</span><span class="content">public_addr</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">10.240.2.201:6789/0</span><span class="delimiter">"</span></span>
},
{
<span class="key"><span class="delimiter">"</span><span class="content">rank</span><span class="delimiter">"</span></span>: <span class="integer">1</span>,
<span class="key"><span class="delimiter">"</span><span class="content">name</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">MSK-SR1-R2-CEPH-HOT1</span><span class="delimiter">"</span></span>,
<span class="key"><span class="delimiter">"</span><span class="content">public_addrs</span><span class="delimiter">"</span></span>: {
<span class="key"><span class="delimiter">"</span><span class="content">addrvec</span><span class="delimiter">"</span></span>: [
{
<span class="key"><span class="delimiter">"</span><span class="content">type</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">v1</span><span class="delimiter">"</span></span>,
<span class="key"><span class="delimiter">"</span><span class="content">addr</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">10.240.2.202:6789</span><span class="delimiter">"</span></span>,
<span class="key"><span class="delimiter">"</span><span class="content">nonce</span><span class="delimiter">"</span></span>: <span class="integer">0</span>
},
{
<span class="key"><span class="delimiter">"</span><span class="content">type</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">v2</span><span class="delimiter">"</span></span>,
<span class="key"><span class="delimiter">"</span><span class="content">addr</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">10.240.2.202:3300</span><span class="delimiter">"</span></span>,
<span class="key"><span class="delimiter">"</span><span class="content">nonce</span><span class="delimiter">"</span></span>: <span class="integer">0</span>
}
]
},
<span class="key"><span class="delimiter">"</span><span class="content">addr</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">10.240.2.202:6789/0</span><span class="delimiter">"</span></span>,
<span class="key"><span class="delimiter">"</span><span class="content">public_addr</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">10.240.2.202:6789/0</span><span class="delimiter">"</span></span>
},
{
<span class="key"><span class="delimiter">"</span><span class="content">rank</span><span class="delimiter">"</span></span>: <span class="integer">2</span>,
<span class="key"><span class="delimiter">"</span><span class="content">name</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">MSK-SR1-R3-CEPH-HOT1</span><span class="delimiter">"</span></span>,
<span class="key"><span class="delimiter">"</span><span class="content">public_addrs</span><span class="delimiter">"</span></span>: {
<span class="key"><span class="delimiter">"</span><span class="content">addrvec</span><span class="delimiter">"</span></span>: [
{
<span class="key"><span class="delimiter">"</span><span class="content">type</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">v2</span><span class="delimiter">"</span></span>,
<span class="key"><span class="delimiter">"</span><span class="content">addr</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">10.240.2.203:3300</span><span class="delimiter">"</span></span>,
<span class="key"><span class="delimiter">"</span><span class="content">nonce</span><span class="delimiter">"</span></span>: <span class="integer">0</span>
},
{
<span class="key"><span class="delimiter">"</span><span class="content">type</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">v1</span><span class="delimiter">"</span></span>,
<span class="key"><span class="delimiter">"</span><span class="content">addr</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">10.240.2.203:6789</span><span class="delimiter">"</span></span>,
<span class="key"><span class="delimiter">"</span><span class="content">nonce</span><span class="delimiter">"</span></span>: <span class="integer">0</span>
}
]
},
<span class="key"><span class="delimiter">"</span><span class="content">addr</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">10.240.2.203:6789/0</span><span class="delimiter">"</span></span>,
<span class="key"><span class="delimiter">"</span><span class="content">public_addr</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">10.240.2.203:6789/0</span><span class="delimiter">"</span></span>
}
]
},
<span class="key"><span class="delimiter">"</span><span class="content">feature_map</span><span class="delimiter">"</span></span>: {
<span class="key"><span class="delimiter">"</span><span class="content">mon</span><span class="delimiter">"</span></span>: [
{
<span class="key"><span class="delimiter">"</span><span class="content">features</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">0x3ffddff8ffacffff</span><span class="delimiter">"</span></span>,
<span class="key"><span class="delimiter">"</span><span class="content">release</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">luminous</span><span class="delimiter">"</span></span>,
<span class="key"><span class="delimiter">"</span><span class="content">num</span><span class="delimiter">"</span></span>: <span class="integer">1</span>
}
],
<span class="key"><span class="delimiter">"</span><span class="content">osd</span><span class="delimiter">"</span></span>: [
{
<span class="key"><span class="delimiter">"</span><span class="content">features</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">0x3ffddff8ffacffff</span><span class="delimiter">"</span></span>,
<span class="key"><span class="delimiter">"</span><span class="content">release</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">luminous</span><span class="delimiter">"</span></span>,
<span class="key"><span class="delimiter">"</span><span class="content">num</span><span class="delimiter">"</span></span>: <span class="integer">200</span>
}
],
<span class="key"><span class="delimiter">"</span><span class="content">client</span><span class="delimiter">"</span></span>: [
{
<span class="key"><span class="delimiter">"</span><span class="content">features</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">0x27018fb86aa42ada</span><span class="delimiter">"</span></span>,
<span class="key"><span class="delimiter">"</span><span class="content">release</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">jewel</span><span class="delimiter">"</span></span>,
<span class="key"><span class="delimiter">"</span><span class="content">num</span><span class="delimiter">"</span></span>: <span class="integer">10</span>
},
{
<span class="key"><span class="delimiter">"</span><span class="content">features</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">0x3ffddff8ffacffff</span><span class="delimiter">"</span></span>,
<span class="key"><span class="delimiter">"</span><span class="content">release</span><span class="delimiter">"</span></span>: <span class="string"><span class="delimiter">"</span><span class="content">luminous</span><span class="delimiter">"</span></span>,
<span class="key"><span class="delimiter">"</span><span class="content">num</span><span class="delimiter">"</span></span>: <span class="integer">786</span>
}
]
}
}
</span></code></pre></p>
RADOS - Bug #24531: Mimic MONs have slow/long running ops
https://tracker.ceph.com/issues/24531?journal_id=166534
2020-05-23T15:11:56Z
Daniel Poelzleithner
poelzleithner@b1-systems.de
<ul></ul><p>We had this issue yesterday. We had a broken mon cluster which I was able to repair by shutting down all mons, scaling monmap to 1 and starting one mon server. After the osds went back online, I zapped all mons and let them rejoin the cluster. I had to restart all mds, as they had strange hickups and cause also SLOW requests.</p>
RADOS - Bug #24531: Mimic MONs have slow/long running ops
https://tracker.ceph.com/issues/24531?journal_id=174753
2020-09-10T04:43:55Z
Kefu Chai
tchaikov@gmail.com
<ul></ul><p>please note, this fix was backported as f0697a9af54bf866572036bd6d582abd5299d0c8<br /><pre>
git tag --contains f0697a9af54bf866572036bd6d582abd5299d0c8 Thu Sep 10 12:41:54 2020
v14.2.10
v14.2.5
v14.2.6
v14.2.7
v14.2.8
v14.2.9
</pre></p>
<p>and is included in nautilus 14.2.5 and up. if you are still using < 14.2.5. might want to consider upgrade at least your monitor to 14.2.5.</p>