Project

General

Profile

Actions
Copy link

Bug #46178

closed

slow request osd_op(... (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49)

Added by Sebastian Wagner almost 4 years ago. Updated almost 4 years ago.

Status:
Duplicate
Priority:
Urgent
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Saw this error yesterday for the first time:

http://pulpito.ceph.com/swagner-2020-06-23_13:15:09-rados:cephadm-wip-swagner3-testing-2020-06-23-1058-distro-basic-smithi/5172444

2020-06-23T14:14:24.479 INFO:tasks.cephadm:Deploying osd.1 on smithi140 with /dev/vg_nvme/lv_3...
...
2020-06-24T01:44:38.508 INFO:ceph.mon.b.smithi180.stdout:Jun 24 01:44:38 smithi180 bash[11465]: cluster 2020-06-24T01:44:37.532712+0000 osd.1 (osd.1) 951804 : cluster [WRN] slow request osd_op(client.34367.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-23T17:02:28.014118+0000 currently delayed
2020-06-24T01:44:38.508 INFO:ceph.mon.b.smithi180.stdout:Jun 24 01:44:38 smithi180 bash[11465]: cluster 2020-06-24T01:44:37.532721+0000 osd.1 (osd.1) 951805 : cluster [WRN] slow request osd_op(client.34367.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-24T01:02:28.112645+0000 currently delayed
2020-06-24T01:44:38.508 INFO:ceph.mon.b.smithi180.stdout:Jun 24 01:44:38 smithi180 bash[11465]: cluster 2020-06-24T01:44:37.532732+0000 osd.1 (osd.1) 951806 : cluster [WRN] slow request osd_op(client.34367.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-23T17:17:28.017258+0000 currently delayed
2020-06-24T01:44:38.508 INFO:ceph.mon.b.smithi180.stdout:Jun 24 01:44:38 smithi180 bash[11465]: cluster 2020-06-24T01:44:37.532741+0000 osd.1 (osd.1) 951807 : cluster [WRN] slow request osd_op(client.34367.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-24T01:17:28.116826+0000 currently delayed
2020-06-24T01:44:38.509 INFO:ceph.mon.b.smithi180.stdout:Jun 24 01:44:38 smithi180 bash[11465]: cluster 2020-06-24T01:44:37.532749+0000 osd.1 (osd.1) 951808 : cluster [WRN] slow request osd_op(client.34367.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-23T17:32:28.021231+0000 currently delayed
2020-06-24T01:44:38.509 INFO:ceph.mon.b.smithi180.stdout:Jun 24 01:44:38 smithi180 bash[11465]: cluster 2020-06-24T01:44:37.532758+0000 osd.1 (osd.1) 951809 : cluster [WRN] slow request osd_op(client.34367.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-24T01:32:28.117176+0000 currently delayed
2020-06-24T01:44:38.509 INFO:ceph.mon.b.smithi180.stdout:Jun 24 01:44:38 smithi180 bash[11465]: cluster 2020-06-24T01:44:37.532770+0000 osd.1 (osd.1) 951810 : cluster [WRN] slow request osd_op(client.34367.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-23T17:47:28.021867+0000 currently delayed
2020-06-24T01:44:38.509 INFO:ceph.mon.b.smithi180.stdout:Jun 24 01:44:38 smithi180 bash[11465]: cluster 2020-06-24T01:44:37.532795+0000 osd.1 (osd.1) 951811 : cluster [WRN] slow request osd_op(client.34367.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-23T18:02:28.024273+0000 currently delayed
2020-06-24T01:44:38.779 INFO:ceph.osd.1.smithi140.stdout:Jun 24 01:44:38 smithi140 bash[20025]: debug 2020-06-24T01:44:38.512+0000 7f660a6f2700 -1 osd.1 49 get_health_metrics reporting 46 slow ops, oldest is osd_op(client.34367.0:13 2.a 2.a (undecoded) ondisk+read+ignore_overlay+known_if_redirected e49)
2020-06-24T01:44:39.499 INFO:ceph.mon.b.smithi180.stdout:Jun 24 01:44:39 smithi180 bash[11465]: cluster 2020-06-24T01:44:38.476728+0000 mgr.x (mgr.34109) 20737 : cluster [DBG] pgmap v20741: 33 pgs: 3 creating+peering, 30 active+clean; 780 B data, 3.4 MiB used, 707 GiB / 715 GiB avail
2020-06-24T01:44:39.500 INFO:ceph.mon.b.smithi180.stdout:Jun 24 01:44:39 smithi180 bash[11465]: cluster 2020-06-24T01:44:38.515272+0000 osd.1 (osd.1) 951812 : cluster [WRN] slow request osd_op(client.34367.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-23T18:17:28.028627+0000 currently delayed
2020-06-24T01:44:39.500 INFO:ceph.mon.b.smithi180.stdout:Jun 24 01:44:39 smithi180 bash[11465]: cluster 2020-06-24T01:44:38.515294+0000 osd.1 (osd.1) 951813 : cluster [WRN] slow request osd_op(client.34367.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-23T18:32:28.033173+0000 currently delayed
2020-06-24T01:44:39.500 INFO:ceph.mon.b.smithi180.stdout:Jun 24 01:44:39 smithi180 bash[11465]: cluster 2020-06-24T01:44:38.515312+0000 osd.1 (osd.1) 951814 : cluster [WRN] slow request osd_op(client.34367.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-23T18:47:28.037863+0000 currently delayed
2020-06-24T01:44:39.501 INFO:ceph.mon.b.smithi180.stdout:Jun 24 01:44:39 smithi180 bash[11465]: cluster 2020-06-24T01:44:38.515329+0000 osd.1 (osd.1) 951815 : cluster [WRN] slow request osd_op(client.34367.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-23T19:02:28.037117+0000 currently delayed
2020-06-24T01:44:39.501 INFO:ceph.mon.b.smithi180.stdout:Jun 24 01:44:39 smithi180 bash[11465]: cluster 2020-06-24T01:44:38.515344+0000 osd.1 (osd.1) 951816 : cluster [WRN] slow request osd_op(client.34367.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-23T19:17:28.041383+0000 currently delayed
2020-06-24T01:44:39.501 INFO:ceph.mon.b.smithi180.stdout:Jun 24 01:44:39 smithi180 bash[11465]: cluster 2020-06-24T01:44:38.515363+0000 osd.1 (osd.1) 951817 : cluster [WRN] slow request osd_op(client.34367.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-23T19:32:28.045582+0000 currently delayed
2020-06-24T01:44:39.501 INFO:ceph.mon.b.smithi180.stdout:Jun 24 01:44:39 smithi180 bash[11465]: cluster 2020-06-24T01:44:38.515379+0000 osd.1 (osd.1) 951818 : cluster [WRN] slow request osd_op(client.34367.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overla

now it happened again:

http://pulpito.ceph.com/swagner-2020-06-24_11:29:20-rados:cephadm-wip-swagner-testing-2020-06-24-1032-distro-basic-smithi/5175427/

2020-06-24T11:56:54.575 INFO:tasks.cephadm:Deploying osd.1 on smithi118 with /dev/vg_nvme/lv_3...
...
7f44d2a96700 -1 osd.1 49 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.34343.0:13 2.a 2.a (undecoded) ondisk+read+ignore_overlay+known_if_redirected e49)
0000 mgr.x (mgr.34103) 1527 : cluster [DBG] pgmap v1531: 33 pgs: 3 creating+peering, 30 active+clean; 780 B data, 3.9 MiB used, 707 GiB / 715 GiB avail
0000 osd.1 (osd.1) 5934 : cluster [WRN] slow request osd_op(client.34343.0:13 2.a 2.a (undecoded) ondisk+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-24T11:58:48.006893+0000 currently delayed
0000 osd.1 (osd.1) 5935 : cluster [WRN] slow request osd_op(client.34343.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-24T12:13:48.005126+0000 currently delayed
0000 osd.1 (osd.1) 5936 : cluster [WRN] slow request osd_op(client.34343.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-24T12:28:48.005918+0000 currently delayed
0000 osd.1 (osd.1) 5937 : cluster [WRN] slow request osd_op(client.34343.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-24T12:43:48.009047+0000 currently delayed
0000 mgr.x (mgr.34103) 1527 : cluster [DBG] pgmap v1531: 33 pgs: 3 creating+peering, 30 active+clean; 780 B data, 3.9 MiB used, 707 GiB / 715 GiB avail
0000 osd.1 (osd.1) 5934 : cluster [WRN] slow request osd_op(client.34343.0:13 2.a 2.a (undecoded) ondisk+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-24T11:58:48.006893+0000 currently delayed
0000 osd.1 (osd.1) 5935 : cluster [WRN] slow request osd_op(client.34343.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-24T12:13:48.005126+0000 currently delayed
0000 osd.1 (osd.1) 5936 : cluster [WRN] slow request osd_op(client.34343.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-24T12:28:48.005918+0000 currently delayed
0000 osd.1 (osd.1) 5937 : cluster [WRN] slow request osd_op(client.34343.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-24T12:43:48.009047+0000 currently delayed
0000 mgr.x (mgr.34103) 1527 : cluster [DBG] pgmap v1531: 33 pgs: 3 creating+peering, 30 active+clean; 780 B data, 3.9 MiB used, 707 GiB / 715 GiB avail
0000 osd.1 (osd.1) 5934 : cluster [WRN] slow request osd_op(client.34343.0:13 2.a 2.a (undecoded) ondisk+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-24T11:58:48.006893+0000 currently delayed
0000 osd.1 (osd.1) 5935 : cluster [WRN] slow request osd_op(client.34343.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-24T12:13:48.005126+0000 currently delayed
0000 osd.1 (osd.1) 5936 : cluster [WRN] slow request osd_op(client.34343.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-24T12:28:48.005918+0000 currently delayed
0000 osd.1 (osd.1) 5937 : cluster [WRN] slow request osd_op(client.34343.0:13 2.a 2.a (undecoded) ondisk+retry+read+ignore_overlay+known_if_redirected e49) initiated 2020-06-24T12:43:48.009047+0000 currently delayed

Unfortunately, I don't know where this comes from.

Related issues 1 (0 open — 1 closed)

Is duplicate of RADOS - Bug #46180: qa: Scrubbing terminated -- not all pgs were active and clean.ResolvedIlya Dryomov

Actions
Actions
Copy link

Also available in: Atom PDF