Project

General

Profile

Actions

Bug #53342

closed

Exiting scrub checking -- not all pgs scrubbed

Added by Neha Ojha over 2 years ago. Updated about 1 month ago.

Status:
Resolved
Priority:
Normal
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

2021-11-18T05:17:58.768 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed.
2021-11-18T05:18:10.289 INFO:tasks.ceph.osd.1.smithi122.stderr:2021-11-18T05:18:10.289+0000 7f274865d700 -1 osd.1 pg_epoch: 246 pg[1.0( empty local-lis/les=130/131 n=0 ec=14/14 lis/c=130/130 les/c/f=131/131/0 sis=130) [1,5,2] r=0 lpr=130 crt=0'0 mlcod 0'0 active+clean MUST_DEEP_SCRUB MUST_SCRUB planned REQ_SCRUB]  scrubber pg(1.0) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 5 (0x5634bef4f340)
2021-11-18T05:18:17.315 INFO:tasks.ceph.osd.1.smithi122.stderr:2021-11-18T05:18:17.316+0000 7f274865d700 -1 osd.1 pg_epoch: 246 pg[1.0( empty local-lis/les=130/131 n=0 ec=14/14 lis/c=130/130 les/c/f=131/131/0 sis=130) [1,5,2] r=0 lpr=130 crt=0'0 mlcod 0'0 active+clean MUST_DEEP_SCRUB MUST_SCRUB planned REQ_SCRUB]  scrubber pg(1.0) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 5 (0x5634bd72d600)
...
2021-11-18T05:18:19.075 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6cfddd5e1c7d5ede1367cf3087c21af44eb88356/teuthology/contextutil.py", line 33, in nested
    yield vars
  File "/home/teuthworker/src/github.com_ceph_ceph-c_c46b441d7343ee6b8ac90208a03c0055394b8e80/qa/tasks/ceph.py", line 1909, in task
    osd_scrub_pgs(ctx, config)
  File "/home/teuthworker/src/github.com_ceph_ceph-c_c46b441d7343ee6b8ac90208a03c0055394b8e80/qa/tasks/ceph.py", line 1303, in osd_scrub_pgs

/a/yuriw-2021-11-18_00:45:42-rados-wip-yuri11-testing-2021-11-17-1514-pacific-distro-basic-smithi/6511073

Actions #1

Updated by Laura Flores about 2 years ago

  • Related to Bug #55009: Scrubbing exits due to error reading object head added
Actions #2

Updated by Radoslaw Zarzynski about 2 years ago

  • Related to deleted (Bug #55009: Scrubbing exits due to error reading object head)
Actions #3

Updated by Sridhar Seshasayee almost 2 years ago

/a/yuriw-2022-06-29_18:22:37-rados-wip-yuri2-testing-2022-06-29-0820-distro-default-smithi/6906076
/a/yuriw-2022-06-29_18:22:37-rados-wip-yuri2-testing-2022-06-29-0820-distro-default-smithi/6906236

Actions #4

Updated by Neha Ojha almost 2 years ago

Sridhar Seshasayee wrote:

/a/yuriw-2022-06-29_18:22:37-rados-wip-yuri2-testing-2022-06-29-0820-distro-default-smithi/6906076
/a/yuriw-2022-06-29_18:22:37-rados-wip-yuri2-testing-2022-06-29-0820-distro-default-smithi/6906236

These runs were failing due to other reasons before we saw scrub terminate.

2022-06-30T05:22:12.933 DEBUG:teuthology.orchestra.run:got remote process result: 124
2022-06-30T05:22:12.934 INFO:tasks.workunit.client.0.smithi005.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlus
2022-06-30T05:22:12.935 INFO:tasks.workunit:Stopping ['rados/test.sh', 'rados/test_pool_quota.sh'] on client.0...
Actions #5

Updated by Kamoltat (Junior) Sirivadhna 3 months ago

/a/yuriw-2024-01-31_19:20:14-rados-wip-yuri3-testing-2024-01-29-1434-distro-default-smithi/7540741/

Actions #6

Updated by Ronen Friedman 2 months ago

  • Status changed from New to In Progress
  • Backport deleted (pacific)
  • Pull request ID set to 55478
  • Affected Versions added

Note: PR#55478 (not part of the code yet) fixes the specific failure reported in note#5.
It is not related to previous occurrences of "not scrubbed in time"

Actions #7

Updated by Radoslaw Zarzynski 2 months ago

  • Assignee set to Ronen Friedman
Actions #8

Updated by Ronen Friedman 2 months ago

I think we can close this bug as 'resolved'

Actions #9

Updated by Radoslaw Zarzynski about 2 months ago

  • Status changed from In Progress to Resolved

Ronen, do we need any backporting?

Actions #10

Updated by Ronen Friedman about 2 months ago

Radoslaw Zarzynski wrote:

Ronen, do we need any backporting?

No. The fix (55478) made it in time for Squid.

Actions #11

Updated by Ilya Dryomov about 1 month ago

  • Affected Versions v18.2.3 added
  • Affected Versions deleted ()
Actions

Also available in: Atom PDF