Project

General

Profile

Bug #50242

test_repair_corrupted_obj fails with assert not inconsistent

Added by Neha Ojha almost 3 years ago. Updated 8 months ago.

Status:
New
Priority:
Normal
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
pacific,octopus,quincy
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

2021-04-05T17:04:53.832 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/run_tasks.py", line 91, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/run_tasks.py", line 70, in run_one_task
    return task(**kwargs)
  File "/home/teuthworker/src/github.com_ideepika_ceph_fc430897399d00a6bce6b5ce19341729eefd3188/qa/tasks/scrub_test.py", line 390, in task
    test_repair_corrupted_obj(ctx, manager, pg, osd_remote, obj_path, 'rbd')
  File "/home/teuthworker/src/github.com_ideepika_ceph_fc430897399d00a6bce6b5ce19341729eefd3188/qa/tasks/scrub_test.py", line 94, in test_repair_corrupted_obj
    repair(manager, pg, pool)
  File "/home/teuthworker/src/github.com_ideepika_ceph_fc430897399d00a6bce6b5ce19341729eefd3188/qa/tasks/scrub_test.py", line 88, in repair
    assert not inconsistent
AssertionError
2021-04-05T17:04:54.213 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=910a44f567234774a573c3027257e874
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/run_tasks.py", line 91, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/run_tasks.py", line 70, in run_one_task
    return task(**kwargs)
  File "/home/teuthworker/src/github.com_ideepika_ceph_fc430897399d00a6bce6b5ce19341729eefd3188/qa/tasks/scrub_test.py", line 390, in task
    test_repair_corrupted_obj(ctx, manager, pg, osd_remote, obj_path, 'rbd')
  File "/home/teuthworker/src/github.com_ideepika_ceph_fc430897399d00a6bce6b5ce19341729eefd3188/qa/tasks/scrub_test.py", line 94, in test_repair_corrupted_obj
    repair(manager, pg, pool)
  File "/home/teuthworker/src/github.com_ideepika_ceph_fc430897399d00a6bce6b5ce19341729eefd3188/qa/tasks/scrub_test.py", line 88, in repair
    assert not inconsistent
AssertionError

/a/ideepika-2021-04-05_14:32:13-rados-wip-deepika-testing-2021-04-05-0643-distro-basic-smithi/6022340
/a/sage-2021-04-06_20:05:38-rados-wip-sage-testing-2021-04-06-1403-distro-basic-smithi/6025317

https://sentry.ceph.com/organizations/ceph/issues/6725/events/910a44f567234774a573c3027257e874/events/?project=2

History

#1 Updated by Ronen Friedman almost 3 years ago

Some observations from /a/sage-2021-04-06_20:05:38-rados-wip-sage-testing-2021-04-06-1403-distro-basic-smithi/6025317:

osd.6 finished scrubbing 2.7 on 13:30:09.363; The PG status changed (per the log of that OSD) to 'clean'.
Still the test code (qa/tasks/scrub_test.py:repair()), executing manager.get_single_pg_stats() some 7 seconds later, and still saw 'inconsistent'. The OSD seems to have published a correct update (sequence 834), but the test still saw seq=833.

2021-04-07T13:30:09.363+0000 7ff637188700 15 osd.6 pg_epoch: 17 pg[2.7( v 17'800 (0'0,17'800] local-lis/les=15/16 n=399 ec=15/15 lis/c=15/15 les/c/f=16/17/0 sis=15) [6,7] r=0 lpr=15 crt=17'800 mlcod 17'799 active+clean+scrubbing+deep [ 2.7: REQ_SCRUB ] ] publish_stats_to_osd 17:834

#2 Updated by Neha Ojha almost 3 years ago

  • Backport set to pacific

/a/teuthology-2021-04-11_03:31:02-rados-pacific-distro-basic-smithi/6034330

#3 Updated by Neha Ojha about 2 years ago

  • Status changed from New to Can't reproduce

#4 Updated by Kamoltat (Junior) Sirivadhna over 1 year ago

/a/yuriw-2022-06-29_13:30:16-rados-wip-yuri3-testing-2022-06-28-1737-distro-default-smithi/6905523/

#5 Updated by Kamoltat (Junior) Sirivadhna over 1 year ago

  • Status changed from Can't reproduce to New

#6 Updated by Radoslaw Zarzynski over 1 year ago

  • Assignee set to Ronen Friedman
  • Priority changed from High to Normal
  • Backport changed from pacific to pacific,octopus,quincy

Hello Ronen. It looks to be somehow scrub-related. Mind taking a look? Nothing urgent.

#7 Updated by Prashant D 8 months ago

/a/yuriw-2023-07-11_14:28:47-rados-wip-yuri10-testing-2023-07-10-1345-distro-default-smithi/7333876/

Also available in: Atom PDF