Bug #50242: test_repair_corrupted_obj fails with assert not inconsistent - RADOS - Ceph

Actions

Copy link

Bug #50242

open

test_repair_corrupted_obj fails with assert not inconsistent

Added by Neha Ojha about 3 years ago. Updated 10 months ago.

Status:

New

Priority:

Normal

Assignee:

Ronen Friedman

Category:

Target version:

% Done:

Source:

Tags:

Backport:

pacific,octopus,quincy

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Component(RADOS):

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

2021-04-05T17:04:53.832 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/run_tasks.py", line 91, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/run_tasks.py", line 70, in run_one_task
    return task(**kwargs)
  File "/home/teuthworker/src/github.com_ideepika_ceph_fc430897399d00a6bce6b5ce19341729eefd3188/qa/tasks/scrub_test.py", line 390, in task
    test_repair_corrupted_obj(ctx, manager, pg, osd_remote, obj_path, 'rbd')
  File "/home/teuthworker/src/github.com_ideepika_ceph_fc430897399d00a6bce6b5ce19341729eefd3188/qa/tasks/scrub_test.py", line 94, in test_repair_corrupted_obj
    repair(manager, pg, pool)
  File "/home/teuthworker/src/github.com_ideepika_ceph_fc430897399d00a6bce6b5ce19341729eefd3188/qa/tasks/scrub_test.py", line 88, in repair
    assert not inconsistent
AssertionError
2021-04-05T17:04:54.213 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=910a44f567234774a573c3027257e874
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/run_tasks.py", line 91, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/run_tasks.py", line 70, in run_one_task
    return task(**kwargs)
  File "/home/teuthworker/src/github.com_ideepika_ceph_fc430897399d00a6bce6b5ce19341729eefd3188/qa/tasks/scrub_test.py", line 390, in task
    test_repair_corrupted_obj(ctx, manager, pg, osd_remote, obj_path, 'rbd')
  File "/home/teuthworker/src/github.com_ideepika_ceph_fc430897399d00a6bce6b5ce19341729eefd3188/qa/tasks/scrub_test.py", line 94, in test_repair_corrupted_obj
    repair(manager, pg, pool)
  File "/home/teuthworker/src/github.com_ideepika_ceph_fc430897399d00a6bce6b5ce19341729eefd3188/qa/tasks/scrub_test.py", line 88, in repair
    assert not inconsistent
AssertionError

/a/ideepika-2021-04-05_14:32:13-rados-wip-deepika-testing-2021-04-05-0643-distro-basic-smithi/6022340
/a/sage-2021-04-06_20:05:38-rados-wip-sage-testing-2021-04-06-1403-distro-basic-smithi/6025317

https://sentry.ceph.com/organizations/ceph/issues/6725/events/910a44f567234774a573c3027257e874/events/?project=2

Actions

Copy link

Updated by Ronen Friedman about 3 years ago

Some observations from /a/sage-2021-04-06_20:05:38-rados-wip-sage-testing-2021-04-06-1403-distro-basic-smithi/6025317:

osd.6 finished scrubbing 2.7 on 13:30:09.363; The PG status changed (per the log of that OSD) to 'clean'.
Still the test code (qa/tasks/scrub_test.py:repair()), executing manager.get_single_pg_stats() some 7 seconds later, and still saw 'inconsistent'. The OSD seems to have published a correct update (sequence 834), but the test still saw seq=833.

2021-04-07T13:30:09.363+0000 7ff637188700 15 osd.6 pg_epoch: 17 pg[2.7( v 17'800 (0'0,17'800] local-lis/les=15/16 n=399 ec=15/15 lis/c=15/15 les/c/f=16/17/0 sis=15) [6,7] r=0 lpr=15 crt=17'800 mlcod 17'799 active+clean+scrubbing+deep [ 2.7: REQ_SCRUB ] ] publish_stats_to_osd 17:834

Actions

Copy link