Bug #24240: qa: 1 mutations had unexpected outcomes - CephFS - Ceph

Actions

Copy link

Bug #24240

closed

qa: 1 mutations had unexpected outcomes

Added by Patrick Donnelly almost 6 years ago. Updated almost 6 years ago.

Status:

Resolved

Priority:

Urgent

Assignee:

Zheng Yan

Category:

Target version:

Ceph - v14.0.0

% Done:

Source:

Q/A

Tags:

Backport:

mimic

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Component(FS):

qa-suite

Labels (FS):

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

2018-05-20T03:00:48.437 INFO:tasks.cephfs.test_damage:Result: Mutation 'Corrupt omap header on 1.00000000' led to DAMAGED state
2018-05-20T03:00:48.437 ERROR:tasks.cephfs.test_damage:1 mutations had unexpected outcomes:
2018-05-20T03:00:48.437 ERROR:tasks.cephfs.test_damage:  Expected 'no damage' actually 'server failed' from 'Corrupt omap header on mds0_openfiles.0'
2018-05-20T03:00:48.439 INFO:tasks.cephfs_test_runner:test_object_deletion (tasks.cephfs.test_damage.TestDamage) ... ERROR
...
2018-05-20T03:00:50.199 INFO:tasks.cephfs_test_runner:======================================================================
2018-05-20T03:00:50.199 INFO:tasks.cephfs_test_runner:ERROR: test_object_deletion (tasks.cephfs.test_damage.TestDamage)
2018-05-20T03:00:50.199 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2018-05-20T03:00:50.200 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2018-05-20T03:00:50.200 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph_mimic/qa/tasks/cephfs/test_damage.py", line 352, in test_object_deletion
2018-05-20T03:00:50.200 INFO:tasks.cephfs_test_runner:    raise RuntimeError("{0} mutations had unexpected outcomes".format(len(failures)))
2018-05-20T03:00:50.200 INFO:tasks.cephfs_test_runner:RuntimeError: 1 mutations had unexpected outcomes

From: /ceph/teuthology-archive/teuthology-2018-05-18_05:20:02-kcephfs-mimic-testing-basic-ovh/2546622/teuthology.log

Related issues 1 (0 open — 1 closed)

Actions

Copy link

Updated by Zheng Yan almost 6 years ago

The test case corrupted open file table’s omap header. One field in omap header is ‘num_objects’. The corrupted header’s num_objects had very large value. which caused mds to resize a vector to very large size. It seems the big memory allocation was very slow.

adding magic number to omap header should fix this issue.

Actions

Copy link