Project

General

Profile

Bug #24240

qa: 1 mutations had unexpected outcomes

Added by Patrick Donnelly 7 months ago. Updated 5 months ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Category:
-
Target version:
Start date:
05/23/2018
Due date:
% Done:

0%

Source:
Q/A
Tags:
Backport:
mimic
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
qa-suite
Labels (FS):
qa
Pull request ID:

Description

2018-05-20T03:00:48.437 INFO:tasks.cephfs.test_damage:Result: Mutation 'Corrupt omap header on 1.00000000' led to DAMAGED state
2018-05-20T03:00:48.437 ERROR:tasks.cephfs.test_damage:1 mutations had unexpected outcomes:
2018-05-20T03:00:48.437 ERROR:tasks.cephfs.test_damage:  Expected 'no damage' actually 'server failed' from 'Corrupt omap header on mds0_openfiles.0'
2018-05-20T03:00:48.439 INFO:tasks.cephfs_test_runner:test_object_deletion (tasks.cephfs.test_damage.TestDamage) ... ERROR
...
2018-05-20T03:00:50.199 INFO:tasks.cephfs_test_runner:======================================================================
2018-05-20T03:00:50.199 INFO:tasks.cephfs_test_runner:ERROR: test_object_deletion (tasks.cephfs.test_damage.TestDamage)
2018-05-20T03:00:50.199 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2018-05-20T03:00:50.200 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2018-05-20T03:00:50.200 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph_mimic/qa/tasks/cephfs/test_damage.py", line 352, in test_object_deletion
2018-05-20T03:00:50.200 INFO:tasks.cephfs_test_runner:    raise RuntimeError("{0} mutations had unexpected outcomes".format(len(failures)))
2018-05-20T03:00:50.200 INFO:tasks.cephfs_test_runner:RuntimeError: 1 mutations had unexpected outcomes

From: /ceph/teuthology-archive/teuthology-2018-05-18_05:20:02-kcephfs-mimic-testing-basic-ovh/2546622/teuthology.log


Related issues

Copied to fs - Backport #24541: mimic: qa: 1 mutations had unexpected outcomes Resolved

History

#1 Updated by Zheng Yan 7 months ago

The test case corrupted open file table’s omap header. One field in omap header is ‘num_objects’. The corrupted header’s num_objects had very large value. which caused mds to resize a vector to very large size. It seems the big memory allocation was very slow.

adding magic number to omap header should fix this issue.

#2 Updated by Zheng Yan 7 months ago

  • Status changed from New to Need Review

#3 Updated by Patrick Donnelly 6 months ago

  • Status changed from Need Review to Pending Backport

#4 Updated by Nathan Cutler 6 months ago

  • Copied to Backport #24541: mimic: qa: 1 mutations had unexpected outcomes added

#6 Updated by Zheng Yan 6 months ago

  • Backport changed from mimic,luminous to mimic

#7 Updated by Nathan Cutler 6 months ago

not indicated for backport to luminous because, as per Zheng, luminous does not have open file table.

#8 Updated by Nathan Cutler 5 months ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF