Project

General

Profile

Bug #50016

qa: test_damage: "RuntimeError: 2 mutations had unexpected outcomes"

Added by Patrick Donnelly 7 months ago. Updated 5 months ago.

Status:
New
Priority:
High
Assignee:
-
Category:
-
Target version:
% Done:

0%

Source:
Q/A
Tags:
Backport:
pacific
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
kceph
Labels (FS):
qa, qa-failure
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

2021-03-26T02:06:47.407 INFO:tasks.ceph.mds.a.smithi045.stderr:2021-03-26T02:06:47.403+0000 7f3e32bc5700 -1 mds.0.cache.dir(0x1) Corrupt fnode in dirfrag 0x1: void fnode_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) no longer understand old encoding version 3 < 101: Malformed input
2021-03-26T02:06:47.408 INFO:tasks.ceph.mds.a.smithi045.stderr:2021-03-26T02:06:47.403+0000 7f3e32bc5700 -1 mds.0.damage notify_dirfrag Damage to fragment * of ino 0x1 is fatal because it is a system directory for this rank
2021-03-26T02:06:47.415 INFO:tasks.ceph.mds.a.smithi045.stderr:   -15> 2021-03-26T02:06:47.403+0000 7f3e32bc5700 -1 mds.0.cache.dir(0x1) Corrupt fnode in dirfrag 0x1: void fnode_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) no longer understand old encoding version 3 < 101: Malformed input
2021-03-26T02:06:47.416 INFO:tasks.ceph.mds.a.smithi045.stderr:   -12> 2021-03-26T02:06:47.403+0000 7f3e32bc5700 -1 mds.0.damage notify_dirfrag Damage to fragment * of ino 0x1 is fatal because it is a system directory for this rank
2021-03-26T02:06:47.459 INFO:tasks.ceph.mds.a.smithi045.stdout:starting mds.a at
2021-03-26T02:06:48.403 INFO:tasks.ceph.mon.a.smithi045.stderr:2021-03-26T02:06:48.403+0000 7fc490deb700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN)
2021-03-26T02:06:48.403 INFO:tasks.ceph.mon.a.smithi045.stderr:2021-03-26T02:06:48.403+0000 7fc490deb700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 mds daemon damaged (MDS_DAMAGE)
...
2021-03-26T02:06:50.346 INFO:tasks.cephfs.test_damage:Result: Mutation 'Corrupt omap header on 1.00000000' led to DAMAGED state
2021-03-26T02:06:50.347 ERROR:tasks.cephfs.test_damage:2 mutations had unexpected outcomes:
2021-03-26T02:06:50.347 ERROR:tasks.cephfs.test_damage:  Expected 'eio' actually 'client failed' from 'Corrupt omap key 10000000000.00000000:sixmegs_head'
2021-03-26T02:06:50.348 ERROR:tasks.cephfs.test_damage:  Expected 'eio' actually 'client failed' from 'Corrupt omap key 1.00000000:subdir_head'
...
2021-03-26T02:17:10.760 INFO:tasks.cephfs_test_runner:test_object_deletion (tasks.cephfs.test_damage.TestDamage) ... ERROR
2021-03-26T02:17:10.760 INFO:tasks.cephfs_test_runner:
2021-03-26T02:17:10.761 INFO:tasks.cephfs_test_runner:======================================================================
2021-03-26T02:17:10.761 INFO:tasks.cephfs_test_runner:ERROR: test_object_deletion (tasks.cephfs.test_damage.TestDamage)
2021-03-26T02:17:10.761 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2021-03-26T02:17:10.762 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2021-03-26T02:17:10.762 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_ceph_ceph-c_e59cbc129006d21160edc03bc105109e762c816b/qa/tasks/cephfs/test_damage.py", line 370, in test_object_deletion
2021-03-26T02:17:10.762 INFO:tasks.cephfs_test_runner:    raise RuntimeError("{0} mutations had unexpected outcomes".format(len(failures)))
2021-03-26T02:17:10.763 INFO:tasks.cephfs_test_runner:RuntimeError: 2 mutations had unexpected outcomes
2021-03-26T02:17:10.763 INFO:tasks.cephfs_test_runner:

From: /ceph/teuthology-archive/yuriw-2021-03-25_21:03:23-fs-wip-yuri-testing-2021-03-25-1105-pacific-distro-basic-smithi/5999489/teuthology.log

This is with the testing kernel.

History

#1 Updated by Patrick Donnelly 7 months ago

/ceph/teuthology-archive/pdonnell-2021-03-24_23:26:35-fs-wip-pdonnell-testing-20210324.190252-distro-basic-smithi/5996081/teuthology.log

Also available in: Atom PDF