Bug #11314: qa: MDS crashed and the runs hung without ever timing out - CephFS - Ceph

Actions

Copy link

Bug #11314

closed

qa: MDS crashed and the runs hung without ever timing out

Added by Greg Farnum about 9 years ago. Updated almost 5 years ago.

Status:

Duplicate

Priority:

High

Assignee:

Category:

Testing

Target version:

% Done:

Source:

Q/A

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Component(FS):

qa-suite

Labels (FS):

Pull request ID:

28378

Crash signature (v1):

Crash signature (v2):

Description

http://pulpito.ceph.com/gregf-2015-04-01_20:53:07-fs-greg-fs-testing---basic-multi/

The MDS crashed in many of these, but for some reason the tasks didn't notice and kept running.

Anyway, we need to make it notice in cases like this so that the failures don't lock up the whole lab overnight and prevent any tests from running. :(

Related issues 1 (0 open — 1 closed)

Actions

Copy link

Updated by John Spray about 9 years ago

This was mentioned in #10369:

Once we have that checker method, we should also introduce a check that all the daemons are really running, so that if e.g. an OSD crashes unexpectedly, we detect it immediately rather than waiting a long time for at timeout of some kind to end the test.

Actions

Copy link

Updated by Greg Farnum about 8 years ago

Subject changed from teuthology: MDS crashed and the runs hung without ever timing out to qa: MDS crashed and the runs hung without ever timing out
Priority changed from High to Normal

We clearly aren't treating this as very important, and I think we've had more trouble with OSDs doing this than MDSes lately, so whatever.

Actions

Copy link

Updated by Patrick Donnelly over 6 years ago

Assignee set to Patrick Donnelly

I added a DaemonWatchdog in the mds_thrash.py code that catches this kind of thing. We should pull it out into its own context so it can be used elsewhere.

Actions

Copy link