Project

General

Profile

Bug #18179

MDS crashes on missing metadata object

Added by Wido den Hollander 5 months ago. Updated 9 days ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
fsck/damage handling
Target version:
-
Start date:
12/07/2016
Due date:
% Done:

0%

Source:
Community (user)
Tags:
jewel,cephfs,mds,metadata
Backport:
jewel, kraken
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Release:
jewel
Component(FS):
MDS
Needs Doc:
No

Description

Saw this crash happening on a Jewel 10.2.3 MDS when it was missing a object in the metadata pool:

2016-12-07 08:29:58.852595 7f3d74c96700 -1 log_channel(cluster) log [ERR] : dir 10011a4767b object missing on disk; some files may be lost
2016-12-07 08:29:58.855070 7f3d74c96700 -1 mds/MDCache.cc: In function 'virtual void C_MDC_OpenInoTraverseDir::finish(int)' thread 7f3d74c96700 time 2016-12-07 08:29:58.852637
mds/MDCache.cc: 8213: FAILED assert(r >= 0)

ceph version 10.2.3 (ecc23778eb545d8dd55e2e4735b53cc93f92e65b)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0x7f3d7fbe3a45]
2: (C_MDC_OpenInoTraverseDir::finish(int)+0x13e) [0x7f3d7f93d59e]
3: (MDSInternalContextBase::complete(int)+0x1eb) [0x7f3d7fa408ab]
4: (CDir::finish_waiting(unsigned long, int)+0x324) [0x7f3d7f9bb5a4]
5: (CDir::_omap_fetched(ceph::buffer::list&, std::map<std::string, ceph::buffer::list, std::less<std::string>, std::allocator<std::pair<std::string const, ceph::buffer::list> > >&, std::string const&, int)+0x65a) [0x7f3d7f9d29ba]
6: (CDir::_tmap_fetched(ceph::buffer::list&, std::string const&, int)+0x368) [0x7f3d7f9d40f8]
7: (MDSIOContextBase::complete(int)+0xa4) [0x7f3d7fa40bd4]
8: (Finisher::finisher_thread_entry()+0x27d) [0x7f3d7fb0f17d]
9: (()+0x7dc5) [0x7f3d7e9ccdc5]
10: (clone()+0x6d) [0x7f3d7d497ced]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

As per discussion on the ceph-users list I'm opening this issue so that the MDS can be patched to record damaged metadata and continue it's operation.


Related issues

Copied to Backport #18565: jewel: MDS crashes on missing metadata object Resolved
Copied to Backport #18566: kraken: MDS crashes on missing metadata object Resolved

History

#1 Updated by Loic Dachary 4 months ago

  • Target version deleted (v10.2.5)

#2 Updated by Zheng Yan 4 months ago

  • Status changed from New to Need Review

#3 Updated by John Spray 4 months ago

  • Status changed from Need Review to Pending Backport
  • Backport set to jewel kraken

#4 Updated by Nathan Cutler 4 months ago

  • Backport changed from jewel kraken to jewel, kraken

#5 Updated by Nathan Cutler 3 months ago

  • Copied to Backport #18565: jewel: MDS crashes on missing metadata object added

#6 Updated by Nathan Cutler 3 months ago

  • Copied to Backport #18566: kraken: MDS crashes on missing metadata object added

#7 Updated by Nathan Cutler 9 days ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF