Project

General

Profile

Actions
Copy link

Bug #11541

closed

MDS is crashed (mds/CDir.cc: 1391: FAILED assert(!is_complete()))

Added by Andrey Matyashov almost 9 years ago. Updated almost 9 years ago.

Status:
Resolved
Priority:
High
Assignee:
Zheng Yan
Category:
-
Target version:
-
% Done:

0%

Source:
other
Tags:
Backport:
hammer
Regression:
Yes
Severity:
1 - critical
Reviewed:
Affected Versions:
ceph-qa-suite:
upgrade/hammer
Component(FS):
Labels (FS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

After update from 0.82 to 0.94.1 mds crached:

root@virt-master:~# /usr/bin/ceph-mds -i virt-master --debug_ms 9 --debug_mds 9 --pid-file /var/run/ceph/mds.virt-master.pid -c /etc/ceph/ceph.conf --cluster ceph -f > mds.virt-master.log 

2015-05-06 13:13:07.076817 7f9e13a85780 -1 mds.-1.0 log_to_monitors {default=true}
mds/CDir.cc: In function 'void CDir::fetch(MDSInternalContextBase*, const string&, bool)' thread 7f9e0a48f700 time 2015-05-06 13:13:11.230459
mds/CDir.cc: 1391: FAILED assert(!is_complete())
 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x72) [0xa38d72]
 2: (CDir::fetch(MDSInternalContextBase*, std::string const&, bool)+0x7fc) [0x86434c]
 3: (CDir::fetch(MDSInternalContextBase*, bool)+0x2b) [0x8643db]
 4: (MDCache::open_undef_inodes_dirfrags()+0x333) [0x74aec3]
 5: (MDCache::rejoin_gather_finish()+0x6c) [0x79745c]
 6: (MDSInternalContextBase::complete(int)+0x15b) [0x8be9bb]
 7: (C_GatherBase<MDSInternalContextBase, MDSInternalContextGather>::delete_me()+0x16) [0x6ba0b6]
 8: (C_GatherBase<MDSInternalContextBase, MDSInternalContextGather>::sub_finish(MDSInternalContextBase*, int)+0x265) [0x6c0bd5]
 9: (C_GatherBase<MDSInternalContextBase, MDSInternalContextGather>::C_GatherSub::finish(int)+0x12) [0x6c0ce2]
 10: (C_GatherBase<MDSInternalContextBase, MDSInternalContextGather>::C_GatherSub::complete(int)+0x9) [0x6afc59]
 11: (MDS::_advance_queues()+0x112) [0x69e642]
 12: (MDS::ProgressThread::entry()+0x4a) [0x69f12a]
 13: (()+0x6b50) [0x7f9e13410b50]
 14: (clone()+0x6d) [0x7f9e1203095d]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2015-05-06 13:13:11.232253 7f9e0a48f700 -1 mds/CDir.cc: In function 'void CDir::fetch(MDSInternalContextBase*, const string&, bool)' thread 7f9e0a48f700 time 2015-05-06 13:13:11.230459
mds/CDir.cc: 1391: FAILED assert(!is_complete())

 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x72) [0xa38d72]
 2: (CDir::fetch(MDSInternalContextBase*, std::string const&, bool)+0x7fc) [0x86434c]
 3: (CDir::fetch(MDSInternalContextBase*, bool)+0x2b) [0x8643db]
 4: (MDCache::open_undef_inodes_dirfrags()+0x333) [0x74aec3]
 5: (MDCache::rejoin_gather_finish()+0x6c) [0x79745c]
 6: (MDSInternalContextBase::complete(int)+0x15b) [0x8be9bb]
 7: (C_GatherBase<MDSInternalContextBase, MDSInternalContextGather>::delete_me()+0x16) [0x6ba0b6]
 8: (C_GatherBase<MDSInternalContextBase, MDSInternalContextGather>::sub_finish(MDSInternalContextBase*, int)+0x265) [0x6c0bd5]
 9: (C_GatherBase<MDSInternalContextBase, MDSInternalContextGather>::C_GatherSub::finish(int)+0x12) [0x6c0ce2]
 10: (C_GatherBase<MDSInternalContextBase, MDSInternalContextGather>::C_GatherSub::complete(int)+0x9) [0x6afc59]
 11: (MDS::_advance_queues()+0x112) [0x69e642]
 12: (MDS::ProgressThread::entry()+0x4a) [0x69f12a]
 13: (()+0x6b50) [0x7f9e13410b50]
 14: (clone()+0x6d) [0x7f9e1203095d]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

     0> 2015-05-06 13:13:11.232253 7f9e0a48f700 -1 mds/CDir.cc: In function 'void CDir::fetch(MDSInternalContextBase*, const string&, bool)' thread 7f9e0a48f700 time 2015-05-06 13:13:11.230459
mds/CDir.cc: 1391: FAILED assert(!is_complete())

 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x72) [0xa38d72]
 2: (CDir::fetch(MDSInternalContextBase*, std::string const&, bool)+0x7fc) [0x86434c]
 3: (CDir::fetch(MDSInternalContextBase*, bool)+0x2b) [0x8643db]
 4: (MDCache::open_undef_inodes_dirfrags()+0x333) [0x74aec3]
 5: (MDCache::rejoin_gather_finish()+0x6c) [0x79745c]
 6: (MDSInternalContextBase::complete(int)+0x15b) [0x8be9bb]
 7: (C_GatherBase<MDSInternalContextBase, MDSInternalContextGather>::delete_me()+0x16) [0x6ba0b6]
 8: (C_GatherBase<MDSInternalContextBase, MDSInternalContextGather>::sub_finish(MDSInternalContextBase*, int)+0x265) [0x6c0bd5]
 9: (C_GatherBase<MDSInternalContextBase, MDSInternalContextGather>::C_GatherSub::finish(int)+0x12) [0x6c0ce2]
 10: (C_GatherBase<MDSInternalContextBase, MDSInternalContextGather>::C_GatherSub::complete(int)+0x9) [0x6afc59]
 11: (MDS::_advance_queues()+0x112) [0x69e642]
 12: (MDS::ProgressThread::entry()+0x4a) [0x69f12a]
 13: (()+0x6b50) [0x7f9e13410b50]
 14: (clone()+0x6d) [0x7f9e1203095d]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

terminate called after throwing an instance of 'ceph::FailedAssertion'
*** Caught signal (Aborted) **
 in thread 7f9e0a48f700
 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff)
 1: /usr/bin/ceph-mds() [0x9556ec]
 2: (()+0xf0a0) [0x7f9e134190a0]
 3: (gsignal()+0x35) [0x7f9e11f87165]
 4: (abort()+0x180) [0x7f9e11f8a3e0]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f9e127dd89d]
 6: (()+0x63996) [0x7f9e127db996]
 7: (()+0x639c3) [0x7f9e127db9c3]
 8: (()+0x63bee) [0x7f9e127dbbee]
 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x220) [0xa38f20]
 10: (CDir::fetch(MDSInternalContextBase*, std::string const&, bool)+0x7fc) [0x86434c]
 11: (CDir::fetch(MDSInternalContextBase*, bool)+0x2b) [0x8643db]
 12: (MDCache::open_undef_inodes_dirfrags()+0x333) [0x74aec3]
 13: (MDCache::rejoin_gather_finish()+0x6c) [0x79745c]
 14: (MDSInternalContextBase::complete(int)+0x15b) [0x8be9bb]
 15: (C_GatherBase<MDSInternalContextBase, MDSInternalContextGather>::delete_me()+0x16) [0x6ba0b6]
 16: (C_GatherBase<MDSInternalContextBase, MDSInternalContextGather>::sub_finish(MDSInternalContextBase*, int)+0x265) [0x6c0bd5]
 17: (C_GatherBase<MDSInternalContextBase, MDSInternalContextGather>::C_GatherSub::finish(int)+0x12) [0x6c0ce2]
 18: (C_GatherBase<MDSInternalContextBase, MDSInternalContextGather>::C_GatherSub::complete(int)+0x9) [0x6afc59]
 19: (MDS::_advance_queues()+0x112) [0x69e642]
 20: (MDS::ProgressThread::entry()+0x4a) [0x69f12a]
 21: (()+0x6b50) [0x7f9e13410b50]
 22: (clone()+0x6d) [0x7f9e1203095d]
2015-05-06 13:13:11.294984 7f9e0a48f700 -1 *** Caught signal (Aborted) **
 in thread 7f9e0a48f700

 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff)
 1: /usr/bin/ceph-mds() [0x9556ec]
 2: (()+0xf0a0) [0x7f9e134190a0]
 3: (gsignal()+0x35) [0x7f9e11f87165]
 4: (abort()+0x180) [0x7f9e11f8a3e0]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f9e127dd89d]
 6: (()+0x63996) [0x7f9e127db996]
 7: (()+0x639c3) [0x7f9e127db9c3]
 8: (()+0x63bee) [0x7f9e127dbbee]
 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x220) [0xa38f20]
 10: (CDir::fetch(MDSInternalContextBase*, std::string const&, bool)+0x7fc) [0x86434c]
 11: (CDir::fetch(MDSInternalContextBase*, bool)+0x2b) [0x8643db]
 12: (MDCache::open_undef_inodes_dirfrags()+0x333) [0x74aec3]
 13: (MDCache::rejoin_gather_finish()+0x6c) [0x79745c]
 14: (MDSInternalContextBase::complete(int)+0x15b) [0x8be9bb]
 15: (C_GatherBase<MDSInternalContextBase, MDSInternalContextGather>::delete_me()+0x16) [0x6ba0b6]
 16: (C_GatherBase<MDSInternalContextBase, MDSInternalContextGather>::sub_finish(MDSInternalContextBase*, int)+0x265) [0x6c0bd5]
 17: (C_GatherBase<MDSInternalContextBase, MDSInternalContextGather>::C_GatherSub::finish(int)+0x12) [0x6c0ce2]
 18: (C_GatherBase<MDSInternalContextBase, MDSInternalContextGather>::C_GatherSub::complete(int)+0x9) [0x6afc59]
 19: (MDS::_advance_queues()+0x112) [0x69e642]
 20: (MDS::ProgressThread::entry()+0x4a) [0x69f12a]
 21: (()+0x6b50) [0x7f9e13410b50]
 22: (clone()+0x6d) [0x7f9e1203095d]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

     0> 2015-05-06 13:13:11.294984 7f9e0a48f700 -1 *** Caught signal (Aborted) **
 in thread 7f9e0a48f700

 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff)
 1: /usr/bin/ceph-mds() [0x9556ec]
 2: (()+0xf0a0) [0x7f9e134190a0]
 3: (gsignal()+0x35) [0x7f9e11f87165]
 4: (abort()+0x180) [0x7f9e11f8a3e0]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f9e127dd89d]
 6: (()+0x63996) [0x7f9e127db996]
 7: (()+0x639c3) [0x7f9e127db9c3]
 8: (()+0x63bee) [0x7f9e127dbbee]
 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x220) [0xa38f20]
 10: (CDir::fetch(MDSInternalContextBase*, std::string const&, bool)+0x7fc) [0x86434c]
 11: (CDir::fetch(MDSInternalContextBase*, bool)+0x2b) [0x8643db]
 12: (MDCache::open_undef_inodes_dirfrags()+0x333) [0x74aec3]
 13: (MDCache::rejoin_gather_finish()+0x6c) [0x79745c]
 14: (MDSInternalContextBase::complete(int)+0x15b) [0x8be9bb]
 15: (C_GatherBase<MDSInternalContextBase, MDSInternalContextGather>::delete_me()+0x16) [0x6ba0b6]
 16: (C_GatherBase<MDSInternalContextBase, MDSInternalContextGather>::sub_finish(MDSInternalContextBase*, int)+0x265) [0x6c0bd5]
 17: (C_GatherBase<MDSInternalContextBase, MDSInternalContextGather>::C_GatherSub::finish(int)+0x12) [0x6c0ce2]
 18: (C_GatherBase<MDSInternalContextBase, MDSInternalContextGather>::C_GatherSub::complete(int)+0x9) [0x6afc59]
 19: (MDS::_advance_queues()+0x112) [0x69e642]
 20: (MDS::ProgressThread::entry()+0x4a) [0x69f12a]
 21: (()+0x6b50) [0x7f9e13410b50]
 22: (clone()+0x6d) [0x7f9e1203095d]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

Aborted

See more info in attached logfile. (start cmd: /usr/bin/ceph-mds -i virt-master --debug_ms 9 --debug_mds 9 -d --pid-file /var/run/ceph/mds.virt-master.pid -c /etc/ceph/ceph.conf --cluster ceph > mds.virt-master.log 2>&1 )

Files

mds.virt-master.log.bz2 (319 KB) mds.virt-master.log.bz2 Andrey Matyashov, 05/06/2015 10:28 AM

Related issues 1 (0 open — 1 closed)

Copied to CephFS - Backport #11737: MDS is crashed (mds/CDir.cc: 1391: FAILED assert(!is_complete()))ResolvedAbhishek LekshmananActions
Actions
Copy link

Also available in: Atom PDF