Bug #608
mds: MDCache::create_system_inode()
% Done:
0%
Source:
Tags:
Backport:
Regression:
Severity:
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Labels (FS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
On a small test cluster I saw that my MDS was not coming up after a fresh mkcephfs, this is what the log showed:
2010-11-23 16:08:47.923106 7fa332d77710 mds0.6 mds has 1 queued contexts 2010-11-23 16:08:47.923115 7fa332d77710 mds0.6 0x299cab0 2010-11-23 16:08:47.923127 7fa332d77710 mds0.cache open_root 2010-11-23 16:08:47.923140 7fa332d77710 mds0.cache.dir(1) fetch on [dir 1 / [2,head] auth v=1 cv=0/0 dir_auth=0 state=1073741824 f(v0 1=0+1) n(v0 1=0+1) hs=0+0,ss=0+0 | subtree 0x29f5000] 2010-11-23 16:08:47.923157 7fa332d77710 2010-11-23 16:08:47.923160 mds0.cache.dir(1) add_waiter 2 0x299cb00 on [dir 1 / [2,head] auth v=1 cv=0/0 dir_auth=0 state=1073741824 f(v0 1=0+1) n(v0 1=0+1) hs=0+0,ss=0+0 | subtree waiter 0x29f5000] 2010-11-23 16:08:47.923175 7fa332d77710 mds0.cache.dir(1) auth_pin by 0x29f5000 on [dir 1 / [2,head] auth v=1 cv=0/0 dir_auth=0 ap=1+0+0 state=1073741824 f(v0 1=0+1) n(v0 1=0+1) hs=0+0,ss=0+0 | subtree waiter authpin 0x29f5000] count now 1 + 0 2010-11-23 16:08:47.923913 7fa332d77710 mds0.cache.dir(1) _fetched 545 bytes for [dir 1 / [2,head] auth v=1 cv=0/0 dir_auth=0 ap=1+0+0 state=1073741952 f(v0 1=0+1) n(v0 1=0+1) hs=0+0,ss=0+0 | subtree waiter authpin 0x29f5000] want_dn= 2010-11-23 16:08:47.923937 7fa332d77710 mds0.cache.dir(1) _fetched version 1, 545 bytes, 1 keys 2010-11-23 16:08:47.923947 7fa332d77710 mds0.cache.snaprealm(1 seq 1 0x29d9000) have_past_parents_open [1,head] 2010-11-23 16:08:47.923957 7fa332d77710 mds0.cache.snaprealm(1 seq 1 0x29d9000) build_snap_set [0,head] on snaprealm(1 seq 1 lc 0 cr 0 cps 1 snaps={} 0x29d9000) 2010-11-23 16:08:47.923976 7fa332d77710 mds0.cache.snaprealm(1 seq 1 0x29d9000) build_snap_trace my_snaps [] 2010-11-23 16:08:47.923987 7fa332d77710 mds0.cache.snaprealm(1 seq 1 0x29d9000) check_cache rebuilt seq 1 cached_seq 1 cached_last_created 0 cached_last_destroyed 0) 2010-11-23 16:08:47.924000 7fa332d77710 mds0.cache.dir(1) lookup (head, '.ceph') 2010-11-23 16:08:47.924015 7fa332d77710 mds0.cache.dir(1) add_primary_dentry [dentry #1/.ceph [2,head] auth (dversion lock) pv=0 v=1 inode=0x29e1240 0x2a063b0] 2010-11-23 16:08:47.924030 7fa332d77710 mds0.cache.dir(1) _fetched got [dentry #1/.ceph [2,head] auth (dversion lock) pv=0 v=1 inode=0x29e1240 0x2a063b0] [inode 2 [...2,head] /.ceph/ auth v1 f() n() (iversion lock) 0x29e1240] 2010-11-23 16:08:47.924049 7fa332d77710 mds0.cache.dir(1) auth_unpin by 0x29f5000 on [dir 1 / [2,head] auth v=1 cv=0/0 dir_auth=0 state=1073741826|complete f(v0 1=0+1) n(v0 1=0+1) hs=1+0,ss=0+0 | child subtree waiter 0x29f5000] count now 0 + 0 2010-11-23 16:08:47.924066 7fa332d77710 mds0.cache.dir(1) finish_waiting mask 2 result 0 on [dir 1 / [2,head] auth v=1 cv=0/0 dir_auth=0 state=1073741826|complete f(v0 1=0+1) n(v0 1=0+1) hs=1+0,ss=0+0 | child subtree waiter 0x29f5000] 2010-11-23 16:08:47.924082 7fa332d77710 2010-11-23 16:08:47.924085 mds0.cache.dir(1) take_waiting mask 2 took 0x299cb00 tag 2 on [dir 1 / [2,head] auth v=1 cv=0/0 dir_auth=0 state=1073741826|complete f(v0 1=0+1) n(v0 1=0+1) hs=1+0,ss=0+0 | child subtree waiter 0x29f5000] 2010-11-23 16:08:47.924103 7fa332d77710 mds0.6 mds has 1 queued contexts 2010-11-23 16:08:47.924112 7fa332d77710 mds0.6 0x299cb00 2010-11-23 16:08:47.924121 7fa332d77710 mds0.cache open_root 2010-11-23 16:08:47.924130 7fa332d77710 mds0.cache populate_mydir [dir 100 ~mds0/ [2,head] auth v=1 cv=0/0 dir_auth=0 state=1073741826|complete f(v0 2=1+1) n(v0 2=1+1) hs=2+0,ss=0+0 | child subtree 0x29f5618] 2010-11-23 16:08:47.924181 7fa332d77710 mds0.cache.dir(100) lookup (head, 'stray0') 2010-11-23 16:08:47.924192 7fa332d77710 mds0.cache creating system inode with ino:600 mds/MDCache.cc: In function 'void MDCache::add_inode(CInode*)': mds/MDCache.cc:189: FAILED assert(inode_map.count(in->vino()) == 0) ceph version 0.24~rc (commit:c327c6a2064f3a14f0221b53dc0c87d744d0e977) 1: /usr/bin/cmds() [0x50f17d] 2: (MDCache::create_system_inode(inodeno_t, int)+0x202) [0x51afe2] 3: (MDCache::populate_mydir()+0x1e8) [0x546298] 4: (MDS::_dispatch(Message*)+0x6e2) [0x4a7062] 5: (MDS::ms_dispatch(Message*)+0x6d) [0x4ab3fd] 6: (SimpleMessenger::dispatch_entry()+0x759) [0x4810e9] 7: (SimpleMessenger::DispatchThread::entry()+0x1c) [0x478edc] 8: (Thread::_entry_func(void*)+0xa) [0x48d78a] 9: (()+0x69ca) [0x7fa33572d9ca] 10: (clone()+0x6d) [0x7fa33449370d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. *** Caught signal (ABRT) *** ceph version 0.24~rc (commit:c327c6a2064f3a14f0221b53dc0c87d744d0e977) 1: (sigabrt_handler(int)+0x7d) [0x6ddf4d] 2: (()+0x33af0) [0x7fa3343e0af0] 3: (gsignal()+0x35) [0x7fa3343e0a75] 4: (abort()+0x180) [0x7fa3343e45c0] 5: (__gnu_cxx::__verbose_terminate_handler()+0x115) [0x7fa334c968e5] 6: (()+0xcad16) [0x7fa334c94d16] 7: (()+0xcad43) [0x7fa334c94d43] 8: (()+0xcae3e) [0x7fa334c94e3e] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x448) [0x6c9bd8] 10: /usr/bin/cmds() [0x50f17d] 11: (MDCache::create_system_inode(inodeno_t, int)+0x202) [0x51afe2] 12: (MDCache::populate_mydir()+0x1e8) [0x546298] 13: (MDS::_dispatch(Message*)+0x6e2) [0x4a7062] 14: (MDS::ms_dispatch(Message*)+0x6d) [0x4ab3fd] 15: (SimpleMessenger::dispatch_entry()+0x759) [0x4810e9] 16: (SimpleMessenger::DispatchThread::entry()+0x1c) [0x478edc] 17: (Thread::_entry_func(void*)+0xa) [0x48d78a] 18: (()+0x69ca) [0x7fa33572d9ca] 19: (clone()+0x6d) [0x7fa33449370d]
I've ran cdebugpack and uploaded the data to logger.ceph.widodh.nl:/srv/ceph/issues/cmds_create_system_inode
If you want to check out the machine itself, it's noisy.ceph.widodh.nl, you can access it through the logger machine.
History
#1 Updated by Sage Weil over 13 years ago
- Status changed from New to Resolved
- Assignee set to Sage Weil
this should be fixed by commit:fc212548aea1d7f001b56ba096a79ba54b8a92c3
Thanks!
#2 Updated by Sage Weil over 13 years ago
- Target version set to v0.24
#3 Updated by John Spray over 7 years ago
- Project changed from Ceph to CephFS
- Category deleted (
1) - Target version deleted (
v0.24)
Bulk updating project=ceph category=mds bugs so that I can remove the MDS category from the Ceph project to avoid confusion.