Ceph » CephFS

Now I have a directory 'ls' in which make crash mds.

BTW, if you know what directory the hard link(s) in this directory reference (it'll be the first time the linked files in the bad dir were original written), you can work around this bug by doing an ls in that old/original dir first, and then doing the ls in the bad dir. Because the link targets will already be in cache the anchortable won't be queried at all.

Updated Ceph to 0.39 and the bug seems to be gone.

Got it again with 0.39. Still there.

Unfortunately there's not enough info in this log either. We're going to need a log with (at minimum) level 10 mds debugging, maybe higher.

Or a reliable reproducer we can run locally, of course!

This log is less illuminating than I'd hoped since it's just replay commits. :(

However, it did give me one idea to look into (are replayed ops failing if they got committed while removing an inode and we didn't notice?); let me check into it and I'll post back her shortly.

After looking into this a lot today, none of my ideas panned out. I also tried a few simple tests to reproduce locally and was unable to do so. :(

Just hit this bug again with 0.47.2 on ext4, so it is still open and it is still an issue.

This bug seems to be reproduced with the deadlock described on the mailing list as
"OSD deadlock with cephfs client and OSD on same machine", it is probably related to a partial sync/commit of osd journal or storage.

We are auto creating up to 200 preconfigured home directories on all four nodes, each home dir consists of ca. 400 dirs and files with ca. 16 MB of data. AFAIK, there are no hard links involved. So it is a massive parallel creation of many small files, probably lots of metadata for them.

Amon, are you sure you're hitting exactly this bug with your users? This particular one requires hard links to be in use to even be triggered.

Anyway, this is still in our queue but will require some in-depth investigation that we unfortunately haven't had time for.

No, I am not sure about that. Only saw the same assert message and a similar trace, so I assumed it to be the same bug.

Hi,

Still have the problem here.

mds/AnchorServer.cc: 249: FAILED assert(anchor_map.count(curino) == 1)

 ceph version 0.48.2argonaut (commit:3e02b2fad88c2a95d9c0c86878f10d1beb780bfe)
 1: (AnchorServer::handle_query(MMDSTableRequest*)+0x749) [0x678d89]
 2: (MDS::handle_deferrable_message(Message*)+0xe84) [0x4af644]
 3: (MDS::_dispatch(Message*)+0xc23) [0x4c24c3]
 4: (MDS::ms_dispatch(Message*)+0x1eb) [0x4c34ab]
 5: (SimpleMessenger::DispatchQueue::entry()+0x903) [0x7579a3]
 6: (SimpleMessenger::dispatch_entry()+0x24) [0x758744]
 7: (SimpleMessenger::DispatchThread::entry()+0xd) [0x717a5d]
 8: (()+0x7e9a) [0x7fe878223e9a]
 9: (clone()+0x6d) [0x7fe8770c4cbd]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

    -4> 2012-11-28 03:42:30.334561 7fe86ff34700  5 throttle(msgr_dispatch_throttler-mds 0x233f9e0) get 118 (234 -> 352)
    -3> 2012-11-28 03:42:30.334689 7fe870c3a700  5 throttle(msgr_dispatch_throttler-mds 0x233f9e0) get 2820 (352 -> 3172)
    -2> 2012-11-28 03:42:30.334736 7fe86fd32700  5 throttle(msgr_dispatch_throttler-mds 0x233f9e0) get 117 (3172 -> 3289)
    -1> 2012-11-28 03:42:30.338286 7fe870136700  5 throttle(msgr_dispatch_throttler-mds 0x233f9e0) get 117 (3289 -> 3406)
     0> 2012-11-28 03:42:30.338311 7fe870136700  5 throttle(msgr_dispatch_throttler-mds 0x233f9e0) get 119 (3406 -> 3525)
--- end dump of recent events ---
2012-11-28 03:42:30.343327 7fe872e40700 -1 *** Caught signal (Aborted) **
 in thread 7fe872e40700

 ceph version 0.48.2argonaut (commit:3e02b2fad88c2a95d9c0c86878f10d1beb780bfe)
 1: /usr/bin/ceph-mds() [0x7e315a]
 2: (()+0xfcb0) [0x7fe87822bcb0]
 3: (gsignal()+0x35) [0x7fe877007425]
 4: (abort()+0x17b) [0x7fe87700ab8b]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7fe87795969d]
 6: (()+0xb5846) [0x7fe877957846]
 7: (()+0xb5873) [0x7fe877957873]
 8: (()+0xb596e) [0x7fe87795796e]
 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1de) [0x7771fe]
 10: (AnchorServer::handle_query(MMDSTableRequest*)+0x749) [0x678d89]
 11: (MDS::handle_deferrable_message(Message*)+0xe84) [0x4af644]
 12: (MDS::_dispatch(Message*)+0xc23) [0x4c24c3]
 13: (MDS::ms_dispatch(Message*)+0x1eb) [0x4c34ab]
 14: (SimpleMessenger::DispatchQueue::entry()+0x903) [0x7579a3]
 15: (SimpleMessenger::dispatch_entry()+0x24) [0x758744]
 16: (SimpleMessenger::DispatchThread::entry()+0xd) [0x717a5d]
 17: (()+0x7e9a) [0x7fe878223e9a]
 18: (clone()+0x6d) [0x7fe8770c4cbd]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

Each time I want to restart mds server it crashes again, someone have an idea?

Regards.
Sam

Sage Weil wrote:

most likely fixed by yan's patch

commit id?

commit:a3aad3c3c58ffd7cd556275cd1c7dbc69b3eb4dd, which is in v0.55.