Ceph » CephFS

Zheng, please take a look.

I'm running into the same problem on luminous 12.2.0 - while removing a directory with lots of files, the MDS crashes, and even after repeated restarts it does not recover. I did not turn directory fragmentation on. I'm attaching the MDS log of the crash (cephtest-mds.xooby04.log.gz).

Also - any procedure to recover would be helpful.

It's dup of http://tracker.ceph.com/issues/21070. fix has already been fixed in master branch

PR for luminous https://github.com/ceph/ceph/pull/17686

description¶

   -15> 2017-09-07 13:27:37.898704 7f79be11a700  4 mds.1.purge_queue push: pushing inode 0x0x1000001d350
   -14> 2017-09-07 13:27:37.898713 7f79be11a700  4 mds.1.purge_queue push: pushing inode 0x0x1000001e354
   -13> 2017-09-07 13:27:37.898730 7f79be11a700  4 mds.1.purge_queue push: pushing inode 0x0x20000003d35
   -12> 2017-09-07 13:27:37.898740 7f79be11a700  4 mds.1.purge_queue push: pushing inode 0x0x20000003cbe
   -11> 2017-09-07 13:27:37.898750 7f79be11a700  4 mds.1.purge_queue push: pushing inode 0x0x20000003d0d
   -10> 2017-09-07 13:27:37.898760 7f79be11a700  4 mds.1.purge_queue push: pushing inode 0x0x20000003cd8
    -9> 2017-09-07 13:27:37.898782 7f79be11a700  4 mds.1.purge_queue push: pushing inode 0x0x10000023fd1
    -8> 2017-09-07 13:27:37.898791 7f79be11a700  4 mds.1.purge_queue push: pushing inode 0x0x10000023fe6
    -7> 2017-09-07 13:27:37.898850 7f79be11a700  1 -- 10.8.128.105:6800/338052715 --> 10.8.128.36:6808/21995 -- osd_op(unknown.0.38:97 2.6 2:6f2c300e:::60f.00000000:head [omap-get-header,omap
-get-vals,getxattr parent] snapc 0=[] ondisk+read+known_if_redirected+full_force e42) v8 -- 0x1c934d3040 con 0
    -6> 2017-09-07 13:27:37.952466 7f79c58c6700  5 -- 10.8.128.105:6800/338052715 >> 10.8.128.36:6808/21995 conn(0x1c91f75000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=189 cs=1 l=
1). rx osd.8 seq 29 0x1c91f14680 osd_op_reply(83 501.00000001 [read 458397~3735907 [fadvise_dontneed]] v0'0 uv1651 ondisk = 0) v8
    -5> 2017-09-07 13:27:37.952506 7f79c58c6700  1 -- 10.8.128.105:6800/338052715 <== osd.8 10.8.128.36:6808/21995 29 ==== osd_op_reply(83 501.00000001 [read 458397~3735907 [fadvise_dontneed]
] v0'0 uv1651 ondisk = 0) v8 ==== 156+0+3735907 (1828211159 0 2860789470) 0x1c91f14680 con 0x1c91f75000
    -4> 2017-09-07 13:27:37.952625 7f79c58c6700  5 -- 10.8.128.105:6800/338052715 >> 10.8.128.36:6808/21995 conn(0x1c91f75000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=189 cs=1 l=
1). rx osd.8 seq 30 0x1c91f14680 osd_op_reply(84 1000000c0da.02c00000 [omap-get-header,omap-get-vals] v0'0 uv1757 ondisk = 0) v8
    -3> 2017-09-07 13:27:37.952646 7f79c58c6700  1 -- 10.8.128.105:6800/338052715 <== osd.8 10.8.128.36:6808/21995 30 ==== osd_op_reply(84 1000000c0da.02c00000 [omap-get-header,omap-get-vals]
 v0'0 uv1757 ondisk = 0) v8 ==== 206+0+24480 (2245149694 0 2706660154) 0x1c91f14680 con 0x1c91f75000
    -2> 2017-09-07 13:27:37.952724 7f79bd919700  1 -- 10.8.128.105:6800/338052715 --> 10.8.128.103:6800/20641 -- osd_op(unknown.0.38:98 1.2 1:55af870c:::1000000d735.00000000:head [delete] sna
pc 1=[] ondisk+write+known_if_redirected+full_force e42) v8 -- 0x1c934c0700 con 0
    -1> 2017-09-07 13:27:37.953525 7f79c58c6700  2 -- 10.8.128.105:6800/338052715 >> 10.8.128.103:6800/20641 conn(0x1c91fb5000 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_con
nection got newly_acked_seq 0 vs out_seq 0
     0> 2017-09-07 13:27:37.955163 7f79be11a700 -1 *** Caught signal (Segmentation fault) **
 in thread 7f79be11a700 thread_name:mds_rank_progr

and backtrace:

ceph version 12.2.0-2redhat1xenial (b661348f156f148d764b998b65b90451f096cb27) luminous (rc)
 1: (()+0x59b344) [0x1c86d06344]
 2: (()+0x11390) [0x7f79c8420390]
 3: (Server::handle_client_readdir(boost::intrusive_ptr<MDRequestImpl>&)+0xc05) [0x1c86a65615]
 4: (Server::dispatch_client_request(boost::intrusive_ptr<MDRequestImpl>&)+0x9f1) [0x1c86a89341]
 5: (MDSInternalContextBase::complete(int)+0x18b) [0x1c86c894bb]
 6: (MDSRank::_advance_queues()+0x6a7) [0x1c86a0e957]
 7: (MDSRank::ProgressThread::entry()+0x4a) [0x1c86a0ef7a]
 8: (()+0x76ba) [0x7f79c84166ba]
 9: (clone()+0x6d) [0x7f79c74823dd]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

Steps to Reproduce:
1. Created a cluster with 3 mds
2. Initially 1 mds was active and 2 were standby
3. moved another mds to be active
4. so effectively 2 mds with 1 standby
5. created directories and files on fuse mount
6. one of the directory had 100k files of 0 size
7. I was just probing the dirfrag ls from admin socket
8. Initiated rm -rf on the directory
9. Was continuously probing "objecter_requests" from admin socket of one mds
10. The other mds was crashed meanwhile.
11. Though standby has taken over client I/Os are hung

BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1489461