Backport #21357
Updated by Nathan Cutler over 6 years ago
https://github.com/ceph/ceph/pull/17686 <pre> -15> 2017-09-07 13:27:37.898704 7f79be11a700 4 mds.1.purge_queue push: pushing inode 0x0x1000001d350 -14> 2017-09-07 13:27:37.898713 7f79be11a700 4 mds.1.purge_queue push: pushing inode 0x0x1000001e354 -13> 2017-09-07 13:27:37.898730 7f79be11a700 4 mds.1.purge_queue push: pushing inode 0x0x20000003d35 -12> 2017-09-07 13:27:37.898740 7f79be11a700 4 mds.1.purge_queue push: pushing inode 0x0x20000003cbe -11> 2017-09-07 13:27:37.898750 7f79be11a700 4 mds.1.purge_queue push: pushing inode 0x0x20000003d0d -10> 2017-09-07 13:27:37.898760 7f79be11a700 4 mds.1.purge_queue push: pushing inode 0x0x20000003cd8 -9> 2017-09-07 13:27:37.898782 7f79be11a700 4 mds.1.purge_queue push: pushing inode 0x0x10000023fd1 -8> 2017-09-07 13:27:37.898791 7f79be11a700 4 mds.1.purge_queue push: pushing inode 0x0x10000023fe6 -7> 2017-09-07 13:27:37.898850 7f79be11a700 1 -- 10.8.128.105:6800/338052715 --> 10.8.128.36:6808/21995 -- osd_op(unknown.0.38:97 2.6 2:6f2c300e:::60f.00000000:head [omap-get-header,omap -get-vals,getxattr parent] snapc 0=[] ondisk+read+known_if_redirected+full_force e42) v8 -- 0x1c934d3040 con 0 -6> 2017-09-07 13:27:37.952466 7f79c58c6700 5 -- 10.8.128.105:6800/338052715 >> 10.8.128.36:6808/21995 conn(0x1c91f75000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=189 cs=1 l= 1). rx osd.8 seq 29 0x1c91f14680 osd_op_reply(83 501.00000001 [read 458397~3735907 [fadvise_dontneed]] v0'0 uv1651 ondisk = 0) v8 -5> 2017-09-07 13:27:37.952506 7f79c58c6700 1 -- 10.8.128.105:6800/338052715 <== osd.8 10.8.128.36:6808/21995 29 ==== osd_op_reply(83 501.00000001 [read 458397~3735907 [fadvise_dontneed] ] v0'0 uv1651 ondisk = 0) v8 ==== 156+0+3735907 (1828211159 0 2860789470) 0x1c91f14680 con 0x1c91f75000 -4> 2017-09-07 13:27:37.952625 7f79c58c6700 5 -- 10.8.128.105:6800/338052715 >> 10.8.128.36:6808/21995 conn(0x1c91f75000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=189 cs=1 l= 1). rx osd.8 seq 30 0x1c91f14680 osd_op_reply(84 1000000c0da.02c00000 [omap-get-header,omap-get-vals] v0'0 uv1757 ondisk = 0) v8 -3> 2017-09-07 13:27:37.952646 7f79c58c6700 1 -- 10.8.128.105:6800/338052715 <== osd.8 10.8.128.36:6808/21995 30 ==== osd_op_reply(84 1000000c0da.02c00000 [omap-get-header,omap-get-vals] v0'0 uv1757 ondisk = 0) v8 ==== 206+0+24480 (2245149694 0 2706660154) 0x1c91f14680 con 0x1c91f75000 -2> 2017-09-07 13:27:37.952724 7f79bd919700 1 -- 10.8.128.105:6800/338052715 --> 10.8.128.103:6800/20641 -- osd_op(unknown.0.38:98 1.2 1:55af870c:::1000000d735.00000000:head [delete] sna pc 1=[] ondisk+write+known_if_redirected+full_force e42) v8 -- 0x1c934c0700 con 0 -1> 2017-09-07 13:27:37.953525 7f79c58c6700 2 -- 10.8.128.105:6800/338052715 >> 10.8.128.103:6800/20641 conn(0x1c91fb5000 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_con nection got newly_acked_seq 0 vs out_seq 0 0> 2017-09-07 13:27:37.955163 7f79be11a700 -1 *** Caught signal (Segmentation fault) ** in thread 7f79be11a700 thread_name:mds_rank_progr </pre> and backtrace: <pre> ceph version 12.2.0-2redhat1xenial (b661348f156f148d764b998b65b90451f096cb27) luminous (rc) 1: (()+0x59b344) [0x1c86d06344] 2: (()+0x11390) [0x7f79c8420390] 3: (Server::handle_client_readdir(boost::intrusive_ptr<MDRequestImpl>&)+0xc05) [0x1c86a65615] 4: (Server::dispatch_client_request(boost::intrusive_ptr<MDRequestImpl>&)+0x9f1) [0x1c86a89341] 5: (MDSInternalContextBase::complete(int)+0x18b) [0x1c86c894bb] 6: (MDSRank::_advance_queues()+0x6a7) [0x1c86a0e957] 7: (MDSRank::ProgressThread::entry()+0x4a) [0x1c86a0ef7a] 8: (()+0x76ba) [0x7f79c84166ba] 9: (clone()+0x6d) [0x7f79c74823dd] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. </pre> Steps to Reproduce: 1. Created a cluster with 3 mds 2. Initially 1 mds was active and 2 were standby 3. moved another mds to be active 4. so effectively 2 mds with 1 standby 5. created directories and files on fuse mount 6. one of the directory had 100k files of 0 size 7. I was just probing the dirfrag ls from admin socket 8. Initiated rm -rf on the directory 9. Was continuously probing "objecter_requests" from admin socket of one mds 10. The other mds was crashed meanwhile. 11. Though standby has taken over client I/Os are hung BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1489461