Ceph » RADOS

Deadlock in Luminous:

Thread 6 (Thread 0x7f236f0b9700 (LWP 21238)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007f2381d3ecf4 in Wait (mutex=..., this=0x7f2383bc8e70) at /home/dzafman/ceph/src/common/Cond.h:48
#2 DBObjectMap::MapHeaderLock::MapHeaderLock (this=0x7f236f0b5ae0, db=0x7f2383bc8dc0, oid=...) at /home/dzafman/ceph/src/os/filestore/DBObjectMap.h:90
Python Exception <class 'ValueError'> Cannot find type const std::set<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::_Rep_type:
#3 0x00007f2381d3b51e in DBObjectMap::rm_keys (this=0x7f2383bc8dc0, oid=..., to_clear=std::set with 19 elements, spos=0x7f236f0b5e00) at /home/dzafman/ceph/src/os/filestore/DBObjectMap.cc:650
Python Exception <class 'ValueError'> Cannot find type const std::set<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::_Rep_type:
#4 0x00007f2381bba6b9 in FileStore::_omap_rmkeys (this=this@entry=0x7f2383bdc000, cid=..., hoid=..., keys=std::set with 19 elements, spos=...) at /home/dzafman/ceph/src/os/filestore/FileStore.cc:5561
#5 0x00007f2381be43b3 in FileStore::_do_transaction (this=this@entry=0x7f2383bdc000, t=..., op_seq=op_seq@entry=57, trans_num=trans_num@entry=0, handle=handle@entry=0x7f236f0b6f70) at /home/dzafman/ceph/src/os/filestore/FileStore.cc:3010
#6 0x00007f2381be7edb in FileStore::_do_transactions (this=this@entry=0x7f2383bdc000, tls=std::vector of length 1, capacity 1 = {...}, op_seq=57, handle=handle@entry=0x7f236f0b6f70) at /home/dzafman/ceph/src/os/filestore/FileStore.cc:2339
#7 0x00007f2381be82fb in FileStore::_do_op (this=0x7f2383bdc000, osr=0x7f23839951e0, handle=...) at /home/dzafman/ceph/src/os/filestore/FileStore.cc:2082
#8 0x00007f2377dbdb4e in ThreadPool::worker (this=0x7f2383bdcaf0, wt=0x7f2383bf86a0) at /home/dzafman/ceph/src/common/WorkQueue.cc:120
#9 0x00007f2377dbea70 in ThreadPool::WorkThread::entry (this=<optimized out>) at /home/dzafman/ceph/src/common/WorkQueue.h:448
#10 0x00007f23775a2182 in start_thread (arg=0x7f236f0b9700) at pthread_create.c:312
#11 0x00007f23768a747d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 1 (Thread 0x7f2382366400 (LWP 21207)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007f2381b916e9 in Wait (mutex=..., this=0x7fff1f488350) at /home/dzafman/ceph/src/common/Cond.h:48
#2 ObjectStore::apply_transactions (this=0x7f2383bdc000, osr=0x7f2383b79600, tls=std::vector of length 0, capacity 0, ondisk=<optimized out>) at /home/dzafman/ceph/src/os/ObjectStore.cc:183
#3 0x00007f238183b97c in ObjectStore::apply_transaction(ObjectStore::Sequencer*, ObjectStore::Transaction&&, Context*) (this=0x7f2383bdc000, osr=0x7f2383b79600, t=<optimized out>, ondisk=0x0) at /home/dzafman/ceph/src/os/ObjectStore.h:1469
#4 0x00007f2381819d33 in do_trim_pg_log (store=0x7f2383bdc000, coll=..., info=..., pgid=..., osr=..., map_epoch=12, past_intervals=...) at /home/dzafman/ceph/src/tools/ceph_objectstore_tool.cc:567
#5 0x00007f238177c50b in main (argc=<optimized out>, argv=<optimized out>) at /home/dzafman/ceph/src/tools/ceph_objectstore_tool.cc:4024

Switching backport to use queue_transactions() at least for the rmkeys portion.