Ceph » rgw

see: http://pulpito.front.sepia.ceph.com/yuvalif-2022-09-13_13:09:52-rgw:multisite-wip-yuval-fix-57518-distro-default-smithi/

backtrace for one RGW is:

ceph version 17.0.0-14925-g2dd12b3f (2dd12b3fa5440a4bc98a44fa2009e41893368c64) quincy (dev)
 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f76d5eaf420]
 2: /lib/x86_64-linux-gnu/libc.so.6(+0x184df0) [0x7f76d5b21df0]
 3: (std::_Rb_tree<rgw_obj_key, std::pair<rgw_obj_key const, RGWSyncShardMarkerTrack<rgw_obj_key, rgw_obj_key>::marker_entry>, std::_Select1st<std::pair<rgw_obj_key const, RGWSyncShardMarkerTrack<rgw_obj_key, rgw_obj_key>::marker_entry> >, std::less<rgw_obj_key>, std::allocator<std::pair<rgw_obj_key const, RGWSyncShardMarkerTrack<rgw_obj_key, rgw_obj_key>::marker_entry> > >::find(rgw_obj_key const&)+0x6f) [0x56264146925f]
 4: (RGWBucketSyncSingleEntryCR<rgw_obj_key, rgw_obj_key>::operate(DoutPrefixProvider const*)+0x8bc) [0x56264149e10c]
 5: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x17c) [0x562641582cac]
 6: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list<RGWCoroutinesStack*, std::allocator<RGWCoroutinesStack*> >&)+0x355) [0x562641584255]
 7: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0x91) [0x562641585541]
 8: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x1fa) [0x562641442f4a]
 9: (RGWDataSyncProcessorThread::process(DoutPrefixProvider const*)+0x58) [0x56264166fb98]
 10: (RGWRadosThread::Worker::entry()+0xbd) [0x56264163cb5d]
 11: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f76d5ea3609]
 12: clone()

and for the other is:

ceph version 17.0.0-14925-g2dd12b3f (2dd12b3fa5440a4bc98a44fa2009e41893368c64) quincy (dev)
 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f277ad0e420]
 2: (tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned int, int)+0x93) [0x7f277af0df83]
 3: (tcmalloc::ThreadCache::Scavenge()+0x4e) [0x7f277af0e2de]
 4: radosgw(+0x97ab3e) [0x55d47d5bcb3e]
 5: (RGWBucketSyncSingleEntryCR<rgw_obj_key, rgw_obj_key>::~RGWBucketSyncSingleEntryCR()+0x56) [0x55d47d5e81f6]
 6: (ceph::common::RefCountedObject::put() const+0x121) [0x7f277b9ce7b1]
 7: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x339) [0x55d47d710e69]
 8: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list<RGWCoroutinesStack*, std::allocator<RGWCoroutinesStack*> >&)+0x355) [0x55d47d712255]
 9: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0x91) [0x55d47d713541]
 10: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x1fa) [0x55d47d5d0f4a]
 11: (RGWDataSyncProcessorThread::process(DoutPrefixProvider const*)+0x58) [0x55d47d7fdb98]
 12: (RGWRadosThread::Worker::entry()+0xbd) [0x55d47d7cab5d]
 13: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f277ad02609]
 14: clone()