2022-09-20 10:19:58 new-croit-host-E4BD74 radosgw -1633> 2022-09-20T14:18:07.093+0000 7fbbe6fb5d40 1 rgw main: int RGWSI_Notify::robust_notify(const DoutPrefixProvider*, RGWSI_RADOS::Obj&, const RGWCacheNotifyInfo&, optional_yield):402 Notify failed on object sync.rgw.log:meta.history: (110) Connection timed out 2022-09-20 10:19:58 new-croit-host-E4BD74 radosgw -1632> 2022-09-20T14:18:07.093+0000 7fbbe6fb5d40 1 rgw main: int RGWSI_Notify::robust_notify(const DoutPrefixProvider*, RGWSI_RADOS::Obj&, const RGWCacheNotifyInfo&, optional_yield):418 Invalidating obj=sync.rgw.log:meta.history tries=0 2022-09-20 10:21:58 new-croit-host-E4BD74 radosgw -1910> 2022-09-20T14:19:58.475+0000 7f5c6bfe1d40 2 auth: KeyRing::load: loaded key file /var/lib/ceph/radosgw/ceph-rgw.new-croit-host-E4BD74/keyring 2022-09-20 10:21:58 new-croit-host-E4BD74 radosgw -1909> 2022-09-20T14:19:58.475+0000 7f5c6bfe1d40 2 auth: KeyRing::load: loaded key file /var/lib/ceph/radosgw/ceph-rgw.new-croit-host-E4BD74/keyring 2022-09-20 10:20:27 new-croit-host-6E1EC6 radosgw 2022-09-20T14:20:27.229+0000 7ff18ef35700 1 ====== starting new request req=0x7ff29063b610 ===== 2022-09-20 10:20:27 new-croit-host-6E1EC6 radosgw 2022-09-20T14:20:27.229+0000 7ff18ef35700 1 ====== req done req=0x7ff29063b610 op status=0 http_status=200 latency=0.000000000s ====== 2022-09-20 10:20:27 new-croit-host-6E1EC6 radosgw 2022-09-20T14:20:27.229+0000 7ff18ef35700 1 beast: 0x7ff29063b610: 172.31.124.6 - syncuser [20/Sep/2022:14:20:27.229 +0000] "GET /admin/log?type=data&rgwx-zonegroup=e740002a-fd74-4834-8b60-c634c935bdc5 HTTP/1.1" 200 19 - - - latency=0.000000000s 2022-09-20 10:21:58 new-croit-host-E4BD74 radosgw 2022-09-20T14:21:58.516+0000 7f5bb9ffb700 0 cb147abbaebccd7e: sync_object: b=:mystetbucket[f28cb046-1b0e-4919-8407-e86aa22fc629.33057.1]) k=synctest versioned_epoch=0 2022-09-20 10:21:58 new-croit-host-E4BD74 radosgw 2022-09-20T14:21:58.520+0000 7f5bb9ffb700 0 AWS: creating bucket rgw-default-cb147abbaebccd7e 2022-09-20 10:21:58 new-croit-host-E4BD74 radosgw terminate called after throwing an instance of 'std::bad_alloc' 2022-09-20 10:21:58 new-croit-host-E4BD74 radosgw what(): std::bad_alloc 2022-09-20 10:21:58 new-croit-host-E4BD74 radosgw *** Caught signal (Aborted) ** 2022-09-20 10:21:58 new-croit-host-E4BD74 radosgw in thread 7f5bb9ffb700 thread_name:data-sync 2022-09-20 10:21:58 new-croit-host-E4BD74 radosgw ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable) 2022-09-20 10:21:58 new-croit-host-E4BD74 radosgw 1: /lib64/libpthread.so.0(+0x168c0) [0x7f5c6ac1b8c0] 2022-09-20 10:21:58 new-croit-host-E4BD74 radosgw 2: gsignal() 2022-09-20 10:21:58 new-croit-host-E4BD74 radosgw 3: abort() 2022-09-20 10:21:58 new-croit-host-E4BD74 radosgw 4: /usr/lib64/libstdc++.so.6(+0xa4fc6) [0x7f5c6a896fc6] 2022-09-20 10:21:58 new-croit-host-E4BD74 radosgw 5: /usr/lib64/libstdc++.so.6(+0xb083c) [0x7f5c6a8a283c] 2022-09-20 10:21:58 new-croit-host-E4BD74 radosgw 6: /usr/lib64/libstdc++.so.6(+0xb08a7) [0x7f5c6a8a28a7] 2022-09-20 10:21:58 new-croit-host-E4BD74 radosgw 7: /usr/lib64/libstdc++.so.6(+0xb0b09) [0x7f5c6a8a2b09] 2022-09-20 10:21:58 new-croit-host-E4BD74 radosgw 8: /usr/lib64/libstdc++.so.6(+0xa4bd1) [0x7f5c6a896bd1] 2022-09-20 10:21:58 new-croit-host-E4BD74 radosgw 9: /usr/lib64/libradosgw.so.2(+0x889412) [0x7f5c6b6b1412] 2022-09-20 10:21:58 new-croit-host-E4BD74 radosgw 10: (RGWRESTConn::get_obj(DoutPrefixProvider const*, rgw::sal::RGWObject const*, RGWRESTConn::get_obj_params const&, bool, RGWRESTStreamRWRequest**)+0x567) [0x7f5c6b6b5b57] 11: (RGWRESTStreamGetCRF::init(DoutPrefixProvider const*)+0xbd) [0x7f5c6b53a7ad] 12: (RGWStreamSpliceCR::operate(DoutPrefixProvider const*)+0x55) [0x7f5c6b5c6235] 13: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x9a) [0x7f5c6b5b19fa] 14: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list >&)+0x2ba) [0x7f5c6b5b28aa] 15: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0x90) [0x7f5c6b5b38a0] 16: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x19e) [0x7f5c6b4cc00e] 17: (RGWDataSyncProcessorThread::process(DoutPrefixProvider const*)+0x59) [0x7f5c6b689569] 18: (RGWRadosThread::Worker::entry()+0x182) [0x7f5c6b648322] 19: /lib64/libpthread.so.0(+0xa6ea) [0x7f5c6ac0f6ea] 20: clone() NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2022-09-20 10:21:58 new-croit-host-E4BD74 radosgw 2022-09-20T14:21:58.536+0000 7f5bb9ffb700 -1 *** Caught signal (Aborted) ** 2022-09-20 10:21:58 new-croit-host-E4BD74 radosgw in thread 7f5bb9ffb700 thread_name:data-sync 2022-09-20 10:21:58 new-croit-host-E4BD74 radosgw ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable) 2022-09-20 10:21:58 new-croit-host-E4BD74 radosgw 1: /lib64/libpthread.so.0(+0x168c0) [0x7f5c6ac1b8c0] 2022-09-20 10:21:58 new-croit-host-E4BD74 radosgw 2: gsignal() 2022-09-20 10:21:58 new-croit-host-E4BD74 radosgw 3: abort()