==> radosgw.8000.log <== -95> 2019-01-23 15:50:55.535 7f0687477dc0 1 lockdep using id 508 -94> 2019-01-23 15:50:55.535 7f0687477dc0 1 lockdep using id 509 -93> 2019-01-23 15:50:55.535 7f0687477dc0 1 lockdep using id 510 -92> 2019-01-23 15:50:55.535 7f0687477dc0 1 lockdep using id 511 -91> 2019-01-23 15:50:55.535 7f0687477dc0 1 lockdep using id 512 -90> 2019-01-23 15:50:55.535 7f0687477dc0 1 lockdep using id 513 -89> 2019-01-23 15:50:55.535 7f0687477dc0 1 lockdep using id 514 -88> 2019-01-23 15:50:55.535 7f0687477dc0 1 lockdep using id 515 -87> 2019-01-23 15:50:55.535 7f0687477dc0 1 lockdep using id 516 -86> 2019-01-23 15:50:55.535 7f0687477dc0 1 lockdep using id 517 -85> 2019-01-23 15:50:55.535 7f0687477dc0 1 lockdep using id 518 -84> 2019-01-23 15:50:55.535 7f0687477dc0 1 lockdep using id 519 -83> 2019-01-23 15:50:55.535 7f0687477dc0 1 lockdep using id 520 -82> 2019-01-23 15:50:55.535 7f0687477dc0 1 lockdep using id 521 -81> 2019-01-23 15:50:55.535 7f0687477dc0 1 lockdep using id 522 -80> 2019-01-23 15:50:55.535 7f0687477dc0 1 lockdep using id 523 -79> 2019-01-23 15:50:55.535 7f0687477dc0 1 lockdep using id 524 -78> 2019-01-23 15:50:55.535 7f0687477dc0 1 lockdep using id 525 -77> 2019-01-23 15:50:55.535 7f0687477dc0 1 lockdep using id 526 -76> 2019-01-23 15:50:55.535 7f0687477dc0 1 lockdep using id 527 -75> 2019-01-23 15:50:55.535 7f0687477dc0 1 lockdep using id 528 -74> 2019-01-23 15:50:55.535 7f0687477dc0 1 lockdep using id 529 -73> 2019-01-23 15:50:55.535 7f0681e42700 1 -- v2:localhost:57842/1692306889 <== osd.0 v2:localhost:6802/21386 3 ==== osd_op_reply(4 gc.20 [call] v0'0 uv1 ondisk = 0) v8 ==== 149+0+15 (3282212732 0 682959608) 0x5643235a0400 con 0x5643243c6480 -72> 2019-01-23 15:50:55.535 7f0687477dc0 1 lockdep using id 530 -71> 2019-01-23 15:50:55.535 7f0687477dc0 1 lockdep using id 531 -70> 2019-01-23 15:50:55.535 7f0687477dc0 1 lockdep using id 532 -69> 2019-01-23 15:50:55.535 7f0687477dc0 1 lockdep using id 533 -68> 2019-01-23 15:50:55.535 7f0687477dc0 1 lockdep using id 534 -67> 2019-01-23 15:50:55.535 7f0687477dc0 1 lockdep using id 535 -66> 2019-01-23 15:50:55.535 7f0687477dc0 1 lockdep using id 536 -65> 2019-01-23 15:50:55.535 7f0687477dc0 1 lockdep using id 537 -64> 2019-01-23 15:50:55.535 7f0687477dc0 1 lockdep using id 538 -63> 2019-01-23 15:50:55.535 7f0687477dc0 1 lockdep using id 539 -62> 2019-01-23 15:50:55.535 7f0687477dc0 1 lockdep using id 540 -61> 2019-01-23 15:50:55.535 7f0687477dc0 1 lockdep using id 541 -60> 2019-01-23 15:50:55.535 7f0687477dc0 1 lockdep using id 542 -59> 2019-01-23 15:50:55.535 7f0687477dc0 1 lockdep using id 543 -58> 2019-01-23 15:50:55.539 7f0687477dc0 1 lockdep using id 544 -57> 2019-01-23 15:50:55.539 7f0687477dc0 1 lockdep using id 545 -56> 2019-01-23 15:50:55.539 7f0687477dc0 1 lockdep using id 546 -55> 2019-01-23 15:50:55.539 7f0687477dc0 1 lockdep using id 547 -54> 2019-01-23 15:50:55.539 7f0687477dc0 1 lockdep using id 548 -53> 2019-01-23 15:50:55.539 7f0687477dc0 1 lockdep using id 549 -52> 2019-01-23 15:50:55.539 7f0687477dc0 1 lockdep using id 550 -51> 2019-01-23 15:50:55.539 7f0687477dc0 1 lockdep using id 551 -50> 2019-01-23 15:50:55.539 7f0687477dc0 1 lockdep using id 552 -49> 2019-01-23 15:50:55.539 7f0687477dc0 1 lockdep using id 553 -48> 2019-01-23 15:50:55.539 7f0687477dc0 1 lockdep using id 554 -47> 2019-01-23 15:50:55.539 7f0687477dc0 1 lockdep using id 555 -46> 2019-01-23 15:50:55.539 7f0687477dc0 1 lockdep using id 556 -45> 2019-01-23 15:50:55.539 7f0687477dc0 1 lockdep using id 557 -44> 2019-01-23 15:50:55.539 7f0687477dc0 1 lockdep using id 558 -43> 2019-01-23 15:50:55.539 7f0687477dc0 1 lockdep using id 559 -42> 2019-01-23 15:50:55.539 7f0687477dc0 1 lockdep using id 560 -41> 2019-01-23 15:50:55.539 7f0687477dc0 1 lockdep using id 561 -40> 2019-01-23 15:50:55.539 7f0687477dc0 1 lockdep using id 562 -39> 2019-01-23 15:50:55.539 7f0687477dc0 1 lockdep using id 563 -38> 2019-01-23 15:50:55.539 7f0687477dc0 1 lockdep using id 564 -37> 2019-01-23 15:50:55.539 7f0687477dc0 1 lockdep using id 565 -36> 2019-01-23 15:50:55.539 7f0687477dc0 1 lockdep using id 566 -35> 2019-01-23 15:50:55.539 7f0687477dc0 1 lockdep using id 567 -34> 2019-01-23 15:50:55.539 7f0681e42700 1 -- v2:localhost:57842/1692306889 <== osd.0 v2:localhost:6802/21386 4 ==== osd_op_reply(5 obj_delete_at_hint.0000000000 [call] v0'0 uv1 ondisk = 0) v8 ==== 173+0+15 (1944739484 0 2149983739) 0x5643235a0400 con 0x5643243c6480 -33> 2019-01-23 15:50:55.539 7f0687477dc0 1 lockdep using id 568 -32> 2019-01-23 15:50:55.539 7f0687477dc0 1 lockdep using id 569 -31> 2019-01-23 15:50:55.539 7f0687477dc0 1 lockdep using id 570 -30> 2019-01-23 15:50:55.539 7f0687477dc0 1 lockdep using id 571 -29> 2019-01-23 15:50:55.539 7f0681e42700 1 -- v2:localhost:57842/1692306889 <== osd.0 v2:localhost:6802/21386 5 ==== osd_op_reply(3 reshard.0000000000 [call] v23'2 uv2 ondisk = 0) v8 ==== 162+0+0 (1519196165 0 0) 0x5643235a0400 con 0x5643243c6480 -28> 2019-01-23 15:50:55.539 7f0687477dc0 1 lockdep using id 572 -27> 2019-01-23 15:50:55.539 7f0678e30700 1 -- v2:localhost:57842/1692306889 --> [v2:localhost:6802/21386,v1:localhost:6803/21386] -- osd_op(unknown.0.0:6 6.7 6:e622f76a:gc::gc.20:head [call lock.unlock] snapc 0=[] ondisk+write+known_if_redirected e23) v8 -- 0x5643235a0d00 con 0x5643243c6480 -26> 2019-01-23 15:50:55.539 7f0687477dc0 1 lockdep using id 573 -25> 2019-01-23 15:50:55.539 7f0687477dc0 1 lockdep using id 574 -24> 2019-01-23 15:50:55.539 7f0687477dc0 1 lockdep using id 575 -23> 2019-01-23 15:50:55.539 7f0687477dc0 1 lockdep using id 576 -22> 2019-01-23 15:50:55.539 7f0687477dc0 1 lockdep using id 577 -21> 2019-01-23 15:50:55.539 7f0687477dc0 1 lockdep using id 578 -20> 2019-01-23 15:50:55.539 7f0687477dc0 1 lockdep using id 579 -19> 2019-01-23 15:50:55.543 7f0687477dc0 1 lockdep reusing last freed id 403 -18> 2019-01-23 15:50:55.543 7f0687477dc0 1 lockdep using id 580 -17> 2019-01-23 15:50:55.543 7f067862f700 1 -- v2:localhost:57842/1692306889 --> [v2:localhost:6802/21386,v1:localhost:6803/21386] -- osd_op(unknown.0.0:7 6.0 6:1a734c59:::obj_delete_at_hint.0000000000:head [call lock.unlock] snapc 0=[] ondisk+write+known_if_redirected e23) v8 -- 0x56432359ec00 con 0x5643243c6480 -16> 2019-01-23 15:50:55.543 7f0662e04700 1 lockdep using id 581 -15> 2019-01-23 15:50:55.543 7f0662e04700 1 -- v2:localhost:57842/1692306889 --> [v2:localhost:6802/21386,v1:localhost:6803/21386] -- osd_op(unknown.0.0:8 6.7 6:e9187ab8:reshard::reshard.0000000000:head [call rgw.reshard_list] snapc 0=[] ondisk+read+known_if_redirected e23) v8 -- 0x5643235a1000 con 0x5643243c6480 -14> 2019-01-23 15:50:55.543 7f0687477dc0 20 RGW hostnames: -13> 2019-01-23 15:50:55.543 7f0687477dc0 20 RGW S3website hostnames: -12> 2019-01-23 15:50:55.543 7f0687477dc0 1 lockdep using id 582 -11> 2019-01-23 15:50:55.543 7f0687477dc0 1 lockdep using id 583 -10> 2019-01-23 15:50:55.551 7f0681e42700 1 -- v2:localhost:57842/1692306889 <== osd.0 v2:localhost:6802/21386 6 ==== osd_op_reply(8 reshard.0000000000 [call] v0'0 uv2 ondisk = 0) v8 ==== 162+0+11 (2888616447 0 1993775135) 0x5643235a1000 con 0x5643243c6480 -9> 2019-01-23 15:50:55.551 7f0662e04700 1 -- v2:localhost:57842/1692306889 --> [v2:localhost:6802/21386,v1:localhost:6803/21386] -- osd_op(unknown.0.0:9 6.7 6:e9187ab8:reshard::reshard.0000000000:head [call lock.unlock] snapc 0=[] ondisk+write+known_if_redirected e23) v8 -- 0x5643235a1300 con 0x5643243c6480 -8> 2019-01-23 15:50:55.559 7f0681e42700 1 -- v2:localhost:57842/1692306889 <== osd.0 v2:localhost:6802/21386 7 ==== osd_op_reply(7 obj_delete_at_hint.0000000000 [call] v23'2 uv2 ondisk = 0) v8 ==== 173+0+0 (3395733901 0 0) 0x5643235a1300 con 0x5643243c6480 -7> 2019-01-23 15:50:55.559 7f067862f700 20 proceeding shard = obj_delete_at_hint.0000000001 -6> 2019-01-23 15:50:55.559 7f067862f700 1 -- v2:localhost:57842/1692306889 --> [v2:localhost:6802/21386,v1:localhost:6803/21386] -- osd_op(unknown.0.0:10 6.7 6:f1983ba0:::obj_delete_at_hint.0000000001:head [call lock.lock] snapc 0=[] ondisk+write+known_if_redirected e23) v8 -- 0x56432439dc00 con 0x5643243c6480 -5> 2019-01-23 15:50:55.559 7f0681e42700 1 -- v2:localhost:57842/1692306889 <== osd.0 v2:localhost:6802/21386 8 ==== osd_op_reply(6 gc.20 [call] v23'3 uv3 ondisk = 0) v8 ==== 149+0+0 (981529768 0 0) 0x56432439dc00 con 0x5643243c6480 -4> 2019-01-23 15:50:55.559 7f0678e30700 1 -- v2:localhost:57842/1692306889 --> [v2:localhost:6810/21787,v1:localhost:6811/21787] -- osd_op(unknown.0.0:11 6.2 6:5127e642:gc::gc.21:head [call lock.lock] snapc 0=[] ondisk+write+known_if_redirected e23) v8 -- 0x5643235a0100 con 0x5643243c6900 -3> 2019-01-23 15:50:55.563 7f0681e42700 1 -- v2:localhost:57842/1692306889 <== osd.0 v2:localhost:6802/21386 9 ==== osd_op_reply(9 reshard.0000000000 [call] v23'4 uv4 ondisk = 0) v8 ==== 162+0+0 (3716503844 0 0) 0x56432439dc00 con 0x5643243c6480 -2> 2019-01-23 15:50:55.563 7f0662e04700 20 proceeding logshard = reshard.0000000001 -1> 2019-01-23 15:50:55.563 7f0662e04700 1 -- v2:localhost:57842/1692306889 --> [v2:localhost:6810/21787,v1:localhost:6811/21787] -- osd_op(unknown.0.0:12 6.4 6:2c0c0c23:reshard::reshard.0000000001:head [call lock.lock] snapc 0=[] ondisk+write+known_if_redirected e23) v8 -- 0x564324371800 con 0x5643243c6900 0> 2019-01-23 15:50:55.567 7f0687477dc0 -1 *** Caught signal (Aborted) ** in thread 7f0687477dc0 thread_name:radosgw ceph version 14.0.1-2864-gba556a7722 (ba556a77222f8c4f604693ea1ccc8db1bad57af7) nautilus (dev) 1: (()+0xd83660) [0x564321193660] 2: (()+0x12080) [0x7f068a842080] 3: (gsignal()+0x10b) [0x7f06891cc9eb] 4: (abort()+0x151) [0x7f06891cdda1] 5: (()+0x2f10a) [0x7f06891c510a] 6: (()+0x2f182) [0x7f06891c5182] 7: (()+0x1e5c4) [0x7f068a3dd5c4] 8: (ldap_sasl_bind_s()+0x8a) [0x7f068a3dd97a] 9: (ldap_simple_bind_s()+0x60) [0x7f068a3de200] 10: (rgw::LDAPHelper::bind()+0x4c) [0x5643215d9db0] 11: (rgw::auth::s3::LDAPEngine::init(CephContext*)+0x209) [0x5643215d1471] 12: (rgw::auth::s3::LDAPEngine::LDAPEngine(CephContext*, RGWRados*, rgw::auth::s3::AWSEngine::VersionAbstractor const&, rgw::auth::RemoteApplier::Factory const*)+0x66) [0x564321048ffc] 13: (rgw::auth::s3::ExternalAuthStrategy::ExternalAuthStrategy(CephContext*, RGWRados*, rgw::auth::s3::AWSEngine::VersionAbstractor*)+0xa6) [0x56432104bf14] 14: (rgw::auth::s3::AWSAuthStrategy::AWSAuthStrategy(CephContext*, RGWRados*)+0xf8) [0x564321053edc] 15: (rgw::auth::StrategyRegistry::s3_main_strategy_t::s3_main_strategy_t(CephContext*, RGWRados*)+0x4a) [0x56432104c0fe] 16: (rgw::auth::StrategyRegistry::StrategyRegistry(CephContext*, RGWRados*)+0x2c) [0x56432104c1c8] 17: (void __gnu_cxx::new_allocator::construct(rgw::auth::StrategyRegistry*, CephContext* const&, RGWRados* const&)+0x5f) [0x5643210782ab] 18: (void std::allocator_traits >::construct(std::allocator&, rgw::auth::StrategyRegistry*, CephContext* const&, RGWRados* const&)+0x4a) [0x56432107622e] 19: (std::_Sp_counted_ptr_inplace, (__gnu_cxx::_Lock_policy)2>::_Sp_counted_ptr_inplace(std::allocator, CephContext* const&, RGWRados* const&)+0xb9) [0x564321073915] 20: (std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count, CephContext* const&, RGWRados* const&>(std::_Sp_make_shared_tag, rgw::auth::StrategyRegistry*, std::allocator const&, CephContext* const&, RGWRados* const&)+0xe0) [0x56432106f498] 21: (std::__shared_ptr::__shared_ptr, CephContext* const&, RGWRados* const&>(std::_Sp_make_shared_tag, std::allocator const&, CephContext* const&, RGWRados* const&)+0x66) [0x56432106a868] 22: (std::shared_ptr::shared_ptr, CephContext* const&, RGWRados* const&>(std::_Sp_make_shared_tag, std::allocator const&, CephContext* const&, RGWRados* const&)+0x4f) [0x564321064427] 23: (std::shared_ptr std::allocate_shared, CephContext* const&, RGWRados* const&>(std::allocator const&, CephContext* const&, RGWRados* const&)+0x4a) [0x56432105cb13] 24: (std::shared_ptr std::make_shared(CephContext* const&, RGWRados* const&)+0x66) [0x564321055529] 25: (rgw::auth::StrategyRegistry::create(CephContext*, RGWRados*)+0x3a) [0x56432104c2b3] 26: (main()+0x2081) [0x56432103d823] 27: (__libc_start_main()+0xe7) [0x7f06891b7a87] 28: (_start()+0x2a) [0x56432103a65a] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 1/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 1/ 1 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 1 reserver 1/ 5 heartbeatmap 1/ 5 perfcounter 20/20 rgw 1/ 5 rgw_sync 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 4/ 5 memdb 1/ 5 kinetic 1/ 5 fuse 1/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /ceph/build/out/radosgw.8000.log --- end dump of recent events --- 2019-01-23 15:50:55.579 7f0682643700 1 -- v2:localhost:57842/1692306889 <== osd.1 v2:localhost:6810/21787 1 ==== osd_op_reply(11 gc.21 [call] v23'1 uv1 ondisk = 0) v8 ==== 149+0+0 (235892003 0 0) 0x564324371800 con 0x5643243c6900 2019-01-23 15:50:55.579 7f0678e30700 1 -- v2:localhost:57842/1692306889 --> [v2:localhost:6810/21787,v1:localhost:6811/21787] -- osd_op(unknown.0.0:13 6.2 6:5127e642:gc::gc.21:head [call rgw.gc_list] snapc 0=[] ondisk+read+known_if_redirected e23) v8 -- 0x5643235a0400 con 0x5643243c6900 2019-01-23 15:50:55.579 7f0681e42700 1 -- v2:localhost:57842/1692306889 <== osd.0 v2:localhost:6802/21386 10 ==== osd_op_reply(10 obj_delete_at_hint.0000000001 [call] v23'5 uv5 ondisk = 0) v8 ==== 173+0+0 (357236170 0 0) 0x56432439d600 con 0x5643243c6480 2019-01-23 15:50:55.579 7f067862f700 1 -- v2:localhost:57842/1692306889 --> [v2:localhost:6802/21386,v1:localhost:6803/21386] -- osd_op(unknown.0.0:14 6.7 6:f1983ba0:::obj_delete_at_hint.0000000001:head [call timeindex.list] snapc 0=[] ondisk+read+known_if_redirected e23) v8 -- 0x5643241ca300 con 0x5643243c6480 ==> radosgw.8001.log <== -97> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 508 -96> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 509 -95> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 510 -94> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 511 -93> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 512 -92> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 513 -91> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 514 -90> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 515 -89> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 516 -88> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 517 -87> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 518 -86> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 519 -85> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 520 -84> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 521 -83> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 522 -82> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 523 -81> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 524 -80> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 525 -79> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 526 -78> 2019-01-23 15:50:55.715 7f513b015700 1 -- v2:localhost:37078/2017474412 --> [v2:localhost:6802/21386,v1:localhost:6803/21386] -- osd_op(unknown.0.0:9 6.0 6:1a734c59:::obj_delete_at_hint.0000000000:head [call lock.unlock] snapc 0=[] ondisk+write+known_if_redirected e23) v8 -- 0x560b9bb32100 con 0x560b9bac6480 -77> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 527 -76> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 528 -75> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 529 -74> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 530 -73> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 531 -72> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 532 -71> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 533 -70> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 534 -69> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 535 -68> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 536 -67> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 537 -66> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 538 -65> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 539 -64> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 540 -63> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 541 -62> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 542 -61> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 543 -60> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 544 -59> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 545 -58> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 546 -57> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 547 -56> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 548 -55> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 549 -54> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 550 -53> 2019-01-23 15:50:55.715 7f514582a700 1 -- v2:localhost:37078/2017474412 <== osd.0 v2:localhost:6802/21386 7 ==== osd_op_reply(5 gc.28 [call] v23'2 uv2 ondisk = 0) v8 ==== 149+0+0 (947611243 0 0) 0x560b9bb32100 con 0x560b9bac6480 -52> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 551 -51> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 552 -50> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 553 -49> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 554 -48> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 555 -47> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 556 -46> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 557 -45> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 558 -44> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 559 -43> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 560 -42> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 561 -41> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 562 -40> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 563 -39> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 564 -38> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 565 -37> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 566 -36> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 567 -35> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 568 -34> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 569 -33> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 570 -32> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 571 -31> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 572 -30> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 573 -29> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 574 -28> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 575 -27> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 576 -26> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 577 -25> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 578 -24> 2019-01-23 15:50:55.715 7f5149e5ddc0 1 lockdep using id 579 -23> 2019-01-23 15:50:55.719 7f5149e5ddc0 1 lockdep using id 580 -22> 2019-01-23 15:50:55.719 7f513b816700 1 -- v2:localhost:37078/2017474412 --> [v2:localhost:6802/21386,v1:localhost:6803/21386] -- osd_op(unknown.0.0:10 6.7 6:e47c9322:gc::gc.29:head [call lock.lock] snapc 0=[] ondisk+write+known_if_redirected e23) v8 -- 0x560b9bb31500 con 0x560b9bac6480 -21> 2019-01-23 15:50:55.719 7f5149e5ddc0 20 RGW hostnames: -20> 2019-01-23 15:50:55.719 7f5149e5ddc0 20 RGW S3website hostnames: -19> 2019-01-23 15:50:55.719 7f5149e5ddc0 1 lockdep using id 581 -18> 2019-01-23 15:50:55.719 7f5149e5ddc0 1 lockdep using id 582 -17> 2019-01-23 15:50:55.719 7f514582a700 1 -- v2:localhost:37078/2017474412 <== osd.0 v2:localhost:6802/21386 8 ==== osd_op_reply(8 reshard.0000000000 [call] v23'8 uv8 ondisk = 0) v8 ==== 162+0+0 (2690286902 0 0) 0x560b9bb31500 con 0x560b9bac6480 -16> 2019-01-23 15:50:55.719 7f51257ea700 20 proceeding logshard = reshard.0000000001 -15> 2019-01-23 15:50:55.719 7f51257ea700 1 -- v2:localhost:37078/2017474412 --> [v2:localhost:6810/21787,v1:localhost:6811/21787] -- osd_op(unknown.0.0:11 6.4 6:2c0c0c23:reshard::reshard.0000000001:head [call lock.lock] snapc 0=[] ondisk+write+known_if_redirected e23) v8 -- 0x560b9acaea00 con 0x560b9bac6900 -14> 2019-01-23 15:50:55.727 7f514582a700 1 -- v2:localhost:37078/2017474412 <== osd.0 v2:localhost:6802/21386 9 ==== osd_op_reply(9 obj_delete_at_hint.0000000000 [call] v23'4 uv4 ondisk = 0) v8 ==== 173+0+0 (4102629252 0 0) 0x560b9bb31500 con 0x560b9bac6480 -13> 2019-01-23 15:50:55.727 7f513b015700 20 proceeding shard = obj_delete_at_hint.0000000001 -12> 2019-01-23 15:50:55.727 7f513b015700 1 -- v2:localhost:37078/2017474412 --> [v2:localhost:6802/21386,v1:localhost:6803/21386] -- osd_op(unknown.0.0:12 6.7 6:f1983ba0:::obj_delete_at_hint.0000000001:head [call lock.lock] snapc 0=[] ondisk+write+known_if_redirected e23) v8 -- 0x560b9bad1300 con 0x560b9bac6480 -11> 2019-01-23 15:50:55.731 7f514582a700 1 -- v2:localhost:37078/2017474412 <== osd.0 v2:localhost:6802/21386 10 ==== osd_op_reply(10 gc.29 [call] v23'9 uv9 ondisk = 0) v8 ==== 149+0+0 (2911071720 0 0) 0x560b9bad1300 con 0x560b9bac6480 -10> 2019-01-23 15:50:55.735 7f513b816700 1 lockdep using id 583 -9> 2019-01-23 15:50:55.735 7f513b816700 1 -- v2:localhost:37078/2017474412 --> [v2:localhost:6802/21386,v1:localhost:6803/21386] -- osd_op(unknown.0.0:13 6.7 6:e47c9322:gc::gc.29:head [call rgw.gc_list] snapc 0=[] ondisk+read+known_if_redirected e23) v8 -- 0x560b9bb30000 con 0x560b9bac6480 -8> 2019-01-23 15:50:55.735 7f514582a700 1 -- v2:localhost:37078/2017474412 <== osd.0 v2:localhost:6802/21386 11 ==== osd_op_reply(13 gc.29 [call] v0'0 uv9 ondisk = 0) v8 ==== 149+0+15 (3922409707 0 682959608) 0x560b9bb30000 con 0x560b9bac6480 -7> 2019-01-23 15:50:55.739 7f514582a700 1 -- v2:localhost:37078/2017474412 <== osd.0 v2:localhost:6802/21386 12 ==== osd_op_reply(12 obj_delete_at_hint.0000000001 [call] v23'10 uv10 ondisk = 0) v8 ==== 173+0+0 (4111789476 0 0) 0x560b9bb30000 con 0x560b9bac6480 -6> 2019-01-23 15:50:55.739 7f513b816700 1 -- v2:localhost:37078/2017474412 --> [v2:localhost:6802/21386,v1:localhost:6803/21386] -- osd_op(unknown.0.0:14 6.7 6:e47c9322:gc::gc.29:head [call lock.unlock] snapc 0=[] ondisk+write+known_if_redirected e23) v8 -- 0x560b9bad1600 con 0x560b9bac6480 -5> 2019-01-23 15:50:55.739 7f513b015700 1 lockdep reusing last freed id 583 -4> 2019-01-23 15:50:55.739 7f513b015700 1 -- v2:localhost:37078/2017474412 --> [v2:localhost:6802/21386,v1:localhost:6803/21386] -- osd_op(unknown.0.0:15 6.7 6:f1983ba0:::obj_delete_at_hint.0000000001:head [call timeindex.list] snapc 0=[] ondisk+read+known_if_redirected e23) v8 -- 0x560b9acadb00 con 0x560b9bac6480 -3> 2019-01-23 15:50:55.739 7f5145029700 1 -- v2:localhost:37078/2017474412 <== osd.1 v2:localhost:6810/21787 1 ==== osd_op_reply(11 reshard.0000000001 [call] v23'2 uv1 ondisk = -16 ((16) Device or resource busy)) v8 ==== 162+0+0 (3081877967 0 0) 0x560b9acaea00 con 0x560b9bac6900 -2> 2019-01-23 15:50:55.739 7f51257ea700 0 RGWReshardLock::lock failed to acquire lock on reshard.0000000001 ret=-16 -1> 2019-01-23 15:50:55.739 7f51257ea700 5 process_single_logshard(): failed to acquire lock on reshard.0000000001 0> 2019-01-23 15:50:55.739 7f5149e5ddc0 -1 *** Caught signal (Aborted) ** in thread 7f5149e5ddc0 thread_name:radosgw ceph version 14.0.1-2864-gba556a7722 (ba556a77222f8c4f604693ea1ccc8db1bad57af7) nautilus (dev) 1: (()+0xd83660) [0x560b97bb7660] 2: (()+0x12080) [0x7f514d228080] 3: (gsignal()+0x10b) [0x7f514bbb29eb] 4: (abort()+0x151) [0x7f514bbb3da1] 5: (()+0x2f10a) [0x7f514bbab10a] 6: (()+0x2f182) [0x7f514bbab182] 7: (()+0x1e5c4) [0x7f514cdc35c4] 8: (ldap_sasl_bind_s()+0x8a) [0x7f514cdc397a] 9: (ldap_simple_bind_s()+0x60) [0x7f514cdc4200] 10: (rgw::LDAPHelper::bind()+0x4c) [0x560b97ffddb0] 11: (rgw::auth::s3::LDAPEngine::init(CephContext*)+0x209) [0x560b97ff5471] 12: (rgw::auth::s3::LDAPEngine::LDAPEngine(CephContext*, RGWRados*, rgw::auth::s3::AWSEngine::VersionAbstractor const&, rgw::auth::RemoteApplier::Factory const*)+0x66) [0x560b97a6cffc] 13: (rgw::auth::s3::ExternalAuthStrategy::ExternalAuthStrategy(CephContext*, RGWRados*, rgw::auth::s3::AWSEngine::VersionAbstractor*)+0xa6) [0x560b97a6ff14] 14: (rgw::auth::s3::AWSAuthStrategy::AWSAuthStrategy(CephContext*, RGWRados*)+0xf8) [0x560b97a77edc] 15: (rgw::auth::StrategyRegistry::s3_main_strategy_t::s3_main_strategy_t(CephContext*, RGWRados*)+0x4a) [0x560b97a700fe] 16: (rgw::auth::StrategyRegistry::StrategyRegistry(CephContext*, RGWRados*)+0x2c) [0x560b97a701c8] 17: (void __gnu_cxx::new_allocator::construct(rgw::auth::StrategyRegistry*, CephContext* const&, RGWRados* const&)+0x5f) [0x560b97a9c2ab] 18: (void std::allocator_traits >::construct(std::allocator&, rgw::auth::StrategyRegistry*, CephContext* const&, RGWRados* const&)+0x4a) [0x560b97a9a22e] 19: (std::_Sp_counted_ptr_inplace, (__gnu_cxx::_Lock_policy)2>::_Sp_counted_ptr_inplace(std::allocator, CephContext* const&, RGWRados* const&)+0xb9) [0x560b97a97915] 20: (std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count, CephContext* const&, RGWRados* const&>(std::_Sp_make_shared_tag, rgw::auth::StrategyRegistry*, std::allocator const&, CephContext* const&, RGWRados* const&)+0xe0) [0x560b97a93498] 21: (std::__shared_ptr::__shared_ptr, CephContext* const&, RGWRados* const&>(std::_Sp_make_shared_tag, std::allocator const&, CephContext* const&, RGWRados* const&)+0x66) [0x560b97a8e868] 22: (std::shared_ptr::shared_ptr, CephContext* const&, RGWRados* const&>(std::_Sp_make_shared_tag, std::allocator const&, CephContext* const&, RGWRados* const&)+0x4f) [0x560b97a88427] 23: (std::shared_ptr std::allocate_shared, CephContext* const&, RGWRados* const&>(std::allocator const&, CephContext* const&, RGWRados* const&)+0x4a) [0x560b97a80b13] 24: (std::shared_ptr std::make_shared(CephContext* const&, RGWRados* const&)+0x66) [0x560b97a79529] 25: (rgw::auth::StrategyRegistry::create(CephContext*, RGWRados*)+0x3a) [0x560b97a702b3] 26: (main()+0x2081) [0x560b97a61823] 27: (__libc_start_main()+0xe7) [0x7f514bb9da87] 28: (_start()+0x2a) [0x560b97a5e65a] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 1/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 1/ 1 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 1 reserver 1/ 5 heartbeatmap 1/ 5 perfcounter 20/20 rgw 1/ 5 rgw_sync 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 4/ 5 memdb 1/ 5 kinetic 1/ 5 fuse 1/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /ceph/build/out/radosgw.8001.log --- end dump of recent events --- 2019-01-23 15:50:55.743 7f514582a700 1 -- v2:localhost:37078/2017474412 <== osd.0 v2:localhost:6802/21386 13 ==== osd_op_reply(15 obj_delete_at_hint.0000000001 [call] v0'0 uv10 ondisk = 0) v8 ==== 173+0+15 (918723173 0 2149983739) 0x560b9acadb00 con 0x560b9bac6480 2019-01-23 15:50:55.743 7f513b015700 1 -- v2:localhost:37078/2017474412 --> [v2:localhost:6802/21386,v1:localhost:6803/21386] -- osd_op(unknown.0.0:16 6.7 6:f1983ba0:::obj_delete_at_hint.0000000001:head [call lock.unlock] snapc 0=[] ondisk+write+known_if_redirected e23) v8 -- 0x560b9bb30f00 con 0x560b9bac6480