Bug #58231
Updated by Casey Bodley over 1 year ago
per-object concurrency was recently added to RGWDeleteMultiObj in https://github.com/ceph/ceph/pull/48679
each object spawns a new yield context, but the notification calls are using the wrong one from s->yield
ex http://qa-proxy.ceph.com/teuthology/cbodley-2022-12-08_14:16:01-rgw-wip-cbodley-testing-distro-default-smithi/7107993/teuthology.log
<pre>
2022-12-09T07:51:10.487 INFO:tasks.rgw.client.0.smithi005.stdout:radosgw: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-1287-ga6ead589/rpm/el8/BUILD/ceph-18.0.0-1287-ga6ead589/x86_64-redhat-linux-gnu/boost/include/boost/context/continuation_fcontext.hpp:260: boost::context::continuation boost::context::continuation::resume() &&: Assertion `nullptr != fctx_' failed.
2022-12-09T07:51:10.487 INFO:tasks.rgw.client.0.smithi005.stdout:*** Caught signal (Aborted) **
2022-12-09T07:51:10.487 INFO:tasks.rgw.client.0.smithi005.stdout: in thread 7f7ea59e1700 thread_name:radosgw
2022-12-09T07:51:10.490 INFO:tasks.rgw.client.0.smithi005.stdout: ceph version 18.0.0-1287-ga6ead589 (a6ead5895323e8028165d6c6614e9b12172f4f59) reef (dev)
2022-12-09T07:51:10.490 INFO:tasks.rgw.client.0.smithi005.stdout: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f7eddfb9cf0]
2022-12-09T07:51:10.490 INFO:tasks.rgw.client.0.smithi005.stdout: 2: gsignal()
2022-12-09T07:51:10.490 INFO:tasks.rgw.client.0.smithi005.stdout: 3: abort()
2022-12-09T07:51:10.490 INFO:tasks.rgw.client.0.smithi005.stdout: 4: /lib64/libc.so.6(+0x21d79) [0x7f7edcd31d79]
2022-12-09T07:51:10.490 INFO:tasks.rgw.client.0.smithi005.stdout: 5: /lib64/libc.so.6(+0x47456) [0x7f7edcd57456]
2022-12-09T07:51:10.491 INFO:tasks.rgw.client.0.smithi005.stdout: 6: (auto rgw_http_req_data::async_wait<boost::asio::io_context, spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul> > > > >(boost::asio::io_context&, spawn::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul> > > >&&)+0x5ee) [0x55ebec5e728e]
2022-12-09T07:51:10.491 INFO:tasks.rgw.client.0.smithi005.stdout: 7: (rgw_http_req_data::wait(optional_yield)+0xd2) [0x55ebec5e7582]
2022-12-09T07:51:10.491 INFO:tasks.rgw.client.0.smithi005.stdout: 8: (RGWPubSubHTTPEndpoint::send_to_completion_async(ceph::common::CephContext*, rgw_pubsub_s3_event const&, optional_yield)+0x67b) [0x55ebeccc4fbb]
2022-12-09T07:51:10.491 INFO:tasks.rgw.client.0.smithi005.stdout: 9: (rgw::notify::publish_commit(rgw::sal::Object*, unsigned long, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rgw::notify::EventType, rgw::notify::reservation_t&, DoutPrefixProvider const*)+0xd28) [0x55ebecbfdb68]
2022-12-09T07:51:10.491 INFO:tasks.rgw.client.0.smithi005.stdout: 10: (rgw::sal::RadosNotification::publish_commit(DoutPrefixProvider const*, unsigned long, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x2f) [0x55ebec8513af]
2022-12-09T07:51:10.491 INFO:tasks.rgw.client.0.smithi005.stdout: 11: (RGWDeleteMultiObj::handle_individual_object(rgw_obj_key const&, optional_yield, boost::asio::basic_deadline_timer<boost::posix_time::ptime, boost::asio::time_traits<boost::posix_time::ptime>, boost::asio::executor>*)+0x81f) [0x55ebec64b87f]
2022-12-09T07:51:10.491 INFO:tasks.rgw.client.0.smithi005.stdout: 12: radosgw(+0x8e4410) [0x55ebec64c410]
2022-12-09T07:51:10.491 INFO:tasks.rgw.client.0.smithi005.stdout: 13: make_fcontext()
</pre>