Project

General

Profile

Bug #40018 » java_s3_tests_crash_log_1.txt

Ali Maredia, 05/23/2019 09:47 PM

 
http://qa-proxy.ceph.com/teuthology/amaredia-2019-05-16_13:43:07-rgw:verify-master-distro-basic-smithi/3958985/teuthology.log

2019-05-16T16:32:11.837 INFO:teuthology.orchestra.run.smithi165.stdout:suite > AWS4 tests > AWS4Test.testUploadFileHLAPIBigFileAWS4 STARTED
2019-05-16T16:33:43.315 INFO:tasks.ceph.mon.b.smithi008.stderr:==00:00:29:23.899 9786== Warning: unimplemented fcntl command: 1036
2019-05-16T16:33:43.356 INFO:tasks.ceph.mon.c.smithi165.stderr:==00:00:29:23.992 9951== Warning: unimplemented fcntl command: 1036
2019-05-16T16:33:43.359 INFO:tasks.ceph.mon.c.smithi165.stderr:==00:00:29:23.996 9951== Warning: unimplemented fcntl command: 1036
2019-05-16T16:33:43.377 INFO:tasks.ceph.mon.a.smithi165.stderr:==00:00:29:24.014 9924== Warning: unimplemented fcntl command: 1036
2019-05-16T16:33:43.378 INFO:tasks.ceph.mon.b.smithi008.stderr:==00:00:29:23.962 9786== Warning: unimplemented fcntl command: 1036
2019-05-16T16:33:43.379 INFO:tasks.ceph.mon.a.smithi165.stderr:==00:00:29:24.016 9924== Warning: unimplemented fcntl command: 1036
2019-05-16T16:33:43.553 INFO:tasks.ceph.mon.b.smithi008.stderr:==00:00:29:24.137 9786== Warning: unimplemented fcntl command: 1036
2019-05-16T16:33:43.561 INFO:tasks.ceph.mon.c.smithi165.stderr:==00:00:29:24.198 9951== Warning: unimplemented fcntl command: 1036
2019-05-16T16:33:43.610 INFO:tasks.ceph.mon.a.smithi165.stderr:==00:00:29:24.247 9924== Warning: unimplemented fcntl command: 1036
2019-05-16T16:34:52.414 INFO:tasks.rgw.client.0.smithi165.stdout:2019-05-16 16:34:52.240 20fc5700 -1 RGWWatcher::handle_error cookie 555117344 err (107) Transport endpoint is not connected
2019-05-16T16:34:52.501 INFO:tasks.rgw.client.0.smithi165.stdout:2019-05-16 16:34:52.405 20fc5700 -1 RGWWatcher::handle_error cookie 555072880 err (107) Transport endpoint is not connected
2019-05-16T16:35:03.281 INFO:tasks.rgw.client.0.smithi165.stdout:2019-05-16 16:35:03.278 20fc5700 -1 RGWWatcher::handle_error cookie 555175968 err (107) Transport endpoint is not connected
2019-05-16T16:35:03.282 INFO:tasks.rgw.client.0.smithi165.stdout:2019-05-16 16:35:03.278 20fc5700 -1 RGWWatcher::handle_error cookie 555396160 err (107) Transport endpoint is not connected
2019-05-16T16:35:03.336 INFO:tasks.rgw.client.0.smithi165.stdout:2019-05-16 16:35:03.279 20fc5700 -1 RGWWatcher::handle_error cookie 555453872 err (107) Transport endpoint is not connected
2019-05-16T16:35:03.338 INFO:tasks.rgw.client.0.smithi165.stdout:2019-05-16 16:35:03.279 20fc5700 -1 RGWWatcher::handle_error cookie 555627776 err (107) Transport endpoint is not connected
2019-05-16T16:35:03.609 INFO:tasks.rgw.client.0.smithi165.stdout:2019-05-16 16:35:03.491 20fc5700 -1 RGWWatcher::handle_error cookie 555350912 err (107) Transport endpoint is not connected
2019-05-16T16:35:20.853 INFO:tasks.rgw.client.0.smithi165.stdout:2019-05-16 16:35:17.709 20fc5700 -1 RGWWatcher::handle_error cookie 555679712 err (107) Transport endpoint is not connected
2019-05-16T16:36:22.752 INFO:tasks.rgw.client.0.smithi165.stdout:2019-05-16 16:36:22.744 20fc5700 -1 RGWWatcher::handle_error cookie 5380047904 err (107) Transport endpoint is not connected
2019-05-16T16:36:22.790 INFO:tasks.rgw.client.0.smithi165.stdout:2019-05-16 16:36:22.787 20fc5700 -1 RGWWatcher::handle_error cookie 553540656 err (107) Transport endpoint is not connected
2019-05-16T16:38:05.711 INFO:tasks.rgw.client.0.smithi165.stdout:2019-05-16 16:38:05.652 20fc5700 -1 RGWWatcher::handle_error cookie 5381925216 err (107) Transport endpoint is not connected
2019-05-16T16:38:05.714 INFO:tasks.rgw.client.0.smithi165.stdout:2019-05-16 16:38:05.704 20fc5700 -1 RGWWatcher::handle_error cookie 357374576 err (107) Transport endpoint is not connected
2019-05-16T16:38:05.793 INFO:tasks.rgw.client.0.smithi165.stdout:2019-05-16 16:38:05.704 20fc5700 -1 RGWWatcher::handle_error cookie 5387014992 err (107) Transport endpoint is not connected
2019-05-16T16:38:05.800 INFO:tasks.rgw.client.0.smithi165.stdout:2019-05-16 16:38:05.787 20fc5700 -1 RGWWatcher::handle_error cookie 5383348144 err (107) Transport endpoint is not connected
2019-05-16T16:38:07.147 INFO:tasks.rgw.client.0.smithi165.stdout:2019-05-16 16:38:05.974 20fc5700 -1 RGWWatcher::handle_error cookie 358086592 err (107) Transport endpoint is not connected
2019-05-16T16:38:07.590 INFO:tasks.rgw.client.0.smithi165.stdout:2019-05-16 16:38:06.841 20fc5700 -1 RGWWatcher::handle_error cookie 378493168 err (107) Transport endpoint is not connected
2019-05-16T16:38:07.953 INFO:tasks.rgw.client.0.smithi165.stdout:2019-05-16 16:38:06.842 20fc5700 -1 RGWWatcher::handle_error cookie 555621312 err (107) Transport endpoint is not connected
2019-05-16T16:38:36.007 INFO:tasks.rgw.client.0.smithi165.stdout:terminate called after throwing an instance of 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::system::system_error> >'
2019-05-16T16:38:36.025 INFO:tasks.rgw.client.0.smithi165.stdout: what(): remote_endpoint: Transport endpoint is not connected
2019-05-16T16:38:36.142 INFO:tasks.rgw.client.0.smithi165.stdout:*** Caught signal (Aborted) **
2019-05-16T16:38:36.142 INFO:tasks.rgw.client.0.smithi165.stdout: in thread 12d7e6700 thread_name:memcheck-amd64-
2019-05-16T16:38:39.350 INFO:tasks.rgw.client.0.smithi165.stdout: ceph version 15.0.0-1290-g2fc991b (2fc991b6b4f4a97a6c904ff51383eab2cdc8cf1f) octopus (dev)
2019-05-16T16:38:39.350 INFO:tasks.rgw.client.0.smithi165.stdout: 1: (()+0xf5d0) [0x113d25d0]
2019-05-16T16:38:39.350 INFO:tasks.rgw.client.0.smithi165.stdout: 2: (gsignal()+0x37) [0x11e34207]
2019-05-16T16:38:39.350 INFO:tasks.rgw.client.0.smithi165.stdout: 3: (abort()+0x148) [0x11e358f8]
2019-05-16T16:38:39.350 INFO:tasks.rgw.client.0.smithi165.stdout: 4: (__gnu_cxx::__verbose_terminate_handler()+0x165) [0x1163f7d5]
2019-05-16T16:38:39.351 INFO:tasks.rgw.client.0.smithi165.stdout: 5: (()+0x5e746) [0x1163d746]
2019-05-16T16:38:39.351 INFO:tasks.rgw.client.0.smithi165.stdout: 6: (()+0x5e773) [0x1163d773]
2019-05-16T16:38:39.351 INFO:tasks.rgw.client.0.smithi165.stdout: 7: (()+0x5e993) [0x1163d993]
2019-05-16T16:38:39.351 INFO:tasks.rgw.client.0.smithi165.stdout: 8: (virtual thunk to boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::system::system_error> >::rethrow() const+0x6e) [0x405ade]
2019-05-16T16:38:39.351 INFO:tasks.rgw.client.0.smithi165.stdout: 9: (boost::rethrow_exception(boost::exception_ptr const&)+0xd) [0x3fba0d]
2019-05-16T16:38:39.351 INFO:tasks.rgw.client.0.smithi165.stdout: 10: (boost::coroutines::detail::push_coroutine_impl<void>::push()+0x4e) [0x3fbafe]
2019-05-16T16:38:39.351 INFO:tasks.rgw.client.0.smithi165.stdout: 11: (boost::asio::detail::executor_op<ceph::async::ForwardingHandler<ceph::async::CompletionHandler<boost::asio::detail::coro_handler<boost::asio::executor_binder<void (*)(), boost::asio::executor>, std::shared_lock<ceph::async::SharedMutex<boost::asio::io_context::executor_type> > >, std::tuple<boost::system::error_code, std::shared_lock<ceph::async::SharedMutex<boost::asio::io_context::executor_type> > > > >, std::allocator<void>, boost::asio::detail::scheduler_operation>::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long)+0x16d) [0x40685d]
2019-05-16T16:38:39.352 INFO:tasks.rgw.client.0.smithi165.stdout: 12: (boost::asio::detail::executor_op<boost::asio::executor::function, std::allocator<void>, boost::asio::detail::scheduler_operation>::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long)+0x8c) [0x3f879c]
2019-05-16T16:38:39.352 INFO:tasks.rgw.client.0.smithi165.stdout: 13: (boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::executor_type const>::operator()()+0x7d) [0x41e5ad]
2019-05-16T16:38:39.352 INFO:tasks.rgw.client.0.smithi165.stdout: 14: (boost::asio::detail::executor_op<boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::executor_type const>, boost::asio::detail::recycling_allocator<void>, boost::asio::detail::scheduler_operation>::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long)+0x94) [0x41e924]
2019-05-16T16:38:39.352 INFO:tasks.rgw.client.0.smithi165.stdout: 15: (boost::asio::detail::scheduler::run(boost::system::error_code&)+0x3f2) [0x3fd3c2]
2019-05-16T16:38:39.352 INFO:tasks.rgw.client.0.smithi165.stdout: 16: (()+0x2de2ab) [0x3e62ab]
2019-05-16T16:38:39.352 INFO:tasks.rgw.client.0.smithi165.stdout: 17: (()+0x7b8f8f) [0x642ff8f]
2019-05-16T16:38:39.352 INFO:tasks.rgw.client.0.smithi165.stdout: 18: (()+0x7dd5) [0x113cadd5]
2019-05-16T16:38:39.353 INFO:tasks.rgw.client.0.smithi165.stdout: 19: (clone()+0x6d) [0x11efbead]
2019-05-16T16:38:39.399 INFO:tasks.rgw.client.0.smithi165.stdout:2019-05-16 16:38:39.387 12d7e6700 -1 *** Caught signal (Aborted) **
2019-05-16T16:38:39.399 INFO:tasks.rgw.client.0.smithi165.stdout: in thread 12d7e6700 thread_name:memcheck-amd64-
2019-05-16T16:38:39.399 INFO:tasks.rgw.client.0.smithi165.stdout:
2019-05-16T16:38:39.399 INFO:tasks.rgw.client.0.smithi165.stdout: ceph version 15.0.0-1290-g2fc991b (2fc991b6b4f4a97a6c904ff51383eab2cdc8cf1f) octopus (dev)
2019-05-16T16:38:39.399 INFO:tasks.rgw.client.0.smithi165.stdout: 1: (()+0xf5d0) [0x113d25d0]
2019-05-16T16:38:39.399 INFO:tasks.rgw.client.0.smithi165.stdout: 2: (gsignal()+0x37) [0x11e34207]
2019-05-16T16:38:39.400 INFO:tasks.rgw.client.0.smithi165.stdout: 3: (abort()+0x148) [0x11e358f8]
2019-05-16T16:38:39.400 INFO:tasks.rgw.client.0.smithi165.stdout: 4: (__gnu_cxx::__verbose_terminate_handler()+0x165) [0x1163f7d5]
2019-05-16T16:38:39.400 INFO:tasks.rgw.client.0.smithi165.stdout: 5: (()+0x5e746) [0x1163d746]
2019-05-16T16:38:39.400 INFO:tasks.rgw.client.0.smithi165.stdout: 6: (()+0x5e773) [0x1163d773]
2019-05-16T16:38:39.400 INFO:tasks.rgw.client.0.smithi165.stdout: 7: (()+0x5e993) [0x1163d993]
2019-05-16T16:38:39.400 INFO:tasks.rgw.client.0.smithi165.stdout: 8: (virtual thunk to boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::system::system_error> >::rethrow() const+0x6e) [0x405ade]
2019-05-16T16:38:39.400 INFO:tasks.rgw.client.0.smithi165.stdout: 9: (boost::rethrow_exception(boost::exception_ptr const&)+0xd) [0x3fba0d]
2019-05-16T16:38:39.401 INFO:tasks.rgw.client.0.smithi165.stdout: 10: (boost::coroutines::detail::push_coroutine_impl<void>::push()+0x4e) [0x3fbafe]
2019-05-16T16:38:39.401 INFO:tasks.rgw.client.0.smithi165.stdout: 11: (boost::asio::detail::executor_op<ceph::async::ForwardingHandler<ceph::async::CompletionHandler<boost::asio::detail::coro_handler<boost::asio::executor_binder<void (*)(), boost::asio::executor>, std::shared_lock<ceph::async::SharedMutex<boost::asio::io_context::executor_type> > >, std::tuple<boost::system::error_code, std::shared_lock<ceph::async::SharedMutex<boost::asio::io_context::executor_type> > > > >, std::allocator<void>, boost::asio::detail::scheduler_operation>::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long)+0x16d) [0x40685d]
2019-05-16T16:38:39.401 INFO:tasks.rgw.client.0.smithi165.stdout: 12: (boost::asio::detail::executor_op<boost::asio::executor::function, std::allocator<void>, boost::asio::detail::scheduler_operation>::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long)+0x8c) [0x3f879c]
2019-05-16T16:38:39.401 INFO:tasks.rgw.client.0.smithi165.stdout: 13: (boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::executor_type const>::operator()()+0x7d) [0x41e5ad]
2019-05-16T16:38:39.401 INFO:tasks.rgw.client.0.smithi165.stdout: 14: (boost::asio::detail::executor_op<boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::executor_type const>, boost::asio::detail::recycling_allocator<void>, boost::asio::detail::scheduler_operation>::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long)+0x94) [0x41e924]
2019-05-16T16:38:39.401 INFO:tasks.rgw.client.0.smithi165.stdout: 15: (boost::asio::detail::scheduler::run(boost::system::error_code&)+0x3f2) [0x3fd3c2]
2019-05-16T16:38:39.402 INFO:tasks.rgw.client.0.smithi165.stdout: 16: (()+0x2de2ab) [0x3e62ab]
2019-05-16T16:38:39.402 INFO:tasks.rgw.client.0.smithi165.stdout: 17: (()+0x7b8f8f) [0x642ff8f]
2019-05-16T16:38:39.402 INFO:tasks.rgw.client.0.smithi165.stdout: 18: (()+0x7dd5) [0x113cadd5]
2019-05-16T16:38:39.402 INFO:tasks.rgw.client.0.smithi165.stdout: 19: (clone()+0x6d) [0x11efbead]
2019-05-16T16:38:39.402 INFO:tasks.rgw.client.0.smithi165.stdout: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2019-05-16T16:38:39.403 INFO:tasks.rgw.client.0.smithi165.stdout:
2019-05-16T16:38:39.986 INFO:tasks.rgw.client.0.smithi165.stdout: -3576> 2019-05-16 16:34:52.240 20fc5700 -1 RGWWatcher::handle_error cookie 555117344 err (107) Transport endpoint is not connected
2019-05-16T16:38:39.987 INFO:tasks.rgw.client.0.smithi165.stdout: -3573> 2019-05-16 16:34:52.405 20fc5700 -1 RGWWatcher::handle_error cookie 555072880 err (107) Transport endpoint is not connected
2019-05-16T16:38:40.035 INFO:tasks.rgw.client.0.smithi165.stdout: -3547> 2019-05-16 16:35:03.278 20fc5700 -1 RGWWatcher::handle_error cookie 555175968 err (107) Transport endpoint is not connected
2019-05-16T16:38:40.074 INFO:tasks.rgw.client.0.smithi165.stdout: -3545> 2019-05-16 16:35:03.278 20fc5700 -1 RGWWatcher::handle_error cookie 555396160 err (107) Transport endpoint is not connected
2019-05-16T16:38:40.076 INFO:tasks.rgw.client.0.smithi165.stdout: -3543> 2019-05-16 16:35:03.279 20fc5700 -1 RGWWatcher::handle_error cookie 555453872 err (107) Transport endpoint is not connected
2019-05-16T16:38:40.076 INFO:tasks.rgw.client.0.smithi165.stdout: -3541> 2019-05-16 16:35:03.279 20fc5700 -1 RGWWatcher::handle_error cookie 555627776 err (107) Transport endpoint is not connected
2019-05-16T16:38:40.085 INFO:tasks.rgw.client.0.smithi165.stdout: -3523> 2019-05-16 16:35:03.491 20fc5700 -1 RGWWatcher::handle_error cookie 555350912 err (107) Transport endpoint is not connected
2019-05-16T16:38:40.087 INFO:tasks.rgw.client.0.smithi165.stdout: -3502> 2019-05-16 16:35:17.709 20fc5700 -1 RGWWatcher::handle_error cookie 555679712 err (107) Transport endpoint is not connected
2019-05-16T16:38:40.232 INFO:tasks.rgw.client.0.smithi165.stdout: -1835> 2019-05-16 16:36:22.744 20fc5700 -1 RGWWatcher::handle_error cookie 5380047904 err (107) Transport endpoint is not connected
2019-05-16T16:38:40.236 INFO:tasks.rgw.client.0.smithi165.stdout: -1831> 2019-05-16 16:36:22.787 20fc5700 -1 RGWWatcher::handle_error cookie 553540656 err (107) Transport endpoint is not connected
2019-05-16T16:38:40.237 INFO:tasks.rgw.client.0.smithi165.stdout:terminate called recursively
2019-05-16T16:38:55.569 INFO:tasks.rgw.client.0.smithi165.stderr:daemon-helper: command crashed with signal 6
2019-05-16T16:39:11.545 INFO:teuthology.orchestra.run.smithi165.stdout:
2019-05-16T16:39:11.545 INFO:teuthology.orchestra.run.smithi165.stdout:suite > AWS4 tests > AWS4Test.testUploadFileHLAPIBigFileAWS4 PASSED
2019-05-16T16:40:01.742 INFO:teuthology.orchestra.run.smithi165.stdout:
2019-05-16T16:40:01.742 INFO:teuthology.orchestra.run.smithi165.stdout:----------------------------------------------------------------------
2019-05-16T16:40:01.742 INFO:teuthology.orchestra.run.smithi165.stdout:| Results: SUCCESS (27 tests, 27 successes, 0 failures, 0 skipped) |
2019-05-16T16:40:01.743 INFO:teuthology.orchestra.run.smithi165.stdout:----------------------------------------------------------------------
2019-05-16T16:40:01.743 INFO:teuthology.orchestra.run.smithi165.stdout:
2019-05-16T16:40:01.743 INFO:teuthology.orchestra.run.smithi165.stdout:BUILD SUCCESSFUL in 23m 18s
2019-05-16T16:40:01.743 INFO:teuthology.orchestra.run.smithi165.stdout:5 actionable tasks: 4 executed, 1 up-to-date
2019-05-16T16:40:02.065 INFO:teuthology.orchestra.run.smithi165:Running:
2019-05-16T16:40:02.065 INFO:teuthology.orchestra.run.smithi165:> radosgw-admin gc process --include-all
2019-05-16T16:40:02.349 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:02.341 7fa922ac56c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:02.378 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:02.371 7fa922ac56c0 20 rados_obj.operate() r=-2 bl.length=0
2019-05-16T16:40:02.379 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:02.371 7fa922ac56c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:02.389 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:02.382 7fa922ac56c0 20 rados_obj.operate() r=-2 bl.length=0
2019-05-16T16:40:02.389 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:02.382 7fa922ac56c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:02.396 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:02.388 7fa922ac56c0 20 rados_obj.operate() r=-2 bl.length=0
2019-05-16T16:40:02.396 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:02.388 7fa922ac56c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:02.405 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:02.398 7fa922ac56c0 20 rados_obj.operate() r=0 bl.length=46
2019-05-16T16:40:02.494 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:02.486 7fa922ac56c0 20 RGWRados::pool_iterate: got zonegroup_info.3e3ec152-13ab-4fe5-bbc5-31d7ce5be4ef
2019-05-16T16:40:02.494 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:02.486 7fa922ac56c0 20 RGWRados::pool_iterate: got zone_info.ec4d9074-b84d-4f70-b1c0-15fa89cb015e
2019-05-16T16:40:02.494 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:02.486 7fa922ac56c0 20 RGWRados::pool_iterate: got zone_names.default
2019-05-16T16:40:02.494 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:02.486 7fa922ac56c0 20 RGWRados::pool_iterate: got zonegroups_names.default
2019-05-16T16:40:02.495 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:02.486 7fa922ac56c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:02.501 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:02.494 7fa922ac56c0 20 rados_obj.operate() r=0 bl.length=46
2019-05-16T16:40:02.501 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:02.494 7fa922ac56c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:02.513 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:02.506 7fa922ac56c0 20 rados_obj.operate() r=0 bl.length=951
2019-05-16T16:40:02.514 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:02.506 7fa922ac56c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:02.521 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:02.513 7fa922ac56c0 20 rados_obj.operate() r=0 bl.length=46
2019-05-16T16:40:02.521 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:02.513 7fa922ac56c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:02.529 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:02.521 7fa922ac56c0 20 rados_obj.operate() r=0 bl.length=382
2019-05-16T16:40:02.529 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:02.521 7fa922ac56c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:02.534 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:02.526 7fa922ac56c0 20 rados_obj.operate() r=-2 bl.length=0
2019-05-16T16:40:02.534 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:02.526 7fa922ac56c0 10 cannot find current period zonegroup using local zonegroup
2019-05-16T16:40:02.534 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:02.526 7fa922ac56c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:02.538 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:02.531 7fa922ac56c0 20 rados_obj.operate() r=-2 bl.length=0
2019-05-16T16:40:02.539 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:02.531 7fa922ac56c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:02.546 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:02.539 7fa922ac56c0 20 rados_obj.operate() r=0 bl.length=46
2019-05-16T16:40:02.546 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:02.539 7fa922ac56c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:02.557 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:02.549 7fa922ac56c0 20 rados_obj.operate() r=0 bl.length=382
2019-05-16T16:40:02.557 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:02.549 7fa922ac56c0 20 zonegroup default
2019-05-16T16:40:02.557 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:02.549 7fa922ac56c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:02.561 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:02.554 7fa922ac56c0 20 rados_obj.operate() r=-2 bl.length=0
2019-05-16T16:40:02.561 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:02.554 7fa922ac56c0 10 Cannot find current period zone using local zone
2019-05-16T16:40:02.562 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:02.554 7fa922ac56c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:02.566 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:02.559 7fa922ac56c0 20 rados_obj.operate() r=-2 bl.length=0
2019-05-16T16:40:02.566 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:02.559 7fa922ac56c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:02.576 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:02.568 7fa922ac56c0 20 rados_obj.operate() r=0 bl.length=46
2019-05-16T16:40:02.576 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:02.568 7fa922ac56c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:02.583 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:02.576 7fa922ac56c0 20 rados_obj.operate() r=0 bl.length=951
2019-05-16T16:40:02.584 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:02.576 7fa922ac56c0 20 zone default
2019-05-16T16:40:03.034 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:03.026 7fa922ac56c0 20 add_watcher() i=0
2019-05-16T16:40:03.034 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:03.027 7fa922ac56c0 20 add_watcher() i=1
2019-05-16T16:40:03.034 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:03.027 7fa922ac56c0 20 add_watcher() i=2
2019-05-16T16:40:03.034 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:03.027 7fa922ac56c0 20 add_watcher() i=3
2019-05-16T16:40:03.035 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:03.027 7fa922ac56c0 20 add_watcher() i=4
2019-05-16T16:40:03.035 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:03.027 7fa922ac56c0 20 add_watcher() i=5
2019-05-16T16:40:03.042 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:03.035 7fa922ac56c0 20 add_watcher() i=6
2019-05-16T16:40:03.078 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:03.071 7fa922ac56c0 20 add_watcher() i=7
2019-05-16T16:40:03.078 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:03.071 7fa922ac56c0 2 all 8 watchers are set, enabling cache
2019-05-16T16:40:03.096 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:03.089 7fa910b4e700 0 -- 172.21.15.165:0/168945780 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x55e8bd505bd0 msgr2=0x55e8bd505fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure
2019-05-16T16:40:03.122 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:03.115 7fa8e7fff700 2 RGWDataChangesLog::ChangesRenewThread: start
2019-05-16T16:40:03.141 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:03.133 7fa8ce7e4700 20 reqs_thread_entry: start
2019-05-16T16:40:03.141 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:03.134 7fa89ffff700 20 reqs_thread_entry: start
2019-05-16T16:40:03.141 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:03.134 7fa922ac56c0 20 init_complete bucket index max shards: 0
2019-05-16T16:40:03.142 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:03.135 7fa89e7fc700 20 reqs_thread_entry: start
2019-05-16T16:40:03.142 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:03.135 7fa922ac56c0 20 garbage collection: RGWGC::process entered with GC index_shard=18, max_secs=3600, expired_only=0
2019-05-16T16:40:03.227 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:03.219 7fa91034d700 0 -- 172.21.15.165:0/4191709180 >> [v2:172.21.15.8:6810/11110,v1:172.21.15.8:6811/11110] conn(0x55e8bd596b30 msgr2=0x55e8bd599050 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure
2019-05-16T16:40:03.227 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:03.219 7fa922ac56c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=3, truncated=0, next_marker=''
2019-05-16T16:40:03.227 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:03.219 7fa922ac56c0 20 garbage collection: RGWGC::process iterating over entry tag='ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4367.560�', time=2019-05-16 18:22:46.0.20858s, chain.objs.size()=9
2019-05-16T16:40:03.227 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:03.219 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.26__multipart_file.mpg.2~UBjAvLgo62J9jvKXyW8Ibuy739IjXR3.1
2019-05-16T16:40:03.227 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:03.219 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.26__shadow_file.mpg.2~UBjAvLgo62J9jvKXyW8Ibuy739IjXR3.1_1
2019-05-16T16:40:03.228 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:03.219 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.26__multipart_file.mpg.2~UBjAvLgo62J9jvKXyW8Ibuy739IjXR3.2
2019-05-16T16:40:03.228 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:03.219 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.26__shadow_file.mpg.2~UBjAvLgo62J9jvKXyW8Ibuy739IjXR3.2_1
2019-05-16T16:40:03.228 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:03.219 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.26__multipart_file.mpg.2~UBjAvLgo62J9jvKXyW8Ibuy739IjXR3.3
2019-05-16T16:40:03.229 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:03.220 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.26__shadow_file.mpg.2~UBjAvLgo62J9jvKXyW8Ibuy739IjXR3.3_1
2019-05-16T16:40:03.229 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:03.220 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.26__multipart_file.mpg.2~UBjAvLgo62J9jvKXyW8Ibuy739IjXR3.4
2019-05-16T16:40:03.229 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:03.220 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.26__shadow_file.mpg.2~UBjAvLgo62J9jvKXyW8Ibuy739IjXR3.4_1
2019-05-16T16:40:03.229 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:03.220 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.26__multipart_file.mpg.2~UBjAvLgo62J9jvKXyW8Ibuy739IjXR3.5
2019-05-16T16:40:03.229 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:03.220 7fa922ac56c0 20 garbage collection: RGWGC::process iterating over entry tag='ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4367.559�', time=2019-05-16 18:22:46.0.7769s, chain.objs.size()=7
2019-05-16T16:40:03.229 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:03.220 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.26__multipart_file2.mpg.2~tpWofrkIoXVpDro8Z3-xGWXnXBhWYnS.1
2019-05-16T16:40:03.230 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:03.221 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.26__shadow_file2.mpg.2~tpWofrkIoXVpDro8Z3-xGWXnXBhWYnS.1_1
2019-05-16T16:40:03.230 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:03.221 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.26__multipart_file2.mpg.2~tpWofrkIoXVpDro8Z3-xGWXnXBhWYnS.2
2019-05-16T16:40:03.408 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:03.400 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.26__shadow_file2.mpg.2~tpWofrkIoXVpDro8Z3-xGWXnXBhWYnS.2_1
2019-05-16T16:40:03.408 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:03.400 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.26__multipart_file2.mpg.2~tpWofrkIoXVpDro8Z3-xGWXnXBhWYnS.3
2019-05-16T16:40:03.408 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:03.400 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.26__shadow_file2.mpg.2~tpWofrkIoXVpDro8Z3-xGWXnXBhWYnS.3_1
2019-05-16T16:40:03.409 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:03.400 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.26__multipart_file2.mpg.2~tpWofrkIoXVpDro8Z3-xGWXnXBhWYnS.4
2019-05-16T16:40:03.409 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:03.400 7fa922ac56c0 20 garbage collection: RGWGC::process iterating over entry tag='ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4367.608�', time=2019-05-16 18:26:19.0.012217s, chain.objs.size()=21
2019-05-16T16:40:03.409 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:03.400 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__multipart_key1.2~bjJUSCQjE6wrPwulGMQ6tcu_5IAEAVK.1
2019-05-16T16:40:03.474 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:03.467 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__shadow_key1.2~bjJUSCQjE6wrPwulGMQ6tcu_5IAEAVK.1_1
2019-05-16T16:40:03.475 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:03.467 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__multipart_key1.2~bjJUSCQjE6wrPwulGMQ6tcu_5IAEAVK.2
2019-05-16T16:40:03.475 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:03.467 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__shadow_key1.2~bjJUSCQjE6wrPwulGMQ6tcu_5IAEAVK.2_1
2019-05-16T16:40:03.475 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:03.467 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__multipart_key1.2~bjJUSCQjE6wrPwulGMQ6tcu_5IAEAVK.3
2019-05-16T16:40:03.475 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:03.467 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__shadow_key1.2~bjJUSCQjE6wrPwulGMQ6tcu_5IAEAVK.3_1
2019-05-16T16:40:03.475 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:03.467 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__multipart_key1.2~bjJUSCQjE6wrPwulGMQ6tcu_5IAEAVK.4
2019-05-16T16:40:03.562 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:03.554 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__shadow_key1.2~bjJUSCQjE6wrPwulGMQ6tcu_5IAEAVK.4_1
2019-05-16T16:40:03.580 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:03.572 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__multipart_key1.2~bjJUSCQjE6wrPwulGMQ6tcu_5IAEAVK.5
2019-05-16T16:40:03.580 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:03.572 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__shadow_key1.2~bjJUSCQjE6wrPwulGMQ6tcu_5IAEAVK.5_1
2019-05-16T16:40:03.580 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:03.573 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__multipart_key1.2~bjJUSCQjE6wrPwulGMQ6tcu_5IAEAVK.6
2019-05-16T16:40:03.580 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:03.573 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__shadow_key1.2~bjJUSCQjE6wrPwulGMQ6tcu_5IAEAVK.6_1
2019-05-16T16:40:03.695 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:03.687 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__multipart_key1.2~bjJUSCQjE6wrPwulGMQ6tcu_5IAEAVK.7
2019-05-16T16:40:03.695 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:03.687 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__shadow_key1.2~bjJUSCQjE6wrPwulGMQ6tcu_5IAEAVK.7_1
2019-05-16T16:40:03.785 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:03.777 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__multipart_key1.2~bjJUSCQjE6wrPwulGMQ6tcu_5IAEAVK.8
2019-05-16T16:40:03.785 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:03.777 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__shadow_key1.2~bjJUSCQjE6wrPwulGMQ6tcu_5IAEAVK.8_1
2019-05-16T16:40:03.785 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:03.777 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__multipart_key1.2~bjJUSCQjE6wrPwulGMQ6tcu_5IAEAVK.9
2019-05-16T16:40:03.785 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:03.777 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__shadow_key1.2~bjJUSCQjE6wrPwulGMQ6tcu_5IAEAVK.9_1
2019-05-16T16:40:03.785 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:03.777 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__multipart_key1.2~bjJUSCQjE6wrPwulGMQ6tcu_5IAEAVK.10
2019-05-16T16:40:03.785 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:03.777 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__shadow_key1.2~bjJUSCQjE6wrPwulGMQ6tcu_5IAEAVK.10_1
2019-05-16T16:40:03.786 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:03.777 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__multipart_key1.2~bjJUSCQjE6wrPwulGMQ6tcu_5IAEAVK.11
2019-05-16T16:40:03.947 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:03.940 7fa922ac56c0 20 garbage collection: RGWGC::process entered with GC index_shard=19, max_secs=3600, expired_only=0
2019-05-16T16:40:03.999 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:03.992 7fa922ac56c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:04.031 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:04.024 7fa922ac56c0 20 garbage collection: RGWGC::process entered with GC index_shard=20, max_secs=3600, expired_only=0
2019-05-16T16:40:04.096 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:04.090 7fa922ac56c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:04.150 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:04.144 7fa922ac56c0 20 garbage collection: RGWGC::process entered with GC index_shard=21, max_secs=3600, expired_only=0
2019-05-16T16:40:04.209 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:04.201 7fa922ac56c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:04.251 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:04.244 7fa922ac56c0 20 garbage collection: RGWGC::process entered with GC index_shard=22, max_secs=3600, expired_only=0
2019-05-16T16:40:04.325 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:04.317 7fa922ac56c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=1, truncated=0, next_marker=''
2019-05-16T16:40:04.325 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:04.317 7fa922ac56c0 20 garbage collection: RGWGC::process iterating over entry tag='ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4367.439�', time=2019-05-16 18:19:46.0.627197s, chain.objs.size()=9
2019-05-16T16:40:04.325 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:04.317 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.20__multipart_key1.2~4xQ8P6ar_KG_eTpp-xDK05ygD-8G0VB.1
2019-05-16T16:40:04.325 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:04.317 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.20__shadow_key1.2~4xQ8P6ar_KG_eTpp-xDK05ygD-8G0VB.1_1
2019-05-16T16:40:04.325 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:04.317 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.20__multipart_key1.2~4xQ8P6ar_KG_eTpp-xDK05ygD-8G0VB.2
2019-05-16T16:40:04.326 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:04.317 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.20__shadow_key1.2~4xQ8P6ar_KG_eTpp-xDK05ygD-8G0VB.2_1
2019-05-16T16:40:04.326 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:04.317 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.20__multipart_key1.2~4xQ8P6ar_KG_eTpp-xDK05ygD-8G0VB.3
2019-05-16T16:40:04.326 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:04.317 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.20__shadow_key1.2~4xQ8P6ar_KG_eTpp-xDK05ygD-8G0VB.3_1
2019-05-16T16:40:04.326 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:04.317 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.20__multipart_key1.2~4xQ8P6ar_KG_eTpp-xDK05ygD-8G0VB.4
2019-05-16T16:40:04.326 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:04.317 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.20__shadow_key1.2~4xQ8P6ar_KG_eTpp-xDK05ygD-8G0VB.4_1
2019-05-16T16:40:04.326 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:04.317 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.20__multipart_key1.2~4xQ8P6ar_KG_eTpp-xDK05ygD-8G0VB.5
2019-05-16T16:40:04.490 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:04.483 7fa922ac56c0 20 garbage collection: RGWGC::process entered with GC index_shard=23, max_secs=3600, expired_only=0
2019-05-16T16:40:04.570 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:04.563 7fa922ac56c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=1, truncated=0, next_marker=''
2019-05-16T16:40:04.571 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:04.563 7fa922ac56c0 20 garbage collection: RGWGC::process iterating over entry tag='ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4367.447�', time=2019-05-16 18:19:57.0.196856s, chain.objs.size()=9
2019-05-16T16:40:04.571 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:04.563 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.20__multipart_key1.2~IsFCu4xqxiYBmSN1-sk6QEDR7kDCvBp.1
2019-05-16T16:40:04.571 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:04.563 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.20__shadow_key1.2~IsFCu4xqxiYBmSN1-sk6QEDR7kDCvBp.1_1
2019-05-16T16:40:04.571 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:04.563 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.20__multipart_key1.2~IsFCu4xqxiYBmSN1-sk6QEDR7kDCvBp.2
2019-05-16T16:40:04.571 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:04.563 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.20__shadow_key1.2~IsFCu4xqxiYBmSN1-sk6QEDR7kDCvBp.2_1
2019-05-16T16:40:04.571 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:04.563 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.20__multipart_key1.2~IsFCu4xqxiYBmSN1-sk6QEDR7kDCvBp.3
2019-05-16T16:40:04.571 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:04.563 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.20__shadow_key1.2~IsFCu4xqxiYBmSN1-sk6QEDR7kDCvBp.3_1
2019-05-16T16:40:04.572 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:04.563 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.20__multipart_key1.2~IsFCu4xqxiYBmSN1-sk6QEDR7kDCvBp.4
2019-05-16T16:40:04.572 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:04.563 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.20__shadow_key1.2~IsFCu4xqxiYBmSN1-sk6QEDR7kDCvBp.4_1
2019-05-16T16:40:04.572 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:04.563 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.20__multipart_key1.2~IsFCu4xqxiYBmSN1-sk6QEDR7kDCvBp.5
2019-05-16T16:40:04.727 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:04.720 7fa922ac56c0 20 garbage collection: RGWGC::process entered with GC index_shard=24, max_secs=3600, expired_only=0
2019-05-16T16:40:04.774 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:04.766 7fa922ac56c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:04.810 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:04.803 7fa922ac56c0 20 garbage collection: RGWGC::process entered with GC index_shard=25, max_secs=3600, expired_only=0
2019-05-16T16:40:04.859 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:04.851 7fa922ac56c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=2, truncated=0, next_marker=''
2019-05-16T16:40:04.859 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:04.851 7fa922ac56c0 20 garbage collection: RGWGC::process iterating over entry tag='2~SOYcdAaxnIBC24rDDIxGxImNbw7wRa5', time=2019-05-16 18:23:03.0.358439s, chain.objs.size()=5
2019-05-16T16:40:04.859 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:04.851 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.27__multipart_key1.2~SOYcdAaxnIBC24rDDIxGxImNbw7wRa5.7
2019-05-16T16:40:04.859 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:04.851 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.27__shadow_key1.2~SOYcdAaxnIBC24rDDIxGxImNbw7wRa5.7_1
2019-05-16T16:40:04.860 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:04.851 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.27__multipart_key1.2~SOYcdAaxnIBC24rDDIxGxImNbw7wRa5.10
2019-05-16T16:40:04.860 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:04.851 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.27__shadow_key1.2~SOYcdAaxnIBC24rDDIxGxImNbw7wRa5.10_1
2019-05-16T16:40:04.860 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:04.851 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.27__multipart_key1.2~SOYcdAaxnIBC24rDDIxGxImNbw7wRa5.13
2019-05-16T16:40:04.860 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:04.851 7fa922ac56c0 20 garbage collection: RGWGC::process iterating over entry tag='ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4367.628�', time=2019-05-16 18:28:19.0.356762s, chain.objs.size()=16
2019-05-16T16:40:04.860 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:04.851 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__multipart_key1.2~tKT3QG3e3i5rSmEwhx2p9U0f3P7WI50.1
2019-05-16T16:40:04.860 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:04.851 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__shadow_key1.2~tKT3QG3e3i5rSmEwhx2p9U0f3P7WI50.1_1
2019-05-16T16:40:04.860 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:04.851 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__shadow_key1.2~tKT3QG3e3i5rSmEwhx2p9U0f3P7WI50.1_2
2019-05-16T16:40:04.860 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:04.851 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__multipart_key1.2~tKT3QG3e3i5rSmEwhx2p9U0f3P7WI50.2
2019-05-16T16:40:04.861 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:04.851 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__shadow_key1.2~tKT3QG3e3i5rSmEwhx2p9U0f3P7WI50.2_1
2019-05-16T16:40:04.861 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:04.851 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__shadow_key1.2~tKT3QG3e3i5rSmEwhx2p9U0f3P7WI50.2_2
2019-05-16T16:40:04.861 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:04.851 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__multipart_key1.2~tKT3QG3e3i5rSmEwhx2p9U0f3P7WI50.3
2019-05-16T16:40:04.959 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:04.951 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__shadow_key1.2~tKT3QG3e3i5rSmEwhx2p9U0f3P7WI50.3_1
2019-05-16T16:40:05.049 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:05.042 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__shadow_key1.2~tKT3QG3e3i5rSmEwhx2p9U0f3P7WI50.3_2
2019-05-16T16:40:05.104 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:05.096 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__multipart_key1.2~tKT3QG3e3i5rSmEwhx2p9U0f3P7WI50.4
2019-05-16T16:40:05.104 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:05.096 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__shadow_key1.2~tKT3QG3e3i5rSmEwhx2p9U0f3P7WI50.4_1
2019-05-16T16:40:05.104 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:05.096 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__shadow_key1.2~tKT3QG3e3i5rSmEwhx2p9U0f3P7WI50.4_2
2019-05-16T16:40:05.141 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:05.133 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__multipart_key1.2~tKT3QG3e3i5rSmEwhx2p9U0f3P7WI50.5
2019-05-16T16:40:05.141 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:05.133 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__shadow_key1.2~tKT3QG3e3i5rSmEwhx2p9U0f3P7WI50.5_1
2019-05-16T16:40:05.141 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:05.133 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__shadow_key1.2~tKT3QG3e3i5rSmEwhx2p9U0f3P7WI50.5_2
2019-05-16T16:40:05.141 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:05.133 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__multipart_key1.2~tKT3QG3e3i5rSmEwhx2p9U0f3P7WI50.6
2019-05-16T16:40:05.186 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:05.179 7fa922ac56c0 20 garbage collection: RGWGC::process entered with GC index_shard=26, max_secs=3600, expired_only=0
2019-05-16T16:40:05.278 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:05.270 7fa922ac56c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:05.349 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:05.342 7fa922ac56c0 20 garbage collection: RGWGC::process entered with GC index_shard=27, max_secs=3600, expired_only=0
2019-05-16T16:40:05.417 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:05.410 7fa922ac56c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:05.445 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:05.438 7fa922ac56c0 20 garbage collection: RGWGC::process entered with GC index_shard=28, max_secs=3600, expired_only=0
2019-05-16T16:40:05.517 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:05.510 7fa922ac56c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:05.579 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:05.572 7fa922ac56c0 20 garbage collection: RGWGC::process entered with GC index_shard=29, max_secs=3600, expired_only=0
2019-05-16T16:40:05.661 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:05.653 7fa922ac56c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=1, truncated=0, next_marker=''
2019-05-16T16:40:05.662 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:05.653 7fa922ac56c0 20 garbage collection: RGWGC::process iterating over entry tag='ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4367.455�', time=2019-05-16 18:20:06.0.341728s, chain.objs.size()=9
2019-05-16T16:40:05.662 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:05.653 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.20__multipart_key1.2~bv_eH8i-srsPAO--gCkIyPh0jT8-edA.1
2019-05-16T16:40:05.662 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:05.653 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.20__shadow_key1.2~bv_eH8i-srsPAO--gCkIyPh0jT8-edA.1_1
2019-05-16T16:40:05.662 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:05.653 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.20__multipart_key1.2~bv_eH8i-srsPAO--gCkIyPh0jT8-edA.2
2019-05-16T16:40:05.662 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:05.653 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.20__shadow_key1.2~bv_eH8i-srsPAO--gCkIyPh0jT8-edA.2_1
2019-05-16T16:40:05.662 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:05.654 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.20__multipart_key1.2~bv_eH8i-srsPAO--gCkIyPh0jT8-edA.3
2019-05-16T16:40:05.663 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:05.654 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.20__shadow_key1.2~bv_eH8i-srsPAO--gCkIyPh0jT8-edA.3_1
2019-05-16T16:40:05.663 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:05.654 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.20__multipart_key1.2~bv_eH8i-srsPAO--gCkIyPh0jT8-edA.4
2019-05-16T16:40:05.663 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:05.654 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.20__shadow_key1.2~bv_eH8i-srsPAO--gCkIyPh0jT8-edA.4_1
2019-05-16T16:40:05.663 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:05.654 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.20__multipart_key1.2~bv_eH8i-srsPAO--gCkIyPh0jT8-edA.5
2019-05-16T16:40:05.779 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:05.772 7fa922ac56c0 20 garbage collection: RGWGC::process entered with GC index_shard=30, max_secs=3600, expired_only=0
2019-05-16T16:40:05.907 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:05.900 7fa922ac56c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=2, truncated=0, next_marker=''
2019-05-16T16:40:05.908 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:05.900 7fa922ac56c0 20 garbage collection: RGWGC::process iterating over entry tag='ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4367.467�', time=2019-05-16 18:20:24.0.277742s, chain.objs.size()=7
2019-05-16T16:40:05.908 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:05.900 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.20__multipart_key1.2~KktTqOZH2754xVY2Ib7uu_H9tzNvHMp.1
2019-05-16T16:40:05.908 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:05.900 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.20__shadow_key1.2~KktTqOZH2754xVY2Ib7uu_H9tzNvHMp.1_1
2019-05-16T16:40:05.908 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:05.900 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.20__shadow_key1.2~KktTqOZH2754xVY2Ib7uu_H9tzNvHMp.1_2
2019-05-16T16:40:05.932 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:05.925 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.20__multipart_key1.2~KktTqOZH2754xVY2Ib7uu_H9tzNvHMp.2
2019-05-16T16:40:05.932 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:05.925 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.20__shadow_key1.2~KktTqOZH2754xVY2Ib7uu_H9tzNvHMp.2_1
2019-05-16T16:40:05.932 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:05.925 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.20__shadow_key1.2~KktTqOZH2754xVY2Ib7uu_H9tzNvHMp.2_2
2019-05-16T16:40:05.968 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:05.961 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.20__multipart_key1.2~KktTqOZH2754xVY2Ib7uu_H9tzNvHMp.3
2019-05-16T16:40:05.969 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:05.961 7fa922ac56c0 20 garbage collection: RGWGC::process iterating over entry tag='2~I2lrozwNiQpMhL7rptMTqobZTaq4ILn', time=2019-05-16 18:30:48.0.230531s, chain.objs.size()=20
2019-05-16T16:40:05.969 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:05.961 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.31__multipart_key1.2~I2lrozwNiQpMhL7rptMTqobZTaq4ILn.1
2019-05-16T16:40:05.970 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:05.961 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.31__multipart_key1.2~I2lrozwNiQpMhL7rptMTqobZTaq4ILn.2
2019-05-16T16:40:05.970 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:05.961 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.31__multipart_key1.2~I2lrozwNiQpMhL7rptMTqobZTaq4ILn.3
2019-05-16T16:40:05.970 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:05.961 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.31__multipart_key1.2~I2lrozwNiQpMhL7rptMTqobZTaq4ILn.4
2019-05-16T16:40:05.970 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:05.961 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.31__multipart_key1.2~I2lrozwNiQpMhL7rptMTqobZTaq4ILn.5
2019-05-16T16:40:06.154 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:06.147 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.31__multipart_key1.2~I2lrozwNiQpMhL7rptMTqobZTaq4ILn.6
2019-05-16T16:40:06.155 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:06.147 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.31__multipart_key1.2~I2lrozwNiQpMhL7rptMTqobZTaq4ILn.7
2019-05-16T16:40:06.177 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:06.170 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.31__multipart_key1.2~I2lrozwNiQpMhL7rptMTqobZTaq4ILn.8
2019-05-16T16:40:06.207 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:06.200 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.31__multipart_key1.2~I2lrozwNiQpMhL7rptMTqobZTaq4ILn.9
2019-05-16T16:40:06.207 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:06.200 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.31__multipart_key1.2~I2lrozwNiQpMhL7rptMTqobZTaq4ILn.10
2019-05-16T16:40:06.347 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:06.340 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.31__multipart_key1.2~I2lrozwNiQpMhL7rptMTqobZTaq4ILn.11
2019-05-16T16:40:06.347 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:06.340 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.31__multipart_key1.2~I2lrozwNiQpMhL7rptMTqobZTaq4ILn.12
2019-05-16T16:40:06.348 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:06.340 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.31__multipart_key1.-vCH3Hnoq2lQDC0zXcYUDNQ0VGcLxQ2�.13
2019-05-16T16:40:06.348 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:06.340 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.31__multipart_key1.2~I2lrozwNiQpMhL7rptMTqobZTaq4ILn.14
2019-05-16T16:40:06.348 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:06.340 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.31__multipart_key1.IHb9V191tULDSLnL_zf8rTYjeeAoe1_�.15
2019-05-16T16:40:06.348 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:06.340 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.31__multipart_key1.gQvvS0aFBWe3-lE7Kn2owk17VcvRtH0�.16
2019-05-16T16:40:06.570 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:06.563 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.31__multipart_key1.ei19pmvEE7R2zf6MQumRl8M7INFDOgs�.17
2019-05-16T16:40:06.571 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:06.563 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.31__multipart_key1.JStTCSab-8jWghqXYPG7fQ-K2Q1GbIn�.18
2019-05-16T16:40:06.571 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:06.563 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.31__multipart_key1.2~I2lrozwNiQpMhL7rptMTqobZTaq4ILn.20
2019-05-16T16:40:06.571 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:06.563 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.31__multipart_key1.2~I2lrozwNiQpMhL7rptMTqobZTaq4ILn.21
2019-05-16T16:40:06.635 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:06.628 7fa922ac56c0 20 garbage collection: RGWGC::process entered with GC index_shard=31, max_secs=3600, expired_only=0
2019-05-16T16:40:06.732 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:06.723 7fa922ac56c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=2, truncated=0, next_marker=''
2019-05-16T16:40:06.732 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:06.723 7fa922ac56c0 20 garbage collection: RGWGC::process iterating over entry tag='ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4367.418�', time=2019-05-16 18:18:55.0.528396s, chain.objs.size()=9
2019-05-16T16:40:06.732 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:06.723 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.17__multipart_key1.2~pxVyzOyEcLylB-jMxr_LkrowjGSatiC.1
2019-05-16T16:40:06.732 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:06.723 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.17__shadow_key1.2~pxVyzOyEcLylB-jMxr_LkrowjGSatiC.1_1
2019-05-16T16:40:06.733 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:06.723 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.17__multipart_key1.2~pxVyzOyEcLylB-jMxr_LkrowjGSatiC.2
2019-05-16T16:40:06.733 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:06.723 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.17__shadow_key1.2~pxVyzOyEcLylB-jMxr_LkrowjGSatiC.2_1
2019-05-16T16:40:06.733 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:06.723 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.17__multipart_key1.2~pxVyzOyEcLylB-jMxr_LkrowjGSatiC.3
2019-05-16T16:40:06.733 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:06.723 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.17__shadow_key1.2~pxVyzOyEcLylB-jMxr_LkrowjGSatiC.3_1
2019-05-16T16:40:06.733 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:06.724 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.17__multipart_key1.2~pxVyzOyEcLylB-jMxr_LkrowjGSatiC.4
2019-05-16T16:40:06.733 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:06.724 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.17__shadow_key1.2~pxVyzOyEcLylB-jMxr_LkrowjGSatiC.4_1
2019-05-16T16:40:06.734 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:06.724 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.17__multipart_key1.2~pxVyzOyEcLylB-jMxr_LkrowjGSatiC.5
2019-05-16T16:40:06.734 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:06.724 7fa922ac56c0 20 garbage collection: RGWGC::process iterating over entry tag='ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4367.636�', time=2019-05-16 18:29:20.0.188735s, chain.objs.size()=16
2019-05-16T16:40:06.734 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:06.724 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__multipart_key1.2~9KuzpBjr2f6-VdK1WhybxX12SUjsQih.1
2019-05-16T16:40:06.734 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:06.724 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__shadow_key1.2~9KuzpBjr2f6-VdK1WhybxX12SUjsQih.1_1
2019-05-16T16:40:06.734 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:06.724 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__shadow_key1.2~9KuzpBjr2f6-VdK1WhybxX12SUjsQih.1_2
2019-05-16T16:40:06.931 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:06.923 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__multipart_key1.2~9KuzpBjr2f6-VdK1WhybxX12SUjsQih.2
2019-05-16T16:40:06.931 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:06.923 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__shadow_key1.2~9KuzpBjr2f6-VdK1WhybxX12SUjsQih.2_1
2019-05-16T16:40:06.931 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:06.923 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__shadow_key1.2~9KuzpBjr2f6-VdK1WhybxX12SUjsQih.2_2
2019-05-16T16:40:06.931 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:06.923 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__multipart_key1.2~9KuzpBjr2f6-VdK1WhybxX12SUjsQih.3
2019-05-16T16:40:06.931 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:06.923 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__shadow_key1.2~9KuzpBjr2f6-VdK1WhybxX12SUjsQih.3_1
2019-05-16T16:40:06.931 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:06.923 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__shadow_key1.2~9KuzpBjr2f6-VdK1WhybxX12SUjsQih.3_2
2019-05-16T16:40:06.932 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:06.923 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__multipart_key1.2~9KuzpBjr2f6-VdK1WhybxX12SUjsQih.4
2019-05-16T16:40:06.932 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:06.923 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__shadow_key1.2~9KuzpBjr2f6-VdK1WhybxX12SUjsQih.4_1
2019-05-16T16:40:06.932 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:06.923 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__shadow_key1.2~9KuzpBjr2f6-VdK1WhybxX12SUjsQih.4_2
2019-05-16T16:40:06.944 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:06.935 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__multipart_key1.2~9KuzpBjr2f6-VdK1WhybxX12SUjsQih.5
2019-05-16T16:40:06.944 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:06.935 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__shadow_key1.2~9KuzpBjr2f6-VdK1WhybxX12SUjsQih.5_1
2019-05-16T16:40:07.095 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:07.087 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__shadow_key1.2~9KuzpBjr2f6-VdK1WhybxX12SUjsQih.5_2
2019-05-16T16:40:07.095 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:07.087 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__multipart_key1.2~9KuzpBjr2f6-VdK1WhybxX12SUjsQih.6
2019-05-16T16:40:07.260 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:07.253 7fa922ac56c0 20 garbage collection: RGWGC::process entered with GC index_shard=0, max_secs=3600, expired_only=0
2019-05-16T16:40:07.307 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:07.299 7fa922ac56c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=1, truncated=0, next_marker=''
2019-05-16T16:40:07.307 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:07.299 7fa922ac56c0 20 garbage collection: RGWGC::process iterating over entry tag='ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4367.431�', time=2019-05-16 18:20:25.0.385121s, chain.objs.size()=5
2019-05-16T16:40:07.307 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:07.299 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.19__shadow_.EhqrTauBMugv9JTjue_c_ypZo59gV-H_1
2019-05-16T16:40:07.307 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:07.299 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.19__shadow_.EhqrTauBMugv9JTjue_c_ypZo59gV-H_2
2019-05-16T16:40:07.307 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:07.299 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.19__shadow_.EhqrTauBMugv9JTjue_c_ypZo59gV-H_3
2019-05-16T16:40:07.308 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:07.299 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.19__shadow_.EhqrTauBMugv9JTjue_c_ypZo59gV-H_4
2019-05-16T16:40:07.308 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:07.299 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.19__shadow_.EhqrTauBMugv9JTjue_c_ypZo59gV-H_5
2019-05-16T16:40:07.381 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:07.374 7fa922ac56c0 20 garbage collection: RGWGC::process entered with GC index_shard=1, max_secs=3600, expired_only=0
2019-05-16T16:40:07.425 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:07.418 7fa922ac56c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:07.456 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:07.449 7fa922ac56c0 20 garbage collection: RGWGC::process entered with GC index_shard=2, max_secs=3600, expired_only=0
2019-05-16T16:40:07.499 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:07.492 7fa922ac56c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:07.534 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:07.527 7fa922ac56c0 20 garbage collection: RGWGC::process entered with GC index_shard=3, max_secs=3600, expired_only=0
2019-05-16T16:40:07.578 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:07.570 7fa922ac56c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=3, truncated=0, next_marker=''
2019-05-16T16:40:07.579 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:07.570 7fa922ac56c0 20 garbage collection: RGWGC::process iterating over entry tag='ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4367.461�', time=2019-05-16 18:20:15.0.081747s, chain.objs.size()=7
2019-05-16T16:40:07.579 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:07.570 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.20__multipart_key1.2~SWy2_VWSyeri5DE_rg1HYNScqEsufJQ.1
2019-05-16T16:40:07.579 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:07.570 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.20__shadow_key1.2~SWy2_VWSyeri5DE_rg1HYNScqEsufJQ.1_1
2019-05-16T16:40:07.579 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:07.570 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.20__shadow_key1.2~SWy2_VWSyeri5DE_rg1HYNScqEsufJQ.1_2
2019-05-16T16:40:07.579 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:07.570 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.20__multipart_key1.2~SWy2_VWSyeri5DE_rg1HYNScqEsufJQ.2
2019-05-16T16:40:07.579 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:07.570 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.20__shadow_key1.2~SWy2_VWSyeri5DE_rg1HYNScqEsufJQ.2_1
2019-05-16T16:40:07.580 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:07.570 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.20__shadow_key1.2~SWy2_VWSyeri5DE_rg1HYNScqEsufJQ.2_2
2019-05-16T16:40:07.580 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:07.570 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.20__multipart_key1.2~SWy2_VWSyeri5DE_rg1HYNScqEsufJQ.3
2019-05-16T16:40:07.580 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:07.570 7fa922ac56c0 20 garbage collection: RGWGC::process iterating over entry tag='ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4367.510�', time=2019-05-16 18:21:05.0.994057s, chain.objs.size()=9
2019-05-16T16:40:07.580 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:07.570 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.24__multipart_key1.2~J8prqaQckyRKPrnJ1BlDltVbMA-rEi-.1
2019-05-16T16:40:07.580 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:07.570 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.24__shadow_key1.2~J8prqaQckyRKPrnJ1BlDltVbMA-rEi-.1_1
2019-05-16T16:40:07.580 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:07.571 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.24__multipart_key1.2~J8prqaQckyRKPrnJ1BlDltVbMA-rEi-.2
2019-05-16T16:40:07.580 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:07.571 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.24__shadow_key1.2~J8prqaQckyRKPrnJ1BlDltVbMA-rEi-.2_1
2019-05-16T16:40:07.581 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:07.571 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.24__multipart_key1.2~J8prqaQckyRKPrnJ1BlDltVbMA-rEi-.3
2019-05-16T16:40:07.631 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:07.624 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.24__shadow_key1.2~J8prqaQckyRKPrnJ1BlDltVbMA-rEi-.3_1
2019-05-16T16:40:07.644 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:07.637 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.24__multipart_key1.2~J8prqaQckyRKPrnJ1BlDltVbMA-rEi-.4
2019-05-16T16:40:07.911 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:07.904 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.24__shadow_key1.2~J8prqaQckyRKPrnJ1BlDltVbMA-rEi-.4_1
2019-05-16T16:40:07.912 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:07.904 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.24__multipart_key1.2~J8prqaQckyRKPrnJ1BlDltVbMA-rEi-.5
2019-05-16T16:40:07.912 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:07.904 7fa922ac56c0 20 garbage collection: RGWGC::process iterating over entry tag='ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4367.620�', time=2019-05-16 18:27:18.0.505662s, chain.objs.size()=19
2019-05-16T16:40:07.912 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:07.904 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__multipart_key1.2~5jzjkk3mMuiXOrzXyid_kLu-ldo-vyr.1
2019-05-16T16:40:07.912 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:07.904 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__shadow_key1.2~5jzjkk3mMuiXOrzXyid_kLu-ldo-vyr.1_1
2019-05-16T16:40:07.912 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:07.904 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__multipart_key1.2~5jzjkk3mMuiXOrzXyid_kLu-ldo-vyr.2
2019-05-16T16:40:07.912 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:07.904 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__shadow_key1.2~5jzjkk3mMuiXOrzXyid_kLu-ldo-vyr.2_1
2019-05-16T16:40:07.912 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:07.904 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__multipart_key1.2~5jzjkk3mMuiXOrzXyid_kLu-ldo-vyr.3
2019-05-16T16:40:07.913 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:07.904 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__shadow_key1.2~5jzjkk3mMuiXOrzXyid_kLu-ldo-vyr.3_1
2019-05-16T16:40:08.000 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:07.992 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__multipart_key1.2~5jzjkk3mMuiXOrzXyid_kLu-ldo-vyr.4
2019-05-16T16:40:08.000 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:07.992 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__shadow_key1.2~5jzjkk3mMuiXOrzXyid_kLu-ldo-vyr.4_1
2019-05-16T16:40:08.000 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:07.992 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__multipart_key1.2~5jzjkk3mMuiXOrzXyid_kLu-ldo-vyr.5
2019-05-16T16:40:08.127 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:08.120 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__shadow_key1.2~5jzjkk3mMuiXOrzXyid_kLu-ldo-vyr.5_1
2019-05-16T16:40:08.128 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:08.120 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__multipart_key1.2~5jzjkk3mMuiXOrzXyid_kLu-ldo-vyr.6
2019-05-16T16:40:08.157 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:08.150 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__shadow_key1.2~5jzjkk3mMuiXOrzXyid_kLu-ldo-vyr.6_1
2019-05-16T16:40:08.158 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:08.150 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__multipart_key1.2~5jzjkk3mMuiXOrzXyid_kLu-ldo-vyr.7
2019-05-16T16:40:08.158 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:08.150 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__shadow_key1.2~5jzjkk3mMuiXOrzXyid_kLu-ldo-vyr.7_1
2019-05-16T16:40:08.158 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:08.150 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__multipart_key1.2~5jzjkk3mMuiXOrzXyid_kLu-ldo-vyr.8
2019-05-16T16:40:08.158 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:08.150 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__shadow_key1.2~5jzjkk3mMuiXOrzXyid_kLu-ldo-vyr.8_1
2019-05-16T16:40:08.158 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:08.150 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__multipart_key1.2~5jzjkk3mMuiXOrzXyid_kLu-ldo-vyr.9
2019-05-16T16:40:08.158 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:08.150 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__shadow_key1.2~5jzjkk3mMuiXOrzXyid_kLu-ldo-vyr.9_1
2019-05-16T16:40:08.158 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:08.150 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__multipart_key1.2~5jzjkk3mMuiXOrzXyid_kLu-ldo-vyr.10
2019-05-16T16:40:08.201 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:08.194 7fa922ac56c0 20 garbage collection: RGWGC::process entered with GC index_shard=4, max_secs=3600, expired_only=0
2019-05-16T16:40:08.266 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:08.259 7fa922ac56c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=2, truncated=0, next_marker=''
2019-05-16T16:40:08.267 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:08.259 7fa922ac56c0 20 garbage collection: RGWGC::process iterating over entry tag='ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4367.416�', time=2019-05-16 18:18:56.0.738637s, chain.objs.size()=9
2019-05-16T16:40:08.267 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:08.259 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.17__multipart_key1.2~pxVyzOyEcLylB-jMxr_LkrowjGSatiC.1
2019-05-16T16:40:08.267 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:08.259 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.17__shadow_key1.2~pxVyzOyEcLylB-jMxr_LkrowjGSatiC.1_1
2019-05-16T16:40:08.267 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:08.259 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.17__multipart_key1.2~pxVyzOyEcLylB-jMxr_LkrowjGSatiC.2
2019-05-16T16:40:08.407 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:08.399 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.17__shadow_key1.2~pxVyzOyEcLylB-jMxr_LkrowjGSatiC.2_1
2019-05-16T16:40:08.407 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:08.399 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.17__multipart_key1.2~pxVyzOyEcLylB-jMxr_LkrowjGSatiC.3
2019-05-16T16:40:08.407 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:08.399 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.17__shadow_key1.2~pxVyzOyEcLylB-jMxr_LkrowjGSatiC.3_1
2019-05-16T16:40:08.407 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:08.399 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.17__multipart_key1.2~pxVyzOyEcLylB-jMxr_LkrowjGSatiC.4
2019-05-16T16:40:08.407 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:08.399 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.17__shadow_key1.2~pxVyzOyEcLylB-jMxr_LkrowjGSatiC.4_1
2019-05-16T16:40:08.409 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:08.402 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.17__multipart_key1.2~pxVyzOyEcLylB-jMxr_LkrowjGSatiC.5
2019-05-16T16:40:08.409 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:08.402 7fa922ac56c0 20 garbage collection: RGWGC::process iterating over entry tag='ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4367.473�', time=2019-05-16 18:20:26.0.657125s, chain.objs.size()=7
2019-05-16T16:40:08.409 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:08.402 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.20__multipart_key1.2~nGiqCIIlxs9zcWjzFyLzIu_Y8g5xrvd.1
2019-05-16T16:40:08.409 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:08.402 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.20__shadow_key1.2~nGiqCIIlxs9zcWjzFyLzIu_Y8g5xrvd.1_1
2019-05-16T16:40:08.409 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:08.402 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.20__shadow_key1.2~nGiqCIIlxs9zcWjzFyLzIu_Y8g5xrvd.1_2
2019-05-16T16:40:08.447 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:08.439 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.20__multipart_key1.2~nGiqCIIlxs9zcWjzFyLzIu_Y8g5xrvd.2
2019-05-16T16:40:08.447 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:08.439 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.20__shadow_key1.2~nGiqCIIlxs9zcWjzFyLzIu_Y8g5xrvd.2_1
2019-05-16T16:40:08.567 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:08.560 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.20__shadow_key1.2~nGiqCIIlxs9zcWjzFyLzIu_Y8g5xrvd.2_2
2019-05-16T16:40:08.567 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:08.560 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.20__multipart_key1.2~nGiqCIIlxs9zcWjzFyLzIu_Y8g5xrvd.3
2019-05-16T16:40:08.665 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:08.658 7fa922ac56c0 20 garbage collection: RGWGC::process entered with GC index_shard=5, max_secs=3600, expired_only=0
2019-05-16T16:40:08.735 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:08.728 7fa922ac56c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=2, truncated=0, next_marker=''
2019-05-16T16:40:08.735 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:08.728 7fa922ac56c0 20 garbage collection: RGWGC::process iterating over entry tag='ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4367.595�', time=2019-05-16 18:25:14.0.343856s, chain.objs.size()=21
2019-05-16T16:40:08.735 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:08.728 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__multipart_key1.2~8bsY4mIIVVcR0JVSMt4BQnvoEbSm-wh.1
2019-05-16T16:40:08.735 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:08.728 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__shadow_key1.2~8bsY4mIIVVcR0JVSMt4BQnvoEbSm-wh.1_1
2019-05-16T16:40:08.735 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:08.728 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__multipart_key1.2~8bsY4mIIVVcR0JVSMt4BQnvoEbSm-wh.2
2019-05-16T16:40:08.736 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:08.728 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__shadow_key1.2~8bsY4mIIVVcR0JVSMt4BQnvoEbSm-wh.2_1
2019-05-16T16:40:08.736 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:08.728 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__multipart_key1.2~8bsY4mIIVVcR0JVSMt4BQnvoEbSm-wh.3
2019-05-16T16:40:08.740 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:08.732 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__shadow_key1.2~8bsY4mIIVVcR0JVSMt4BQnvoEbSm-wh.3_1
2019-05-16T16:40:08.740 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:08.732 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__multipart_key1.2~8bsY4mIIVVcR0JVSMt4BQnvoEbSm-wh.4
2019-05-16T16:40:08.741 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:08.732 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__shadow_key1.2~8bsY4mIIVVcR0JVSMt4BQnvoEbSm-wh.4_1
2019-05-16T16:40:08.741 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:08.732 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__multipart_key1.2~8bsY4mIIVVcR0JVSMt4BQnvoEbSm-wh.5
2019-05-16T16:40:08.741 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:08.732 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__shadow_key1.2~8bsY4mIIVVcR0JVSMt4BQnvoEbSm-wh.5_1
2019-05-16T16:40:08.741 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:08.732 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__multipart_key1.2~8bsY4mIIVVcR0JVSMt4BQnvoEbSm-wh.6
2019-05-16T16:40:08.741 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:08.733 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__shadow_key1.2~8bsY4mIIVVcR0JVSMt4BQnvoEbSm-wh.6_1
2019-05-16T16:40:08.987 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:08.980 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__multipart_key1.2~8bsY4mIIVVcR0JVSMt4BQnvoEbSm-wh.7
2019-05-16T16:40:08.987 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:08.980 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__shadow_key1.2~8bsY4mIIVVcR0JVSMt4BQnvoEbSm-wh.7_1
2019-05-16T16:40:08.987 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:08.980 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__multipart_key1.2~8bsY4mIIVVcR0JVSMt4BQnvoEbSm-wh.8
2019-05-16T16:40:08.987 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:08.980 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__shadow_key1.2~8bsY4mIIVVcR0JVSMt4BQnvoEbSm-wh.8_1
2019-05-16T16:40:08.987 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:08.980 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__multipart_key1.2~8bsY4mIIVVcR0JVSMt4BQnvoEbSm-wh.9
2019-05-16T16:40:09.003 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:08.996 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__shadow_key1.2~8bsY4mIIVVcR0JVSMt4BQnvoEbSm-wh.9_1
2019-05-16T16:40:09.004 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:08.996 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__multipart_key1.2~8bsY4mIIVVcR0JVSMt4BQnvoEbSm-wh.10
2019-05-16T16:40:09.017 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:09.010 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__shadow_key1.2~8bsY4mIIVVcR0JVSMt4BQnvoEbSm-wh.10_1
2019-05-16T16:40:09.053 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:09.046 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__multipart_key1.2~8bsY4mIIVVcR0JVSMt4BQnvoEbSm-wh.11
2019-05-16T16:40:09.053 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:09.046 7fa922ac56c0 20 garbage collection: RGWGC::process iterating over entry tag='ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4367.644�', time=2019-05-16 18:29:21.0.220395s, chain.objs.size()=16
2019-05-16T16:40:09.053 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:09.046 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__multipart_key1.2~RLtZw41Argx0XsARXHCVzsPGYTwdyRa.1
2019-05-16T16:40:09.053 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:09.046 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__shadow_key1.2~RLtZw41Argx0XsARXHCVzsPGYTwdyRa.1_1
2019-05-16T16:40:09.200 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:09.192 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__shadow_key1.2~RLtZw41Argx0XsARXHCVzsPGYTwdyRa.1_2
2019-05-16T16:40:09.200 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:09.192 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__multipart_key1.2~RLtZw41Argx0XsARXHCVzsPGYTwdyRa.2
2019-05-16T16:40:09.200 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:09.192 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__shadow_key1.2~RLtZw41Argx0XsARXHCVzsPGYTwdyRa.2_1
2019-05-16T16:40:09.213 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:09.206 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__shadow_key1.2~RLtZw41Argx0XsARXHCVzsPGYTwdyRa.2_2
2019-05-16T16:40:09.213 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:09.206 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__multipart_key1.2~RLtZw41Argx0XsARXHCVzsPGYTwdyRa.3
2019-05-16T16:40:09.213 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:09.206 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__shadow_key1.2~RLtZw41Argx0XsARXHCVzsPGYTwdyRa.3_1
2019-05-16T16:40:09.223 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:09.215 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__shadow_key1.2~RLtZw41Argx0XsARXHCVzsPGYTwdyRa.3_2
2019-05-16T16:40:09.223 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:09.215 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__multipart_key1.2~RLtZw41Argx0XsARXHCVzsPGYTwdyRa.4
2019-05-16T16:40:09.223 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:09.215 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__shadow_key1.2~RLtZw41Argx0XsARXHCVzsPGYTwdyRa.4_1
2019-05-16T16:40:09.419 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:09.412 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__shadow_key1.2~RLtZw41Argx0XsARXHCVzsPGYTwdyRa.4_2
2019-05-16T16:40:09.420 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:09.412 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__multipart_key1.2~RLtZw41Argx0XsARXHCVzsPGYTwdyRa.5
2019-05-16T16:40:09.470 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:09.462 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__shadow_key1.2~RLtZw41Argx0XsARXHCVzsPGYTwdyRa.5_1
2019-05-16T16:40:09.470 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:09.462 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__shadow_key1.2~RLtZw41Argx0XsARXHCVzsPGYTwdyRa.5_2
2019-05-16T16:40:09.470 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:09.462 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.28__multipart_key1.2~RLtZw41Argx0XsARXHCVzsPGYTwdyRa.6
2019-05-16T16:40:09.520 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:09.513 7fa922ac56c0 20 garbage collection: RGWGC::process entered with GC index_shard=6, max_secs=3600, expired_only=0
2019-05-16T16:40:09.675 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:09.667 7fa922ac56c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=1, truncated=0, next_marker=''
2019-05-16T16:40:09.675 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:09.667 7fa922ac56c0 20 garbage collection: RGWGC::process iterating over entry tag='ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4367.387�', time=2019-05-16 18:18:17.0.30749s, chain.objs.size()=9
2019-05-16T16:40:09.676 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:09.667 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.15__multipart_key1.2~hKfb-0nLT5Icj1O3Xrus0Ax7Tbr9zF2.1
2019-05-16T16:40:09.676 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:09.667 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.15__shadow_key1.2~hKfb-0nLT5Icj1O3Xrus0Ax7Tbr9zF2.1_1
2019-05-16T16:40:09.676 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:09.668 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.15__multipart_key1.2~hKfb-0nLT5Icj1O3Xrus0Ax7Tbr9zF2.2
2019-05-16T16:40:09.676 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:09.668 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.15__shadow_key1.2~hKfb-0nLT5Icj1O3Xrus0Ax7Tbr9zF2.2_1
2019-05-16T16:40:09.676 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:09.668 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.15__multipart_key1.2~hKfb-0nLT5Icj1O3Xrus0Ax7Tbr9zF2.3
2019-05-16T16:40:09.676 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:09.668 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.15__shadow_key1.2~hKfb-0nLT5Icj1O3Xrus0Ax7Tbr9zF2.3_1
2019-05-16T16:40:09.676 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:09.668 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.15__multipart_key1.2~hKfb-0nLT5Icj1O3Xrus0Ax7Tbr9zF2.4
2019-05-16T16:40:09.677 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:09.668 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.15__shadow_key1.2~hKfb-0nLT5Icj1O3Xrus0Ax7Tbr9zF2.4_1
2019-05-16T16:40:09.677 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:09.668 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.15__multipart_key1.2~hKfb-0nLT5Icj1O3Xrus0Ax7Tbr9zF2.5
2019-05-16T16:40:09.783 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:09.776 7fa922ac56c0 20 garbage collection: RGWGC::process entered with GC index_shard=7, max_secs=3600, expired_only=0
2019-05-16T16:40:09.887 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:09.880 7fa922ac56c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:09.933 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:09.925 7fa922ac56c0 20 garbage collection: RGWGC::process entered with GC index_shard=8, max_secs=3600, expired_only=0
2019-05-16T16:40:09.981 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:09.973 7fa922ac56c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=1, truncated=0, next_marker=''
2019-05-16T16:40:09.982 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:09.974 7fa922ac56c0 20 garbage collection: RGWGC::process iterating over entry tag='ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4367.674�', time=2019-05-16 18:30:27.0.400849s, chain.objs.size()=9
2019-05-16T16:40:09.982 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:09.974 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.30__multipart_key1.2~29IVaKjheLJ2-Wfiikb5a_Iqy5TgkVz.1
2019-05-16T16:40:09.982 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:09.974 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.30__shadow_key1.2~29IVaKjheLJ2-Wfiikb5a_Iqy5TgkVz.1_1
2019-05-16T16:40:09.982 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:09.974 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.30__multipart_key1.2~29IVaKjheLJ2-Wfiikb5a_Iqy5TgkVz.2
2019-05-16T16:40:09.982 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:09.974 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.30__shadow_key1.2~29IVaKjheLJ2-Wfiikb5a_Iqy5TgkVz.2_1
2019-05-16T16:40:09.982 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:09.974 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.30__multipart_key1.2~29IVaKjheLJ2-Wfiikb5a_Iqy5TgkVz.3
2019-05-16T16:40:09.982 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:09.974 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.30__shadow_key1.2~29IVaKjheLJ2-Wfiikb5a_Iqy5TgkVz.3_1
2019-05-16T16:40:09.983 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:09.974 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.30__multipart_key1.2~29IVaKjheLJ2-Wfiikb5a_Iqy5TgkVz.4
2019-05-16T16:40:09.983 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:09.974 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.30__shadow_key1.2~29IVaKjheLJ2-Wfiikb5a_Iqy5TgkVz.4_1
2019-05-16T16:40:09.983 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:09.974 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.30__multipart_key1.2~29IVaKjheLJ2-Wfiikb5a_Iqy5TgkVz.5
2019-05-16T16:40:10.072 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:10.064 7fa922ac56c0 20 garbage collection: RGWGC::process entered with GC index_shard=9, max_secs=3600, expired_only=0
2019-05-16T16:40:10.154 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:10.146 7fa922ac56c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:10.199 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:10.192 7fa922ac56c0 20 garbage collection: RGWGC::process entered with GC index_shard=10, max_secs=3600, expired_only=0
2019-05-16T16:40:10.238 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:10.230 7fa922ac56c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=1, truncated=0, next_marker=''
2019-05-16T16:40:10.238 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:10.230 7fa922ac56c0 20 garbage collection: RGWGC::process iterating over entry tag='2~3TPeQ9ill_LjXXnIW3HL4XrAMNsabb-', time=2019-05-16 18:17:36.0.985539s, chain.objs.size()=9
2019-05-16T16:40:10.238 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:10.230 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.13__multipart_key1.2~3TPeQ9ill_LjXXnIW3HL4XrAMNsabb-.1
2019-05-16T16:40:10.238 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:10.230 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.13__shadow_key1.2~3TPeQ9ill_LjXXnIW3HL4XrAMNsabb-.1_1
2019-05-16T16:40:10.238 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:10.230 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.13__multipart_key1.2~3TPeQ9ill_LjXXnIW3HL4XrAMNsabb-.2
2019-05-16T16:40:10.239 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:10.230 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.13__shadow_key1.2~3TPeQ9ill_LjXXnIW3HL4XrAMNsabb-.2_1
2019-05-16T16:40:10.239 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:10.230 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.13__multipart_key1.2~3TPeQ9ill_LjXXnIW3HL4XrAMNsabb-.3
2019-05-16T16:40:10.240 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:10.230 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.13__shadow_key1.2~3TPeQ9ill_LjXXnIW3HL4XrAMNsabb-.3_1
2019-05-16T16:40:10.240 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:10.230 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.13__multipart_key1.2~3TPeQ9ill_LjXXnIW3HL4XrAMNsabb-.4
2019-05-16T16:40:10.240 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:10.230 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.13__shadow_key1.2~3TPeQ9ill_LjXXnIW3HL4XrAMNsabb-.4_1
2019-05-16T16:40:10.240 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:10.230 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.13__multipart_key1.2~3TPeQ9ill_LjXXnIW3HL4XrAMNsabb-.5
2019-05-16T16:40:10.309 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:10.302 7fa922ac56c0 20 garbage collection: RGWGC::process entered with GC index_shard=11, max_secs=3600, expired_only=0
2019-05-16T16:40:10.408 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:10.400 7fa922ac56c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:10.443 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:10.436 7fa922ac56c0 20 garbage collection: RGWGC::process entered with GC index_shard=12, max_secs=3600, expired_only=0
2019-05-16T16:40:10.553 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:10.546 7fa922ac56c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:10.589 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:10.581 7fa922ac56c0 20 garbage collection: RGWGC::process entered with GC index_shard=13, max_secs=3600, expired_only=0
2019-05-16T16:40:10.646 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:10.639 7fa922ac56c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:10.698 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:10.692 7fa922ac56c0 20 garbage collection: RGWGC::process entered with GC index_shard=14, max_secs=3600, expired_only=0
2019-05-16T16:40:10.779 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:10.771 7fa922ac56c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:10.839 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:10.832 7fa922ac56c0 20 garbage collection: RGWGC::process entered with GC index_shard=15, max_secs=3600, expired_only=0
2019-05-16T16:40:10.907 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:10.900 7fa922ac56c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=1, truncated=0, next_marker=''
2019-05-16T16:40:10.907 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:10.900 7fa922ac56c0 20 garbage collection: RGWGC::process iterating over entry tag='ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4367.529�', time=2019-05-16 18:21:43.0.625932s, chain.objs.size()=9
2019-05-16T16:40:10.908 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:10.900 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.25__multipart_key1.2~vRGAAworcAXe2w8f8h-QQzajCt82wxv.1
2019-05-16T16:40:10.908 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:10.900 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.25__shadow_key1.2~vRGAAworcAXe2w8f8h-QQzajCt82wxv.1_1
2019-05-16T16:40:10.908 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:10.900 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.25__multipart_key1.2~vRGAAworcAXe2w8f8h-QQzajCt82wxv.2
2019-05-16T16:40:10.908 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:10.900 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.25__shadow_key1.2~vRGAAworcAXe2w8f8h-QQzajCt82wxv.2_1
2019-05-16T16:40:10.908 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:10.900 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.25__multipart_key1.2~vRGAAworcAXe2w8f8h-QQzajCt82wxv.3
2019-05-16T16:40:10.908 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:10.900 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.25__shadow_key1.2~vRGAAworcAXe2w8f8h-QQzajCt82wxv.3_1
2019-05-16T16:40:10.909 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:10.900 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.25__multipart_key1.2~vRGAAworcAXe2w8f8h-QQzajCt82wxv.4
2019-05-16T16:40:10.909 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:10.900 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.25__shadow_key1.2~vRGAAworcAXe2w8f8h-QQzajCt82wxv.4_1
2019-05-16T16:40:10.909 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:10.900 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.25__multipart_key1.2~vRGAAworcAXe2w8f8h-QQzajCt82wxv.5
2019-05-16T16:40:11.136 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:11.129 7fa922ac56c0 20 garbage collection: RGWGC::process entered with GC index_shard=16, max_secs=3600, expired_only=0
2019-05-16T16:40:11.201 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:11.194 7fa922ac56c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:11.245 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:11.238 7fa922ac56c0 20 garbage collection: RGWGC::process entered with GC index_shard=17, max_secs=3600, expired_only=0
2019-05-16T16:40:11.299 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:11.291 7fa922ac56c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=1, truncated=0, next_marker=''
2019-05-16T16:40:11.299 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:11.291 7fa922ac56c0 20 garbage collection: RGWGC::process iterating over entry tag='ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4367.659�', time=2019-05-16 18:29:57.0.401454s, chain.objs.size()=9
2019-05-16T16:40:11.299 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:11.291 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.29__multipart_key1.2~6Eo8kJHmdMRym4T7AqfLnDFUYe-SIqY.1
2019-05-16T16:40:11.299 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:11.291 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.29__shadow_key1.2~6Eo8kJHmdMRym4T7AqfLnDFUYe-SIqY.1_1
2019-05-16T16:40:11.299 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:11.291 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.29__multipart_key1.2~6Eo8kJHmdMRym4T7AqfLnDFUYe-SIqY.2
2019-05-16T16:40:11.300 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:11.291 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.29__shadow_key1.2~6Eo8kJHmdMRym4T7AqfLnDFUYe-SIqY.2_1
2019-05-16T16:40:11.300 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:11.291 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.29__multipart_key1.2~6Eo8kJHmdMRym4T7AqfLnDFUYe-SIqY.3
2019-05-16T16:40:11.300 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:11.291 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.29__shadow_key1.2~6Eo8kJHmdMRym4T7AqfLnDFUYe-SIqY.3_1
2019-05-16T16:40:11.300 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:11.291 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.29__multipart_key1.2~6Eo8kJHmdMRym4T7AqfLnDFUYe-SIqY.4
2019-05-16T16:40:11.300 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:11.291 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.29__shadow_key1.2~6Eo8kJHmdMRym4T7AqfLnDFUYe-SIqY.4_1
2019-05-16T16:40:11.301 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:11.291 7fa922ac56c0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4370.29__multipart_key1.2~6Eo8kJHmdMRym4T7AqfLnDFUYe-SIqY.5
2019-05-16T16:40:11.521 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:11.512 7fa922ac56c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=0, size=1, entries=[ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4367.431�]
2019-05-16T16:40:11.521 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:11.512 7fa922ac56c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=1, size=0, entries=[]
2019-05-16T16:40:11.521 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:11.512 7fa922ac56c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=2, size=0, entries=[]
2019-05-16T16:40:11.521 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:11.512 7fa922ac56c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=3, size=3, entries=[ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4367.461�,ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4367.510�,ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4367.620�]
2019-05-16T16:40:11.522 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:11.512 7fa922ac56c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=4, size=2, entries=[ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4367.416�,ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4367.473�]
2019-05-16T16:40:11.522 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:11.512 7fa922ac56c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=5, size=2, entries=[ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4367.595�,ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4367.644�]
2019-05-16T16:40:11.522 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:11.512 7fa922ac56c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=6, size=1, entries=[ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4367.387�]
2019-05-16T16:40:11.522 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:11.512 7fa922ac56c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=7, size=0, entries=[]
2019-05-16T16:40:11.522 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:11.512 7fa922ac56c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=8, size=1, entries=[ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4367.674�]
2019-05-16T16:40:11.522 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:11.512 7fa922ac56c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=9, size=0, entries=[]
2019-05-16T16:40:11.523 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:11.512 7fa922ac56c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=10, size=1, entries=[2~3TPeQ9ill_LjXXnIW3HL4XrAMNsabb-]
2019-05-16T16:40:11.523 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:11.512 7fa922ac56c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=11, size=0, entries=[]
2019-05-16T16:40:11.523 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:11.512 7fa922ac56c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=12, size=0, entries=[]
2019-05-16T16:40:11.523 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:11.512 7fa922ac56c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=13, size=0, entries=[]
2019-05-16T16:40:11.523 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:11.512 7fa922ac56c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=14, size=0, entries=[]
2019-05-16T16:40:11.523 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:11.512 7fa922ac56c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=15, size=1, entries=[ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4367.529�]
2019-05-16T16:40:11.523 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:11.513 7fa922ac56c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=16, size=0, entries=[]
2019-05-16T16:40:11.523 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:11.513 7fa922ac56c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=17, size=1, entries=[ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4367.659�]
2019-05-16T16:40:11.524 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:11.513 7fa922ac56c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=18, size=3, entries=[ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4367.560�,ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4367.559�,ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4367.608�]
2019-05-16T16:40:11.524 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:11.513 7fa922ac56c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=19, size=0, entries=[]
2019-05-16T16:40:11.524 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:11.513 7fa922ac56c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=20, size=0, entries=[]
2019-05-16T16:40:11.524 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:11.513 7fa922ac56c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=21, size=0, entries=[]
2019-05-16T16:40:11.524 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:11.513 7fa922ac56c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=22, size=1, entries=[ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4367.439�]
2019-05-16T16:40:11.524 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:11.513 7fa922ac56c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=23, size=1, entries=[ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4367.447�]
2019-05-16T16:40:11.524 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:11.513 7fa922ac56c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=24, size=0, entries=[]
2019-05-16T16:40:11.525 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:11.513 7fa922ac56c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=25, size=2, entries=[2~SOYcdAaxnIBC24rDDIxGxImNbw7wRa5,ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4367.628�]
2019-05-16T16:40:11.525 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:11.513 7fa922ac56c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=26, size=0, entries=[]
2019-05-16T16:40:11.525 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:11.513 7fa922ac56c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=27, size=0, entries=[]
2019-05-16T16:40:11.525 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:11.513 7fa922ac56c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=28, size=0, entries=[]
2019-05-16T16:40:11.525 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:11.513 7fa922ac56c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=29, size=1, entries=[ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4367.455�]
2019-05-16T16:40:11.525 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:11.513 7fa922ac56c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=30, size=2, entries=[ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4367.467�,2~I2lrozwNiQpMhL7rptMTqobZTaq4ILn]
2019-05-16T16:40:11.525 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:11.513 7fa922ac56c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=31, size=2, entries=[ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4367.418�,ec4d9074-b84d-4f70-b1c0-15fa89cb015e.4367.636�]
2019-05-16T16:40:11.805 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:11.798 7fa922ac56c0 20 remove_watcher() i=0
2019-05-16T16:40:11.805 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:11.798 7fa922ac56c0 2 removed watcher, disabling cache
2019-05-16T16:40:11.835 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:11.827 7fa922ac56c0 20 remove_watcher() i=1
2019-05-16T16:40:11.865 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:11.858 7fa922ac56c0 20 remove_watcher() i=2
2019-05-16T16:40:11.904 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:11.897 7fa922ac56c0 20 remove_watcher() i=3
2019-05-16T16:40:11.935 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:11.928 7fa922ac56c0 20 remove_watcher() i=4
2019-05-16T16:40:11.986 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:11.979 7fa922ac56c0 20 remove_watcher() i=5
2019-05-16T16:40:12.045 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:12.038 7fa922ac56c0 20 remove_watcher() i=6
2019-05-16T16:40:12.108 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:12.101 7fa922ac56c0 20 remove_watcher() i=7
2019-05-16T16:40:12.116 INFO:teuthology.orchestra.run.smithi165:Running:
2019-05-16T16:40:12.116 INFO:teuthology.orchestra.run.smithi165:> radosgw-admin gc process --include-all
2019-05-16T16:40:12.300 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:12.293 7fbf6cdee6c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:12.332 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:12.325 7fbf6cdee6c0 20 rados_obj.operate() r=-2 bl.length=0
2019-05-16T16:40:12.332 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:12.325 7fbf6cdee6c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:12.348 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:12.341 7fbf6cdee6c0 20 rados_obj.operate() r=-2 bl.length=0
2019-05-16T16:40:12.348 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:12.341 7fbf6cdee6c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:12.354 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:12.346 7fbf6cdee6c0 20 rados_obj.operate() r=-2 bl.length=0
2019-05-16T16:40:12.354 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:12.346 7fbf6cdee6c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:12.361 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:12.354 7fbf6cdee6c0 20 rados_obj.operate() r=0 bl.length=46
2019-05-16T16:40:12.456 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:12.448 7fbf6cdee6c0 20 RGWRados::pool_iterate: got zonegroup_info.3e3ec152-13ab-4fe5-bbc5-31d7ce5be4ef
2019-05-16T16:40:12.456 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:12.448 7fbf6cdee6c0 20 RGWRados::pool_iterate: got zone_info.ec4d9074-b84d-4f70-b1c0-15fa89cb015e
2019-05-16T16:40:12.456 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:12.448 7fbf6cdee6c0 20 RGWRados::pool_iterate: got zone_names.default
2019-05-16T16:40:12.456 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:12.448 7fbf6cdee6c0 20 RGWRados::pool_iterate: got zonegroups_names.default
2019-05-16T16:40:12.456 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:12.448 7fbf6cdee6c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:12.467 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:12.460 7fbf6cdee6c0 20 rados_obj.operate() r=0 bl.length=46
2019-05-16T16:40:12.467 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:12.460 7fbf6cdee6c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:12.479 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:12.471 7fbf6cdee6c0 20 rados_obj.operate() r=0 bl.length=951
2019-05-16T16:40:12.479 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:12.471 7fbf6cdee6c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:12.486 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:12.479 7fbf6cdee6c0 20 rados_obj.operate() r=0 bl.length=46
2019-05-16T16:40:12.487 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:12.479 7fbf6cdee6c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:12.493 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:12.486 7fbf6cdee6c0 20 rados_obj.operate() r=0 bl.length=382
2019-05-16T16:40:12.493 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:12.486 7fbf6cdee6c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:12.499 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:12.492 7fbf6cdee6c0 20 rados_obj.operate() r=-2 bl.length=0
2019-05-16T16:40:12.499 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:12.492 7fbf6cdee6c0 10 cannot find current period zonegroup using local zonegroup
2019-05-16T16:40:12.499 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:12.492 7fbf6cdee6c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:12.506 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:12.499 7fbf6cdee6c0 20 rados_obj.operate() r=-2 bl.length=0
2019-05-16T16:40:12.506 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:12.499 7fbf6cdee6c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:12.516 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:12.509 7fbf6cdee6c0 20 rados_obj.operate() r=0 bl.length=46
2019-05-16T16:40:12.516 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:12.509 7fbf6cdee6c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:12.526 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:12.519 7fbf6cdee6c0 20 rados_obj.operate() r=0 bl.length=382
2019-05-16T16:40:12.526 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:12.519 7fbf6cdee6c0 20 zonegroup default
2019-05-16T16:40:12.526 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:12.519 7fbf6cdee6c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:12.537 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:12.529 7fbf6cdee6c0 20 rados_obj.operate() r=-2 bl.length=0
2019-05-16T16:40:12.537 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:12.530 7fbf6cdee6c0 10 Cannot find current period zone using local zone
2019-05-16T16:40:12.537 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:12.530 7fbf6cdee6c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:12.546 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:12.539 7fbf6cdee6c0 20 rados_obj.operate() r=-2 bl.length=0
2019-05-16T16:40:12.546 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:12.539 7fbf6cdee6c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:12.562 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:12.555 7fbf6cdee6c0 20 rados_obj.operate() r=0 bl.length=46
2019-05-16T16:40:12.563 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:12.555 7fbf6cdee6c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:12.570 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:12.563 7fbf6cdee6c0 20 rados_obj.operate() r=0 bl.length=951
2019-05-16T16:40:12.571 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:12.563 7fbf6cdee6c0 20 zone default
2019-05-16T16:40:12.991 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:12.983 7fbf6cdee6c0 20 add_watcher() i=0
2019-05-16T16:40:12.992 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:12.983 7fbf6cdee6c0 20 add_watcher() i=1
2019-05-16T16:40:12.993 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:12.983 7fbf6cdee6c0 20 add_watcher() i=2
2019-05-16T16:40:12.993 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:12.983 7fbf6cdee6c0 20 add_watcher() i=3
2019-05-16T16:40:12.993 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:12.984 7fbf6cdee6c0 20 add_watcher() i=4
2019-05-16T16:40:12.994 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:12.984 7fbf6cdee6c0 20 add_watcher() i=5
2019-05-16T16:40:12.994 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:12.984 7fbf6cdee6c0 20 add_watcher() i=6
2019-05-16T16:40:13.037 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:13.031 7fbf6cdee6c0 20 add_watcher() i=7
2019-05-16T16:40:13.038 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:13.031 7fbf6cdee6c0 2 all 8 watchers are set, enabling cache
2019-05-16T16:40:13.057 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:13.050 7fbf5a676700 0 -- 172.21.15.165:0/1555924976 >> v2:172.21.15.165:3300/0 conn(0x55a9ae3d2fe0 msgr2=0x55a9ae3d33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure
2019-05-16T16:40:13.081 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:13.073 7fbf31ffb700 2 RGWDataChangesLog::ChangesRenewThread: start
2019-05-16T16:40:13.104 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:13.097 7fbf5ae77700 0 -- 172.21.15.165:0/1555924976 >> [v2:172.21.15.8:6800/9787,v1:172.21.15.8:6801/9787] conn(0x7fbf34017130 msgr2=0x7fbf34019540 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure
2019-05-16T16:40:13.105 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:13.098 7fbf18fe1700 20 reqs_thread_entry: start
2019-05-16T16:40:13.106 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:13.098 7fbf127fc700 20 reqs_thread_entry: start
2019-05-16T16:40:13.106 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:13.098 7fbf6cdee6c0 20 init_complete bucket index max shards: 0
2019-05-16T16:40:13.106 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:13.099 7fbefbfff700 20 reqs_thread_entry: start
2019-05-16T16:40:13.107 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:13.099 7fbf6cdee6c0 20 garbage collection: RGWGC::process entered with GC index_shard=25, max_secs=3600, expired_only=0
2019-05-16T16:40:13.196 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:13.190 7fbf6cdee6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:13.245 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:13.239 7fbf6cdee6c0 20 garbage collection: RGWGC::process entered with GC index_shard=26, max_secs=3600, expired_only=0
2019-05-16T16:40:13.334 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:13.326 7fbf6cdee6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:13.401 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:13.394 7fbf6cdee6c0 20 garbage collection: RGWGC::process entered with GC index_shard=27, max_secs=3600, expired_only=0
2019-05-16T16:40:13.504 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:13.496 7fbf6cdee6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:13.550 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:13.543 7fbf6cdee6c0 20 garbage collection: RGWGC::process entered with GC index_shard=28, max_secs=3600, expired_only=0
2019-05-16T16:40:13.656 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:13.649 7fbf6cdee6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:13.723 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:13.716 7fbf6cdee6c0 20 garbage collection: RGWGC::process entered with GC index_shard=29, max_secs=3600, expired_only=0
2019-05-16T16:40:13.810 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:13.804 7fbf6cdee6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:13.880 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:13.873 7fbf6cdee6c0 20 garbage collection: RGWGC::process entered with GC index_shard=30, max_secs=3600, expired_only=0
2019-05-16T16:40:13.969 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:13.962 7fbf6cdee6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:14.040 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:14.033 7fbf6cdee6c0 20 garbage collection: RGWGC::process entered with GC index_shard=31, max_secs=3600, expired_only=0
2019-05-16T16:40:14.125 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:14.119 7fbf6cdee6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:14.188 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:14.181 7fbf6cdee6c0 20 garbage collection: RGWGC::process entered with GC index_shard=0, max_secs=3600, expired_only=0
2019-05-16T16:40:14.274 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:14.266 7fbf6cdee6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:14.322 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:14.316 7fbf6cdee6c0 20 garbage collection: RGWGC::process entered with GC index_shard=1, max_secs=3600, expired_only=0
2019-05-16T16:40:14.411 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:14.404 7fbf6cdee6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:14.468 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:14.462 7fbf6cdee6c0 20 garbage collection: RGWGC::process entered with GC index_shard=2, max_secs=3600, expired_only=0
2019-05-16T16:40:14.573 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:14.566 7fbf6cdee6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:14.610 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:14.603 7fbf6cdee6c0 20 garbage collection: RGWGC::process entered with GC index_shard=3, max_secs=3600, expired_only=0
2019-05-16T16:40:14.684 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:14.674 7fbf6cdee6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:14.749 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:14.741 7fbf6cdee6c0 20 garbage collection: RGWGC::process entered with GC index_shard=4, max_secs=3600, expired_only=0
2019-05-16T16:40:14.796 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:14.788 7fbf6cdee6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:14.832 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:14.825 7fbf6cdee6c0 20 garbage collection: RGWGC::process entered with GC index_shard=5, max_secs=3600, expired_only=0
2019-05-16T16:40:14.924 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:14.918 7fbf6cdee6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:14.990 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:14.983 7fbf6cdee6c0 20 garbage collection: RGWGC::process entered with GC index_shard=6, max_secs=3600, expired_only=0
2019-05-16T16:40:15.068 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:15.061 7fbf6cdee6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:15.128 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:15.121 7fbf6cdee6c0 20 garbage collection: RGWGC::process entered with GC index_shard=7, max_secs=3600, expired_only=0
2019-05-16T16:40:15.213 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:15.206 7fbf6cdee6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:15.278 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:15.271 7fbf6cdee6c0 20 garbage collection: RGWGC::process entered with GC index_shard=8, max_secs=3600, expired_only=0
2019-05-16T16:40:15.364 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:15.357 7fbf6cdee6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:15.420 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:15.413 7fbf6cdee6c0 20 garbage collection: RGWGC::process entered with GC index_shard=9, max_secs=3600, expired_only=0
2019-05-16T16:40:15.486 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:15.479 7fbf6cdee6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:15.547 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:15.540 7fbf6cdee6c0 20 garbage collection: RGWGC::process entered with GC index_shard=10, max_secs=3600, expired_only=0
2019-05-16T16:40:15.630 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:15.623 7fbf6cdee6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:15.702 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:15.694 7fbf6cdee6c0 20 garbage collection: RGWGC::process entered with GC index_shard=11, max_secs=3600, expired_only=0
2019-05-16T16:40:15.780 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:15.773 7fbf6cdee6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:15.826 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:15.819 7fbf6cdee6c0 20 garbage collection: RGWGC::process entered with GC index_shard=12, max_secs=3600, expired_only=0
2019-05-16T16:40:15.925 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:15.919 7fbf6cdee6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:15.997 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:15.991 7fbf6cdee6c0 20 garbage collection: RGWGC::process entered with GC index_shard=13, max_secs=3600, expired_only=0
2019-05-16T16:40:16.066 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:16.059 7fbf6cdee6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:16.129 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:16.123 7fbf6cdee6c0 20 garbage collection: RGWGC::process entered with GC index_shard=14, max_secs=3600, expired_only=0
2019-05-16T16:40:16.219 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:16.211 7fbf6cdee6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:16.287 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:16.280 7fbf6cdee6c0 20 garbage collection: RGWGC::process entered with GC index_shard=15, max_secs=3600, expired_only=0
2019-05-16T16:40:16.387 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:16.380 7fbf6cdee6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:16.459 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:16.452 7fbf6cdee6c0 20 garbage collection: RGWGC::process entered with GC index_shard=16, max_secs=3600, expired_only=0
2019-05-16T16:40:16.534 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:16.527 7fbf6cdee6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:16.659 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:16.652 7fbf6cdee6c0 20 garbage collection: RGWGC::process entered with GC index_shard=17, max_secs=3600, expired_only=0
2019-05-16T16:40:16.729 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:16.722 7fbf6cdee6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:16.790 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:16.782 7fbf6cdee6c0 20 garbage collection: RGWGC::process entered with GC index_shard=18, max_secs=3600, expired_only=0
2019-05-16T16:40:16.878 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:16.872 7fbf6cdee6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:16.932 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:16.926 7fbf6cdee6c0 20 garbage collection: RGWGC::process entered with GC index_shard=19, max_secs=3600, expired_only=0
2019-05-16T16:40:17.014 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:17.007 7fbf6cdee6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:17.078 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:17.071 7fbf6cdee6c0 20 garbage collection: RGWGC::process entered with GC index_shard=20, max_secs=3600, expired_only=0
2019-05-16T16:40:17.175 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:17.168 7fbf6cdee6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:17.231 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:17.224 7fbf6cdee6c0 20 garbage collection: RGWGC::process entered with GC index_shard=21, max_secs=3600, expired_only=0
2019-05-16T16:40:17.303 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:17.296 7fbf6cdee6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:17.357 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:17.350 7fbf6cdee6c0 20 garbage collection: RGWGC::process entered with GC index_shard=22, max_secs=3600, expired_only=0
2019-05-16T16:40:17.444 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:17.437 7fbf6cdee6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:17.503 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:17.497 7fbf6cdee6c0 20 garbage collection: RGWGC::process entered with GC index_shard=23, max_secs=3600, expired_only=0
2019-05-16T16:40:17.598 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:17.592 7fbf6cdee6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:17.675 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:17.668 7fbf6cdee6c0 20 garbage collection: RGWGC::process entered with GC index_shard=24, max_secs=3600, expired_only=0
2019-05-16T16:40:17.755 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:17.747 7fbf6cdee6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:17.805 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:17.796 7fbf6cdee6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=0, size=0, entries=[]
2019-05-16T16:40:17.805 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:17.796 7fbf6cdee6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=1, size=0, entries=[]
2019-05-16T16:40:17.805 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:17.796 7fbf6cdee6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=2, size=0, entries=[]
2019-05-16T16:40:17.805 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:17.796 7fbf6cdee6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=3, size=0, entries=[]
2019-05-16T16:40:17.806 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:17.796 7fbf6cdee6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=4, size=0, entries=[]
2019-05-16T16:40:17.806 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:17.796 7fbf6cdee6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=5, size=0, entries=[]
2019-05-16T16:40:17.806 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:17.796 7fbf6cdee6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=6, size=0, entries=[]
2019-05-16T16:40:17.806 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:17.796 7fbf6cdee6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=7, size=0, entries=[]
2019-05-16T16:40:17.806 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:17.796 7fbf6cdee6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=8, size=0, entries=[]
2019-05-16T16:40:17.806 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:17.796 7fbf6cdee6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=9, size=0, entries=[]
2019-05-16T16:40:17.806 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:17.796 7fbf6cdee6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=10, size=0, entries=[]
2019-05-16T16:40:17.807 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:17.796 7fbf6cdee6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=11, size=0, entries=[]
2019-05-16T16:40:17.807 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:17.796 7fbf6cdee6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=12, size=0, entries=[]
2019-05-16T16:40:17.807 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:17.796 7fbf6cdee6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=13, size=0, entries=[]
2019-05-16T16:40:17.807 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:17.796 7fbf6cdee6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=14, size=0, entries=[]
2019-05-16T16:40:17.807 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:17.796 7fbf6cdee6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=15, size=0, entries=[]
2019-05-16T16:40:17.807 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:17.796 7fbf6cdee6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=16, size=0, entries=[]
2019-05-16T16:40:17.807 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:17.796 7fbf6cdee6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=17, size=0, entries=[]
2019-05-16T16:40:17.808 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:17.796 7fbf6cdee6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=18, size=0, entries=[]
2019-05-16T16:40:17.808 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:17.796 7fbf6cdee6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=19, size=0, entries=[]
2019-05-16T16:40:17.808 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:17.796 7fbf6cdee6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=20, size=0, entries=[]
2019-05-16T16:40:17.808 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:17.796 7fbf6cdee6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=21, size=0, entries=[]
2019-05-16T16:40:17.808 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:17.796 7fbf6cdee6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=22, size=0, entries=[]
2019-05-16T16:40:17.808 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:17.796 7fbf6cdee6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=23, size=0, entries=[]
2019-05-16T16:40:17.808 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:17.796 7fbf6cdee6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=24, size=0, entries=[]
2019-05-16T16:40:17.809 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:17.796 7fbf6cdee6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=25, size=0, entries=[]
2019-05-16T16:40:17.809 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:17.796 7fbf6cdee6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=26, size=0, entries=[]
2019-05-16T16:40:17.809 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:17.796 7fbf6cdee6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=27, size=0, entries=[]
2019-05-16T16:40:17.809 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:17.796 7fbf6cdee6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=28, size=0, entries=[]
2019-05-16T16:40:17.809 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:17.796 7fbf6cdee6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=29, size=0, entries=[]
2019-05-16T16:40:17.809 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:17.796 7fbf6cdee6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=30, size=0, entries=[]
2019-05-16T16:40:17.809 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:17.796 7fbf6cdee6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=31, size=0, entries=[]
2019-05-16T16:40:18.089 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:18.082 7fbf6cdee6c0 20 remove_watcher() i=0
2019-05-16T16:40:18.089 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:18.082 7fbf6cdee6c0 2 removed watcher, disabling cache
2019-05-16T16:40:18.127 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:18.119 7fbf6cdee6c0 20 remove_watcher() i=1
2019-05-16T16:40:18.168 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:18.161 7fbf6cdee6c0 20 remove_watcher() i=2
2019-05-16T16:40:18.214 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:18.206 7fbf6cdee6c0 20 remove_watcher() i=3
2019-05-16T16:40:18.248 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:18.241 7fbf6cdee6c0 20 remove_watcher() i=4
2019-05-16T16:40:18.312 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:18.305 7fbf6cdee6c0 20 remove_watcher() i=5
2019-05-16T16:40:18.362 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:18.355 7fbf6cdee6c0 20 remove_watcher() i=6
2019-05-16T16:40:18.415 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:18.408 7fbf6cdee6c0 20 remove_watcher() i=7
2019-05-16T16:40:18.425 INFO:teuthology.orchestra.run.smithi165:Running:
2019-05-16T16:40:18.426 INFO:teuthology.orchestra.run.smithi165:> radosgw-admin gc process --include-all
2019-05-16T16:40:18.619 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:18.612 7ff368b3b6c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:18.655 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:18.648 7ff368b3b6c0 20 rados_obj.operate() r=-2 bl.length=0
2019-05-16T16:40:18.655 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:18.648 7ff368b3b6c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:18.669 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:18.662 7ff368b3b6c0 20 rados_obj.operate() r=-2 bl.length=0
2019-05-16T16:40:18.669 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:18.662 7ff368b3b6c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:18.675 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:18.668 7ff368b3b6c0 20 rados_obj.operate() r=-2 bl.length=0
2019-05-16T16:40:18.675 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:18.668 7ff368b3b6c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:18.691 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:18.684 7ff368b3b6c0 20 rados_obj.operate() r=0 bl.length=46
2019-05-16T16:40:18.819 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:18.812 7ff368b3b6c0 20 RGWRados::pool_iterate: got zonegroup_info.3e3ec152-13ab-4fe5-bbc5-31d7ce5be4ef
2019-05-16T16:40:18.820 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:18.812 7ff368b3b6c0 20 RGWRados::pool_iterate: got zone_info.ec4d9074-b84d-4f70-b1c0-15fa89cb015e
2019-05-16T16:40:18.820 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:18.812 7ff368b3b6c0 20 RGWRados::pool_iterate: got zone_names.default
2019-05-16T16:40:18.820 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:18.812 7ff368b3b6c0 20 RGWRados::pool_iterate: got zonegroups_names.default
2019-05-16T16:40:18.820 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:18.812 7ff368b3b6c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:18.836 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:18.829 7ff368b3b6c0 20 rados_obj.operate() r=0 bl.length=46
2019-05-16T16:40:18.836 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:18.829 7ff368b3b6c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:18.853 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:18.846 7ff368b3b6c0 20 rados_obj.operate() r=0 bl.length=951
2019-05-16T16:40:18.853 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:18.846 7ff368b3b6c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:18.870 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:18.863 7ff368b3b6c0 20 rados_obj.operate() r=0 bl.length=46
2019-05-16T16:40:18.870 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:18.863 7ff368b3b6c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:18.889 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:18.882 7ff368b3b6c0 20 rados_obj.operate() r=0 bl.length=382
2019-05-16T16:40:18.889 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:18.882 7ff368b3b6c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:18.899 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:18.892 7ff368b3b6c0 20 rados_obj.operate() r=-2 bl.length=0
2019-05-16T16:40:18.899 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:18.892 7ff368b3b6c0 10 cannot find current period zonegroup using local zonegroup
2019-05-16T16:40:18.899 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:18.892 7ff368b3b6c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:18.907 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:18.900 7ff368b3b6c0 20 rados_obj.operate() r=-2 bl.length=0
2019-05-16T16:40:18.907 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:18.900 7ff368b3b6c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:18.923 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:18.916 7ff368b3b6c0 20 rados_obj.operate() r=0 bl.length=46
2019-05-16T16:40:18.923 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:18.916 7ff368b3b6c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:18.940 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:18.932 7ff368b3b6c0 20 rados_obj.operate() r=0 bl.length=382
2019-05-16T16:40:18.940 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:18.932 7ff368b3b6c0 20 zonegroup default
2019-05-16T16:40:18.940 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:18.932 7ff368b3b6c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:18.950 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:18.943 7ff368b3b6c0 20 rados_obj.operate() r=-2 bl.length=0
2019-05-16T16:40:18.950 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:18.943 7ff368b3b6c0 10 Cannot find current period zone using local zone
2019-05-16T16:40:18.950 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:18.943 7ff368b3b6c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:18.959 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:18.952 7ff368b3b6c0 20 rados_obj.operate() r=-2 bl.length=0
2019-05-16T16:40:18.959 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:18.952 7ff368b3b6c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:18.976 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:18.969 7ff368b3b6c0 20 rados_obj.operate() r=0 bl.length=46
2019-05-16T16:40:18.976 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:18.969 7ff368b3b6c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:18.992 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:18.984 7ff368b3b6c0 20 rados_obj.operate() r=0 bl.length=951
2019-05-16T16:40:18.992 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:18.984 7ff368b3b6c0 20 zone default
2019-05-16T16:40:19.431 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:19.424 7ff368b3b6c0 20 add_watcher() i=0
2019-05-16T16:40:19.431 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:19.424 7ff368b3b6c0 20 add_watcher() i=1
2019-05-16T16:40:19.432 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:19.424 7ff368b3b6c0 20 add_watcher() i=2
2019-05-16T16:40:19.432 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:19.424 7ff368b3b6c0 20 add_watcher() i=3
2019-05-16T16:40:19.432 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:19.424 7ff368b3b6c0 20 add_watcher() i=4
2019-05-16T16:40:19.432 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:19.424 7ff368b3b6c0 20 add_watcher() i=5
2019-05-16T16:40:19.442 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:19.435 7ff368b3b6c0 20 add_watcher() i=6
2019-05-16T16:40:19.488 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:19.482 7ff368b3b6c0 20 add_watcher() i=7
2019-05-16T16:40:19.488 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:19.482 7ff368b3b6c0 2 all 8 watchers are set, enabling cache
2019-05-16T16:40:19.512 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:19.505 7ff355bc2700 0 -- 172.21.15.165:0/197262678 >> v2:172.21.15.8:3300/0 conn(0x562c0a717c80 msgr2=0x562c0a718080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure
2019-05-16T16:40:19.532 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:19.524 7ff32dffb700 2 RGWDataChangesLog::ChangesRenewThread: start
2019-05-16T16:40:19.550 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:19.543 7ff2f7fff700 20 reqs_thread_entry: start
2019-05-16T16:40:19.551 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:19.543 7ff2f5ffb700 20 reqs_thread_entry: start
2019-05-16T16:40:19.551 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:19.543 7ff368b3b6c0 20 init_complete bucket index max shards: 0
2019-05-16T16:40:19.551 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:19.544 7ff2affff700 20 reqs_thread_entry: start
2019-05-16T16:40:19.552 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:19.544 7ff368b3b6c0 20 garbage collection: RGWGC::process entered with GC index_shard=7, max_secs=3600, expired_only=0
2019-05-16T16:40:19.622 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:19.615 7ff368b3b6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:19.683 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:19.676 7ff368b3b6c0 20 garbage collection: RGWGC::process entered with GC index_shard=8, max_secs=3600, expired_only=0
2019-05-16T16:40:19.684 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:19.677 7ff356bc4700 0 -- 172.21.15.165:0/197262678 >> [v2:172.21.15.165:6808/10886,v1:172.21.15.165:6809/10886] conn(0x562c0a7a6060 msgr2=0x562c0a7a8520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure
2019-05-16T16:40:20.392 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:20.385 7ff368b3b6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:20.433 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:20.426 7ff368b3b6c0 20 garbage collection: RGWGC::process entered with GC index_shard=9, max_secs=3600, expired_only=0
2019-05-16T16:40:20.501 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:20.494 7ff368b3b6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:20.545 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:20.538 7ff368b3b6c0 20 garbage collection: RGWGC::process entered with GC index_shard=10, max_secs=3600, expired_only=0
2019-05-16T16:40:20.594 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:20.587 7ff368b3b6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:20.630 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:20.623 7ff368b3b6c0 20 garbage collection: RGWGC::process entered with GC index_shard=11, max_secs=3600, expired_only=0
2019-05-16T16:40:20.672 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:20.665 7ff368b3b6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:20.706 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:20.698 7ff368b3b6c0 20 garbage collection: RGWGC::process entered with GC index_shard=12, max_secs=3600, expired_only=0
2019-05-16T16:40:20.814 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:20.807 7ff368b3b6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:20.875 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:20.868 7ff368b3b6c0 20 garbage collection: RGWGC::process entered with GC index_shard=13, max_secs=3600, expired_only=0
2019-05-16T16:40:20.957 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:20.949 7ff368b3b6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:21.019 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:21.013 7ff368b3b6c0 20 garbage collection: RGWGC::process entered with GC index_shard=14, max_secs=3600, expired_only=0
2019-05-16T16:40:21.093 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:21.085 7ff368b3b6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:21.157 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:21.150 7ff368b3b6c0 20 garbage collection: RGWGC::process entered with GC index_shard=15, max_secs=3600, expired_only=0
2019-05-16T16:40:21.230 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:21.224 7ff368b3b6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:21.279 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:21.272 7ff368b3b6c0 20 garbage collection: RGWGC::process entered with GC index_shard=16, max_secs=3600, expired_only=0
2019-05-16T16:40:21.344 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:21.337 7ff368b3b6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:21.389 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:21.381 7ff368b3b6c0 20 garbage collection: RGWGC::process entered with GC index_shard=17, max_secs=3600, expired_only=0
2019-05-16T16:40:21.478 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:21.471 7ff368b3b6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:21.543 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:21.535 7ff368b3b6c0 20 garbage collection: RGWGC::process entered with GC index_shard=18, max_secs=3600, expired_only=0
2019-05-16T16:40:21.636 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:21.629 7ff368b3b6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:21.693 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:21.686 7ff368b3b6c0 20 garbage collection: RGWGC::process entered with GC index_shard=19, max_secs=3600, expired_only=0
2019-05-16T16:40:21.761 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:21.754 7ff368b3b6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:21.817 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:21.809 7ff368b3b6c0 20 garbage collection: RGWGC::process entered with GC index_shard=20, max_secs=3600, expired_only=0
2019-05-16T16:40:21.894 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:21.888 7ff368b3b6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:21.953 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:21.946 7ff368b3b6c0 20 garbage collection: RGWGC::process entered with GC index_shard=21, max_secs=3600, expired_only=0
2019-05-16T16:40:22.020 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:22.013 7ff368b3b6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:22.071 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:22.063 7ff368b3b6c0 20 garbage collection: RGWGC::process entered with GC index_shard=22, max_secs=3600, expired_only=0
2019-05-16T16:40:22.162 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:22.155 7ff368b3b6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:22.232 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:22.226 7ff368b3b6c0 20 garbage collection: RGWGC::process entered with GC index_shard=23, max_secs=3600, expired_only=0
2019-05-16T16:40:22.321 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:22.314 7ff368b3b6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:22.385 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:22.378 7ff368b3b6c0 20 garbage collection: RGWGC::process entered with GC index_shard=24, max_secs=3600, expired_only=0
2019-05-16T16:40:22.458 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:22.451 7ff368b3b6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:22.516 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:22.509 7ff368b3b6c0 20 garbage collection: RGWGC::process entered with GC index_shard=25, max_secs=3600, expired_only=0
2019-05-16T16:40:22.614 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:22.608 7ff368b3b6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:22.683 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:22.677 7ff368b3b6c0 20 garbage collection: RGWGC::process entered with GC index_shard=26, max_secs=3600, expired_only=0
2019-05-16T16:40:22.776 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:22.769 7ff368b3b6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:22.837 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:22.831 7ff368b3b6c0 20 garbage collection: RGWGC::process entered with GC index_shard=27, max_secs=3600, expired_only=0
2019-05-16T16:40:22.916 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:22.909 7ff368b3b6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:22.985 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:22.978 7ff368b3b6c0 20 garbage collection: RGWGC::process entered with GC index_shard=28, max_secs=3600, expired_only=0
2019-05-16T16:40:23.080 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:23.072 7ff368b3b6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:23.130 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:23.123 7ff368b3b6c0 20 garbage collection: RGWGC::process entered with GC index_shard=29, max_secs=3600, expired_only=0
2019-05-16T16:40:23.193 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:23.187 7ff368b3b6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:23.238 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:23.231 7ff368b3b6c0 20 garbage collection: RGWGC::process entered with GC index_shard=30, max_secs=3600, expired_only=0
2019-05-16T16:40:23.324 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:23.318 7ff368b3b6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:23.495 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:23.487 7ff368b3b6c0 20 garbage collection: RGWGC::process entered with GC index_shard=31, max_secs=3600, expired_only=0
2019-05-16T16:40:23.552 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:23.544 7ff368b3b6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:23.602 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:23.595 7ff368b3b6c0 20 garbage collection: RGWGC::process entered with GC index_shard=0, max_secs=3600, expired_only=0
2019-05-16T16:40:23.679 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:23.672 7ff368b3b6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:23.722 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:23.715 7ff368b3b6c0 20 garbage collection: RGWGC::process entered with GC index_shard=1, max_secs=3600, expired_only=0
2019-05-16T16:40:23.804 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:23.798 7ff368b3b6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:23.866 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:23.859 7ff368b3b6c0 20 garbage collection: RGWGC::process entered with GC index_shard=2, max_secs=3600, expired_only=0
2019-05-16T16:40:23.944 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:23.937 7ff368b3b6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:23.996 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:23.989 7ff368b3b6c0 20 garbage collection: RGWGC::process entered with GC index_shard=3, max_secs=3600, expired_only=0
2019-05-16T16:40:24.068 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:24.060 7ff368b3b6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:24.128 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:24.120 7ff368b3b6c0 20 garbage collection: RGWGC::process entered with GC index_shard=4, max_secs=3600, expired_only=0
2019-05-16T16:40:24.206 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:24.199 7ff368b3b6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:24.259 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:24.252 7ff368b3b6c0 20 garbage collection: RGWGC::process entered with GC index_shard=5, max_secs=3600, expired_only=0
2019-05-16T16:40:24.349 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:24.343 7ff368b3b6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:24.421 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:24.414 7ff368b3b6c0 20 garbage collection: RGWGC::process entered with GC index_shard=6, max_secs=3600, expired_only=0
2019-05-16T16:40:24.517 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:24.510 7ff368b3b6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:24.578 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:24.568 7ff368b3b6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=0, size=0, entries=[]
2019-05-16T16:40:24.578 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:24.568 7ff368b3b6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=1, size=0, entries=[]
2019-05-16T16:40:24.578 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:24.568 7ff368b3b6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=2, size=0, entries=[]
2019-05-16T16:40:24.578 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:24.568 7ff368b3b6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=3, size=0, entries=[]
2019-05-16T16:40:24.578 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:24.568 7ff368b3b6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=4, size=0, entries=[]
2019-05-16T16:40:24.578 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:24.568 7ff368b3b6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=5, size=0, entries=[]
2019-05-16T16:40:24.579 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:24.568 7ff368b3b6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=6, size=0, entries=[]
2019-05-16T16:40:24.579 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:24.568 7ff368b3b6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=7, size=0, entries=[]
2019-05-16T16:40:24.579 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:24.568 7ff368b3b6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=8, size=0, entries=[]
2019-05-16T16:40:24.579 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:24.568 7ff368b3b6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=9, size=0, entries=[]
2019-05-16T16:40:24.579 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:24.568 7ff368b3b6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=10, size=0, entries=[]
2019-05-16T16:40:24.579 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:24.568 7ff368b3b6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=11, size=0, entries=[]
2019-05-16T16:40:24.579 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:24.568 7ff368b3b6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=12, size=0, entries=[]
2019-05-16T16:40:24.580 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:24.569 7ff368b3b6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=13, size=0, entries=[]
2019-05-16T16:40:24.580 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:24.569 7ff368b3b6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=14, size=0, entries=[]
2019-05-16T16:40:24.580 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:24.569 7ff368b3b6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=15, size=0, entries=[]
2019-05-16T16:40:24.580 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:24.569 7ff368b3b6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=16, size=0, entries=[]
2019-05-16T16:40:24.580 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:24.569 7ff368b3b6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=17, size=0, entries=[]
2019-05-16T16:40:24.580 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:24.569 7ff368b3b6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=18, size=0, entries=[]
2019-05-16T16:40:24.581 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:24.569 7ff368b3b6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=19, size=0, entries=[]
2019-05-16T16:40:24.581 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:24.569 7ff368b3b6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=20, size=0, entries=[]
2019-05-16T16:40:24.581 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:24.569 7ff368b3b6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=21, size=0, entries=[]
2019-05-16T16:40:24.581 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:24.569 7ff368b3b6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=22, size=0, entries=[]
2019-05-16T16:40:24.581 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:24.569 7ff368b3b6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=23, size=0, entries=[]
2019-05-16T16:40:24.582 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:24.569 7ff368b3b6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=24, size=0, entries=[]
2019-05-16T16:40:24.582 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:24.569 7ff368b3b6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=25, size=0, entries=[]
2019-05-16T16:40:24.582 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:24.569 7ff368b3b6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=26, size=0, entries=[]
2019-05-16T16:40:24.582 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:24.569 7ff368b3b6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=27, size=0, entries=[]
2019-05-16T16:40:24.582 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:24.569 7ff368b3b6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=28, size=0, entries=[]
2019-05-16T16:40:24.582 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:24.569 7ff368b3b6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=29, size=0, entries=[]
2019-05-16T16:40:24.582 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:24.569 7ff368b3b6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=30, size=0, entries=[]
2019-05-16T16:40:24.583 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:24.569 7ff368b3b6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=31, size=0, entries=[]
2019-05-16T16:40:24.871 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:24.865 7ff368b3b6c0 20 remove_watcher() i=0
2019-05-16T16:40:24.872 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:24.865 7ff368b3b6c0 2 removed watcher, disabling cache
2019-05-16T16:40:24.920 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:24.913 7ff368b3b6c0 20 remove_watcher() i=1
2019-05-16T16:40:24.960 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:24.952 7ff368b3b6c0 20 remove_watcher() i=2
2019-05-16T16:40:25.013 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:25.006 7ff368b3b6c0 20 remove_watcher() i=3
2019-05-16T16:40:25.079 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:25.072 7ff368b3b6c0 20 remove_watcher() i=4
2019-05-16T16:40:25.126 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:25.119 7ff368b3b6c0 20 remove_watcher() i=5
2019-05-16T16:40:25.184 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:25.178 7ff368b3b6c0 20 remove_watcher() i=6
2019-05-16T16:40:25.251 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:25.244 7ff368b3b6c0 20 remove_watcher() i=7
2019-05-16T16:40:25.263 INFO:teuthology.orchestra.run.smithi165:Running:
2019-05-16T16:40:25.263 INFO:teuthology.orchestra.run.smithi165:> radosgw-admin gc process --include-all
2019-05-16T16:40:25.490 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:25.482 7f3f32e1d6c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:25.512 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:25.504 7f3f32e1d6c0 20 rados_obj.operate() r=-2 bl.length=0
2019-05-16T16:40:25.512 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:25.504 7f3f32e1d6c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:25.525 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:25.518 7f3f32e1d6c0 20 rados_obj.operate() r=-2 bl.length=0
2019-05-16T16:40:25.525 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:25.518 7f3f32e1d6c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:25.531 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:25.523 7f3f32e1d6c0 20 rados_obj.operate() r=-2 bl.length=0
2019-05-16T16:40:25.531 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:25.523 7f3f32e1d6c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:25.540 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:25.533 7f3f32e1d6c0 20 rados_obj.operate() r=0 bl.length=46
2019-05-16T16:40:25.656 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:25.649 7f3f32e1d6c0 20 RGWRados::pool_iterate: got zonegroup_info.3e3ec152-13ab-4fe5-bbc5-31d7ce5be4ef
2019-05-16T16:40:25.657 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:25.649 7f3f32e1d6c0 20 RGWRados::pool_iterate: got zone_info.ec4d9074-b84d-4f70-b1c0-15fa89cb015e
2019-05-16T16:40:25.657 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:25.649 7f3f32e1d6c0 20 RGWRados::pool_iterate: got zone_names.default
2019-05-16T16:40:25.657 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:25.649 7f3f32e1d6c0 20 RGWRados::pool_iterate: got zonegroups_names.default
2019-05-16T16:40:25.657 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:25.649 7f3f32e1d6c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:25.666 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:25.659 7f3f32e1d6c0 20 rados_obj.operate() r=0 bl.length=46
2019-05-16T16:40:25.666 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:25.659 7f3f32e1d6c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:25.683 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:25.676 7f3f32e1d6c0 20 rados_obj.operate() r=0 bl.length=951
2019-05-16T16:40:25.683 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:25.676 7f3f32e1d6c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:25.695 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:25.688 7f3f32e1d6c0 20 rados_obj.operate() r=0 bl.length=46
2019-05-16T16:40:25.695 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:25.688 7f3f32e1d6c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:25.706 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:25.699 7f3f32e1d6c0 20 rados_obj.operate() r=0 bl.length=382
2019-05-16T16:40:25.706 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:25.699 7f3f32e1d6c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:25.711 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:25.704 7f3f32e1d6c0 20 rados_obj.operate() r=-2 bl.length=0
2019-05-16T16:40:25.712 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:25.704 7f3f32e1d6c0 10 cannot find current period zonegroup using local zonegroup
2019-05-16T16:40:25.713 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:25.704 7f3f32e1d6c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:25.717 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:25.710 7f3f32e1d6c0 20 rados_obj.operate() r=-2 bl.length=0
2019-05-16T16:40:25.718 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:25.710 7f3f32e1d6c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:25.726 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:25.719 7f3f32e1d6c0 20 rados_obj.operate() r=0 bl.length=46
2019-05-16T16:40:25.726 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:25.719 7f3f32e1d6c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:25.735 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:25.728 7f3f32e1d6c0 20 rados_obj.operate() r=0 bl.length=382
2019-05-16T16:40:25.735 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:25.728 7f3f32e1d6c0 20 zonegroup default
2019-05-16T16:40:25.735 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:25.728 7f3f32e1d6c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:25.744 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:25.737 7f3f32e1d6c0 20 rados_obj.operate() r=-2 bl.length=0
2019-05-16T16:40:25.745 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:25.737 7f3f32e1d6c0 10 Cannot find current period zone using local zone
2019-05-16T16:40:25.745 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:25.737 7f3f32e1d6c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:25.752 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:25.745 7f3f32e1d6c0 20 rados_obj.operate() r=-2 bl.length=0
2019-05-16T16:40:25.752 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:25.745 7f3f32e1d6c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:25.762 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:25.755 7f3f32e1d6c0 20 rados_obj.operate() r=0 bl.length=46
2019-05-16T16:40:25.762 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:25.755 7f3f32e1d6c0 20 rados->read ofs=0 len=0
2019-05-16T16:40:25.770 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:25.763 7f3f32e1d6c0 20 rados_obj.operate() r=0 bl.length=951
2019-05-16T16:40:25.771 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:25.763 7f3f32e1d6c0 20 zone default
2019-05-16T16:40:26.427 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:26.419 7f3f32e1d6c0 20 add_watcher() i=0
2019-05-16T16:40:26.427 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:26.419 7f3f32e1d6c0 20 add_watcher() i=1
2019-05-16T16:40:26.427 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:26.419 7f3f32e1d6c0 20 add_watcher() i=2
2019-05-16T16:40:26.427 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:26.419 7f3f32e1d6c0 20 add_watcher() i=3
2019-05-16T16:40:26.428 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:26.419 7f3f32e1d6c0 20 add_watcher() i=4
2019-05-16T16:40:26.428 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:26.419 7f3f32e1d6c0 20 add_watcher() i=5
2019-05-16T16:40:26.428 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:26.419 7f3f32e1d6c0 20 add_watcher() i=6
2019-05-16T16:40:26.464 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:26.458 7f3f32e1d6c0 20 add_watcher() i=7
2019-05-16T16:40:26.464 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:26.458 7f3f32e1d6c0 2 all 8 watchers are set, enabling cache
2019-05-16T16:40:26.478 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:26.471 7f3f1fea4700 0 -- 172.21.15.165:0/1668828216 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x5593ce90c320 msgr2=0x5593ce90a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure
2019-05-16T16:40:26.509 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:26.501 7f3ef7fff700 2 RGWDataChangesLog::ChangesRenewThread: start
2019-05-16T16:40:26.520 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:26.513 7f3f20ea6700 0 -- 172.21.15.165:0/2851067721 >> [v2:172.21.15.8:6800/9787,v1:172.21.15.8:6801/9787] conn(0x7f3ef8014240 msgr2=0x7f3ef8016760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure
2019-05-16T16:40:26.522 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:26.515 7f3ede7e4700 20 reqs_thread_entry: start
2019-05-16T16:40:26.523 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:26.516 7f3e67fff700 20 reqs_thread_entry: start
2019-05-16T16:40:26.523 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:26.516 7f3f32e1d6c0 20 init_complete bucket index max shards: 0
2019-05-16T16:40:26.523 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:26.516 7f3e667fc700 20 reqs_thread_entry: start
2019-05-16T16:40:26.524 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:26.516 7f3f32e1d6c0 20 garbage collection: RGWGC::process entered with GC index_shard=27, max_secs=3600, expired_only=0
2019-05-16T16:40:26.588 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:26.580 7f3f32e1d6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:26.623 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:26.615 7f3f32e1d6c0 20 garbage collection: RGWGC::process entered with GC index_shard=28, max_secs=3600, expired_only=0
2019-05-16T16:40:26.736 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:26.729 7f3f32e1d6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:26.807 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:26.801 7f3f32e1d6c0 20 garbage collection: RGWGC::process entered with GC index_shard=29, max_secs=3600, expired_only=0
2019-05-16T16:40:26.920 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:26.913 7f3f32e1d6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:26.992 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:26.985 7f3f32e1d6c0 20 garbage collection: RGWGC::process entered with GC index_shard=30, max_secs=3600, expired_only=0
2019-05-16T16:40:27.105 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:27.099 7f3f32e1d6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:27.175 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:27.168 7f3f32e1d6c0 20 garbage collection: RGWGC::process entered with GC index_shard=31, max_secs=3600, expired_only=0
2019-05-16T16:40:27.260 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:27.252 7f3f32e1d6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:27.306 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:27.300 7f3f32e1d6c0 20 garbage collection: RGWGC::process entered with GC index_shard=0, max_secs=3600, expired_only=0
2019-05-16T16:40:27.387 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:27.380 7f3f32e1d6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:27.451 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:27.444 7f3f32e1d6c0 20 garbage collection: RGWGC::process entered with GC index_shard=1, max_secs=3600, expired_only=0
2019-05-16T16:40:27.515 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:27.508 7f3f32e1d6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:27.582 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:27.574 7f3f32e1d6c0 20 garbage collection: RGWGC::process entered with GC index_shard=2, max_secs=3600, expired_only=0
2019-05-16T16:40:27.654 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:27.647 7f3f32e1d6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:27.704 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:27.696 7f3f32e1d6c0 20 garbage collection: RGWGC::process entered with GC index_shard=3, max_secs=3600, expired_only=0
2019-05-16T16:40:27.769 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:27.761 7f3f32e1d6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:27.812 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:27.805 7f3f32e1d6c0 20 garbage collection: RGWGC::process entered with GC index_shard=4, max_secs=3600, expired_only=0
2019-05-16T16:40:27.857 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:27.850 7f3f32e1d6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:27.890 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:27.883 7f3f32e1d6c0 20 garbage collection: RGWGC::process entered with GC index_shard=5, max_secs=3600, expired_only=0
2019-05-16T16:40:28.013 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:28.006 7f3f32e1d6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:28.089 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:28.082 7f3f32e1d6c0 20 garbage collection: RGWGC::process entered with GC index_shard=6, max_secs=3600, expired_only=0
2019-05-16T16:40:28.177 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:28.170 7f3f32e1d6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:28.244 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:28.237 7f3f32e1d6c0 20 garbage collection: RGWGC::process entered with GC index_shard=7, max_secs=3600, expired_only=0
2019-05-16T16:40:28.324 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:28.317 7f3f32e1d6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:28.385 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:28.377 7f3f32e1d6c0 20 garbage collection: RGWGC::process entered with GC index_shard=8, max_secs=3600, expired_only=0
2019-05-16T16:40:28.460 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:28.453 7f3f32e1d6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:28.500 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:28.493 7f3f32e1d6c0 20 garbage collection: RGWGC::process entered with GC index_shard=9, max_secs=3600, expired_only=0
2019-05-16T16:40:28.550 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:28.543 7f3f32e1d6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:28.604 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:28.597 7f3f32e1d6c0 20 garbage collection: RGWGC::process entered with GC index_shard=10, max_secs=3600, expired_only=0
2019-05-16T16:40:28.654 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:28.647 7f3f32e1d6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:28.683 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:28.676 7f3f32e1d6c0 20 garbage collection: RGWGC::process entered with GC index_shard=11, max_secs=3600, expired_only=0
2019-05-16T16:40:28.729 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:28.722 7f3f32e1d6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:28.761 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:28.754 7f3f32e1d6c0 20 garbage collection: RGWGC::process entered with GC index_shard=12, max_secs=3600, expired_only=0
2019-05-16T16:40:28.871 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:28.864 7f3f32e1d6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:28.932 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:28.926 7f3f32e1d6c0 20 garbage collection: RGWGC::process entered with GC index_shard=13, max_secs=3600, expired_only=0
2019-05-16T16:40:28.991 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:28.985 7f3f32e1d6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:29.052 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:29.045 7f3f32e1d6c0 20 garbage collection: RGWGC::process entered with GC index_shard=14, max_secs=3600, expired_only=0
2019-05-16T16:40:29.146 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:29.139 7f3f32e1d6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:29.219 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:29.212 7f3f32e1d6c0 20 garbage collection: RGWGC::process entered with GC index_shard=15, max_secs=3600, expired_only=0
2019-05-16T16:40:29.306 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:29.299 7f3f32e1d6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:29.355 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:29.348 7f3f32e1d6c0 20 garbage collection: RGWGC::process entered with GC index_shard=16, max_secs=3600, expired_only=0
2019-05-16T16:40:29.427 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:29.420 7f3f32e1d6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:29.473 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:29.466 7f3f32e1d6c0 20 garbage collection: RGWGC::process entered with GC index_shard=17, max_secs=3600, expired_only=0
2019-05-16T16:40:29.547 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:29.540 7f3f32e1d6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:29.594 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:29.587 7f3f32e1d6c0 20 garbage collection: RGWGC::process entered with GC index_shard=18, max_secs=3600, expired_only=0
2019-05-16T16:40:29.678 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:29.672 7f3f32e1d6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:29.744 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:29.737 7f3f32e1d6c0 20 garbage collection: RGWGC::process entered with GC index_shard=19, max_secs=3600, expired_only=0
2019-05-16T16:40:29.828 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:29.821 7f3f32e1d6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:29.884 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:29.877 7f3f32e1d6c0 20 garbage collection: RGWGC::process entered with GC index_shard=20, max_secs=3600, expired_only=0
2019-05-16T16:40:29.971 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:29.964 7f3f32e1d6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:30.041 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:30.035 7f3f32e1d6c0 20 garbage collection: RGWGC::process entered with GC index_shard=21, max_secs=3600, expired_only=0
2019-05-16T16:40:30.122 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:30.115 7f3f32e1d6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:30.172 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:30.165 7f3f32e1d6c0 20 garbage collection: RGWGC::process entered with GC index_shard=22, max_secs=3600, expired_only=0
2019-05-16T16:40:30.269 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:30.261 7f3f32e1d6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:30.331 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:30.325 7f3f32e1d6c0 20 garbage collection: RGWGC::process entered with GC index_shard=23, max_secs=3600, expired_only=0
2019-05-16T16:40:30.402 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:30.396 7f3f32e1d6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:30.449 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:30.442 7f3f32e1d6c0 20 garbage collection: RGWGC::process entered with GC index_shard=24, max_secs=3600, expired_only=0
2019-05-16T16:40:30.530 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:30.523 7f3f32e1d6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:30.575 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:30.568 7f3f32e1d6c0 20 garbage collection: RGWGC::process entered with GC index_shard=25, max_secs=3600, expired_only=0
2019-05-16T16:40:30.667 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:30.661 7f3f32e1d6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:30.729 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:30.722 7f3f32e1d6c0 20 garbage collection: RGWGC::process entered with GC index_shard=26, max_secs=3600, expired_only=0
2019-05-16T16:40:30.818 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:30.811 7f3f32e1d6c0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker=''
2019-05-16T16:40:30.896 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:30.887 7f3f32e1d6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=0, size=0, entries=[]
2019-05-16T16:40:30.896 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:30.887 7f3f32e1d6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=1, size=0, entries=[]
2019-05-16T16:40:30.896 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:30.887 7f3f32e1d6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=2, size=0, entries=[]
2019-05-16T16:40:30.896 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:30.887 7f3f32e1d6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=3, size=0, entries=[]
2019-05-16T16:40:30.896 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:30.887 7f3f32e1d6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=4, size=0, entries=[]
2019-05-16T16:40:30.896 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:30.887 7f3f32e1d6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=5, size=0, entries=[]
2019-05-16T16:40:30.897 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:30.887 7f3f32e1d6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=6, size=0, entries=[]
2019-05-16T16:40:30.897 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:30.887 7f3f32e1d6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=7, size=0, entries=[]
2019-05-16T16:40:30.897 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:30.887 7f3f32e1d6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=8, size=0, entries=[]
2019-05-16T16:40:30.897 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:30.887 7f3f32e1d6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=9, size=0, entries=[]
2019-05-16T16:40:30.897 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:30.887 7f3f32e1d6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=10, size=0, entries=[]
2019-05-16T16:40:30.897 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:30.887 7f3f32e1d6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=11, size=0, entries=[]
2019-05-16T16:40:30.897 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:30.887 7f3f32e1d6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=12, size=0, entries=[]
2019-05-16T16:40:30.897 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:30.887 7f3f32e1d6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=13, size=0, entries=[]
2019-05-16T16:40:30.898 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:30.887 7f3f32e1d6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=14, size=0, entries=[]
2019-05-16T16:40:30.898 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:30.887 7f3f32e1d6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=15, size=0, entries=[]
2019-05-16T16:40:30.898 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:30.887 7f3f32e1d6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=16, size=0, entries=[]
2019-05-16T16:40:30.898 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:30.887 7f3f32e1d6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=17, size=0, entries=[]
2019-05-16T16:40:30.898 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:30.887 7f3f32e1d6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=18, size=0, entries=[]
2019-05-16T16:40:30.898 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:30.887 7f3f32e1d6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=19, size=0, entries=[]
2019-05-16T16:40:30.898 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:30.887 7f3f32e1d6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=20, size=0, entries=[]
2019-05-16T16:40:30.899 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:30.887 7f3f32e1d6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=21, size=0, entries=[]
2019-05-16T16:40:30.899 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:30.887 7f3f32e1d6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=22, size=0, entries=[]
2019-05-16T16:40:30.899 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:30.887 7f3f32e1d6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=23, size=0, entries=[]
2019-05-16T16:40:30.899 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:30.887 7f3f32e1d6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=24, size=0, entries=[]
2019-05-16T16:40:30.899 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:30.887 7f3f32e1d6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=25, size=0, entries=[]
2019-05-16T16:40:30.899 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:30.887 7f3f32e1d6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=26, size=0, entries=[]
2019-05-16T16:40:30.899 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:30.887 7f3f32e1d6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=27, size=0, entries=[]
2019-05-16T16:40:30.899 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:30.887 7f3f32e1d6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=28, size=0, entries=[]
2019-05-16T16:40:30.900 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:30.888 7f3f32e1d6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=29, size=0, entries=[]
2019-05-16T16:40:30.900 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:30.888 7f3f32e1d6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=30, size=0, entries=[]
2019-05-16T16:40:30.900 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:30.888 7f3f32e1d6c0 20 garbage collection: flush_remove_tags removing entries from gc log shard index=31, size=0, entries=[]
2019-05-16T16:40:31.209 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:31.203 7f3f32e1d6c0 20 remove_watcher() i=0
2019-05-16T16:40:31.209 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:31.203 7f3f32e1d6c0 2 removed watcher, disabling cache
2019-05-16T16:40:31.253 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:31.245 7f3f32e1d6c0 20 remove_watcher() i=1
2019-05-16T16:40:31.287 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:31.279 7f3f32e1d6c0 20 remove_watcher() i=2
2019-05-16T16:40:31.323 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:31.316 7f3f32e1d6c0 20 remove_watcher() i=3
2019-05-16T16:40:31.352 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:31.345 7f3f32e1d6c0 20 remove_watcher() i=4
2019-05-16T16:40:31.399 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:31.392 7f3f32e1d6c0 20 remove_watcher() i=5
2019-05-16T16:40:31.463 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:31.456 7f3f32e1d6c0 20 remove_watcher() i=6
2019-05-16T16:40:31.516 INFO:teuthology.orchestra.run.smithi165.stderr:2019-05-16 16:40:31.509 7f3f32e1d6c0 20 remove_watcher() i=7
2019-05-16T16:40:31.525 INFO:teuthology.orchestra.run.smithi165:Running:
2019-05-16T16:40:31.525 INFO:teuthology.orchestra.run.smithi165:> cd /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle-4.7/bin/gradle clean test --rerun-tasks --no-build-cache --tests BucketTest
2019-05-16T16:40:32.057 INFO:teuthology.orchestra.run.smithi165.stdout:> Task :clean
2019-05-16T16:40:32.850 INFO:teuthology.orchestra.run.smithi165.stdout:> Task :compileJava
2019-05-16T16:40:32.850 INFO:teuthology.orchestra.run.smithi165.stdout:> Task :processResources
2019-05-16T16:40:32.851 INFO:teuthology.orchestra.run.smithi165.stdout:> Task :classes
2019-05-16T16:40:33.149 INFO:teuthology.orchestra.run.smithi165.stdout:
2019-05-16T16:40:33.150 INFO:teuthology.orchestra.run.smithi165.stdout:> Task :compileTestJava
2019-05-16T16:40:33.150 INFO:teuthology.orchestra.run.smithi165.stdout:Note: /home/ubuntu/cephtest/s3-tests-java/src/test/java/BucketTest.java uses or overrides a deprecated API.
2019-05-16T16:40:33.150 INFO:teuthology.orchestra.run.smithi165.stdout:Note: Recompile with -Xlint:deprecation for details.
2019-05-16T16:40:33.150 INFO:teuthology.orchestra.run.smithi165.stdout:
2019-05-16T16:40:33.150 INFO:teuthology.orchestra.run.smithi165.stdout:> Task :processTestResources NO-SOURCE
2019-05-16T16:40:33.150 INFO:teuthology.orchestra.run.smithi165.stdout:> Task :testClasses
2019-05-16T16:41:03.148 INFO:teuthology.orchestra.run.smithi165.stdout:> Task :test
2019-05-16T16:46:58.418 INFO:tasks.ceph.mon.b.smithi008.stderr:==00:00:42:39.000 9786== Warning: unimplemented fcntl command: 1036
2019-05-16T16:46:58.460 INFO:tasks.ceph.mon.c.smithi165.stderr:==00:00:42:39.092 9951== Warning: unimplemented fcntl command: 1036
2019-05-16T16:46:58.474 INFO:tasks.ceph.mon.b.smithi008.stderr:==00:00:42:39.055 9786== Warning: unimplemented fcntl command: 1036
2019-05-16T16:46:58.497 INFO:tasks.ceph.mon.a.smithi165.stderr:==00:00:42:39.129 9924== Warning: unimplemented fcntl command: 1036
2019-05-16T16:46:58.503 INFO:tasks.ceph.mon.a.smithi165.stderr:==00:00:42:39.136 9924== Warning: unimplemented fcntl command: 1036
2019-05-16T16:46:58.512 INFO:tasks.ceph.mon.c.smithi165.stderr:==00:00:42:39.144 9951== Warning: unimplemented fcntl command: 1036
2019-05-16T16:46:58.853 INFO:tasks.ceph.mon.c.smithi165.stderr:==00:00:42:39.485 9951== Warning: unimplemented fcntl command: 1036
2019-05-16T16:46:58.895 INFO:tasks.ceph.mon.a.smithi165.stderr:==00:00:42:39.528 9924== Warning: unimplemented fcntl command: 1036
2019-05-16T16:46:58.953 INFO:tasks.ceph.mon.b.smithi008.stderr:==00:00:42:39.535 9786== Warning: unimplemented fcntl command: 1036
2019-05-16T16:47:30.840 INFO:tasks.ceph.mon.b.smithi008.stderr:==00:00:43:11.421 9786== Warning: unimplemented fcntl command: 1036
2019-05-16T16:47:30.862 INFO:tasks.ceph.mon.b.smithi008.stderr:==00:00:43:11.444 9786== Warning: unimplemented fcntl command: 1036
2019-05-16T16:47:30.869 INFO:tasks.ceph.mon.c.smithi165.stderr:==00:00:43:11.501 9951== Warning: unimplemented fcntl command: 1036
2019-05-16T16:47:30.872 INFO:tasks.ceph.mon.a.smithi165.stderr:==00:00:43:11.505 9924== Warning: unimplemented fcntl command: 1036
2019-05-16T16:47:30.878 INFO:tasks.ceph.mon.c.smithi165.stderr:==00:00:43:11.511 9951== Warning: unimplemented fcntl command: 1036
2019-05-16T16:47:30.883 INFO:tasks.ceph.mon.a.smithi165.stderr:==00:00:43:11.516 9924== Warning: unimplemented fcntl command: 1036
2019-05-16T16:47:30.942 INFO:tasks.ceph.mon.c.smithi165.stderr:==00:00:43:11.574 9951== Warning: unimplemented fcntl command: 1036
2019-05-16T16:47:30.942 INFO:tasks.ceph.mon.a.smithi165.stderr:==00:00:43:11.575 9924== Warning: unimplemented fcntl command: 1036
2019-05-16T16:47:30.980 INFO:tasks.ceph.mon.b.smithi008.stderr:==00:00:43:11.562 9786== Warning: unimplemented fcntl command: 1036
2019-05-16T16:54:27.042 INFO:tasks.ceph.mon.b.smithi008.stderr:==00:00:50:07.625 9786== Warning: unimplemented fcntl command: 1036
2019-05-16T16:54:27.048 INFO:tasks.ceph.mon.b.smithi008.stderr:==00:00:50:07.630 9786== Warning: unimplemented fcntl command: 1036
2019-05-16T16:54:27.064 INFO:tasks.ceph.mon.a.smithi165.stderr:==00:00:50:07.698 9924== Warning: unimplemented fcntl command: 1036
2019-05-16T16:54:27.066 INFO:tasks.ceph.mon.c.smithi165.stderr:==00:00:50:07.699 9951== Warning: unimplemented fcntl command: 1036
2019-05-16T16:54:27.071 INFO:tasks.ceph.mon.a.smithi165.stderr:==00:00:50:07.704 9924== Warning: unimplemented fcntl command: 1036
2019-05-16T16:54:27.087 INFO:tasks.ceph.mon.c.smithi165.stderr:==00:00:50:07.720 9951== Warning: unimplemented fcntl command: 1036
2019-05-16T16:54:27.320 INFO:tasks.ceph.mon.a.smithi165.stderr:==00:00:50:07.953 9924== Warning: unimplemented fcntl command: 1036
2019-05-16T16:54:27.328 INFO:tasks.ceph.mon.c.smithi165.stderr:==00:00:50:07.962 9951== Warning: unimplemented fcntl command: 1036
2019-05-16T16:54:27.403 INFO:tasks.ceph.mon.b.smithi008.stderr:==00:00:50:07.986 9786== Warning: unimplemented fcntl command: 1036
2019-05-16T16:55:19.537 INFO:tasks.ceph.mon.b.smithi008.stderr:==00:00:51:00.120 9786== Warning: unimplemented fcntl command: 1036
2019-05-16T16:55:19.541 INFO:tasks.ceph.mon.b.smithi008.stderr:==00:00:51:00.124 9786== Warning: unimplemented fcntl command: 1036
2019-05-16T16:55:19.581 INFO:tasks.ceph.mon.b.smithi008.stderr:==00:00:51:00.164 9786== Warning: unimplemented fcntl command: 1036
2019-05-16T16:55:19.608 INFO:tasks.ceph.mon.a.smithi165.stderr:==00:00:51:00.241 9924== Warning: unimplemented fcntl command: 1036
2019-05-16T16:55:19.611 INFO:tasks.ceph.mon.a.smithi165.stderr:==00:00:51:00.245 9924== Warning: unimplemented fcntl command: 1036
2019-05-16T16:55:19.616 INFO:tasks.ceph.mon.c.smithi165.stderr:==00:00:51:00.249 9951== Warning: unimplemented fcntl command: 1036
2019-05-16T16:55:19.636 INFO:tasks.ceph.mon.c.smithi165.stderr:==00:00:51:00.269 9951== Warning: unimplemented fcntl command: 1036
2019-05-16T16:55:19.688 INFO:tasks.ceph.mon.a.smithi165.stderr:==00:00:51:00.321 9924== Warning: unimplemented fcntl command: 1036
2019-05-16T16:55:19.689 INFO:tasks.ceph.mon.c.smithi165.stderr:==00:00:51:00.322 9951== Warning: unimplemented fcntl command: 1036
2019-05-16T16:56:13.748 INFO:teuthology.orchestra.run.smithi165.stdout:
2019-05-16T16:56:13.748 INFO:teuthology.orchestra.run.smithi165.stdout:suite > Bucket tests > BucketTest.testBucketCreateBadExpectEmpty STARTED
2019-05-16T16:56:13.748 INFO:teuthology.orchestra.run.smithi165.stdout:
2019-05-16T16:56:13.748 INFO:teuthology.orchestra.run.smithi165.stdout:suite > Bucket tests > BucketTest.testBucketCreateBadExpectEmpty FAILED
2019-05-16T16:56:13.748 INFO:teuthology.orchestra.run.smithi165.stdout: com.amazonaws.SdkClientException at BucketTest.java:147
2019-05-16T16:56:13.749 INFO:teuthology.orchestra.run.smithi165.stdout: Caused by: org.apache.http.conn.HttpHostConnectException at BucketTest.java:147
2019-05-16T16:56:13.749 INFO:teuthology.orchestra.run.smithi165.stdout: Caused by: java.net.ConnectException at BucketTest.java:147
2019-05-16T16:57:03.748 INFO:teuthology.orchestra.run.smithi165.stdout:
2019-05-16T16:57:03.748 INFO:teuthology.orchestra.run.smithi165.stdout:suite > Bucket tests > BucketTest.testBucketCreateBadExpectMismatch STARTED
2019-05-16T16:57:03.748 INFO:teuthology.orchestra.run.smithi165.stdout:
2019-05-16T16:57:03.748 INFO:teuthology.orchestra.run.smithi165.stdout:suite > Bucket tests > BucketTest.testBucketCreateBadExpectMismatch FAILED
2019-05-16T16:57:03.748 INFO:teuthology.orchestra.run.smithi165.stdout: com.amazonaws.SdkClientException at BucketTest.java:137
2019-05-16T16:57:03.749 INFO:teuthology.orchestra.run.smithi165.stdout: Caused by: org.apache.http.conn.HttpHostConnectException at BucketTest.java:137
2019-05-16T16:57:03.749 INFO:teuthology.orchestra.run.smithi165.stdout: Caused by: java.net.ConnectException at BucketTest.java:137
2019-05-16T16:57:53.848 INFO:teuthology.orchestra.run.smithi165.stdout:
2019-05-16T16:57:53.848 INFO:teuthology.orchestra.run.smithi165.stdout:suite > Bucket tests > BucketTest.testBucketCreateBadExpectUnreadable STARTED
2019-05-16T16:57:53.848 INFO:teuthology.orchestra.run.smithi165.stdout:
2019-05-16T16:57:53.848 INFO:teuthology.orchestra.run.smithi165.stdout:suite > Bucket tests > BucketTest.testBucketCreateBadExpectUnreadable FAILED
2019-05-16T16:57:53.849 INFO:teuthology.orchestra.run.smithi165.stdout: com.amazonaws.SdkClientException at BucketTest.java:157
2019-05-16T16:57:53.849 INFO:teuthology.orchestra.run.smithi165.stdout: Caused by: org.apache.http.conn.HttpHostConnectException at BucketTest.java:157
2019-05-16T16:57:53.849 INFO:teuthology.orchestra.run.smithi165.stdout: Caused by: java.net.ConnectException at BucketTest.java:157
2019-05-16T16:58:43.848 INFO:teuthology.orchestra.run.smithi165.stdout:
2019-05-16T16:58:43.849 INFO:teuthology.orchestra.run.smithi165.stdout:suite > Bucket tests > BucketTest.testBucketCreateReadDelete STARTED
2019-05-16T16:58:43.948 INFO:teuthology.orchestra.run.smithi165.stdout:
2019-05-16T16:58:43.948 INFO:teuthology.orchestra.run.smithi165.stdout:suite > Bucket tests > BucketTest.testBucketCreateReadDelete FAILED
2019-05-16T16:58:43.948 INFO:teuthology.orchestra.run.smithi165.stdout: com.amazonaws.SdkClientException at BucketTest.java:91
2019-05-16T16:58:43.948 INFO:teuthology.orchestra.run.smithi165.stdout: Caused by: org.apache.http.conn.HttpHostConnectException at BucketTest.java:91
2019-05-16T16:58:43.948 INFO:teuthology.orchestra.run.smithi165.stdout: Caused by: java.net.ConnectException at BucketTest.java:91
2019-05-16T16:59:33.948 INFO:teuthology.orchestra.run.smithi165.stdout:
2019-05-16T16:59:33.948 INFO:teuthology.orchestra.run.smithi165.stdout:suite > Bucket tests > BucketTest.testBucketDeleteNonEmpty STARTED
2019-05-16T16:59:33.948 INFO:teuthology.orchestra.run.smithi165.stdout:
2019-05-16T16:59:33.949 INFO:teuthology.orchestra.run.smithi165.stdout:suite > Bucket tests > BucketTest.testBucketDeleteNonEmpty FAILED
2019-05-16T16:59:33.949 INFO:teuthology.orchestra.run.smithi165.stdout: com.amazonaws.SdkClientException at BucketTest.java:75
2019-05-16T16:59:33.949 INFO:teuthology.orchestra.run.smithi165.stdout: Caused by: org.apache.http.conn.HttpHostConnectException at BucketTest.java:75
2019-05-16T16:59:33.949 INFO:teuthology.orchestra.run.smithi165.stdout: Caused by: java.net.ConnectException at BucketTest.java:75
2019-05-16T17:00:24.048 INFO:teuthology.orchestra.run.smithi165.stdout:
2019-05-16T17:00:24.048 INFO:teuthology.orchestra.run.smithi165.stdout:suite > Bucket tests > BucketTest.testBucketDeleteNotExist STARTED
2019-05-16T17:00:24.048 INFO:teuthology.orchestra.run.smithi165.stdout:
2019-05-16T17:00:24.048 INFO:teuthology.orchestra.run.smithi165.stdout:suite > Bucket tests > BucketTest.testBucketDeleteNotExist FAILED
2019-05-16T17:00:24.049 INFO:teuthology.orchestra.run.smithi165.stdout: com.amazonaws.SdkClientException at BucketTest.java:64
2019-05-16T17:00:24.049 INFO:teuthology.orchestra.run.smithi165.stdout: Caused by: org.apache.http.conn.HttpHostConnectException at BucketTest.java:64
2019-05-16T17:00:24.049 INFO:teuthology.orchestra.run.smithi165.stdout: Caused by: java.net.ConnectException at BucketTest.java:64
2019-05-16T17:01:14.048 INFO:teuthology.orchestra.run.smithi165.stdout:
2019-05-16T17:01:14.048 INFO:teuthology.orchestra.run.smithi165.stdout:suite > Bucket tests > BucketTest.testBucketListDistinct STARTED
2019-05-16T17:01:14.049 INFO:teuthology.orchestra.run.smithi165.stdout:
2019-05-16T17:01:14.049 INFO:teuthology.orchestra.run.smithi165.stdout:suite > Bucket tests > BucketTest.testBucketListDistinct FAILED
2019-05-16T17:01:14.049 INFO:teuthology.orchestra.run.smithi165.stdout: com.amazonaws.SdkClientException at BucketTest.java:108
2019-05-16T17:01:14.049 INFO:teuthology.orchestra.run.smithi165.stdout: Caused by: org.apache.http.conn.HttpHostConnectException at BucketTest.java:108
2019-05-16T17:01:14.049 INFO:teuthology.orchestra.run.smithi165.stdout: Caused by: java.net.ConnectException at BucketTest.java:108
2019-05-16T17:01:21.361 INFO:tasks.ceph.mon.b.smithi008.stderr:==00:00:57:01.945 9786== Warning: unimplemented fcntl command: 1036
2019-05-16T17:01:21.373 INFO:tasks.ceph.mon.b.smithi008.stderr:==00:00:57:01.957 9786== Warning: unimplemented fcntl command: 1036
2019-05-16T17:01:21.395 INFO:tasks.ceph.mon.a.smithi165.stderr:==00:00:57:02.028 9924== Warning: unimplemented fcntl command: 1036
2019-05-16T17:01:21.399 INFO:tasks.ceph.mon.c.smithi165.stderr:==00:00:57:02.032 9951== Warning: unimplemented fcntl command: 1036
2019-05-16T17:01:21.401 INFO:tasks.ceph.mon.a.smithi165.stderr:==00:00:57:02.034 9924== Warning: unimplemented fcntl command: 1036
2019-05-16T17:01:21.411 INFO:tasks.ceph.mon.c.smithi165.stderr:==00:00:57:02.044 9951== Warning: unimplemented fcntl command: 1036
2019-05-16T17:01:21.589 INFO:tasks.ceph.mon.b.smithi008.stderr:==00:00:57:02.173 9786== Warning: unimplemented fcntl command: 1036
2019-05-16T17:01:21.695 INFO:tasks.ceph.mon.c.smithi165.stderr:==00:00:57:02.328 9951== Warning: unimplemented fcntl command: 1036
2019-05-16T17:01:21.714 INFO:tasks.ceph.mon.a.smithi165.stderr:==00:00:57:02.347 9924== Warning: unimplemented fcntl command: 1036
2019-05-16T17:02:04.148 INFO:teuthology.orchestra.run.smithi165.stdout:
2019-05-16T17:02:04.148 INFO:teuthology.orchestra.run.smithi165.stdout:suite > Bucket tests > BucketTest.testBucketListEmpty STARTED
2019-05-16T17:02:04.148 INFO:teuthology.orchestra.run.smithi165.stdout:
2019-05-16T17:02:04.149 INFO:teuthology.orchestra.run.smithi165.stdout:suite > Bucket tests > BucketTest.testBucketListEmpty FAILED
2019-05-16T17:02:04.149 INFO:teuthology.orchestra.run.smithi165.stdout: com.amazonaws.SdkClientException at BucketTest.java:53
2019-05-16T17:02:04.149 INFO:teuthology.orchestra.run.smithi165.stdout: Caused by: org.apache.http.conn.HttpHostConnectException at BucketTest.java:53
2019-05-16T17:02:04.149 INFO:teuthology.orchestra.run.smithi165.stdout: Caused by: java.net.ConnectException at BucketTest.java:53
2019-05-16T17:02:54.148 INFO:teuthology.orchestra.run.smithi165.stdout:
2019-05-16T17:02:54.148 INFO:teuthology.orchestra.run.smithi165.stdout:suite > Bucket tests > BucketTest.testBucketNotExist STARTED
2019-05-16T17:02:54.248 INFO:teuthology.orchestra.run.smithi165.stdout:
2019-05-16T17:02:54.248 INFO:teuthology.orchestra.run.smithi165.stdout:suite > Bucket tests > BucketTest.testBucketNotExist FAILED
2019-05-16T17:02:54.248 INFO:teuthology.orchestra.run.smithi165.stdout: com.amazonaws.SdkClientException at BucketTest.java:123
2019-05-16T17:02:54.248 INFO:teuthology.orchestra.run.smithi165.stdout: Caused by: org.apache.http.conn.HttpHostConnectException at BucketTest.java:123
2019-05-16T17:02:54.249 INFO:teuthology.orchestra.run.smithi165.stdout: Caused by: java.net.ConnectException at BucketTest.java:123
2019-05-16T17:03:40.639 INFO:tasks.ceph.mon.b.smithi008.stderr:==00:00:59:21.223 9786== Warning: unimplemented fcntl command: 1036
2019-05-16T17:03:40.646 INFO:tasks.ceph.mon.b.smithi008.stderr:==00:00:59:21.231 9786== Warning: unimplemented fcntl command: 1036
2019-05-16T17:03:40.717 INFO:tasks.ceph.mon.c.smithi165.stderr:==00:00:59:21.350 9951== Warning: unimplemented fcntl command: 1036
2019-05-16T17:03:40.718 INFO:tasks.ceph.mon.a.smithi165.stderr:==00:00:59:21.351 9924== Warning: unimplemented fcntl command: 1036
2019-05-16T17:03:40.720 INFO:tasks.ceph.mon.c.smithi165.stderr:==00:00:59:21.352 9951== Warning: unimplemented fcntl command: 1036
2019-05-16T17:03:40.722 INFO:tasks.ceph.mon.a.smithi165.stderr:==00:00:59:21.355 9924== Warning: unimplemented fcntl command: 1036
2019-05-16T17:03:40.750 INFO:tasks.ceph.mon.b.smithi008.stderr:==00:00:59:21.334 9786== Warning: unimplemented fcntl command: 1036
2019-05-16T17:03:40.806 INFO:tasks.ceph.mon.c.smithi165.stderr:==00:00:59:21.439 9951== Warning: unimplemented fcntl command: 1036
2019-05-16T17:03:40.833 INFO:tasks.ceph.mon.a.smithi165.stderr:==00:00:59:21.465 9924== Warning: unimplemented fcntl command: 1036
2019-05-16T17:03:44.248 INFO:teuthology.orchestra.run.smithi165.stdout:
2019-05-16T17:03:44.248 INFO:teuthology.orchestra.run.smithi165.stdout:--------------------------------------------------------------------
2019-05-16T17:03:44.248 INFO:teuthology.orchestra.run.smithi165.stdout:| Results: FAILURE (9 tests, 0 successes, 9 failures, 0 skipped) |
2019-05-16T17:03:44.248 INFO:teuthology.orchestra.run.smithi165.stdout:--------------------------------------------------------------------
2019-05-16T17:03:44.350 INFO:teuthology.orchestra.run.smithi165.stdout:
2019-05-16T17:03:44.351 INFO:teuthology.orchestra.run.smithi165.stdout:9 tests completed, 9 failed
2019-05-16T17:03:44.351 INFO:teuthology.orchestra.run.smithi165.stdout:
2019-05-16T17:03:44.351 INFO:teuthology.orchestra.run.smithi165.stdout:> Task :test FAILED
2019-05-16T17:03:44.351 INFO:teuthology.orchestra.run.smithi165.stderr:
2019-05-16T17:03:44.351 INFO:teuthology.orchestra.run.smithi165.stderr:FAILURE: Build failed with an exception.
2019-05-16T17:03:44.352 INFO:teuthology.orchestra.run.smithi165.stderr:
2019-05-16T17:03:44.352 INFO:teuthology.orchestra.run.smithi165.stderr:* What went wrong:
2019-05-16T17:03:44.352 INFO:teuthology.orchestra.run.smithi165.stderr:Execution failed for task ':test'.
2019-05-16T17:03:44.352 INFO:teuthology.orchestra.run.smithi165.stderr:> There were failing tests. See the report at: file:///home/ubuntu/cephtest/s3-tests-java/build/reports/tests/test/index.html
2019-05-16T17:03:44.352 INFO:teuthology.orchestra.run.smithi165.stderr:
2019-05-16T17:03:44.353 INFO:teuthology.orchestra.run.smithi165.stderr:* Try:
2019-05-16T17:03:44.354 INFO:teuthology.orchestra.run.smithi165.stderr:Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
2019-05-16T17:03:44.355 INFO:teuthology.orchestra.run.smithi165.stderr:
2019-05-16T17:03:44.355 INFO:teuthology.orchestra.run.smithi165.stderr:* Get more help at https://help.gradle.org
2019-05-16T17:03:44.355 INFO:teuthology.orchestra.run.smithi165.stdout:
2019-05-16T17:03:44.355 INFO:teuthology.orchestra.run.smithi165.stdout:BUILD FAILED in 23m 12s
2019-05-16T17:03:44.355 INFO:teuthology.orchestra.run.smithi165.stdout:5 actionable tasks: 5 executed
2019-05-16T17:03:44.377 DEBUG:teuthology.orchestra.run:got remote process result: 1
2019-05-16T17:03:44.377 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 89, in run_tasks
manager.__enter__()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/__init__.py", line 123, in __enter__
self.begin()
File "/home/teuthworker/src/github.com_alimaredia_ceph_wip-s3java/qa/tasks/s3tests_java.py", line 97, in begin
self.run_tests()
File "/home/teuthworker/src/github.com_alimaredia_ceph_wip-s3java/qa/tasks/s3tests_java.py", line 358, in run_tests
stdout=StringIO()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/cluster.py", line 64, in run
return [remote.run(**kwargs) for remote in remotes]
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 205, in run
r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 437, in run
r.wait()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 162, in wait
self._raise_for_status()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 184, in _raise_for_status
node=self.hostname, label=self.label
CommandFailedError: Command failed on smithi165 with status 1: 'cd /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle-4.7/bin/gradle clean test --rerun-tasks --no-build-cache --tests BucketTest'
2019-05-16T17:03:44.379 DEBUG:teuthology.run_tasks:Unwinding manager s3tests-java
2019-05-16T17:03:44.394 DEBUG:tasks.s3tests_java:S3 Tests Java: END
2019-05-16T17:03:44.394 INFO:tasks.s3tests_java:S3 Tests Java: Cleaning up s3-tests-java...
2019-05-16T17:03:44.395 INFO:teuthology.orchestra.run.smithi165:Running:
2019-05-16T17:03:44.395 INFO:teuthology.orchestra.run.smithi165:> rm -rf /home/ubuntu/cephtest/s3-tests-java
2019-05-16T17:03:44.556 INFO:tasks.s3tests_java:S3 Tests Java: Deleting S3 users...
(1-1/2)