Ceph » rgw

Hi,

The RGW instances in our cluster are crashing regularly with this message. They can be restarted, but will crash after a short period even when not exposed to client requests. It seems to be down to our multi-site replication, but the other sites are fine. Here's the relevant log entries, what can we do to troubleshoot or figure out what the root cause is here .

 -338> 2022-12-09T18:04:32.916+0000 7f6980719700  1 ====== starting new request req=0x7f682c7e85c0 =====
 -337> 2022-12-09T18:04:32.916+0000 7f6980719700  2 req 9777807639303135910 0.000000000s initializing for trans_id = tx0000087b1c03e2ec3a2a6-00639378b0-b992fa-nvan
 -336> 2022-12-09T18:04:32.916+0000 7f6980719700  2 req 9777807639303135910 0.000000000s getting op 0
 -335> 2022-12-09T18:04:32.916+0000 7f6980719700  2 req 9777807639303135910 0.000000000s s3:get_obj verifying requester
 -334> 2022-12-09T18:04:32.916+0000 7f6980719700  2 req 9777807639303135910 0.000000000s s3:get_obj normalizing buckets and tenants
 -333> 2022-12-09T18:04:32.916+0000 7f6980719700  2 req 9777807639303135910 0.000000000s s3:get_obj init permissions
 -332> 2022-12-09T18:04:32.916+0000 7f6980719700  2 req 9777807639303135910 0.000000000s s3:get_obj recalculating target
 -331> 2022-12-09T18:04:32.916+0000 7f6980719700  2 req 9777807639303135910 0.000000000s s3:get_obj reading permissions
 -330> 2022-12-09T18:04:32.927+0000 7f6993f40700  2 req 10732979668358407264 0.140001565s s3:get_obj init op
 -329> 2022-12-09T18:04:32.927+0000 7f6993f40700  2 req 10732979668358407264 0.140001565s s3:get_obj verifying op mask
 -328> 2022-12-09T18:04:32.927+0000 7f6993f40700  2 req 10732979668358407264 0.140001565s s3:get_obj verifying op permissions
 -327> 2022-12-09T18:04:32.927+0000 7f6993f40700  5 req 10732979668358407264 0.140001565s s3:get_obj Searching permissions for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=synchronization-user, acct_name=synchronization-user, subuser=, perm_mask=15, is_admin=0) mask=49
 -326> 2022-12-09T18:04:32.927+0000 7f6993f40700  5 req 10732979668358407264 0.140001565s s3:get_obj Searching permissions for uid=synchronization-user
 -325> 2022-12-09T18:04:32.927+0000 7f6993f40700  5 req 10732979668358407264 0.140001565s s3:get_obj Permissions for user not found
 -324> 2022-12-09T18:04:32.927+0000 7f6993f40700  5 req 10732979668358407264 0.140001565s s3:get_obj Searching permissions for group=1 mask=49
 -323> 2022-12-09T18:04:32.927+0000 7f6993f40700  5 req 10732979668358407264 0.140001565s s3:get_obj Permissions for group not found
 -322> 2022-12-09T18:04:32.927+0000 7f6993f40700  5 req 10732979668358407264 0.140001565s s3:get_obj Searching permissions for group=2 mask=49
 -321> 2022-12-09T18:04:32.927+0000 7f6993f40700  5 req 10732979668358407264 0.140001565s s3:get_obj Permissions for group not found
 -320> 2022-12-09T18:04:32.927+0000 7f6993f40700  5 req 10732979668358407264 0.140001565s s3:get_obj -- Getting permissions done for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=synchronization-user, acct_name=synchronization-user, subuser=, perm_mask=15, is_admin=0), owner=admin, perm=0
 -319> 2022-12-09T18:04:32.927+0000 7f6993f40700  5 req 10732979668358407264 0.140001565s s3:get_obj Searching permissions for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=synchronization-user, acct_name=synchronization-user, subuser=, perm_mask=15, is_admin=0) mask=48
 -318> 2022-12-09T18:04:32.927+0000 7f6993f40700  5 req 10732979668358407264 0.140001565s s3:get_obj Searching permissions for uid=synchronization-user
 -317> 2022-12-09T18:04:32.927+0000 7f6993f40700  5 req 10732979668358407264 0.140001565s s3:get_obj Permissions for user not found
 -316> 2022-12-09T18:04:32.927+0000 7f6993f40700  5 req 10732979668358407264 0.140001565s s3:get_obj Searching permissions for group=1 mask=48
 -315> 2022-12-09T18:04:32.927+0000 7f6993f40700  5 req 10732979668358407264 0.140001565s s3:get_obj Permissions for group not found
 -314> 2022-12-09T18:04:32.927+0000 7f6993f40700  5 req 10732979668358407264 0.140001565s s3:get_obj Searching permissions for group=2 mask=48
 -313> 2022-12-09T18:04:32.927+0000 7f6993f40700  5 req 10732979668358407264 0.140001565s s3:get_obj Permissions for group not found
 -312> 2022-12-09T18:04:32.927+0000 7f6993f40700  5 req 10732979668358407264 0.140001565s s3:get_obj -- Getting permissions done for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=synchronization-user, acct_name=synchronization-user, subuser=, perm_mask=15, is_admin=0), owner=admin, perm=0
 -311> 2022-12-09T18:04:32.927+0000 7f6993f40700  2 overriding permissions due to system operation
 -310> 2022-12-09T18:04:32.927+0000 7f6993f40700  2 req 10732979668358407264 0.140001565s s3:get_obj verifying op params
 -309> 2022-12-09T18:04:32.927+0000 7f6993f40700  2 req 10732979668358407264 0.140001565s s3:get_obj pre-executing
 -308> 2022-12-09T18:04:32.927+0000 7f6993f40700  2 req 10732979668358407264 0.140001565s s3:get_obj check rate limiting
 -307> 2022-12-09T18:04:32.927+0000 7f6993f40700  2 req 10732979668358407264 0.140001565s s3:get_obj executing
 -306> 2022-12-09T18:04:32.927+0000 7f6993f40700  2 req 10732979668358407264 0.140001565s s3:get_obj completing
 -305> 2022-12-09T18:04:32.928+0000 7f6993f40700  2 req 10732979668358407264 0.141001567s s3:get_obj op status=-2016
 -304> 2022-12-09T18:04:32.928+0000 7f6993f40700  2 req 10732979668358407264 0.141001567s s3:get_obj http status=304
 -303> 2022-12-09T18:04:32.928+0000 7f6993f40700  1 ====== req done req=0x7f682c35f5c0 op status=-2016 http_status=304 latency=0.141001567s ======
 -302> 2022-12-09T18:04:32.928+0000 7f6993f40700  1 beast: 0x7f682c35f5c0: 10.200.125.13 - synchronization-user [09/Dec/2022:18:04:32.787 +0000] "GET /mimic.da.spoolers.perf3/0OQNSl04%2F1517.hydozRLPKmLv0l4W.rnd?rgwx-zonegroup=56082fa6-e41f-40aa-9c45-b83b869effd1&rgwx-prepend-metadata=true&rgwx-sync-manifest&rgwx-skip-decrypt HTTP/1.1" 304 0 - - - latency=0.141001567s
 -301> 2022-12-09T18:04:32.970+0000 7f6980f1a700  2 req 3767462689007815227 0.077000856s s3:get_obj init op
 -300> 2022-12-09T18:04:32.970+0000 7f6980f1a700  2 req 3767462689007815227 0.077000856s s3:get_obj verifying op mask
 -299> 2022-12-09T18:04:32.970+0000 7f6980f1a700  2 req 3767462689007815227 0.077000856s s3:get_obj verifying op permissions
 -298> 2022-12-09T18:04:32.970+0000 7f6980f1a700  5 req 3767462689007815227 0.077000856s s3:get_obj Searching permissions for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=synchronization-user, acct_name=synchronization-user, subuser=, perm_mask=15, is_admin=0) mask=49
 -297> 2022-12-09T18:04:32.970+0000 7f6980f1a700  5 req 3767462689007815227 0.077000856s s3:get_obj Searching permissions for uid=synchronization-user
 -296> 2022-12-09T18:04:32.970+0000 7f6980f1a700  5 req 3767462689007815227 0.077000856s s3:get_obj Permissions for user not found
 -295> 2022-12-09T18:04:32.970+0000 7f6980f1a700  5 req 3767462689007815227 0.077000856s s3:get_obj Searching permissions for group=1 mask=49
 -294> 2022-12-09T18:04:32.970+0000 7f6980f1a700  5 req 3767462689007815227 0.077000856s s3:get_obj Permissions for group not found
 -293> 2022-12-09T18:04:32.970+0000 7f6980f1a700  5 req 3767462689007815227 0.077000856s s3:get_obj Searching permissions for group=2 mask=49
 -292> 2022-12-09T18:04:32.970+0000 7f6980f1a700  5 req 3767462689007815227 0.077000856s s3:get_obj Permissions for group not found
 -291> 2022-12-09T18:04:32.970+0000 7f6980f1a700  5 req 3767462689007815227 0.077000856s s3:get_obj -- Getting permissions done for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=synchronization-user, acct_name=synchronization-user, subuser=, perm_mask=15, is_admin=0), owner=admin, perm=0
 -290> 2022-12-09T18:04:32.970+0000 7f6980f1a700  5 req 3767462689007815227 0.077000856s s3:get_obj Searching permissions for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=synchronization-user, acct_name=synchronization-user, subuser=, perm_mask=15, is_admin=0) mask=48
 -289> 2022-12-09T18:04:32.970+0000 7f6980f1a700  5 req 3767462689007815227 0.077000856s s3:get_obj Searching permissions for uid=synchronization-user
 -288> 2022-12-09T18:04:32.970+0000 7f6980f1a700  5 req 3767462689007815227 0.077000856s s3:get_obj Permissions for user not found
 -287> 2022-12-09T18:04:32.970+0000 7f6980f1a700  5 req 3767462689007815227 0.077000856s s3:get_obj Searching permissions for group=1 mask=48
 -286> 2022-12-09T18:04:32.970+0000 7f6980f1a700  5 req 3767462689007815227 0.077000856s s3:get_obj Permissions for group not found
 -285> 2022-12-09T18:04:32.970+0000 7f6980f1a700  5 req 3767462689007815227 0.077000856s s3:get_obj Searching permissions for group=2 mask=48
 -284> 2022-12-09T18:04:32.970+0000 7f6980f1a700  5 req 3767462689007815227 0.077000856s s3:get_obj Permissions for group not found
 -283> 2022-12-09T18:04:32.970+0000 7f6980f1a700  5 req 3767462689007815227 0.077000856s s3:get_obj -- Getting permissions done for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=synchronization-user, acct_name=synchronization-user, subuser=, perm_mask=15, is_admin=0), owner=admin, perm=0
 -282> 2022-12-09T18:04:32.970+0000 7f6980f1a700  2 overriding permissions due to system operation
 -281> 2022-12-09T18:04:32.970+0000 7f6980f1a700  2 req 3767462689007815227 0.077000856s s3:get_obj verifying op params
 -280> 2022-12-09T18:04:32.970+0000 7f6980f1a700  2 req 3767462689007815227 0.077000856s s3:get_obj pre-executing
 -279> 2022-12-09T18:04:32.970+0000 7f6980f1a700  2 req 3767462689007815227 0.077000856s s3:get_obj check rate limiting
 -278> 2022-12-09T18:04:32.970+0000 7f6980f1a700  2 req 3767462689007815227 0.077000856s s3:get_obj executing
 -277> 2022-12-09T18:04:32.970+0000 7f6980f1a700  2 req 3767462689007815227 0.077000856s s3:get_obj completing
 -276> 2022-12-09T18:04:32.970+0000 7f6980f1a700  2 req 3767462689007815227 0.077000856s s3:get_obj op status=-2016
 -275> 2022-12-09T18:04:32.970+0000 7f6980f1a700  2 req 3767462689007815227 0.077000856s s3:get_obj http status=304
 -274> 2022-12-09T18:04:32.970+0000 7f6980f1a700  1 ====== req done req=0x7f6b6fde75c0 op status=-2016 http_status=304 latency=0.077000856s ======
 -273> 2022-12-09T18:04:32.971+0000 7f6980f1a700  1 beast: 0x7f6b6fde75c0: 10.200.125.14 - synchronization-user [09/Dec/2022:18:04:32.893 +0000] "GET /mimic.da.spoolers.perf3/1SJ1eyPp%2F7293.c)ZwwC9CAzfuj2vn.rnd?rgwx-zonegroup=56082fa6-e41f-40aa-9c45-b83b869effd1&rgwx-prepend-metadata=true&rgwx-sync-manifest&rgwx-skip-decrypt HTTP/1.1" 304 0 - - - latency=0.077000856s
 -272> 2022-12-09T18:04:33.026+0000 7f6980719700  2 req 9777807639303135910 0.110001221s s3:get_obj init op
 -271> 2022-12-09T18:04:33.026+0000 7f6980719700  2 req 9777807639303135910 0.110001221s s3:get_obj verifying op mask
 -270> 2022-12-09T18:04:33.026+0000 7f6980719700  2 req 9777807639303135910 0.110001221s s3:get_obj verifying op permissions
 -269> 2022-12-09T18:04:33.026+0000 7f6980719700  5 req 9777807639303135910 0.110001221s s3:get_obj Searching permissions for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=synchronization-user, acct_name=synchronization-user, subuser=, perm_mask=15, is_admin=0) mask=49
 -268> 2022-12-09T18:04:33.026+0000 7f6980719700  5 req 9777807639303135910 0.110001221s s3:get_obj Searching permissions for uid=synchronization-user
 -267> 2022-12-09T18:04:33.026+0000 7f6980719700  5 req 9777807639303135910 0.110001221s s3:get_obj Permissions for user not found
 -266> 2022-12-09T18:04:33.026+0000 7f6980719700  5 req 9777807639303135910 0.110001221s s3:get_obj Searching permissions for group=1 mask=49
 -265> 2022-12-09T18:04:33.026+0000 7f6980719700  5 req 9777807639303135910 0.110001221s s3:get_obj Permissions for group not found
 -264> 2022-12-09T18:04:33.026+0000 7f6980719700  5 req 9777807639303135910 0.110001221s s3:get_obj Searching permissions for group=2 mask=49
 -263> 2022-12-09T18:04:33.026+0000 7f6980719700  5 req 9777807639303135910 0.110001221s s3:get_obj Permissions for group not found
 -262> 2022-12-09T18:04:33.026+0000 7f6980719700  5 req 9777807639303135910 0.110001221s s3:get_obj -- Getting permissions done for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=synchronization-user, acct_name=synchronization-user, subuser=, perm_mask=15, is_admin=0), owner=admin, perm=0
 -261> 2022-12-09T18:04:33.026+0000 7f6980719700  5 req 9777807639303135910 0.110001221s s3:get_obj Searching permissions for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=synchronization-user, acct_name=synchronization-user, subuser=, perm_mask=15, is_admin=0) mask=48
 -260> 2022-12-09T18:04:33.026+0000 7f6980719700  5 req 9777807639303135910 0.110001221s s3:get_obj Searching permissions for uid=synchronization-user
 -259> 2022-12-09T18:04:33.026+0000 7f6980719700  5 req 9777807639303135910 0.110001221s s3:get_obj Permissions for user not found
 -258> 2022-12-09T18:04:33.026+0000 7f6980719700  5 req 9777807639303135910 0.110001221s s3:get_obj Searching permissions for group=1 mask=48
 -257> 2022-12-09T18:04:33.026+0000 7f6980719700  5 req 9777807639303135910 0.110001221s s3:get_obj Permissions for group not found
 -256> 2022-12-09T18:04:33.026+0000 7f6980719700  5 req 9777807639303135910 0.110001221s s3:get_obj Searching permissions for group=2 mask=48
 -255> 2022-12-09T18:04:33.026+0000 7f6980719700  5 req 9777807639303135910 0.110001221s s3:get_obj Permissions for group not found
 -254> 2022-12-09T18:04:33.026+0000 7f6980719700  5 req 9777807639303135910 0.110001221s s3:get_obj -- Getting permissions done for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=synchronization-user, acct_name=synchronization-user, subuser=, perm_mask=15, is_admin=0), owner=admin, perm=0
 -253> 2022-12-09T18:04:33.026+0000 7f6980719700  2 overriding permissions due to system operation
 -252> 2022-12-09T18:04:33.026+0000 7f6980719700  2 req 9777807639303135910 0.110001221s s3:get_obj verifying op params
 -251> 2022-12-09T18:04:33.026+0000 7f6980719700  2 req 9777807639303135910 0.110001221s s3:get_obj pre-executing
 -250> 2022-12-09T18:04:33.026+0000 7f6980719700  2 req 9777807639303135910 0.110001221s s3:get_obj check rate limiting
 -249> 2022-12-09T18:04:33.026+0000 7f6980719700  2 req 9777807639303135910 0.110001221s s3:get_obj executing
 -248> 2022-12-09T18:04:33.026+0000 7f6980719700  2 req 9777807639303135910 0.110001221s s3:get_obj completing
 -247> 2022-12-09T18:04:33.026+0000 7f6980719700  2 req 9777807639303135910 0.110001221s s3:get_obj op status=-2016
 -246> 2022-12-09T18:04:33.026+0000 7f6980719700  2 req 9777807639303135910 0.110001221s s3:get_obj http status=304
 -245> 2022-12-09T18:04:33.026+0000 7f6980719700  1 ====== req done req=0x7f682c7e85c0 op status=-2016 http_status=304 latency=0.110001221s ======
 -244> 2022-12-09T18:04:33.026+0000 7f6980719700  1 beast: 0x7f682c7e85c0: 10.200.125.16 - synchronization-user [09/Dec/2022:18:04:32.916 +0000] "GET /mimic.da.spoolers.perf3/(Ja5Foko%2F10722.GggLHf2oGIRAcnhV.rnd?rgwx-zonegroup=56082fa6-e41f-40aa-9c45-b83b869effd1&rgwx-prepend-metadata=true&rgwx-sync-manifest&rgwx-skip-decrypt HTTP/1.1" 304 0 - - - latency=0.110001221s
 -243> 2022-12-09T18:04:33.056+0000 7f69aef76700  1 ====== starting new request req=0x7f682c5e45c0 =====
 -242> 2022-12-09T18:04:33.056+0000 7f69aef76700  2 req 9849936671185163943 0.000000000s initializing for trans_id = tx0000088b201371a15d6a7-00639378b1-b992fa-nvan
 -241> 2022-12-09T18:04:33.056+0000 7f69aef76700  2 req 9849936671185163943 0.000000000s getting op 0
 -240> 2022-12-09T18:04:33.056+0000 7f69aef76700  2 req 9849936671185163943 0.000000000s s3:get_obj verifying requester
 -239> 2022-12-09T18:04:33.056+0000 7f69aef76700  2 req 9849936671185163943 0.000000000s s3:get_obj normalizing buckets and tenants
 -238> 2022-12-09T18:04:33.056+0000 7f69aef76700  2 req 9849936671185163943 0.000000000s s3:get_obj init permissions
 -237> 2022-12-09T18:04:33.056+0000 7f69aef76700  2 req 9849936671185163943 0.000000000s s3:get_obj recalculating target
 -236> 2022-12-09T18:04:33.056+0000 7f69aef76700  2 req 9849936671185163943 0.000000000s s3:get_obj reading permissions
 -235> 2022-12-09T18:04:33.083+0000 7f697f717700  1 ====== starting new request req=0x7f682b94b5c0 =====
 -234> 2022-12-09T18:04:33.083+0000 7f697f717700  2 req 8762047531418507817 0.000000000s initializing for trans_id = tx0000079990bce7749da29-00639378b1-b992fa-nvan
 -233> 2022-12-09T18:04:33.083+0000 7f697f717700  2 req 8762047531418507817 0.000000000s getting op 0
 -232> 2022-12-09T18:04:33.083+0000 7f697f717700  2 req 8762047531418507817 0.000000000s s3:list_bucket verifying requester
 -231> 2022-12-09T18:04:33.083+0000 7f697f717700  2 req 8762047531418507817 0.000000000s s3:list_bucket normalizing buckets and tenants
 -230> 2022-12-09T18:04:33.083+0000 7f697f717700  2 req 8762047531418507817 0.000000000s s3:list_bucket init permissions
 -229> 2022-12-09T18:04:33.085+0000 7f69dcfd2700  2 req 8762047531418507817 0.002000023s s3:list_bucket recalculating target
 -228> 2022-12-09T18:04:33.085+0000 7f69dcfd2700  2 req 8762047531418507817 0.002000023s s3:list_bucket reading permissions
 -227> 2022-12-09T18:04:33.085+0000 7f69dcfd2700  2 req 8762047531418507817 0.002000023s s3:list_bucket init op
 -226> 2022-12-09T18:04:33.085+0000 7f69dcfd2700  2 req 8762047531418507817 0.002000023s s3:list_bucket verifying op mask
 -225> 2022-12-09T18:04:33.085+0000 7f69dcfd2700  2 req 8762047531418507817 0.002000023s s3:list_bucket verifying op permissions
 -224> 2022-12-09T18:04:33.085+0000 7f69dcfd2700  5 req 8762047531418507817 0.002000023s s3:list_bucket Searching permissions for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=synchronization-user, acct_name=synchronization-user, subuser=, perm_mask=15, is_admin=0) mask=49
 -223> 2022-12-09T18:04:33.085+0000 7f69dcfd2700  5 req 8762047531418507817 0.002000023s s3:list_bucket Searching permissions for uid=synchronization-user
 -222> 2022-12-09T18:04:33.085+0000 7f69dcfd2700  5 req 8762047531418507817 0.002000023s s3:list_bucket Permissions for user not found
 -221> 2022-12-09T18:04:33.085+0000 7f69dcfd2700  5 req 8762047531418507817 0.002000023s s3:list_bucket Searching permissions for group=1 mask=49
 -220> 2022-12-09T18:04:33.085+0000 7f69dcfd2700  5 req 8762047531418507817 0.002000023s s3:list_bucket Permissions for group not found
 -219> 2022-12-09T18:04:33.085+0000 7f69dcfd2700  5 req 8762047531418507817 0.002000023s s3:list_bucket Searching permissions for group=2 mask=49
 -218> 2022-12-09T18:04:33.085+0000 7f69dcfd2700  5 req 8762047531418507817 0.002000023s s3:list_bucket Permissions for group not found
 -217> 2022-12-09T18:04:33.085+0000 7f69dcfd2700  5 req 8762047531418507817 0.002000023s s3:list_bucket -- Getting permissions done for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=synchronization-user, acct_name=synchronization-user, subuser=, perm_mask=15, is_admin=0), owner=admin, perm=0
 -216> 2022-12-09T18:04:33.085+0000 7f69dcfd2700  2 overriding permissions due to system operation
 -215> 2022-12-09T18:04:33.085+0000 7f69dcfd2700  2 req 8762047531418507817 0.002000023s s3:list_bucket verifying op params
 -214> 2022-12-09T18:04:33.085+0000 7f69dcfd2700  2 req 8762047531418507817 0.002000023s s3:list_bucket pre-executing
 -213> 2022-12-09T18:04:33.085+0000 7f69dcfd2700  2 req 8762047531418507817 0.002000023s s3:list_bucket check rate limiting
 -212> 2022-12-09T18:04:33.085+0000 7f69dcfd2700  2 req 8762047531418507817 0.002000023s s3:list_bucket executing
 -211> 2022-12-09T18:04:33.090+0000 7f69dcfd2700 -1 *** Caught signal (Floating point exception) **
in thread 7f69dcfd2700 thread_name:radosgw

ceph version 17.2.3 (dff484dfc9e19a9819f375586300b3b79d80034d) quincy (stable)
1: /lib64/libpthread.so.0(+0x12ce0) [0x7f6b6bcbbce0]
2: (RGWRados::calc_ordered_bucket_list_per_shard(unsigned int, unsigned int)+0xd) [0x7f6b6edd2b8d]
3: (RGWRados::cls_bucket_list_ordered(DoutPrefixProvider const*, RGWBucketInfo&, int, cls_rgw_obj_key const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int, bool, unsigned short, boost::container::flat_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, rgw_bucket_dir_entry, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, void>&, bool*, bool*, cls_rgw_obj_key*, optional_yield, std::function<bool (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>)+0x1d2e) [0x7f6b6ee147ce]
4: (RGWRados::Bucket::List::list_objects_ordered(DoutPrefixProvider const*, long, std::vector<rgw_bucket_dir_entry, std::allocator<rgw_bucket_dir_entry> >*, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, bool> > >*, bool*, optional_yield)+0x787) [0x7f6b6ee16267]
5: (rgw::sal::RadosBucket::list(DoutPrefixProvider const*, rgw::sal::Bucket::ListParams&, int, rgw::sal::Bucket::ListResults&, optional_yield)+0x3fa) [0x7f6b6ef0a46a]
6: (RGWListBucket::execute(optional_yield)+0x347) [0x7f6b6ed7e697]
7: (rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, optional_yield, rgw::sal::Store*, bool)+0xb3f) [0x7f6b6e9d969f]
8: (process_request(rgw::sal::Store*, RGWREST*, RGWRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSink*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*, std::shared_ptr<RateLimiter>, int*)+0x2616) [0x7f6b6e9dc606]
9: /lib64/libradosgw.so.2(+0x65c62d) [0x7f6b6e94b62d]
10: make_fcontext()

NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this