Ceph » rgw

The renew of the lock logic is run in process_single_shard but a single buckert resharding can take a long time.
The renew code needs to move into do_reshard and executed before and after any long operation like bi_list and bi_put.
In case the lock has expired the resharding operation should be aborted and all objects created should be deleted.

Hello,

I’m experiencing this myself at the moment. Is there a workaround? I’m running Ceph 12.2.7 on a 4-node cluster, each node with one 2.8TB SSD OSD, and 20 CPU cores (3GHz).

Scenario: I have several large buckets, the largest one with over 9 million objects. I have 2 rgw clients that seem to interfere with each other, as they alternate “failed to acquire lock on obj_delete_at_hint.{number}” log entries and appear to be processing at around the same hint entry.

I don’t think either rgw client finishes the resharding process, as they reach:
object expiration: stop
object expiration: start
at which point they loop back to object_delete_at_hint.0000000000

I’ve tried increasing rgw_reshard_hints_num_shards to 1024, hoping that the rgw clients would finish resharding before reaching the rgw_reshard_hints_num_shards limit, with no success. The resharding process ran all night, over and over, without resharding any bucket (there's 6 buckets that need resharding).

Is there any other workaround I can try to let resharding finish? Also, any advice on how I can speed up the resharding process? I don't mind using lots more CPU for this to minimize the amount of time that my large buckets are locked (which prevents my app from writing to it).

I'm new to ceph, but have been devouring online documentation and bug reports to learn the ceph internals and search for a workaround, but haven't found a solution to this yet.

Best regards,
Jorge

Hi Joreg,
You can try disabling dynamic resharding in the ceph conf file as a temporary workaround.
You can use "reshard cancel command" to cancel ongoing resharding

Thanks Orit.
I disabled resharding and have no problems writing to my buckets. I’ll await for the next major development regarding rgw resharding before re-enabling it.

Also, I upgraded to Mimic, and it’s working great so far! My deep scrubs were taking too long due to stupidalloc dumps (I store over 30M objects), but the new bitmap allocator seems to have solved those issues.

Cheers,
Jorge

Update: I just noticed Mimic still uses stupidalloc by default. Either way, I’m not experiencing the same slowdown and stupdalloc dumps during scrubs.

I’m tempted to try bitmap allocator given that my cluster stores over 30M small objects. Do you recommend I make the switch at this point?

Cheers,
Jorge