Ceph » rgw

One of our user complained that they'd deleted a bucket with lots of part-uploaded bits in but their quota was still being treated as if the contents were still there.

rgw-admin user stats --sync-stats reports:
"total_entries": 6590,
"total_bytes": 1767693041863,
"total_bytes_rounded": 1767700045824

if I do bucket stats --uid=as45 (or search the output of bucket stats by hand), I find 4 buckets who sum to: (details in footnote 1)
num_objects: 3370
size_kb: 774880722
size_kb_actual: 774887560

taking the larger of these x1024 is 793,484,861,440, considerably smaller than the quota number above.

We have done "bucket check" on all the users' buckets, all return 0. We have done "orphan find" and removed all the leaked objects returned.

We are running Ubuntu 16.04, with packaged rgw version 10.2.7-0ubuntu0.16.04.2~sanger1; the Sanger1 suffix is a RH-provided patch to fix a MIME issue with uploads.

I've attached the (compressed) output of radosgw-admin -n client.rgw.sto-1-2 user stats --sync-stats --uid=as45 --debug_rgw=20 >/tmp/rgwoutput2 2>&1

This is a significant problem for a number of reasons including that it makes effective quota management very difficult (and I wonder whether this reflects storage that is still being used and can't be reclaimed).

[1] 4 buckets:
"size_kb": 33599604,
"size_kb_actual": 33602384,
"num_objects": 1390
"size_kb": 0,
"size_kb_actual": 0,
"num_objects": 0
"size_kb": 707556170,
"size_kb_actual": 707556172,
"num_objects": 2
"size_kb": 33724948,
"size_kb_actual": 33729004,
"num_objects": 1978