Project

General

Profile

Bug #3127

rgw: negative bucket stats

Added by Yehuda Sadeh over 11 years ago. Updated over 11 years ago.

Status:
Resolved
Priority:
High
Assignee:
-
Target version:
-
% Done:

0%

Source:
Development
Tags:
Backport:
argonaut
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

We see a case where after all the objects were removed out of a bucket (40TB of 64k objects) the bucket index stats end up negative.

Associated revisions

Revision b12c1477 (diff)
Added by Yehuda Sadeh over 11 years ago

cls_rgw: change scoping of suggested changes vars

Fixes: #3127
Bad variable scoping made it so that specific variables
weren't initialized between suggested changes iterations.
This specifically affected a case where in a specific
change we had an updated followed by a remove, and the
remove was on a non-existent key (e.g., was already
removed earlier). We ended up re-substracting the
object stats, as the entry wasn't reset between
the iterations (and we didn't read it because the
key didn't exist).

backport:argonaut
Signed-off-by: Yehuda Sadeh <>

Revision 92be908c (diff)
Added by Yehuda Sadeh over 11 years ago

cls_rgw: change scoping of suggested changes vars

Fixes: #3127
Bad variable scoping made it so that specific variables
weren't initialized between suggested changes iterations.
This specifically affected a case where in a specific
change we had an updated followed by a remove, and the
remove was on a non-existent key (e.g., was already
removed earlier). We ended up re-substracting the
object stats, as the entry wasn't reset between
the iterations (and we didn't read it because the
key didn't exist).

backport:argonaut
Signed-off-by: Yehuda Sadeh <>

History

#1 Updated by JuanJose Galvez over 11 years ago

DHO reported the following details regarding this issue:

root@peon7935:~# radosgw-admin bucket stats --uid=andfar3
[ { "bucket": "filesforawhile",
"pool": ".rgw.data.1",
"id": "8132259.4520",
"marker": "8132259.4520",
"owner": "andfar3",
"usage": { "rgw.main": { "size_kb": 18014398509473792,
"size_kb_actual": 18014398509473792,
"num_objects": -16}}}]

User andfar3 had one bucket 'filesforawhile' with 40TB in 64KB-sized objects, then deleted them all. He did not delete the bucket however, and it still exists so can be inspected if that is helpful. The gigantic number above resolves to 8192PB.

#2 Updated by Yehuda Sadeh over 11 years ago

managed to reproduce with some code changes to simulate identified scenario. The code changes:
1. Randomly skip the bucket index update complete operation (simulates rgw crash / restart)
2. Call suggest_changes twice with the same suggested changes (simulates racing operations)
3. Reduce the bucket index pending timeout to 60 seconds (instead of 24 hours)

Then, run the following:

# doing this twice on purpose
$ for i in `seq 1 100`; do ./s3 -u put bucket/dasd$i filename=1; done
$ for i in `seq 1 100`; do ./s3 -u put bucket/dasd$i filename=1; done
$ sleep 60
$ for i in `seq 50 100`; do ./s3 -u delete bucket/dasd$i; done
$ ./radosgw-admin bucket stats --bucket=bucket
{ "bucket": "bucket",
  "pool": ".rgw.buckets",
  "id": "6702.1",
  "marker": "6702.1",
  "owner": "yehsad",
  "usage": { "rgw.none": { "size_kb": 3,
          "size_kb_actual": 96,
          "num_objects": 24},
      "rgw.main": { "size_kb": 5,
          "size_kb_actual": 196,
          "num_objects": 49}}}
$ sleep 60
$ ./s3 -u list bucket | grep dasd | wc -l
49
$ ./radosgw-admin bucket stats --bucket=bucket
{ "bucket": "bucket",
  "pool": ".rgw.buckets",
  "id": "6702.1",
  "marker": "6702.1",
  "owner": "yehsad",
  "usage": { "rgw.none": { "size_kb": 3,
          "size_kb_actual": 96,
          "num_objects": 24},
      "rgw.main": { "size_kb": 4,
          "size_kb_actual": 152,
          "num_objects": 38}}}
$ for i in `seq 1 49`; do ./s3 -u delete bucket/dasd$i; done
$ ./radosgw-admin bucket stats --bucket=bucket
{ "bucket": "bucket",
  "pool": ".rgw.buckets",
  "id": "6702.1",
  "marker": "6702.1",
  "owner": "yehsad",
  "usage": { "rgw.none": { "size_kb": 3,
          "size_kb_actual": 96,
          "num_objects": 24},
      "rgw.main": { "size_kb": 18014398509481983,
          "size_kb_actual": 18014398509481940,
          "num_objects": -11}}}

#3 Updated by Yehuda Sadeh over 11 years ago

  • Status changed from New to Fix Under Review

There were a few issues in the mechanism that recovers from failed operations. When we write/remove an object we go through a two phase commit in order to update the bucket index. If the second phase doesn't complete (that is: the object was updated but the bucket index didn't complete the update, usually due to rgw process going down) then we'd later detect it and update the index according to what's actually in the object store. The accounting went a bit off there.
I've pushed a few fixes to wip-3127 (still pending review), created a unitest, and created a new check/recovery command for radosgw-admin.

#4 Updated by Yehuda Sadeh over 11 years ago

merged to master at commit:9b628efee1fc31fcb1f46818af4f6b262b460840

#5 Updated by Yehuda Sadeh over 11 years ago

  • Status changed from Fix Under Review to Resolved
  • Backport set to argonaut

Also wip-3127-stable contains the required fix and an additional band aid solution for argonaut.

Also available in: Atom PDF