Project

General

Profile

Bug #51786

could not reset user stats: Value too large for defined data type

Added by JS Landry over 2 years ago. Updated 3 months ago.

Status:
Triaged
Priority:
High
Assignee:
Target version:
-
% Done:

0%

Source:
Tags:
resetstats
Backport:
pacific quincy reef
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
ceph-ansible
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Hi, I have some kind of double accounting on a shared bucket between two users.
I was hopping to fix the situation with reset-stats, but failed.

$ radosgw-admin user stats --uid student --reset-stats
ERROR: could not reset user stats: (75) Value too large for defined data type
$ radosgw-admin user stats --uid teacher --reset-stats
ERROR: could not reset user stats: (75) Value too large for defined data type

running ceph octopus 15.2.13
rgw_bucket_default_quota_max_objects: 300000

one "shared_bucket", quota enabled: false

user 1: teacher (bucket owner)
quota enable: true
max size: 4TB
max objects: -1

user 2: student
quota enable: true
max size: 25GB
max objects: -1

Currently, the "shared_bucket" have 1268 objects in it and the rgw.main.size is around 240GB
student is the owner of about 1000 objects of the 1268 in this bucket.

Using radosgw-admin user stats, both the teacher and the student have 240GB stats.size usage.
and the student is now 900% over-quota.

student don't have any other buckets and/or any objects elsewhere on the cluster, everything is in this bucket.
teacher have a 2nd empty bucket.

What I expected is that the quota usage should be add to the bucket owner, and the quota usage of the "student" should be near zero.
It's look like the "--reset-stats / Value too large for defined data type", is a bug, so how can I reset the usage stats?
The "radosgw-admin user stats --uid student --sync-stats" doesn't change anything.

Cheers!

radosgw-admin-info-stats.out - radosgw-admin bucket/user stats/info output (3.81 KB) JS Landry, 07/21/2021 05:49 PM

History

#1 Updated by Casey Bodley over 2 years ago

  • Assignee set to Matt Benjamin
  • Tags set to resetstats
  • Backport set to pacific octopus

#2 Updated by Casey Bodley over 2 years ago

Matt, was this one related to the max number of bytes a write request can return? do you know if this has been resolved on master yet?

#3 Updated by Matt Benjamin over 2 years ago

I don't know--I thought I pushed a PR, maybe I didn't. Didn't RADOS agree to lift this to 128 bytes (size of md5 checksum, iirc)?

Matt

#4 Updated by Matt Benjamin 12 months ago

  • Status changed from New to Triaged
  • Priority changed from Normal to High

It looks to me as if this is a returnvec overflow.

When we planned this, we were told we could have 128 bytes of return buffer, I'd asked for 512. What we got was 64 bytes, which seems needlessly small.

The reset operation serializes a cls_user_reset_stats2_ret as its return type. The stats variables being summed into consume 24 bytes, which is already tight. The problem then is
the marker. Depending on what bucket names the user has, it probably overflows fairly easily.

struct cls_user_reset_stats2_ret {
  std::string marker;
  cls_user_stats acc_stats; /* 0-initialized */
  bool truncated;

  cls_user_reset_stats2_ret()
    : truncated(false) {}

#5 Updated by Matt Benjamin 12 months ago

Let me correct. 64-bytes is exactly the size of a sha-512 hash, so it is I guess the absolute minimal size that could be durably useful (forget MD5 above).

That said, we need more buffer for almost any other radosgw use.

Matt

#6 Updated by Casey Bodley 12 months ago

  • Backport changed from pacific octopus to pacific quincy reef

#7 Updated by Matt Benjamin 3 months ago

maybe this can be adapted to complete after multiple invocations

Also available in: Atom PDF