Ceph » rgw

Hi Taha,

I was not able to reproduce this issue on the main branch after setting bucket policy and trying to trigger quota overruns for either a max-objects quota or a max-size quota.

Sometimes quota is prone to overrun because it takes time for the quota cache and the users/bucket stats to sync up. Do you notice this issue repeatedly occurs after long pauses where the quota could be properly refreshed?

If so could you send me more detailed steps to reproduce this issue?

- Ali

I'm able to replicate this running quay.io/ceph/ceph:v18.2.1 deployed with rook

• create two users, A and B, each with quotas enabled. I chose 1GB quotas and no object count quota
• create a bucket owned by A
• apply policy to A's bucket
  

  {
     "Version":"2012-10-17",
     "Statement":[
        {
           "Effect":"Allow",
           "Principal":{
              "AWS":[
                 "arn:aws:iam:::user/B" 
              ]
           },
           "Action":"s3:*",
           "Resource":[
              "arn:aws:s3:::*" 
           ]
        }
     ]
  }
  

• as B, write an object to A's bucket. This action will not be accounted for in either A's or B's size stat, as shown by
  

  [rook@ceph-1 /]$ radosgw-admin user stats --uid A 
  {
      "stats": {
          "size": 0,
          "size_actual": 0,
          "size_kb": 0,
          "size_kb_actual": 0,
          "num_objects": 0
      },
      "last_stats_sync": "2024-01-31T22:32:17.367195Z",
      "last_stats_update": "2024-01-31T22:32:17.366219Z" 
  }
  [rook@ceph-1 /]$ radosgw-admin user stats --uid B
  {
      "stats": {
          "size": 0,
          "size_actual": 0,
          "size_kb": 0,
          "size_kb_actual": 0,
          "num_objects": 0
      },
      "last_stats_sync": "2024-02-06T22:32:19.139449Z",
      "last_stats_update": "2024-02-06T22:32:19.138487Z" 
  }
  
  

  
  I can also confirm that B can continue uploading objects beyond the quota size, including after a stat sync has occurred

• Perhaps related, B does not see A's bucket when listing all buckets
  

  [rook@ceph-1 /]$ aws --endpoint-url http://s3.private s3api list-buckets
  {
      "Buckets": [],
      "Owner": {
          "DisplayName": "B",
          "ID": "b" 
      }
  }
  
  

Paul,

The user stats not being update makes it seem as if the user stats are not being updated fast enough. Can you show in this issue being reproduced if you run `radosgw-admin user stats --uid {uid} --sync-stats` when you're trying to take a look at the user stats and between trying to overrun quota from user B.

See:
https://docs.ceph.com/en/latest/radosgw/admin/#update-quota-stats

The problem persists even after running sync-stats. In fact, I identified the issue weeks after the quota had already been exceeded.