Ceph » rgw

@Liam,
Will attempt to reproduce - is it possible to provide the ceph.conf please?

Similar issue here. Users are hitting their quotas very quickly even though they are not writing anywhere near the reported space usage.

It seems the "size_*" and "num_objects" counts increase every time a 'user stats' command is issued that includes either "--reset-stats" or "--sync-stats":

$ sudo radosgw-admin --rgw-realm=obs1 user stats --reset-stats  --uid <uid>
{
    "stats": {
        "size": 881499743033166,
        "size_actual": 881500029788160,
        "size_utilized": 0,
        "size_kb": 860839592806,
        "size_kb_actual": 860839872840,
        "size_kb_utilized": 0,
        "num_objects": 251406
    },
    "last_stats_sync": "0.000000",
    "last_stats_update": "2020-08-03T15:06:06.461464Z" 
}
$ sudo radosgw-admin --rgw-realm=obs1 user stats --reset-stats  --uid <uid>
{
    "stats": {
        "size": 930471950979453,
        "size_actual": 930472253665280,
        "size_utilized": 0,
        "size_kb": 908664014629,
        "size_kb_actual": 908664310220,
        "size_kb_utilized": 0,
        "num_objects": 265373
    },
    "last_stats_sync": "0.000000",
    "last_stats_update": "2020-08-03T15:06:09.689322Z" 
}

Thank you for the additional detailes, was not able to reproduce prior.

Reproduces on master, running the command below increases the stats continiously:

watch -d "./bin/radosgw-admin user stats --sync-stats --uid=cosbench 2>/dev/null" 
#                                              ^^^^^^^^^^^^

- Does not occur with "--reset-stats" - only with "--sync-stats".
- Does not occur in v14.2.9 - Proceedingh to bisect and debug.

Following the above PR, the issue no longer reproduces.
(tested on a user with multiple buckets and multiple shards)

does not reproduce on nautilus
dos reproduce on master,pacific,octopus

copying comment from github - https://github.com/ceph/ceph/pull/36542#issuecomment-672213315 :
...
we built 15.2.4/master with this PR cherry-picked. We upgraded from 15.2.4 to 15.2.4 with this patch. Initially we rolled it out to a single RGW and things looked fine; we then rolled it out to the rest of the RGWs and again things seemed fine; the RGWs were serving requests as normal. An hour or so later we noticed radosgw-admin segfaulted whenever we called radosgw-admin user stats. After about two hours or so, all of the RGWs segfaulted with the following: https://gist.github.com/monschein/027f3e45e2adb3f93f8efd493ea77663#file-cash1-txt

0> 2020-08-11T11:25:26.308-0400 7f1bd29a3700 -1 *** Caught signal (Segmentation fault) **
 in thread 7f1bd29a3700 thread_name:rgw_user_st_syn

 ceph version 15.2.4-537-gde523422e7 (de523422e7396e1237aa21a7a92ecc04651d9bf9) octopus (stable)
 1: (()+0x12dd0) [0x7f1c02e64dd0]
 2: (rgw_user::to_str(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) const+0x18) [0x7f1c0d8de1e8]
 3: (RGWSI_User_RADOS::get_buckets_obj(rgw_user const&) const+0x4b) [0x7f1c0dd7d3eb]
 4: (RGWSI_User_RADOS::flush_bucket_stats(RGWSI_MetaBackend::Context*, rgw_user const&, RGWBucketEnt const&)+0x36) [0x7f1c0dd81ee6]
 5: (()+0x8fb92e) [0x7f1c0dd6492e]
 6: (RGWSI_MetaBackend_SObj::call(std::optional<std::variant<RGWSI_MetaBackend_CtxParams_SObj> >, std::function<int (RGWSI_MetaBackend::Context*)>)+0x9e) [0x7f1c0dd6789e]
 7: (RGWSI_MetaBackend_Handler::call(std::optional<std::variant<RGWSI_MetaBackend_CtxParams_SObj> >, std::function<int (RGWSI_MetaBackend_Handler::Op*)>)+0x5f) [0x7f1c0dd6475f]
 8: (RGWSI_MetaBackend_Handler::call(std::function<int (RGWSI_MetaBackend_Handler::Op*)>)+0x78) [0x7f1c0dcc7fa8]
 9: (RGWUserCtl::flush_bucket_stats(rgw_user const&, RGWBucketEnt const&)+0x73) [0x7f1c0dcbb453]
 10: (RGWBucketCtl::sync_user_stats(rgw_user const&, RGWBucketInfo const&, RGWBucketEnt*)+0x367) [0x7f1c0d9a0b47]
 11: (rgw_user_sync_all_stats(rgw::sal::RGWRadosStore*, rgw_user const&)+0x1b1) [0x7f1c0dcc59f1]
 12: (RGWUserStatsCache::sync_user(rgw_user const&)+0x289) [0x7f1c0dbdce59]
 13: (RGWUserStatsCache::sync_all_users()+0x63a) [0x7f1c0dbdd77a]
 14: (RGWUserStatsCache::UserSyncThread::entry()+0x10b) [0x7f1c0dbe404b]
 15: (()+0x82de) [0x7f1c02e5a2de]
 16: (clone()+0x43) [0x7f1c0156de83]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

@Liam,
For reproduction purposes, may I ask -
what is the largest number of buckets that a single user has?

We have one user with 172 buckets, and most users have far fewer than that.

total users: 1484

• mean: 1.1617250673854447
• median: 0.0
• mode: 0
• max: 172

The segfault reproduces on 15.2.4
but not on master , bisecting to find the commit that fixes it.

git branch -vv
* (HEAD detached at v15.2.4) 7447c15c6f 15.2.4
  master                     f93105e895 [origin/master] Merge pull request #36532 from simon-rock/crash_on_restart

2020-08-13T11:06:44.699+0000 7efc82ffd700  2 req 34 0s s3:create_bucket init permissions                                                                                        
2020-08-13T11:06:44.699+0000 7efc8ffff700 -1 *** Caught signal (Segmentation fault) **                                                                                           
 in thread 7efc8ffff700 thread_name:rgw_user_st_syn                                                                                                                                  

 ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable)                                                                                                 
 1: (()+0x37ec0) [0x7efd82e82ec0]                                                                                                                                                
 2: (RGWSI_User_RADOS::get_buckets_obj(rgw_user const&) const+0x2f) [0x7efd83a3eaff]                                                                                             
 3: (RGWSI_User_RADOS::flush_bucket_stats(RGWSI_MetaBackend::Context*, rgw_user const&, RGWBucketEnt const&)+0x38) [0x7efd83a44ed8]                                             
 4: (()+0xa050f2) [0x7efd83a250f2]                                                                                                                                              
 5: (RGWSI_MetaBackend_SObj::call(std::optional<std::variant<RGWSI_MetaBackend_CtxParams_SObj> >, std::function<int (RGWSI_MetaBackend::Context*)>)+0x9f) [0x7efd83a2805f]           
 6: (RGWSI_MetaBackend_Handler::call(std::optional<std::variant<RGWSI_MetaBackend_CtxParams_SObj> >, std::function<int (RGWSI_MetaBackend_Handler::Op*)>)+0x5f) [0x7efd83a24f2f]     
 7: (RGWUserCtl::flush_bucket_stats(rgw_user const&, RGWBucketEnt const&)+0xbf) [0x7efd8396b66f]                                                                                
 8: (RGWBucketCtl::sync_user_stats(rgw_user const&, RGWBucketInfo const&, RGWBucketEnt*)+0x273) [0x7efd83593e73]                                                                
 9: (rgw_user_sync_all_stats(rgw::sal::RGWRadosStore*, rgw_user const&)+0x480) [0x7efd8397e5e0]                                                                                 
 10: (RGWUserStatsCache::sync_user(rgw_user const&)+0x334) [0x7efd8385a884]                                                                                                     
 11: (RGWUserStatsCache::sync_all_users()+0x2f3) [0x7efd8385b6d3]                                                                                                               
 12: (RGWUserStatsCache::UserSyncThread::entry()+0x102) [0x7efd83864262]                                                                                                        
 13: (()+0x84c0) [0x7efd79ede4c0]                                                                                                                                               
 14: (clone()+0x43) [0x7efd82f47133]                                                                                                                                            
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.                                                                                         
  -30> 2020-08-13T11:06:44.699+0000 7efc8dffb700  2 req 36 0s s3:create_bucket verifying op mask                                              
  -29> 2020-08-13T11:06:44.699+0000 7efc8dffb700  2 req 36 0s s3:create_bucket verifying op permissions               

copying from github comment: https://github.com/ceph/ceph/pull/36542#issuecomment-674856572

In version 15.2.4 the cherry-pick needs to be modified slightly:

please modify the cherry-pick to:

return store->ctl()->bucket->sync_user_stats(user.info.user_id, info, &ent);

@Mark:

(noting that `user.info.user_id` in 15.2.4 changed to `owner->get_id()` in master)

Can you tell us which master PR made this change?

@Nathan Weinberg, It was changed by this -

commit: 99f7c4aa1286edfea6961b92bb44bb8fe22bd599

just adding for clarification, the backported line should look similar to:

return store->ctl()->bucket->sync_user_stats(user.info.user_id, info);

return store->ctl()->bucket->sync_user_stats(owner->get_id(), info, &ent);

And it is not applicable to 14.x and below.

Mark Kogan wrote:

@Nathan Weinberg, It was changed by this -
[...]
pr: https://github.com/ceph/ceph/pull/35851

Thanks, Mark. Presumably it's a feature that is not going to be backported to Octopus.

And it is not applicable to 14.x and below.

So the "Backport" field of this issue is set correctly. Good.