Project

General

Profile

Bug #22650

couldn't fetch user stats at slave zone, return ERROR: could not sync bucket stats: ret=-2

Added by Amine Liu about 6 years ago. Updated about 6 years ago.

Status:
Need More Info
Priority:
Normal
Assignee:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
rgw
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

at master:
[root@sx-3f3r-ceph-s3-c1-03 conf]# radosgw-admin user stats --sync-stats --uid=Ymliu {
"stats": {
"total_entries": 7,
"total_bytes": 132701990,
"total_bytes_rounded": 132710400
},
"last_stats_sync": "2018-01-10 08:53:44.586164Z",
"last_stats_update": "2018-01-10 08:53:44.560790Z"
}

But error return at slave zone:

[root@xxy-2f201r-ceph-s3-c1-01 my-cluster]# radosgw-admin user stats --sync-stats --uid=Ymliu
ERROR: failed to sync user stats: (2) No such file or directory
2018-01-10 17:18:07.019280 7fcbbcb919c0 0 ERROR: could not sync bucket stats: ret=-2

my zone config :

{
"id": "cf47478b-b040-4649-82d2-97510979d738",
"name": "imu",
"api_name": "imu",
"is_master": "true",
"endpoints": [
"http:\/\/s3gz.imu.cn:80"
],
"hostnames": [],
"hostnames_s3website": [],
"master_zone": "e3026b1f-be0f-4ce3-a159-b4bf6c5bf416",
"zones": [ {
"id": "52996319-9058-4973-a36a-d4acd649e185",
"name": "imu-shxxy",
"endpoints": [
"http:\/\/s3sh.imu.cn:80"
],
"log_meta": "false",
"log_data": "true",
"bucket_index_max_shards": 0,
"read_only": "false"
}, {
"id": "e3026b1f-be0f-4ce3-a159-b4bf6c5bf416",
"name": "imu-gzsx",
"endpoints": [
"http:\/\/s3gz.imu.cn:80"
],
"log_meta": "false",
"log_data": "true",
"bucket_index_max_shards": 0,
"read_only": "false"
}
],
"placement_targets": [ {
"name": "default-placement",
"tags": []
}
],
"default_placement": "default-placement",
"realm_id": "701fa769-bf8a-4cb6-ab58-66ca362da2ee"
}

[root@xxy-2f201r-ceph-s3-c1-01 my-cluster]# radosgw-admin sync status
realm 701fa769-bf8a-4cb6-ab58-66ca362da2ee (imusic)
zonegroup cf47478b-b040-4649-82d2-97510979d738 (imu)
zone 52996319-9058-4973-a36a-d4acd649e185 (imu-shxxy)
metadata sync syncing
full sync: 0/64 shards
incremental sync: 64/64 shards
metadata is caught up with master
data sync source: e3026b1f-be0f-4ce3-a159-b4bf6c5bf416 (imu-gzsx)
syncing
full sync: 0/128 shards
incremental sync: 128/128 shards
data is caught up with source

ceph-client.rgw.xxy-2f201r-ceph-s3-c1-01.rar - debug log as `radosgw-admin user stats` on slave (719 KB) Amine Liu, 01/12/2018 01:42 AM

History

#1 Updated by Casey Bodley about 6 years ago

  • Status changed from New to Need More Info

if you list the user's buckets on both sides, do you see the same results?

you can check `radosgw-admin sync error list` for errors on the specific user or their buckets

and it would also help to include the debug log (--debug-rgw 20 --debug-ms 1) of your `radosgw-admin user stats` command that fails on the secondary zone

#2 Updated by Amine Liu about 6 years ago

Casey Bodley wrote:

if you list the user's buckets on both sides, do you see the same results?

you can check `radosgw-admin sync error list` for errors on the specific user or their buckets

and it would also help to include the debug log (--debug-rgw 20 --debug-ms 1) of your `radosgw-admin user stats` command that fails on the secondary zone

list the user's buckets on both sides, list each bucket, all the results(bucket size, bucket objs) are correct:
master zone:
[root@sx-3f3r-ceph-s3-c1-03 my-cluster]# radosgw-admin bucket stats --bucket=imusic {
"bucket": "imusic",
"pool": "imu-gzsx.rgw.buckets.data",
"index_pool": "imu-gzsx.rgw.buckets.index",
"id": "e3026b1f-be0f-4ce3-a159-b4bf6c5bf416.15360.1",
"marker": "e3026b1f-be0f-4ce3-a159-b4bf6c5bf416.15360.1",
"owner": "Ymliu",
"ver": "0#25,1#3,2#21",
"master_ver": "0#0,1#0,2#0",
"mtime": "2018-01-08 17:35:28.767400",
"max_marker": "0#00000000024.24.3,1#00000000002.3.3,2#00000000020.20.3",
"usage": {
"rgw.main": {
"size_kb": 16953,
"size_kb_actual": 16960,
"num_objects": 5
},
"rgw.multimeta": {
"size_kb": 0,
"size_kb_actual": 0,
"num_objects": 0
}
},
"bucket_quota": {
"enabled": false,
"max_size_kb": -1,
"max_objects": -1
}
}
slave zone: {
"bucket": "imusic",
"pool": "imu-shxxy.rgw.buckets.data",
"index_pool": "imu-shxxy.rgw.buckets.index",
"id": "e3026b1f-be0f-4ce3-a159-b4bf6c5bf416.15360.1",
"marker": "e3026b1f-be0f-4ce3-a159-b4bf6c5bf416.15360.1",
"owner": "Ymliu",
"ver": "0#7,1#3,2#11",
"master_ver": "0#0,1#0,2#0",
"mtime": "2018-01-08 17:35:28.767400",
"max_marker": "0#00000000006.6.3,1#00000000002.2.3,2#00000000010.10.3",
"usage": {
"rgw.main": {
"size_kb": 16953,
"size_kb_actual": 16960,
"num_objects": 5
}
},
"bucket_quota": {
"enabled": false,
"max_size_kb": -1,
"max_objects": -1
}
}

but, `radosgw-admin user stats ` is not error about some values on slave zone (as total_entries,last_stats_sync):
master zone:
[root@sx-3f3r-ceph-s3-c1-03 my-cluster]# radosgw-admin user stats --uid=Ymliu {
"stats": {
"total_entries": 118829,
"total_bytes": 11899758630,
"total_bytes_rounded": 12184944640
},
"last_stats_sync": "2018-01-11 09:41:10.394107Z",
"last_stats_update": "2018-01-11 10:11:14.170735Z"
}

slave zone:
[root@xxy-2f201r-ceph-s3-c1-01 ~]# radosgw-admin user stats --uid=Ymliu {
"stats": {
"total_entries": 123250,
"total_bytes": 12341758630,
"total_bytes_rounded": 12637552640
},
"last_stats_sync": "0.000000",
"last_stats_update": "2018-01-11 10:11:47.163477Z"
}

on both zone , `radosgw-admin sync error list` all error message was:
"error_code": 16,
"message": "failed to sync bucket instance: (16) Device or resource busy"

Also available in: Atom PDF