Bug #22331
closedLuminous 12.2.2 multisite sync failed when objects bigger than 4MB
Added by Amine Liu over 6 years ago. Updated about 6 years ago.
0%
Description
OS: centos 7
I updated my ceph to 12.2.2 from 12.2.1 ， But now multisite sync failed when objects bigger than 4MB，and upload failed to local bucket when objects bigger than 4MB return http_codes 416 , why ?
{
"id": "1_1512544263.567288_61715.1",
"section": "data",
"name": "test1:031f5809-360a-4885-8356-88c660c79c76.94256.1",
"timestamp": "2017-12-06 07:11:03.567288Z",
"info": {
"source_zone": "af404015-a2fd-4b5e-84e4-63178a64790f",
"error_code": 16,
"message": "failed to sync bucket instance: (16) Device or resource busy"
}
}Updated by Amine Liu over 6 years ago
zonegroup：
{
"id": "30d51ec6-5ccc-4eb6-92c4-2b9f91dbcaf7",
"name": "sxs3",
"api_name": "sxs3",
"is_master": "true",
"endpoints": [
"http://s3test1.imu.cn:80"
],
"hostnames": [],
"hostnames_s3website": [],
"master_zone": "031f5809-360a-4885-8356-88c660c79c76",
"zones": [
{
"id": "031f5809-360a-4885-8356-88c660c79c76",
"name": "sxs3-1",
"endpoints": [
"http://s3test1.imu.cn:80"
],
"log_meta": "false",
"log_data": "true",
"bucket_index_max_shards": 0,
"read_only": "false",
"tier_type": "",
"sync_from_all": "true",
"sync_from": []
},
{
"id": "af404015-a2fd-4b5e-84e4-63178a64790f",
"name": "sxs3-2",
"endpoints": [
"http://s3test2.imu.cn:80"
],
"log_meta": "false",
"log_data": "true",
"bucket_index_max_shards": 0,
"read_only": "false",
"tier_type": "",
"sync_from_all": "true",
"sync_from": []
}
],
"placement_targets": [
{
"name": "default-placement",
"tags": []
},
{
"name": "new",
"tags": []
}
],
"default_placement": "default-placement",
"realm_id": "a598fa89-421f-4c09-897b-9c4634399a22"
}
master zone bucket list：
[root@s3-1-7 ~]# s3cmd -c Ymliu.cfg la
2017-12-06 03:58 3663646 s3://My-bucket1/abc.mp3
2017-12-06 06:28 15922438 s3://My-bucket1/abc.mp4
2017-11-28 07:48 10 s3://My-bucket1/hello.html
2017-12-07 01:18 15922438 s3://test1/abc.mp4
[root@s3-1-7 ~]#
second zone bucket list：
[root@s3-2-1 ~]# s3cmd -c Ymliu.cfg la
2017-12-06 03:58 3663646 s3://My-bucket1/abc.mp3
2017-11-28 07:48 10 s3://My-bucket1/hello.html
[root@s3-2-1 ~]#
radosgw-admin sync error list :
"message": "failed to sync bucket instance: (16) Device or resource busy"
Updated by Amine Liu over 6 years ago
upload objects to local bucket failed :
[root@s3-1-7 ~]# s3cmd --debug c Ymliu.cfg put abc.mp4 s3://My-bucket1>Myliu
DEBUG: s3cmd version 1.6.1
DEBUG: ConfigParser: Reading file 'Ymliu.cfg'
DEBUG: ConfigParser: user
DEBUG: ConfigParser: access_key->K9...18_chars...
DEBUG: ConfigParser: secret_key->vo...37_chars...f
DEBUG: ConfigParser: host_base->s3test1.imu.cn
DEBUG: ConfigParser: host_bucket->test-bucket.s3test1.imu.cn
DEBUG: ConfigParser: use_https->False
DEBUG: Updating Config.Config cache_file > )." using UTF-8
DEBUG: Updating Config.Config follow_symlinks -> False
DEBUG: Updating Config.Config verbosity -> 10
DEBUG: Unicodising 'put' using UTF-8
DEBUG: Unicodising 'abc.mp4' using UTF-8
DEBUG: Unicodising 's3://My-bucket1' using UTF-8
DEBUG: Command: put
DEBUG: DeUnicodising u'abc.mp4' using UTF-8
INFO: Compiling list of local files...
DEBUG: DeUnicodising u'abc.mp4' using UTF-8
DEBUG: DeUnicodising u'abc.mp4' using UTF-8
DEBUG: Unicodising '' using UTF-8
DEBUG: DeUnicodising u'abc.mp4' using UTF-8
DEBUG: DeUnicodising u'abc.mp4' using UTF-8
DEBUG: Applying --exclude/--include
DEBUG: CHECK: abc.mp4
DEBUG: PASS: u'abc.mp4'
INFO: Running stat() and reading/calculating MD5 values on 1 files, this may take some time...
DEBUG: DeUnicodising u'abc.mp4' using UTF-8
DEBUG: doing file I/O to read md5 of abc.mp4
DEBUG: DeUnicodising u'abc.mp4' using UTF-8
INFO: Summary: 1 local files to upload
DEBUG: attr_header: {'x-amz-meta-s3cmd-attrs': 'uid:0/gname:root/uname:root/gid:0/mode:33188/mtime:1502099202/atime:1512541273/md5:5aff4055d76f6716aa9ba6a8477db4fa/ctime:1509948754'}
DEBUG: DeUnicodising u'abc.mp4' using UTF-8
DEBUG: DeUnicodising u'abc.mp4' using UTF-8
DEBUG: DeUnicodising u'abc.mp4' using UTF-8
DEBUG: String 'abc.mp4' encoded to 'abc.mp4'
DEBUG: CreateRequest: resource[uri]=/abc.mp4?uploads
DEBUG: Unicodising "Bucket name 'My-bucket1' contains disallowed character 'M'. The only supported ones are: lowercase us-ascii letters (a-z), digits (0-9), dot (.) and hyphen (
DEBUG: Using signature v2
DEBUG: SignHeaders: 'POST\n\nvideo/mp4\n\nx-amz-date:Thu, 07 Dec 2017 01:27:15 0000\nx-amz-meta-s3cmd-attrs:uid:0/gname:root/uname:root/gid:0/mode:33188/mtime:1502099202/atime:1512541273/md5:5aff4055d76f6716aa9ba6a8477db4fa/ctime:1509948754\nx-amz-storage-class:STANDARD\n/My-bucket1/abc.mp4?uploads'
DEBUG: Processing request, please wait...
DEBUG: get_hostname(My-bucket1): s3test1.imu.cn
DEBUG: ConnMan.get(): creating new connection: http://s3test1.imu.cn
DEBUG: non-proxied HTTPConnection(s3test1.imu.cn)
DEBUG: format_uri(): /My-bucket1/abc.mp4?uploads
DEBUG: Sending request method_string='POST', uri='/My-bucket1/abc.mp4?uploads', headers={'x-amz-meta-s3cmd-attrs': 'uid:0/gname:root/uname:root/gid:0/mode:33188/mtime:1502099202/atime:1512541273/md5:5aff4055d76f6716aa9ba6a8477db4fa/ctime:1509948754', 'content-type': 'video/mp4', 'Authorization': 'AWS K9S70QXR5ATG2Y1AW4VD:vkKkzwjDjMo3WdtAgddFGK1RzuY=', 'x-amz-date': 'Thu, 07 Dec 2017 01:27:15 +0000', 'x-amz-storage-class': 'STANDARD'}, body=(0 bytes)
DEBUG: Response: {'status': 200, 'headers': {'date': 'Thu, 07 Dec 2017 01:27:15 GMT', 'content-length': '248', 'x-amz-request-id': 'tx0000000000000000004f5-005a2898f3-1709c-sxs3-1', 'content-type': 'application/xml'}, 'reason': 'OK', 'data': '<InitiateMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Bucket>My-bucket1</Bucket><Key>abc.mp4</Key><UploadId>2~UzmCMtOBDHOawc6T7GKPahs0PvoK8yH</UploadId></InitiateMultipartUploadResult>'}
DEBUG: ConnMan.put(): connection put back to pool (http://s3test1.imu.cn#1)
DEBUG: Unicodising 'abc.mp4' using UTF-8
DEBUG: DeUnicodising u'abc.mp4' using UTF-8
DEBUG: MultiPart: Uploading abc.mp4 in 2 parts
DEBUG: Uploading part 1 of u'2~UzmCMtOBDHOawc6T7GKPahs0PvoK8yH' (15728640 bytes)
DEBUG: String 'abc.mp4' encoded to 'abc.mp4'
DEBUG: CreateRequest: resource[uri]=/abc.mp4?partNumber=1&uploadId=2~UzmCMtOBDHOawc6T7GKPahs0PvoK8yH
DEBUG: Unicodising "Bucket name 'My-bucket1' contains disallowed character 'M'. The only supported ones are: lowercase us-ascii letters (a-z), digits (0-9), dot (.) and hyphen ()." using UTF-8)." using UTF-8
DEBUG: Using signature v2
DEBUG: SignHeaders: 'PUT\n\n\n\nx-amz-date:Thu, 07 Dec 2017 01:27:15 +0000\n/My-bucket1/abc.mp4?partNumber=1&uploadId=2~UzmCMtOBDHOawc6T7GKPahs0PvoK8yH'
DEBUG: Unicodising 'abc.mp4' using UTF-8
upload: 'abc.mp4' -> 's3://My-bucket1/abc.mp4' [part 1 of 2, 15MB] [1 of 1]
DEBUG: DeUnicodising u'abc.mp4' using UTF-8
DEBUG: Unicodising "Bucket name 'My-bucket1' contains disallowed character 'M'. The only supported ones are: lowercase us-ascii letters (a-z), digits (0-9), dot (.) and hyphen (
DEBUG: Using signature v2
DEBUG: SignHeaders: 'PUT\n\n\n\nx-amz-date:Thu, 07 Dec 2017 01:27:15 +0000\n/My-bucket1/abc.mp4?partNumber=1&uploadId=2~UzmCMtOBDHOawc6T7GKPahs0PvoK8yH'
DEBUG: get_hostname(My-bucket1): s3test1.imu.cn
DEBUG: ConnMan.get(): re-using connection: http://s3test1.imu.cn#1
DEBUG: format_uri(): /My-bucket1/abc.mp4?partNumber=1&uploadId=2~UzmCMtOBDHOawc6T7GKPahs0PvoK8yH
65536 of 15728640 0% in 0s 522.65 kB/sDEBUG: ConnMan.put(): connection put back to pool (http://s3test1.imu.cn#2)
DEBUG: Response: {'status': 200, 'headers': {'date': 'Thu, 07 Dec 2017 01:27:16 GMT', 'content-length': '0', 'etag': '"f09bf3ace14481aca0bd02e8233ecd82"', 'accept-ranges': 'bytes', 'x-amz-request-id': 'tx0000000000000000004f6-005a2898f3-1709c-sxs3-1'}, 'reason': 'OK', 'data': '', 'size': 15728640L}
15728640 of 15728640 100% in 0s 27.45 MB/s done
DEBUG: MD5 sums: computed=f09bf3ace14481aca0bd02e8233ecd82, received="f09bf3ace14481aca0bd02e8233ecd82"
DEBUG: Uploading part 2 of u'2~UzmCMtOBDHOawc6T7GKPahs0PvoK8yH' (193798 bytes)
DEBUG: String 'abc.mp4' encoded to 'abc.mp4'
DEBUG: CreateRequest: resource[uri]=/abc.mp4?partNumber=2&uploadId=2~UzmCMtOBDHOawc6T7GKPahs0PvoK8yH
DEBUG: Unicodising "Bucket name 'My-bucket1' contains disallowed character 'M'. The only supported ones are: lowercase us-ascii letters (a-z), digits (0-9), dot (.) and hyphen ()." using UTF-8)." using UTF-8
DEBUG: Using signature v2
DEBUG: SignHeaders: 'PUT\n\n\n\nx-amz-date:Thu, 07 Dec 2017 01:27:16 +0000\n/My-bucket1/abc.mp4?partNumber=2&uploadId=2~UzmCMtOBDHOawc6T7GKPahs0PvoK8yH'
DEBUG: Unicodising 'abc.mp4' using UTF-8
upload: 'abc.mp4' -> 's3://My-bucket1/abc.mp4' [part 2 of 2, 189kB] [1 of 1]
DEBUG: DeUnicodising u'abc.mp4' using UTF-8
DEBUG: Unicodising "Bucket name 'My-bucket1' contains disallowed character 'M'. The only supported ones are: lowercase us-ascii letters (a-z), digits (0-9), dot (.) and hyphen (
DEBUG: Using signature v2
DEBUG: SignHeaders: 'PUT\n\n\n\nx-amz-date:Thu, 07 Dec 2017 01:27:16 +0000\n/My-bucket1/abc.mp4?partNumber=2&uploadId=2~UzmCMtOBDHOawc6T7GKPahs0PvoK8yH'
DEBUG: get_hostname(My-bucket1): s3test1.imu.cn
DEBUG: ConnMan.get(): re-using connection: http://s3test1.imu.cn#2
DEBUG: format_uri(): /My-bucket1/abc.mp4?partNumber=2&uploadId=2~UzmCMtOBDHOawc6T7GKPahs0PvoK8yH
65536 of 193798 33% in 0s 14.51 MB/sDEBUG: ConnMan.put(): connection put back to pool (http://s3test1.imu.cn#3)
DEBUG: Response: {'status': 200, 'headers': {'date': 'Thu, 07 Dec 2017 01:27:16 GMT', 'content-length': '0', 'etag': '"49997ba8244bee9b27ee7528de76e8ca"', 'accept-ranges': 'bytes', 'x-amz-request-id': 'tx0000000000000000004f7-005a2898f4-1709c-sxs3-1'}, 'reason': 'OK', 'data': '', 'size': 193798L}
193798 of 193798 100% in 0s 1473.91 kB/s done
DEBUG: MD5 sums: computed=49997ba8244bee9b27ee7528de76e8ca, received="49997ba8244bee9b27ee7528de76e8ca"
DEBUG: MultiPart: Upload finished: 2 parts
DEBUG: MultiPart: Completing upload: 2~UzmCMtOBDHOawc6T7GKPahs0PvoK8yH
DEBUG: String 'abc.mp4' encoded to 'abc.mp4'
DEBUG: CreateRequest: resource[uri]=/abc.mp4?uploadId=2~UzmCMtOBDHOawc6T7GKPahs0PvoK8yH
DEBUG: Unicodising "Bucket name 'My-bucket1' contains disallowed character 'M'. The only supported ones are: lowercase us-ascii letters (a-z), digits (0-9), dot (.) and hyphen (-)." using UTF-8
DEBUG: Using signature v2
DEBUG: SignHeaders: 'POST\n\n\n\nx-amz-date:Thu, 07 Dec 2017 01:27:16 +0000\n/My-bucket1/abc.mp4?uploadId=2~UzmCMtOBDHOawc6T7GKPahs0PvoK8yH'
DEBUG: Processing request, please wait...
DEBUG: get_hostname(My-bucket1): s3test1.imu.cn
DEBUG: ConnMan.get(): re-using connection: http://s3test1.imu.cn#3
DEBUG: format_uri(): /My-bucket1/abc.mp4?uploadId=2~UzmCMtOBDHOawc6T7GKPahs0PvoK8yH
DEBUG: Sending request method_string='POST', uri='/My-bucket1/abc.mp4?uploadId=2~UzmCMtOBDHOawc6T7GKPahs0PvoK8yH', headers={'content-length': '223', 'Authorization': 'AWS K9S70QXR5ATG2Y1AW4VD:FNh+r1/CQtw+QhzR7R6QRJeeuk=', 'x-amz-date': 'Thu, 07 Dec 2017 01:27:16 +0000'}, body=(223 bytes)
DEBUG: Response: {'status': 200, 'headers': {'date': 'Thu, 07 Dec 2017 01:27:16 GMT', 'content-length': '287', 'x-amz-request-id': 'tx0000000000000000004f8-005a2898f4-1709c-sxs3-1', 'content-type': 'application/xml'}, 'reason': 'OK', 'data': '<CompleteMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Location>My-bucket1.s3test1.imu.cn</Location><Bucket>My-bucket1</Bucket><Key>abc.mp4</Key><ETag>65bffc7e3853c1f31369bfddeb003620-2</ETag></CompleteMultipartUploadResult>'}
DEBUG: ConnMan.put(): connection put back to pool (http://s3test1.imu.cn#4)
My local bucket info:
[root@s3-1-7 ~]# radosgw-admin bucket stats --bucket localtest2
{
"bucket": "localtest2",
"zonegroup": "30d51ec6-5ccc-4eb6-92c4-2b9f91dbcaf7",
"placement_rule": "new",
"explicit_placement": {
"data_pool": "",
"data_extra_pool": "",
"index_pool": ""
},
"id": "031f5809-360a-4885-8356-88c660c79c76.75970.1",
"marker": "031f5809-360a-4885-8356-88c660c79c76.75970.1",
"index_type": "Normal",
"owner": "Myliu",
"ver": "0#17",
"master_ver": "0#0",
"mtime": "2017-11-28 14:53:40.121601",
"max_marker": "0#00000000016.17.1",
"usage": {
"rgw.main": {
"size": 0,
"size_actual": 0,
"size_utilized": 0,
"size_kb": 0,
"size_kb_actual": 0,
"size_kb_utilized": 0,
"num_objects": 0
}
},
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
}
}
Updated by Casey Bodley over 6 years ago
Can you share the output of 'radosgw-admin sync status'? The 'Device or resource busy' error you see is unrelated (those get retried automatically).
Updated by Casey Bodley over 6 years ago
- Status changed from New to Need More Info
Updated by Amine Liu over 6 years ago
Casey Bodley wrote:
Can you share the output of 'radosgw-admin sync status'? The 'Device or resource busy' error you see is unrelated (those get retried automatically).
sorry， I've drop my test env for testing jewel 10.2.10, But I remember that the output of 'radosgw-admin sync status' was normal. and put a small text file can sync successfull.
Updated by Yehuda Sadeh about 6 years ago
- Status changed from Need More Info to Can't reproduce