Actions
Bug #22982
closedObjects only serving first 512K
Status:
Resolved
Priority:
High
Assignee:
-
Target version:
-
% Done:
0%
Source:
Community (dev)
Tags:
rgw
Backport:
luminous
Regression:
Yes
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
rgw
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
This has be seen in previous releases, but it's back again in Luminous! Too late for 12.2.3 tag, but I'll apply the fix to our internal build when it's available.
$ wget ... --2018-02-11 23:44:57-- https://objects-us-west-1.dream.io/BUCKET-CENSORED/FILENAME-CENSORED?Expires=1518424203&Signature=CENSORED&AWSAccessKeyId=CENSORED Resolving objects-us-west-1.dream.io... 2607:f298:4:143:acce:55:2:1, 64.90.32.8 Connecting to objects-us-west-1.dream.io|2607:f298:4:143:acce:55:2:1|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 11127936 (11M) [audio/x-wav] Saving to: ‘/dev/null’ /dev/null 4%[===> ] 512.00K 1.90MB/s in 0.3s 2018-02-11 23:44:58 (1.90 MB/s) - Connection closed at byte 524288. Retrying.
JSON of object stat
$ radosgw-admin --bucket BUCKET-CENSORED --object 'FILENAME-CENSORED' object stat
{
"name": "FILENAME-CENSORED",
"size": 11127936,
"policy": {
"acl": {
"acl_user_map": [
{
"user": "USER-CENSORED",
"acl": 15
}
],
"acl_group_map": [],
"grant_map": [
{
"id": "USER-CENSORED",
"grant": {
"type": {
"type": 0
},
"id": "USER-CENSORED",
"email": "",
"permission": {
"flags": 15
},
"name": "USER-CENSORED",
"group": 0,
"url_spec": ""
}
}
]
},
"owner": {
"id": "USER-CENSORED",
"display_name": "USER-CENSORED"
}
},
"etag": "722e7c40151930b38cdf285659a92d94",
"tag": "default.52038511.39424258",
"manifest": {
"objs": [
0,
{
"loc": {
"bucket": {
"name": "BUCKET-CENSORED",
"marker": "default.51695731.142363",
"bucket_id": "default.51695731.142363",
"tenant": "",
"explicit_placement": {
"data_pool": ".rgw.data.1",
"data_extra_pool": "",
"index_pool": ".rgw.data.1"
}
},
"key": {
"name": "FILENAME-CENSORED",
"instance": "",
"ns": ""
}
},
"loc_ofs": 0,
"size": 524288
},
524288,
{
"loc": {
"bucket": {
"name": "BUCKET-CENSORED",
"marker": "default.51695731.142363",
"bucket_id": "default.51695731.142363",
"tenant": "",
"explicit_placement": {
"data_pool": ".rgw.data.1",
"data_extra_pool": "",
"index_pool": ".rgw.data.1"
}
},
"key": {
"name": "__JbcJZZ4eXDtuKeFN8SakI46t0Erh4T5_1",
"instance": "",
"ns": "shadow"
}
},
"loc_ofs": 0,
"size": 4194304
},
4718592,
{
"loc": {
"bucket": {
"name": "BUCKET-CENSORED",
"marker": "default.51695731.142363",
"bucket_id": "default.51695731.142363",
"tenant": "",
"explicit_placement": {
"data_pool": ".rgw.data.1",
"data_extra_pool": "",
"index_pool": ".rgw.data.1"
}
},
"key": {
"name": "__JbcJZZ4eXDtuKeFN8SakI46t0Erh4T5_2",
"instance": "",
"ns": "shadow"
}
},
"loc_ofs": 0,
"size": 4194304
},
8912896,
{
"loc": {
"bucket": {
"name": "BUCKET-CENSORED",
"marker": "default.51695731.142363",
"bucket_id": "default.51695731.142363",
"tenant": "",
"explicit_placement": {
"data_pool": ".rgw.data.1",
"data_extra_pool": "",
"index_pool": ".rgw.data.1"
}
},
"key": {
"name": "__JbcJZZ4eXDtuKeFN8SakI46t0Erh4T5_3",
"instance": "",
"ns": "shadow"
}
},
"loc_ofs": 0,
"size": 2215040
}
],
"obj_size": 11127936,
"explicit_objs": "true",
"head_size": 524288,
"max_head_size": 524288,
"prefix": "",
"rules": [],
"tail_instance": "",
"tail_placement": {
"bucket": {
"name": "",
"marker": "",
"bucket_id": "",
"tenant": "",
"explicit_placement": {
"data_pool": "",
"data_extra_pool": "",
"index_pool": ""
}
},
"placement_rule": ""
}
},
"attrs": {
"user.rgw.content_type": "audio/x-wav"
}
}
Files
Updated by Matt Benjamin about 6 years ago
@robin, can you point out the distinguishing characteristics of this? how is our CI testing missing this?
Matt
Updated by Yehuda Sadeh about 6 years ago
it's an old object with explicit placement, so maybe pre-firefly (?). The tail objects are starting with underscore, so maybe there's some regression there.
Updated by Robin Johnson about 6 years ago
It doesn't start with an underscore in this case, but the data is from at least Dumpling, maybe older (Dumpling->Firefly upgrade happened 2015/Jan)
# rados -p .rgw.data.1 stat default.51695731.142363__shadow__JbcJZZ4eXDtuKeFN8SakI46t0Erh4T5_1 .rgw.data.1/default.51695731.142363__shadow__JbcJZZ4eXDtuKeFN8SakI46t0Erh4T5_1 mtime 2014-03-31 03:52:59.000000, size 4194304
Updated by Robin Johnson about 6 years ago
Ok, Yehuda's underscore could still be correct. This is what the RGW is looking for:
default.51695731.142363__shadow___JbcJZZ4eXDtuKeFN8SakI46t0Erh4T5_1- looking fordefault.51695731.142363__shadow__JbcJZZ4eXDtuKeFN8SakI46t0Erh4T5_1- actual
2018-02-12 21:01:40.932636 7f3771123700 1 -- [XXXX::7590]:0/382522710 --> [XXXX::8880]:6809/3072030 -- osd_op(unknown.0.0:69623801 5.cb53 5:cad336d1:::default.51695731.142363__shadow___JbcJZZ4eXDtuKeFN8SakI46t0Erh4T5_1:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e1289786) v8 -- 0x56283b8a59c0 con 0 2018-02-12 21:01:40.933380 7f37be3c3700 1 -- [XXXX::7590]:0/382522710 <== osd.186 [XXXX::8880]:6809/3072030 5444 ==== osd_op_reply(69623801 default.51695731.142363__shadow___JbcJZZ4eXDtuKeFN8SakI46t0Erh4T5_1 [read 0~4194304] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (880413640 0 0) 0x56283c198580 con 0x56283eb10800
Updated by Yehuda Sadeh about 6 years ago
Can you provide the raw manifest somehow? I want to know if the issue is the way we stored the original objects (that might have been buggy) or the new decoding. To do that you will need to read the manifest off the manifest xattr in the head object.
Updated by Robin Johnson about 6 years ago
Updated by Robin Johnson about 6 years ago
Private attachment made w/ manifest & hexdump of manifest.
Updated by Yehuda Sadeh about 6 years ago
Manifest has appropriate raw rados objects there, so we do an unneeded conversion when reading it.
Updated by Yehuda Sadeh about 6 years ago
a possible fix:
https://github.com/ceph/ceph/pull/20425
Updated by Robin Johnson about 6 years ago
This change has been deployed at Dreamhost, seems to work so far, watching for other breakage
Updated by Casey Bodley about 6 years ago
- Status changed from 12 to Pending Backport
Updated by Nathan Cutler about 6 years ago
- Copied to Backport #23102: luminous: Objects only serving first 512K added
Updated by Nathan Cutler over 4 years ago
- Status changed from Pending Backport to Resolved
While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are in status "Resolved".
Actions