Project

General

Profile

Bug #22982

Objects only serving first 512K

Added by Robin Johnson about 1 year ago. Updated about 1 year ago.

Status:
Pending Backport
Priority:
High
Assignee:
-
Target version:
-
Start date:
02/12/2018
Due date:
% Done:

0%

Source:
Community (dev)
Tags:
rgw
Backport:
luminous
Regression:
Yes
Severity:
2 - major
Reviewed:
ceph-qa-suite:
rgw
Pull request ID:

Description

This has be seen in previous releases, but it's back again in Luminous! Too late for 12.2.3 tag, but I'll apply the fix to our internal build when it's available.

$ wget ...
--2018-02-11 23:44:57--  https://objects-us-west-1.dream.io/BUCKET-CENSORED/FILENAME-CENSORED?Expires=1518424203&Signature=CENSORED&AWSAccessKeyId=CENSORED
Resolving objects-us-west-1.dream.io... 2607:f298:4:143:acce:55:2:1, 64.90.32.8
Connecting to objects-us-west-1.dream.io|2607:f298:4:143:acce:55:2:1|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 11127936 (11M) [audio/x-wav]
Saving to: ‘/dev/null’

/dev/null                                         4%[===>                                                                                                 ] 512.00K  1.90MB/s    in 0.3s    

2018-02-11 23:44:58 (1.90 MB/s) - Connection closed at byte 524288. Retrying.

JSON of object stat

$ radosgw-admin --bucket BUCKET-CENSORED --object 'FILENAME-CENSORED' object stat
{
    "name": "FILENAME-CENSORED",
    "size": 11127936,
    "policy": {
        "acl": {
            "acl_user_map": [
                {
                    "user": "USER-CENSORED",
                    "acl": 15
                }
            ],
            "acl_group_map": [],
            "grant_map": [
                {
                    "id": "USER-CENSORED",
                    "grant": {
                        "type": {
                            "type": 0
                        },
                        "id": "USER-CENSORED",
                        "email": "",
                        "permission": {
                            "flags": 15
                        },
                        "name": "USER-CENSORED",
                        "group": 0,
                        "url_spec": "" 
                    }
                }
            ]
        },
        "owner": {
            "id": "USER-CENSORED",
            "display_name": "USER-CENSORED" 
        }
    },
    "etag": "722e7c40151930b38cdf285659a92d94",
    "tag": "default.52038511.39424258",
    "manifest": {
        "objs": [
            0,
            {
                "loc": {
                    "bucket": {
                        "name": "BUCKET-CENSORED",
                        "marker": "default.51695731.142363",
                        "bucket_id": "default.51695731.142363",
                        "tenant": "",
                        "explicit_placement": {
                            "data_pool": ".rgw.data.1",
                            "data_extra_pool": "",
                            "index_pool": ".rgw.data.1" 
                        }
                    },
                    "key": {
                        "name": "FILENAME-CENSORED",
                        "instance": "",
                        "ns": "" 
                    }
                },
                "loc_ofs": 0,
                "size": 524288
            },
            524288,
            {
                "loc": {
                    "bucket": {
                        "name": "BUCKET-CENSORED",
                        "marker": "default.51695731.142363",
                        "bucket_id": "default.51695731.142363",
                        "tenant": "",
                        "explicit_placement": {
                            "data_pool": ".rgw.data.1",
                            "data_extra_pool": "",
                            "index_pool": ".rgw.data.1" 
                        }
                    },
                    "key": {
                        "name": "__JbcJZZ4eXDtuKeFN8SakI46t0Erh4T5_1",
                        "instance": "",
                        "ns": "shadow" 
                    }
                },
                "loc_ofs": 0,
                "size": 4194304
            },
            4718592,
            {
                "loc": {
                    "bucket": {
                        "name": "BUCKET-CENSORED",
                        "marker": "default.51695731.142363",
                        "bucket_id": "default.51695731.142363",
                        "tenant": "",
                        "explicit_placement": {
                            "data_pool": ".rgw.data.1",
                            "data_extra_pool": "",
                            "index_pool": ".rgw.data.1" 
                        }
                    },
                    "key": {
                        "name": "__JbcJZZ4eXDtuKeFN8SakI46t0Erh4T5_2",
                        "instance": "",
                        "ns": "shadow" 
                    }
                },
                "loc_ofs": 0,
                "size": 4194304
            },
            8912896,
            {
                "loc": {
                    "bucket": {
                        "name": "BUCKET-CENSORED",
                        "marker": "default.51695731.142363",
                        "bucket_id": "default.51695731.142363",
                        "tenant": "",
                        "explicit_placement": {
                            "data_pool": ".rgw.data.1",
                            "data_extra_pool": "",
                            "index_pool": ".rgw.data.1" 
                        }
                    },
                    "key": {
                        "name": "__JbcJZZ4eXDtuKeFN8SakI46t0Erh4T5_3",
                        "instance": "",
                        "ns": "shadow" 
                    }
                },
                "loc_ofs": 0,
                "size": 2215040
            }
        ],
        "obj_size": 11127936,
        "explicit_objs": "true",
        "head_size": 524288,
        "max_head_size": 524288,
        "prefix": "",
        "rules": [],
        "tail_instance": "",
        "tail_placement": {
            "bucket": {
                "name": "",
                "marker": "",
                "bucket_id": "",
                "tenant": "",
                "explicit_placement": {
                    "data_pool": "",
                    "data_extra_pool": "",
                    "index_pool": "" 
                }
            },
            "placement_rule": "" 
        }
    },
    "attrs": {
        "user.rgw.content_type": "audio/x-wav" 
    }
}

Dupont.wav-user.rgw.manifest View - user.rgw.manifest of object (1.06 KB) Robin Johnson, 02/13/2018 04:24 AM


Related issues

Copied to rgw - Backport #23102: luminous: Objects only serving first 512K Resolved

History

#1 Updated by Robin Johnson about 1 year ago

  • Backport set to luminous

#2 Updated by Matt Benjamin about 1 year ago

@robin, can you point out the distinguishing characteristics of this? how is our CI testing missing this?

Matt

#3 Updated by Yehuda Sadeh about 1 year ago

it's an old object with explicit placement, so maybe pre-firefly (?). The tail objects are starting with underscore, so maybe there's some regression there.

#4 Updated by Robin Johnson about 1 year ago

It doesn't start with an underscore in this case, but the data is from at least Dumpling, maybe older (Dumpling->Firefly upgrade happened 2015/Jan)

# rados -p .rgw.data.1 stat default.51695731.142363__shadow__JbcJZZ4eXDtuKeFN8SakI46t0Erh4T5_1
.rgw.data.1/default.51695731.142363__shadow__JbcJZZ4eXDtuKeFN8SakI46t0Erh4T5_1 mtime 2014-03-31 03:52:59.000000, size 4194304

#5 Updated by Robin Johnson about 1 year ago

Ok, Yehuda's underscore could still be correct. This is what the RGW is looking for:
  • default.51695731.142363__shadow___JbcJZZ4eXDtuKeFN8SakI46t0Erh4T5_1 - looking for
  • default.51695731.142363__shadow__JbcJZZ4eXDtuKeFN8SakI46t0Erh4T5_1 - actual
2018-02-12 21:01:40.932636 7f3771123700  1 -- [XXXX::7590]:0/382522710 --> [XXXX::8880]:6809/3072030 -- osd_op(unknown.0.0:69623801 5.cb53 5:cad336d1:::default.51695731.142363__shadow___JbcJZZ4eXDtuKeFN8SakI46t0Erh4T5_1:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e1289786) v8 -- 0x56283b8a59c0 con 0
2018-02-12 21:01:40.933380 7f37be3c3700  1 -- [XXXX::7590]:0/382522710 <== osd.186 [XXXX::8880]:6809/3072030 5444 ==== osd_op_reply(69623801 default.51695731.142363__shadow___JbcJZZ4eXDtuKeFN8SakI46t0Erh4T5_1 [read 0~4194304] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 211+0+0 (880413640 0 0) 0x56283c198580 con 0x56283eb10800

#6 Updated by Yehuda Sadeh about 1 year ago

Can you provide the raw manifest somehow? I want to know if the issue is the way we stored the original objects (that might have been buggy) or the new decoding. To do that you will need to read the manifest off the manifest xattr in the head object.

#9 Updated by Robin Johnson about 1 year ago

Private attachment made w/ manifest & hexdump of manifest.

#10 Updated by Yehuda Sadeh about 1 year ago

Manifest has appropriate raw rados objects there, so we do an unneeded conversion when reading it.

#12 Updated by Robin Johnson about 1 year ago

This change has been deployed at Dreamhost, seems to work so far, watching for other breakage

#13 Updated by Casey Bodley about 1 year ago

  • Status changed from New to Verified

#14 Updated by Casey Bodley about 1 year ago

  • Status changed from Verified to Pending Backport

#15 Updated by Nathan Cutler about 1 year ago

  • Copied to Backport #23102: luminous: Objects only serving first 512K added

Also available in: Atom PDF