Project

General

Profile

Bug #23939

etag/hash and content_type fields were concatenated wit \u0000

Added by Syed Armani almost 6 years ago. Updated almost 6 years ago.

Status:
Triaged
Priority:
Normal
Target version:
% Done:

0%

Source:
Community (user)
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

We are running radosgw(Luminous) with Swift API enabled. We observed
that after updating an object the "hash" and "content_type"
fields were concatenated with "\u000".

Steps to reproduce the issue.

[1] Create a container(Swift nomenclature)
[2] Upload a file with "swift --debug upload <name-of-container> <filename>"
[3] Make a change to the file.
[4] Update with "swift --debug post <container-name> <filename>"
[5] List again with "swift --debug list <container>"

After step 5, the RESP BODY is like this:

{
"bytes": 251,
"content_type": "text/plain\u0000",
"hash": "8df6558bcaed354cbf24aa3b4a790fad\u0000",
"last_modified": "2018-04-23T06:21:52.574Z",
"name": "test.txt"
}

As you can see there is a "\u0000" character in both "content_type" and "hash" fields.

Also, when we used the "radosgw-admin object stat" we observed this:

  1. diff beforeUpdate.txt afterUpdate.txt
    39c39
    < "tag": "7626b755-713f-4be3-9062-9328ad3fa425.6729.9",
    ---

"tag": "_GmhB3jfJWOV7yxkzk0GdrL_8Z71pVN7",

79,80c79
< "user.rgw.tail_tag": "7626b755-713f-4be3-9062-9328ad3fa425.6729.9",
< "user.rgw.x-amz-meta-mtime": "1524468290.332091"
---

"user.rgw.tail_tag": "7626b755-713f-4be3-9062-9328ad3fa425.6729.9"

Please note that the "tag" field changed after update and "user.rgw.x-amz-meta-mtime" disappeared.

beforeUpdate.txt --> http://paste.openstack.org/show/719723/
afterUpdate.txt --> http://paste.openstack.org/show/719724/

We are using a combination of native swift clusters and radsogw based swift clusters.
Our ambition is to switch to radosgw completely. At the moment issues like above are
breaking clients which are not able to parse information because of the presence of
extra "\u0000" character. We donot see any such issue with the native Swift clusters.

History

#1 Updated by Syed Armani almost 6 years ago

Syed Armani wrote:

We are running radosgw(Luminous) with Swift API enabled. We observed
that after updating an object the "hash" and "content_type"
fields were concatenated with "\u000".

Steps to reproduce the issue.

[1] Create a container(Swift nomenclature)
[2] Upload a file with swift --debug upload <name-of-container> <filename>
[3] Make a change to the file.
[4] Update with swift --debug post <container-name> <filename>
[5] List again with swift --debug list <container>

#2 Updated by Nathan Cutler almost 6 years ago

  • Project changed from Ceph to rgw
  • Category deleted (openstack)

#3 Updated by Florian Haas almost 6 years ago

I can confirm that this issue is still present in 12.2.5:

Ceph and Swift versions

$ swift --version
python-swiftclient 3.5.0
$ sudo radosgw --version
ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable)
$ sudo ceph version
ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable)


Also, for brevity, I use this alias for the swift command:
$ alias swift
alias swift='swift -A http://192.168.122.100:7480/auth/1.0 -U <uid>:swift -K <swift key>'

Container/object creation

Create a container and upload an object, then list the container (observe that the RESP BODY on the etc/issue object contains nothing unexpected):

$ swift list     
$ swift post test
$ swift upload test /etc/issue
etc/issue
$ swift list --debug 2>&1 test | grep RESP
DEBUG:swiftclient:RESP STATUS: 204 No Content
DEBUG:swiftclient:RESP HEADERS: {u'X-Storage-Token': u'AUTH_rgwtk130000004a616e65576869726c706f6f6c3a7377696674174cce08dedc8ac24e2ee85aca113c34318e7249da9d4eefc19de7331a78faf2ced30c77', u'X-Auth-Token': u'AUTH_rgwtk130000004a616e65576869726c706f6f6c3a7377696674174cce08dedc8ac24e2ee85aca113c34318e7249da9d4eefc19de7331a78faf2ced30c77', u'X-Trans-Id': u'tx000000000000000000062-005ae6dcce-1971b-default', u'Date': u'Mon, 30 Apr 2018 09:07:26 GMT', u'X-Storage-Url': u'http://192.168.122.100:7480/swift/v1', u'Content-Type': u'application/json; charset=utf-8', u'X-Openstack-Request-Id': u'tx000000000000000000062-005ae6dcce-1971b-default'}
DEBUG:swiftclient:RESP STATUS: 200 OK
DEBUG:swiftclient:RESP HEADERS: {u'Content-Length': u'118', u'X-Container-Object-Count': u'1', u'Accept-Ranges': u'bytes', u'X-Storage-Policy': u'default-placement', u'X-Container-Bytes-Used-Actual': u'4096', u'Date': u'Mon, 30 Apr 2018 09:07:26 GMT', u'X-Timestamp': u'1525079218.04990', u'X-Trans-Id': u'tx000000000000000000063-005ae6dcce-1971b-default', u'X-Container-Bytes-Used': u'26', u'Content-Type': u'application/json; charset=utf-8', u'X-Openstack-Request-Id': u'tx000000000000000000063-005ae6dcce-1971b-default'}
DEBUG:swiftclient:RESP BODY: [{"name":"etc/issue","hash":"df83af441843aaf15b83ebec82fb2c5a","bytes":26,"last_modified":"2018-04-30T09:07:07.795Z"}]
DEBUG:swiftclient:RESP STATUS: 200 OK
DEBUG:swiftclient:RESP HEADERS: {u'Content-Length': u'2', u'X-Container-Object-Count': u'1', u'Accept-Ranges': u'bytes', u'X-Storage-Policy': u'default-placement', u'X-Container-Bytes-Used-Actual': u'4096', u'Date': u'Mon, 30 Apr 2018 09:07:26 GMT', u'X-Timestamp': u'1525079218.04990', u'X-Trans-Id': u'tx000000000000000000064-005ae6dcce-1971b-default', u'X-Container-Bytes-Used': u'26', u'Content-Type': u'application/json; charset=utf-8', u'X-Openstack-Request-Id': u'tx000000000000000000064-005ae6dcce-1971b-default'}
DEBUG:swiftclient:RESP BODY: []

Object update

Issue a swift post command, observe \u0000 being appended to the object hash in the container list (the first RESP BODY in the debug trace below).

$ swift post test etc/issue 
$ swift list --debug 2>&1 test | grep RESP
DEBUG:swiftclient:RESP STATUS: 204 No Content
DEBUG:swiftclient:RESP HEADERS: {u'X-Storage-Token': u'AUTH_rgwtk130000004a616e65576869726c706f6f6c3a737769667431854bb43d258489672ee85a3b4ae50874939940049b80415763b31134cca8710b8810ab', u'X-Auth-Token': u'AUTH_rgwtk130000004a616e65576869726c706f6f6c3a737769667431854bb43d258489672ee85a3b4ae50874939940049b80415763b31134cca8710b8810ab', u'X-Trans-Id': u'tx000000000000000000069-005ae6dce7-1971b-default', u'Date': u'Mon, 30 Apr 2018 09:07:51 GMT', u'X-Storage-Url': u'http://192.168.122.100:7480/swift/v1', u'Content-Type': u'application/json; charset=utf-8', u'X-Openstack-Request-Id': u'tx000000000000000000069-005ae6dce7-1971b-default'}
DEBUG:swiftclient:RESP STATUS: 200 OK
DEBUG:swiftclient:RESP HEADERS: {u'Content-Length': u'124', u'X-Container-Object-Count': u'1', u'Accept-Ranges': u'bytes', u'X-Storage-Policy': u'default-placement', u'X-Container-Bytes-Used-Actual': u'4096', u'Date': u'Mon, 30 Apr 2018 09:07:51 GMT', u'X-Timestamp': u'1525079218.04990', u'X-Trans-Id': u'tx00000000000000000006a-005ae6dce7-1971b-default', u'X-Container-Bytes-Used': u'26', u'Content-Type': u'application/json; charset=utf-8', u'X-Openstack-Request-Id': u'tx00000000000000000006a-005ae6dce7-1971b-default'}
DEBUG:swiftclient:RESP BODY: [{"name":"etc/issue","hash":"df83af441843aaf15b83ebec82fb2c5a\u0000","bytes":26,"last_modified":"2018-04-30T09:07:47.091Z"}]
DEBUG:swiftclient:RESP STATUS: 200 OK
DEBUG:swiftclient:RESP HEADERS: {u'Content-Length': u'2', u'X-Container-Object-Count': u'1', u'Accept-Ranges': u'bytes', u'X-Storage-Policy': u'default-placement', u'X-Container-Bytes-Used-Actual': u'4096', u'Date': u'Mon, 30 Apr 2018 09:07:51 GMT', u'X-Timestamp': u'1525079218.04990', u'X-Trans-Id': u'tx00000000000000000006b-005ae6dce7-1971b-default', u'X-Container-Bytes-Used': u'26', u'Content-Type': u'application/json; charset=utf-8', u'X-Openstack-Request-Id': u'tx00000000000000000006b-005ae6dce7-1971b-default'}
DEBUG:swiftclient:RESP BODY: []

Somewhat confusingly, the appended \u0000 seems to present only in the object metadata as listed in a swift list command issued against a container, whereas swift stat against the individual object appears to be fine. See the swift stat command below, where the etag does not contain the null character:

$ swift stat --debug test etc/issue
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): 192.168.122.100
DEBUG:urllib3.connectionpool:http://192.168.122.100:7480 "GET /auth/1.0 HTTP/1.1" 204 0
DEBUG:swiftclient:REQ: curl -i http://192.168.122.100:7480/auth/1.0 -X GET
DEBUG:swiftclient:RESP STATUS: 204 No Content
DEBUG:swiftclient:RESP HEADERS: {u'X-Storage-Token': u'AUTH_rgwtk130000004a616e65576869726c706f6f6c3a7377696674a5fd19d5733d57cf853ae85a55cb66253b0bbd3af83c4e93729ddc4b27b0ece8e26ffc5a', u'X-Auth-Token': u'AUTH_rgwtk130000004a616e65576869726c706f6f6c3a7377696674a5fd19d5733d57cf853ae85a55cb66253b0bbd3af83c4e93729ddc4b27b0ece8e26ffc5a', u'X-Trans-Id': u'tx0000000000000000000a2-005ae6e905-1971b-default', u'Date': u'Mon, 30 Apr 2018 09:59:33 GMT', u'X-Storage-Url': u'http://192.168.122.100:7480/swift/v1', u'Content-Type': u'application/json; charset=utf-8', u'X-Openstack-Request-Id': u'tx0000000000000000000a2-005ae6e905-1971b-default'}
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): 192.168.122.100
DEBUG:urllib3.connectionpool:http://192.168.122.100:7480 "HEAD /swift/v1/test/etc/issue HTTP/1.1" 200 0
DEBUG:swiftclient:REQ: curl -i http://192.168.122.100:7480/swift/v1/test/etc/issue -I -H "X-Auth-Token: AUTH_rgwtk130000004a616e65576869726c706f6f6c3a7377696674a5fd19d5733d57cf853ae85a55cb66253b0bbd3af83c4e93729ddc4b27b0ece8e26ffc5a" 
DEBUG:swiftclient:RESP STATUS: 200 OK
DEBUG:swiftclient:RESP HEADERS: {u'Content-Length': u'26', u'Accept-Ranges': u'bytes', u'Last-Modified': u'Mon, 30 Apr 2018 09:59:21 GMT', u'etag': u'df83af441843aaf15b83ebec82fb2c5a', u'X-Timestamp': u'1525082361.86576', u'X-Trans-Id': u'tx0000000000000000000a3-005ae6e905-1971b-default', u'Date': u'Mon, 30 Apr 2018 09:59:33 GMT', u'Content-Type': u'binary/octet-stream', u'X-Openstack-Request-Id': u'tx0000000000000000000a3-005ae6e905-1971b-default'}
               Account: v1
             Container: test
                Object: etc/issue
          Content Type: binary/octet-stream
        Content Length: 26
         Last Modified: Mon, 30 Apr 2018 09:59:21 GMT
                  ETag: df83af441843aaf15b83ebec82fb2c5a
         Accept-Ranges: bytes
           X-Timestamp: 1525082361.86576
            X-Trans-Id: tx0000000000000000000a3-005ae6e905-1971b-default
X-Openstack-Request-Id: tx0000000000000000000a3-005ae6e905-1971b-default

#4 Updated by Syed Armani almost 6 years ago

Affected Versions: 12.2.2 and 12.2.5

Syed Armani wrote:

Syed Armani wrote:

We are running radosgw(Luminous) with Swift API enabled. We observed
that after updating an object the "hash" and "content_type"
fields were concatenated with "\u000".

Steps to reproduce the issue.

[1] Create a container(Swift nomenclature)
[2] Upload a file with swift --debug upload <name-of-container> <filename>
[3] Make a change to the file.
[4] Update with swift --debug post <container-name> <filename>
[5] List again with swift --debug list <container>

#5 Updated by Orit Wasserman almost 6 years ago

  • Assignee set to Orit Wasserman

#6 Updated by Matt Benjamin almost 6 years ago

  • Status changed from New to Triaged

yehuda thinks may be fixed in mimic

#7 Updated by Florian Haas almost 6 years ago

Matt Benjamin wrote:

yehuda thinks may be fixed in mimic

Could you clarify what is meant here please? Should we read this as "we may be able to fix this in the mimic cycle" or as "it's possible that this is already fixed in the mimic RC, could you give that a whirl"?

#8 Updated by Yehuda Sadeh almost 6 years ago

It should be fixed in Mimic. However, I'm not completely sure that objects that were created prior will not have the same issue (need to be tested).

Also available in: Atom PDF