Bug #23939
etag/hash and content_type fields were concatenated wit \u0000
0%
Description
We are running radosgw(Luminous) with Swift API enabled. We observed
that after updating an object the "hash" and "content_type"
fields were concatenated with "\u000".
Steps to reproduce the issue.
[1] Create a container(Swift nomenclature)
[2] Upload a file with "swift --debug upload <name-of-container> <filename>"
[3] Make a change to the file.
[4] Update with "swift --debug post <container-name> <filename>"
[5] List again with "swift --debug list <container>"
After step 5, the RESP BODY is like this:
{
"bytes": 251,
"content_type": "text/plain\u0000",
"hash": "8df6558bcaed354cbf24aa3b4a790fad\u0000",
"last_modified": "2018-04-23T06:21:52.574Z",
"name": "test.txt"
}
As you can see there is a "\u0000" character in both "content_type" and "hash" fields.
Also, when we used the "radosgw-admin object stat" we observed this:
- diff beforeUpdate.txt afterUpdate.txt
39c39
< "tag": "7626b755-713f-4be3-9062-9328ad3fa425.6729.9",
---
"tag": "_GmhB3jfJWOV7yxkzk0GdrL_8Z71pVN7",
79,80c79
< "user.rgw.tail_tag": "7626b755-713f-4be3-9062-9328ad3fa425.6729.9",
< "user.rgw.x-amz-meta-mtime": "1524468290.332091"
---
"user.rgw.tail_tag": "7626b755-713f-4be3-9062-9328ad3fa425.6729.9"
Please note that the "tag" field changed after update and "user.rgw.x-amz-meta-mtime" disappeared.
beforeUpdate.txt --> http://paste.openstack.org/show/719723/
afterUpdate.txt --> http://paste.openstack.org/show/719724/
We are using a combination of native swift clusters and radsogw based swift clusters.
Our ambition is to switch to radosgw completely. At the moment issues like above are
breaking clients which are not able to parse information because of the presence of
extra "\u0000" character. We donot see any such issue with the native Swift clusters.
History
#1 Updated by Syed Armani almost 6 years ago
Syed Armani wrote:
We are running radosgw(Luminous) with Swift API enabled. We observed
that after updating an object the "hash" and "content_type"
fields were concatenated with "\u000".Steps to reproduce the issue.
[1] Create a container(Swift nomenclature)
[2] Upload a file with swift --debug upload <name-of-container> <filename>
[3] Make a change to the file.
[4] Update with swift --debug post <container-name> <filename>
[5] List again with swift --debug list <container>
#2 Updated by Nathan Cutler almost 6 years ago
- Project changed from Ceph to rgw
- Category deleted (
openstack)
#3 Updated by Florian Haas almost 6 years ago
I can confirm that this issue is still present in 12.2.5:
Ceph and Swift versions¶
$ swift --version python-swiftclient 3.5.0 $ sudo radosgw --version ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable) $ sudo ceph version ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable)
Also, for brevity, I use this alias for the
swift
command:
$ alias swift alias swift='swift -A http://192.168.122.100:7480/auth/1.0 -U <uid>:swift -K <swift key>'
Container/object creation¶
Create a container and upload an object, then list the container (observe that the RESP BODY
on the etc/issue
object contains nothing unexpected):
$ swift list $ swift post test $ swift upload test /etc/issue etc/issue $ swift list --debug 2>&1 test | grep RESP DEBUG:swiftclient:RESP STATUS: 204 No Content DEBUG:swiftclient:RESP HEADERS: {u'X-Storage-Token': u'AUTH_rgwtk130000004a616e65576869726c706f6f6c3a7377696674174cce08dedc8ac24e2ee85aca113c34318e7249da9d4eefc19de7331a78faf2ced30c77', u'X-Auth-Token': u'AUTH_rgwtk130000004a616e65576869726c706f6f6c3a7377696674174cce08dedc8ac24e2ee85aca113c34318e7249da9d4eefc19de7331a78faf2ced30c77', u'X-Trans-Id': u'tx000000000000000000062-005ae6dcce-1971b-default', u'Date': u'Mon, 30 Apr 2018 09:07:26 GMT', u'X-Storage-Url': u'http://192.168.122.100:7480/swift/v1', u'Content-Type': u'application/json; charset=utf-8', u'X-Openstack-Request-Id': u'tx000000000000000000062-005ae6dcce-1971b-default'} DEBUG:swiftclient:RESP STATUS: 200 OK DEBUG:swiftclient:RESP HEADERS: {u'Content-Length': u'118', u'X-Container-Object-Count': u'1', u'Accept-Ranges': u'bytes', u'X-Storage-Policy': u'default-placement', u'X-Container-Bytes-Used-Actual': u'4096', u'Date': u'Mon, 30 Apr 2018 09:07:26 GMT', u'X-Timestamp': u'1525079218.04990', u'X-Trans-Id': u'tx000000000000000000063-005ae6dcce-1971b-default', u'X-Container-Bytes-Used': u'26', u'Content-Type': u'application/json; charset=utf-8', u'X-Openstack-Request-Id': u'tx000000000000000000063-005ae6dcce-1971b-default'} DEBUG:swiftclient:RESP BODY: [{"name":"etc/issue","hash":"df83af441843aaf15b83ebec82fb2c5a","bytes":26,"last_modified":"2018-04-30T09:07:07.795Z"}] DEBUG:swiftclient:RESP STATUS: 200 OK DEBUG:swiftclient:RESP HEADERS: {u'Content-Length': u'2', u'X-Container-Object-Count': u'1', u'Accept-Ranges': u'bytes', u'X-Storage-Policy': u'default-placement', u'X-Container-Bytes-Used-Actual': u'4096', u'Date': u'Mon, 30 Apr 2018 09:07:26 GMT', u'X-Timestamp': u'1525079218.04990', u'X-Trans-Id': u'tx000000000000000000064-005ae6dcce-1971b-default', u'X-Container-Bytes-Used': u'26', u'Content-Type': u'application/json; charset=utf-8', u'X-Openstack-Request-Id': u'tx000000000000000000064-005ae6dcce-1971b-default'} DEBUG:swiftclient:RESP BODY: []
Object update¶
Issue a swift post
command, observe \u0000
being appended to the object hash in the container list (the first RESP BODY
in the debug trace below).
$ swift post test etc/issue $ swift list --debug 2>&1 test | grep RESP DEBUG:swiftclient:RESP STATUS: 204 No Content DEBUG:swiftclient:RESP HEADERS: {u'X-Storage-Token': u'AUTH_rgwtk130000004a616e65576869726c706f6f6c3a737769667431854bb43d258489672ee85a3b4ae50874939940049b80415763b31134cca8710b8810ab', u'X-Auth-Token': u'AUTH_rgwtk130000004a616e65576869726c706f6f6c3a737769667431854bb43d258489672ee85a3b4ae50874939940049b80415763b31134cca8710b8810ab', u'X-Trans-Id': u'tx000000000000000000069-005ae6dce7-1971b-default', u'Date': u'Mon, 30 Apr 2018 09:07:51 GMT', u'X-Storage-Url': u'http://192.168.122.100:7480/swift/v1', u'Content-Type': u'application/json; charset=utf-8', u'X-Openstack-Request-Id': u'tx000000000000000000069-005ae6dce7-1971b-default'} DEBUG:swiftclient:RESP STATUS: 200 OK DEBUG:swiftclient:RESP HEADERS: {u'Content-Length': u'124', u'X-Container-Object-Count': u'1', u'Accept-Ranges': u'bytes', u'X-Storage-Policy': u'default-placement', u'X-Container-Bytes-Used-Actual': u'4096', u'Date': u'Mon, 30 Apr 2018 09:07:51 GMT', u'X-Timestamp': u'1525079218.04990', u'X-Trans-Id': u'tx00000000000000000006a-005ae6dce7-1971b-default', u'X-Container-Bytes-Used': u'26', u'Content-Type': u'application/json; charset=utf-8', u'X-Openstack-Request-Id': u'tx00000000000000000006a-005ae6dce7-1971b-default'} DEBUG:swiftclient:RESP BODY: [{"name":"etc/issue","hash":"df83af441843aaf15b83ebec82fb2c5a\u0000","bytes":26,"last_modified":"2018-04-30T09:07:47.091Z"}] DEBUG:swiftclient:RESP STATUS: 200 OK DEBUG:swiftclient:RESP HEADERS: {u'Content-Length': u'2', u'X-Container-Object-Count': u'1', u'Accept-Ranges': u'bytes', u'X-Storage-Policy': u'default-placement', u'X-Container-Bytes-Used-Actual': u'4096', u'Date': u'Mon, 30 Apr 2018 09:07:51 GMT', u'X-Timestamp': u'1525079218.04990', u'X-Trans-Id': u'tx00000000000000000006b-005ae6dce7-1971b-default', u'X-Container-Bytes-Used': u'26', u'Content-Type': u'application/json; charset=utf-8', u'X-Openstack-Request-Id': u'tx00000000000000000006b-005ae6dce7-1971b-default'} DEBUG:swiftclient:RESP BODY: []
Somewhat confusingly, the appended \u0000
seems to present only in the object metadata as listed in a swift list
command issued against a container, whereas swift stat
against the individual object appears to be fine. See the swift stat
command below, where the etag
does not contain the null character:
$ swift stat --debug test etc/issue DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): 192.168.122.100 DEBUG:urllib3.connectionpool:http://192.168.122.100:7480 "GET /auth/1.0 HTTP/1.1" 204 0 DEBUG:swiftclient:REQ: curl -i http://192.168.122.100:7480/auth/1.0 -X GET DEBUG:swiftclient:RESP STATUS: 204 No Content DEBUG:swiftclient:RESP HEADERS: {u'X-Storage-Token': u'AUTH_rgwtk130000004a616e65576869726c706f6f6c3a7377696674a5fd19d5733d57cf853ae85a55cb66253b0bbd3af83c4e93729ddc4b27b0ece8e26ffc5a', u'X-Auth-Token': u'AUTH_rgwtk130000004a616e65576869726c706f6f6c3a7377696674a5fd19d5733d57cf853ae85a55cb66253b0bbd3af83c4e93729ddc4b27b0ece8e26ffc5a', u'X-Trans-Id': u'tx0000000000000000000a2-005ae6e905-1971b-default', u'Date': u'Mon, 30 Apr 2018 09:59:33 GMT', u'X-Storage-Url': u'http://192.168.122.100:7480/swift/v1', u'Content-Type': u'application/json; charset=utf-8', u'X-Openstack-Request-Id': u'tx0000000000000000000a2-005ae6e905-1971b-default'} DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): 192.168.122.100 DEBUG:urllib3.connectionpool:http://192.168.122.100:7480 "HEAD /swift/v1/test/etc/issue HTTP/1.1" 200 0 DEBUG:swiftclient:REQ: curl -i http://192.168.122.100:7480/swift/v1/test/etc/issue -I -H "X-Auth-Token: AUTH_rgwtk130000004a616e65576869726c706f6f6c3a7377696674a5fd19d5733d57cf853ae85a55cb66253b0bbd3af83c4e93729ddc4b27b0ece8e26ffc5a" DEBUG:swiftclient:RESP STATUS: 200 OK DEBUG:swiftclient:RESP HEADERS: {u'Content-Length': u'26', u'Accept-Ranges': u'bytes', u'Last-Modified': u'Mon, 30 Apr 2018 09:59:21 GMT', u'etag': u'df83af441843aaf15b83ebec82fb2c5a', u'X-Timestamp': u'1525082361.86576', u'X-Trans-Id': u'tx0000000000000000000a3-005ae6e905-1971b-default', u'Date': u'Mon, 30 Apr 2018 09:59:33 GMT', u'Content-Type': u'binary/octet-stream', u'X-Openstack-Request-Id': u'tx0000000000000000000a3-005ae6e905-1971b-default'} Account: v1 Container: test Object: etc/issue Content Type: binary/octet-stream Content Length: 26 Last Modified: Mon, 30 Apr 2018 09:59:21 GMT ETag: df83af441843aaf15b83ebec82fb2c5a Accept-Ranges: bytes X-Timestamp: 1525082361.86576 X-Trans-Id: tx0000000000000000000a3-005ae6e905-1971b-default X-Openstack-Request-Id: tx0000000000000000000a3-005ae6e905-1971b-default
#4 Updated by Syed Armani almost 6 years ago
Affected Versions: 12.2.2 and 12.2.5
Syed Armani wrote:
Syed Armani wrote:
We are running radosgw(Luminous) with Swift API enabled. We observed
that after updating an object the "hash" and "content_type"
fields were concatenated with "\u000".Steps to reproduce the issue.
[1] Create a container(Swift nomenclature)
[2] Upload a file with swift --debug upload <name-of-container> <filename>
[3] Make a change to the file.
[4] Update with swift --debug post <container-name> <filename>
[5] List again with swift --debug list <container>
#5 Updated by Orit Wasserman almost 6 years ago
- Assignee set to Orit Wasserman
#6 Updated by Matt Benjamin almost 6 years ago
- Status changed from New to Triaged
yehuda thinks may be fixed in mimic
#7 Updated by Florian Haas almost 6 years ago
Matt Benjamin wrote:
yehuda thinks may be fixed in mimic
Could you clarify what is meant here please? Should we read this as "we may be able to fix this in the mimic cycle" or as "it's possible that this is already fixed in the mimic RC, could you give that a whirl"?
#8 Updated by Yehuda Sadeh almost 6 years ago
It should be fixed in Mimic. However, I'm not completely sure that objects that were created prior will not have the same issue (need to be tested).