Ceph » rgw

I'm working to validate Ceph Nautilus with OpenStack Stein release.

One of the scenarios we test is using radosgw as a swift backend for glance images which uses dynamic large objects. This scenario is failing to upload the manifest to swift when specifying etag in the request headers with the checksum of empty string:

Put of segment:

PUT /swift/v1/glance/643f4dda-4bc5-46bd-85f4-a6cd3ca8c910-00001 HTTP/1.1
    Host: [::1]:8080
    user-agent: python-swiftclient-3.7.0
    x-auth-token: gAAAAABcrMzaXzoSj38LNsVDo8WMVMjlTm4Os9XCNICiFLsv00dB5xyVdVU3OU1sYh9huDyUYHurWQYpKoZ-JKOUYNrhAgG0XMifSadSm4NDXvvo41ciqP-Xet3LIx_T2XsvF57nt5ghHSOPY3Qr2kkaX7eELXoebqZJ8d0I7tqsVI9QWQdD19c
    Transfer-Encoding: chunked

Response of segment 1:

HTTP/1.1 201 Created
    etag: 9ed16eaedad2389750bcee3aca0c6a8e
    Last-Modified: Tue, 09 Apr 2019 16:48:28 GMT
    X-Trans-Id: tx000000000000000000002-005cacccdb-b491-default
    X-Openstack-Request-Id: tx000000000000000000002-005cacccdb-b491-default
    Content-Type: text/plain; charset=utf-8
    Content-Length: 0
    Date: Tue, 09 Apr 2019 16:48:28 GMT

Request of the manifest:

PUT /swift/v1/glance/643f4dda-4bc5-46bd-85f4-a6cd3ca8c910 HTTP/1.1
    Host: [::1]:8080
    Accept-Encoding: identity
    x-auth-token: gAAAAABcrMzaXzoSj38LNsVDo8WMVMjlTm4Os9XCNICiFLsv00dB5xyVdVU3OU1sYh9huDyUYHurWQYpKoZ-JKOUYNrhAgG0XMifSadSm4NDXvvo41ciqP-Xet3LIx_T2XsvF57nt5ghHSOPY3Qr2kkaX7eELXoebqZJ8d0I7tqsVI9QWQdD19c
    content-length: 0
    etag: d41d8cd98f00b204e9800998ecf8427e
    user-agent: python-swiftclient-3.7.0
    x-object-manifest: glance/643f4dda-4bc5-46bd-85f4-a6cd3ca8c910-

Response for the manifest put:

HTTP/1.1 422 Unprocessable Entity
    Content-Length: 19
    etag: b0e641c998cc3eae6fa2f8726d98cddd
    Last-Modified: Thu, 01 Jan 1970 00:00:00 GMT
    X-Trans-Id: tx000000000000000000003-005cacccdc-b491-default
    X-Openstack-Request-Id: tx000000000000000000003-005cacccdc-b491-default
    Accept-Ranges: bytes
    Content-Type: text/plain; charset=utf-8
    Date: Tue, 09 Apr 2019 16:48:29 GMT

Note the 422 http code in response.

The same exact workflow with same etag in the request work for previous versions of ceph, luminous and mimic and using swift project (as provided by OpenStack), so i guess something has changed in how etag is handled in chunked uploads which makes it to behave differently.

As a note, if i don't include etag header in the manifest upload request, it works fine and returns following reply:

HTTP/1.1 201 Created
    etag: b0e641c998cc3eae6fa2f8726d98cddd
    Last-Modified: Tue, 09 Apr 2019 16:26:56 GMT
    X-Trans-Id: tx00000000000000000006d-005cacc7cf-8b72-default
    X-Openstack-Request-Id: tx00000000000000000006d-005cacc7cf-8b72-default
    Content-Type: text/plain; charset=utf-8
    Content-Length: 0
    Date: Tue, 09 Apr 2019 16:26:56 GMT

I can't find where that etag: b0e641c998cc3eae6fa2f8726d98cddd is coming from.