Project

General

Profile

Bug #46295

RGW returns 404 code for unauthorized instead of 401

Added by Dmitriy Rabotyagov about 1 month ago. Updated about 1 month ago.

Status:
Fix Under Review
Priority:
Normal
Assignee:
-
Target version:
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
ceph-ansible, rgw
Pull request ID:
Crash signature:

Description

Right after the release of ceph 14.2.10 and 15.2.4 our CI start failing because of the changed return code for unauthorized actions from 401 to the 404. We run the following test: https://opendev.org/openstack/tempest/src/branch/master/tempest/scenario/test_object_storage_basic_ops.py#L65

Ceph is deployed with ceph-ansible. And our patches that were passing 3 days ago on 14.2.9 and 15.2.3 (but were not merged in time) are now failing with MismatchError: 404 != 401

We have rgw log here

These logs are from 14.2.10 installation but it's the same for 15.2.4

As a result return codes now are incompatible with swift ones.

All logs are placed here for 14.2.10 and 15.2.4

History

#1 Updated by Matthew Oliver about 1 month ago

Interesting there was a patch to fix returning 401 over 404 for an anon user was trying to GET an object that didn't exist. https://github.com/ceph/ceph/commit/b03d9754e113d24221f1ce0bac17556ab0017a8a

Let me see if this somehow effected the opposite. 401 for anon GET to a obj that exists.

#2 Updated by Matthew Oliver about 1 month ago

OK so I built up the current master then had a quick play and it seems to work fine for Swift V1 auth:

Put an object:

suse@ironic-moliver:~/code/ceph/build(master)$ curl -i -H "X-Auth-Token: $OS_AUTH_TOKEN" $OS_STORAGE_URL/cont/obj -X PUT --data-binary '1234'                                                                                                
HTTP/1.1 201 Created
etag: 81dc9bdb52d04dc20036dbd8313ed055
Last-Modified: Sun, 05 Jul 2020 23:15:30 GMT
X-Trans-Id: tx000000000000000000005-005f025f0f-1145-default
X-Openstack-Request-Id: tx000000000000000000005-005f025f0f-1145-default
Content-Type: text/plain; charset=utf-8
Content-Length: 0
Date: Sun, 05 Jul 2020 23:15:30 GMT
Connection: Keep-Alive

GET it as the auth user:

suse@ironic-moliver:~/code/ceph/build(master)$ curl -i -H "X-Auth-Token: $OS_AUTH_TOKEN" $OS_STORAGE_URL/cont/obj
HTTP/1.1 200 OK
Content-Length: 4
Accept-Ranges: bytes
Last-Modified: Sun, 05 Jul 2020 23:15:30 GMT
X-Timestamp: 1593990930.18153
etag: 81dc9bdb52d04dc20036dbd8313ed055
X-Trans-Id: tx000000000000000000006-005f025f19-1145-default
X-Openstack-Request-Id: tx000000000000000000006-005f025f19-1145-default
Content-Type: application/x-www-form-urlencoded
Date: Sun, 05 Jul 2020 23:15:37 GMT
Connection: Keep-Alive
1234

When I attempt to get it unauthorised (minus the token) we get a 401:

suse@ironic-moliver:~/code/ceph/build(master)$ curl -i $OS_STORAGE_URL/cont/obj                                                                                                                                                              
HTTP/1.1 401 Unauthorized
Content-Length: 12
X-Trans-Id: tx000000000000000000007-005f025f29-1145-default
X-Openstack-Request-Id: tx000000000000000000007-005f025f29-1145-default
Accept-Ranges: bytes
Content-Type: text/plain; charset=utf-8
Date: Sun, 05 Jul 2020 23:15:53 GMT
Connection: Keep-Alive

So maybe it's related to anon keystone auth? So will set up keystone, test again, and if I still fail to reproduce, I'll then at least have an endpoint to point tempest at :)

#3 Updated by Dmitriy Rabotyagov about 1 month ago

Ok, so that's what I get in my sandbox, I've just created:

root@aio1-utility-container-4a6caa82:/# curl http://172.29.236.100:8080/swift/v1/AUTH_4d39f320a46f4a51970b64fd0fce912d/test/obj1 -i -H "X-Auth-Token: $TOKEN" -X PUT --data-binary '1234'
HTTP/1.1 201 Created
etag: 81dc9bdb52d04dc20036dbd8313ed055
Last-Modified: Mon, 06 Jul 2020 16:35:16 GMT
X-Trans-Id: tx000000000000000000013-005f0352c4-15ff-default
X-Openstack-Request-Id: tx000000000000000000013-005f0352c4-15ff-default
Content-Type: text/plain; charset=utf-8
Content-Length: 0
Date: Mon, 06 Jul 2020 16:35:16 GMT
Connection: close

root@aio1-utility-container-4a6caa82:/# curl http://172.29.236.100:8080/swift/v1/AUTH_4d39f320a46f4a51970b64fd0fce912d/test/obj1 -i -H "X-Auth-Token: $TOKEN" -X GET
HTTP/1.1 200 OK
Content-Length: 4
Accept-Ranges: bytes
Last-Modified: Mon, 06 Jul 2020 16:35:16 GMT
X-Timestamp: 1594053316.48704
etag: 81dc9bdb52d04dc20036dbd8313ed055
X-Trans-Id: tx000000000000000000016-005f0352d7-15ff-default
X-Openstack-Request-Id: tx000000000000000000016-005f0352d7-15ff-default
Content-Type: application/x-www-form-urlencoded
Date: Mon, 06 Jul 2020 16:35:35 GMT
Connection: close

1234root@aio1-utility-container-4a6caa82:/# curl http://172.29.236.100:8080/swift/v1/AUTH_4d39f320a46f4a51970b64fd0fce912d/test/obj1 -i -X GET
HTTP/1.1 404 Not Found
Content-Length: 12
X-Trans-Id: tx000000000000000000022-005f035338-15ff-default
X-Openstack-Request-Id: tx000000000000000000022-005f035338-15ff-default
Accept-Ranges: bytes
Content-Type: text/plain; charset=utf-8
Date: Mon, 06 Jul 2020 16:37:12 GMT
Connection: close

NoSuchBucketroot@aio1-utility-container-4a6caa82:/# 

So as we can see, it returns 404 and NoSuchBucket. I can put your keys on that VM if that helps. And we can reproduce that pretty easily, as we're deploying with ansible.

#4 Updated by Matthew Oliver about 1 month ago

You right, it doesn't work, but only when your using keystone. I've now also confirmed it:

suse@ironic-moliver:~/code/ceph/build(master)$ curl -i -H "X-Auth-Token: $TOKEN" http://192.168.121.1:8000/swift/v1/AUTH_a87da23093c345529e29f5b08ba94ac9/cont/ -X PUT
HTTP/1.1 201 Created
Content-Length: 0
X-Trans-Id: tx000000000000000000004-005f03ed73-11c3-default
X-Openstack-Request-Id: tx000000000000000000004-005f03ed73-11c3-default
Accept-Ranges: bytes
Content-Type: text/plain; charset=utf-8
Date: Tue, 07 Jul 2020 03:35:20 GMT
Connection: Keep-Alive

suse@ironic-moliver:~/code/ceph/build(master)$ curl -i -H "X-Auth-Token: $TOKEN" http://192.168.121.1:8000/swift/v1/AUTH_a87da23093c345529e29f5b08ba94ac9/cont/obj --data-binary 1234 -X PUT
HTTP/1.1 201 Created
etag: 81dc9bdb52d04dc20036dbd8313ed055
Last-Modified: Tue, 07 Jul 2020 03:35:39 GMT
X-Trans-Id: tx000000000000000000005-005f03ed88-11c3-default
X-Openstack-Request-Id: tx000000000000000000005-005f03ed88-11c3-default
Content-Type: text/plain; charset=utf-8
Content-Length: 0
Date: Tue, 07 Jul 2020 03:35:39 GMT
Connection: Keep-Alive

suse@ironic-moliver:~/code/ceph/build(master)$ curl -i -H "X-Auth-Token: $TOKEN" http://192.168.121.1:8000/swift/v1/AUTH_a87da23093c345529e29f5b08ba94ac9/cont/obj
HTTP/1.1 200 OK
Content-Length: 4
Accept-Ranges: bytes
Last-Modified: Tue, 07 Jul 2020 03:35:39 GMT
X-Timestamp: 1594092939.16920
etag: 81dc9bdb52d04dc20036dbd8313ed055
X-Trans-Id: tx000000000000000000006-005f03ed93-11c3-default
X-Openstack-Request-Id: tx000000000000000000006-005f03ed93-11c3-default
Content-Type: application/x-www-form-urlencoded
Date: Tue, 07 Jul 2020 03:35:47 GMT
Connection: Keep-Alive

1234
suse@ironic-moliver:~/code/ceph/build(master)$ curl -i http://192.168.121.1:8000/swift/v1/AUTH_a87da23093c345529e29f5b08ba94ac9/cont/obj
HTTP/1.1 404 Not Found
Content-Length: 12
X-Trans-Id: tx000000000000000000007-005f03ed9f-11c3-default
X-Openstack-Request-Id: tx000000000000000000007-005f03ed9f-11c3-default
Accept-Ranges: bytes
Content-Type: text/plain; charset=utf-8
Date: Tue, 07 Jul 2020 03:35:59 GMT
Connection: Keep-Alive

NoSuchBucket

Interesting it says NoSuchBucket, maybe that'll help me pinpoint in the code where it's failing!
Now that we've reproduced it, finally time to debug :)

#5 Updated by Matthew Oliver about 1 month ago

OK I have a first version of a fix. Works in the above case, I'll just run some tests, and maybe tempest this time, to see if it's effects anything else.

#6 Updated by Matthew Oliver about 1 month ago

turns out the earlier patch didn't properly account for tenants so although it worked for Swift auth v1, it didn't behave as expected with keystone, worse, it gave keystone container ACLs problems too.

I've created a patch that reworks the original and makes it behave for both keystone and swift v1: https://github.com/ceph/ceph/pull/35984

#7 Updated by Casey Bodley about 1 month ago

  • Status changed from New to Fix Under Review
  • Pull request ID set to 35984

Also available in: Atom PDF