Ceph » rgw

This issue can be reproduced every time with "swift" CLI command (python-swiftclient package)
or with OpenStack Horizon web interface.

When creating a new bucket the 'swift' command in the background first checks if bucket exists (HTTP POST method)
and then actually creates the bucket with another consequential request (HTTP PUT method).
Those two HTTP requests end up on different rgws in case of round-robin connection balancing policy.
When listing the previously created bucket, every HTTP GET request ends up on different rgw node.
The rgw node that handled POST request always returns "404 Not found" on bucket list request due to cache hit (negative entry).

The problem doesn't occur when using "openstack" CLI,
which sends only HTTP PUT request when creating bucket.

How to reproduce:

Specific RGW settings:
- rgw_swift_account_in_url=true
- rgw_keystone_token_cache_size=0

1. Configure two RGWs with balancer in front and round-robin connection balancing policy
2. swift post TestBucket --debug
3. swift list TestBucket --debug
4. swift list TestBucket --debug

Swift CLI Debug log and correlated RGWs log:

STEP 2. - swift post TestBucket --debug

// Please correlate with rgw1-POST.log

INFO:swiftclient:REQ: curl -i http://192.168.255.74:8080/swift/v1/AUTH_5085b75263394f95b1bce78a4566126c/TestBucket -X POST -H "X-Auth-Token: gAAAAABfhqNZceCYoMOduztkkeBa0bBKOSULMvl24INxANb-haBz4T_qha8Rf-MFbSETUJYdTQVF3E5UMQypj_TfERXe5iQ_3nTJrSF-dxGTpKiD4kgcI-qJme2VY0kwi2IBn11QseAEw1Rbtw-M6bLT-HA7Kgzg-CX5-aY1Gb5w9JA8GYnKu-k" -H "Content-Length: 0"
INFO:swiftclient:RESP STATUS: 404 Not Found
INFO:swiftclient:RESP HEADERS: {'Content-Length': '12', 'X-Trans-Id': 'tx00000000000000000000a-005f86a359-275023-default', 'X-Openstack-Request-Id': 'tx00000000000000000000a-005f86a359-275023-default', 'Accept-Ranges': 'bytes', 'Content-Type': 'text/plain; charset=utf-8', 'Date': 'Wed, 14 Oct 2020 07:06:01 GMT'}
INFO:swiftclient:RESP BODY: b'NoSuchBucket'

// Please correlate with rgw2-PUT.log

DEBUG:swiftclient:REQ: curl -i http://192.168.255.74:8080/swift/v1/AUTH_5085b75263394f95b1bce78a4566126c/TestBucket -X PUT -H "X-Auth-Token: gAAAAABfhqNZceCYoMOduztkkeBa0bBKOSULMvl24INxANb-haBz4T_qha8Rf-MFbSETUJYdTQVF3E5UMQypj_TfERXe5iQ_3nTJrSF-dxGTpKiD4kgcI-qJme2VY0kwi2IBn11QseAEw1Rbtw-M6bLT-HA7Kgzg-CX5-aY1Gb5w9JA8GYnKu-k" -H "Content-Length: 0"
DEBUG:swiftclient:RESP STATUS: 201 Created
DEBUG:swiftclient:RESP HEADERS: {'Content-Length': '0', 'X-Trans-Id': 'tx000000000000000000009-005f86a359-2777a8-default', 'X-Openstack-Request-Id': 'tx000000000000000000009-005f86a359-2777a8-default', 'Accept-Ranges': 'bytes', 'Content-Type': 'text/plain; charset=utf-8', 'Date': 'Wed, 14 Oct 2020 07:06:01 GMT'}

STEP 3. swift list TestBucket --debug

// Please correlate with rgw1-GET.log

INFO:swiftclient:REQ: curl -i http://192.168.255.74:8080/swift/v1/AUTH_5085b75263394f95b1bce78a4566126c/TestBucket?format=json -X GET -H "X-Auth-Token: gAAAAABfhqNf8c32u7aWlSacApUOA18GPyvPUbPrukOsizjVTDSebP0Ng3oGeq4koPfmrAyLbrdz_NbM6WNqY5yKbXqx2x8RCXIoD9PrPW2mQOJGp4aGoiJvKbmxRkW9TV34iYSOTL9pXflkun8EDqltbv-dhYIdOEYISfTR1LB3D1LZxNKngeg" -H "Accept-Encoding: gzip"
INFO:swiftclient:RESP STATUS: 404 Not Found
INFO:swiftclient:RESP HEADERS: {'Content-Length': '147', 'X-Trans-Id': 'tx000000000000000000014-005f86a35f-275023-default', 'X-Openstack-Request-Id': 'tx000000000000000000014-005f86a35f-275023-default', 'Accept-Ranges': 'bytes', 'Content-Type': 'application/json; charset=utf-8', 'Date': 'Wed, 14 Oct 2020 07:06:07 GMT'}
INFO:swiftclient:RESP BODY: b'{"Code":"NoSuchBucket","BucketName":"TestBucket","RequestId":"tx000000000000000000014-005f86a35f-275023-default","HostId":"275023-default-default"}'

STEP 4. swift list TestBucket --debug

// Please correlate with rgw2-GET.log

DEBUG:swiftclient:REQ: curl -i http://192.168.255.74:8080/swift/v1/AUTH_5085b75263394f95b1bce78a4566126c/TestBucket?format=json -X GET -H "X-Auth-Token: gAAAAABfhqNjkWGPkbhoc8bQyHNC_eyOnH2wrNhF9L6qEbAYxLWoKwV2gBuUi3dvNZwhiS1ieLGvoHiANNQmVip2CDU8JoyTgfAZrvUTgEdJRVDwtLbu0ij95e7Kp8XZ0iWP_abrnHLg--Pln9XUczJ3fDqgbtpQaKYMWkjaGlnCzWxNnT2nivU" -H "Accept-Encoding: gzip"
DEBUG:swiftclient:RESP STATUS: 200 OK
DEBUG:swiftclient:RESP HEADERS: {'Content-Length': '2', 'X-Timestamp': '1602659161.68947', 'X-Container-Object-Count': '0', 'X-Container-Bytes-Used': '0', 'X-Container-Bytes-Used-Actual': '0', 'X-Storage-Policy': 'default-placement', 'X-Storage-Class': 'STANDARD', 'Last-Modified': 'Wed, 14 Oct 2020 07:06:01 GMT', 'X-Trans-Id': 'tx000000000000000000019-005f86a363-2777a8-default', 'X-Openstack-Request-Id': 'tx000000000000000000019-005f86a363-2777a8-default', 'Accept-Ranges': 'bytes', 'Content-Type': 'application/json; charset=utf-8', 'Date': 'Wed, 14 Oct 2020 07:06:11 GMT'}
DEBUG:swiftclient:RESP BODY: b'[]'