Actions
Bug #46806
closedtempest tests fail in test_create_container_with_remove_metadata_key
Added by Yuval Lifshitz almost 4 years ago. Updated over 3 years ago.
Status:
Resolved
Priority:
Normal
Assignee:
-
Target version:
-
% Done:
0%
Source:
Development
Tags:
tempest
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Updated by Yuval Lifshitz almost 4 years ago
- Subject changed from tempest tests fail in test_create_container_with_remove_metadata_key" to tempest tests fail in test_create_container_with_remove_metadata_key
Updated by Matthew Oliver over 3 years ago
I wanted to take a look at this, but the links seem to timeout for me.
I'd love to see the error message.
Updated by Casey Bodley over 3 years ago
Updated by Matthew Oliver over 3 years ago
Great, thanks Casey, now I can see the failed tests:
2020-07-31T20:09:32.520 INFO:teuthology.orchestra.run.smithi012.stdout:==============================
2020-07-31T20:09:32.521 INFO:teuthology.orchestra.run.smithi012.stdout:Failed 2 tests - output below:
2020-07-31T20:09:32.521 INFO:teuthology.orchestra.run.smithi012.stdout:==============================
2020-07-31T20:09:32.521 INFO:teuthology.orchestra.run.smithi012.stdout:
2020-07-31T20:09:32.522 INFO:teuthology.orchestra.run.smithi012.stdout:tempest.api.object_storage.test_container_services.ContainerTest.test_create_container_with_remove_metadata_key[id-24d16451-1c0c-4e4f-b59c-9840a3aba40e]
2020-07-31T20:09:32.522 INFO:teuthology.orchestra.run.smithi012.stdout:--------------------------------------------------------------------------------------------------------------------------------------------------------
2020-07-31T20:09:32.522 INFO:teuthology.orchestra.run.smithi012.stdout:
2020-07-31T20:09:32.523 INFO:teuthology.orchestra.run.smithi012.stdout:Captured traceback:
2020-07-31T20:09:32.523 INFO:teuthology.orchestra.run.smithi012.stdout:~~~~~~~~~~~~~~~~~~~
2020-07-31T20:09:32.523 INFO:teuthology.orchestra.run.smithi012.stdout: Traceback (most recent call last):
2020-07-31T20:09:32.524 INFO:teuthology.orchestra.run.smithi012.stdout:
2020-07-31T20:09:32.524 INFO:teuthology.orchestra.run.smithi012.stdout: File "/home/ubuntu/cephtest/tempest/tempest/api/object_storage/test_container_services.py", line 96, in test_create_container_with_remove_metadata_key
2020-07-31T20:09:32.524 INFO:teuthology.orchestra.run.smithi012.stdout: self.assertNotIn('x-container-meta-test-container-meta', resp)
2020-07-31T20:09:32.525 INFO:teuthology.orchestra.run.smithi012.stdout:
2020-07-31T20:09:32.525 INFO:teuthology.orchestra.run.smithi012.stdout: File "/home/ubuntu/cephtest/tempest/.tox/venv/lib/python3.6/site-packages/testtools/testcase.py", line 459, in assertNotIn
2020-07-31T20:09:32.526 INFO:teuthology.orchestra.run.smithi012.stdout: self.assertThat(haystack, matcher, message)
2020-07-31T20:09:32.526 INFO:teuthology.orchestra.run.smithi012.stdout:
2020-07-31T20:09:32.527 INFO:teuthology.orchestra.run.smithi012.stdout: File "/home/ubuntu/cephtest/tempest/.tox/venv/lib/python3.6/site-packages/testtools/testcase.py", line 502, in assertThat
2020-07-31T20:09:32.527 INFO:teuthology.orchestra.run.smithi012.stdout: raise mismatch_error
2020-07-31T20:09:32.527 INFO:teuthology.orchestra.run.smithi012.stdout:
2020-07-31T20:09:32.528 INFO:teuthology.orchestra.run.smithi012.stdout: testtools.matchers._impl.MismatchError: {'x-timestamp': '1596226130.64688', 'x-container-object-count': '0', 'x-container-bytes-used': '0', 'x-container-bytes-used-actual': '0', 'x-storage-policy': 'default-placement', 'x-storage-class': 'STANDARD', 'x-container-meta-test-container-meta': 'Meta1', 'last-modified': 'Fri, 31 Jul 2020 20:08:50 GMT', 'x-trans-id': 'tx00000000000000000023d-005f247a52-1091-default', 'x-openstack-request-id': 'tx00000000000000000023d-005f247a52-1091-default', 'accept-ranges': 'bytes', 'content-type': 'application/json; charset=utf-8', 'date': 'Fri, 31 Jul 2020 20:08:50 GMT', 'connection': 'close', 'status': '204', 'content-location': 'http://smithi012.front.sepia.ceph.com:80/v1/KEY_be344e62a85747669e377574df6716fa/tempest-TestContainer-1896483384'} matches Contains('x-container-meta-test-container-meta')
2020-07-31T20:09:32.528 INFO:teuthology.orchestra.run.smithi012.stdout:
2020-07-31T20:09:32.528 INFO:teuthology.orchestra.run.smithi012.stdout:
2020-07-31T20:09:32.529 INFO:teuthology.orchestra.run.smithi012.stdout:Captured pythonlogging:
2020-07-31T20:09:32.529 INFO:teuthology.orchestra.run.smithi012.stdout:~~~~~~~~~~~~~~~~~~~~~~~
2020-07-31T20:09:32.530 INFO:teuthology.orchestra.run.smithi012.stdout: 2020-07-31 20:08:50,663 24431 INFO [tempest.lib.common.rest_client] Request (ContainerTest:test_create_container_with_remove_metadata_key): 201 PUT http://smithi012.front.sepia.ceph.com:80/v1/KEY_be344e62a85747669e377574df6716fa/tempest-TestContainer-1896483384 0.019s
2020-07-31T20:09:32.530 INFO:teuthology.orchestra.run.smithi012.stdout:2020-07-31 20:08:50,668 24431 INFO [tempest.lib.common.rest_client] Request (ContainerTest:test_create_container_with_remove_metadata_key): 201 PUT http://smithi012.front.sepia.ceph.com:80/v1/KEY_be344e62a85747669e377574df6716fa/tempest-TestContainer-1896483384 0.005s
2020-07-31T20:09:32.530 INFO:teuthology.orchestra.run.smithi012.stdout:2020-07-31 20:08:50,673 24431 INFO [tempest.lib.common.rest_client] Request (ContainerTest:test_create_container_with_remove_metadata_key): 204 HEAD http://smithi012.front.sepia.ceph.com:80/v1/KEY_be344e62a85747669e377574df6716fa/tempest-TestContainer-1896483384 0.004s
2020-07-31T20:09:32.531 INFO:teuthology.orchestra.run.smithi012.stdout:2020-07-31 20:08:50,676 24431 INFO [tempest.lib.common.rest_client] Request (ContainerTest:tearDown): 404 GET http://smithi012.front.sepia.ceph.com:80/v1/KEY_be344e62a85747669e377574df6716fa/tempest-TestContainer-1366595854?&limit=9999&format=json 0.002s
2020-07-31T20:09:32.531 INFO:teuthology.orchestra.run.smithi012.stdout:2020-07-31 20:08:50,678 24431 INFO [tempest.lib.common.rest_client] Request (ContainerTest:tearDown): 404 GET http://smithi012.front.sepia.ceph.com:80/v1/KEY_be344e62a85747669e377574df6716fa/tempest-TestContainer-566889470?&limit=9999&format=json 0.002s
2020-07-31T20:09:32.531 INFO:teuthology.orchestra.run.smithi012.stdout:2020-07-31 20:08:50,680 24431 INFO [tempest.lib.common.rest_client] Request (ContainerTest:tearDown): 404 GET http://smithi012.front.sepia.ceph.com:80/v1/KEY_be344e62a85747669e377574df6716fa/tempest-TestContainer-1599579299?&limit=9999&format=json 0.002s
2020-07-31T20:09:32.532 INFO:teuthology.orchestra.run.smithi012.stdout:2020-07-31 20:08:50,683 24431 INFO [tempest.lib.common.rest_client] Request (ContainerTest:tearDown): 404 GET http://smithi012.front.sepia.ceph.com:80/v1/KEY_be344e62a85747669e377574df6716fa/tempest-TestContainer-1706779172?&limit=9999&format=json 0.002s
2020-07-31T20:09:32.532 INFO:teuthology.orchestra.run.smithi012.stdout:2020-07-31 20:08:50,689 24431 INFO [tempest.lib.common.rest_client] Request (ContainerTest:tearDown): 200 GET http://smithi012.front.sepia.ceph.com:80/v1/KEY_be344e62a85747669e377574df6716fa/tempest-TestContainer-1896483384?&limit=9999&format=json 0.005s
2020-07-31T20:09:32.532 INFO:teuthology.orchestra.run.smithi012.stdout:2020-07-31 20:08:52,773 24431 INFO [tempest.lib.common.rest_client] Request (ContainerTest:tearDown): 204 DELETE http://smithi012.front.sepia.ceph.com:80/v1/KEY_be344e62a85747669e377574df6716fa/tempest-TestContainer-1896483384 0.081s
2020-07-31T20:09:32.533 INFO:teuthology.orchestra.run.smithi012.stdout:
2020-07-31T20:09:32.533 INFO:teuthology.orchestra.run.smithi012.stdout:
2020-07-31T20:09:32.533 INFO:teuthology.orchestra.run.smithi012.stdout:tempest.api.object_storage.test_container_services.ContainerTest.test_create_container_with_remove_metadata_value[id-8a21ebad-a5c7-4e29-b428-384edc8cd156]
2020-07-31T20:09:32.534 INFO:teuthology.orchestra.run.smithi012.stdout:----------------------------------------------------------------------------------------------------------------------------------------------------------
2020-07-31T20:09:32.534 INFO:teuthology.orchestra.run.smithi012.stdout:
2020-07-31T20:09:32.535 INFO:teuthology.orchestra.run.smithi012.stdout:Captured traceback:
2020-07-31T20:09:32.535 INFO:teuthology.orchestra.run.smithi012.stdout:~~~~~~~~~~~~~~~~~~~
2020-07-31T20:09:32.535 INFO:teuthology.orchestra.run.smithi012.stdout: Traceback (most recent call last):
2020-07-31T20:09:32.536 INFO:teuthology.orchestra.run.smithi012.stdout:
2020-07-31T20:09:32.536 INFO:teuthology.orchestra.run.smithi012.stdout: File "/home/ubuntu/cephtest/tempest/tempest/api/object_storage/test_container_services.py", line 113, in test_create_container_with_remove_metadata_value
2020-07-31T20:09:32.536 INFO:teuthology.orchestra.run.smithi012.stdout: self.assertNotIn('x-container-meta-test-container-meta', resp)
2020-07-31T20:09:32.537 INFO:teuthology.orchestra.run.smithi012.stdout:
2020-07-31T20:09:32.537 INFO:teuthology.orchestra.run.smithi012.stdout: File "/home/ubuntu/cephtest/tempest/.tox/venv/lib/python3.6/site-packages/testtools/testcase.py", line 459, in assertNotIn
2020-07-31T20:09:32.537 INFO:teuthology.orchestra.run.smithi012.stdout: self.assertThat(haystack, matcher, message)
2020-07-31T20:09:32.538 INFO:teuthology.orchestra.run.smithi012.stdout:
2020-07-31T20:09:32.538 INFO:teuthology.orchestra.run.smithi012.stdout: File "/home/ubuntu/cephtest/tempest/.tox/venv/lib/python3.6/site-packages/testtools/testcase.py", line 502, in assertThat
2020-07-31T20:09:32.538 INFO:teuthology.orchestra.run.smithi012.stdout: raise mismatch_error
2020-07-31T20:09:32.539 INFO:teuthology.orchestra.run.smithi012.stdout:
2020-07-31T20:09:32.539 INFO:teuthology.orchestra.run.smithi012.stdout: testtools.matchers._impl.MismatchError: {'x-timestamp': '1596226132.77845', 'x-container-object-count': '0', 'x-container-bytes-used': '0', 'x-container-bytes-used-actual': '0', 'x-storage-policy': 'default-placement', 'x-storage-class': 'STANDARD', 'x-container-meta-test-container-meta': 'Meta1', 'last-modified': 'Fri, 31 Jul 2020 20:08:52 GMT', 'x-trans-id': 'tx000000000000000000247-005f247a54-1091-default', 'x-openstack-request-id': 'tx000000000000000000247-005f247a54-1091-default', 'accept-ranges': 'bytes', 'content-type': 'application/json; charset=utf-8', 'date': 'Fri, 31 Jul 2020 20:08:52 GMT', 'connection': 'close', 'status': '204', 'content-location': 'http://smithi012.front.sepia.ceph.com:80/v1/KEY_be344e62a85747669e377574df6716fa/tempest-TestContainer-440795581'} matches Contains('x-container-meta-test-container-meta')
2020-07-31T20:09:32.539 INFO:teuthology.orchestra.run.smithi012.stdout:
2020-07-31T20:09:32.540 INFO:teuthology.orchestra.run.smithi012.stdout:
2020-07-31T20:09:32.540 INFO:teuthology.orchestra.run.smithi012.stdout:Captured pythonlogging:
2020-07-31T20:09:32.540 INFO:teuthology.orchestra.run.smithi012.stdout:~~~~~~~~~~~~~~~~~~~~~~~
2020-07-31T20:09:32.541 INFO:teuthology.orchestra.run.smithi012.stdout: 2020-07-31 20:08:52,797 24431 INFO [tempest.lib.common.rest_client] Request (ContainerTest:test_create_container_with_remove_metadata_value): 201 PUT http://smithi012.front.sepia.ceph.com:80/v1/KEY_be344e62a85747669e377574df6716fa/tempest-TestContainer-440795581 0.023s
2020-07-31T20:09:32.541 INFO:teuthology.orchestra.run.smithi012.stdout:2020-07-31 20:08:52,802 24431 INFO [tempest.lib.common.rest_client] Request (ContainerTest:test_create_container_with_remove_metadata_value): 201 PUT http://smithi012.front.sepia.ceph.com:80/v1/KEY_be344e62a85747669e377574df6716fa/tempest-TestContainer-440795581 0.005s
2020-07-31T20:09:32.541 INFO:teuthology.orchestra.run.smithi012.stdout:2020-07-31 20:08:52,807 24431 INFO [tempest.lib.common.rest_client] Request (ContainerTest:test_create_container_with_remove_metadata_value): 204 HEAD http://smithi012.front.sepia.ceph.com:80/v1/KEY_be344e62a85747669e377574df6716fa/tempest-TestContainer-440795581 0.004s
2020-07-31T20:09:32.542 INFO:teuthology.orchestra.run.smithi012.stdout:2020-07-31 20:08:52,810 24431 INFO [tempest.lib.common.rest_client] Request (ContainerTest:tearDown): 404 GET http://smithi012.front.sepia.ceph.com:80/v1/KEY_be344e62a85747669e377574df6716fa/tempest-TestContainer-1366595854?&limit=9999&format=json 0.002s
2020-07-31T20:09:32.542 INFO:teuthology.orchestra.run.smithi012.stdout:2020-07-31 20:08:52,812 24431 INFO [tempest.lib.common.rest_client] Request (ContainerTest:tearDown): 404 GET http://smithi012.front.sepia.ceph.com:80/v1/KEY_be344e62a85747669e377574df6716fa/tempest-TestContainer-566889470?&limit=9999&format=json 0.002s
2020-07-31T20:09:32.549 INFO:teuthology.orchestra.run.smithi012.stdout:2020-07-31 20:08:52,815 24431 INFO [tempest.lib.common.rest_client] Request (ContainerTest:tearDown): 404 GET http://smithi012.front.sepia.ceph.com:80/v1/KEY_be344e62a85747669e377574df6716fa/tempest-TestContainer-1599579299?&limit=9999&format=json 0.002s
2020-07-31T20:09:32.549 INFO:teuthology.orchestra.run.smithi012.stdout:2020-07-31 20:08:52,817 24431 INFO [tempest.lib.common.rest_client] Request (ContainerTest:tearDown): 404 GET http://smithi012.front.sepia.ceph.com:80/v1/KEY_be344e62a85747669e377574df6716fa/tempest-TestContainer-1706779172?&limit=9999&format=json 0.002s
2020-07-31T20:09:32.549 INFO:teuthology.orchestra.run.smithi012.stdout:2020-07-31 20:08:52,820 24431 INFO [tempest.lib.common.rest_client] Request (ContainerTest:tearDown): 404 GET http://smithi012.front.sepia.ceph.com:80/v1/KEY_be344e62a85747669e377574df6716fa/tempest-TestContainer-1896483384?&limit=9999&format=json 0.003s
2020-07-31T20:09:32.549 INFO:teuthology.orchestra.run.smithi012.stdout:2020-07-31 20:08:52,826 24431 INFO [tempest.lib.common.rest_client] Request (ContainerTest:tearDown): 200 GET http://smithi012.front.sepia.ceph.com:80/v1/KEY_be344e62a85747669e377574df6716fa/tempest-TestContainer-440795581?&limit=9999&format=json 0.006s
2020-07-31T20:09:32.550 INFO:teuthology.orchestra.run.smithi012.stdout:2020-07-31 20:08:54,907 24431 INFO [tempest.lib.common.rest_client] Request (ContainerTest:tearDown): 204 DELETE http://smithi012.front.sepia.ceph.com:80/v1/KEY_be344e62a85747669e377574df6716fa/tempest-TestContainer-440795581 0.079s
Will run some tempest tests and see what happens. There was a keystone RGW issue I fixed (https://github.com/ceph/ceph/pull/35984), So am curious if it's happens to fix this too.. but not sure it's quite as related. But let's see. If not I'll figure out whats wrong.
Updated by Matthew Oliver over 3 years ago
Huh, very interesting behaviour.
When I run with a container name similar to one used in tempest `CONT_NAME=tempest-TestContainer-18964833345345`. I get:
$ curl -i -H "X-Auth-Token: $TOKEN" $STORAGE_URL/$CONT_NAME -X PUT -H "X-Container-Meta-Foo: bar"; curl -i -H "X-Auth-Token: $TOKEN" $STORAGE_URL/$CONT_NAME -X PUT -H "X-Remove-Container-Meta-Foo: bar"; curl -i -H "X-Auth-Token: $TOKEN" $STORAGE_URL/$CONT_NAME -X HEAD
HTTP/1.1 201 Created
Content-Length: 0
X-Trans-Id: tx0000000000000000003d5-005f55ce58-1152-default
X-Openstack-Request-Id: tx0000000000000000003d5-005f55ce58-1152-default
Accept-Ranges: bytes
Content-Type: text/plain; charset=utf-8
Date: Mon, 07 Sep 2020 06:08:24 GMT
Connection: Keep-Alive
HTTP/1.1 201 Created
Content-Length: 0
X-Trans-Id: tx0000000000000000003d6-005f55ce58-1152-default
X-Openstack-Request-Id: tx0000000000000000003d6-005f55ce58-1152-default
Accept-Ranges: bytes
Content-Type: text/plain; charset=utf-8
Date: Mon, 07 Sep 2020 06:08:24 GMT
Connection: Keep-Alive
Warning: Setting custom HTTP method to HEAD with -X/--request may not work the
Warning: way you want. Consider using -I/--head instead.
HTTP/1.1 204 No Content
X-Timestamp: 1599458901.92610
X-Container-Object-Count: 0
X-Container-Bytes-Used: 0
X-Container-Bytes-Used-Actual: 0
X-Storage-Policy: default-placement
X-Storage-Class: STANDARD
X-Container-Meta-Foo: bar
Last-Modified: Mon, 07 Sep 2020 06:08:21 GMT
X-Trans-Id: tx0000000000000000003d7-005f55ce58-1152-default
X-Openstack-Request-Id: tx0000000000000000003d7-005f55ce58-1152-default
Accept-Ranges: bytes
Content-Type: text/plain; charset=utf-8
Date: Mon, 07 Sep 2020 06:08:24 GMT
Connection: Keep-Alive
Note: the `X-Container-Meta-Foo: bar` is still there.
However if I use a simple container name, like `cont` the `X-Remove-Container-..` works:
$ curl -i -H "X-Auth-Token: $TOKEN" $STORAGE_URL/cont -X PUT -H "X-Container-Meta-Foo: bar"; curl -i -H "X-Auth-Token: $TOKEN" $STORAGE_URL/cont -X PUT -H "X-Remove-Container-Meta-Foo: bar"; curl -i -H "X-Auth-Token: $TOKEN" $STORAGE_URL/cont -X HEAD
HTTP/1.1 201 Created
Content-Length: 0
X-Trans-Id: tx0000000000000000003d8-005f55cef3-1152-default
X-Openstack-Request-Id: tx0000000000000000003d8-005f55cef3-1152-default
Accept-Ranges: bytes
Content-Type: text/plain; charset=utf-8
Date: Mon, 07 Sep 2020 06:10:59 GMT
Connection: Keep-Alive
HTTP/1.1 201 Created
Content-Length: 0
X-Trans-Id: tx0000000000000000003d9-005f55cef3-1152-default
X-Openstack-Request-Id: tx0000000000000000003d9-005f55cef3-1152-default
Accept-Ranges: bytes
Content-Type: text/plain; charset=utf-8
Date: Mon, 07 Sep 2020 06:10:59 GMT
Connection: Keep-Alive
Warning: Setting custom HTTP method to HEAD with -X/--request may not work the
Warning: way you want. Consider using -I/--head instead.
HTTP/1.1 204 No Content
X-Timestamp: 1599457572.37999
X-Container-Object-Count: 0
X-Container-Bytes-Used: 0
X-Container-Bytes-Used-Actual: 0
X-Storage-Policy: default-placement
X-Storage-Class: STANDARD
Last-Modified: Mon, 07 Sep 2020 06:02:05 GMT
X-Trans-Id: tx0000000000000000003da-005f55cef3-1152-default
X-Openstack-Request-Id: tx0000000000000000003da-005f55cef3-1152-default
Accept-Ranges: bytes
Content-Type: text/plain; charset=utf-8
Date: Mon, 07 Sep 2020 06:10:59 GMT
Connection: Keep-Alive
Now that I've recreated the issue, time to debug.
Updated by Casey Bodley over 3 years ago
- Status changed from New to Resolved
it looks like this was resolved by https://github.com/ceph/ceph/pull/36978
Actions