Bug #46806: tempest tests fail in test_create_container_with_remove_metadata_key - rgw - Ceph

Actions

Copy link

Bug #46806

closed

tempest tests fail in test_create_container_with_remove_metadata_key

Added by Yuval Lifshitz over 3 years ago. Updated over 3 years ago.

Status:

Resolved

Priority:

Normal

Assignee:

Target version:

% Done:

Source:

Development

Tags:

tempest

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

both in civetweb and beast:

http://pulpito.front.sepia.ceph.com/yuvalif-2020-07-31_16:16:13-rgw-wip-yuval-persistent-notif4-distro-basic-smithi/5273446
http://pulpito.front.sepia.ceph.com/yuvalif-2020-07-31_16:16:13-rgw-wip-yuval-persistent-notif4-distro-basic-smithi/5273500

Actions

Copy link

Updated by Yuval Lifshitz over 3 years ago

Subject changed from tempest tests fail in test_create_container_with_remove_metadata_key" to tempest tests fail in test_create_container_with_remove_metadata_key

Actions

Copy link

Updated by Matthew Oliver over 3 years ago

I wanted to take a look at this, but the links seem to timeout for me.

I'd love to see the error message.

Actions

Copy link

Updated by Casey Bodley over 3 years ago

these are links available outside the vpn:
http://pulpito.ceph.com/yuvalif-2020-07-31_16:16:13-rgw-wip-yuval-persistent-notif4-distro-basic-smithi/5273446
http://pulpito.ceph.com/yuvalif-2020-07-31_16:16:13-rgw-wip-yuval-persistent-notif4-distro-basic-smithi/5273500

Actions

Copy link

Updated by Matthew Oliver over 3 years ago

Great, thanks Casey, now I can see the failed tests:

2020-07-31T20:09:32.520 INFO:teuthology.orchestra.run.smithi012.stdout:==============================
2020-07-31T20:09:32.521 INFO:teuthology.orchestra.run.smithi012.stdout:Failed 2 tests - output below:
2020-07-31T20:09:32.521 INFO:teuthology.orchestra.run.smithi012.stdout:==============================
2020-07-31T20:09:32.521 INFO:teuthology.orchestra.run.smithi012.stdout:
2020-07-31T20:09:32.522 INFO:teuthology.orchestra.run.smithi012.stdout:tempest.api.object_storage.test_container_services.ContainerTest.test_create_container_with_remove_metadata_key[id-24d16451-1c0c-4e4f-b59c-9840a3aba40e]
2020-07-31T20:09:32.522 INFO:teuthology.orchestra.run.smithi012.stdout:--------------------------------------------------------------------------------------------------------------------------------------------------------
2020-07-31T20:09:32.522 INFO:teuthology.orchestra.run.smithi012.stdout:
2020-07-31T20:09:32.523 INFO:teuthology.orchestra.run.smithi012.stdout:Captured traceback:
2020-07-31T20:09:32.523 INFO:teuthology.orchestra.run.smithi012.stdout:~~~~~~~~~~~~~~~~~~~
2020-07-31T20:09:32.523 INFO:teuthology.orchestra.run.smithi012.stdout:    Traceback (most recent call last):
2020-07-31T20:09:32.524 INFO:teuthology.orchestra.run.smithi012.stdout:
2020-07-31T20:09:32.524 INFO:teuthology.orchestra.run.smithi012.stdout:      File "/home/ubuntu/cephtest/tempest/tempest/api/object_storage/test_container_services.py", line 96, in test_create_container_with_remove_metadata_key
2020-07-31T20:09:32.524 INFO:teuthology.orchestra.run.smithi012.stdout:    self.assertNotIn('x-container-meta-test-container-meta', resp)
2020-07-31T20:09:32.525 INFO:teuthology.orchestra.run.smithi012.stdout:
2020-07-31T20:09:32.525 INFO:teuthology.orchestra.run.smithi012.stdout:      File "/home/ubuntu/cephtest/tempest/.tox/venv/lib/python3.6/site-packages/testtools/testcase.py", line 459, in assertNotIn
2020-07-31T20:09:32.526 INFO:teuthology.orchestra.run.smithi012.stdout:    self.assertThat(haystack, matcher, message)
2020-07-31T20:09:32.526 INFO:teuthology.orchestra.run.smithi012.stdout:
2020-07-31T20:09:32.527 INFO:teuthology.orchestra.run.smithi012.stdout:      File "/home/ubuntu/cephtest/tempest/.tox/venv/lib/python3.6/site-packages/testtools/testcase.py", line 502, in assertThat
2020-07-31T20:09:32.527 INFO:teuthology.orchestra.run.smithi012.stdout:    raise mismatch_error
2020-07-31T20:09:32.527 INFO:teuthology.orchestra.run.smithi012.stdout:
2020-07-31T20:09:32.528 INFO:teuthology.orchestra.run.smithi012.stdout:    testtools.matchers._impl.MismatchError: {'x-timestamp': '1596226130.64688', 'x-container-object-count': '0', 'x-container-bytes-used': '0', 'x-container-bytes-used-actual': '0', 'x-storage-policy': 'default-placement', 'x-storage-class': 'STANDARD', 'x-container-meta-test-container-meta': 'Meta1', 'last-modified': 'Fri, 31 Jul 2020 20:08:50 GMT', 'x-trans-id': 'tx00000000000000000023d-005f247a52-1091-default', 'x-openstack-request-id': 'tx00000000000000000023d-005f247a52-1091-default', 'accept-ranges': 'bytes', 'content-type': 'application/json; charset=utf-8', 'date': 'Fri, 31 Jul 2020 20:08:50 GMT', 'connection': 'close', 'status': '204', 'content-location': 'http://smithi012.front.sepia.ceph.com:80/v1/KEY_be344e62a85747669e377574df6716fa/tempest-TestContainer-1896483384'} matches Contains('x-container-meta-test-container-meta')
2020-07-31T20:09:32.528 INFO:teuthology.orchestra.run.smithi012.stdout:
2020-07-31T20:09:32.528 INFO:teuthology.orchestra.run.smithi012.stdout:
2020-07-31T20:09:32.529 INFO:teuthology.orchestra.run.smithi012.stdout:Captured pythonlogging:
2020-07-31T20:09:32.529 INFO:teuthology.orchestra.run.smithi012.stdout:~~~~~~~~~~~~~~~~~~~~~~~
2020-07-31T20:09:32.530 INFO:teuthology.orchestra.run.smithi012.stdout:    2020-07-31 20:08:50,663 24431 INFO     [tempest.lib.common.rest_client] Request (ContainerTest:test_create_container_with_remove_metadata_key): 201 PUT http://smithi012.front.sepia.ceph.com:80/v1/KEY_be344e62a85747669e377574df6716fa/tempest-TestContainer-1896483384 0.019s
2020-07-31T20:09:32.530 INFO:teuthology.orchestra.run.smithi012.stdout:2020-07-31 20:08:50,668 24431 INFO     [tempest.lib.common.rest_client] Request (ContainerTest:test_create_container_with_remove_metadata_key): 201 PUT http://smithi012.front.sepia.ceph.com:80/v1/KEY_be344e62a85747669e377574df6716fa/tempest-TestContainer-1896483384 0.005s
2020-07-31T20:09:32.530 INFO:teuthology.orchestra.run.smithi012.stdout:2020-07-31 20:08:50,673 24431 INFO     [tempest.lib.common.rest_client] Request (ContainerTest:test_create_container_with_remove_metadata_key): 204 HEAD http://smithi012.front.sepia.ceph.com:80/v1/KEY_be344e62a85747669e377574df6716fa/tempest-TestContainer-1896483384 0.004s
2020-07-31T20:09:32.531 INFO:teuthology.orchestra.run.smithi012.stdout:2020-07-31 20:08:50,676 24431 INFO     [tempest.lib.common.rest_client] Request (ContainerTest:tearDown): 404 GET http://smithi012.front.sepia.ceph.com:80/v1/KEY_be344e62a85747669e377574df6716fa/tempest-TestContainer-1366595854?&limit=9999&format=json 0.002s
2020-07-31T20:09:32.531 INFO:teuthology.orchestra.run.smithi012.stdout:2020-07-31 20:08:50,678 24431 INFO     [tempest.lib.common.rest_client] Request (ContainerTest:tearDown): 404 GET http://smithi012.front.sepia.ceph.com:80/v1/KEY_be344e62a85747669e377574df6716fa/tempest-TestContainer-566889470?&limit=9999&format=json 0.002s
2020-07-31T20:09:32.531 INFO:teuthology.orchestra.run.smithi012.stdout:2020-07-31 20:08:50,680 24431 INFO     [tempest.lib.common.rest_client] Request (ContainerTest:tearDown): 404 GET http://smithi012.front.sepia.ceph.com:80/v1/KEY_be344e62a85747669e377574df6716fa/tempest-TestContainer-1599579299?&limit=9999&format=json 0.002s
2020-07-31T20:09:32.532 INFO:teuthology.orchestra.run.smithi012.stdout:2020-07-31 20:08:50,683 24431 INFO     [tempest.lib.common.rest_client] Request (ContainerTest:tearDown): 404 GET http://smithi012.front.sepia.ceph.com:80/v1/KEY_be344e62a85747669e377574df6716fa/tempest-TestContainer-1706779172?&limit=9999&format=json 0.002s
2020-07-31T20:09:32.532 INFO:teuthology.orchestra.run.smithi012.stdout:2020-07-31 20:08:50,689 24431 INFO     [tempest.lib.common.rest_client] Request (ContainerTest:tearDown): 200 GET http://smithi012.front.sepia.ceph.com:80/v1/KEY_be344e62a85747669e377574df6716fa/tempest-TestContainer-1896483384?&limit=9999&format=json 0.005s
2020-07-31T20:09:32.532 INFO:teuthology.orchestra.run.smithi012.stdout:2020-07-31 20:08:52,773 24431 INFO     [tempest.lib.common.rest_client] Request (ContainerTest:tearDown): 204 DELETE http://smithi012.front.sepia.ceph.com:80/v1/KEY_be344e62a85747669e377574df6716fa/tempest-TestContainer-1896483384 0.081s
2020-07-31T20:09:32.533 INFO:teuthology.orchestra.run.smithi012.stdout:
2020-07-31T20:09:32.533 INFO:teuthology.orchestra.run.smithi012.stdout:
2020-07-31T20:09:32.533 INFO:teuthology.orchestra.run.smithi012.stdout:tempest.api.object_storage.test_container_services.ContainerTest.test_create_container_with_remove_metadata_value[id-8a21ebad-a5c7-4e29-b428-384edc8cd156]
2020-07-31T20:09:32.534 INFO:teuthology.orchestra.run.smithi012.stdout:----------------------------------------------------------------------------------------------------------------------------------------------------------
2020-07-31T20:09:32.534 INFO:teuthology.orchestra.run.smithi012.stdout:
2020-07-31T20:09:32.535 INFO:teuthology.orchestra.run.smithi012.stdout:Captured traceback:
2020-07-31T20:09:32.535 INFO:teuthology.orchestra.run.smithi012.stdout:~~~~~~~~~~~~~~~~~~~
2020-07-31T20:09:32.535 INFO:teuthology.orchestra.run.smithi012.stdout:    Traceback (most recent call last):
2020-07-31T20:09:32.536 INFO:teuthology.orchestra.run.smithi012.stdout:
2020-07-31T20:09:32.536 INFO:teuthology.orchestra.run.smithi012.stdout:      File "/home/ubuntu/cephtest/tempest/tempest/api/object_storage/test_container_services.py", line 113, in test_create_container_with_remove_metadata_value
2020-07-31T20:09:32.536 INFO:teuthology.orchestra.run.smithi012.stdout:    self.assertNotIn('x-container-meta-test-container-meta', resp)
2020-07-31T20:09:32.537 INFO:teuthology.orchestra.run.smithi012.stdout:
2020-07-31T20:09:32.537 INFO:teuthology.orchestra.run.smithi012.stdout:      File "/home/ubuntu/cephtest/tempest/.tox/venv/lib/python3.6/site-packages/testtools/testcase.py", line 459, in assertNotIn
2020-07-31T20:09:32.537 INFO:teuthology.orchestra.run.smithi012.stdout:    self.assertThat(haystack, matcher, message)
2020-07-31T20:09:32.538 INFO:teuthology.orchestra.run.smithi012.stdout:
2020-07-31T20:09:32.538 INFO:teuthology.orchestra.run.smithi012.stdout:      File "/home/ubuntu/cephtest/tempest/.tox/venv/lib/python3.6/site-packages/testtools/testcase.py", line 502, in assertThat
2020-07-31T20:09:32.538 INFO:teuthology.orchestra.run.smithi012.stdout:    raise mismatch_error
2020-07-31T20:09:32.539 INFO:teuthology.orchestra.run.smithi012.stdout:
2020-07-31T20:09:32.539 INFO:teuthology.orchestra.run.smithi012.stdout:    testtools.matchers._impl.MismatchError: {'x-timestamp': '1596226132.77845', 'x-container-object-count': '0', 'x-container-bytes-used': '0', 'x-container-bytes-used-actual': '0', 'x-storage-policy': 'default-placement', 'x-storage-class': 'STANDARD', 'x-container-meta-test-container-meta': 'Meta1', 'last-modified': 'Fri, 31 Jul 2020 20:08:52 GMT', 'x-trans-id': 'tx000000000000000000247-005f247a54-1091-default', 'x-openstack-request-id': 'tx000000000000000000247-005f247a54-1091-default', 'accept-ranges': 'bytes', 'content-type': 'application/json; charset=utf-8', 'date': 'Fri, 31 Jul 2020 20:08:52 GMT', 'connection': 'close', 'status': '204', 'content-location': 'http://smithi012.front.sepia.ceph.com:80/v1/KEY_be344e62a85747669e377574df6716fa/tempest-TestContainer-440795581'} matches Contains('x-container-meta-test-container-meta')
2020-07-31T20:09:32.539 INFO:teuthology.orchestra.run.smithi012.stdout:
2020-07-31T20:09:32.540 INFO:teuthology.orchestra.run.smithi012.stdout:
2020-07-31T20:09:32.540 INFO:teuthology.orchestra.run.smithi012.stdout:Captured pythonlogging:
2020-07-31T20:09:32.540 INFO:teuthology.orchestra.run.smithi012.stdout:~~~~~~~~~~~~~~~~~~~~~~~
2020-07-31T20:09:32.541 INFO:teuthology.orchestra.run.smithi012.stdout:    2020-07-31 20:08:52,797 24431 INFO     [tempest.lib.common.rest_client] Request (ContainerTest:test_create_container_with_remove_metadata_value): 201 PUT http://smithi012.front.sepia.ceph.com:80/v1/KEY_be344e62a85747669e377574df6716fa/tempest-TestContainer-440795581 0.023s
2020-07-31T20:09:32.541 INFO:teuthology.orchestra.run.smithi012.stdout:2020-07-31 20:08:52,802 24431 INFO     [tempest.lib.common.rest_client] Request (ContainerTest:test_create_container_with_remove_metadata_value): 201 PUT http://smithi012.front.sepia.ceph.com:80/v1/KEY_be344e62a85747669e377574df6716fa/tempest-TestContainer-440795581 0.005s
2020-07-31T20:09:32.541 INFO:teuthology.orchestra.run.smithi012.stdout:2020-07-31 20:08:52,807 24431 INFO     [tempest.lib.common.rest_client] Request (ContainerTest:test_create_container_with_remove_metadata_value): 204 HEAD http://smithi012.front.sepia.ceph.com:80/v1/KEY_be344e62a85747669e377574df6716fa/tempest-TestContainer-440795581 0.004s
2020-07-31T20:09:32.542 INFO:teuthology.orchestra.run.smithi012.stdout:2020-07-31 20:08:52,810 24431 INFO     [tempest.lib.common.rest_client] Request (ContainerTest:tearDown): 404 GET http://smithi012.front.sepia.ceph.com:80/v1/KEY_be344e62a85747669e377574df6716fa/tempest-TestContainer-1366595854?&limit=9999&format=json 0.002s
2020-07-31T20:09:32.542 INFO:teuthology.orchestra.run.smithi012.stdout:2020-07-31 20:08:52,812 24431 INFO     [tempest.lib.common.rest_client] Request (ContainerTest:tearDown): 404 GET http://smithi012.front.sepia.ceph.com:80/v1/KEY_be344e62a85747669e377574df6716fa/tempest-TestContainer-566889470?&limit=9999&format=json 0.002s
2020-07-31T20:09:32.549 INFO:teuthology.orchestra.run.smithi012.stdout:2020-07-31 20:08:52,815 24431 INFO     [tempest.lib.common.rest_client] Request (ContainerTest:tearDown): 404 GET http://smithi012.front.sepia.ceph.com:80/v1/KEY_be344e62a85747669e377574df6716fa/tempest-TestContainer-1599579299?&limit=9999&format=json 0.002s
2020-07-31T20:09:32.549 INFO:teuthology.orchestra.run.smithi012.stdout:2020-07-31 20:08:52,817 24431 INFO     [tempest.lib.common.rest_client] Request (ContainerTest:tearDown): 404 GET http://smithi012.front.sepia.ceph.com:80/v1/KEY_be344e62a85747669e377574df6716fa/tempest-TestContainer-1706779172?&limit=9999&format=json 0.002s
2020-07-31T20:09:32.549 INFO:teuthology.orchestra.run.smithi012.stdout:2020-07-31 20:08:52,820 24431 INFO     [tempest.lib.common.rest_client] Request (ContainerTest:tearDown): 404 GET http://smithi012.front.sepia.ceph.com:80/v1/KEY_be344e62a85747669e377574df6716fa/tempest-TestContainer-1896483384?&limit=9999&format=json 0.003s
2020-07-31T20:09:32.549 INFO:teuthology.orchestra.run.smithi012.stdout:2020-07-31 20:08:52,826 24431 INFO     [tempest.lib.common.rest_client] Request (ContainerTest:tearDown): 200 GET http://smithi012.front.sepia.ceph.com:80/v1/KEY_be344e62a85747669e377574df6716fa/tempest-TestContainer-440795581?&limit=9999&format=json 0.006s
2020-07-31T20:09:32.550 INFO:teuthology.orchestra.run.smithi012.stdout:2020-07-31 20:08:54,907 24431 INFO     [tempest.lib.common.rest_client] Request (ContainerTest:tearDown): 204 DELETE http://smithi012.front.sepia.ceph.com:80/v1/KEY_be344e62a85747669e377574df6716fa/tempest-TestContainer-440795581 0.079s

Will run some tempest tests and see what happens. There was a keystone RGW issue I fixed (https://github.com/ceph/ceph/pull/35984), So am curious if it's happens to fix this too.. but not sure it's quite as related. But let's see. If not I'll figure out whats wrong.

Actions

Copy link

Updated by Matthew Oliver over 3 years ago

Huh, very interesting behaviour.

When I run with a container name similar to one used in tempest `CONT_NAME=tempest-TestContainer-18964833345345`. I get:

$ curl -i -H "X-Auth-Token: $TOKEN" $STORAGE_URL/$CONT_NAME -X PUT -H "X-Container-Meta-Foo: bar"; curl -i -H "X-Auth-Token: $TOKEN" $STORAGE_URL/$CONT_NAME -X PUT -H "X-Remove-Container-Meta-Foo: bar"; curl -i -H "X-Auth-Token: $TOKEN" $STORAGE_URL/$CONT_NAME -X HEAD
HTTP/1.1 201 Created
Content-Length: 0
X-Trans-Id: tx0000000000000000003d5-005f55ce58-1152-default
X-Openstack-Request-Id: tx0000000000000000003d5-005f55ce58-1152-default
Accept-Ranges: bytes
Content-Type: text/plain; charset=utf-8
Date: Mon, 07 Sep 2020 06:08:24 GMT
Connection: Keep-Alive

HTTP/1.1 201 Created
Content-Length: 0
X-Trans-Id: tx0000000000000000003d6-005f55ce58-1152-default
X-Openstack-Request-Id: tx0000000000000000003d6-005f55ce58-1152-default
Accept-Ranges: bytes
Content-Type: text/plain; charset=utf-8
Date: Mon, 07 Sep 2020 06:08:24 GMT
Connection: Keep-Alive

Warning: Setting custom HTTP method to HEAD with -X/--request may not work the 
Warning: way you want. Consider using -I/--head instead.
HTTP/1.1 204 No Content
X-Timestamp: 1599458901.92610
X-Container-Object-Count: 0
X-Container-Bytes-Used: 0
X-Container-Bytes-Used-Actual: 0
X-Storage-Policy: default-placement
X-Storage-Class: STANDARD
X-Container-Meta-Foo: bar
Last-Modified: Mon, 07 Sep 2020 06:08:21 GMT
X-Trans-Id: tx0000000000000000003d7-005f55ce58-1152-default
X-Openstack-Request-Id: tx0000000000000000003d7-005f55ce58-1152-default
Accept-Ranges: bytes
Content-Type: text/plain; charset=utf-8
Date: Mon, 07 Sep 2020 06:08:24 GMT
Connection: Keep-Alive

Note: the `X-Container-Meta-Foo: bar` is still there.

However if I use a simple container name, like `cont` the `X-Remove-Container-..` works:

$ curl -i -H "X-Auth-Token: $TOKEN" $STORAGE_URL/cont -X PUT -H "X-Container-Meta-Foo: bar"; curl -i -H "X-Auth-Token: $TOKEN" $STORAGE_URL/cont -X PUT -H "X-Remove-Container-Meta-Foo: bar"; curl -i -H "X-Auth-Token: $TOKEN" $STORAGE_URL/cont -X HEAD
HTTP/1.1 201 Created
Content-Length: 0
X-Trans-Id: tx0000000000000000003d8-005f55cef3-1152-default
X-Openstack-Request-Id: tx0000000000000000003d8-005f55cef3-1152-default
Accept-Ranges: bytes
Content-Type: text/plain; charset=utf-8
Date: Mon, 07 Sep 2020 06:10:59 GMT
Connection: Keep-Alive

HTTP/1.1 201 Created
Content-Length: 0
X-Trans-Id: tx0000000000000000003d9-005f55cef3-1152-default
X-Openstack-Request-Id: tx0000000000000000003d9-005f55cef3-1152-default
Accept-Ranges: bytes
Content-Type: text/plain; charset=utf-8
Date: Mon, 07 Sep 2020 06:10:59 GMT
Connection: Keep-Alive

Warning: Setting custom HTTP method to HEAD with -X/--request may not work the 
Warning: way you want. Consider using -I/--head instead.
HTTP/1.1 204 No Content
X-Timestamp: 1599457572.37999
X-Container-Object-Count: 0
X-Container-Bytes-Used: 0
X-Container-Bytes-Used-Actual: 0
X-Storage-Policy: default-placement
X-Storage-Class: STANDARD
Last-Modified: Mon, 07 Sep 2020 06:02:05 GMT
X-Trans-Id: tx0000000000000000003da-005f55cef3-1152-default
X-Openstack-Request-Id: tx0000000000000000003da-005f55cef3-1152-default
Accept-Ranges: bytes
Content-Type: text/plain; charset=utf-8
Date: Mon, 07 Sep 2020 06:10:59 GMT
Connection: Keep-Alive

Now that I've recreated the issue, time to debug.

Actions

Copy link