Project

General

Profile

Actions
Copy link

Bug #47801

closed

rgw/tempest test failure in test_get_object_with_x_object_manifest

Added by Casey Bodley over 3 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
High
Assignee:
Matthew Oliver
Target version:
-
% Done:

0%

Source:
Tags:
tempest
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
37734
Crash signature (v1):
Crash signature (v2):

Description

2020-10-08T17:21:54.080 INFO:teuthology.orchestra.run.smithi198.stdout:{0} tempest.api.object_storage.test_object_services.ObjectTest.test_get_object_with_x_object_manifest [0.130526s] ... FAILED
2020-10-08T17:21:54.080 INFO:teuthology.orchestra.run.smithi198.stdout:
2020-10-08T17:21:54.080 INFO:teuthology.orchestra.run.smithi198.stdout:Captured traceback:
2020-10-08T17:21:54.080 INFO:teuthology.orchestra.run.smithi198.stdout:~~~~~~~~~~~~~~~~~~~
2020-10-08T17:21:54.080 INFO:teuthology.orchestra.run.smithi198.stdout:    Traceback (most recent call last):
2020-10-08T17:21:54.080 INFO:teuthology.orchestra.run.smithi198.stdout:
2020-10-08T17:21:54.081 INFO:teuthology.orchestra.run.smithi198.stdout:      File "/home/ubuntu/cephtest/tempest/tempest/api/object_storage/test_object_services.py", line 622, in test_get_object_with_x_object_manifest
2020-10-08T17:21:54.081 INFO:teuthology.orchestra.run.smithi198.stdout:    self.assertEqual(''.join(data_segments), body.decode())
2020-10-08T17:21:54.081 INFO:teuthology.orchestra.run.smithi198.stdout:
2020-10-08T17:21:54.081 INFO:teuthology.orchestra.run.smithi198.stdout:      File "/home/ubuntu/cephtest/tempest/.tox/venv/lib/python3.6/site-packages/testtools/testcase.py", line 415, in assertEqual
2020-10-08T17:21:54.081 INFO:teuthology.orchestra.run.smithi198.stdout:    self.assertThat(observed, matcher, message)
2020-10-08T17:21:54.081 INFO:teuthology.orchestra.run.smithi198.stdout:
2020-10-08T17:21:54.081 INFO:teuthology.orchestra.run.smithi198.stdout:      File "/home/ubuntu/cephtest/tempest/.tox/venv/lib/python3.6/site-packages/testtools/testcase.py", line 502, in assertThat
2020-10-08T17:21:54.082 INFO:teuthology.orchestra.run.smithi198.stdout:    raise mismatch_error
2020-10-08T17:21:54.082 INFO:teuthology.orchestra.run.smithi198.stdout:
2020-10-08T17:21:54.082 INFO:teuthology.orchestra.run.smithi198.stdout:    testtools.matchers._impl.MismatchError: 'test0test1test2test3test4test5test6test7test8test9' != 'test4'
2020-10-08T17:21:54.082 INFO:teuthology.orchestra.run.smithi198.stdout:
2020-10-08T17:21:54.082 INFO:teuthology.orchestra.run.smithi198.stdout:
2020-10-08T17:21:54.082 INFO:teuthology.orchestra.run.smithi198.stdout:Captured pythonlogging:
2020-10-08T17:21:54.082 INFO:teuthology.orchestra.run.smithi198.stdout:~~~~~~~~~~~~~~~~~~~~~~~
2020-10-08T17:21:54.083 INFO:teuthology.orchestra.run.smithi198.stdout:    2020-10-08 17:21:53,963 24839 INFO     [tempest.lib.common.rest_client] Request (ObjectTest:test_get_object_with_x_object_manifest): 201 PUT http://smithi198.front.sepia.ceph.com:80/v1/KEY_3f726e9d69e04ea082c068d0569e7960/tempest-TestContainer-724220972/tempest-LObject-937969082/0 0.010s
2020-10-08T17:21:54.083 INFO:teuthology.orchestra.run.smithi198.stdout:2020-10-08 17:21:53,974 24839 INFO     [tempest.lib.common.rest_client] Request (ObjectTest:test_get_object_with_x_object_manifest): 201 PUT http://smithi198.front.sepia.ceph.com:80/v1/KEY_3f726e9d69e04ea082c068d0569e7960/tempest-TestContainer-724220972/tempest-LObject-937969082/1 0.010s
2020-10-08T17:21:54.083 INFO:teuthology.orchestra.run.smithi198.stdout:2020-10-08 17:21:53,983 24839 INFO     [tempest.lib.common.rest_client] Request (ObjectTest:test_get_object_with_x_object_manifest): 201 PUT http://smithi198.front.sepia.ceph.com:80/v1/KEY_3f726e9d69e04ea082c068d0569e7960/tempest-TestContainer-724220972/tempest-LObject-937969082/2 0.008s
2020-10-08T17:21:54.083 INFO:teuthology.orchestra.run.smithi198.stdout:2020-10-08 17:21:53,995 24839 INFO     [tempest.lib.common.rest_client] Request (ObjectTest:test_get_object_with_x_object_manifest): 201 PUT http://smithi198.front.sepia.ceph.com:80/v1/KEY_3f726e9d69e04ea082c068d0569e7960/tempest-TestContainer-724220972/tempest-LObject-937969082/3 0.011s
2020-10-08T17:21:54.083 INFO:teuthology.orchestra.run.smithi198.stdout:2020-10-08 17:21:54,005 24839 INFO     [tempest.lib.common.rest_client] Request (ObjectTest:test_get_object_with_x_object_manifest): 201 PUT http://smithi198.front.sepia.ceph.com:80/v1/KEY_3f726e9d69e04ea082c068d0569e7960/tempest-TestContainer-724220972/tempest-LObject-937969082/4 0.010s
2020-10-08T17:21:54.083 INFO:teuthology.orchestra.run.smithi198.stdout:2020-10-08 17:21:54,013 24839 INFO     [tempest.lib.common.rest_client] Request (ObjectTest:test_get_object_with_x_object_manifest): 201 PUT http://smithi198.front.sepia.ceph.com:80/v1/KEY_3f726e9d69e04ea082c068d0569e7960/tempest-TestContainer-724220972/tempest-LObject-937969082/5 0.007s
2020-10-08T17:21:54.083 INFO:teuthology.orchestra.run.smithi198.stdout:2020-10-08 17:21:54,024 24839 INFO     [tempest.lib.common.rest_client] Request (ObjectTest:test_get_object_with_x_object_manifest): 201 PUT http://smithi198.front.sepia.ceph.com:80/v1/KEY_3f726e9d69e04ea082c068d0569e7960/tempest-TestContainer-724220972/tempest-LObject-937969082/6 0.011s
2020-10-08T17:21:54.084 INFO:teuthology.orchestra.run.smithi198.stdout:2020-10-08 17:21:54,034 24839 INFO     [tempest.lib.common.rest_client] Request (ObjectTest:test_get_object_with_x_object_manifest): 201 PUT http://smithi198.front.sepia.ceph.com:80/v1/KEY_3f726e9d69e04ea082c068d0569e7960/tempest-TestContainer-724220972/tempest-LObject-937969082/7 0.009s
2020-10-08T17:21:54.084 INFO:teuthology.orchestra.run.smithi198.stdout:2020-10-08 17:21:54,049 24839 INFO     [tempest.lib.common.rest_client] Request (ObjectTest:test_get_object_with_x_object_manifest): 201 PUT http://smithi198.front.sepia.ceph.com:80/v1/KEY_3f726e9d69e04ea082c068d0569e7960/tempest-TestContainer-724220972/tempest-LObject-937969082/8 0.014s
2020-10-08T17:21:54.084 INFO:teuthology.orchestra.run.smithi198.stdout:2020-10-08 17:21:54,066 24839 INFO     [tempest.lib.common.rest_client] Request (ObjectTest:test_get_object_with_x_object_manifest): 201 PUT http://smithi198.front.sepia.ceph.com:80/v1/KEY_3f726e9d69e04ea082c068d0569e7960/tempest-TestContainer-724220972/tempest-LObject-937969082/9 0.016s
2020-10-08T17:21:54.084 INFO:teuthology.orchestra.run.smithi198.stdout:2020-10-08 17:21:54,073 24839 INFO     [tempest.lib.common.rest_client] Request (ObjectTest:test_get_object_with_x_object_manifest): 201 PUT http://smithi198.front.sepia.ceph.com:80/v1/KEY_3f726e9d69e04ea082c068d0569e7960/tempest-TestContainer-724220972/tempest-LObject-937969082 0.006s
2020-10-08T17:21:54.084 INFO:teuthology.orchestra.run.smithi198.stdout:2020-10-08 17:21:54,082 24839 INFO     [tempest.lib.common.rest_client] Request (ObjectTest:test_get_object_with_x_object_manifest): 200 GET http://smithi198.front.sepia.ceph.com:80/v1/KEY_3f726e9d69e04ea082c068d0569e7960/tempest-TestContainer-724220972/tempest-LObject-937969082 0.008s

ex: http://qa-proxy.ceph.com/teuthology/cbodley-2020-10-08_16:39:10-rgw-wip-cbodley-testing-distro-basic-smithi/5506944/teuthology.log

Actions
Copy link
 #1

Updated by Matthew Oliver over 3 years ago

  • Assignee set to Matthew Oliver
  • Pull request ID set to 37734

I've recreated the problem. You can simply do it by making a dlo in the Swift api and then try and read it back.

And I think I've gotten to the bottom of it. When we go list the container/bucket we're sending in shard_id = 0 (the default). so we only ever check objects at that shard. Thus the truncated results. As in my testing it wasn't always the last item (as what happens in the tempest test). As we want all the results back all we need to do is set shard_id = -1, so all shards are checked.

Added https://github.com/ceph/ceph/pull/37734

Actions
Copy link
 #2

Updated by Casey Bodley over 3 years ago

  • Status changed from New to Pending Backport
Actions
Copy link
 #3

Updated by Nathan Cutler over 3 years ago

  • Backport set to octopus
Actions
Copy link
 #5

Updated by Nathan Cutler over 3 years ago

  • Status changed from Pending Backport to Fix Under Review
Actions
Copy link
 #6

Updated by Nathan Cutler over 3 years ago

  • Backport deleted (octopus)
Actions
Copy link
 #7

Updated by Casey Bodley over 3 years ago

  • Status changed from Fix Under Review to Resolved
Actions
Copy link

Also available in: Atom PDF