Project

General

Profile

Actions
Copy link

Bug #47800

closed

rgw/tempest test failure in test_object_upload_in_segments

Added by Casey Bodley over 3 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
High
Assignee:
Matthew Oliver
Target version:
-
% Done:

0%

Source:
Tags:
tempest
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
37734
Crash signature (v1):
Crash signature (v2):

Description

2020-10-08T17:21:54.341 INFO:teuthology.orchestra.run.smithi198.stdout:{0} tempest.api.object_storage.test_object_services.ObjectTest.test_object_upload_in_segments [0.141846s] ... FAILED
2020-10-08T17:21:54.341 INFO:teuthology.orchestra.run.smithi198.stdout:
2020-10-08T17:21:54.341 INFO:teuthology.orchestra.run.smithi198.stdout:Captured traceback:
2020-10-08T17:21:54.341 INFO:teuthology.orchestra.run.smithi198.stdout:~~~~~~~~~~~~~~~~~~~
2020-10-08T17:21:54.341 INFO:teuthology.orchestra.run.smithi198.stdout:    Traceback (most recent call last):
2020-10-08T17:21:54.341 INFO:teuthology.orchestra.run.smithi198.stdout:
2020-10-08T17:21:54.342 INFO:teuthology.orchestra.run.smithi198.stdout:      File "/home/ubuntu/cephtest/tempest/tempest/api/object_storage/test_object_services.py", line 946, in test_object_upload_in_segments
2020-10-08T17:21:54.342 INFO:teuthology.orchestra.run.smithi198.stdout:    self.assertEqual(''.join(data_segments), body.decode())
2020-10-08T17:21:54.342 INFO:teuthology.orchestra.run.smithi198.stdout:
2020-10-08T17:21:54.342 INFO:teuthology.orchestra.run.smithi198.stdout:      File "/home/ubuntu/cephtest/tempest/.tox/venv/lib/python3.6/site-packages/testtools/testcase.py", line 415, in assertEqual
2020-10-08T17:21:54.342 INFO:teuthology.orchestra.run.smithi198.stdout:    self.assertThat(observed, matcher, message)
2020-10-08T17:21:54.342 INFO:teuthology.orchestra.run.smithi198.stdout:
2020-10-08T17:21:54.342 INFO:teuthology.orchestra.run.smithi198.stdout:      File "/home/ubuntu/cephtest/tempest/.tox/venv/lib/python3.6/site-packages/testtools/testcase.py", line 502, in assertThat
2020-10-08T17:21:54.343 INFO:teuthology.orchestra.run.smithi198.stdout:    raise mismatch_error
2020-10-08T17:21:54.343 INFO:teuthology.orchestra.run.smithi198.stdout:
2020-10-08T17:21:54.343 INFO:teuthology.orchestra.run.smithi198.stdout:    testtools.matchers._impl.MismatchError: 'test0test1test2test3test4test5test6test7test8test9' != 'test0'
2020-10-08T17:21:54.343 INFO:teuthology.orchestra.run.smithi198.stdout:
2020-10-08T17:21:54.343 INFO:teuthology.orchestra.run.smithi198.stdout:
2020-10-08T17:21:54.343 INFO:teuthology.orchestra.run.smithi198.stdout:Captured pythonlogging:
2020-10-08T17:21:54.343 INFO:teuthology.orchestra.run.smithi198.stdout:~~~~~~~~~~~~~~~~~~~~~~~
2020-10-08T17:21:54.344 INFO:teuthology.orchestra.run.smithi198.stdout:    2020-10-08 17:21:54,210 24839 INFO     [tempest.lib.common.rest_client] Request (ObjectTest:test_object_upload_in_segments): 201 PUT http://smithi198.front.sepia.ceph.com:80/v1/KEY_3f726e9d69e04ea082c068d0569e7960/tempest-TestContainer-724220972/tempest-LObject-870378192/0 0.008s
2020-10-08T17:21:54.344 INFO:teuthology.orchestra.run.smithi198.stdout:2020-10-08 17:21:54,222 24839 INFO     [tempest.lib.common.rest_client] Request (ObjectTest:test_object_upload_in_segments): 201 PUT http://smithi198.front.sepia.ceph.com:80/v1/KEY_3f726e9d69e04ea082c068d0569e7960/tempest-TestContainer-724220972/tempest-LObject-870378192/1 0.011s
2020-10-08T17:21:54.344 INFO:teuthology.orchestra.run.smithi198.stdout:2020-10-08 17:21:54,231 24839 INFO     [tempest.lib.common.rest_client] Request (ObjectTest:test_object_upload_in_segments): 201 PUT http://smithi198.front.sepia.ceph.com:80/v1/KEY_3f726e9d69e04ea082c068d0569e7960/tempest-TestContainer-724220972/tempest-LObject-870378192/2 0.009s
2020-10-08T17:21:54.344 INFO:teuthology.orchestra.run.smithi198.stdout:2020-10-08 17:21:54,240 24839 INFO     [tempest.lib.common.rest_client] Request (ObjectTest:test_object_upload_in_segments): 201 PUT http://smithi198.front.sepia.ceph.com:80/v1/KEY_3f726e9d69e04ea082c068d0569e7960/tempest-TestContainer-724220972/tempest-LObject-870378192/3 0.008s
2020-10-08T17:21:54.344 INFO:teuthology.orchestra.run.smithi198.stdout:2020-10-08 17:21:54,250 24839 INFO     [tempest.lib.common.rest_client] Request (ObjectTest:test_object_upload_in_segments): 201 PUT http://smithi198.front.sepia.ceph.com:80/v1/KEY_3f726e9d69e04ea082c068d0569e7960/tempest-TestContainer-724220972/tempest-LObject-870378192/4 0.009s
2020-10-08T17:21:54.344 INFO:teuthology.orchestra.run.smithi198.stdout:2020-10-08 17:21:54,258 24839 INFO     [tempest.lib.common.rest_client] Request (ObjectTest:test_object_upload_in_segments): 201 PUT http://smithi198.front.sepia.ceph.com:80/v1/KEY_3f726e9d69e04ea082c068d0569e7960/tempest-TestContainer-724220972/tempest-LObject-870378192/5 0.008s
2020-10-08T17:21:54.344 INFO:teuthology.orchestra.run.smithi198.stdout:2020-10-08 17:21:54,270 24839 INFO     [tempest.lib.common.rest_client] Request (ObjectTest:test_object_upload_in_segments): 201 PUT http://smithi198.front.sepia.ceph.com:80/v1/KEY_3f726e9d69e04ea082c068d0569e7960/tempest-TestContainer-724220972/tempest-LObject-870378192/6 0.010s
2020-10-08T17:21:54.345 INFO:teuthology.orchestra.run.smithi198.stdout:2020-10-08 17:21:54,283 24839 INFO     [tempest.lib.common.rest_client] Request (ObjectTest:test_object_upload_in_segments): 201 PUT http://smithi198.front.sepia.ceph.com:80/v1/KEY_3f726e9d69e04ea082c068d0569e7960/tempest-TestContainer-724220972/tempest-LObject-870378192/7 0.013s
2020-10-08T17:21:54.345 INFO:teuthology.orchestra.run.smithi198.stdout:2020-10-08 17:21:54,293 24839 INFO     [tempest.lib.common.rest_client] Request (ObjectTest:test_object_upload_in_segments): 201 PUT http://smithi198.front.sepia.ceph.com:80/v1/KEY_3f726e9d69e04ea082c068d0569e7960/tempest-TestContainer-724220972/tempest-LObject-870378192/8 0.009s
2020-10-08T17:21:54.345 INFO:teuthology.orchestra.run.smithi198.stdout:2020-10-08 17:21:54,306 24839 INFO     [tempest.lib.common.rest_client] Request (ObjectTest:test_object_upload_in_segments): 201 PUT http://smithi198.front.sepia.ceph.com:80/v1/KEY_3f726e9d69e04ea082c068d0569e7960/tempest-TestContainer-724220972/tempest-LObject-870378192/9 0.012s
2020-10-08T17:21:54.345 INFO:teuthology.orchestra.run.smithi198.stdout:2020-10-08 17:21:54,315 24839 INFO     [tempest.lib.common.rest_client] Request (ObjectTest:test_object_upload_in_segments): 201 PUT http://smithi198.front.sepia.ceph.com:80/v1/KEY_3f726e9d69e04ea082c068d0569e7960/tempest-TestContainer-724220972/tempest-LObject-870378192 0.008s
2020-10-08T17:21:54.345 INFO:teuthology.orchestra.run.smithi198.stdout:2020-10-08 17:21:54,328 24839 INFO     [tempest.lib.common.rest_client] Request (ObjectTest:test_object_upload_in_segments): 202 POST http://smithi198.front.sepia.ceph.com:80/v1/KEY_3f726e9d69e04ea082c068d0569e7960/tempest-TestContainer-724220972/tempest-LObject-870378192 0.012s
2020-10-08T17:21:54.345 INFO:teuthology.orchestra.run.smithi198.stdout:2020-10-08 17:21:54,333 24839 INFO     [tempest.lib.common.rest_client] Request (ObjectTest:test_object_upload_in_segments): 200 HEAD http://smithi198.front.sepia.ceph.com:80/v1/KEY_3f726e9d69e04ea082c068d0569e7960/tempest-TestContainer-724220972/tempest-LObject-870378192 0.004s
2020-10-08T17:21:54.345 INFO:teuthology.orchestra.run.smithi198.stdout:2020-10-08 17:21:54,342 24839 INFO     [tempest.lib.common.rest_client] Request (ObjectTest:test_object_upload_in_segments): 200 GET http://smithi198.front.sepia.ceph.com:80/v1/KEY_3f726e9d69e04ea082c068d0569e7960/tempest-TestContainer-724220972/tempest-LObject-870378192 0.009s

ex: http://qa-proxy.ceph.com/teuthology/cbodley-2020-10-08_16:39:10-rgw-wip-cbodley-testing-distro-basic-smithi/5506944/teuthology.log

Actions
Copy link
 #1

Updated by Matthew Oliver over 3 years ago

  • Assignee set to Matthew Oliver
  • Pull request ID set to 37734

I've recreated the problem. You can simply do it by making a dlo in the Swift api and then try and read it back.

And I think I've gotten to the bottom of it. When we go list the container/bucket we're sending in shard_id = 0 (the default). so we only ever check objects at that shard. Thus the truncated results. As in my testing it wasn't always the last item (as what happens in the tempest test). As we want all the results back all we need to do is set shard_id = -1, so all shards are checked.

Added https://github.com/ceph/ceph/pull/37734

Actions
Copy link
 #2

Updated by Casey Bodley over 3 years ago

  • Status changed from New to Fix Under Review
Actions
Copy link
 #3

Updated by Casey Bodley over 3 years ago

  • Status changed from Fix Under Review to Resolved
Actions
Copy link

Also available in: Atom PDF