Project

General

Profile

Actions
Copy link

Bug #55475

closed

"ERROR: ragweed.tests.tests.r_test_multipart_defer_complete.test" in upgrade:octopus-x-pacific

Added by Yuri Weinstein almost 2 years ago. Updated almost 2 years ago.

Status:
Closed
Priority:
Urgent
Assignee:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
Ceph - v16.2.8
ceph-qa-suite:
rgw
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

This is for 16.2.8 release

Run: http://pulpito.front.sepia.ceph.com/yuriw-2022-04-27_14:24:25-upgrade:octopus-x-pacific-distro-default-smithi
Job: 6808910
Logs: http://qa-proxy.ceph.com/teuthology/yuriw-2022-04-27_14:24:25-upgrade:octopus-x-pacific-distro-default-smithi/6808910/teuthology.log

022-04-27T18:41:41.375 INFO:teuthology.orchestra.run.smithi157.stderr:======================================================================
2022-04-27T18:41:41.375 INFO:teuthology.orchestra.run.smithi157.stderr:ERROR: ragweed.tests.tests.r_test_multipart_defer_complete.test
2022-04-27T18:41:41.375 INFO:teuthology.orchestra.run.smithi157.stderr:----------------------------------------------------------------------
2022-04-27T18:41:41.375 INFO:teuthology.orchestra.run.smithi157.stderr:Traceback (most recent call last):
2022-04-27T18:41:41.375 INFO:teuthology.orchestra.run.smithi157.stderr:  File "/home/ubuntu/cephtest/ragweed/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest
2022-04-27T18:41:41.375 INFO:teuthology.orchestra.run.smithi157.stderr:    self.test(*self.arg)
2022-04-27T18:41:41.375 INFO:teuthology.orchestra.run.smithi157.stderr:  File "/home/ubuntu/cephtest/ragweed/ragweed/framework.py", line 345, in test
2022-04-27T18:41:41.376 INFO:teuthology.orchestra.run.smithi157.stderr:    self.check()
2022-04-27T18:41:41.376 INFO:teuthology.orchestra.run.smithi157.stderr:  File "/home/ubuntu/cephtest/ragweed/ragweed/tests/tests.py", line 332, in check
2022-04-27T18:41:41.376 INFO:teuthology.orchestra.run.smithi157.stderr:    uploader.complete()
2022-04-27T18:41:41.376 INFO:teuthology.orchestra.run.smithi157.stderr:  File "/home/ubuntu/cephtest/ragweed/ragweed/tests/tests.py", line 231, in complete
2022-04-27T18:41:41.376 INFO:teuthology.orchestra.run.smithi157.stderr:    self.mp.complete_upload()
2022-04-27T18:41:41.376 INFO:teuthology.orchestra.run.smithi157.stderr:AttributeError: 'MultipartUploader' object has no attribute 'mp'
2022-04-27T18:41:41.377 INFO:teuthology.orchestra.run.smithi157.stderr:-------------------- >> begin captured stdout << ---------------------
Actions
Copy link
 #1

Updated by Neha Ojha almost 2 years ago

  • Project changed from Ceph to rgw
Actions
Copy link
 #2

Updated by Casey Bodley almost 2 years ago

  • Assignee set to Casey Bodley
Actions
Copy link
 #3

Updated by Casey Bodley almost 2 years ago

  • Assignee deleted (Casey Bodley)

still trying to figure out what the test is doing, but it may be a regression in ListMultipartUploads

Actions
Copy link
 #4

Updated by Casey Bodley almost 2 years ago

there are other ragweed test failures here that look different:

======================================================================
ERROR: ragweed.tests.tests.r_test_multipart_defer_update_complete.test
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ubuntu/cephtest/ragweed/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/ubuntu/cephtest/ragweed/ragweed/framework.py", line 344, in test
    self.load()
  File "/home/ubuntu/cephtest/ragweed/ragweed/framework.py", line 335, in load
    suite.zone.refresh_rbucket(rb)
  File "/home/ubuntu/cephtest/ragweed/ragweed/framework.py", line 285, in refresh_rbucket
    rbucket.bucket = self.get_raw_bucket(rbucket.bucket.name)
  File "/home/ubuntu/cephtest/ragweed/ragweed/framework.py", line 282, in get_raw_bucket
    return self.conn.regular.get_bucket(name)
  File "/home/ubuntu/cephtest/ragweed/ragweed/framework.py", line 39, in get_bucket
    return self.conn.get_bucket(name, validate=validate)
  File "/home/ubuntu/cephtest/ragweed/virtualenv/lib/python3.6/site-packages/boto/s3/connection.py", line 509, in get_bucket
    return self.head_bucket(bucket_name, headers=headers)
  File "/home/ubuntu/cephtest/ragweed/virtualenv/lib/python3.6/site-packages/boto/s3/connection.py", line 553, in head_bucket
    raise err
boto.exception.S3ResponseError: S3ResponseError: 404 Not Found

here's the RGW log's view of events from before the upgrade:

// CreateBucket
2022-04-27T18:39:41.967+0000 7f93f54e4700  1 civetweb: 0x5636ca97c000: 172.21.15.138 - - [27/Apr/2022:18:39:39 +0000] "PUT /test-client.1-r-test-multipart-defer-update-complete.1/ HTTP/1.1" 200 142 - Boto/2.49.0 Python/3.6.8 Linux/4.18.0-383.el8.x86_64

// InitMultipart
2022-04-27T18:39:41.976+0000 7f93f54e4700  1 civetweb: 0x5636ca97c000: 172.21.15.138 - - [27/Apr/2022:18:39:39 +0000] "POST /test-client.1-r-test-multipart-defer-update-complete.1/foo?uploads HTTP/1.1" 200 463 - Boto/2.49.0 Python/3.6.8 Linux/4.18.0-383.el8.x86_64

// PutObject for part 1
2022-04-27T18:39:42.076+0000 7f93f54e4700  1 civetweb: 0x5636ca97c000: 172.21.15.138 - - [27/Apr/2022:18:39:39 +0000] "PUT /test-client.1-r-test-multipart-defer-update-complete.1/foo?uploadId=2~oLWXWG2CG0Q0y7avQ9iLJNOHZOBSxxV&partNumber=1 HTTP/1.1" 200 231 - Boto/2.49.0 Python/3.6.8 Linux/4.18.0-383.el8.x86_64

after the upgrade, we fail a HEAD request on this bucket with 404 Not Found:

// HeadBucket
2022-04-27T18:44:34.140+0000 7ff0d596c700  1 civetweb: 0x55f734afa000: 172.21.15.138 - - [27/Apr/2022:18:44:33 +0000] "HEAD /test-client.1-r-test-multipart-defer-update-complete.1/ HTTP/1.1" 404 204 - Boto/2.49.0 Python/3.6.8 Linux/4.18.0-383.el8.x86_64

Actions
Copy link
 #5

Updated by Casey Bodley almost 2 years ago

  • Assignee set to Casey Bodley
Actions
Copy link
 #6

Updated by Casey Bodley almost 2 years ago

  • Assignee deleted (Casey Bodley)

trace for r_test_multipart_defer_complete:

// CreateBucket
2022-04-27T18:39:39.921+0000 7f93f54e4700  1 civetweb: 0x5636ca97c000: 172.21.15.138 - - [27/Apr/2022:18:39:39 +0000] "PUT /test-client.1-r-test-multipart-defer-complete.1/ HTTP/1.1" 200 142 - Boto/2.49.0 Python/3.6.8 Linux/4.18.0-383.el8.x86_64

// InitMultipart
2022-04-27T18:39:41.601+0000 7f93f54e4700  1 civetweb: 0x5636ca97c000: 172.21.15.138 - - [27/Apr/2022:18:39:39 +0000] "POST /test-client.1-r-test-multipart-defer-complete.1/foo?uploads HTTP/1.1" 200 456 - Boto/2.49.0 Python/3.6.8 Linux/4.18.0-383.el8.x86_64

// part uploads
2022-04-27T18:39:41.728+0000 7f93f54e4700  1 civetweb: 0x5636ca97c000: 172.21.15.138 - - [27/Apr/2022:18:39:39 +0000] "PUT /test-client.1-r-test-multipart-defer-complete.1/foo?uploadId=2~k-D0_jBzc1QsSRS07vX_bcA9Sy_8M1j&partNumber=1 HTTP/1.1" 200 231 - Boto/2.49.0 Python/3.6.8 Linux/4.18.0-383.el8.x86_64
2022-04-27T18:39:41.800+0000 7f93f54e4700  1 civetweb: 0x5636ca97c000: 172.21.15.138 - - [27/Apr/2022:18:39:39 +0000] "PUT /test-client.1-r-test-multipart-defer-complete.1/foo?uploadId=2~k-D0_jBzc1QsSRS07vX_bcA9Sy_8M1j&partNumber=2 HTTP/1.1" 200 231 - Boto/2.49.0 Python/3.6.8 Linux/4.18.0-383.el8.x86_64
2022-04-27T18:39:41.861+0000 7f93f54e4700  1 civetweb: 0x5636ca97c000: 172.21.15.138 - - [27/Apr/2022:18:39:39 +0000] "PUT /test-client.1-r-test-multipart-defer-complete.1/foo?uploadId=2~k-D0_jBzc1QsSRS07vX_bcA9Sy_8M1j&partNumber=3 HTTP/1.1" 200 231 - Boto/2.49.0 Python/3.6.8 Linux/4.18.0-383.el8.x86_64
2022-04-27T18:39:41.910+0000 7f93f54e4700  1 civetweb: 0x5636ca97c000: 172.21.15.138 - - [27/Apr/2022:18:39:39 +0000] "PUT /test-client.1-r-test-multipart-defer-complete.1/foo?uploadId=2~k-D0_jBzc1QsSRS07vX_bcA9Sy_8M1j&partNumber=4 HTTP/1.1" 200 231 - Boto/2.49.0 Python/3.6.8 Linux/4.18.0-383.el8.x86_64

after upgrade:
// HeadBucket
2022-04-27T18:44:34.043+0000 7ff0d596c700  1 civetweb: 0x55f734afa000: 172.21.15.138 - - [27/Apr/2022:18:44:33 +0000] "HEAD /test-client.1-r-test-multipart-defer-complete.1/ HTTP/1.1" 200 342 - Boto/2.49.0 Python/3.6.8 Linux/4.18.0-383.el8.x86_64

// ListMultipartUploads
2022-04-27T18:44:34.050+0000 7ff0d596c700  1 civetweb: 0x55f734afa000: 172.21.15.138 - - [27/Apr/2022:18:44:33 +0000] "GET /test-client.1-r-test-multipart-defer-complete.1/?uploads HTTP/1.1" 200 462 - Boto/2.49.0 Python/3.6.8 Linux/4.18.0-383.el8.x86_64

Actions
Copy link
 #7

Updated by Casey Bodley almost 2 years ago

  • Status changed from New to Closed

this was a regression in bluestore, and was addressed by https://github.com/ceph/ceph/pull/46095

Actions
Copy link

Also available in: Atom PDF