Project

General

Profile

Bug #53003

Performance regression on rgw/s3 copy operation

Added by Casey Bodley about 1 month ago. Updated about 1 month ago.

Status:
Pending Backport
Priority:
High
Assignee:
Target version:
-
% Done:

0%

Source:
Tags:
copy
Backport:
pacific
Regression:
Yes
Severity:
3 - minor
Reviewed:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

reported on by Tim Foerster:

we noticed a massive latency increase on object copy since the pacific
release. Prior pacific the copy operation finished in always less than a
second. The reproducer is quite simple:

```
s3cmd mb s3://moo
truncate -s 10G moo.img
s3cmd put moo.img s3://moo/hui --multipart-chunk-size-mb=5000

  1. expect the time to be less than a second (at least for our env)
  2. there is a huge gap between latest and latest-octopus )
    time s3cmd modify s3://moo/hui --add-header=x-amz-meta-foo3:Bar
    ```

We followed the developer instructions to spin-up the cluster and
bisected the following commit.

https://github.com/ceph/ceph/commit/99f7c4aa1286edfea6961b92bb44bb8fe22bd599

I'm not that involved to easily identify the cause from this commit, so
it looks more or less like the issue were introduced earlier it's just
getting used after the wide refactoring.


Related issues

Copied to rgw - Backport #53145: pacific: Performance regression on rgw/s3 copy operation In Progress

History

#1 Updated by Casey Bodley about 1 month ago

  • Assignee set to Casey Bodley

#2 Updated by Jonas Keidel about 1 month ago

I've tested it with the latest pacific version (16.2.6) and it still happens:

```
  1. time s3cmd modify s3://test/test --add-header=x-amz-meta-foo3:Bar
    modify: 's3://test/test'

real 0m10.203s
user 0m0.154s
sys 0m0.048s
```

#3 Updated by Casey Bodley about 1 month ago

  • Status changed from New to Fix Under Review
  • Pull request ID set to 43710

#4 Updated by Casey Bodley about 1 month ago

thanks for the help in tracking this down!

#5 Updated by Casey Bodley about 1 month ago

  • Status changed from Fix Under Review to Pending Backport

#6 Updated by Backport Bot about 1 month ago

  • Copied to Backport #53145: pacific: Performance regression on rgw/s3 copy operation added

Also available in: Atom PDF