Bug #53003
closedPerformance regression on rgw/s3 copy operation
100%
Description
reported on dev@ceph.io by Tim Foerster:
we noticed a massive latency increase on object copy since the pacific
release. Prior pacific the copy operation finished in always less than a
second. The reproducer is quite simple:```
s3cmd mb s3://moo
truncate -s 10G moo.img
s3cmd put moo.img s3://moo/hui --multipart-chunk-size-mb=5000
- expect the time to be less than a second (at least for our env)
- there is a huge gap between latest and latest-octopus )
time s3cmd modify s3://moo/hui --add-header=x-amz-meta-foo3:Bar
```We followed the developer instructions to spin-up the cluster and
bisected the following commit.https://github.com/ceph/ceph/commit/99f7c4aa1286edfea6961b92bb44bb8fe22bd599
I'm not that involved to easily identify the cause from this commit, so
it looks more or less like the issue were introduced earlier it's just
getting used after the wide refactoring.
Updated by Jonas Keidel over 2 years ago
I've tested it with the latest pacific version (16.2.6) and it still happens:
```- time s3cmd modify s3://test/test --add-header=x-amz-meta-foo3:Bar
modify: 's3://test/test'
real 0m10.203s
user 0m0.154s
sys 0m0.048s
```
Updated by Casey Bodley over 2 years ago
- Status changed from New to Fix Under Review
- Pull request ID set to 43710
Updated by Casey Bodley over 2 years ago
thanks for the help in tracking this down!
Updated by Casey Bodley over 2 years ago
- Status changed from Fix Under Review to Pending Backport
Updated by Backport Bot over 2 years ago
- Copied to Backport #53145: pacific: Performance regression on rgw/s3 copy operation added
Updated by Backport Bot almost 2 years ago
- Tags changed from copy to copy backport_processed
Updated by Konstantin Shalygin over 1 year ago
- Status changed from Pending Backport to Resolved
- % Done changed from 0 to 100
- Tags changed from copy backport_processed to copy