Bug #61769
closedBulk upload feature not working
0%
Description
As part of this - https://github.com/ceph/ceph/pull/14775, ceph rgw should support bulkupload feature. But this isn't working as expected in latest quincy release. It is failing with below message when we try to upload files using bulk upload feature -
curl -i -H "X-Auth-Token: $TOKEN" https://<url>/swift/v1/test_1234?extract-archive=tar -X PUT --data-binary @test.tar
HTTP/1.1 200 OK
transfer-encoding: chunked
x-trans-id: tx00000d353477433a6ae6e-006493d898-26a0393-default
x-openstack-request-id: tx00000d353477433a6ae6e-006493d898-26a0393-default
content-type: text/plain; charset=utf-8
date: Thu, 22 Jun 2023 05:14:03 GMT
Number Files Created: 3
Response Body:
Response Status: 400 Bad Request
Errors:
`��DV, 404 Not Found
� ��DV, 404 Not Found
This issue is encountered for few file formats - (ex: .html and .bin)
During upload of these files, rgw doesn't identify the bucket name properly and it gets some garbage value. This results in 404 error.
Snippet from the rgw debug logs -
023-06-22T05:38:52.191+0000 7f90fc66b700 2 req 3378194986423204620 0.000000000s s3:list_buckets executing
2023-06-22T05:38:52.191+0000 7f90fc66b700 20 req 3378194986423204620 0.000000000s s3:list_buckets RGWSI_User_RADOS::list_buckets(): anonymous user
2023-06-22T05:38:52.191+0000 7f90cce0c700 20 req 17170992831263490010 1.939950347s swift:bulk_upload bulk_upload: get_exactly ret=512
2023-06-22T05:38:52.191+0000 7f90cce0c700 2 req 17170992831263490010 1.939950347s swift:bulk_upload handling regular file
2023-06-22T05:38:52.191+0000 7f90cce0c700 20 req 17170992831263490010 1.939950347s swift:bulk_upload got file=<A0><C3>^Y^@V^4/files.html, size=8118^Y
2023-06-22T05:38:52.191+0000 7f90cce0c700 20 req 17170992831263490010 1.939950347s swift:bulk_upload get_system_obj_state: rctx=0x7f90cad84f60 obj=default.rgw.meta:root:<A0><C3>V^4 state=0x560019200760 s->prefetch_data=0^Y
2023-06-22T05:38:52.191+0000 7f90cce0c700 10 req 17170992831263490010 1.939950347s swift:bulk_upload cache get: name=default.rgw.meta+root+<A0><C3>V^4 : miss%19%00V%00%004:head [call version.read in=11b,getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58800) v8 -- 0x560019d45800 con 0x56001863c000
2023-06-22T05:38:52.191+0000 7f90cce0c700 1 -- -- osd_op(unknown.0.0:8362 3.17 3:eebde558:root::%a0%c3
2023-06-22T05:38:52.191+0000 7f90e7641700 2 req 3378194986423204620 0.000000000s s3:list_buckets completing
2023-06-22T05:38:52.191+0000 7f90e7641700 20 req 3378194986423204620 0.000000000s get_system_obj_state: rctx=0x7f90cae06790 obj=default.rgw.log:script.postrequest. state=0x56001b5e7de0 s->prefetch_data=0
2023-06-22T05:38:52.191+0000 7f90e7641700 10 req 3378194986423204620 0.000000000s cache get: name=default.rgw.log++script.postrequest. : hit (negative entry)
2023-06-22T05:38:52.191+0000 7f90e7641700 2 req 3378194986423204620 0.000000000s s3:list_buckets op status=0
2023-06-22T05:38:52.191+0000 7f90e7641700 2 req 3378194986423204620 0.000000000s s3:list_buckets http status=200
2023-06-22T05:38:52.191+0000 7f90e7641700 1 ====== req done req=0x7f90cae07710 op status=0 http_status=200 latency=0.000000000s ======
2023-06-22T05:38:52.191+0000 7f90e7641700 1 beast: 0x7f90cae07710: - anonymous [22/Jun/2023:05:38:52.191 +0000] "GET / HTTP/1.0" 200 231 - - - latency=0.000000000s
2023-06-22T05:38:52.191+0000 7f91f7081700 1 -- <== osd.1179 1 ==== osd_op_reply(8362 <A0><C3>^Y^@V^4 [call,getxattrs,stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 237+0+0 (crc 0 0 0) 0x5600157598c0 con 0x56001863c000Y
2023-06-22T05:38:52.191+0000 7f91c8002700 10 req 17170992831263490010 1.939950347s swift:bulk_upload cache put: name=default.rgw.meta+root+<A0><C3>V^4 info.flags=0x0^Y
2023-06-22T05:38:52.191+0000 7f91c8002700 10 req 17170992831263490010 1.939950347s swift:bulk_upload adding default.rgw.meta+root+<A0><C3>V^4 to cache LRU end^Y
2023-06-22T05:38:52.191+0000 7f91c8002700 20 req 17170992831263490010 1.939950347s swift:bulk_upload non existent directory=<A0><C3>V^^@4
This case was reported with Redhat before and it has already been fixed in RHCS releases but this issue is still seen in upstream releases. Referring BZ case here -
https://bugzilla.redhat.com/show_bug.cgi?id=2132481
https://bugzilla.redhat.com/show_bug.cgi?id=2132482
Files
Updated by Casey Bodley 11 months ago
- Status changed from New to Need More Info
Prajwal Kabbinale wrote:
This case was reported with Redhat before and it has already been fixed in RHCS releases but this issue is still seen in upstream releases. Referring BZ case here -
https://bugzilla.redhat.com/show_bug.cgi?id=2132481
https://bugzilla.redhat.com/show_bug.cgi?id=2132482
thanks Prajwal. https://tracker.ceph.com/issues/57326 is the tracker issue associated with those BZs, but it's not clear to me whether it's the same issue you've reported. are you sure it's the same thing?
the pacific and quincy backports for https://tracker.ceph.com/issues/57326 hadn't been processed yet, so i updated them both so they can make the next point releases
Updated by Casey Bodley 11 months ago
- Related to Bug #57326: using a string_view on a temporary string object added
Updated by Prajwal Kabbinale 11 months ago
I am not sure if this is the same thing. I had raised a case with RH support and they had provided me that the issue I reported was tracked as part of the BZ cases that I provided.
But definitely this is something to do with getting the correct filename during bulk upload.
https://github.com/cbodley/ceph/blob/e9c2ac02f1f59714677e3e1b984367df3c0ec166/src/rgw/rgw_op.cc#L7209 - in this line of code, currently its not getting correct bucket_path
Updated by Casey Bodley 11 months ago
Prajwal Kabbinale wrote:
https://github.com/cbodley/ceph/blob/e9c2ac02f1f59714677e3e1b984367df3c0ec166/src/rgw/rgw_op.cc#L7209 - in this line of code, currently its not getting correct bucket_path
thanks, that helps. the request path here should be "/swift/v1/test_1234" - can you clarify whether test_1234 is supposed to be an account name or a bucket name?
this url_bucket variable is parsed in RGWHandler_REST_SWIFT::init_from_header(), and that logs the value of first which ends up in url_bucket:
ldpp_dout(s, 10) << "ver=" << ver << " first=" << first << " req=" << req << dendl;
are you able to set debug_rgw to 10 or above to capture that log output?
Updated by Prajwal Kabbinale 11 months ago
Yes. The bucket name test_1234 is supported.
Updated by Prajwal Kabbinale 10 months ago
Prajwal Kabbinale wrote:
Yes. The bucket name test_1234 is supported. Provided logs captured by setting debug_rgw to 20.
Updated by Casey Bodley 10 months ago
Prajwal Kabbinale wrote:
Prajwal Kabbinale wrote:
Yes. The bucket name test_1234 is supported. Provided logs captured by setting debug_rgw to 20.
where can i find that log output?
Updated by Prajwal Kabbinale 10 months ago
- File bulk_upload_failed_logs bulk_upload_failed_logs added
Updated by Casey Bodley 10 months ago
2023-06-22T05:38:50.251+0000 7f9127ec2700 10 req 17170992831263490010 0.000000000s ver=v1 first=test_1234 req=
thanks Prajwal, this part looks correct. a grep for 'got file' shows the correct path in some cases:
2023-06-22T05:38:52.167+0000 7f90fee70700 20 req 17170992831263490010 1.915951014s swift:bulk_upload got file=test_1234/1.txt, size=6 2023-06-22T05:38:52.179+0000 7f910c68b700 20 req 17170992831263490010 1.927950621s swift:bulk_upload got file=test_1234/2.txt, size=8 2023-06-22T05:38:52.191+0000 7f90cce0c700 20 req 17170992831263490010 1.939950347s swift:bulk_upload got file=<A0><C3>@^Y^@V^@^@4/files.html, size=8118 2023-06-22T05:38:52.191+0000 7f91c8002700 20 req 17170992831263490010 1.939950347s swift:bulk_upload got file=test_1234/3.txt, size=20 2023-06-22T05:38:52.203+0000 7f9112697700 20 req 17170992831263490010 1.951950073s swift:bulk_upload got file=@<D7><E9>^Z^@V^@^@4/cli.bin, size=0
any idea what's special about
files.html and cli.bin? are they archived under a subdirectory or something?Updated by Prajwal Kabbinale 10 months ago
No Casey. All files were archived same way.
This is the RH case we raised before for the same issue we observed in downstream nautilus cluster - https://access.redhat.com/support/cases/#/case/03317033 as part of this case, RH raise a following BZ case https://bugzilla.redhat.com/show_bug.cgi?id=2130119 for nautilus and they cloned this for both Pacific and Quincy release too.
https://bugzilla.redhat.com/show_bug.cgi?id=2132481
https://bugzilla.redhat.com/show_bug.cgi?id=2132482
As part above BZ cases, RH fixed it in pacific and quincy release (we tested this only in RHCS 5.3 so far and this issue is disappeared)
I was in a hope that this will be fixed in the upstream releases too but that wasn't a case. So I reported here.
Thanks,
Prajwal
Updated by Prajwal Kabbinale 10 months ago
Hi Casey Bodley, any update on this case?
Updated by Casey Bodley 10 months ago
Prajwal Kabbinale wrote:
No Casey. All files were archived same way.
This is the RH case we raised before for the same issue we observed in downstream nautilus cluster - https://access.redhat.com/support/cases/#/case/03317033 as part of this case, RH raise a following BZ case https://bugzilla.redhat.com/show_bug.cgi?id=2130119 for nautilus and they cloned this for both Pacific and Quincy release too.https://bugzilla.redhat.com/show_bug.cgi?id=2132481
https://bugzilla.redhat.com/show_bug.cgi?id=2132482As part above BZ cases, RH fixed it in pacific and quincy release (we tested this only in RHCS 5.3 so far and this issue is disappeared)
I was in a hope that this will be fixed in the upstream releases too but that wasn't a case. So I reported here.
afaik the rhcs 5.3 fix is the same thing as https://tracker.ceph.com/issues/57326. if that resolved the issue, then we're just waiting on its backports
Updated by Casey Bodley 10 months ago
- Status changed from Need More Info to Duplicate
Updated by Prajwal Kabbinale 10 months ago
Any idea on which releases we can expect this backports?
Updated by Casey Bodley 10 months ago
Prajwal Kabbinale wrote:
Any idea on which releases we can expect this backports?
they should make the next pacific/quincy point releases