Backport #23237
Updated by Casey Bodley almost 5 years ago
[backport pr https://github.com/ceph/ceph/pull/27982]
We have a problem in civetweb downloads, our clients have slow connections and download large files using multi-segment downloads (download managers), using fastcgi, everything works fine, using civetweb, lots of downloads are corrupted.
We were able to reproduce the problem, using these steps:
The RGW uses "rgw_frontends = fastcgi, civetweb port=7480" and nginx listens on port 8080 with "fastcgi_pass unix:/run/ceph/ceph.radosgw.gateway.fastcgi.sock"
Inside a test dir, run these commands
1) Download using 8 segments, with 350 KB/s speed limit, using fastcgi with nginx, no problems (all MD5s match)
for i in {1..10}; do aria2c --max-overall-download-limit=350K -s 8 -x 8 -o test$i "http://192.168.217.234:8080/mustafa-test/CentOS-7-x86_64-NetInstall-1708.iso?AWSAccessKeyId=ZTCEWB8UCH1HR47ZQJYT&Expires=1521316274&Signature=XL4E4s8%2F%2FUiL6mFPnby%2BAyokvIk%3D" && md5sum test$i >> testmd5 & done
cat testmd5
8fc7bb301bccbf84030ff4faf3a4bbb5 test8
8fc7bb301bccbf84030ff4faf3a4bbb5 test9
8fc7bb301bccbf84030ff4faf3a4bbb5 test2
8fc7bb301bccbf84030ff4faf3a4bbb5 test1
8fc7bb301bccbf84030ff4faf3a4bbb5 test4
8fc7bb301bccbf84030ff4faf3a4bbb5 test10
8fc7bb301bccbf84030ff4faf3a4bbb5 test5
8fc7bb301bccbf84030ff4faf3a4bbb5 test6
8fc7bb301bccbf84030ff4faf3a4bbb5 test7
8fc7bb301bccbf84030ff4faf3a4bbb5 test3
2) Download using 1 segment, with 350 KB/s speed limit, using civetweb, no problems (all MD5s match)
rm -f test*; for i in {1..10}; do aria2c --max-overall-download-limit=350K -o test$i "http://192.168.217.234:7480/mustafa-test/CentOS-7-x86_64-NetInstall-1708.iso?AWSAccessKeyId=ZTCEWB8UCH1HR47ZQJYT&Expires=1521316274&Signature=XL4E4s8%2F%2FUiL6mFPnby%2BAyokvIk%3D" && md5sum test$i >> testmd5 & done
cat testmd5
8fc7bb301bccbf84030ff4faf3a4bbb5 test6
8fc7bb301bccbf84030ff4faf3a4bbb5 test7
8fc7bb301bccbf84030ff4faf3a4bbb5 test3
8fc7bb301bccbf84030ff4faf3a4bbb5 test2
8fc7bb301bccbf84030ff4faf3a4bbb5 test9
8fc7bb301bccbf84030ff4faf3a4bbb5 test10
8fc7bb301bccbf84030ff4faf3a4bbb5 test4
8fc7bb301bccbf84030ff4faf3a4bbb5 test8
8fc7bb301bccbf84030ff4faf3a4bbb5 test1
8fc7bb301bccbf84030ff4faf3a4bbb5 test5
3) Download using 8 segments, with 350 KB/s speed limit, using civetweb, corrupted files (different MD5 for each file)
rm -f test*; for i in {1..10}; do aria2c --max-overall-download-limit=350K -s 8 -x 8 -o test$i "http://192.168.217.234:7480/mustafa-test/CentOS-7-x86_64-NetInstall-1708.iso?AWSAccessKeyId=ZTCEWB8UCH1HR47ZQJYT&Expires=1521316274&Signature=XL4E4s8%2F%2FUiL6mFPnby%2BAyokvIk%3D" && md5sum test$i >> testmd5 & done
cat testmd5
50b2bee56e19460e165e24eb7bd490d0 test1
934b1c7e5bdeee64632d3fac89b17226 test8
13655ac02a174fb11eff02a21ee92c4d test9
5053d361eeb569057ea8df191460bf96 test7
aeb85750cec63c16214bc5a732cd3411 test2
120a4e4f77f1994f6939aa722c678749 test5
362f146feb8461deca297f6a6dd5eec0 test6
de77a8d528fece0c48b7a330c2c3dbf5 test4
475bff03d5fd6ddd0acfce3a57f048dd test3
ab87c95fc5e5561786b23ef58ac65efd test10
Starts happening when the download takes about 10~15 minutes, and increases with download time, you can see all the downloads are corrupt after 20~25 minutes (last case), result files have correct size but corrupt data.
We tested using the publicly available file CentOS-7-x86_64-NetInstall-1708.iso so you can reproduce exactly as we did.
Also attached is the RGW log