Bug #19085
closedObjects incorrectly report size 0 bytes after upgrade to 11.2.0
0%
Description
I'm running a docker registry using the s3aws storage driver on a ceph cluster. This setup was running very well until I upgraded the ceph cluster including radosgw from 10.2.0 to 11.2.0.
After the upgrade, pulling images from the registry resulted in "image config verification failed for digest sha256:<sha256>" errors. After some investigation it turned out only layers of images that were uploaded after the upgrade were affected.
When I download one of the affected layers from the registry manually, the response is an HTTP 200:
10.20.14.0 - - [24/Feb/2017:16:34:57 +0000] "GET /v2/ays/consumejsonservice/blobs/sha256:6fcd5d005ecfab86dd716406a83dbc2f817e26ff826e4f7db21701171a701718 HTTP/1.1" 200 0 "" "curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.19.1 Basic ECC zlib/1.2.3
Looking at the radowsgw log this looks like this:
2017-02-24 17:43:08.791663 7f3a455ee700 1 civetweb: 0x7f3a8ffd3000: 10.152.47.17 - - [24/Feb/2017:17:43:08 +0100] "GET http://<hostname>/poc-registry2?max-keys=1&prefix=docker%2Fregistry%2Fv2%2Fblobs%2Fsha256%2F6f%2F6fcd5d005ecfab86dd716406a83dbc2f817e26ff826e4f7db21701171a701718%2Fdata HTTP/1.1" 1 0 - aws-sdk-go/1.2.4 (go1.7.3; linux; amd64)
So the registry checks the size of this object, and immediatly returns the HTTP/200 without getting the object from radosgw. The reason is that radosgw is reporting the size as being 0.
When I do an s3cmd ls on this object, it also reports the size as 0:
- s3cmd ls s3://poc-registry2/docker/registry/v2/blobs/sha256/6f/6fcd5d005ecfab86dd716406a83dbc2f817e26ff826e4f7db21701171a701718/data
2017-02-15 07:50 0 s3://poc-registry2/docker/registry/v2/blobs/sha256/6f/6fcd5d005ecfab86dd716406a83dbc2f817e26ff826e4f7db21701171a701718/data
When I get the object, it returns the object correctly, and the sha256sum matches:
- s3cmd get s3://poc-registry2/docker/registry/v2/blobs/sha256/6f/6fcd5d005ecfab86dd716406a83dbc2f817e26ff826e4f7db21701171a701718/data
download: 's3://poc-registry2/docker/registry/v2/blobs/sha256/6f/6fcd5d005ecfab86dd716406a83dbc2f817e26ff826e4f7db21701171a701718/data' -> './data' [1 of 1]
215 of 215 100% in 0s 20.90 kB/s done
- sha256sum data
6fcd5d005ecfab86dd716406a83dbc2f817e26ff826e4f7db21701171a701718 data
So, it looks to me that the object is intact, but for some reason the size is not correct in the metadata...
Updated by Anonymous about 7 years ago
In the meantime I found out that if I put an object with s3cmd the size of the object is correct, so the problem is only when the docker registry2 uploads the objects through the aws-sdk-go/1.2.4 libraries.
Updated by Pranjal Agrawal about 7 years ago
I'm working on this. Can someone set the assignee to me (Pranjal Agrawal), since I don't have sufficient privileges yet?
Updated by Matt Benjamin about 7 years ago
Hi Pranjal,
Can you update this ticket with your analysis? If you have a proposed fix, create a githup pull request and update here!
Thanks!
Matt
Updated by Anonymous over 6 years ago
Tested this again today on Luminous, and I don't see any blobs with size 0 anymore. I guess this got fixed somewhere along the way. This issue can be closed.