Actions
Bug #23092
openRGW leaking/orphan data with Jewel
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
This was previously reported under #18331 but this popped up again on a Jewel cluster.
Some information:
- Jewel 10.2.9
- Two RADOS Gateway Instances
- Docker DTR as client ( https://docs.docker.com/registry/spec/api/#detail )
Using this command orphans can be found and removed, but that should not be the 'normal'.
radosgw-admin orphans find --pool default.rgw.buckets.data --job-id=checkorph > orphans2
awk '/leaked/ { print $2 }' orphans2 | while read line; do rados -p default.rgw.buckets.data rm $line; done
radosgw-admin orphans finish --job-id=checkorph
I've checked a few things:
- RADOS Gateways do not crash
- Object versioning is not used
Currently we don't have any log files regarding this because running with debug_rgw=20 would generate a lot of data and might need to run for a very long time.
Multipart uploads and possible overwrites of objects are done by the Docker DTR and I have a feeling it's related to that when I see output of the orphan scan:
2018-02-22 09:49:08.694513 7fe3c4fd0a00 0 run(): building index of all bucket indexes storing 1 entries at orphan.scan.checkorph.buckets.9 2018-02-22 09:49:08.712446 7fe3c4fd0a00 0 run(): building index of all linked objects 2018-02-22 09:49:08.712673 7fe3c4fd0a00 0 building linked oids index: 0/64 ERROR: read_entries(orphan.scan.checkorph.buckets.0) returned ret=-2 2018-02-22 09:49:08.714690 7fe3c4fd0a00 0 building linked oids index: 1/64 ERROR: read_entries(orphan.scan.checkorph.buckets.1) returned ret=-2 2018-02-22 09:49:08.717263 7fe3c4fd0a00 0 building linked oids index: 2/64 ERROR: read_entries(orphan.scan.checkorph.buckets.2) returned ret=-2 2018-02-22 09:49:08.719710 7fe3c4fd0a00 0 building linked oids index: 3/64 ERROR: read_entries(orphan.scan.checkorph.buckets.3) returned ret=-2 2018-02-22 09:49:08.721511 7fe3c4fd0a00 0 building linked oids index: 4/64 ERROR: read_entries(orphan.scan.checkorph.buckets.4) returned ret=-2 2018-02-22 09:49:08.723848 7fe3c4fd0a00 0 building linked oids index: 5/64 ERROR: read_entries(orphan.scan.checkorph.buckets.5) returned ret=-2 2018-02-22 09:49:08.725737 7fe3c4fd0a00 0 building linked oids index: 6/64 ERROR: read_entries(orphan.scan.checkorph.buckets.6) returned ret=-2 2018-02-22 09:49:08.727379 7fe3c4fd0a00 0 building linked oids index: 7/64 ERROR: read_entries(orphan.scan.checkorph.buckets.7) returned ret=-2 2018-02-22 09:49:08.729231 7fe3c4fd0a00 0 building linked oids index: 8/64 ERROR: read_entries(orphan.scan.checkorph.buckets.8) returned ret=-2 2018-02-22 09:49:08.731622 7fe3c4fd0a00 0 building linked oids index: 9/64 2018-02-22 09:49:08.750650 7fe3c4fd0a00 0 building linked oids index: 10/64 ERROR: read_entries(orphan.scan.checkorph.buckets.10) returned ret=-2 2018-02-22 09:49:08.752870 7fe3c4fd0a00 0 building linked oids index: 11/64 ERROR: read_entries(orphan.scan.checkorph.buckets.11) returned ret=-2 2018-02-22 09:49:08.754326 7fe3c4fd0a00 0 building linked oids index: 12/64 ERROR: read_entries(orphan.scan.checkorph.buckets.12) returned ret=-2 2018-02-22 09:49:08.756223 7fe3c4fd0a00 0 building linked oids index: 13/64 ERROR: read_entries(orphan.scan.checkorph.buckets.13) returned ret=-2 2018-02-22 09:49:08.757784 7fe3c4fd0a00 0 building linked oids index: 14/64 ERROR: read_entries(orphan.scan.checkorph.buckets.14) returned ret=-2 2018-02-22 09:49:08.759495 7fe3c4fd0a00 0 building linked oids index: 15/64 ERROR: read_entries(orphan.scan.checkorph.buckets.15) returned ret=-2 2018-02-22 09:49:08.761156 7fe3c4fd0a00 0 building linked oids index: 16/64 ERROR: read_entries(orphan.scan.checkorph.buckets.16) returned ret=-2 2018-02-22 09:49:08.763629 7fe3c4fd0a00 0 building linked oids index: 17/64 ERROR: read_entries(orphan.scan.checkorph.buckets.17) returned ret=-2 2018-02-22 09:49:08.765347 7fe3c4fd0a00 0 building linked oids index: 18/64 ERROR: read_entries(orphan.scan.checkorph.buckets.18) returned ret=-2 2018-02-22 09:49:08.767029 7fe3c4fd0a00 0 building linked oids index: 19/64 ERROR: read_entries(orphan.scan.checkorph.buckets.19) returned ret=-2 2018-02-22 09:49:08.768641 7fe3c4fd0a00 0 building linked oids index: 20/64 ERROR: read_entries(orphan.scan.checkorph.buckets.20) returned ret=-2 2018-02-22 09:49:08.770190 7fe3c4fd0a00 0 building linked oids index: 21/64 ERROR: read_entries(orphan.scan.checkorph.buckets.21) returned ret=-2 2018-02-22 09:49:08.771814 7fe3c4fd0a00 0 building linked oids index: 22/64 ERROR: read_entries(orphan.scan.checkorph.buckets.22) returned ret=-2 2018-02-22 09:49:08.774429 7fe3c4fd0a00 0 building linked oids index: 23/64 ERROR: read_entries(orphan.scan.checkorph.buckets.23) returned ret=-2 2018-02-22 09:49:08.776749 7fe3c4fd0a00 0 building linked oids index: 24/64 ERROR: read_entries(orphan.scan.checkorph.buckets.24) returned ret=-2 2018-02-22 09:49:08.778995 7fe3c4fd0a00 0 building linked oids index: 25/64 ERROR: read_entries(orphan.scan.checkorph.buckets.25) returned ret=-2 2018-02-22 09:49:08.781064 7fe3c4fd0a00 0 building linked oids index: 26/64 ERROR: read_entries(orphan.scan.checkorph.buckets.26) returned ret=-2 2018-02-22 09:49:08.782814 7fe3c4fd0a00 0 building linked oids index: 27/64 ERROR: read_entries(orphan.scan.checkorph.buckets.27) returned ret=-2 2018-02-22 09:49:08.785029 7fe3c4fd0a00 0 building linked oids index: 28/64 ERROR: read_entries(orphan.scan.checkorph.buckets.28) returned ret=-2 2018-02-22 09:49:08.786391 7fe3c4fd0a00 0 building linked oids index: 29/64 ERROR: read_entries(orphan.scan.checkorph.buckets.29) returned ret=-2 2018-02-22 09:49:08.787968 7fe3c4fd0a00 0 building linked oids index: 30/64 ERROR: read_entries(orphan.scan.checkorph.buckets.30) returned ret=-2 2018-02-22 09:49:08.789261 7fe3c4fd0a00 0 building linked oids index: 31/64 ERROR: read_entries(orphan.scan.checkorph.buckets.31) returned ret=-2 2018-02-22 09:49:08.791160 7fe3c4fd0a00 0 building linked oids index: 32/64 ERROR: read_entries(orphan.scan.checkorph.buckets.32) returned ret=-2 2018-02-22 09:49:08.793280 7fe3c4fd0a00 0 building linked oids index: 33/64 ERROR: read_entries(orphan.scan.checkorph.buckets.33) returned ret=-2 2018-02-22 09:49:08.795083 7fe3c4fd0a00 0 building linked oids index: 34/64 ERROR: read_entries(orphan.scan.checkorph.buckets.34) returned ret=-2 2018-02-22 09:49:08.796738 7fe3c4fd0a00 0 building linked oids index: 35/64 ERROR: read_entries(orphan.scan.checkorph.buckets.35) returned ret=-2 2018-02-22 09:49:08.798553 7fe3c4fd0a00 0 building linked oids index: 36/64 ERROR: read_entries(orphan.scan.checkorph.buckets.36) returned ret=-2 2018-02-22 09:49:08.801154 7fe3c4fd0a00 0 building linked oids index: 37/64 ERROR: read_entries(orphan.scan.checkorph.buckets.37) returned ret=-2 2018-02-22 09:49:08.802854 7fe3c4fd0a00 0 building linked oids index: 38/64 ERROR: read_entries(orphan.scan.checkorph.buckets.38) returned ret=-2 2018-02-22 09:49:08.804554 7fe3c4fd0a00 0 building linked oids index: 39/64 ERROR: read_entries(orphan.scan.checkorph.buckets.39) returned ret=-2 2018-02-22 09:49:08.806850 7fe3c4fd0a00 0 building linked oids index: 40/64 ERROR: read_entries(orphan.scan.checkorph.buckets.40) returned ret=-2 2018-02-22 09:49:08.808575 7fe3c4fd0a00 0 building linked oids index: 41/64 ERROR: read_entries(orphan.scan.checkorph.buckets.41) returned ret=-2 2018-02-22 09:49:08.810261 7fe3c4fd0a00 0 building linked oids index: 42/64 ERROR: read_entries(orphan.scan.checkorph.buckets.42) returned ret=-2 2018-02-22 09:49:08.811851 7fe3c4fd0a00 0 building linked oids index: 43/64 ERROR: read_entries(orphan.scan.checkorph.buckets.43) returned ret=-2 2018-02-22 09:49:08.814315 7fe3c4fd0a00 0 building linked oids index: 44/64 ERROR: read_entries(orphan.scan.checkorph.buckets.44) returned ret=-2 2018-02-22 09:49:08.815980 7fe3c4fd0a00 0 building linked oids index: 45/64 ERROR: read_entries(orphan.scan.checkorph.buckets.45) returned ret=-2 2018-02-22 09:49:08.818174 7fe3c4fd0a00 0 building linked oids index: 46/64 ERROR: read_entries(orphan.scan.checkorph.buckets.46) returned ret=-2 2018-02-22 09:49:08.819920 7fe3c4fd0a00 0 building linked oids index: 47/64 ERROR: read_entries(orphan.scan.checkorph.buckets.47) returned ret=-2 2018-02-22 09:49:08.821492 7fe3c4fd0a00 0 building linked oids index: 48/64 ERROR: read_entries(orphan.scan.checkorph.buckets.48) returned ret=-2 2018-02-22 09:49:08.823463 7fe3c4fd0a00 0 building linked oids index: 49/64 ERROR: read_entries(orphan.scan.checkorph.buckets.49) returned ret=-2 2018-02-22 09:49:08.825118 7fe3c4fd0a00 0 building linked oids index: 50/64 ERROR: read_entries(orphan.scan.checkorph.buckets.50) returned ret=-2 2018-02-22 09:49:08.826757 7fe3c4fd0a00 0 building linked oids index: 51/64 ERROR: read_entries(orphan.scan.checkorph.buckets.51) returned ret=-2 2018-02-22 09:49:08.828403 7fe3c4fd0a00 0 building linked oids index: 52/64 ERROR: read_entries(orphan.scan.checkorph.buckets.52) returned ret=-2 2018-02-22 09:49:08.830000 7fe3c4fd0a00 0 building linked oids index: 53/64 ERROR: read_entries(orphan.scan.checkorph.buckets.53) returned ret=-2 2018-02-22 09:49:08.831462 7fe3c4fd0a00 0 building linked oids index: 54/64 ERROR: read_entries(orphan.scan.checkorph.buckets.54) returned ret=-2 2018-02-22 09:49:08.833956 7fe3c4fd0a00 0 building linked oids index: 55/64 ERROR: read_entries(orphan.scan.checkorph.buckets.55) returned ret=-2 2018-02-22 09:49:08.835377 7fe3c4fd0a00 0 building linked oids index: 56/64 ERROR: read_entries(orphan.scan.checkorph.buckets.56) returned ret=-2 2018-02-22 09:49:08.837581 7fe3c4fd0a00 0 building linked oids index: 57/64 ERROR: read_entries(orphan.scan.checkorph.buckets.57) returned ret=-2 2018-02-22 09:49:08.839159 7fe3c4fd0a00 0 building linked oids index: 58/64 ERROR: read_entries(orphan.scan.checkorph.buckets.58) returned ret=-2 2018-02-22 09:49:08.841316 7fe3c4fd0a00 0 building linked oids index: 59/64 ERROR: read_entries(orphan.scan.checkorph.buckets.59) returned ret=-2 2018-02-22 09:49:08.842964 7fe3c4fd0a00 0 building linked oids index: 60/64 ERROR: read_entries(orphan.scan.checkorph.buckets.60) returned ret=-2 2018-02-22 09:49:08.845283 7fe3c4fd0a00 0 building linked oids index: 61/64 ERROR: read_entries(orphan.scan.checkorph.buckets.61) returned ret=-2 2018-02-22 09:49:08.847025 7fe3c4fd0a00 0 building linked oids index: 62/64 ERROR: read_entries(orphan.scan.checkorph.buckets.62) returned ret=-2 2018-02-22 09:49:08.849227 7fe3c4fd0a00 0 building linked oids index: 63/64 ERROR: read_entries(orphan.scan.checkorph.buckets.63) returned ret=-2 storing 2 entries at orphan.scan.checkorph.linked.0 storing 1 entries at orphan.scan.checkorph.linked.2 storing 1 entries at orphan.scan.checkorph.linked.3 storing 2 entries at orphan.scan.checkorph.linked.9 storing 1 entries at orphan.scan.checkorph.linked.10 storing 1 entries at orphan.scan.checkorph.linked.12 storing 1 entries at orphan.scan.checkorph.linked.13 storing 1 entries at orphan.scan.checkorph.linked.16 storing 1 entries at orphan.scan.checkorph.linked.20 storing 1 entries at orphan.scan.checkorph.linked.25 storing 1 entries at orphan.scan.checkorph.linked.27 storing 1 entries at orphan.scan.checkorph.linked.31 storing 1 entries at orphan.scan.checkorph.linked.33 storing 1 entries at orphan.scan.checkorph.linked.41 storing 1 entries at orphan.scan.checkorph.linked.44 storing 1 entries at orphan.scan.checkorph.linked.55 storing 1 entries at orphan.scan.checkorph.linked.60 storing 1 entries at orphan.scan.checkorph.linked.62 leaked: cb83e8f5-515d-458a-905c-5c718e1c8352.14097.2__multipart_docker/registry/v2/repositories/admin/nginx/_uploads/fbb24482-8c3a-4951-869a-5f4b2aed8fb1/data.2~iUCoJQLFh_AWXyfAO8_1uN6py0tf428.1 leaked: cb83e8f5-515d-458a-905c-5c718e1c8352.14097.2__multipart_docker/registry/v2/repositories/admin/nginx/_uploads/7b71139a-b01b-4a08-9364-2492693b6037/data.2~RG4YW1HsWmb2UT1leHWm5_Pp5Daiyy3.1 leaked: cb83e8f5-515d-458a-905c-5c718e1c8352.14097.2__multipart_docker/registry/v2/repositories/admin/nginx/_uploads/7b71139a-b01b-4a08-9364-2492693b6037/data.2~RG4YW1HsWmb2UT1leHWm5_Pp5Daiyy3.10 leaked: cb83e8f5-515d-458a-905c-5c718e1c8352.14097.2__multipart_docker/registry/v2/repositories/admin/nginx/_uploads/7b71139a-b01b-4a08-9364-2492693b6037/data.2~RG4YW1HsWmb2UT1leHWm5_Pp5Daiyy3.11 leaked: cb83e8f5-515d-458a-905c-5c718e1c8352.14097.2__multipart_docker/registry/v2/repositories/admin/nginx/_uploads/7b71139a-b01b-4a08-9364-2492693b6037/data.2~RG4YW1HsWmb2UT1leHWm5_Pp5Daiyy3.12 leaked: cb83e8f5-515d-458a-905c-5c718e1c8352.14097.2__multipart_docker/registry/v2/repositories/admin/nginx/_uploads/7b71139a-b01b-4a08-9364-2492693b6037/data.2~RG4YW1HsWmb2UT1leHWm5_Pp5Daiyy3.13 leaked: cb83e8f5-515d-458a-905c-5c718e1c8352.14097.2__multipart_docker/registry/v2/repositories/admin/nginx/_uploads/7b71139a-b01b-4a08-9364-2492693b6037/data.2~RG4YW1HsWmb2UT1leHWm5_Pp5Daiyy3.2 leaked: cb83e8f5-515d-458a-905c-5c718e1c8352.14097.2__multipart_docker/registry/v2/repositories/admin/nginx/_uploads/7b71139a-b01b-4a08-9364-2492693b6037/data.2~RG4YW1HsWmb2UT1leHWm5_Pp5Daiyy3.3 ... ... ... leaked: cb83e8f5-515d-458a-905c-5c718e1c8352.14097.2__shadow_docker/registry/v2/repositories/admin/nginx/_uploads/f00d6919-9739-4e62-acfd-34eec348efcd/data.2~2Ozda0EauUKukcDBnzRGA4FwA1Cwyf0.6_2 leaked: cb83e8f5-515d-458a-905c-5c718e1c8352.14097.2__shadow_docker/registry/v2/repositories/admin/nginx/_uploads/f00d6919-9739-4e62-acfd-34eec348efcd/data.2~2Ozda0EauUKukcDBnzRGA4FwA1Cwyf0.7_1 leaked: cb83e8f5-515d-458a-905c-5c718e1c8352.14097.2__shadow_docker/registry/v2/repositories/admin/nginx/_uploads/f00d6919-9739-4e62-acfd-34eec348efcd/data.2~2Ozda0EauUKukcDBnzRGA4FwA1Cwyf0.7_2 leaked: cb83e8f5-515d-458a-905c-5c718e1c8352.14097.2__shadow_docker/registry/v2/repositories/admin/nginx/_uploads/f00d6919-9739-4e62-acfd-34eec348efcd/data.2~2Ozda0EauUKukcDBnzRGA4FwA1Cwyf0.8_1 leaked: cb83e8f5-515d-458a-905c-5c718e1c8352.14097.2__shadow_docker/registry/v2/repositories/admin/nginx/_uploads/f00d6919-9739-4e62-acfd-34eec348efcd/data.2~2Ozda0EauUKukcDBnzRGA4FwA1Cwyf0.8_2 leaked: cb83e8f5-515d-458a-905c-5c718e1c8352.14097.2__shadow_docker/registry/v2/repositories/admin/nginx/_uploads/f00d6919-9739-4e62-acfd-34eec348efcd/data.2~2Ozda0EauUKukcDBnzRGA4FwA1Cwyf0.9_1 leaked: cb83e8f5-515d-458a-905c-5c718e1c8352.14097.2__shadow_docker/registry/v2/repositories/admin/nginx/_uploads/f00d6919-9739-4e62-acfd-34eec348efcd/data.2~2Ozda0EauUKukcDBnzRGA4FwA1Cwyf0.9_2 leaked: cb83e8f5-515d-458a-905c-5c718e1c8352.14097.2__multipart_docker/registry/v2/repositories/admin/nginx/_uploads/7af6c6a4-3c7d-41ba-b258-98b04a303fd6/data.2~V-7rpyG2O1AysCIioFJiz_QXVqoLPTE.1 leaked: cb83e8f5-515d-458a-905c-5c718e1c8352.14097.2__multipart_docker/registry/v2/repositories/admin/nginx/_uploads/fe9dc324-41ae-4709-b556-daaf6ec5539c/data.2~_IIrN5zhPSky2KL_DAuZMtE67oGOic7.1
A test cluster has the same issue and will be upgraded to Luminous to verify if it's still present there.
Is this a known issue?
Updated by Matt Benjamin almost 6 years ago
I don't think it's a known issue on Jewel. It would be useful to confirm the workflow for affected objects.
Matt
Updated by Abhishek Varshney almost 6 years ago
Hi Orit/Matt,
I am able to reproduce a similar issue in 12.2.5. Here are the steps for the same.
TL;DR : This can happen in cases where a client initiates multipart upload, uploads different parts through different threads and the upload is aborted while other threads continue uploading the objects. Although, in this particular case radosgw-admin orphans find was not able to detect these leaked objects.
Initiate a multipart upload
# s3cmd put ~/Downloads/eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz s3://test/eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz
While the upload is in process, get the upload id and abort it.
# s3cmd multipart s3://test s3://test/ Initiated Path Id 2018-05-31T10:30:36.247Z s3://test/eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz 2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6 # s3cmd abortmp s3://test/eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz 2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6 s3://test/eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz
The original upload ends with 400 (InvalidPart) error
# s3cmd put ~/Downloads/eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz s3://test/eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz WARNING: Module python-magic is not available. Guessing MIME types based on file extensions. upload: '/Users/abhishek.varshney/Downloads/eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz' -> 's3://test/eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz' [part 1 of 12, 15MB] [1 of 1] 15728640 of 15728640 100% in 4s 3.62 MB/s done upload: '/Users/abhishek.varshney/Downloads/eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz' -> 's3://test/eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz' [part 2 of 12, 15MB] [1 of 1] 15728640 of 15728640 100% in 1s 8.64 MB/s done upload: '/Users/abhishek.varshney/Downloads/eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz' -> 's3://test/eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz' [part 3 of 12, 15MB] [1 of 1] 15728640 of 15728640 100% in 1s 9.09 MB/s done upload: '/Users/abhishek.varshney/Downloads/eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz' -> 's3://test/eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz' [part 4 of 12, 15MB] [1 of 1] 15728640 of 15728640 100% in 5s 2.72 MB/s done upload: '/Users/abhishek.varshney/Downloads/eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz' -> 's3://test/eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz' [part 5 of 12, 15MB] [1 of 1] 15728640 of 15728640 100% in 7s 2.06 MB/s done upload: '/Users/abhishek.varshney/Downloads/eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz' -> 's3://test/eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz' [part 6 of 12, 15MB] [1 of 1] 15728640 of 15728640 100% in 5s 2.76 MB/s done upload: '/Users/abhishek.varshney/Downloads/eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz' -> 's3://test/eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz' [part 7 of 12, 15MB] [1 of 1] 15728640 of 15728640 100% in 4s 3.18 MB/s done upload: '/Users/abhishek.varshney/Downloads/eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz' -> 's3://test/eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz' [part 8 of 12, 15MB] [1 of 1] 15728640 of 15728640 100% in 7s 2.07 MB/s done upload: '/Users/abhishek.varshney/Downloads/eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz' -> 's3://test/eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz' [part 9 of 12, 15MB] [1 of 1] 15728640 of 15728640 100% in 5s 2.66 MB/s done upload: '/Users/abhishek.varshney/Downloads/eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz' -> 's3://test/eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz' [part 10 of 12, 15MB] [1 of 1] 15728640 of 15728640 100% in 4s 3.36 MB/s done upload: '/Users/abhishek.varshney/Downloads/eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz' -> 's3://test/eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz' [part 11 of 12, 15MB] [1 of 1] 15728640 of 15728640 100% in 3s 4.73 MB/s done upload: '/Users/abhishek.varshney/Downloads/eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz' -> 's3://test/eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz' [part 12 of 12, 10MB] [1 of 1] 11022833 of 11022833 100% in 3s 2.90 MB/s done ERROR: S3 error: 400 (InvalidPart)
Now, there are no pending multiparts.
s3cmd multipart s3://test s3://test/ Initiated Path Id
gc list has the following entries
# ./bin/radosgw-admin gc list --include-all
2018-05-31 10:36:54.357234 7fa4752d8b80 -1 WARNING: all dangerous and experimental features are enabled.
2018-05-31 10:36:54.357477 7fa4752d8b80 -1 WARNING: all dangerous and experimental features are enabled.
2018-05-31 10:36:54.359430 7fa4752d8b80 -1 WARNING: all dangerous and experimental features are enabled.
[
{
"tag": "2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6",
"time": "2018-05-31 10:30:45.0.82138s",
"objs": [
{
"pool": "default.rgw.buckets.data",
"oid": "494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__multipart_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.1",
"key": "",
"instance": ""
},
{
"pool": "default.rgw.buckets.data",
"oid": "494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__shadow_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.1_1",
"key": "",
"instance": ""
},
{
"pool": "default.rgw.buckets.data",
"oid": "494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__shadow_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.1_2",
"key": "",
"instance": ""
},
{
"pool": "default.rgw.buckets.data",
"oid": "494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__shadow_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.1_3",
"key": "",
"instance": ""
},
{
"pool": "default.rgw.buckets.data",
"oid": "494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__multipart_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.2",
"key": "",
"instance": ""
},
{
"pool": "default.rgw.buckets.data",
"oid": "494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__shadow_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.2_1",
"key": "",
"instance": ""
},
{
"pool": "default.rgw.buckets.data",
"oid": "494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__shadow_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.2_2",
"key": "",
"instance": ""
},
{
"pool": "default.rgw.buckets.data",
"oid": "494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__shadow_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.2_3",
"key": "",
"instance": ""
},
{
"pool": "default.rgw.buckets.data",
"oid": "494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__multipart_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.3",
"key": "",
"instance": ""
},
{
"pool": "default.rgw.buckets.data",
"oid": "494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__shadow_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.3_1",
"key": "",
"instance": ""
},
{
"pool": "default.rgw.buckets.data",
"oid": "494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__shadow_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.3_2",
"key": "",
"instance": ""
},
{
"pool": "default.rgw.buckets.data",
"oid": "494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__shadow_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.3_3",
"key": "",
"instance": ""
}
]
}
]
After processing gc, the data pool has following leaked objects
# ./bin/radosgw-admin gc process # ./bin/rados -p default.rgw.buckets.data ls 2018-05-31 10:39:03.703799 7f288c32de00 -1 WARNING: all dangerous and experimental features are enabled. 2018-05-31 10:39:03.703979 7f288c32de00 -1 WARNING: all dangerous and experimental features are enabled. 2018-05-31 10:39:03.705670 7f288c32de00 -1 WARNING: all dangerous and experimental features are enabled. 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__shadow_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.12_1 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__shadow_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.10_2 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__multipart_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.10 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__multipart_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.4 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__shadow_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.9_3 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__shadow_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.8_3 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__multipart_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.12 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__shadow_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.6_2 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__shadow_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.8_2 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__shadow_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.4_2 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__shadow_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.6_1 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__multipart_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.11 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__shadow_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.10_1 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__shadow_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.7_1 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__shadow_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.10_3 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__shadow_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.11_2 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__shadow_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.4_3 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__shadow_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.5_3 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__shadow_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.12_2 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__shadow_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.11_1 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__multipart_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.7 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__shadow_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.7_3 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__shadow_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.5_2 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__shadow_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.9_2 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__shadow_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.9_1 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__shadow_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.6_3 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__multipart_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.5 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__shadow_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.4_1 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__shadow_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.11_3 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__shadow_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.5_1 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__multipart_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.6 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__multipart_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.9 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__multipart_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.8 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__shadow_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.8_1 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__shadow_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.7_2
The bucket index contains the following entries
# ./bin/rados -p default.rgw.buckets.index listomapkeys .dir.494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1 2018-05-31 10:42:42.103853 7f9d15508e00 -1 WARNING: all dangerous and experimental features are enabled. 2018-05-31 10:42:42.104057 7f9d15508e00 -1 WARNING: all dangerous and experimental features are enabled. 2018-05-31 10:42:42.105861 7f9d15508e00 -1 WARNING: all dangerous and experimental features are enabled. _multipart_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.10 _multipart_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.11 _multipart_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.12 _multipart_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.4 _multipart_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.5 _multipart_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.6 _multipart_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.7 _multipart_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.8 _multipart_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.9
the meta object has following entries
# ./bin/rados -p default.rgw.buckets.non-ec listomapkeys 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__multipart_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.meta 2018-05-31 10:44:19.709229 7f543c3a8e00 -1 WARNING: all dangerous and experimental features are enabled. 2018-05-31 10:44:19.709558 7f543c3a8e00 -1 WARNING: all dangerous and experimental features are enabled. 2018-05-31 10:44:19.711354 7f543c3a8e00 -1 WARNING: all dangerous and experimental features are enabled. part.00000004 part.00000005 part.00000006 part.00000007 part.00000008 part.00000009 part.00000010 part.00000011 part.00000012
Now, let's delete this bucket
# s3cmd rb s3://test Bucket 's3://test/' removed
gc list is now empty
# ./bin/radosgw-admin gc list --include-all 2018-05-31 10:45:55.058937 7fec3dd87b80 -1 WARNING: all dangerous and experimental features are enabled. 2018-05-31 10:45:55.059206 7fec3dd87b80 -1 WARNING: all dangerous and experimental features are enabled. 2018-05-31 10:45:55.061340 7fec3dd87b80 -1 WARNING: all dangerous and experimental features are enabled. []
But, the meta object and the objects in the data pool still exist.
# ./bin/rados -p default.rgw.buckets.non-ec listomapkeys 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__multipart_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.meta 2018-05-31 10:46:31.445700 7fa7de8bbe00 -1 WARNING: all dangerous and experimental features are enabled. 2018-05-31 10:46:31.445931 7fa7de8bbe00 -1 WARNING: all dangerous and experimental features are enabled. 2018-05-31 10:46:31.447662 7fa7de8bbe00 -1 WARNING: all dangerous and experimental features are enabled. part.00000004 part.00000005 part.00000006 part.00000007 part.00000008 part.00000009 part.00000010 part.00000011 part.00000012 # ./bin/rados -p default.rgw.buckets.data ls 2018-05-31 10:46:45.906260 7f3306cafe00 -1 WARNING: all dangerous and experimental features are enabled. 2018-05-31 10:46:45.906462 7f3306cafe00 -1 WARNING: all dangerous and experimental features are enabled. 2018-05-31 10:46:45.908207 7f3306cafe00 -1 WARNING: all dangerous and experimental features are enabled. 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__shadow_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.12_1 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__shadow_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.10_2 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__multipart_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.10 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__multipart_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.4 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__shadow_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.9_3 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__shadow_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.8_3 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__multipart_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.12 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__shadow_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.6_2 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__shadow_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.8_2 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__shadow_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.4_2 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__shadow_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.6_1 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__multipart_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.11 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__shadow_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.10_1 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__shadow_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.7_1 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__shadow_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.10_3 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__shadow_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.11_2 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__shadow_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.4_3 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__shadow_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.5_3 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__shadow_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.12_2 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__shadow_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.11_1 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__multipart_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.7 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__shadow_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.7_3 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__shadow_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.5_2 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__shadow_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.9_2 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__shadow_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.9_1 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__shadow_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.6_3 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__multipart_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.5 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__shadow_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.4_1 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__shadow_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.11_3 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__shadow_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.5_1 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__multipart_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.6 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__multipart_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.9 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__multipart_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.8 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__shadow_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.8_1 494a99d5-0781-4e56-8cd5-d2b412f30f50.4161.1__shadow_eclipse-cpp-mars-2-macosx-cocoa-x86_64.tar.gz.2~acBR_ThRBXS5FKGRWlarDtRoznsKnS6.7_2
Actions