Ceph » rgw

Multi-part uploads to S3 on RGW silently cause corruption if you use a non-default chunk size, e.g. 5242881 bytes, which is the default plus one.

I ran into this using Alpakka S3, which can be used to upload streams of (beforehand) unknown size to S3. This triggers the bug because only a minimum chunk size can be specified, not an exact size.

This problem does not occur with AWS S3.

Steps to reproduce:

• install the AWS CLI as per https://docs.aws.amazon.com/cli/latest/userguide/install-linux.html:
  pip3 install awscli --upgrade --user

• configure AWS CLI - provide credentials and use region default:
  aws configure

• Create a SSE-C key. Corruption will occur consistently with any key, but different keys will yield distinct corruption patterns.
  dd if=/dev/zero of=/tmp/ssec.key bs=1 count=32

• Generate test data - 10MB of NUL bytes (0x00):
  dd if=/dev/zero of=/tmp/foo bs=10000 count=1000

• Upload and download the file - this will be a multi-part upload because the input file is larger than the default multi-part chunk size (5242880, i.e. 5 *1024*1024 = 80 * 2^16 bytes)
  aws s3 cp /tmp/foo s3://<bucket_name>/ --sse-c AES256 --sse-c-key fileb:///tmp/ssec.key --endpoint-url https://<rados gateway host>
aws s3 cp s3://<bucket_name>/foo /tmp/bar --sse-c AES256 --sse-c-key fileb:///tmp/ssec.key --endpoint-url https://<rados gateway host>

• Verify file integrity - no surprises here:
  sha256sum /tmp/foo -> f5e02aa71e67f41d79023a128ca35bad86cf7b6656967bfe0884b3a3c4325eaf
sha256sum /tmp/bar -> f5e02aa71e67f41d79023a128ca35bad86cf7b6656967bfe0884b3a3c4325eaf

Now, it gets interesting.

• Set the chunk size to a non-default value and repeat the process:
  aws configure set default.s3.multipart_chunksize 5242881
aws s3 cp /tmp/foo s3://<bucket_name>/ --sse-c AES256 --sse-c-key fileb:///tmp/ssec.key --endpoint-url https://<rados gateway host>
aws s3 cp s3://<bucket_name>/foo /tmp/bar --sse-c AES256 --sse-c-key fileb:///tmp/ssec.key --endpoint-url https://<rados gateway host>
• Now, the downloaded file is corrupt!
  sha256sum /tmp/foo -> f5e02aa71e67f41d79023a128ca35bad86cf7b6656967bfe0884b3a3c4325eaf
sha256sum /tmp/bar -> 2351da404fde47c360d201cf77311afd1d5e8cbfb601a1db1cee0b8f82124554

Taking a closer look at the files:

• There are indeed 10M bytes in the file:
  cat foo | wc -c
• Filter out all NUL bytes and count the remainder; this outputs 0, confirming that the original foo contains only NUL bytes:
  tr -d '\000' < foo | wc -c

Then, from the corrupted bar file, we take the first n bytes, filter the NULs, and count how many remain.

• For the first chunksize bytes, there are 0, meaning the contents of the first chunk are not corrupted:
  head -c 5242881 bar | tr -d '\000' | wc -c
• Take one more byte, and the output changes to 1, meaning corruption starts at the part uploaded second:
  head -c 5242882 bar | tr -d '\000' | wc -c

Examining bar in a file editor shows that the remainder of the file is indeed garbage.

Uploading and downloading the file multiple times with a particular SSE-C key yields the exact same result, so this is a deterministic bug.