Ceph » rgw

I should mention: awscli does v4 signatures by default, but when properly coaxed (by giving it a specially prepared "endpoints.json" file), it can be coerced into doing v2 signatures. The url it emits is a faulty host based url: "bucket.s3.amazonaws.com", but the signature formed is using /bucket/object matching the behavior of s3cmd. And just as in the s3cmd case the signature works with radosgw when the URL is munged into a valid path style URL.

Giving awscli a differently prepared "endpoints.json" file (or using --endpoint=) causes it to generate v4 signatures instead, which work fine with ceph.

Either you're doing something weird, or it's broken between Jewel & master.

Here is working output on Jewel (not quite 10.2.7, 10.2.6 + most RGW stuff backported by me before the release) it DOES work to generate Vhost calling-format URLs, and you can manually re-arrange it to have working path-format access.

$ .../s3cmd -c $MYCFG signurl s3://robjoh84-congress-private-test/testfile-private `date -d 'now + 1 year' +%s`
http://robjoh84-congress-private-test.objects-us-west-1.dream.io/testfile-private?AWSAccessKeyId=TUqXKJSgG_c8wRL4222l&Expires=1530857403&Signature=lPU7VxM9hWVAlAUDSbIQDPlo2e8%3D

Feel free to check it.

And manually re-arrange it:
http://objects-us-west-1.dream.io/robjoh84-congress-private-test/testfile-private?AWSAccessKeyId=TUqXKJSgG_c8wRL4222l&Expires=1530857403&Signature=lPU7VxM9hWVAlAUDSbIQDPlo2e8%3D

Running s3cmd v1.6.1-2-gf0340a6, which is 1.6.1 w/ a minor packaging fix.

@mwatts, can you update with info from last night?

Matt

I checked: jewel has the same behavior.

I don't think Robin is describing different behavior than what I saw. We agree: it works if you manually rearrange the URL to force path style access. So the only question is whether that behavior is a bug or a feature.

In my case v2 worked in both calling formats

@mwatts I can't get it to fail in my production setups, maybe your setup had a weird rgw dns name or zone config?

Hello!

My team is trying to update from Hammer to Jewel, and we believe we have run into this issue. We are testing with s3curl (https://github.com/rtdp/s3curl/blob/master/s3curl.pl), which uses the V2 signature. The command we use (apologies for vaguery here):

s3curl --id the_test_user_profile -- -s http://region_url_mapped_to_rgw:7480/mytestbucket/sometest.file

This results in a 403. I turned on

debug rgw = 20

How far along are you on figuring out where the algorithm changed?

I was able to reproduce this by running s3curl against demo Docker containers.

ceph/demo:tag-build-master-hammer-ubuntu-14.04 returns successfully
ceph/demo:tag-build-master-jewel-ubuntu-14.04 returns the 403

$ cat test_ceph.sh
#!/bin/bash

version=$1

if [ "$version" == "" ]; then
    echo "Specify hammer or jewel as a parameter" 
    exit 1
fi

CEPH_IMAGE="ceph/demo:tag-build-master-${version}-ubuntu-14.04" 

docker rm -f ceph
docker run -p 7280:8080 -d -e RGW_CIVETWEB_PORT=8080 -e MON_IP=127.0.0.1 -e CEPH_PUBLIC_NETWORK=172.18.0.0/24 --name ceph ${CEPH_IMAGE}
docker exec ceph radosgw-admin user create --uid=uat_admin --display-name="Ceph user for testing" --access-key=CEPHUSER --secret=thisisasecret --access=full --system

$ cat ~/.s3curl
@endpoints = (
  'localhost',
);

%awsSecretAccessKeys = (
    sigtester => {
      id => 'CEPHUSER',
      key => 'thisisasecret',
    },
);

$ ./test_ceph.sh hammer
...

$ s3curl --id sigtester http://localhost:7280
<?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>uat_admin</ID><DisplayName>Ceph user for testing</DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>

$ ./test_ceph.sh jewel
...

$ s3curl --id sigtester http://localhost:7280
<?xml version="1.0" encoding="UTF-8"?><Error><Code>SignatureDoesNotMatch</Code><RequestId>tx000000000000000000001-0059764eee-100d-default</RequestId><HostId>100d-default-default</HostId></Error>

Discovered that Hammer's auth_hdr debug output ends with:

/mytestbucket/sometest.file

But in Jewel, it ends with:

/region_url_mapped_to_rgw/mytestbucket/sometest.file

This is why the signatures do not match. Jewel incorrectly includes the host, probably assuming it is the bucket? (Reading above suggests folks already knew that.)

We can ignore my comments here. They are caused by an incorrect configuration of "rgw dns name" in /etc/ceph/ceph.conf. RGW uses that to parse out the host and detect bucket vs host.