Ceph » rgw

Is that reproducible? If so, a gateway log could help (with debug ms = 1, debug rgw = 20)

I made a small logging change to our tests (https://gerrit.wikimedia.org/r/#/c/29346/1) which causes a large number of tests to fail when using ceph. Adding some var_dump() statements reveals that the metadata is:
["sha1base36"]=>
string(31) "0jkfoql5r3blelktbtcm9izn2df5l84"

Does rgw normalize metadata keys to lower case or something? It seems to be different behavoir than Swift.

According to https://bugs.launchpad.net/swift/+bug/939982 , clients should not rely on casing. I guess MediaWiki can use strtolower() to work around this.

Unfortunetely, the language bindings (at least the PHP one) doesn't do this automatically nor does it have a "getMetadata" type function - it just returns an associative array with whatever the proxy/gateway gave. If we accept the "follow HTTP spec which says headers in case insensitive" argument, then I suppose it's not wrong to randomly mangle the case or have swift and rgw use different case mangling. Actually, I suppose ceph's makes more sense, since it fully lower case. Maybe that is just something for the docs to give warnings about.

In any case, I still don't know where the csv came from.

Not sure exactly where the csv is coming from, may be the client library doing something. I'll need to see radosgw logs for that. In any case, swift returns the meta field with first char capitalized, radosgw returns it lower cased.

Aaron are you still seeing this?

Ian Colle wrote:

Aaron are you still seeing this?

Sorry I need to get the time to try and reproduce this (and on the dev branch). How easy is it to go from the stable (via apt-get) to the latest dev release of ceph?

Aaron Schulz wrote:

Ian Colle wrote:

Aaron are you still seeing this?

Sorry I need to get the time to try and reproduce this (and on the dev branch). How easy is it to go from the stable (via apt-get) to the latest dev release of ceph?

If you're on argonaut, you should probably move to the bobtail dev branch. That's just

deb http://gitbuilder.ceph.com/ceph-deb-precise-x86_64-basic/ref/bobtail precise main

or similar; adjust for whatever distro you are running (you can see the list at the to level listing).

Thanks!

I'm having a hard time reproducing this on bobtail. If I remove the metadata normalization code in the MediaWiki/CloudFiles and thus reproduce the loop of trying to store a Sha1base36 metadata key (which normalizes to sha1base36 in ceph) again and again nothing strange happens. HEADs/GETs for both the s3 and swift API only show the header one (sha1base36). In the application I see:

array(1) {
["sha1base36"]=>
string(31) "bzv12n7m8g0a0khdo1oibnn68pc9hwz"
}

Maybe this was fixed somewhere along the line. This can probably be closed.

Sage Weil wrote:

Aaron Schulz wrote:

Ian Colle wrote:

Aaron are you still seeing this?

Sorry I need to get the time to try and reproduce this (and on the dev branch). How easy is it to go from the stable (via apt-get) to the latest dev release of ceph?

If you're on argonaut, you should probably move to the bobtail dev branch. That's just

deb http://gitbuilder.ceph.com/ceph-deb-precise-x86_64-basic/ref/bobtail precise main

or similar; adjust for whatever distro you are running (you can see the list at the to level listing).

Thanks!

For the record, yes, I was on argonaut before.