Bug #3365
closedBroken metadata (duplicated as CSV)
0%
Description
I am having a problem running MediaWiki with rados+rgw. The code has worked fine with Swift for months (and in production). The error I'm seeing is that I'll PUT and object with one key/value metadata pair, called Sha1base36. I expect that when I HEAD the object via PHP CloudFiles (granted we forked our code there a bit) I'll get back the following metadata:
["Sha1base36"]=> "r2baidcig82irc3n9xfozku4mgo8b96"
If at least 2 files, what I am actually getting is:
["sha1base36"]=>
string(262) "r2baidcig82irc3n9xfozku4mgo8b96, r2baidcig82irc3n9xfozku4mgo8b96, r2baidcig82irc3n9xfozku4mgo8b96, r2baidcig82irc3n9xfozku4mgo8b96, r2baidcig82irc3n9xfozku4mgo8b96, r2baidcig82irc3n9xfozku4mgo8b96, r2baidcig82irc3n9xfozku4mgo8b96, r2baidcig82irc3n9xfozku4mgo8b96"
The case is wrong (Swift always makes the first char of the key capitalized) and I have know idea why the value got duplicated many times into a CSV. I think maybe there is some capitilization normalization issue going on here? I'm still trying to rule out bugs purely on the client end though. I ran the same code with MediaWiki using Swift and didn't run into this.
Note that I am using BTRFS if that helps.
Updated by Yehuda Sadeh over 11 years ago
Is that reproducible? If so, a gateway log could help (with debug ms = 1, debug rgw = 20)
Updated by Aaron Schulz over 11 years ago
I made a small logging change to our tests (https://gerrit.wikimedia.org/r/#/c/29346/1) which causes a large number of tests to fail when using ceph. Adding some var_dump() statements reveals that the metadata is:
["sha1base36"]=>
string(31) "0jkfoql5r3blelktbtcm9izn2df5l84"
Does rgw normalize metadata keys to lower case or something? It seems to be different behavoir than Swift.
Updated by Aaron Schulz over 11 years ago
According to https://bugs.launchpad.net/swift/+bug/939982 , clients should not rely on casing. I guess MediaWiki can use strtolower() to work around this.
Unfortunetely, the language bindings (at least the PHP one) doesn't do this automatically nor does it have a "getMetadata" type function - it just returns an associative array with whatever the proxy/gateway gave. If we accept the "follow HTTP spec which says headers in case insensitive" argument, then I suppose it's not wrong to randomly mangle the case or have swift and rgw use different case mangling. Actually, I suppose ceph's makes more sense, since it fully lower case. Maybe that is just something for the docs to give warnings about.
In any case, I still don't know where the csv came from.
Updated by Yehuda Sadeh over 11 years ago
Not sure exactly where the csv is coming from, may be the client library doing something. I'll need to see radosgw logs for that. In any case, swift returns the meta field with first char capitalized, radosgw returns it lower cased.
Updated by Ian Colle about 11 years ago
- Status changed from New to Need More Info
Updated by Aaron Schulz about 11 years ago
Ian Colle wrote:
Aaron are you still seeing this?
Sorry I need to get the time to try and reproduce this (and on the dev branch). How easy is it to go from the stable (via apt-get) to the latest dev release of ceph?
Updated by Sage Weil about 11 years ago
Aaron Schulz wrote:
Ian Colle wrote:
Aaron are you still seeing this?
Sorry I need to get the time to try and reproduce this (and on the dev branch). How easy is it to go from the stable (via apt-get) to the latest dev release of ceph?
If you're on argonaut, you should probably move to the bobtail dev branch. That's just
deb http://gitbuilder.ceph.com/ceph-deb-precise-x86_64-basic/ref/bobtail precise mainor similar; adjust for whatever distro you are running (you can see the list at the to level listing).
Thanks!
Updated by Aaron Schulz about 11 years ago
I'm having a hard time reproducing this on bobtail. If I remove the metadata normalization code in the MediaWiki/CloudFiles and thus reproduce the loop of trying to store a Sha1base36 metadata key (which normalizes to sha1base36 in ceph) again and again nothing strange happens. HEADs/GETs for both the s3 and swift API only show the header one (sha1base36). In the application I see:
array(1) {
["sha1base36"]=>
string(31) "bzv12n7m8g0a0khdo1oibnn68pc9hwz"
}
Maybe this was fixed somewhere along the line. This can probably be closed.
Updated by Ian Colle about 11 years ago
- Status changed from Need More Info to Can't reproduce
Thanks for trying to reproduce this on Bobtail, Aaron. I'm moving it to Can't Reproduce.
Updated by Aaron Schulz about 11 years ago
Sage Weil wrote:
Aaron Schulz wrote:
Ian Colle wrote:
Aaron are you still seeing this?
Sorry I need to get the time to try and reproduce this (and on the dev branch). How easy is it to go from the stable (via apt-get) to the latest dev release of ceph?
If you're on argonaut, you should probably move to the bobtail dev branch. That's just
deb http://gitbuilder.ceph.com/ceph-deb-precise-x86_64-basic/ref/bobtail precise main
or similar; adjust for whatever distro you are running (you can see the list at the to level listing).
Thanks!
For the record, yes, I was on argonaut before.