Ceph » rgw

While attempting to write a python script around radosgw-admin object stat, I found that I was occasionally receiving UnicodeDecodeError exceptions when loading the output as JSON, such as:

UnicodeDecodeError: 'utf-8' codec cannot decode byte 0xe4 in position 4802: invalid continuation byte

Tracking down the offending bytes, I noticed that they are always in the attrs section, and are found in the values of user.rgw.pg_ver and user.rgw.source_zone. Looking through the source, it looks like these are uint64_t and uint32_t objects, which is perfectly reasonable for the underlying data.

The trouble is that when radosgw-admin dumps these attributes it assumes that they're all stringly typed, calling ceph::buffer::list::to_str(), which seems to whack all the contained bytes into a string. As the underlying data are integer types, there's no guarantee that this will form a valid UTF-8 encoded string, and even when it does the output is not really what we want.

I think the easiest solution here is to add special handling for these attrs alongside other system attributes that are already treated differently, such as etag or delete_at. I'm happy to make that change. I think any fix is going to require special handling like this, as there isn't a way to inspect the serialised type of an arbitrary bufferlist at runtime.