Ceph

SO ANNOYING.

Ok, so snapdir is really just a whiteout head. Time to admit that. Path forward:

1) Update existing code to completely ban snapdir except when generating the backend operation. That is, update the existing code paths to use the whiteout field in code, but not to write it out to disk, instead writing a snapdir. get_object_context etc should transparently check for snapdir as needed.

2) Add code which simply writes out the object_info_t with the whiteout field once the right OSDMap flag is set cluster wide banning osds which assume that a snapdir exists. When an osd reads a snapdir into an object info, add an ephemeral is_snapdir flag to the ObjectContext and transparently kill it the next time we do a write on that object (like a snap trim!).
ObjectContext { bool snapdir_exists; }

3) Add a may_have_snapdir set to true to the pg_info_t. Once we scrub the whole PG without finding any snapdir objects, flip the bool and share_pg_info(). If that flag is set in the info, no need to check for snapdir in get_object_context.

pg_info_t { may_have_snapdir = true; };

4) Should add a tool to scan through a pg and cause the snapdir->head transition. How about sending a transaction which does a touch,delete?