Ceph

h1. Osd - tiering - object redirects

h3. Summary

Create a RADOS redirect primitive and methods for making use of them.  A redirect should function analogously to a symlink, allowing an object to be moved to a different pool but still be accessible transparently by clients looking in the old location.  This would be underlying infrastructure to support tiering.

h3. Owners

* Sage Weil (Inktank)

h3. Interested Parties

* Loic Dachary <loic@dachary.org>

* Sam Just (Inktank)

h3. Current Status

h3. Detailed Description

*--- data types ---*

terminology

p((. origin: original object in original location

target: alternative location of object

new fields for object_info_t:

p((. @enum redir_state;                ///< [origin, target]

object_locator_t redir_oloc;     ///< [origin] locator for target object

eversion_t redir_version;        ///< [origin, target] when this redirect was set to this target

u8 flags;                        ///< [origin]

object_locator_t owner_oloc;     ///< [target] locator for the origin

eversion_t owner_user_version;   ///< [target] user_version, not version!@

where the origin states are:

p((. *NONE*

 *REDIRECT*    we are pointing to another object

 *PROMOTING*    we are copying the target object back to the origin location

 *DEMOTING*    we are copying the primary object to the origin location

 *CLEANUP*   we have the object, but need to delete the demoted object

 *DELETING*   local object is logically non-existent, but we need to clean up target location.

flags are:

p((. PROMOTE_ON_READ

 PROMOTE_ON_WRITE

- we may want to make PROMOTE_ON_WRITE the only behavior for the initial implementation.

- the demoted object has only 2 states:

p((. *NONE*

 *TARGET*      we are pointed to by primary

- primary osd will handle object promote, demote operations (copying to/from alternate location)

  - use backend cluster interface to avoid deadlock from throttling ( loic : how can it deadlock from throttling ? sage: hmm, might not be a problem, as long as no recovery operations can block on the redirect state. )

- objecter can also do a SET_REDIRECT operation:

   - will erase local object and set redirect metadata

- return redirect metadata with GET_REDIRECT ( loic : without GET_REDIRECT it would transparently try again when receiving a EAGAIN, in the same way an http client would on a 302 ? sage: yeah this is like lstat().. we want to find out if we are a redirect origin or target )

*--- osd behavior ---*

on read (no flags):

 NONE, DEMOTING, CLEANUP: do the read

 REDIRECT: send EAGAIN with redirect metadata to client

 PROMOTING: block or forward. ( loic : what does "forward" mean in this context ? I would understand "block then do the read" )

 DELETING: enoent

on read (PROMOTE_ON_READ):

 NONE, CLEANUP: do the read

 DEMOTING: abort the demotion move to CLEANUP and do the the read

 REDIRECT: move to PROMOTING, block then do the read

 PROMOTING: block then do the read

 DELETING: enoent

on write (no flag);

 DEMOTING: block

 REDIRECT: forward

 PROMOTING: block

 DELETING: CLEANUP, proceed.

on write (promote on write);

 DEMOTING:

   move to CLEANUP

 REDIRECT:

   move to PROMOTING, block

 PROMOTING: block

 DELETING: CLEANUP, proceed.

on delete:

 DEMOTING, REDIRECT, PROMOTING, CLEANUP: move to DELETING and queue target object for deletion (as with CLEANUP)

 DELETING: no change.

on any op:

 TARGET: verify the redir_version matches, or EAGAIN

- if we are doing the redirect request and the target does not exist or the version does not match what the redirect/primary had, retry

- the CLEANUP and DELETING states mean the osd needs to remove the redirect and then transition to NONE or delete (respectively)

*--- objecter behavior ---*

- send op to normal location

- on EAGAIN with redirect metadata,

p((.  - note redirect version

  - if this is a retry and version hasn't changed, return error to caller.

  - resend op to alternate location, *including* the primary's eversion_t

  - if we get an error (ENOENT on read), retry from the top

*--- pg log events ---*

redir_demote_start -- we are now allowed to start copying to target pool.  move to DEMOTING

redir_demote_finish -- target is in place; delete local data and set redirect metadata. move to REDIRECT

redir_promote_cleanup -- did copy from target back to origin; still need to clean up old target.  move to CLEANUP

redir_cleanup_finish -- old target is cleaned up.  move to NONE

redir_delete_start -- can remove target, move to DELETING

remove (existing event) -- finished removing target, delete object.

*--- common races ---*

- read vs demote

p((.  - if we hit primary while DEMOTING, we get the result

  - if we get EAGAIN, we read from teh demoted copy

- read vs promote (or read vs demote+prmote)

p((.  - try primary

  - if REDIRECT:

    - EAGAIN, try alternate location

    - result, or ENOENT and back to primary (and block->success or success)

- if PROMOTING, block, then success

*--- in-memory osd state ---*

For each PG, we maintain:

* set<Demotion*> redir_demoting;   ///< all pending demotions

* set<Promotion*> redir_promotion; ///< all pending promotions

* set<Cleanup*> redir_cleanup;       ///< all pending cleanups/deletions.

These structs will have a ref to the ObjectContext and will need to orchestrate the push/pull to do the promotion/demotion.  They will reuse all of the push/pull helpers used by recovery.

*--- snapshots ---*

We can start with a simple approach, and add more complex bheavior from there.

# Force promote-on-write if a non-empty SnapContext is specified.  This ensures that all the snap metadata lives in the main pool and makes sense.  Similarly, we refuse to demote anything that is snapped.

# Allow snaps to be demoted.  For teh primary pool, recovery needs to be adjusted so that the clone_range stuff falls back to a full copy when the snap is a redirect.  In the target pool, recovery needs to behave when we have a subset of the snapset... i.e. just the snapped object.  It may be simplest if it is not a snap at all: foo @12 -> foo_$version @nosnap with key foo.  And writes/cow never happen in the cold pool.

*--- clonerange ---*

If a source item for a clonerange is a redirect, block and promote.

h3. Work items

h3. Coding tasks

# osd: add object_info_t fields for redirects

# add redirect metadata to MOSDOp, MOSDOpReply.  

# add a feature bit.

# osd, objecter, librados, api tests: SET_REDIRECT, GET_REDIRECT operations

# osd: basic redirect logic: reply with EAGAIN on primary, verify or EAGAIN on target.  

# osd: EINVAL or similar if client lacks feature.

# objecter: handle EAGAIN redirects

# osd: pg log entries to indicate state changes (none -> demoting -> redirect -> promoting -> cleanup, deleting, etc.)

# osd: per-PG map of pending redirect states (demoting, promoting, cleanup, tombstone)

# osd: log replay to update pending redirect states

# osd: support deletion.  refactoring to support tombstones.

# osd: promote

# osd: demote

# osd: allow snap

h3. Build / release tasks

# add promote/demote to RadosModel

h3. Documentation tasks

# Task 1