Ceph

h1. Multi-object transaction support

*Summary*

This is for multi-object transaction support

(previous blueprint: https://wiki.ceph.com/Planning/Blueprints/Infernalis/osd%3A_Transactions )

*Owners*

Li Wang (Ubuntukylin)

Yunchuan Wen (Ubuntukylin)

Name

*Interested Parties*

If you are interested in contributing to this blueprint, or want to be a "speaker" during the Summit session, list your name here.

Name (Affiliation)

Name (Affiliation)

Name

*Current Status*

Please describe the current status of Ceph as it relates to this blueprint.  Is there something that this replaces?  Are there current features that are related?

*Detailed Description*

Algorithm

(1) Allow client to define the following struct, 

struct MultiObjectTransaction {

> map<hobject_t, ObjectWriteOperation> object_ops;

> hobject_t master;

};

based on this, client could send a group of ops through MOSDOp to the PG coressponding to master object, call it MASTER

(2) Master receive MOSDOp from client, extract ObjectWriteOperations by PGs, and send them to corresponding PGs, call them SLAVE

(3) SLAVE receive MOSDOp from MASTER, if there are pending conventional single-object transactions with operations on the same object, wait until them finished; If there are pending multi-object transactions with the operations on the same object, return EAGAIN (we can not wait here, otherwise may dead lock, see http://article.gmane.org/gmane.comp.file-systems.ceph.devel/23783); Otherwise, contruct the transaction in the conventional way, return MASTER PREPARE_ACK, if error occured, for example, a 'read-then-comparation' failed, return MASTER ERROR

(4) If MASTER receive ERROR from SLAVE, it send ROLLBACK to all SLAVES, cancel the transaction, return client ERROR

(5) MASTER collect all PREPARE_ACK, send SLAVES PREPARE_COMMIT

(6) SLAVE receive PREPARE_COMMIT from MASTER, write pending transaction into PG metadata, return MASTER PREPARE_COMMIT_ACK

(7) MASTER collect all PREPARE_COMMIT_ACK, construct a transaction, including a write to PG metadata to indicat the transaction COMMITTING, and the writes operations on its own data objects, submit the transaction to ObjectStore, then return client ACK, enable client to return from write operation, and send SLAVES COMMIT 

(8) SLAVE receive COMMIT, submit the pending transaction to ObjectStore, after the transaction finished (written to the ultimate location), delete the transaction from PG metadata, return MASTER COMMIT_ACK

(9) MASTER collect COMMIT_ACK, return client COMMIT, to indicate the transaction has been really finished, enable client to do read, delete the transaction from the PG metadata

Error Process

(1) SLAVE down

MASTER could be aware of SLAVE down by OSDMAP, and start a super-peering (inter-PG peering, in constrast to intra-PG peering) to ask SLAVE's state after the SLAVE PG has returned to active-and-clean by conventional peering, and process according to the following different situations,

> (a) No transaction found in SLAVE's PG metadata

> > (1) If MASTER not in COMMITTING, MASTER resend SLAVE MOSDOp, let SLAVE restart from Step (3) as described above;

> > (2) If MASTER in COMMITTING, that imply SLAVE has finished Step (6), and finished the submittion of transaction from PG metadata to its ultimate location, then MASTER do nothing

> (b) Transaction found in SLAVE's PG metadata

> >  (1) MASTER send SLAVE COMMIT, let SLAVE restart from Step (8)

on the other hand, when SLAVE recover from down, if it found transaction in its PG metadata, then it will also start a super-peering to ask MASTER's state, and process according to the following different situations,

> (a) MASTER know nothing about the transaction, which imply that MASTER has just also recovered from down, and lost all the in memory transaction informations, then SLAVE rollback;

> (b) MASTER know this transaction, then MASTER will direct SLAVE to finish the transaction as described above

(2) MASTER down

SLAVE could be aware of MASTER down by OSDMAP, and start a super-peering to ask MASTER's state after MASTER PG returned to active-and-clean, and process according to the following different situations,

> (a) MASTER know nothing about the transaction, imply MASTER has lost all its in memory transaction informations, SLAVE rollback;

> (b) MASTER in COMMITTING, then SLAVE restart from Step (8)

When MASTER recover from down, and found there exists transaction in COMMITTING, then start super-peering to all SLAVES, ask their state, and process according to the following different situations,

> (a) no transaction found in SLAVE's PG metadata, imply SLAVE has done, do nothing;

> (b) transaction found in SLAVE's PG metadata, send SLAVE COMMIT, let SLAVE restart from Step (8) 

Others

(a) A flag could be introduced, to decide whether need a special PREPARE_COMMIT step (Step 3), this step is meant to do a fast deadlock and read-and-comparasion check, if it is not desirable, then Step 3 and 6 could be coalcesed, and Step 5 and 7 could be coalesced, nevertheless, it has no impact on error process described above

(b) For Step 8, we could make use of metadata-only journal mode (http://tracker.ceph.com/projects/ceph/wiki/Rados_-_metadata-only_journal_mode) to speedup the write from PG metadata to ultimate location

*Work items*

This section should contain a list of work tasks created by this blueprint.  Please include engineering tasks as well as related build/release and documentation work.  If this blueprint requires cleanup of deprecated features, please list those tasks as well.

*Coding tasks*

(1) Revise MOSDOp to allow multi-object transaction ops (nearly done);

(2) Revise the call chain from do_op to prepare_transaction to allow transaction pending (ongoing);

(3) Operate PG metadata (done);

(4) Super-peering (ongoing)

*Build / release tasks*

Task 1

Task 2

Task 3

*Documentation tasks*

Task 1

Task 2

Task 3

*Deprecation tasks*

Task 1

Task 2

Task 3