Libradosobjecter - improve threading


The Objecter and librados currently use a single global lock for all state. Break this down to improve multithread performance.


  • Sage Weil (Inktank)
  • Name (Affiliation)
  • Name

Interested Parties

  • Somnath Ray (SanDisk)
  • Haomai Wang (UnitedStack)
  • Guang Yang (Yahoo!)

Current Status

  • Objecter uses a single lock for all state.
  • Objecter doesn't own the lock; it is provided by the caller/owner (e.g., librados's lock, or MDS's lock, or Client's lock)
  • Objecter has two basic categories of state:
    • OSDMap state
    • per-OSD session state
      • list of requests, Connection*, etc.

Detailed Description

Work items

Coding tasks

  1. objecter: take over ownership of lock
    1. do not take caller's lock
    2. remove assumption that completion callbacks are invoked with locks held
    3. remove assumption that messages are passed in with locks held
  2. add read/write lock protecting just OSDMap and session map
    1. use this in place of global lock where possible
  3. replace global counters with atomics
  4. add new lock protecting non-osd operations (stuff for the mon)
  5. move ops, linger_ops, etc into per-session maps
    1. with luck, this will simplify the code that verifies the reply came back from the right now
    2. it will complicate somewhat the code the recalculates targets for ops bc they will move around betwen sessions.held
  6. add per-OSDSession r/w lock
    1. use this in place of the global lock when initiating operations
    2. allocate tid via an atomic inside the session lock
    3. do a readlock/unlock/writelock retry loop when the needed session is not open
  7. modify smalliobench or similar tool to submit requests via many threads