Libradosobjecter - improve threading¶
Summary¶
The Objecter and librados currently use a single global lock for all state. Break this down to improve multithread performance.
Owners¶
- Sage Weil (Inktank)
- Name (Affiliation)
- Name
Interested Parties¶
- Somnath Ray (SanDisk)
- Haomai Wang (UnitedStack)
- Guang Yang (Yahoo!)
Current Status¶
- Objecter uses a single lock for all state.
- Objecter doesn't own the lock; it is provided by the caller/owner (e.g., librados's lock, or MDS's lock, or Client's lock)
- Objecter has two basic categories of state:
- OSDMap state
- per-OSD session state
- list of requests, Connection*, etc.
Detailed Description¶
Work items¶
Coding tasks¶
- objecter: take over ownership of lock
- do not take caller's lock
- remove assumption that completion callbacks are invoked with locks held
- remove assumption that messages are passed in with locks held
- add read/write lock protecting just OSDMap and session map
- use this in place of global lock where possible
- replace global counters with atomics
- add new lock protecting non-osd operations (stuff for the mon)
- move ops, linger_ops, etc into per-session maps
- with luck, this will simplify the code that verifies the reply came back from the right now
- it will complicate somewhat the code the recalculates targets for ops bc they will move around betwen sessions.held
- add per-OSDSession r/w lock
- use this in place of the global lock when initiating operations
- allocate tid via an atomic inside the session lock
- do a readlock/unlock/writelock retry loop when the needed session is not open
- modify smalliobench or similar tool to submit requests via many threads