Ceph

h1. Libradosobjecter - improve threading

h3. Summary

The Objecter and librados currently use a single global lock for all state.  Break this down to improve multithread performance.

h3. Owners

* Sage Weil (Inktank)

* Name (Affiliation)

* Name

h3. Interested Parties

* Somnath Ray (SanDisk)

* Haomai Wang (UnitedStack)

* Guang Yang (Yahoo!)

h3. Current Status

* Objecter uses a single lock for all state.

* Objecter doesn't own the lock; it is provided by the caller/owner (e.g., librados's lock, or MDS's lock, or Client's lock)

* Objecter has two basic categories of state:

** OSDMap state

** per-OSD session state

*** list of requests, Connection*, etc.

h3. Detailed Description

h3. Work items

h4. Coding tasks

# objecter: take over ownership of lock

## do not take caller's lock

## remove assumption that completion callbacks are invoked with locks held

## remove assumption that messages are passed in with locks held

# add read/write lock protecting just OSDMap and session map

## use this in place of global lock where possible

# replace global counters with atomics

# add new lock protecting non-osd operations (stuff for the mon)

# move ops, linger_ops, etc into per-session maps

## with luck, this will simplify the code that verifies the reply came back from the right now

## it will complicate somewhat the code the recalculates targets for ops bc they will move around betwen sessions.held 

# add per-OSDSession r/w lock

## use this in place of the global lock when initiating operations

## allocate tid via an atomic inside the session lock

## do a readlock/unlock/writelock retry loop when the needed session is not open

# modify smalliobench or similar tool to submit requests via many threads