OSD: do not do PG creates in-line with incoming request
We got a report that splitting a pool from 8 PGs to 2040 PGs caused a number of OSDs in the cluster to go down temporarily. I checked out the code and believe I've identified the issue:
OSD::handle_pg_create() will synchronously, under the OSD lock, create every PG that is specified in a single message.
It does so by doing whatever lookups are necessary (I'm not sure how much this is) and creating a single transaction that does all the PG creation steps, then dispatching it for execution. In the case above, that meant 254 PG creates within a single transaction, which I think is a bit large. We should avoid doing the work in uninterruptible chunks and split it into reasonable sizes for processing, instead.