Ceph

3849ce2fa428dcfab57d6a7fdde1870f1448a33f I think

Thanks for looking at this!

In terms of folder splitting, I am wondering if people are interested in an alternative solution, that is to pre-create all sub-folders (e.g. 5 level, 6 level) when provisioning a new cluster after estimating the number of filers that would be put into a pool, With use of this approach, we can avoid the (unplanned) folder splitting which could brings long tail latency (even after we fixed the lock contention issue as we still need to protect the operation atomicity).

Hmm, the problem with that is:
1) people are really bad at estimating how much stuff they're going to put into a storage system, and if they're wrong it'll be expensive.
2) You can remove the cost of the splits from runtime, but it's a tradeoff because you're adding a bunch of extra metadata the OSDs need to cache/lookup for the entire lifetime of the cluster whenever they do an IO.

It might be worth exploring, but I don't know that's worth it until we've got more data on when we should actually be doing the splits to begin with.

Greg Farnum wrote:

Hmm, the problem with that is:
1) people are really bad at estimating how much stuff they're going to put into a storage system, and if they're wrong it'll be expensive.
2) You can remove the cost of the splits from runtime, but it's a tradeoff because you're adding a bunch of extra metadata the OSDs need to cache/lookup for the entire lifetime of the cluster whenever they do an IO.

It might be worth exploring, but I don't know that's worth it until we've got more data on when we should actually be doing the splits to begin with.

After checking the fix, I think there is another potential improvement over the splitting, that is to only trigger the splitting from within a background thread (e.g. filestore op thread) but not from within the op thread which is in the critical path, as if the two op threads accidentally trigger splitting at the same time (which is likely to happen as objects are evenly distributed across different folders), the entire OSD would block a couple of seconds.

I see Guang's concern is that the all threads within a thread pool might be working on splitting, as most of the folders at the same level are all close to split at the same time. Perhaps we can stagger the folder splits by its PG name/replica id/folder name.

Another concern is that multiple threads might be splitting a same folder and may cause reentry problem. But I am not sure whether it is a real problem.

Guang Yang wrote:

After checking the fix, I think there is another potential improvement over the splitting, that is to only trigger the splitting from within a background thread (e.g. filestore op thread) but not from within the op thread which is in the critical path, as if the two op threads accidentally trigger splitting at the same time (which is likely to happen as objects are evenly distributed across different folders), the entire OSD would block a couple of seconds.

That I find to be a pretty unlikely problem — random balances aren't going to result in that happening very often. Maybe if we observe it happening in practice we can investigate solutions (a background thread won't work without being very complicated, we need strong consistency about where things are located). Somehow staggering it would work better (and we already serialize based on the PG, so we don't need to worry about colliding splits if we do it in-line).

> That I find to be a pretty unlikely problem — random balances aren't going to result in that happening very often. Maybe if we observe it happening in practice we can investigate solutions (a background thread won't work without being very complicated, we need strong consistency about where things are located). Somehow staggering it would work better (and we already serialize based on the PG, so we don't need to worry about colliding splits if we do it in-line).

Thanks Greg! We can port the fix and test against it on the cluster which is trying to do splitting (from level 5 to level 6).

By triggering it from within a background thread (filestore thread), I mean FileStore::lfn_open could be called either from an op thread (e.g. from read op) or from a filestore thread. And if it is called from a op thread which happens to trigger the splitting, that op thread would work on the splitting stuff for a couple of seconds (6 seconds per our testing with 320 files to split), the throughput of the OSD drops a half during the time period, even worse if two op threads hang the same way, entire OSD could block for a couple of seconds.

An alternative is to only trigger (and do) the splitting in the background filestore thread which does not block op anyway (is there any other concern?).

After checking the code, one potential small change to achieve this is, check the #create# parameter within FileStore::lfn_open and only trigger folder splitting when #create# is true.

Well, we already only do folder split/merge when creating or removing objects, which isn't going to happen on a read. ;) I hadn't considered that doing the split would block other ops on different PGs from progressing; I don't remember how the internal interfaces are constructed for the work queues but it's possible we could stick anything doing a split onto a callback and release the op thread to work on other operations.

Greg Farnum wrote:

Well, we already only do folder split/merge when creating or removing objects,

Oh right, it checks #create# parameter within the function (sorry missing it) before checking if it is good time to do splitting. So that I assume the splitting will never happen in an op thread (am I right?) but only filestore thread?

Another change I would suggest is to add some logs before and after folder splitting so that it is easier to check when it happens and how long does it take. Does that make sense?

Which op threads and filestore threads are you talking about? The splitting happens whenever the filestore actually does the create that triggers it.

Greg Farnum wrote:

Which op threads and filestore threads are you talking about? The splitting happens whenever the filestore actually does the create that triggers it.

By #op thread#, I mean the thread handling messages (opWQ), by filestore thread, I mean the filestore op thread (filestoreWQ).

That is correct, I was wrong in terms of that such path (folder splitting) could be triggered and done within a op thread which potential blocks other message processing. As long as it is only happen from within a background filestore thread, we can go ahead to test the fix.

Thanks Greg!

Hi Greg,
I got a chance to test against the fix in our cluster (the leaf folder has around 1000 files), there are two observations:
1) Disk saturation when folder started splitting, the util is 100 and it could last minutes.
2) Start seeing client message throttle.
3) Latency increasing (up to tens of seconds).

Things could be worse if we increase the max objects per folder config.

Hello Greg,
Can we backport this fix to dumpling?

As a performance patch that only impacts splitting, I don't think this is really appropriate for a backport.