Ceph » mgr » Dashboard

Here are some observation:

• Cherrypy spawns a 10-threads pool by default to serve requests, shared resources between threads should be handle carefully (with locks).
• There is a shared resource between backend threads: `GaneshaConf.exports`. For each new request, current existing exports are loaded from RADOS objects. If thread A is manipulating exports and thread B is spawned to serve another request, the exports that thread B sees might not be reflected yet, which might cause problems.
• (Guesswork) Problems might be seen more easily when a system is under heavy workloads because loading and saving exports/configs involved I/O to the RADOS.

Here are some race condition examples in the implementation (can be artificially created):

• Creating multiple exports simultaneously
  Say we have no exports and two create requests are received. Two threads are spawned and each one sees no existing exports. For the export creation, each thread then generates a new export ID 1 for the new export and the final result might be that there is only one export is created (Both writes to `cephfs_data/EXPORT-1`)
• Deleting export and creating at the same time
  If the deleting takes longer, the result might be that new export is created, but daemon doesn't reference it because the delete call erases the reference.