Ceph

The problem is init is done in the background (background_init) but shutdown is called synchronously from Mgr::handle_mgr_map. Shutdown should probably also be async, but i think we need some explicit states to manage this. Like activating[-needs-stop], active, stopping[-needs-start], stopped (or similar).

I looked at this one a bit more. I started trying to create states in Mgr and then MgrStandby for starting, active, stopping, standby, and then a target state, but it's a bit tedious to restructure it all, and even if we do do that, we can still get in a state where init() is blocked in some background async thread and so even though we want to shut down we are stuck in the 'starting' state until init completes (which might never happen if, say, the mons are down).

I think a better path would be to make Mgr state-driven instead of imperitive, so that init() does not block and background_init() can go away. Instead, it would have internal states for the various parts of its initialization, and at any point we can tell it to shut down and it will throw it all away and become deleteable.

Is that feasible, given that the python modules may also be written in a blocking fashion, or are we stuck in a world where mgr is doing blocking things and we have to wait for them?

/a/sage-2017-05-02_10:48:00-rados-wip-sage-testing---basic-smithi/1092427

/a/sage-2017-05-03_20:20:09-rados-wip-sage-testing---basic-smithi/1098337

void MgrStandby::shutdown()
{
  // Expect already to be locked as we're called from signal handler
  assert(lock.is_locked_by_me());

  if (active_mgr) {
    active_mgr->shutdown();
  }

  objecter->shutdown();

  timer.shutdown();

  monc->shutdown();
  client_messenger->shutdown();
}

2017-05-03 22:03:27.793623 7fe4aa7a7700  1 -- 172.21.15.164:0/781609493 <== mon.1 172.21.15.166:6789/0 6 ==== mgrmap(e 6) v1 ==== 74+0+0 (455897865 0 0) 0x7fe4b9992d00 con 0x7fe4b9e5e700
2017-05-03 22:02:33.789129 7fe4aa7a7700  4 mgr handle_mgr_map received map epoch 4
2017-05-03 22:02:33.789130 7fe4aa7a7700  4 mgr handle_mgr_map active in map: 0 active is 0
2017-05-03 22:02:33.789132 7fe4aa7a7700 -1 mgr handle_mgr_map I was active but no longer am
2017-05-03 22:02:33.789152 7fe4a649c700 10 mgr.server shutdown shutdown
2017-05-03 22:02:33.789160 7fe4a649c700  1 -- 172.21.15.164:6812/23102 mark_down_all
2017-05-03 22:02:33.789574 7fe4a649c700  1 -- 172.21.15.164:6812/23102 shutdown complete.
2017-05-03 22:02:33.789592 7fe4a649c700 10 mgr.server shutdown shutdown done

2017-05-03 22:02:34.480112 7fe4aa7a7700  1 -- 172.21.15.164:0/781609493 <== mon.7 172.21.15.166:6792/0 9 ==== mgrmap(e 5) v1 ==== 75+0+0 (455823694 0 0) 0x7fe4b99921c0 con 0x7fe4b9e5ef00
2017-05-03 22:02:34.480147 7fe4aa7a7700  4 mgr handle_mgr_map received map epoch 5?
2017-05-03 22:02:34.480150 7fe4aa7a7700  4 mgr handle_mgr_map active in map: 1 active is 4111
2017-05-03 22:02:34.480162 7fe4aa7a7700  1 mgr handle_mgr_map Activating!

2017-05-03 22:03:27.793623 7fe4aa7a7700  1 -- 172.21.15.164:0/781609493 <== mon.1 172.21.15.166:6789/0 6 ==== mgrmap(e 6) v1 ==== 74+0+0 (455897865 0 0) 0x7fe4b9992d00 con 0x7fe4b9e5e700
2017-05-03 22:03:27.793637 7fe4aa7a7700 -1 mgr handle_mgr_map I was active but no longer am