Ceph

In the new implementation Sam set up, and apparently in the old code as well, a notify can succeed without getting a response from one of the watchers — and the watcher doesn't get told that it failed to respond in time, and it doesn't get disconnected, and the notifier doesn't get told that somebody timed out.

This means that watch-notify actually provides guarantees consisting of...pretty much nothing.

I suspect that notify needs to lose the ability to provide timeouts shorter than the watch timeout, and when a watch fails the client needs to get a disconnect or something. Or else allow notifies to provide an attack space, and to force-disconnect (or force-fail the Watch) on a client which doesn't ack a Notify as quickly as requested.