Ceph

It won't be in dumpling v0.67.12 but ... it could be in v0.80.10 ;-) It looks like an important fix.

As far as I understand this hurts snapshots. I'm on Firefly and getting bitten by this. Is there a work-around to get back to a usable snapshot state once this has kicked in?

A workaround is to detach and reattach your images. This reopens them and reestablishes the watch.

Ah. So in that case restarting Qemu would be the specific action for that, right?

I'm currently trying this out. I'm a bit unclear on the specifics of the trigger. We have some automation code that causes pg_num and pgp_num to be automatically (slowly) adjusted for growing pools.

However, this was running for a while without the cluster exhibiting the issue in a way that we would notice. The specific point when we noticed was when we updated our tunables to the recommended settings for Firefly (and caused a large CRUSH rearrangement).

Would you think that in running operations stopping our automatic pg_num, pgp_num adaption would be sufficient to avoid this bug?

For further clarification: does this bug apply on a per-pool basis, per-image basis or cluster-wide? My guess would be this applies on a per-pool basis.

Thanks for the hint!

Ok, so I restarted one of the VMs exiting Qemu and starting afresh. Took a snapshot immediately after that and it's been giving a consistent hash of the mapped rbd device multiple times after that.

Yes, restarting qemu will fix it. The trigger for the issue is pg split, so it would only affect pools where you had increased pg_num and pgp_num. If you avoid splitting, you avoid this bug. Other crush changes like straw2 or new tunables should not cause this issue.

That's a relief! Thanks for the explanation, I hope other people stumbling over this bug will find this helpful, too. :)