Ceph

The bug description claims that cluster is running v0.61.5 but attached log says v0.61.7. Could there be a mix of nodes?

I haven't yet been able to reproduce with all machines running v0.61.7.

I was wrong - we are indeed on 0.61.7

root@ineri ~$ ndo all_nodes ceph --version
h0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
h1 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
h2 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
h3 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
h4 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
h5 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
s0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
s1 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
s2 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
s4 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)

This bug has not resurfaced (luckily) and I'm not particularly keen on trying to reproduce it :)

not sure what the best way forward is at this point.

/jc

There is a race already fixed in a later release by (01d3e094) which could allow start_recovery_ops() to be called with a negative value for "max" arg. The way to see this assert is for recovery to be required on one or more replicas but not the primary. In that case no replica operations would be started and the code would attempt to transition to Recovered state.

I think we should add a call to needs_recovery() before assuming that recovery must be done. There are other possible obscure error paths that could result in this assert. That change could be backported too.