Ceph » CephFS

https://github.com/ceph/ceph/pull/21923

Xuehan Xu wrote:

In our online clusters, we encountered the bug #19593. Although we cherry-pick the fixing commits, the purge queue's journal is already damaged. When trying to repair the journal, we found that the journal's head has not been updated for a long time, which is caused by PurgeQueue::_consume() method always returning true. So we think it might be necessary for this method to return false when there were actually no purge queue item executed, even if the bug #19593 is fixed. After all, there could be other bugs that could damage the journal.

As we encountered bug #19593, the journal was damanged and the purge queue could not issue read as the journal is not readable, so the journaler's flush could not be triggered by the purge queue's read, and on the other hand, PurgeQueue:::_consume kept on returning true as the can_consume() method returned true, so the journaler's flush was never executed, so the journal's head was never updated.

Hi Xuehan,
I'm just curious about that how do you repair your purge queue journal ？

dongdong tao wrote:

Hi Xuehan,
I'm just curious about that how do you repair your purge queue journal ？

Actually, We have not repaired it yet. But we plan to rewrite the journal head manually, setting its expire_pos to the next journal entry beyond the damaged one and write_pos to the end of the journal. There are lots of journal entries beyond the damaged one, and we just can't afford to abandon them.

dongdong tao wrote:

Hi Xuehan,
I'm just curious about that how do you repair your purge queue journal ？

By the way, we also noticed your patch https://github.com/ceph/ceph/pull/19471 which enables cephfs-journal-tool to operate on purge queue journals. We plan to backport it to our code, and use it to rewrite the purge queue journal's head

Yeah, that‘s what i want to recommend to you, it can work as you expected.

dongdong tao wrote:

Yeah, that‘s what i want to recommend to you, it can work as you expected.

Thank you:-) That's very nice of you.