Ceph

Wait, 'force_create_pg' cannot recreate incomplete PGs? If that, could you please let me know how to deal with the incomplete PGs?

Probably ceph_objectstore_tool --[unsafe?]-mark-complete <pgid> on a down osd.

Definitely need to set:
info.last_epoch_started to current
info.history.last_epoch_started current
info.history.last_epoch_clean to current
info.last_backfill = MAX

I have this exact problem on my 0.94.1.2 cluster. It's production but the data is mostly-replaceable, so I'm willing to test patches if you need testers. ceph_objectstore_tool --mark-complete <pgid> gets my vote :)

I have a patch for ceph-objectstore-tool, which adds mark-complete operation:

https://github.com/ceph/ceph/pull/5031

Aaron, you might want to try it on your own risk.

Mykola Golub wrote:

I have a patch for ceph-objectstore-tool, which adds mark-complete operation:

https://github.com/ceph/ceph/pull/5031

Aaron, you might want to try it on your own risk.

Mykola,

Thank you for the opportunity to help test -- unfortunately, we just finished rebuilding our cluster not 24 hours before your comment. We're running 9.0.1 now and playing with erasure coding :) Hopefully no more incomplete PGs will occur, but if they do, we'll be more than happy to test your patch.

-Aaron