Ceph » RADOS

Sage was right, the MDSMonitor unconditionally calls do_application_enable() and that unconditionally sets application metadata on the pending_inc's pool.

So if this is done prior to setting the luminous flags, that might be able to prompt a divergence in what's encoded where?

We're encoding with the quorum features, though, so I don't think that could actually cause a problem, Maybe though.

The bug I hit before was doing the right checks on encoding, but the pending_inc was applied to the in-memory mon copy. So the in-memory osdmap had the fields populated but didn't encode them. then later when the flag was set we would get a crc mismatch because suddenly mon encoded fields that were populated with values but the osds didn't get that when they applied their incrementals.

It sounds like this is a somewhat different problem, but I suspect the root cause is the same: we shouldn't populate pending_inc app metadata unless require_osd_release>=luminous is set.

Hmm, this still doesn't make sense. The cluster started out as luminous and so the maps would always have the luminous flags set. (And indeed, on the first dump it's got "require_osd_release":"luminous".)

Okay, unlike the previous log I looked at, the "fs new" command is clearly not triggering a new osd map commit. We run do_application_enable against osdmap epoch 19:

2017-08-09 06:51:53.748483 7f6980098700  7 mon.a@0(leader).mds e1 prepare_update mon_command({"prefix": "fs new", "data": "cephfs_data", "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1
2017-08-09 06:51:53.748505 7f6980098700 20 mon.a@0(leader).osd e19 do_application_enable: pool_id=3, app_name=cephfs
2017-08-09 06:51:53.748513 7f6980098700 20 mon.a@0(leader).osd e19 do_application_enable: pool_id=2, app_name=cephfs

2017-08-09 07:13:54.959684 7f6980098700 10 mon.a@0(leader).osd e19 should_propose

As I reported in #20891 I am seeing this on fresh luminous clusters.

https://github.com/ceph/ceph/pull/16954