https://tracker.ceph.com/https://tracker.ceph.com/favicon.ico2016-09-23T10:39:14ZCeph Ceph - Bug #17386: Upgrading 0.94.6 -> 0.94.9 saturating mon node networkinghttps://tracker.ceph.com/issues/17386?journal_id=789132016-09-23T10:39:14ZKefu Chaitchaikov@gmail.com
<ul></ul><blockquote>
<p>It appears that starting with 0.94.7 that the osdmap encoding changed (which was unexpected by developers</p>
</blockquote>
<p>the CRC mismatch warning is expected:</p>
<p><code>pg_pool_t</code> is a field in <code>OSDMap::Incremental</code>, and <code>OSDMap</code> itself. in 0.94.6, <code>pg_pool_t</code> is encoded with v17 scheme, while in 0.94.9, this structure is encoded using v21. after upgrade, the monitors encode the (inc) osdmap using the new scheme, while OSD running 0.94.6 is still re-encoding the full osdmap using the v17, and then compare the crc of the re-encoded full map with the crc of the original fullmap encoded using v21. that's why the CRCs mismatch.</p>
<p>in a large cluster, resending the fullmap could be burden to monitor and saturates the cluster network. maybe we can have</p>
<ul>
<li>we do have the machinery to re-encode osdmap for old client. but we need to do this explicitly, i.e.
<ol>
<li>add CEPH_FEATURE_RESERVED (the non-exist feature bit) to the feature bits</li>
<li>encode the MOSDMap message in OSDMonitor::send_incremental() before sending it down to messenger, which will just put the pre-encoded incremental maps and full maps into the payload buffer. (downside: larger memory foot print)</li>
</ol>
</li>
<li>or, we can add an option to disable the crc checking (or full map upon CRC mismatch) on the OSD side. so we can disable it at run-time at seeing the performance degradation due to this problem. (downside: yet another knob)</li>
</ul> Ceph - Bug #17386: Upgrading 0.94.6 -> 0.94.9 saturating mon node networkinghttps://tracker.ceph.com/issues/17386?journal_id=789142016-09-23T10:39:32ZKefu Chaitchaikov@gmail.com
<ul><li><strong>Assignee</strong> set to <i>Kefu Chai</i></li></ul> Ceph - Bug #17386: Upgrading 0.94.6 -> 0.94.9 saturating mon node networkinghttps://tracker.ceph.com/issues/17386?journal_id=789152016-09-23T10:39:51ZKefu Chaitchaikov@gmail.com
<ul><li><strong>Target version</strong> deleted (<del><i>v0.94.8</i></del>)</li></ul> Ceph - Bug #17386: Upgrading 0.94.6 -> 0.94.9 saturating mon node networkinghttps://tracker.ceph.com/issues/17386?journal_id=789272016-09-23T22:07:12ZKen Dreyerkdreyer@redhat.com
<ul><li><strong>Backport</strong> set to <i>jewel, hammer</i></li></ul> Ceph - Bug #17386: Upgrading 0.94.6 -> 0.94.9 saturating mon node networkinghttps://tracker.ceph.com/issues/17386?journal_id=789392016-09-26T04:17:45ZKefu Chaitchaikov@gmail.com
<ul></ul><blockquote>
<p>option 1: re-encode the MOSDMap message if GMT_HITSET feature bit is missing.</p>
</blockquote>
<p>- downside</p>
<ul>
<li>larger memory foot print</li>
<li>waste CPU cycles on re-encoding the OSDMap again and again for the mon clients without GMT_HITSET.</li>
<li>the assert for detecting the caller of <code>Incremental::encode()</code> is disabled. this is just a precaution, but would be good to have it.<br /><pre>
// only a select set of callers should *ever* be encoding new
// OSDMaps. others should be passing around the canonical encoded
// buffers from on high. select out those callers by passing in an
// "impossible" feature bit.
// assert(features & CEPH_FEATURE_RESERVED);
</pre></li>
</ul>
<blockquote>
<p>option 2: we can add an option to disable the crc checking (or full map upon CRC mismatch) on the OSD side. so we can disable it at run-time at seeing the performance degradation due to this problem. (downside: yet another knob)</p>
</blockquote>
<p>- downside: it requires user to a hotfix of OSD just for upgrading the mon or OSD, to prevent the old version of OSDs from asking for full osdmaps.</p>
<p>i think this does not make sense, and is not a viable solution. so guess the fix on monitor side is better.</p> Ceph - Bug #17386: Upgrading 0.94.6 -> 0.94.9 saturating mon node networkinghttps://tracker.ceph.com/issues/17386?journal_id=790052016-09-27T15:22:03ZKefu Chaitchaikov@gmail.com
<ul></ul><p>so we need three patches for fixing this problem:</p>
<ol>
<li>for this specific issue. we need to re-encode the OSDMap when sending it to client if the GMT_HITSET feature bit is missing. but this fix is not supposed to be merged into master</li>
<li>add an option on the OSD side, so it won't ask for the full map upon CRC mismatch, and also document the suggested steps for upgrade from 0.94.6 to 0.94.6+: upgrade the OSD (and other monitor clients) side first with this option enabled, <strong>then</strong> upgrade the monitor.</li>
<li>add an option on the OSD side, so it will ask for the latest <strong>full</strong> map upon boot. so the osdmaps won't diverge over time.</li>
</ol> Ceph - Bug #17386: Upgrading 0.94.6 -> 0.94.9 saturating mon node networkinghttps://tracker.ceph.com/issues/17386?journal_id=790402016-09-29T09:22:07ZKefu Chaitchaikov@gmail.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Fix Under Review</i></li></ul><p><a class="external" href="https://github.com/ceph/ceph/pull/11258">https://github.com/ceph/ceph/pull/11258</a><br /><a class="external" href="https://github.com/ceph/ceph/pull/11284">https://github.com/ceph/ceph/pull/11284</a><br /><a class="external" href="https://github.com/ceph/ceph/pull/11596">https://github.com/ceph/ceph/pull/11596</a></p> Ceph - Bug #17386: Upgrading 0.94.6 -> 0.94.9 saturating mon node networkinghttps://tracker.ceph.com/issues/17386?journal_id=793872016-10-08T14:17:05ZLoïc Dacharyloic@dachary.org
<ul><li><strong>Copied to</strong> <i><a class="issue tracker-9 status-3 priority-5 priority-high3 closed" href="/issues/17534">Backport #17534</a>: hammer: doc: document the changed upgrade steps for hammer</i> added</li></ul> Ceph - Bug #17386: Upgrading 0.94.6 -> 0.94.9 saturating mon node networkinghttps://tracker.ceph.com/issues/17386?journal_id=799652016-10-18T13:05:18ZNathan Cutlerncutler@suse.cz
<ul></ul><p>Kefu, <a class="external" href="https://github.com/ceph/ceph/pull/11284">https://github.com/ceph/ceph/pull/11284</a> should be backported to jewel only, correct? I.e. not to hammer.</p> Ceph - Bug #17386: Upgrading 0.94.6 -> 0.94.9 saturating mon node networkinghttps://tracker.ceph.com/issues/17386?journal_id=802552016-10-24T10:13:37ZKefu Chaitchaikov@gmail.com
<ul></ul><p>@Nathan</p>
<p>sorry for the latency, right. for clarification:</p>
for jewel
<ul>
<li><del><a class="external" href="https://github.com/ceph/ceph/pull/11258">https://github.com/ceph/ceph/pull/11258</a></del> // not needed anymore.</li>
<li><a class="external" href="https://github.com/ceph/ceph/pull/11284">https://github.com/ceph/ceph/pull/11284</a></li>
<li><a class="external" href="https://github.com/ceph/ceph/pull/11596">https://github.com/ceph/ceph/pull/11596</a></li>
<li><a class="external" href="https://github.com/ceph/ceph/pull/11180">https://github.com/ceph/ceph/pull/11180</a> // this is tracked by <a class="issue tracker-1 status-3 priority-6 priority-high2 closed" title="Bug: mon: forwarded message is encoded with sending client's features (Resolved)" href="https://tracker.ceph.com/issues/17365">#17365</a>, which will also be backported to jewel. as commented by sage as below, it is a prereq for PR#11610.</li>
<li><a class="external" href="https://github.com/ceph/ceph/pull/11610">https://github.com/ceph/ceph/pull/11610</a></li>
</ul>
for hammer
<ul>
<li><a class="external" href="https://github.com/ceph/ceph/pull/11372">https://github.com/ceph/ceph/pull/11372</a></li>
<li><del><a class="external" href="https://github.com/ceph/ceph/pull/11258">https://github.com/ceph/ceph/pull/11258</a></del></li>
</ul> Ceph - Bug #17386: Upgrading 0.94.6 -> 0.94.9 saturating mon node networkinghttps://tracker.ceph.com/issues/17386?journal_id=802562016-10-24T10:25:02ZSage Weilsage@newdream.net
<ul></ul><p>note that d4f5e88f36e5388ae9e062c4bc49ac1c684a3f3c is a prereq for <a class="external" href="https://github.com/ceph/ceph/pull/11610">https://github.com/ceph/ceph/pull/11610</a></p> Ceph - Bug #17386: Upgrading 0.94.6 -> 0.94.9 saturating mon node networkinghttps://tracker.ceph.com/issues/17386?journal_id=802582016-10-24T12:10:12ZKefu Chaitchaikov@gmail.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-1 status-3 priority-6 priority-high2 closed" href="/issues/17365">Bug #17365</a>: mon: forwarded message is encoded with sending client's features</i> added</li></ul> Ceph - Bug #17386: Upgrading 0.94.6 -> 0.94.9 saturating mon node networkinghttps://tracker.ceph.com/issues/17386?journal_id=804992016-10-28T12:47:56ZKefu Chaitchaikov@gmail.com
<ul><li><strong>Status</strong> changed from <i>Fix Under Review</i> to <i>Pending Backport</i></li></ul> Ceph - Bug #17386: Upgrading 0.94.6 -> 0.94.9 saturating mon node networkinghttps://tracker.ceph.com/issues/17386?journal_id=805042016-10-28T12:52:59ZLoïc Dacharyloic@dachary.org
<ul><li><strong>Copied to</strong> <i><a class="issue tracker-9 status-3 priority-4 priority-default closed" href="/issues/17734">Backport #17734</a>: jewel: Upgrading 0.94.6 -> 0.94.9 saturating mon node networking</i> added</li></ul> Ceph - Bug #17386: Upgrading 0.94.6 -> 0.94.9 saturating mon node networkinghttps://tracker.ceph.com/issues/17386?journal_id=805382016-10-31T13:20:16ZLoïc Dacharyloic@dachary.org
<ul></ul><p><a class="external" href="https://github.com/ceph/ceph/pull/11284/files">https://github.com/ceph/ceph/pull/11284/files</a> needs the matching ceph-qa-suite commit <a class="external" href="https://github.com/ceph/ceph-qa-suite/commit/322363a41741d212dd82c02aec148647e447d055">https://github.com/ceph/ceph-qa-suite/commit/322363a41741d212dd82c02aec148647e447d055</a></p> Ceph - Bug #17386: Upgrading 0.94.6 -> 0.94.9 saturating mon node networkinghttps://tracker.ceph.com/issues/17386?journal_id=810512016-11-10T11:14:14ZLoïc Dacharyloic@dachary.org
<ul><li><strong>Status</strong> changed from <i>Pending Backport</i> to <i>Resolved</i></li></ul> Ceph - Bug #17386: Upgrading 0.94.6 -> 0.94.9 saturating mon node networkinghttps://tracker.ceph.com/issues/17386?journal_id=2496982023-11-06T19:03:40ZRadoslaw Zarzynskirzarzyns@redhat.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-1 status-14 priority-5 priority-high3" href="/issues/63389">Bug #63389</a>: Failed to encode map X with expected CRC</i> added</li></ul>