https://tracker.ceph.com/https://tracker.ceph.com/favicon.ico2022-07-19T20:55:57ZCeph bluestore - Bug #56640: RGW S3 workload has a huge performance boost in quincy 17.2.0 as compared to 17.2.1https://tracker.ceph.com/issues/56640?journal_id=2207852022-07-19T20:55:57ZVikhyat Umrao
<ul></ul><p>This has been already verified that by default COSbench uses random not zero.</p>
<p>COSBench document - page 50 - <a class="external" href="https://raw.githubusercontent.com/intel-cloud/cosbench/master/COSBenchUserGuide.pdf">https://raw.githubusercontent.com/intel-cloud/cosbench/master/COSBenchUserGuide.pdf</a><br />content | String | “random”(default) ”zero” | Fill object content with random data or all-zeros</p> bluestore - Bug #56640: RGW S3 workload has a huge performance boost in quincy 17.2.0 as compared to 17.2.1https://tracker.ceph.com/issues/56640?journal_id=2207862022-07-19T21:06:08ZVikhyat Umrao
<ul></ul><p>The `bluestore_zero_block_detection` was set to false in 17.2.1. For more details please check:</p>
<p><a class="external" href="https://tracker.ceph.com/issues/55521">https://tracker.ceph.com/issues/55521</a><br /><a class="external" href="https://github.com/ceph/ceph/pull/46193/files">https://github.com/ceph/ceph/pull/46193/files</a></p>
<p>Release notes:</p>
<blockquote>
<p>=17.2.1</p>
</blockquote>
<ul>
<li>The "BlueStore zero block detection" feature (first introduced to Quincy in<br /><a class="external" href="https://github.com/ceph/ceph/pull/43337">https://github.com/ceph/ceph/pull/43337</a>) has been turned off by default with a<br />new global configuration called `bluestore_zero_block_detection`. This feature,<br />intended for large-scale synthetic testing, does not interact well with some RBD<br />and CephFS features. Any side effects experienced in previous Quincy versions<br />would no longer occur, provided that the configuration remains set to false.<br />Relevant tracker: <a class="external" href="https://tracker.ceph.com/issues/55521">https://tracker.ceph.com/issues/55521</a></li>
</ul> bluestore - Bug #56640: RGW S3 workload has a huge performance boost in quincy 17.2.0 as compared to 17.2.1https://tracker.ceph.com/issues/56640?journal_id=2207872022-07-19T21:07:28ZVikhyat Umrao
<ul></ul><p>Vikhyat Umrao wrote:</p>
<blockquote>
<p>This has been already verified that by default COSbench uses random not zero.</p>
<p>COSBench document - page 50 - <a class="external" href="https://raw.githubusercontent.com/intel-cloud/cosbench/master/COSBenchUserGuide.pdf">https://raw.githubusercontent.com/intel-cloud/cosbench/master/COSBenchUserGuide.pdf</a><br />content | String | “random”(default) ”zero” | Fill object content with random data or all-zeros</p>
</blockquote>
<p>and in our testing results mentioned in this attached doc do not use zero, it by default uses random.</p> bluestore - Bug #56640: RGW S3 workload has a huge performance boost in quincy 17.2.0 as compared to 17.2.1https://tracker.ceph.com/issues/56640?journal_id=2208662022-07-20T16:58:09ZVikhyat Umrao
<ul></ul><p>There are two test cases that would be executed to find more details on what is going on with this feature and how it is giving this much perf boost in 17.2.0 and in 17.2.1(when the feature is enabled).</p>
<p>Cluster 1 - Testcase 1 - bluestore_zero_block_detection=false - Default - no need to set</p>
<p>- Deploy 17.2.1 cluster <br />- Do a fill workload to fill the cluster</p>
<p>- Enable debug_bluestore 20<br /> ceph config set osd debug_bluestore 20</p>
<p>- Do a 1-hr hybrid with 10 minutes interval for osd perf dump</p>
<p>Cluster 2 - Testcase 2 - bluestore_zero_block_detection=true non-Default - need to set</p>
<p>- Deploy 17.2.1 cluster with bluestore_zero_block_detection=true<br /> ceph config set osd bluestore_zero_block_detection true</p>
<p>- Do a fill workload to fill the cluster<br />- Enable debug_bluestore 20<br /> ceph config set osd debug_bluestore 20</p>
<p>- Do a 1-hr hybrid with 10 minutes interval for osd perf dump</p> bluestore - Bug #56640: RGW S3 workload has a huge performance boost in quincy 17.2.0 as compared to 17.2.1https://tracker.ceph.com/issues/56640?journal_id=2208882022-07-20T18:19:19ZVikhyat Umrao
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/220888/diff?detail_id=234015">diff</a>)</li></ul> bluestore - Bug #56640: RGW S3 workload has a huge performance boost in quincy 17.2.0 as compared to 17.2.1https://tracker.ceph.com/issues/56640?journal_id=2208892022-07-20T18:20:24ZVikhyat Umrao
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/220889/diff?detail_id=234016">diff</a>)</li></ul> bluestore - Bug #56640: RGW S3 workload has a huge performance boost in quincy 17.2.0 as compared to 17.2.1https://tracker.ceph.com/issues/56640?journal_id=2218262022-07-28T16:18:40ZTim Wilkinsontwilkins@redhat.com
<ul></ul><p>Two rounds of testing have occurred, one with osd debug_bluestore=20 but this one did not reproduce the perf boost seen in hybrid-1hr jobs w/BSZBD enabled. Another test without BSZBD did reproduce the boost. Both sets of ceph logs were copied to /root on all players.</p>
<pre><code>Testing w/debug 20: /root/ceph_220722-1609<br /> Testing w/out debug 20: /root/220725-1513_{sa,ceph}</code></pre>
<p>The clusters have since been redeployed.</p> bluestore - Bug #56640: RGW S3 workload has a huge performance boost in quincy 17.2.0 as compared to 17.2.1https://tracker.ceph.com/issues/56640?journal_id=2218272022-07-28T16:20:36ZTim Wilkinsontwilkins@redhat.com
<ul></ul><p>Correction, each test had site1 w/out BSZBD and the other with it enabled. The second round of tests were the same but without osd debug_bluestore=20.</p>