Ceph : Issueshttps://tracker.ceph.com/https://tracker.ceph.com/favicon.ico2017-06-28T04:48:22ZCeph
Redmine Ceph - Backport #20443 (Resolved): kraken: osd: client IOPS drops to zero frequentlyhttps://tracker.ceph.com/issues/204432017-06-28T04:48:22ZAlexey Sheplyakovasheplyakov@mirantis.com
<p><a class="external" href="https://github.com/ceph/ceph/pull/15962">https://github.com/ceph/ceph/pull/15962</a></p> Ceph - Backport #20428 (Resolved): jewel: osd: client IOPS drops to zero frequentlyhttps://tracker.ceph.com/issues/204282017-06-27T11:13:38ZAlexey Sheplyakovasheplyakov@mirantis.com
<p><a class="external" href="https://github.com/ceph/ceph/pull/15947">https://github.com/ceph/ceph/pull/15947</a></p> Ceph - Bug #20427 (Resolved): osd: client IOPS drops to zero frequentlyhttps://tracker.ceph.com/issues/204272017-06-27T10:17:03ZAlexey Sheplyakovasheplyakov@mirantis.com
<p>[From <a class="external" href="http://www.spinics.net/lists/ceph-devel/msg37163.html">http://www.spinics.net/lists/ceph-devel/msg37163.html</a>]</p>
<p>At Alibaba, we experienced unstable performance with Jewel on one<br />production cluster, and we can easily reproduce it now with several<br />small test clusters. One test cluster has 30 SSDs, and another test<br />one has 120 SSDs, we are using filestore+async messenger on the<br />backend and fio+librbd to test them. When this issue happens, client<br />fio IOPS drops to zero (or close to zero) frequently during fio runs.<br />And the durations of those drops were very short, about 1 second or<br />so.</p>
<p>For the 30 SSDs test cluster, we use 135 client fio writing into 135<br />rbd images individually, each fio has only 1 job and rate limit is<br />3MB/s. On this fresh created test cluster, for all 135 client fio<br />runs, during first 15 minutes or so, client IOPS were very stable and<br />each OSD server's throughput was very stable as well. After 15 minutes<br />and 360 GB data written, the test cluster entered an unstable state,<br />client fio IOPS dropped to zero (or close) frequently and each OSD<br />server's throughput became very spiky as well (from 500MB/s to less<br />1MB/s). We tried let all fio keeping writing for about 16 hours,<br />cluster was still in this swing state.</p>
<p>This is very easily reproducible. I don't think it's caused by<br />filestore folder splitting, since they were all done during the first<br />15 minutes. And also, OSD server mem/cpu/disk were far from saturated.<br />One thing we noticed from perf counter is that op_latency increased<br />from 0.7 ms to >20 ms after entering this unstable state. Is this<br />normal Jewel/filestore behavior? Anyone knows what causes it?</p> rbd - Backport #19957 (Resolved): jewel: rbd: Lock release requests not honored after watch is re...https://tracker.ceph.com/issues/199572017-05-17T09:14:43ZAlexey Sheplyakovasheplyakov@mirantis.com
<p><a class="external" href="https://github.com/ceph/ceph/pull/17385">https://github.com/ceph/ceph/pull/17385</a></p> Ceph - Backport #19928 (Resolved): kraken: mon crash on shutdown, lease_ack_timeout eventhttps://tracker.ceph.com/issues/199282017-05-15T09:32:40ZAlexey Sheplyakovasheplyakov@mirantis.com
<p><a class="external" href="https://github.com/ceph/ceph/pull/15084">https://github.com/ceph/ceph/pull/15084</a></p> Ceph - Backport #19926 (Resolved): jewel: mon crash on shutdown, lease_ack_timeout eventhttps://tracker.ceph.com/issues/199262017-05-15T08:16:42ZAlexey Sheplyakovasheplyakov@mirantis.com
<p><a class="external" href="https://github.com/ceph/ceph/pull/15083">https://github.com/ceph/ceph/pull/15083</a></p> Ceph - Backport #19323 (Rejected): hammer: segfault in FileStore::fiemap()https://tracker.ceph.com/issues/193232017-03-21T12:18:47ZAlexey Sheplyakovasheplyakov@mirantis.com
<p><a class="external" href="https://github.com/ceph/ceph/pull/14069">https://github.com/ceph/ceph/pull/14069</a></p> rgw - Backport #19322 (Resolved): kraken: multisite: possible infinite loop in RGWFetchAllMetaCRhttps://tracker.ceph.com/issues/193222017-03-21T11:00:41ZAlexey Sheplyakovasheplyakov@mirantis.com
<p><a class="external" href="https://github.com/ceph/ceph/pull/14067">https://github.com/ceph/ceph/pull/14067</a></p> rgw - Backport #19321 (Resolved): jewel: multisite: possible infinite loop in RGWFetchAllMetaCRhttps://tracker.ceph.com/issues/193212017-03-21T10:42:15ZAlexey Sheplyakovasheplyakov@mirantis.com
<p><a class="external" href="http://tracker.ceph.com/issues/17655">http://tracker.ceph.com/issues/17655</a></p> rgw - Backport #19115 (Resolved): jewel: rgw_file: ensure valid_s3_object_name for directorieshttps://tracker.ceph.com/issues/191152017-03-01T07:03:14ZAlexey Sheplyakovasheplyakov@mirantis.com
<p><a class="external" href="https://github.com/ceph/ceph/pull/13717">https://github.com/ceph/ceph/pull/13717</a></p> rgw - Backport #18827 (Resolved): jewel: RGW leaking datahttps://tracker.ceph.com/issues/188272017-02-06T08:17:28ZAlexey Sheplyakovasheplyakov@mirantis.com
<p><a class="external" href="https://github.com/ceph/ceph/pull/13358">https://github.com/ceph/ceph/pull/13358</a></p> Ceph - Backport #18729 (Resolved): jewel: ceph-disk: error on _bytes2strhttps://tracker.ceph.com/issues/187292017-01-30T12:00:48ZAlexey Sheplyakovasheplyakov@mirantis.com
<p><a class="external" href="https://github.com/ceph/ceph/pull/13187">https://github.com/ceph/ceph/pull/13187</a></p> Ceph - Backport #18581 (Rejected): jewel: osd: ENOENT on clonehttps://tracker.ceph.com/issues/185812017-01-18T11:10:46ZAlexey Sheplyakovasheplyakov@mirantis.com
<p><a class="external" href="https://github.com/ceph/ceph/pull/12978">https://github.com/ceph/ceph/pull/12978</a></p> rgw - Backport #17908 (Resolved): jewel: rgw: the response element "X-Timestamp" of swift stat bu...https://tracker.ceph.com/issues/179082016-11-15T09:23:37ZAlexey Sheplyakovasheplyakov@mirantis.com
<p><a class="external" href="https://github.com/ceph/ceph/pull/11990">https://github.com/ceph/ceph/pull/11990</a></p> sepia - Bug #17274 (Resolved): Access to sepia lab: Alexey Sheplyakovhttps://tracker.ceph.com/issues/172742016-09-14T07:52:26ZAlexey Sheplyakovasheplyakov@mirantis.com
<p>1. I don't intend to run jobs manually<br />2. Username: asheplyakov<br />3. ssh key is attached (id_rsa.pub)<br />4. Hashed VPN password: <br /><a class="email" href="mailto:asheplyakov@asheplyakov.srt.mirantis.net">asheplyakov@asheplyakov.srt.mirantis.net</a> wFW0ZgT4cNhKRAGXiUtevQ 1b11f0702b2db42a42aae6579737ece2caad3b80a8186b971686575cb76b3051</p>