https://tracker.ceph.com/
https://tracker.ceph.com/favicon.ico
2018-01-29T14:31:00Z
Ceph
rgw - Bug #22804: multisite Synchronization failed when read and write delete at the same time
https://tracker.ceph.com/issues/22804?journal_id=106116
2018-01-29T14:31:00Z
John Spray
jcspray@gmail.com
<ul><li><strong>Project</strong> changed from <i>Ceph</i> to <i>rgw</i></li><li><strong>Description</strong> updated (<a title="View differences" href="/journals/106116/diff?detail_id=103346">diff</a>)</li></ul>
rgw - Bug #22804: multisite Synchronization failed when read and write delete at the same time
https://tracker.ceph.com/issues/22804?journal_id=107120
2018-02-08T08:10:58Z
Amine Liu
<ul></ul><p>#!/bin/sh</p>
<p>usercfg="Ymliu.cfg" <br />bucket="s3://testmix" <br />file="s-mix011.txt" <br />for i in {1..100};<br />do
{<br /> s3cmd -c $usercfg put $file $bucket/$i & <br /> sleep 5<br /> #s3cmd -c $usercfg rm $bucket/$i <br /> s3cmd -c $usercfg put $file $bucket/$i &<br /> s3cmd -c $usercfg rm $bucket/$i &<br />}&<br />done</p>
<p>This script can also be reproduced. master ` s3cmd ls` is 99 objects, bucket stats is'"num_objects": 96';and slave `s3cmd ls `is 89 objects,bucket stats is'"num_objects": 89'.</p>
rgw - Bug #22804: multisite Synchronization failed when read and write delete at the same time
https://tracker.ceph.com/issues/22804?journal_id=107121
2018-02-08T08:23:44Z
Amine Liu
<ul></ul><p>[root@sx-3f3r-ceph-s3-c1-03 my-cluster]# radosgw-admin bucket stats --bucket=testmix
{<br /> "bucket": "testmix",<br /> "zonegroup": "e1d5f39f-70f6-443e-98e8-3dc0b3b312f8",<br /> "placement_rule": "default-placement",<br /> "explicit_placement": {<br /> "data_pool": "",<br /> "data_extra_pool": "",<br /> "index_pool": "" <br /> },<br /> "id": "d49d824f-76c0-4d15-9219-ca7acf5c31b3.1514569.3",<br /> "marker": "d49d824f-76c0-4d15-9219-ca7acf5c31b3.1514569.3",<br /> "index_type": "Normal",<br /> "owner": "Ymliu",<br /> "ver": "0#163,1#74,2#265,3#246,4#161,5#164,6#179,7#244,8#166,9#220,10#179,11#180,12#134,13#180,14#194,15#103,16#78,17#154,18#170,19#214",<br /> "master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0,11#0,12#0,13#0,14#0,15#0,16#0,17#0,18#0,19#0",<br /> "mtime": "2018-02-08 15:11:52.323452",<br /> "max_marker": "0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#,11#,12#,13#,14#,15#,16#,17#,18#,19#",<br /> "usage": {<br /> "rgw.none": {<br /> "size": 0,<br /> "size_actual": 0,<br /> "size_utilized": 0,<br /> "size_kb": 0,<br /> "size_kb_actual": 0,<br /> "size_kb_utilized": 0,<br /> "num_objects": 3<br /> },<br /> "rgw.main": {<br /> "size": 4785792,<br /> "size_actual": 5111808,<br /> "size_utilized": 4785792,<br /> "size_kb": 4674,<br /> "size_kb_actual": 4992,<br /> "size_kb_utilized": 4674,<br /> "num_objects": 96<br /> }<br /> },<br /> "bucket_quota": {<br /> "enabled": false,<br /> "check_on_raw": false,<br /> "max_size": -1,<br /> "max_size_kb": 0,<br /> "max_objects": -1<br /> }<br />}</p>
<p>[root@sx-3f3r-ceph-s3-c1-03 my-cluster]# s3cmd -c Ymliu.cfg ls s3://testmix|wc -l <br />99<br />[root@sx-3f3r-ceph-s3-c1-03 my-cluster]#</p>
rgw - Bug #22804: multisite Synchronization failed when read and write delete at the same time
https://tracker.ceph.com/issues/22804?journal_id=107122
2018-02-08T08:24:18Z
Amine Liu
<ul></ul><p>[root@sx-3f3r-ceph-s3-c1-03 my-cluster]# radosgw-admin bucket check --fix --check-objects --bucket=testmix<br />[]
{<br /> "object": "1",<br /> "object": "10",<br /> "object": "100",<br /> "object": "11",<br /> "object": "12",<br /> "object": "13",<br /> "object": "14",<br /> "object": "15",<br /> "object": "16",<br /> "object": "17",<br /> "object": "18",<br /> "object": "19",<br /> "object": "2",<br /> "object": "20",<br /> "object": "21",<br /> "object": "22",<br /> "object": "23",<br /> "object": "24",<br /> "object": "25",<br /> "object": "26",<br /> "object": "27",<br /> "object": "28",<br /> "object": "29",<br /> "object": "3",<br /> "object": "30",<br /> "object": "31",<br /> "object": "32",<br /> "object": "33",<br /> "object": "34",<br /> "object": "35",<br /> "object": "36",<br /> "object": "37",<br /> "object": "38",<br /> "object": "39",<br /> "object": "4",<br /> "object": "40",<br /> "object": "41",<br /> "object": "42",<br /> "object": "43",<br /> "object": "44",<br /> "object": "45",<br /> "object": "46",<br /> "object": "48",<br /> "object": "49",<br /> "object": "5",<br /> "object": "50",<br /> "object": "51",<br /> "object": "52",<br /> "object": "53",<br /> "object": "54",<br /> "object": "55",<br /> "object": "56",<br /> "object": "57",<br /> "object": "58",<br /> "object": "59",<br /> "object": "6",<br /> "object": "60",<br /> "object": "61",<br /> "object": "62",<br /> "object": "63",<br /> "object": "64",<br /> "object": "65",<br /> "object": "66",<br /> "object": "67",<br /> "object": "68",<br /> "object": "69",<br /> "object": "7",<br /> "object": "70",<br /> "object": "71",<br /> "object": "72",<br /> "object": "73",<br /> "object": "74",<br /> "object": "75",<br /> "object": "76",<br /> "object": "77",<br /> "object": "78",<br /> "object": "79",<br /> "object": "8",<br /> "object": "80",<br /> "object": "81",<br /> "object": "82",<br /> "object": "83",<br /> "object": "84",<br /> "object": "85",<br /> "object": "86",<br /> "object": "87",<br /> "object": "88",<br /> "object": "89",<br /> "object": "9",<br /> "object": "90",<br /> "object": "91",<br /> "object": "92",<br /> "object": "93",<br /> "object": "94",<br /> "object": "95",<br /> "object": "96",<br /> "object": "97",<br /> "object": "98",<br /> "object": "99" <br />}
{<br /> "existing_header": {<br /> "usage": {<br /> "rgw.none": {<br /> "size": 0,<br /> "size_actual": 0,<br /> "size_utilized": 0,<br /> "size_kb": 0,<br /> "size_kb_actual": 0,<br /> "size_kb_utilized": 0,<br /> "num_objects": 3<br /> },<br /> "rgw.main": {<br /> "size": 4785792,<br /> "size_actual": 5111808,<br /> "size_utilized": 4785792,<br /> "size_kb": 4674,<br /> "size_kb_actual": 4992,<br /> "size_kb_utilized": 4674,<br /> "num_objects": 96<br /> }<br /> }<br /> },<br /> "calculated_header": {<br /> "usage": {<br /> "rgw.none": {<br /> "size": 0,<br /> "size_actual": 0,<br /> "size_utilized": 0,<br /> "size_kb": 0,<br /> "size_kb_actual": 0,<br /> "size_kb_utilized": 0,<br /> "num_objects": 3<br /> },<br /> "rgw.main": {<br /> "size": 4785792,<br /> "size_actual": 5111808,<br /> "size_utilized": 4785792,<br /> "size_kb": 4674,<br /> "size_kb_actual": 4992,<br /> "size_kb_utilized": 4674,<br /> "num_objects": 96<br /> }<br /> }<br /> }<br />}</p>
rgw - Bug #22804: multisite Synchronization failed when read and write delete at the same time
https://tracker.ceph.com/issues/22804?journal_id=107123
2018-02-08T09:44:40Z
Amine Liu
<ul><li><strong>File</strong> <a href="/attachments/download/3228/slave-bilog.txt">slave-bilog.txt</a> <a class="icon-only icon-magnifier" title="View" href="/attachments/3228/slave-bilog.txt">View</a> added</li><li><strong>File</strong> <a href="/attachments/download/3227/master-bilog.txt">master-bilog.txt</a> <a class="icon-only icon-magnifier" title="View" href="/attachments/3227/master-bilog.txt">View</a> added</li></ul><p>Another test. dump of bilog:</p>
<p>master merge op(write,delete),but slave miss some op.</p>
rgw - Bug #22804: multisite Synchronization failed when read and write delete at the same time
https://tracker.ceph.com/issues/22804?journal_id=107468
2018-02-15T18:58:18Z
Casey Bodley
cbodley@redhat.com
<ul></ul><p>a couple of prs that are potentially related? <a class="external" href="https://github.com/ceph/ceph/pull/20396">https://github.com/ceph/ceph/pull/20396</a> <a class="external" href="https://github.com/ceph/ceph/pull/19895">https://github.com/ceph/ceph/pull/19895</a></p>
rgw - Bug #22804: multisite Synchronization failed when read and write delete at the same time
https://tracker.ceph.com/issues/22804?journal_id=107898
2018-02-23T01:42:18Z
Amine Liu
<ul></ul><p>Casey Bodley wrote:</p>
<blockquote>
<p>a couple of prs that are potentially related? <a class="external" href="https://github.com/ceph/ceph/pull/20396">https://github.com/ceph/ceph/pull/20396</a> <a class="external" href="https://github.com/ceph/ceph/pull/19895">https://github.com/ceph/ceph/pull/19895</a></p>
</blockquote>
<p>yes, "rgw: fix index cancel op miss update header <a class="issue tracker-1 status-9 priority-5 priority-high3 closed" title="Bug: segv in python somewhere from restful test (Can't reproduce)" href="https://tracker.ceph.com/issues/20396">#20396</a>" that is why index not equal by `ls`;</p>
<p>but "rgw: do not add cancel op in squash_map <a class="issue tracker-1 status-9 priority-7 priority-highest closed" title="Bug: test/osd/RadosModel.h: 1169: FAILED assert(version == old_value.version) during upgrade (Can't reproduce)" href="https://tracker.ceph.com/issues/19895">#19895</a>" that does not seem to involve a OP merger, like miss `DELETE` on slave zone.</p>
rgw - Bug #22804: multisite Synchronization failed when read and write delete at the same time
https://tracker.ceph.com/issues/22804?journal_id=108512
2018-03-05T18:59:41Z
Yehuda Sadeh
yehuda@redhat.com
<ul><li><strong>Subject</strong> changed from <i>multipsite Synchronization failed when read and write delete at the same time</i> to <i>multisite Synchronization failed when read and write delete at the same time</i></li><li><strong>Assignee</strong> set to <i>Casey Bodley</i></li><li><strong>Priority</strong> changed from <i>Normal</i> to <i>High</i></li></ul>
rgw - Bug #22804: multisite Synchronization failed when read and write delete at the same time
https://tracker.ceph.com/issues/22804?journal_id=108809
2018-03-09T03:33:19Z
Pengju Niu
<ul></ul><p>it beacase that master try to del first write objA, and it has been canceled however bilog record del op behind second write.So master rgw has objA,slave rgw can't find. pr is <a class="external" href="https://github.com/ceph/ceph/pull/20814">https://github.com/ceph/ceph/pull/20814</a>.</p>
rgw - Bug #22804: multisite Synchronization failed when read and write delete at the same time
https://tracker.ceph.com/issues/22804?journal_id=108810
2018-03-09T03:39:57Z
Amine Liu
<ul></ul><p>Pengju Niu wrote:</p>
<blockquote>
<p>it beacase that master try to del first write objA, and it has been canceled however bilog record del op behind second write.So master rgw has objA,slave rgw can't find. pr is <a class="external" href="https://github.com/ceph/ceph/pull/20814">https://github.com/ceph/ceph/pull/20814</a>.</p>
</blockquote>
<p>Yes, your pull solved my problem, thank you!</p>
rgw - Bug #22804: multisite Synchronization failed when read and write delete at the same time
https://tracker.ceph.com/issues/22804?journal_id=109189
2018-03-15T17:50:52Z
Yehuda Sadeh
yehuda@redhat.com
<ul></ul><p>I commented on the PR.</p>
rgw - Bug #22804: multisite Synchronization failed when read and write delete at the same time
https://tracker.ceph.com/issues/22804?journal_id=109338
2018-03-20T02:03:05Z
Amine Liu
<ul></ul><p>Yehuda Sadeh wrote:</p>
<blockquote>
<p>I commented on the PR.</p>
</blockquote>
<p>sorry, Two days after this pull, bug reappeared.</p>
rgw - Bug #22804: multisite Synchronization failed when read and write delete at the same time
https://tracker.ceph.com/issues/22804?journal_id=110281
2018-04-05T19:18:44Z
Casey Bodley
cbodley@redhat.com
<ul></ul><p><a class="external" href="https://github.com/ceph/ceph/pull/21262">https://github.com/ceph/ceph/pull/21262</a></p>
rgw - Bug #22804: multisite Synchronization failed when read and write delete at the same time
https://tracker.ceph.com/issues/22804?journal_id=110282
2018-04-05T19:19:18Z
Casey Bodley
cbodley@redhat.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Fix Under Review</i></li><li><strong>Tags</strong> changed from <i>multipsite sync failed</i> to <i>multisite sync failed</i></li><li><strong>Backport</strong> set to <i>jewel luminous</i></li></ul>
rgw - Bug #22804: multisite Synchronization failed when read and write delete at the same time
https://tracker.ceph.com/issues/22804?journal_id=110424
2018-04-08T02:26:19Z
Amine Liu
<ul></ul><p>Casey Bodley wrote:</p>
<blockquote>
<p><a class="external" href="https://github.com/ceph/ceph/pull/21262">https://github.com/ceph/ceph/pull/21262</a></p>
</blockquote>
<p>Miss something as error status code</p>
rgw - Bug #22804: multisite Synchronization failed when read and write delete at the same time
https://tracker.ceph.com/issues/22804?journal_id=111064
2018-04-12T17:33:10Z
Casey Bodley
cbodley@redhat.com
<ul><li><strong>Status</strong> changed from <i>Fix Under Review</i> to <i>Pending Backport</i></li></ul>
rgw - Bug #22804: multisite Synchronization failed when read and write delete at the same time
https://tracker.ceph.com/issues/22804?journal_id=111067
2018-04-12T17:37:21Z
Abhishek Lekshmanan
abhishek.lekshmanan@gmail.com
<ul></ul><p>master pr : <a class="external" href="https://github.com/ceph/ceph/pull/20814">https://github.com/ceph/ceph/pull/20814</a></p>
rgw - Bug #22804: multisite Synchronization failed when read and write delete at the same time
https://tracker.ceph.com/issues/22804?journal_id=111069
2018-04-12T17:38:48Z
Abhishek Lekshmanan
abhishek.lekshmanan@gmail.com
<ul><li><strong>Copied to</strong> <i><a class="issue tracker-9 status-3 priority-5 priority-high3 closed" href="/issues/23690">Backport #23690</a>: luminous: multisite Synchronization failed when read and write delete at the same time</i> added</li></ul>
rgw - Bug #22804: multisite Synchronization failed when read and write delete at the same time
https://tracker.ceph.com/issues/22804?journal_id=111091
2018-04-12T19:34:07Z
Nathan Cutler
ncutler@suse.cz
<ul><li><strong>Copied to</strong> <i><a class="issue tracker-9 status-6 priority-4 priority-default closed" href="/issues/23692">Backport #23692</a>: jewel: multisite Synchronization failed when read and write delete at the same time</i> added</li></ul>
rgw - Bug #22804: multisite Synchronization failed when read and write delete at the same time
https://tracker.ceph.com/issues/22804?journal_id=111314
2018-04-16T04:08:03Z
Amine Liu
<ul><li><strong>File</strong> <a href="/attachments/download/3414/master-index100k_objs_591-bilog.rar">master-index100k_objs_591-bilog.rar</a> added</li></ul><p>cosbench with read/write/delete at the same time, master's objects and index was still more than slave's.</p>
rgw - Bug #22804: multisite Synchronization failed when read and write delete at the same time
https://tracker.ceph.com/issues/22804?journal_id=123617
2018-10-26T11:04:21Z
Nathan Cutler
ncutler@suse.cz
<ul><li><strong>Status</strong> changed from <i>Pending Backport</i> to <i>Resolved</i></li></ul>