https://tracker.ceph.com/
https://tracker.ceph.com/favicon.ico
2018-09-21T02:53:05Z
Ceph
rgw - Bug #36092: Radosgw elastic search sync module not working properly (all result same)
https://tracker.ceph.com/issues/36092?journal_id=121116
2018-09-21T02:53:05Z
hoan nv
<ul></ul><p>My ceph cluster is running on 13.2.1 version.</p>
rgw - Bug #36092: Radosgw elastic search sync module not working properly (all result same)
https://tracker.ceph.com/issues/36092?journal_id=122535
2018-10-11T18:12:30Z
Abhishek Lekshmanan
abhishek.lekshmanan@gmail.com
<ul></ul><p>the log cuts off at the end, can you paste the output of query on the cli and the rest of hte log?</p>
rgw - Bug #36092: Radosgw elastic search sync module not working properly (all result same)
https://tracker.ceph.com/issues/36092?journal_id=122536
2018-10-11T18:12:40Z
Abhishek Lekshmanan
abhishek.lekshmanan@gmail.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Need More Info</i></li></ul>
rgw - Bug #36092: Radosgw elastic search sync module not working properly (all result same)
https://tracker.ceph.com/issues/36092?journal_id=122537
2018-10-11T18:19:01Z
Abhishek Lekshmanan
abhishek.lekshmanan@gmail.com
<ul><li><strong>Assignee</strong> set to <i>Abhishek Lekshmanan</i></li></ul>
rgw - Bug #36092: Radosgw elastic search sync module not working properly (all result same)
https://tracker.ceph.com/issues/36092?journal_id=125290
2018-11-28T04:35:04Z
hoan nv
<ul></ul><p>This is rgw log.</p>
<pre>
2018-11-28 11:29:34.891 7fd4583ad700 10 cache get: name=hn-1-es.rgw.meta+root+.bucket.meta.buck:492c72b3-f13f-44ab-8d30-a11a96729b74.107808.1 : expiry miss
2018-11-28 11:29:34.891 7fd4583ad700 10 cache put: name=hn-1-es.rgw.meta+root+.bucket.meta.buck:492c72b3-f13f-44ab-8d30-a11a96729b74.107808.1 info.flags=0x16
2018-11-28 11:29:34.891 7fd4583ad700 10 adding hn-1-es.rgw.meta+root+.bucket.meta.buck:492c72b3-f13f-44ab-8d30-a11a96729b74.107808.1 to cache LRU end
2018-11-28 11:29:34.891 7fd4583ad700 10 updating xattr: name=user.rgw.acl bl.length()=141
2018-11-28 11:29:34.891 7fd4583ad700 20 get_system_obj_state: s->obj_tag was set empty
2018-11-28 11:29:34.891 7fd4583ad700 20 Read xattr: user.rgw.acl
2018-11-28 11:29:34.891 7fd4583ad700 20 Read xattr: user.rgw.idtag
2018-11-28 11:29:34.891 7fd4583ad700 10 cache get: name=hn-1-es.rgw.meta+root+.bucket.meta.buck:492c72b3-f13f-44ab-8d30-a11a96729b74.107808.1 : type miss (requested=0x11, cached=0x16)
2018-11-28 11:29:34.891 7fd4583ad700 20 rados->read ofs=0 len=524288
2018-11-28 11:29:34.891 7fd4583ad700 20 rados->read r=0 bl.length=272
2018-11-28 11:29:34.891 7fd4583ad700 10 cache put: name=hn-1-es.rgw.meta+root+.bucket.meta.buck:492c72b3-f13f-44ab-8d30-a11a96729b74.107808.1 info.flags=0x11
2018-11-28 11:29:34.891 7fd4583ad700 10 moving hn-1-es.rgw.meta+root+.bucket.meta.buck:492c72b3-f13f-44ab-8d30-a11a96729b74.107808.1 to cache LRU end
2018-11-28 11:29:34.891 7fd4583ad700 10 chain_cache_entry: cache_locator=hn-1-es.rgw.meta+root+buck
2018-11-28 11:29:34.891 7fd4583ad700 10 chain_cache_entry: cache_locator=hn-1-es.rgw.meta+root+.bucket.meta.buck:492c72b3-f13f-44ab-8d30-a11a96729b74.107808.1
2018-11-28 11:29:34.891 7fd4583ad700 15 decode_policy Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>hoannv</ID><DisplayName>hoannv</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>hoannv</ID><DisplayName>hoannv</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy>
2018-11-28 11:29:34.891 7fd4583ad700 2 req 11:0.010648:s3:GET /buck/:metadata_search:recalculating target
2018-11-28 11:29:34.891 7fd4583ad700 2 req 11:0.010664:s3:GET /buck/:metadata_search:reading permissions
2018-11-28 11:29:34.891 7fd4583ad700 2 req 11:0.010673:s3:GET /buck/:metadata_search:init op
2018-11-28 11:29:34.891 7fd4583ad700 2 req 11:0.010676:s3:GET /buck/:metadata_search:verifying op mask
2018-11-28 11:29:34.891 7fd4583ad700 20 required_mask= 1 user.op_mask=7
2018-11-28 11:29:34.891 7fd4583ad700 2 req 11:0.010680:s3:GET /buck/:metadata_search:verifying op permissions
2018-11-28 11:29:34.891 7fd4583ad700 2 req 11:0.010683:s3:GET /buck/:metadata_search:verifying op params
2018-11-28 11:29:34.891 7fd4583ad700 2 req 11:0.010685:s3:GET /buck/:metadata_search:pre-executing
2018-11-28 11:29:34.891 7fd4583ad700 2 req 11:0.010687:s3:GET /buck/:metadata_search:executing
2018-11-28 11:29:34.891 7fd4583ad700 20 sending request to elasticsearch, payload={"query":{"bool":{"must":[{"term":{"bucket":"buck"}},{"bool":{"must":[{"term":{"permissions":"hoannv"}},{"range":{"name":{"gte":"foo"}}}]}}]}}}
2018-11-28 11:29:34.891 7fd4583ad700 20 sending request to http://10.5.8.176:9200/rgw-ss-5fc11232/_search?size=100
2018-11-28 11:29:34.891 7fd4583ad700 20 register_request mgr=0x55c69de68000 req_data->id=12, curl_handle=0x55c69fc84140
2018-11-28 11:29:34.891 7fd477df3700 20 link_request req_data=0x55c69fbeed80 req_data->id=12, curl_handle=0x55c69fc84140
2018-11-28 11:29:34.903 7fd477df3700 10 receive_http_header
2018-11-28 11:29:34.903 7fd477df3700 10 received header:HTTP/1.1 200 OK
2018-11-28 11:29:34.903 7fd477df3700 10 receive_http_header
2018-11-28 11:29:34.903 7fd477df3700 10 received header:content-type: application/json; charset=UTF-8
2018-11-28 11:29:34.903 7fd477df3700 10 receive_http_header
2018-11-28 11:29:34.903 7fd477df3700 10 received header:content-length: 2011
2018-11-28 11:29:34.903 7fd477df3700 10 receive_http_header
2018-11-28 11:29:34.903 7fd477df3700 10 received header:
2018-11-28 11:29:34.903 7fd4583ad700 20 response: {"took":7,"timed_out":false,"_shards":{"total":10,"successful":10,"skipped":0,"failed":0},"hits":{"total":4,"max_score":1.0,"hits":[{"_index":"rgw-ss-5fc11232","_type":"object","_id":"492c72b3-f13f-44ab-8d30-a11a96729b74.106743.1%3A4.jpg%3Anull","_score":1.0,"_source":{"bucket":"hoannv123","name":"4.jpg","instance":"null","versioned_epoch":0,"owner":{"id":"hoannv","display_name":"hoannv"},"permissions":["hoannv"],"meta":{"size":30494,"mtime":"2018-11-23T08:39:24.213Z","content_type":"image/jpeg","etag":"3a4dea97c6567be869aec8aecf5662c","tail_tag":"492c72b3-f13f-44ab-8d30-a11a96729b74.106743.15176"}}},{"_index":"rgw-ss-5fc11232","_type":"object","_id":"492c72b3-f13f-44ab-8d30-a11a96729b74.107808.1%3Afoo%3Anull","_score":1.0,"_source":{"bucket":"buck","name":"foo","instance":"null","versioned_epoch":0,"owner":{"id":"hoannv","display_name":"hoannv"},"permissions":["hoannv"],"meta":{"size":7,"mtime":"2018-11-27T10:44:20.894Z","content_type":"text/plain","etag":"26f3808cdec7fb1ba79cd549cba6e20","tail_tag":"492c72b3-f13f-44ab-8d30-a11a96729b74.107808.5136"}}},{"_index":"rgw-ss-5fc11232","_type":"object","_id":"492c72b3-f13f-44ab-8d30-a11a96729b74.107808.1%3Afoo1%3Anull","_score":1.0,"_source":{"bucket":"buck","name":"foo1","instance":"null","versioned_epoch":0,"owner":{"id":"hoannv","display_name":"hoannv"},"permissions":["hoannv"],"meta":{"size":7,"mtime":"2018-11-27T10:44:27.946Z","content_type":"text/plain","etag":"26f3808cdec7fb1ba79cd549cba6e20","tail_tag":"492c72b3-f13f-44ab-8d30-a11a96729b74.107808.5275"}}},{"_index":"rgw-ss-5fc11232","_type":"object","_id":"492c72b3-f13f-44ab-8d30-a11a96729b74.106743.1%3A1.jpg%3Anull","_score":1.0,"_source":{"bucket":"hoannv123","name":"1.jpg","instance":"null","versioned_epoch":0,"owner":{"id":"hoannv","display_name":"hoannv"},"permissions":["hoannv"],"meta":{"size":30494,"mtime":"2018-11-27T10:41:20.885Z","content_type":"image/jpeg","etag":"3a4dea97c6567be869aec8aecf5662c","tail_tag":"492c72b3-f13f-44ab-8d30-a11a96729b74.107808.3376"}}}]}}
2018-11-28 11:29:34.903 7fd4583ad700 2 req 11:0.022545:s3:GET /buck/:metadata_search:completing
2018-11-28 11:29:34.903 7fd4583ad700 2 req 11:0.022916:s3:GET /buck/:metadata_search:op status=0
2018-11-28 11:29:34.903 7fd4583ad700 2 req 11:0.022920:s3:GET /buck/:metadata_search:http status=200
2018-11-28 11:29:34.903 7fd4583ad700 1 ====== req done req=0x7fd4583a4870 op status=0 http_status=200 ======
</pre>
<p>This is my elasticsearch query<br /><pre>
curl -XPOST 10.5.8.176:9200/rgw-ss-5fc11232/_search?pretty -d '{"query":{"bool":{"must":[{"term":{"bucket":"buck"}},{"bool":{"must":[{"term":{"permissions":"hoannv"}},{"range":{"name":{"gte":"foo"}}}]}}]}}}'
{
"took" : 6,
"timed_out" : false,
"_shards" : {
"total" : 10,
"successful" : 10,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 2,
"max_score" : 1.5753641,
"hits" : [
{
"_index" : "rgw-ss-5fc11232",
"_type" : "object",
"_id" : "492c72b3-f13f-44ab-8d30-a11a96729b74.107808.1%3Afoo%3Anull",
"_score" : 1.5753641,
"_source" : {
"bucket" : "buck",
"name" : "foo",
"instance" : "null",
"versioned_epoch" : 0,
"owner" : {
"id" : "hoannv",
"display_name" : "hoannv"
},
"permissions" : [
"hoannv"
],
"meta" : {
"size" : 7,
"mtime" : "2018-11-27T10:44:20.894Z",
"content_type" : "text/plain",
"etag" : "26f3808cdec7fb1ba79cd549cba6e20",
"tail_tag" : "492c72b3-f13f-44ab-8d30-a11a96729b74.107808.5136"
}
}
},
{
"_index" : "rgw-ss-5fc11232",
"_type" : "object",
"_id" : "492c72b3-f13f-44ab-8d30-a11a96729b74.107808.1%3Afoo1%3Anull",
"_score" : 1.5753641,
"_source" : {
"bucket" : "buck",
"name" : "foo1",
"instance" : "null",
"versioned_epoch" : 0,
"owner" : {
"id" : "hoannv",
"display_name" : "hoannv"
},
"permissions" : [
"hoannv"
],
"meta" : {
"size" : 7,
"mtime" : "2018-11-27T10:44:27.946Z",
"content_type" : "text/plain",
"etag" : "26f3808cdec7fb1ba79cd549cba6e20",
"tail_tag" : "492c72b3-f13f-44ab-8d30-a11a96729b74.107808.5275"
}
}
}
]
}
}
</pre></p>
<p>I use obo tool to search <br />obo mdsearch buck --query='name>=foo'</p>
<pre>
obo mdsearch buck --query='name>=foo'
{
"Marker": "",
"Objects": [
{
"ContentType": "image/jpeg",
"LastModified": "2018-11-23T08:39:24.213Z",
"Bucket": "hoannv123",
"Instance": "null",
"ETag": "\"3a4dea97c6567be869aec8aecf5662c\"",
"VersionedEpoch": 0,
"CustomMetadata": [],
"Key": "4.jpg",
"Owner": {
"DisplayName": "hoannv",
"ID": "hoannv"
},
"Size": 30494
},
{
"ContentType": "text/plain",
"LastModified": "2018-11-27T10:44:20.894Z",
"Bucket": "buck",
"Instance": "null",
"ETag": "\"26f3808cdec7fb1ba79cd549cba6e20\"",
"VersionedEpoch": 0,
"CustomMetadata": [],
"Key": "foo",
"Owner": {
"DisplayName": "hoannv",
"ID": "hoannv"
},
"Size": 7
},
{
"ContentType": "text/plain",
"LastModified": "2018-11-27T10:44:27.946Z",
"Bucket": "buck",
"Instance": "null",
"ETag": "\"26f3808cdec7fb1ba79cd549cba6e20\"",
"VersionedEpoch": 0,
"CustomMetadata": [],
"Key": "foo1",
"Owner": {
"DisplayName": "hoannv",
"ID": "hoannv"
},
"Size": 7
},
{
"ContentType": "image/jpeg",
"LastModified": "2018-11-27T10:41:20.885Z",
"Bucket": "hoannv123",
"Instance": "null",
"ETag": "\"3a4dea97c6567be869aec8aecf5662c\"",
"VersionedEpoch": 0,
"CustomMetadata": [],
"Key": "1.jpg",
"Owner": {
"DisplayName": "hoannv",
"ID": "hoannv"
},
"Size": 30494
}
],
"IsTruncated": "false"
}
[
{
"name": "1.jpg",
"version_id": "null",
"last_modified": "2018-11-27T10:41:20.885Z",
"etag": "3a4dea97c6567be869aec8aecf5662c",
"content_type": "image/jpeg",
"owner": "hoannv",
"size": 30494
},
{
"name": "4.jpg",
"version_id": "null",
"last_modified": "2018-11-23T08:39:24.213Z",
"etag": "3a4dea97c6567be869aec8aecf5662c",
"content_type": "image/jpeg",
"owner": "hoannv",
"size": 30494
},
{
"name": "foo",
"version_id": "null",
"last_modified": "2018-11-27T10:44:20.894Z",
"etag": "26f3808cdec7fb1ba79cd549cba6e20",
"content_type": "text/plain",
"owner": "hoannv",
"size": 7
},
{
"name": "foo1",
"version_id": "null",
"last_modified": "2018-11-27T10:44:27.946Z",
"etag": "26f3808cdec7fb1ba79cd549cba6e20",
"content_type": "text/plain",
"owner": "hoannv",
"size": 7
}
]
</pre>
<p>As you see.</p>
<p>Query in elasticsearch work: result only include foo object.<br />But reponse from rgw display all object.</p>
<p>Thanks.</p>
rgw - Bug #36092: Radosgw elastic search sync module not working properly (all result same)
https://tracker.ceph.com/issues/36092?journal_id=125302
2018-11-28T10:01:25Z
hoan nv
<ul></ul><p>I enabled elaticsearch log</p>
<pre>
curl -XPUT '10.5.8.176:9200/rgw-ss-5fc11232/_settings' -H 'Content-Type: application/json' -d '{
"index.indexing.slowlog.threshold.index.warn": "0ms",
"index.indexing.slowlog.threshold.index.info": "0ms",
"index.indexing.slowlog.threshold.index.debug": "0ms",
"index.indexing.slowlog.threshold.index.trace": "0ms",
"index.indexing.slowlog.level": "trace",
"index.search.slowlog.threshold.fetch.warn": "0ms",
"index.search.slowlog.threshold.fetch.info": "0ms",
"index.search.slowlog.threshold.fetch.debug": "0ms",
"index.search.slowlog.threshold.fetch.trace": "0ms",
"index.indexing.slowlog.source": "1000"
}'
</pre>
<p>elasticsearch log while i am use curl to query elasticsearch</p>
<pre>
[2018-11-28T16:44:58,952][WARN ][index.search.slowlog.fetch] [aQHdoZA] [rgw-ss-5fc11232][4] took[728.6micros], took_millis[0], total_hits[1], types[], stats[], search_type[QUERY_THEN_FETCH], total_shards[10], source[{"query":{"bool":{"must":[{"term":{"bucket":{"value":"buck","boost":1.0}}},{"bool":{"must":[{"term":{"permissions":{"value":"hoannv","boost":1.0}}},{"range":{"name":{"from":"foo","to":null,"include_lower":false,"include_upper":true,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}}], id[],
</pre>
<p>elasticsearch log while i am use obo to query</p>
<pre>
[2018-11-28T16:45:19,930][WARN ][index.search.slowlog.fetch] [aQHdoZA] [rgw-ss-5fc11232][7] took[685.8micros], took_millis[0], total_hits[1], types[], stats[], search_type[QUERY_THEN_FETCH], total_shards[10], source[{"size":100}], id[],
</pre>
<p>Thanks</p>
rgw - Bug #36092: Radosgw elastic search sync module not working properly (all result same)
https://tracker.ceph.com/issues/36092?journal_id=125627
2018-12-05T04:24:28Z
hoan nv
<ul></ul><p>I tested on mimic, rgw not return correct data, it return all result.</p>
<p>On luminous, it return correct data.</p>
<p>Thanks.</p>
rgw - Bug #36092: Radosgw elastic search sync module not working properly (all result same)
https://tracker.ceph.com/issues/36092?journal_id=128013
2019-01-23T23:42:11Z
Yehuda Sadeh
yehuda@redhat.com
<ul><li><strong>Status</strong> changed from <i>Need More Info</i> to <i>Fix Under Review</i></li><li><strong>Priority</strong> changed from <i>Normal</i> to <i>High</i></li><li><strong>Backport</strong> set to <i>mimic</i></li></ul><p>This fixes it:<br /><a class="external" href="https://github.com/ceph/ceph/pull/26103">https://github.com/ceph/ceph/pull/26103</a></p>
rgw - Bug #36092: Radosgw elastic search sync module not working properly (all result same)
https://tracker.ceph.com/issues/36092?journal_id=128722
2019-02-05T01:55:25Z
Yehuda Sadeh
yehuda@redhat.com
<ul><li><strong>Status</strong> changed from <i>Fix Under Review</i> to <i>Pending Backport</i></li></ul>
rgw - Bug #36092: Radosgw elastic search sync module not working properly (all result same)
https://tracker.ceph.com/issues/36092?journal_id=128777
2019-02-05T17:03:44Z
Nathan Cutler
ncutler@suse.cz
<ul><li><strong>Copied to</strong> <i><a class="issue tracker-9 status-3 priority-4 priority-default closed" href="/issues/38194">Backport #38194</a>: mimic: Radosgw elastic search sync module not working properly (all result same)</i> added</li></ul>
rgw - Bug #36092: Radosgw elastic search sync module not working properly (all result same)
https://tracker.ceph.com/issues/36092?journal_id=131629
2019-03-12T12:04:10Z
Nathan Cutler
ncutler@suse.cz
<ul><li><strong>Backport</strong> changed from <i>mimic</i> to <i>mimic luminous</i></li></ul>
rgw - Bug #36092: Radosgw elastic search sync module not working properly (all result same)
https://tracker.ceph.com/issues/36092?journal_id=131630
2019-03-12T12:04:29Z
Nathan Cutler
ncutler@suse.cz
<ul><li><strong>Copied to</strong> <i><a class="issue tracker-9 status-3 priority-4 priority-default closed" href="/issues/38694">Backport #38694</a>: luminous: Radosgw elastic search sync module not working properly (all result same)</i> added</li></ul>
rgw - Bug #36092: Radosgw elastic search sync module not working properly (all result same)
https://tracker.ceph.com/issues/36092?journal_id=133062
2019-03-28T16:13:37Z
Nathan Cutler
ncutler@suse.cz
<ul><li><strong>Status</strong> changed from <i>Pending Backport</i> to <i>Resolved</i></li></ul>