Bug #36092
closedRadosgw elastic search sync module not working properly (all result same)
0%
Description
Hi all.
Currently, i create a zone with elasticsearch tier.
My radosgw log
2018-09-19 14:45:00.281 7f5b8272b700 15 decode_policy Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>hoannv</I D><DisplayName>hoannv</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>hoannv </ID><DisplayName>hoannv</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy> 2018-09-19 14:45:00.281 7f5b8272b700 2 req 3:0.000333:s3:GET /hoannv-test/:metadata_search:recalculating target 2018-09-19 14:45:00.281 7f5b8272b700 2 req 3:0.000340:s3:GET /hoannv-test/:metadata_search:reading permissions 2018-09-19 14:45:00.281 7f5b8272b700 2 req 3:0.000342:s3:GET /hoannv-test/:metadata_search:init op 2018-09-19 14:45:00.281 7f5b8272b700 2 req 3:0.000343:s3:GET /hoannv-test/:metadata_search:verifying op mask 2018-09-19 14:45:00.281 7f5b8272b700 20 required_mask= 1 user.op_mask=7 2018-09-19 14:45:00.281 7f5b8272b700 2 req 3:0.000345:s3:GET /hoannv-test/:metadata_search:verifying op permissions 2018-09-19 14:45:00.281 7f5b8272b700 2 req 3:0.000346:s3:GET /hoannv-test/:metadata_search:verifying op params 2018-09-19 14:45:00.281 7f5b8272b700 2 req 3:0.000347:s3:GET /hoannv-test/:metadata_search:pre-executing 2018-09-19 14:45:00.281 7f5b8272b700 2 req 3:0.000348:s3:GET /hoannv-test/:metadata_search:executing 2018-09-19 14:45:00.281 7f5b8272b700 20 sending request to elasticsearch, payload={"query":{"bool":{"must":[{"term":{"bucket":"hoannv-test"}},{"bool":{"must":[{"term": {"permissions":"hoannv"}},{"term":{"name":"abc"}}]}}]}}} 2018-09-19 14:45:00.281 7f5b8272b700 20 sending request to http://10.5.8.176:9200/rgw-ss-realm-a9f0523f/_search?size=100 2018-09-19 14:45:00.281 7f5b8272b700 20 register_request mgr=0x563b55f92000 req_data->id=4, curl_handle=0x563b564bad00 2018-09-19 14:45:00.281 7f5ba2171700 20 link_request req_data=0x563b565a2d80 req_data->id=4, curl_handle=0x563b564bad00 2018-09-19 14:45:00.293 7f5ba2171700 10 receive_http_header 2018-09-19 14:45:00.293 7f5ba2171700 10 received header:HTTP/1.1 200 OK 2018-09-19 14:45:00.293 7f5ba2171700 10 receive_http_header 2018-09-19 14:45:00.293 7f5ba2171700 10 received header:access-control-allow-credentials: true 2018-09-19 14:45:00.293 7f5ba2171700 10 receive_http_header 2018-09-19 14:45:00.293 7f5ba2171700 10 received header:content-type: application/json; charset=UTF-8 2018-09-19 14:45:00.293 7f5ba2171700 10 receive_http_header 2018-09-19 14:45:00.293 7f5ba2171700 10 received header:content-length: 1198 2018-09-19 14:45:00.293 7f5ba2171700 10 receive_http_header 2018-09-19 14:45:00.293 7f5ba2171700 10 received header: 2018-09-19 14:45:00.293 7f5b8272b700 20 response: {"took":7,"timed_out":false,"_shards":{"total":10,"successful":10,"skipped":0,"failed":0},"hits":{"total":2,"max_scor e":1.0,"hits":[{"_index":"rgw-ss-realm-a9f0523f","_type":"object","_id":"6df1b9b2-cf4d-425e-b30d-9f37dd58d615.24821.2%3Afoo%3AYVFOWWDceKspKX8EAa9eyuwZghW5X2X","_score" :1.0,"_source":{"bucket":"hoannv-test","name":"foo","instance":"YVFOWWDceKspKX8EAa9eyuwZghW5X2X","versioned_epoch":2,"owner":{"id":"hoannv","display_name":"hoannv"},"p ermissions":["hoannv"],"meta":{"size":47,"mtime":"2018-09-19T02:53:25.973Z","content_type":"text/plain","etag":"0a278abf92bcfe8ecb645f463f3f218","tail_tag":"6df1b9b2-c f4d-425e-b30d-9f37dd58d615.26307.1205368"}}},{"_index":"rgw-ss-realm-a9f0523f","_type":"object","_id":"6df1b9b2-cf4d-425e-b30d-9f37dd58d615.24821.2%3Afoo1%3AUzoRWOircq w4CKawvO4Wm-GmrwAmkJz","_score":1.0,"_source":{"bucket":"hoannv-test","name":"foo1","instance":"UzoRWOircqw4CKawvO4Wm-GmrwAmkJz","versioned_epoch":2,"owner":{"id":"hoa nnv","display_name":"hoannv"},"permissions":["hoannv"],"meta":{"size":47,"mtime":"2018-09-19T07:19:34.978Z","content_type":"text/plain","etag":"0a278abf92bcfe8ecb645f4 63f3f218","tail_tag":"6df1b9b2-cf4d-425e-b30d-9f37dd58d615.26307.1362851"}}}]}}
I see this log call a request to
http://10.5.8.176:9200/rgw-ss-realm-a9f0523f/_search?size=100 has payload
{"query":{"bool":{"must":[{"term":{"bucket":"hoannv-test"}},{"bool":{"must":[{"term":{"permissions":"hoannv"}},{"term":{"name":"abc"}}]}}]}}}
I use curl to create a request to this endpoint.
curl -XGET 'http://10.5.8.176:9200/rgw-ss-realm-a9f0523f/_search?size=100' -d '{"query":{"bool":{"must":[{"term":{"bucket":"hoannv-test"}},{"bool":{"must":[{"term":{"permissions":"hoannv"}},{"term":{"name":"foo1"}}]}}]}}}'
{"took":4,"timed_out":false,"_shards":{"total":10,"successful":10,"skipped":0,"failed":0},"hits":{"total":1,"max_score":0.8630463,"hits":[{"_index":"rgw-ss-realm-a9f0523f","_type":"object","_id":"6df1b9b2-cf4d-425e-b30d-9f37dd58d615.24821.2%3Afoo1%3AUzoRWOircqw4CKawvO4Wm-GmrwAmkJz","_score":0.8630463,"_source":{"bucket":"hoannv-test","name":"foo1","instance":"UzoRWOircqw4CKawvO4Wm-GmrwAmkJz","versioned_epoch":2,"owner":{"id":"hoannv","display_name":"hoannv"},"permissions":["hoannv"],"meta":{"size":47,"mtime":"2018-09-19T07:19:34.978Z","content_type":"text/plain","etag":"0a278abf92bcfe8ecb645f463f3f218","tail_tag":"6df1b9b2-cf4d-425e-b30d-9f37dd58d615.26307.1362851"}}}]}}
Result of this query not same result of radosgw log. I tested the result always all
Thanks
Updated by hoan nv over 5 years ago
My ceph cluster is running on 13.2.1 version.
Updated by Abhishek Lekshmanan over 5 years ago
the log cuts off at the end, can you paste the output of query on the cli and the rest of hte log?
Updated by Abhishek Lekshmanan over 5 years ago
- Status changed from New to Need More Info
Updated by Abhishek Lekshmanan over 5 years ago
- Assignee set to Abhishek Lekshmanan
Updated by hoan nv over 5 years ago
This is rgw log.
2018-11-28 11:29:34.891 7fd4583ad700 10 cache get: name=hn-1-es.rgw.meta+root+.bucket.meta.buck:492c72b3-f13f-44ab-8d30-a11a96729b74.107808.1 : expiry miss 2018-11-28 11:29:34.891 7fd4583ad700 10 cache put: name=hn-1-es.rgw.meta+root+.bucket.meta.buck:492c72b3-f13f-44ab-8d30-a11a96729b74.107808.1 info.flags=0x16 2018-11-28 11:29:34.891 7fd4583ad700 10 adding hn-1-es.rgw.meta+root+.bucket.meta.buck:492c72b3-f13f-44ab-8d30-a11a96729b74.107808.1 to cache LRU end 2018-11-28 11:29:34.891 7fd4583ad700 10 updating xattr: name=user.rgw.acl bl.length()=141 2018-11-28 11:29:34.891 7fd4583ad700 20 get_system_obj_state: s->obj_tag was set empty 2018-11-28 11:29:34.891 7fd4583ad700 20 Read xattr: user.rgw.acl 2018-11-28 11:29:34.891 7fd4583ad700 20 Read xattr: user.rgw.idtag 2018-11-28 11:29:34.891 7fd4583ad700 10 cache get: name=hn-1-es.rgw.meta+root+.bucket.meta.buck:492c72b3-f13f-44ab-8d30-a11a96729b74.107808.1 : type miss (requested=0x11, cached=0x16) 2018-11-28 11:29:34.891 7fd4583ad700 20 rados->read ofs=0 len=524288 2018-11-28 11:29:34.891 7fd4583ad700 20 rados->read r=0 bl.length=272 2018-11-28 11:29:34.891 7fd4583ad700 10 cache put: name=hn-1-es.rgw.meta+root+.bucket.meta.buck:492c72b3-f13f-44ab-8d30-a11a96729b74.107808.1 info.flags=0x11 2018-11-28 11:29:34.891 7fd4583ad700 10 moving hn-1-es.rgw.meta+root+.bucket.meta.buck:492c72b3-f13f-44ab-8d30-a11a96729b74.107808.1 to cache LRU end 2018-11-28 11:29:34.891 7fd4583ad700 10 chain_cache_entry: cache_locator=hn-1-es.rgw.meta+root+buck 2018-11-28 11:29:34.891 7fd4583ad700 10 chain_cache_entry: cache_locator=hn-1-es.rgw.meta+root+.bucket.meta.buck:492c72b3-f13f-44ab-8d30-a11a96729b74.107808.1 2018-11-28 11:29:34.891 7fd4583ad700 15 decode_policy Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>hoannv</ID><DisplayName>hoannv</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>hoannv</ID><DisplayName>hoannv</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy> 2018-11-28 11:29:34.891 7fd4583ad700 2 req 11:0.010648:s3:GET /buck/:metadata_search:recalculating target 2018-11-28 11:29:34.891 7fd4583ad700 2 req 11:0.010664:s3:GET /buck/:metadata_search:reading permissions 2018-11-28 11:29:34.891 7fd4583ad700 2 req 11:0.010673:s3:GET /buck/:metadata_search:init op 2018-11-28 11:29:34.891 7fd4583ad700 2 req 11:0.010676:s3:GET /buck/:metadata_search:verifying op mask 2018-11-28 11:29:34.891 7fd4583ad700 20 required_mask= 1 user.op_mask=7 2018-11-28 11:29:34.891 7fd4583ad700 2 req 11:0.010680:s3:GET /buck/:metadata_search:verifying op permissions 2018-11-28 11:29:34.891 7fd4583ad700 2 req 11:0.010683:s3:GET /buck/:metadata_search:verifying op params 2018-11-28 11:29:34.891 7fd4583ad700 2 req 11:0.010685:s3:GET /buck/:metadata_search:pre-executing 2018-11-28 11:29:34.891 7fd4583ad700 2 req 11:0.010687:s3:GET /buck/:metadata_search:executing 2018-11-28 11:29:34.891 7fd4583ad700 20 sending request to elasticsearch, payload={"query":{"bool":{"must":[{"term":{"bucket":"buck"}},{"bool":{"must":[{"term":{"permissions":"hoannv"}},{"range":{"name":{"gte":"foo"}}}]}}]}}} 2018-11-28 11:29:34.891 7fd4583ad700 20 sending request to http://10.5.8.176:9200/rgw-ss-5fc11232/_search?size=100 2018-11-28 11:29:34.891 7fd4583ad700 20 register_request mgr=0x55c69de68000 req_data->id=12, curl_handle=0x55c69fc84140 2018-11-28 11:29:34.891 7fd477df3700 20 link_request req_data=0x55c69fbeed80 req_data->id=12, curl_handle=0x55c69fc84140 2018-11-28 11:29:34.903 7fd477df3700 10 receive_http_header 2018-11-28 11:29:34.903 7fd477df3700 10 received header:HTTP/1.1 200 OK 2018-11-28 11:29:34.903 7fd477df3700 10 receive_http_header 2018-11-28 11:29:34.903 7fd477df3700 10 received header:content-type: application/json; charset=UTF-8 2018-11-28 11:29:34.903 7fd477df3700 10 receive_http_header 2018-11-28 11:29:34.903 7fd477df3700 10 received header:content-length: 2011 2018-11-28 11:29:34.903 7fd477df3700 10 receive_http_header 2018-11-28 11:29:34.903 7fd477df3700 10 received header: 2018-11-28 11:29:34.903 7fd4583ad700 20 response: {"took":7,"timed_out":false,"_shards":{"total":10,"successful":10,"skipped":0,"failed":0},"hits":{"total":4,"max_score":1.0,"hits":[{"_index":"rgw-ss-5fc11232","_type":"object","_id":"492c72b3-f13f-44ab-8d30-a11a96729b74.106743.1%3A4.jpg%3Anull","_score":1.0,"_source":{"bucket":"hoannv123","name":"4.jpg","instance":"null","versioned_epoch":0,"owner":{"id":"hoannv","display_name":"hoannv"},"permissions":["hoannv"],"meta":{"size":30494,"mtime":"2018-11-23T08:39:24.213Z","content_type":"image/jpeg","etag":"3a4dea97c6567be869aec8aecf5662c","tail_tag":"492c72b3-f13f-44ab-8d30-a11a96729b74.106743.15176"}}},{"_index":"rgw-ss-5fc11232","_type":"object","_id":"492c72b3-f13f-44ab-8d30-a11a96729b74.107808.1%3Afoo%3Anull","_score":1.0,"_source":{"bucket":"buck","name":"foo","instance":"null","versioned_epoch":0,"owner":{"id":"hoannv","display_name":"hoannv"},"permissions":["hoannv"],"meta":{"size":7,"mtime":"2018-11-27T10:44:20.894Z","content_type":"text/plain","etag":"26f3808cdec7fb1ba79cd549cba6e20","tail_tag":"492c72b3-f13f-44ab-8d30-a11a96729b74.107808.5136"}}},{"_index":"rgw-ss-5fc11232","_type":"object","_id":"492c72b3-f13f-44ab-8d30-a11a96729b74.107808.1%3Afoo1%3Anull","_score":1.0,"_source":{"bucket":"buck","name":"foo1","instance":"null","versioned_epoch":0,"owner":{"id":"hoannv","display_name":"hoannv"},"permissions":["hoannv"],"meta":{"size":7,"mtime":"2018-11-27T10:44:27.946Z","content_type":"text/plain","etag":"26f3808cdec7fb1ba79cd549cba6e20","tail_tag":"492c72b3-f13f-44ab-8d30-a11a96729b74.107808.5275"}}},{"_index":"rgw-ss-5fc11232","_type":"object","_id":"492c72b3-f13f-44ab-8d30-a11a96729b74.106743.1%3A1.jpg%3Anull","_score":1.0,"_source":{"bucket":"hoannv123","name":"1.jpg","instance":"null","versioned_epoch":0,"owner":{"id":"hoannv","display_name":"hoannv"},"permissions":["hoannv"],"meta":{"size":30494,"mtime":"2018-11-27T10:41:20.885Z","content_type":"image/jpeg","etag":"3a4dea97c6567be869aec8aecf5662c","tail_tag":"492c72b3-f13f-44ab-8d30-a11a96729b74.107808.3376"}}}]}} 2018-11-28 11:29:34.903 7fd4583ad700 2 req 11:0.022545:s3:GET /buck/:metadata_search:completing 2018-11-28 11:29:34.903 7fd4583ad700 2 req 11:0.022916:s3:GET /buck/:metadata_search:op status=0 2018-11-28 11:29:34.903 7fd4583ad700 2 req 11:0.022920:s3:GET /buck/:metadata_search:http status=200 2018-11-28 11:29:34.903 7fd4583ad700 1 ====== req done req=0x7fd4583a4870 op status=0 http_status=200 ======
This is my elasticsearch query
curl -XPOST 10.5.8.176:9200/rgw-ss-5fc11232/_search?pretty -d '{"query":{"bool":{"must":[{"term":{"bucket":"buck"}},{"bool":{"must":[{"term":{"permissions":"hoannv"}},{"range":{"name":{"gte":"foo"}}}]}}]}}}' { "took" : 6, "timed_out" : false, "_shards" : { "total" : 10, "successful" : 10, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : 2, "max_score" : 1.5753641, "hits" : [ { "_index" : "rgw-ss-5fc11232", "_type" : "object", "_id" : "492c72b3-f13f-44ab-8d30-a11a96729b74.107808.1%3Afoo%3Anull", "_score" : 1.5753641, "_source" : { "bucket" : "buck", "name" : "foo", "instance" : "null", "versioned_epoch" : 0, "owner" : { "id" : "hoannv", "display_name" : "hoannv" }, "permissions" : [ "hoannv" ], "meta" : { "size" : 7, "mtime" : "2018-11-27T10:44:20.894Z", "content_type" : "text/plain", "etag" : "26f3808cdec7fb1ba79cd549cba6e20", "tail_tag" : "492c72b3-f13f-44ab-8d30-a11a96729b74.107808.5136" } } }, { "_index" : "rgw-ss-5fc11232", "_type" : "object", "_id" : "492c72b3-f13f-44ab-8d30-a11a96729b74.107808.1%3Afoo1%3Anull", "_score" : 1.5753641, "_source" : { "bucket" : "buck", "name" : "foo1", "instance" : "null", "versioned_epoch" : 0, "owner" : { "id" : "hoannv", "display_name" : "hoannv" }, "permissions" : [ "hoannv" ], "meta" : { "size" : 7, "mtime" : "2018-11-27T10:44:27.946Z", "content_type" : "text/plain", "etag" : "26f3808cdec7fb1ba79cd549cba6e20", "tail_tag" : "492c72b3-f13f-44ab-8d30-a11a96729b74.107808.5275" } } } ] } }
I use obo tool to search
obo mdsearch buck --query='name>=foo'
obo mdsearch buck --query='name>=foo' { "Marker": "", "Objects": [ { "ContentType": "image/jpeg", "LastModified": "2018-11-23T08:39:24.213Z", "Bucket": "hoannv123", "Instance": "null", "ETag": "\"3a4dea97c6567be869aec8aecf5662c\"", "VersionedEpoch": 0, "CustomMetadata": [], "Key": "4.jpg", "Owner": { "DisplayName": "hoannv", "ID": "hoannv" }, "Size": 30494 }, { "ContentType": "text/plain", "LastModified": "2018-11-27T10:44:20.894Z", "Bucket": "buck", "Instance": "null", "ETag": "\"26f3808cdec7fb1ba79cd549cba6e20\"", "VersionedEpoch": 0, "CustomMetadata": [], "Key": "foo", "Owner": { "DisplayName": "hoannv", "ID": "hoannv" }, "Size": 7 }, { "ContentType": "text/plain", "LastModified": "2018-11-27T10:44:27.946Z", "Bucket": "buck", "Instance": "null", "ETag": "\"26f3808cdec7fb1ba79cd549cba6e20\"", "VersionedEpoch": 0, "CustomMetadata": [], "Key": "foo1", "Owner": { "DisplayName": "hoannv", "ID": "hoannv" }, "Size": 7 }, { "ContentType": "image/jpeg", "LastModified": "2018-11-27T10:41:20.885Z", "Bucket": "hoannv123", "Instance": "null", "ETag": "\"3a4dea97c6567be869aec8aecf5662c\"", "VersionedEpoch": 0, "CustomMetadata": [], "Key": "1.jpg", "Owner": { "DisplayName": "hoannv", "ID": "hoannv" }, "Size": 30494 } ], "IsTruncated": "false" } [ { "name": "1.jpg", "version_id": "null", "last_modified": "2018-11-27T10:41:20.885Z", "etag": "3a4dea97c6567be869aec8aecf5662c", "content_type": "image/jpeg", "owner": "hoannv", "size": 30494 }, { "name": "4.jpg", "version_id": "null", "last_modified": "2018-11-23T08:39:24.213Z", "etag": "3a4dea97c6567be869aec8aecf5662c", "content_type": "image/jpeg", "owner": "hoannv", "size": 30494 }, { "name": "foo", "version_id": "null", "last_modified": "2018-11-27T10:44:20.894Z", "etag": "26f3808cdec7fb1ba79cd549cba6e20", "content_type": "text/plain", "owner": "hoannv", "size": 7 }, { "name": "foo1", "version_id": "null", "last_modified": "2018-11-27T10:44:27.946Z", "etag": "26f3808cdec7fb1ba79cd549cba6e20", "content_type": "text/plain", "owner": "hoannv", "size": 7 } ]
As you see.
Query in elasticsearch work: result only include foo object.
But reponse from rgw display all object.
Thanks.
Updated by hoan nv over 5 years ago
I enabled elaticsearch log
curl -XPUT '10.5.8.176:9200/rgw-ss-5fc11232/_settings' -H 'Content-Type: application/json' -d '{ "index.indexing.slowlog.threshold.index.warn": "0ms", "index.indexing.slowlog.threshold.index.info": "0ms", "index.indexing.slowlog.threshold.index.debug": "0ms", "index.indexing.slowlog.threshold.index.trace": "0ms", "index.indexing.slowlog.level": "trace", "index.search.slowlog.threshold.fetch.warn": "0ms", "index.search.slowlog.threshold.fetch.info": "0ms", "index.search.slowlog.threshold.fetch.debug": "0ms", "index.search.slowlog.threshold.fetch.trace": "0ms", "index.indexing.slowlog.source": "1000" }'
elasticsearch log while i am use curl to query elasticsearch
[2018-11-28T16:44:58,952][WARN ][index.search.slowlog.fetch] [aQHdoZA] [rgw-ss-5fc11232][4] took[728.6micros], took_millis[0], total_hits[1], types[], stats[], search_type[QUERY_THEN_FETCH], total_shards[10], source[{"query":{"bool":{"must":[{"term":{"bucket":{"value":"buck","boost":1.0}}},{"bool":{"must":[{"term":{"permissions":{"value":"hoannv","boost":1.0}}},{"range":{"name":{"from":"foo","to":null,"include_lower":false,"include_upper":true,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}}], id[],
elasticsearch log while i am use obo to query
[2018-11-28T16:45:19,930][WARN ][index.search.slowlog.fetch] [aQHdoZA] [rgw-ss-5fc11232][7] took[685.8micros], took_millis[0], total_hits[1], types[], stats[], search_type[QUERY_THEN_FETCH], total_shards[10], source[{"size":100}], id[],
Thanks
Updated by hoan nv over 5 years ago
I tested on mimic, rgw not return correct data, it return all result.
On luminous, it return correct data.
Thanks.
Updated by Yehuda Sadeh about 5 years ago
- Status changed from Need More Info to Fix Under Review
- Priority changed from Normal to High
- Backport set to mimic
This fixes it:
https://github.com/ceph/ceph/pull/26103
Updated by Yehuda Sadeh about 5 years ago
- Status changed from Fix Under Review to Pending Backport
Updated by Nathan Cutler about 5 years ago
- Copied to Backport #38194: mimic: Radosgw elastic search sync module not working properly (all result same) added
Updated by Nathan Cutler about 5 years ago
- Backport changed from mimic to mimic luminous
Updated by Nathan Cutler about 5 years ago
- Copied to Backport #38694: luminous: Radosgw elastic search sync module not working properly (all result same) added
Updated by Nathan Cutler about 5 years ago
- Status changed from Pending Backport to Resolved