Project

General

Profile

Actions
Copy link

Bug #23432

open

Elastic Search Indexing fails for encrypted objects

Added by Vik Tara about 6 years ago. Updated about 5 years ago.

Status:
In Progress
Priority:
Normal
Assignee:
Vik Tara
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

When using encrypted objects (server side encryption using barbican) indexing with elastic search fails.

Ceph throws errors like:

2018-03-13 21:20:40.187236 7ff8bfd49700 1 meta sync: ERROR: failed to read mdlog info with (2) No such file or directory
2018-03-13 21:20:43.238247 7ff8ab520700 0 RGWStatRemoteObjCR() callback returned -22
2018-03-13 21:20:43.238266 7ff8ab520700 0 data sync: ERROR: failed to sync object: propcodms:ee560b67-c330-4fd0-af50-aefff93735d2.4163.1:6/testr/
2018-03-13 21:20:43.278427 7ff8ab520700 0 data sync: ERROR: a sync operation returned error
2018-03-13 21:20:43.339337 7ff8ab520700 0 RGWStatRemoteObjCR() callback returned -22
2018-03-13 21:20:43.339357 7ff8ab520700 0 data sync: ERROR: failed to sync object: propcodms:ee560b67-c330-4fd0-af50-aefff93735d2.4163.1:1/new4
2018-03-13 21:20:43.339453 7ff8ab520700 0 data sync: ERROR: failure in sync, backing out (sync_status=-22)
2018-03-13 21:20:43.456843 7ff8ab520700 0 data sync: ERROR: a sync operation returned error
2018-03-13 21:20:43.456873 7ff8ab520700 0 data sync: ERROR: failure in sync, backing out (sync_status=-22)

and elasticsearch says:

org.elasticsearch.index.mapper.MapperParsingException: failed to parse

Caused by: com.fasterxml.jackson.core.JsonParseException: Invalid UTF-8 middle byte 0x55
at [Source: org.elasticsearch.common.bytes.BytesReference$MarkSupportingStreamInputWrapper@4b162b44; line: 1, column: 323]

Actions
Copy link
 #1

Updated by Vik Tara about 6 years ago

Taking a look at the request that ceph sends to elastic we see this:

failed to execute bulk item (index) BulkShardRequest [[brontodms]4] containing [index {[brontodms][object][ee560b67-c330-4fd0-af50-aefff93735d2.4163.1:newone:null], source[{"bucket":"acmedms","name":"newone","instance":"null","versioned_epoch":0,"owner":{"id":"acme","display_name":"Acme Admin"},"permissions":["acme"],"meta":{"size":8,"mtime":"2018-03-13T14:53:38.447Z","content_type":"text/plain","crypt.keyid":"cd4d0092-41dd-4144-b2ce-4be41c0b558","crypt.keysel":"\u0004\u0002v%�{l�QN�Zo��\u0002���İ\u000bK`J'��T�v","crypt.mode":"SSE-KM","etag":"eb1a3227cdc3fedbaec2fe38bf6c044a","tail_tag":"ee560b67-c330-4fd0-af50-aefff93735d2.54110.136607","x-amz-content-sha256":"12a61f4e173fb3a11c05d6471f74728f76231b4a5fcd9667cef3af87a3ae4dc2","x-amz-date":"20180313T145337Z","x-amz-server-side-encryption":"aws:kms","x-amz-server-side-encryption-aws-kms-key-id":"cd4d0092-41dd-4144-b2ce-4be41c0b5582","x-amz-storage-class":"STANDARD","custom-string":[{"name":"s3cmd-attrs","value":"uid:1000/gname:amardeep/uname:amardeep/gid:1000/mode:33204/mtime:1518453135/atime:1520936897/md5:eb1a3227cdc3fedbaec2fe38bf6c044a/ctime:1518453135"}]}}]}]

Which shows incorrect encoding here:
"crypt.keysel":"\u0004\u0002v%�{l�QN�Zo��\u0002���İ\u000bK`J'��T�v"

Caused by:
https://github.com/ceph/ceph/blob/master/src/rgw/rgw_crypt.cc#L1120

Working on a fix for this.

Actions
Copy link
 #2

Updated by Abhishek Lekshmanan about 6 years ago

  • Status changed from New to In Progress
  • Assignee set to Vik Tara
Actions
Copy link
 #3

Updated by Vik Tara about 5 years ago

Here's a patch for this, it's been sitting with us for a while!

---
 src/rgw/rgw_crypt.cc | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/src/rgw/rgw_crypt.cc b/src/rgw/rgw_crypt.cc
index 81a84ad698..db3677d603 100644
--- a/src/rgw/rgw_crypt.cc
+++ b/src/rgw/rgw_crypt.cc
@@ -826,10 +826,13 @@ int RGWPutObj_BlockEncrypt::throttle_data(void *handle,

 std::string create_random_key_selector(CephContext * const cct) {
   char random[AES_256_KEYSIZE];
-  if (get_random_bytes(&random[0], sizeof(random)) != 0) {
-    ldout(cct, 0) << "ERROR: cannot get_random_bytes. " << dendl;
-    for (char& v:random) v=rand();
-  }
+  char* first = begin(random);
+  char* last = end(random);
+  do {
+    get_random_bytes(first, last - first);
+    first = find_if(first, last,
+                    [](char c) { return !isprint(c) || c == '"' || c == '\\'; });
+  } while (first != last);
   return std::string(random, sizeof(random));
 }

--
Actions
Copy link
 #4

Updated by Vik Tara about 5 years ago 

---
 src/rgw/rgw_crypt.cc | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/src/rgw/rgw_crypt.cc b/src/rgw/rgw_crypt.cc
index 81a84ad698..db3677d603 100644
--- a/src/rgw/rgw_crypt.cc
+++ b/src/rgw/rgw_crypt.cc
@@ -826,10 +826,13 @@ int RGWPutObj_BlockEncrypt::throttle_data(void *handle,

 std::string create_random_key_selector(CephContext * const cct) {
   char random[AES_256_KEYSIZE];
-  if (get_random_bytes(&random[0], sizeof(random)) != 0) {
-    ldout(cct, 0) << "ERROR: cannot get_random_bytes. " << dendl;
-    for (char& v:random) v=rand();
-  }
+  char* first = begin(random);
+  char* last = end(random);
+  do {
+    get_random_bytes(first, last - first);
+    first = find_if(first, last,
+                    [](char c) { return !isprint(c) || c == '"' || c == '\\'; });
+  } while (first != last);
   return std::string(random, sizeof(random));
 }

--
Actions
Copy link

Also available in: Atom PDF