Project

General

Profile

Actions
Copy link

Bug #24938

open

luminous: rados listomapkeys & listomapvals don't return data.

Added by Magnus Grönlund almost 6 years ago. Updated almost 6 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
Ceph - v12.2.4
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Hi,

rados listomapkeys & rados listomapvals don't return data when running Luminous, tested on 12.2.4 and 12.2.6:

# rados -p default.rgw.buckets.index listomapkeys .dir.aac73a5b-8198-4cd8-9935-6f9c56a0d3f6.71829993.10 
#

# rados -p default.rgw.buckets.index listomapvals .dir.aac73a5b-8198-4cd8-9935-6f9c56a0d3f6.71829993.10 
#

Expected behavior as per jewel:

# rados -p default.rgw.buckets.index listomapkeys .dir.d7b3188d-c51f-44a3-97e1-638fc4e43f6e.7419712.6230
out.mp4
#

# rados -p default.rgw.buckets.index listomapvals .dir.d7b3188d-c51f-44a3-97e1-638fc4e43f6e.7419712.6230
out.mp4
value (238 bytes) :
00000000  08 03 e8 00 00 00 07 00  00 00 6f 75 74 2e 6d 70  |..........out.mp|
00000010  34 cb 0a 00 00 00 00 00  00 01 04 03 77 00 00 00  |4...........w...|
00000020  01 fc 90 29 00 00 00 00  00 4b c8 4b 59 5d 7c f7  |...).....K.KY]|.|
00000030  36 20 00 00 00 65 63 36  36 34 32 35 31 65 38 39  |6 ...ec664251e89|
00000040  64 33 62 31 61 62 66 32  61 66 38 32 61 65 37 66  |d3b1abf2af82ae7f|
...
...
...

Actions
Copy link
 #1

Updated by Greg Farnum almost 6 years ago

Did you check that this bucket actually has any entries? These commands are tested in our suite.

Actions
Copy link
 #2

Updated by Magnus Grönlund almost 6 years ago

Sorry that was just a single example to keep it short. listomapkeys doesn't return any data for any bucket in this cluster or our test cluster.

The ceph-cluster is a production cluster that has been in production for more than a year and contains 13TB of user data. It was recently upgraded from jewel to luminous.
I discovered the issue when I tested the "Get OMAP Key/Value Size" script from https://ceph.com/geen-categorie/get-omap-keyvalue-size/, at first I thought it was something wrong with the script since it only returned zeros for every object in every pool, but I soon figured out that is was because listomapvals didn't return any data.

An expanded example using the default.rgw.log pool and the Get OMAP Key/Value Size script as an example:
# rados df
POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR
.rgw.root 6876 18 0 54 0 0 0 6339 4651k 86 60416
default.rgw.buckets.data 193G 74915 0 224745 0 0 0 22886831 10946G 8625524 299G
default.rgw.buckets.index 0 68 0 204 0 0 0 273918033 262G 6154735 0
default.rgw.buckets.non-ec 0 0 0 0 0 0 0 0 0 0 0
default.rgw.control 0 8 0 24 0 0 0 0 0 0 0
default.rgw.data.root 48900 136 0 408 0 0 0 2830499 2323M 2198921 429M
default.rgw.gc 0 32 0 96 0 0 0 7461476 7868M 8139626 0
*default.rgw.log 187 239 0 717 0 0 0 122748591 119G 85615306 559k*
default.rgw.meta 98472k 259004 0 777012 0 0 0 0 0 6334810 1275M
default.rgw.users.keys 12 1 0 3 0 0 0 0 0 1 1024
default.rgw.users.swift 12 1 0 3 0 0 0 5 3072 1 1024
default.rgw.users.uid 26620 150 0 450 0 0 0 633983 615M 571489 126k
...
...

# rados -p default.rgw.log ls -
obj_delete_at_hint.0000000078
meta.history
obj_delete_at_hint.0000000070
obj_delete_at_hint.0000000104
meta.log.128a376b-8807-4e19-9ddc-8220fd50d7c1.56
obj_delete_at_hint.0000000026
meta.log.128a376b-8807-4e19-9ddc-8220fd50d7c1.22
obj_delete_at_hint.0000000028
meta.log.128a376b-8807-4e19-9ddc-8220fd50d7c1.34
meta.log.128a376b-8807-4e19-9ddc-8220fd50d7c1.43
meta.log.128a376b-8807-4e19-9ddc-8220fd50d7c1.49
---180 additional lines removed--

# bash ./get-omap-keyvalue-size.sh
object size_keys(kB) size_values(kB) total(kB) nr_keys nr_values
obj_delete_at_hint.0000000078 0 0 0 0 0
meta.history 0 0 0 0 0
obj_delete_at_hint.0000000070 0 0 0 0 0
obj_delete_at_hint.0000000104 0 0 0 0 0
meta.log.128a376b-8807-4e19-9ddc-8220fd50d7c1.56 0 0 0 0 0
obj_delete_at_hint.0000000026 0 0 0 0 0
meta.log.128a376b-8807-4e19-9ddc-8220fd50d7c1.22 0 0 0 0 0
obj_delete_at_hint.0000000028 0 0 0 0 0
meta.log.128a376b-8807-4e19-9ddc-8220fd50d7c1.34 0 0 0 0 0
meta.log.128a376b-8807-4e19-9ddc-8220fd50d7c1.43 0 0 0 0 0
meta.log.128a376b-8807-4e19-9ddc-8220fd50d7c1.49 0 0 0 0 0
---180 lines of 180 objects with all zeros removed---

And as I said, this can be repeated for every object in every pool in both this cluster and our small test cluster.
Any more information I should provide?

Actions
Copy link
 #3

Updated by Dan van der Ster almost 6 years ago

This sounds familiar: http://tracker.ceph.com/issues/16211
We used the workaround to set and rm a dummy key/val and it revived the omaps in our rbd image headers (which disappeared in Luminous).

Actions
Copy link

Also available in: Atom PDF