Bug #24938
openluminous: rados listomapkeys & listomapvals don't return data.
0%
Description
Hi,
rados listomapkeys & rados listomapvals don't return data when running Luminous, tested on 12.2.4 and 12.2.6:
# rados -p default.rgw.buckets.index listomapkeys .dir.aac73a5b-8198-4cd8-9935-6f9c56a0d3f6.71829993.10 # # rados -p default.rgw.buckets.index listomapvals .dir.aac73a5b-8198-4cd8-9935-6f9c56a0d3f6.71829993.10 #
Expected behavior as per jewel:
# rados -p default.rgw.buckets.index listomapkeys .dir.d7b3188d-c51f-44a3-97e1-638fc4e43f6e.7419712.6230 out.mp4 # # rados -p default.rgw.buckets.index listomapvals .dir.d7b3188d-c51f-44a3-97e1-638fc4e43f6e.7419712.6230 out.mp4 value (238 bytes) : 00000000 08 03 e8 00 00 00 07 00 00 00 6f 75 74 2e 6d 70 |..........out.mp| 00000010 34 cb 0a 00 00 00 00 00 00 01 04 03 77 00 00 00 |4...........w...| 00000020 01 fc 90 29 00 00 00 00 00 4b c8 4b 59 5d 7c f7 |...).....K.KY]|.| 00000030 36 20 00 00 00 65 63 36 36 34 32 35 31 65 38 39 |6 ...ec664251e89| 00000040 64 33 62 31 61 62 66 32 61 66 38 32 61 65 37 66 |d3b1abf2af82ae7f| ... ... ...
Updated by Greg Farnum almost 6 years ago
Did you check that this bucket actually has any entries? These commands are tested in our suite.
Updated by Magnus Grönlund almost 6 years ago
Sorry that was just a single example to keep it short. listomapkeys doesn't return any data for any bucket in this cluster or our test cluster.
The ceph-cluster is a production cluster that has been in production for more than a year and contains 13TB of user data. It was recently upgraded from jewel to luminous.
I discovered the issue when I tested the "Get OMAP Key/Value Size" script from https://ceph.com/geen-categorie/get-omap-keyvalue-size/, at first I thought it was something wrong with the script since it only returned zeros for every object in every pool, but I soon figured out that is was because listomapvals didn't return any data.
An expanded example using the default.rgw.log pool and the Get OMAP Key/Value Size script as an example: # rados df
POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR
.rgw.root 6876 18 0 54 0 0 0 6339 4651k 86 60416
default.rgw.buckets.data 193G 74915 0 224745 0 0 0 22886831 10946G 8625524 299G
default.rgw.buckets.index 0 68 0 204 0 0 0 273918033 262G 6154735 0
default.rgw.buckets.non-ec 0 0 0 0 0 0 0 0 0 0 0
default.rgw.control 0 8 0 24 0 0 0 0 0 0 0
default.rgw.data.root 48900 136 0 408 0 0 0 2830499 2323M 2198921 429M
default.rgw.gc 0 32 0 96 0 0 0 7461476 7868M 8139626 0
*default.rgw.log 187 239 0 717 0 0 0 122748591 119G 85615306 559k*
default.rgw.meta 98472k 259004 0 777012 0 0 0 0 0 6334810 1275M
default.rgw.users.keys 12 1 0 3 0 0 0 0 0 1 1024
default.rgw.users.swift 12 1 0 3 0 0 0 5 3072 1 1024
default.rgw.users.uid 26620 150 0 450 0 0 0 633983 615M 571489 126k
...
...
# rados -p default.rgw.log ls -
obj_delete_at_hint.0000000078
meta.history
obj_delete_at_hint.0000000070
obj_delete_at_hint.0000000104
meta.log.128a376b-8807-4e19-9ddc-8220fd50d7c1.56
obj_delete_at_hint.0000000026
meta.log.128a376b-8807-4e19-9ddc-8220fd50d7c1.22
obj_delete_at_hint.0000000028
meta.log.128a376b-8807-4e19-9ddc-8220fd50d7c1.34
meta.log.128a376b-8807-4e19-9ddc-8220fd50d7c1.43
meta.log.128a376b-8807-4e19-9ddc-8220fd50d7c1.49
---180 additional lines removed--
# bash ./get-omap-keyvalue-size.sh
object size_keys(kB) size_values(kB) total(kB) nr_keys nr_values
obj_delete_at_hint.0000000078 0 0 0 0 0
meta.history 0 0 0 0 0
obj_delete_at_hint.0000000070 0 0 0 0 0
obj_delete_at_hint.0000000104 0 0 0 0 0
meta.log.128a376b-8807-4e19-9ddc-8220fd50d7c1.56 0 0 0 0 0
obj_delete_at_hint.0000000026 0 0 0 0 0
meta.log.128a376b-8807-4e19-9ddc-8220fd50d7c1.22 0 0 0 0 0
obj_delete_at_hint.0000000028 0 0 0 0 0
meta.log.128a376b-8807-4e19-9ddc-8220fd50d7c1.34 0 0 0 0 0
meta.log.128a376b-8807-4e19-9ddc-8220fd50d7c1.43 0 0 0 0 0
meta.log.128a376b-8807-4e19-9ddc-8220fd50d7c1.49 0 0 0 0 0
---180 lines of 180 objects with all zeros removed---
And as I said, this can be repeated for every object in every pool in both this cluster and our small test cluster.
Any more information I should provide?
Updated by Dan van der Ster almost 6 years ago
This sounds familiar: http://tracker.ceph.com/issues/16211
We used the workaround to set and rm a dummy key/val and it revived the omaps in our rbd image headers (which disappeared in Luminous).