Actions
Bug #61761
open[crimson] number of objects in the pool reduced after OSD restart
Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:
0%
Source:
Tags:
good-first-issue
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
Underlying issue could be similar to https://tracker.ceph.com/issues/61748
This could become a duplicate; adding for tracking purpose.
Observed on a Reef based Crimson cluster build with this image - https://shaman.ceph.com/builds/ceph/main/ff8144fac0bdb12d803d6c3905e68584dd10bb19/crimson/347418/
Created a replicated pool named test_omap with default configuration
Wrote omap entries to this pool using this script from the client node - https://raw.githubusercontent.com/red-hat-storage/cephci/master/utility/generate_omap_entries.py
python3 generate_omap_entries.py --pool test_omap --start 0 --end 100 --key-count 100000
# date Wed Jun 21 02:33:57 UTC 2023 [ceph: root@dell-r640-039 /]# ceph df detail 2023-06-21T02:34:47.199+0000 7f12931f3700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-06-21T02:34:47.200+0000 7f12931f3700 -1 WARNING: the following dangerous and experimental features are enabled: crimson --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED TOTAL 1.3 TiB 1.3 TiB 3.3 GiB 3.3 GiB 0.25 --- POOLS --- POOL ID PGS STORED (DATA) (OMAP) OBJECTS USED (DATA) (OMAP) %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR UNDER COMPR .mgr 1 1 0 B 0 B 0 B 0 0 B 0 B 0 B 0 426 GiB N/A N/A N/A 0 B 0 B test_bench 2 32 45 MiB 45 MiB 0 B 11.48k 45 MiB 45 MiB 0 B 0 426 GiB N/A N/A N/A 0 B 0 B test_bench_objs 3 32 53 MiB 53 MiB 0 B 13.62k 53 MiB 53 MiB 0 B 0 426 GiB N/A N/A N/A 0 B 0 B test_bench_objs_2 4 32 8.6 MiB 8.6 MiB 0 B 2.20k 8.6 MiB 8.6 MiB 0 B 0 426 GiB N/A N/A N/A 0 B 0 B test_omap 5 32 0 B 0 B 0 B 100 0 B 0 B 0 B 0 426 GiB N/A N/A N/A 0 B 0 B [ceph: root@dell-r640-039 /]# exit exit [root@dell-r640-039 ~]# systemctl restart ceph-203a849c-0fcb-11ee-918b-78ac443b3604@osd.5.service [root@dell-r640-039 ~]# systemctl -a | grep osd dev-ceph\x2d524ff1ec\x2d924b\x2d49ce\x2db07c\x2da5088cb8efe5-osd\x2dblock\x2df5c33760\x2db36c\x2d42fe\x2d8f7e\x2dee4c6271a912.device loaded active plugged /dev/ceph-524ff1ec-924b-49ce-b07c-a5088cb8efe5/osd-block-f5c33760-b36c-42fe-8f7e-ee4c6271a912 dev-ceph\x2dc4098c30\x2dd657\x2d4e17\x2d9c81\x2d0999de38a731-osd\x2dblock\x2ddff805cc\x2d5cac\x2d484e\x2d8d48\x2de2d836da55e7.device loaded active plugged /dev/ceph-c4098c30-d657-4e17-9c81-0999de38a731/osd-block-dff805cc-5cac-484e-8d48-e2d836da55e7 dev-ceph\x2df9f8632a\x2d363e\x2d420a\x2dbf22\x2db279755e2687-osd\x2dblock\x2d8d6dbea5\x2dc0c7\x2d4aed\x2d9b84\x2d2ef5ff0e055a.device loaded active plugged /dev/ceph-f9f8632a-363e-420a-bf22-b279755e2687/osd-block-8d6dbea5-c0c7-4aed-9b84-2ef5ff0e055a dev-disk-by\x2did-dm\x2dname\x2dceph\x2d\x2d524ff1ec\x2d\x2d924b\x2d\x2d49ce\x2d\x2db07c\x2d\x2da5088cb8efe5\x2dosd\x2d\x2dblock\x2d\x2df5c33760\x2d\x2db36c\x2d\x2d42fe\x2d\x2d8f7e\x2d\x2dee4c6271a912.device loaded active plugged /dev/disk/by-id/dm-name-ceph--524ff1ec--924b--49ce--b07c--a5088cb8efe5-osd--block--f5c33760--b36c--42fe--8f7e--ee4c6271a912 dev-disk-by\x2did-dm\x2dname\x2dceph\x2d\x2dc4098c30\x2d\x2dd657\x2d\x2d4e17\x2d\x2d9c81\x2d\x2d0999de38a731\x2dosd\x2d\x2dblock\x2d\x2ddff805cc\x2d\x2d5cac\x2d\x2d484e\x2d\x2d8d48\x2d\x2de2d836da55e7.device loaded active plugged /dev/disk/by-id/dm-name-ceph--c4098c30--d657--4e17--9c81--0999de38a731-osd--block--dff805cc--5cac--484e--8d48--e2d836da55e7 dev-disk-by\x2did-dm\x2dname\x2dceph\x2d\x2df9f8632a\x2d\x2d363e\x2d\x2d420a\x2d\x2dbf22\x2d\x2db279755e2687\x2dosd\x2d\x2dblock\x2d\x2d8d6dbea5\x2d\x2dc0c7\x2d\x2d4aed\x2d\x2d9b84\x2d\x2d2ef5ff0e055a.device loaded active plugged /dev/disk/by-id/dm-name-ceph--f9f8632a--363e--420a--bf22--b279755e2687-osd--block--8d6dbea5--c0c7--4aed--9b84--2ef5ff0e055a dev-mapper-ceph\x2d\x2d524ff1ec\x2d\x2d924b\x2d\x2d49ce\x2d\x2db07c\x2d\x2da5088cb8efe5\x2dosd\x2d\x2dblock\x2d\x2df5c33760\x2d\x2db36c\x2d\x2d42fe\x2d\x2d8f7e\x2d\x2dee4c6271a912.device loaded active plugged /dev/mapper/ceph--524ff1ec--924b--49ce--b07c--a5088cb8efe5-osd--block--f5c33760--b36c--42fe--8f7e--ee4c6271a912 dev-mapper-ceph\x2d\x2dc4098c30\x2d\x2dd657\x2d\x2d4e17\x2d\x2d9c81\x2d\x2d0999de38a731\x2dosd\x2d\x2dblock\x2d\x2ddff805cc\x2d\x2d5cac\x2d\x2d484e\x2d\x2d8d48\x2d\x2de2d836da55e7.device loaded active plugged /dev/mapper/ceph--c4098c30--d657--4e17--9c81--0999de38a731-osd--block--dff805cc--5cac--484e--8d48--e2d836da55e7 dev-mapper-ceph\x2d\x2df9f8632a\x2d\x2d363e\x2d\x2d420a\x2d\x2dbf22\x2d\x2db279755e2687\x2dosd\x2d\x2dblock\x2d\x2d8d6dbea5\x2d\x2dc0c7\x2d\x2d4aed\x2d\x2d9b84\x2d\x2d2ef5ff0e055a.device loaded active plugged /dev/mapper/ceph--f9f8632a--363e--420a--bf22--b279755e2687-osd--block--8d6dbea5--c0c7--4aed--9b84--2ef5ff0e055a ceph-203a849c-0fcb-11ee-918b-78ac443b3604@osd.2.service loaded active running Ceph osd.2 for 203a849c-0fcb-11ee-918b-78ac443b3604 ceph-203a849c-0fcb-11ee-918b-78ac443b3604@osd.5.service loaded active running Ceph osd.5 for 203a849c-0fcb-11ee-918b-78ac443b3604 ceph-203a849c-0fcb-11ee-918b-78ac443b3604@osd.8.service loaded active running Ceph osd.8 for 203a849c-0fcb-11ee-918b-78ac443b3604 [root@dell-r640-039 ~]# cephadm shell Inferring fsid 203a849c-0fcb-11ee-918b-78ac443b3604 Inferring config /var/lib/ceph/203a849c-0fcb-11ee-918b-78ac443b3604/mon.dell-r640-039/config Using ceph image with id 'e50e16176f87' and tag 'ff8144fac0bdb12d803d6c3905e68584dd10bb19-crimson' created on 2023-06-20 23:00:38 +0000 UTC quay.ceph.io/ceph-ci/ceph@sha256:87239ee25da7bd08962ffe94a73ff5429fa4391108bad2efed02617933edded1 [ceph: root@dell-r640-039 /]# ceph df detail 2023-06-21T02:36:51.792+0000 7fdb206f6700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-06-21T02:36:51.793+0000 7fdb206f6700 -1 WARNING: the following dangerous and experimental features are enabled: crimson --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED TOTAL 1.3 TiB 1.3 TiB 3.2 GiB 3.2 GiB 0.24 --- POOLS --- POOL ID PGS STORED (DATA) (OMAP) OBJECTS USED (DATA) (OMAP) %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR UNDER COMPR .mgr 1 1 0 B 0 B 0 B 0 0 B 0 B 0 B 0 426 GiB N/A N/A N/A 0 B 0 B test_bench 2 32 38 MiB 38 MiB 0 B 9.66k 38 MiB 38 MiB 0 B 0 426 GiB N/A N/A N/A 0 B 0 B test_bench_objs 3 32 47 MiB 47 MiB 0 B 11.97k 47 MiB 47 MiB 0 B 0 426 GiB N/A N/A N/A 0 B 0 B test_bench_objs_2 4 32 7.8 MiB 7.8 MiB 0 B 1.99k 7.8 MiB 7.8 MiB 0 B 0 426 GiB N/A N/A N/A 0 B 0 B test_omap 5 32 0 B 0 B 0 B 98 0 B 0 B 0 B 0 426 GiB N/A N/A N/A 0 B 0 B [ceph: root@dell-r640-039 /]# ceph df detail 2023-06-21T02:37:41.698+0000 7f80b552b700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-06-21T02:37:41.699+0000 7f80b552b700 -1 WARNING: the following dangerous and experimental features are enabled: crimson --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED TOTAL 1.3 TiB 1.3 TiB 3.2 GiB 3.2 GiB 0.24 --- POOLS --- POOL ID PGS STORED (DATA) (OMAP) OBJECTS USED (DATA) (OMAP) %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR UNDER COMPR .mgr 1 1 0 B 0 B 0 B 0 0 B 0 B 0 B 0 426 GiB N/A N/A N/A 0 B 0 B test_bench 2 32 38 MiB 38 MiB 0 B 9.66k 38 MiB 38 MiB 0 B 0 426 GiB N/A N/A N/A 0 B 0 B test_bench_objs 3 32 47 MiB 47 MiB 0 B 11.97k 47 MiB 47 MiB 0 B 0 426 GiB N/A N/A N/A 0 B 0 B test_bench_objs_2 4 32 7.8 MiB 7.8 MiB 0 B 1.99k 7.8 MiB 7.8 MiB 0 B 0 426 GiB N/A N/A N/A 0 B 0 B test_omap 5 32 0 B 0 B 0 B 98 0 B 0 B 0 B 0 426 GiB N/A N/A N/A 0 B 0 B
As seen from the 'ceph df detail' stats, number of objects in the pools were higher before the OSD 5 service was restarted, and upon restart, the numbers decreased
Cluster version
# cephadm shell -- ceph version Inferring fsid 203a849c-0fcb-11ee-918b-78ac443b3604 Inferring config /var/lib/ceph/203a849c-0fcb-11ee-918b-78ac443b3604/mon.dell-r640-039/config Using ceph image with id 'e50e16176f87' and tag 'ff8144fac0bdb12d803d6c3905e68584dd10bb19-crimson' created on 2023-06-20 23:00:38 +0000 UTC quay.ceph.io/ceph-ci/ceph@sha256:87239ee25da7bd08962ffe94a73ff5429fa4391108bad2efed02617933edded1 2023-06-21T02:19:02.294+0000 7f15d7a4f700 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-06-21T02:19:02.295+0000 7f15d7a4f700 -1 WARNING: the following dangerous and experimental features are enabled: crimson ceph version 18.0.0-4505-gff8144fa (ff8144fac0bdb12d803d6c3905e68584dd10bb19) reef (dev)
Cluster logs - http://magna002.ceph.redhat.com/ceph-qe-logs/harsh/crimson_osd_restart/
Updated by Harsh Kumar 11 months ago
Consequently the data also gets removed with those objects.
Updated by Matan Breizman about 1 month ago
- Tags changed from crimson to crimson, good-first-issue
Updated by Matan Breizman about 1 month ago
- Tags changed from crimson, good-first-issue to good-first-issue
Actions