Ceph » bluestore

Reported in ceph-users

I am trying to apply the resharding to a containerized OSD (16.2.4) as described here:

https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/#rocksdb-sharding

# ceph osd set noout
# ceph orch daemon stop osd.13
# cephadm shell --name osd.13
# ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-13 --sharding="m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L P" reshard
2021-07-08T15:12:03.392+0000 7f2e2173b3c0 -1 rocksdb: prepare_for_reshard failure parsing column options: block_cache={type=binned_lru}
error resharding: (22) Invalid argument
# exit
# ceph orch daemon start osd.13
# ceph osd unset noout

The OSD cannot start any more and has these errors in its log:

Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.293+0000 7efc32db4080  0 osd.13:7.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196)
Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.293+0000 7efc32db4080  1 bluestore(/var/lib/ceph/osd/ceph-13) _mount path /var/lib/ceph/osd/ceph-13
Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.293+0000 7efc32db4080  0 bluestore(/var/lib/ceph/osd/ceph-13) _open_db_and_around read-only:0 repair:0
Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.293+0000 7efc32db4080  1 bdev(0x564271210800 /var/lib/ceph/osd/ceph-13/block) open path /var/lib/ceph/osd/ceph-13/block
Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.293+0000 7efc32db4080  1 bdev(0x564271210800 /var/lib/ceph/osd/ceph-13/block) open size 107369988096 (0x18ffc00000, 100 GiB) block_size 4096 (4 KiB) non-rotational discard supported
Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.293+0000 7efc32db4080  1 bluestore(/var/lib/ceph/osd/ceph-13) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06
Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.293+0000 7efc32db4080  1 bdev(0x564271210c00 /var/lib/ceph/osd/ceph-13/block) open path /var/lib/ceph/osd/ceph-13/block
Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.293+0000 7efc32db4080  1 bdev(0x564271210c00 /var/lib/ceph/osd/ceph-13/block) open size 107369988096 (0x18ffc00000, 100 GiB) block_size 4096 (4 KiB) non-rotational discard supported
Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.293+0000 7efc32db4080  1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-13/block size 100 GiB
Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.293+0000 7efc32db4080  1 bluefs mount
Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.293+0000 7efc32db4080  1 bluefs _init_alloc shared, id 1, capacity 0x18ffc00000, block size 0x10000
Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.325+0000 7efc32db4080  1 bluefs mount shared_bdev_used = 0
Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.325+0000 7efc32db4080 -1 rocksdb: verify_sharding extra columns in rocksdb. rocksdb columns = [default,m-0,m-1,m-2,p-0,p-1,p-2] target columns = [reshardingXcommencingXlocked]
Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.325+0000 7efc32db4080 -1 bluestore(/var/lib/ceph/osd/ceph-13) _open_db erroring opening db:
Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.325+0000 7efc32db4080  1 bluefs umount
Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.325+0000 7efc32db4080  1 bdev(0x564271210c00 /var/lib/ceph/osd/ceph-13/block) close
Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.577+0000 7efc32db4080  1 bdev(0x564271210800 /var/lib/ceph/osd/ceph-13/block) close
Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.825+0000 7efc32db4080 -1 osd.13 0 OSD:init: unable to mount object store
Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.825+0000 7efc32db4080 -1  ** ERROR: osd init failed: (5) Input/output error