Ceph » RADOS

We're running Ceph Octopus 15.2.8 with the same problem. Our monitors ran out of space after enabling autoscale as osdmaps are not pruned until the leading monitor is restarted. Whilst the merging process is safe, in that it only decrements pg_num again once misplaced objected are moved around, due to it not trimming osdmaps as it regains full health it leads to situations were clusters would be inaccessible due to monitors running out of space.

Restarting the leading monitor during a merge does not trim the osdmap, how would one recover in a situation where monitors are offline due to them having run out of space?

We have another pool that we would also like to reduce placement group numbers on and can reproduce the problem at will by simply reducing pg_num by 100 (was 1024, target is 64). For example if the current pg_num is 800, cluster is healthy and monitors are using <3 GiB:

[admin@kvm7b ~]# ceph osd pool set ec_hdd pg_num 700

After some hours everything completes but store grows to 23 GiB:

After reducing from 800 to 700 pg_num:

[admin@kvm7b ~]# ceph osd dump | grep ec_hdd
pool 8 'ec_hdd' erasure profile ec32_hdd size 5 min_size 4 crush_rule 3 object_hash rjenkins pg_num 700 pgp_num 700 autoscale_mode warn last_change 77572 lfor 384/77572/77570 flags hashpspool,ec_overwrites,selfmanaged_snaps tiers 10 read_tier 10 write_tier 10 stripe_width 12288 application rbd

[admin@kvm7b ~]# ceph -s
  cluster:
    id:     cf07a431-7d5c-45df-ab7d-7c92ea7b5cd1
    health: HEALTH_WARN
            mons kvm7a,kvm7b,kvm7e are using a lot of disk space
            1 pool(s) have non-power-of-two pg_num
            2 pools have too many placement groups

  services:
    mon:         3 daemons, quorum kvm7a,kvm7b,kvm7e (age 80m)
    mgr:         kvm7e(active, since 3d), standbys: kvm7b, kvm7a
    mds:         cephfs:1 {0=kvm7b=up:active} 2 up:standby
    osd:         60 osds: 59 up (since 3d), 59 in (since 6M)
    rgw:         3 daemons active (radosgw.kvm7a, radosgw.kvm7b, radosgw.kvm7e)
    tcmu-runner: 12 daemons active (kvm7a:iscsi/vm-169-disk-2, kvm7b:iscsi/vm-169-disk-2, kvm7e:iscsi/vm-169-disk-2)

  task status:

  data:
    pools:   15 pools, 1629 pgs
    objects: 18.92M objects, 20 TiB
    usage:   40 TiB used, 268 TiB / 308 TiB avail
    pgs:     1626 active+clean
             3    active+clean+scrubbing

  io:
    client:   8.1 KiB/s rd, 693 KiB/s wr, 7 op/s rd, 49 op/s wr

[admin@kvm7b ~]# ceph daemon osd.200 status
{
    "cluster_fsid": "cf07a431-7d5c-45df-ab7d-7c92ea7b5cd1",
    "osd_fsid": "71d27d56-59bf-4b89-ab19-a43602fee8cf",
    "whoami": 200,
    "state": "active",
    "oldest_map": 70170,
    "newest_map": 77656,
    "num_pgs": 41
}

[admin@kvm7b ~]# ceph report | jq .osdmap_first_committed
report 2501770158
70170

[admin@kvm7b ~]# ceph report | jq .osdmap_last_committed
report 2229346450
77656

[admin@kvm7b ~]# ceph report | grep "min_last_epoch_clean" 
report 535617613
        "min_last_epoch_clean": 0,

[admin@kvm7b ~]# ceph report | jq '.osdmap_manifest.pinned_maps | length'
report 1861220632
0

[admin@kvm7b ~]# ceph report | jq .osdmap_clean_epochs
report 1705570990
{
  "min_last_epoch_clean": 0,
  "last_epoch_clean": {
    "per_pool": []
  },
  "osd_epochs": []
}

[admin@kvm7b ~]# du -sh /var/lib/ceph/mon/ceph-kvm7b/store.db
23G     /var/lib/ceph/mon/ceph-kvm7b/store.db

After then running 'systemctl restart ceph-mon@kvm7a/b/c' it immediately starts trimming. 2 minutes later:

[admin@kvm7b ~]# du -sh /var/lib/ceph/mon/ceph-kvm7b/store.db
1.8G    /var/lib/ceph/mon/ceph-kvm7b/store.db

[admin@kvm7b ~]# ceph daemon osd.200 status
{
    "cluster_fsid": "cf07a431-7d5c-45df-ab7d-7c92ea7b5cd1",
    "osd_fsid": "71d27d56-59bf-4b89-ab19-a43602fee8cf",
    "whoami": 200,
    "state": "active",
    "oldest_map": 70170,
    "newest_map": 77656,
    "num_pgs": 41
}
[admin@kvm7b ~]# ceph report | jq .osdmap_first_committed
report 1859194375
77156
[admin@kvm7b ~]# ceph report | jq .osdmap_last_committed
report 3598010931
77656
[admin@kvm7b ~]# ceph report | grep "min_last_epoch_clean" 
report 3944254309
        "min_last_epoch_clean": 77655,
[admin@kvm7b ~]# ceph report | jq '.osdmap_manifest.pinned_maps | length'
report 1361140713
0
[admin@kvm7b ~]# ceph report | jq .osdmap_clean_epochs
report 2361317492
{
  "min_last_epoch_clean": 0,
  "last_epoch_clean": {
    "per_pool": []
  },
  "osd_epochs": []
}

We got a little relief by reducing mon_osdmap_full_prune_min from the default 10,000 to 1,000 but osdmaps still grew to 15K+ whilst merging. kvm7b was the first to fill to 100% and couldn't be started again until we'd cleared all old log files. This gave it 2% free disc space so the service still wouldn't start. Running just a 'list' cleared up another GiB but running compact shrunk the store.db folder from 54 GiB to 1.4 GiB instantly:

[root@kvm7b ~]# systemctl stop ceph-mon@kvm7b
[root@kvm7b ~]# systemctl reset-failed
[root@kvm7b ~]# sudo -u ceph ceph-kvstore-tool rocksdb /var/lib/ceph/mon/kvm7b/store.db list
[root@kvm7b ~]# sudo -u ceph ceph-kvstore-tool rocksdb /var/lib/ceph/mon/kvm7b/store.db compact
[root@kvm7b ~]# systemctl start ceph-mon@kvm7b

Reducing mon_osdmap_full_prune_min results in the rocksdb monitor store being pruned constantly, resulting in the saw tooth pattern in the grows below, but overall utilisation still grows until the leading monitor is restarted. This doesn't release all space though and on occasion appears to result in monitors not purging osdmaps resulting in the sudden increase which started at about 22:15:

Nautilus is EOL