Ceph » rbd

The reproduce steps as the below

[root@ceph-node01 ~]# ceph osd pool create test1 16
pool 'test1' created
[root@ceph-node01 ~]# rbd pool init test1
[root@ceph-node01 ~]# ceph osd pool set-quota test1 max_bytes $((1 * 1024 * 1024 * 1024))
set-quota max_bytes = 1073741824 for pool test1
[root@ceph-node01 ~]# rbd create --size 600 test1/img1g001 --thick-provision
Thick provisioning: 100% complete...done.
[root@ceph-node01 ~]# rbd create --size 600 test1/img1g002 --thick-provision
Thick provisioning: 100% complete...done.
[root@ceph-node01 ~]# rbd list test1 -l
NAME SIZE PARENT FMT PROT LOCK
img1g001 600 MiB 2
img1g002 600 MiB 2
[root@ceph-node01 ~]# ceph health detail
HEALTH_WARN 1 pool(s) full
[WRN] POOL_FULL: 1 pool(s) full
pool 'test1' is full (running out of quota)
[root@ceph-node01 ~]# rbd rm test1/img1g001
CTL+C
The rm action is hang ...

Hi Kevin,

What version of Ceph is installed on the client side, i.e. on the node where you are running this test on? What is the output of "rbd --version"?

The reason I ask is this was fixed in 16.2.8 and later releases with the caveat that the "problematic" remove should not be the first remove in that pool, see https://tracker.ceph.com/issues/52734. Your test case passes for me with that slight modification:

$ ceph osd pool create test1 16
pool 'test1' created
$ rbd pool init test1
$ ceph osd pool set-quota test1 max_bytes $((1 * 1024 * 1024 * 1024))
set-quota max_bytes = 1073741824 for pool test1
$ rbd create --size 1 test1/dummy                                           <------
$ rbd rm test1/dummy                                                        <------
Removing image: 100% complete...done.
$ rbd create --size 600 test1/img1g001 --thick-provision
Thick provisioning: 100% complete...done.
$ rbd create --size 600 test1/img1g002 --thick-provision
Thick provisioning: 100% complete...done.
$ ceph health detail
HEALTH_WARN 1 pool(s) full
[WRN] POOL_FULL: 1 pool(s) full
    pool 'test1' is full (running out of quota)
$ rbd rm test1/img1g001
Removing image: 100% complete...done.
$ rbd rm test1/img1g002
Removing image: 100% complete...done.
$ ceph health detail
HEALTH_OK

The need for the "dummy" remove is just an oversight -- it is only needed in the "reached quota" case, not when the pool actually becomes full. In practice, people tend to run into ENOSPC ("No space left on device") far more often than they run into EDQUOT ("Disk quota exceeded"), possibly because pool quota is not a widely used feature. Nevertheless, I'm going to address it ASAP.

kevin huang wrote:

[root@ceph-node01 ~]# ceph osd pool set-quota test1 max_bytes $((1 * 1024 * 1024 * 1024))
set-quota max_bytes = 1073741824 for pool test1
[root@ceph-node01 ~]# rbd create --size 600 test1/img1g001 --thick-provision
Thick provisioning: 100% complete...done.
[root@ceph-node01 ~]# rbd create --size 600 test1/img1g002 --thick-provision
Thick provisioning: 100% complete...done.

Also note that you may need to CTRL+C this command as well. Because pool quota is not precise -- it lags behind by a few seconds, sometimes you would be able to write 600M + 600M = 1.2G into a pool with max_bytes set to 1G and sometimes not.