Ceph » RADOS

In Jewel (ceph-10.2.1+git.1464009581.b8ee9e4-1.2), flushing a cache pool appears to work for all objects except rbd related objects (i.e. rbd_directory and one rbd_header per image.) I believe this is working as intended (i.e. some rbd objects must remain in the cache), but the cache-flush-evict-all will report an error. For example,

1. rados -p swimming-cache ls
   rbd_header.1153238e1f29
   rbd_directory
   rbd_header.5ec42ae8944a

1. rados -p swimming-cache cache-flush-evict-all
   rbd_header.1153238e1f29
   failed to flush /rbd_header.1153238e1f29: (16) Device or resource busy
   rbd_directory
   failed to flush /rbd_directory: (16) Device or resource busy
   rbd_header.5ec42ae8944a
   failed to flush /rbd_header.5ec42ae8944a: (16) Device or resource busy
   error from cache-flush-evict-all: (1) Operation not permitted

1. rados -p swimming-cache ls
   rbd_header.1153238e1f29
   rbd_directory
   rbd_header.5ec42ae8944a

To reproduce:

ceph osd erasure-code-profile set three-one k=3 m=1
ceph osd pool create swimming 128 erasure three-one
rbd -p swimming create media --size=2048
ceph osd pool create swimming-cache 256 256
ceph osd tier add swimming swimming-cache
ceph osd tier cache-mode swimming-cache writeback
ceph osd tier set-overlay swimming swimming-cache
ceph osd pool set swimming-cache hit_set_type bloom
ceph osd pool set swimming-cache hit_set_period 4
ceph osd pool set swimming-cache hit_set_count 1200

All other objects in the swimming-cache pool are evicted successfully to the backing store swimming pool. Should cache-flush-evict-all ignore these objects and report success or should these objects be pushed to the backing store pool as well?