Ceph » RADOS

I found this problem with v0.80.7， but in master branch ,it's still exists.

Here are the steps to reproduce:

1. environment
   

   # ./ceph --version
   *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
   ceph version 0.80.7 (6c0127fcb58008793d3c8b62d925bc91963672a3)
   
   # ./ceph osd tree
   *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
   # id    weight  type name       up/down reweight
   -3      3       root cache
   -4      3               host host-cache
   3       1                       osd.3   up      1
   4       1                       osd.4   up      1
   5       1                       osd.5   up      1
   -1      3       root default
   -5      3               host ceph-test
   0       1                       osd.0   up      1
   1       1                       osd.1   up      1
   2       1                       osd.2   up      1
   
   # ./ceph osd crush rule dump
   *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
   [
       { "rule_id": 0,
         "rule_name": "replicated_ruleset",
         "ruleset": 0,
         "type": 1,
         "min_size": 1,
         "max_size": 10,
         "steps": [
               { "op": "take",
                 "item": -1,
                 "item_name": "default"},
               { "op": "choose_firstn",
                 "num": 0,
                 "type": "osd"},
               { "op": "emit"}]},
       { "rule_id": 1,
         "rule_name": "cache_ruleset",
         "ruleset": 1,
         "type": 1,
         "min_size": 1,
         "max_size": 10,
         "steps": [
               { "op": "take",
                 "item": -3,
                 "item_name": "cache"},
               { "op": "choose_firstn",
                 "num": 0,
                 "type": "osd"},
               { "op": "emit"}]}]
   
   # ./ceph osd pool create base 8                                                                                                
   # ./ceph osd pool create cache 8                                                                                               
   # ./ceph osd pool set cache crush_ruleset 1
   

2. with cache tier
   

   # echo 'abc' > test_rbd
   # ./rbd -p base import --image-format 2 test_rbd test_rbd
   # ./rbd snap create base/test_rbd@snap
   # ./rbd snap protect base/test_rbd@snap
   # ./rbd clone base/test_rbd@snap rbd/test_rbd_clone
   # ./rbd snap create rbd/test_rbd_clone@snap
   
   # ./rbd diff rbd/test_rbd_clone@snap
   Offset Length Type 
   0      4      data
   

3. with writeback cache tier
   

   # ./ceph osd tier add rbd cache
   # ./ceph osd tier cache-mode cache writeback
   # ./ceph osd pool set cache hit_set_type bloom                                                                                 
   # ./ceph osd pool set cache hit_set_count 1                                                                                    
   # ./ceph osd pool set cache hit_set_period 3600                                                                                
   # ./ceph osd pool set cache target_max_bytes 10737418240
   # ./ceph osd tier set-overlay rbd cache
   
   # ./rbd diff rbd/test_rbd_clone@snap
   

   
   HERE IS NOTHING.

In master branch, when cache tier is set, the behavior is slightly different:

1. rbd object is not promoted immediately, so when run rbd diff for the first time, the result is correct
   

   # ./rbd diff rbd/test_rbd_clone@snap
   Offset Length Type 
   0      4      data
   

2. after that, rbd object is promoted, rbd diff outputs nothing
   

   # ./rados -p cache ls
   rbd_object_map.5e3f2ae8944a.0000000000000004
   rbd_data.5e3f2ae8944a.0000000000000000
   rbd_id.test_rbd_clone
   rbd_header.5e3f2ae8944a
   # ./rbd diff rbd/test_rbd_clone@snap
   

3. if I call cache-flush-evict-all, rbd diff works correctly again:
   

   # ./rados -p cache cache-flush-evict-all
   # ./rbd diff rbd/test_rbd_clone@snap
   Offset Length Type 
   0      4      data
   

I looked into the source code, rbd diff would look parent diff only if calling list-snaps on the rbd object returns ENOENT.
Wth a writeback cache tier, when doing list-snaps on a base pool object(non-exists), a whiteout object will be created in cache pool.
Calling list-snaps on this whiteouted object will normally return , then rbd diff will not work correctly.