Ceph » rbd

Recently, in our test, we found that there are VM images, that we exported from the original cluster and imported to another cluster, whose images on those two clusters are not the same.

The details of test is as follows: at first, we fully exported the VM's images from the original cluster and imported them to another cluster; after that, we "export-diff"ed the VM images once every day and apply those diffs to the other cluster.

We did some extra test to narrow this issue, and we found that it seems that the difference of the images on those two cluster are that some sections of the data of the image on the original cluster is all 0, while non-zero data on the other cluster. In our VM, we had a crond task to call fstrim. After doing some reading of the source code of ceph, we doubt that some discard OPs are not recorded in the diffs.

To confirm the cause of this phenomenon, we did the following test:
We created an image, write data into it, then protect it and clone a new image. Then we create a snapshot, snap1, of the cloned image, then truncate it and create a snapshot, snap2. In the diff of the two snapshot, we found that only the data that are not truncated is recorded in the diff, which means the truncate operation is not reflected by the diff.

Is export-diff suppose to run in this way? Or is this a bug?