Bug #56181
Updated by Ilya Dryomov almost 2 years ago
<pre>
$ rbd create --size 2G data/testimg
$ rbd mirror image enable data/testimg snapshot
Mirroring enabled
$ sudo rbd device map data/testimg
$ dd if=/dev/urandom of=/dev/rbd0 bs=4M oflag=direct
dd: error writing '/dev/rbd0': No space left on device
513+0 records in
512+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 37.0197 s, 58.0 MB/s
$ sudo rbd device unmap data/testimg
$ rbd mirror image snapshot data/testimg
Snapshot ID: 8
</pre>
Monitor last_copied_object_number and complete fields on the secondary cluster:
<pre>
$ rbd snap ls --all --format=json data/testimg | jq '.[-1] | .namespace.last_copied_object_number,.namespace.complete'
</pre>
For a fully allocated image, last_copied_object_number is gradually increasing from 1 to 512 (image size / object size). The snapshot is marked complete when last_copied_object_number reaches 512, as expected:
<pre>
1
false
---
19
false
---
42
false
---
50
false
---
72
false
---
92
false
---
107
false
---
124
false
---
142
false
---
161
false
---
181
false
---
200
false
---
214
false
---
235
false
---
255
false
---
270
false
---
288
false
---
305
false
---
320
false
---
338
false
---
355
false
---
370
false
---
386
false
---
402
false
---
418
false
---
432
false
---
451
false
---
465
false
---
484
false
---
502
false
---
512
true
---
512
true
---
512
true
</pre>
For an image with holes in it, last_copied_object_number gets tripped over the first hole. The sync continues but last_copied_object_number isn't updated. Eventually the snapshot is marked complete with last_copied_object_number still stuck:
<pre>
$ rbd create --size 2G data/testimg
$ rbd mirror image enable data/testimg snapshot
Mirroring enabled
$ sudo rbd device map data/testimg
$ dd if=/dev/urandom of=/dev/rbd0 bs=4M count=100 oflag=direct
100+0 records in
100+0 records out
419430400 bytes (419 MB, 400 MiB) copied, 15.9774 s, 26.3 MB/s
$ dd if=/dev/urandom of=/dev/rbd0 bs=4M seek=110 oflag=direct
dd: error writing '/dev/rbd0': No space left on device
403+0 records in
402+0 records out
1686110208 bytes (1.7 GB, 1.6 GiB) copied, 65.5053 s, 25.7 MB/s
$ sudo rbd device unmap data/testimg
$ rbd mirror image snapshot data/testimg
Snapshot ID: 16
</pre>
<pre>
1
false
---
5
false
---
29
false
---
40
false
---
51
false
---
64
false
---
73
false
---
85
false
---
87
false
---
99
false
---
99
false
---
99
false
---
[...]
---
99
false
---
99
false
---
99
false
---
100
true
---
100
true
---
100
true
</pre>
This can also be obselved by monitoring syncing_percent in "rbd mirror image status" output.