https://tracker.ceph.com/https://tracker.ceph.com/favicon.ico2010-07-15T10:46:37ZCeph Ceph - Bug #278: data corruption after rbd rollbackhttps://tracker.ceph.com/issues/278?journal_id=6202010-07-15T10:46:37ZSage Weilsage@newdream.net
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Resolved</i></li></ul><p>fixed by <a class="changeset" title="osd: use client or pool snapc based on _pool_ snap mode, not client op The pool snap mode determ..." href="https://tracker.ceph.com/projects/ceph/repository/revisions/e8991f19526939ee843c7b04c167fe290f113602">e8991f19526939ee843c7b04c167fe290f113602</a></p> Ceph - Bug #278: data corruption after rbd rollbackhttps://tracker.ceph.com/issues/278?journal_id=9292010-08-31T06:28:37ZWido den Hollanderwido@42on.com
<ul><li><strong>File</strong> <a href="/attachments/download/77/Screenshot-Untitled_Window.png">Screenshot-Untitled_Window.png</a> <a class="icon-only icon-magnifier" title="View" href="/attachments/77/Screenshot-Untitled_Window.png">View</a> added</li><li><strong>Status</strong> changed from <i>Resolved</i> to <i>7</i></li></ul><p>I'm setting this back to "Testing" since i'm seeing this too.</p>
<pre>root@client01:~# rbd snap create --snap=charlie001 charlie
root@client01:~# rbd snap ls charlie
4 charlie001 53687091200
< DELETE ALL DATA INSIDE VM >
root@client01:~# virsh destroy charlie
Domain charlie destroyed
root@client01:~# time rbd snap rollback --snap=charlie001 charlie
real 9m26.463s
user 0m2.240s
sys 0m2.170s
root@client01:~# virsh start charlie
Domain charlie started
root@client01:~#</pre>
<p>After creating the snapshot I did a "rm -rf /*" in the virtual machine.</p>
<p>As you can see, the VM's data got corrupted.</p>
<p>I'm not sure which logfile to look into.</p> Ceph - Bug #278: data corruption after rbd rollbackhttps://tracker.ceph.com/issues/278?journal_id=9322010-09-01T01:44:45ZWido den Hollanderwido@42on.com
<ul><li><strong>File</strong> <a href="/attachments/download/78/filelist.txt">filelist.txt</a> <a class="icon-only icon-magnifier" title="View" href="/attachments/78/filelist.txt">View</a> added</li></ul><p>I just exported the "charlie" image and mounted it through a loop device.</p>
<p>Attached is a filelist which I got from the VM.</p>
<p>A df -h shows me:</p>
<pre>root@logger:/mnt/loop1# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 9.2G 3.7G 5.1G 43% /
none 2.0G 176K 2.0G 1% /dev
none 2.0G 0 2.0G 0% /dev/shm
none 2.0G 72K 2.0G 1% /var/run
none 2.0G 0 2.0G 0% /var/lock
none 2.0G 0 2.0G 0% /lib/init/rw
/dev/mapper/data-logs
500G 121G 380G 25% /srv/ceph
/dev/mapper/loop0p1 48G 180M 45G 1% /mnt/loop1
root@logger:/mnt/loop1# </pre>
<p>To me it seems that the rollback didn't do anything at all, the data is still erased.</p>
<p>Searching through the OSD logs I didn't see anything about a "rollback", but this could be due to my low debug level.</p> Ceph - Bug #278: data corruption after rbd rollbackhttps://tracker.ceph.com/issues/278?journal_id=9332010-09-01T04:20:28ZWido den Hollanderwido@42on.com
<ul></ul><p>Did another test:</p>
<ul>
<li>Created a second disk for the VM "alpha" </li>
<li>Formatted the disk with ext4 inside the VM</li>
<li>Mounted the VM</li>
<li>Downloaded several Ubuntu ISO's onto that disk</li>
<li>Snapshotted the "alpah-second" disk</li>
<li>Removed all the ISO's</li>
<li>Halted "alpha" </li>
<li>Rolled back the snapshot</li>
<li>Started "alpha" again</li>
<li>Mounted the disk</li>
</ul>
<p>I then found out that all the ISO's were still gone, it seems no rollback has been done at all.</p>
<p>The filesystem mounted without errors or whatsoever, so it seems that the rollback didn't do anything.</p>
<pre>root@client01:~# virsh start alpha
Domain alpha started
root@client01:~# rbd ls
alpha
alpha-second
beta
charlie
root@client01:~# rbd snap create --snap=alpha-second-snap alpha-second
root@client01:~# rbd snap ls alpha-second
6 alpha-second-snap 10737418240
root@client01:~# time rbd snap rollback --snap=alpha-second-snap alpha-second
real 1m40.516s
user 0m0.460s
sys 0m0.520s
root@client01:~# virsh start alpha
Domain alpha started
root@client01:~#</pre> Ceph - Bug #278: data corruption after rbd rollbackhttps://tracker.ceph.com/issues/278?journal_id=9342010-09-01T12:30:37ZWido den Hollanderwido@42on.com
<ul><li><strong>Status</strong> changed from <i>7</i> to <i>Closed</i></li></ul><p>Just spoke to Yehudasa, but this seems to be a synchronization problem between RBD and qemu-rbd</p>
<p>When using qemu-rbd you should shutdown the VM first before creating the snapshot via the "rbd" tool, or use "qemu-img" to use the snapshot.</p>
<p>Also, it's recommended to do the rollback while the VM is down.</p> Ceph - Bug #278: data corruption after rbd rollbackhttps://tracker.ceph.com/issues/278?journal_id=18662010-12-01T15:12:10ZSage Weilsage@newdream.net
<ul><li><strong>Project</strong> changed from <i>3</i> to <i>Ceph</i></li></ul>