Ceph » rbd

I have been able to repeat this a number of times. Essentially, I create a small rbd device, using the map command in rbd utility. (Assume root with every command)

rbd --pool rbd info lab8-1.img --name client.admin --key ...some-secret-key...

Next, I apply a filesystem to the block device and mount it.

/dev/rbd0 on /mnt/rbd/lab8-1 type ext4 (rw)

I then create some directories and some files to test and have no issues writing. Everything is still mounted and I attempt to create a snapshot of the block device.

rbd --name client.admin --pool rbd snap create --snap=lab8-1.img-snap-20120607-001 lab8-1.img

As soon as I create a snapshot, I can no longer modify the mounted block device.

touch /mnt/rbd/lab8-1/labadm/after-snapshot
touch: cannot touch `/mnt/rbd/lab8-1/labadm/after-snapshot': Read-only file system

I test by unmounting and re-mounting the block device.

umount /mnt/rbd/lab8-1/; sudo mount /dev/rbd0 /mnt/rbd/lab8-1

I try to create another file, and again seemingly a r-o filesystem situation.

touch /mnt/rbd/lab8-1/labadm/after-snapshot-n
touch: cannot touch `/mnt/rbd/lab8-1/labadm/after-snapshot-n': Read-only file system

At the same time I tested ability to touch an existing file, and it appears that in fact I can touch a file that's already there.

touch /mnt/rbd/lab8-1/labadm/after-snapshot
ls -ltr /mnt/rbd/lab8-1/labadm/

total 0
-rw-rw-r-- 1 labadm labadm 0 Jun  7 14:15 after-snapshot

touch /mnt/rbd/lab8-1/labadm/after-snapshot
ls -ltr /mnt/rbd/lab8-1/labadm/

total 0
-rw-rw-r-- 1 labadm labadm 0 Jun  7 14:18 after-snapshot

Creating a new file definitely fails.

touch /mnt/rbd/lab8-1/labadm/after-snapshot-n
touch: cannot touch `/mnt/rbd/lab8-1/labadm/after-snapshot-n': Read-only file system

Trying to actually modify the already existing file also fails.

echo "This is a test" > /mnt/rbd/lab8-1/labadm/after-snapshot
-bash: /mnt/rbd/lab8-1/labadm/after-snapshot: Read-only file system

At the moment I have three snapshots, which I am going to purge and try again.

rbd --name client.admin --pool rbd snap ls lab8-1.img

8    lab8-1.img-snap-20120607-001    104857600
9    lab8-1.img-snap-20120607-002    104857600
10    lab8-1.img-snap-20120607-003    104857600

rbd --name client.admin --pool rbd snap purge lab8-1.img

Removing all snapshots: 100% complete...done.

Then I unmount and remount again.

sudo umount /mnt/rbd/lab8-1/; sudo mount /dev/rbd0 /mnt/rbd/lab8-1

Now once again we try to write to existing file, and succeed without any issues.

echo "This is a test" > /mnt/rbd/lab8-1/labadm/after-snapshot && echo $?
0

And we can now also touch new files that do not yet exist.

touch /mnt/rbd/lab8-1/labadm/after-snapshot-n && echo $?
0

ls -ltr /mnt/rbd/lab8-1/labadm/
total 1
-rw-rw-r-- 1 labadm labadm 15 Jun  7 14:26 after-snapshot
-rw-rw-r-- 1 labadm labadm  0 Jun  7 14:27 after-snapshot-n

It appears as though for a short period of time the mounted filesystem is not read-only, but shortly after it seems to become read-only.

I repeated the same test after creating another pool and results appear to be consistent. However, if I undo the mapping, then re-create the mapping and mount again, problem seems to disappear and the image is r/w again. I create additional images and repeat test to see if filesystem on the block device is again read-only, but it does not appear to be so. Seemingly this happens with the first image.

Add'l info:

uname -a
Linux server-lab8 3.2.0-24-generic #39-Ubuntu SMP Mon May 21 16:52:17 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

dpkg -l ceph* *rbd* *rados*|grep ^ii
ii  ceph                             0.47.2-1precise            distributed storage and file system
ii  librados2                        0.47.2-1precise            RADOS distributed object store client library
ii  librbd1                          0.47.2-1precise            RADOS block device client library