Ceph

Hi Sage,

does it matter that the OSD is now down for around 1-2 days or will it just pickup any changes made to the data/ cluster so far?

I double checked syslog, kernel log, dmesg and there were not a single error/ strangeness. Our cluster is also quite good monitored (each machine has newrelic and munin client, we also have internal smokeping to measure latencies) and nothing indicates any errors, hangs, load peaks, ...

When I reenable the osd within the next hours (after your ok), which debug settings should I enable so we can track down the bug in case it occurs again after a few days? I'll not upgrade any other osds till I know 0.48.3 is as stable as 0.48.2 always was for us... :)

Thanks,
Corin

Both osd.7 and osd.15 have corrupted leveldb state. It's likely related to downgrading and then upgrading leveldb.

Well, after downgrading them they seem to work stable again. If it's related to leveldb, then upgrading leveldb as this patch does is not possible. It's not the downgrading, which makes troubles.

After the downgrade my cluster is still stable and no osd crashed so far.

What can I do to upgrade to latest argonaut? I'd even prefer upgrading to latest botail, but I assume it has the same problem. And I don't really want to try, because I'd not be able to downgrade then I guess.

If there's anything I can do to help you track down and fix this bug please let me know.

Corin Langosch wrote:

After the downgrade my cluster is still stable and no osd crashed so far.

What can I do to upgrade to latest argonaut? I'd even prefer upgrading to latest botail, but I assume it has the same problem. And I don't really want to try, because I'd not be able to downgrade then I guess.

If there's anything I can do to help you track down and fix this bug please let me know.

#3945 is tracking the dynamic linking to leveldb. Once this is done, we can cherry-pick it for your cluster against bobtail. Alternatively, I can push a branch with the newer leveldb linked in statically as a temporary fix (the same version quantal has).

It sounds like the up/downgrade may have caused issues on some of your leveldb databases, though, so it may be necessary to migrate data off those osds and replace them. It probably depends on whether you're in a hurry to upgrade..

It's not really urgent, but being able to upgrade to latest argonaut (and if that works for 2-3 days) to latest bobtail within the next 1-2 weeks would be great. Bobtail has several fixes I'm quite in need of (for example windows kvm guests don't crash anymore #3521 as has been fixed). It just has to be rock stable as 0.48.2 currently seems to be :)

Do you really think it's only leveldb data? I mean if the data would be corrupted, why does the downgraded version work perfectly fine with this data (for several days now on several osds)? I'd be quite stunned if leveldb wouldn't check if the data format on disk is supported and just returns wrong data to the user. Normally a program with invalid data just crashes (especially a database should take data integrity really serious). Should I report this on the leveldb tracker?

Will you also add checks to ensure ceph doesn't start if the on disk data is incompatible with the version of the library in use?

Unfortunately there doesn't appear to be a way to detect that with leveldb's current API. (If there was, I would expect that leveldb would be checking for incompatible up/downgrades internally and wouldn't hit this issue in the first place. :)

Future releases, starting with v0.58, will use the installed libleveldb. And we'll make a bobtail branch for you so you can safely stick with v0.56.x.

Created branch bobtail-leveldb with leveldb changes.

Great, thanks. Can only upgrade a single osd (out of around 16) to this branch and keep the others running using argonaut 0.48.2? If the upgraded osd runs stable for a couple of days then I'd upgrade them all.

Yes

I just tried to update one of my osds, but it doesn't start. I tried it twice:

ceph version 0.56.3-28-g2457211 (24572111607b3f2a89c2db8bd4acd5f9bf3fd22c)

2013-03-08 19:05:16.420338 7f3218cba780 0 filestore(/var/lib/ceph/osd/ceph-8) mount FIEMAP ioctl is supported and appears to work
2013-03-08 19:05:16.420403 7f3218cba780 0 filestore(/var/lib/ceph/osd/ceph-8) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
2013-03-08 19:05:16.424565 7f3218cba780 0 filestore(/var/lib/ceph/osd/ceph-8) mount did NOT detect btrfs
2013-03-08 19:05:16.429690 7f3218cba780 0 filestore(/var/lib/ceph/osd/ceph-8) mount syncfs(2) syscall fully supported (by glibc and kernel)
2013-03-08 19:05:16.429956 7f3218cba780 0 filestore(/var/lib/ceph/osd/ceph-8) mount found snaps <>
2013-03-08 19:05:16.541282 7f3218cba780 0 filestore(/var/lib/ceph/osd/ceph-8) mount: enabling WRITEAHEAD journal mode: btrfs not detected
2013-03-08 19:05:16.549733 7f3218cba780 1 journal _open /var/lib/ceph/osd/ceph-8/journal fd 19: 1048576000 bytes, block size 4096 bytes, directio = 1, aio = 0
2013-03-08 19:05:16.666846 7f3218cba780 1 journal _open /var/lib/ceph/osd/ceph-8/journal fd 19: 1048576000 bytes, block size 4096 bytes, directio = 1, aio = 0
2013-03-08 19:05:16.698963 7f3218cba780 1 journal close /var/lib/ceph/osd/ceph-8/journal
2013-03-08 19:05:16.717722 7f3218cba780 0 filestore(/var/lib/ceph/osd/ceph-8) mount FIEMAP ioctl is supported and appears to work
2013-03-08 19:05:16.717746 7f3218cba780 0 filestore(/var/lib/ceph/osd/ceph-8) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
2013-03-08 19:05:16.718180 7f3218cba780 0 filestore(/var/lib/ceph/osd/ceph-8) mount did NOT detect btrfs
2013-03-08 19:05:16.723162 7f3218cba780 0 filestore(/var/lib/ceph/osd/ceph-8) mount syncfs(2) syscall fully supported (by glibc and kernel)
2013-03-08 19:05:16.723806 7f3218cba780 0 filestore(/var/lib/ceph/osd/ceph-8) mount found snaps <>
2013-03-08 19:05:16.732368 7f3218cba780 0 filestore(/var/lib/ceph/osd/ceph-8) mount: enabling WRITEAHEAD journal mode: btrfs not detected
2013-03-08 19:05:16.744101 7f3218cba780 1 journal _open /var/lib/ceph/osd/ceph-8/journal fd 26: 1048576000 bytes, block size 4096 bytes, directio = 1, aio = 0
2013-03-08 19:05:16.745044 7f3218cba780 1 journal _open /var/lib/ceph/osd/ceph-8/journal fd 26: 1048576000 bytes, block size 4096 bytes, directio = 1, aio = 0
2013-03-08 19:05:16.778834 7f3218cba780 0 osd.8 13882 crush map has features 262144, adjusting msgr requires for clients
2013-03-08 19:05:16.778877 7f3218cba780 0 osd.8 13882 crush map has features 262144, adjusting msgr requires for osds
2013-03-08 19:05:17.701661 7f3218cba780 1 journal close /var/lib/ceph/osd/ceph-8/journal
2013-03-08 19:05:17.702858 7f3218cba780 -1 * ERROR: osd init failed: (95) Operation not supported
2013-03-08 19:06:06.117936 7f73dd9d1780 0 filestore(/var/lib/ceph/osd/ceph-8) mount FIEMAP ioctl is supported and appears to work
2013-03-08 19:06:06.117989 7f73dd9d1780 0 filestore(/var/lib/ceph/osd/ceph-8) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
2013-03-08 19:06:06.118961 7f73dd9d1780 0 filestore(/var/lib/ceph/osd/ceph-8) mount did NOT detect btrfs
2013-03-08 19:06:06.122584 7f73dd9d1780 0 filestore(/var/lib/ceph/osd/ceph-8) mount syncfs(2) syscall fully supported (by glibc and kernel)
2013-03-08 19:06:06.122864 7f73dd9d1780 0 filestore(/var/lib/ceph/osd/ceph-8) mount found snaps <>
2013-03-08 19:06:06.127184 7f73dd9d1780 0 filestore(/var/lib/ceph/osd/ceph-8) mount: enabling WRITEAHEAD journal mode: btrfs not detected
2013-03-08 19:06:06.145721 7f73dd9d1780 1 journal _open /var/lib/ceph/osd/ceph-8/journal fd 18: 1048576000 bytes, block size 4096 bytes, directio = 1, aio = 0
2013-03-08 19:06:06.189452 7f73dd9d1780 1 journal _open /var/lib/ceph/osd/ceph-8/journal fd 18: 1048576000 bytes, block size 4096 bytes, directio = 1, aio = 0
2013-03-08 19:06:06.196632 7f73dd9d1780 1 journal close /var/lib/ceph/osd/ceph-8/journal
2013-03-08 19:06:06.217373 7f73dd9d1780 0 filestore(/var/lib/ceph/osd/ceph-8) mount FIEMAP ioctl is supported and appears to work
2013-03-08 19:06:06.217403 7f73dd9d1780 0 filestore(/var/lib/ceph/osd/ceph-8) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
2013-03-08 19:06:06.218319 7f73dd9d1780 0 filestore(/var/lib/ceph/osd/ceph-8) mount did NOT detect btrfs
2013-03-08 19:06:06.231650 7f73dd9d1780 0 filestore(/var/lib/ceph/osd/ceph-8) mount syncfs(2) syscall fully supported (by glibc and kernel)
2013-03-08 19:06:06.231868 7f73dd9d1780 0 filestore(/var/lib/ceph/osd/ceph-8) mount found snaps <>
2013-03-08 19:06:06.236043 7f73dd9d1780 0 filestore(/var/lib/ceph/osd/ceph-8) mount: enabling WRITEAHEAD journal mode: btrfs not detected
2013-03-08 19:06:06.243854 7f73dd9d1780 1 journal _open /var/lib/ceph/osd/ceph-8/journal fd 26: 1048576000 bytes, block size 4096 bytes, directio = 1, aio = 0
2013-03-08 19:06:06.244595 7f73dd9d1780 1 journal _open /var/lib/ceph/osd/ceph-8/journal fd 26: 1048576000 bytes, block size 4096 bytes, directio = 1, aio = 0
2013-03-08 19:06:06.246056 7f73dd9d1780 0 osd.8 13882 crush map has features 262144, adjusting msgr requires for clients
2013-03-08 19:06:06.246065 7f73dd9d1780 0 osd.8 13882 crush map has features 262144, adjusting msgr requires for osds
2013-03-08 19:06:06.298262 7f73dd9d1780 1 journal close /var/lib/ceph/osd/ceph-8/journal
2013-03-08 19:06:06.298551 7f73dd9d1780 -1 * ERROR: osd init failed: (95) Operation not supported

Seems like I can't even downgrade anymore... :-(. I'll delete all files of this osd and restart it using the old version of ceph to get my cluster clean again.

This might be an authentication issue. That is one of the possible causes for the osd init failed error. I believe the defaults changed between Argonaut and Bobtail.

This is the wrong area to be authentication. Sam probably has a better idea what precisely it is; I would need to go code diving. Usually it's something silly like xattrs on ext4 but I assume that's not the issue here.

I'm using XFS everywhere. Kernel is stock ubuntu quantal (3.5.x) on these systems.

can you reproduce the EOPNOTSUPP with debug filestore = 20 and debug osd = 20 ?