Ceph

This morning I upgraded to the latest unstable ( 0b7c1afc43202953123f335057b9a5da428bc9a2 ), but when doing so, 10 of my 12 OSD's crashed with the following back trace:

Core was generated by `/usr/bin/cosd -i 2 -c /etc/ceph/ceph.conf'.
Program terminated with signal 6, Aborted.
#0  0x00007f9f117b0a75 in raise () from /lib/libc.so.6
(gdb) bt
#0  0x00007f9f117b0a75 in raise () from /lib/libc.so.6
#1  0x00007f9f117b45c0 in abort () from /lib/libc.so.6
#2  0x00007f9f120668e5 in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/libstdc++.so.6
#3  0x00007f9f12064d16 in ?? () from /usr/lib/libstdc++.so.6
#4  0x00007f9f12064d43 in std::terminate() () from /usr/lib/libstdc++.so.6
#5  0x00007f9f12064e3e in __cxa_throw () from /usr/lib/libstdc++.so.6
#6  0x00000000005c6ba8 in ceph::__ceph_assert_fail (assertion=0x5f7d98 "!store->collection_exists(coll_t(pgid))", 
    file=<value optimized out>, line=902, func=<value optimized out>) at common/assert.cc:30
#7  0x00000000004dd01d in OSD::_create_lock_pg (this=0x1c9f000, pgid=DWARF-2 expression error: DW_OP_reg operations must be used either alone or in conjuction with DW_OP_piece.
) at osd/OSD.cc:902
#8  0x00000000004f3ebb in OSD::handle_pg_notify (this=0x1c9f000, m=<value optimized out>) at osd/OSD.cc:3547
#9  0x00000000004f483d in OSD::_dispatch (this=0x1c9f000, m=0x1cb4c40) at osd/OSD.cc:2127
#10 0x00000000004f5129 in OSD::ms_dispatch (this=0x1c9f000, m=0x1cb4c40) at osd/OSD.cc:1956
#11 0x0000000000460af9 in Messenger::ms_deliver_dispatch (this=0x1c9d000) at msg/Messenger.h:97
#12 SimpleMessenger::dispatch_entry (this=0x1c9d000) at msg/SimpleMessenger.cc:342
#13 0x000000000045a37c in SimpleMessenger::DispatchThread::entry (this=0x1c9d488) at msg/SimpleMessenger.h:558
#14 0x000000000046e85a in Thread::_entry_func (arg=0x6bb7) at ./common/Thread.h:39
#15 0x00007f9f128ab9ca in start_thread () from /lib/libpthread.so.0
#16 0x00007f9f118636fd in clone () from /lib/libc.so.6
#17 0x0000000000000000 in ?? ()

I've uploaded the logs, cores and binaries to logger.ceph.widodh.nl:/srv/ceph/issues/osd_crash_create_lock_pg

The timestamps were all preserved, so they should match the log files.

After starting all the OSD's manually (with some pauze in between) they started to run, but the cluster wouldn't recover:

2010-10-02 10:42:01.477025    pg v339914: 3296 pgs: 3 creating, 101 active+clean, 168 peering, 2788 active+clean+degraded, 236 degraded+peering; 1442 GB data, 593 GB used, 1584 GB / 2178 GB avail; 1511545/3227908 degraded (46.827%)
2010-10-02 10:42:03.486299    pg v339915: 3296 pgs: 1 creating, 103 active+clean, 168 peering, 2788 active+clean+degraded, 236 degraded+peering; 1442 GB data, 1130 GB used, 3022 GB / 4153 GB avail; 1511545/3227908 degraded (46.827%)
2010-10-02 10:42:24.247506   osd e17614: 12 osds: 12 up, 12 in
2010-10-02 10:42:24.544186    pg v339916: 3295 pgs: 103 active+clean, 168 peering, 2788 active+clean+degraded, 236 degraded+peering; 1442 GB data, 1130 GB used, 3022 GB / 4153 GB avail; 1511545/3227908 degraded (46.827%)
2010-10-02 10:42:47.117378   log 2010-10-02 10:41:45.737347 mon0 [2001:16f8:10:2::c3c3:3f9b]:6789/0 67 : [INF] osd0 [2001:16f8:10:2::c3c3:8f6b]:6800/4074 failed (by osd6 [2001:16f8:10:2::c3c3:3f6c]:6800/7925)
2010-10-02 10:42:47.117378   log 2010-10-02 10:41:45.737500 mon0 [2001:16f8:10:2::c3c3:3f9b]:6789/0 68 : [INF] osd5 [2001:16f8:10:2::c3c3:3b6a]:6800/5449 failed (by osd6 [2001:16f8:10:2::c3c3:3f6c]:6800/7925)
2010-10-02 10:42:52.280467   log 2010-10-02 10:41:50.738303 mon0 [2001:16f8:10:2::c3c3:3f9b]:6789/0 69 : [INF] osd0 [2001:16f8:10:2::c3c3:8f6b]:6800/4074 failed (by osd6 [2001:16f8:10:2::c3c3:3f6c]:6800/7925)
2010-10-02 10:42:52.280467   log 2010-10-02 10:41:50.738477 mon0 [2001:16f8:10:2::c3c3:3f9b]:6789/0 70 : [INF] osd5 [2001:16f8:10:2::c3c3:3b6a]:6800/5449 failed (by osd6 [2001:16f8:10:2::c3c3:3f6c]:6800/7925)
2010-10-02 10:42:57.403565   osd e17615: 12 osds: 10 up, 12 in
2010-10-02 10:42:57.530072   log 2010-10-02 10:41:56.000109 mon0 [2001:16f8:10:2::c3c3:3f9b]:6789/0 71 : [INF] osd0 [2001:16f8:10:2::c3c3:8f6b]:6800/4074 failed (by osd6 [2001:16f8:10:2::c3c3:3f6c]:6800/7925)
2010-10-02 10:42:57.530072   log 2010-10-02 10:41:56.000275 mon0 [2001:16f8:10:2::c3c3:3f9b]:6789/0 72 : [INF] osd5 [2001:16f8:10:2::c3c3:3b6a]:6800/5449 failed (by osd6 [2001:16f8:10:2::c3c3:3f6c]:6800/7925)
2010-10-02 10:42:57.782387    pg v339917: 3295 pgs: 103 active+clean, 168 peering, 2788 active+clean+degraded, 236 degraded+peering; 1442 GB data, 1130 GB used, 3022 GB / 4153 GB avail; 1511545/3227908 degraded (46.827%)
2010-10-02 10:42:58.930449   osd e17616: 12 osds: 12 up, 12 in
2010-10-02 10:42:59.405541    pg v339918: 3295 pgs: 103 active+clean, 168 peering, 2788 active+clean+degraded, 236 degraded+peering; 1442 GB data, 1130 GB used, 3022 GB / 4153 GB avail; 1511545/3227908 degraded (46.827%)
2010-10-02 10:42:59.547091   log 2010-10-02 10:40:49.769174 osd5 [2001:16f8:10:2::c3c3:3b6a]:6800/5449 1 : [WRN] map e17615 wrongly marked me down
2010-10-02 10:42:59.547091   log 2010-10-02 10:40:49.784113 osd0 [2001:16f8:10:2::c3c3:8f6b]:6800/4074 1 : [WRN] map e17615 wrongly marked me down
2010-10-02 10:42:59.547091   log 2010-10-02 10:41:58.732677 mon0 [2001:16f8:10:2::c3c3:3f9b]:6789/0 73 : [INF] osd0 [2001:16f8:10:2::c3c3:8f6b]:6800/4074 boot
2010-10-02 10:42:59.547091   log 2010-10-02 10:41:58.732780 mon0 [2001:16f8:10:2::c3c3:3f9b]:6789/0 74 : [INF] osd5 [2001:16f8:10:2::c3c3:3b6a]:6800/5449 boot

My cluster has a total of 6300GB with all 12 OSD's up, but here it stays stuck at 4100GB, missing 2200GB of space.

Also, osd0 and osd5 (which got marked down) were never down after I started them.

A few minutes later osd8 also got marked down (wrongly).

These OSD's weren't busy at all, their load was 0.00 at that moment, so it can't be because they were responding too slow.

Doing a write to the cluster (for example creating a pool) gave it a small "kick" where it would start to recover a bit more, but it won't recover fully.

Am I in a situation because of my cluster which is already running for a long time and new users will never get into this situation? Would a fresh mkcephfs be a bit easier?