Ceph

I currently have 2 OSDs that won't start and it's preventing my
cluster from running my VMs.
My cluster is running on:

1. ceph -v
   ceph version 0.67.7 (d7ab4244396b57aac8b7e80812115bbd079e6b73)

My OSD's are found on different hosts as per the default CRUSH Map
rules and I got the logs from the problem OSDs as follows

osd.1:
-10> 2014-03-16 11:51:25.183093 7f50923e4780 20 read_log
12794'1377363 (12794'1377357) modify
5ee8f77d/rb.0.1f7b.238e1f29.0000000118fb/head//2 by
client.1077470.0:376704 2014-03-15 12:37:43.175319
-9> 2014-03-16 11:51:25.183122 7f50923e4780 20 read_log
12794'1377364 (12794'1377362) modify
af8f477d/rb.0.24f4.238e1f29.00000000e378/head//2 by
client.1074837.0:1295631 2014-03-15 12:38:03.613604
-8> 2014-03-16 11:51:25.183146 7f50923e4780 20 read_log
12794'1377365 (12794'1377348) modify
cbb2fb7d/rb.0.2355.2ae8944a.00000000849e/head//2 by
client.1077379.0:1396709 2014-03-15 12:38:33.720354
-7> 2014-03-16 11:51:25.183179 7f50923e4780 20 read_log
12794'1377366 (12794'1377365) modify
cbb2fb7d/rb.0.2355.2ae8944a.00000000849e/head//2 by
client.1077379.0:1396712 2014-03-15 12:38:33.726419
-6> 2014-03-16 11:51:25.183207 7f50923e4780 20 read_log
12794'1377367 (12794'1377366) modify
cbb2fb7d/rb.0.2355.2ae8944a.00000000849e/head//2 by
client.1077379.0:1397305 2014-03-15 12:39:09.863260
-5> 2014-03-16 11:51:25.183231 7f50923e4780 20 read_log
12794'1377368 (12794'1377367) modify
cbb2fb7d/rb.0.2355.2ae8944a.00000000849e/head//2 by
client.1077379.0:1398903 2014-03-15 12:40:13.096258
-4> 2014-03-16 11:51:25.183258 7f50923e4780 20 read_log
12794'1377369 (12794'1377363) modify
5ee8f77d/rb.0.1f7b.238e1f29.0000000118fb/head//2 by
client.1077470.0:377159 2014-03-15 12:40:13.105469
-3> 2014-03-16 11:51:25.183282 7f50923e4780 20 read_log
12794'1377370 (12794'1377360) modify
19463f7d/rb.0.ecb29.238e1f29.000000000101/head//2 by
client.1058212.1:358750 2014-03-15 12:40:24.998076
-2> 2014-03-16 11:51:25.183309 7f50923e4780 20 read_log
12794'1377371 (12794'1377368) modify
cbb2fb7d/rb.0.2355.2ae8944a.00000000849e/head//2 by
client.1077379.0:1399253 2014-03-15 12:40:28.134624
-1> 2014-03-16 11:51:25.183333 7f50923e4780 20 read_log
13740'1377371 (12605'1364064) modify
94c6137d/rb.0.d07d.2ae8944a.000000002524/head//2 by
client.1088028.0:10498 2014-03-16 00:06:12.643968
0> 2014-03-16 11:51:25.185685 7f50923e4780 -1 osd/PGLog.cc: In
function 'static bool PGLog::read_log(ObjectStore*, coll_t, hobject_t,
const pg_info_t&, std::map<eversion_t, hobject_t>&,
PGLog::IndexedLog&, pg_missing_t&, std::ostringstream&,
std::set<std::basic_string<char> >*)' thread 7f50923e4780 time
2014-03-16 11:51:25.183350
osd/PGLog.cc: 677: FAILED assert(last_e.version.version < e.version.version)

osd.2:
-10> 2014-03-16 11:28:45.015366 7fe5a7539780 20 read_log 12794'1377363
(12794'1377357) modify
5ee8f77d/rb.0.1f7b.238e1f29.0000000118fb/head//2 by
client.1077470.0:376704 2014-03-15 12:37:43.175319
-9> 2014-03-16 11:28:45.015381 7fe5a7539780 20 read_log
12794'1377364 (12794'1377362) modify
af8f477d/rb.0.24f4.238e1f29.00000000e378/head//2 by
client.1074837.0:1295631 2014-03-15 12:38:03.613604
-8> 2014-03-16 11:28:45.015394 7fe5a7539780 20 read_log
12794'1377365 (12794'1377348) modify
cbb2fb7d/rb.0.2355.2ae8944a.00000000849e/head//2 by
client.1077379.0:1396709 2014-03-15 12:38:33.720354
-7> 2014-03-16 11:28:45.015405 7fe5a7539780 20 read_log
12794'1377366 (12794'1377365) modify
cbb2fb7d/rb.0.2355.2ae8944a.00000000849e/head//2 by
client.1077379.0:1396712 2014-03-15 12:38:33.726419
-6> 2014-03-16 11:28:45.015418 7fe5a7539780 20 read_log
12794'1377367 (12794'1377366) modify
cbb2fb7d/rb.0.2355.2ae8944a.00000000849e/head//2 by
client.1077379.0:1397305 2014-03-15 12:39:09.863260
-5> 2014-03-16 11:28:45.015428 7fe5a7539780 20 read_log
12794'1377368 (12794'1377367) modify
cbb2fb7d/rb.0.2355.2ae8944a.00000000849e/head//2 by
client.1077379.0:1398903 2014-03-15 12:40:13.096258
-4> 2014-03-16 11:28:45.015441 7fe5a7539780 20 read_log
12794'1377369 (12794'1377363) modify
5ee8f77d/rb.0.1f7b.238e1f29.0000000118fb/head//2 by
client.1077470.0:377159 2014-03-15 12:40:13.105469
-3> 2014-03-16 11:28:45.015452 7fe5a7539780 20 read_log
12794'1377370 (12794'1377360) modify
19463f7d/rb.0.ecb29.238e1f29.000000000101/head//2 by
client.1058212.1:358750 2014-03-15 12:40:24.998076
-2> 2014-03-16 11:28:45.015464 7fe5a7539780 20 read_log
12794'1377371 (12794'1377368) modify
cbb2fb7d/rb.0.2355.2ae8944a.00000000849e/head//2 by
client.1077379.0:1399253 2014-03-15 12:40:28.134624
-1> 2014-03-16 11:28:45.015475 7fe5a7539780 20 read_log
13740'1377371 (12605'1364064) modify
94c6137d/rb.0.d07d.2ae8944a.000000002524/head//2 by
client.1088028.0:10498 2014-03-16 00:06:12.643968
0> 2014-03-16 11:28:45.016656 7fe5a7539780 -1 osd/PGLog.cc: In
function 'static bool PGLog::read_log(ObjectStore*, coll_t, hobject_t,
const pg_info_t&, std::map<eversion_t, hobject_t>&,
PGLog::IndexedLog&, pg_missing_t&, std::ostringstream&,
std::set<std::basic_string<char> >*)' thread 7fe5a7539780 time
2014-03-16 11:28:45.015497
osd/PGLog.cc: 677: FAILED assert(last_e.version.version < e.version.version)

It seems that the latest version of the object taken from omap is not
newer than the ones represented in the logs and this failed assert is
preventing those 2 OSDs to start.

Here's a link to the code.
https://github.com/ceph/ceph/blob/dumpling/src/osd/PGLog.cc#L667