Ceph » RADOS

My Ceph cluster is down when the server is powered off,
and when i restart my osd, it failed in read_log.
As fllow:

by client.765048.0:2510440 2017-05-25 14:37:44.210324
2017-05-27 01:14:32.791414 7fc20486f7c0 20 read_log 346'6529417 (346'6529402) modify   2/ccdae3a1/rbd_data.4e2875a574153.0000000000001ac6/head by client.576715.0:4770236 2017-05-25 14:37:28.097068
2017-05-27 01:14:32.791421 7fc20486f7c0 20 read_log 346'6529418 (346'6529417) modify   2/ccdae3a1/rbd_data.4e2875a574153.0000000000001ac6/head by client.576715.0:4770237 2017-05-25 14:37:28.097097
2017-05-27 01:14:32.791429 7fc20486f7c0 20 read_log 346'6529419 (346'6529418) modify   2/ccdae3a1/rbd_data.4e2875a574153.0000000000001ac6/head by client.576715.0:4770239 2017-05-25 14:37:28.100208
2017-05-27 01:14:32.791435 7fc20486f7c0 20 read_log 346'6529420 (346'6528484) modify   2/c173e4c1/rbd_data.5d4e9238e1f29.00000000000009a8/head by client.561926.0:2389556 2017-05-25 14:37:39.001718
2017-05-27 01:14:32.791443 7fc20486f7c0 20 read_log 346'6529421 (346'6529420) modify   2/c173e4c1/rbd_data.5d4e9238e1f29.00000000000009a8/head by client.561926.0:2389557 2017-05-25 14:37:39.003726
2017-05-27 01:14:32.791457 7fc20486f7c0 20 read_log 346'6529422 (346'6529391) modify   2/67ac01c1/rbd_data.ba87d238e1f29.0000000000000374/head by client.764086.0:136303 2017-05-25 14:37:28.961843
2017-05-27 01:14:32.791465 7fc20486f7c0 20 read_log 346'6529423 (346'6529397) modify   2/3ca0c401/rbd_data.ba86e10117899.0000000000000830/head by client.764087.0:1603325 2017-05-25 14:37:29.038499
2017-05-27 01:14:32.791473 7fc20486f7c0 20 read_log 346'6529424 (346'6529404) modify   2/ab76dbe1/rbd_data.4de9374b0dc51.00000000000004a6/head by client.561676.0:3091323 2017-05-25 14:37:29.053888
2017-05-27 01:14:32.791479 7fc20486f7c0 20 read_log 346'6529425 (346'6529424) modify   2/ab76dbe1/rbd_data.4de9374b0dc51.00000000000004a6/head by client.561676.0:3091325 2017-05-25 14:37:29.055988
2017-05-27 01:14:32.791486 7fc20486f7c0 20 read_log 346'6529426 (346'6529249) modify   2/61f5a41/rbd_data.4e2875a574153.00000000000028bb/head by client.576715.0:4770241 2017-05-25 14:37:29.283908
2017-05-27 01:14:32.791494 7fc20486f7c0 20 read_log 346'6529427 (346'6529416) modify   2/232529e1/rbd_data.ba26f5a13810.0000000000007d72/head by client.765048.0:2510476 2017-05-25 14:37:47.221941
2017-05-27 01:14:32.791502 7fc20486f7c0 20 read_log 346'6529428 (346'6529392) modify   2/763a1381/rbd_data.5d4f7594a48ee.0000000000000049/head by client.561927.0:7376282 2017-05-25 14:37:40.231966
2017-05-27 01:14:32.791510 7fc20486f7c0 20 read_log 406'6529418 (346'6514935) modify   2/709a41c1/rbd_header.bb711e3fe196/head by client.794682.0:6 2017-05-26 02:09:57.942893
2017-05-27 01:14:32.792691 7fc20486f7c0 -1 osd/PGLog.cc: In function 'static void PGLog::read_log(ObjectStore*, coll_t, coll_t, ghobject_t, const pg_info_t&, std::map<eversion_t, hobject_t>&, PGLog::IndexedLog&, pg_missing_t&, std::ostringstream&, std::set<std::basic_string<char> >*)' thread 7fc20486f7c0 time 2017-05-27 01:14:32.791516
osd/PGLog.cc: 911: FAILED assert(last_e.version.version < e.version.version)

 ceph version 0.94.9-3.el7cp (7358f71bebe44c463df4d91c2770149e812bbeaa)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0xb11da5]
 2: (PGLog::read_log(ObjectStore*, coll_t, coll_t, ghobject_t, pg_info_t const&, std::map<eversion_t, hobject_t, std::less<eversion_t>, std::allocator<std::pair<eversion_t const, hobject_t> > >&, PGLog::IndexedLog&, pg_missing_t&, std::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >&, std::set<std::string, std::less<std::string>, std::allocator<std::string> >*)+0x1a38) [0x733d88]
 3: (PG::read_state(ObjectStore*, ceph::buffer::list&)+0x34f) [0x7b73df]
 4: (OSD::load_pgs()+0xa99) [0x67c499]
 5: (OSD::init()+0x181a) [0x67ff5a]
 6: (main()+0x2aec) [0x619ecc]
 7: (__libc_start_main()+0xf5) [0x7fc202233b35]
 8: /usr/bin/ceph-osd() [0x620e99]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.