Ceph » RADOS

Version: 12.2.7

The following crash is observed during normal operation of the cluster, so no particular steps to reproduce it.
Now the OSD crashes at every start.

-28> 2018-08-29 22:19:11.823123 7f79fb9f0d00 3 journal journal_replay: applying op seq 158041971
-27> 2018-08-29 22:19:11.823126 7f79fb9f0d00 10 journal op_apply_start 158041971 open_ops 0 -> 1
-26> 2018-08-29 22:19:11.823128 7f79fb9f0d00 10 filestore(/var/lib/ceph/osd/internal-146) do_transaction(2623): on 0x7f7a0642f300
-25> 2018-08-29 22:19:11.823134 7f79fb9f0d00 20 filestore(/var/lib/ceph/osd/internal-146) _check_global_replay_guard(2412): no xattr
-24> 2018-08-29 22:19:11.823149 7f79fb9f0d00 20 filestore(/var/lib/ceph/osd/internal-146) _check_replay_guard(2585): no xattr
-23> 2018-08-29 22:19:11.823151 7f79fb9f0d00 15 filestore(/var/lib/ceph/osd/internal-146) _omap_setkeys(5515): 9.64c_head/#9:32600000::::head#
-22> 2018-08-29 22:19:11.823152 7f79fb9f0d00 20 filestore(/var/lib/ceph/osd/internal-146) _omap_setkeys(5539): set 0000714470.00000000000008021654
-21> 2018-08-29 22:19:11.823153 7f79fb9f0d00 20 filestore(/var/lib/ceph/osd/internal-146) _omap_setkeys(5539): set _fastinfo
-20> 2018-08-29 22:19:11.823154 7f79fb9f0d00 10 filestore oid: #9:32600000::::head# not skipping op, spos 158041971.0.0
-19> 2018-08-29 22:19:11.823155 7f79fb9f0d00 10 filestore > header.spos 0.0.0
-18> 2018-08-29 22:19:11.823161 7f79fb9f0d00 20 filestore(/var/lib/ceph/osd/internal-146) _omap_setkeys(5543): 9.64c_head/#9:32600000::::head# = 0
-17> 2018-08-29 22:19:11.823163 7f79fb9f0d00 10 filestore(/var/lib/ceph/osd/internal-146) _do_transaction(2623): on 0x7f7a0642f468
-16> 2018-08-29 22:19:11.823167 7f79fb9f0d00 20 filestore(/var/lib/ceph/osd/internal-146) _check_global_replay_guard(2412): no xattr
-15> 2018-08-29 22:19:11.823183 7f79fb9f0d00 10 filestore(/var/lib/ceph/osd/internal-146) _check_replay_guard(2613): object has 155735759.0.2 < current pos 158041971.1.0, in past, will replay
-14> 2018-08-29 22:19:11.823200 7f79fb9f0d00 20 filestore(/var/lib/ceph/osd/internal-146) _fgetattrs(4341): 29 getting ''
-13> 2018-08-29 22:19:11.823204 7f79fb9f0d00 20 filestore(/var/lib/ceph/osd/internal-146) _fgetattrs(4341): 29 getting 'snapset'
-12> 2018-08-29 22:19:11.823207 7f79fb9f0d00 15 filestore(/var/lib/ceph/osd/internal-146) _setattrs(4567): 9.64c_head/#9:32606f77:::.dir.default.32460656.12:head#
-11> 2018-08-29 22:19:11.823228 7f79fb9f0d00 10 filestore(/var/lib/ceph/osd/internal-146) _setattrs(4636): 9.64c_head/#9:32606f77:::.dir.default.32460656.12:head# = 0
-10> 2018-08-29 22:19:11.823236 7f79fb9f0d00 20 filestore(/var/lib/ceph/osd/internal-146) _check_global_replay_guard(2412): no xattr
-9> 2018-08-29 22:19:11.823250 7f79fb9f0d00 10 filestore(/var/lib/ceph/osd/internal-146) _check_replay_guard(2613): object has 155735759.0.2 < current pos 158041971.1.1, in past, will replay
-8> 2018-08-29 22:19:11.823253 7f79fb9f0d00 15 filestore(/var/lib/ceph/osd/internal-146) _omap_setheader(5595): 9.64c_head/#9:32606f77:::.dir.default.32460656.12:head#
-7> 2018-08-29 22:19:11.823266 7f79fb9f0d00 10 filestore oid: #9:32606f77:::.dir.default.32460656.12:head# not skipping op, *spos 158041971.1.1
-6> 2018-08-29 22:19:11.823267 7f79fb9f0d00 10 filestore > header.spos 155735759.0.2
-5> 2018-08-29 22:19:11.823280 7f79fb9f0d00 20 filestore(/var/lib/ceph/osd/internal-146) _check_global_replay_guard(2412): no xattr
-4> 2018-08-29 22:19:11.823295 7f79fb9f0d00 10 filestore(/var/lib/ceph/osd/internal-146) _check_replay_guard(2613): object has 155735759.0.2 < current pos 158041971.1.2, in past, will replay
-3> 2018-08-29 22:19:11.823297 7f79fb9f0d00 15 filestore(/var/lib/ceph/osd/internal-146) _omap_rmkeys(5550): 9.64c_head/#9:32606f77:::.dir.default.32460656.12:head#
-2> 2018-08-29 22:19:11.823311 7f79fb9f0d00 10 filestore oid: #9:32606f77:::.dir.default.32460656.12:head# not skipping op, *spos 158041971.1.2
-1> 2018-08-29 22:19:11.823313 7f79fb9f0d00 10 filestore > header.spos 155735759.0.2
0> 2018-08-29 22:19:11.825746 7f79fb9f0d00 -1 /build/ceph-12.2.7/src/os/filestore/DBObjectMap.cc: In function 'virtual int DBObjectMap::rm_keys(const ghobject_t&, const std::set<std::basic_string<cha
r> >&, const SequencerPosition)' thread 7f79fb9f0d00 time 2018-08-29 22:19:11.823318
/build/ceph-12.2.7/src/os/filestore/DBObjectMap.cc: 662: FAILED assert(state.legacy)