Ceph » CephFS

On both my MDS'es I'm seeing the following crash:

Core was generated by `/usr/bin/cmds -i 0 -c /etc/ceph/ceph.conf'.
Program terminated with signal 11, Segmentation fault.
#0  0x0000000000000000 in ?? ()
(gdb) bt
#0  0x0000000000000000 in ?? ()
#1  0x00000000006d7be1 in sigabrt_handler (signum=6) at config.cc:238
#2  <signal handler called>
#3  0x00007fde092f3a75 in raise () from /lib/libc.so.6
#4  0x00007fde092f75c0 in abort () from /lib/libc.so.6
#5  0x00007fde09ba98e5 in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/libstdc++.so.6
#6  0x00007fde09ba7d16 in ?? () from /usr/lib/libstdc++.so.6
#7  0x00007fde09ba7d43 in std::terminate() () from /usr/lib/libstdc++.so.6
#8  0x00007fde09ba7e3e in __cxa_throw () from /usr/lib/libstdc++.so.6
#9  0x00000000006c3e08 in ceph::__ceph_assert_fail (assertion=0x7064c9 "p.end()", file=<value optimized out>, line=82, func=<value optimized out>) at common/assert.cc:30
#10 0x0000000000655e7b in LogEvent::decode (bl=...) at mds/LogEvent.cc:82
#11 0x000000000063a5c8 in MDLog::_replay_thread (this=0xcbe300) at mds/MDLog.cc:524
#12 0x00000000004a97ad in MDLog::ReplayThread::entry() ()
#13 0x000000000048bfea in Thread::_entry_func (arg=0x13e2) at ./common/Thread.h:39
#14 0x00007fde0a3ee9ca in start_thread () from /lib/libpthread.so.0
#15 0x00007fde093a66fd in clone () from /lib/libc.so.6
#16 0x0000000000000000 in ?? ()

The last log lines (100 show):

2010-10-13 10:28:53.444873 7fde0497a710 mds0.cache.dir(1000007f42b) link_primary_inode [dentry #1/static/kernel/scm/fs/ext2/val/e2fsprogs.git/objects/d4 [2,head] auth NULL (dversion lock) v=0 inode=0 0x1a8cff30] [inode 1000007f56e [...2,head] #1000007f56e/ auth v14098 f(v0 m2010-10-09 19:28:29.610558 11=11+0) n(v0 rc2010-10-09 19:28:29.610558 b23272 12=11+1) (iversion lock) 0x1efc8ce0]
2010-10-13 10:28:53.444904 7fde0497a710 mds0.journal EMetaBlob.replay added [inode 1000007f56e [...2,head] /static/kernel/scm/fs/ext2/val/e2fsprogs.git/objects/d4/ auth v14098 f(v0 m2010-10-09 19:28:29.610558 11=11+0) n(v0 rc2010-10-09 19:28:29.610558 b23272 12=11+1) (iversion lock) 0x1efc8ce0]
2010-10-13 10:28:53.444932 7fde0497a710 mds0.log _replay 8707369153~699 / 8878399754 : EOpen [metablob 1000007f42b, 1 dirs], 1 open files
2010-10-13 10:28:53.444943 7fde0497a710 mds0.journal EOpen.replay 
2010-10-13 10:28:53.444951 7fde0497a710 mds0.journal EMetaBlob.replay 1 dirlumps
2010-10-13 10:28:53.444959 7fde0497a710 mds0.journal EMetaBlob.replay dir 1000007f42b
2010-10-13 10:28:53.444968 7fde0497a710 mds0.journal EMetaBlob.replay updated dir [dir 1000007f42b /static/kernel/scm/fs/ext2/val/e2fsprogs.git/objects/ [2,head] auth v=15003 cv=0/0 state=1073741824 f(v0 m2010-10-09 19:28:12.553073 258=0+258) n(v8 rc2010-10-09 19:28:38.788973 b33928568 1780=1522+258) hs=197+0,ss=0+0 | child 0x24550a20]
2010-10-13 10:28:53.444994 7fde0497a710 mds0.cache.dir(1000007f42b) add_null_dentry [dentry #1/static/kernel/scm/fs/ext2/val/e2fsprogs.git/objects/d5 [2,head] auth NULL (dversion lock) pv=0 v=15003 inode=0 0x1a8d0100]
2010-10-13 10:28:53.445008 7fde0497a710 mds0.journal EMetaBlob.replay added [dentry #1/static/kernel/scm/fs/ext2/val/e2fsprogs.git/objects/d5 [2,head] auth NULL (dversion lock) v=0 inode=0 0x1a8d0100]
2010-10-13 10:28:53.445028 7fde0497a710 mds0.cache.dir(1000007f42b) link_primary_inode [dentry #1/static/kernel/scm/fs/ext2/val/e2fsprogs.git/objects/d5 [2,head] auth NULL (dversion lock) v=0 inode=0 0x1a8d0100] [inode 1000007f56f [...2,head] #1000007f56f/ auth v14100 f(v0 m2010-10-09 19:28:29.619583 7=7+0) n(v0 rc2010-10-09 19:28:29.619583 b15173 8=7+1) (iversion lock) 0x1efc9570]
2010-10-13 10:28:53.445052 7fde0497a710 mds0.journal EMetaBlob.replay added [inode 1000007f56f [...2,head] /static/kernel/scm/fs/ext2/val/e2fsprogs.git/objects/d5/ auth v14100 f(v0 m2010-10-09 19:28:29.619583 7=7+0) n(v0 rc2010-10-09 19:28:29.619583 b15173 8=7+1) (iversion lock) 0x1efc9570]
2010-10-13 10:28:53.445079 7fde0497a710 mds0.log _replay 8707369856~699 / 8878399754 : EOpen [metablob 1000007f42b, 1 dirs], 1 open files
2010-10-13 10:28:53.445090 7fde0497a710 mds0.journal EOpen.replay 
2010-10-13 10:28:53.445099 7fde0497a710 mds0.journal EMetaBlob.replay 1 dirlumps
2010-10-13 10:28:53.445107 7fde0497a710 mds0.journal EMetaBlob.replay dir 1000007f42b
2010-10-13 10:28:53.445116 7fde0497a710 mds0.journal EMetaBlob.replay updated dir [dir 1000007f42b /static/kernel/scm/fs/ext2/val/e2fsprogs.git/objects/ [2,head] auth v=15003 cv=0/0 state=1073741824 f(v0 m2010-10-09 19:28:12.553073 258=0+258) n(v8 rc2010-10-09 19:28:38.788973 b33928568 1780=1522+258) hs=198+0,ss=0+0 | child 0x24550a20]
2010-10-13 10:28:53.445140 7fde0497a710 mds0.cache.dir(1000007f42b) add_null_dentry [dentry #1/static/kernel/scm/fs/ext2/val/e2fsprogs.git/objects/d6 [2,head] auth NULL (dversion lock) pv=0 v=15003 inode=0 0x1a8d02d0]
2010-10-13 10:28:53.445159 7fde0497a710 mds0.journal EMetaBlob.replay added [dentry #1/static/kernel/scm/fs/ext2/val/e2fsprogs.git/objects/d6 [2,head] auth NULL (dversion lock) v=0 inode=0 0x1a8d02d0]
2010-10-13 10:28:53.445179 7fde0497a710 mds0.cache.dir(1000007f42b) link_primary_inode [dentry #1/static/kernel/scm/fs/ext2/val/e2fsprogs.git/objects/d6 [2,head] auth NULL (dversion lock) v=0 inode=0 0x1a8d02d0] [inode 1000007f571 [...2,head] #1000007f571/ auth v14102 f(v0 m2010-10-09 19:28:29.626329 5=5+0) n(v0 rc2010-10-09 19:28:29.626329 b4555 6=5+1) (iversion lock) 0x1efc9e00]
2010-10-13 10:28:53.445203 7fde0497a710 mds0.journal EMetaBlob.replay added [inode 1000007f571 [...2,head] /static/kernel/scm/fs/ext2/val/e2fsprogs.git/objects/d6/ auth v14102 f(v0 m2010-10-09 19:28:29.626329 5=5+0) n(v0 rc2010-10-09 19:28:29.626329 b4555 6=5+1) (iversion lock) 0x1efc9e00]
2010-10-13 10:28:53.445230 7fde0497a710 mds0.log _replay 8707370559~699 / 8878399754 : EOpen [metablob 1000007f42b, 1 dirs], 1 open files
2010-10-13 10:28:53.445249 7fde0497a710 mds0.journal EOpen.replay 
2010-10-13 10:28:53.445258 7fde0497a710 mds0.journal EMetaBlob.replay 1 dirlumps
2010-10-13 10:28:53.445266 7fde0497a710 mds0.journal EMetaBlob.replay dir 1000007f42b
2010-10-13 10:28:53.445275 7fde0497a710 mds0.journal EMetaBlob.replay updated dir [dir 1000007f42b /static/kernel/scm/fs/ext2/val/e2fsprogs.git/objects/ [2,head] auth v=15003 cv=0/0 state=1073741824 f(v0 m2010-10-09 19:28:12.553073 258=0+258) n(v8 rc2010-10-09 19:28:38.788973 b33928568 1780=1522+258) hs=199+0,ss=0+0 | child 0x24550a20]
2010-10-13 10:28:53.445300 7fde0497a710 mds0.cache.dir(1000007f42b) add_null_dentry [dentry #1/static/kernel/scm/fs/ext2/val/e2fsprogs.git/objects/d7 [2,head] auth NULL (dversion lock) pv=0 v=15003 inode=0 0x1a8d04a0]
2010-10-13 10:28:53.445315 7fde0497a710 mds0.journal EMetaBlob.replay added [dentry #1/static/kernel/scm/fs/ext2/val/e2fsprogs.git/objects/d7 [2,head] auth NULL (dversion lock) v=0 inode=0 0x1a8d04a0]
2010-10-13 10:28:53.445336 7fde0497a710 mds0.cache.dir(1000007f42b) link_primary_inode [dentry #1/static/kernel/scm/fs/ext2/val/e2fsprogs.git/objects/d7 [2,head] auth NULL (dversion lock) v=0 inode=0 0x1a8d04a0] [inode 1000007f572 [...2,head] #1000007f572/ auth v14104 f(v0 m2010-10-09 19:28:29.632638 5=5+0) n(v0 rc2010-10-09 19:28:29.632638 b44631 6=5+1) (iversion lock) 0x1efca690]
2010-10-13 10:28:53.445360 7fde0497a710 mds0.journal EMetaBlob.replay added [inode 1000007f572 [...2,head] /static/kernel/scm/fs/ext2/val/e2fsprogs.git/objects/d7/ auth v14104 f(v0 m2010-10-09 19:28:29.632638 5=5+0) n(v0 rc2010-10-09 19:28:29.632638 b44631 6=5+1) (iversion lock) 0x1efca690]
2010-10-13 10:28:53.445387 7fde0497a710 mds0.log _replay 8707371262~699 / 8878399754 : EOpen [metablob 1000007f42b, 1 dirs], 1 open files
2010-10-13 10:28:53.445399 7fde0497a710 mds0.journal EOpen.replay 
2010-10-13 10:28:53.445407 7fde0497a710 mds0.journal EMetaBlob.replay 1 dirlumps
2010-10-13 10:28:53.445415 7fde0497a710 mds0.journal EMetaBlob.replay dir 1000007f42b
2010-10-13 10:28:53.445424 7fde0497a710 mds0.journal EMetaBlob.replay updated dir [dir 1000007f42b /static/kernel/scm/fs/ext2/val/e2fsprogs.git/objects/ [2,head] auth v=15003 cv=0/0 state=1073741824 f(v0 m2010-10-09 19:28:12.553073 258=0+258) n(v8 rc2010-10-09 19:28:38.788973 b33928568 1780=1522+258) hs=200+0,ss=0+0 | child 0x24550a20]
2010-10-13 10:28:53.445449 7fde0497a710 mds0.cache.dir(1000007f42b) add_null_dentry [dentry #1/static/kernel/scm/fs/ext2/val/e2fsprogs.git/objects/d8 [2,head] auth NULL (dversion lock) pv=0 v=15003 inode=0 0x1a8d0670]
2010-10-13 10:28:53.445464 7fde0497a710 mds0.journal EMetaBlob.replay added [dentry #1/static/kernel/scm/fs/ext2/val/e2fsprogs.git/objects/d8 [2,head] auth NULL (dversion lock) v=0 inode=0 0x1a8d0670]
2010-10-13 10:28:53.445484 7fde0497a710 mds0.cache.dir(1000007f42b) link_primary_inode [dentry #1/static/kernel/scm/fs/ext2/val/e2fsprogs.git/objects/d8 [2,head] auth NULL (dversion lock) v=0 inode=0 0x1a8d0670] [inode 1000007f574 [...2,head] #1000007f574/ auth v14106 f(v0 m2010-10-09 19:28:29.643937 9=9+0) n(v0 rc2010-10-09 19:28:29.643937 b10773 10=9+1) (iversion lock) 0x1efcaf20]
2010-10-13 10:28:53.445508 7fde0497a710 mds0.journal EMetaBlob.replay added [inode 1000007f574 [...2,head] /static/kernel/scm/fs/ext2/val/e2fsprogs.git/objects/d8/ auth v14106 f(v0 m2010-10-09 19:28:29.643937 9=9+0) n(v0 rc2010-10-09 19:28:29.643937 b10773 10=9+1) (iversion lock) 0x1efcaf20]
2010-10-13 10:28:53.445535 7fde0497a710 mds0.log _replay 8707371965~699 / 8878399754 : EOpen [metablob 1000007f42b, 1 dirs], 1 open files
2010-10-13 10:28:53.445546 7fde0497a710 mds0.journal EOpen.replay 
2010-10-13 10:28:53.445554 7fde0497a710 mds0.journal EMetaBlob.replay 1 dirlumps
2010-10-13 10:28:53.445562 7fde0497a710 mds0.journal EMetaBlob.replay dir 1000007f42b
2010-10-13 10:28:53.445572 7fde0497a710 mds0.journal EMetaBlob.replay updated dir [dir 1000007f42b /static/kernel/scm/fs/ext2/val/e2fsprogs.git/objects/ [2,head] auth v=15003 cv=0/0 state=1073741824 f(v0 m2010-10-09 19:28:12.553073 258=0+258) n(v8 rc2010-10-09 19:28:38.788973 b33928568 1780=1522+258) hs=201+0,ss=0+0 | child 0x24550a20]
2010-10-13 10:28:53.445605 7fde0497a710 mds0.cache.dir(1000007f42b) add_null_dentry [dentry #1/static/kernel/scm/fs/ext2/val/e2fsprogs.git/objects/d9 [2,head] auth NULL (dversion lock) pv=0 v=15003 inode=0 0x1a8d0840]
2010-10-13 10:28:53.445620 7fde0497a710 mds0.journal EMetaBlob.replay added [dentry #1/static/kernel/scm/fs/ext2/val/e2fsprogs.git/objects/d9 [2,head] auth NULL (dversion lock) v=14667 inode=0 0x1a8d0840]
2010-10-13 10:28:53.445640 7fde0497a710 mds0.cache.dir(1000007f42b) link_primary_inode [dentry #1/static/kernel/scm/fs/ext2/val/e2fsprogs.git/objects/d9 [2,head] auth NULL (dversion lock) v=14667 inode=0 0x1a8d0840] [inode 1000007f575 [...2,head] #1000007f575/ auth v14667 f(v0 m2010-10-09 19:28:32.338921 3=3+0) n(v0 rc2010-10-09 19:28:32.338921 b3036 4=3+1) (iversion lock) 0x1efcb7b0]
2010-10-13 10:28:53.445664 7fde0497a710 mds0.journal EMetaBlob.replay added [inode 1000007f575 [...2,head] /static/kernel/scm/fs/ext2/val/e2fsprogs.git/objects/d9/ auth v14667 f(v0 m2010-10-09 19:28:32.338921 3=3+0) n(v0 rc2010-10-09 19:28:32.338921 b3036 4=3+1) (iversion lock) 0x1efcb7b0]
2010-10-13 10:28:53.445691 7fde0497a710 mds0.log _replay 8707372668~699 / 8878399754 : EOpen [metablob 1000007f42b, 1 dirs], 1 open files
2010-10-13 10:28:53.445702 7fde0497a710 mds0.journal EOpen.replay 
2010-10-13 10:28:53.445710 7fde0497a710 mds0.journal EMetaBlob.replay 1 dirlumps
2010-10-13 10:28:53.445719 7fde0497a710 mds0.journal EMetaBlob.replay dir 1000007f42b
2010-10-13 10:28:53.445727 7fde0497a710 mds0.journal EMetaBlob.replay updated dir [dir 1000007f42b /static/kernel/scm/fs/ext2/val/e2fsprogs.git/objects/ [2,head] auth v=15003 cv=0/0 state=1073741824 f(v0 m2010-10-09 19:28:12.553073 258=0+258) n(v8 rc2010-10-09 19:28:38.788973 b33928568 1780=1522+258) hs=202+0,ss=0+0 | child 0x24550a20]
2010-10-13 10:28:53.445753 7fde0497a710 mds0.cache.dir(1000007f42b) add_null_dentry [dentry #1/static/kernel/scm/fs/ext2/val/e2fsprogs.git/objects/da [2,head] auth NULL (dversion lock) pv=0 v=15003 inode=0 0x1a8d0a10]
2010-10-13 10:28:53.445768 7fde0497a710 mds0.journal EMetaBlob.replay added [dentry #1/static/kernel/scm/fs/ext2/val/e2fsprogs.git/objects/da [2,head] auth NULL (dversion lock) v=14669 inode=0 0x1a8d0a10]
2010-10-13 10:28:53.445789 7fde0497a710 mds0.cache.dir(1000007f42b) link_primary_inode [dentry #1/static/kernel/scm/fs/ext2/val/e2fsprogs.git/objects/da [2,head] auth NULL (dversion lock) v=14669 inode=0 0x1a8d0a10] [inode 1000007f577 [...2,head] #1000007f577/ auth v14669 f(v0 m2010-10-09 19:28:32.603833 10=10+0) n(v0 rc2010-10-09 19:28:32.603833 b16270 11=10+1) (iversion lock) 0x1efcc040]
2010-10-13 10:28:53.445812 7fde0497a710 mds0.journal EMetaBlob.replay added [inode 1000007f577 [...2,head] /static/kernel/scm/fs/ext2/val/e2fsprogs.git/objects/da/ auth v14669 f(v0 m2010-10-09 19:28:32.603833 10=10+0) n(v0 rc2010-10-09 19:28:32.603833 b16270 11=10+1) (iversion lock) 0x1efcc040]
2010-10-13 10:28:53.445839 7fde0497a710 mds0.log _replay 8707373371~699 / 8878399754 : EOpen [metablob 1000007f42b, 1 dirs], 1 open files
2010-10-13 10:28:53.445850 7fde0497a710 mds0.journal EOpen.replay 
2010-10-13 10:28:53.445858 7fde0497a710 mds0.journal EMetaBlob.replay 1 dirlumps
2010-10-13 10:28:53.445867 7fde0497a710 mds0.journal EMetaBlob.replay dir 1000007f42b
2010-10-13 10:28:53.445875 7fde0497a710 mds0.journal EMetaBlob.replay updated dir [dir 1000007f42b /static/kernel/scm/fs/ext2/val/e2fsprogs.git/objects/ [2,head] auth v=15003 cv=0/0 state=1073741824 f(v0 m2010-10-09 19:28:12.553073 258=0+258) n(v8 rc2010-10-09 19:28:38.788973 b33928568 1780=1522+258) hs=203+0,ss=0+0 | child 0x24550a20]
2010-10-13 10:28:53.445900 7fde0497a710 mds0.cache.dir(1000007f42b) add_null_dentry [dentry #1/static/kernel/scm/fs/ext2/val/e2fsprogs.git/objects/db [2,head] auth NULL (dversion lock) pv=0 v=15003 inode=0 0x1a8d0be0]
2010-10-13 10:28:53.445915 7fde0497a710 mds0.journal EMetaBlob.replay added [dentry #1/static/kernel/scm/fs/ext2/val/e2fsprogs.git/objects/db [2,head] auth NULL (dversion lock) v=14671 inode=0 0x1a8d0be0]
2010-10-13 10:28:53.445943 7fde0497a710 mds0.cache.dir(1000007f42b) link_primary_inode [dentry #1/static/kernel/scm/fs/ext2/val/e2fsprogs.git/objects/db [2,head] auth NULL (dversion lock) v=14671 inode=0 0x1a8d0be0] [inode 1000007f578 [...2,head] #1000007f578/ auth v14671 f(v0 m2010-10-09 19:28:32.690528 4=4+0) n(v0 rc2010-10-09 19:28:32.690528 b90164 5=4+1) (iversion lock) 0x1efcc8d0]
2010-10-13 10:28:53.445967 7fde0497a710 mds0.journal EMetaBlob.replay added [inode 1000007f578 [...2,head] /static/kernel/scm/fs/ext2/val/e2fsprogs.git/objects/db/ auth v14671 f(v0 m2010-10-09 19:28:32.690528 4=4+0) n(v0 rc2010-10-09 19:28:32.690528 b90164 5=4+1) (iversion lock) 0x1efcc8d0]
2010-10-13 10:28:53.445994 7fde0497a710 mds0.log _replay 8707374074~699 / 8878399754 : EOpen [metablob 1000007f42b, 1 dirs], 1 open files
2010-10-13 10:28:53.446006 7fde0497a710 mds0.journal EOpen.replay 
2010-10-13 10:28:53.446014 7fde0497a710 mds0.journal EMetaBlob.replay 1 dirlumps
2010-10-13 10:28:53.446022 7fde0497a710 mds0.journal EMetaBlob.replay dir 1000007f42b
2010-10-13 10:28:53.446031 7fde0497a710 mds0.journal EMetaBlob.replay updated dir [dir 1000007f42b /static/kernel/scm/fs/ext2/val/e2fsprogs.git/objects/ [2,head] auth v=15003 cv=0/0 state=1073741824 f(v0 m2010-10-09 19:28:12.553073 258=0+258) n(v8 rc2010-10-09 19:28:38.788973 b33928568 1780=1522+258) hs=204+0,ss=0+0 | child 0x24550a20]
2010-10-13 10:28:53.446056 7fde0497a710 mds0.cache.dir(1000007f42b) add_null_dentry [dentry #1/static/kernel/scm/fs/ext2/val/e2fsprogs.git/objects/dc [2,head] auth NULL (dversion lock) pv=0 v=15003 inode=0 0x1a8d0db0]
2010-10-13 10:28:53.446071 7fde0497a710 mds0.journal EMetaBlob.replay added [dentry #1/static/kernel/scm/fs/ext2/val/e2fsprogs.git/objects/dc [2,head] auth NULL (dversion lock) v=14990 inode=0 0x1a8d0db0]
2010-10-13 10:28:53.446091 7fde0497a710 mds0.cache.dir(1000007f42b) link_primary_inode [dentry #1/static/kernel/scm/fs/ext2/val/e2fsprogs.git/objects/dc [2,head] auth NULL (dversion lock) v=14990 inode=0 0x1a8d0db0] [inode 1000007f57a [...2,head] #1000007f57a/ auth v14990 f(v0 m2010-10-09 19:28:33.069855 5=5+0) n(v0 rc2010-10-09 19:28:33.069855 b14136 6=5+1) (iversion lock) 0x1efcd160]
2010-10-13 10:28:53.446115 7fde0497a710 mds0.journal EMetaBlob.replay added [inode 1000007f57a [...2,head] /static/kernel/scm/fs/ext2/val/e2fsprogs.git/objects/dc/ auth v14990 f(v0 m2010-10-09 19:28:33.069855 5=5+0) n(v0 rc2010-10-09 19:28:33.069855 b14136 6=5+1) (iversion lock) 0x1efcd160]
mds/LogEvent.cc: In function 'static LogEvent* LogEvent::decode(ceph::bufferlist&)':
mds/LogEvent.cc:82: FAILED assert(p.end())
 ceph version 0.22~rc (53decffc7e30fb70cd49da2382db00698913961c)
 1: (MDLog::_replay_thread()+0x468) [0x63a5c8]
 2: (MDLog::ReplayThread::entry()+0xd) [0x4a97ad]
 3: (Thread::_entry_func(void*)+0xa) [0x48bfea]
 4: (()+0x69ca) [0x7fde0a3ee9ca]
 5: (clone()+0x6d) [0x7fde093a66fd]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
*** Caught signal (ABRT) ***
 ceph version 0.22~rc (53decffc7e30fb70cd49da2382db00698913961c)
 1: (sigabrt_handler(int)+0x7d) [0x6d7bad]
 2: (()+0x33af0) [0x7fde092f3af0]
 3: (gsignal()+0x35) [0x7fde092f3a75]
 4: (abort()+0x180) [0x7fde092f75c0]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x115) [0x7fde09ba98e5]
 6: (()+0xcad16) [0x7fde09ba7d16]
 7: (()+0xcad43) [0x7fde09ba7d43]
 8: (()+0xcae3e) [0x7fde09ba7e3e]
 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x448) [0x6c3e08]
 10: (LogEvent::decode(ceph::buffer::list&)+0x97b) [0x655e7b]
 11: (MDLog::_replay_thread()+0x468) [0x63a5c8]
 12: (MDLog::ReplayThread::entry()+0xd) [0x4a97ad]
 13: (Thread::_entry_func(void*)+0xa) [0x48bfea]
 14: (()+0x69ca) [0x7fde0a3ee9ca]
 15: (clone()+0x6d) [0x7fde093a66fd]

I used cdebugpack to gather the information, data is located at logger.ceph.widodh.nl:/srv/ceph/issues/cmds_crash_logevent_decode

There is a node13 and node14 tar.gz