Bug #15223: mon crash on 0.94.6 decoding mdsmap during upgrade test - Ceph - Ceph

Actions

Copy link

Bug #15223

closed

mon crash on 0.94.6 decoding mdsmap during upgrade test

Added by Sage Weil about 8 years ago. Updated about 8 years ago.

Status:

Resolved

Priority:

Urgent

Assignee:

John Spray

Category:

Target version:

% Done:

Source:

Q/A

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

2016-03-20T12:05:40.583 INFO:teuthology.orchestra.run.vpm086:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd pool get rbd pg_num'
2016-03-20T12:05:40.908 INFO:tasks.ceph.mon.c.vpm152.stderr:terminate called after throwing an instance of 'ceph::buffer::malformed_input'
2016-03-20T12:05:40.908 INFO:tasks.ceph.mon.c.vpm152.stderr:  what():  buffer::malformed_input: __PRETTY_FUNCTION__ unknown encoding version > 5
2016-03-20T12:05:40.909 INFO:tasks.ceph.mon.c.vpm152.stderr:*** Caught signal (Aborted) **
2016-03-20T12:05:40.909 INFO:tasks.ceph.mon.c.vpm152.stderr: in thread 7f8b46b4f700
2016-03-20T12:05:40.909 INFO:tasks.ceph.mon.c.vpm152.stderr: ceph version 0.94.6-199-g0418943 (0418943c6ef8c9649a58003444daeb4b6224fbab)
2016-03-20T12:05:40.909 INFO:tasks.ceph.mon.c.vpm152.stderr: 1: ceph-mon() [0x9ce362]
2016-03-20T12:05:40.910 INFO:tasks.ceph.mon.c.vpm152.stderr: 2: (()+0xf100) [0x7f8b4ccb6100]
2016-03-20T12:05:40.910 INFO:tasks.ceph.mon.c.vpm152.stderr: 3: (gsignal()+0x37) [0x7f8b4b6cf5f7]
2016-03-20T12:05:40.910 INFO:tasks.ceph.mon.c.vpm152.stderr: 4: (abort()+0x148) [0x7f8b4b6d0ce8]
2016-03-20T12:05:40.910 INFO:tasks.ceph.mon.c.vpm152.stderr: 5: (__gnu_cxx::__verbose_terminate_handler()+0x165) [0x7f8b4bfd39d5]
2016-03-20T12:05:40.911 INFO:tasks.ceph.mon.c.vpm152.stderr: 6: (()+0x5e946) [0x7f8b4bfd1946]
2016-03-20T12:05:40.911 INFO:tasks.ceph.mon.c.vpm152.stderr: 7: (()+0x5e973) [0x7f8b4bfd1973]
2016-03-20T12:05:40.911 INFO:tasks.ceph.mon.c.vpm152.stderr: 8: (()+0x5eb93) [0x7f8b4bfd1b93]
2016-03-20T12:05:40.911 INFO:tasks.ceph.mon.c.vpm152.stderr: 9: (MDSMap::decode(ceph::buffer::list::iterator&)+0x9db) [0x74950b]
2016-03-20T12:05:40.911 INFO:tasks.ceph.mon.c.vpm152.stderr: 10: (MDSMonitor::update_from_paxos(bool*)+0x358) [0x673018]
2016-03-20T12:05:40.912 INFO:tasks.ceph.mon.c.vpm152.stderr: 11: (PaxosService::refresh(bool*)+0x1a5) [0x608dc5]
2016-03-20T12:05:40.912 INFO:tasks.ceph.mon.c.vpm152.stderr: 12: (Monitor::refresh_from_paxos(bool*)+0x1fb) [0x5b2bbb]
2016-03-20T12:05:40.912 INFO:tasks.ceph.mon.c.vpm152.stderr: 13: (Paxos::do_refresh()+0x41) [0x5f3121]
2016-03-20T12:05:40.912 INFO:tasks.ceph.mon.c.vpm152.stderr: 14: (Paxos::handle_commit(MMonPaxos*)+0x293) [0x5fad93]
2016-03-20T12:05:40.913 INFO:tasks.ceph.mon.c.vpm152.stderr: 15: (Paxos::dispatch(PaxosServiceMessage*)+0x1db) [0x6020cb]
2016-03-20T12:05:40.913 INFO:tasks.ceph.mon.c.vpm152.stderr: 16: (Monitor::dispatch(MonSession*, Message*, bool)+0x943) [0x5d2753]
2016-03-20T12:05:40.913 INFO:tasks.ceph.mon.c.vpm152.stderr: 17: (Monitor::_ms_dispatch(Message*)+0x1a6) [0x5d2bb6]
2016-03-20T12:05:40.913 INFO:tasks.ceph.mon.c.vpm152.stderr: 18: (Monitor::ms_dispatch(Message*)+0x23) [0x5f2143]
2016-03-20T12:05:40.913 INFO:tasks.ceph.mon.c.vpm152.stderr: 19: (DispatchQueue::entry()+0x62a) [0x946cba]
2016-03-20T12:05:40.913 INFO:tasks.ceph.mon.c.vpm152.stderr: 20: (DispatchQueue::DispatchThread::entry()+0xd) [0x7dbf5d]
2016-03-20T12:05:40.914 INFO:tasks.ceph.mon.c.vpm152.stderr: 21: (()+0x7dc5) [0x7f8b4ccaedc5]
2016-03-20T12:05:40.914 INFO:tasks.ceph.mon.c.vpm152.stderr: 22: (clone()+0x6d) [0x7f8b4b79028d]
2016-03-20T12:05:40.914 INFO:tasks.ceph.mon.c.vpm152.stderr:2016-03-20 19:05:40.897017 7f8b46b4f700 -1 *** Caught signal (Aborted) **
2016-03-20T12:05:40.914 INFO:tasks.ceph.mon.c.vpm152.stderr: in thread 7f8b46b4f700

Actions

Copy link

Updated by Sage Weil about 8 years ago

teuthology:75866  04:18 AM $ grep ' ceph version' remote/*/log/*mon* | sort -k 2 
remote/vpm152/log/ceph-mon.b.log:2016-03-20 19:01:31.174632 7f8c06184880  0 ceph version 0.94.6-199-g0418943 (0418943c6ef8c9649a58003444daeb4b6224fbab), process ceph-mon, pid 28447
remote/vpm152/log/ceph-mon.c.log:2016-03-20 19:01:31.185969 7f8b4e120880  0 ceph version 0.94.6-199-g0418943 (0418943c6ef8c9649a58003444daeb4b6224fbab), process ceph-mon, pid 28448
remote/vpm086/log/ceph-mon.a.log:2016-03-20 19:01:31.358511 7f1303923880  0 ceph version 0.94.6-199-g0418943 (0418943c6ef8c9649a58003444daeb4b6224fbab), process ceph-mon, pid 28728
remote/vpm086/log/ceph-mon.a.log:2016-03-20 19:03:07.004560 7fa7b09d74c0  0 ceph version 10.0.5-2678-geeabbd0 (eeabbd05f55b06044a62a54d4d4eaf4c309e125d), process ceph-mon, pid 31101

looks like the primary was jewel, and encoded a new-style FSMap.

Actions

Copy link