Bug #10643
closed"FAILED assert(other->is_writeable())" in upgrade:giant-x-next-distro-basic-vps run
0%
Description
Run: http://pulpito.ceph.com/teuthology-2015-01-24_17:05:02-upgrade:giant-x-next-distro-basic-vps/
Jobs: ['721645', '721647', '721656']
and
Run: http://pulpito.ceph.com/teuthology-2015-01-25_17:05:01-upgrade:giant-x-next-distro-basic-vps/
Jobs: ['722194', '722196']
Logs for one: http://qa-proxy.ceph.com/teuthology/teuthology-2015-01-25_17:05:01-upgrade:giant-x-next-distro-basic-vps/722194/
2015-01-25T23:47:27.159 INFO:teuthology.orchestra.run.vpm084:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config get mon_pg_warn_min_per_osd' 2015-01-25T23:47:27.483 INFO:tasks.cephfs.filesystem:_json_asok output: { "mon_pg_warn_min_per_osd": "2"} 2015-01-25T23:47:27.484 INFO:teuthology.orchestra.run.vpm084:Running: 'sudo ceph osd pool create metadata 28' 2015-01-25T23:47:28.410 INFO:teuthology.orchestra.run.vpm084.stderr:pool 'metadata' created 2015-01-25T23:47:28.532 INFO:teuthology.orchestra.run.vpm084:Running: 'sudo ceph osd pool create data 28' 2015-01-25T23:47:29.800 INFO:teuthology.orchestra.run.vpm084.stderr:pool 'data' created 2015-01-25T23:47:30.090 INFO:teuthology.orchestra.run.vpm084:Running: 'sudo ceph fs new default metadata data' 2015-01-25T23:47:30.717 INFO:tasks.ceph.mon.a.vpm084.stderr:mon/PaxosService.h: In function 'void PaxosService::request_proposal(PaxosService*)' thread 7f757f089700 time 2015-01-26 07:47:30.714985 2015-01-25T23:47:30.718 INFO:tasks.ceph.mon.a.vpm084.stderr:mon/PaxosService.h: 309: FAILED assert(other->is_writeable()) 2015-01-25T23:47:31.780 INFO:tasks.ceph.mon.a.vpm084.stderr: ceph version 0.87-81-ga463b92 (a463b92e475cd1f4cdb963e402033ebc9d37dbdc) 2015-01-25T23:47:31.781 INFO:tasks.ceph.mon.a.vpm084.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x72) [0x860662] 2015-01-25T23:47:31.781 INFO:tasks.ceph.mon.a.vpm084.stderr: 2: ceph-mon() [0x702287] 2015-01-25T23:47:31.781 INFO:tasks.ceph.mon.a.vpm084.stderr: 3: (MDSMonitor::management_command(std::string const&, std::map<std::string, boost::variant<std::string, bool, long, double, std::vector<std::string, std::allocator<std::string> >, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>, std::less<std::string>, std::allocator<std::pair<std::string const, boost::variant<std::string, bool, long, double, std::vector<std::string, std::allocator<std::string> >, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> > > >&, std::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >&, int&)+0x448) [0x70c188] 2015-01-25T23:47:31.781 INFO:tasks.ceph.mon.a.vpm084.stderr: 4: (MDSMonitor::prepare_command(MMonCommand*)+0x26a) [0x7104ca] 2015-01-25T23:47:31.782 INFO:tasks.ceph.mon.a.vpm084.stderr: 5: (MDSMonitor::prepare_update(PaxosServiceMessage*)+0x16b) [0x7108fb] 2015-01-25T23:47:31.782 INFO:tasks.ceph.mon.a.vpm084.stderr: 6: (PaxosService::dispatch(PaxosServiceMessage*)+0x5f1) [0x6afbd1] 2015-01-25T23:47:31.782 INFO:tasks.ceph.mon.a.vpm084.stderr: 7: (Monitor::handle_command(MMonCommand*)+0xa16) [0x66fd96] 2015-01-25T23:47:31.782 INFO:tasks.ceph.mon.a.vpm084.stderr: 8: (Monitor::dispatch(MonSession*, Message*, bool)+0x22b) [0x6798eb] 2015-01-25T23:47:31.783 INFO:tasks.ceph.mon.a.vpm084.stderr: 9: (Monitor::_ms_dispatch(Message*)+0x1a7) [0x67a1b7] 2015-01-25T23:47:31.783 INFO:tasks.ceph.mon.a.vpm084.stderr: 10: (Monitor::ms_dispatch(Message*)+0x32) [0x698dc2] 2015-01-25T23:47:31.783 INFO:tasks.ceph.mon.a.vpm084.stderr: 11: (DispatchQueue::entry()+0x45b) [0x99ec5b] 2015-01-25T23:47:31.783 INFO:tasks.ceph.mon.a.vpm084.stderr: 12: (DispatchQueue::DispatchThread::entry()+0xd) [0x84708d] 2015-01-25T23:47:31.784 INFO:tasks.ceph.mon.a.vpm084.stderr: 13: (()+0x6b50) [0x7f7585093b50] 2015-01-25T23:47:31.784 INFO:tasks.ceph.mon.a.vpm084.stderr: 14: (clone()+0x6d) [0x7f7583aafa7d] 2015-01-25T23:47:31.784 INFO:tasks.ceph.mon.a.vpm084.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. 2015-01-25T23:47:31.784 INFO:tasks.ceph.mon.a.vpm084.stderr:2015-01-26 07:47:31.777074 7f757f089700 -1 mon/PaxosService.h: In function 'void PaxosService::request_proposal(PaxosService*)' thread 7f757f089700 time 2015-01-26 07:47:30.714985 2015-01-25T23:47:31.785 INFO:tasks.ceph.mon.a.vpm084.stderr:mon/PaxosService.h: 309: FAILED assert(other->is_writeable()) 2015-01-25T23:47:31.785 INFO:tasks.ceph.mon.a.vpm084.stderr: 2015-01-25T23:47:31.785 INFO:tasks.ceph.mon.a.vpm084.stderr: ceph version 0.87-81-ga463b92 (a463b92e475cd1f4cdb963e402033ebc9d37dbdc) 2015-01-25T23:47:31.785 INFO:tasks.ceph.mon.a.vpm084.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x72) [0x860662] 2015-01-25T23:47:31.786 INFO:tasks.ceph.mon.a.vpm084.stderr: 2: ceph-mon() [0x702287] 2015-01-25T23:47:31.786 INFO:tasks.ceph.mon.a.vpm084.stderr: 3: (MDSMonitor::management_command(std::string const&, std::map<std::string, boost::variant<std::string, bool, long, double, std::vector<std::string, std::allocator<std::string> >, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>, std::less<std::string>, std::allocator<std::pair<std::string const, boost::variant<std::string, bool, long, double, std::vector<std::string, std::allocator<std::string> >, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> > > >&, std::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >&, int&)+0x448) [0x70c188] 2015-01-25T23:47:31.786 INFO:tasks.ceph.mon.a.vpm084.stderr: 4: (MDSMonitor::prepare_command(MMonCommand*)+0x26a) [0x7104ca] 2015-01-25T23:47:31.786 INFO:tasks.ceph.mon.a.vpm084.stderr: 5: (MDSMonitor::prepare_update(PaxosServiceMessage*)+0x16b) [0x7108fb] 2015-01-25T23:47:31.786 INFO:tasks.ceph.mon.a.vpm084.stderr: 6: (PaxosService::dispatch(PaxosServiceMessage*)+0x5f1) [0x6afbd1] 2015-01-25T23:47:31.787 INFO:tasks.ceph.mon.a.vpm084.stderr: 7: (Monitor::handle_command(MMonCommand*)+0xa16) [0x66fd96] 2015-01-25T23:47:31.787 INFO:tasks.ceph.mon.a.vpm084.stderr: 8: (Monitor::dispatch(MonSession*, Message*, bool)+0x22b) [0x6798eb] 2015-01-25T23:47:31.787 INFO:tasks.ceph.mon.a.vpm084.stderr: 9: (Monitor::_ms_dispatch(Message*)+0x1a7) [0x67a1b7] 2015-01-25T23:47:31.787 INFO:tasks.ceph.mon.a.vpm084.stderr: 10: (Monitor::ms_dispatch(Message*)+0x32) [0x698dc2] 2015-01-25T23:47:31.787 INFO:tasks.ceph.mon.a.vpm084.stderr: 11: (DispatchQueue::entry()+0x45b) [0x99ec5b] 2015-01-25T23:47:31.787 INFO:tasks.ceph.mon.a.vpm084.stderr: 12: (DispatchQueue::DispatchThread::entry()+0xd) [0x84708d] 2015-01-25T23:47:31.788 INFO:tasks.ceph.mon.a.vpm084.stderr: 13: (()+0x6b50) [0x7f7585093b50] 2015-01-25T23:47:31.788 INFO:tasks.ceph.mon.a.vpm084.stderr: 14: (clone()+0x6d) [0x7f7583aafa7d] 2015-01-25T23:47:31.788 INFO:tasks.ceph.mon.a.vpm084.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Updated by Sage Weil over 9 years ago
- Assignee set to Joao Eduardo Luis
- Priority changed from Normal to Urgent
Updated by Joao Eduardo Luis over 9 years ago
- Category set to Monitor
- Status changed from New to Pending Backport
- Backport set to giant
This was caused by missing backports on giant. Pull request has been created: https://github.com/ceph/ceph/pull/3548
Updated by Yuri Weinstein about 9 years ago
Updated by Yuri Weinstein about 9 years ago
Updated by Yuri Weinstein about 9 years ago
Same in hammer
Run: http://pulpito.ceph.com/teuthology-2015-02-22_17:05:02-upgrade:giant-x-hammer-distro-basic-vps/
Job: 774610
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2015-02-22_17:05:02-upgrade:giant-x-hammer-distro-basic-vps/774610/
Consider hammer backport as well.
Updated by Yuri Weinstein about 9 years ago
Run: http://pulpito.ceph.com/teuthology-2015-03-06_17:05:02-upgrade:giant-x-hammer-distro-basic-vps/
Jobs: ['794291', '794297']
Assertion: mon/PaxosService.h: 309: FAILED assert(other->is_writeable()) ceph version 0.87.1-1-g938e036 (938e03630e075af03780da139ae879b5b0377734) 1: (PaxosService::request_proposal(PaxosService*)+0x47) [0x62e877] 2: (MDSMonitor::management_command(std::string const&, std::map<std::string, boost::variant<std::string, bool, long, double, std::vector<std::string, std::allocator<std::string> >, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>, std::less<std::string>, std::allocator<std::pair<std::string const, boost::variant<std::string, bool, long, double, std::vector<std::string, std::allocator<std::string> >, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> > > >&, std::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >&, int&)+0xa1a) [0x620c4a] 3: (MDSMonitor::prepare_command(MMonCommand*)+0x49e) [0x62795e] 4: (MDSMonitor::prepare_update(PaxosServiceMessage*)+0xa3) [0x627f13] 5: (PaxosService::dispatch(PaxosServiceMessage*)+0x927) [0x5c0847] 6: (Monitor::handle_command(MMonCommand*)+0xede) [0x5897ee] 7: (Monitor::dispatch(MonSession*, Message*, bool)+0x24b) [0x58bafb] 8: (Monitor::_ms_dispatch(Message*)+0x214) [0x58c3f4] 9: (Monitor::ms_dispatch(Message*)+0x32) [0x5aa722] 10: (DispatchQueue::entry()+0x4fa) [0x82c99a] 11: (DispatchQueue::DispatchThread::entry()+0xd) [0x80b19d] 12: (()+0x79d1) [0x7fe7954449d1] 13: (clone()+0x6d) [0x7fe7943d486d]
Updated by Yuri Weinstein about 9 years ago
Updated by Yuri Weinstein about 9 years ago
Run: http://pulpito.ceph.com/teuthology-2015-03-15_17:05:01-upgrade:giant-x-hammer-distro-basic-vps/
Job: ['805064', '805071']
Updated by Yuri Weinstein about 9 years ago
Run: http://pulpito.ceph.com/teuthology-2015-03-23_17:05:01-upgrade:giant-x-hammer-distro-basic-multi/
Job: ['818518']
It seemed to pass already on hammer, maybe backporing?
Assertion: mon/PaxosService.h: 309: FAILED assert(other->is_writeable()) ceph version 0.87.1-99-g2ccbc14 (2ccbc14d17b54ea4fd4126cb04a7b83cd64c7f1e) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x7c573b] 2: ceph-mon() [0x64d797] 3: (MDSMonitor::management_command(std::string const&, std::map<std::string, boost::variant<std::string, bool, long, double, std::vector<std::string, std::allocator<std::string> >, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>, std::less<std::string>, std::allocator<std::pair<std::string const, boost::variant<std::string, bool, long, double, std::vector<std::string, std::allocator<std::string> >, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> > > >&, std::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >&, int&)+0x7d5) [0x657885] 4: (MDSMonitor::prepare_command(MMonCommand*)+0x430) [0x65ab80] 5: (MDSMonitor::prepare_update(PaxosServiceMessage*)+0x19b) [0x65b28b] 6: (PaxosService::dispatch(PaxosServiceMessage*)+0xbcc) [0x5f4b9c] 7: (Monitor::handle_command(MMonCommand*)+0xfcd) [0x5bb60d] 8: (Monitor::dispatch(MonSession*, Message*, bool)+0x32b) [0x5be7eb] 9: (Monitor::_ms_dispatch(Message*)+0x1a5) [0x5bf245] 10: (Monitor::handle_forward(MForward*)+0x619) [0x5be0f9] 11: (Monitor::dispatch(MonSession*, Message*, bool)+0x7cb) [0x5bec8b] 12: (Monitor::_ms_dispatch(Message*)+0x1a5) [0x5bf245] 13: (Monitor::ms_dispatch(Message*)+0x23) [0x5de9a3] 14: (DispatchQueue::entry()+0x649) [0x8fb5a9] 15: (DispatchQueue::DispatchThread::entry()+0xd) [0x7aa98d] 16: (()+0x8182) [0x7fe198f97182] 17: (clone()+0x6d) [0x7fe19750330d]
Updated by Sage Weil about 9 years ago
- Status changed from Pending Backport to 12
Joao, that last failure commit contains the backported fix. :(
Updated by Loïc Dachary about 9 years ago
- Status changed from 12 to Pending Backport
- Backport set to giant
since this breaks upgrades, let say we wait on it before publishing giant v0.87.2 ... unless someone has a different opinion ?
Updated by Yuri Weinstein about 9 years ago
Updated by Joao Eduardo Luis about 9 years ago
missing backport is the cause. Will open a pull request once it finishes compiling.
Updated by Joao Eduardo Luis about 9 years ago
Updated by Loïc Dachary about 9 years ago
- Status changed from Pending Backport to Resolved
passed tests (rados & fs) and merged https://github.com/ceph/ceph/pull/4258 . Let's bet it will disapear from the other tests as well ;-)