Bug #10643: "FAILED assert(other->is_writeable())" in upgrade:giant-x-next-distro-basic-vps run - Ceph - Ceph

Actions

Copy link

Bug #10643

closed

"FAILED assert(other->is_writeable())" in upgrade:giant-x-next-distro-basic-vps run

Added by Yuri Weinstein about 9 years ago. Updated about 9 years ago.

Status:

Resolved

Priority:

Urgent

Assignee:

Joao Eduardo Luis

Category:

Monitor

Target version:

% Done:

Source:

Q/A

Tags:

Backport:

giant

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

Run: http://pulpito.ceph.com/teuthology-2015-01-24_17:05:02-upgrade:giant-x-next-distro-basic-vps/
Jobs: ['721645', '721647', '721656']

and

Run: http://pulpito.ceph.com/teuthology-2015-01-25_17:05:01-upgrade:giant-x-next-distro-basic-vps/
Jobs: ['722194', '722196']

Logs for one: http://qa-proxy.ceph.com/teuthology/teuthology-2015-01-25_17:05:01-upgrade:giant-x-next-distro-basic-vps/722194/

2015-01-25T23:47:27.159 INFO:teuthology.orchestra.run.vpm084:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config get mon_pg_warn_min_per_osd'
2015-01-25T23:47:27.483 INFO:tasks.cephfs.filesystem:_json_asok output: { "mon_pg_warn_min_per_osd": "2"}

2015-01-25T23:47:27.484 INFO:teuthology.orchestra.run.vpm084:Running: 'sudo ceph osd pool create metadata 28'
2015-01-25T23:47:28.410 INFO:teuthology.orchestra.run.vpm084.stderr:pool 'metadata' created
2015-01-25T23:47:28.532 INFO:teuthology.orchestra.run.vpm084:Running: 'sudo ceph osd pool create data 28'
2015-01-25T23:47:29.800 INFO:teuthology.orchestra.run.vpm084.stderr:pool 'data' created
2015-01-25T23:47:30.090 INFO:teuthology.orchestra.run.vpm084:Running: 'sudo ceph fs new default metadata data'
2015-01-25T23:47:30.717 INFO:tasks.ceph.mon.a.vpm084.stderr:mon/PaxosService.h: In function 'void PaxosService::request_proposal(PaxosService*)' thread 7f757f089700 time 2015-01-26 07:47:30.714985
2015-01-25T23:47:30.718 INFO:tasks.ceph.mon.a.vpm084.stderr:mon/PaxosService.h: 309: FAILED assert(other->is_writeable())
2015-01-25T23:47:31.780 INFO:tasks.ceph.mon.a.vpm084.stderr: ceph version 0.87-81-ga463b92 (a463b92e475cd1f4cdb963e402033ebc9d37dbdc)
2015-01-25T23:47:31.781 INFO:tasks.ceph.mon.a.vpm084.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x72) [0x860662]
2015-01-25T23:47:31.781 INFO:tasks.ceph.mon.a.vpm084.stderr: 2: ceph-mon() [0x702287]
2015-01-25T23:47:31.781 INFO:tasks.ceph.mon.a.vpm084.stderr: 3: (MDSMonitor::management_command(std::string const&, std::map<std::string, boost::variant<std::string, bool, long, double, std::vector<std::string, std::allocator<std::string> >, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>, std::less<std::string>, std::allocator<std::pair<std::string const, boost::variant<std::string, bool, long, double, std::vector<std::string, std::allocator<std::string> >, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> > > >&, std::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >&, int&)+0x448) [0x70c188]
2015-01-25T23:47:31.781 INFO:tasks.ceph.mon.a.vpm084.stderr: 4: (MDSMonitor::prepare_command(MMonCommand*)+0x26a) [0x7104ca]
2015-01-25T23:47:31.782 INFO:tasks.ceph.mon.a.vpm084.stderr: 5: (MDSMonitor::prepare_update(PaxosServiceMessage*)+0x16b) [0x7108fb]
2015-01-25T23:47:31.782 INFO:tasks.ceph.mon.a.vpm084.stderr: 6: (PaxosService::dispatch(PaxosServiceMessage*)+0x5f1) [0x6afbd1]
2015-01-25T23:47:31.782 INFO:tasks.ceph.mon.a.vpm084.stderr: 7: (Monitor::handle_command(MMonCommand*)+0xa16) [0x66fd96]
2015-01-25T23:47:31.782 INFO:tasks.ceph.mon.a.vpm084.stderr: 8: (Monitor::dispatch(MonSession*, Message*, bool)+0x22b) [0x6798eb]
2015-01-25T23:47:31.783 INFO:tasks.ceph.mon.a.vpm084.stderr: 9: (Monitor::_ms_dispatch(Message*)+0x1a7) [0x67a1b7]
2015-01-25T23:47:31.783 INFO:tasks.ceph.mon.a.vpm084.stderr: 10: (Monitor::ms_dispatch(Message*)+0x32) [0x698dc2]
2015-01-25T23:47:31.783 INFO:tasks.ceph.mon.a.vpm084.stderr: 11: (DispatchQueue::entry()+0x45b) [0x99ec5b]
2015-01-25T23:47:31.783 INFO:tasks.ceph.mon.a.vpm084.stderr: 12: (DispatchQueue::DispatchThread::entry()+0xd) [0x84708d]
2015-01-25T23:47:31.784 INFO:tasks.ceph.mon.a.vpm084.stderr: 13: (()+0x6b50) [0x7f7585093b50]
2015-01-25T23:47:31.784 INFO:tasks.ceph.mon.a.vpm084.stderr: 14: (clone()+0x6d) [0x7f7583aafa7d]
2015-01-25T23:47:31.784 INFO:tasks.ceph.mon.a.vpm084.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2015-01-25T23:47:31.784 INFO:tasks.ceph.mon.a.vpm084.stderr:2015-01-26 07:47:31.777074 7f757f089700 -1 mon/PaxosService.h: In function 'void PaxosService::request_proposal(PaxosService*)' thread 7f757f089700 time 2015-01-26 07:47:30.714985
2015-01-25T23:47:31.785 INFO:tasks.ceph.mon.a.vpm084.stderr:mon/PaxosService.h: 309: FAILED assert(other->is_writeable())
2015-01-25T23:47:31.785 INFO:tasks.ceph.mon.a.vpm084.stderr:
2015-01-25T23:47:31.785 INFO:tasks.ceph.mon.a.vpm084.stderr: ceph version 0.87-81-ga463b92 (a463b92e475cd1f4cdb963e402033ebc9d37dbdc)
2015-01-25T23:47:31.785 INFO:tasks.ceph.mon.a.vpm084.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x72) [0x860662]
2015-01-25T23:47:31.786 INFO:tasks.ceph.mon.a.vpm084.stderr: 2: ceph-mon() [0x702287]
2015-01-25T23:47:31.786 INFO:tasks.ceph.mon.a.vpm084.stderr: 3: (MDSMonitor::management_command(std::string const&, std::map<std::string, boost::variant<std::string, bool, long, double, std::vector<std::string, std::allocator<std::string> >, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>, std::less<std::string>, std::allocator<std::pair<std::string const, boost::variant<std::string, bool, long, double, std::vector<std::string, std::allocator<std::string> >, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> > > >&, std::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >&, int&)+0x448) [0x70c188]
2015-01-25T23:47:31.786 INFO:tasks.ceph.mon.a.vpm084.stderr: 4: (MDSMonitor::prepare_command(MMonCommand*)+0x26a) [0x7104ca]
2015-01-25T23:47:31.786 INFO:tasks.ceph.mon.a.vpm084.stderr: 5: (MDSMonitor::prepare_update(PaxosServiceMessage*)+0x16b) [0x7108fb]
2015-01-25T23:47:31.786 INFO:tasks.ceph.mon.a.vpm084.stderr: 6: (PaxosService::dispatch(PaxosServiceMessage*)+0x5f1) [0x6afbd1]
2015-01-25T23:47:31.787 INFO:tasks.ceph.mon.a.vpm084.stderr: 7: (Monitor::handle_command(MMonCommand*)+0xa16) [0x66fd96]
2015-01-25T23:47:31.787 INFO:tasks.ceph.mon.a.vpm084.stderr: 8: (Monitor::dispatch(MonSession*, Message*, bool)+0x22b) [0x6798eb]
2015-01-25T23:47:31.787 INFO:tasks.ceph.mon.a.vpm084.stderr: 9: (Monitor::_ms_dispatch(Message*)+0x1a7) [0x67a1b7]
2015-01-25T23:47:31.787 INFO:tasks.ceph.mon.a.vpm084.stderr: 10: (Monitor::ms_dispatch(Message*)+0x32) [0x698dc2]
2015-01-25T23:47:31.787 INFO:tasks.ceph.mon.a.vpm084.stderr: 11: (DispatchQueue::entry()+0x45b) [0x99ec5b]
2015-01-25T23:47:31.787 INFO:tasks.ceph.mon.a.vpm084.stderr: 12: (DispatchQueue::DispatchThread::entry()+0xd) [0x84708d]
2015-01-25T23:47:31.788 INFO:tasks.ceph.mon.a.vpm084.stderr: 13: (()+0x6b50) [0x7f7585093b50]
2015-01-25T23:47:31.788 INFO:tasks.ceph.mon.a.vpm084.stderr: 14: (clone()+0x6d) [0x7f7583aafa7d]
2015-01-25T23:47:31.788 INFO:tasks.ceph.mon.a.vpm084.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

Actions

Copy link

Updated by Sage Weil about 9 years ago

Assignee set to Joao Eduardo Luis
Priority changed from Normal to Urgent

Actions

Copy link

Updated by Joao Eduardo Luis about 9 years ago

Category set to Monitor
Status changed from New to Pending Backport
Backport set to giant

This was caused by missing backports on giant. Pull request has been created: https://github.com/ceph/ceph/pull/3548

Actions

Copy link

Updated by Yuri Weinstein about 9 years ago

Same in
Run: http://pulpito.ceph.com/teuthology-2015-02-01_17:05:01-upgrade:giant-x-next-distro-basic-vps/
Jobs: ['734955', '734963']
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2015-02-01_17:05:01-upgrade:giant-x-next-distro-basic-vps/734963/

Actions

Copy link

Updated by Yuri Weinstein about 9 years ago

Same in
Run: http://pulpito.ceph.com/teuthology-2015-02-02_17:05:01-upgrade:giant-x-next-distro-basic-vps/
Job: 737468
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2015-02-02_17:05:01-upgrade:giant-x-next-distro-basic-vps/737468/

Actions

Copy link

Updated by Yuri Weinstein about 9 years ago

Same in
Run: http://pulpito.ceph.com/teuthology-2015-02-03_17:05:01-upgrade:giant-x-next-distro-basic-vps/
Job: ['739079']
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2015-02-03_17:05:01-upgrade:giant-x-next-distro-basic-vps/739079/

Actions

Copy link

Updated by Yuri Weinstein about 9 years ago

Same in hammer
Run: http://pulpito.ceph.com/teuthology-2015-02-22_17:05:02-upgrade:giant-x-hammer-distro-basic-vps/
Job: 774610
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2015-02-22_17:05:02-upgrade:giant-x-hammer-distro-basic-vps/774610/

Consider hammer backport as well.

Actions

Copy link

Updated by Yuri Weinstein about 9 years ago

Run: http://pulpito.ceph.com/teuthology-2015-03-06_17:05:02-upgrade:giant-x-hammer-distro-basic-vps/
Jobs: ['794291', '794297']

Assertion: mon/PaxosService.h: 309: FAILED assert(other->is_writeable())
ceph version 0.87.1-1-g938e036 (938e03630e075af03780da139ae879b5b0377734)
 1: (PaxosService::request_proposal(PaxosService*)+0x47) [0x62e877]
 2: (MDSMonitor::management_command(std::string const&, std::map<std::string, boost::variant<std::string, bool, long, double, std::vector<std::string, std::allocator<std::string> >, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>, std::less<std::string>, std::allocator<std::pair<std::string const, boost::variant<std::string, bool, long, double, std::vector<std::string, std::allocator<std::string> >, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> > > >&, std::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >&, int&)+0xa1a) [0x620c4a]
 3: (MDSMonitor::prepare_command(MMonCommand*)+0x49e) [0x62795e]
 4: (MDSMonitor::prepare_update(PaxosServiceMessage*)+0xa3) [0x627f13]
 5: (PaxosService::dispatch(PaxosServiceMessage*)+0x927) [0x5c0847]
 6: (Monitor::handle_command(MMonCommand*)+0xede) [0x5897ee]
 7: (Monitor::dispatch(MonSession*, Message*, bool)+0x24b) [0x58bafb]
 8: (Monitor::_ms_dispatch(Message*)+0x214) [0x58c3f4]
 9: (Monitor::ms_dispatch(Message*)+0x32) [0x5aa722]
 10: (DispatchQueue::entry()+0x4fa) [0x82c99a]
 11: (DispatchQueue::DispatchThread::entry()+0xd) [0x80b19d]
 12: (()+0x79d1) [0x7fe7954449d1]
 13: (clone()+0x6d) [0x7fe7943d486d]

Actions

Copy link

Updated by Yuri Weinstein about 9 years ago

Run: http://pulpito.ceph.com/teuthology-2015-03-09_17:05:01-upgrade:giant-x-hammer-distro-basic-vps/
Job: ['797710']
Logs: http://pulpito.ceph.com/teuthology-2015-03-09_17:05:01-upgrade:giant-x-hammer-distro-basic-vps/797710/

Actions

Copy link

Updated by Yuri Weinstein about 9 years ago

Run: http://pulpito.ceph.com/teuthology-2015-03-11_17:05:01-upgrade:giant-x-hammer-distro-basic-multi/
Job: ['801128']

Actions

Copy link

#10

Updated by Yuri Weinstein about 9 years ago

Run: http://pulpito.ceph.com/teuthology-2015-03-15_17:05:01-upgrade:giant-x-hammer-distro-basic-vps/
Job: ['805064', '805071']

Actions

Copy link

#11

Updated by Yuri Weinstein about 9 years ago

Run: http://pulpito.ceph.com/teuthology-2015-03-23_17:05:01-upgrade:giant-x-hammer-distro-basic-multi/
Job: ['818518']

It seemed to pass already on hammer, maybe backporing?

Assertion: mon/PaxosService.h: 309: FAILED assert(other->is_writeable())
ceph version 0.87.1-99-g2ccbc14 (2ccbc14d17b54ea4fd4126cb04a7b83cd64c7f1e)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x7c573b]
 2: ceph-mon() [0x64d797]
 3: (MDSMonitor::management_command(std::string const&, std::map<std::string, boost::variant<std::string, bool, long, double, std::vector<std::string, std::allocator<std::string> >, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>, std::less<std::string>, std::allocator<std::pair<std::string const, boost::variant<std::string, bool, long, double, std::vector<std::string, std::allocator<std::string> >, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> > > >&, std::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >&, int&)+0x7d5) [0x657885]
 4: (MDSMonitor::prepare_command(MMonCommand*)+0x430) [0x65ab80]
 5: (MDSMonitor::prepare_update(PaxosServiceMessage*)+0x19b) [0x65b28b]
 6: (PaxosService::dispatch(PaxosServiceMessage*)+0xbcc) [0x5f4b9c]
 7: (Monitor::handle_command(MMonCommand*)+0xfcd) [0x5bb60d]
 8: (Monitor::dispatch(MonSession*, Message*, bool)+0x32b) [0x5be7eb]
 9: (Monitor::_ms_dispatch(Message*)+0x1a5) [0x5bf245]
 10: (Monitor::handle_forward(MForward*)+0x619) [0x5be0f9]
 11: (Monitor::dispatch(MonSession*, Message*, bool)+0x7cb) [0x5bec8b]
 12: (Monitor::_ms_dispatch(Message*)+0x1a5) [0x5bf245]
 13: (Monitor::ms_dispatch(Message*)+0x23) [0x5de9a3]
 14: (DispatchQueue::entry()+0x649) [0x8fb5a9]
 15: (DispatchQueue::DispatchThread::entry()+0xd) [0x7aa98d]
 16: (()+0x8182) [0x7fe198f97182]
 17: (clone()+0x6d) [0x7fe19750330d]

Actions

Copy link

#12