Bug #16453: "mon/Paxos.cc: 197: FAILED assert(mon->is_peon())" in rados-hammer-distro-basic-vps - Ceph - Ceph

Actions

Copy link

Bug #16453

closed

"mon/Paxos.cc: 197: FAILED assert(mon->is_peon())" in rados-hammer-distro-basic-vps

Added by Yuri Weinstein almost 8 years ago. Updated almost 7 years ago.

Status:

Can't reproduce

Priority:

Normal

Assignee:

Category:

Target version:

% Done:

Source:

Q/A

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

rados

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

Run: http://pulpito.ceph.com/teuthology-2016-06-22_03:00:02-rados-hammer-distro-basic-vps/
Job: 271151
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2016-06-22_03:00:02-rados-hammer-distro-basic-vps/271151/teuthology.log

2016-06-22T09:28:53.571 INFO:teuthology.orchestra.run.vpm167.stderr:2016-06-22 16:28:53.654055 7f3388e098c0 -1 filestore(/var/lib/ceph/osd/ceph-5) could not find -1/23c2fcde/osd_superblock/0 in index: (2) No such file or directory
2016-06-22T09:28:53.597 INFO:teuthology.orchestra.run.vpm167.stderr:2016-06-22 16:28:53.679871 7f3388e098c0 -1 created object store /var/lib/ceph/osd/ceph-5 journal /var/lib/ceph/osd/ceph-5/journal for osd.5 fsid 9f9c8125-9aed-4f34-b657-6415d1b9f50c
2016-06-22T09:28:53.597 INFO:teuthology.orchestra.run.vpm167.stderr:2016-06-22 16:28:53.679908 7f3388e098c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory
2016-06-22T09:28:53.598 INFO:teuthology.orchestra.run.vpm167.stderr:2016-06-22 16:28:53.680015 7f3388e098c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring
2016-06-22T09:28:53.599 INFO:tasks.ceph:Reading keys from all nodes...

... and

2016-06-22T09:28:56.625 INFO:teuthology.orchestra.run.vpm055:Running: 'sudo ceph osd crush tunables default'
2016-06-22T09:28:56.682 INFO:tasks.ceph.mon.b.vpm167.stdout:starting mon.b rank 1 at 172.21.2.167:6789/0 mon_data /var/lib/ceph/mon/ceph-b fsid 9f9c8125-9aed-4f34-b657-6415d1b9f50c
2016-06-22T09:28:56.721 INFO:tasks.ceph.mon.a.vpm055.stdout:starting mon.a rank 0 at 172.21.2.55:6789/0 mon_data /var/lib/ceph/mon/ceph-a fsid 9f9c8125-9aed-4f34-b657-6415d1b9f50c
2016-06-22T09:28:56.723 INFO:tasks.ceph.mon.c.vpm055.stdout:starting mon.c rank 2 at 172.21.2.55:6790/0 mon_data /var/lib/ceph/mon/ceph-c fsid 9f9c8125-9aed-4f34-b657-6415d1b9f50c
2016-06-22T09:29:16.706 INFO:tasks.ceph.mon.b.vpm167.stderr:mon/Paxos.cc: In function 'void Paxos::handle_collect(MMonPaxos*)' thread 7f0a80d69700 time 2016-06-22 16:29:16.788170
2016-06-22T09:29:16.706 INFO:tasks.ceph.mon.b.vpm167.stderr:mon/Paxos.cc: 197: FAILED assert(mon->is_peon())
2016-06-22T09:29:16.707 INFO:tasks.ceph.mon.b.vpm167.stderr: ceph version 0.94.7-26-g2e156d7 (2e156d7ad4b9f4ffd6028df3a460b50b30c8b0d3)
2016-06-22T09:29:16.707 INFO:tasks.ceph.mon.b.vpm167.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x7e51ab]
2016-06-22T09:29:16.707 INFO:tasks.ceph.mon.b.vpm167.stderr: 2: (Paxos::handle_collect(MMonPaxos*)+0xe7b) [0x5faa3b]
2016-06-22T09:29:16.707 INFO:tasks.ceph.mon.b.vpm167.stderr: 3: (Paxos::dispatch(PaxosServiceMessage*)+0x1db) [0x5ff34b]
2016-06-22T09:29:16.708 INFO:tasks.ceph.mon.b.vpm167.stderr: 4: (Monitor::dispatch(MonSession*, Message*, bool)+0x923) [0x5d1753]
2016-06-22T09:29:16.708 INFO:tasks.ceph.mon.b.vpm167.stderr: 5: (Monitor::_ms_dispatch(Message*)+0x1a6) [0x5d1ba6]
2016-06-22T09:29:16.708 INFO:tasks.ceph.mon.b.vpm167.stderr: 6: (Monitor::ms_dispatch(Message*)+0x23) [0x5efa23]
2016-06-22T09:29:16.708 INFO:tasks.ceph.mon.b.vpm167.stderr: 7: (DispatchQueue::entry()+0x649) [0x92d929]
2016-06-22T09:29:16.708 INFO:tasks.ceph.mon.b.vpm167.stderr: 8: (DispatchQueue::DispatchThread::entry()+0xd) [0x7cdc2d]
2016-06-22T09:29:16.709 INFO:tasks.ceph.mon.b.vpm167.stderr: 9: (()+0x8184) [0x7f0a87663184]
2016-06-22T09:29:16.709 INFO:tasks.ceph.mon.b.vpm167.stderr: 10: (clone()+0x6d) [0x7f0a85bce37d]
2016-06-22T09:29:16.709 INFO:tasks.ceph.mon.b.vpm167.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.2016-06-22T09:28:53.570 INFO:teuthology.orchestra.run.vpm167.stderr:2016-06-22 16:28:53.653255 7f3388e098c0 -1 journal FileJournal::_open: disabling aio for non-block journal.  Use journal_force_aio to force use of aio anyway
2016-06-22T09:28:53.571 INFO:teuthology.orchestra.run.vpm167.stderr:2016-06-22 16:28:53.654055 7f3388e098c0 -1 filestore(/var/lib/ceph/osd/ceph-5) could not find -1/23c2fcde/osd_superblock/0 in index: (2) No such file or directory
2016-06-22T09:28:53.597 INFO:teuthology.orchestra.run.vpm167.stderr:2016-06-22 16:28:53.679871 7f3388e098c0 -1 created object store /var/lib/ceph/osd/ceph-5 journal /var/lib/ceph/osd/ceph-5/journal for osd.5 fsid 9f9c8125-9aed-4f34-b657-6415d1b9f50c
2016-06-22T09:28:53.597 INFO:teuthology.orchestra.run.vpm167.stderr:2016-06-22 16:28:53.679908 7f3388e098c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory
2016-06-22T09:28:53.598 INFO:teuthology.orchestra.run.vpm167.stderr:2016-06-22 16:28:53.680015 7f3388e098c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring2016-06-22T09:28:53.570 INFO:teuthology.orchestra.run.vpm167.stderr:2016-06-22 16:28:53.653255 7f3388e098c0 -1 journal FileJournal::_open: disabling aio for non-block journal.  Use journal_force_aio to force use of aio anyway
2016-06-22T09:28:53.571 INFO:teuthology.orchestra.run.vpm167.stderr:2016-06-22 16:28:53.654055 7f3388e098c0 -1 filestore(/var/lib/ceph/osd/ceph-5) could not find -1/23c2fcde/osd_superblock/0 in index: (2) No such file or directory
2016-06-22T09:28:53.597 INFO:teuthology.orchestra.run.vpm167.stderr:2016-06-22 16:28:53.679871 7f3388e098c0 -1 created object store /var/lib/ceph/osd/ceph-5 journal /var/lib/ceph/osd/ceph-5/journal for osd.5 fsid 9f9c8125-9aed-4f34-b657-6415d1b9f50c
2016-06-22T09:28:53.597 INFO:teuthology.orchestra.run.vpm167.stderr:2016-06-22 16:28:53.679908 7f3388e098c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory
2016-06-22T09:28:53.598 INFO:teuthology.orchestra.run.vpm167.stderr:2016-06-22 16:28:53.680015 7f3388e098c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring2016-06-22T09:28:53.570 INFO:teuthology.orchestra.run.vpm167.stderr:2016-06-22 16:28:53.653255 7f3388e098c0 -1 journal FileJournal::_open: disabling aio for non-block journal.  Use journal_force_aio to force use of aio anyway
2016-06-22T09:28:53.571 INFO:teuthology.orchestra.run.vpm167.stderr:2016-06-22 16:28:53.654055 7f3388e098c0 -1 filestore(/var/lib/ceph/osd/ceph-5) could not find -1/23c2fcde/osd_superblock/0 in index: (2) No such file or directory
2016-06-22T09:28:53.597 INFO:teuthology.orchestra.run.vpm167.stderr:2016-06-22 16:28:53.679871 7f3388e098c0 -1 created object store /var/lib/ceph/osd/ceph-5 journal /var/lib/ceph/osd/ceph-5/journal for osd.5 fsid 9f9c8125-9aed-4f34-b657-6415d1b9f50c
2016-06-22T09:28:2016-06-22T09:28:53.570 INFO:teuthology.orchestra.run.vpm167.stderr:2016-06-22 16:28:53.653255 7f3388e098c0 -1 j2016-06-22T09:28:53.570 INFO:teuthology.orchestra.run.vpm167.stderr:2016-06-22 16:28:53.653255 7f3388e098c0 -1 journal FileJournal::_open: disabling aio for non-block journal.  Use journal_force_aio to force use of aio anyway
2016-06-22T09:28:53.571 INFO:teuthology.orchestra.run.vpm167.stderr:2016-06-22 16:28:53.654055 7f3388e098c0 -1 filestore(/var/lib/ceph/osd/ceph-5) could not find -1/23c2fcde/osd_superblock/0 in index: (2) No such file or directory
2016-06-22T09:28:53.597 INFO:teuthology.orchestra.run.vpm167.stderr:2016-06-22 16:28:53.679871 7f3388e098c0 -1 created object store /var/lib/ceph/osd/ceph-5 journal /var/lib/ceph/osd/ceph-5/journal for osd.5 fsid 9f9c8125-9aed-4f34-b657-6415d1b9f50c
2016-06-22T09:28:53.597 INFO:teuthology.orchestra.run.vpm167.stderr:2016-06-22 16:28:53.679908 7f3388e098c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory
2016-06-22T09:28:53.598 INFO:teuthology.orchestra.run.vpm167.stderr:2016-06-22 16:28:53.680015 7f3388e098c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyringournal FileJournal::_open: disabling aio for non-block journal.  Use journal_force_aio to force use of aio anyway
2016-06-22T09:28:53.571 INFO:teuthology.orchestra.run.vpm167.stderr:2016-06-22 16:28:53.654055 7f3388e098c0 -1 filestore(/var/lib/ceph/osd/ceph-5) could not find -1/23c2fcde/osd_superblock/0 in index: (2) No such file or directory
2016-06-22T09:28:53.597 INFO:teuthology.orchestra.run.vpm167.stderr:2016-06-22 16:28:53.679871 7f3388e098c0 -1 created object store /var/lib/ceph/osd/ceph-5 journal /var/lib/ceph/osd/ceph-5/journal for osd.5 fsid 9f9c8125-9aed-4f34-b657-6415d1b9f50c
2016-06-22T09:28:53.597 INFO:teuthology.orchestra.run.vpm167.stderr:2016-06-22 16:28:53.679908 7f3388e098c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory
2016-06-22T09:28:53.598 INFO:teuthology.orchestra.run.vpm167.stderr:2016-06-22 16:28:53.680015 7f3388e098c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring53.597 INFO:teuthology.orchestra.run.vpm167.stderr:2016-06-22 16:28:53.679908 7f3388e098c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory
2016-06-22T09:28:53.598 INFO:teuthology.orchestra.run.vpm167.stderr:2016-06-22 16:28:53.680015 7f3388e098c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring