Bug #16453
Updated by Yuri Weinstein almost 8 years ago
Run: http://pulpito.ceph.com/teuthology-2016-06-22_03:00:02-rados-hammer-distro-basic-vps/ Job: 271151 Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2016-06-22_03:00:02-rados-hammer-distro-basic-vps/271151/teuthology.log <pre> 2016-06-22T09:28:53.571 INFO:teuthology.orchestra.run.vpm167.stderr:2016-06-22 16:28:53.654055 7f3388e098c0 -1 filestore(/var/lib/ceph/osd/ceph-5) could not find -1/23c2fcde/osd_superblock/0 in index: (2) No such file or directory 2016-06-22T09:28:53.597 INFO:teuthology.orchestra.run.vpm167.stderr:2016-06-22 16:28:53.679871 7f3388e098c0 -1 created object store /var/lib/ceph/osd/ceph-5 journal /var/lib/ceph/osd/ceph-5/journal for osd.5 fsid 9f9c8125-9aed-4f34-b657-6415d1b9f50c 2016-06-22T09:28:53.597 INFO:teuthology.orchestra.run.vpm167.stderr:2016-06-22 16:28:53.679908 7f3388e098c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2016-06-22T09:28:53.598 INFO:teuthology.orchestra.run.vpm167.stderr:2016-06-22 16:28:53.680015 7f3388e098c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2016-06-22T09:28:53.599 INFO:tasks.ceph:Reading keys from all nodes... vpm167 </pre> ... and <pre> 2016-06-22T09:28:56.625 INFO:teuthology.orchestra.run.vpm055:Running: 'sudo ceph osd crush tunables default' 2016-06-22T09:28:56.682 INFO:tasks.ceph.mon.b.vpm167.stdout:starting mon.b rank 1 at 172.21.2.167:6789/0 mon_data /var/lib/ceph/mon/ceph-b fsid 9f9c8125-9aed-4f34-b657-6415d1b9f50c 2016-06-22T09:28:56.721 INFO:tasks.ceph.mon.a.vpm055.stdout:starting mon.a rank 0 at 172.21.2.55:6789/0 mon_data /var/lib/ceph/mon/ceph-a fsid 9f9c8125-9aed-4f34-b657-6415d1b9f50c 2016-06-22T09:28:56.723 INFO:tasks.ceph.mon.c.vpm055.stdout:starting mon.c rank 2 at 172.21.2.55:6790/0 mon_data /var/lib/ceph/mon/ceph-c fsid 9f9c8125-9aed-4f34-b657-6415d1b9f50c 2016-06-22T09:29:16.706 INFO:tasks.ceph.mon.b.vpm167.stderr:mon/Paxos.cc: In function 'void Paxos::handle_collect(MMonPaxos*)' thread 7f0a80d69700 time 2016-06-22 16:29:16.788170 2016-06-22T09:29:16.706 INFO:tasks.ceph.mon.b.vpm167.stderr:mon/Paxos.cc: 197: FAILED assert(mon->is_peon()) 2016-06-22T09:29:16.707 INFO:tasks.ceph.mon.b.vpm167.stderr: ceph version 0.94.7-26-g2e156d7 (2e156d7ad4b9f4ffd6028df3a460b50b30c8b0d3) 2016-06-22T09:29:16.707 INFO:tasks.ceph.mon.b.vpm167.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x7e51ab] 2016-06-22T09:29:16.707 INFO:tasks.ceph.mon.b.vpm167.stderr: 2: (Paxos::handle_collect(MMonPaxos*)+0xe7b) [0x5faa3b] 2016-06-22T09:29:16.707 INFO:tasks.ceph.mon.b.vpm167.stderr: 3: (Paxos::dispatch(PaxosServiceMessage*)+0x1db) [0x5ff34b] 2016-06-22T09:29:16.708 INFO:tasks.ceph.mon.b.vpm167.stderr: 4: (Monitor::dispatch(MonSession*, Message*, bool)+0x923) [0x5d1753] 2016-06-22T09:29:16.708 INFO:tasks.ceph.mon.b.vpm167.stderr: 5: (Monitor::_ms_dispatch(Message*)+0x1a6) [0x5d1ba6] 2016-06-22T09:29:16.708 INFO:tasks.ceph.mon.b.vpm167.stderr: 6: (Monitor::ms_dispatch(Message*)+0x23) [0x5efa23] 2016-06-22T09:29:16.708 INFO:tasks.ceph.mon.b.vpm167.stderr: 7: (DispatchQueue::entry()+0x649) [0x92d929] 2016-06-22T09:29:16.708 INFO:tasks.ceph.mon.b.vpm167.stderr: 8: (DispatchQueue::DispatchThread::entry()+0xd) [0x7cdc2d] 2016-06-22T09:29:16.709 INFO:tasks.ceph.mon.b.vpm167.stderr: 9: (()+0x8184) [0x7f0a87663184] 2016-06-22T09:29:16.709 INFO:tasks.ceph.mon.b.vpm167.stderr: 10: (clone()+0x6d) [0x7f0a85bce37d] 2016-06-22T09:29:16.709 INFO:tasks.ceph.mon.b.vpm167.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.2016-06-22T09:28:53.570 INFO:teuthology.orchestra.run.vpm167.stderr:2016-06-22 16:28:53.653255 7f3388e098c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2016-06-22T09:28:53.571 INFO:teuthology.orchestra.run.vpm167.stderr:2016-06-22 16:28:53.654055 7f3388e098c0 -1 filestore(/var/lib/ceph/osd/ceph-5) could not find -1/23c2fcde/osd_superblock/0 in index: (2) No such file or directory 2016-06-22T09:28:53.597 INFO:teuthology.orchestra.run.vpm167.stderr:2016-06-22 16:28:53.679871 7f3388e098c0 -1 created object store /var/lib/ceph/osd/ceph-5 journal /var/lib/ceph/osd/ceph-5/journal for osd.5 fsid 9f9c8125-9aed-4f34-b657-6415d1b9f50c 2016-06-22T09:28:53.597 INFO:teuthology.orchestra.run.vpm167.stderr:2016-06-22 16:28:53.679908 7f3388e098c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2016-06-22T09:28:53.598 INFO:teuthology.orchestra.run.vpm167.stderr:2016-06-22 16:28:53.680015 7f3388e098c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring2016-06-22T09:28:53.570 INFO:teuthology.orchestra.run.vpm167.stderr:2016-06-22 16:28:53.653255 7f3388e098c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2016-06-22T09:28:53.571 INFO:teuthology.orchestra.run.vpm167.stderr:2016-06-22 16:28:53.654055 7f3388e098c0 -1 filestore(/var/lib/ceph/osd/ceph-5) could not find -1/23c2fcde/osd_superblock/0 in index: (2) No such file or directory 2016-06-22T09:28:53.597 INFO:teuthology.orchestra.run.vpm167.stderr:2016-06-22 16:28:53.679871 7f3388e098c0 -1 created object store /var/lib/ceph/osd/ceph-5 journal /var/lib/ceph/osd/ceph-5/journal for osd.5 fsid 9f9c8125-9aed-4f34-b657-6415d1b9f50c 2016-06-22T09:28:53.597 INFO:teuthology.orchestra.run.vpm167.stderr:2016-06-22 16:28:53.679908 7f3388e098c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2016-06-22T09:28:53.598 INFO:teuthology.orchestra.run.vpm167.stderr:2016-06-22 16:28:53.680015 7f3388e098c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring2016-06-22T09:28:53.570 INFO:teuthology.orchestra.run.vpm167.stderr:2016-06-22 16:28:53.653255 7f3388e098c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2016-06-22T09:28:53.571 INFO:teuthology.orchestra.run.vpm167.stderr:2016-06-22 16:28:53.654055 7f3388e098c0 -1 filestore(/var/lib/ceph/osd/ceph-5) could not find -1/23c2fcde/osd_superblock/0 in index: (2) No such file or directory 2016-06-22T09:28:53.597 INFO:teuthology.orchestra.run.vpm167.stderr:2016-06-22 16:28:53.679871 7f3388e098c0 -1 created object store /var/lib/ceph/osd/ceph-5 journal /var/lib/ceph/osd/ceph-5/journal for osd.5 fsid 9f9c8125-9aed-4f34-b657-6415d1b9f50c 2016-06-22T09:28:2016-06-22T09:28:53.570 INFO:teuthology.orchestra.run.vpm167.stderr:2016-06-22 16:28:53.653255 7f3388e098c0 -1 j2016-06-22T09:28:53.570 INFO:teuthology.orchestra.run.vpm167.stderr:2016-06-22 16:28:53.653255 7f3388e098c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2016-06-22T09:28:53.571 INFO:teuthology.orchestra.run.vpm167.stderr:2016-06-22 16:28:53.654055 7f3388e098c0 -1 filestore(/var/lib/ceph/osd/ceph-5) could not find -1/23c2fcde/osd_superblock/0 in index: (2) No such file or directory 2016-06-22T09:28:53.597 INFO:teuthology.orchestra.run.vpm167.stderr:2016-06-22 16:28:53.679871 7f3388e098c0 -1 created object store /var/lib/ceph/osd/ceph-5 journal /var/lib/ceph/osd/ceph-5/journal for osd.5 fsid 9f9c8125-9aed-4f34-b657-6415d1b9f50c 2016-06-22T09:28:53.597 INFO:teuthology.orchestra.run.vpm167.stderr:2016-06-22 16:28:53.679908 7f3388e098c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2016-06-22T09:28:53.598 INFO:teuthology.orchestra.run.vpm167.stderr:2016-06-22 16:28:53.680015 7f3388e098c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyringournal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2016-06-22T09:28:53.571 INFO:teuthology.orchestra.run.vpm167.stderr:2016-06-22 16:28:53.654055 7f3388e098c0 -1 filestore(/var/lib/ceph/osd/ceph-5) could not find -1/23c2fcde/osd_superblock/0 in index: (2) No such file or directory 2016-06-22T09:28:53.597 INFO:teuthology.orchestra.run.vpm167.stderr:2016-06-22 16:28:53.679871 7f3388e098c0 -1 created object store /var/lib/ceph/osd/ceph-5 journal /var/lib/ceph/osd/ceph-5/journal for osd.5 fsid 9f9c8125-9aed-4f34-b657-6415d1b9f50c 2016-06-22T09:28:53.597 INFO:teuthology.orchestra.run.vpm167.stderr:2016-06-22 16:28:53.679908 7f3388e098c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2016-06-22T09:28:53.598 INFO:teuthology.orchestra.run.vpm167.stderr:2016-06-22 16:28:53.680015 7f3388e098c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring53.597 INFO:teuthology.orchestra.run.vpm167.stderr:2016-06-22 16:28:53.679908 7f3388e098c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2016-06-22T09:28:53.598 INFO:teuthology.orchestra.run.vpm167.stderr:2016-06-22 16:28:53.680015 7f3388e098c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring </pre>