Bug #3695
closedmonitor crashed after an upgrade in Monitor::timecheck
0%
Description
ceph version : 0.55.1-329-g01376d4 (01376d44d73189080d207f701fc7e38cf55c738d)
cluster:
burnupi15[running osd.1, osd.2 and mon.a on argonaut]
burnupi19 [osd.3 and osd.4 are stopped]
burnupi20 [running osd.5, osd.6, mon.c and mds.a on v0.55.1]
Initially, argonaut was running on all the three nodes and i had to bring burnupi19 daemons down for some other test. so left with burnupi15 and burnupi20.
I upgraded burnupi20 with no ceph auth entries in ceph.conf and when trying to access rbd pool, it errored out:
ubuntu@burnupi20:~$ rbd ls
rbd: couldn't connect to the cluster!
2012-12-28 17:35:14.798345 7f894f333780 -1 monclient(hunting): failed to open keyring: (2) No such file or directory
2012-12-28 17:35:14.798364 7f894f333780 0 librados: client.admin initialization error (2) No such file or directory
so, i modified ceph.conf with auth settings set to none and upgraded again, restarted ceph service,
[global]
auth service required = none
auth client required = none
auth cluster required = none
[osd]
osd journal size = 1000
filestore xattr use omap = true
osd min pg log entries = 10
[osd.1]
host = burnupi15
[osd.2]
host = burnupi15
[osd.3]
host = burnupi19
[osd.4]
host = burnupi19
[osd.5]
host = burnupi20
[osd.6]
host = burnupi20
[mon.a]
host = burnupi15
mon addr = 10.214.134.22:6789
[mon.b]
host = burnupi19
mon addr = 10.214.134.14:6789
[mon.c]
host = burnupi20
mon addr = 10.214.134.12:6789
[mds.a]
host = burnupi20
and this crashed mon.c running on burnupi20,
mon/Monitor.cc: 2318: FAILED assert(is_leader()) ceph version 0.55.1-329-g01376d4 (01376d44d73189080d207f701fc7e38cf55c738d) 1: (Monitor::timecheck()+0x989) [0x48d669] 2: (SafeTimer::timer_thread()+0x425) [0x5ed6d5] 3: (SafeTimerThread::entry()+0xd) [0x5ee31d] 4: (()+0x7e9a) [0x7f30ea7a6e9a] 5: (clone()+0x6d) [0x7f30e8f5ccbd] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- begin dump of recent events ---