Project

General

Profile

Actions
Copy link

Bug #9535

closed

monitor crashed after restarting

Added by Xinxin Shu over 9 years ago. Updated over 9 years ago.

Status:
Duplicate
Priority:
High
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
other
Tags:
Backport:
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

recently when i restarted my ceph cluster , the monitor crashed , below is the output of monitor log

2014-09-19 11:29:30.568856 7f89c94077a0 0 ceph version 0.80.1 (a38fe1169b6d2ac98b427334c12d7cf81f809b74), process ceph-mon, pid 39903
2014-09-19 11:29:30.743139 7f89c94077a0 0 starting mon.a rank 0 at 192.101.12.51:6789/0 mon_data /var/lib/ceph/mon/ceph-a fsid beb286e8-9c6f-4687-babe-1b8924479760
2014-09-19 11:29:30.743496 7f89c94077a0 1 mon.a@-1(probing) e1 preinit fsid beb286e8-9c6f-4687-babe-1b8924479760
2014-09-19 11:29:30.744615 7f89c94077a0 1 mon.a@-1(probing).paxosservice(pgmap 1229030..1229550) refresh upgraded, format 0 -> 1
2014-09-19 11:29:30.744637 7f89c94077a0 1 mon.a@-1(probing).pg v0 on_upgrade discarding in-core PGMap
2014-09-19 11:29:30.834870 7f89c94077a0 0 mon.a@-1(probing).mds e1 print_map
epoch 1
flags 0
created 2014-07-15 15:55:02.229706
modified 2014-07-15 15:55:02.229738
tableserver 0
root 0
session_timeout 60
session_autoclose 300
max_file_size 1099511627776
last_failure 0
last_failure_osd_epoch 0
compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap}
max_mds 1
in
up {}
failed
stopped
data_pools 0
metadata_pool 1
inline_data disabled

2014-09-19 11:29:30.835568 7f89c94077a0 0 mon.a@-1(probing).osd e2668 crush map has features 2200130813952, adjusting msgr requires
2014-09-19 11:29:30.835579 7f89c94077a0 0 mon.a@-1(probing).osd e2668 crush map has features 2200130813952, adjusting msgr requires
2014-09-19 11:29:30.835581 7f89c94077a0 0 mon.a@-1(probing).osd e2668 crush map has features 2200130813952, adjusting msgr requires
2014-09-19 11:29:30.835582 7f89c94077a0 0 mon.a@-1(probing).osd e2668 crush map has features 2200130813952, adjusting msgr requires
2014-09-19 11:29:30.836276 7f89c94077a0 1 mon.a@-1(probing).paxosservice(auth 1501..1695) refresh upgraded, format 0 -> 1
2014-09-19 11:29:30.837200 7f89c94077a0 -1 mon/AuthMonitor.cc: In function 'virtual void AuthMonitor::update_from_paxos(bool*)' thread 7f89c94077a0 time 2014-09-19 11:29:30.836402
mon/AuthMonitor.cc: 155: FAILED assert(ret == 0)

ceph version 0.80.1 (a38fe1169b6d2ac98b427334c12d7cf81f809b74)
 1: (AuthMonitor::update_from_paxos(bool*)+0x2611) [0x5e8041]
 2: (PaxosService::refresh(bool*)+0x193) [0x530c43]
 3: (Monitor::refresh_from_paxos(bool*)+0x57) [0x4ced07]
 4: (Monitor::init_paxos()+0xd5) [0x4cee95]
 5: (Monitor::preinit()+0x981) [0x4f3d51]
 6: (main()+0x3caa) [0x4b620a]
 7: (__libc_start_main()+0xfd) [0x376f41ecdd]
 8: /usr/bin/ceph-mon() [0x4b0af9]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
36> 2014-09-19 11:29:30.563118 7f89c94077a0 5 asok(0x3b80e00) register_command perfcounters_dump hook 0x3b38010
-35> 2014-09-19 11:29:30.563159 7f89c94077a0 5 asok(0x3b80e00) register_command 1 hook 0x3b38010
-34> 2014-09-19 11:29:30.563191 7f89c94077a0 5 asok(0x3b80e00) register_command perf dump hook 0x3b38010
-33> 2014-09-19 11:29:30.563198 7f89c94077a0 5 asok(0x3b80e00) register_command perfcounters_schema hook 0x3b38010
-32> 2014-09-19 11:29:30.563204 7f89c94077a0 5 asok(0x3b80e00) register_command 2 hook 0x3b38010
-31> 2014-09-19 11:29:30.563210 7f89c94077a0 5 asok(0x3b80e00) register_command perf schema hook 0x3b38010
-30> 2014-09-19 11:29:30.563227 7f89c94077a0 5 asok(0x3b80e00) register_command config show hook 0x3b38010
-29> 2014-09-19 11:29:30.563234 7f89c94077a0 5 asok(0x3b80e00) register_command config set hook 0x3b38010
-28> 2014-09-19 11:29:30.563243 7f89c94077a0 5 asok(0x3b80e00) register_command config get hook 0x3b38010
-27> 2014-09-19 11:29:30.563257 7f89c94077a0 5 asok(0x3b80e00) register_command log flush hook 0x3b38010
-26> 2014-09-19 11:29:30.563264 7f89c94077a0 5 asok(0x3b80e00) register_command log dump hook 0x3b38010
-25> 2014-09-19 11:29:30.563270 7f89c94077a0 5 asok(0x3b80e00) register_command log reopen hook 0x3b38010
-24> 2014-09-19 11:29:30.568856 7f89c94077a0 0 ceph version 0.80.1 (a38fe1169b6d2ac98b427334c12d7cf81f809b74), process ceph-mon, pid 39903
-23> 2014-09-19 11:29:30.571827 7f89c94077a0 5 asok(0x3b80e00) init /var/run/ceph/ceph-mon.a.asok
-22> 2014-09-19 11:29:30.571868 7f89c94077a0 5 asok(0x3b80e00) bind_and_listen /var/run/ceph/ceph-mon.a.asok
-21> 2014-09-19 11:29:30.572135 7f89c94077a0 5 asok(0x3b80e00) register_command 0 hook 0x3b30030
-20> 2014-09-19 11:29:30.572159 7f89c94077a0 5 asok(0x3b80e00) register_command version hook 0x3b30030
-19> 2014-09-19 11:29:30.572172 7f89c94077a0 5 asok(0x3b80e00) register_command git_version hook 0x3b30030
-18> 2014-09-19 11:29:30.572180 7f89c94077a0 5 asok(0x3b80e00) register_command help hook 0x3b38030
-17> 2014-09-19 11:29:30.572188 7f89c94077a0 5 asok(0x3b80e00) register_command get_command_descriptions hook 0x3b38090
-16> 2014-09-19 11:29:30.572274 7f89c8404700 5 asok(0x3b80e00) entry start
-15> 2014-09-19 11:29:30.743139 7f89c94077a0 0 starting mon.a rank 0 at 192.101.12.51:6789/0 mon_data /var/lib/ceph/mon/ceph-a fsid beb286e8-9c6f-4687-babe-1b8924479760
-14> 2014-09-19 11:29:30.743237 7f89c94077a0 1 - 192.101.12.51:6789/0 learned my addr 192.101.12.51:6789/0
-13> 2014-09-19 11:29:30.743255 7f89c94077a0 1 accepter.accepter.bind my_inst.addr is 192.101.12.51:6789/0 need_addr=0
-12> 2014-09-19 11:29:30.743282 7f89c94077a0 5 adding auth protocol: none
-11> 2014-09-19 11:29:30.743289 7f89c94077a0 5 adding auth protocol: none
-10> 2014-09-19 11:29:30.743496 7f89c94077a0 1 mon.a@-1(probing) e1 preinit fsid beb286e8-9c6f-4687-babe-1b8924479760
-9> 2014-09-19 11:29:30.744615 7f89c94077a0 1 mon.a@-1(probing).paxosservice(pgmap 1229030..1229550) refresh upgraded, format 0 -> 1
-8> 2014-09-19 11:29:30.744637 7f89c94077a0 1 mon.a@-1(probing).pg v0 on_upgrade discarding in-core PGMap
-7> 2014-09-19 11:29:30.834858 7f89c94077a0 4 mon.a@-1(probing).mds e1 new map
-6> 2014-09-19 11:29:30.834870 7f89c94077a0 0 mon.a@-1(probing).mds e1 print_map
epoch 1
flags 0
created 2014-07-15 15:55:02.229706
modified 2014-07-15 15:55:02.229738
tableserver 0
root 0
session_timeout 60
session_autoclose 300
max_file_size 1099511627776
last_failure 0
last_failure_osd_epoch 0
compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap}
max_mds 1
in
up {}
failed
stopped
data_pools 0
metadata_pool 1
inline_data disabled

-5> 2014-09-19 11:29:30.835568 7f89c94077a0  0 mon.a@-1(probing).osd e2668 crush map has features 2200130813952, adjusting msgr requires
    -4> 2014-09-19 11:29:30.835579 7f89c94077a0  0 mon.a@-1(probing).osd e2668 crush map has features 2200130813952, adjusting msgr requires
    -3> 2014-09-19 11:29:30.835581 7f89c94077a0  0 mon.a@-1(probing).osd e2668 crush map has features 2200130813952, adjusting msgr requires
    -2> 2014-09-19 11:29:30.835582 7f89c94077a0  0 mon.a@-1(probing).osd e2668 crush map has features 2200130813952, adjusting msgr requires
    -1> 2014-09-19 11:29:30.836276 7f89c94077a0  1 mon.a@-1(probing).paxosservice(auth 1501..1695) refresh upgraded, format 0 -> 1
     0> 2014-09-19 11:29:30.837200 7f89c94077a0 -1 mon/AuthMonitor.cc: In function 'virtual void AuthMonitor::update_from_paxos(bool*)' thread 7f89c94077a0 time 2014-09-19 11:29:30.836402
mon/AuthMonitor.cc: 155: FAILED assert(ret == 0)
ceph version 0.80.1 (a38fe1169b6d2ac98b427334c12d7cf81f809b74)
 1: (AuthMonitor::update_from_paxos(bool*)+0x2611) [0x5e8041]
 2: (PaxosService::refresh(bool*)+0x193) [0x530c43]
 3: (Monitor::refresh_from_paxos(bool*)+0x57) [0x4ced07]
 4: (Monitor::init_paxos()+0xd5) [0x4cee95]
 5: (Monitor::preinit()+0x981) [0x4f3d51]
 6: (main()+0x3caa) [0x4b620a]
 7: (__libc_start_main()+0xfd) [0x376f41ecdd]
 8: /usr/bin/ceph-mon() [0x4b0af9]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
0/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 keyvaluestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/mon.a.log
-- end dump of recent events ---
2014-09-19 11:29:30.838931 7f89c94077a0 -1 ** Caught signal (Aborted) *
in thread 7f89c94077a0

ceph version 0.80.1 (a38fe1169b6d2ac98b427334c12d7cf81f809b74)
 1: /usr/bin/ceph-mon() [0x7edb41]
 2: /lib64/libpthread.so.0() [0x376fc0f500]
 3: (gsignal()+0x35) [0x376f4328e5]
 4: (abort()+0x175) [0x376f4340c5]
 5: (_gnu_cxx::_verbose_terminate_handler()+0x12d) [0x3c6cabea5d]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
0> 2014-09-19 11:29:30.838931 7f89c94077a0 -1 ** Caught signal (Aborted) *
in thread 7f89c94077a0

ceph version 0.80.1 (a38fe1169b6d2ac98b427334c12d7cf81f809b74)
 1: /usr/bin/ceph-mon() [0x7edb41]
 2: /lib64/libpthread.so.0() [0x376fc0f500]
 3: (gsignal()+0x35) [0x376f4328e5]
 4: (abort()+0x175) [0x376f4340c5]
 5: (_gnu_cxx::_verbose_terminate_handler()+0x12d) [0x3c6cabea5d]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
0/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 keyvaluestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/mon.a.log
-- end dump of recent events ---

Related issues 1 (0 open — 1 closed)

Is duplicate of Ceph - Bug #8851: Mon crash after update to 0.80.4ResolvedJoao Eduardo Luis07/16/2014

Actions
Actions
Copy link
 #1

Updated by Joao Eduardo Luis over 9 years ago

  • Status changed from New to Duplicate
Actions
Copy link

Also available in: Atom PDF