Actions
Bug #4974
closednightlies: failed assert at Monitor::StoreConverter::_convert_machines
% Done:
0%
Source:
Q/A
Tags:
Backport:
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
this seems to happen after upgrading from bobtail to next branch, when the daemons are restarted. all the monitors seems to crash.
logs: ubuntu@teuthology:/a/teuthology-2013-05-09_01:30:04-upgrade-master-testing-basic/9561
2013-05-09T01:39:02.816 INFO:teuthology.misc.health.err:2013-05-09 01:39:18.541333 7f8ee9310700 0 -- :/9994 >> 10.214.132.26:6790/0 pipe(0x7f8ed4000c00 sd=8 :0 s=1 pgs=0 cs=0 l=1).fault 2013-05-09T01:39:04.938 INFO:teuthology.task.ceph.mon.a.err:mon/Monitor.cc: In function 'void Monitor::StoreConverter::_convert_machines(std::string)' thread 7fe9998af780 time 2013-05-09 01:39:20.662331 2013-05-09T01:39:04.938 INFO:teuthology.task.ceph.mon.a.err:mon/Monitor.cc: 4413: FAILED assert(0 == "Duplicate GV -- something is wrong!") 2013-05-09T01:39:04.938 INFO:teuthology.task.ceph.mon.a.err: ceph version 0.61-120-g6c1e479 (6c1e4791782ce2b3e101ee80640d896bcda684de) 2013-05-09T01:39:04.938 INFO:teuthology.task.ceph.mon.a.err: 1: (Monitor::StoreConverter::_convert_machines(std::string)+0x1021) [0x4ad6f1] 2013-05-09T01:39:04.938 INFO:teuthology.task.ceph.mon.a.err: 2: (Monitor::StoreConverter::_convert_machines()+0xf1) [0x4ae6e1] 2013-05-09T01:39:04.939 INFO:teuthology.task.ceph.mon.a.err: 3: (Monitor::StoreConverter::convert()+0x185) [0x4ae9a5] 2013-05-09T01:39:04.939 INFO:teuthology.task.ceph.mon.a.err: 4: (main()+0x416) [0x48c3a6] 2013-05-09T01:39:04.939 INFO:teuthology.task.ceph.mon.a.err: 5: (__libc_start_main()+0xed) [0x7fe99796976d] 2013-05-09T01:39:04.939 INFO:teuthology.task.ceph.mon.a.err: 6: ceph-mon() [0x48fddd] 2013-05-09T01:39:04.939 INFO:teuthology.task.ceph.mon.a.err: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. 2013-05-09T01:39:04.939 INFO:teuthology.task.ceph.mon.a.err:2013-05-09 01:39:20.663233 7fe9998af780 -1 mon/Monitor.cc: In function 'void Monitor::StoreConverter::_convert_machines(std::string)' thread 7fe9998af780 time 2013-05-09 01:39:20.662331 2013-05-09T01:39:04.940 INFO:teuthology.task.ceph.mon.a.err:mon/Monitor.cc: 4413: FAILED assert(0 == "Duplicate GV -- something is wrong!") 2013-05-09T01:39:04.940 INFO:teuthology.task.ceph.mon.a.err: 2013-05-09T01:39:04.940 INFO:teuthology.task.ceph.mon.a.err: ceph version 0.61-120-g6c1e479 (6c1e4791782ce2b3e101ee80640d896bcda684de) 2013-05-09T01:39:04.940 INFO:teuthology.task.ceph.mon.a.err: 1: (Monitor::StoreConverter::_convert_machines(std::string)+0x1021) [0x4ad6f1] 2013-05-09T01:39:04.940 INFO:teuthology.task.ceph.mon.a.err: 2: (Monitor::StoreConverter::_convert_machines()+0xf1) [0x4ae6e1] 2013-05-09T01:39:04.940 INFO:teuthology.task.ceph.mon.a.err: 3: (Monitor::StoreConverter::convert()+0x185) [0x4ae9a5] 2013-05-09T01:39:04.941 INFO:teuthology.task.ceph.mon.a.err: 4: (main()+0x416) [0x48c3a6] 2013-05-09T01:39:04.941 INFO:teuthology.task.ceph.mon.a.err: 5: (__libc_start_main()+0xed) [0x7fe99796976d] 2013-05-09T01:39:04.941 INFO:teuthology.task.ceph.mon.a.err: 6: ceph-mon() [0x48fddd] 2013-05-09T01:39:04.941 INFO:teuthology.task.ceph.mon.a.err: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. 2013-05-09T01:39:04.941 INFO:teuthology.task.ceph.mon.a.err: 2013-05-09T01:39:04.949 INFO:teuthology.task.ceph.mon.a.err: 0> 2013-05-09 01:39:20.663233 7fe9998af780 -1 mon/Monitor.cc: In function 'void Monitor::StoreConverter::_convert_machines(std::string)' thread 7fe9998af780 time 2013-05-09 01:39:20.662331 2013-05-09T01:39:04.949 INFO:teuthology.task.ceph.mon.a.err:mon/Monitor.cc: 4413: FAILED assert(0 == "Duplicate GV -- something is wrong!") 2013-05-09T01:39:04.950 INFO:teuthology.task.ceph.mon.a.err: 2013-05-09T01:39:04.950 INFO:teuthology.task.ceph.mon.a.err: ceph version 0.61-120-g6c1e479 (6c1e4791782ce2b3e101ee80640d896bcda684de) 2013-05-09T01:39:04.950 INFO:teuthology.task.ceph.mon.a.err: 1: (Monitor::StoreConverter::_convert_machines(std::string)+0x1021) [0x4ad6f1] 2013-05-09T01:39:04.950 INFO:teuthology.task.ceph.mon.a.err: 2: (Monitor::StoreConverter::_convert_machines()+0xf1) [0x4ae6e1] 2013-05-09T01:39:04.950 INFO:teuthology.task.ceph.mon.a.err: 3: (Monitor::StoreConverter::convert()+0x185) [0x4ae9a5] 2013-05-09T01:39:04.950 INFO:teuthology.task.ceph.mon.a.err: 4: (main()+0x416) [0x48c3a6] 2013-05-09T01:39:04.951 INFO:teuthology.task.ceph.mon.a.err: 5: (__libc_start_main()+0xed) [0x7fe99796976d] 2013-05-09T01:39:04.951 INFO:teuthology.task.ceph.mon.a.err: 6: ceph-mon() [0x48fddd] 2013-05-09T01:39:04.951 INFO:teuthology.task.ceph.mon.a.err: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. ubuntu@teuthology:/a/teuthology-2013-05-09_01:30:04-upgrade-master-testing-basic/9561$ cat orig.config.yaml kernel: kdb: true sha1: b5b09be30cf99f9c699e825629f02e3bce555d44 machine_type: plana nuke-on-error: true overrides: ceph: conf: mon: debug mon: 20 debug ms: 20 debug paxos: 20 log-whitelist: - slow request sha1: 5177fcb6bf384d5c9caed9b28f28f656823b3d64 s3tests: branch: master workunit: sha1: 5177fcb6bf384d5c9caed9b28f28f656823b3d64 roles: - - mon.a - mds.a - osd.0 - osd.1 - - mon.b - mon.c - osd.2 - osd.3 - - client.0 tasks: - chef: null - clock.check: null - install: branch: bobtail - ceph: null - workunit: branch: bobtail clients: client.0: - rbd/import_export.sh env: RBD_CREATE_ARGS: --new-format - install.upgrade: all: branch: next - ceph.restart: - osd.0 - osd.1 - osd.2 - osd.3 - mon.a - mon.b - mon.c - mds.a - workunit: clients: client.0: - rbd/import_export.sh env: RBD_CREATE_ARGS: --new-format
Actions