Ceph

this seems to happen after upgrading from bobtail to next branch, when the daemons are restarted. all the monitors seems to crash.

logs: ubuntu@teuthology:/a/teuthology-2013-05-09_01:30:04-upgrade-master-testing-basic/9561

2013-05-09T01:39:02.816 INFO:teuthology.misc.health.err:2013-05-09 01:39:18.541333 7f8ee9310700  0 -- :/9994 >> 10.214.132.26:6790/0 pipe(0x7f8ed4000c00 sd=8 :0 s=1 pgs=0 cs=0 l=1).fault
2013-05-09T01:39:04.938 INFO:teuthology.task.ceph.mon.a.err:mon/Monitor.cc: In function 'void Monitor::StoreConverter::_convert_machines(std::string)' thread 7fe9998af780 time 2013-05-09 01:39:20.662331
2013-05-09T01:39:04.938 INFO:teuthology.task.ceph.mon.a.err:mon/Monitor.cc: 4413: FAILED assert(0 == "Duplicate GV -- something is wrong!")
2013-05-09T01:39:04.938 INFO:teuthology.task.ceph.mon.a.err: ceph version 0.61-120-g6c1e479 (6c1e4791782ce2b3e101ee80640d896bcda684de)
2013-05-09T01:39:04.938 INFO:teuthology.task.ceph.mon.a.err: 1: (Monitor::StoreConverter::_convert_machines(std::string)+0x1021) [0x4ad6f1]
2013-05-09T01:39:04.938 INFO:teuthology.task.ceph.mon.a.err: 2: (Monitor::StoreConverter::_convert_machines()+0xf1) [0x4ae6e1]
2013-05-09T01:39:04.939 INFO:teuthology.task.ceph.mon.a.err: 3: (Monitor::StoreConverter::convert()+0x185) [0x4ae9a5]
2013-05-09T01:39:04.939 INFO:teuthology.task.ceph.mon.a.err: 4: (main()+0x416) [0x48c3a6]
2013-05-09T01:39:04.939 INFO:teuthology.task.ceph.mon.a.err: 5: (__libc_start_main()+0xed) [0x7fe99796976d]
2013-05-09T01:39:04.939 INFO:teuthology.task.ceph.mon.a.err: 6: ceph-mon() [0x48fddd]
2013-05-09T01:39:04.939 INFO:teuthology.task.ceph.mon.a.err: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2013-05-09T01:39:04.939 INFO:teuthology.task.ceph.mon.a.err:2013-05-09 01:39:20.663233 7fe9998af780 -1 mon/Monitor.cc: In function 'void Monitor::StoreConverter::_convert_machines(std::string)' thread 7fe9998af780 time 2013-05-09 01:39:20.662331
2013-05-09T01:39:04.940 INFO:teuthology.task.ceph.mon.a.err:mon/Monitor.cc: 4413: FAILED assert(0 == "Duplicate GV -- something is wrong!")
2013-05-09T01:39:04.940 INFO:teuthology.task.ceph.mon.a.err:
2013-05-09T01:39:04.940 INFO:teuthology.task.ceph.mon.a.err: ceph version 0.61-120-g6c1e479 (6c1e4791782ce2b3e101ee80640d896bcda684de)
2013-05-09T01:39:04.940 INFO:teuthology.task.ceph.mon.a.err: 1: (Monitor::StoreConverter::_convert_machines(std::string)+0x1021) [0x4ad6f1]
2013-05-09T01:39:04.940 INFO:teuthology.task.ceph.mon.a.err: 2: (Monitor::StoreConverter::_convert_machines()+0xf1) [0x4ae6e1]
2013-05-09T01:39:04.940 INFO:teuthology.task.ceph.mon.a.err: 3: (Monitor::StoreConverter::convert()+0x185) [0x4ae9a5]
2013-05-09T01:39:04.941 INFO:teuthology.task.ceph.mon.a.err: 4: (main()+0x416) [0x48c3a6]
2013-05-09T01:39:04.941 INFO:teuthology.task.ceph.mon.a.err: 5: (__libc_start_main()+0xed) [0x7fe99796976d]
2013-05-09T01:39:04.941 INFO:teuthology.task.ceph.mon.a.err: 6: ceph-mon() [0x48fddd]
2013-05-09T01:39:04.941 INFO:teuthology.task.ceph.mon.a.err: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2013-05-09T01:39:04.941 INFO:teuthology.task.ceph.mon.a.err:
2013-05-09T01:39:04.949 INFO:teuthology.task.ceph.mon.a.err:     0> 2013-05-09 01:39:20.663233 7fe9998af780 -1 mon/Monitor.cc: In function 'void Monitor::StoreConverter::_convert_machines(std::string)' thread 7fe9998af780 time 2013-05-09 01:39:20.662331
2013-05-09T01:39:04.949 INFO:teuthology.task.ceph.mon.a.err:mon/Monitor.cc: 4413: FAILED assert(0 == "Duplicate GV -- something is wrong!")
2013-05-09T01:39:04.950 INFO:teuthology.task.ceph.mon.a.err:
2013-05-09T01:39:04.950 INFO:teuthology.task.ceph.mon.a.err: ceph version 0.61-120-g6c1e479 (6c1e4791782ce2b3e101ee80640d896bcda684de)
2013-05-09T01:39:04.950 INFO:teuthology.task.ceph.mon.a.err: 1: (Monitor::StoreConverter::_convert_machines(std::string)+0x1021) [0x4ad6f1]
2013-05-09T01:39:04.950 INFO:teuthology.task.ceph.mon.a.err: 2: (Monitor::StoreConverter::_convert_machines()+0xf1) [0x4ae6e1]
2013-05-09T01:39:04.950 INFO:teuthology.task.ceph.mon.a.err: 3: (Monitor::StoreConverter::convert()+0x185) [0x4ae9a5]
2013-05-09T01:39:04.950 INFO:teuthology.task.ceph.mon.a.err: 4: (main()+0x416) [0x48c3a6]
2013-05-09T01:39:04.951 INFO:teuthology.task.ceph.mon.a.err: 5: (__libc_start_main()+0xed) [0x7fe99796976d]
2013-05-09T01:39:04.951 INFO:teuthology.task.ceph.mon.a.err: 6: ceph-mon() [0x48fddd]
2013-05-09T01:39:04.951 INFO:teuthology.task.ceph.mon.a.err: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

ubuntu@teuthology:/a/teuthology-2013-05-09_01:30:04-upgrade-master-testing-basic/9561$ cat orig.config.yaml 
kernel:
  kdb: true
  sha1: b5b09be30cf99f9c699e825629f02e3bce555d44
machine_type: plana
nuke-on-error: true
overrides:
  ceph:
    conf:
      mon:
        debug mon: 20
        debug ms: 20
        debug paxos: 20
    log-whitelist:
    - slow request
    sha1: 5177fcb6bf384d5c9caed9b28f28f656823b3d64
  s3tests:
    branch: master
  workunit:
    sha1: 5177fcb6bf384d5c9caed9b28f28f656823b3d64
roles:
- - mon.a
  - mds.a
  - osd.0
  - osd.1
- - mon.b
  - mon.c
  - osd.2
  - osd.3
- - client.0
tasks:
- chef: null
- clock.check: null
- install:
    branch: bobtail
- ceph: null
- workunit:
    branch: bobtail
    clients:
      client.0:
      - rbd/import_export.sh
    env:
      RBD_CREATE_ARGS: --new-format
- install.upgrade:
    all:
      branch: next
- ceph.restart:
  - osd.0
  - osd.1
  - osd.2
  - osd.3
  - mon.a
  - mon.b
  - mon.c
  - mds.a
- workunit:
    clients:
      client.0:
      - rbd/import_export.sh
    env:
      RBD_CREATE_ARGS: --new-format