Ceph

2013-09-08T01:44:01.214 INFO:teuthology.orchestra.run.err:[10.214.133.21]: *** Caught signal (Segmentation fault) **
2013-09-08T01:44:01.214 INFO:teuthology.orchestra.run.err:[10.214.133.21]:  in thread 7f48a26a8780
2013-09-08T01:44:01.220 INFO:teuthology.orchestra.run.err:[10.214.133.21]:  ceph version 0.68-269-gb2caa79 (b2caa799d693ca1a963667ed003cb6c670cc586d)
2013-09-08T01:44:01.220 INFO:teuthology.orchestra.run.err:[10.214.133.21]:  1: rados() [0x43be1a]
2013-09-08T01:44:01.220 INFO:teuthology.orchestra.run.err:[10.214.133.21]:  2: (()+0xfcb0) [0x7f48a142ecb0]
2013-09-08T01:44:01.220 INFO:teuthology.orchestra.run.err:[10.214.133.21]:  3: (ceph_clock_now(CephContext*)+0x3a) [0x7f48a184283a]
2013-09-08T01:44:01.221 INFO:teuthology.orchestra.run.err:[10.214.133.21]:  4: (ObjBencher::out(std::ostream&)+0x1d) [0x42cebd]
2013-09-08T01:44:01.221 INFO:teuthology.orchestra.run.err:[10.214.133.21]:  5: (ObjBencher::write_bench(int, int, int)+0x50) [0x4331d0]
2013-09-08T01:44:01.221 INFO:teuthology.orchestra.run.err:[10.214.133.21]:  6: (ObjBencher::aio_bench(int, int, int, int, int, bool)+0x27a) [0x43592a]
2013-09-08T01:44:01.221 INFO:teuthology.orchestra.run.err:[10.214.133.21]:  7: rados() [0x41acad]
2013-09-08T01:44:01.221 INFO:teuthology.orchestra.run.err:[10.214.133.21]:  8: (main()+0xbb5) [0x4103e5]
2013-09-08T01:44:01.221 INFO:teuthology.orchestra.run.err:[10.214.133.21]:  9: (__libc_start_main()+0xed) [0x7f48a086f76d]

(gdb) bt
#0  0x00007f48a142eb7b in raise (sig=<optimized out>)
    at ../nptl/sysdeps/unix/sysv/linux/pt-raise.c:42
#1  0x000000000043bf6e in reraise_fatal (signum=11)
    at global/signal_handler.cc:59
#2  handle_fatal_signal (signum=11) at global/signal_handler.cc:105
#3  <signal handler called>
#4  0x00007f48a184283a in ceph_clock_now (cct=0x7fff70ef4d50)
    at common/Clock.cc:29
#5  0x000000000042cebd in ObjBencher::out (this=0x7fff70ef4bd0, os=...)
    at common/obj_bencher.cc:82
#6  0x00000000004331d0 in ObjBencher::write_bench (this=0x7fff70ef4bd0, 
    secondsToRun=15, maxObjectsToCreate=0, concurrentios=16)
    at common/obj_bencher.cc:306
#7  0x000000000043592a in ObjBencher::aio_bench (this=0x7fff70ef4bd0, 
    operation=1, secondsToRun=15, maxObjectsToCreate=0, concurrentios=16, 
    op_size=<optimized out>, cleanup=false) at common/obj_bencher.cc:209
#8  0x000000000041acad in rados_tool_common (opts=..., nargs=...)
    at tools/rados/rados.cc:1994
#9  0x00000000004103e5 in main (argc=<optimized out>, argv=<optimized out>)
    at tools/rados/rados.cc:2351
(gdb) f 8
#8  0x000000000041acad in rados_tool_common (opts=..., nargs=...)
    at tools/rados/rados.cc:1994
1994    tools/rados/rados.cc: No such file or directory.
(gdb) p bencher.cct
$1 = (CephContext *) 0x7fff70ef4d50
(gdb) g_ceph_context
Undefined command: "g_ceph_context".  Try "help".
(gdb) p g_ceph_context
$2 = (CephContext *) 0xca7ee0

wtf!

ubuntu@teuthology:/a/teuthology-2013-09-08_01:00:04-rados-master-testing-basic-plana/25783$ cat orig.config.yaml 
kernel:
  kdb: true
  sha1: a8d436f015b627a55ec3b1d15f13d6ab92dd892b
machine_type: plana
nuke-on-error: true
os_type: ubuntu
overrides:
  admin_socket:
    branch: master
  ceph:
    conf:
      global:
        ms inject socket failures: 500
      mon:
        debug mon: 20
        debug ms: 1
        debug paxos: 20
      osd:
        debug ms: 1
        debug osd: 5
        osd op thread timeout: 60
    fs: btrfs
    log-whitelist:
    - slow request
    sha1: b2caa799d693ca1a963667ed003cb6c670cc586d
  ceph-deploy:
    branch:
      dev: master
    conf:
      client:
        log file: /var/log/ceph/ceph-$name.$pid.log
      mon:
        debug mon: 1
        debug ms: 20
        debug paxos: 20
  install:
    ceph:
      sha1: b2caa799d693ca1a963667ed003cb6c670cc586d
  s3tests:
    branch: master
  workunit:
    sha1: b2caa799d693ca1a963667ed003cb6c670cc586d
roles:
- - mon.a
  - mon.b
  - mon.c
  - mds.a
  - osd.0
  - osd.1
  - osd.2
tasks:
- chef: null
- clock.check: null
- install: null
- ceph:
    conf:
      osd:
        osd min pg log entries: 5
    log-whitelist:
    - wrongly marked me down
- osd_backfill: null
teuthology_branch: master