Ceph

ubuntu@teuthology:/var/lib/teuthworker/archive/sage-2013-02-08_12:11:07-rados-master-testing-basic/2870$

2013-02-08 14:48:31.384540 7f00cf167780  0 filestore(/tmp/cephtest/data/osd.3.data) mount FIEMAP ioctl is supported and appears to work
2013-02-08 14:48:31.384554 7f00cf167780  0 filestore(/tmp/cephtest/data/osd.3.data) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
2013-02-08 14:48:31.384844 7f00cf167780  0 filestore(/tmp/cephtest/data/osd.3.data) mount did NOT detect btrfs
2013-02-08 14:48:31.569382 7f00cf167780  0 filestore(/tmp/cephtest/data/osd.3.data) mount syncfs(2) syscall fully supported (by glibc and kernel)
2013-02-08 14:48:31.569522 7f00cf167780  0 filestore(/tmp/cephtest/data/osd.3.data) mount found snaps <>
2013-02-08 14:48:31.836521 7f00cf167780  0 filestore(/tmp/cephtest/data/osd.3.data) mount: enabling WRITEAHEAD journal mode: btrfs not detected
2013-02-08 14:48:31.839333 7f00cf167780  1 journal _open /tmp/cephtest/data/osd.3.journal fd 23: 104857600 bytes, block size 4096 bytes, directio = 1, aio = 1
2013-02-08 14:48:31.960639 7f00cf167780 -1 filestore(/tmp/cephtest/data/osd.3.data) error getting collection index for FORREMOVAL_215_0.13: (2) No such file or directory
2013-02-08 14:48:31.960674 7f00cf167780 -1 filestore(/tmp/cephtest/data/osd.3.data) error getting collection index for FORREMOVAL_215_0.13: (2) No such file or directory
2013-02-08 14:48:31.960694 7f00cf167780 -1 filestore(/tmp/cephtest/data/osd.3.data) error getting collection index for FORREMOVAL_215_0.13: (2) No such file or directory
2013-02-08 14:48:31.960714 7f00cf167780 -1 filestore(/tmp/cephtest/data/osd.3.data) error getting collection index for FORREMOVAL_215_0.13: (2) No such file or directory
2013-02-08 14:48:31.960733 7f00cf167780 -1 filestore(/tmp/cephtest/data/osd.3.data) error getting collection index for FORREMOVAL_215_0.13: (2) No such file or directory
2013-02-08 14:48:31.960754 7f00cf167780 -1 filestore(/tmp/cephtest/data/osd.3.data) error getting collection index for FORREMOVAL_215_0.13: (2) No such file or directory
2013-02-08 14:48:31.960773 7f00cf167780 -1 filestore(/tmp/cephtest/data/osd.3.data) error getting collection index for FORREMOVAL_215_0.13: (2) No such file or directory
2013-02-08 14:48:31.960792 7f00cf167780 -1 filestore(/tmp/cephtest/data/osd.3.data) error getting collection index for FORREMOVAL_215_0.13: (2) No such file or directory
2013-02-08 14:48:31.960812 7f00cf167780 -1 filestore(/tmp/cephtest/data/osd.3.data) error getting collection index for FORREMOVAL_215_0.13: (2) No such file or directory
2013-02-08 14:48:31.960831 7f00cf167780 -1 filestore(/tmp/cephtest/data/osd.3.data) error getting collection index for FORREMOVAL_215_0.13: (2) No such file or directory
2013-02-08 14:48:31.960850 7f00cf167780 -1 filestore(/tmp/cephtest/data/osd.3.data) error getting collection index for FORREMOVAL_215_0.13: (2) No such file or directory
2013-02-08 14:48:31.960869 7f00cf167780 -1 filestore(/tmp/cephtest/data/osd.3.data) error getting collection index for FORREMOVAL_215_0.13: (2) No such file or directory
2013-02-08 14:48:32.123259 7f00cf167780 -1 filestore(/tmp/cephtest/data/osd.3.data) error getting collection index for FORREMOVAL_217_1.10: (2) No such file or directory
2013-02-08 14:48:32.123288 7f00cf167780 -1 filestore(/tmp/cephtest/data/osd.3.data) error getting collection index for FORREMOVAL_217_1.10: (2) No such file or directory
2013-02-08 14:48:32.123308 7f00cf167780 -1 filestore(/tmp/cephtest/data/osd.3.data) error getting collection index for FORREMOVAL_217_1.10: (2) No such file or directory
2013-02-08 14:48:32.123328 7f00cf167780 -1 filestore(/tmp/cephtest/data/osd.3.data) error getting collection index for FORREMOVAL_217_1.10: (2) No such file or directory
2013-02-08 14:48:32.148526 7f00cf167780  0 filestore(/tmp/cephtest/data/osd.3.data)  error (39) Directory not empty not handled on operation 21 (18509.0.1, or op 1, counting from 0)
2013-02-08 14:48:32.148545 7f00cf167780  0 filestore(/tmp/cephtest/data/osd.3.data) ENOTEMPTY suggests garbage data in osd data dir
2013-02-08 14:48:32.148547 7f00cf167780  0 filestore(/tmp/cephtest/data/osd.3.data)  transaction dump:
{ "ops": [
        { "op_num": 0,
          "op_name": "remove",
          "collection": "FORREMOVAL_219_0.a",
          "oid": "190173fa\/plana8923779-290\/head\/\/0"},
        { "op_num": 1,
          "op_name": "rmcoll",
          "collection": "FORREMOVAL_219_0.a"}]}
2013-02-08 14:48:32.162177 7f00cf167780 -1 os/FileStore.cc: In function 'unsigned int FileStore::_do_transaction(ObjectStore::Transaction&, uint64_t, int)' thread 7f00cf167780 time 2013-02-08 14:48:32.148606
os/FileStore.cc: 2667: FAILED assert(0 == "unexpected error")

 ceph version 0.56-696-g2bdf753 (2bdf753d7d771062b03f47d79f6783c75d18ea38)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x95) [0x11bdc11]
 2: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long, int)+0x2cf3) [0x1060ba9]
 3: (FileStore::_do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long, ThreadPool::TPHandle*)+0xec) [0x105c0fe]
 4: (FileStore::do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long)+0x2d) [0x107ae63]
 5: (JournalingObjectStore::journal_replay(unsigned long)+0x8c8) [0x1088442]
 6: (FileStore::mount()+0x48c9) [0x105886b]
 7: (OSD::do_convertfs(ObjectStore*)+0x32) [0xea0e72]
 8: (OSD::convertfs(std::string const&, std::string const&)+0x61) [0xea1dc5]
 9: (main()+0x30bb) [0xdd6825]
 10: (__libc_start_main()+0xed) [0x7f00ccbfb76d]
 11: /tmp/cephtest/binary/usr/local/bin/ceph-osd() [0xdd32c9]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

ubuntu@teuthology:/var/lib/teuthworker/archive/sage-2013-02-08_12:11:07-rados-master-testing-basic/2870$ cat orig.config.yaml 
kernel:
  kdb: true
  sha1: 012d5bda1c0f229494c67098d00edfa24c531ea5
nuke-on-error: true
overrides:
  ceph:
    conf:
      global:
        ms inject delay max: 1
        ms inject delay probability: 0.005
        ms inject delay type: osd
        ms inject socket failures: 2500
    fs: ext4
    log-whitelist:
    - slow request
    sha1: 2bdf753d7d771062b03f47d79f6783c75d18ea38
  s3tests:
    branch: master
  workunit:
    sha1: 2bdf753d7d771062b03f47d79f6783c75d18ea38
roles:
- - mon.a
  - mon.b
  - osd.0
  - osd.1
  - osd.2
- - mon.c
  - mds.a
  - osd.3
  - osd.4
  - osd.5
- - client.0
tasks:
- chef: null
- clock: null
- ceph:
    log-whitelist:
    - wrongly marked me down
    - objects unfound and apparently lost
- thrashosds:
    timeout: 1200
- rados:
    clients:
    - client.0
    objects: 500
    op_weights:
      delete: 50
      read: 100
      rollback: 50
      snap_create: 50
      snap_remove: 50
      write: 100
    ops: 4000