Ceph » rbd

recent log: ubuntu@teuthology:/a/teuthology-2013-02-09_20:00:03-regression-bobtail-master-basic/3977

2013-02-10T00:35:17.538 INFO:teuthology.orchestra.run.err:osdc/ObjectCacher.cc: In function 'ObjectCacher::~ObjectCacher()' thread 7f3fad397780 time 2013-02-10 00:35:12.654134
2013-02-10T00:35:17.538 INFO:teuthology.orchestra.run.err:osdc/ObjectCacher.cc: 517: FAILED assert(!i->size())
2013-02-10T00:35:17.538 INFO:teuthology.orchestra.run.err: ceph version 0.56.2-29-gf11beb9 (f11beb954976f66bfae75e847937f84958ebeaf3)
2013-02-10T00:35:17.538 INFO:teuthology.orchestra.run.err: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x95) [0x7f3fac0eaf8d]
2013-02-10T00:35:17.539 INFO:teuthology.orchestra.run.err: 2: (ObjectCacher::~ObjectCacher()+0x67) [0x7f3facf23b4f]
2013-02-10T00:35:17.539 INFO:teuthology.orchestra.run.err: 3: (librbd::ImageCtx::~ImageCtx()+0x41) [0x7f3facef5a19]
2013-02-10T00:35:17.539 INFO:teuthology.orchestra.run.err: 4: (librbd::close_image(librbd::ImageCtx*)+0x1a8) [0x7f3facf10986]
2013-02-10T00:35:17.539 INFO:teuthology.orchestra.run.err: 5: (librbd::close_image(librbd::ImageCtx*)+0x160) [0x7f3facf1093e]
2013-02-10T00:35:17.539 INFO:teuthology.orchestra.run.err: 6: (rbd_close()+0x20) [0x7f3facee0fbf]
2013-02-10T00:35:17.539 INFO:teuthology.orchestra.run.err: 7: (main()+0x825) [0x4031e5]
2013-02-10T00:35:17.540 INFO:teuthology.orchestra.run.err: 8: (__libc_start_main()+0xed) [0x7f3fab29576d]
2013-02-10T00:35:17.540 INFO:teuthology.orchestra.run.err: 9: /tmp/cephtest/binary/usr/local/bin/test_librbd_fsx() [0x403535]
2013-02-10T00:35:17.540 INFO:teuthology.orchestra.run.err: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

ubuntu@teuthology:/a/teuthology-2013-02-09_20:00:03-regression-bobtail-master-basic/3977$ cat summary.yaml 
ceph-sha1: f11beb954976f66bfae75e847937f84958ebeaf3
client.0-kernel-sha1: 0fa6ebc600bc8e830551aee47a0e929e818a1868
description: collection:rbd-thrash clusters:6-osd-3-machine.yaml fs:xfs.yaml msgr-failures:few.yaml
  thrashers:default.yaml workloads:rbd_fsx_cache_writethrough.yaml
duration: 696.5752260684967
failure_reason: 'Command crashed: ''CEPH_CONF=/tmp/cephtest/ceph.conf LD_LIBRARY_PATH=/tmp/cephtest/binary/usr/local/lib
  /tmp/cephtest/enable-coredump /tmp/cephtest/binary/usr/local/bin/ceph-coverage /tmp/cephtest/archive/coverage
  /tmp/cephtest/binary/usr/local/bin/test_librbd_fsx -d -W -R -p 100 -P /tmp/cephtest/archive
  -t 1 -l 1073741824 -S 0 -N 2000 pool_client.0 image_client.0'''
flavor: basic
mds.a-kernel-sha1: 0fa6ebc600bc8e830551aee47a0e929e818a1868
mon.a-kernel-sha1: 0fa6ebc600bc8e830551aee47a0e929e818a1868
owner: scheduled_teuthology@teuthology
success: false
ubuntu@teuthology:/a/teuthology-2013-02-09_20:00:03-regression-bobtail-master-basic/3977$ cat config.yaml 
kernel: &id001
  kdb: true
  sha1: 0fa6ebc600bc8e830551aee47a0e929e818a1868
nuke-on-error: true
overrides:
  ceph:
    conf:
      client:
        rbd cache: true
        rbd cache max dirty: 0
      global:
        ms inject socket failures: 5000
    fs: xfs
    log-whitelist:
    - slow request
    sha1: f11beb954976f66bfae75e847937f84958ebeaf3
  s3tests:
    branch: bobtail
  workunit:
    sha1: f11beb954976f66bfae75e847937f84958ebeaf3
roles:
- - mon.a
  - osd.0
  - osd.1
  - osd.2
- - mds.a
  - osd.3
  - osd.4
  - osd.5
- - client.0
targets:
  ubuntu@plana46.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCrD6s9otJ5xCNH4nyv0iJu6AoqmlNTFd8D0X9RfFBnmOMrMBWU9kwsFzPIOsuJGbSYbA8LCtjWUwaWoXmbEFtTMitxaDXp47gbVNXknHq7TGZHkWWOwKKu+tlSQBpCVzO/rzBbvJ9fcG7tewq5XcIHz0IUXsUFuEuXR1HaTUJKic2twBpaeAGNvdd6IZ9Sz9TMkfiRV/aVdcHJ/yF8bsXi3pfRPR3puMK/Nyfq5Hz/aabQo1TSyK2o0weoWV7D8vD6S8f3D7p5/5ScBhL3zUcP85SsV47W+/hTFbU8kN1Grlv2sx0fVMB/TUB/UNVdsHKGn5Nv6zb/qMqBEx9nSeZ9
  ubuntu@plana59.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2ijQopwlP92QQaUVB/Dx8HLmRokvfKoN4+2xGBvej1Qb/P67By9WNF92tQspMpLSpvqodRaXxcKixvrlLEJJQ16lY9xje2jNDWRjJ/AjnAg79qRA7N6L6BC7rccZ8oZWXqOpNKIKRNUUyn+XwtVgkbbcUr5NjQG2CfsLPtoNkdX98a/UyvhefomanU6oStJHFIptLXe2WjgW6DoXofu9qIpyZrO75XFOIlEpC2QTXU6eTVGRxhL2qtVVR42z7bcrNuVFr4TTTNvqaabyvW2mPZNz3a8ojh/pb8gIZGFYydiG8aOsS9dapltOysmdwfBPXHsAoD/49vbPEHIdyjarp
  ubuntu@plana60.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5sa91O7pjxDijkXWrqS3slUWFtQJZtC3hfN5cKt+oISIVRZoy4CNxFrOWhmXwKOZaIV7fS2PzmhZQe9jfA5iRP7G/opl/5sMImzTIUUz8Qrlqb7epmwYHkoE196uN45y8iW/IDwsD7o69+rOGIj0SgdT0nGBYXB27OhdECdumxHcST/vM8dFfIwwMienvhKKB+ghrS2bYa33DcfYHBQWdIPXrUf2brb/XRYDwVMg8W5uPyW8dx3GIONVxSWequnTO1m0FuMKB7BrVpH19+/H4DFOwj6vqSvvOYUWm3Go3FsQTvparxrg7/Rliee0RBWcBIE3D0gWk6r74HV0C4pnT
tasks:
- internal.lock_machines: 3
- internal.save_config: null
- internal.check_lock: null
- internal.connect: null
- internal.check_conflict: null
- kernel: *id001
- internal.base: null
- internal.archive: null
- internal.coredump: null
- internal.syslog: null
- internal.timer: null
- chef: null
- clock: null
- ceph:
    log-whitelist:
    - wrongly marked me down
    - objects unfound and apparently lost
- thrashosds:
    timeout: 1200
- rbd_fsx:
    clients:
    - client.0
    ops: 2000

Need to reproduce with:

client:
  debug rbd: 20
  debug objectcacher: 20
  debug ms: 1

Reproduced with client side logging after running in a loop for several days.

ubuntu@teuthology:/var/lib/teuthworker/archive/teuthology-2013-03-20_07:49:52-rbd-master-testing-basic/36

Josh - any update? Are we still going to get this in Cuttlefish?

Unfortunately the logs aren't there anymore (they weren't saved when a power failure restarted the machine running teuthology). Trying to reproduce again, although I've still got the core file at least.

Nevermind, the logs were saved after all. Hooray!

ubuntu@teuthology:/a/teuthology-2013-04-23_19:55:59-rbd-next-testing-basic$ less 148/teuthology.log

ubuntu@teuthology:/a/teuthology-2013-04-23_19:55:59-rbd-next-testing-basic$ less 155/teuthology.log