Ceph

Running test_librbd_fsx I encountered this again (with master, including
the changes in wip-rbd-read):

272 read 0x6b5a thru 0x110bf (0xa566 bytes)
osdc/Striper.cc: In function 'void Striper::StripedReadResult::assemble_result(CephContext*, ceph::bufferlist&, bool)' thread 7fa644870700 time 2012-11-19 12:38:29.464006
osdc/Striper.cc: 278: FAILED assert(p->first == end - p->second.second)
ceph version 0.54-669-gde12ae9 (de12ae9803e7aeb19f94aa4c9c587b54b12e5c32)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x95) [0x7fa64a455569]
2: (Striper::StripedReadResult::assemble_result(CephContext*, ceph::buffer::list&, bool)+0x3c5) [0x7fa64b2e440d]
3: (librbd::AioCompletion::finalize(CephContext*, long)+0x1ec) [0x7fa64b27ff5a]
4: (librbd::AioCompletion::complete_request(CephContext*, long)+0x295) [0x7fa64b2805a7]
5: (librbd::C_AioRead::finish(int)+0x3f2) [0x7fa64b2809e4]
6: (Context::complete(int)+0x2b) [0x7fa64b280ea9]
7: (librbd::AioRequest::complete(int)+0x6f) [0x7fa64b280f39]
8: (librbd::rados_req_cb(void*, void*)+0x34) [0x7fa64b2b0e14]
9: (librados::C_AioComplete::finish(int)+0x4d) [0x7fa64a3dde79]
10: (Finisher::finisher_thread_entry()+0x348) [0x7fa64a454554]
11: (Finisher::FinisherThread::entry()+0x1c) [0x7fa64a3c57d0]
12: (Thread::_entry_func(void*)+0x23) [0x7fa64a601011]
13: (()+0x7e9a) [0x7fa6493bee9a]
14: (clone()+0x6d) [0x7fa6496c7cbd]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
terminate called after throwing an instance of 'ceph::FailedAssertion'
Aborted

...and this seems to reproduce it so far:

./test_librbd_fsx -d -W -R -p 10 -t 1 -S 4 -N 300 rbd fsx

er, b35e37fb73c942c9be6c551d9c52d9507bc7fd7f

Recent logs: ubuntu@teuthology:/a/teuthology-2012-12-04_19:00:05-regression-next-testing-basic/8664

2012-12-04T23:44:34.023 INFO:teuthology.orchestra.run.err:osdc/Striper.cc: 281: FAILED assert(p->first == end - p->second.second)
2012-12-04T23:44:34.023 INFO:teuthology.orchestra.run.err: ceph version 0.55-135-g4cdc30b (4cdc30b94319ba1ef72bd2d976651dc889793d61)
2012-12-04T23:44:34.024 INFO:teuthology.orchestra.run.err: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x95) [0x7f6dea4b698d]
2012-12-04T23:44:34.024 INFO:teuthology.orchestra.run.err: 2: (Striper::StripedReadResult::assemble_result(CephContext*, ceph::buffer::list&, bool)+0x38f) [0x7f6deb30e725]
2012-12-04T23:44:34.024 INFO:teuthology.orchestra.run.err: 3: (librbd::AioCompletion::finalize(CephContext*, long)+0x1d7) [0x7f6deb2ae8af]
2012-12-04T23:44:34.024 INFO:teuthology.orchestra.run.err: 4: (librbd::AioCompletion::complete_request(CephContext*, long)+0x27a) [0x7f6deb2aeea2]
2012-12-04T23:44:34.024 INFO:teuthology.orchestra.run.err: 5: (librbd::C_AioRead::finish(int)+0x3c8) [0x7f6deb2af2b2]
2012-12-04T23:44:34.024 INFO:teuthology.orchestra.run.err: 6: (Context::complete(int)+0x2b) [0x7f6deb2af707]
2012-12-04T23:44:34.024 INFO:teuthology.orchestra.run.err: 7: (librbd::AioRequest::complete(int)+0x6f) [0x7f6deb2af797]
2012-12-04T23:44:34.024 INFO:teuthology.orchestra.run.err: 8: (librbd::rbd_req_cb(void*, void*)+0x3c) [0x7f6deb2debf0]
2012-12-04T23:44:34.025 INFO:teuthology.orchestra.run.err: 9: (librbd::AioCompletion::complete()+0xd4) [0x7f6deb2af906]
2012-12-04T23:44:34.025 INFO:teuthology.orchestra.run.err: 10: (librbd::AioCompletion::complete_request(CephContext*, long)+0x289) [0x7f6deb2aeeb1]
2012-12-04T23:44:34.025 INFO:teuthology.orchestra.run.err: 11: (librbd::C_AioRead::finish(int)+0x3c8) [0x7f6deb2af2b2]
2012-12-04T23:44:34.025 INFO:teuthology.orchestra.run.err: 12: (Context::complete(int)+0x2b) [0x7f6deb2af707]
2012-12-04T23:44:34.025 INFO:teuthology.orchestra.run.err: 13: (librbd::AioRequest::complete(int)+0x6f) [0x7f6deb2af797]
2012-12-04T23:44:34.025 INFO:teuthology.orchestra.run.err: 14: (librbd::C_CacheRead::finish(int)+0x24) [0x7f6deb2af31e]
2012-12-04T23:44:34.026 INFO:teuthology.orchestra.run.err: 15: (Context::complete(int)+0x2b) [0x7f6deb2af707]
2012-12-04T23:44:34.026 INFO:teuthology.orchestra.run.err: 16: (ObjectCacher::C_RetryRead::finish(int)+0xb4) [0x7f6deb2fd1b6]
2012-12-04T23:44:34.026 INFO:teuthology.orchestra.run.err: 17: (Context::complete(int)+0x2b) [0x7f6deb2af707]
2012-12-04T23:44:34.026 INFO:teuthology.orchestra.run.err: 18: (finish_contexts(CephContext*, std::list<Context*, std::allocator<Context*> >&, int)+0x267) [0x7f6deb2fa7b5]
2012-12-04T23:44:34.026 INFO:teuthology.orchestra.run.err: 19: (ObjectCacher::bh_read_finish(long, sobject_t, long, unsigned long, ceph::buffer::list&, int)+0x13af) [0x7f6deb2ee785]
2012-12-04T23:44:34.026 INFO:teuthology.orchestra.run.err: 20: (ObjectCacher::C_ReadFinish::finish(int)+0x6e) [0x7f6deb2fcf10]
2012-12-04T23:44:34.027 INFO:teuthology.orchestra.run.err: 21: (Context::complete(int)+0x2b) [0x7f6deb2af707]
2012-12-04T23:44:34.027 INFO:teuthology.orchestra.run.err: 22: (librbd::C_Request::finish(int)+0x142) [0x7f6deb2e73f4]
2012-12-04T23:44:34.027 INFO:teuthology.orchestra.run.err: 23: (librbd::context_cb(void*, void*)+0x40) [0x7f6deb2e693c]
2012-12-04T23:44:34.027 INFO:teuthology.orchestra.run.err: 24: (librados::C_AioComplete::finish(int)+0x4d) [0x7f6dea440b41]
2012-12-04T23:44:34.027 INFO:teuthology.orchestra.run.err: 25: (Finisher::finisher_thread_entry()+0x342) [0x7f6dea4b5afe]
2012-12-04T23:44:34.027 INFO:teuthology.orchestra.run.err: 26: (Finisher::FinisherThread::entry()+0x1c) [0x7f6dea42891a]
2012-12-04T23:44:34.028 INFO:teuthology.orchestra.run.err: 27: (Thread::_entry_func(void*)+0x23) [0x7f6dea6571b9]
2012-12-04T23:44:34.028 INFO:teuthology.orchestra.run.err: 28: (()+0x7e9a) [0x7f6de9424e9a]
2012-12-04T23:44:34.028 INFO:teuthology.orchestra.run.err: 29: (clone()+0x6d) [0x7f6de972c4bd]
2012-12-04T23:44:34.028 INFO:teuthology.orchestra.run.err: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2012-12-04T23:44:34.028 INFO:teuthology.orchestra.run.err:terminate called after throwing an instance of 'ceph::FailedAssertion'
ubuntu@teuthology:/a/teuthology-2012-12-04_19:00:05-regression-next-testing-basic/8664$ cat config.yaml 
kernel: &id001
  kdb: true
  sha1: 9954d934f23883b95fade3eb56448f2169735c72
nuke-on-error: true
overrides:
  ceph:
    conf:
      client:
        rbd cache: true
      global:
        ms inject socket failures: 5000
    fs: ext4
    log-whitelist:
    - slow request
    sha1: 4cdc30b94319ba1ef72bd2d976651dc889793d61
  s3tests:
    branch: next
  workunit:
    sha1: 4cdc30b94319ba1ef72bd2d976651dc889793d61
roles:
- - mon.a
  - osd.0
  - osd.1
  - osd.2
- - mds.a
  - osd.3
  - osd.4
  - osd.5
- - client.0
targets:
  ubuntu@plana16.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDI5A8ibfCXo5uFpfa+hLZoUoMjRl54060vpyvkYSeFtVz0iX7rCJRkZBqHiwz940yqs2+qnoczg/SVucIFMZSv/c14V/8hhRNHDK7n0lADYF5tkTs9fupeAQSHEPytW/RxOLlNappd1rd7M56rVxeUMPgkVJAdJoQnkXyhw9KqizSj4vFej4qSKUMfaOlXwxCNjmUrtyBdPkk6kOgWODtgOn6d39Tva0hwn7jfeFaYc/ixNPFgiO5um2Fz3XHJFPMWPSLt5X3eEn54ZTO33B6ybYIgoYGn0kVBl+Wks/VTQkky5wJnbS5mghhPNK/+U0KJ5rcQlIgMGgTrBBROcFw3
  ubuntu@plana26.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDx0US96hot7gygZ69W4nxJQ9myYnn3I22YtOaSPe+yFWOPJVQUuOST+aw5K6JDcjdO2Gq0aS6s01mgoWpZlO/FVDKss7vZ2KjMp3uPkGMpDZarNbR3QTe5YZYrl7Wfw4pMu4jh92hCWJEzy5nH0H3X2YJhOd5BdOYz0P97qsMSPQGxhlvDBYBhDl9MLgsS3lKm/Js/OPLO+Uf3/SZceCjUqO2m3WsrJSiQJKh8XUWUu3z+6C1Wg6TXSSlA/jdVCiokDg7WYwPN9zMwzzGkGv+GUGHKMZaPGRZb9LQJLTBf/OjwRSgclAVdDc3vnZeYAS5+sDnt2grnJnlBd1rBUj3n
  ubuntu@plana35.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCtCEUU7ZjJxMBOYq9dlPZppwlahU15OIq1a/xqI0YIpAF7ZLf1eis04s6Kzvl45ooKP2J6DW598Lr9zTtFb6j9RoAkgIbIOkWZmLQjeZesi//7B8uXBw7niNzU1iBI83K8dPL2cwTsgwXXtoaD03pW9HCv4ADhH12/CsfoqNPfql7ze4JlI07seHmUqkBuBFxLVgdZDXb+Cjdg2sQnMS5qZo0OGBp2yfBla01Pk8V1sHGg39miF0EDJZyIOL2ziZOcm1f02WVkUAD8MMOtfQbf33DT/ya4Th20YtRMQxNLkUYG6rxcjutzILFKDr7ugbvOwtBhVS7qplqLWAwHdi0p
tasks:
- internal.lock_machines: 3
- internal.save_config: null
- internal.check_lock: null
- internal.connect: null
- internal.check_conflict: null
- kernel: *id001
- internal.base: null
- internal.archive: null
- internal.coredump: null
- internal.syslog: null
- internal.timer: null
- chef: null
- clock: null
- ceph:
    log-whitelist:
    - wrongly marked me down
    - objects unfound and apparently lost
- thrashosds:
    timeout: 1200
- rbd_fsx:
    clients:
    - client.0
    ops: 2000