Ceph

Logs are in http://qa-proxy.ceph.com/teuthology/teuthology-2014-03-07_02:30:02-rados-firefly-testing-basic-plana/121141/

What is interesting that it passed manually.

2014-03-07T07:46:07.515 INFO:teuthology.task.rados.rados.0.out:[10.214.131.30]: update_object_version oid 42 v 1617 (ObjNum 394 snap 103 seq_num 394) dirty exists
2014-03-07T07:46:07.515 INFO:teuthology.task.rados.rados.0.out:[10.214.131.30]: 1021:  finishing write tid 4 to plana105665-15
2014-03-07T07:46:07.515 INFO:teuthology.task.rados.rados.0.out:[10.214.131.30]: update_object_version oid 15 v 1685 (ObjNum 395 snap 103 seq_num 395) dirty exists
2014-03-07T07:46:07.515 INFO:teuthology.task.rados.rados.0.err:[10.214.131.30]: 1020: Error: did not find header attr, has_contents: 1
2014-03-07T07:46:07.516 INFO:teuthology.task.rados.rados.0.err:[10.214.131.30]: ./test/osd/RadosModel.h: In function 'virtual void ReadOp::_finish(TestOp::CallbackInfo*)' thread 7f8682ffd700 time 2014-03-07 07:46:07.512668
2014-03-07T07:46:07.516 INFO:teuthology.task.rados.rados.0.err:[10.214.131.30]: ./test/osd/RadosModel.h: 1037: FAILED assert(!old_value.has_contents())
2014-03-07T07:46:07.516 INFO:teuthology.task.rados.rados.0.err:[10.214.131.30]:  ceph version 0.77-757-g8221a8e (8221a8ecba14f80eb7e35e3b1e6fe8487502b2d9)
2014-03-07T07:46:07.516 INFO:teuthology.task.rados.rados.0.err:[10.214.131.30]:  1: (ReadOp::_finish(TestOp::CallbackInfo*)+0x12d7) [0x41bf57]
2014-03-07T07:46:07.516 INFO:teuthology.task.rados.rados.0.err:[10.214.131.30]:  2: (librados::C_AioComplete::finish(int)+0x1d) [0x7f868d47c83d]
2014-03-07T07:46:07.517 INFO:teuthology.task.rados.rados.0.err:[10.214.131.30]:  3: (Context::complete(int)+0x9) [0x7f868d459f69]
2014-03-07T07:46:07.517 INFO:teuthology.task.rados.rados.0.err:[10.214.131.30]:  4: (Finisher::finisher_thread_entry()+0x1c0) [0x7f868d50cc80]
2014-03-07T07:46:07.517 INFO:teuthology.task.rados.rados.0.err:[10.214.131.30]:  5: (()+0x7e9a) [0x7f868d0b3e9a]
2014-03-07T07:46:07.517 INFO:teuthology.task.rados.rados.0.err:[10.214.131.30]:  6: (clone()+0x6d) [0x7f868c8ca3fd]
2014-03-07T07:46:07.517 INFO:teuthology.task.rados.rados.0.err:[10.214.131.30]:  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2014-03-07T07:46:07.517 INFO:teuthology.task.rados.rados.0.err:[10.214.131.30]: terminate called after throwing an instance of 'ceph::FailedAssertion'
2014-03-07T07:46:07.851 ERROR:teuthology.run_tasks:Manager failed: rados
Traceback (most recent call last):
  File "/home/teuthworker/teuthology-firefly/teuthology/run_tasks.py", line 84, in run_tasks
    suppress = manager.__exit__(*exc_info)
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/home/teuthworker/teuthology-firefly/teuthology/task/rados.py", line 170, in task
    running.get()
  File "/usr/lib/python2.7/dist-packages/gevent/greenlet.py", line 331, in get
    raise self._exception
CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --ec-pool --op read 100 --op write 0 --op delete 50 --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op copy_from 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op append 100 --pool unique_pool_0'

archive_path: /var/lib/teuthworker/archive/teuthology-2014-03-07_02:30:02-rados-firefly-testing-basic-plana/121141
description: rados/thrash/{clusters/fixed-2.yaml fs/ext4.yaml msgr-failures/osd-delay.yaml
  thrashers/default.yaml workloads/ec-snaps-few-objects.yaml}
email: null
job_id: '121141'
kernel: &id001
  kdb: true
  sha1: f31a96afabfad92cb917fd52a421b23275cdb6da
last_in_suite: false
machine_type: plana
name: teuthology-2014-03-07_02:30:02-rados-firefly-testing-basic-plana
nuke-on-error: true
os_type: ubuntu
overrides:
  admin_socket:
    branch: firefly
  ceph:
    conf:
      global:
        ms inject delay max: 1
        ms inject delay probability: 0.005
        ms inject delay type: osd
        ms inject internal delays: 0.002
        ms inject socket failures: 2500
      mon:
        debug mon: 20
        debug ms: 1
        debug paxos: 20
      osd:
        debug filestore: 20
        debug ms: 1
        debug osd: 20
    fs: ext4
    log-whitelist:
    - slow request
    sha1: 8221a8ecba14f80eb7e35e3b1e6fe8487502b2d9
  ceph-deploy:
    branch:
      dev: firefly
    conf:
      client:
        log file: /var/log/ceph/ceph-$name.$pid.log
      mon:
        debug mon: 1
        debug ms: 20
        debug paxos: 20
        osd default pool size: 2
  install:
    ceph:
      sha1: 8221a8ecba14f80eb7e35e3b1e6fe8487502b2d9
  s3tests:
    branch: master
  workunit:
    sha1: 8221a8ecba14f80eb7e35e3b1e6fe8487502b2d9
owner: scheduled_teuthology@teuthology
roles:
- - mon.a
  - mon.c
  - osd.0
  - osd.1
  - osd.2
  - client.0
- - mon.b
  - mds.a
  - osd.3
  - osd.4
  - osd.5
  - client.1
targets:
  ubuntu@plana10.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZM11QYmJVb8X/BHwLAOgozCK/ZG4XtzQZb5znJiFPwyi7vSZtCM7/DGWm6FGTUXg1bEcJHdD83EpvVgqTue/c6iTKCglzuB8eadW7gErwXjJgy/DCmJOHjmx/0fNCZi60nBVZQW3V1nptIgaFYHSFgPMhQSvEUg01ohfPlLKgnyeb6ciRxcDmJcrGpUrDNutxLf0aUC9gDF8tJ1nE5WV6tOSMNDpgVf9L3WwkxYj+G/77l+eSPqgDHs751C5iIzvt5eWTU+vfUd4uymPlBihPQXqnyfcriFJK8Tx80qofODJNxOLd9f+8wAUKXvY/pxSHL9h6sIrRI17TXPR9ip/V
  ubuntu@plana21.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUcR+fVqP6cDG9ax0r2ZiO1kX3BPmYU6jD3j44V9oejFMr7C3zCi82gsGcmbFEKz3JGYK5wWYrn6ueBxC1lH1FQO8qafvOuF5M87BU/l+AFAn8mqt6ohJZd5G+ze5+oxzDDHBeLBl0Mth7siHfODtTv7L8sOh/uPZRPUM0GH3wrftpu/adv1yvNdbgJGPoKSZNmDlvBfhvvBIBY4X8/5F8DfA8zEB1dQE96bIFaf56yoh7wDiumM56q4h4Hm7X9X+gLwYtw1BnLSqqqgGHhcQEwRhhXTP99iHswNJ0GDlqXpCcEgCwBqP1y7YbBS3p7SLsd4q4uG0+D9QV5Q9pfBIp
tasks:
- internal.lock_machines:
  - 2
  - plana
- internal.save_config: null
- internal.check_lock: null
- internal.connect: null
- internal.check_conflict: null
- internal.check_ceph_data: null
- internal.vm_setup: null
- kernel: *id001
- internal.base: null
- internal.archive: null
- internal.coredump: null
- internal.sudo: null
- internal.syslog: null
- internal.timer: null
- chef: null
- clock.check: null
- install: null
- ceph:
    log-whitelist:
    - wrongly marked me down
    - objects unfound and apparently lost
- thrashosds:
    chance_pgnum_grow: 1
    chance_pgpnum_fix: 1
    timeout: 1200
- rados:
    clients:
    - client.0
    ec_pool: true
    objects: 50
    op_weights:
      append: 100
      copy_from: 50
      delete: 50
      read: 100
      rollback: 50
      snap_create: 50
      snap_remove: 50
      write: 0
    ops: 4000
teuthology_branch: firefly
verbose: true
worker_log: /var/lib/teuthworker/archive/worker_logs/worker.plana.11185

description: rados/thrash/{clusters/fixed-2.yaml fs/ext4.yaml msgr-failures/osd-delay.yaml
  thrashers/default.yaml workloads/ec-snaps-few-objects.yaml}
duration: 979.2814102172852
failure_reason: 'Command crashed: ''CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage
  /home/ubuntu/cephtest/archive/coverage ceph_test_rados --ec-pool --op read 100 --op
  write 0 --op delete 50 --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000
  --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op copy_from
  50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op append 100 --pool
  unique_pool_0'''
flavor: basic
mon.a-kernel-sha1: f31a96afabfad92cb917fd52a421b23275cdb6da
mon.b-kernel-sha1: f31a96afabfad92cb917fd52a421b23275cdb6da
owner: scheduled_teuthology@teuthology
success: false