Ceph

Logs: ubuntu@teuthology:/a/teuthology-2012-09-07_19:00:06-regression-master-testing-gcov/18598

2012-09-07T20:12:23.158 INFO:teuthology.task.peer:pg is {u'last_scrub': u"0'0", u'log_start': u"0'0", u'last_active': u'2012-09-07 20:11:29.678453', u'log_size': 512, u'last_deep_scrub': u"0'0", u'parent_split_bits': 0, u'ondisk_log_size': 512, u'mapping_epoch': 6, u'state': u'active+recovering', u'version': u"5'4", u'pgid': u'1.0', u'parent': u'0.0', u'reported': u"3'18", u'last_epoch_clean': 1, u'last_deep_scrub_stamp': u'0.000000', u'stat_cat_sum': {}, u'last_fresh': u'2012-09-07 20:11:29.678453', u'last_change': u'2012-09-07 20:11:29.519861', u'created': 1, u'up': [0, 2], u'stat_sum': {u'num_objects_unfound': 0, u'num_objects_missing_on_primary': 0, u'num_write': 12, u'num_object_clones': 0, u'num_objects': 4, u'num_object_copies': 8, u'num_bytes': 820, u'num_read_kb': 0, u'num_read': 4, u'num_write_kb': 4, u'num_objects_degraded': 8}, u'acting': [0, 2], u'last_clean': u'2012-09-07 20:11:19.669425', u'last_unstale': u'2012-09-07 20:11:29.678453', u'last_scrub_stamp': u'0.000000', u'ondisk_log_start': u"0'0"}, query json is {u'info': {u'last_backfill': u'MAX', u'dne': 0, u'pgid': u'1.0', u'log_tail': u"0'0", u'last_update': u"5'4", u'purged_snaps': u'[]', u'last_complete': u"5'4", u'incomplete': 0, u'stats': {u'last_scrub': u"5'4", u'log_start': u"0'0", u'last_active': u'2012-09-07 20:11:45.673667', u'log_size': 512, u'last_deep_scrub': u"0'0", u'parent_split_bits': 0, u'ondisk_log_size': 512, u'mapping_epoch': 6, u'state': u'active+clean', u'version': u"5'4", u'parent': u'0.0', u'reported': u"3'25", u'last_epoch_clean': 1, u'last_deep_scrub_stamp': u'0.000000', u'stat_cat_sum': {}, u'last_fresh': u'2012-09-07 20:11:45.673667', u'last_change': u'2012-09-07 20:11:45.673667', u'created': 1, u'up': [0, 2], u'ondisk_log_start': u"0'0", u'acting': [0, 2], u'last_clean': u'2012-09-07 20:11:45.673667', u'last_unstale': u'2012-09-07 20:11:45.673667', u'last_scrub_stamp': u'2012-09-07 20:11:45.673592', u'stat_sum': {u'num_objects_unfound': 0, u'num_objects_missing_on_primary': 0, u'num_write': 12, u'num_object_clones': 0, u'num_objects': 4, u'num_object_copies': 0, u'num_bytes': 820, u'num_read_kb': 0, u'num_read': 4, u'num_write_kb': 4, u'num_objects_degraded': 0}}, u'empty': 0, u'history': {u'last_scrub': u"5'4", u'epoch_created': 1, u'last_deep_scrub_stamp': u'0.000000', u'same_interval_since': 10, u'same_primary_since': 3, u'last_epoch_split': 0, u'same_up_since': 10, u'last_deep_scrub': u"0'0", u'last_epoch_clean': 11, u'last_epoch_started': 11, u'last_scrub_stamp': u'2012-09-07 20:11:45.673592'}}, u'recovery_state': [{u'recovery_progress': {u'pushing': [], u'backfill_info': {u'begin': u'0//0//-1', u'objects': [], u'end': u'0//0//-1'}, u'pull_from_peer': [], u'peer_backfill_info': {u'begin': u'0//0//-1', u'objects': [], u'end': u'0//0//-1'}, u'waiting_on_backfill': 0, u'backfills_in_flight': [], u'backfill_target': -1, u'backfill_pos': u'0//0//-1'}, u'scrub': {u'scrubber.block_writes': 0, u'scrubber.finalizing': 0, u'scrubber.active': 0, u'scrubber.epoch_start': u'10', u'scrubber.waiting_on_whom': [], u'scrubber.waiting_on': 0}, u'enter_time': u'2012-09-07 20:11:29.519821', u'name': u'Started/Primary/Active', u'might_have_unfound': []}, {u'enter_time': u'2012-09-07 20:11:29.075243', u'name': u'Started'}], u'state': u'active+clean', u'up': [0, 2], u'acting': [0, 2]}
2012-09-07T20:12:23.158 ERROR:teuthology.run_tasks:Saw exception from tasks
Traceback (most recent call last):
  File "/var/lib/teuthworker/teuthology/teuthology/run_tasks.py", line 25, in run_tasks
    manager = _run_one_task(taskname, ctx=ctx, config=config)
  File "/var/lib/teuthworker/teuthology/teuthology/run_tasks.py", line 14, in _run_one_task
    return fn(**kwargs)
  File "/var/lib/teuthworker/teuthology/teuthology/task/peer.py", line 84, in task
    assert j['state'].replace('+scrubbing','') == pg['state'].replace('+scrubbing','')
AssertionError
2012-09-07T20:12:23.179 DEBUG:teuthology.run_tasks:Unwinding manager <contextlib.GeneratorContextManager object at 0x1a978d0>
2012-09-07T20:12:23.180 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
  File "/var/lib/teuthworker/teuthology/teuthology/contextutil.py", line 27, in nested
    yield vars
  File "/var/lib/teuthworker/teuthology/teuthology/task/ceph.py", line 1077, in task
    yield
  File "/var/lib/teuthworker/teuthology/teuthology/run_tasks.py", line 25, in run_tasks
    manager = _run_one_task(taskname, ctx=ctx, config=config)
  File "/var/lib/teuthworker/teuthology/teuthology/run_tasks.py", line 14, in _run_one_task
    return fn(**kwargs)
  File "/var/lib/teuthworker/teuthology/teuthology/task/peer.py", line 84, in task
    assert j['state'].replace('+scrubbing','') == pg['state'].replace('+scrubbing','')
AssertionError
ubuntu@teuthology:/a/teuthology-2012-09-07_19:00:06-regression-master-testing-gcov/18598$ cat config.yaml
kernel: &id001
  kdb: true
  sha1: e81d5d695a03a141b9a4a4e75b8e009ecba43c64
nuke-on-error: true
overrides:
  ceph:
    conf:
      global:
        ms inject socket failures: 5000
    coverage: true
    fs: btrfs
    log-whitelist:
    - slow request
    sha1: 06290f6dffec33f4a9f47e4c3733f6779173f595
  workunit:
    sha1: 06290f6dffec33f4a9f47e4c3733f6779173f595
roles:
- - mon.0
  - mon.1
  - mon.2
  - mds.a
  - osd.0
  - osd.1
  - osd.2
targets:
  ubuntu@plana45.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDp3cwfZhOipCot6NiKX4cRMn4zx43QY0+5HdqzCQU2y7OrOJt3d0qvifnZPyeq8/d+aW2WL2OM8m4taz380JsP0SLmlpY8D0pGY/tN0pQDqIFd8EboMtKY6tR8unQrVzuczMqup/tkKSfdRp0zAeTiJ8qH7l9MaVcOw6WfRACb8f7APJE2gVRBrzPAdbqKzAphTRzZSz0cq722AX7XQDPT2dz7NoTp5Tk7xaQdDu2II+78B1H27IWdyYeonfy17yf9N+IA2Xzna/g5zu8apg7UvzyFmHunLyjr78dhPtR39201A0QJ5x5Qli9/UaB3LwiqnbCiGfx4xWFazdUFzxiD
tasks:
- internal.lock_machines: 1
- internal.save_config: null
- internal.check_lock: null
- internal.connect: null
- internal.check_conflict: null
- kernel: *id001
- internal.base: null
- internal.archive: null
- internal.coredump: null
- internal.syslog: null
- internal.timer: null
- chef: null
- clock: null
- ceph:
    log-whitelist:
    - objects unfound and apparently lost
- peer: null
ubuntu@teuthology:/a/teuthology-2012-09-07_19:00:06-regression-master-testing-gcov/18598$ cat summary.yaml
ceph-sha1: 06290f6dffec33f4a9f47e4c3733f6779173f595
description: collection:rados-singleton all:peer.yaml fs:btrfs.yaml msgr-failures:few.yaml
duration: 151.63367486000061
failure_reason: ''
flavor: gcov
mon.0-kernel-sha1: e81d5d695a03a141b9a4a4e75b8e009ecba43c64
owner: scheduled_teuthology@teuthology
success: false