Actions
Bug #3125
closedAssertion Error in peer.py - failure from the nightly run
% Done:
0%
Source:
Q/A
Tags:
Backport:
Regression:
Severity:
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
Logs: ubuntu@teuthology:/a/teuthology-2012-09-07_19:00:06-regression-master-testing-gcov/18598
2012-09-07T20:12:23.158 INFO:teuthology.task.peer:pg is {u'last_scrub': u"0'0", u'log_start': u"0'0", u'last_active': u'2012-09-07 20:11:29.678453', u'log_size': 512, u'last_deep_scrub': u"0'0", u'parent_split_bits': 0, u'ondisk_log_size': 512, u'mapping_epoch': 6, u'state': u'active+recovering', u'version': u"5'4", u'pgid': u'1.0', u'parent': u'0.0', u'reported': u"3'18", u'last_epoch_clean': 1, u'last_deep_scrub_stamp': u'0.000000', u'stat_cat_sum': {}, u'last_fresh': u'2012-09-07 20:11:29.678453', u'last_change': u'2012-09-07 20:11:29.519861', u'created': 1, u'up': [0, 2], u'stat_sum': {u'num_objects_unfound': 0, u'num_objects_missing_on_primary': 0, u'num_write': 12, u'num_object_clones': 0, u'num_objects': 4, u'num_object_copies': 8, u'num_bytes': 820, u'num_read_kb': 0, u'num_read': 4, u'num_write_kb': 4, u'num_objects_degraded': 8}, u'acting': [0, 2], u'last_clean': u'2012-09-07 20:11:19.669425', u'last_unstale': u'2012-09-07 20:11:29.678453', u'last_scrub_stamp': u'0.000000', u'ondisk_log_start': u"0'0"}, query json is {u'info': {u'last_backfill': u'MAX', u'dne': 0, u'pgid': u'1.0', u'log_tail': u"0'0", u'last_update': u"5'4", u'purged_snaps': u'[]', u'last_complete': u"5'4", u'incomplete': 0, u'stats': {u'last_scrub': u"5'4", u'log_start': u"0'0", u'last_active': u'2012-09-07 20:11:45.673667', u'log_size': 512, u'last_deep_scrub': u"0'0", u'parent_split_bits': 0, u'ondisk_log_size': 512, u'mapping_epoch': 6, u'state': u'active+clean', u'version': u"5'4", u'parent': u'0.0', u'reported': u"3'25", u'last_epoch_clean': 1, u'last_deep_scrub_stamp': u'0.000000', u'stat_cat_sum': {}, u'last_fresh': u'2012-09-07 20:11:45.673667', u'last_change': u'2012-09-07 20:11:45.673667', u'created': 1, u'up': [0, 2], u'ondisk_log_start': u"0'0", u'acting': [0, 2], u'last_clean': u'2012-09-07 20:11:45.673667', u'last_unstale': u'2012-09-07 20:11:45.673667', u'last_scrub_stamp': u'2012-09-07 20:11:45.673592', u'stat_sum': {u'num_objects_unfound': 0, u'num_objects_missing_on_primary': 0, u'num_write': 12, u'num_object_clones': 0, u'num_objects': 4, u'num_object_copies': 0, u'num_bytes': 820, u'num_read_kb': 0, u'num_read': 4, u'num_write_kb': 4, u'num_objects_degraded': 0}}, u'empty': 0, u'history': {u'last_scrub': u"5'4", u'epoch_created': 1, u'last_deep_scrub_stamp': u'0.000000', u'same_interval_since': 10, u'same_primary_since': 3, u'last_epoch_split': 0, u'same_up_since': 10, u'last_deep_scrub': u"0'0", u'last_epoch_clean': 11, u'last_epoch_started': 11, u'last_scrub_stamp': u'2012-09-07 20:11:45.673592'}}, u'recovery_state': [{u'recovery_progress': {u'pushing': [], u'backfill_info': {u'begin': u'0//0//-1', u'objects': [], u'end': u'0//0//-1'}, u'pull_from_peer': [], u'peer_backfill_info': {u'begin': u'0//0//-1', u'objects': [], u'end': u'0//0//-1'}, u'waiting_on_backfill': 0, u'backfills_in_flight': [], u'backfill_target': -1, u'backfill_pos': u'0//0//-1'}, u'scrub': {u'scrubber.block_writes': 0, u'scrubber.finalizing': 0, u'scrubber.active': 0, u'scrubber.epoch_start': u'10', u'scrubber.waiting_on_whom': [], u'scrubber.waiting_on': 0}, u'enter_time': u'2012-09-07 20:11:29.519821', u'name': u'Started/Primary/Active', u'might_have_unfound': []}, {u'enter_time': u'2012-09-07 20:11:29.075243', u'name': u'Started'}], u'state': u'active+clean', u'up': [0, 2], u'acting': [0, 2]} 2012-09-07T20:12:23.158 ERROR:teuthology.run_tasks:Saw exception from tasks Traceback (most recent call last): File "/var/lib/teuthworker/teuthology/teuthology/run_tasks.py", line 25, in run_tasks manager = _run_one_task(taskname, ctx=ctx, config=config) File "/var/lib/teuthworker/teuthology/teuthology/run_tasks.py", line 14, in _run_one_task return fn(**kwargs) File "/var/lib/teuthworker/teuthology/teuthology/task/peer.py", line 84, in task assert j['state'].replace('+scrubbing','') == pg['state'].replace('+scrubbing','') AssertionError 2012-09-07T20:12:23.179 DEBUG:teuthology.run_tasks:Unwinding manager <contextlib.GeneratorContextManager object at 0x1a978d0> 2012-09-07T20:12:23.180 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/var/lib/teuthworker/teuthology/teuthology/contextutil.py", line 27, in nested yield vars File "/var/lib/teuthworker/teuthology/teuthology/task/ceph.py", line 1077, in task yield File "/var/lib/teuthworker/teuthology/teuthology/run_tasks.py", line 25, in run_tasks manager = _run_one_task(taskname, ctx=ctx, config=config) File "/var/lib/teuthworker/teuthology/teuthology/run_tasks.py", line 14, in _run_one_task return fn(**kwargs) File "/var/lib/teuthworker/teuthology/teuthology/task/peer.py", line 84, in task assert j['state'].replace('+scrubbing','') == pg['state'].replace('+scrubbing','') AssertionError ubuntu@teuthology:/a/teuthology-2012-09-07_19:00:06-regression-master-testing-gcov/18598$ cat config.yaml kernel: &id001 kdb: true sha1: e81d5d695a03a141b9a4a4e75b8e009ecba43c64 nuke-on-error: true overrides: ceph: conf: global: ms inject socket failures: 5000 coverage: true fs: btrfs log-whitelist: - slow request sha1: 06290f6dffec33f4a9f47e4c3733f6779173f595 workunit: sha1: 06290f6dffec33f4a9f47e4c3733f6779173f595 roles: - - mon.0 - mon.1 - mon.2 - mds.a - osd.0 - osd.1 - osd.2 targets: ubuntu@plana45.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDp3cwfZhOipCot6NiKX4cRMn4zx43QY0+5HdqzCQU2y7OrOJt3d0qvifnZPyeq8/d+aW2WL2OM8m4taz380JsP0SLmlpY8D0pGY/tN0pQDqIFd8EboMtKY6tR8unQrVzuczMqup/tkKSfdRp0zAeTiJ8qH7l9MaVcOw6WfRACb8f7APJE2gVRBrzPAdbqKzAphTRzZSz0cq722AX7XQDPT2dz7NoTp5Tk7xaQdDu2II+78B1H27IWdyYeonfy17yf9N+IA2Xzna/g5zu8apg7UvzyFmHunLyjr78dhPtR39201A0QJ5x5Qli9/UaB3LwiqnbCiGfx4xWFazdUFzxiD tasks: - internal.lock_machines: 1 - internal.save_config: null - internal.check_lock: null - internal.connect: null - internal.check_conflict: null - kernel: *id001 - internal.base: null - internal.archive: null - internal.coredump: null - internal.syslog: null - internal.timer: null - chef: null - clock: null - ceph: log-whitelist: - objects unfound and apparently lost - peer: null ubuntu@teuthology:/a/teuthology-2012-09-07_19:00:06-regression-master-testing-gcov/18598$ cat summary.yaml ceph-sha1: 06290f6dffec33f4a9f47e4c3733f6779173f595 description: collection:rados-singleton all:peer.yaml fs:btrfs.yaml msgr-failures:few.yaml duration: 151.63367486000061 failure_reason: '' flavor: gcov mon.0-kernel-sha1: e81d5d695a03a141b9a4a4e75b8e009ecba43c64 owner: scheduled_teuthology@teuthology success: false
Updated by Sage Weil over 11 years ago
- Status changed from New to Resolved
this is fixed up now, most recent commit was 3772d437dd4c562a6490f84124eb4757e22eca92
Actions