Ceph » RADOS

Description: rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2}

Failure Reason: hit max job timeout

/a/yuriw-2022-01-11_19:17:55-rados-wip-yuri5-testing-2022-01-11-0843-distro-default-smithi/6608450

2022-01-11T20:49:27.596 INFO:tasks.ceph.ceph_manager.ceph:PG 3.41 is not active+clean
2022-01-11T20:49:27.597 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.41', 'version': "458'317", 'reported_seq': 1036, 'reported_epoch': 1029, 'state': 'recovery_wait+undersized+degraded+remapped+peered', 'last_fresh': '2022-01-11T20:49:2
0.883156+0000', 'last_change': '2022-01-11T20:28:20.445497+0000', 'last_active': '2022-01-11T20:28:18.367398+0000', 'last_peered': '2022-01-11T20:49:20.883156+0000', 'last_clean': '2022-01-11T20:27:59.375878+0000', 'last_became_active': '
2022-01-11T20:28:13.397766+0000', 'last_became_peered': '2022-01-11T20:28:20.443480+0000', 'last_unstale': '2022-01-11T20:49:20.883156+0000', 'last_undegraded': '2022-01-11T20:28:20.381645+0000', 'last_fullsized': '2022-01-11T20:28:20.363
365+0000', 'mapping_epoch': 464, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 340, 'last_epoch_clean': 384, 'parent': '0.0', 'parent_split_bits': 7, 'last_scrub': "415'285", 'last_scrub_stamp': '2022-01-11T20:27:31.399421+000
0', 'last_deep_scrub': "104'133", 'last_deep_scrub_stamp': '2022-01-11T20:22:21.657242+0000', 'last_clean_scrub_stamp': '2022-01-11T20:27:31.399421+0000', 'log_size': 317, 'ondisk_log_size': 317, 'stats_invalid': False, 'dirty_stats_inval
id': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 10, 'last_scrub_duration': 1, 'scrub_schedule': 'qu
eued for scrub', 'stat_sum': {'num_bytes': 24592384, 'num_objects': 5, 'num_object_clones': 4, 'num_object_copies': 30, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 60, 'num_objects_misplaced': 0,
 'num_objects_unfound': 0, 'num_objects_dirty': 5, 'num_whiteouts': 1, 'num_read': 7, 'num_read_kb': 22383, 'num_write': 7, 'num_write_kb': 3088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objec
ts_recovered': 1, 'num_bytes_recovered': 7258112, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num
_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_byte
s': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4, 1, 8, 0, 2], 'acting': [6, 2147483647, 1, 8, 0, 2147483647], 'avail_no_missing': ['6(0)', '0(4)', '4(5)', '5(1)', '8(3)', '10(2)'], 'object_location_counts': [{'shards': 
'0(4),2(2),4(5),5(1),6(0),8(3),10(2)', 'objects': 1}, {'shards': '0(4),4(5),5(1),6(0),8(3)', 'objects': 3}, {'shards': '0(4),4(5),5(1),6(0),8(3),10(2)', 'objects': 1}], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps
': []}
2022-01-11T20:49:27.598 INFO:tasks.thrashosds.thrasher:Traceback (most recent call last):
  File "/home/teuthworker/src/github.com_ceph_ceph-c_12e70511ad0d4669cc9b87f2162909e36a0361eb/qa/tasks/ceph_manager.py", line 189, in wrapper
    return func(self)
  File "/home/teuthworker/src/github.com_ceph_ceph-c_12e70511ad0d4669cc9b87f2162909e36a0361eb/qa/tasks/ceph_manager.py", line 1405, in _do_thrash
    timeout=self.config.get('timeout')
  File "/home/teuthworker/src/github.com_ceph_ceph-c_12e70511ad0d4669cc9b87f2162909e36a0361eb/qa/tasks/ceph_manager.py", line 2822, in wait_for_recovery
    'wait_for_recovery: failed before timeout expired'
AssertionError: wait_for_recovery: failed before timeout expired