Ceph » RADOS

/a/lflores-2023-03-27_02:17:31-rados-wip-aclamk-bs-elastic-shared-blob-save-25.03.2023-a-distro-default-smithi/7220990

2023-03-27T07:36:12.704 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.5', 'version': "781'981", 'reported_seq': 2565, 'reported_epoch': 985, 'state': 'active+undersized+degraded', 'last_fresh': '2023-03-27T07:36:02.849903+0000', 'last_change': '2023-03-27T07:15:53.567376+0000', 'last_active': '2023-03-27T07:36:02.849903+0000', 'last_peered': '2023-03-27T07:36:02.849903+0000', 'last_clean': '2023-03-27T07:15:50.823756+0000', 'last_became_active': '2023-03-27T07:15:53.567376+0000', 'last_became_peered': '2023-03-27T07:15:53.567376+0000', 'last_unstale': '2023-03-27T07:36:02.849903+0000', 'last_undegraded': '2023-03-27T07:15:53.555753+0000', 'last_fullsized': '2023-03-27T07:15:53.555539+0000', 'mapping_epoch': 162, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 143, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "105'164", 'last_scrub_stamp': '2023-03-27T07:14:56.445168+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-03-27T07:13:27.693605+0000', 'last_clean_scrub_stamp': '2023-03-27T07:14:56.445168+0000', 'objects_scrubbed': 15, 'log_size': 981, 'log_dups_size': 0, 'ondisk_log_size': 981, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 305, 'last_scrub_duration': 1, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0.012991803, 'objects_trimmed': 0, 'snaptrim_duration': 0.025207018, 'stat_sum': {'num_bytes': 232095744, 'num_objects': 156, 'num_object_clones': 151, 'num_object_copies': 936, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 156, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 156, 'num_whiteouts': 0, 'num_read': 383, 'num_read_kb': 890442, 'num_write': 1256, 'num_write_kb': 223520, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 57, 'num_bytes_recovered': 117489664, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2147483647, 2, 3, 7, 5, 1], 'acting': [2147483647, 2, 3, 7, 5, 1], 'avail_no_missing': ['2(1)', '1(5)', '3(2)', '5(4)', '7(3)'], 'object_location_counts': [{'shards': '1(5),2(1),3(2),5(4),7(3)', 'objects': 156}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []}
2023-03-27T07:36:12.705 INFO:tasks.thrashosds.thrasher:Traceback (most recent call last):
  File "/home/teuthworker/src/github.com_ljflores_ceph_5e717292106ca2d310770101bfebb345837be8e1/qa/tasks/ceph_manager.py", line 190, in wrapper
    return func(self)
  File "/home/teuthworker/src/github.com_ljflores_ceph_5e717292106ca2d310770101bfebb345837be8e1/qa/tasks/ceph_manager.py", line 1401, in _do_thrash
    self.choose_action()()
  File "/home/teuthworker/src/github.com_ljflores_ceph_5e717292106ca2d310770101bfebb345837be8e1/qa/tasks/ceph_manager.py", line 967, in test_pool_min_size
    self.ceph_manager.wait_for_clean(timeout=self.config.get('timeout'))
  File "/home/teuthworker/src/github.com_ljflores_ceph_5e717292106ca2d310770101bfebb345837be8e1/qa/tasks/ceph_manager.py", line 2747, in wait_for_clean
    assert time.time() - start < timeout, \
AssertionError: wait_for_clean: failed before timeout expired

Last attempt to fix was in https://tracker.ceph.com/issues/51904.