Ceph » RADOS

2021-12-19T02:15:16.582 INFO:tasks.backfill_toofull:pg={'pgid': '2.0', 'version': "23'14551", 'reported_seq': 14906, 'reported_epoch': 33, 'state': 'active+undersized+degraded+remapped+backfill_toofull', 'last_fresh': '2021-12-19T02:14:54.380853+0000', 'last_change': '2021-12-19T02:14:54.380853+0000', 'last_active': '2021-12-19T02:14:54.380853+0000', 'last_peered': '2021-12-19T02:14:54.380853+0000', 'last_clean': '2021-12-19T01:34:28.459149+0000', 'last_became_active': '2021-12-19T01:35:54.242082+0000', 'last_became_peered': '2021-12-19T01:35:54.242082+0000', 'last_unstale': '2021-12-19T02:14:54.380853+0000', 'last_undegraded': '2021-12-19T01:35:54.225980+0000', 'last_fullsized': '2021-12-19T01:35:54.221427+0000', 'mapping_epoch': 28, 'log_start': "23'14469", 'ondisk_log_start': "23'14469", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2021-12-19T01:33:15.714744+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2021-12-19T01:33:15.714744+0000', 'last_clean_scrub_stamp': '2021-12-19T01:33:15.714744+0000', 'log_size': 82, 'ondisk_log_size': 82, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2021-12-20T03:48:01.242929+0000', 'stat_sum': {'num_bytes': 25794969624, 'num_objects': 6151, 'num_object_clones': 0, 'num_object_copies': 18453, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 6151, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6151, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 14551, 'num_write_kb': 42389506, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 2147483647, 2], 'avail_no_missing': ['3(0)', '2(2)'], 'object_location_counts': [{'shards': '2(2),3(0)', 'objects': 6151}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []}
2021-12-19T02:15:16.582 DEBUG:tasks.backfill_toofull:not backfilling
2021-12-19T02:15:16.583 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/teuthology/run_tasks.py", line 91, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/teuthology/run_tasks.py", line 70, in run_one_task
    return task(**kwargs)
  File "/home/teuthworker/src/github.com_ceph_ceph-c_9f128e94b494f6aacaa2c4dc4adc35fe6fbd0e12/qa/tasks/backfill_toofull.py", line 169, in task
    wait_for_pg_state(manager, pgid, 'backfilling', target)
  File "/home/teuthworker/src/github.com_ceph_ceph-c_9f128e94b494f6aacaa2c4dc4adc35fe6fbd0e12/qa/tasks/backfill_toofull.py", line 30, in wait_for_pg_state
    assert False, '%s not in %s' % (pgid, state)
AssertionError: 2.0 not in backfilling

/a/yuriw-2021-12-18_18:14:24-rados-wip-yuriw-master-12.18.21-distro-default-smithi/6570206

https://sentry.ceph.com/organizations/ceph/issues/15974/events/612b123bed3c427184a6a1e65e3799b6/events/?project=2

Mykola: could you please take a look?