Project

General

Profile

Bug #42312

qa: failure to start mimic

Added by Patrick Donnelly 4 months ago. Updated 3 months ago.

Status:
Can't reproduce
Priority:
Urgent
Assignee:
-
Category:
-
Target version:
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
fs
Pull request ID:
Crash signature:

Description

2019-10-13T01:31:16.516 INFO:teuthology.run_tasks:Running task ceph...
...
2019-10-13T01:49:56.157 DEBUG:teuthology.misc:Ceph health: HEALTH_WARN too few PGs per OSD (12 < min 30)
2019-10-13T01:49:57.158 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges...
2019-10-13T01:49:57.158 INFO:teuthology.orchestra.run.smithi035:Running:
2019-10-13T01:49:57.159 INFO:teuthology.orchestra.run.smithi035:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json
...
2019-10-13T01:49:57.366 INFO:teuthology.orchestra.run.smithi035.stdout:{"epoch":15,"fsid":"e45efc46-c594-4f6f-822b-7717ce12506c","created":"2019-10-13 01:31:39.313908","modified":"2019-10-13 01:31:53.766997","flags":"sortbitwise,recovery_deletes,purged_snapdirs","flags_num":1605632,"flags_set":["purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.950000,"backfillfull_ratio":0.900000,"nearfull_ratio":0.850000,"cluster_snapshot":"","pool_max":3,"max_osd":4,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"mimic","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-10-13 01:31:49.219576","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_num":8,"pg_placement_num":8,"last_change":"11","last_force_op_resend":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2019-10-13 01:31:51.484940","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_num":8,"pg_placement_num":8,"last_change":"14","last_force_op_resend":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2019-10-13 01:31:51.767238","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_num":8,"pg_placement_num":8,"last_change":"14","last_force_op_resend":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"df28a38d-e440-46a9-9e93-93578d1a5c8a","up":1,"in":1,"weight":1.000000,"primary_affinity":1.000000,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addr":"172.21.15.35:6805/11230","cluster_addr":"172.21.15.35:6806/11230","heartbeat_back_addr":"172.21.15.35:6807/11230","heartbeat_front_addr":"172.21.15.35:6808/11230","state":["exists","up"]},{"osd":1,"uuid":"cac44433-2450-4d61-8a0a-418d0211480a","up":1,"in":1,"weight":1.000000,"primary_affinity":1.000000,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addr":"172.21.15.35:6801/11229","cluster_addr":"172.21.15.35:6802/11229","heartbeat_back_addr":"172.21.15.35:6803/11229","heartbeat_front_addr":"172.21.15.35:6804/11229","state":["exists","up"]},{"osd":2,"uuid":"4cdca9f0-d167-4d79-a834-5f4a58d46e51","up":1,"in":1,"weight":1.000000,"primary_affinity":1.000000,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addr":"172.21.15.35:6809/11233","cluster_addr":"172.21.15.35:6812/11233","heartbeat_back_addr":"172.21.15.35:6814/11233","heartbeat_front_addr":"172.21.15.35:6816/11233","state":["exists","up"]},{"osd":3,"uuid":"3ec273b7-d438-4662-a8f7-1e3c4b8dfacd","up":1,"in":1,"weight":1.000000,"primary_affinity":1.000000,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addr":"172.21.15.35:6810/11232","cluster_addr":"172.21.15.35:6811/11232","heartbeat_back_addr":"172.21.15.35:6813/11232","heartbeat_front_addr":"172.21.15.35:6815/11232","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0.000000,"laggy_interval":0,"features":4611087854031667195,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0.000000,"laggy_interval":0,"features":4611087854031667195,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0.000000,"laggy_interval":0,"features":4611087854031667195,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0.000000,"laggy_interval":0,"features":4611087854031667195,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[]}
2019-10-13T01:49:57.367 INFO:teuthology.orchestra.run.smithi035.stderr:2019-10-13 01:49:57.360 7febbfdc7700  1 -- 172.21.15.35:0/2890315975 >> 172.21.15.35:6800/10859 conn(0x7feba0006de0 :-1 s=STATE_OPEN pgs=221 cs=1 l=1).mark_down
2019-10-13T01:49:57.367 INFO:teuthology.orchestra.run.smithi035.stderr:2019-10-13 01:49:57.364 7febbfdc7700  1 -- 172.21.15.35:0/2890315975 >> 172.21.15.35:6789/0 conn(0x7febb80a28f0 :-1 s=STATE_OPEN pgs=269 cs=1 l=1).mark_down
2019-10-13T01:49:57.367 INFO:teuthology.orchestra.run.smithi035.stderr:2019-10-13 01:49:57.364 7febbfdc7700  1 -- 172.21.15.35:0/2890315975 shutdown_connections
2019-10-13T01:49:57.367 INFO:teuthology.orchestra.run.smithi035.stderr:2019-10-13 01:49:57.364 7febbfdc7700  1 -- 172.21.15.35:0/2890315975 shutdown_connections
2019-10-13T01:49:57.367 INFO:teuthology.orchestra.run.smithi035.stderr:2019-10-13 01:49:57.364 7febbfdc7700  1 -- 172.21.15.35:0/2890315975 wait complete.
2019-10-13T01:49:57.368 INFO:teuthology.orchestra.run.smithi035.stderr:2019-10-13 01:49:57.364 7febbfdc7700  1 -- 172.21.15.35:0/2890315975 >> 172.21.15.35:0/2890315975 conn(0x7febb8098b60 :-1 s=STATE_NONE pgs=0 cs=0 l=0).mark_down
2019-10-13T01:49:57.377 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 32, in nested
    yield vars
  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20191012.025926/qa/tasks/ceph.py", line 1944, in task
    ctx.managers[config['cluster']].stop_pg_num_changes()
  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20191012.025926/qa/tasks/ceph_manager.py", line 1900, in stop_pg_num_changes
    if pool['pg_num'] != pool['pg_num_target']:
KeyError: 'pg_num_target'

From: /ceph/teuthology-archive/pdonnell-2019-10-12_07:19:34-fs-wip-pdonnell-testing-20191012.025926-distro-basic-smithi/4407515/teuthology.log

History

#1 Updated by Patrick Donnelly 4 months ago

  • Description updated (diff)

#2 Updated by Patrick Donnelly 3 months ago

  • Status changed from New to Can't reproduce

Haven't seen this anymore.

Also available in: Atom PDF