Actions
Bug #59124
open"Health check failed: 1/3 mons down, quorum b,c (MON_DOWN)" during quincy p2p upgrade test
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
/a/yuriw-2023-03-14_21:36:24-upgrade:quincy-p2p-quincy-release-distro-default-smithi/7208396
2023-03-16T08:56:52.709 DEBUG:teuthology.orchestra.run.smithi115:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'reached quota' | egrep -v scrub | egrep -v osd_map_max_advance | egrep -v 'wrongly marked' | egrep -v FS_DEGRADED | egrep -v POOL_APP_NOT_ENABLED | egrep -v CACHE_POOL_NO_HIT_SET | egrep -v POOL_FULL | egrep -v SMALLER_PG | egrep -v 'pool\(s\) full' | egrep -v OSD_DOWN | egrep -v 'missing hit_sets' | egrep -v CACHE_POOL_NEAR_FULL | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'application not enabled' | egrep -v 'cache pools at or near target size' | egrep -v 'filesystem is degraded' | egrep -v OBJECT_MISPLACED | head -n 1
2023-03-16T08:56:53.502 INFO:teuthology.orchestra.run.smithi115.stdout:2023-03-16T07:37:38.814552+0000 mon.b (mon.1) 7 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,c (MON_DOWN)
2023-03-16T08:56:54.798 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log
2023-03-16T08:56:54.799 DEBUG:teuthology.orchestra.run.smithi115:> sudo egrep '\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'reached quota' | egrep -v scrub | egrep -v osd_map_max_advance | egrep -v 'wrongly marked' | egrep -v FS_DEGRADED | egrep -v POOL_APP_NOT_ENABLED | egrep -v CACHE_POOL_NO_HIT_SET | egrep -v POOL_FULL | egrep -v SMALLER_PG | egrep -v 'pool\(s\) full' | egrep -v OSD_DOWN | egrep -v 'missing hit_sets' | egrep -v CACHE_POOL_NEAR_FULL | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'application not enabled' | egrep -v 'cache pools at or near target size' | egrep -v 'filesystem is degraded' | egrep -v OBJECT_MISPLACED | head -n 1
2023-03-16T08:56:54.862 DEBUG:teuthology.orchestra.run.smithi115:> sudo egrep '\[ERR\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'reached quota' | egrep -v scrub | egrep -v osd_map_max_advance | egrep -v 'wrongly marked' | egrep -v FS_DEGRADED | egrep -v POOL_APP_NOT_ENABLED | egrep -v CACHE_POOL_NO_HIT_SET | egrep -v POOL_FULL | egrep -v SMALLER_PG | egrep -v 'pool\(s\) full' | egrep -v OSD_DOWN | egrep -v 'missing hit_sets' | egrep -v CACHE_POOL_NEAR_FULL | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'application not enabled' | egrep -v 'cache pools at or near target size' | egrep -v 'filesystem is degraded' | egrep -v OBJECT_MISPLACED | head -n 1
2023-03-16T08:56:56.516 DEBUG:teuthology.orchestra.run.smithi115:> sudo egrep '\[WRN\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'reached quota' | egrep -v scrub | egrep -v osd_map_max_advance | egrep -v 'wrongly marked' | egrep -v FS_DEGRADED | egrep -v POOL_APP_NOT_ENABLED | egrep -v CACHE_POOL_NO_HIT_SET | egrep -v POOL_FULL | egrep -v SMALLER_PG | egrep -v 'pool\(s\) full' | egrep -v OSD_DOWN | egrep -v 'missing hit_sets' | egrep -v CACHE_POOL_NEAR_FULL | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'application not enabled' | egrep -v 'cache pools at or near target size' | egrep -v 'filesystem is degraded' | egrep -v OBJECT_MISPLACED | head -n 1
2023-03-16T08:56:57.633 INFO:teuthology.orchestra.run.smithi115.stdout:2023-03-16T07:37:38.814552+0000 mon.b (mon.1) 7 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,c (MON_DOWN)
Updated by Laura Flores about 1 year ago
Perhaps something needs to be whitelisted here.
Actions