Project

General

Profile

Bug #58258

rook: kubelet fails from connection refused

Added by Laura Flores over 1 year ago. Updated about 1 year ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

/a/yuriw-2022-12-09_22:27:10-rados-main-distro-default-smithi/7110406

2022-12-10T07:31:03.876 INFO:teuthology.orchestra.run.smithi174.stdout:[kubelet-check] It seems like the kubelet isn't running or healthy.
2022-12-10T07:31:03.876 INFO:teuthology.orchestra.run.smithi174.stdout:[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp [::1]:10248: connect: connection refused.
2022-12-10T07:31:03.876 INFO:teuthology.orchestra.run.smithi174.stdout:
2022-12-10T07:31:03.877 INFO:teuthology.orchestra.run.smithi174.stdout:Unfortunately, an error has occurred:
2022-12-10T07:31:03.877 INFO:teuthology.orchestra.run.smithi174.stdout:    timed out waiting for the condition
2022-12-10T07:31:03.877 INFO:teuthology.orchestra.run.smithi174.stdout:
2022-12-10T07:31:03.877 INFO:teuthology.orchestra.run.smithi174.stdout:This error is likely caused by:
2022-12-10T07:31:03.877 INFO:teuthology.orchestra.run.smithi174.stdout:    - The kubelet is not running
2022-12-10T07:31:03.877 INFO:teuthology.orchestra.run.smithi174.stdout:    - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
2022-12-10T07:31:03.877 INFO:teuthology.orchestra.run.smithi174.stdout:
2022-12-10T07:31:03.877 INFO:teuthology.orchestra.run.smithi174.stdout:If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
2022-12-10T07:31:03.877 INFO:teuthology.orchestra.run.smithi174.stdout:    - 'systemctl status kubelet'
2022-12-10T07:31:03.877 INFO:teuthology.orchestra.run.smithi174.stdout:    - 'journalctl -xeu kubelet'
2022-12-10T07:31:03.878 INFO:teuthology.orchestra.run.smithi174.stdout:
2022-12-10T07:31:03.878 INFO:teuthology.orchestra.run.smithi174.stdout:Additionally, a control plane component may have crashed or exited when started by the container runtime.
2022-12-10T07:31:03.878 INFO:teuthology.orchestra.run.smithi174.stdout:To troubleshoot, list all containers using your preferred container runtimes CLI.
2022-12-10T07:31:03.878 INFO:teuthology.orchestra.run.smithi174.stdout:Here is one example how you may list all running Kubernetes containers by using crictl:
2022-12-10T07:31:03.878 INFO:teuthology.orchestra.run.smithi174.stdout:    - 'crictl --runtime-endpoint unix:///var/run/containerd/containerd.sock ps -a | grep kube | grep -v pause'
2022-12-10T07:31:03.878 INFO:teuthology.orchestra.run.smithi174.stdout:    Once you have found the failing container, you can inspect its logs with:
2022-12-10T07:31:03.878 INFO:teuthology.orchestra.run.smithi174.stdout:    - 'crictl --runtime-endpoint unix:///var/run/containerd/containerd.sock logs CONTAINERID'
2022-12-10T07:31:03.878 INFO:teuthology.orchestra.run.smithi174.stderr:error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
2022-12-10T07:31:03.879 INFO:teuthology.orchestra.run.smithi174.stderr:To see the stack trace of this error execute with --v=5 or higher
2022-12-10T07:31:03.882 DEBUG:teuthology.orchestra.run:got remote process result: 1

Older attempt to fix this failure was: https://tracker.ceph.com/issues/52116


Related issues

Related to Orchestrator - Bug #52116: kubeadm task fails with error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster Resolved

History

#1 Updated by Laura Flores over 1 year ago

  • Related to Bug #52116: kubeadm task fails with error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster added

#2 Updated by Laura Flores over 1 year ago

/a/yuriw-2022-12-12_21:48:52-rados-wip-yuri-testing-2022-12-12-1136-distro-default-smithi/7113446

2022-12-12T22:12:05.931 INFO:teuthology.orchestra.run.smithi140.stdout:[kubelet-check] It seems like the kubelet isn't running or healthy.
2022-12-12T22:12:05.931 INFO:teuthology.orchestra.run.smithi140.stdout:[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp [::1]:10248: connect: connection refused.
2022-12-12T22:12:05.931 INFO:teuthology.orchestra.run.smithi140.stdout:
2022-12-12T22:12:05.932 INFO:teuthology.orchestra.run.smithi140.stdout:Unfortunately, an error has occurred:
2022-12-12T22:12:05.932 INFO:teuthology.orchestra.run.smithi140.stdout: timed out waiting for the condition
2022-12-12T22:12:05.932 INFO:teuthology.orchestra.run.smithi140.stdout:
2022-12-12T22:12:05.932 INFO:teuthology.orchestra.run.smithi140.stdout:This error is likely caused by:
2022-12-12T22:12:05.932 INFO:teuthology.orchestra.run.smithi140.stdout: - The kubelet is not running
2022-12-12T22:12:05.932 INFO:teuthology.orchestra.run.smithi140.stdout: - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
2022-12-12T22:12:05.932 INFO:teuthology.orchestra.run.smithi140.stdout:
2022-12-12T22:12:05.932 INFO:teuthology.orchestra.run.smithi140.stdout:If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
2022-12-12T22:12:05.932 INFO:teuthology.orchestra.run.smithi140.stdout: - 'systemctl status kubelet'
2022-12-12T22:12:05.932 INFO:teuthology.orchestra.run.smithi140.stdout: - 'journalctl -xeu kubelet'
2022-12-12T22:12:05.933 INFO:teuthology.orchestra.run.smithi140.stdout:
2022-12-12T22:12:05.933 INFO:teuthology.orchestra.run.smithi140.stdout:Additionally, a control plane component may have crashed or exited when started by the container runtime.
2022-12-12T22:12:05.933 INFO:teuthology.orchestra.run.smithi140.stdout:To troubleshoot, list all containers using your preferred container runtimes CLI.
2022-12-12T22:12:05.933 INFO:teuthology.orchestra.run.smithi140.stdout:Here is one example how you may list all running Kubernetes containers by using crictl:
2022-12-12T22:12:05.933 INFO:teuthology.orchestra.run.smithi140.stdout: - 'crictl --runtime-endpoint unix:///var/run/containerd/containerd.sock ps -a | grep kube | grep -v pause'
2022-12-12T22:12:05.933 INFO:teuthology.orchestra.run.smithi140.stdout: Once you have found the failing container, you can inspect its logs with:
2022-12-12T22:12:05.933 INFO:teuthology.orchestra.run.smithi140.stdout: - 'crictl --runtime-endpoint unix:///var/run/containerd/containerd.sock logs CONTAINERID'
2022-12-12T22:12:05.933 INFO:teuthology.orchestra.run.smithi140.stderr:error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
2022-12-12T22:12:05.934 INFO:teuthology.orchestra.run.smithi140.stderr:To see the stack trace of this error execute with --v=5 or higher
2022-12-12T22:12:05.939 DEBUG:teuthology.orchestra.run:got remote process result: 1
2022-12-12T22:12:05.939 ERROR:tasks.kubeadm:Command failed on smithi140 with status 1: 'sudo kubeadm init --node-name smithi140 --token abcdef.t6ki4p7hie6bufkn --pod-network-cidr 10.252.88.0/21'

#3 Updated by Kamoltat (Junior) Sirivadhna over 1 year ago

/a/ksirivad-2022-12-15_06:28:05-rados-wip-ksirivad-testing-main-distro-default-smithi/7117979/

#4 Updated by Matan Breizman over 1 year ago

/a/yuriw-2022-12-20_16:13:57-rados-wip-yuri3-testing-2022-12-14-0855-pacific-distro-default-smithi/7122655

#5 Updated by Kamoltat (Junior) Sirivadhna over 1 year ago

/a/ksirivad-2022-12-21_15:23:02-rados-wip-ksirivad-test-quincy-distro-default-smithi/7123774/

#6 Updated by Matan Breizman over 1 year ago

/a/yuriw-2022-12-22_15:18:04-rados-pacific_16.2.11_RC3-distro-default-smithi/7124779

#7 Updated by Kamoltat (Junior) Sirivadhna over 1 year ago

/a/ksirivad-2022-12-22_17:58:01-rados-wip-ksirivad-testing-pacific-distro-default-smithi/7124917/

#8 Updated by Laura Flores about 1 year ago

/a/yuriw-2023-01-12_20:11:41-rados-main-distro-default-smithi/7138617

Also available in: Atom PDF