Project

General

Profile

Actions
Copy link

Bug #52923

closed

centos8 stream systemd version causes ceph-ansible tests to fail

Added by Brad Hubbard over 2 years ago. Updated over 2 years ago.

Status:
Resolved
Priority:
High
Assignee:
Brad Hubbard
Category:
Test Node
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):

Description

http://qa-proxy.ceph.com/teuthology/bhubbard-2021-10-12_03:15:53-ceph-ansible-wip-octopus-badone-testing-distro-basic-smithi/6432186/teuthology.log

2021-10-12T03:35:09.709 INFO:teuthology.orchestra.run.smithi028.stdout:TASK [ceph-infra : start firewalld] ********************************************
2021-10-12T03:35:09.709 INFO:teuthology.orchestra.run.smithi028.stdout:task path: /home/ubuntu/ceph-ansible/roles/ceph-infra/tasks/configure_firewall.yml:22
2021-10-12T03:35:09.709 INFO:teuthology.orchestra.run.smithi028.stdout:Tuesday 12 October 2021  03:35:09 +0000 (0:00:05.029)       0:00:20.562 *******
2021-10-12T03:35:10.232 INFO:teuthology.orchestra.run.smithi028.stdout:FAILED - RETRYING: start firewalld (5 retries left).
2021-10-12T03:35:10.239 INFO:teuthology.orchestra.run.smithi028.stdout:FAILED - RETRYING: start firewalld (5 retries left).
2021-10-12T03:35:10.243 INFO:teuthology.orchestra.run.smithi028.stdout:FAILED - RETRYING: start firewalld (5 retries left).
2021-10-12T03:35:13.499 INFO:teuthology.orchestra.run.smithi028.stdout:FAILED - RETRYING: start firewalld (4 retries left).
2021-10-12T03:35:13.517 INFO:teuthology.orchestra.run.smithi028.stdout:FAILED - RETRYING: start firewalld (4 retries left).
2021-10-12T03:35:13.526 INFO:teuthology.orchestra.run.smithi028.stdout:FAILED - RETRYING: start firewalld (4 retries left).
2021-10-12T03:35:16.769 INFO:teuthology.orchestra.run.smithi028.stdout:FAILED - RETRYING: start firewalld (3 retries left).
2021-10-12T03:35:16.800 INFO:teuthology.orchestra.run.smithi028.stdout:FAILED - RETRYING: start firewalld (3 retries left).
2021-10-12T03:35:16.803 INFO:teuthology.orchestra.run.smithi028.stdout:FAILED - RETRYING: start firewalld (3 retries left).
2021-10-12T03:35:20.036 INFO:teuthology.orchestra.run.smithi028.stdout:FAILED - RETRYING: start firewalld (2 retries left).
2021-10-12T03:35:20.084 INFO:teuthology.orchestra.run.smithi028.stdout:FAILED - RETRYING: start firewalld (2 retries left).
2021-10-12T03:35:20.086 INFO:teuthology.orchestra.run.smithi028.stdout:FAILED - RETRYING: start firewalld (2 retries left).
2021-10-12T03:35:23.305 INFO:teuthology.orchestra.run.smithi028.stdout:FAILED - RETRYING: start firewalld (1 retries left).
2021-10-12T03:35:23.364 INFO:teuthology.orchestra.run.smithi028.stdout:FAILED - RETRYING: start firewalld (1 retries left).
2021-10-12T03:35:23.366 INFO:teuthology.orchestra.run.smithi028.stdout:FAILED - RETRYING: start firewalld (1 retries left).
2021-10-12T03:35:26.576 INFO:teuthology.orchestra.run.smithi028.stdout:fatal: [smithi170.front.sepia.ceph.com]: FAILED! => {
2021-10-12T03:35:26.577 INFO:teuthology.orchestra.run.smithi028.stdout:    "attempts": 5,
2021-10-12T03:35:26.577 INFO:teuthology.orchestra.run.smithi028.stdout:    "changed": false
2021-10-12T03:35:26.577 INFO:teuthology.orchestra.run.smithi028.stdout:}
2021-10-12T03:35:26.577 INFO:teuthology.orchestra.run.smithi028.stdout:
2021-10-12T03:35:26.578 INFO:teuthology.orchestra.run.smithi028.stdout:MSG:
2021-10-12T03:35:26.578 INFO:teuthology.orchestra.run.smithi028.stdout:
2021-10-12T03:35:26.578 INFO:teuthology.orchestra.run.smithi028.stdout:Could not find the requested service firewalld: host
2021-10-12T03:35:26.641 INFO:teuthology.orchestra.run.smithi028.stdout:fatal: [smithi028.front.sepia.ceph.com]: FAILED! => {
2021-10-12T03:35:26.642 INFO:teuthology.orchestra.run.smithi028.stdout:    "attempts": 5,
2021-10-12T03:35:26.642 INFO:teuthology.orchestra.run.smithi028.stdout:    "changed": false
2021-10-12T03:35:26.642 INFO:teuthology.orchestra.run.smithi028.stdout:}
2021-10-12T03:35:26.642 INFO:teuthology.orchestra.run.smithi028.stdout:
2021-10-12T03:35:26.643 INFO:teuthology.orchestra.run.smithi028.stdout:MSG:
2021-10-12T03:35:26.643 INFO:teuthology.orchestra.run.smithi028.stdout:
2021-10-12T03:35:26.643 INFO:teuthology.orchestra.run.smithi028.stdout:Could not find the requested service firewalld: host
2021-10-12T03:35:26.645 INFO:teuthology.orchestra.run.smithi028.stdout:fatal: [smithi029.front.sepia.ceph.com]: FAILED! => {
2021-10-12T03:35:26.645 INFO:teuthology.orchestra.run.smithi028.stdout:    "attempts": 5,
2021-10-12T03:35:26.646 INFO:teuthology.orchestra.run.smithi028.stdout:    "changed": false
2021-10-12T03:35:26.646 INFO:teuthology.orchestra.run.smithi028.stdout:}
2021-10-12T03:35:26.646 INFO:teuthology.orchestra.run.smithi028.stdout:
2021-10-12T03:35:26.646 INFO:teuthology.orchestra.run.smithi028.stdout:MSG:
2021-10-12T03:35:26.647 INFO:teuthology.orchestra.run.smithi028.stdout:
2021-10-12T03:35:26.647 INFO:teuthology.orchestra.run.smithi028.stdout:Could not find the requested service firewalld: host

Finally tracked this to https://github.com/ansible/ansible/issues/48786#issuecomment-709408800 and sure enough upgrading systemd by hand resolves the issue.

Current version is systemd-239-40.el8.x86_64 and systemd-239-45.el8.x86_64 or above should work.

I hope this is the right place to ask for the systemd version on the centos8 stream image to be updated but, if not, please advise or feel free to move it.

Actions
Copy link
 #1

Updated by adam kraitman over 2 years ago

  • Status changed from New to In Progress
  • Assignee set to adam kraitman
Actions
Copy link
 #2

Updated by adam kraitman over 2 years ago

Hey Brad please try now

Actions
Copy link
 #3

Updated by Brad Hubbard over 2 years ago

Hi Adam,

If I do the following I still see 239-40.el8.

$ teuthology-lock --lock-many 1 --machine-type smithi --os-version 8.3 --os-type centos
...
2021-10-21 00:51:21,283.283 INFO:teuthology.orchestra.run.smithi153.stdout:x86_64
2021-10-21 00:51:21,338.338 INFO:teuthology.orchestra.run.smithi153.stdout:NAME="CentOS Linux" 
2021-10-21 00:51:21,338.338 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION="8" 
2021-10-21 00:51:21,338.338 INFO:teuthology.orchestra.run.smithi153.stdout:ID="centos" 
2021-10-21 00:51:21,338.338 INFO:teuthology.orchestra.run.smithi153.stdout:ID_LIKE="rhel fedora" 
2021-10-21 00:51:21,338.338 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION_ID="8" 
2021-10-21 00:51:21,338.338 INFO:teuthology.orchestra.run.smithi153.stdout:PLATFORM_ID="platform:el8" 
2021-10-21 00:51:21,338.338 INFO:teuthology.orchestra.run.smithi153.stdout:PRETTY_NAME="CentOS Linux 8" 
2021-10-21 00:51:21,338.338 INFO:teuthology.orchestra.run.smithi153.stdout:ANSI_COLOR="0;31" 
2021-10-21 00:51:21,338.338 INFO:teuthology.orchestra.run.smithi153.stdout:CPE_NAME="cpe:/o:centos:centos:8" 
2021-10-21 00:51:21,338.338 INFO:teuthology.orchestra.run.smithi153.stdout:HOME_URL="https://centos.org/" 
2021-10-21 00:51:21,338.338 INFO:teuthology.orchestra.run.smithi153.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 
2021-10-21 00:51:21,339.339 INFO:teuthology.orchestra.run.smithi153.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 
2021-10-21 00:51:21,339.339 INFO:teuthology.orchestra.run.smithi153.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 
...

$ ssh smithi153 rpm -q systemd
systemd-239-40.el8.x86_64

I'm trying to do the same with a gibba machine to see if the image is the same but I can't seem to lock one of those machines at the moment.

Which image(s) did you change Adam?

Actions
Copy link
 #4

Updated by adam kraitman over 2 years ago

Hey Brad,
I added the change to centos 8.x image
PR - https://github.com/ceph/ceph-cm-ansible/pull/655

Actions
Copy link
 #5

Updated by Brad Hubbard over 2 years ago

Hi Adam,

How would I specify I want to use the image you changed?

Actions
Copy link
 #6

Updated by David Galloway over 2 years ago

ceph-cm-ansible still needs to run inside the teuthology job.

Or we can manually capture new FOG images.

Actions
Copy link
 #7

Updated by Brad Hubbard over 2 years ago

  • Status changed from In Progress to Fix Under Review

Never mind guys, I've resolved this by changing the test to use the 8.4 image.

Hopefully no one else hits this same issue with the 8.3 image.

https://github.com/ceph/ceph/pull/43658

Actions
Copy link
 #8

Updated by Brad Hubbard over 2 years ago

  • Assignee changed from adam kraitman to Brad Hubbard
Actions
Copy link
 #9

Updated by Yuri Weinstein over 2 years ago

Brad Hubbard wrote:

Never mind guys, I've resolved this by changing the test to use the 8.4 image.

Hopefully no one else hits this same issue with the 8.3 image.

https://github.com/ceph/ceph/pull/43658

merged

Actions
Copy link
 #10

Updated by Brad Hubbard over 2 years ago

  • Status changed from Fix Under Review to Resolved
Actions
Copy link

Also available in: Atom PDF