Project

General

Profile

Bug #52923

centos8 stream systemd version causes ceph-ansible tests to fail

Added by Brad Hubbard about 2 months ago. Updated 21 days ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
Test Node
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):

Description

http://qa-proxy.ceph.com/teuthology/bhubbard-2021-10-12_03:15:53-ceph-ansible-wip-octopus-badone-testing-distro-basic-smithi/6432186/teuthology.log

2021-10-12T03:35:09.709 INFO:teuthology.orchestra.run.smithi028.stdout:TASK [ceph-infra : start firewalld] ********************************************
2021-10-12T03:35:09.709 INFO:teuthology.orchestra.run.smithi028.stdout:task path: /home/ubuntu/ceph-ansible/roles/ceph-infra/tasks/configure_firewall.yml:22
2021-10-12T03:35:09.709 INFO:teuthology.orchestra.run.smithi028.stdout:Tuesday 12 October 2021  03:35:09 +0000 (0:00:05.029)       0:00:20.562 *******
2021-10-12T03:35:10.232 INFO:teuthology.orchestra.run.smithi028.stdout:FAILED - RETRYING: start firewalld (5 retries left).
2021-10-12T03:35:10.239 INFO:teuthology.orchestra.run.smithi028.stdout:FAILED - RETRYING: start firewalld (5 retries left).
2021-10-12T03:35:10.243 INFO:teuthology.orchestra.run.smithi028.stdout:FAILED - RETRYING: start firewalld (5 retries left).
2021-10-12T03:35:13.499 INFO:teuthology.orchestra.run.smithi028.stdout:FAILED - RETRYING: start firewalld (4 retries left).
2021-10-12T03:35:13.517 INFO:teuthology.orchestra.run.smithi028.stdout:FAILED - RETRYING: start firewalld (4 retries left).
2021-10-12T03:35:13.526 INFO:teuthology.orchestra.run.smithi028.stdout:FAILED - RETRYING: start firewalld (4 retries left).
2021-10-12T03:35:16.769 INFO:teuthology.orchestra.run.smithi028.stdout:FAILED - RETRYING: start firewalld (3 retries left).
2021-10-12T03:35:16.800 INFO:teuthology.orchestra.run.smithi028.stdout:FAILED - RETRYING: start firewalld (3 retries left).
2021-10-12T03:35:16.803 INFO:teuthology.orchestra.run.smithi028.stdout:FAILED - RETRYING: start firewalld (3 retries left).
2021-10-12T03:35:20.036 INFO:teuthology.orchestra.run.smithi028.stdout:FAILED - RETRYING: start firewalld (2 retries left).
2021-10-12T03:35:20.084 INFO:teuthology.orchestra.run.smithi028.stdout:FAILED - RETRYING: start firewalld (2 retries left).
2021-10-12T03:35:20.086 INFO:teuthology.orchestra.run.smithi028.stdout:FAILED - RETRYING: start firewalld (2 retries left).
2021-10-12T03:35:23.305 INFO:teuthology.orchestra.run.smithi028.stdout:FAILED - RETRYING: start firewalld (1 retries left).
2021-10-12T03:35:23.364 INFO:teuthology.orchestra.run.smithi028.stdout:FAILED - RETRYING: start firewalld (1 retries left).
2021-10-12T03:35:23.366 INFO:teuthology.orchestra.run.smithi028.stdout:FAILED - RETRYING: start firewalld (1 retries left).
2021-10-12T03:35:26.576 INFO:teuthology.orchestra.run.smithi028.stdout:fatal: [smithi170.front.sepia.ceph.com]: FAILED! => {
2021-10-12T03:35:26.577 INFO:teuthology.orchestra.run.smithi028.stdout:    "attempts": 5,
2021-10-12T03:35:26.577 INFO:teuthology.orchestra.run.smithi028.stdout:    "changed": false
2021-10-12T03:35:26.577 INFO:teuthology.orchestra.run.smithi028.stdout:}
2021-10-12T03:35:26.577 INFO:teuthology.orchestra.run.smithi028.stdout:
2021-10-12T03:35:26.578 INFO:teuthology.orchestra.run.smithi028.stdout:MSG:
2021-10-12T03:35:26.578 INFO:teuthology.orchestra.run.smithi028.stdout:
2021-10-12T03:35:26.578 INFO:teuthology.orchestra.run.smithi028.stdout:Could not find the requested service firewalld: host
2021-10-12T03:35:26.641 INFO:teuthology.orchestra.run.smithi028.stdout:fatal: [smithi028.front.sepia.ceph.com]: FAILED! => {
2021-10-12T03:35:26.642 INFO:teuthology.orchestra.run.smithi028.stdout:    "attempts": 5,
2021-10-12T03:35:26.642 INFO:teuthology.orchestra.run.smithi028.stdout:    "changed": false
2021-10-12T03:35:26.642 INFO:teuthology.orchestra.run.smithi028.stdout:}
2021-10-12T03:35:26.642 INFO:teuthology.orchestra.run.smithi028.stdout:
2021-10-12T03:35:26.643 INFO:teuthology.orchestra.run.smithi028.stdout:MSG:
2021-10-12T03:35:26.643 INFO:teuthology.orchestra.run.smithi028.stdout:
2021-10-12T03:35:26.643 INFO:teuthology.orchestra.run.smithi028.stdout:Could not find the requested service firewalld: host
2021-10-12T03:35:26.645 INFO:teuthology.orchestra.run.smithi028.stdout:fatal: [smithi029.front.sepia.ceph.com]: FAILED! => {
2021-10-12T03:35:26.645 INFO:teuthology.orchestra.run.smithi028.stdout:    "attempts": 5,
2021-10-12T03:35:26.646 INFO:teuthology.orchestra.run.smithi028.stdout:    "changed": false
2021-10-12T03:35:26.646 INFO:teuthology.orchestra.run.smithi028.stdout:}
2021-10-12T03:35:26.646 INFO:teuthology.orchestra.run.smithi028.stdout:
2021-10-12T03:35:26.646 INFO:teuthology.orchestra.run.smithi028.stdout:MSG:
2021-10-12T03:35:26.647 INFO:teuthology.orchestra.run.smithi028.stdout:
2021-10-12T03:35:26.647 INFO:teuthology.orchestra.run.smithi028.stdout:Could not find the requested service firewalld: host

Finally tracked this to https://github.com/ansible/ansible/issues/48786#issuecomment-709408800 and sure enough upgrading systemd by hand resolves the issue.

Current version is systemd-239-40.el8.x86_64 and systemd-239-45.el8.x86_64 or above should work.

I hope this is the right place to ask for the systemd version on the centos8 stream image to be updated but, if not, please advise or feel free to move it.

History

#1 Updated by adam kraitman about 1 month ago

  • Status changed from New to In Progress
  • Assignee set to adam kraitman

#2 Updated by adam kraitman about 1 month ago

Hey Brad please try now

#3 Updated by Brad Hubbard about 1 month ago

Hi Adam,

If I do the following I still see 239-40.el8.

$ teuthology-lock --lock-many 1 --machine-type smithi --os-version 8.3 --os-type centos
...
2021-10-21 00:51:21,283.283 INFO:teuthology.orchestra.run.smithi153.stdout:x86_64
2021-10-21 00:51:21,338.338 INFO:teuthology.orchestra.run.smithi153.stdout:NAME="CentOS Linux" 
2021-10-21 00:51:21,338.338 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION="8" 
2021-10-21 00:51:21,338.338 INFO:teuthology.orchestra.run.smithi153.stdout:ID="centos" 
2021-10-21 00:51:21,338.338 INFO:teuthology.orchestra.run.smithi153.stdout:ID_LIKE="rhel fedora" 
2021-10-21 00:51:21,338.338 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION_ID="8" 
2021-10-21 00:51:21,338.338 INFO:teuthology.orchestra.run.smithi153.stdout:PLATFORM_ID="platform:el8" 
2021-10-21 00:51:21,338.338 INFO:teuthology.orchestra.run.smithi153.stdout:PRETTY_NAME="CentOS Linux 8" 
2021-10-21 00:51:21,338.338 INFO:teuthology.orchestra.run.smithi153.stdout:ANSI_COLOR="0;31" 
2021-10-21 00:51:21,338.338 INFO:teuthology.orchestra.run.smithi153.stdout:CPE_NAME="cpe:/o:centos:centos:8" 
2021-10-21 00:51:21,338.338 INFO:teuthology.orchestra.run.smithi153.stdout:HOME_URL="https://centos.org/" 
2021-10-21 00:51:21,338.338 INFO:teuthology.orchestra.run.smithi153.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 
2021-10-21 00:51:21,339.339 INFO:teuthology.orchestra.run.smithi153.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 
2021-10-21 00:51:21,339.339 INFO:teuthology.orchestra.run.smithi153.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 
...

$ ssh smithi153 rpm -q systemd
systemd-239-40.el8.x86_64

I'm trying to do the same with a gibba machine to see if the image is the same but I can't seem to lock one of those machines at the moment.

Which image(s) did you change Adam?

#4 Updated by adam kraitman about 1 month ago

Hey Brad,
I added the change to centos 8.x image
PR - https://github.com/ceph/ceph-cm-ansible/pull/655

#5 Updated by Brad Hubbard about 1 month ago

Hi Adam,

How would I specify I want to use the image you changed?

#6 Updated by David Galloway about 1 month ago

ceph-cm-ansible still needs to run inside the teuthology job.

Or we can manually capture new FOG images.

#7 Updated by Brad Hubbard about 1 month ago

  • Status changed from In Progress to Fix Under Review

Never mind guys, I've resolved this by changing the test to use the 8.4 image.

Hopefully no one else hits this same issue with the 8.3 image.

https://github.com/ceph/ceph/pull/43658

#8 Updated by Brad Hubbard about 1 month ago

  • Assignee changed from adam kraitman to Brad Hubbard

#9 Updated by Yuri Weinstein 21 days ago

Brad Hubbard wrote:

Never mind guys, I've resolved this by changing the test to use the 8.4 image.

Hopefully no one else hits this same issue with the 8.3 image.

https://github.com/ceph/ceph/pull/43658

merged

#10 Updated by Brad Hubbard 21 days ago

  • Status changed from Fix Under Review to Resolved

Also available in: Atom PDF