Bug #54992
pacific: rados/dashboard: tasks/dashboard: cannot stat '/etc/containers/registries.conf': No such file or directory
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Tags:
Description
/a/yuriw-2022-03-17_14:54:32-rados-wip-yuri10-testing-2022-03-16-1432-pacific-distro-default-smithi/6742377
2022-03-17T17:06:57.876 DEBUG:teuthology.orchestra.run.smithi040:> TESTDIR=/home/ubuntu/cephtest bash -s
2022-03-17T17:06:57.877 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi125.front.sepia.ceph.com
2022-03-17T17:06:57.877 DEBUG:teuthology.orchestra.run.smithi125:> TESTDIR=/home/ubuntu/cephtest bash -s
2022-03-17T17:06:57.918 INFO:teuthology.orchestra.run.smithi125.stderr:cp: cannot stat '/etc/containers/registries.conf': No such file or directory
2022-03-17T17:06:57.919 DEBUG:teuthology.orchestra.run:got remote process result: 1
2022-03-17T17:06:57.920 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_a09009223dc44ca923b2e6ad4cd4c4fe88381b22/teuthology/run_tasks.py", line 91, in run_tasks
manager = run_one_task(taskname, ctx=ctx, config=config)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_a09009223dc44ca923b2e6ad4cd4c4fe88381b22/teuthology/run_tasks.py", line 70, in run_one_task
return task(**kwargs)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_a09009223dc44ca923b2e6ad4cd4c4fe88381b22/teuthology/task/pexec.py", line 149, in task
p.spawn(_exec_host, barrier, barrier_queue, remote[0], sudo, testdir, remote[1])
File "/home/teuthworker/src/git.ceph.com_git_teuthology_a09009223dc44ca923b2e6ad4cd4c4fe88381b22/teuthology/parallel.py", line 84, in __exit__
for result in self:
File "/home/teuthworker/src/git.ceph.com_git_teuthology_a09009223dc44ca923b2e6ad4cd4c4fe88381b22/teuthology/parallel.py", line 98, in __next__
resurrect_traceback(result)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_a09009223dc44ca923b2e6ad4cd4c4fe88381b22/teuthology/parallel.py", line 30, in resurrect_traceback
raise exc.exc_info[1]
File "/home/teuthworker/src/git.ceph.com_git_teuthology_a09009223dc44ca923b2e6ad4cd4c4fe88381b22/teuthology/parallel.py", line 23, in capture_traceback
return func(*args, **kwargs)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_a09009223dc44ca923b2e6ad4cd4c4fe88381b22/teuthology/task/pexec.py", line 60, in _exec_host
tor.wait([r])
File "/home/teuthworker/src/git.ceph.com_git_teuthology_a09009223dc44ca923b2e6ad4cd4c4fe88381b22/teuthology/orchestra/run.py", line 479, in wait
proc.wait()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_a09009223dc44ca923b2e6ad4cd4c4fe88381b22/teuthology/orchestra/run.py", line 161, in wait
self._raise_for_status()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_a09009223dc44ca923b2e6ad4cd4c4fe88381b22/teuthology/orchestra/run.py", line 183, in _raise_for_status
node=self.hostname, label=self.label
teuthology.exceptions.CommandFailedError: Command failed on smithi125 with status 1: 'TESTDIR=/home/ubuntu/cephtest bash -s'
2022-03-17T17:06:58.017 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=9636d9d49b2b403ca15242337274d90f
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_a09009223dc44ca923b2e6ad4cd4c4fe88381b22/teuthology/run_tasks.py", line 91, in run_tasks
manager = run_one_task(taskname, ctx=ctx, config=config)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_a09009223dc44ca923b2e6ad4cd4c4fe88381b22/teuthology/run_tasks.py", line 70, in run_one_task
return task(**kwargs)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_a09009223dc44ca923b2e6ad4cd4c4fe88381b22/teuthology/task/pexec.py", line 149, in task
p.spawn(_exec_host, barrier, barrier_queue, remote[0], sudo, testdir, remote[1])
File "/home/teuthworker/src/git.ceph.com_git_teuthology_a09009223dc44ca923b2e6ad4cd4c4fe88381b22/teuthology/parallel.py", line 84, in __exit__
for result in self:
File "/home/teuthworker/src/git.ceph.com_git_teuthology_a09009223dc44ca923b2e6ad4cd4c4fe88381b22/teuthology/parallel.py", line 98, in __next__
resurrect_traceback(result)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_a09009223dc44ca923b2e6ad4cd4c4fe88381b22/teuthology/parallel.py", line 30, in resurrect_traceback
raise exc.exc_info[1]
File "/home/teuthworker/src/git.ceph.com_git_teuthology_a09009223dc44ca923b2e6ad4cd4c4fe88381b22/teuthology/parallel.py", line 23, in capture_traceback
return func(*args, **kwargs)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_a09009223dc44ca923b2e6ad4cd4c4fe88381b22/teuthology/task/pexec.py", line 60, in _exec_host
tor.wait([r])
File "/home/teuthworker/src/git.ceph.com_git_teuthology_a09009223dc44ca923b2e6ad4cd4c4fe88381b22/teuthology/orchestra/run.py", line 479, in wait
proc.wait()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_a09009223dc44ca923b2e6ad4cd4c4fe88381b22/teuthology/orchestra/run.py", line 161, in wait
self._raise_for_status()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_a09009223dc44ca923b2e6ad4cd4c4fe88381b22/teuthology/orchestra/run.py", line 183, in _raise_for_status
node=self.hostname, label=self.label
teuthology.exceptions.CommandFailedError: Command failed on smithi125 with status 1: 'TESTDIR=/home/ubuntu/cephtest bash -s'
Related issues
History
#1 Updated by Kamoltat (Junior) Sirivadhna over 1 year ago
/a/yuriw-2022-03-26_19:49:47-rados-wip-yuri10-testing-2022-03-22-1809-pacific-distro-default-smithi/6762737
#2 Updated by Laura Flores over 1 year ago
/a/yuriw-2022-05-04_20:09:21-rados-pacific-distro-default-smithi/6821407
#3 Updated by Laura Flores over 1 year ago
- Duplicated by Bug #54086: Permission denied when trying to unlink and open /var/log/ntpstats/... added
#4 Updated by Laura Flores over 1 year ago
/a/yuriw-2022-06-02_00:50:42-rados-wip-yuri4-testing-2022-06-01-1350-pacific-distro-default-smithi/6859624
#5 Updated by Matan Breizman about 1 year ago
/a/yuriw-2022-08-22_21:19:34-rados-wip-yuri4-testing-2022-08-18-1020-pacific-distro-default-smithi/6986466
#6 Updated by Laura Flores about 1 year ago
- Tags set to test-failure
- Priority changed from Normal to High
Marking this as high since it appears in the rados suite a lot.
/a/amathuri-2022-09-08_14:19:23-rados-wip-yuri5-testing-2022-09-06-1334-pacific-distro-default-smithi/7020554
#7 Updated by Adam King about 1 year ago
- Project changed from Orchestrator to Dashboard
- Subject changed from cannot stat '/etc/containers/registries.conf': No such file or directory to pacific: rados/dashboard: tasks/dashboard: cannot stat '/etc/containers/registries.conf': No such file or directory
moving this to dashboard component given the failure only exists in a dashboard specific test. My guess is this is an issue with something the test is doing rather than an actual bug in Ceph or the dashboard, but maybe someone who knows the dashboard tests a bit better can investigate. It also seems to only happen in pacific if that info is useful.
#8 Updated by Aishwarya Mathuria 10 months ago
/a/yuriw-2022-11-29_22:29:58-rados-wip-yuri10-testing-2022-11-29-1005-pacific-distro-default-smithi/7097412
#9 Updated by Laura Flores 10 months ago
This bug occurs on Pacific whenever the test is run on ubuntu.
There are clear messages about ubuntu not supporting Selinux in the log:
2022-11-30T01:15:07.924 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}
2022-11-30T01:15:07.925 DEBUG:teuthology.task.selinux:Excluding smithi143: OS 'ubuntu' does not support SELinux
2022-11-30T01:15:07.925 DEBUG:teuthology.task.selinux:Excluding smithi175: OS 'ubuntu' does not support SELinux
Which leads to:
2022-11-30T01:19:48.277 INFO:teuthology.orchestra.run.smithi175.stdout:30 Nov 01:19:48 ntpd[17296]: Listening on routing socket on fd #22 for interface updates
2022-11-30T01:19:49.259 INFO:teuthology.orchestra.run.smithi143.stderr:30 Nov 01:19:49 ntpd[17328]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied
2022-11-30T01:19:49.260 INFO:teuthology.orchestra.run.smithi143.stderr:30 Nov 01:19:49 ntpd[17328]: can't open /var/log/ntpstats/rawstats.20221130: Permission denied
2022-11-30T01:19:49.260 INFO:teuthology.orchestra.run.smithi143.stderr:30 Nov 01:19:49 ntpd[17328]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied
2022-11-30T01:19:49.260 INFO:teuthology.orchestra.run.smithi143.stderr:30 Nov 01:19:49 ntpd[17328]: can't open /var/log/ntpstats/rawstats.20221130: Permission denied
2022-11-30T01:19:49.261 INFO:teuthology.orchestra.run.smithi143.stderr:30 Nov 01:19:49 ntpd[17328]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied
2022-11-30T01:19:49.261 INFO:teuthology.orchestra.run.smithi143.stderr:30 Nov 01:19:49 ntpd[17328]: can't open /var/log/ntpstats/peerstats.20221130: Permission denied
2022-11-30T01:19:50.262 INFO:teuthology.orchestra.run.smithi175.stderr:30 Nov 01:19:50 ntpd[17296]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied
2022-11-30T01:19:50.262 INFO:teuthology.orchestra.run.smithi175.stderr:30 Nov 01:19:50 ntpd[17296]: can't open /var/log/ntpstats/rawstats.20221130: Permission denied
2022-11-30T01:19:50.263 INFO:teuthology.orchestra.run.smithi175.stderr:30 Nov 01:19:50 ntpd[17296]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied
2022-11-30T01:19:50.263 INFO:teuthology.orchestra.run.smithi175.stderr:30 Nov 01:19:50 ntpd[17296]: can't open /var/log/ntpstats/rawstats.20221130: Permission denied
2022-11-30T01:19:50.264 INFO:teuthology.orchestra.run.smithi175.stderr:30 Nov 01:19:50 ntpd[17296]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied
2022-11-30T01:19:50.264 INFO:teuthology.orchestra.run.smithi175.stderr:30 Nov 01:19:50 ntpd[17296]: can't open /var/log/ntpstats/rawstats.20221130: Permission denied
2022-11-30T01:19:50.265 INFO:teuthology.orchestra.run.smithi175.stderr:30 Nov 01:19:50 ntpd[17296]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied
2022-11-30T01:19:50.265 INFO:teuthology.orchestra.run.smithi175.stderr:30 Nov 01:19:50 ntpd[17296]: can't open /var/log/ntpstats/rawstats.20221130: Permission denied
2022-11-30T01:19:50.265 INFO:teuthology.orchestra.run.smithi175.stderr:30 Nov 01:19:50 ntpd[17296]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied
2022-11-30T01:19:50.265 INFO:teuthology.orchestra.run.smithi175.stderr:30 Nov 01:19:50 ntpd[17296]: can't open /var/log/ntpstats/peerstats.20221130: Permission denied
And eventually..
2022-11-30T01:19:56.318 INFO:teuthology.run_tasks:Running task pexec...
2022-11-30T01:19:56.328 INFO:teuthology.task.pexec:Executing custom commands...
2022-11-30T01:19:56.329 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi143.front.sepia.ceph.com
2022-11-30T01:19:56.329 DEBUG:teuthology.orchestra.run.smithi143:> TESTDIR=/home/ubuntu/cephtest bash -s
2022-11-30T01:19:56.331 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi175.front.sepia.ceph.com
2022-11-30T01:19:56.331 DEBUG:teuthology.orchestra.run.smithi175:> TESTDIR=/home/ubuntu/cephtest bash -s
2022-11-30T01:19:56.349 INFO:teuthology.orchestra.run.smithi143.stderr:cp: cannot stat '/etc/containers/registries.conf': No such file or directory
As we've established, this bug only seems to occur on Pacific. The difference between tasks/dashboard tests on main/quincy vs. pacific is that we don't symlink to "supported-random-distros$" in main/quincy. There must be a commit somewhere where we dropped support for ubuntu that just needs to be backported.
tasks/dashboard for main: https://github.com/ceph/ceph/tree/main/qa/suites/rados/dashboard
tasks/dashboard for quincy: https://github.com/ceph/ceph/tree/quincy/qa/suites/rados/dashboard
tasks/dashboard for pacific: https://github.com/ceph/ceph/tree/pacific/qa/suites/rados/dashboard
#10 Updated by Laura Flores 10 months ago
These two commits pinned dashboard e2e test on centos only:
https://github.com/ceph/ceph/pull/42084
https://github.com/ceph/ceph/pull/42158
Both are from Kefu, so I may ask his opinion on this to make sure we are backporting the right things.
#11 Updated by Laura Flores 10 months ago
- Status changed from New to Fix Under Review
- Pull request ID set to 49337
I think all that's needed is to remove the random distros facet, so I'm proposing backporting that commit. But I will ask for Kefu's opinion.
#12 Updated by Laura Flores 10 months ago
- Assignee set to Laura Flores
#13 Updated by Laura Flores 9 months ago
- Status changed from Fix Under Review to Resolved