Bug #49771
ceph ansible fails on rhel 7+
0%
Description
Failure object was: {'smithi104.front.sepia.ceph.com': {'changed': False, 'results': ['yum-plugin-priorities-1.1.31-52.el7.noarch Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b7dbd64ff8500a97d0659a105ea1a1f415ec323b/virtualenv/lib/python3.6/site-p return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b7dbd64ff8500a97d0659a105ea1a1f415ec323b/virtualenv/lib/python3.6/site-p dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b7dbd64ff8500a97d0659a105ea1a1f415ec323b/virtualenv/lib/python3.6/site-p node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b7dbd64ff8500a97d0659a105ea1a1f415ec323b/virtualenv/lib/python3.6/site-p node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b7dbd64ff8500a97d0659a105ea1a1f415ec323b/virtualenv/lib/python3.6/site-p return self.represent_mapping('tag:yaml.org,2002:map', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b7dbd64ff8500a97d0659a105ea1a1f415ec323b/virtualenv/lib/python3.6/site-p node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b7dbd64ff8500a97d0659a105ea1a1f415ec323b/virtualenv/lib/python3.6/site-p node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b7dbd64ff8500a97d0659a105ea1a1f415ec323b/virtualenv/lib/python3.6/site-p return self.represent_mapping('tag:yaml.org,2002:map', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b7dbd64ff8500a97d0659a105ea1a1f415ec323b/virtualenv/lib/python3.6/site-p node_key = self.represent_data(item_key) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b7dbd64ff8500a97d0659a105ea1a1f415ec323b/virtualenv/lib/python3.6/site-p node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b7dbd64ff8500a97d0659a105ea1a1f415ec323b/virtualenv/lib/python3.6/site-p raise RepresenterError("cannot represent an object", data) yaml.representer.RepresenterError: ('cannot represent an object', 'changed')
/ceph/teuthology-archive/yuriw-2021-03-11_19:01:40-rados-octopus-distro-basic-smithi/5956609/teuthology.log
Related issues
History
#1 Updated by Deepika Upadhyay about 3 years ago
- Project changed from RADOS to Infrastructure
#2 Updated by Deepika Upadhyay about 3 years ago
root cause seems to be targeted for rhel 7.9:
"No package matching 'qemu-kvm' found available, installed or updated"],
see sentry events: https://sentry.ceph.com/organizations/ceph/issues/36/events/9a28868d3b8a4e19a3c12f89ae152bfc/events/?project=2
#3 Updated by Deepika Upadhyay about 3 years ago
- Subject changed from ceph ansible fails on rhel 7.7 to ceph ansible fails on rhel 7+
#4 Updated by Deepika Upadhyay about 3 years ago
- Related to Bug #49865: yaml.representer.RepresenterError: cannot represent an object added
#5 Updated by David Galloway about 3 years ago
- Assignee set to David Galloway
So far I've determined qemu-kvm should be in the RHEL7.9 image. I think something is wrong with the repo on the Satellite server.
[root@gibba032 ~]# yum whatprovides /usr/libexec/qemu-kvm Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-disabled-repos, subscription-manager Loading mirror speeds from cached hostfile * epel: download-cc-rdu01.fedoraproject.org epel | 4.7 kB 00:00:00 lab-extras | 2.9 kB 00:00:00 rhel-7-fcgi-ceph | 951 B 00:00:00 rhel-7-server-extras-rpms | 2.0 kB 00:00:00 rhel-7-server-optional-rpms | 2.0 kB 00:00:00 rhel-7-server-rpms | 2.0 kB 00:00:00 rhel-ha-for-rhel-7-server-rpms | 2.0 kB 00:00:00 One of the configured repositories failed (Unknown), and yum doesn't have enough cached data to continue. At this point the only safe thing yum can do is fail. There are a few ways to work "fix" this: 1. Contact the upstream for the repository and get them to fix the problem. 2. Reconfigure the baseurl/etc. for the repository, to point to a working upstream. This is most often useful if you are using a newer distribution release than is supported by the repository (and the packages for the previous distribution release still work). 3. Run the command with the repository temporarily disabled yum --disablerepo=<repoid> ... 4. Disable the repository permanently, so yum won't use it by default. Yum will then just ignore the repository until you permanently enable it again or use --enablerepo for temporary usage: yum-config-manager --disable <repoid> or subscription-manager repos --disable=<repoid> 5. Configure the failing repository to be skipped, if it is unavailable. Note that yum will try to contact the repo. when it runs most commands, so will have to try and fail each time (and thus. yum will be be much slower). If it is a very temporary problem though, this is often a nice compromise: yum-config-manager --save --setopt=<repoid>.skip_if_unavailable=true pkgKey 476 doesn't exist in repo rhel-7-server-rpms
#6 Updated by Deepika Upadhyay about 3 years ago
/ceph/teuthology-archive/yuriw-2021-03-18_20:45:33-rados-wip-yuri3-testing-2021-03-18-1149-octopus-distro-basic-smithi/5978020/teuthology.log"
#7 Updated by Deepika Upadhyay about 3 years ago
just for indexing, looks like another variation of this failure:
os_type: rhel os_version: '7.7'
2021-03-23T19:13:58.334 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/run_tasks.py", line 94, in run_tasks manager.__enter__() File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/task/internal/syslog.py", line 50, in syslog rem.chcon(log_path, log_context) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 318, in chcon if teuthology.lock.query.is_vm(self.shortname): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/lock/query.py", line 48, in is_vm return status.get('is_vm', False) AttributeError: 'NoneType' object has no attribute 'get'
#8 Updated by Deepika Upadhyay almost 3 years ago
seems like another related consequence, adding for debugging purpose:
description: rados/basic/{ceph clusters/{fixed-2 openstack} msgr-failures/few msgr/random objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_7} tasks/rados_cls_all}
2021-03-27T18:22:56.965 INFO:teuthology.task.ansible.out:FAILED - RETRYING: Register with subscription-manager. (2 retries left).^M 2021-03-27T18:22:56.998 INFO:teuthology.task.ansible.out:FAILED - RETRYING: Register with subscription-manager. (2 retries left).^M 2021-03-27T18:23:07.827 INFO:teuthology.task.ansible.out:FAILED - RETRYING: Register with subscription-manager. (1 retries left).^M 2021-03-27T18:23:07.863 INFO:teuthology.task.ansible.out:FAILED - RETRYING: Register with subscription-manager. (1 retries left).^M
2021-03-27T18:23:19.406 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=35662c88a9a946e98893121464676bbb Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_d5a60f86b5bc836fe1569aac8a1c7ce60b0a3cf2/teuthology/run_tasks.py", line 94, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_git_teuthology_d5a60f86b5bc836fe1569aac8a1c7ce60b0a3cf2/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_git_teuthology_d5a60f86b5bc836fe1569aac8a1c7ce60b0a3cf2/teuthology/task/ansible.py", line 421, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_git_teuthology_d5a60f86b5bc836fe1569aac8a1c7ce60b0a3cf2/teuthology/task/ansible.py", line 263, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_git_teuthology_d5a60f86b5bc836fe1569aac8a1c7ce60b0a3cf2/teuthology/task/ansible.py", line 290, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d5a60f86b5bc836fe1569aac8a1c7ce60b0a3cf2/teuthology/task/ansible.py", line 315, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: {'smithi062.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}, 'smithi013.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}}
/ceph/teuthology-archive/yuriw-2021-03-27_15:31:42-rados-wip-yuri-testing-2021-03-26-1233-nautilus-distro-basic-smithi/6004595/teuthology.log
#9 Updated by adam kraitman almost 3 years ago
Fixed by setting the releasevar on rhel machines https://github.com/ceph/ceph-cm-ansible/pull/617
#10 Updated by Deepika Upadhyay almost 3 years ago
- Status changed from New to Resolved