Project

General

Profile

Bug #49771

ceph ansible fails on rhel 7+

Added by Deepika Upadhyay about 3 years ago. Updated almost 3 years ago.

Status:
Resolved
Priority:
Normal
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):

Description

Failure object was: {'smithi104.front.sepia.ceph.com': {'changed': False, 'results': ['yum-plugin-priorities-1.1.31-52.el7.noarch
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/callback_plugins/failure_log.py", line 44, in log_failure
    log.error(yaml.safe_dump(failure))
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_b7dbd64ff8500a97d0659a105ea1a1f415ec323b/virtualenv/lib/python3.6/site-p
    return dump_all([data], stream, Dumper=SafeDumper, **kwds)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_b7dbd64ff8500a97d0659a105ea1a1f415ec323b/virtualenv/lib/python3.6/site-p
    dumper.represent(data)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_b7dbd64ff8500a97d0659a105ea1a1f415ec323b/virtualenv/lib/python3.6/site-p
    node = self.represent_data(data)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_b7dbd64ff8500a97d0659a105ea1a1f415ec323b/virtualenv/lib/python3.6/site-p
    node = self.yaml_representers[data_types[0]](self, data)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_b7dbd64ff8500a97d0659a105ea1a1f415ec323b/virtualenv/lib/python3.6/site-p
    return self.represent_mapping('tag:yaml.org,2002:map', data)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_b7dbd64ff8500a97d0659a105ea1a1f415ec323b/virtualenv/lib/python3.6/site-p
    node_value = self.represent_data(item_value)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_b7dbd64ff8500a97d0659a105ea1a1f415ec323b/virtualenv/lib/python3.6/site-p
    node = self.yaml_representers[data_types[0]](self, data)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_b7dbd64ff8500a97d0659a105ea1a1f415ec323b/virtualenv/lib/python3.6/site-p
    return self.represent_mapping('tag:yaml.org,2002:map', data)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_b7dbd64ff8500a97d0659a105ea1a1f415ec323b/virtualenv/lib/python3.6/site-p
    node_key = self.represent_data(item_key)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_b7dbd64ff8500a97d0659a105ea1a1f415ec323b/virtualenv/lib/python3.6/site-p
    node = self.yaml_representers[None](self, data)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_b7dbd64ff8500a97d0659a105ea1a1f415ec323b/virtualenv/lib/python3.6/site-p
    raise RepresenterError("cannot represent an object", data)
yaml.representer.RepresenterError: ('cannot represent an object', 'changed')

/ceph/teuthology-archive/yuriw-2021-03-11_19:01:40-rados-octopus-distro-basic-smithi/5956609/teuthology.log


Related issues

Related to Infrastructure - Bug #49865: yaml.representer.RepresenterError: cannot represent an object New

History

#1 Updated by Deepika Upadhyay about 3 years ago

  • Project changed from RADOS to Infrastructure

#2 Updated by Deepika Upadhyay about 3 years ago

root cause seems to be targeted for rhel 7.9:

"No package matching 'qemu-kvm' found available, installed or updated"],

see sentry events: https://sentry.ceph.com/organizations/ceph/issues/36/events/9a28868d3b8a4e19a3c12f89ae152bfc/events/?project=2

#3 Updated by Deepika Upadhyay about 3 years ago

  • Subject changed from ceph ansible fails on rhel 7.7 to ceph ansible fails on rhel 7+

#4 Updated by Deepika Upadhyay about 3 years ago

  • Related to Bug #49865: yaml.representer.RepresenterError: cannot represent an object added

#5 Updated by David Galloway about 3 years ago

  • Assignee set to David Galloway

So far I've determined qemu-kvm should be in the RHEL7.9 image. I think something is wrong with the repo on the Satellite server.

[root@gibba032 ~]# yum whatprovides /usr/libexec/qemu-kvm
Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-disabled-repos, subscription-manager
Loading mirror speeds from cached hostfile
 * epel: download-cc-rdu01.fedoraproject.org
epel                                                                                                                                                                                                                                                     | 4.7 kB  00:00:00     
lab-extras                                                                                                                                                                                                                                               | 2.9 kB  00:00:00     
rhel-7-fcgi-ceph                                                                                                                                                                                                                                         |  951 B  00:00:00     
rhel-7-server-extras-rpms                                                                                                                                                                                                                                | 2.0 kB  00:00:00     
rhel-7-server-optional-rpms                                                                                                                                                                                                                              | 2.0 kB  00:00:00     
rhel-7-server-rpms                                                                                                                                                                                                                                       | 2.0 kB  00:00:00     
rhel-ha-for-rhel-7-server-rpms                                                                                                                                                                                                                           | 2.0 kB  00:00:00     

 One of the configured repositories failed (Unknown),
 and yum doesn't have enough cached data to continue. At this point the only
 safe thing yum can do is fail. There are a few ways to work "fix" this:

     1. Contact the upstream for the repository and get them to fix the problem.

     2. Reconfigure the baseurl/etc. for the repository, to point to a working
        upstream. This is most often useful if you are using a newer
        distribution release than is supported by the repository (and the
        packages for the previous distribution release still work).

     3. Run the command with the repository temporarily disabled
            yum --disablerepo=<repoid> ...

     4. Disable the repository permanently, so yum won't use it by default. Yum
        will then just ignore the repository until you permanently enable it
        again or use --enablerepo for temporary usage:

            yum-config-manager --disable <repoid>
        or
            subscription-manager repos --disable=<repoid>

     5. Configure the failing repository to be skipped, if it is unavailable.
        Note that yum will try to contact the repo. when it runs most commands,
        so will have to try and fail each time (and thus. yum will be be much
        slower). If it is a very temporary problem though, this is often a nice
        compromise:

            yum-config-manager --save --setopt=<repoid>.skip_if_unavailable=true

pkgKey 476 doesn't exist in repo rhel-7-server-rpms

#6 Updated by Deepika Upadhyay about 3 years ago

/ceph/teuthology-archive/yuriw-2021-03-18_20:45:33-rados-wip-yuri3-testing-2021-03-18-1149-octopus-distro-basic-smithi/5978020/teuthology.log"

#7 Updated by Deepika Upadhyay about 3 years ago

just for indexing, looks like another variation of this failure:

  os_type: rhel
  os_version: '7.7'

2021-03-23T19:13:58.334 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/run_tasks.py", line 94, in run_tasks
    manager.__enter__()
  File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/task/internal/syslog.py", line 50, in syslog
    rem.chcon(log_path, log_context)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 318, in chcon
    if teuthology.lock.query.is_vm(self.shortname):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/lock/query.py", line 48, in is_vm
    return status.get('is_vm', False)
AttributeError: 'NoneType' object has no attribute 'get'

#8 Updated by Deepika Upadhyay almost 3 years ago

seems like another related consequence, adding for debugging purpose:

  description: rados/basic/{ceph clusters/{fixed-2 openstack} msgr-failures/few msgr/random
    objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_7} tasks/rados_cls_all}


2021-03-27T18:22:56.965 INFO:teuthology.task.ansible.out:FAILED - RETRYING: Register with subscription-manager. (2 retries left).^M

2021-03-27T18:22:56.998 INFO:teuthology.task.ansible.out:FAILED - RETRYING: Register with subscription-manager. (2 retries left).^M

2021-03-27T18:23:07.827 INFO:teuthology.task.ansible.out:FAILED - RETRYING: Register with subscription-manager. (1 retries left).^M

2021-03-27T18:23:07.863 INFO:teuthology.task.ansible.out:FAILED - RETRYING: Register with subscription-manager. (1 retries left).^M
2021-03-27T18:23:19.406 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=35662c88a9a946e98893121464676bbb
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d5a60f86b5bc836fe1569aac8a1c7ce60b0a3cf2/teuthology/run_tasks.py", line 94, in run_tasks
    manager.__enter__()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d5a60f86b5bc836fe1569aac8a1c7ce60b0a3cf2/teuthology/task/__init__.py", line 123, in __enter__
    self.begin()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d5a60f86b5bc836fe1569aac8a1c7ce60b0a3cf2/teuthology/task/ansible.py", line 421, in begin
    super(CephLab, self).begin()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d5a60f86b5bc836fe1569aac8a1c7ce60b0a3cf2/teuthology/task/ansible.py", line 263, in begin
    self.execute_playbook()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d5a60f86b5bc836fe1569aac8a1c7ce60b0a3cf2/teuthology/task/ansible.py", line 290, in execute_playbook
    self._handle_failure(command, status)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d5a60f86b5bc836fe1569aac8a1c7ce60b0a3cf2/teuthology/task/ansible.py", line 315, in _handle_failure
    raise AnsibleFailedError(failures)
teuthology.exceptions.AnsibleFailedError: {'smithi062.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}, 'smithi013.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}}

/ceph/teuthology-archive/yuriw-2021-03-27_15:31:42-rados-wip-yuri-testing-2021-03-26-1233-nautilus-distro-basic-smithi/6004595/teuthology.log

#9 Updated by adam kraitman almost 3 years ago

Fixed by setting the releasevar on rhel machines https://github.com/ceph/ceph-cm-ansible/pull/617

#10 Updated by Deepika Upadhyay almost 3 years ago

  • Status changed from New to Resolved

Also available in: Atom PDF