Actions
Bug #19065
openansible error while scanning a plain scalar in "/tmp/teuth_ansible_failures_TasNV4", line 1, column 103 found unexpected ':
Added by Vasu Kulkarni about 7 years ago. Updated over 5 years ago.
Status:
New
Priority:
Normal
Assignee:
-
Category:
-
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):
Description
some rare runs fail due to following error, I tried to scan the logs but couldn't find anything interesting.
while scanning a plain scalar in "/tmp/teuth_ansible_failures_TasNV4", line 1, column 103 found unexpected ':http://pulpito.ceph.redhat.com/vasu-2017-02-01_13:54:15-smoke-jewel---basic-multi/263379
and I think Sam's run has seen quite a few today:
http://pulpito.ceph.com/samuelj-2017-02-23_06:09:28-rados-wip-19023---basic-smithi/
Updated by David Galloway about 7 years ago
The real error for Vasu's run was:
2017-02-01T15:18:04.264 INFO:teuthology.task.ansible.out:fatal: [clara011.ceph.redhat.com]: FAILED! => {"changed": false, "cmd": ["yum-complete-transaction", "--cleanup-only"], "delta": "0:00:01.366641", "end": "2017-02-01 20:18:04.654618", "failed": true, "rc": 1, "start": "2017-02-01 20:18:03.287977", "stderr": "", "stdout": "Loaded plugins: fastestmirror, langpacks, priorities, product-id\nhttp://download-node-02.eng.bos.redhat.com/rcm-guest/ceph-drops/auto/rhscon-2-rhel-7-compose/latest-RHSCON-2-RHEL-7/compose/Installer/x86_64/os/repodata/repomd.xml: [Errno 14] HTTP Error 404 - Not Found\nTrying other mirror.\nTo address this issue please refer to the below knowledge base article \n\nhttps://access.redhat.com/articles/1320623\n\nIf above article doesn't help to resolve this issue please open a ticket with Red Hat Support.\n\nfailure: repodata/repomd.xml from ceph-Installer: [Errno 256] No more mirrors to try.\nhttp://download-node-02.eng.bos.redhat.com/rcm-guest/ceph-drops/auto/rhscon-2-rhel-7-compose/latest-RHSCON-2-RHEL-7/compose/Installer/x86_64/os/repodata/repomd.xml: [Errno 14] HTTP Error 404 - Not Found", "stdout_lines": ["Loaded plugins: fastestmirror, langpacks, priorities, product-id", "http://download-node-02.eng.bos.redhat.com/rcm-guest/ceph-drops/auto/rhscon-2-rhel-7-compose/latest-RHSCON-2-RHEL-7/compose/Installer/x86_64/os/repodata/repomd.xml: [Errno 14] HTTP Error 404 - Not Found", "Trying other mirror.", "To address this issue please refer to the below knowledge base article ", "", "https://access.redhat.com/articles/1320623", "", "If above article doesn't help to resolve this issue please open a ticket with Red Hat Support.", "", "failure: repodata/repomd.xml from ceph-Installer: [Errno 256] No more mirrors to try.", "http://download-node-02.eng.bos.redhat.com/rcm-guest/ceph-drops/auto/rhscon-2-rhel-7-compose/latest-RHSCON-2-RHEL-7/compose/Installer/x86_64/os/repodata/repomd.xml: [Errno 14] HTTP Error 404 - Not Found"], "warnings": []}
The first failure in Sam's run was:
TASK [pcp : Install pcp] *******************************************************
fatal: [smithi154.front.sepia.ceph.com]: FAILED! => {"changed": false, "failed": true, "msg": "https://2.chacra.ceph.com/r/samba/master/ce9b72c17abb156de8185b100f27d1ddd3c89b15/centos/7/flavors/default/x86_64/repodata/repomd.xml: [Errno 14] HTTPS Error 404 - Not Found\nTrying other mirror.\nTo address this issue please refer to the below knowledge base article \n\nhttps://access.redhat.com/articles/1320623\n\nIf above article doesn't help to resolve this issue please create a bug on https://bugs.centos.org/\n\n\n\n One of the configured repositories failed (samba packages for \\x86_64),\n and yum doesn't have enough cached data to continue. At this point the only\n safe thing yum can do is fail. There are a few ways to work \"fix\" this:\n\n 1. Contact the upstream for the repository and get them to fix the problem.\n\n 2. Reconfigure the baseurl/etc. for the repository, to point to a working\n upstream. This is most often useful if you are using a newer\n distribution release than is supported by the repository (and the\n packages for the previous distribution release still work).\n\n 3. Run the command with the repository temporarily disabled\n yum --disablerepo=samba ...\n\n 4. Disable the repository permanently, so yum won't use it by default. Yum\n will then just ignore the repository until you permanently enable it\n again or use --enablerepo for temporary usage:\n\n yum-config-manager --disable samba\n or\n subscription-manager repos --disable=samba\n\n 5. Configure the failing repository to be skipped, if it is unavailable.\n Note that yum will try to contact the repo. when it runs most commands,\n so will have to try and fail each time (and thus. yum will be be much\n slower). If it is a very temporary problem though, this is often a nice\n compromise:\n
2017-02-23T06:15:47.284 INFO:teuthology.task.ansible.out:\n yum-config-manager --save --setopt=samba.skip_if_unavailable=true\n\nfailure: repodata/repomd.xml from samba: [Errno 256] No more mirrors to try.\nhttps://2.chacra.ceph.com/r/samba/master/ce9b72c17abb156de8185b100f27d1ddd3c89b15/centos/7/flavors/default/x86_64/repodata/repomd.xml: [Errno 14] HTTPS Error 404 - Not Found\n", "rc": 1, "results": []}
The problem Sam's jobs hit is fixed in http://tracker.ceph.com/issues/19061
Updated by Vasu Kulkarni about 7 years ago
Yeah I checked other logs in that same run, for some reason 404 during that test for few jobs:
2017-02-01T15:18:08.886 INFO:teuthology.task.ansible.out:
TASK [testnode : Removing saved yum transactions] ******************************
2017-02-01T15:18:08.887 INFO:teuthology.task.ansible.out:fatal: [clara007.ceph.redhat.com]: FAILED! => {"changed": false, "cmd": ["yum-complete-transaction", "--cleanup-only"], "delta": "0:00:01.382613", "end": "2017-02-01 20:18:09.104991", "failed": true, "rc": 1, "start": "2017-02-01 20:18:07.722378", "stderr": "", "stdout": "Loaded plugins: fastestmirror, langpacks, priorities, product-id\nhttp://download-node-02.eng.bos.redhat.com/rcm-guest/ceph-drops/auto/rhscon-2-rhel-7-compose/latest-RHSCON-
Is there a way to report this as error in ansible or stop right after that first failure? or the only way is scan for some keywords?
Updated by David Galloway about 7 years ago
I'm guessing what really happened here is ansible changed its output for failures and teuthology needs to be updated for the true ansible failure to show up as the job failure reason.
Updated by David Galloway almost 7 years ago
- Project changed from sepia to teuthology
Throwing over the wall. Feel free to throw it back at me if I'm wrong.
Updated by David Galloway almost 7 years ago
Here's another example:
The real error
2017-05-05T17:37:52.387 INFO:teuthology.task.ansible.out:fatal: [mira078.front.sepia.ceph.com]: FAILED! => {"changed": false, "failed": true, "msg": "https://2.chacra.ceph.com/r/calamari/1.5/1be5ff4b365dd91a38180cde996ecb5b150b01de/centos/7/flavors/default/x86_64/repodata/repomd.xml: [Errno 14] HTTPS Error 404 - Not Found\nTrying other mirror.\nTo address this issue please refer to the below knowledge base article \n\nhttps://access.redhat.com/articles/1320623\n\nIf above article doesn't help to resolve this issue please create a bug on https://bugs.centos.org/\n\n\n\n One of the configured repositories failed (calamari packages for \\x86_64),\n and yum doesn't have enough cached data to continue. At this point the only\n safe thing yum can do is fail. There are a few ways to work \"fix\" this:\n\n 1. Contact the upstream for the repository and get them to fix the problem.\n\n 2. Reconfigure the baseurl/etc. for the repository, to point to a working\n upstream. This is most often useful if you are using a newer\n distribution release than is supported by the repository (and the\n packages for the previous distribution release still work).\n\n 3. Disable the repository, so yum won't use it by default. Yum will then\n just ignore the repository until you permanently enable it again or use\n --enablerepo for temporary usage:\n\n yum-config-manager --disable calamari\n\n 4. Configure the failing repository to be skipped, if it is unavailable.\n Note that yum will try to contact the repo. when it runs most commands,\n so will have to try and fail each time (and thus. yum will be be much\n slower). If it is a very temporary problem though, this is often a nice\n compromise:\n\n yum-config-manager --save --setopt=calamari.skip_if_unavailable=true\n\nfailure: repodata/repomd.xml from calamari: [Errno 256] No more mirrors to try.\nhttps://2.chacra.ceph.com/r/calamari/1.5/1be5ff4b365dd91a38180cde996ecb5b150b01de/centos/7/flavors/def
2017-05-05T17:37:52.389 INFO:teuthology.task.ansible.out:ault/x86_64/repodata/repomd.xml: [Errno 14] HTTPS Error 404 - Not Found\n", "rc": 1, "results": []}
2017-05-05T17:37:52.519 INFO:teuthology.task.ansible.out:fatal: [mira038.front.sepia.ceph.com]: FAILED! => {"changed": false, "failed": true, "msg": "https://2.chacra.ceph.com/r/calamari/1.4-something/ef04582d4bc0dcd1b9ffaf939b05bb13d32ae785/centos/7/flavors/default/x86_64/repodata/repomd.xml: [Errno 14] HTTPS Error 404 - Not Found\nTrying other mirror.\nTo address this issue please refer to the below knowledge base article \n\nhttps://access.redhat.com/articles/1320623\n\nIf above article doesn't help to resolve this issue please create a bug on https://bugs.centos.org/\n\n\n\n One of the configured repositories failed (calamari packages for \\x86_64),\n and yum doesn't have enough cached data to continue. At this point the only\n safe thing yum can do is fail. There are a few ways to work \"fix\" this:\n\n 1. Contact the upstream for the repository and get them to fix the problem.\n\n 2. Reconfigure the baseurl/etc. for the repository, to point to a working\n upstream. This is most often useful if you are using a newer\n distribution release than is supported by the repository (and the\n packages for the previous distribution release still work).\n\n 3. Disable the repository, so yum won't use it by default. Yum will then\n just ignore the repository until you permanently enable it again or use\n --enablerepo for temporary usage:\n\n yum-config-manager --disable calamari\n\n 4. Configure the failing repository to be skipped, if it is unavailable.\n Note that yum will try to contact the repo. when it runs most commands,\n so will have to try and fail each time (and thus. yum will be be much\n slower). If it is a very temporary problem though, this is often a nice\n compromise:\n\n yum-config-manager --save --setopt=calamari.skip_if_unavailable=true\n\nfailure: repodata/repomd.xml from calamari: [Errno 256] No more mirrors to try.\nhttps://2.chacra.ceph.com/r/calamari/1.4-something/ef04582d4bc0dcd1b9ffaf939b05bb13d32ae785/
2017-05-05T17:37:52.521 INFO:teuthology.task.ansible.out:centos/7/flavors/default/x86_64/repodata/repomd.xml: [Errno 14] HTTPS Error 404 - Not Found\n", "rc": 1, "results": []}
Python traceback:
2017-05-05T17:42:49.886 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 89, in run_tasks
manager.__enter__()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/__init__.py", line 121, in __enter__
self.begin()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/ansible.py", line 390, in begin
super(CephLab, self).begin()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/ansible.py", line 237, in begin
self.execute_playbook()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/ansible.py", line 264, in execute_playbook
self._handle_failure(command, status)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/ansible.py", line 277, in _handle_failure
failures = yaml.safe_load(fail_log)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/__init__.py", line 93, in safe_load
return load(stream, SafeLoader)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/__init__.py", line 71, in load
return loader.get_single_data()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/constructor.py", line 37, in get_single_data
node = self.get_single_node()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/composer.py", line 36, in get_single_node
document = self.compose_document()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/composer.py", line 55, in compose_document
node = self.compose_node(None, None)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/composer.py", line 84, in compose_node
node = self.compose_mapping_node(anchor)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/composer.py", line 133, in compose_mapping_node
item_value = self.compose_node(node, item_key)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/composer.py", line 84, in compose_node
node = self.compose_mapping_node(anchor)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/composer.py", line 133, in compose_mapping_node
item_value = self.compose_node(node, item_key)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/composer.py", line 84, in compose_node
node = self.compose_mapping_node(anchor)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/composer.py", line 133, in compose_mapping_node
item_value = self.compose_node(node, item_key)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/composer.py", line 64, in compose_node
if self.check_event(AliasEvent):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/parser.py", line 98, in check_event
self.current_event = self.state()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/parser.py", line 572, in parse_flow_mapping_value
if not self.check_token(FlowEntryToken, FlowMappingEndToken):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/scanner.py", line 116, in check_token
self.fetch_more_tokens()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/scanner.py", line 252, in fetch_more_tokens
return self.fetch_plain()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/scanner.py", line 676, in fetch_plain
self.tokens.append(self.scan_plain())
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/scanner.py", line 1305, in scan_plain
"Please check http://pyyaml.org/wiki/YAMLColonInFlowContext for details.")
ScannerError: while scanning a plain scalar
in "/tmp/teuth_ansible_failures_KOQre5", line 1, column 498
found unexpected ':'
in "/tmp/teuth_ansible_failures_KOQre5", line 1, column 505
Please check http://pyyaml.org/wiki/YAMLColonInFlowContext for details.
2017-05-05T17:42:50.092 ERROR:teuthology.run_tasks: Sentry event: http://sentry.ceph.com/sepia/teuthology/?q=4bcf8ccd6aef46668abd9146b23c8ce7
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 89, in run_tasks
manager.__enter__()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/__init__.py", line 121, in __enter__
self.begin()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/ansible.py", line 390, in begin
super(CephLab, self).begin()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/ansible.py", line 237, in begin
self.execute_playbook()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/ansible.py", line 264, in execute_playbook
self._handle_failure(command, status)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/ansible.py", line 277, in _handle_failure
failures = yaml.safe_load(fail_log)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/__init__.py", line 93, in safe_load
return load(stream, SafeLoader)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/__init__.py", line 71, in load
return loader.get_single_data()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/constructor.py", line 37, in get_single_data
node = self.get_single_node()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/composer.py", line 36, in get_single_node
document = self.compose_document()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/composer.py", line 55, in compose_document
node = self.compose_node(None, None)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/composer.py", line 84, in compose_node
node = self.compose_mapping_node(anchor)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/composer.py", line 133, in compose_mapping_node
item_value = self.compose_node(node, item_key)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/composer.py", line 84, in compose_node
node = self.compose_mapping_node(anchor)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/composer.py", line 133, in compose_mapping_node
item_value = self.compose_node(node, item_key)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/composer.py", line 84, in compose_node
node = self.compose_mapping_node(anchor)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/composer.py", line 133, in compose_mapping_node
item_value = self.compose_node(node, item_key)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/composer.py", line 64, in compose_node
if self.check_event(AliasEvent):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/parser.py", line 98, in check_event
self.current_event = self.state()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/parser.py", line 572, in parse_flow_mapping_value
if not self.check_token(FlowEntryToken, FlowMappingEndToken):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/scanner.py", line 116, in check_token
self.fetch_more_tokens()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/scanner.py", line 252, in fetch_more_tokens
return self.fetch_plain()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/scanner.py", line 676, in fetch_plain
self.tokens.append(self.scan_plain())
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/scanner.py", line 1305, in scan_plain
"Please check http://pyyaml.org/wiki/YAMLColonInFlowContext for details.")
ScannerError: while scanning a plain scalar
in "/tmp/teuth_ansible_failures_KOQre5", line 1, column 498
found unexpected ':'
in "/tmp/teuth_ansible_failures_KOQre5", line 1, column 505
Please check http://pyyaml.org/wiki/YAMLColonInFlowContext for details.
Updated by huang jun over 5 years ago
Did it is a problem? we have same output for some cases but when i rerun those cases, it's ok.
Actions