Bug #12675: Ansible error handler throws ParserError - teuthology - Ceph

Actions

Copy link

Bug #12675

closed

Ansible error handler throws ParserError

Added by John Spray over 8 years ago. Updated over 8 years ago.

Status:

Resolved

Priority:

High

Assignee:

Andrew Schoen

Category:

% Done:

Source:

other

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Crash signature (v1):

Crash signature (v2):

Description

2015-08-10T20:22:11.142 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 56, in run_tasks
    manager.__enter__()
  File "/home/teuthworker/src/teuthology_master/teuthology/task/__init__.py", line 121, in __enter__
    self.begin()
  File "/home/teuthworker/src/teuthology_master/teuthology/task/ansible.py", line 230, in begin
    self.execute_playbook()
  File "/home/teuthworker/src/teuthology_master/teuthology/task/ansible.py", line 255, in execute_playbook
    self._handle_failure(command, status)
  File "/home/teuthworker/src/teuthology_master/teuthology/task/ansible.py", line 266, in _handle_failure
    failures = yaml.safe_load(log)
  File "/home/teuthworker/src/teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/__init__.py", line 93, in safe_load
    return load(stream, SafeLoader)
  File "/home/teuthworker/src/teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/__init__.py", line 71, in load
    return loader.get_single_data()
  File "/home/teuthworker/src/teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/constructor.py", line 37, in get_single_data
    node = self.get_single_node()
  File "/home/teuthworker/src/teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/composer.py", line 39, in get_single_node
    if not self.check_event(StreamEndEvent):
  File "/home/teuthworker/src/teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/parser.py", line 98, in check_event
    self.current_event = self.state()
  File "/home/teuthworker/src/teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/parser.py", line 174, in parse_document_start
    self.peek_token().start_mark)
ParserError: expected '<document start>', but found '{'
  in "/tmp/teuth_ansible_failures_d8JX7N", line 4, column 1

Same area as #12656. Also fallout from https://github.com/ceph/teuthology/commit/fdc11d9e5922811533e51968a34d05a2d94feec2

Actions

Copy link

Updated by John Spray over 8 years ago

/a/teuthology-2015-08-08_23:18:02-multimds-master-testing-basic-multi/1007525

Actions

Copy link

Updated by Andrew Schoen over 8 years ago

Status changed from New to In Progress

I added some error handling on that ParserError here: https://github.com/ceph/teuthology/pull/595

This will at least let the task complete and raise a CommandFailedError if it can't parse the log. We can then go and inspect the log after the fact to see what pyyaml is tripping on.

Actions

Copy link

Updated by Andrew Schoen over 8 years ago

I'm looking at the log linked here: http://qa-proxy.ceph.com/teuthology/teuthology-2015-08-08_23:18:02-multimds-master-testing-basic-multi/1007525/teuthology.log

I see:

2015-08-10T20:22:10.992 INFO:teuthology.task.ansible.out:[0;31mfatal: [plana91.front.sepia.ceph.com] => error while evaluating conditional: ssh_key_update.state == 'present'[0m

This is an indication that the ansible being used here is not 1.9.2. That ssh key issue was fixed in 1.9.2 I believe.

Actions

Copy link

Updated by Andrew Schoen over 8 years ago

PR 595 was merged. If we see a CommandFailedError instead of an AnsibleFailedError we can inspect the ansible log and hopefully get a hint as to what's causing the ParserError.

https://github.com/ceph/teuthology/commit/80376b8ea5736f94f5ff8bf6036d6277151f9172

Actions

Copy link