Project

General

Profile

Bug #12868

Github timeout when fetching ssh keys for admin users.

Added by Greg Farnum about 4 years ago. Updated about 4 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
ansible
Target version:
-
Start date:
08/31/2015
Due date:
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:

Description

2015-08-28T16:25:29.233 INFO:teuthology.task.ansible.out:ESC[0;33mchanged: [burnupi42.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 3, "state": "file", "uid": 0}ESC[0m

2015-08-28T16:25:29.235 INFO:teuthology.task.ansible.out:ESC[0;33mchanged: [plana68.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 3, "state": "file", "uid": 0}ESC[0m

2015-08-28T16:25:29.248 INFO:teuthology.task.ansible.out:
PLAY [cobbler] **************************************************************** 

2015-08-28T16:25:29.252 INFO:teuthology.task.ansible.out:ESC[0;36mskipping: no hosts matchedESC[0m

2015-08-28T16:25:29.254 INFO:teuthology.task.ansible.out:
PLAY RECAP ******************************************************************** 
           to retry, use: --limit @/var/lib/teuthworker/cephlab.retry

ESC[0;33mburnupi42.front.sepia.ceph.comESC[0m : ESC[0;32mokESC[0mESC[0;32m=ESC[0mESC[0;32m71ESC[0m   ESC[0;33mchangedESC[0mESC[0;33m=ESC[0mESC[0;33m3ESC[0m    unreachable=0    failed=0   
ESC[0;31mmira080.front.sepia.ceph.comESC[0m : ESC[0;32mokESC[0mESC[0;32m=ESC[0mESC[0;32m8ESC[0m    changed=0    unreachable=0    ESC[0;31mfailedESC[0mESC[0;31m=ESC[0mESC[0;31m1ESC[0m   
ESC[0;33mplana68.front.sepia.ceph.comESC[0m : ESC[0;32mokESC[0mESC[0;32m=ESC[0mESC[0;32m71ESC[0m   ESC[0;33mchangedESC[0mESC[0;33m=ESC[0mESC[0;33m3ESC[0m    unreachable=0    failed=0   

2015-08-28T16:25:29.382 INFO:teuthology.task.ansible:Archiving ansible failure log at: /var/lib/teuthworker/archive/teuthology-2015-08-24_23:08:02-kcephfs-master-testing-basic-multi/1030722/ansible_failures.yaml
2015-08-28T16:25:32.180 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 56, in run_tasks
    manager.__enter__()
  File "/home/teuthworker/src/teuthology_master/teuthology/task/__init__.py", line 121, in __enter__
    self.begin()
  File "/home/teuthworker/src/teuthology_master/teuthology/task/ansible.py", line 231, in begin
    self.execute_playbook()
  File "/home/teuthworker/src/teuthology_master/teuthology/task/ansible.py", line 256, in execute_playbook
    self._handle_failure(command, status)
  File "/home/teuthworker/src/teuthology_master/teuthology/task/ansible.py", line 279, in _handle_failure
    raise AnsibleFailedError(failures)
AnsibleFailedError: {'mira080.front.sepia.ceph.com': {'invocation': {'module_name': 'authorized_key', 'module_args': ''}, 'failed': True, 'attempts': 3, 'item': {'name': 'dmick', 'key': 'https://raw.githubusercontent.com/ceph/keys/master/ssh/dmick.pub'}, 'msg': 'Task fail
ed as maximum retries was encountered'}}

I can't tell what's going on here. This is from http://qa-proxy.ceph.com/teuthology/teuthology-2015-08-24_23:08:02-kcephfs-master-testing-basic-multi/1030722/ but searching my email it's been happening sporadically for a week or two.

History

#1 Updated by Andrew Schoen about 4 years ago

  • Subject changed from "Task failed as maximum retries was encountered" to Github timeout when fetching ssh keys for admin users.
  • Status changed from New to In Progress
  • Assignee set to Andrew Schoen

This is a github timeout. We've worked code into ceph-cm-ansible to retry 3 times with a delay of 5 seconds between when trying to fetch ssh keys from github.

A simple solution would be just to increase those values. Perhaps just upping the retry count to 10 might be enough.

Another solution would be to not talk to github for those keys, they can be stored in the secrets repos as plain strings.

#2 Updated by Andrew Schoen about 4 years ago

  • Status changed from In Progress to Need Review

Also available in: Atom PDF