Bug #12868
Github timeout when fetching ssh keys for admin users.
% Done:
0%
Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):
Description
2015-08-28T16:25:29.233 INFO:teuthology.task.ansible.out:ESC[0;33mchanged: [burnupi42.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 3, "state": "file", "uid": 0}ESC[0m 2015-08-28T16:25:29.235 INFO:teuthology.task.ansible.out:ESC[0;33mchanged: [plana68.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 3, "state": "file", "uid": 0}ESC[0m 2015-08-28T16:25:29.248 INFO:teuthology.task.ansible.out: PLAY [cobbler] **************************************************************** 2015-08-28T16:25:29.252 INFO:teuthology.task.ansible.out:ESC[0;36mskipping: no hosts matchedESC[0m 2015-08-28T16:25:29.254 INFO:teuthology.task.ansible.out: PLAY RECAP ******************************************************************** to retry, use: --limit @/var/lib/teuthworker/cephlab.retry ESC[0;33mburnupi42.front.sepia.ceph.comESC[0m : ESC[0;32mokESC[0mESC[0;32m=ESC[0mESC[0;32m71ESC[0m ESC[0;33mchangedESC[0mESC[0;33m=ESC[0mESC[0;33m3ESC[0m unreachable=0 failed=0 ESC[0;31mmira080.front.sepia.ceph.comESC[0m : ESC[0;32mokESC[0mESC[0;32m=ESC[0mESC[0;32m8ESC[0m changed=0 unreachable=0 ESC[0;31mfailedESC[0mESC[0;31m=ESC[0mESC[0;31m1ESC[0m ESC[0;33mplana68.front.sepia.ceph.comESC[0m : ESC[0;32mokESC[0mESC[0;32m=ESC[0mESC[0;32m71ESC[0m ESC[0;33mchangedESC[0mESC[0;33m=ESC[0mESC[0;33m3ESC[0m unreachable=0 failed=0 2015-08-28T16:25:29.382 INFO:teuthology.task.ansible:Archiving ansible failure log at: /var/lib/teuthworker/archive/teuthology-2015-08-24_23:08:02-kcephfs-master-testing-basic-multi/1030722/ansible_failures.yaml 2015-08-28T16:25:32.180 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 56, in run_tasks manager.__enter__() File "/home/teuthworker/src/teuthology_master/teuthology/task/__init__.py", line 121, in __enter__ self.begin() File "/home/teuthworker/src/teuthology_master/teuthology/task/ansible.py", line 231, in begin self.execute_playbook() File "/home/teuthworker/src/teuthology_master/teuthology/task/ansible.py", line 256, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/teuthology_master/teuthology/task/ansible.py", line 279, in _handle_failure raise AnsibleFailedError(failures) AnsibleFailedError: {'mira080.front.sepia.ceph.com': {'invocation': {'module_name': 'authorized_key', 'module_args': ''}, 'failed': True, 'attempts': 3, 'item': {'name': 'dmick', 'key': 'https://raw.githubusercontent.com/ceph/keys/master/ssh/dmick.pub'}, 'msg': 'Task fail ed as maximum retries was encountered'}}
I can't tell what's going on here. This is from http://qa-proxy.ceph.com/teuthology/teuthology-2015-08-24_23:08:02-kcephfs-master-testing-basic-multi/1030722/ but searching my email it's been happening sporadically for a week or two.
History
#1 Updated by Andrew Schoen over 8 years ago
- Subject changed from "Task failed as maximum retries was encountered" to Github timeout when fetching ssh keys for admin users.
- Status changed from New to In Progress
- Assignee set to Andrew Schoen
This is a github timeout. We've worked code into ceph-cm-ansible to retry 3 times with a delay of 5 seconds between when trying to fetch ssh keys from github.
A simple solution would be just to increase those values. Perhaps just upping the retry count to 10 might be enough.
Another solution would be to not talk to github for those keys, they can be stored in the secrets repos as plain strings.
#2 Updated by Andrew Schoen over 8 years ago
- Status changed from In Progress to Fix Under Review
#3 Updated by Andrew Schoen over 8 years ago
- Status changed from Fix Under Review to Resolved