Bug #9908
closedChannelException: Administratively prohibited
100%
Description
Something is causing tracebacks like this:
2014-10-27T10:12:36.305 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 55, in run_tasks manager.__enter__() File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/var/lib/teuthworker/src/ceph-qa-suite_giant/tasks/ceph.py", line 1262, in task lambda: run_daemon(ctx=ctx, config=config, type_='mds'), File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/home/teuthworker/src/teuthology_master/teuthology/contextutil.py", line 27, in nested vars.append(enter()) File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/var/lib/teuthworker/src/ceph-qa-suite_giant/tasks/ceph.py", line 185, in cephfs_setup stdout=StringIO()) File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/remote.py", line 128, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 325, in run r.execute() File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 73, in execute self.client.exec_command(self.command) File "/usr/lib/python2.7/dist-packages/paramiko/client.py", line 363, in exec_command chan = self._transport.open_session() File "/usr/lib/python2.7/dist-packages/paramiko/transport.py", line 658, in open_session return self.open_channel('session') File "/usr/lib/python2.7/dist-packages/paramiko/transport.py", line 755, in open_channel raise e ChannelException: Administratively prohibited
Updated by Zack Cerza over 9 years ago
I asked paddles how often this has happened:
>>> jobs = Job.query.filter(Job.failure_reason.contains('Administratively prohibited')).all() >>> for job in jobs: print str(job) ... <Job u'teuthology-2014-09-08_13:37:10-upgrade:firefly-x:stress-split-master-distro-basic-vps' u'472558'> <Job u'teuthology-2014-09-08_13:37:10-upgrade:firefly-x:stress-split-master-distro-basic-vps' u'472564'> <Job u'teuthology-2014-09-08_13:37:10-upgrade:firefly-x:stress-split-master-distro-basic-vps' u'472560'> <Job u'teuthology-2014-09-08_11:22:41-upgrade:firefly-x:stress-split-master-distro-basic-vps' u'472534'> <Job u'teuthology-2014-09-08_11:22:41-upgrade:firefly-x:stress-split-master-distro-basic-vps' u'472527'> <Job u'teuthology-2014-09-08_11:22:41-upgrade:firefly-x:stress-split-master-distro-basic-vps' u'472549'> <Job u'teuthology-2014-09-08_11:22:41-upgrade:firefly-x:stress-split-master-distro-basic-vps' u'472537'> <Job u'teuthology-2014-09-08_11:22:41-upgrade:firefly-x:stress-split-master-distro-basic-vps' u'472545'> <Job u'teuthology-2014-09-08_13:37:10-upgrade:firefly-x:stress-split-master-distro-basic-vps' u'472566'> <Job u'teuthology-2014-10-27_09:50:24-upgrade:firefly-x-giant-distro-basic-vps' u'573681'> <Job u'teuthology-2014-10-27_09:50:24-upgrade:firefly-x-giant-distro-basic-vps' u'573687'> <Job u'teuthology-2014-10-27_10:58:31-upgrade:firefly-x-giant-distro-basic-vps' u'573696'> <Job u'loic-2014-10-22_10:04:57-upgrade:firefly-x-giant-testing-basic-vps' u'566248'> <Job u'teuthology-2014-10-23_10:53:07-upgrade:firefly-x-giant-distro-basic-vps' u'567521'> <Job u'teuthology-2014-10-23_10:53:07-upgrade:firefly-x-giant-distro-basic-vps' u'567508'> <Job u'teuthology-2014-10-23_10:53:07-upgrade:firefly-x-giant-distro-basic-vps' u'567520'> <Job u'teuthology-2014-10-23_10:53:07-upgrade:firefly-x-giant-distro-basic-vps' u'567515'> <Job u'teuthology-2014-10-23_10:53:07-upgrade:firefly-x-giant-distro-basic-vps' u'567512'> <Job u'teuthology-2014-10-23_10:53:07-upgrade:firefly-x-giant-distro-basic-vps' u'567518'> <Job u'teuthology-2014-10-23_10:53:07-upgrade:firefly-x-giant-distro-basic-vps' u'567514'> <Job u'teuthology-2014-10-23_10:53:07-upgrade:firefly-x-giant-distro-basic-vps' u'567509'> <Job u'teuthology-2014-10-23_10:53:07-upgrade:firefly-x-giant-distro-basic-vps' u'567506'> <Job u'teuthology-2014-10-17_19:25:01-upgrade:firefly-x-giant-distro-basic-vps' u'554637'> <Job u'teuthology-2014-10-17_19:25:01-upgrade:firefly-x-giant-distro-basic-vps' u'554666'> <Job u'teuthology-2014-10-17_19:25:01-upgrade:firefly-x-giant-distro-basic-vps' u'554648'> <Job u'teuthology-2014-10-17_19:25:01-upgrade:firefly-x-giant-distro-basic-vps' u'554636'> <Job u'teuthology-2014-10-17_19:25:01-upgrade:firefly-x-giant-distro-basic-vps' u'554660'> <Job u'teuthology-2014-10-17_19:25:01-upgrade:firefly-x-giant-distro-basic-vps' u'554649'> <Job u'teuthology-2014-10-17_19:25:01-upgrade:firefly-x-giant-distro-basic-vps' u'554654'> <Job u'teuthology-2014-10-17_19:25:01-upgrade:firefly-x-giant-distro-basic-vps' u'554661'> <Job u'teuthology-2014-10-17_19:25:01-upgrade:firefly-x-giant-distro-basic-vps' u'554655'> <Job u'teuthology-2014-10-17_19:25:01-upgrade:firefly-x-giant-distro-basic-vps' u'554642'> <Job u'teuthology-2014-10-17_19:25:01-upgrade:firefly-x-giant-distro-basic-vps' u'554679'> <Job u'teuthology-2014-10-17_19:25:01-upgrade:firefly-x-giant-distro-basic-vps' u'554667'> <Job u'teuthology-2014-10-17_19:25:01-upgrade:firefly-x-giant-distro-basic-vps' u'554672'> <Job u'teuthology-2014-10-17_19:25:01-upgrade:firefly-x-giant-distro-basic-vps' u'554673'> <Job u'teuthology-2014-10-17_19:25:01-upgrade:firefly-x-giant-distro-basic-vps' u'554678'> <Job u'teuthology-2014-10-20_15:01:14-upgrade:firefly-x-giant-distro-basic-vps' u'560042'> <Job u'teuthology-2014-10-20_15:09:31-upgrade:firefly-x:stress-split-giant-distro-basic-vps' u'560083'> <Job u'teuthology-2014-10-20_15:09:31-upgrade:firefly-x:stress-split-giant-distro-basic-vps' u'560080'> <Job u'teuthology-2014-10-27_10:58:31-upgrade:firefly-x-giant-distro-basic-vps' u'573695'> <Job u'teuthology-2014-10-27_10:58:31-upgrade:firefly-x-giant-distro-basic-vps' u'573700'> <Job u'teuthology-2014-10-27_09:50:24-upgrade:firefly-x-giant-distro-basic-vps' u'573672'> <Job u'teuthology-2014-10-27_09:50:24-upgrade:firefly-x-giant-distro-basic-vps' u'573678'> <Job u'teuthology-2014-10-27_09:50:24-upgrade:firefly-x-giant-distro-basic-vps' u'573675'> <Job u'teuthology-2014-10-27_09:50:24-upgrade:firefly-x-giant-distro-basic-vps' u'573674'> <Job u'teuthology-2014-10-27_09:50:24-upgrade:firefly-x-giant-distro-basic-vps' u'573680'> <Job u'teuthology-2014-10-27_09:50:24-upgrade:firefly-x-giant-distro-basic-vps' u'573686'> <Job u'teuthology-2014-10-27_10:58:31-upgrade:firefly-x-giant-distro-basic-vps' u'573698'> <Job u'teuthology-2014-10-27_10:58:31-upgrade:firefly-x-giant-distro-basic-vps' u'573699'> <Job u'teuthology-2014-10-27_10:58:31-upgrade:firefly-x-giant-distro-basic-vps' u'573697'> <Job u'teuthology-2014-07-30_12:36:02-upgrade:firefly-firefly-testing-basic-vps' u'387311'> <Job u'teuthology-2014-07-30_12:36:02-upgrade:firefly-firefly-testing-basic-vps' u'387383'> <Job u'teuthology-2014-07-30_12:36:02-upgrade:firefly-firefly-testing-basic-vps' u'387317'> <Job u'teuthology-2014-07-30_12:36:02-upgrade:firefly-firefly-testing-basic-vps' u'387374'> <Job u'teuthology-2014-07-30_12:36:02-upgrade:firefly-firefly-testing-basic-vps' u'387524'> <Job u'teuthology-2014-07-30_12:36:02-upgrade:firefly-firefly-testing-basic-vps' u'387508'>
Updated by Tamilarasi muthamizhan over 9 years ago
- Assignee set to Sandon Van Ness
Updated by Shambhu Rajak over 9 years ago
I had the same issue, so i understood that my Linux box was not allowing more than 10 sessions, so i added
Maxsession 100 in/etc/ssh/sshd_config and it restarted the service, and now Paramiko is able to make more than 10 concurrent session.
Updated by Sandon Van Ness over 9 years ago
- Status changed from New to Resolved
- % Done changed from 0 to 100
This should be fixed as of commit 7ba6e104551e343702b4975570610332624e365e in ceph-qa-chef since Oct 27th. If someone is sitll seeing this on our lab machines let me know!
As per the comment above it was due to a 10 connection limit.
Updated by Zack Cerza over 9 years ago
>>> jobs = Job.query.filter(Job.failure_reason.contains('Administratively prohibited')).all() >>> print [job for job in jobs if '2014-11' in str(job.updated)] []
Yup, we're good
Updated by Yuri Weinstein almost 9 years ago
- Status changed from Resolved to New
- Assignee changed from Sandon Van Ness to Zack Cerza
- Source changed from other to Q/A
- Regression set to No
Updated by Yuri Weinstein almost 9 years ago
Persisting in run:
http://pulpito.ceph.com/teuthology-2015-07-12_17:18:02-upgrade:firefly-x-next-distro-basic-vps/
Jobs: ['971058', '971061', '971062', '971063', '971067', '971070', '971071', '971072', '971079', '971080', '971081', '971085', '971088', '971089']
Updated by Zack Cerza almost 9 years ago
Updating paramiko did not help:
http://pulpito.ceph.com/zack-2015-07-14_11:00:51-upgrade:firefly-x-next-distro-basic-vps/
Updated by Yuri Weinstein almost 9 years ago
Updated by Yuri Weinstein almost 9 years ago
The same job passed here http://pulpito.ceph.com/teuthology-2015-07-17_11:11:27-upgrade:hammer-hammer-distro-basic-vps/
Maybe comparing them will hold some more information?
Updated by Zack Cerza almost 9 years ago
From one of the jobs:
2015-07-14T21:20:07.820 INFO:teuthology.task.ansible.out: NOTIFIED: [testnode | restart ssh] ******************************************** 2015-07-14T21:20:08.661 INFO:teuthology.task.ansible.out:[0;33mchanged: [vpm121.front.sepia.ceph.com] => {"changed": true, "name": "sshd", "state": "started"}[0m 2015-07-14T21:20:09.164 INFO:teuthology.task.ansible.out:[0;33mchanged: [vpm138.front.sepia.ceph.com] => {"changed": true, "name": "sshd", "state": "started"}[0m [0;33mchanged: [vpm070.front.sepia.ceph.com] => {"changed": true, "name": "sshd", "state": "started"}[0m
Should that not read 'restarted' ?
Updated by Zack Cerza almost 9 years ago
- Status changed from New to Fix Under Review
I think I found the problem. We're shipping a new sshd_config
and restarting sshd
, but we're still connected and thus might be affected by settings in the old config, not the new one. Reconnecting after ansible-playbook
is run would fix that.
Updated by Zack Cerza almost 9 years ago
- Status changed from Fix Under Review to Resolved