Bug #9908: ChannelException: Administratively prohibited - teuthology - Ceph

Actions

Copy link

Bug #9908

closed

ChannelException: Administratively prohibited

Added by Zack Cerza over 9 years ago. Updated almost 9 years ago.

Status:

Resolved

Priority:

Normal

Assignee:

Zack Cerza

Category:

% Done:

100%

Source:

Q/A

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Crash signature (v1):

Crash signature (v2):

Description

Something is causing tracebacks like this:

2014-10-27T10:12:36.305 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 55, in run_tasks
    manager.__enter__()
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/var/lib/teuthworker/src/ceph-qa-suite_giant/tasks/ceph.py", line 1262, in task
    lambda: run_daemon(ctx=ctx, config=config, type_='mds'),
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/src/teuthology_master/teuthology/contextutil.py", line 27, in nested
    vars.append(enter())
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/var/lib/teuthworker/src/ceph-qa-suite_giant/tasks/ceph.py", line 185, in cephfs_setup
    stdout=StringIO())
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/remote.py", line 128, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 325, in run
    r.execute()
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 73, in execute
    self.client.exec_command(self.command)
  File "/usr/lib/python2.7/dist-packages/paramiko/client.py", line 363, in exec_command
    chan = self._transport.open_session()
  File "/usr/lib/python2.7/dist-packages/paramiko/transport.py", line 658, in open_session
    return self.open_channel('session')
  File "/usr/lib/python2.7/dist-packages/paramiko/transport.py", line 755, in open_channel
    raise e
ChannelException: Administratively prohibited

Actions

Copy link

Updated by Zack Cerza over 9 years ago

I asked paddles how often this has happened:

>>> jobs = Job.query.filter(Job.failure_reason.contains('Administratively prohibited')).all()
>>> for job in jobs: print str(job)
...
<Job u'teuthology-2014-09-08_13:37:10-upgrade:firefly-x:stress-split-master-distro-basic-vps' u'472558'>
<Job u'teuthology-2014-09-08_13:37:10-upgrade:firefly-x:stress-split-master-distro-basic-vps' u'472564'>
<Job u'teuthology-2014-09-08_13:37:10-upgrade:firefly-x:stress-split-master-distro-basic-vps' u'472560'>
<Job u'teuthology-2014-09-08_11:22:41-upgrade:firefly-x:stress-split-master-distro-basic-vps' u'472534'>
<Job u'teuthology-2014-09-08_11:22:41-upgrade:firefly-x:stress-split-master-distro-basic-vps' u'472527'>
<Job u'teuthology-2014-09-08_11:22:41-upgrade:firefly-x:stress-split-master-distro-basic-vps' u'472549'>
<Job u'teuthology-2014-09-08_11:22:41-upgrade:firefly-x:stress-split-master-distro-basic-vps' u'472537'>
<Job u'teuthology-2014-09-08_11:22:41-upgrade:firefly-x:stress-split-master-distro-basic-vps' u'472545'>
<Job u'teuthology-2014-09-08_13:37:10-upgrade:firefly-x:stress-split-master-distro-basic-vps' u'472566'>
<Job u'teuthology-2014-10-27_09:50:24-upgrade:firefly-x-giant-distro-basic-vps' u'573681'>
<Job u'teuthology-2014-10-27_09:50:24-upgrade:firefly-x-giant-distro-basic-vps' u'573687'>
<Job u'teuthology-2014-10-27_10:58:31-upgrade:firefly-x-giant-distro-basic-vps' u'573696'>
<Job u'loic-2014-10-22_10:04:57-upgrade:firefly-x-giant-testing-basic-vps' u'566248'>
<Job u'teuthology-2014-10-23_10:53:07-upgrade:firefly-x-giant-distro-basic-vps' u'567521'>
<Job u'teuthology-2014-10-23_10:53:07-upgrade:firefly-x-giant-distro-basic-vps' u'567508'>
<Job u'teuthology-2014-10-23_10:53:07-upgrade:firefly-x-giant-distro-basic-vps' u'567520'>
<Job u'teuthology-2014-10-23_10:53:07-upgrade:firefly-x-giant-distro-basic-vps' u'567515'>
<Job u'teuthology-2014-10-23_10:53:07-upgrade:firefly-x-giant-distro-basic-vps' u'567512'>
<Job u'teuthology-2014-10-23_10:53:07-upgrade:firefly-x-giant-distro-basic-vps' u'567518'>
<Job u'teuthology-2014-10-23_10:53:07-upgrade:firefly-x-giant-distro-basic-vps' u'567514'>
<Job u'teuthology-2014-10-23_10:53:07-upgrade:firefly-x-giant-distro-basic-vps' u'567509'>
<Job u'teuthology-2014-10-23_10:53:07-upgrade:firefly-x-giant-distro-basic-vps' u'567506'>
<Job u'teuthology-2014-10-17_19:25:01-upgrade:firefly-x-giant-distro-basic-vps' u'554637'>
<Job u'teuthology-2014-10-17_19:25:01-upgrade:firefly-x-giant-distro-basic-vps' u'554666'>
<Job u'teuthology-2014-10-17_19:25:01-upgrade:firefly-x-giant-distro-basic-vps' u'554648'>
<Job u'teuthology-2014-10-17_19:25:01-upgrade:firefly-x-giant-distro-basic-vps' u'554636'>
<Job u'teuthology-2014-10-17_19:25:01-upgrade:firefly-x-giant-distro-basic-vps' u'554660'>
<Job u'teuthology-2014-10-17_19:25:01-upgrade:firefly-x-giant-distro-basic-vps' u'554649'>
<Job u'teuthology-2014-10-17_19:25:01-upgrade:firefly-x-giant-distro-basic-vps' u'554654'>
<Job u'teuthology-2014-10-17_19:25:01-upgrade:firefly-x-giant-distro-basic-vps' u'554661'>
<Job u'teuthology-2014-10-17_19:25:01-upgrade:firefly-x-giant-distro-basic-vps' u'554655'>
<Job u'teuthology-2014-10-17_19:25:01-upgrade:firefly-x-giant-distro-basic-vps' u'554642'>
<Job u'teuthology-2014-10-17_19:25:01-upgrade:firefly-x-giant-distro-basic-vps' u'554679'>
<Job u'teuthology-2014-10-17_19:25:01-upgrade:firefly-x-giant-distro-basic-vps' u'554667'>
<Job u'teuthology-2014-10-17_19:25:01-upgrade:firefly-x-giant-distro-basic-vps' u'554672'>
<Job u'teuthology-2014-10-17_19:25:01-upgrade:firefly-x-giant-distro-basic-vps' u'554673'>
<Job u'teuthology-2014-10-17_19:25:01-upgrade:firefly-x-giant-distro-basic-vps' u'554678'>
<Job u'teuthology-2014-10-20_15:01:14-upgrade:firefly-x-giant-distro-basic-vps' u'560042'>
<Job u'teuthology-2014-10-20_15:09:31-upgrade:firefly-x:stress-split-giant-distro-basic-vps' u'560083'>
<Job u'teuthology-2014-10-20_15:09:31-upgrade:firefly-x:stress-split-giant-distro-basic-vps' u'560080'>
<Job u'teuthology-2014-10-27_10:58:31-upgrade:firefly-x-giant-distro-basic-vps' u'573695'>
<Job u'teuthology-2014-10-27_10:58:31-upgrade:firefly-x-giant-distro-basic-vps' u'573700'>
<Job u'teuthology-2014-10-27_09:50:24-upgrade:firefly-x-giant-distro-basic-vps' u'573672'>
<Job u'teuthology-2014-10-27_09:50:24-upgrade:firefly-x-giant-distro-basic-vps' u'573678'>
<Job u'teuthology-2014-10-27_09:50:24-upgrade:firefly-x-giant-distro-basic-vps' u'573675'>
<Job u'teuthology-2014-10-27_09:50:24-upgrade:firefly-x-giant-distro-basic-vps' u'573674'>
<Job u'teuthology-2014-10-27_09:50:24-upgrade:firefly-x-giant-distro-basic-vps' u'573680'>
<Job u'teuthology-2014-10-27_09:50:24-upgrade:firefly-x-giant-distro-basic-vps' u'573686'>
<Job u'teuthology-2014-10-27_10:58:31-upgrade:firefly-x-giant-distro-basic-vps' u'573698'>
<Job u'teuthology-2014-10-27_10:58:31-upgrade:firefly-x-giant-distro-basic-vps' u'573699'>
<Job u'teuthology-2014-10-27_10:58:31-upgrade:firefly-x-giant-distro-basic-vps' u'573697'>
<Job u'teuthology-2014-07-30_12:36:02-upgrade:firefly-firefly-testing-basic-vps' u'387311'>
<Job u'teuthology-2014-07-30_12:36:02-upgrade:firefly-firefly-testing-basic-vps' u'387383'>
<Job u'teuthology-2014-07-30_12:36:02-upgrade:firefly-firefly-testing-basic-vps' u'387317'>
<Job u'teuthology-2014-07-30_12:36:02-upgrade:firefly-firefly-testing-basic-vps' u'387374'>
<Job u'teuthology-2014-07-30_12:36:02-upgrade:firefly-firefly-testing-basic-vps' u'387524'>
<Job u'teuthology-2014-07-30_12:36:02-upgrade:firefly-firefly-testing-basic-vps' u'387508'>

Actions

Copy link

Updated by Tamilarasi muthamizhan over 9 years ago

Assignee set to Sandon Van Ness

Actions

Copy link

Updated by Shambhu Rajak over 9 years ago

I had the same issue, so i understood that my Linux box was not allowing more than 10 sessions, so i added
Maxsession 100 in/etc/ssh/sshd_config and it restarted the service, and now Paramiko is able to make more than 10 concurrent session.

Actions

Copy link

Updated by Sandon Van Ness over 9 years ago

Status changed from New to Resolved
% Done changed from 0 to 100

This should be fixed as of commit 7ba6e104551e343702b4975570610332624e365e in ceph-qa-chef since Oct 27th. If someone is sitll seeing this on our lab machines let me know!

As per the comment above it was due to a 10 connection limit.

Actions

Copy link

Updated by Zack Cerza over 9 years ago

>>> jobs = Job.query.filter(Job.failure_reason.contains('Administratively prohibited')).all()
>>> print [job for job in jobs if '2014-11' in str(job.updated)]
[]

Yup, we're good

Actions

Copy link

Updated by Yuri Weinstein almost 9 years ago

Status changed from Resolved to New
Assignee changed from Sandon Van Ness to Zack Cerza
Source changed from other to Q/A
Regression set to No

See again in run: http://pulpito.ceph.com/teuthology-2015-07-09_14:34:37-upgrade:hammer-x-wip-12021-next-distro-basic-vps/
Job: http://qa-proxy.ceph.com/teuthology/teuthology-2015-07-09_14:34:37-upgrade:hammer-x-wip-12021-next-distro-basic-vps/967519/teuthology.log

Actions

Copy link

Updated by Yuri Weinstein almost 9 years ago

Persisting in run:
http://pulpito.ceph.com/teuthology-2015-07-12_17:18:02-upgrade:firefly-x-next-distro-basic-vps/
Jobs: ['971058', '971061', '971062', '971063', '971067', '971070', '971071', '971072', '971079', '971080', '971081', '971085', '971088', '971089']

Actions

Copy link

Updated by Zack Cerza almost 9 years ago

Updating paramiko did not help:
http://pulpito.ceph.com/zack-2015-07-14_11:00:51-upgrade:firefly-x-next-distro-basic-vps/

Actions

Copy link

Updated by Yuri Weinstein almost 9 years ago

Also in run: http://pulpito.ceph.com/teuthology-2015-07-14_17:10:10-upgrade:hammer-x-next-distro-basic-vps/

Actions

Copy link

#10

Updated by Yuri Weinstein almost 9 years ago

One more - http://qa-proxy.ceph.com/teuthology/teuthology-2015-07-17_08:46:22-upgrade:hammer-hammer-distro-basic-vps/977126/teuthology.log

The same job passed here http://pulpito.ceph.com/teuthology-2015-07-17_11:11:27-upgrade:hammer-hammer-distro-basic-vps/

Maybe comparing them will hold some more information?

Actions

Copy link

#11

Updated by Zack Cerza almost 9 years ago

From one of the jobs:

2015-07-14T21:20:07.820 INFO:teuthology.task.ansible.out:
NOTIFIED: [testnode | restart ssh] ******************************************** 

2015-07-14T21:20:08.661 INFO:teuthology.task.ansible.out:[0;33mchanged: [vpm121.front.sepia.ceph.com] => {"changed": true, "name": "sshd", "state": "started"}[0m

2015-07-14T21:20:09.164 INFO:teuthology.task.ansible.out:[0;33mchanged: [vpm138.front.sepia.ceph.com] => {"changed": true, "name": "sshd", "state": "started"}[0m
[0;33mchanged: [vpm070.front.sepia.ceph.com] => {"changed": true, "name": "sshd", "state": "started"}[0m

Should that not read 'restarted' ?

Actions

Copy link

#12

Updated by Zack Cerza almost 9 years ago

Status changed from New to Fix Under Review

I think I found the problem. We're shipping a new sshd_config and restarting sshd, but we're still connected and thus might be affected by settings in the old config, not the new one. Reconnecting after ansible-playbook is run would fix that.

https://github.com/ceph/teuthology/pull/574

Actions

Copy link

#13