Bug #16893
closedOVH throttling token creation
0%
Description
OVH announced they will limit the number of authentication tokens issued on a daily basis http://travaux.ovh.net/?do=details&id=19345.
The teuthology openstack backend should be modified to re-use the same token (with --os-token <auth-token>) when possible. Tokens are typically valid for ~24h.
https://wiki.openstack.org/wiki/OpenStackClient/Authentication
- export OS_AUTH_TYPE=v2token
- $ openstack token issue
+------------+----------------------------------+ | Field | Value | +------------+----------------------------------+ | expires | 2016-08-12T09:53:50Z | | id | da27cded0556439e85e7468343f1a257 | | project_id | 131b886b156a4f84b5f41baf2fbe646c | | user_id | 291dde1633154837be2693c6ffa6315c | +------------+----------------------------------+
- export OS_TOKEN=da27cded0556439e85e7468343f1a257
Updated by Loïc Dachary over 7 years ago
- Has duplicate Bug #16972: dead jobs - provisioning nodes (Command 'openstack flavor list -f json') added
Updated by Loïc Dachary over 7 years ago
- Status changed from 12 to Fix Under Review
- Assignee set to Loïc Dachary
Updated by Nathan Cutler over 7 years ago
Updated by Nathan Cutler over 7 years ago
I can trigger Error 429 in OVH even with the throttling patches, by running a single test 10 times (--num 10). For example:
cw -v ceph-qa-suite --simultaneous-jobs 10 --teuthology-git-url http://github.com/SUSE/teuthology --teuthology-branch wip-opensuse --ceph-qa-suite-git-url http://github.com/SUSE/ceph-qa-suite --suite-branch wip-opensuse-jewel --suite rados --ceph-git-url http://github.com/ceph/ceph --ceph jewel --filter 'rados/objectstore/objectstore.yaml' --upload --num 10
Updated by Zack Cerza over 7 years ago
Yeah... the jobs I scheduled after merging immediately died. :-/
Updated by Loïc Dachary over 7 years ago
- Status changed from Fix Under Review to In Progress
Trying to reproduce the problem to analyze the logs.
ceph-workbench ceph-qa-suite --num 10 --verbose --ceph-qa-suite-git-url http://github.com/ceph/ceph-qa-suite --suite-branch jewel --ceph-git-url http://github.com/ceph/ceph --ceph jewel --suite rados/objectstore/alloc-hint.yaml
- pass http://158.69.92.129:8081/ubuntu-2016-08-31_11:54:22-rados:objectstore:alloc-hint.yaml-jewel---basic-openstack/
- dead because of a 404 error on the server creation, not because of a 429, filed https://www.ovh.com/manager/web/index.html#/ticket/3659251 http://158.69.92.129:8081/ubuntu-2016-08-31_11:54:22-rados:objectstore:alloc-hint.yaml-jewel---basic-openstack/6
ceph-workbench ceph-qa-suite --simultaneous-jobs 100 --num 200 --verbose --ceph-qa-suite-git-url http://github.com/ceph/ceph-qa-suite --suite-branch jewel --ceph-git-url http://github.com/ceph/ceph --ceph jewel --suite rados/objectstore/alloc-hint.yaml
Updated by Nathan Cutler over 7 years ago
Loic, sorry for not posting the logs - they are still available (for a limited time) at http://149.202.166.202:8081/ubuntu-2016-08-30_10:59:33-rados-jewel---basic-openstack/ (some, or all, of the "dead" jobs fail because of Error 429)
Updated by Loïc Dachary over 7 years ago
2016-08-30T10:59:41.851 DEBUG:teuthology.misc::sh: openstack -q token issue -c id -f value 2016-08-30T10:59:43.385 DEBUG:teuthology.misc:585bb4248906465994a797afec95696a 2016-08-30T10:59:43.522 INFO:teuthology.openstack:caching OS_TOKEN and setting OS_AUTH_TYPE=v2token during 3600 seconds ... 2016-08-30T11:00:33.580 DEBUG:teuthology.provision:ProvisionOpenStack:create 2016-08-30T11:00:33.580 DEBUG:teuthology.misc::sh: openstack --quiet image list -f json --property name='teuthology-ubuntu-14.04-x86_64' 2016-08-30T11:00:39.760 DEBUG:teuthology.misc:HTTP Client Error (HTTP 429)
I don't understand where the 429 error comes from. Since there is a token available in the environment, it won't query the auth server to obtain a new token, therefore it should not fail with 429.
Updated by Loïc Dachary over 7 years ago
- Status changed from In Progress to Fix Under Review
Updated by Loïc Dachary over 7 years ago
- Status changed from Fix Under Review to 12
- Assignee deleted (
Loïc Dachary)