Project

General

Profile

Actions
Copy link

Bug #16893

closed

OVH throttling token creation

Added by Loïc Dachary over 7 years ago. Updated 12 months ago.

Status:
Resolved
Priority:
Urgent
Assignee:
-
% Done:

0%

Source:
other
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):

Description

OVH announced they will limit the number of authentication tokens issued on a daily basis http://travaux.ovh.net/?do=details&id=19345.

The teuthology openstack backend should be modified to re-use the same token (with --os-token <auth-token>) when possible. Tokens are typically valid for ~24h.

https://wiki.openstack.org/wiki/OpenStackClient/Authentication

  • export OS_AUTH_TYPE=v2token
  • $ openstack token issue
    +------------+----------------------------------+
| Field      | Value                            |
+------------+----------------------------------+
| expires    | 2016-08-12T09:53:50Z             |
| id         | da27cded0556439e85e7468343f1a257 |
| project_id | 131b886b156a4f84b5f41baf2fbe646c |
| user_id    | 291dde1633154837be2693c6ffa6315c |
+------------+----------------------------------+
  • export OS_TOKEN=da27cded0556439e85e7468343f1a257

Related issues 1 (0 open — 1 closed)

Has duplicate ovh - Bug #16972: dead jobs - provisioning nodes (Command 'openstack flavor list -f json')Duplicate08/09/2016

Actions
Actions
Copy link
 #1

Updated by Loïc Dachary over 7 years ago

  • Description updated (diff)
Actions
Copy link
 #2

Updated by Loïc Dachary over 7 years ago

  • Description updated (diff)
Actions
Copy link
 #3

Updated by Loïc Dachary over 7 years ago

  • Has duplicate Bug #16972: dead jobs - provisioning nodes (Command 'openstack flavor list -f json') added
Actions
Copy link
 #4

Updated by Loïc Dachary over 7 years ago

  • Description updated (diff)
Actions
Copy link
 #5

Updated by Loïc Dachary over 7 years ago

  • Status changed from 12 to Fix Under Review
  • Assignee set to Loïc Dachary
Actions
Copy link
 #7

Updated by Nathan Cutler over 7 years ago

I can trigger Error 429 in OVH even with the throttling patches, by running a single test 10 times (--num 10). For example:

cw -v ceph-qa-suite --simultaneous-jobs 10 --teuthology-git-url http://github.com/SUSE/teuthology --teuthology-branch wip-opensuse --ceph-qa-suite-git-url http://github.com/SUSE/ceph-qa-suite --suite-branch wip-opensuse-jewel --suite rados --ceph-git-url http://github.com/ceph/ceph --ceph jewel --filter 'rados/objectstore/objectstore.yaml' --upload --num 10
Actions
Copy link
 #8

Updated by Zack Cerza over 7 years ago

Yeah... the jobs I scheduled after merging immediately died. :-/

Actions
Copy link
 #9

Updated by Loïc Dachary over 7 years ago

  • Status changed from Fix Under Review to In Progress

Trying to reproduce the problem to analyze the logs.

ceph-workbench ceph-qa-suite --num 10 --verbose --ceph-qa-suite-git-url http://github.com/ceph/ceph-qa-suite --suite-branch jewel --ceph-git-url http://github.com/ceph/ceph --ceph jewel --suite rados/objectstore/alloc-hint.yaml ceph-workbench ceph-qa-suite --simultaneous-jobs 100 --num 200 --verbose --ceph-qa-suite-git-url http://github.com/ceph/ceph-qa-suite --suite-branch jewel --ceph-git-url http://github.com/ceph/ceph --ceph jewel --suite rados/objectstore/alloc-hint.yaml
Actions
Copy link
 #10

Updated by Nathan Cutler over 7 years ago

Loic, sorry for not posting the logs - they are still available (for a limited time) at http://149.202.166.202:8081/ubuntu-2016-08-30_10:59:33-rados-jewel---basic-openstack/ (some, or all, of the "dead" jobs fail because of Error 429)

Actions
Copy link
 #11

Updated by Loïc Dachary over 7 years ago 

2016-08-30T10:59:41.851 DEBUG:teuthology.misc::sh: openstack -q token issue -c id -f value
2016-08-30T10:59:43.385 DEBUG:teuthology.misc:585bb4248906465994a797afec95696a
2016-08-30T10:59:43.522 INFO:teuthology.openstack:caching OS_TOKEN and setting OS_AUTH_TYPE=v2token during 3600 seconds
...
2016-08-30T11:00:33.580 DEBUG:teuthology.provision:ProvisionOpenStack:create
2016-08-30T11:00:33.580 DEBUG:teuthology.misc::sh: openstack --quiet image list -f json --property name='teuthology-ubuntu-14.04-x86_64'
2016-08-30T11:00:39.760 DEBUG:teuthology.misc:HTTP Client Error (HTTP 429)

I don't understand where the 429 error comes from. Since there is a token available in the environment, it won't query the auth server to obtain a new token, therefore it should not fail with 429.
Actions
Copy link
 #12

Updated by Loïc Dachary over 7 years ago

  • Status changed from In Progress to Fix Under Review
Actions
Copy link
 #13

Updated by Loïc Dachary over 7 years ago

  • Status changed from Fix Under Review to 12
  • Assignee deleted (Loïc Dachary)
Actions
Copy link
 #14

Updated by Patrick Donnelly over 4 years ago

  • Status changed from 12 to New
Actions
Copy link
 #15

Updated by Laura Flores 12 months ago

  • Status changed from New to Resolved
Actions
Copy link

Also available in: Atom PDF