Actions
Bug #17659
openOpenStack: volume created, but cannot immediately be added to server (404)
% Done:
0%
Source:
Community (dev)
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):
Description
Sometimes, the OpenStack provider's infrastructure cannot immediately add a newly-created volume to a server:
2016-10-21T10:30:14.413 DEBUG:teuthology.misc::sh: openstack --quiet volume show -f json target158069068026-0
2016-10-21T10:30:16.960 DEBUG:teuthology.misc:{
2016-10-21T10:30:16.960 DEBUG:teuthology.misc:"size": 10,
2016-10-21T10:30:16.961 DEBUG:teuthology.misc:"status": "available",
2016-10-21T10:30:16.961 DEBUG:teuthology.misc:"properties": "ownedby='158.69.95.136'",
2016-10-21T10:30:16.961 DEBUG:teuthology.misc:"user_id": "159a1258770041e49c4808fc92e01a26",
2016-10-21T10:30:16.961 DEBUG:teuthology.misc:"description": null,
2016-10-21T10:30:16.962 DEBUG:teuthology.misc:"availability_zone": "nova",
2016-10-21T10:30:16.962 DEBUG:teuthology.misc:"bootable": "false",
2016-10-21T10:30:16.962 DEBUG:teuthology.misc:"encrypted": false,
2016-10-21T10:30:16.962 DEBUG:teuthology.misc:"created_at": "2016-10-21T10:30:17.000000",
2016-10-21T10:30:16.962 DEBUG:teuthology.misc:"multiattach": false,
2016-10-21T10:30:16.963 DEBUG:teuthology.misc:"os-volume-replication:driver_data": null,
2016-10-21T10:30:16.963 DEBUG:teuthology.misc:"name": "target158069068026-0",
2016-10-21T10:30:16.963 DEBUG:teuthology.misc:"snapshot_id": null,
2016-10-21T10:30:16.963 DEBUG:teuthology.misc:"consistencygroup_id": null,
2016-10-21T10:30:16.963 DEBUG:teuthology.misc:"replication_status": "disabled",
2016-10-21T10:30:16.964 DEBUG:teuthology.misc:"os-vol-tenant-attr:tenant_id": "d966438c9cbb46c894789b10478ecf04",
2016-10-21T10:30:16.964 DEBUG:teuthology.misc:"source_volid": null,
2016-10-21T10:30:16.964 DEBUG:teuthology.misc:"os-volume-replication:extended_status": null,
2016-10-21T10:30:16.964 DEBUG:teuthology.misc:"type": "classic",
2016-10-21T10:30:16.964 DEBUG:teuthology.misc:"id": "7aebfe59-9aa4-40a3-8bf7-0a61f9a3dea1",
2016-10-21T10:30:16.964 DEBUG:teuthology.misc:"attachments": []
2016-10-21T10:30:16.999 DEBUG:teuthology.misc:}
2016-10-21T10:30:17.000 DEBUG:teuthology.misc::sh: openstack server add volume target158069068026 target158069068026-0
2016-10-21T10:30:25.343 DEBUG:teuthology.misc:Volume 7aebfe59-9aa4-40a3-8bf7-0a61f9a3dea1 could not be found. (HTTP 404) (Request-ID: req-7bffb1d3-94fd-40c5-bbac-6ddc009f574a)
2016-10-21T10:30:25.381 ERROR:teuthology.provision:Command 'openstack server add volume target158069068026 target158069068026-0' returned non-zero exit status 1
Traceback (most recent call last):
File "/home/ubuntu/teuthology/teuthology/provision.py", line 366, in create
self.attach_volumes(name, resources_hint['volumes'])
File "/home/ubuntu/teuthology/teuthology/provision.py", line 293, in attach_volumes
misc.sh("openstack server add volume " + name + " " + volume_name)
File "/home/ubuntu/teuthology/teuthology/misc.py", line 1371, in sh
output=output
CalledProcessError: Command 'openstack server add volume target158069068026 target158069068026-0' returned non-zero exit status 1
Fortunately this is rare so a workaround is simply to re-schedule the run.
However, since it's a transient error it should be handled as one - i.e. retry several times in a loop before giving up.
Updated by Nathan Cutler over 7 years ago
(01:00:07 PM) loicd: I suggest retrying after a long delay (5 min) and try 20 times or so. If the infrastructure is overloaded, it may take at least a few minutes for it to regain enough space to allocate devices.
Updated by Nathan Cutler about 7 years ago
This has been happening more frequently of late. It seems OVH-specific. Probability of it happening seems to increase as number of simultaneously-running jobs increases.
Actions