Actions
Bug #14541
closed"No space left on device" when locking vpm097
% Done:
0%
Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):
Description
2016-01-27T19:56:52.684 INFO:teuthology.provision:Downburst failed on ubuntu@vpm097.front.sepia.ceph.com: INFO:urllib3.connectionpool:Starting new HTTP connection (1): ceph.com
INFO:urllib3.connectionpool:Starting new HTTPS connection (1): git.ceph.com
libvirt: Storage Driver error : cannot fill file '/srv/libvirtpool/vpm097/vpm097-3.img': No space left on device
Traceback (most recent call last):
File "/home/ubuntu/src/downburst/virtualenv/bin/downburst", line 9, in <module>
load_entry_point('downburst==0.0.1', 'console_scripts', 'downburst')()
File "/home/ubuntu/src/downburst/downburst/cli.py", line 59, in main
return args.func(args)
File "/home/ubuntu/src/downburst/downburst/create.py", line 158, in create
additional_disks_key.append(pool.createXML(etree.tostring(diskxml), flags=0).key())
File "/usr/lib/python2.7/dist-packages/libvirt.py", line 2831, in createXML
if ret is None:raise libvirtError('virStorageVolCreateXML() failed', pool=self)
libvirt.libvirtError: cannot fill file '/srv/libvirtpool/vpm097/vpm097-3.img': No space left on device
2016-01-27T19:56:52.684 ERROR:teuthology.lock:Unable to create virtual machine: ubuntu@vpm097.front.sepia.ceph.com
2016-01-27T19:56:54.009 ERROR:teuthology.provision:Error destroying vpm097.front.sepia.ceph.com: libvirt: QEMU Driver error : Domain not found: no domain with matching name 'vpm097'
Updated by Dan Mick about 8 years ago
- Subject changed from "No space left on device" vpm097 to "No space left on device" when locking vpm097
I can't figure this one out. By all accounts there is only about 15GB used on /, but df shows 628GB used. Normally that would mean "there are deleted but still open files", but lsof cannot find them.
fsck agrees that there is about that much of the disk allocated. find -ls | awk-to-sum-up-sizes
agrees that there's only about 15GB in files.
My best guess is "try a reboot", but there are jobs on the other vpms on mira017. Maybe we have to mark them down and let them become unlocked so we can reboot to see.
Updated by Dan Mick about 8 years ago
- Status changed from New to Resolved
- Assignee changed from David Galloway to Dan Mick
There were files under vpm104's mountpoint consuming space from root. Embarrassing that it took so long to figure that out, but, it's figured out.
Actions