Actions
Bug #18687
closedbluestore: ENOSPC writing to XFS block file on smithi
Status:
Resolved
Priority:
High
Assignee:
-
Category:
-
Target version:
-
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
-14> 2017-01-25 23:52:14.783592 7f48267c3700 10 bdev(/var/lib/ceph/osd/ceph-4/block) _aio_thread finished aio 0x7f48403fe680 r 524288 ioc 0x7f485026eb28 with aios left -13> 2017-01-25 23:52:14.783630 7f48167a3700 20 bdev(/var/lib/ceph/osd/ceph-4/block) aio_submit aio 0x7f4849f00010 fd 23 0x241790000~80000 -12> 2017-01-25 23:52:14.783713 7f48267c3700 20 bdev(/var/lib/ceph/osd/ceph-4/block) _aio_log_finish 1 0x241710000~80000 -11> 2017-01-25 23:52:14.783719 7f48267c3700 10 bdev(/var/lib/ceph/osd/ceph-4/block) _aio_thread finished aio 0x7f48404002e0 r 524288 ioc 0x7f485026eb28 with aios left -10> 2017-01-25 23:52:14.790490 7f48267c3700 20 bdev(/var/lib/ceph/osd/ceph-4/block) _aio_log_finish 1 0x241790000~80000 -9> 2017-01-25 23:52:14.790490 7f48167a3700 20 bdev(/var/lib/ceph/osd/ceph-4/block) aio_submit aio 0x7f4849f01780 fd 23 0x241810000~80000 -8> 2017-01-25 23:52:14.790498 7f48267c3700 10 bdev(/var/lib/ceph/osd/ceph-4/block) _aio_thread finished aio 0x7f4849f00010 r -28 ioc 0x7f485026eb28 with aios left
/a/sage-2017-01-25_23:09:16-rados-wip-sage-testing---basic-smithi/748443
As far as I can tell the disk was far from empty:
2017-01-25 23:52:05.834245 mon.0 172.21.15.70:6789/0 1933 : cluster [INF] pgmap v620: 160 pgs: 3 peering, 10 creating, 147 active+clean; 13372 MB data, 22095 MB used, 428 GB / 450 GB avail
and
-6182> 2017-01-25 23:52:14.271402 7f481279b700 20 bluestore(/var/lib/ceph/osd/ceph-4) statfsstore_statfs(0x151f6b0000/0x1680000000, stored 0x156d45da4/0x159950000, compress 0x0/0x0/0x0) -6181> 2017-01-25 23:52:14.271445 7f481279b700 20 osd.4 362 update_osd_stat osd_stat(5641 MB used, 86518 MB avail, 92160 MB total, peers [0,1,2,3,5] op hist [])
so I'm not sure why XFS gave us ENOSPC.
In any case, I think this is an issues mainly specific to the lab (where we run on top of an XFS file and not a raw block device), so it's not super-critical. Let's see if it happens again, though.
Actions