Ceph » RADOS

   -14> 2017-01-25 23:52:14.783592 7f48267c3700 10 bdev(/var/lib/ceph/osd/ceph-4/block) _aio_thread finished aio 0x7f48403fe680 r 524288 ioc 0x7f485026eb28 with  aios left
   -13> 2017-01-25 23:52:14.783630 7f48167a3700 20 bdev(/var/lib/ceph/osd/ceph-4/block) aio_submit  aio 0x7f4849f00010 fd 23 0x241790000~80000
   -12> 2017-01-25 23:52:14.783713 7f48267c3700 20 bdev(/var/lib/ceph/osd/ceph-4/block) _aio_log_finish 1 0x241710000~80000
   -11> 2017-01-25 23:52:14.783719 7f48267c3700 10 bdev(/var/lib/ceph/osd/ceph-4/block) _aio_thread finished aio 0x7f48404002e0 r 524288 ioc 0x7f485026eb28 with  aios left
   -10> 2017-01-25 23:52:14.790490 7f48267c3700 20 bdev(/var/lib/ceph/osd/ceph-4/block) _aio_log_finish 1 0x241790000~80000
    -9> 2017-01-25 23:52:14.790490 7f48167a3700 20 bdev(/var/lib/ceph/osd/ceph-4/block) aio_submit  aio 0x7f4849f01780 fd 23 0x241810000~80000
    -8> 2017-01-25 23:52:14.790498 7f48267c3700 10 bdev(/var/lib/ceph/osd/ceph-4/block) _aio_thread finished aio 0x7f4849f00010 r -28 ioc 0x7f485026eb28 with  aios left

/a/sage-2017-01-25_23:09:16-rados-wip-sage-testing---basic-smithi/748443

As far as I can tell the disk was far from empty:

2017-01-25 23:52:05.834245 mon.0 172.21.15.70:6789/0 1933 : cluster [INF] pgmap v620: 160 pgs: 3 peering, 10 creating, 147 active+clean; 13372 MB data, 22095 MB used, 428 GB / 450 GB avail

and

 -6182> 2017-01-25 23:52:14.271402 7f481279b700 20 bluestore(/var/lib/ceph/osd/ceph-4) statfsstore_statfs(0x151f6b0000/0x1680000000, stored 0x156d45da4/0x159950000, compress 0x0/0x0/0x0)
 -6181> 2017-01-25 23:52:14.271445 7f481279b700 20 osd.4 362 update_osd_stat osd_stat(5641 MB used, 86518 MB avail, 92160 MB total, peers [0,1,2,3,5] op hist [])

so I'm not sure why XFS gave us ENOSPC.

In any case, I think this is an issues mainly specific to the lab (where we run on top of an XFS file and not a raw block device), so it's not super-critical. Let's see if it happens again, though.