Bug #15050
closeddeleting striped file in cephfs doesn't free up file's space
0%
Description
I've been creating large (16-64 GB) files in an otherwise empty test cephfs (mounted via the kernel client in linux 4.3.3 (CoreOS 899.5.0)). When a file is deleted, it usually takes 5-10 seconds for the file's space to be reclaimed (df still shows the space used during this period) - but eventually df shows the FS has no space used. However, if I create a directory and change the striping to non-default (ceph.dir.layout="stripe_unit=1048576 stripe_count=4 object_size=8388608 pool=cephfs_data"), then create & delete files, that space is never fully reclaimed.
For example, I've created a 64 GB striped file (confirmed with getfattr to match the above directory layout), then deleted it. I see some space (maybe as much as 16 GB) reclaimed, but the rest stays in limbo - even after a couple days pass. There are no cache pools here; just a single MDS, single MON, and 3x nodes with 3x OSDs each. Both data and metadata pools have size=2, and 128 and 64 PGs respectively.
After doing this several times, I have a filesystem where du claims 270k are in use but df claims 394 GB are used (and rados confirms about 200 GB of objects in my cephfs_data pool).
Am I doing something wrong here? Is there any way to find orphaned FS objects, or a cron job I'm omitting to automatically purge stuff like this? Or is this a bug?
Updated by Greg Farnum about 8 years ago
Can you describe in a little more detail how you were creating and deleting these files? You talk about them being 10s of GB — were you writing out data at a very large offset to force them to that size? Actually writing out that much data?
Can you dump the cache on your MDS (it's an admin socket command) and upload it with ceph-post-file? And also dump the perfcounters and share them?
Stuff should be automatically deleted, but neither of your descriptions of time (<10 seconds, nor multiple days) really make sense for files of this size. :/ We did add a "StrayManager" in Infernalis that prevents the MDS from spewing out an infinite number of delete ops at the OSDs which could be going wrong, but it hasn't had any failure reports so far.
Updated by Zheng Yan about 8 years ago
- Status changed from New to 12
we don't handle 'stripe_count > 1' properly when purging stray.
if (in->is_file()) {
uint64_t period = (uint64_t)in->inode.layout.fl_object_size *
(uint64_t)in->inode.layout.fl_stripe_count;
uint64_t to = in->inode.get_max_size();
to = MAX(in->inode.size, to);
// when truncating a file, the filer does not delete stripe objects that are
// truncated to zero. so we need to purge stripe objects up to the max size
// the file has ever been.
to = MAX(in->inode.max_size_ever, to);
if (to && period) {
uint64_t num = (to + period - 1) / period;
^^^^
dout(10) << __func__ << " 0~" << to << " objects 0~" << num
<< " snapc " << snapc << " on " << *in << dendl;
filer.purge_range(in->inode.ino, &in->inode.layout, *snapc,
0, num, ceph::real_clock::now(g_ceph_context), 0,
gather.new_sub());
}
}
Updated by Greg Farnum about 8 years ago
- Assignee set to Zheng Yan
- Priority changed from Normal to High
Well spotted.
Updated by Loïc Dachary about 8 years ago
- Subject changed from Infernalis: deleting striped file in cephfs doesn't free up file's space to deleting striped file in cephfs doesn't free up file's space
Updated by Zheng Yan about 8 years ago
- Status changed from 12 to Fix Under Review
Updated by Nathan Cutler about 8 years ago
- Copied to Backport #15056: hammer: deleting striped file in cephfs doesn't free up file's space added
Updated by Nathan Cutler about 8 years ago
- Copied to Backport #15057: infernalis: deleting striped file in cephfs doesn't free up file's space added
Updated by Greg Farnum about 8 years ago
- Status changed from Fix Under Review to Pending Backport
It's been merged.
Updated by Greg Farnum about 8 years ago
- Priority changed from High to Normal
Just waiting on backports, FS team doesn't need to worry about those.
Updated by John Spray almost 8 years ago
Florent: yes, you can check that kind of thing yourself with a git clone and something like:
git log --grep "mds: fix stray purging in 'stripe_count > 1' case" v10.2.1
Updated by Loïc Dachary over 7 years ago
- Status changed from Pending Backport to Resolved