Project

General

Profile

Bug #15050

deleting striped file in cephfs doesn't free up file's space

Added by Mike Shuey almost 3 years ago. Updated over 2 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
-
Target version:
-
Start date:
03/10/2016
Due date:
% Done:

0%

Source:
other
Tags:
Backport:
hammer,infernalis
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
kcephfs
Component(FS):
Labels (FS):
Pull request ID:

Description

I've been creating large (16-64 GB) files in an otherwise empty test cephfs (mounted via the kernel client in linux 4.3.3 (CoreOS 899.5.0)). When a file is deleted, it usually takes 5-10 seconds for the file's space to be reclaimed (df still shows the space used during this period) - but eventually df shows the FS has no space used. However, if I create a directory and change the striping to non-default (ceph.dir.layout="stripe_unit=1048576 stripe_count=4 object_size=8388608 pool=cephfs_data"), then create & delete files, that space is never fully reclaimed.

For example, I've created a 64 GB striped file (confirmed with getfattr to match the above directory layout), then deleted it. I see some space (maybe as much as 16 GB) reclaimed, but the rest stays in limbo - even after a couple days pass. There are no cache pools here; just a single MDS, single MON, and 3x nodes with 3x OSDs each. Both data and metadata pools have size=2, and 128 and 64 PGs respectively.

After doing this several times, I have a filesystem where du claims 270k are in use but df claims 394 GB are used (and rados confirms about 200 GB of objects in my cephfs_data pool).

Am I doing something wrong here? Is there any way to find orphaned FS objects, or a cron job I'm omitting to automatically purge stuff like this? Or is this a bug?


Related issues

Copied to fs - Backport #15056: hammer: deleting striped file in cephfs doesn't free up file's space Resolved
Copied to fs - Backport #15057: infernalis: deleting striped file in cephfs doesn't free up file's space Rejected

Associated revisions

Revision 99ec1839 (diff)
Added by Yan, Zheng almost 3 years ago

mds: fix stray purging in 'stripe_count > 1' case

Fixes: #15050
Signed-off-by: Yan, Zheng <>

Revision 3ecdedd5 (diff)
Added by Yan, Zheng almost 3 years ago

mds: fix stray purging in 'stripe_count > 1' case

Fixes: #15050
Signed-off-by: Yan, Zheng <>
(cherry picked from commit 99ec183973e0f4686673d93df02e5381543be51f)

History

#1 Updated by Nathan Cutler almost 3 years ago

  • Project changed from Ceph to fs

#2 Updated by Greg Farnum almost 3 years ago

Can you describe in a little more detail how you were creating and deleting these files? You talk about them being 10s of GB — were you writing out data at a very large offset to force them to that size? Actually writing out that much data?

Can you dump the cache on your MDS (it's an admin socket command) and upload it with ceph-post-file? And also dump the perfcounters and share them?

Stuff should be automatically deleted, but neither of your descriptions of time (<10 seconds, nor multiple days) really make sense for files of this size. :/ We did add a "StrayManager" in Infernalis that prevents the MDS from spewing out an infinite number of delete ops at the OSDs which could be going wrong, but it hasn't had any failure reports so far.

#3 Updated by Zheng Yan almost 3 years ago

  • Status changed from New to Verified

we don't handle 'stripe_count > 1' properly when purging stray.

  if (in->is_file()) {
    uint64_t period = (uint64_t)in->inode.layout.fl_object_size *
                      (uint64_t)in->inode.layout.fl_stripe_count;
    uint64_t to = in->inode.get_max_size();
    to = MAX(in->inode.size, to);
    // when truncating a file, the filer does not delete stripe objects that are
    // truncated to zero. so we need to purge stripe objects up to the max size
    // the file has ever been.
    to = MAX(in->inode.max_size_ever, to);
    if (to && period) {
      uint64_t num = (to + period - 1) / period;
               ^^^^
      dout(10) << __func__ << " 0~" << to << " objects 0~" << num
               << " snapc " << snapc << " on " << *in << dendl;
      filer.purge_range(in->inode.ino, &in->inode.layout, *snapc,
                        0, num, ceph::real_clock::now(g_ceph_context), 0,
                        gather.new_sub());
    }
  }

#4 Updated by Greg Farnum almost 3 years ago

  • Assignee set to Zheng Yan
  • Priority changed from Normal to High

Well spotted.

#5 Updated by Loic Dachary almost 3 years ago

  • Subject changed from Infernalis: deleting striped file in cephfs doesn't free up file's space to deleting striped file in cephfs doesn't free up file's space

#7 Updated by Nathan Cutler almost 3 years ago

  • Backport set to hammer,infernalis

#8 Updated by Nathan Cutler almost 3 years ago

  • Copied to Backport #15056: hammer: deleting striped file in cephfs doesn't free up file's space added

#9 Updated by Nathan Cutler almost 3 years ago

  • Copied to Backport #15057: infernalis: deleting striped file in cephfs doesn't free up file's space added

#10 Updated by Greg Farnum almost 3 years ago

  • Status changed from Need Review to Pending Backport

It's been merged.

#11 Updated by Greg Farnum almost 3 years ago

  • Priority changed from High to Normal

Just waiting on backports, FS team doesn't need to worry about those.

#12 Updated by Florent B over 2 years ago

Is it fixed in v10.2.1 ?

#13 Updated by John Spray over 2 years ago

Florent: yes, you can check that kind of thing yourself with a git clone and something like:

git log --grep "mds: fix stray purging in 'stripe_count > 1' case" v10.2.1

#14 Updated by Loic Dachary over 2 years ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF