Project

General

Profile

Actions
Copy link

Bug #39704

closed

When running multiple filesystems, directories do not fragment

Added by Nathan Fish almost 5 years ago. Updated almost 5 years ago.

Status:
Won't Fix
Priority:
Normal
Assignee:
-
Category:
Performance/Resource Usage
Target version:
Ceph - v15.0.0
% Done:

0%

Source:
Community (user)
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
fs
Component(FS):
Labels (FS):
multifs
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Nautilus, Ubuntu 18.04.2, HWE kernel 4.18.0-18-generic.
I have created multiple ceph filesystems:

root@mc-3015-201:/mnt/ceph1# ceph fs ls
name: cephfs_bs_lvm, metadata pool: cephfs_metadata, data pools: [cephfs_data ]
name: cephfs1, metadata pool: cephfs1_metadata, data pools: [cephfs1_data ]
name: cephfs2, metadata pool: cephfs2_metadata, data pools: [cephfs2_data ]
name: cephfs3, metadata pool: cephfs3_metadata, data pools: [cephfs3_data ]
name: cephfs4, metadata pool: cephfs4_metadata, data pools: [cephfs4_data ]

I can create 1 million files in cephfs_bs_lvm, the first one created.
But when I try to do the same on any of the others, I get ENOSPC at exactly 100k.
This implies I am hitting mds_bal_fragment_size_max, and that the directories are not fragmenting.

Please let me know if any other information is required.

Actions
Copy link
 #1

Updated by Patrick Donnelly almost 5 years ago

  • Status changed from New to Need More Info
  • Target version changed from v14.2.2 to v15.0.0
  • Start date deleted (05/10/2019)

This is with Nautilus v14.2.1? Can you bump up debugging on the MDS during the event and share the log?

Actions
Copy link
 #2

Updated by Nathan Fish almost 5 years ago

Patrick Donnelly wrote:

This is with Nautilus v14.2.1? Can you bump up debugging on the MDS during the event and share the log?

root@m3-3101-422:/etc/ceph# ceph versions {
"mon": {
"ceph version 14.2.1 (d555a9489eb35f84f2e1ef49b77e19da9d113972) nautilus (stable)": 3
},
"mgr": {
"ceph version 14.2.1 (d555a9489eb35f84f2e1ef49b77e19da9d113972) nautilus (stable)": 3
},
"osd": {
"ceph version 14.2.1 (d555a9489eb35f84f2e1ef49b77e19da9d113972) nautilus (stable)": 63
},
"mds": {
"ceph version 14.2.1 (d555a9489eb35f84f2e1ef49b77e19da9d113972) nautilus (stable)": 12
},
"overall": {
"ceph version 14.2.1 (d555a9489eb35f84f2e1ef49b77e19da9d113972) nautilus (stable)": 81
}
}

Edited ceph.conf:

[mds]
  1. https://tracker.ceph.com/issues/39704
    debug mds = 5
    debug mds balancer = 5

Even debug 5 shows nothing when trying to touch a new file while at max. I'll delete all of them and start creating them again:

root@mc-3015-201:/mnt/ceph1# find . type f -delete
root@mc-3015-201:/mnt/ceph1# time parallel -i sh -c "fastrandom | split -d -b2 -a5 - {}" - {0..9}

Here's the log from the active mds on cephfs1; it's large uncompressed so it's a .gz.
https://vault.cs.uwaterloo.ca/s/PFsXSEDqXTSyjdr/download

I don't see anything useful there myself... Any other logs or tests that would help?

Actions
Copy link
 #3

Updated by Zheng Yan almost 5 years ago

the log show you were creating files in root directory. mds never fragment root directory.

Actions
Copy link
 #4

Updated by Nathan Fish almost 5 years ago

Zheng Yan wrote:

the log show you were creating files in root directory. mds never fragment root directory.

I see! That's a difference between the filesystems which I didn't even think about. I just tested with a subdirectory and I could create 1 million files. I guess this can be closed; I may make a PR to update the documentation later.

Actions
Copy link
 #5

Updated by Patrick Donnelly almost 5 years ago

  • Status changed from Need More Info to Won't Fix
Actions
Copy link

Also available in: Atom PDF