Project

General

Profile

Bug #37726

mds: high debug logging with many subtrees is slow

Added by Patrick Donnelly 11 months ago. Updated 19 days ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Category:
Performance/Resource Usage
Target version:
Start date:
Due date:
% Done:

0%

Source:
Development
Tags:
Backport:
nautilus,mimic,luminous
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
MDS
Labels (FS):
Pull request ID:
Crash signature:

Description

In various places the MDS prints subtrees to the debug log. We should truncate the list if the number of subtrees is large.


Related issues

Copied to fs - Backport #38875: mimic: mds: high debug logging with many subtrees is slow Resolved
Copied to fs - Backport #38876: nautilus: mds: high debug logging with many subtrees is slow Resolved
Copied to fs - Backport #38877: luminous: mds: high debug logging with many subtrees is slow Resolved

History

#1 Updated by Patrick Donnelly 10 months ago

  • Assignee changed from Patrick Donnelly to Rishabh Dave

#2 Updated by Rishabh Dave 10 months ago

I created 100 directories and wrote 10 files in each directory after pinning all the directories on a MDS. If I reproduced it correctly this must be the is the huge subtree you were talking about - https://paste.fedoraproject.org/paste/3E5EtAAEcdqOHovpxMWTlw ?

What should the threshold be for deciding that the number of subtrees is huge? And since we would be truncating the list, do we have a preference on which subtrees we would like to print? Also, at some other log level do we want to print all the subtrees?

#3 Updated by Patrick Donnelly 10 months ago

  • Status changed from New to In Progress

Rishabh Dave wrote:

I created 100 directories and wrote 10 files in each directory after pinning all the directories on a MDS. If I reproduced it correctly this must be the is the huge subtree you were talking about - https://paste.fedoraproject.org/paste/3E5EtAAEcdqOHovpxMWTlw ?

Yes, that's it!

What should the threshold be for deciding that the number of subtrees is huge? And since we would be truncating the list, do we have a preference on which subtrees we would like to print? Also, at some other log level do we want to print all the subtrees?

If the list is large, I think the default should be printing the number of subtrees and maybe some other metadata in a single message. Maybe if debugging is >=25 then we should always print the full list.

I think MDBalancer::handle_heartbeat could be the place that prints the full list. That should happen every few seconds.

#4 Updated by Patrick Donnelly 8 months ago

  • Target version changed from v14.0.0 to v15.0.0

#5 Updated by Patrick Donnelly 8 months ago

  • Category set to Performance/Resource Usage
  • Target version changed from v15.0.0 to v14.0.0

#6 Updated by Patrick Donnelly 8 months ago

  • Target version changed from v14.0.0 to v15.0.0
  • Backport changed from mimic,luminous to nautilus,mimic,luminous
  • Pull request ID set to 26056

#7 Updated by Patrick Donnelly 8 months ago

  • Status changed from In Progress to Pending Backport

#8 Updated by Nathan Cutler 8 months ago

  • Copied to Backport #38875: mimic: mds: high debug logging with many subtrees is slow added

#9 Updated by Nathan Cutler 8 months ago

  • Copied to Backport #38876: nautilus: mds: high debug logging with many subtrees is slow added

#10 Updated by Nathan Cutler 8 months ago

  • Copied to Backport #38877: luminous: mds: high debug logging with many subtrees is slow added

#11 Updated by Nathan Cutler 5 months ago

  • Status changed from Pending Backport to Resolved

#12 Updated by Nathan Cutler 5 months ago

  • Status changed from Resolved to Pending Backport

mimic backport is still open

#13 Updated by Nathan Cutler 19 days ago

  • Status changed from Pending Backport to Resolved

While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are in status "Resolved" or "Rejected".

Also available in: Atom PDF