Project

General

Profile

Actions

Bug #49902

open

monitor store.db directory has been increasing and has exceeded 450GB

Added by Hughen X about 3 years ago. Updated almost 3 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
monitor big
Backport:
Regression:
No
Severity:
1 - critical
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

My monitor keeps growing. After it exceeds 300GB, it cannot be started because the disk is not enough. After the space is restored, two (mon-t, mon-u) of the original three monitor (mon-r, mon-t, mon-u) services cannot work normally, and it has been in the electing stage.

Then I select mon-t data and perform the following operations:

ceph-mon -i t --mon-data /var/lib/ceph/mon/ceph-t/data --extract-monmap /tmp/monmap

monmaptool /tmp/monmap --rm r
monmaptool /tmp/monmap --rm u

ceph-mon -i t --mon-data /var/lib/ceph/mon/ceph-t/data --inject-monmap /tmp/monmap

Finally, restart mon-t, but the store.db directory is still growing, and it has exceeded 400GB in 5 hours. During the entire time, in addition to the short up/down of some osd, ceph also accompanied a large number of warnings about slow ops on mon-t, and all pgs were in active+clean and scrubbing state.

And neither the offline ceph-monstore-tool nor the online compact command can make the space occupied by mon-t smaller

Ceph version: v14.2.11

Actions #1

Updated by Sage Weil almost 3 years ago

  • Project changed from Ceph to RADOS
  • Category deleted (Monitor)
Actions

Also available in: Atom PDF