Project

General

Profile

Bug #14537

should compact full epochs in monitor

Added by Kefu Chai almost 3 years ago. Updated over 2 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Monitor
Target version:
-
Start date:
01/28/2016
Due date:
% Done:

0%

Source:
Support
Tags:
Backport:
hammer,infernalis
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:

Description

it is reported that there is inconsistency between mon store leveldbs in the same cluster.

4.0G /var/lib/ceph/mon
219M /var/lib/ceph/mon
5.3G /var/lib/ceph/mon
5.3G /var/lib/ceph/mon
5.3G /var/lib/ceph/mon

this issue was originally spotted in a firefly cluster, but the symptom persists in hammer. when time elapses the size keep increasing:

5.6G /var/lib/ceph/mon
2.6G /var/lib/ceph/mon
7.7G /var/lib/ceph/mon
7.7G /var/lib/ceph/mon
7.7G /var/lib/ceph/mon

after compaction using "ceph tell mon.* compact" or ceph-monstore-tool, the larger ones shrank to 260MB.

we found leveldb 1.12 has a relevant bug https://groups.google.com/forum/#!msg/leveldb/yL6h1mAOc20/vLU64RylIdMJ. but we have an option "mon_compact_on_trim" for compacting the trimmed epochs. and it's "true" by default. so the space used by the trimmed epochs should always be compacted and reclaimed even without the bug fix from leveldb.

in other words, we do call:
CompactRange(prefix, trimmed_incremental_begin-1, trimmed_incremental_end)

but seems not all trimmed range is compacted. assuming prefix is "osdmap", begin is "123", end is "373", the range being compacted is like:

osdmap\0123..osdmap\0373

which might not cover the range of

osdmap\0full_123..osdmap\0full_373

and as https://groups.google.com/forum/#!msg/leveldb/yL6h1mAOc20/vLU64RylIdMJ put, the key range no longer touched will not be reclaimed.

so i guess we need to compact the full epochs also.


Related issues

Copied to Ceph - Backport #14580: infernalis: should compact full epochs in monitor Rejected
Copied to Ceph - Backport #14581: hammer: should compact full epochs in monitor Resolved

Associated revisions

Revision 93d633a2 (diff)
Added by Kefu Chai almost 3 years ago

mon: compact full epochs also

by compacting the ${prefix}.${start}..${prefix}..${end} does not
necessary compact the range of ${prefix}."full_"${start}..
${prefix}."full_"${end}. so when more and more epochs get trimmed
with out a full range compaction, the size of monitor store could
be very large.

Fixes: #14537
Signed-off-by: Kefu Chai <>

Revision 91093041 (diff)
Added by Kefu Chai almost 3 years ago

mon: compact full epochs also

by compacting the ${prefix}.${start}..${prefix}..${end} does not
necessary compact the range of ${prefix}."full_"${start}..
${prefix}."full_"${end}. so when more and more epochs get trimmed
with out a full range compaction, the size of monitor store could
be very large.

Fixes: #14537
Signed-off-by: Kefu Chai <>
(cherry picked from commit 93d633a25ad8c36c972bb766c38187e2612041e1)

History

#1 Updated by Kefu Chai almost 3 years ago

  • Status changed from In Progress to Need Review

#2 Updated by Sage Weil almost 3 years ago

  • Status changed from Need Review to Pending Backport

#3 Updated by Nathan Cutler almost 3 years ago

  • Backport changed from hammer, jewel to hammer,infernalis

#4 Updated by Kefu Chai almost 3 years ago

hammer backport was merged at https://github.com/ceph/ceph/pull/7446

#5 Updated by Loic Dachary almost 3 years ago

  • Copied to Backport #14580: infernalis: should compact full epochs in monitor added

#6 Updated by Loic Dachary almost 3 years ago

  • Copied to Backport #14581: hammer: should compact full epochs in monitor added

#7 Updated by Loic Dachary over 2 years ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF