Bug #39646: ceph-mgr log is flooded with pgmap info every two seconds - mgr - Ceph

Actions

Copy link

Bug #39646

closed

ceph-mgr log is flooded with pgmap info every two seconds

Added by Ricardo Dias almost 5 years ago. Updated almost 2 years ago.

Status:

Won't Fix

Priority:

High

Assignee:

Category:

ceph-mgr

Target version:

% Done:

Source:

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

Ceph - v14.2.0

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

The ceph-mgr daemon with debug-mgr=0 logs pgmap information every two second:

Example:

2019-05-08 17:15:54.443 7fc43d8b5700  0 log_channel(cluster) log [DBG] : pgmap v188: 480 pgs: 480 active+clean; 4.7 KiB data, 71 MiB used, 216 GiB / 228 GiB avail; 5.0 KiB/s rd, 4 op/s
2019-05-08 17:15:56.443 7fc43d8b5700  0 log_channel(cluster) log [DBG] : pgmap v189: 480 pgs: 480 active+clean; 4.7 KiB data, 71 MiB used, 216 GiB / 228 GiB avail; 3.3 KiB/s rd, 3 op/s
2019-05-08 17:15:58.443 7fc43d8b5700  0 log_channel(cluster) log [DBG] : pgmap v190: 480 pgs: 480 active+clean; 4.7 KiB data, 71 MiB used, 216 GiB / 228 GiB avail; 5.0 KiB/s rd, 4 op/s
2019-05-08 17:16:00.443 7fc43d8b5700  0 log_channel(cluster) log [DBG] : pgmap v191: 480 pgs: 480 active+clean; 4.7 KiB data, 71 MiB used, 216 GiB / 228 GiB avail; 3.3 KiB/s rd, 3 op/s
2019-05-08 17:16:02.443 7fc43d8b5700  0 log_channel(cluster) log [DBG] : pgmap v192: 480 pgs: 480 active+clean; 4.7 KiB data, 71 MiB used, 216 GiB / 228 GiB avail; 3.3 KiB/s rd, 3 op/s
2019-05-08 17:16:04.447 7fc43d8b5700  0 log_channel(cluster) log [DBG] : pgmap v193: 480 pgs: 480 active+clean; 4.7 KiB data, 71 MiB used, 216 GiB / 228 GiB avail; 5.0 KiB/s rd, 4 op/s

Do we really need to show this info in the log?

Actions

Copy link

Updated by Lenz Grimmer almost 5 years ago

Is this related to #37886 by any chance?

Actions

Copy link

Updated by Sebastian Wagner almost 5 years ago

I'd be :+1: for remove this, Iff creating a new pgmap every two seconds is not a bug.

Actions

Copy link

Updated by Vikhyat Umrao almost 5 years ago

Hi - This is not a bug this was changed because of some reason during troubleshooting/RCA we need previous historic IOPS data. If you want to stop these messages just simply change the log level to `info`.

ceph tell mon.* injectargs '--mon_cluster_log_file_level info'

Actions

Copy link

Updated by Sebastian Wagner almost 5 years ago

Vikhyat Umrao wrote:

Hi - This is not a bug this was changed because of some reason during troubleshooting/RCA we need previous historic IOPS data. If you want to stop these messages just simply change the log level to `info`.

[...]

Interesting, thanks for the backgroud. To me, this raises the question, if there are better places to store history IOPS data, like prometheus and if the MGR log file is really the best place for this. Especially as this pgmap output is spamming the mgr log file in vstart clusters.

Actions

Copy link

Updated by Vikhyat Umrao almost 5 years ago

Sebastian Wagner wrote:

Vikhyat Umrao wrote:

Hi - This is not a bug this was changed because of some reason during troubleshooting/RCA we need previous historic IOPS data. If you want to stop these messages just simply change the log level to `info`.

[...]

Interesting, thanks for the backgroud. To me, this raises the question, if there are better places to store history IOPS data, like prometheus and if the MGR log file is really the best place for this. Especially as this pgmap output is spamming the mgr log file in vstart clusters.

I think you can fix vstart issue by adding mon_cluster_log_file_level = info in vstart.sh [global] section.

Actions

Copy link