Project

General

Profile

Actions
Copy link

Bug #13135

closed

high mon memory usage after upgrade firefly -> hammer

Added by Corin Langosch over 8 years ago. Updated over 8 years ago.

Status:
Rejected
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
other
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Yesterday I upgraded a firefly cluster to hammer 0.94.3. More info about my cluster and another issue can be found here: http://tracker.ceph.com/issues/13134

I noticed the mon daemons are constantly growing in memory usage, starting with 50-80 MB and then growing to over 2000 MB (and then I restarted them). The cluster was always in healthy state after restarting all mons for the last time. But after only about 30 minutes memory usage of for example mon.c grew from 80MB to 190MB. Here are the memory usages:

root@r-ch103:~# ps aux | grep ceph
root     12081  2.9  0.9 1020820 640492 ?      Sl   Sep16  19:16 /usr/bin/ceph-mon -i a --pid-file /var/run/ceph/mon.a.pid -c /etc/ceph/ceph.conf --cluster ceph
root     19382 23.2  0.7 1637528 525732 ?      Ssl  Sep16 141:49 /usr/bin/ceph-osd -i 16 --pid-file /var/run/ceph/osd.16.pid -c /etc/ceph/ceph.conf --cluster ceph
root     27656  0.0  0.0   9388   928 pts/0    S+   08:18   0:00 grep --color=auto ceph
root     28243 21.1  0.7 1663072 498556 ?      Ssl  Sep16 125:01 /usr/bin/ceph-osd -i 3 --pid-file /var/run/ceph/osd.3.pid -c /etc/ceph/ceph.conf --cluster ceph
root@r-ch103:~# /etc/init.d/ceph restart mon
=== mon.a ===
=== mon.a ===
Stopping Ceph mon.a on r-ch103...kill 12081...done
=== mon.a ===
Starting Ceph mon.a on r-ch103...
2015-09-17 08:19:29.843757 7eff5a918780 -1 compacting monitor store ...
Starting ceph-create-keys on r-ch103...
root@r-ch103:~# ps aux | grep ceph
root     19382 23.2  0.7 1637528 524852 ?      Ssl  Sep16 142:13 /usr/bin/ceph-osd -i 16 --pid-file /var/run/ceph/osd.16.pid -c /etc/ceph/ceph.conf --cluster ceph
root     28068  5.0  0.0 224456 36252 pts/0    Sl   08:19   0:01 /usr/bin/ceph-mon -i a --pid-file /var/run/ceph/mon.a.pid -c /etc/ceph/ceph.conf --cluster ceph
root     28205  0.0  0.0   9388   932 pts/0    S+   08:20   0:00 grep --color=auto ceph
root     28243 21.1  0.7 1662304 497800 ?      Ssl  Sep16 125:22 /usr/bin/ceph-osd -i 3 --pid-file /var/run/ceph/osd.3.pid -c /etc/ceph/ceph.conf --cluster ceph
root@r-ch103:~# ps aux | grep ceph
root     19382 23.2  0.7 1637016 508476 ?      Ssl  Sep16 143:07 /usr/bin/ceph-osd -i 16 --pid-file /var/run/ceph/osd.16.pid -c /etc/ceph/ceph.conf --cluster ceph
root     28068  2.0  0.0 252144 56416 pts/0    Sl   08:19   0:05 /usr/bin/ceph-mon -i a --pid-file /var/run/ceph/mon.a.pid -c /etc/ceph/ceph.conf --cluster ceph
root     28243 21.1  0.7 1663072 478140 ?      Ssl  Sep16 126:15 /usr/bin/ceph-osd -i 3 --pid-file /var/run/ceph/osd.3.pid -c /etc/ceph/ceph.conf --cluster ceph
root     28667  0.0  0.0   9388   932 pts/0    S+   08:23   0:00 grep --color=auto ceph
root@r-ch103:~# ps aux | grep ceph
root     19382 23.2  0.7 1637528 482944 ?      Ssl  Sep16 145:13 /usr/bin/ceph-osd -i 16 --pid-file /var/run/ceph/osd.16.pid -c /etc/ceph/ceph.conf --cluster ceph
root     28068  1.8  0.1 276136 80460 pts/0    Sl   08:19   0:15 /usr/bin/ceph-mon -i a --pid-file /var/run/ceph/mon.a.pid -c /etc/ceph/ceph.conf --cluster ceph
root     28243 21.1  0.6 1663072 460252 ?      Ssl  Sep16 128:12 /usr/bin/ceph-osd -i 3 --pid-file /var/run/ceph/osd.3.pid -c /etc/ceph/ceph.conf --cluster ceph
root     29426  0.0  0.0   9388   932 pts/0    S+   08:33   0:00 grep --color=auto ceph
root@r-ch103:~# ps aux | grep ceph
root     19382 23.2  0.7 1637528 461884 ?      Ssl  Sep16 148:52 /usr/bin/ceph-osd -i 16 --pid-file /var/run/ceph/osd.16.pid -c /etc/ceph/ceph.conf --cluster ceph
root     28068  1.9  0.2 329844 134076 pts/0   Sl   08:19   0:34 /usr/bin/ceph-mon -i a --pid-file /var/run/ceph/mon.a.pid -c /etc/ceph/ceph.conf --cluster ceph
root     28243 21.1  0.6 1662176 452392 ?      Ssl  Sep16 131:33 /usr/bin/ceph-osd -i 3 --pid-file /var/run/ceph/osd.3.pid -c /etc/ceph/ceph.conf --cluster ceph
root     30495  0.0  0.0   9388   928 pts/0    S+   08:49   0:00 grep --color=auto ceph
root@r-ch103:~# ps aux | grep ceph
root     19382 23.2  0.6 1637528 453500 ?      Ssl  Sep16 151:13 /usr/bin/ceph-osd -i 16 --pid-file /var/run/ceph/osd.16.pid -c /etc/ceph/ceph.conf --cluster ceph
root     28068  2.0  0.2 368436 171068 pts/0   Sl   08:19   0:48 /usr/bin/ceph-mon -i a --pid-file /var/run/ceph/mon.a.pid -c /etc/ceph/ceph.conf --cluster ceph
root     28243 21.1  0.6 1663072 431108 ?      Ssl  Sep16 133:47 /usr/bin/ceph-osd -i 3 --pid-file /var/run/ceph/osd.3.pid -c /etc/ceph/ceph.conf --cluster ceph
root     31300  0.0  0.0   9388   932 pts/0    S+   08:59   0:00 grep --color=auto ceph

root@r-ch104:~# ps aux | grep ceph
root     13772  0.0  0.0   9384   928 pts/0    S+   08:18   0:00 grep --color=auto ceph
root     23698  1.7  3.6 2659996 2416880 ?     Sl   Sep16  11:24 /usr/bin/ceph-mon -i b --pid-file /var/run/ceph/mon.b.pid -c /etc/ceph/ceph.conf --cluster ceph
root     26540 20.7  0.7 1620936 504968 ?      Ssl  Sep16 131:21 /usr/bin/ceph-osd -i 17 --pid-file /var/run/ceph/osd.17.pid -c /etc/ceph/ceph.conf --cluster ceph
root     29198 22.3  0.7 1728640 510976 ?      Ssl  Sep16 136:12 /usr/bin/ceph-osd -i 6 --pid-file /var/run/ceph/osd.6.pid -c /etc/ceph/ceph.conf --cluster ceph
root@r-ch104:~# /etc/init.d/ceph restart mon
=== mon.b ===
=== mon.b ===
Stopping Ceph mon.b on r-ch104...kill 23698...done
=== mon.b ===
Starting Ceph mon.b on r-ch104...
2015-09-17 08:19:18.013282 7fa8e167a780 -1 compacting monitor store ...
Starting ceph-create-keys on r-ch104...
root@r-ch104:~# ps aux | grep ceph
root     14082  3.9  0.0 326068 57272 pts/0    Sl   08:19   0:01 /usr/bin/ceph-mon -i b --pid-file /var/run/ceph/mon.b.pid -c /etc/ceph/ceph.conf --cluster ceph
root     14336  0.0  0.0   9384   932 pts/0    S+   08:20   0:00 grep --color=auto ceph
root     26540 20.7  0.7 1620936 487968 ?      Ssl  Sep16 131:41 /usr/bin/ceph-osd -i 17 --pid-file /var/run/ceph/osd.17.pid -c /etc/ceph/ceph.conf --cluster ceph
root     29198 22.3  0.7 1728640 511332 ?      Ssl  Sep16 136:34 /usr/bin/ceph-osd -i 6 --pid-file /var/run/ceph/osd.6.pid -c /etc/ceph/ceph.conf --cluster ceph
root@r-ch104:~# ps aux | grep ceph
root     14082  1.6  0.1 354812 82920 pts/0    Sl   08:19   0:04 /usr/bin/ceph-mon -i b --pid-file /var/run/ceph/mon.b.pid -c /etc/ceph/ceph.conf --cluster ceph
root     14431  0.0  0.0   9384   924 pts/0    S+   08:23   0:00 grep --color=auto ceph
root     26540 20.8  0.7 1620936 487556 ?      Ssl  Sep16 132:34 /usr/bin/ceph-osd -i 17 --pid-file /var/run/ceph/osd.17.pid -c /etc/ceph/ceph.conf --cluster ceph
root     29198 22.3  0.7 1728640 493852 ?      Ssl  Sep16 137:33 /usr/bin/ceph-osd -i 6 --pid-file /var/run/ceph/osd.6.pid -c /etc/ceph/ceph.conf --cluster ceph
root@r-ch104:~# ps aux | grep ceph
root     14082  1.5  0.1 382004 111520 pts/0   Sl   08:19   0:12 /usr/bin/ceph-mon -i b --pid-file /var/run/ceph/mon.b.pid -c /etc/ceph/ceph.conf --cluster ceph
root     14722  0.0  0.0   9384   932 pts/0    S+   08:33   0:00 grep --color=auto ceph
root     26540 20.8  0.7 1620936 472456 ?      Ssl  Sep16 134:34 /usr/bin/ceph-osd -i 17 --pid-file /var/run/ceph/osd.17.pid -c /etc/ceph/ceph.conf --cluster ceph
root     29198 22.3  0.7 1728640 481416 ?      Ssl  Sep16 139:40 /usr/bin/ceph-osd -i 6 --pid-file /var/run/ceph/osd.6.pid -c /etc/ceph/ceph.conf --cluster ceph
root@r-ch104:~# ps aux | grep ceph
root     14082  1.5  0.2 441696 171620 pts/0   Sl   08:19   0:27 /usr/bin/ceph-mon -i b --pid-file /var/run/ceph/mon.b.pid -c /etc/ceph/ceph.conf --cluster ceph
root     15222  0.0  0.0   9384   932 pts/0    S+   08:49   0:00 grep --color=auto ceph
root     26540 20.8  0.6 1620936 438600 ?      Ssl  Sep16 138:05 /usr/bin/ceph-osd -i 17 --pid-file /var/run/ceph/osd.17.pid -c /etc/ceph/ceph.conf --cluster ceph
root     29198 22.4  0.6 1728640 453088 ?      Ssl  Sep16 143:36 /usr/bin/ceph-osd -i 6 --pid-file /var/run/ceph/osd.6.pid -c /etc/ceph/ceph.conf --cluster ceph
root@r-ch104:~# ps aux | grep ceph
root     14082  1.6  0.3 482292 210324 pts/0   Sl   08:19   0:39 /usr/bin/ceph-mon -i b --pid-file /var/run/ceph/mon.b.pid -c /etc/ceph/ceph.conf --cluster ceph
root     15708  0.0  0.0   9384   932 pts/0    S+   09:00   0:00 grep --color=auto ceph
root     26540 20.8  0.6 1620936 430284 ?      Ssl  Sep16 140:24 /usr/bin/ceph-osd -i 17 --pid-file /var/run/ceph/osd.17.pid -c /etc/ceph/ceph.conf --cluster ceph
root     29198 22.4  0.6 1728640 425096 ?      Ssl  Sep16 146:07 /usr/bin/ceph-osd -i 6 --pid-file /var/run/ceph/osd.6.pid -c /etc/ceph/ceph.conf --cluster ceph

root@r-ch105:~# ps aux | grep ceph
root     20835  0.0  0.0   9384   932 pts/0    S+   08:10   0:00 grep --color=auto ceph
root     23550  1.7  3.5 2659800 2364780 ?     Sl   Sep16  11:18 /usr/bin/ceph-mon -i c --pid-file /var/run/ceph/mon.c.pid -c /etc/ceph/ceph.conf --cluster ceph
root     25512 21.3  0.8 1671084 555572 ?      Ssl  Sep16 137:32 /usr/bin/ceph-osd -i 18 --pid-file /var/run/ceph/osd.18.pid -c /etc/ceph/ceph.conf --cluster ceph
root     28444 22.7  1.0 1728056 668176 ?      Ssl  Sep16 142:01 /usr/bin/ceph-osd -i 9 --pid-file /var/run/ceph/osd.9.pid -c /etc/ceph/ceph.conf --cluster ceph
root@r-ch105:/var/log/ceph# ps aux | grep ceph
root     21467 25.8  0.7 1506260 466856 ?      Ssl  08:16   1:48 /usr/bin/ceph-osd -i 9 --pid-file /var/run/ceph/osd.9.pid -c /etc/ceph/ceph.conf --cluster ceph
root     22286  1.5  0.1 397484 83584 pts/0    Sl   08:18   0:04 /usr/bin/ceph-mon -i c --pid-file /var/run/ceph/mon.c.pid -c /etc/ceph/ceph.conf --cluster ceph
root     22744  0.0  0.0   9388   932 pts/0    R+   08:23   0:00 grep --color=auto ceph
root     25512 21.4  0.7 1671980 521548 ?      Ssl  Sep16 140:45 /usr/bin/ceph-osd -i 18 --pid-file /var/run/ceph/osd.18.pid -c /etc/ceph/ceph.conf --cluster ceph
root@r-ch105:/var/log/ceph# ps aux | grep ceph
root     21467 25.7  0.7 1506260 467868 ?      Ssl  08:16   1:49 /usr/bin/ceph-osd -i 9 --pid-file /var/run/ceph/osd.9.pid -c /etc/ceph/ceph.conf --cluster ceph
root     22286  1.6  0.1 399052 85124 pts/0    Sl   08:18   0:05 /usr/bin/ceph-mon -i c --pid-file /var/run/ceph/mon.c.pid -c /etc/ceph/ceph.conf --cluster ceph
root     22746  0.0  0.0   9388   932 pts/0    R+   08:24   0:00 grep --color=auto ceph
root     25512 21.4  0.7 1671980 521752 ?      Ssl  Sep16 140:47 /usr/bin/ceph-osd -i 18 --pid-file /var/run/ceph/osd.18.pid -c /etc/ceph/ceph.conf --cluster ceph
root@r-ch105:/var/log/ceph# ps aux | grep ceph
root     21467 25.7  0.7 1506260 467868 ?      Ssl  08:16   1:49 /usr/bin/ceph-osd -i 9 --pid-file /var/run/ceph/osd.9.pid -c /etc/ceph/ceph.conf --cluster ceph
root     22286  1.6  0.1 399052 85124 pts/0    Sl   08:18   0:05 /usr/bin/ceph-mon -i c --pid-file /var/run/ceph/mon.c.pid -c /etc/ceph/ceph.conf --cluster ceph
root     22748  0.0  0.0   9388   932 pts/0    R+   08:24   0:00 grep --color=auto ceph
root     25512 21.4  0.7 1671980 521936 ?      Ssl  Sep16 140:47 /usr/bin/ceph-osd -i 18 --pid-file /var/run/ceph/osd.18.pid -c /etc/ceph/ceph.conf --cluster ceph
root@r-ch105:/var/log/ceph# ps aux | grep ceph
root     21467 22.0  0.7 1512428 469344 ?      Ssl  08:16   3:40 /usr/bin/ceph-osd -i 9 --pid-file /var/run/ceph/osd.9.pid -c /etc/ceph/ceph.conf --cluster ceph
root     22286  1.4  0.1 418364 104800 pts/0   Sl   08:18   0:12 /usr/bin/ceph-mon -i c --pid-file /var/run/ceph/mon.c.pid -c /etc/ceph/ceph.conf --cluster ceph
root     23295  0.0  0.0   9388   932 pts/0    R+   08:33   0:00 grep --color=auto ceph
root     25512 21.4  0.7 1671980 505556 ?      Ssl  Sep16 142:44 /usr/bin/ceph-osd -i 18 --pid-file /var/run/ceph/osd.18.pid -c /etc/ceph/ceph.conf --cluster ceph
root@r-ch105:/var/log/ceph# ps aux | grep ceph
root     21467 22.0  0.6 1512428 444644 ?      Ssl  08:16   3:40 /usr/bin/ceph-osd -i 9 --pid-file /var/run/ceph/osd.9.pid -c /etc/ceph/ceph.conf --cluster ceph
root     22286  1.4  0.1 418364 104876 pts/0   Sl   08:18   0:12 /usr/bin/ceph-mon -i c --pid-file /var/run/ceph/mon.c.pid -c /etc/ceph/ceph.conf --cluster ceph
root     23299  0.0  0.0   9388   928 pts/0    S+   08:33   0:00 grep --color=auto ceph
root     25512 21.4  0.7 1671980 505556 ?      Ssl  Sep16 142:44 /usr/bin/ceph-osd -i 18 --pid-file /var/run/ceph/osd.18.pid -c /etc/ceph/ceph.conf --cluster ceph
root@r-ch105:/var/log/ceph# ps aux | grep ceph
root     21467 22.0  0.6 1512428 444660 ?      Ssl  08:16   3:40 /usr/bin/ceph-osd -i 9 --pid-file /var/run/ceph/osd.9.pid -c /etc/ceph/ceph.conf --cluster ceph
root     22286  1.4  0.1 418364 103960 pts/0   Sl   08:18   0:12 /usr/bin/ceph-mon -i c --pid-file /var/run/ceph/mon.c.pid -c /etc/ceph/ceph.conf --cluster ceph
root     23301  0.0  0.0   9388   928 pts/0    S+   08:33   0:00 grep --color=auto ceph
root     25512 21.4  0.7 1671980 505812 ?      Ssl  Sep16 142:45 /usr/bin/ceph-osd -i 18 --pid-file /var/run/ceph/osd.18.pid -c /etc/ceph/ceph.conf --cluster ceph
root@r-ch105:/var/log/ceph# ps aux | grep ceph
root     21467 22.0  0.6 1512428 444660 ?      Ssl  08:16   3:40 /usr/bin/ceph-osd -i 9 --pid-file /var/run/ceph/osd.9.pid -c /etc/ceph/ceph.conf --cluster ceph
root     22286  1.4  0.1 418364 103960 pts/0   Sl   08:18   0:12 /usr/bin/ceph-mon -i c --pid-file /var/run/ceph/mon.c.pid -c /etc/ceph/ceph.conf --cluster ceph
root     23303  0.0  0.0   9388   928 pts/0    R+   08:33   0:00 grep --color=auto ceph
root     25512 21.4  0.7 1671980 505812 ?      Ssl  Sep16 142:45 /usr/bin/ceph-osd -i 18 --pid-file /var/run/ceph/osd.18.pid -c /etc/ceph/ceph.conf --cluster ceph
root@r-ch105:/var/log/ceph# ps aux | grep ceph
root     21467 22.0  0.6 1512428 443740 ?      Ssl  08:16   3:40 /usr/bin/ceph-osd -i 9 --pid-file /var/run/ceph/osd.9.pid -c /etc/ceph/ceph.conf --cluster ceph
root     22286  1.4  0.1 418364 105524 pts/0   Sl   08:18   0:12 /usr/bin/ceph-mon -i c --pid-file /var/run/ceph/mon.c.pid -c /etc/ceph/ceph.conf --cluster ceph
root     23325  0.0  0.0   9388   932 pts/0    R+   08:33   0:00 grep --color=auto ceph
root     25512 21.4  0.7 1671980 505812 ?      Ssl  Sep16 142:45 /usr/bin/ceph-osd -i 18 --pid-file /var/run/ceph/osd.18.pid -c /etc/ceph/ceph.conf --cluster ceph
root@r-ch105:/var/log/ceph# ps aux | grep ceph
root     21467 22.0  0.6 1512428 443740 ?      Ssl  08:16   3:40 /usr/bin/ceph-osd -i 9 --pid-file /var/run/ceph/osd.9.pid -c /etc/ceph/ceph.conf --cluster ceph
root     22286  1.4  0.1 418364 105524 pts/0   Sl   08:18   0:12 /usr/bin/ceph-mon -i c --pid-file /var/run/ceph/mon.c.pid -c /etc/ceph/ceph.conf --cluster ceph
root     23328  0.0  0.0   9388   932 pts/0    R+   08:33   0:00 grep --color=auto ceph
root     25512 21.4  0.7 1671980 505812 ?      Ssl  Sep16 142:45 /usr/bin/ceph-osd -i 18 --pid-file /var/run/ceph/osd.18.pid -c /etc/ceph/ceph.conf --cluster ceph
root@r-ch105:/var/log/ceph# ps aux | grep ceph
root     21467 22.0  0.6 1512428 443740 ?      Ssl  08:16   3:40 /usr/bin/ceph-osd -i 9 --pid-file /var/run/ceph/osd.9.pid -c /etc/ceph/ceph.conf --cluster ceph
root     22286  1.4  0.1 418364 105524 pts/0   Sl   08:18   0:12 /usr/bin/ceph-mon -i c --pid-file /var/run/ceph/mon.c.pid -c /etc/ceph/ceph.conf --cluster ceph
root     23330  0.0  0.0   9388   932 pts/0    S+   08:33   0:00 grep --color=auto ceph
root     25512 21.4  0.7 1671980 505812 ?      Ssl  Sep16 142:45 /usr/bin/ceph-osd -i 18 --pid-file /var/run/ceph/osd.18.pid -c /etc/ceph/ceph.conf --cluster ceph
root@r-ch105:/var/log/ceph# ps aux | grep ceph
root     21467 21.7  0.6 1516540 428032 ?      Ssl  08:16   7:05 /usr/bin/ceph-osd -i 9 --pid-file /var/run/ceph/osd.9.pid -c /etc/ceph/ceph.conf --cluster ceph
root     22286  1.4  0.2 469632 157040 pts/0   Sl   08:18   0:26 /usr/bin/ceph-mon -i c --pid-file /var/run/ceph/mon.c.pid -c /etc/ceph/ceph.conf --cluster ceph
root     23950  0.0  0.0   9388   928 pts/0    S+   08:49   0:00 grep --color=auto ceph
root     25512 21.4  0.7 1671980 470492 ?      Ssl  Sep16 146:24 /usr/bin/ceph-osd -i 18 --pid-file /var/run/ceph/osd.18.pid -c /etc/ceph/ceph.conf --cluster ceph
root@r-ch105:/var/log/ceph# ps aux | grep ceph
root     21467 21.7  0.6 1517696 408620 ?      Ssl  08:16   9:24 /usr/bin/ceph-osd -i 9 --pid-file /var/run/ceph/osd.9.pid -c /etc/ceph/ceph.conf --cluster ceph
root     22286  1.4  0.2 508536 194144 pts/0   Sl   08:18   0:36 /usr/bin/ceph-mon -i c --pid-file /var/run/ceph/mon.c.pid -c /etc/ceph/ceph.conf --cluster ceph
root     24513  0.0  0.0   9388   932 pts/0    S+   09:00   0:00 grep --color=auto ceph
root     25512 21.4  0.6 1671980 453616 ?      Ssl  Sep16 148:49 /usr/bin/ceph-osd -i 18 --pid-file /var/run/ceph/osd.18.pid -c /etc/ceph/ceph.conf --cluster ceph
  • Why are the memory usages that different (mon.a = 640 MB, while mon.b and mon.c around 2000 MB)?
  • What can I do to further investigate and fix this? (it's a production cluster so I cannot "experiment freely")
Actions
Copy link

Also available in: Atom PDF