Bug #48732
openMarking OSDs out causes mon daemons to crash following tcmalloc: large alloc
0%
Description
On a 14.2.11 zero-load cluster I am taking some osd servers out of service.
I began marking OSDs out in preparation and encountered an issue:
Immediately following marking an osd out the quorum became very laggy. The following state was recorded close to the quorum become unresponsive to client requests (which lasted about five minutes)
HEALTH_WARN no active mgr; Reduced data availability: 6 pgs inactive, 35 pgs peering; Degraded data redundancy: 2920/16956 objects degraded (17.221%), 8 pgs degraded; 1/3 mons down
Looking at the mon log I saw the following:
3 instances of " ceph-mon: tcmalloc: large alloc 8589934592 bytes"
followed by the mon going down:
Dec 30 12:20:50 redacted systemd: Stopping Ceph cluster monitor daemon...
Dec 30 12:20:50 redacted ceph-mon: 2020-12-30 12:20:50.017 7f9ec4f21700 -1 received signal: Terminated from /usr/lib/systemd/systemd --system --deserialize 25 (PID: 1) UID: 0
this cascaded to a second monitor going down shortly thereafter. The mon daemon was able to be restarted immediately and after about five minutes the clusters monitors and mgrs all came back but its unclear to me why simply marking an osd out would cause the mon daemons to crash. The mgr daemons never crashed.
looking at the memory graphs of the mon servers there was always plenty of free memory and load never seemed to spike. The highest number of PGs per osd in the cluster is 176. All spinning disks on bluestore.
I can try and recreate this issue if it helps.