Ceph

Back in 0.34, my cluster with 3210 PGs 3-plicated across 3 OSD required some 4GB of RAM for the OSDs, i.e., with 3 OSDs up it would use 1.3GB on each; 2GB for 2 OSDs or 4GB for a single OSD.

With 0.35 and 3594, each OSD eats up 4.5GB of RAM just reading the local state, before even trying to contact a monitor. Two OSDs complete recovery using some 6 or 7GB each, and a single OSD skyrockets to 14+ GB before even moving PGs to peering. I could never complete recovery with a single 0.35 OSD :-(

I'm attaching the results of some memory profiling. I selected 3 relevant snapshots in the following sequence of events:

0. I brought osd.2 down and let the others recover, then brought them all down
1. I started osd.2 with memory profiling, and took snapshot alldown when it completed transitioning all PGs to crashed+down+degraded+peering
2. I started osd.1, and took snapshot cleanboth of osd.2 when all PGs were active+clean+degraded
3. I stopped osd.1, and took snapshot recovering of osd.2 when I ran out of time

Then I generated graphs out of each snapshot, as well as graphs with the incremental memory use between consecutive snapshots. They're all attached.