Ceph » RADOS

Hi Ruben,
please share full dump_mempools output.

Hi Igor,

See attachment.

One thing I tried was to set osd_max_pg_log_entries to 500 instead of the default of 10000, but this doesn't help.

Ruben Kerkhof wrote:

One thing I tried was to set osd_max_pg_log_entries to 500 instead of the default of 10000, but this doesn't help.

What exactly did you do to try this?

In our clusters when osd_pglog mempool exploded we set osd_max_pg_log_entries and osd_min_pg_log_entries to 500 in the ceph.conf [osd] section and restarted OSDs.
The active OSD for a PG needs to have these settings for the trim to take effect.
Also, you can check the LOG columns in `ceph pg dump` to see the current num entries per PG.

Hi Ruben, Did you make any more progress on this?

I'm going through all the osd pglog memory usage tickets, and it looks like most or all of them are happening to SSDs.
I'm guessing here, but one notable difference is that SSDs have 2 op threads per shard by default, HDDs have 1.

Does the memory usage look more reasonable if you configure the SSD osds to use the _hdd defaults for shards/threads?

[osd]
osd_op_num_shards = 5
osd_op_num_threads_per_shard = 1

Dan van der Ster wrote:

Hi Ruben, Did you make any more progress on this?

Hi Dan, I missed your update, sorry. Not yet, the customer did upgrade a few hosts to Ubuntu 20.04, but a few of those OSDs also have high memory usage so it seems like tcmalloc is a red herring and the os doesn't affect this.

I'm going through all the osd pglog memory usage tickets, and it looks like most or all of them are happening to SSDs.
I'm guessing here, but one notable difference is that SSDs have 2 op threads per shard by default, HDDs have 1.

Does the memory usage look more reasonable if you configure the SSD osds to use the _hdd defaults for shards/threads?

[osd]
osd_op_num_shards = 5
osd_op_num_threads_per_shard = 1

Excellent idea! I'll ask the customer and get back with the results.

Excellent idea! I'll ask the customer and get back with the results.

We restarted the OSDs on a single node with these settings, and left them running for a few days, but no change.
The OSDs still use between 3GiB and 4Gib for the pglog mempool, and around 8GB rss.

Ruben Kerkhof wrote:

Excellent idea! I'll ask the customer and get back with the results.

We restarted the OSDs on a single node with these settings, and left them running for a few days, but no change.
The OSDs still use between 3GiB and 4Gib for the pglog mempool, and around 8GB rss.

Okay, it is likely this: https://tracker.ceph.com/issues/53729 (see the last week or so of comments)

You can confirm by stopping the osd and checking the number of dups in the PGs (it is normally under 3k per PG -- you may have somethign like 100k to a million in a few PGs, based on your memory usage):

PGS=$(ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0/ --op list-pgs)
for p in $PGS; do echo $p; ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0/ --op log --pgid $p | jq '(.pg_log_t.log|length),(.pg_log_t.dups|length)'; echo; done

There's nothing you can do until that fix is available.

Dan van der Ster wrote:

Ruben Kerkhof wrote:

Excellent idea! I'll ask the customer and get back with the results.

We restarted the OSDs on a single node with these settings, and left them running for a few days, but no change.
The OSDs still use between 3GiB and 4Gib for the pglog mempool, and around 8GB rss.

Okay, it is likely this: https://tracker.ceph.com/issues/53729 (see the last week or so of comments)

You can confirm by stopping the osd and checking the number of dups in the PGs (it is normally under 3k per PG -- you may have somethign like 100k to a million in a few PGs, based on your memory usage):

[...]

There's nothing you can do until that fix is available.

Hi Dan,

Great catch, that seems to be it. Most pgs show:
11.e4
554
2499

11.b4
579
2499

11.97
599
2499

11.8b
544
2499

But there are certainly outliers:

11.3b1
520
1589213

11.37a
519
602198

11.2c
595
2794697

11.3f
511
998223