Ceph » RADOS

Can you dump the pg log using the ceph-objectstore-tool when the OSD is consuming high memory and share it with us?

Attached a pglog at the peak of one prod issue. I had to redact the object names since it's prod but let me know if you need them and I can easily reproduce this in our staging environment.

That was faster than I thought. Attached massif outfile (let me know if that's what you expect not super familiar with valgrind) and mempools before/after.
Note that valgrind slowed the OSD down to a crawl so s3cmd objectcopy timed out/failed in most instances.

Workload sent:
- s3 objectcopy of 1x 10GB file from bucket1 to bucket2 - 500 times in parallel
- osd.32 is OSD I monitored. It appears to only have one part.

Edit: Even though s3cmd timed out, the workload actually finished in the bg. Updated after mempool below:

# ceph daemon osd.32 dump_mempools
{
    "mempool": {
        "by_pool": {
            "bloom_filter": {
                "items": 0,
                "bytes": 0
            },
            "bluestore_alloc": {
                "items": 15523110,
                "bytes": 182819808
            },
            "bluestore_cache_data": {
                "items": 6509,
                "bytes": 1391216
            },
            "bluestore_cache_onode": {
                "items": 6706,
                "bytes": 4130896
            },
            "bluestore_cache_meta": {
                "items": 1742472,
                "bytes": 11319642
            },
            "bluestore_cache_other": {
                "items": 23988,
                "bytes": 581688
            },
            "bluestore_Buffer": {
                "items": 133,
                "bytes": 12768
            },
            "bluestore_Extent": {
                "items": 9858,
                "bytes": 473184
            },
            "bluestore_Blob": {
                "items": 9858,
                "bytes": 1025232
            },
            "bluestore_SharedBlob": {
                "items": 9858,
                "bytes": 1104096
            },
            "bluestore_inline_bl": {
                "items": 75,
                "bytes": 13588
            },
            "bluestore_fsck": {
                "items": 0,
                "bytes": 0
            },
            "bluestore_txc": {
                "items": 1,
                "bytes": 784
            },
            "bluestore_writing_deferred": {
                "items": 7,
                "bytes": 28288
            },
            "bluestore_writing": {
                "items": 64,
                "bytes": 927056
            },
            "bluefs": {
                "items": 191935,
                "bytes": 2578592
            },
            "bluefs_file_reader": {
                "items": 1038,
                "bytes": 132864
            },
            "bluefs_file_writer": {
                "items": 3,
                "bytes": 576
            },
            "buffer_anon": {
                "items": 2204,
                "bytes": 4591510
            },
            "buffer_meta": {
                "items": 39234,
                "bytes": 3452592
            },
            "osd": {
                "items": 256,
                "bytes": 2895872
            },
            "osd_mapbl": {
                "items": 0,
                "bytes": 0
            },
            "osd_pglog": {
                "items": 1111350,
                "bytes": 1193066303
            },
            "osdmap": {
                "items": 47045,
                "bytes": 1842072
            },
            "osdmap_mapping": {
                "items": 0,
                "bytes": 0
            },
            "pgmap": {
                "items": 0,
                "bytes": 0
            },
            "mds_co": {
                "items": 0,
                "bytes": 0
            },
            "unittest_1": {
                "items": 0,
                "bytes": 0
            },
            "unittest_2": {
                "items": 0,
                "bytes": 0
            }
        },
        "total": {
            "items": 18725704,
            "bytes": 1412388627
        }
    }
}

Attached the debug_osd 20 logs for one the OSD. I turned off (deep)scrub cause the logs were spammed with scrub errors which I'm guessing is related to the patch as well

I have also experienced this situation here

- Create 30 objects in bucket1 using put
- Using 30 objects as the source, append write 1K for the 30 objects and copy to the bucket1，Perform hundreds of cycles like this
- append write use boto url, copy use boto3:copy_object

- one pglog entry size reaches 5kB
cause the nodes to run ouf of memory

Tested on 15.2.13