Ceph » RADOS

rados/singleton/{all/thrash-eio mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8}}

/a/yuriw-2021-08-06_16:31:19-rados-wip-yuri-master-8.6.21-distro-basic-smithi/6324604

This only happens with rados/singleton, and every time the job goes dead.

/a/yuriw-2023-04-24_22:54:45-rados-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7250551

Something interesting at the end of the mgr log:
/a/yuriw-2023-04-24_22:54:45-rados-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7250551/remote/smithi121/log/ceph-mgr.x.log.gz

2023-04-27T23:51:41.761+0000 7f5d406fa640 10 mgr.server operator() 24 pgs: 3 active+recovery_unfound+undersized+degraded+remapped, 21 active+clean; 4.0 GiB data, 8.8 GiB used, 441 GiB / 450 GiB avail; 24/8218 objects degraded (0.292%); 12/4109 objects unfound (0.292%)
2023-04-27T23:51:41.761+0000 7f5d406fa640 10 mgr.server operator() 4 health checks
2023-04-27T23:51:41.761+0000 7f5d406fa640 20 mgr.server operator() health checks:
{
    "OBJECT_UNFOUND": {
        "severity": "HEALTH_WARN",
        "summary": {
            "message": "12/4109 objects unfound (0.292%)",
            "count": 12
        },
        "detail": [
            {
                "message": "pg 2.9 has 3 unfound objects" 
            },
            {
                "message": "pg 2.2 has 4 unfound objects" 
            },
            {
                "message": "pg 2.6 has 5 unfound objects" 
            }
        ]
    },
    "OSD_TOO_MANY_REPAIRS": {
        "severity": "HEALTH_WARN",
        "summary": {
            "message": "Too many repaired reads on 1 OSDs",
            "count": 1
        },
        "detail": [
            {
                "message": "osd.0 had 16 reads repaired" 
            }
        ]
    },
    "PG_DAMAGED": {
        "severity": "HEALTH_ERR",
        "summary": {
            "message": "Possible data damage: 3 pgs recovery_unfound",
            "count": 3
        },
        "detail": [
            {
                "message": "pg 2.2 is active+recovery_unfound+undersized+degraded+remapped, acting [0], 4 unfound" 
            },
            {
                "message": "pg 2.6 is active+recovery_unfound+undersized+degraded+remapped, acting [0], 5 unfound" 
            },
            {
                "message": "pg 2.9 is active+recovery_unfound+undersized+degraded+remapped, acting [0], 3 unfound" 
            }
        ]
    },
    "PG_DEGRADED": {
        "severity": "HEALTH_WARN",
        "summary": {
            "message": "Degraded data redundancy: 24/8218 objects degraded (0.292%), 3 pgs degraded, 3 pgs undersized",
            "count": 6
        },
        "detail": [
            {
                "message": "pg 2.2 is stuck undersized for 11h, current state active+recovery_unfound+undersized+degraded+remapped, last acting [0]" 
            },
            {
                "message": "pg 2.6 is stuck undersized for 11h, current state active+recovery_unfound+undersized+degraded+remapped, last acting [0]" 
            },
            {
                "message": "pg 2.9 is stuck undersized for 11h, current state active+recovery_unfound+undersized+degraded+remapped, last acting [0]" 
            }
        ]
    }
}

If my above comment is correct, teuthology also have that incorrect configuration as default, in placeholder.py we thought that we are setting it correctly:

       'ceph-deploy': {
            'conf': {
                'client': {
                    'log file': '/var/log/ceph/ceph-$name.$pid.log'
                },
                'mon': {
                    'osd default pool size': 2
                }
            }