Ceph » Orchestrator

Happens also (sometimes?) after re-provisioning an ingress server. After OS installed and when cephadm configures the server for the first time it attempts to start HAProxy before keepalived and then HAProxy errors.

As of the current design of the ingress service, the keepalived starts before haproxy because the daemon depends on the haproxy as it needs/uses it's health script for the alive check (see below). Thus, we can's start keepalived before haproxy. If the error reported above is persistent then this can be considered as a BUG, however, if the error is fixed (haproxy get deployed correctly) once keepalived is up&running then I think this shouldn't be an issue.

        # script to monitor health
        script = '/usr/bin/false'
        for d in daemons:
            if d.hostname == host:
                if d.daemon_type == 'haproxy':
                    assert d.ports
                    port = d.ports[1]   # monitoring port
                    script = f'/usr/bin/curl {build_url(scheme="http", host=d.ip or "localhost", port=port)}/health'
        assert script

this generates the keepalived.conf section of:

vrrp_script check_backend {
    script "/usr/bin/curl http://localhost:9049/health" 
    weight -20
    interval 2
    rise 2
    fall 2
}

The HAProxy service does not start until it is manually started or the server is restarted.

The problem root is the recent change to bind HAProxy to the IP address that keepalived creates and not to "*", now a circular dependency has been created if HAProxy actually creates the keepalived configuration.

I am pretty sure I saw systemd trying to restart the HAProxy container a few times which of course errored immediately because of the missing IP address. Maybe systemd unit parameters for restarting could be tuned, to allow time for keepalived container to start meanwhile? Currently systemd just tries to restart it a few times but gives up because the unit exits immediately.

I believe this is duplicate of https://tracker.ceph.com/issues/57563