Ceph » Orchestrator

After increasing the count for the alertmanager service, we see the number of alertmanager instances increase and prometheus is updated to reflect this.

However, the alertmanager daemons are not correctly peered. Each daemon should have a --cluster.peer <addr> for each peer in the cluster, but this is not the case

After increasing the count to 3 (from 1), the alertmanager execution looks like this
host1 : /bin/alertmanager --cluster.listen-address=:9094 --web.listen-address=:9093 --config.file=/etc/alertmanager/alertmanager.yml
host2 : /bin/alertmanager --cluster.listen-address=:9094 --web.listen-address=:9093 --cluster.peer=172.16.37.35:9094 --config.file=/etc/alertmanager/alertmanager.yml
host3 : /bin/alertmanager --cluster.listen-address=:9094 --web.listen-address=:9093 --cluster.peer=172.16.37.35:9094 --cluster.peer=10.1.36.191:9094 --config.file=/etc/alertmanager/alertmanager.yml

What should happen is that each instance should be pointing to the other peers, but as this output shows
host1 doesn't even reference any peers
host2 only references 1 peer (in this case host1's IP)
host3 gets it right!