Ceph » mgr

Node RAM utilization >95% (maybe some other number here?)
MDS down
RGW down
Network utilization >95%
downrev daemons
hosts with nearly filled root or /var/log partitions?

What's the preferred distribution format? Should there simply be a sample *.rules file included in some package?

Regarding collaborating:

• Should we just post some alert rules here?
• Should we use node_exporter for the disk utilization?

Example Alert: Ceph health not "OK":

alert: ceph_Health_status
expr: ceph_health_status != 0
for: 10m
annotations:
  description: Ceph unhealthy for > 10m
  summary: Ceph unhealthy

Tobias Florek wrote:

What's the preferred distribution format? Should there simply be a sample *.rules file included in some package?

For now I was thinking another subfolder in https://github.com/ceph/ceph/tree/master/monitoring would be good. Lets maybe start with a single rules file that can just be dropped into a prometheus deployment.

Regarding collaborating:

• Should we just post some alert rules here?

Either that or attach a file with alerts. Or even just some general remarks...all is welcome.

• Should we use node_exporter for the disk utilization?

Yes

Example Alert: Ceph health not "OK":

[...]

I was thinking it would make sense to group alerts my severity. We should maybe sort the into, say, 3 severity buckets (critical, warning, info or some such) so user can use the default alerts in different alert channels (no body wants a text message about a HEALTH_WARN cluster at 3am Saturday I assume). But again, I'm open to suggestions from experienced operators.

1. DeepSea (by @Jan)
2. Ceph-Ansible (by @Boris)
3. Ceph-mixins

First attempt: https://github.com/ceph/ceph/pull/27596