Monitoring¶
Prometheus and Grafana for one or more Haify clusters.
The controller serves Prometheus metrics on [metrics] listen_address:port
in controller.toml. By default metrics are on, at 0.0.0.0, port 9433.
Point Prometheus at the controller VIP, so it keeps scraping the right node
after a Self-HA failover.
# one target per cluster: <controller VIP>:<[metrics] port>, with a cluster label
vi prometheus/prometheus.yml
printf 'GF_SECURITY_ADMIN_USER=admin\nGF_SECURITY_ADMIN_PASSWORD=%s\n' "$(openssl rand -hex 16)" > .env
chmod 600 .env
docker compose up -d
The shipped prometheus.yml scrapes two placeholder clusters
(controller-a.example.com:9433, controller-b.example.com:9433). Replace
them with your own targets. Keep job_name: haify, and keep a cluster label
on every target. The alert rules match job="haify", and the dashboard's
Cluster selector lists the cluster values of up{job="haify"}.
Node memory, CPU, disk and network¶
Haify's own metrics cover the cluster: pools, DRBD, quorum, nodes reachable, the controller process. Memory, CPU, load, disk I/O and network per node come from Prometheus' node_exporter, installed on every node with:
scripts/install-node-exporter.sh node1,node2,node3
It installs the distribution's package (prometheus-node-exporter on
Debian/Ubuntu, node_exporter from EPEL on EL) and moves it to port 39117
(--port to change it; keep [metrics] node_exporter_port in
controller.toml the same). Prometheus finds the nodes through the
controller: http://<controller VIP>:9433/discovery/node-exporter lists one
target per registered node, labelled node, and ?cluster=<name> adds the
cluster label. The shipped prometheus.yml has a node job reading it, one
URL per cluster, so a node added to or removed from Haify is scraped or
dropped by itself. The dashboard's Nodes row shows memory used, CPU busy, I/O
on the DRBD devices, network, load per CPU and the root filesystem.
| Host port | Notes | |
|---|---|---|
| Prometheus | 39417 | Keeps 90 days of data. --web.enable-lifecycle is on, so POST /-/reload reloads the config |
| Grafana | 41863 | Home dashboard "Haify" in folder Haify |
Anyone who can reach Grafana can view the dashboards without logging in
(anonymous Viewer). Editing needs the admin login from .env. To require a
login for viewing too, remove the GF_AUTH_ANONYMOUS_* variables from
docker-compose.yml.
Dashboard¶
The dashboard shows one cluster at a time. It has:
- number tiles: controller up, nodes reachable, resources, degraded replicas, quorum lost, critical alerts, warnings;
- a table of resources and a table of nodes;
- graphs over time: pool use, alerts firing, out-of-sync bytes, and the controller API's request rate and latency;
- a table of backups;
- per node, from node_exporter: memory, CPU, DRBD device I/O, network, load and root filesystem.
grafana/dashboards/haify.json is generated by grafana/gen_dashboard.py. To
change the dashboard, edit the script and run it again
(python3 grafana/gen_dashboard.py) instead of editing the JSON. Grafana loads
the new file from its dashboards directory by itself.
Alert rules¶
prometheus-rules.yml, mounted into Prometheus as
/etc/haify/prometheus-rules.yml:
| Alert | Severity | Fires when |
|---|---|---|
HaifyControllerDown |
critical | up{job="haify"} == 0 for 2m |
HaifyObservationStale |
warning | the controller has not refreshed a data source (source label) for over 10m, for 5m |
HaifyNodeUnreachable |
critical | controller cannot reach a node, for 2m |
HaifyQuorumLost |
critical | haify_drbd_quorum == 0 for 1m |
HaifyResourceUnreadable |
critical | no node answered with the status of a resource, for 5m |
HaifyReplicaNotUpToDate |
warning | disk state other than UpToDate/Diskless for 10m |
HaifyOutOfSync |
warning | out-of-sync bytes on an Established connection for 30m (run haify resource verify) |
HaifyThinPoolNearFull |
warning | thin pool data or metadata over 85% for 10m |
HaifyThinPoolFull |
critical | thin pool data or metadata over 95% for 1m |
HaifyBackupStale |
warning | no completed backup of a resource to a target for over two days, for 1h |
HaifyOneFailureDomain |
info | losing one fault domain would take a resource down, for 1h |
HaifyNodeExporterDown |
warning | a node's node_exporter does not answer, for 5m |
HaifyNodeMemoryHigh |
warning | a node has used over 90% of its memory, for 10m |
No Alertmanager is included. The controller sends its own alerts through its
notification channels (haify channel); add an Alertmanager only if those are
not enough.