Generate an overall health score for a Kubernetes cluster from key operational metrics.
This score starts at 100 and subtracts weighted penalties for four common cluster distress signals: the proportion of unhealthy nodes (up to 30 points), pending/unschedulable pods (up to 20 points), pod restart churn (up to 25 points), and elevated etcd latency, which signals control-plane stress (up to 25 points). Formula: health = 100 − unhealthy_node_penalty − pending_pod_penalty − restart_penalty − etcd_penalty. Scores map to Healthy (90+), Degraded (75-89), At Risk (50-74) or Critical (<50).
Cluster health score
health = 100 - (unhealthy_nodes / total_nodes) × 30 - min(20, pending_pods / 10 × 20) - min(25, restarts / 5) - min(25, max(0, (etcd_latency_ms - 10) / 90 × 25))
etcd is the cluster's source of truth — the API server, scheduler and controllers all depend on fast etcd reads/writes. Elevated etcd latency (often from disk I/O contention or too many watchers) can cascade into slow scheduling, delayed reconciliation, and API timeouts cluster-wide.
Healthy etcd clusters typically see disk write / consensus latency (p99) under 10-25ms; sustained latency above 100ms is a strong signal of disk or network issues and often precedes broader cluster instability.
A handful of pending pods is often transient (waiting for a scale-up event) rather than a systemic issue, so the penalty scales linearly but is capped to avoid over-weighting a metric that self-resolves quickly compared to, say, persistent node failures.
Not always — some restarts are expected (rolling deploys, liveness probe recovery). This metric is best tracked as a rate/trend over a fixed window rather than a cumulative all-time count, so recalibrate the input to your chosen window.