Operations & Resilience

Tuning gets you speed; operations keep you alive. This section is the runbook — what breaks, what it looks like, and exactly what to do about it.

The critical threshold is quorum

For a 3-node cluster, losing 1 node is fine (2/3 quorum holds). Losing 2 is catastrophic (1/3, no quorum). This is why node placement across AZs matters: never put 2 of 3 nodes in the same AZ. Spread them — 1 node per AZ for a 3-node cluster.

In this section

• Cluster Standby (Premium) — HA without the latency penalty: fast snapshots, in-place identity switch, cross-AZ hot standby.
• Disaster recovery — restoring from snapshots, log replay, and the 4th-node async backup.
• Backup strategies — archiving to durable object storage and backing up the backup.
• Failure-mode runbook — node, network, disk, snapshot, determinism, client, human-error, resource-exhaustion, and multi-AZ scenarios.

Failure-mode quick reference

Scenario	Data loss	Availability	Action
Single follower crash	None	None (quorum held)	Restart; auto-catchup
Single leader crash	Uncommitted msgs	Brief (election)	Wait for auto-election; restart node
Minority crash	None	None	Restart ASAP to restore fault tolerance
Majority / quorum loss	Uncommitted msgs	Total	Manual: restore quorum, then snapshot + log
All nodes crash	Possible	Total	Cold-start; replay snapshot + log; else external backup

Self-healing vs. manual

Minority failures self-heal — Aeron handles them automatically. Majority failures require manual intervention. Knowing which is which under pressure is the whole game.