Tuning Cluster Failover Time

Leader failover time in an Aeron® Cluster is fully tunable — and by default, slow. The stock configuration takes ~11 seconds to recover from a leader loss, almost all of it one timeout. You can cut that to sub-second, but the knob cuts both ways: tighten it too far and a healthy leader that goes briefly silent gets voted out for no reason. This page is how to reason about that trade-off.

One-line takeaway

Failover time is a detection + election budget dominated by leader.heartbeat.timeout (default 10 s). It has no enforced floor, but lowering it trades a rare, planned recovery stall for a continuous risk of false failovers. 1–2 s is the engineering sweet spot for most deployments; sub-second is achievable but only under a hard SLA and only after a soak test.

For the byte-level election internals — terms, canvass, vote records — see The Aeron Files. This page is the operational view: what to turn, what you gain, and what you risk. It pairs with Leader Placement and Graceful Step-Down (where the new leader lands) — this page is about how fast it gets there.

What “failover time” is actually made of

Failover = detection (notice the leader is gone) + election (vote in a new leader) + client relink (the client re-points ingress at the new leader). Measured from the kill:

Detection and election run in series, so the floor is roughly heartbeat.timeout + election.timeout + relink.

The knobs

property	default	role	notes
aeron.cluster.leader.heartbeat.timeout	10,000 ms	detection — how long a follower waits after the last leader heartbeat before declaring it dead	~90% of stock failover time; the single biggest lever
aeron.cluster.leader.heartbeat.interval	200 ms	how often the leader heartbeats	must stay ≪ timeout (want ≥3–5 beats per window)
aeron.cluster.election.timeout	1,000 ms	the voting/election phase after detection	runs after detection, in series
aeron.cluster.election.status.interval	100 ms	election round cadence
aeron.cluster.session.timeout	10,000 ms	client session liveness	separate from leader failover; governs when the cluster evicts a silent client
aeron.cluster.startup.canvass.timeout	60,000 ms	cold-start leader canvass	irrelevant to failover (startup only)

By default none of the failover knobs are set, so the stock 10 s detection dominates everything else.

Why lower it — the upside

• Faster recovery from a real leader loss. The obvious win: ~11 s → ~0.85 s in our test (~13×). Less time with the data plane stalled and back-pressured after a genuine crash.
• Shorter client-visible outage. Client ingress is back-pressured for the whole failover window, so a shorter window means a shorter throughput gap and a smaller backlog to drain on resume.
• Tighter, more predictable SLA. If you must promise “recovers in < N seconds,” lowering the timeouts is the only way to shrink N.
• Frees stuck resources sooner. Subscriptions, session state, and flow-control windows tied to the dead leader clear faster.

Why not to — the risks

False failovers (the primary risk)

heartbeat.timeout is a dead-man’s switch. If a healthy leader goes silent longer than the window — for any reason — the followers declare it dead and run a real election: a genuine ~0.85–1.5 s stall plus client back-pressure, self-inflicted, with no actual failure. The tighter the window, the more benign stalls it catches.

Stalls that can trip a too-tight window on a healthy leader:

stall source	typical duration	mitigation
GC pause	ZGC sub-ms; G1/Parallel or a big heap → 100 ms–seconds	a low-pause GC (ZGC)
CPU scheduling / preemption	ms–100s ms if cores are shared	isolcpus + nohz_full pinning
Archive fsync / disk stall	10s–100s ms on network storage	archive on fast local storage / tmpfs
Network jitter / dropped heartbeat	µs–ms	same-AZ placement group (low RTT, no loss)
VM steal / noisy neighbor	ms–100s ms	bare metal (no hypervisor steal)
Safepoint / JIT / THP compaction	ms–100s ms	(residual tail risk)

The safe floor is a property of your environment, not a universal number

Every “mitigation” above is a specific environment choice. Swap the GC, grow the heap, move the archive to slower disk, or go cross-AZ, and a window that was flap-free may start flapping. There is no magic minimum timeout — the floor is set by your worst realistic healthy-leader stall, usually a GC pause.

The other failure modes

• Interval-to-timeout ratio. heartbeat.interval must stay well below heartbeat.timeout (aim for ≥3–5 beats per window). Cut the timeout but not the interval and a single delayed or lost packet triggers a false failover. This is the most common misconfiguration.
• Election churn. If election.timeout is too short, a legitimate election that needs slightly longer (canvassing, log catch-up) can itself time out and restart → the election thrashes, making effective failover longer or non-convergent — the opposite of the goal.
• Leadership thrashing under load. Tight timeouts can bounce leadership between nodes under high load; each bounce forces a client relink and a term change. A cluster that flaps repeatedly can have worse availability than one with a slow-but-stable failover.
• Short-soak false confidence. A few-minute run that shows zero flaps does not prove safety over days — production adds variable load, long-uptime tail GC events, and bursty traffic. Confidence needs a long soak (tens of minutes to hours), not a short benchmark.

What is not at greater risk

Lowering the timeout converts one failure mode (slow recovery) into another (self-inflicted stalls). It does not introduce a correctness hazard:

• No data loss. Aeron Cluster commit is quorum-based (Raft); committed data survives failover regardless of timing. See Cluster and Raft Overview.
• No split-brain. A new leader needs a majority vote, so a false election cannot produce two leaders. The damage from a false failover is purely an availability blip — never corruption or two writers.

Measured findings

Measured on a bare-metal fleet (Granite Rapids, SNC-3), 800K tps, 128B messages, MTU 8192, with a kill-leader-under-load test and a failover-aware client (one that implements EgressListener.onNewLeader and calls pollEgress()). Verified against Aeron 1.48.11; property names/defaults confirmed by reflecting the running jar.

Stock timeouts → ~11 s, and the client trap

With stock timeouts, the cluster election itself is healthy — node killed → new leader elected, 0 NAKs / 0 retransmits / 0 errors, and the surviving follower loses nothing (it was fully replicated at kill, so promotion needed no re-sync; the new leader logged 2 benign catch-up NAKs).

True failover measured ≈ 11.0 s — three independent signals agree (new leader’s raft-log send position, client bytes resuming, client NEWLEADER event). That’s just the stock 10 s heartbeat timeout + ~1 s election, exactly as the model predicts.

Planned step-down skips the 10 s detection entirely — single-digit-ms blocking

That ~11 s is for an unplanned leader death, where detection must wait out leader.heartbeat.timeout. A planned, graceful step-down is a different regime: closing the leader cleanly raises end-of-stream on the followers, which triggers an immediate election — so recovery is bounded by election.timeout (~1 s), not the 10 s heartbeat timeout. In a single-machine embedded test an in-place RESIGN step-down blocked clients for only ~1–2 ms — single-digit milliseconds (order-of-magnitude, not a production SLA). So maintenance handoffs and preferred-leader reclaims are a single-digit-ms blip rather than a failover event, without touching any timeout — see Leader Placement and Preferred-Leader Control. The timeouts below only govern the unplanned-death path.

A send-only client never recovers — at any timeout

The most important finding is not a number. A send-only client that never calls pollEgress() never sees the new-leader redirect and stays back-pressured forever — no failover timeout can fix that, because the client never follows the new leader. Ship a failover-aware client (pollEgress() + EgressListener) or none of this tuning matters. See Client-Cluster Communication.

Aggressive timeouts → ~0.85 s, no floor enforced

Setting heartbeat.timeout=250ms, heartbeat.interval=50ms, election.timeout=300ms, election.status.interval=25ms — Aeron accepted every value verbatim; there is no enforced floor.

trial	detection (ms)	election (ms)	client relink (ms)	total (ms)	false failovers
1	337	637	764	809	0
2	338	695	808	833	0
3	339	805	902	947	0
median	338	695	808	833	0

• ~0.85 s reliably — ~13× faster than stock, all clients fully recovered.
• Sub-500 ms was not reached, structurally: detection (~338 ms ≈ heartbeat.timeout) and election.timeout (300 ms) run in series → ~640 ms to a leader before relink. Both terms must shrink to hit 500 ms, and doing so erodes the flap margin.
• False failovers = 0 across ~28,000 steady-state samples — election_state=CLOSED, flat role for the whole pre-kill window. 250 ms / 50 ms was flap-free on this rig (low-pause GC, isolcpus, fast local archive, same-AZ) — which is exactly the point of the caution above: it’s safe here.

Measuring a sub-second failover

Resolving a sub-second event needs a long-lived counter sampler at a true ~25 ms cadence. A fresh-JVM-per-sample aeron-stat is really ~500 ms/sample and cannot see a sub-500 ms failover. Read detection from the election_state counter, not the un-timestamped heartbeat timeout warning.

The trade-off curve and the recommendation

The two costs move at very different rates as you tighten the timeout:

• Recovery-speed benefit is nearly flat below ~2 s — a leader failure is rare, and 1.5 s vs 0.85 s is rarely user-visible.
• False-failover risk grows steeply as you tighten — this is a continuous exposure (every GC pause, safepoint, packet delay, every second of every day). Roughly, halving the timeout doubles how many benign stalls trip a spurious election.

Marginal gain from 2 s → 0.85 s is small; marginal risk is large. That is the knee of the curve.

tier	timeouts	failover	flap margin (tolerates leader stall of)	when to use
Conservative	stock (10 s / 1 s)	~11 s	~10 s (never flaps)	you never kill leaders / batch systems
Balanced (recommended)	hb 1000 / int 200 / elec 500 / stat 50	~1.5 s	~1 s — covers virtually all GC/safepoint/scheduling tails; robust to non-ideal setups	most production deployments
Aggressive (tested)	hb 250 / int 50 / elec 300 / stat 25	~0.85 s	~250 ms — a single G1 pause / THP compaction can exceed	hard sub-second SLA, ideal rig, after a soak

Recommendation:

1. Default to the balanced ~1–2 s tier (heartbeat.timeout=1000ms, interval=200ms, election.timeout=500ms, election.status.interval=50ms). It is ~7× faster than stock, keeps ~4× the flap margin of the aggressive tier, and stays safe even if the environment drifts off the ideal.
2. Keep the interval ≪ timeout (≥3–5 beats per window). Never lower the timeout alone.
3. Go sub-second only under a hard SLA, and only after a multi-minute-to-hours soak at production-like load and heap/GC that counts false failovers. The safe floor is set by your worst realistic healthy-leader stall, not the number you wish for.
4. session.timeout is separate — tune it for how fast the cluster should evict a silent client, independent of leader failover.
5. Ship a failover-aware client (pollEgress() + EgressListener) — without it, no failover time matters at all.

Checklist before lowering failover time in production

• Which GC, and what is its worst-case pause under load? (ZGC ≪ G1/Parallel.) This sets your floor.
• Are hot threads pinned (isolcpus/nohz_full) or competing for cores?
• Where does the archive fsync — tmpfs, local NVMe, or network storage? (Disk stalls trip detection.)
• Same-AZ / placement group, or cross-AZ? (Network jitter widens the needed margin.)
• Bare metal or VM? (VM steal time adds tail latency.)
• Is interval ≤ ~1/5 of timeout?
• Have you soaked at production load and heap/GC long enough to see the tail (≥ tens of minutes)?
• Does the ingress client handle onNewLeader + poll egress?
• Is the target justified by a real SLA, or is 1–2 s already good enough?

See also: Leader Placement and Graceful Step-Down, Cluster Standby and Multi-AZ HA Design, and the failure-mode runbook.