Skip to content
LowLatencyOps

Hot-Path Logging & Metrics Capture

Everywhere else on this site the rule is the same: no allocation, no blocking I/O, no external calls on the matching thread — any of them shows up directly in p99 (see client ↔ cluster communication and the determinism runbook). That rule raises an obvious question this page answers: then how do I log an event, measure a latency, or expose a counter from code that isn’t allowed to do any of those things?

The answer is a single principle applied three ways: capture cheaply on the hot path, do the expensive work off it. Write a value into pre-allocated memory in nanoseconds; let another thread format, aggregate, and ship it.

The one principle: split capture from emission

Section titled “The one principle: split capture from emission”

Every technique below is the same shape.

The hot path only ever does a bounded, allocation-free store. Formatting a string, acquiring a lock, flushing to disk, or hitting the network — all of it happens on a thread you don’t care about the latency of. This is the same architecture Aeron itself uses for its counters and error log, which is why publishing into that infrastructure (below) is the most natural fit of all.

1 · Logging: the event log is your audit trail; app logs go async

Section titled “1 · Logging: the event log is your audit trail; app logs go async”

The first thing to internalize on a clustered, event-sourced engine: you already have a perfect, totally-ordered, durable log of every input — the Aeron Cluster log itself. For audit, reconstruction, and “what did the engine do at 14:32:07,” the answer is replay the log, not grep application logs. Don’t duplicate the event stream into a logging framework on the hot path.

For the operational logging you still want (warnings, state transitions, rare errors), keep it off the matching thread:

  • Log via a ring buffer, drain on a background thread. Write a compact, pre-encoded event into an Agrona ManyToOneRingBuffer (or OneToOneRingBuffer for a single producer) from the hot path — a bounded, lock-free, allocation-free put. A separate agent drains it and does the formatting and file I/O. This is exactly the pattern behind async loggers (e.g. Log4j2’s async appender is backed by a ring buffer); on a latency-critical thread, make the async-ness explicit and own the buffer.
  • Never format on the hot path. Put the raw fields (order id, price as long cents, an enum ordinal) into the buffer; let the drainer turn them into text. String building is the allocation you are trying to avoid.
  • Aeron’s own error log works this way. The driver records exceptions into a distinct-error buffer in the CnC file that external tools dump point-in-time — no synchronous logging on its duty cycle. Your engine can follow the same model.

2 · Latency capture: HdrHistogram recorded in-process, reported off-thread

Section titled “2 · Latency capture: HdrHistogram recorded in-process, reported off-thread”

You want per-stage latencies (ingress → match, match → egress) at p50/p99/p99.9 from the live engine, not just from a benchmark harness. HdrHistogram is the right tool, and it’s built for exactly this: recording a value is a cheap, allocation-free array increment.

  • Record on the hot path, report off it. Use a Recorder (HdrHistogram’s concurrent front-end). The matching thread calls recorder.recordValue(latencyNs) — no allocation, no lock in the common case. A background thread periodically calls recorder.getIntervalHistogram() (which does the allocation, off the hot path) and logs/exports the percentiles.
  • Capture the timestamp cheaply. Use a monotonic System.nanoTime() for interval math (not wall-clock), and read it outside the deterministic matching logic — timestamp at the transport boundary, subtract, record. Keep the clock out of the state machine (see determinism).
  • Beware coordinated omission. If you time only requests that made it into the handler, you hide the queueing delay of the ones that were stuck behind a stall — the exact latency you care about. Measure against an expected arrival cadence, or record at the earliest possible point (transport ingress). The Benchmarking page covers why coordinated omission flatters you; the same discipline applies to live capture.
// Hot path — allocation-free record
final long startNs = /* nanoTime captured at ingress boundary */;
// ... deterministic matching ...
recorder.recordValue(System.nanoTime() - startNs);   // cheap array bump


// Background reporter thread — do the expensive part here
final Histogram interval = recorder.getIntervalHistogram();  // allocates, OFF hot path
log.info("match p99={}us p999={}us",
    interval.getValueAtPercentile(99.0) / 1000,
    interval.getValueAtPercentile(99.9) / 1000);

3 · Metrics: publish your own counters into Aeron’s counter infrastructure

Section titled “3 · Metrics: publish your own counters into Aeron’s counter infrastructure”

This is the payoff, and the piece most people miss. Aeron’s counters are not just its internal metrics — the same shared-memory infrastructure is a public API you can publish your own metrics into. A counter is a single 64-bit value in the CnC file, updated lock-free with an ordered store and read out-of-process by anyone. That makes it the ideal hot-path metric: the matching thread does one memory write; every scraper reads it for free.

Allocate a counter from your Aeron client and update it on the hot path:

// Setup (once, off the hot path): allocate a counter in the CnC file.
// typeId is your app-level routing key; pick a range outside Aeron's
// (0–99 client/driver, 100–199 archive, 200–299 cluster).
final Counter ordersMatched = aeron.addCounter(1001, "matching: orders matched");
final Counter bookDepth     = aeron.addCounter(1002, "matching: bid depth");


// Hot path: ordered store, no CAS, no allocation, no lock.
ordersMatched.incrementOrdered();       // count an event
bookDepth.setOrdered(currentBidDepth);  // publish a gauge
  • incrementOrdered() / setOrdered() are the hot-path-safe methods. They do a plain store with a release fence (single-writer ordered semantics) — no compare-and-swap contention. Counter extends Agrona’s AtomicCounter; use the *Ordered (or *Release) variants from the single writer, not the CAS-based ones.
  • Reading is already solved. Because the value lands in the same cnc.dat that aeron-stat and CountersReader already read, your custom counters show up alongside Aeron’s with zero extra plumbing — and the exporter pattern in Metrics and Alerting scrapes them to Prometheus/Datadog automatically. Route them by your typeId.
  • Counters, not push. The hot path never talks to Prometheus. It bumps a number in shared memory; the out-of-process scraper does the pull. This is the whole reason Aeron’s own telemetry adds “no significant impact on performance,” and your metrics inherit that property.

Where this lands on the maturity matrix

Section titled “Where this lands on the maturity matrix”

This is the how behind the Monitoring pillar of the maturity matrix:

  • L2 Production wants “app-level metrics — latency histograms, order/fill counts.” §2 (HdrHistogram Recorder) and §3 (your own counters) are exactly how you capture them without a p99 tax.
  • L4 Top-tier wants those signals “measured under sustained load.” Because capture is allocation-free and emission is off-thread, your instrumentation itself doesn’t move the number it’s measuring — a prerequisite for trusting the measurement at all.

Sources

Section titled “Sources”