Inside ServiceRadar's Durable Edge Record Protocol

A ServiceRadar agent produces a lot of evidence. Reachability for every address in a range, latency per host, per-hop traces when a path degrades, inventory pulled from a dozen integrations. All of it has to get from a box at the edge of a customer's network into a database, intact, over links that are frequently not good.

We rebuilt how that happens. The new edge record protocol replaces transport-shaped payloads with durable, versioned, self-describing records. They are compressed on the agent, written to disk before anything acknowledges them, carried over mTLS gRPC, committed to NATS JetStream, and projected into Postgres by a consumer that can tell a redelivery from a new fact. They are about six times smaller on the wire than what they replace, and an acknowledgement now means the data is actually safe.

What we replaced

The old path was shaped by whatever RPC happened to carry it. Our scanner finished a run of ICMP and TCP probes, built a single JSON document describing the entire run, and pushed it through a general-purpose status RPC. The agent then parsed its own JSON back into generic maps to build a second protobuf message from it. At a million hosts that document is a few hundred megabytes, going into a gRPC stream with a 64 MiB request window.

Size was the obvious problem. The subtler one was meaning. A successful RPC response told the agent that a function call had returned, not that anything durable had happened, and the data behind it could still be sitting in a memory-only queue. Per-hop trace data had nowhere structured to live, so it got flattened into scalar timeseries rows. Every new producer arrived with its own queue, its own size limits, its own retry loop, and its own private definition of "done."

A record instead of a payload

The central idea is small. An observation should carry everything a downstream component needs in order to trust and process it, so that nothing important depends on the connection it arrived over. That connection is long gone by the time the data is replayed off a stream three hours later.

So a record names the exact versioned contract it conforms to, rather than leaving a receiver to guess against whatever schema happens to be newest locally. It carries a hash of its exact payload bytes and a runtime-neutral digest of its meaning, which lets a Go publisher and an Elixir consumer agree on identity without agreeing on byte layout. It carries a network-scoped event identity, so a projector recognizes a redelivery instead of writing a second row. It carries signed capabilities establishing who was authorized to produce it, which is a different and far more durable question than which certificate terminated the TLS session. And it carries the correlation context tying it to the assignment, range, and time window it was collected under, because a well-formed observation filed against the wrong assignment is still wrong.

1. 1. Producer

   Scanner, plugin, add-on, or integration

2. 2. Agent sink and spool

   Compress once, build once, fsync

3. 3. mTLS gRPC

   Authenticated, flow-controlled delivery

4. 4. JetStream

   Durable commit, replay, backpressure

5. 5. Live consumers

   Detection, correlation, automation

6. 6. Projection

   Idempotent writes into Postgres

Each layer does one job. gRPC is still the right authenticated edge transport and is not going anywhere. JetStream is where durable ownership, replay, and backpressure become explicit. The projector decides what any of it means in relational terms. None of those layers gets to reinterpret the record on the way through.

Six times smaller on the wire

Scan results are the most repetitive data we produce. The same prefix repeats across a whole /24, the same handful of port numbers shows up on every host, round-trip times cluster within a few hundred microseconds of each other, and outcome values arrive in long identical runs. That is close to an ideal case for a compressor, and zstd takes full advantage of it.

Stack the changes together and a million-host scan goes from a few hundred megabytes of JSON to roughly 11 MB actually crossing the network, split across batches that individually measure in the tens of kilobytes. An operator watching 250,000 hosts every five minutes moves about 850 MB a day instead of about 5 GB, for identical coverage.

That ratio decides real things. It is the difference between a remote site on an LTE modem getting scanned every five minutes or every six hours, between continuous discovery fitting inside a cloud egress budget or not, and between an MPLS circuit sized years ago absorbing a modern monitoring workload or being the reason someone turns it off. The efficiency is not a benchmark trophy. It is what makes continuous visibility affordable in the places that need it most.

An acknowledgement that means something

Every acknowledgement in the new path is backed by a specific durable event. When the agent tells a producer it has the data, the record is already appended to a crash-safe local spool. When the gateway tells the agent it has the data, JetStream has already confirmed the publish. Nothing reports success on the strength of a handoff to volatile memory, so an agent can lose power mid-run and pick up exactly where it left off.

Retries stop being frightening once identity is stable. Networks retry, brokers redeliver, and consumers restart, and the protocol does not pretend otherwise. What it guarantees is that a redelivery is recognizable as one, because the event identity, payload hash, and contract reference are all still attached to the record when it comes around again.

Bounded by design

Accepting compressed data from a remote agent means accepting an instruction to allocate memory, so every limit is declared in the contract and enforced before the work it bounds. Raw size is checked against the bytes as received, before a decoder is involved. The expansion ratio is settled from header fields, before anything is decompressed. The decode itself streams through a small fixed buffer with a capped window and never reserves the full output, and the result has to match its declared size exactly. A compression bomb gets refused at the only point in the pipeline where refusing it is free.

One boundary for every producer

Built-in collectors, Wasm plugins, native add-ons, and integration adapters all submit bounded payloads to the same agent-owned sink. The sink constructs the record and owns the subjects, broker credentials, retries, and transport authority. A plugin author never picks a NATS subject, never holds a credential, and never writes a database schema. They produce an observation covered by an approved contract, and the platform handles getting it somewhere durable.

This is the part that compounds. Adding a new source of telemetry used to mean adding transport code to the agent. Now it means writing a contract and a producer, and the durability, ordering, backpressure, and replay properties come along for free.

Two runtimes, one answer

ServiceRadar is Go at the edge and Elixir in the control plane, which means two independent implementations have to agree on what a record means. They are held to shared byte corpora rather than to parallel prose descriptions: the same committed bytes produce the same decoded values, the same admission verdict, and the same rejection reason in both runtimes. New fields and enum values fail closed until someone deliberately assigns their semantics.

The practical payoff is that we can move a workload between runtimes, or add a third consumer in a third language, without relitigating what the data means. The contract is the specification, and both sides are provably reading it the same way.

Where it stands

The record carrier, the signed capability grammar, correlation against collection authority, compression admission, and cross-runtime conformance are implemented. The producer sink and spool, the gateway relay, JetStream provisioning, and projection into Postgres are rolling out behind them. Existing paths keep running throughout, and telemetry families move across by contract and deployment cohort rather than in one switchover.

Getting the scan fast was never the hard part. Getting the answer off the box intact, cheap enough to run continuously, and available to anything that wants to read it, is what decides how much of a network you can actually watch. That is the number we are chasing next.