Engineering blog
November 18, 2026 Michael Freeman
Solution Packs and Declarative Checks in ServiceRadar
Declarative checks model active service probes and internal platform posture as versioned, secret-free JSON DAGs stored in CNPG and routed to either serviceradar_core or a single first-party WASI runner on the edge agent. Solution Packs bundle those checks with alert rules and dashboards into deterministic, Ed25519-signed .srpack archives distributed through signed catalogs and promoted across environments over pure-HTTPS two-way GitOps.
Earlier this year we shipped a
capability-based Wasm plugin system
for
serviceradar-agent
on top of wazero.
It solved safe extensibility for bespoke protocol probes: an operator can compile a Rust
or TinyGo module, declare its exact egress and credential capabilities, sign it, and run
it in a sandboxed edge runtime.
In practice, two recurring classes of monitoring work did not fit that workflow well. First, most service checks do not warrant a compiler toolchain. Checking an authenticated HTTP JSON endpoint, asserting a TLS certificate's remaining days, running a multi-step synthetic journey, or verifying a DNS record should not require building, signing, and uploading a new Wasm binary every time an assertion changes. Second, probing internal platform state over HTTP from an edge agent is inside-out. Questions such as whether archive storage is approaching its entitlement cap, whether active legal holds are pinning too much of the retention budget, or whether a continuous NVR recording policy has gone silent already live in CNPG and SRQL. Shipping those checks out to an edge agent only to have it call back into the control plane adds latency, credentials, and extra failure modes for no benefit.
A third problem shows up as soon as a team gets those checks right in one environment:
moving a coordinated bundle of checks, alert rules, and dashboards from
dev
to
staging
and
prod
without copying secrets, breaking local threshold overrides, or wiring SSH keys and host
git
binaries into application pods. Declarative Service & Posture Checks and Solution
Packs address all three with one data model.
Canonical JSON DAGs
Checks live in CNPG as versioned, secret-free JSONB graphs over a closed operator set—no embedded JavaScript, Lua, shell, or raw SQL.
Two runners, one pipeline
Core evaluates SRQL and platform posture in-process; Edge executes network probes in one shipped WASI runner. Both project into the same state and alert pipeline.
Signed packs & GitOps
Checks, rules, and dashboards package into deterministic Ed25519-signed
.srpack
archives and sync over pure-HTTPS Git REST APIs.
Canonical JSON-in-CNPG DAGs and the two-way visual builder
Every declarative check is stored in CNPG (platform.declarative_checks)
as a versioned (schema_version: 1),
secret-free JSONB Directed Acyclic Graph. Every node belongs to a closed, schema-validated
operator family: data acquisition nodes feed extraction, threshold assertion, and
decision-table nodes, terminating in an
alert_output
verdict. Because there is no arbitrary scripting runtime in the graph, the compiler can
statically validate every edge, verify parameter types, and route each branch to the right
execution environment before the check ever runs.
{
"schema_version": 1,
"slug": "api-health-and-tls",
"target_binding": { "kind": "singleton" },
"credential_slots": [
{
"slot": "target_http_api",
"provider": "generic_http",
"role": "bearer_token",
"purpose": "Authenticate synthetic HTTP API health requests",
"required": true
}
],
"nodes": [
{
"id": "probe",
"kind": "http_journey",
"config": {
"url": "https://api.example.com/healthz",
"credential_slot": "target_http_api",
"timeout_ms": 3000
}
},
{
"id": "tls",
"kind": "tls_handshake",
"config": { "host": "api.example.com", "port": 443 }
},
{
"id": "verdict",
"kind": "decision_table",
"config": { "default_status": "unknown" }
},
{ "id": "out", "kind": "alert_output" }
]
}
Opening a check at
/settings/declarative-checks/:slug
launches a two-way visual pipeline builder. The canvas combines
Palantir Blueprint v6
(@blueprintjs/core, @blueprintjs/icons, @blueprintjs/select),
strictly CSS-scoped inside
.sr-bp6-scope
with
.bp6-dark
theme synchronization and a local
portalContainer
so Blueprint styles never leak into the surrounding Tailwind v4 interface. Node wiring is
handled by @xyflow/react,
with
elkjs
(layered algorithm, elk.direction = "RIGHT")
providing deterministic auto-layout.
Beside the canvas sits a Monaco JSON editor
synchronized bidirectionally
with the graph. Dragging a node or changing a threshold on the canvas immediately emits
canonical 2-space JSON; editing the JSON in Monaco validates against the
schema_version: 1
contract with inline error diagnostics and updates the visual graph. New checks can start
from four 1-click starter graphs (HTTP JSON API Check, SRQL Posture Check, Multi-Step
Synthetic Journey, and Multi-Vantage Composite Check), and existing
ServiceCheck
and
CompositeCheck
resources keep running unchanged while 1-click import adapters clone them into
draft
DAGs for side-by-side verification without dual-alerting.
Clicking Test Run
executes an ephemeral sandbox evaluation of the draft
graph and decorates every node on the canvas with its live status, duration_ms,
extracted fields, redacted inputs ("[REDACTED]" for brokered credentials), and final verdict—without mutating
platform.service_state
or opening alert incidents. The same check definition handles both single-target and
fleet-wide evaluation through a single
target_binding
contract:
singleton
evaluates once per schedule tick (target_key: "singleton"),
srql_devices
fans out across a paged
in:devices ...
population (pruning stale instances when devices leave scope and reassigning them on DIRE
device merges), and
service_targets
evaluates across standalone or device-linked
ServiceTarget
records.
Two execution runners, one alerting pipeline
The DAG compiler inspects the node families in a check graph and routes execution
automatically. Internal platform questions stay inside
serviceradar_core
with zero Wasm overhead; active network probes run on the edge agent through one
first-party WASI runner; hybrid checks combine both.
| Topology | Node families | Execution path |
|---|---|---|
| Core-Only |
srql_query, platform_posture, extract_field, threshold_assert, decision_table,
alert_output
|
Evaluated in-process by Oban (CoreEvaluationWorker) in serviceradar_core. Zero Wasm or agent assignment overhead.
|
| Edge-Only |
http_request, http_journey, tcp_connect, tls_handshake,
dns_lookup
|
Compiled by
DeclarativeCheckAssignmentCompiler
for the single shipped
declarative-check-runner
WASI plugin on serviceradar-agent.
|
| Hybrid Edge + Core |
Edge probes from one or more agents joined with Core
srql_query
or platform_posture
|
Edge branches persist per-agent signal observations;
CoreEvaluationWorker
combines them with Core signals in a
decision_table
enforcing per-input
max_age_seconds
staleness and a mandatory catch-all row.
|
Core-Only checks are where internal posture monitoring belongs.
srql_query
runs bounded, read-only SRQL queries through the Ash adapter, while
platform_posture
calls into a closed first-party registry of domain evaluators. Built-in Pack #1
(io.serviceradar.storage-and-nvr-health),
shipped offline inside serviceradar_core,
uses four
platform_posture
evaluators to monitor archive storage entitlement utilization
(billable_bytes / plan_bytes or self-hosted object_budget_bytes),
legal hold-pin pressure (non_reapable_bytes as a share of the archive budget so evidence holds never silently starve retention),
archive credential and daily usage report freshness, and continuous NVR recording silence
(confirming every enabled, non-suspended continuous
CameraRecordingPolicy
has sealed a non-quarantined block within the silence window).
Edge-Only checks keep the security boundary we built for Wasm plugins without asking
operators to compile one binary per check. Instead, ServiceRadar ships a single
first-party WASI plugin, io.serviceradar.declarative-check-runner,
that interprets compiled edge DAGs for
http_journey
(with full
httptrace
phase waterfalls: dns_ms, tcp_ms, tls_ms, ttfb_ms, transfer_ms, total_ms), tcp_connect,
tls_handshake
(leaf_days_remaining),
and dns_lookup.
Because it runs inside the existing
wazero
plugin host, every declarative edge check automatically inherits per-assignment egress
allowlists, host-side CNPG
SecretBroker
credential injection (so raw tokens are never serialized into check definitions), and
durable
RecordSpool
to NATS JetStream delivery.
Regardless of where evaluation happened, every instance converges into the same
downstream pipeline: verdicts (healthy, degraded, down,
or unknown)
materialize in platform.declarative_check_instances,
project availability into
platform.service_state
(service_type: "declarative_check"),
and emit OCSF
health.state_change
events to NATS JetStream for
StatefulAlertRule
incident promotion.
Solution Packs (.srpack) and secret-free portability
A check on its own is rarely a complete operational unit; it usually ships alongside
alert rules and an overview dashboard. A Solution Pack
(pack_format_version: 1)
is an immutable, secret-free directory tree that maps 1:1 to a deterministic
.srpack
(tar.zst)
archive:
packs/storage-and-nvr-health/
manifest.json # metadata, typed config, credential_slots, files SHA-256 map
README.md # sanitized catalog documentation
checks/ # schema_version: 1 DeclarativeCheck DAGs
entitlement-utilization.json
hold-pin-pressure.json
archive-credential-health.json
camera-recording-silence.json
rules/ # StatefulAlertRule, EventRule, ZenRule (enabled or paused)
storage-hard-cap-alert.json
camera-silence-alert.json
dashboards/ # portable authored dashboard JSON
storage-and-nvr-overview.json
assets/
architecture.svg # allowlist-sanitized inline preview
Archives are built deterministically—lexicographically sorted relative paths, normalized
POSIX ustar headers (uid=0, gid=0, mtime=0,
mode 0644),
and Zstandard level 3 with no timestamp header—so two builds of the same tree produce the
exact same SHA-256 digest. On upload or catalog download,
ServiceRadar.Catalog.PackArchive
enforces strict limits (25 MB compressed, 64 MB uncompressed, 256 entries), rejects path
traversal, symlinks, hardlinks, and unlisted files fail-closed, verifies every file's
SHA-256 against manifest.files,
and runs server-side allowlist sanitization over SVGs and Markdown.
Two invariants make packs safe to share and safe to upgrade. First, packs are strictly secret-free. Neither
manifest.json
nor any resource file may carry plaintext credentials, AshCloak ciphertext, API tokens,
private keys, or instance-local credential UUIDs. Checks declare abstract
credential_slots
(provider, role, purpose) and typed
{"$pack_param": "<key>"}
bindings. During installation, the Guided Install Wizard maps each slot to a local
NetworkCredentialRule
or NetworkCredentialSecret.
On export or GitOps push,
SecretFreeValidator
strips local UUIDs and rejects forbidden secret field names, common provider token
prefixes, and PEM private key headers before any archive or Git blob is written.
Second, upgrades follow a RuleSeeder-style
reconciliation contract. Every pack-managed check, rule, and dashboard records pack_id, pack_version, managed,
template_fingerprint
(a canonical SHA-256 of structural fields, excluding layout and overrides), and parameter_overrides.
Tuning a threshold, schedule interval, or
enabled
toggle writes into
parameter_overrides
and keeps managed: true,
so upgrading a pack from
0.1.0
to
0.2.0
rolls forward structural fixes automatically while preserving operator thresholds and
local credential bindings. Structural edits (adding nodes or rewriting SRQL) detach the
resource (managed: false),
and diverged resources open a side-by-side Monaco diff in the Guided Upgrade Wizard
rather than being silently overwritten.
Signed catalogs, CLI tooling, and pure-HTTPS two-way GitOps
Published first-party and community packs are browsable today at
developer.serviceradar.cloud/packs
, backed by the Ed25519-signed catalog index at
developer.serviceradar.cloud/packs/index.json
.
Every ServiceRadar deployment seeds the official catalog with trusted first-party Ed25519
keys, and administrators can register additional public or private
:release_catalog
repositories (portal, GitHub, or Gitea). Each downloaded
.srpack
is fetched through
ServiceRadar.HTTP.EgressClient
(stripping
Authorization
headers across cross-origin pre-signed asset redirects) and verified against that
repository's Ed25519 public key before extraction. Private catalogs never call public
transparency logs, keeping internal pack metadata confidential.
Pack authors can validate, build, sign, and index packs offline in CI using @carverauto/serviceradar-cli:
serviceradar pack validate ./packs/example-http-health
serviceradar pack build ./packs/example-http-health \
--refresh-files -o ./dist/example-http-health-0.1.0.srpack
serviceradar pack sign ./dist/example-http-health-0.1.0.srpack \
--key ./keys/example-pack-signing-1.key \
--key-id example-pack-signing-1 \
-o ./dist/example-http-health-0.1.0.srpack.sig
serviceradar pack index ./dist \
-o ./dist/index.json \
--base-url https://packs.example.com/releases/v0.1.0
For teams promoting monitoring packs across
dev -> staging -> prod
through normal code review,
CatalogRepository
also supports
mode: :gitops_tree
for GitHub, GitLab, and Gitea / Forgejo. All reads and writes run strictly over HTTPS Git
Data REST APIs through
ServiceRadar.HTTP.EgressClient
(honoring
SERVICERADAR_EGRESS_PROXY
and setting
redirect: false
on mutating JSON calls). There is no host
git
binary, no persistent worktree, and no SSH private key on pod filesystems. Authentication
uses an AshCloak-encrypted
:api_token
PAT stored in platform.network_credential_secrets,
decrypted only at sync time inside
serviceradar_core
and never exposed to UI assigns or audit payloads.
In practice, an engineer authors and tests checks in dev,
exports the pack to a GitOps repository (by default creating branch
serviceradar/pack-<slug>-v<version>-<shortsha>
and opening a Pull Request or Merge Request after
SecretFreeValidator
passes), and merges the JSON diff after review.
CatalogSyncWorker
detects the new commit in
staging
and prod,
renders a dry-run preview of created, upgraded, unchanged, and diverged resources, binds
environment-local credential slots on first install, and records an OCSF audit event with
the actor, repository URL, branch, pack version, commit SHA, and PR URL. The result is
one portable JSON representation from visual authoring all the way to production
reconciliation.