Skip to content

ServiceRadar

Open navigation

Engineering blog

November 18, 2026 Michael Freeman

Solution Packs and Declarative Checks in ServiceRadar

Declarative checks model active service probes and internal platform posture as versioned, secret-free JSON DAGs stored in CNPG and routed to either serviceradar_core or a single first-party WASI runner on the edge agent. Solution Packs bundle those checks with alert rules and dashboards into deterministic, Ed25519-signed .srpack archives distributed through signed catalogs and promoted across environments over pure-HTTPS two-way GitOps.

solution-packs declarative-checks gitops wasm elixir srql observability

Earlier this year we shipped a capability-based Wasm plugin system for serviceradar-agent on top of wazero. It solved safe extensibility for bespoke protocol probes: an operator can compile a Rust or TinyGo module, declare its exact egress and credential capabilities, sign it, and run it in a sandboxed edge runtime.

In practice, two recurring classes of monitoring work did not fit that workflow well. First, most service checks do not warrant a compiler toolchain. Checking an authenticated HTTP JSON endpoint, asserting a TLS certificate's remaining days, running a multi-step synthetic journey, or verifying a DNS record should not require building, signing, and uploading a new Wasm binary every time an assertion changes. Second, probing internal platform state over HTTP from an edge agent is inside-out. Questions such as whether archive storage is approaching its entitlement cap, whether active legal holds are pinning too much of the retention budget, or whether a continuous NVR recording policy has gone silent already live in CNPG and SRQL. Shipping those checks out to an edge agent only to have it call back into the control plane adds latency, credentials, and extra failure modes for no benefit.

A third problem shows up as soon as a team gets those checks right in one environment: moving a coordinated bundle of checks, alert rules, and dashboards from dev to staging and prod without copying secrets, breaking local threshold overrides, or wiring SSH keys and host git binaries into application pods. Declarative Service & Posture Checks and Solution Packs address all three with one data model.

Canonical JSON DAGs

Checks live in CNPG as versioned, secret-free JSONB graphs over a closed operator set—no embedded JavaScript, Lua, shell, or raw SQL.

Two runners, one pipeline

Core evaluates SRQL and platform posture in-process; Edge executes network probes in one shipped WASI runner. Both project into the same state and alert pipeline.

Signed packs & GitOps

Checks, rules, and dashboards package into deterministic Ed25519-signed .srpack archives and sync over pure-HTTPS Git REST APIs.

Canonical JSON-in-CNPG DAGs and the two-way visual builder

Every declarative check is stored in CNPG (platform.declarative_checks) as a versioned (schema_version: 1), secret-free JSONB Directed Acyclic Graph. Every node belongs to a closed, schema-validated operator family: data acquisition nodes feed extraction, threshold assertion, and decision-table nodes, terminating in an alert_output verdict. Because there is no arbitrary scripting runtime in the graph, the compiler can statically validate every edge, verify parameter types, and route each branch to the right execution environment before the check ever runs.

{
  "schema_version": 1,
  "slug": "api-health-and-tls",
  "target_binding": { "kind": "singleton" },
  "credential_slots": [
    {
      "slot": "target_http_api",
      "provider": "generic_http",
      "role": "bearer_token",
      "purpose": "Authenticate synthetic HTTP API health requests",
      "required": true
    }
  ],
  "nodes": [
    {
      "id": "probe",
      "kind": "http_journey",
      "config": {
        "url": "https://api.example.com/healthz",
        "credential_slot": "target_http_api",
        "timeout_ms": 3000
      }
    },
    {
      "id": "tls",
      "kind": "tls_handshake",
      "config": { "host": "api.example.com", "port": 443 }
    },
    {
      "id": "verdict",
      "kind": "decision_table",
      "config": { "default_status": "unknown" }
    },
    { "id": "out", "kind": "alert_output" }
  ]
}

Opening a check at /settings/declarative-checks/:slug launches a two-way visual pipeline builder. The canvas combines Palantir Blueprint v6 (@blueprintjs/core, @blueprintjs/icons, @blueprintjs/select), strictly CSS-scoped inside .sr-bp6-scope with .bp6-dark theme synchronization and a local portalContainer so Blueprint styles never leak into the surrounding Tailwind v4 interface. Node wiring is handled by @xyflow/react, with elkjs (layered algorithm, elk.direction = "RIGHT") providing deterministic auto-layout.

Beside the canvas sits a Monaco JSON editor synchronized bidirectionally with the graph. Dragging a node or changing a threshold on the canvas immediately emits canonical 2-space JSON; editing the JSON in Monaco validates against the schema_version: 1 contract with inline error diagnostics and updates the visual graph. New checks can start from four 1-click starter graphs (HTTP JSON API Check, SRQL Posture Check, Multi-Step Synthetic Journey, and Multi-Vantage Composite Check), and existing ServiceCheck and CompositeCheck resources keep running unchanged while 1-click import adapters clone them into draft DAGs for side-by-side verification without dual-alerting.

Clicking Test Run executes an ephemeral sandbox evaluation of the draft graph and decorates every node on the canvas with its live status, duration_ms, extracted fields, redacted inputs ("[REDACTED]" for brokered credentials), and final verdict—without mutating platform.service_state or opening alert incidents. The same check definition handles both single-target and fleet-wide evaluation through a single target_binding contract: singleton evaluates once per schedule tick (target_key: "singleton"), srql_devices fans out across a paged in:devices ... population (pruning stale instances when devices leave scope and reassigning them on DIRE device merges), and service_targets evaluates across standalone or device-linked ServiceTarget records.

Two execution runners, one alerting pipeline

The DAG compiler inspects the node families in a check graph and routes execution automatically. Internal platform questions stay inside serviceradar_core with zero Wasm overhead; active network probes run on the edge agent through one first-party WASI runner; hybrid checks combine both.

Topology Node families Execution path
Core-Only srql_query, platform_posture, extract_field, threshold_assert, decision_table, alert_output Evaluated in-process by Oban (CoreEvaluationWorker) in serviceradar_core. Zero Wasm or agent assignment overhead.
Edge-Only http_request, http_journey, tcp_connect, tls_handshake, dns_lookup Compiled by DeclarativeCheckAssignmentCompiler for the single shipped declarative-check-runner WASI plugin on serviceradar-agent.
Hybrid Edge + Core Edge probes from one or more agents joined with Core srql_query or platform_posture Edge branches persist per-agent signal observations; CoreEvaluationWorker combines them with Core signals in a decision_table enforcing per-input max_age_seconds staleness and a mandatory catch-all row.
Table 1. Compiler routing across Core-Only, Edge-Only, and Hybrid declarative check topologies.

Core-Only checks are where internal posture monitoring belongs. srql_query runs bounded, read-only SRQL queries through the Ash adapter, while platform_posture calls into a closed first-party registry of domain evaluators. Built-in Pack #1 (io.serviceradar.storage-and-nvr-health), shipped offline inside serviceradar_core, uses four platform_posture evaluators to monitor archive storage entitlement utilization (billable_bytes / plan_bytes or self-hosted object_budget_bytes), legal hold-pin pressure (non_reapable_bytes as a share of the archive budget so evidence holds never silently starve retention), archive credential and daily usage report freshness, and continuous NVR recording silence (confirming every enabled, non-suspended continuous CameraRecordingPolicy has sealed a non-quarantined block within the silence window).

Edge-Only checks keep the security boundary we built for Wasm plugins without asking operators to compile one binary per check. Instead, ServiceRadar ships a single first-party WASI plugin, io.serviceradar.declarative-check-runner, that interprets compiled edge DAGs for http_journey (with full httptrace phase waterfalls: dns_ms, tcp_ms, tls_ms, ttfb_ms, transfer_ms, total_ms), tcp_connect, tls_handshake (leaf_days_remaining), and dns_lookup. Because it runs inside the existing wazero plugin host, every declarative edge check automatically inherits per-assignment egress allowlists, host-side CNPG SecretBroker credential injection (so raw tokens are never serialized into check definitions), and durable RecordSpool to NATS JetStream delivery.

Regardless of where evaluation happened, every instance converges into the same downstream pipeline: verdicts (healthy, degraded, down, or unknown) materialize in platform.declarative_check_instances, project availability into platform.service_state (service_type: "declarative_check"), and emit OCSF health.state_change events to NATS JetStream for StatefulAlertRule incident promotion.

Solution Packs (.srpack) and secret-free portability

A check on its own is rarely a complete operational unit; it usually ships alongside alert rules and an overview dashboard. A Solution Pack (pack_format_version: 1) is an immutable, secret-free directory tree that maps 1:1 to a deterministic .srpack (tar.zst) archive:

packs/storage-and-nvr-health/
  manifest.json                  # metadata, typed config, credential_slots, files SHA-256 map
  README.md                      # sanitized catalog documentation
  checks/                        # schema_version: 1 DeclarativeCheck DAGs
    entitlement-utilization.json
    hold-pin-pressure.json
    archive-credential-health.json
    camera-recording-silence.json
  rules/                         # StatefulAlertRule, EventRule, ZenRule (enabled or paused)
    storage-hard-cap-alert.json
    camera-silence-alert.json
  dashboards/                    # portable authored dashboard JSON
    storage-and-nvr-overview.json
  assets/
    architecture.svg             # allowlist-sanitized inline preview

Archives are built deterministically—lexicographically sorted relative paths, normalized POSIX ustar headers (uid=0, gid=0, mtime=0, mode 0644), and Zstandard level 3 with no timestamp header—so two builds of the same tree produce the exact same SHA-256 digest. On upload or catalog download, ServiceRadar.Catalog.PackArchive enforces strict limits (25 MB compressed, 64 MB uncompressed, 256 entries), rejects path traversal, symlinks, hardlinks, and unlisted files fail-closed, verifies every file's SHA-256 against manifest.files, and runs server-side allowlist sanitization over SVGs and Markdown.

Two invariants make packs safe to share and safe to upgrade. First, packs are strictly secret-free. Neither manifest.json nor any resource file may carry plaintext credentials, AshCloak ciphertext, API tokens, private keys, or instance-local credential UUIDs. Checks declare abstract credential_slots (provider, role, purpose) and typed {"$pack_param": "<key>"} bindings. During installation, the Guided Install Wizard maps each slot to a local NetworkCredentialRule or NetworkCredentialSecret. On export or GitOps push, SecretFreeValidator strips local UUIDs and rejects forbidden secret field names, common provider token prefixes, and PEM private key headers before any archive or Git blob is written.

Second, upgrades follow a RuleSeeder-style reconciliation contract. Every pack-managed check, rule, and dashboard records pack_id, pack_version, managed, template_fingerprint (a canonical SHA-256 of structural fields, excluding layout and overrides), and parameter_overrides. Tuning a threshold, schedule interval, or enabled toggle writes into parameter_overrides and keeps managed: true, so upgrading a pack from 0.1.0 to 0.2.0 rolls forward structural fixes automatically while preserving operator thresholds and local credential bindings. Structural edits (adding nodes or rewriting SRQL) detach the resource (managed: false), and diverged resources open a side-by-side Monaco diff in the Guided Upgrade Wizard rather than being silently overwritten.

Signed catalogs, CLI tooling, and pure-HTTPS two-way GitOps

Published first-party and community packs are browsable today at developer.serviceradar.cloud/packs , backed by the Ed25519-signed catalog index at developer.serviceradar.cloud/packs/index.json . Every ServiceRadar deployment seeds the official catalog with trusted first-party Ed25519 keys, and administrators can register additional public or private :release_catalog repositories (portal, GitHub, or Gitea). Each downloaded .srpack is fetched through ServiceRadar.HTTP.EgressClient (stripping Authorization headers across cross-origin pre-signed asset redirects) and verified against that repository's Ed25519 public key before extraction. Private catalogs never call public transparency logs, keeping internal pack metadata confidential.

Pack authors can validate, build, sign, and index packs offline in CI using @carverauto/serviceradar-cli:

serviceradar pack validate ./packs/example-http-health
serviceradar pack build ./packs/example-http-health \
  --refresh-files -o ./dist/example-http-health-0.1.0.srpack
serviceradar pack sign ./dist/example-http-health-0.1.0.srpack \
  --key ./keys/example-pack-signing-1.key \
  --key-id example-pack-signing-1 \
  -o ./dist/example-http-health-0.1.0.srpack.sig
serviceradar pack index ./dist \
  -o ./dist/index.json \
  --base-url https://packs.example.com/releases/v0.1.0

For teams promoting monitoring packs across dev -> staging -> prod through normal code review, CatalogRepository also supports mode: :gitops_tree for GitHub, GitLab, and Gitea / Forgejo. All reads and writes run strictly over HTTPS Git Data REST APIs through ServiceRadar.HTTP.EgressClient (honoring SERVICERADAR_EGRESS_PROXY and setting redirect: false on mutating JSON calls). There is no host git binary, no persistent worktree, and no SSH private key on pod filesystems. Authentication uses an AshCloak-encrypted :api_token PAT stored in platform.network_credential_secrets, decrypted only at sync time inside serviceradar_core and never exposed to UI assigns or audit payloads.

In practice, an engineer authors and tests checks in dev, exports the pack to a GitOps repository (by default creating branch serviceradar/pack-<slug>-v<version>-<shortsha> and opening a Pull Request or Merge Request after SecretFreeValidator passes), and merges the JSON diff after review. CatalogSyncWorker detects the new commit in staging and prod, renders a dry-run preview of created, upgraded, unchanged, and diverged resources, binds environment-local credential slots on first install, and records an OCSF audit event with the actor, repository URL, branch, pack version, commit SHA, and PR URL. The result is one portable JSON representation from visual authoring all the way to production reconciliation.