Operations · Anomaly engine
Anomaly detection
Surface operationally meaningful metric changes without drowning the NOC in per-sample noise.
Catch sustained metric spikes, drift, and capacity runway at the edge—then fold findings into bounded episodes that feed the normal event and alert workflow instead of one alarm per noisy sample.
How it fits together
Edge scores fast. Core remembers the long view.
Detection is split on purpose. The edge agent scores live metric streams for low-latency spikes and drift. The platform builds seasonal baselines, stores durable episodes, runs capacity forecasting, and drives alerts and findings in the UI.
Anomaly detection flow
Metrics in → detectors → episode transitions → alerts and findings
Edge agent
Where samples are scored
Metric streams
Host, interface, and other numeric series already collected on the agent
Anomaly add-on
Rolling robust spike scoring · deseasonalized CUSUM drift · local episode state
Episode transitions
Open · update · clear — not one event per sample
Core platform
Where the long view lives
Seasonal baselines
Hour-of-week shape delivered back to the edge so drift judges against the right reference—not raw daily peaks
Episode store
Durable open / update / clear state across producers
Capacity forecasting
Runway episodes for consumable resources
Alerts + findings UI
Stateful alert rules on transitions · device and observability panels operators already use
Loop back: baselines flow edge ← core; transitions flow edge → core. Settings project fleet defaults to the add-on without a separate management plane on every host.
Detection methods
Three detectors, three kinds of “something changed”
The engine is intentionally not a black-box “AI score.” Each path has a clear statistical job, conservative defaults, and guards so quiet networks stay quiet and real shifts still surface.
Spike
Robust z-score on a rolling window
Spikes use a rolling median / MAD baseline (median absolute deviation)—resistant to outliers that would wreck a plain mean and standard deviation. Each sample is scored as a robust z-score against that window. Extreme values are admitted at a clipped decision boundary so a surge cannot train itself into the baseline forever, while every sample still ages the window.
- Warmup: needs enough clean samples before findings emit
- Confirm: consecutive breaching slots before an episode opens
- Adopt: a stable new level clears as adopted instead of paging forever
- Guards: dispersion floors and saturation checks so near-zero noise and empty gauges stay quiet
Drift
CUSUM on a deseasonalized series
Drift uses CUSUM (cumulative sum) to catch a level that has shifted and stayed shifted—the kind of change a rolling spike detector can absorb into “the new normal” too quickly. For classes with daily or weekly shape, the edge scores a deseasonalized residual against hour-of-week baselines from the platform—not the raw diurnal peaks.
- No baseline → no drift: missing seasonal coverage is quiet, never raw unseasonalized CUSUM
- Latch and confirm: practical effect-size gates before an episode opens
- Open is not noisy: ongoing drift does not emit every poll cycle
- Recover or adopt: clear when the series returns, or when a non-saturated new level is accepted
Capacity
Significance-gated runway forecasts
Capacity forecasting runs on the platform for monotone consumable resources by default—disk usage, memory working set—not every bursty gauge. A runway finding needs enough history, a robust slope that excludes zero, persistence across runs, and an ETA gated on a prediction-interval lower bound—not a single optimistic fit.
- Default targets: disk and similar consumables
- Off by default: CPU, interface util, and other mean-reverting series unless you opt in
- Transition emits: projected ↔ cleared, not once per series every hour
- Explainable skips: series the forecaster declines are counted with reasons on health views
Severity is bounded evidence plus impact—not an unbounded accumulator mapped straight to Critical. Edge drift tops out at High; Critical needs high evidence, an impact test (for CPU/memory, host aggregate saturation), and minimum duration. Full knobs and proof gates live in the Anomaly Engine reference and tuning guide .
Episode model
Operators work episodes, not detector ticks
Each OCSF row is a lifecycle transition. Heartbeats refresh state without spawning a new finding every cycle. A benign regime change should cost one open and one clear.
-
Open
Evidence confirmed
Consecutive breaches and effect-size gates pass. A deterministic episode identity is created.
-
Update
Escalate or flap-fold
Severity-band escalation, or a short re-open that reuses the same episode instead of pager thrash.
-
Clear
Recovered or adopted
Series returned inside the clear band, or a stable non-saturated level became the new baseline.
Why volume stays bounded
Volume tracks transitions, not sample count. Emission budgets and cooldowns prioritize clears and high-severity opens in a storm; shed rollups account for what was suppressed so nothing disappears without a trail.
Silence is monitored too
Tripwires watch for a dead pipeline—alert-path liveness, ingest silence while metrics still flow, and stale seasonal baseline delivery—so “no findings” is not always healthy.
Sustained signal, not sample spam
The goal is not to store an event for every noisy sample. ServiceRadar looks for sustained, operationally useful changes, opens a finding when evidence holds, and clears it when the series recovers or a new level becomes normal—so a benign regime change costs one open and one clear, not a finding every poll.
Detect at the edge, operate from the center
Low-latency scoring runs on the edge agent so you do not ship every sample for central evaluation. Seasonal baselines and durable episode state live on the platform. Fleet defaults come from Settings; profiles and assignments still win for canaries and site-specific overrides.
- Edge evaluation next to the metrics you already collect
- Hour-of-week baselines where daily or weekly shape matters
- Settings project fleet defaults; explicit profile params override for canaries
- Same agent and add-on assignment model as other edge capabilities
Tunable for the real network
Defaults are deliberately conservative. Turn down a noisy metric family without rewriting the whole fleet, denylist series that should never produce findings, and keep emission budgets so a storm still prioritizes clears and high-severity opens.
- Per-class overrides instead of one global sensitivity for everything
- Denylist for metrics that are metadata or meaningless as anomalies
- Emission budgets and cooldowns so storms stay accountable
- Liveness and silence checks so a dead pipeline is visible
Detect and alert—not auto-remediate
An anomaly finding means the detector saw a statistically and operationally meaningful transition. It does not automatically restart services, throttle traffic, resize resources, or change policy. Remediation stays a human and workflow decision on top of clear findings.
FAQs
Where does anomaly detection run?
Scoring for spikes and drift runs on the edge agent against metric streams you already collect. Seasonal baselines, durable episode state, capacity forecasting, and presentation live on the platform.
Will every sample create an alert?
No. Confirmed transitions open or update episodes; heartbeats refresh state without new finding rows. A stable regime change should cost one open and one clear, not a finding every poll cycle.
How do daily and weekly patterns work?
Where seasonality matters, the platform delivers hour-of-week baselines so drift can judge against the right reference. Without a usable baseline for a series, drift stays quiet for that series rather than firing on raw seasonal swings.
What is capacity runway?
Capacity forecasting projects when a consumable resource such as disk usage may cross a limit, and emits runway episodes on state changes—not once per series every hour. Bursty gauges such as CPU are excluded by default unless you opt them in.
Can we auto-remediate from anomaly findings?
Not from this path today. Anomaly detection is detect-and-alert: findings feed events and alerts so people and existing workflows can act. Automated throttle, restart, resize, and policy mutation are out of scope for the current release.
Where do operators tune sensitivity?
Start in Settings > Anomaly Detection for fleet defaults. Use profile or assignment overrides for canaries and site-specific knobs, and per-class settings when only one metric family is noisy.
Where are the operator docs?
Tuning ownership, spikes and drift, severity, episodes, and rollout guidance are in the Anomaly Detection docs .
Where is the engine reference?
Architecture, episode contract, severity rules, and verification gates are in the Anomaly Engine reference .
Next step
Ready for findings that respect episode lifecycle?
Walk through edge scoring, seasonal baselines, and alert transitions with our team—or open the demo and the anomaly docs.