Anomaly Investigator
An LLM investigation agent for operational time series, built on one rule: the model never touches raw data and never states a number it cannot cite.
How it works
- A user points the agent at a metric that behaved strangely. The model investigates through five deterministic tools (detectors, window statistics, figures); every tool call is logged under a stable id.
- Detectors report only rank based quantities (alarm budgets), so numbers from different detectors are comparable and cannot be inflated by rescaling a score.
- The report must cite the tool call behind every number. A provenance checker then re-reads the report as untrusted input and labels each numeric claim: verified, derived, wrong citation, uncited, or unsupported.
- On synthetic series with injected incidents, a deterministic judge scores the whole loop: did the agent find the real events, name them correctly, and avoid inventing numbers? The judge is code, because the thing that grades a non deterministic system should not itself be one.
Live evaluation
Three synthetic cases with exact ground truth, three runs per case per model, through an OpenAI compatible gateway. The spread across runs is part of the result, not noise to hide.
| metric | anthropic/claude-sonnet-4-5 9 runs | google/gemini-3.1-flash-lite 9 runs |
|---|---|---|
| event recall | 0.778 (sd 0.441) | 0.815 (sd 0.176) |
| type accuracy | 1.000 (sd 0.000) | 0.926 (sd 0.147) |
| finding precision | 0.943 (sd 0.151) | 1.000 (sd 0.000) |
| unsupported numbers | 0.012 (sd 0.015) | 0.016 (sd 0.033) |
| weak or missing citations | 0.352 (sd 0.222) | 0.541 (sd 0.226) |
The failure modes are more instructive than the means. The smaller model repeatedly skips the frequency shift, the one incident invisible to amplitude reasoning. The stronger model twice delivered no findings at all: it exhausted the turn budget on twenty-plus tool calls of due diligence before writing its report, which is where its recall variance comes from. Its one precision drop came from a run that promoted two of a detector's bought false alarms into findings, labeled "subtle" in its own words. Neither model invented an event from nothing.
What the trust layer caught
In evaluation run synthetic-0-r2 the model wrote
a sentence containing the number 3.297
that no tool call ever produced:
-3.297 (vs baseline min ~-1.05)
The provenance checker marked it unsupported
automatically. Nothing about this was staged; it is what happens when
a language model summarizes under pressure, and it is why the checker
exists. The full sentence is in the linked run report.
Demo investigations
Synthetic series, ground truth hidden from the agent
openai-compatible:google/gemini-3.1-flash-lite 7 tool calls 5 findings 0.0% unsupported numbers
Three injected incidents (spike burst, frequency shift, flatline). The agent sweeps the detectors, renders one figure per confirmed episode, and ends with structured findings the judge can score.
Real 2017 production metric, reviewing its own detector
openai-compatible:google/gemini-3.1-flash-lite 7 tool calls 2 findings 0.0% unsupported numbers
12,087 points at 5 minute sampling, shipped with the score, forecast band and 644 fired alerts of the production detector that originally watched it. The agent reviews that detector's work, including the real outage it finds on its own.
Honesty section
The evaluation numbers above describe the listed models on these three synthetic cases, nothing more. On the real metric there is no ground truth, and the reports say so instead of pretending. Alarm budgets on unlabeled data rest on a self reference that is circular by construction; the tool output repeats that caveat on every call. A deterministic mock provider exists so the whole pipeline runs and is tested without any key; mock numbers measure the pipeline, never a model, and none are shown on this page.