AI failure observability and causal diagnosis
An AI product's worst failures return HTTP 200, in perfectly valid JSON, at normal latency, and every classical monitor stays green.
Traditional software fails loudly. A 500, a stack trace, a latency spike. You find out because something goes red. An AI product fails quietly: it returns a confident, well-formatted, wrong answer with a 200 status and normal timings. Uptime looks perfect while the product is failing its users.
The existing tools cover two of the three questions. Tracing platforms answer what happened. Eval platforms answer whether the output was good. Neither answers the one that actually matters when you are on call at midnight: why did it fail, and which step caused it.
Worse, the answer they do give is a number. A groundedness score of 0.62 tells you nothing actionable. It does not tell you that your reranker is misconfigured, or that a chunk-size change three weeks ago broke a whole route.
Anvaya adds the third thing: diagnosis. It ingests execution traces, detects failures against a named taxonomy, and attributes each one causally across the span tree. The output is not a confidence score. It is an incident that names the failure mode, points at the originating span, lists the downstream symptoms, and suggests remediation.
The taxonomy is the foundation. 56 named failure modes across 8 families, built from published research rather than invented: it covers all 14 modes from the MAST multi-agent study, all 7 failure points from the Barnett RAG paper, and 6 entries from the OWASP LLM Top 10. Each mode carries a definition, a source, and a remediation note.
Detection runs cheapest-first across four tiers, and that ordering is driven by the data rather than by frugality. The most frequent failure modes in the research are also the cheapest to catch. Step repetition, the single most common multi-agent failure, is a cycle in the span tree. Paying a language model to detect a graph cycle is indefensible.
Observability that can take down production is worse than no observability. The SDK never throws, and the transport is fire-and-forget with bounded buffers. You can stop the collector mid-run and every instrumented request still completes at exactly the same speed. This is enforced as an architectural decision record, not a code-review habit.
Every finding renders a qualitative confidence band: possible, likely, or confirmed. A lexical-overlap groundedness check is capped so that it structurally cannot present as confirmed, no matter how strong the overlap. False-positive fatigue kills an observability tool faster than missed detections do, so the ceiling is built into the detector rather than left to a threshold someone will tune away.
The structural and statistical tiers work entirely on metadata, so the default configuration never ships prompt or completion text off the machine. Redaction runs in-process before transport and again server-side as defence in depth. Security findings record the class of secret that leaked, never the value, because a finding that quotes the secret has re-created the exposure it is reporting.
Two taxonomy modes, loss of conversation history and conversation reset, are defined across turns. In a real chat application each turn is its own trace, so a trace-scoped detector can never see them. Anvaya therefore analyses sessions as well, comparing each trace against its predecessor. This works with content capture off, because message counts are metadata rather than content.
The judge tier is billed and off by default. With no key configured it reports as skipped rather than failing, which is an informational outcome and not an error.
Lexical groundedness is a heuristic, not entailment. It is capped at a confidence that cannot read as confirmed for exactly that reason.
Cohort analysis needs a minimum sample before it will claim a lift, so a low-traffic route will stay silent rather than produce a confident-looking result from four requests.