The specific problem
A Text-to-SQL agent does not fail the way a dashboard fails. A broken dashboard is obviously broken. A broken agent returns a well-formed, confidently-worded, plausible number — and the reason nobody catches it is that catching it would require re-deriving the answer by hand, which is the entire reason the agent exists.
The defects that do this are structural, not linguistic: a join that fans out and triples revenue, a date filter placed in WHERE instead of the join condition, an SCD-2 lookup that silently picks the current row for a historical question. Generic LLM evaluation does not detect these because the language is fine.
What you get
- Coverage against a frozen denominator. Risk-Weighted Assurance Coverage over a schema surface fixed before execution, so the number cannot be improved by quietly shrinking what was tested. How exclusions are counted.
- Defects classified by consequence, not by how alarming they sound. A Sev-1 is specifically “confidently wrong with nothing in the output to signal it”. The rubric.
- A defensible answer. “An independent party ran 400 adversarial cases against a frozen configuration, found these defects at these severities, and here is the signed bundle” survives a follow-up question. “Our evals pass” does not.
- The things we could not test, stated and counted against coverage rather than dropped.
What it does not give you
Not a certification, not a badge, and not a pass/fail verdict — we do not yet have a defensible cut score and will not manufacture one. You keep the go/no-go decision and the residual-risk acceptance. That is a feature: a vendor that takes the decision off your desk has also taken the judgement out of your hands.
The first conversation
A readiness review establishes three things: which warehouse and agent, whether the semantic layer has a named Business Definition Owner, and whether we can read INFORMATION_SCHEMA and the dbt manifest. If the second is missing we will tell you the engagement is not worth running yet — ambiguity you pay to discover is a bad trade.
See the report → The eighteen failure domains Executive track