1. Ten of eighteen failure domains are not validated
The taxonomy on failure domains lists eighteen families. Eight carry a validated marker against the calibration benchmark. Ten are marked pilot or unvalidated, which means we have a detector and a rationale but not a measured sensitivity. A defect in an unvalidated domain may be found; we cannot tell you how often it is missed.
2. Sensitivity is measured on a benchmark, not on your data
Detection sensitivity comes from a sequestered 500-item Ground-Truth Calibration Benchmark — 250 seeded defects, 250 clean controls, eight failure families. fact That is a statement about the benchmark. Transfer to your warehouse is an assumption, and the direction of the error is unknown: real schemas are messier than seeded ones, and real agents fail in ways a seeded corpus does not anticipate.
3. The gold set is thin in exactly the places you care about
Seeded defects are densest where defects are easy to seed: join fan-out, SCD-2 temporal boundaries, filter placement. They are thinnest where correctness is contested rather than computable — a metric that two departments define differently, a business rule nobody has written down. Those are the expensive failures, and our coverage of them depends on your Business Definition Owner, not on our detectors.
4. We test the agent’s reasoning, not the data underneath it
Freshness, volume anomalies, upstream schema drift and pipeline breakage are explicitly out of scope. An agent can produce flawless SQL over stale or wrong data and pass every invariant we run. A clean EvalQA bundle is not a statement that the answer was correct — only that the reasoning that produced it held.
5. Prompt sensitivity means results have a shelf life
Agent behaviour changes when the system prompt, the semantic model, the tool definitions or the underlying model version change — sometimes substantially, for edits that look cosmetic. A bundle is evidence about a frozen configuration on a date, which is why the commitment hash pins all four. Treating a three-month-old bundle as current is a misuse we cannot prevent.
6. Contamination risk in adversarial cases
Our challenge suites are reused across engagements. If a customer’s team sees the cases and tunes against them, the suite measures tuning rather than capability. The 30% floor of independent adversarial cases exists to limit this, and we rotate. We have no way to detect that it has happened.
7. Two dialects parse but are not benchmarked
BigQuery and Redshift are supported by the parser and have no benchmark partition. Findings there are reported with the dialect flagged, and the absence of findings means considerably less than it does on Snowflake. See warehouse coverage.
8. Independence is structural, not perfect
We are paid by the organisation whose agent we evaluate. The Independence Charter sets out the conflict rules, the fixed-fee model with no success fee, and the work we refuse. Those constraints reduce the incentive to produce a flattering result; they do not eliminate it. The published judge–human agreement statistics and the verifiable bundle exist so you do not have to take our word for the result.
9. What would change our mind
Our own kill rule: if Incremental Consequential Finding Yield — the share of critical and high defects we find that the native warehouse evaluation did not — comes in under 20% across five customers, the product should not exist and we will say so. planning That measurement is not yet complete.
How the method works → Scope & exclusions Agreement statistics