Where internal suites are systematically blind
- Relational invariants nobody wrote a test for. We compile them from your dbt manifest and
INFORMATION_SCHEMA— grain, cardinality, referential integrity, temporal boundaries — rather than from anyone’s expectations about the agent. - Authorisation under generated SQL. Row access policies and dynamic masking behave differently when the query is machine-authored. A path that returns a row the invoking identity could not have selected directly is a Sev-1 and rarely appears in an internal suite.
- Non-determinism. Identical inputs, different relation. Easy to test for, almost never tested for.
- Adversarial phrasing. A published floor of independent cases — at least 30% of every suite, 50% for Tier-1 agents — because a suite built only from your expectations tests your imagination. The floor.
How it fits your pipeline
The qabit adapter observes tool selection, parameters, loops and latency at runtime. The CLI and stored procedure give you install, auth, commands, exit codes and a CI recipe, so a run is a gate rather than a report someone emails you. Everything executes inside your perimeter: in the default evidence mode only hashes, pass/fail status, row counts, latency and a signed manifest cross the boundary.
Output you can act on
Case-level records — input, configuration, output, tool trajectory, error states and cost — in a documented non-proprietary schema. Confirmed defects are meant to become permanent regression cases in your own harness. We are not trying to be your eval platform; we are trying to hand you the cases you did not have.
Honest engineering constraints
BigQuery and Redshift parse but have no benchmark partition, so absence of findings means less there. Ten of eighteen failure domains are pilot or unvalidated. We test the agent’s reasoning, not the data underneath it — freshness and pipeline breakage stay with your observability stack. All of it, in one place.