What we detect
Eighteen named failure domains, of which eight are validated against the calibration benchmark and ten are marked pilot or unvalidated. The families that matter most are the ones that produce plausible output: relational join fan-out, SCD-2 temporal boundary errors, filter placement, grain violations, row access policy leakage and prompt injection reaching the query layer. The full taxonomy, with the validation status of each →
How the tests are generated
- Deterministic relational invariants compiled from your dbt manifest and
INFORMATION_SCHEMA— grain, cardinality, referential integrity, temporal boundaries. Not written by hand, not written from expectations. - Adversarial challenge queries authored independently, at a published floor of 30% of every suite (50% for Tier-1 agents). The floor.
- Counterfactual data probes that change the data and check whether the agent’s answer changes the way it should.
How results are adjudicated
Every invariant lands in one of four states: PASS, DEFECT_CONFIRMED, EXCLUDED_UNSUPPORTED or INDETERMINATE. Excluded items count against coverage and are never reclassified as passes. Confirmed defects are severity-classified against a published rubric by paired raters with sequential lead calibration.
Where it runs
Inside your perimeter. In the default evidence mode no raw query, result row or prompt crosses the boundary — only hashes, pass/fail status, row counts, latency and a signed manifest. Two further modes exist for exception triage and regulated schemas, both requiring your written authorisation. Evidence modes and EALs →
What you get out
- A signed assurance bundle with a commitment hash pinning the exact configuration tested, verifiable by a third party.
- Case-level records — input, configuration, output, tool trajectory, error states, cost — in a documented non-proprietary schema.
- Risk-Weighted Assurance Coverage against a denominator frozen before execution.
How it connects
The qabit adapter for runtime observation; a CLI and stored procedure with exit codes for CI gating; Slack and Asana for submission routing. Integrations →
Warehouse coverage
Snowflake Cortex Agents, Databricks Genie and custom Text-to-SQL pipelines are the validated surface. BigQuery and Redshift parse but have no benchmark partition. Coverage matrix →