Warehouses
| Warehouse | Status | Execution modality | What is specific to it |
|---|---|---|---|
| Snowflake | Validated | Native stored procedure EVALQA_RUN; evalqa CLI | Micro-partition pruning, clustering keys, remote spilling, row access policies, dynamic data masking, Cortex AI functions, Cortex Agents tool calls. Snowflake page → |
| Databricks (Unity Catalog, Genie) | Pilot | evalqa CLI against SQL warehouses; notebook runner | Unity Catalog row filters and column masks, Delta time travel for counterfactual probes, Genie space instructions as the semantic layer. Phase 2 extension per operating-plan risk RSK-05. |
| Google BigQuery | Roadmap | evalqa CLI | Partition and cluster pruning, row-level security policies, authorised views. Dialect parses; no benchmark partition yet. |
| AWS Redshift | Roadmap | evalqa CLI | Sort and distribution keys, RLS policies, dynamic data masking. Dialect parses; no benchmark partition yet. |
Agents
| Agent | Status | How we observe it | Notes |
|---|---|---|---|
| Snowflake Cortex Agents / Cortex Analyst | Validated | Generated SQL and tool-call traces via the agent API; runtime spans via qabit | Semantic model YAML treated as the definition source; tool parameter abuse family validated. |
| Databricks Genie | Pilot | Generated SQL via the conversation API | Space instructions and trusted assets treated as the semantic layer. |
| Custom LangChain / LlamaIndex Text-to-SQL | Validated | qabit OpenTelemetry adapter; or SQL log ingestion | Any agent whose generated SQL and tool calls we can capture. Framework-agnostic by design. |
| BI-native NL assistants (Looker, Tableau, Power BI) | Roadmap | SQL log ingestion only | Verifiable where the generated SQL is observable; not where it is opaque. |
Failure families by dialect
Validation status is per family and per dialect. A family validated on Snowflake is not thereby validated on Databricks; the benchmark partitions are separate.
| Failure family | Snowflake | Databricks | BigQuery | Redshift |
|---|---|---|---|---|
| Join fan-outs (1:N) | VALIDATED | PILOT | UNVALIDATED | UNVALIDATED |
| SCD-2 temporal logic | VALIDATED | PILOT | UNVALIDATED | UNVALIDATED |
| Fiscal calendar shift | VALIDATED | PILOT | UNVALIDATED | UNVALIDATED |
| Null predicate traps | VALIDATED | PILOT | UNVALIDATED | UNVALIDATED |
| Micro-partition / pruning scans | VALIDATED | n/a — different mechanics | UNVALIDATED | UNVALIDATED |
| Multi-tenant row leak | VALIDATED | PILOT | UNVALIDATED | UNVALIDATED |
| Tool parameter abuse | VALIDATED | PILOT | UNVALIDATED | UNVALIDATED |
| Prompt injection leak | VALIDATED | PILOT | UNVALIDATED | UNVALIDATED |
VALIDATED = family has a partition in the 500-item Ground-Truth Calibration Benchmark for that dialect and has passed the Sev-1 gate. PILOT = invariants defined and running in sprints; corpus expanding. UNVALIDATED = defined in the taxonomy only. The remaining ten of the 18 domains are PILOT or UNVALIDATED on every dialect.
Contractual consequence
Queries in a dialect or family we have not validated are recorded as EXCLUDED_UNSUPPORTED, counted against coverage, and shown on the scorecard. They are never quietly passed. See scope & exclusions.