In scope
- The agent’s generated SQL and tool calls against the pre-frozen schema surface: every model, column and relationship listed in the signed denominator.
- The semantic layer the agent reasons over — Cortex semantic model YAML, Genie space instructions, or the dbt metrics layer — as the definition source for invariants.
- Runtime behaviour observed through the
qabitadapter: tool selection, parameters, loops, latency. - Authorisation behaviour: row access policies, dynamic masking and tenant isolation as exercised by generated queries.
Technical exclusions
| Exclusion | Why | How it shows up |
|---|---|---|
| Insufficient access or definitions | We cannot verify a surface we cannot read (no INFORMATION_SCHEMA / manifest access) or a definition nobody owns (no named Business Definition Owner). Earlier drafts called this “data access intransigence”; the professional term is the one on the left, and the engagement is declined in writing. | Declined at readiness review, or scoped out of the denominator before it is frozen. |
| Unvalidated dialect or family | BigQuery and Redshift parse but have no benchmark partition; ten of eighteen failure domains are PILOT or UNVALIDATED. See coverage. | EXCLUDED_UNSUPPORTED, counted against RWAC and shown on the scorecard. |
| Queries outside the contracted surface | The denominator is frozen before execution precisely so that it cannot grow or shrink to flatter the coverage number. | EXCLUDED_UNSUPPORTED with reason code OUT_OF_SURFACE. |
| Pipeline and table data quality | Freshness, volume, schema drift in the data itself are observability problems. We test what the agent’s reasoning does with the data, not the data. | Not tested; referred to your observability tool. |
| The foundation model itself | We do not evaluate the LLM in the abstract, publish model comparisons, or run public leaderboards. | Not tested. See the Independence Charter. |
| Remediation | Developmental separation: we do not write your prompts, semantic models or dbt models. | Suggested correction in the Tier-2 report; retest after your fix. |
| Statutory certification | We produce technical evidence. Compliance determinations are your counsel’s. | Standards mapping only. |
How exclusions are counted
Every invariant lands in one of four states and the counts are printed on the Tier-1 scorecard. EXCLUDED_UNSUPPORTED items reduce Risk-Weighted Assurance Coverage; they are never reclassified as passes. INDETERMINATE items trigger the Customer Business Definition Owner’s contractual 48-hour window; if the window lapses the item stays INDETERMINATE in the bundle, with the date it was raised.
The 30% floor
Regardless of how much golden truth a customer supplies, at least 30% of every challenge suite is independent EvalQA adversarial cases; Tier-1 agents require at least 50%. A suite built only from the customer’s own expectations tests the customer’s imagination, not the agent.
Changing scope mid-engagement
The denominator can be extended by a signed amendment, which starts a new bundle lineage; the original bundle is never edited. Shrinking the denominator after execution is not possible: the commitment hash would no longer match.