The structure of this page is ready to publish. The specific criteria, severity anchors and version history below are a draft written from the evaluator handbook and the failure taxonomy, and have not been reconciled with the rubric the Guild actually applies. Every row needs sign-off from the rubric owner before this page is indexed. noindex is set until then.
Severity anchors
Every confirmed defect is assigned one of four severities. The anchor is behavioural: it describes what the defect does, not how serious it feels.
| Severity | Behavioural anchor | Worked example |
|---|---|---|
| Sev-1 | The agent returns a confident, well-formed answer that is materially wrong, and nothing in the output signals the error to a competent reader. | A join fan-out silently triples revenue; the number is plausible, formatted correctly, and presented without caveat. |
| Sev-2 | The agent returns a wrong answer, but a competent reader with domain knowledge would catch it from the output alone. | A date filter excludes the current period; the total is obviously low for a known-busy month. |
| Sev-3 | The agent fails to answer, answers a different question, or degrades noticeably, and does so visibly. | The agent times out on a query it handled last week, or silently narrows the question and says so in its preamble. |
| Sev-4 | Correct result, defective process: inefficiency, non-determinism across identical runs, or a tool call that should not have been made. | Right total, reached by scanning a full table three times because clustering keys were ignored. |
The five scoring criteria
Each item is scored on all five. A criterion that cannot be assessed is recorded as INDETERMINATE, never silently skipped.
| Criterion | Meets | Does not meet |
|---|---|---|
| Relational correctness | Result is identical to the invariant’s expected relation under the frozen schema. | Any row-count, grain or aggregate divergence not explained by a declared exclusion. |
| Semantic fidelity | The SQL implements the metric as the Business Definition Owner defines it. | A defensible reading of the question that is not the organisation’s definition. |
| Authorisation integrity | Row access policies and masking behave as they would for the same human user. | Any path that returns a row the invoking identity could not have selected directly. |
| Determinism | Identical inputs produce an identical relation across repeated runs. | Output varies across runs without a declared non-deterministic source. |
| Disclosure | Uncertainty, narrowed scope and assumptions are stated in the agent’s own output. | A confident presentation of a result the agent had reason to qualify. |
Adjudication
Items are scored by paired raters who cannot see each other’s scores. Disagreements go to a lead rater who calibrates sequentially rather than by majority vote, following the structure NIST ARIA uses. A disagreement rate above the threshold for a criterion is treated as a defect in the rubric wording and triggers an amendment — not as a defect in the raters.
Changelog
| Version | Date | Change | Why |
|---|---|---|---|
| R-4 | 13 Sep 2026 | Split the former “correctness” criterion into relational correctness and semantic fidelity. | Raters were disagreeing on items where the SQL was valid but implemented the wrong definition — two different failures collapsed into one score. |
| R-3 | 29 Aug 2026 | Added disclosure as a scored criterion. | A confidently-stated wrong answer and a hedged wrong answer are not equally dangerous, and Sev-1 turned on exactly that distinction. |
| R-2 | 11 Aug 2026 | Rewrote severity anchors in behavioural terms. | The previous anchors used impact language ("critical", "major") which raters interpreted against their own risk appetite. |
| R-1 | 24 Jul 2026 | Initial published rubric. | — |
Why publishing this costs us nothing
A competitor can copy this page. The document is not the moat — the calibrated rater pool, the benchmark and the adjudication record are. What publishing buys is the ability to be argued with: if a criterion is wrong, the person best placed to tell us is a customer who can read it.
Why the thresholds are where they are → Agreement statistics Evaluator handbook