How a finding becomes a number, and what the number means

1. The four outcomes a test can have

Every invariant run terminates in exactly one of four mutually exclusive states. There is no “mostly passed”.

2. The assurance case: twelve elements

An evaluation is not a score; it is a case. Every signed bundle contains these twelve elements, in this order, so that two bundles from two customers are comparable and an auditor knows where to look.

1. Agent identity and target architecture — which agent, which model version, which semantic layer.
2. In-perimeter runtime boundary definition — where execution happened and what left it.
3. Consequence tier and threat model classification.
4. Pre-frozen schema surface denominator — signed by the customer owner and our lead methodologist before execution, so coverage cannot be inflated afterwards.
5. Invariant assertion specifications.
6. Test-suite lineage and challenge-nonce commitments.
7. Observed defect trace and AST execution plan.
8. Counterfactual data-state verification.
9. Human adjudication and rater competency signatures.
10. Evidence Assurance Level (EAL-1 to EAL-4; see evidence).
11. Residual risk acceptance formulation — written for the customer executive to sign, not for us.
12. Dual cryptographic attestation signatures — customer tenant key and EvalQA verification key.

3. Where invariants come from

Earlier versions of this method scaled linearly with human headcount: fifty-plus hours of bespoke case authoring per client. Version 9.3 replaces that with a software pipeline and reserves human time for the logic only a human can know.

1. Metadata and manifest ingestion

   Customer dbt manifest.json and warehouse INFORMATION_SCHEMA: model grain, declared tests, relationships, effective-dated columns, row access policies, clustering keys. No table contents.

2. Automated invariant generation

   Standard relational invariants are synthesised: join fan-out traps, null foreign keys, temporal range boundaries, tenant predicates, sargability on clustering keys. Target: 70% of a suite in under 15 minutes PLANNING ASSUMPTION, tested at the Day-30 gate (≥25 valid rules from a real manifest).

3. Targeted human authoring

   Type C fixtures for proprietary business logic — custom UDFs, multi-table rollups, stored procedures. Authoring time reported in every readout; the trajectory that matters is 50 hours in Sprint 1 to under 5 by Sprint 10 TARGET.

4. Adversarial challenge queries and counterfactual probes

   Independent of customer golden truth: parameterised generators with rotating holdout seeds, so that a static public query set never becomes an LLM training target.

4. Risk-adaptive sequential testing

Agents are stochastic. A fixed sample of five trials is mathematically insufficient for a high-consequence invariant: five passes still leave a one-sided 95% upper bound on the failure probability of about 45%. So the trial count depends on what the invariant protects.

Sequential probability ratio testing lets a high-risk invariant stop early on a confirmed failure, but never early on a pass.

5. Consequence tiers and suite allocation

Suite composition is governed by the agent’s consequence tier and threat model, not by a fixed split.

Under no circumstances does an engagement proceed with fewer than 30% independent EvalQA challenge cases. Customer-supplied golden truth is validated at intake — that is not free, and it is priced as a capped addition rather than hidden.

6. Dynamic severity

Severity is never dictated by defect family alone. The same formula error is Sev-3 in an internal dashboard and Sev-1 in a billing agent.

7. Tiered human review

Five human raters per item is an unprofitable services model and, worse, an unnecessary one. Review is routed by what the item needs.

Who qualifies as a reviewer, and how, is on the Evaluator Guild page. In short: ≥98% Sev-1 recall on a 50-item challenge battery (at most one miss), ≤5% false positives on matched clean controls, Brier score ≤0.15 with positive skill, and 5% hidden gold canaries in production with a Bayesian rolling error rate and an emergency suspension after two consecutive misses. That is an observed qualifying score, not a claim about true underlying recall.

8. Who evaluates EvalQA: the calibration benchmark

A common failure in AI evaluation is to conflate the benchmark used to validate the instrument with the suites run on customer systems. We keep them apart. The Ground-Truth Calibration Benchmark (GTCB) is a synthetic, mathematically verified reference suite used only to measure our own detection sensitivity, specificity and inter-panel reproducibility. Customers never see or modify it.

It contains exactly 500 balanced items: 250 seeded defect positives and 250 clean controls across eight failure families, split into three governance partitions to stop engineering from overfitting the detector to its own test.

Per family the split is 30 / 50 / 20 percent (e.g. join fan-outs: 11/11, 17/17, 7/7). Reuse contamination is fought with parameterised generators and rotating holdout seeds. Methodological honesty note: the benchmark covers eight families; the commercial taxonomy has eighteen domains. We market only the eight as measured capabilities. The other ten are PILOT or UNVALIDATED, and a family marked validated was earlier — wrongly — marked so before it had been tested; audit finding #57 relabelled them.

9. The Sev-1 safety gate, with the arithmetic

The question a buyer should ask is not “what is your sensitivity?” but “what is the worst your miss rate could plausibly be?”. We answer with a one-sided upper confidence bound, not a point estimate.

Under independent Bernoulli trials, if zero Sev-1 misses are observed across n_eff qualifying cases, the one-sided 95% upper bound on the true miss rate is:

To guarantee that bound is at most 2.0%:

The gate is stratified: zero misses across all qualified runs, drawn from the sequestered and external partitions, with at least ten qualifying Sev-1 runs per validated family.

Clustering, and why 149 becomes 239

Test items in a real schema cluster around tables and dbt models. If items in a cluster share intra-cluster correlation ρ, 149 raw items are not 149 independent observations. The raw count is inflated by the variance inflation factor:

Status: TARGET. The gate is defined and the corpus is reconciled; a release passes it or does not, and the result is published on this page when a release ships.

10. Every statistic we report, and how

Uncertainty intervals

Pass rates, defect prevalence, sensitivity and specificity carry 95% Wilson score intervals, computed with scipy.stats.binomtest and statsmodels.stats.proportion. We do not hand-derive interval algebra; an earlier draft did, and got it wrong (audit finding #15).

Inter-rater reliability

Unit of analysis: the categorical rating (Pass, Defect-Family-ID, Indeterminate) assigned to one query trace. We publish observed percentage agreement P_o alongside chance-adjusted coefficients — never one without the other.

• Fleiss’ κ for m ≥ 2 fixed raters per item. Target κ ≥ 0.75.
• Krippendorff’s α on nominal and ordinal scales, tolerant of missing observations. Target α ≥ 0.75.
• Gwet’s AC₁ reported next to κ so that the Feinstein-Cicchetti paradox — high agreement, depressed κ under skewed prevalence — is visible rather than hidden. Target AC₁ ≥ 0.80.

Rater calibration

Reviewers give a probability f_i ∈ [0,1] that a flagged anomaly is a true Sev-1/2 defect. Against adjudicated truth o_i ∈ {0,1}:

Competency is a vector — gold accuracy, n_gold, recency weight w(t) = exp(−λΔt) with a 90-day half-life, dialect competency, domain score, overturn rate, BS, BSS — not three heuristics multiplied into a scalar.

Latent class models

Dawid-Skene EM is used only on the dedicated calibration graph, where raters evaluate dense overlapping sets of ≥50 challenge queries. Sparse production routing (one or two raters per item) is not identifiable and we do not pretend otherwise.

11. The five-metric assurance stack

ICFY

Incremental Consequential Finding Yield. Share of critical/high defects found beyond the native platform evaluation. Target ≥ 40%. Kill signal < 20% across five customers.

CDDR

Consequential Defect Discovery Rate. Critical findings per 100 evaluated agent traces.

RWAC

Risk-Weighted Assurance Coverage. Share of the pre-frozen schema surface under active invariants.

DCR

Decision Change Rate. Share of findings that changed a go/no-go or prompt decision. Target ≥ 30%.

FOR

Finding Overturn Rate. Share of reported defects overturned on adjudication. Target < 5%.

ICFY replaced an earlier metric whose denominator — all defects that exist in production — is unknowable (audit finding #56).

12. Limitations and validation status

• Ten of eighteen domains are not benchmark-validated. Catalog hallucination, cross-turn memory contamination, decision latency and governance refusal are UNVALIDATED; six more are PILOT.
• Snowflake is the only validated dialect. Databricks is in pilot; BigQuery and Redshift parse but have no benchmark partition. See coverage.
• The automation targets are assumptions. 70% automatic synthesis, 30% human triage share, 12 minutes per reviewed item, 50% sprint-to-platform conversion — each has a decision gate and a date in the 90-day programme.
• No customer results exist yet. Everything above describes the instrument; nothing above describes an outcome we have delivered.
• Sequential testing bounds the failure probability, not the consequence. A 5% bound on a Tier-1 invariant may still be unacceptable; that judgement belongs to the customer’s residual-risk acceptance.

13. Independence and conflicts

We take no success fees, never author production prompts or dbt models for an audit client, report confirmed Sev-1s without redaction, and have suspended public model leaderboards so that a benchmark publisher is never also an auditor of the benchmarked. The full text is the Independence Charter. When our own method is found wrong, the recall protocol applies to us.

See what a bundle looks like → The 18 failure domains Glossary