The estimators below are implemented and validated against reference implementations, and they run today inside every engagement. The public, dated release of the population figures is scheduled with the first design-partner cohort. planning Until a number appears in the table with an n and an interval beside it, treat this page as a description of method, not a result. We would rather ship an empty table than a figure we cannot defend.
What gets measured
For a stratified sample of adjudicated items in every engagement, the same item is scored independently by the EvalQA semantic judge and by at least two trained Guild raters who cannot see the judge’s output. Agreement is then computed three ways, because one coefficient is never enough.
| Statistic | What it answers | Reporting threshold |
|---|---|---|
| Cohen’s κ | Agreement between two raters on a categorical call, corrected for the agreement you would get by chance alone. | Reported with the confusion matrix; no single floor, because κ is distorted by prevalence. |
| ICC(2,1) | Absolute agreement on an ordinal severity score, treating raters as a random sample rather than the only raters who matter. | ≥ 0.70 target |
| Krippendorff’s α | Agreement across any number of raters, tolerant of missing data and of more than two categories. | ≥ 0.67 exploratory · ≥ 0.70 operational · ≥ 0.80 consequential target |
Current value for all three: pending publication. Consequence-level thresholds follow Caban (2026), a review of 55 papers on reliability reporting in applied evaluation. Intervals are bias-corrected and accelerated bootstrap, 10,000 resamples, reported at 95%.
Why three coefficients and not one
Each one fails differently, and the failures are informative. κ collapses when one category dominates — a suite where 95% of items pass can show poor κ while the judge is behaving correctly. ICC assumes an interval-like scale and rewards a judge that tracks severity ordering even when it disagrees on the absolute call. α survives missing data and more than two raters but hides which pair disagreed. Publishing all three, with the confusion matrix, lets you see which of those is happening rather than taking a single reassuring number on trust.
What an honest agreement figure has to carry
- An n — how many items, and how they were sampled. A coefficient computed on 30 convenient items is not a population estimate.
- An interval — a point estimate with no uncertainty attached is not a measurement.
- A date and a configuration — judge model, prompt version and rubric version, because all three move.
- The disagreements — the confusion matrix, not just the summary. Where the judge and the humans part company is the finding.
- The rater baseline — human–human agreement on the same items. A judge that matches humans as well as humans match each other has hit the ceiling of the task; a judge compared only against itself has not been tested.
Why this page exists at all
Mahdi et al. (NeurIPS 2025) reviewed 445 published LLM benchmarks and found that only 16% ran any statistical test at all. fact A category that measures things for a living, mostly not measuring whether its own measurements agree with anything, is the gap this company exists to fill. Publishing our own agreement statistics is the smallest honest version of that argument — and it is the number we would want from a vendor selling us the same thing.
Where the estimators live
The coefficients are computed by the same code path that produces the numbers in your assurance report, and their outputs are included in the signed bundle, so a customer can recompute them from the case-level records rather than trusting the summary. The export schema is documented on portability.
The method behind the numbers → Known limitations The rubric