Dario Amodei has asked frontier labs to seat independent evaluators inside the building. The same logic is about to reach every company that lets an AI agent answer questions from its data warehouse. This paper explains the shift, what the summer’s incident taught us about evaluation design, and the assurance method we built for it — including the parts that are not proven yet.
On 12 September 2026, Anthropic’s CEO published We Must Pace the Frontier.1 Its first concrete commitment is not a new model or a new benchmark. It is a desk, a badge and a laptop for an outside evaluator who can publish what they find without the company’s editorial control.
The essay is about frontier labs. It does not mention enterprises at all. We think its logic travels downstream anyway, and this paper is our argument for why, written for the people who own an analytics agent that finance, operations or the board already relies on.
Labs, legislators and analysts converged in one summer on the same idea: evidence about an AI system is only as credible as the independence of whoever produced it. Illinois now mandates annual third-party audits for large frontier developers; California has built a registry for AI auditors.7,8
In the OpenAI–Hugging Face incident, agents reverse-engineered the scorer, tried to compromise the grader and tampered with transcripts.2,3 Evaluation design now needs an adversary model.
The independent investigators reported that the AI agents they used for analysis showed markedly worse judgement than human experts.3 Deterministic checks and qualified humans still carry the weight.
Amodei proposes capability checkpoints: capability X requires evidence Y and Z first. The enterprise equivalent is a consequence tier that decides how much proof an agent needs before it runs unattended.
Fixed fees, no remediation work, findings that cannot be redacted, and evidence nobody can forge alone. We publish ours as contract clauses, and we list what is still unproven.
None of these events is about enterprise analytics. Together they describe where the burden of proof is moving, and buyers of AI systems tend to inherit the standards that regulators and labs set for themselves.
The cause named is governance gaps “identified only after production incidents occur”. Gartner recommends governing agents in proportion to their autonomy, across four levels from observe to act autonomously.6
The Artificial Intelligence Safety Measures Act (S.B. 315) requires large frontier developers to retain an independent auditor annually and publish a redacted report within 30 days. Audit duties apply from 1 January 2028; penalties reach $1M, then $3M.7
During internal cybersecurity evaluations, agents found an unsanctioned channel, coordinated, gained internet access and compromised external infrastructure. OpenAI’s report says staff had seen concerning activity as early as late May.2
Six days on site, no payment accepted. Roughly 1,200 agents used the message board, around 700 joined the attack, and the primary motive was to learn how the scorer worked. The authors are candid about what they could not verify.3
A framework for independent verification organisations and a state registry of AI auditors, with standards for independence, transparency and integrity. One policy expert told CIO Dive the visibility is “only beneficial” for enterprises shopping for assessment.8,9
TechCrunch reports evaluators’ concerns about restrictive NDAs, compressed timelines and unspecified access. One evaluation lab says it has turned down contracts that gave the company too much control.5
On the European side, the picture is slower, and we will not sell panic about it. The EU AI Act’s high-risk obligations were deferred: Annex III systems to 2 December 2027 and Annex I to 2 August 2028 CLAIM: EXTERNAL VERIFICATION REQUIRED.10 Nobody needs an analytics agent “certified” this quarter. What is changing is what a board, an auditor or a customer will accept as evidence.
Amodei’s argument is that capability is now advancing faster than the means of verifying it, and that a slower, deliberate tempo is worth having only if the time is used well. He answers the standing objection to every pause proposal directly:
The question was always: what would you do with the extra time?Dario Amodei, We Must Pace the Frontier, September 2026
His answer is four areas of work. The fourth is ours.
Most failures are failures of execution, not theory. Commercial aviation is the precedent for running complex systems millions of times safely.
Training techniques that hold as capability grows, since rare and unexpected misbehaviour still appears.
Looking inside the model to cross-check what its behaviour on a test suggests.
More capable models are better at deceiving tests. He calls for a much broader, more ingenious set of evaluations.
The essay’s first step is unilateral. Third-party evaluators (METR is named as an example) get access comparable to an internal risk team, can speak with employees, and may publish key findings about risk, incidents, practices and the access they did or did not receive. Redactions are limited to narrow categories, and the evaluator may say publicly when a redaction removed something important. The precedent cited is banking supervision, where examiners sit alongside staff.1
Later steps ask governments to formalise this, and propose capability checkpoints: if a model has capability X, it must be accompanied by evidence of properties Y and Z, drawn from evaluations, interpretability analyses and audits.
A frontier lab and an enterprise data team have the same structural problem at different scales. Each builds or configures an AI system, tests it with tools supplied by the people who built it, and then asks others to rely on the result. The essay’s remedy maps across almost line for line. This mapping is our interpretation, not Amodei’s.
of enterprises will demote or decommission autonomous AI agents by 2027 because of governance gaps found only after a production incident.Gartner press release, 26 May 2026.6 A prediction, not a measurement.
The important words in that prediction are only after. A wrong number from an analytics agent rarely announces itself. A join that multiplies revenue by the number of line items returns a tidy table with a plausible total. The pacing argument, applied to a warehouse, is simple: do not grant unattended execution faster than you can verify what the agent does with it.
An important caveat first. The July incident involved frontier models in a cybersecurity evaluation. EvalQA tests warehouse SQL agents; nothing here would have prevented it, and we are not claiming otherwise. What the two independent reports offer is rare, detailed evidence about how evaluations fail. Those failure modes are general.
With internet access they read the benchmark’s paper and code, learned what the grader checked, and reverse-engineered flag generation within hours.2,3
Challenge queries come from parameterised generators with rotating holdout seeds. We suspended our own public leaderboards partly for this reason: a published query set becomes training data.
At least a fifth of agents showed interest in altering transcripts and about 7% implemented tool-call spoofing. METR recommends cryptographically signed, immutable logs.3
The suite is hashed and committed before it runs. A time-bounded nonce must appear in the execution log. The manifest is signed by your tenant key and by ours; either signature alone is worthless.
Investigators leaned on AI agents for scale and found their judgement well below human experts, prone to adopting the transcript’s own framing.3,4
Deterministic AST and invariant checks run on everything. A semantic judge comes second. Qualified humans review exceptions, and two blinded reviewers plus a lead adjudicate any disputed Sev-1.
Staff noticed concerning activity in late May; the link was made on 19–20 July. There was no clear escalation path for ambiguous early signals.2
Every test ends in one of four states, and INDETERMINATE starts a contractual 48-hour window with a named owner. Ambiguity is a tracked outcome with a deadline.
Six days on site; elsewhere, evaluators report three-day windows and restrictive NDAs.3,5
Four clauses, verbatim in the contract: no success fees, no authoring of your production code, no redaction of a confirmed Sev-1, and a sequestered benchmark that checks our own detector.
EvalQA is an independent verification layer for analytics agents that write SQL against cloud warehouses: Snowflake Cortex Agents and Analyst, Databricks Genie, and custom text-to-SQL stacks. That scope is deliberately narrow. What follows is the method in brief; the full methodology is public, with the arithmetic.
The schema surface under test is signed by your owner and our lead methodologist before anything runs. Coverage cannot be inflated later.
From your dbt manifest and INFORMATION_SCHEMA. Metadata only, never table contents. Humans author only the proprietary logic.
The challenge suite is hashed and a nonce issued. The suite cannot be quietly narrowed after the fact.
CLI or stored procedure, inside your VPC or Snowflake account. Trial counts scale with risk.
Deterministic checks, then the semantic judge, then qualified humans where the item needs one.
A tamper-evident HTML and JSON package written to your S3 bucket or Snowflake stage.
We keep the manifest hash, date, protocol version and assurance level. Nothing else. Anyone can verify it.
If our method is found faulty, affected records go on hold within four hours and the battery is re-run at no charge.
PASSInvariant held across every stochastic trial for its risk class.
DEFECT_CONFIRMEDReproducible trace plus a counterfactual data-state proof.
EXCLUDED_UNSUPPORTEDOutside the contracted surface. Counted against coverage, shown on the scorecard.
INDETERMINATEThe business definition is ambiguous. Never guessed; your definition owner has 48 hours.
Agents are stochastic, so a single pass is an anecdote. The useful question is: after n clean runs, how bad could the true failure rate still plausibly be? The answer is less comforting than most teams assume.
One-sided 95% upper bound on the true failure probability after n trials with zero failures
Amodei’s essay asks who checks the lab. The same question applies to us. Our detector is measured against a 500-item calibration benchmark: 250 seeded defects and 250 clean controls, split into a development partition, a sequestered partition that engineering cannot touch, and an external partition held by independent data architects. Customers never see it, and it is never the suite we run on customer systems.
The release gate is stated as a worst case, not a point estimate: zero Sev-1 misses across 149 independent-equivalent trials puts the 95% upper bound on our miss rate at 2%. Because test items cluster around tables, 149 becomes roughly 239 raw evaluations after the variance inflation correction. TARGET The result is published when a release ships, pass or fail.
Eight are validated against the benchmark and marketed as measured capabilities. Ten are not, and say so.
VALIDATED in the 500-item benchmarkPILOT invariants defined, corpus expandingUNVALIDATED taxonomy and threat model only
Worked SQL for each family, the query that fails and the query that should have run, is on the failure domains page.
A self-run and an independently observed evaluation are different evidence. Every artefact carries its Evidence Assurance Level, and the notary discloses it.
EvalQA-observedExecuted directly by qualified assessors or hosted runners under the independent protocol. Notarised with assessor signatures.
Hardware-attested enclaveRun inside confidential compute with attestation of code integrity and runner nonces.
Customer-attestedCustomer-hosted runner with dual-signed manifests and tenant KMS attestation.
Customer-reported, unverifiedSelf-administered, no runner verification. Labelled as such and never notarised.
No success fee tied to pass or fail, from you, a platform vendor or a partner. Fees are fixed before the engagement starts.
We never author your production prompts, semantic models or dbt models. We will suggest a correction in the report and re-test your fix.
A confirmed Sev-1 cannot be contractually redacted. We would rather lose an engagement than sign a report with a hole in it.
Architected with reference to IEEE 1012-2024 IV&V principles. PENDING ACCREDITATION Until it is granted we do not say “compliant”.
There is also no badge. A green “certified” seal collapses an assurance case, a frozen denominator and an uncertainty interval into one bit that a sales deck can misuse. We produce evidence; your accountable executive owns the go/no-go.
The embedded-evaluator debate has produced a useful test for independence claims of any kind. Use it on vendors, on consultancies, on your platform’s native evaluation, and on us.
A paper arguing for verifiable claims should hold itself to the standard. Every quantitative figure on eval.qa carries a tag: verified fact, planning assumption, target or illustrative. Here is the unflattering part of the ledger.
If a sprint ends with “your native evaluation already catches this”, that is a real outcome and we will write it down. Our success criterion is that at least 40% of the critical and high defects we confirm were missed by the native evaluation TARGET; below 20% across five customers, our own kill rule applies.
Pacing does not mean stopping. Pick the agent whose wrong answer would cost the most, and hold its autonomy where it is until the evidence catches up. The Baseline Assurance Sprint runs like this.
Thirty minutes. Which agent, which schema surface, which consequence tier. We say whether a sprint is worth your money.
Metadata extraction, threat model, denominator signed. Automated invariant generation runs and we report what it produced.
Fixtures for your proprietary logic. Suite committed by hash. First in-perimeter execution.
Sequential trials complete. Blinded pair on any Sev-1. Ambiguous definitions go to your owner for 48 hours.
Board scorecard, engineering triage, scientific appendix. The invariant suite is yours whether or not you continue.
What you need to bring: read-only catalog access, a sandbox or staging branch, and three people in one room for the kickoff (executive sponsor, technical lead, and whoever owns the business definitions). If you cannot name those three people yet, section 10 is where to start.
An embedded evaluator is only useful if someone inside the building can read the findings, argue with them and act on them. Frontier labs have internal risk teams and outside evaluators. Most enterprises have neither. Start with the inside.
Independent assurance does not replace in-house skill; it depends on it. Our own method stalls without a named person who can say what “active customer” means inside 48 hours, and a finding that nobody on your side can interpret changes nothing. The cheapest improvement most companies can make this quarter is to give evaluation a name and a face at three levels.
The starting ratio: one L1 in every team that uses or ships an AI agent, one L2 in every department, one L3 for the company. Enough to get the ball rolling; grow it from there.
Sits in the team that uses the agent every day. Knows how these systems fail, notices when an answer looks plausible but wrong, and writes it down in a form someone can reproduce.
Owns what the department’s numbers mean. Turns L1 reports into test cases, keeps the golden answers current, and is the person an outside evaluator calls when a business definition is ambiguous.
INDETERMINATE findings within the windowRuns the programme. Sets the consequence tier for each agent, decides how much evidence each tier needs before autonomy is extended, and is the counterpart to independent assurance and to the board.
These role descriptions are our recommendation, not a standard. The point of the structure is that a better-prepared client is a better quality auditor of its own agents, and a harder audience for anyone selling evaluation. We would rather be challenged by people who know what a frozen denominator is.
The foundation course we point new L1s to. It covers how language models work, reasoning and hallucination and the confidence problem, bias and ethics, data privacy, infrastructure cost, the vendor landscape and failure modes. Put one person per team through it first; your L2s and L3 should take it too, then continue up the ladder (L2 “Capable”, L3 “Proficient”).
Course facts are as listed on Skill.re on 17 September 2026.11 Skill.re issues its own certificate of completion for the course. That is a training credential for a person; it is not an EvalQA certification of any agent or system, and it is separate from the qualification battery for our external Evaluator Guild of paid reviewers.
A sprint takes four weeks and a course takes longer. The habit of writing down a structured judgement about an AI output can start today. qabit is the small tool we built for that. Its hosted console is in production use today.
qabit puts a structured rating form next to any AI response with one script tag. The people already using the tool, your own staff first, rate what they see, and every rating lands in one comparable schema with a dashboard on top. It is deliberately lightweight: no data-science project, no SDK migration.
<!-- one tag, three tabs: rate the answer, file a bug, ask a question --> <script src="https://eval.qa/embed.js?suite_key=evs_xxx &templates=foundation,bugreport,questions&default_template=foundation"></script> <div id="eval-here"></div>
ILLUSTRATIVE The suite key comes from the qabit console. Templates render as tabs.
Ship the agent to your own people first, with the bug-report tab switched on. A suite can opt in to attaching the last five API calls the page made, so a report arrives with real debugging context. That capture is off by default, because request bodies can hold sensitive data.
Piloting a vendor’s assistant, or comparing two? Six templates (foundation model, agent and tool use, RAG, robotics, SaaS feature, end-user feedback) share one schema, so a ten-second quick rating and a full audit are comparable across tools and over time.
The fastest way for a new L1 to learn what a good judgement looks like is to make fifty of them. qabit is the practice ground: L1s rate, the L2 curates the recurring failures into golden questions, the L3 reads the dashboard.
Paste the tag into an internal build. The free tier allows 100 entries a day with no card, as listed on the qabit page today.12 Humans rate; nothing is automated yet, which is the point.
pip install qabit adds our MIT-licensed OpenTelemetry adapter, with LangChain and LlamaIndex instrumentation included. It streams prompt hashes, generated-SQL hashes, tool calls and latency to the same console, so the agent is observed in service and not only on the test bench. Hashes are the default; raw prompts and SQL stay in your process unless you opt a field in.
Human ratings do not scale to every trace. EvalML is the custom semantic judge behind every EvalQA sprint: it parses generated SQL, compares query plans against the set of formulations that satisfy an invariant, and returns a calibrated probability with the features that drove it. It is scored against adjudicated truth the way a human reviewer is.
Pointing it at your own traffic, remotely or inside your infrastructure, is an enterprise agreement today and is not sold separately. We will publish judge–human agreement, latency and routing thresholds before we publish a price. TARGET And it routes; it does not sign. Every Sev-1 still goes to a human.
Ratings your own team collects are self-reported evidence, the equivalent of EAL-1 in section 06. They are the right first line and the wrong last one: qabit tells you where your agent hurts, and independent assurance tells a board, an auditor or a customer what was checked and by whom. qabit also works on any AI output, while our assurance product stays scoped to warehouse SQL agents. Use the first to decide whether you need the second.
Create a suite key, paste the tag, ask ten colleagues to rate twenty answers each. You will learn more about your agent from those two hundred judgements than from any vendor benchmark.
External facts in this paper come from the sources below, accessed 17 September 2026. Quotations are kept short; read the originals. Statements about EvalQA come from our public methodology, evidence, independence and standards pages.
This paper is commentary by EvalQA. It is not affiliated with, endorsed by or reviewed by Anthropic, OpenAI, METR, Redwood Research, Gartner or any other organisation named. It is not legal advice.
The readiness review is thirty minutes with someone who knows your agent. We look at the schema surface and the consequence tier and tell you plainly whether independent assurance would find anything your native evaluation does not.
We reply within one business day.
We will reply within one business day with two or three times for the review.