What EvalQA does
The independent technical verification, validation and continuous assurance layer for enterprise analytics agents that write SQL against cloud data warehouses — Snowflake Cortex Agents, Databricks Genie and custom Text-to-SQL pipelines.
Any LLM-driven system that turns a natural-language question into SQL (and tool calls) executed against a production warehouse: Snowflake Cortex Analyst and Cortex Agents, Databricks Genie, or a LangChain / LlamaIndex pipeline your team built.
Software with a mandatory onboarding sprint. The Continuous Assurance Platform runs in your CI and your warehouse; the four-week Baseline Assurance Sprint freezes the coverage denominator and produces the first signed bundle. Human reviewers adjudicate exceptions; they are not the product.
No. We removed that claim on 12 September 2026. Scope is warehouse SQL agents. The Learn library still covers evaluation broadly as education; it is not a product claim.
Evidence and data
In Mode 1 (default), no. Only hashes, invariant pass/fail status, row counts, latency and the signed manifest leave. Mode 2 (sanitised diagnostics, 14-day TTL, per-tenant key) and Mode 3 (review inside your VDI) require your explicit authorisation.
Paste the manifest SHA-256 at eval.qa/verify. The notary returns the evaluation date, protocol version, Evidence Assurance Level and hold status — and nothing else, because it holds nothing else.
Signed runner builds that verify themselves, an ephemeral nonce embedded in every query, a challenge-suite hash committed before execution, dual signatures (customer tenant key + EvalQA key), and an EAL that says how the evidence was produced. Self-runs are EAL-1 and are never notarised.
No, and it never will be. It is technical evidence supporting your own go/no-go gate. Your accountable executive owns the decision. We map the evidence to NIST AI 100-1, NIST AI 600-1, ISO/IEC 42001 and IEEE 1012-2024 so an auditor can use it.
Method
A sequestered 500-item Ground-Truth Calibration Benchmark (250 seeded defects, 250 clean controls, eight families) with a stratified Sev-1 gate: zero misses across at least 149 independent-equivalent trials, giving a one-sided 95% upper bound of 2% on the miss rate. Ten of eighteen failure domains are still PILOT or UNVALIDATED and are labelled so everywhere.
It is recorded as INDETERMINATE, never guessed. Your Business Definition Owner has a contractual 48 hours to resolve the definition; then the invariant is re-run.
Agents are stochastic. Five passes leave a 45% upper bound on the failure probability. Fifty-nine zero-failure trials establish, at 95% confidence, that the true failure probability is under 5% — which is the least a Tier-1 financial invariant deserves.
Yes — EvalML, the Tier-2 automated semantic judge. It is scored with a Brier score like a human reviewer, it routes rather than signs, and it is not sold separately until its judge–human agreement and latency are published.
Commercials
An annual platform subscription plus a mandatory Baseline Assurance Sprint: fixed, no success fees, no hourly billing. The commercial model is still being settled with our first design partners, so we quote the figures in the readiness review rather than publish them here.
Enterprise procurement takes 60–120 days. A card-billed, upfront sprint means verification starts before the AP cycle completes, and kickoff is gated on receipt so nobody works for free.
Never. It is the first clause of the Independence Charter.
The first 3 engagements run under a 90-day falsification programme: two paid sprints, a Day-60 ICFY gate, and a Day-90 conversion decision. If neither of the first two partners converts to the platform, we halt the platform build. It is published so you know the offer is a test.
People
Qualified reviewers from the Evaluator Guild: senior analytics engineers who passed a 50-item battery (at most one Sev-1 miss), are paid a published rate from the first calibration hour, never pay us anything, and are continuously measured against hidden gold canaries.
No — developmental separation. A Qualified Implementation Partner or your own team remediates; we retest.
You do. They are committed to your repository, and the signed bundles are in your perimeter. See the portability and exit guarantee.