Status, honestly

LiveThe qabit adapter and its hosted console are in production use today.
PrototypeThe evalqa CLI is the Day-30 deliverable of the 90-day programme; the commands below are its specification, and the Day-60 gate requires it to be running in a design partner’s weekly dbt CI.
PrototypeThe EVALQA_RUN stored procedure is specified for the Snowflake wedge and is exercised in the first sprints; the interface below is the contract we are building to.

On this page

The evalqa Python CLI
The Snowflake native stored procedure
The qabit OpenTelemetry adapter
The telemetry payload schema (v9.3)
Which one to use

1. The `evalqa` Python CLI

Runs the invariant suite from any CI runner against a Snowflake staging branch, a Databricks SQL warehouse or a local sandbox. Exits non-zero when a Sev-1 is confirmed, so a merge can be gated on it. Requires only the scoped runner role; the binary is signed with Sigstore/Cosign and verifies itself before it runs anything.

# .github/workflows/assurance.yml — illustrative
- name: Verify agent SQL against the suite
  run: |
    pip install evalqa
    evalqa verify --cosign             # checks the runner's own signature
    evalqa run --suite finance \
               --branch "${{ github.head_ref }}" \
               --consequence tier-1 \
               --evidence mode-1-local \
               --fail-on sev-1
  env:
    SNOWFLAKE_ROLE: EVALQA_RUNNER
    EVALQA_TOKEN:   ${{ secrets.EVALQA_TOKEN }}   # nonce + notary only; no data

Useful subcommands:

Command	What it does
`evalqa extract`	Reads dbt `manifest.json` and `INFORMATION_SCHEMA`; writes a metadata pack (no table contents) for invariant synthesis.
`evalqa synth`	Synthesises the standard relational invariants from the metadata pack; prints how many were generated and how long it took, so the Day-30 gate is measurable.
`evalqa commit`	Hashes the challenge suite and records the commitment with the notary before execution.
`evalqa run`	Executes with risk-adaptive sequential trials; writes the signed bundle to your stage or bucket.
`evalqa adjudicate`	Opens the console for INDETERMINATE and Sev-1 items; never needed for a clean run.
`evalqa verify <sha256>`	Same check as /verify, from the terminal.

2. The Snowflake native stored procedure

For teams that want nothing outside the Snowflake account: EVALQA_RUN is installed in a schema you own and executes the whole evaluation in your compute. The only outbound calls are the 15-minute nonce and the manifest hash. Details, privileges and an install script are on the Snowflake page.

CALL EVALQA_RUN(
  suite       => 'finance',
  consequence => 'TIER_1',
  evidence    => 'MODE_1_LOCAL',
  nonce       => :nonce
);

3. The `qabit` OpenTelemetry adapter

qabit is our open-source client: an MIT-licensed OpenTelemetry adapter and the packaging for the evalqa CLI. It streams runtime agent spans — the natural-language prompt hash, the generated SQL hash, tool calls and their parameters, latency — so the agent can be observed in service, not only on the test bench. It is deliberately small: the operating plan sized it as “a tiny OTel/CLI adapter”, not a product.

Hashes by default. Raw prompts and SQL never leave the process unless you opt a field in, and the classification matrix says what happens to it if you do.
Works with any Python agent framework that emits OpenTelemetry; a LangChain and a LlamaIndex instrumentation are included.
The hosted console at /labs/qabit/ is where those spans and human ratings land. It is in production use today and is unchanged by anything on this page.

# pip install qabit
from qabit import instrument
instrument(service="finance-agent", tenant="acme",
           export="hashes")   # "hashes" (default) | "sanitized" (Mode 2, opt-in)

4. The telemetry payload schema (v9.3)

Every assurance event, whether produced by the CLI, the stored procedure or the adapter, conforms to evalqa.schema.json. Required fields are listed; everything is hashes, enums, booleans and counts.

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "title": "EvalQATelemetryPayload_v9_3",
  "required": ["assurance_event_id", "tenant_id", "timestamp", "agent_id",
               "evidence_mode", "evidence_assurance_level",
               "challenge_commitment_sha256", "nonce",
               "invariant_results", "dual_signature"],
  "properties": {
    "evidence_mode":            { "enum": ["MODE_1_LOCAL", "MODE_2_SANITIZED", "MODE_3_HOSTED_VDI"] },
    "evidence_assurance_level": { "enum": ["EAL-1", "EAL-2", "EAL-3", "EAL-4"] },
    "runner_provenance":        { "required": ["build_version", "cosign_verified"] },
    "invariant_results": { "items": { "required": ["invariant_id", "status"],
      "properties": { "status": { "enum": ["PASSED", "VIOLATED", "INDETERMINATE"] },
                      "repeated_trial_count": { "type": "integer" },
                      "failure_frequency":    { "type": "number" } } } },
    "dual_signature": { "required": ["customer_tenant_sig", "evalqa_witness_sig"] }
  }
}

5. Which one to use

You want to…	Use	Evidence mode
Block a pull request that introduces a Sev-1	`evalqa` CLI in CI	Mode 1
Keep everything inside the Snowflake account, no CI runner involved	`EVALQA_RUN` stored procedure	Mode 1
See what the agent does in production, week to week	`qabit` adapter → console	Mode 1 (hashes) or Mode 2 (opt-in)
Have an external expert review a regulated schema	Console inside your VDI	Mode 3

Book a readiness review → Evidence modes & EALs

1. The evalqa Python CLI

2. The Snowflake native stored procedure

3. The qabit OpenTelemetry adapter

4. The telemetry payload schema (v9.3)

5. Which one to use

1. The `evalqa` Python CLI

3. The `qabit` OpenTelemetry adapter