1. The three evidence modes
Customer data protection and external human review pull in opposite directions. Rather than pretend otherwise, every engagement runs in one of three explicit modes, named in the order form.
| Mode | Data boundary and processing scope | What is transmitted or stored externally |
|---|---|---|
| Mode 1 — Local-only automated execution Default | 100% automated execution inside the customer VPC or Snowflake sandbox. No human review. | Cryptographic hashes, invariant status, row counts, latency and signed manifests only. Zero raw queries or data leave the perimeter. |
| Mode 2 — Sanitised evidence escalation On explicit authorisation | An invariant failure needs diagnosis; the client authorises a diagnostic export for EvalQA triage. | Ephemeral export of a pseudonymised AST snippet and execution plan. Strict 14-day TTL, encrypted at rest under a per-tenant KMS key. |
| Mode 3 — Customer-hosted restricted VDI review Regulated schemas | High-consequence or regulated client; external expert review required on sensitive schemas. | Nothing. The external rater works inside a customer-controlled VDI with clipboard, downloads and session capture disabled. Remote visual inspection is treated as regulated processing and documented as such. |
A confidential-computing enclave does not make a human’s screen confidential; that is why Mode 3 exists alongside hardware attestation rather than instead of it (audit finding #29).
2. The four Evidence Assurance Levels
An unverified self-run and an independently observed evaluation are not the same evidence, and an auditor should never have to guess which one they are looking at. Every artefact carries its EAL, and the notary discloses it.
| Level | Name | How the evidence was produced | Label on the artefact |
|---|---|---|---|
| EAL-1 | Customer-reported | Self-administered runs executed by the customer without runner verification. | [CUSTOMER REPORTED - UNVERIFIED] — never notarised |
| EAL-2 | Customer-attested | Customer-hosted runner with dual-signed cryptographic manifests and tenant KMS attestation. | Notarised with EAL-2 |
| EAL-3 | Hardware-attested enclave | Executed inside a hardware-isolated confidential compute enclave (AWS Nitro, GCP Confidential VM) with attestation of code integrity and runner nonces. | Notarised with EAL-3 and attestation document hash |
| EAL-4 | EvalQA-observed direct | Directly observed and executed by EvalQA qualified assessors or hosted runners under the independent protocol. | Notarised with EAL-4 and assessor signatures |
3. The runner adversary model
If a dishonest customer controls the execution environment, the inputs and the keys, they could in principle tamper with execution to manufacture passing evidence for a board or an auditor. The defence is layered so that no single party can produce a valid bundle alone.
Cryptographically signed runner builds
The runner binary and containers are signed with Sigstore/Cosign using keyless OIDC and a public transparency log. The runner verifies its own signature before executing; the result records
cosign_verified.Ephemeral challenge nonces
Before execution, eval.qa issues a time-bounded nonce that must appear in query comments and in the signed execution log. A replayed or offline run cannot carry a valid nonce.
Challenge-suite pre-commitments
The suite manifest is hashed and committed before execution. Results are valid only if the execution log matches the committed hash — a suite cannot be quietly narrowed after the fact.
Dual-signed manifests
The final manifest is signed by the customer’s tenant KMS key (attesting execution) and by EvalQA’s verification key (attesting invariant evaluation). Either signature alone is worthless.
Evidence provenance disclosure
The notary states the EAL. A reader is never left to assume an EAL-1 self-run was independently observed.
Signing keys are held in an HSM with automatic rotation and a cryptographic revocation log (risk RSK-17). If a key is ever compromised, every record signed under it is placed on hold and re-signed after re-evaluation.
4. The in-perimeter signed bundle
An earlier design published a per-engagement page at verify/{uuid}. It was withdrawn (audit finding #40): an external web portal holding a customer’s vulnerability list is exactly what InfoSec will refuse. The primary deliverable is now a digitally signed, tamper-evident HTML and JSON package written directly to the customer’s S3 bucket or Snowflake internal stage.
- What it contains: the twelve-element assurance case, the three-tier readout, every finding with its trace, and the dual signatures.
- What EvalQA keeps: the manifest hash, evaluation date, protocol version, EAL and hold status. Nothing else.
- Immutability without a GDPR collision: raw queries, prompts and error strings are never placed in the immutable object-lock store; it holds only hashes, booleans, nonces, EAL and signatures. Mode 2 diagnostics are encrypted per tenant, and destroying the tenant KMS key at contract end renders them cryptographically unrecoverable. See key custody.
5. The stateless notary
Given a manifest SHA-256, eval.qa/verify returns one sentence:
Verification Record Valid: Manifest [hash] evaluated on [date] under Protocol v9.3 at Evidence Assurance Level EAL-2. Signature Verified. Zero customer data stored.
Or, if the record is on hold under the recall protocol: REVIEW REQUIRED. Or, if we have no record: nothing, because EAL-1 runs are never notarised and we hold no data with which to guess. The endpoint is public, rate-limited and documented at /api/notary.php?h=.
6. Why there is no badge
EvalQA never issues an unqualified green “Certified” seal. A badge collapses an assurance case with a stated EAL, a frozen denominator and an uncertainty interval into a single bit that a sales deck can misuse. What we issue is evidence supporting the customer’s own go/no-go gate and residual-risk acceptance. The customer’s accountable executive owns the decision; we own the method. That division is in the Independence Charter and in every contract.