We are a small, early company. We hold no SOC 2 report and no ISO certification today. Everything below is either implemented or explicitly marked as not yet done. We would rather lose a deal on an honest answer than win one on an implied certification — and we have written that preference into the Independence Charter.
The boundary, in one table
Every engagement runs in a named evidence mode. In the default mode the runner executes inside your VPC or Snowflake account and no human at EvalQA sees a query, a row or a prompt.
| Mode | Who sees customer data | What leaves the perimeter |
|---|---|---|
| Mode 1 local-only Default | Nobody at EvalQA | Hashes, invariant pass/fail, row counts, latency, signed manifest |
| Mode 2 sanitised escalation | Named EvalQA triage staff, for 14 days | Pseudonymised AST snippet and execution plan, encrypted under your tenant key |
| Mode 3 restricted VDI review | A qualified reviewer, inside your VDI | Nothing |
Data-classification matrix for every evidence field
All customer-derived evidence — invariant failure names, error strings, prompt metadata — is CONFIDENTIAL by default. The matrix below is the one the platform enforces.
| Evidence field | Classification | Redaction protocol | Retention |
|---|---|---|---|
| Natural-language prompt | CONFIDENTIAL | Named-entity removal | Encrypted; customer opt-out |
| Generated SQL query | CONFIDENTIAL | Literal values masked | 30 days, or never leaves (Mode 1) |
| Query result-set data | RESTRICTED | Zero transmission | Local customer boundary only |
| Invariant pass/fail | CONFIDENTIAL | Tenant pseudonymised | Immutable object-lock store |
| Execution latency | CONFIDENTIAL | Aggregate summary | Telemetry database, 90 days |
| Error stack traces | CONFIDENTIAL | Path and table pseudonymisation | 14-day diagnostic TTL |
Column redaction is done by stable semantic pseudonymisation with role tagging rather than by masking a column into column_19: the latter destroys the semantic utility that evaluation depends on (audit finding #31).
Key custody, deletion and immutability
There is a real collision between an immutable evidence store and GDPR Article 17 / DPDP deletion rights. We resolve it by separation rather than by hoping nobody asks:
- Raw customer queries, prompts and error strings are never placed in the immutable store. The object-locked store holds only SHA-256 hashes, invariant pass/fail booleans, execution nonces, EAL and dual-signed manifests.
- Mode 2 diagnostics are encrypted under a dedicated per-tenant AWS KMS key. On contract termination or a formal deletion request, the tenant key is destroyed and every historical diagnostic payload becomes cryptographically unrecoverable. Confirmed in writing.
- Our signing keys are held in an HSM with automatic rotation and a cryptographic revocation log; a compromised key places every record it signed on hold under the recall protocol.
Who can see your data, when a human is involved at all
| Control | Status | Detail |
|---|---|---|
| Signed confidentiality agreement | In place | Every reviewer signs before any access, covering all customer content indefinitely. |
| Identity verification | In place | Government ID verification for every reviewer before their first paid task. |
| Per-engagement, time-boxed access | In place | A reviewer sees the items they are assigned, for as long as that adjudication is open. Access is not standing. |
| Access logging | In place | Every item view and rating is attributed and timestamped. You can request the log for your data at any time. |
| Named roster on request | In place | How many reviewers, their qualification tier and their processing location — not merely residency, since residency does not determine where processing happens. |
| Background screening | Regulated work only | Run for Tier-1 financial, health or legal schemas. Not by default. |
| Locked-down VDI for review | Available (Mode 3) | Customer-hosted VDI with clipboard, downloads and screen capture disabled. Changes the delivery model and the price; say so early. |
| Automated PII/PHI redaction | In place | Regex and NER tokenisation (Presidio / GLiNER) before any Mode 2 export. |
Platform controls
In place
- TLS for all traffic; HSTS on the production origin
- Strict Content-Security-Policy on every page: no inline script, no inline styles, named origins only
- Authentication on every endpoint that returns stored evaluation data; the notary returns hashes and dates only
- Allow-listed CORS on data endpoints
- CSRF protection on state-changing requests; rate limiting on public write and read endpoints
- Secrets outside the source tree; private data store outside the document root
- Runner builds signed with Sigstore/Cosign; the runner verifies itself before executing
- Automated regression tests that fail the build if a private endpoint becomes public or an inline style appears
Not yet
- SOC 2 Type 2 — scoped; the plan is to start it at customer #4 with a full-time head of security
- ISO 27001 / ISO 42001
- Third-party penetration test
- EAL-3 hardware-attested enclaves (AWS Nitro) — designed, not yet offered
- SAML single sign-on (SSO through Werify is supported today)
- Data residency choice for Mode 2 diagnostics
- E&O insurance at the $5M limit enterprise MSAs ask for — being placed
If one of the items on the right is a hard requirement, tell us at the first conversation. Some are weeks of work and we will say so; some are not on our roadmap at our size and we will say that too.
International transfers
Standard contractual clauses are not a magic solution. Where Mode 2 diagnostics or Mode 3 review involve a transfer out of the UK or EEA we perform a Transfer Impact Assessment, apply sub-processor controls and document the local-law analysis. Reviewer processing location and access authorisation are modelled explicitly, since a reviewer travelling invalidates a residency-based assumption.
Security review pack
We provide a pre-completed CAIQ v4 (CSA STAR Level 1) and standard answers for SIG Lite and bespoke questionnaires. The fastest path through review is almost always Mode 1: nothing leaves, so most of the questionnaire is answered by architecture. Download the DPA and the portability guarantee alongside it.
Reporting a vulnerability
Email [email protected]; also published at /.well-known/security.txt. We acknowledge within 2 business days and give an assessment within 10. Report in good faith — no scanning that degrades service, no accessing data that is not yours, reasonable time before publishing — and we will not pursue legal action. We credit you publicly if you wish. There is no paid bounty programme yet.
If something goes wrong
We notify affected customers within 48 hours of confirming an incident that affects their data, or within 24 hours where your contract or regulator requires it. The notice says what happened, what data was involved, what we have done and what we are asking you to do. Flaws in our method, as opposed to our security, run under the recall protocol.
Evidence modes & the runner adversary model → Data Processing Addendum