The independence argument, stated precisely
IEEE 1012-2024 defines independent verification and validation along three axes: technical, managerial and financial independence. Native warehouse evaluation fails all three by construction, and no amount of engineering quality changes that:
- Technical — the evaluation is built by the team that built the agent, from the same assumptions about what the agent should do.
- Managerial — a finding that reflects badly on the platform reports up to the same executive who owns the platform’s adoption target.
- Financial — the vendor is paid on compute consumption. Its incentive is that you run more queries, not that you run fewer and better ones.
This is not a claim that native evaluation is bad. It is a claim about what a self-assessment can be used for, and it is the same claim your auditors make about your own internal controls.
What native evaluation does better than us
- Integration cost. Zero. It is already in the platform you are paying for.
- Platform awareness. It sees internals we reach only through
INFORMATION_SCHEMAand the query profile. - Continuity. It runs on every query, all the time. We run a contracted suite on a frozen configuration.
- Breadth. It covers the whole platform surface, not the eighteen relational failure domains we specialise in.
What we add
Adversarial cases authored by someone with no stake in the agent passing, invariants compiled from your dbt manifest rather than from the platform’s own model of correctness, and a signed evidence bundle that a third party can verify without trusting either us or the platform vendor. Plus the specific defect families that generic evaluation misses because they produce plausible output: join fan-out, SCD-2 temporal boundary errors, filter placement, row access policy leakage. See the eighteen failure domains.
The number we hold ourselves to
Our contract commits us to reporting the share of critical and high defects we find that the native evaluation did not. If that figure comes in under 20% across five customers, our own kill rule says this product should not exist, and we will say so publicly. planning The measurement is not yet complete — five customers is the threshold, and we are earlier than that.
We publish this because it is the single number that could kill the company, and a vendor unwilling to name one has not thought hard enough about whether it should exist.
The honest recommendation
Run native evaluation continuously. Add independent verification at the points where being wrong has consequences that leave the building — a release gate on a regulated report, a board-level risk statement, a customer-facing number. If native evaluation is catching everything that matters in your environment, our own kill rule says you should not buy us, and we would rather you found that out from this page.
Snowflake Cortex specifics → Independence Charter All comparisons