What Langfuse is genuinely good at
Open-source tracing and scoring you can self-host, keeping every prompt and completion inside your own infrastructure. For teams with data-residency constraints or an aversion to per-seat SaaS, that is a real structural advantage, and it publishes its unit economics openly — a standard we think more of this category should meet.
What self-hosting does not change
Ownership of the tooling is not independence of the judgement. Whoever operates Langfuse still writes the scorers, picks the thresholds and decides what a pass means. Self-hosting removes the vendor from your data path; it does not put a second pair of eyes on the agent.
For most systems that is fine. It stops being fine at the point where somebody outside your team has to rely on the result — and that is a governance requirement, not a tooling one.
| Langfuse | EvalQA | |
|---|---|---|
| Deployment | Self-hosted or cloud | Runs inside your perimeter; only hashes and status leave |
| Data residency | Fully yours | Fully yours in the default evidence mode (modes) |
| Who authors the scoring | Your team | Us, plus invariants compiled from your dbt manifest |
| Licence cost | Open source | Fixed annual fee |
| Operational burden | Yours | Ours |
| Third-party verifiable output | No | Yes — signed bundle + notary |
Where we agree with Langfuse’s philosophy
On lock-in. OpenAI is shutting its Evals platform on 30 November 2026 and Humanloop was absorbed into Anthropic; current buyer guidance is to run an export-and-recreate drill before signing anything. We publish a documented non-proprietary export schema and a written exit guide for the same reason open source is attractive: the ability to leave is what makes staying a choice.
Pick Langfuse if
- Data residency or cost control is the binding constraint.
- You have the platform engineering capacity to run it well.
- The agent’s failures are recoverable and internally visible.
Add EvalQA if
- An audit committee, regulator or customer has to accept the evidence.
- You need adversarial cases your own team would not have authored.
- The failure domains that matter are relational — fan-out, SCD-2 boundaries, row access policy leakage — which generic LLM scoring does not detect.