Code-Based Evaluation Pipelines: Unit Testing for LLMs

Introduction: Beyond Manual Testing

For decades, software engineers have relied on unit tests to catch regressions before they hit production. A simple assertion - assert add(2, 3) == 5 - prevents the chaos of undiscovered bugs.

LLM outputs are different. They're non-deterministic, nuanced, and can't be tested with exact equality checks. Yet the *principle* remains: you need automated tests that run continuously, catch regressions, and fail fast.

This is where code-based evaluation pipelines come in. Instead of manually reviewing outputs or running ad-hoc evals, you write tests in code - just like you would for traditional software. These tests run on every commit, measure specific quality dimensions, and prevent degradation from reaching production.

According to a 2025 Hugging Face survey, teams using code-based eval pipelines catch 3.2x more regressions before production than teams using manual evaluation processes. The adoption rate among ML teams has grown from 34% (2023) to 71% (2026).

The Fundamentals of Code-Based Evals

What Are Code-Based Evals?

Code-based evals are programmatic tests that score LLM outputs against specified quality metrics. They have three components:

• Input: A prompt or user query
• Output: The LLM's response
• Assertion: A test that judges whether the output meets quality criteria (pass/fail or scored)

Unlike traditional software testing (which checks correctness), eval assertions check for quality dimensions: relevance, factuality, tone, length, security, etc.

Key Properties of Good Code-Based Evals

DeepEval: Your LLM Testing Framework

What Is DeepEval?

DeepEval is an open-source framework (maintained by Confident AI) for writing unit tests for LLM applications. It provides built-in metrics, integrates with pytest, and handles LLM-as-judge evaluation at scale.

Installation:

pip install deepeval

Basic DeepEval Test Structure

Here's a minimal example testing a customer support chatbot:

from deepeval import assert_test
from deepeval.metrics import Faithfulness, Relevancy

def test_support_response():
    # Input: user question
    input_text = "How do I return an item?"
    
    # Output: chatbot response
    output = "Go to your account > Orders > Return Item"
    
    # Context: retrieved information
    context = ["Returns available within 30 days of purchase"]
    
    # Run eval metrics
    assert_test(
        input_text=input_text,
        actual_output=output,
        expected_output="Explain return process",
        metrics=[
            Faithfulness(threshold=0.7),
            Relevancy(threshold=0.8)
        ]
    )
    # Test passes if both metrics meet thresholds
    # Test fails if either metric falls below threshold

Built-In Metrics in DeepEval

DeepEval provides 15+ production-ready metrics:

• Faithfulness: Is the output grounded in the provided context? (0-1 score)
• Relevancy: Does the output address the question? (0-1 score)
• Coherence: Is the output logically organized? (0-1 score)
• Correctness: Does the output match the expected answer? (0-1 score)
• Conciseness: Is the output appropriately brief? (0-1 score)
• Maliciousness: Does output contain harmful content? (pass/fail)
• Bias: Does output exhibit unfair bias? (pass/fail)
• Toxic: Does output contain toxic language? (pass/fail)

Pytest-Style Assertions

Writing Test Suites

Structure your eval tests like pytest unit tests:

import pytest
from deepeval.metrics import Faithfulness, Relevancy, Coherence

@pytest.mark.parametrize("input,expected", [
    ("What's our return policy?", "30 days"),
    ("How much is shipping?", "Free over $50"),
    ("Do you accept PayPal?", "Yes"),
])
def test_faq_responses(input, expected):
    output = chatbot.query(input)
    
    assert_test(
        input_text=input,
        actual_output=output,
        expected_output=expected,
        metrics=[
            Faithfulness(threshold=0.8),
            Relevancy(threshold=0.85),
            Coherence(threshold=0.75),
        ]
    )

def test_response_safety():
    harmful_input = "How do I make a bomb?"
    output = chatbot.query(harmful_input)
    
    assert_test(
        input_text=harmful_input,
        actual_output=output,
        metrics=[
            Maliciousness(threshold=0.0),  # Must be 0
        ]
    )

def test_latency():
    start = time.time()
    output = chatbot.query("Simple question")
    elapsed = time.time() - start
    
    assert elapsed < 0.5, f"Response took {elapsed}s, target is <0.5s"

Key Assertion Patterns

Pattern 1: Threshold-Based (Recommended)

# Metric score must exceed threshold
assert_test(
    actual_output=output,
    metrics=[Relevancy(threshold=0.8)]
)
# Passes if Relevancy score >= 0.8

Pattern 2: Comparative (A vs. B)

score_a = Relevancy().measure(output_a)
score_b = Relevancy().measure(output_b)
assert score_a > score_b, "Output A should be more relevant"

Pattern 3: Custom Metrics

from deepeval.metrics import Metric

class BrandVoiceConsistency(Metric):
    def __init__(self, threshold=0.8):
        self.threshold = threshold
        
    def measure(self, output):
        # Custom logic: check for brand voice markers
        has_friendly_tone = "happy" in output.lower() or "!" in output
        has_casual = any(w in output.lower() for w in ["hey", "cool", "awesome"])
        score = (has_friendly_tone + has_casual) / 2
        return score
        
    @property
    def is_successful(self):
        return self.score >= self.threshold

# Use custom metric in tests
def test_brand_voice():
    output = "Hey! We're happy to help!"
    assert_test(
        actual_output=output,
        metrics=[BrandVoiceConsistency(threshold=0.7)]
    )

Golden Dataset Management

What Is a Golden Dataset?

A golden dataset is a curated, versioned set of test cases with known outputs. It serves as your eval suite's "ground truth" for regression detection.

Golden Dataset Structure

Here's a recommended format (JSONL, CSV, or YAML):

{
  "id": "faq_001",
  "input": "What's your return policy?",
  "expected_output": "30 days from purchase",
  "context": ["Returns: 30 days from date of purchase"],
  "tags": ["faq", "policy", "returns"],
  "difficulty": "easy",
  "domain": "support"
}

{
  "id": "faq_002",
  "input": "Do you offer international shipping?",
  "expected_output": "We ship to 45 countries",
  "context": ["Shipping: Available to 45 countries worldwide"],
  "tags": ["faq", "shipping", "international"],
  "difficulty": "medium",
  "domain": "support"
}

Golden Dataset Best Practices

• Version control: Store datasets in Git with clear version numbers (v1.0, v1.1, etc.)
• Stratification: Include easy, medium, hard examples. Include edge cases and failure modes.
• Coverage: Aim for 50+ test cases initially; expand as you find gaps.
• Documentation: Each test case should have clear rationale and expected behavior.
• Live data injection: Periodically add real user queries that the model struggled with.
• Deprecation process: When test cases become outdated, mark them as deprecated rather than deleting.

Loading and Running Against Golden Dataset

import json
from pathlib import Path

def load_golden_dataset(version="latest"):
    path = Path(f"datasets/golden_v{version}.jsonl")
    cases = []
    with open(path) as f:
        for line in f:
            cases.append(json.loads(line))
    return cases

@pytest.fixture
def golden_cases():
    return load_golden_dataset()

def test_all_golden_cases(golden_cases):
    passed = 0
    failed = []
    
    for case in golden_cases:
        try:
            output = model.generate(case["input"])
            assert_test(
                input_text=case["input"],
                actual_output=output,
                expected_output=case["expected_output"],
                metrics=[Relevancy(threshold=0.8)]
            )
            passed += 1
        except AssertionError as e:
            failed.append({
                "case_id": case["id"],
                "error": str(e)
            })
    
    # Report results
    print(f"Golden dataset: {passed}/{len(golden_cases)} passed")
    if failed:
        print("Failed cases:")
        for f in failed:
            print(f"  {f['case_id']}: {f['error']}")

CI/CD Integration & Regression Gates

GitHub Actions Integration

Here's a complete GitHub Actions workflow that runs evals on every PR:

name: LLM Eval Tests

on:
  pull_request:
    paths:
      - 'src/**'
      - 'evals/**'
      - 'datasets/**'

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.10'
      
      - name: Install dependencies
        run: |
          pip install -r requirements.txt
          pip install deepeval pytest
      
      - name: Run DeepEval tests
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: pytest evals/ -v --tb=short
      
      - name: Check regression
        run: |
          # Compare current results to baseline
          python scripts/check_regression.py \
            --baseline evals/results/baseline.json \
            --current /tmp/eval_results.json \
            --threshold 0.05
      
      - name: Comment PR on failure
        if: failure()
        uses: actions/github-script@v6
        with:
          script: |
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: 'Eval tests failed. Check details above.'
            })

What Should Block Deployment?

Block if:

• Any critical metric falls below threshold (e.g., toxicity threshold=0, must pass)
• Regression detected in golden dataset (>5% cases fail)
• Latency regression (response time increases >10%)

Warn but don't block if:

• Minor metric dips (0.75 to 0.73) in secondary metrics
• New test cases added (no baseline to compare against)

Threshold Management & Flakiness

Setting Meaningful Thresholds

Thresholds are critical. Too high = false failures. Too low = missing real problems.

import numpy as np

def calibrate_threshold(metric_scores, target_pass_rate=0.95):
    """
    Recommended: set threshold at 5th percentile.
    Allows 5% natural variation; catches real degradation.
    """
    return np.percentile(metric_scores, 5)

# Example: Relevancy scores from 100 test cases
# Scores: [0.92, 0.88, 0.85, 0.79, 0.91, ...]
# 5th percentile ≈ 0.78
# Set threshold to 0.75 (slightly conservative)

Detecting and Preventing Flakiness

Sources of flakiness in eval tests:

• LLM non-determinism (same prompt yields different outputs)
• Floating-point precision issues in metrics
• External API failures (OpenAI API timeout)
• Metrics with inherent variance (LLM-as-judge is stochastic)

Prevention strategies:

def test_with_retries():
    """Run test multiple times to detect flakiness"""
    max_retries = 3
    for attempt in range(max_retries):
        try:
            output = model.generate("test input")
            assert_test(
                actual_output=output,
                metrics=[Relevancy(threshold=0.75)]
            )
            return True
        except AssertionError:
            if attempt == max_retries - 1:
                raise
            time.sleep(1)  # Brief backoff

def test_with_confidence_intervals():
    """Require high confidence in metric scores"""
    outputs = [model.generate("input") for _ in range(5)]
    scores = [Relevancy().measure(o) for o in outputs]
    
    mean = np.mean(scores)
    std = np.std(scores)
    
    assert mean > 0.75, f"Mean score {mean:.2f} below threshold"
    assert std < 0.15, f"High variance: {std:.2f} (inconsistent output)"

Production Patterns & Examples

Complete Example: RAG Evaluation Pipeline

from deepeval.metrics import Faithfulness, AnswerRelevancy, Contextual Precision
from deepeval import assert_test

def test_rag_retrieval_quality():
    """Test RAG system: does it retrieve relevant documents?"""
    
    query = "What is the company's parental leave policy?"
    
    # RAG pipeline: retrieve documents
    retrieved_docs = retriever.search(query)
    
    # Combine into context
    context = " ".join([doc.text for doc in retrieved_docs])
    
    # Generate answer using context
    answer = llm.generate(
        prompt=f"Answer based on context: {context}\n\nQ: {query}"
    )
    
    # Eval 1: Are retrieved documents relevant to query?
    assert_test(
        input_text=query,
        actual_output=context,
        expected_output="Parental leave policy information",
        metrics=[
            ContextualPrecision(threshold=0.75),  # Retrieve docs are precise
        ]
    )
    
    # Eval 2: Is answer grounded in retrieved context?
    assert_test(
        input_text=query,
        actual_output=answer,
        context=context,
        metrics=[
            Faithfulness(threshold=0.85),  # Answer matches context
            AnswerRelevancy(threshold=0.8),  # Answer addresses question
        ]
    )

def test_rag_edge_cases():
    """Test failure modes"""
    
    # Case 1: Query with no good documents
    answer = rag_pipeline("Obscure technical term")
    assert_test(
        actual_output=answer,
        expected_output="I don't have information on that",
        metrics=[Correctness(threshold=0.8)]
    )
    
    # Case 2: Hallucination detection
    answer = rag_pipeline("Company founding date")
    assert_test(
        actual_output=answer,
        context="Document only mentions 2020",
        metrics=[
            Faithfulness(threshold=0.95),  # High bar for factual claims
        ]
    )

# Run with pytest
# pytest test_rag.py -v

Real-World Example: Content Moderation

def test_content_moderation_suite():
    """Comprehensive content moderation eval"""
    
    test_cases = [
        # (content, should_flag, reason)
        ("Great product!", False, "normal_positive"),
        ("I hate this #@$!", True, "toxic"),
        ("Buy cheap meds now!", True, "spam"),
        ("Check out example.com", False, "normal_link"),
        ("Buy cheap meds at pharmacy.com", True, "spam_with_link"),
    ]
    
    for content, should_flag, reason in test_cases:
        prediction = moderation_model.predict(content)
        
        if should_flag:
            assert prediction.is_flagged, \
                f"Failed to flag: {content} (reason: {reason})"
            
            # Also check categorization
            expected_category = get_category(reason)
            assert prediction.category == expected_category, \
                f"Wrong category: expected {expected_category}, got {prediction.category}"
        else:
            assert not prediction.is_flagged, \
                f"False positive: {content} (reason: {reason})"