Annotation Quality at Scale

Introduction: The Scaling Problem

Annotation quality at scale is one of the most consequential and underestimated challenges in AI development. As projects grow from hundreds to thousands, then tens of thousands of labeled items, maintaining consistent accuracy becomes exponentially harder. A single expert annotator can produce meticulous work, but scaling that expertise across a team of dozens or a crowd of hundreds introduces systematic quality degradation that, left unchecked, can render an entire dataset unusable.

The fundamental challenge is this: annotator behavior is unstable. Fatigue sets in. Mental models drift. Instructions become reinterpreted. Motivation fluctuates. Quality control systems that work for small teams—having a senior expert review every label—become prohibitively expensive at scale. What's needed instead is a systems approach: automated checks, tiered annotator assignments, statistical sampling strategies, and continuous monitoring that can detect quality problems before they contaminate your dataset.

This guide walks you through the complete architecture for maintaining annotation quality at any scale, from 1K to 1M items.

Quality Tier Definitions and Task Routing

Understanding the Tier System

Not all annotation tasks are equally difficult, and not all annotators should work on equally difficult tasks. The tier system solves this by (a) defining what level of accuracy is needed for different task types, and (b) assigning annotators to tiers based on demonstrated capability.

Tier 1: Expert Annotation (>95% Accuracy)

Tier 1 annotators are your elite pool—typically PhDs, domain SMEs, or annotators with years of specialized experience. They should achieve >95% accuracy on gold-standard test sets and maintain that performance across months of work.

Example tasks for Tier 1:

• Medical record classification: ICD-10 code assignment from clinical notes (requires medical domain knowledge)
• Legal contract annotation: Identifying liability clauses and payment terms (requires legal expertise)
• Scientific paper categorization: Assigning research papers to specific subdisciplines (requires disciplinary knowledge)
• Audio speech act classification: Labeling prosody, speaker intent, emotional tone (requires linguistic and acoustic training)

Tier 1 cost: Typically $25-80/hour depending on specialty. For a 50K item project at ~4 items per hour, this is $312K-1M. Used selectively for the most critical subsets.

Tier 2: Trained Annotators (88-95% Accuracy)

Tier 2 annotators are trained specialists—people who may not be domain experts but have been intensively trained on your specific annotation schema. They demonstrate 88-95% accuracy on test sets and maintain stability over weeks to months.

Example tasks for Tier 2:

• Sentiment analysis of customer reviews: 3-5 class sentiment (positive/neutral/negative) with some nuance
• Entity extraction from financial documents: Identifying company names, dates, financial figures (rule-based but requires attention)
• Product image categorization: Assigning e-commerce images to product categories (visual pattern matching)
• Intent classification in chatbot logs: Is this customer asking for a refund, product info, technical support, etc.?
• Software bug severity rating: Is this critical, high, medium, or low impact? (requires some technical context but not deep expertise)

Tier 2 cost: Typically $12-25/hour. For the same 50K item project at ~6 items per hour, this is $100K-208K.

Tier 3: Crowdsourced Annotation (75-88% Accuracy)

Tier 3 annotation uses crowdsourcing platforms (Amazon Mechanical Turk, Scale AI, etc.) for tasks that are relatively clear and don't require specialized training. Individual crowdworker accuracy is 75-88%, but by collecting 3-5 independent labels per item and using majority vote, you can achieve Tier 2 equivalent accuracy.

Example tasks for Tier 3:

• Image classification: Is this image a cat or a dog? (clear visual distinction)
• Paraphrase identification: Do these two sentences mean the same thing? (binary, intuitive)
• Toxicity classification: Is this comment toxic or not? (binary with guidelines)
• Duplicate detection: Are these two product listings for the same item?
• Relevance judgment: Is this search result relevant to the query? (yes/no)

Tier 3 cost: Typically $1-5/hour per worker. But since you need 3-5 labels per item, the effective cost per item is comparable to Tier 2 when quality is adjusted. However, upfront cost is lower: for 50K items at $0.05/item × 3 labels = $7.5K. Throughput is extremely fast (days vs. weeks).

Automated QC Pipeline Design

Why Automation Matters

Manual quality assurance—having a senior annotator review every label—doesn't scale. At 1000 items/week, that's 250 hours of review work. Automation can pre-filter items for manual review, reducing the manual QA burden from 100% to 10-20% of items.

The Five-Layer Automated QC Pipeline

Layer 1: Format and Completeness Checks

• Is the label field populated? (binary check)
• Does the label belong to the allowed set? (e.g., if classes are {positive, neutral, negative}, reject "POSITIVE")
• Are confidence scores in valid range [0, 1]? (if provided)
• Are all required metadata fields filled? (e.g., time spent, annotator ID, timestamp)

Automated flagging rate: Typically 2-5% of submissions. Cost: near-zero (regex checks).

Layer 2: Logical Consistency Checks

• Cross-field validation: If annotation is "name entity → Company" but the entity text is a known currency, flag for review
• Schema violations: If tagging relationship between two entities, verify both entities were labeled in the same item
• Numeric bounds: If annotating numeric quantities, check that the value falls within domain-realistic ranges (e.g., salary not 1B dollars)
• Temporal consistency: If items have temporal ordering, check that annotations respect that ordering

Automated flagging rate: 1-3%. Cost: SQL/rule-based logic.

Layer 3: Gold Standard Comparison

• Maintain a set of gold standard items (100-500 items with known correct labels, usually annotated by Tier 1 experts)
• Periodically insert gold items into the regular workflow without marking them as such
• Compare each annotator's gold performance against a threshold (e.g., 90% accuracy on gold items)
• If an annotator falls below threshold, escalate to quality review

Typical gold set injection: 5-10% of items shown to each annotator are gold. At a typical project pace (5 items/hour), each annotator gets ~2-4 gold items per shift.

Automated flagging rate: 5-15% of annotators per cycle (depending on threshold). Cost: relatively low (pre-existing gold set comparison).

Layer 4: Anomaly Detection on Label Distributions

• Per-annotator class distribution: Does this annotator assign "positive" sentiment to 95% of items while team average is 30%? Potential bias
• Time-based anomalies: Did labeling speed suddenly drop by 50%? Could indicate fatigue or confusion
• Confidence anomalies: If annotator is marking high-confidence labels on items that are objectively ambiguous, red flag
• Statistical process control: Track a z-score or control chart for each annotator's key metrics over time

Automated flagging rate: 2-8% of items. Cost: requires statistical modeling but worth the investment.

Layer 5: Honeypot Traps

• Occasionally insert obviously wrong labels (marked as gold) to test annotator attention. E.g., a product clearly labeled as a "house" for an image of a coffee cup.
• If annotator agrees with honeypot (i.e., they should have caught the error), they fail the honeypot and go into review
• Honeypots should be rare (1-2 per 100 items) to avoid demoralizing annotators

Honeypot pass rate: Healthy teams should pass 95%+ of honeypots. <95% suggests careless work.

Statistical Sampling Strategies

The Sampling Pyramid

You can't manually audit every annotation. Instead, use statistical sampling to detect problems with high confidence while reviewing a manageable percentage of items. The choice of sampling method depends on your risk tolerance and budget.

Square Root Rule (Simple Stratified)

For each annotator completing n items, manually audit √n items chosen at random.

Example: An annotator completes 100 items. Audit √100 = 10 items. If those 10 are correct, your confidence is high that the full batch is good.

Statistical basis: This rule yields approximately 68% confidence that true accuracy is within ±10% of observed sample accuracy (rough binomial approximation).

Cost: For a 50K item project, this means auditing 223 items (√50K ≈ 223). At 2 minutes per audit, that's 7.4 hours of review work—very manageable.

Fixed-Rate Sampling

Audit a fixed percentage of all items, e.g., 10% or 20%.

Advantage: Simple to implement and communicate. "We review 10% of all submissions."

Disadvantage: Doesn't adjust for annotator volume. A high-volume annotator doing 1000 items gets 100 audited (10%), while a low-volume annotator doing 50 items gets 5 audited. Both yield similar confidence, but one uses 20x more resources.

When to use: When annotators have relatively equal volume and you want a simple, transparent sampling policy.

Risk-Based Sampling

Allocate audit resources based on risk score for each annotator. Annotators with low gold-standard performance, high anomaly flags, or new to the project get higher sampling rates.

Risk score formula (example):

risk_score = (1 - gold_accuracy) × 0.5 + (z_score_of_speed_anomaly / 3) × 0.3 + (is_new_annotator) × 0.2

Then: Sampling rate = baseline_rate × (1 + risk_score)

Example calculation:

• New annotator (is_new = 1), gold accuracy 88%, no speed anomaly
• risk_score = (1 - 0.88) × 0.5 + 0 × 0.3 + 1 × 0.2 = 0.06 + 0.2 = 0.26
• baseline_rate = 5% (√n for volume ~400)
• actual_rate = 0.05 × (1 + 0.26) = 6.3%

Advantage: Focuses scarce review resources where most needed. Can catch problems faster.

Disadvantage: More complex to implement and communicate.

Sequential Sampling (SPRT)

Continue auditing items from an annotator until you accumulate enough evidence to confidently accept or reject their work.

How it works: Set two thresholds (acceptable accuracy = 90%, reject threshold = 70%). As you audit items, track the cumulative number of successes and failures. Once you have enough data points to cross one threshold, stop and make a decision.

Example: Audit 15 items, find 2 errors. That's 87% accuracy, which is below your 90% accept threshold but above your 70% reject threshold. Keep auditing. After 5 more items (20 total), you find 2 more errors. That's 80% accuracy—still ambiguous. At 25 items with 3 errors, you hit 88%—still auditing. At 30 items with 3 errors (90%), you can now accept this annotator's work with statistical confidence.

Advantage: Efficient—you audit just enough to make a confident decision, no more.

Disadvantage: Requires statistical expertise to set up correctly.

Annotation Quality Metrics and Targets

The Core Metrics

Precision: Of the items labeled as class A by the annotator, what fraction actually are class A (according to gold standard)? Formula: TP / (TP + FP).

Example: Annotator labels 50 items as "positive" sentiment. Upon review, 45 are actually positive, 5 are neutral. Precision = 45/50 = 90%.

Target by Tier: Tier 1 ≥95%, Tier 2 ≥90%, Tier 3 ≥80% (per-label aggregation).

Recall: Of all items that actually are class A, what fraction did the annotator label as class A? Formula: TP / (TP + FN).

Example: There are 30 positive sentiment items in the test set. The annotator correctly identified 28. Recall = 28/30 = 93%.

Target by Tier: Tier 1 ≥95%, Tier 2 ≥90%, Tier 3 ≥80%.

F1 Score: The harmonic mean of precision and recall. Useful when you care about both precision and recall equally. Formula: 2 × (precision × recall) / (precision + recall).

Consistency Rate (Agreement): When showing the same item twice (at different times), does the annotator give the same label? This measures label reproducibility, independent of correctness.

Example: Annotator labels item X as "positive" on day 1, then sees a disguised version of the same item on day 7 and labels it "positive" again. Consistency score goes up by 1. After 20 re-shown items, annotator has 18 consistent (90% consistency).

Target: Tier 1 ≥95%, Tier 2 ≥90%, Tier 3 ≥85%. Note: consistency should be higher than accuracy, since it doesn't require knowledge of ground truth.

Honeypot Pass Rate: Percentage of obviously incorrect (honeypot) items that the annotator correctly rejected.

Example: 5 honeypot items inserted. Annotator falls for 1 (agrees with obvious error), passes 4. Pass rate = 80%.

Target: ≥95% for all tiers.

Metric Dashboard Template

Annotator ID	Precision	Recall	F1 Score	Consistency	Honeypot Pass	Status
Ann-001	94%	92%	0.93	96%	100%	PASS
Ann-002	88%	85%	0.86	91%	100%	CAUTION
Ann-003	75%	79%	0.77	84%	80%	FAIL
Ann-004	91%	93%	0.92	94%	100%	PASS

Interpreting the Metrics

High precision, low recall: Annotator is conservative. They only label items as class A when very confident, but miss many true positives. This is sometimes acceptable if false positives are more costly than false negatives.

Low precision, high recall: Annotator is aggressive. They label many items as class A, catching most true positives but also many false positives. This is acceptable if false negatives are very costly.

Low precision AND low recall: The annotator is confused or not trying. Serious issue. Likely need retraining or removal.

High consistency, low accuracy: Annotator is consistently wrong. They have a stable mental model that doesn't match the task definition. Needs retraining on schema.

Quality-Based Annotator Assignment

The Routing System

Instead of assigning items to annotators randomly, use a quality-aware routing system that assigns harder items to higher-performing annotators and easier items to newer or lower-performing annotators. This maximizes overall quality per dollar spent.

Implementation Steps

Step 1: Calibrate Task Difficulty

• Have Tier 1 annotator label a sample of 500 items across the full project
• For each item, record whether it was labeled correctly (as determined by gold standard or multiple expert agreement)
• Calculate item difficulty: (1 - % correct) across tier 1 annotators
• Items with 95%+ agreement = easy, items with 70-85% agreement = medium, items with <70% agreement = hard

Step 2: Assess Annotator Capability

• On a calibration set of 50-100 items (with known labels), measure each annotator's accuracy
• This becomes their capability score

Step 3: Assign Based on Difficulty-Capability Match

• Annotators with 95%+ capability get priority on hard items
• Annotators with 85-95% capability get medium items
• Annotators with <85% capability get easy items (to avoid further degradation)

Example: A project has 10,000 items: 3,000 easy, 5,000 medium, 2,000 hard. You have 4 Tier 2 annotators with capabilities [92%, 88%, 81%, 79%].

• Ann-001 (92%): Gets 1,500 hard + 1,500 medium = 3,000 items expected quality 92%
• Ann-002 (88%): Gets 500 hard + 2,000 medium = 2,500 items expected quality 87%
• Ann-003 (81%): Gets 2,500 easy + 500 medium = 3,000 items expected quality 81%
• Ann-004 (79%): Gets 500 easy = 500 items expected quality 79%

Overall expected accuracy = (3000 × 0.92 + 2500 × 0.87 + 3000 × 0.81 + 500 × 0.79) / 9500 ≈ 85%. Without routing, with equal distribution, you'd expect (10000 × 85%) / 10000 = 85%. But with routing, you're also reducing burnout of lower-capability annotators on hard tasks (which is demoralizing), improving retention.

Quality Degradation Patterns and Recovery

The Degradation Lifecycle

Annotation quality rarely crashes suddenly. Instead, it follows a predictable degradation lifecycle. Understanding this cycle lets you intervene before quality becomes unacceptable.

Phase 1: Initial Stability (Days 1-7)

Annotator is fresh, motivated, and carefully following instructions. Accuracy stable at 90-95%. Consistency high. Honeypot pass rate 100%.

Phase 2: Fatigue Begins (Days 8-21)

Labeling speed increases (efficiency improving), but first sign of fatigue: honeypot pass rate drops from 100% to 95-98%. Accuracy still 88-93%. Early intervention window—retrain or rotate task.

Phase 3: Evident Drift (Days 22-35)

Accuracy drops to 82-88%. Class distribution becomes unbalanced (e.g., annotator labels 40% positive when baseline is 25%). Speed has leveled off. Consistency metrics drop to 88-92%. Critical intervention window—must act now.

Phase 4: Systemic Degradation (Days 36+)

Accuracy below 75%. Honeypot pass rate <80%. Annotator may be explicitly cutting corners. Quality data is now contaminated and difficult to fix retroactively. At this stage, remove annotator.

Degradation Causes

Cognitive Fatigue: After 6-8 hours of intense concentration, human performance degrades. Solution: enforce breaks or daily hour limits (max 4-5 hours focused annotation per day).

Concept Drift in Instructions: Annotator's mental model of the task schema gradually shifts. E.g., what counts as "negative" sentiment drifts from "clearly negative" to "anything not positive". Solution: periodic re-calibration with gold items.

Domain Adaptation: As annotator labels more items, they build expectations about what's "normal" for this dataset. These expectations can be wrong. E.g., if first 100 items are all easy, annotator may lower guard for harder items later. Solution: randomize item order or show difficulty distribution statistics.

Social Influence: In team annotation, annotators influence each other. If a senior annotator is lenient, juniors become lenient. Solution: provide individual feedback (not comparative) and rotate team compositions.

Payment or Motivation Changes: If an annotator's pay is cut or project feels less valued, quality drops. Solution: maintain consistent pay and communicate project importance.

Recovery Protocols

When Gold Standard Performance Drops Below 90%:

1. Pause new assignments (within hours, not days)
2. Schedule 30-minute retraining call: walk through 5-10 recent errors, clarify schema
3. Have annotator re-label last 50 completed items for quality comparison
4. Resume with easy items, not hard items
5. Re-check gold standard in 2 days; if still <90%, escalate

When Speed Anomaly Detected (>30% change):

1. Don't assume it's good (faster = more errors). Investigate.
2. Check last 20 items for accuracy—if still ≥90%, speed increase is fine
3. If accuracy dropped, annotator is rushing. Discuss pacing

When Multiple Quality Signals Fail (gold <85%, honeypot <85%, consistency <85%):

1. Schedule one-on-one call with annotator
2. Discuss challenges, workload, motivation
3. Decision point: intensive retraining, task reassignment, or removal

Real-World Cost Analysis

The Economic Case for Quality

IBM Research published a landmark study (Redman, 2017) estimating that poor data quality costs U.S. businesses $1.5 trillion annually. For AI projects specifically, the cost is typically 10-20% of total project spend due to downstream retraining and business impact.

Cost Breakdown: A 50K Item Project

Scenario: Sentiment analysis of customer reviews, 50K items, 3-class (positive, neutral, negative)

Option A: Tier 2 Annotators Only

• 50,000 items ÷ 6 items/hour = 8,333 annotator-hours
• 8,333 hours × $15/hour = $125,000 (annotation cost)
• QC infrastructure (5 layers of automated checks): ~$20,000
• Manual review (10% of items at 2 min each): 50,000 × 0.1 × 2 min / 60 = 166 hours × $25/hour (QA manager) = $4,150
• Total: $149,150
• Expected accuracy: 89% (assuming Tier 2 average)
• Quality-adjusted cost: $149,150 / 0.89 = $167,528 per acceptable annotation

Option B: Hybrid (Tier 3 + Tier 2 for disagreements)

• 50,000 items × $0.08/item (3 crowd labels + aggregation) = $12,000
• Crowd accuracy: 82% (individual annotators ~78%, but 3-way majority vote lifts to 82%)
• Identify 20% of items where crowd is uncertain (entropy-based): 10,000 items
• Send uncertain items to Tier 2: 10,000 ÷ 6 = 1,667 hours × $15 = $25,000
• QC infrastructure: $20,000 (same as Option A)
• Manual review (5% of items—only the Tier 2 batch): 2,500 items × 2 min / 60 × $25 = $2,083
• Total: $59,083
• Expected accuracy: 85% (82% from crowd on easy items, 89% from Tier 2 on hard items, weighted)
• Quality-adjusted cost: $59,083 / 0.85 = $69,508 per acceptable annotation

Option C: Overinvest in Tier 1 (Expert Consensus)

• 50,000 items ÷ 3 items/hour (expert pace) = 16,667 annotator-hours
• 16,667 hours × $50/hour (domain expert rate) = $833,350
• Two experts per item (for consensus): 2 × $833,350 = $1,666,700
• QC: $20,000
• Total: $1,686,700
• Expected accuracy: 97%
• Quality-adjusted cost: $1,686,700 / 0.97 = $1,738,866 per acceptable annotation

Option D: Underinvest (No QC, Cheapest Crowd)

• 50,000 items × $0.02/item (single crowd label, no aggregation, no QC) = $1,000
• Total: $1,000
• Expected accuracy: 65% (single crowd annotator with no aggregation)
• Quality-adjusted cost: $1,000 / 0.65 = $1,538 per acceptable annotation
• HOWEVER: The 35% contaminated labels will likely cause your downstream model to fail or require significant retraining, costing $50K-200K+ to fix.

Option	Upfront Cost	Expected Accuracy	Cost per Acceptable Item	Recommendation
A: Tier 2 Only	$149,150	89%	$1.68	Good default
B: Hybrid (Tier 3 + 2)	$59,083	85%	$0.70	Best value
C: Tier 1 Expert	$1,686,700	97%	$17.39	Only if mission-critical
D: No QC	$1,000	65%	$0.02*	Never use

Worked Example: 50K Items/Month Sentiment Pipeline

Project Setup

Goal: Annotate 50,000 customer review snippets per month for sentiment (positive, neutral, negative). This data trains a deployed sentiment classifier used by the customer success team to identify at-risk customers.

Data characteristics:

• Average review length: 20-50 words
• Class distribution in holdout set: 45% positive, 30% neutral, 25% negative
• Difficult cases: ~8% of reviews have sarcasm, mixed sentiment, or ambiguous intent

Quality requirement: ≥88% accuracy so downstream model converges and doesn't degrade live performance

Phase 1: Calibration (Week 1)

Step 1a: Create Gold Standard

• Have 2 Tier 1 experts (in-house product managers) annotate 200 diverse review samples
• If they disagree on <5% of items, freeze this as gold standard. If >5% disagreement, have them align on definition
• Result: 200-item gold standard with clear positive/neutral/negative examples, including sarcasm samples

Step 1b: Calibrate Difficulty

• Have 3 Tier 2 annotators independently label the full 200-item gold set
• Calculate per-item accuracy: items where all 3 agree with gold = easy (100% accuracy)
• Result: 50 easy items, 120 medium, 30 hard (sarcasm, mixed sentiment)

Step 1c: Recruit Annotators

• Recruit 8 Tier 2 annotators (trained specialists, $12-15/hour)
• Have each complete a 50-item calibration batch with feedback
• Result: 6 annotators pass calibration at 88-93% accuracy; 2 don't meet bar and are not hired

Phase 2: Setup Automation (Week 1)

Build the QC Pipeline:

Layer 1: Format checks - Script checks that label is one of {positive, neutral, negative}

Layer 2: Logical consistency - No special checks needed for this task (single label per item)

Layer 3: Gold comparison - Inject 5% gold items (~25 per 500 items) into the workflow unmarked. Track each annotator's gold performance daily.

Layer 4: Anomaly detection - Flag annotators if:

• Gold accuracy drops below 85%
• Class distribution shifts by >10% from baseline
• Labeling speed drops by >25% (potential confusion or loss of motivation)

Layer 5: Honeypots - Create 5 honeypot items (e.g., "This product is amazing and I love it!!!" labeled as neutral). Inject 1 honeypot per 100 items.

Phase 3: Execution (Weeks 2-4)

Daily Operations:

• Route items: 20% hard (to top 3 annotators), 50% medium (to all 6), 30% easy (to all 6)
• Target throughput: 50,000 items / 22 business days ≈ 2,273 items/day
• At 6 items/hour, 6 annotators × 6 hours/day = 216 items/day... need to scale to 8 annotators or increase hours
• Actual plan: 8 annotators × 8 hours/day (with 1-hour break) = 384 items/day. Over 22 days: 8,448 items. Need 6 batches of this size.

Weekly Monitoring:

Week	Items Completed	Avg. Gold Accuracy	Honeypot Pass Rate	Notes
Week 1 (Calibration)	400	90%	100%	Team settling in, all pass
Week 2	10,000	88%	98%	Ann-004 dips to 84% gold. Schedule retraining.
Week 3	19,500	87%	97%	Ann-004 back to 89% after retraining. Ann-005 shows 92% — promote to hard items.
Week 4	30,000	88%	96%	All annotators stable. Final 20,000 items on schedule.

Cost Tracking:

• Annotation: 50,000 items ÷ 6 items/hour = 8,333 hours × $14/hour (average) = $116,667
• Gold standard creation: 200 items × 30 min = 100 hours × $40/hour (expert) = $4,000
• Calibration/recruitment: 8 annotators × 50 min screening = 400 hours expert time = $16,000
• Retraining (Ann-004): 1 hour × $40/hour = $400
• QC automation build: ~$15,000 (one-time, includes script development and validation)
• Manual review (10% of items): 5,000 items × 2 min = 167 hours × $25/hour = $4,167
• Total: $156,234
• Per-item cost: $3.12
• Expected accuracy: 88%
• Cost per acceptable item: $3.54

Phase 4: Validation & Handoff

Hold-Out Test Set:

• Randomly select 500 items from final 50,000 (haven't been audited yet)
• Have Tier 1 expert label them (gold standard)
• Measure accuracy of the full 50K set against this test set: 87.2%
• Meets requirement of ≥88%? Just barely missed. Options:
  • Re-label 5-10% of lower-confidence items (those with close vote splits or low annotator confidence)
  • Accept 87.2% and acknowledge 1.2% degradation vs. requirement
  • For next month, tighten QC to push toward 89%

Lessons Learned / Next Month:

• Ann-004's mid-month dip could have been prevented with more frequent check-ins (daily vs. weekly gold comparison)
• Hard item accuracy (85%) was lower than expected—may need to invest in more training for hard cases or use Tier 3 → Tier 2 escalation instead
• Consider rotating annotators monthly to prevent long-term fatigue

Summary and Key Takeaways