The Demand Proof Protocol: Validating Market Demand for Eval Services

Why Demand Validation Is Non-Negotiable

Building an AI evaluation service or practice without first validating that customers actually want it is the #1 failure mode for eval-focused startups and consulting practices. The pattern is predictable: a capable engineer or consultant spends 6-12 months building a "comprehensive eval platform" or establishing a "specialized eval practice," only to discover that customers don't perceive a problem requiring their solution. The market isn't ready, the price is wrong, or the target customer was misidentified entirely.

Demand validation isn't about building a business case—it's about de-risking your time investment. Before you invest months of effort, you need concrete evidence that (1) customers recognize a problem in this space, (2) they believe your proposed solution addresses it, (3) they're willing to pay for it, and (4) the willingness to pay aligns with your cost structure. Each of these is testable without building anything.

The stakes are particularly high in the eval space because building a credible eval offering takes genuine technical depth. You can't fake it. This means your opportunity cost is high—the time you spend building evaluation infrastructure is time not spent on something customers are already paying for. Demand validation compresses the feedback loop from months to weeks.

The fundamental principle: invest in validating demand incrementally, investing increasingly substantial resources only as evidence accumulates. Start with low-cost signals (interviews), move to medium-cost signals (landing pages, surveys), and only build product when strong demand signals exist. Strong demand looks like: customers who independently recognize the problem, are actively searching for solutions, and express willingness to pay before you've built anything.

The Demand Proof Hierarchy: What Each Signal Proves

Not all demand signals are created equal. The hierarchy ranges from weak (people say they're interested) to strong (people are already paying you). Understanding what each signal type proves—and doesn't prove—prevents misinterpreting enthusiasm for demand.

Level 1: Vague Survey Interest is the weakest signal. When you email your network asking "would you pay for eval-as-a-service?", most will say yes out of politeness. This proves nothing except that you have contacts. Conversion rates mean little without context. The problem: surveys ask hypothetical questions, and people answer hypothetically.

Level 2: Validated Interview Interest is stronger. When you conduct 10+ in-depth interviews with target customers where they articulate the problem unprompted, describe their current unsatisfactory solution, and express frustration with existing alternatives, you're seeing real demand signals. This proves that the problem exists and is felt acutely enough to spend an hour discussing it. The limitation: interviews show problem recognition, not necessarily willingness to pay.

Level 3: Landing Page Conversion tests demand by building a minimal product description and measuring how many people willingly give you their email or book a call. Landing page tests eliminate the hypothetical—people are actually considering conversion. A 5% conversion rate on relevant traffic is strong signal; less than 1% suggests weak demand. This proves people are interested enough to consider your solution.

Level 4: Letters of Intent are formal expressions of future purchase intent. A prospect says, "If you build X, we'll buy it." LOIs are serious—they commit organizational resources to evaluation. However, LOIs still aren't cash. They prove willingness to purchase but not immediate need. Some LOI holders will evaporate when you ask for a contract.

Level 5: Pre-Sales and Pilots are the strongest proof short of deployed paying customers. When a prospect signs a pilot contract and pays (even a reduced amount), they've moved from talk to action. Pilots are expensive for the customer—they require integrating new vendor, evaluating the solution, and committing staff time. The existence of pilots proves real, urgent demand.

Defining Your Eval Service Hypothesis: The 4-Part Structure

Before running any validation, you need to articulate a testable hypothesis. Many eval service ideas fail because they're vague: "provide better eval infrastructure" or "democratize evaluation." These aren't hypotheses, they're directions. A testable hypothesis has four components: customer segment, problem, solution, and price.

Customer Segment: Which organizations will you serve? "AI companies" is too broad. Better: "series A-B LLM application startups deploying RAG systems." Even better: "Series A-B startups in financial services deploying RAG for customer support." Narrower segments are easier to validate—there are fewer of them, they cluster geographically or on platforms, and their needs are homogeneous. Validation in a narrow segment also de-risks your go-to-market; you can eventually expand but you start focused.

Problem: What specific problem does your target customer face? Not "they need better evaluation" but "they spend 4 weeks per model release manually validating RAG quality, delaying releases and creating bottlenecks." Describe the problem in customer language, not technical language. How does it manifest? What's the cost? How acute is it right now?

Solution: What is your proposed solution? For the RAG validation problem, maybe it's a specialized RAG evaluation platform with pre-built metrics and seamless Langchain integration. The solution should directly address the stated problem. Include what you'll build and what you won't. Most eval services fail because they try to solve "all eval problems" instead of one specific problem for one segment.

Price: At what price will you offer this? Price is testable and you should test it explicitly. The four-part hypothesis might be: "Series A-B fintech startups deploying RAG for customer support will pay $5K-10K/month for a specialized RAG evaluation platform that reduces validation time from 4 weeks to 1 week." This is testable—you can interview prospects and ask whether they'd pay $7.5K/month for that outcome.

Your hypothesis will evolve as you validate, but having an explicit starting hypothesis prevents wandering aimlessly and makes validation efficient. Write it down before you start talking to customers—it focuses your interview questions and helps you avoid confirmation bias.

The 10-Interview Sprint: Structured Customer Discovery in 2 Weeks

The most efficient validation method is structured customer interviews. You can conduct 10 quality interviews in 2-3 weeks if you're focused. The goal isn't to sell—it's to learn whether prospects recognize the problem and whether your proposed solution is relevant.

Recruit the right participants: You need 10 interviews with people in your target customer segment. They should fit your hypothesis. If your hypothesis targets "series A-B fintech startups with RAG systems," recruit VP Engineering or ML leads from companies matching that profile. Avoid talking to people who are too similar to you or whom you already know well—they'll tell you what you want to hear. Reach out via warm intros, conferences, or Cold outreach. Offer them 30 minutes and no commitment.

The interview script: Your script should have three phases. First, problem discovery: "Tell me about your current evaluation process. How do you ensure RAG quality before releasing to customers?" Listen more than you talk. The goal is to hear them describe the problem in their own words. Second, reaction to your proposed solution: "Imagine a platform that did X, Y, Z. How would that affect your workflow?" Don't oversell. Describe your solution objectively and listen to their reaction. Third, price sensitivity: "What would you pay for a tool that reduced validation time from 4 weeks to 1 week?" This is awkward but necessary. Many will deflect ("depends on features"), but some will give numbers.

Analysis framework: After 10 interviews, analyze the patterns. Did prospects independently describe a problem similar to what you hypothesized? (If you had to explain the problem, demand is weak.) Did they react positively to your solution, or did they suggest alternatives? Did anyone offer price anchors? Did anyone ask for a follow-up conversation or pilot? Count the number of "strong signals" (unsolicited problem articulation, enthusiastic solution reaction, price indication, follow-up request). If 7+ of 10 show strong signals, you have validated demand. If fewer than 5 show strong signals, validate further before building.

The Landing Page Test: Measuring Interest With Email Capture

A landing page test validates demand by building a minimal product description and measuring how many people willingly give you their email address or book a call. This eliminates the hypothetical nature of interviews—people are actually considering engaging with your solution.

Build a simple landing page (Webflow, Carrd, or Notion) describing your eval service. Include: problem statement (in customer language), how your solution works, what benefit they'll get, and a CTA (email signup or Calendly link for a call). Keep it short—under 500 words. Run paid traffic to this page (Google Ads, LinkedIn ads) targeting your customer segment. Budget $500-1000 to get meaningful traffic.

Interpretation: A 5%+ conversion rate on relevant traffic (people clicking "Learn More" or booking a call) is strong signal. 1-3% is moderate. Under 1% suggests the page isn't resonating. Analyze which messaging variations convert best. Do people respond more to the problem statement or the solution benefit? This informs your go-to-market framing.

Critically, who is converting? If your traffic is mostly developer tools and you're targeting enterprises, your traffic source is wrong. Refine your targeting and re-run. Real demand shows up consistently across traffic sources and messaging variations.

The Consulting Pilot: Validating Value Before Product

Some eval services are discovered best through consulting pilots. Instead of building a platform, deliver your core value manually first. If your hypothesis is that financial services companies need customized eval frameworks, run a pilot engagement: spend 4-6 weeks with a prospect, designing their evaluation program, building custom metrics, and implementing the system. Charge for this work (at least cost recovery, ideally profit).

The consulting pilot accomplishes several things simultaneously: (1) You validate that the problem is real and urgent enough for customers to pay; (2) You learn what a solution actually needs to include; (3) You build a case study and reference customer; (4) You generate revenue while validating. A successful consulting pilot that leads to a customer asking for a productized offering is the strongest possible demand signal.

Structure pilots to test specific hypotheses. If you think "companies need customized eval taxonomies," design the pilot to validate that. If a pilot prospect says "actually, we need simpler metrics, not more taxonomies," that's learning. Good pilots answer yes-or-no questions about demand. At the end, ask explicitly: "Would you pay for a productized version of this?"

Price Sensitivity Testing: Van Westendorp and Willingness-to-Pay Methods

Pricing is the most validated element of demand. You can test whether customers will actually pay without building a single line of code. The Van Westendorp Price Sensitivity Meter is a simple four-question methodology: "At what price would you consider this too cheap (poor quality)?" "At what price would you consider this a bargain?" "At what price would you consider this expensive?" "At what price would you consider this too expensive?" Plot responses and find the price range with the highest percentage of people saying they'd buy.

Alternatively, use direct willingness-to-pay questions. During interviews, ask: "What would you pay for a tool that cuts validation time from 4 weeks to 1 week? $1K/month? $5K/month? $15K/month?" See where they balk. Most will express a price ceiling. If the median across 10 interviews is $8K/month and your costs are $2K/month, you have a viable business model. If the median is $2K/month and your costs are $5K/month, you have a viability problem.

Pricing validation reveals whether demand is strong enough to support a business. Many consultants discover they can build eval services people want, but at price points that don't work economically. For instance, you might find that mid-market companies want eval services but will only pay $3K/month (monthly retainer), while your cost structure requires $8K/month. That's a constraint to solve (can you achieve higher margins?) or a reason to pivot segments (target enterprises instead of mid-market).

Analyzing Demand Signal Quality: Strong vs. Weak Signals

Not all positive responses are created equal. After running interviews, landing pages, and pilots, you need to distinguish strong demand signals from weak signals. Weak signals feel good but don't predict business success. Strong signals are repeatable, specific, and price-validated.

Weak signals: "This is a great idea and I'd be interested if you ever launch." "We have some evaluation challenges." "We'd definitely consider trying this." These are polite, non-committal responses. They show interest but no urgency or specificity. If most of your signals are vague, demand isn't validated.

Strong signals: "Our current eval process is killing us—we spend 4 weeks per release and it's the blocker to shipping faster. We'd pay $10K/month to cut that to 1 week." "We have three ongoing projects that need this. Can we start a pilot?" "We evaluated tool X for this but it doesn't fit our RAG workflow." Strong signals are specific (they articulate the problem precisely), urgent (they indicate the problem is affecting business today), and validated (they include price anchors or willingness to commit).

When Demand Validation Fails: Pivoting and Persisting

Sometimes validation reveals that your original hypothesis is wrong. This is valuable learning. The question is: how do you know whether to pivot or persist?

You should pivot if: (1) 80%+ of interviews reveal no problem recognition (you had to explain the problem repeatedly), (2) Landing page conversion is under 0.5% even with well-targeted traffic, (3) Multiple consulting pilots revealed that the problem isn't as acute as hypothesized. You should persist if: (1) 70%+ of interviews show strong signal but only in a specific sub-segment (you may have misidentified the segment), (2) Landing page conversion is 3%+, (3) You have one paid pilot with a customer willing to extend.

When you pivot: Change one variable at a time. If problem recognition was weak, test a different customer segment with the same problem hypothesis. If problem recognition was strong but price anchors were low, test a different solution that's cheaper to deliver. Run 5 new interviews before concluding the pivot is right. Many teams pivot prematurely—they get one negative interview and assume the hypothesis is wrong. 5 interviews isn't enough to confirm or deny anything.

The hardest pivot is when you discover the problem is real and customers will pay, but the market is smaller than you thought. Maybe you targeted "all AI companies" but discovered that only 50 companies in the world fit your customer segment and can actually pay your price. That's not bad—it's a focused market you can own. But it means your business model needs to be efficient (50 customers at $20K each is $1M ARR, which works; 50 customers at $2K each is $100K ARR, which doesn't).

Demand Validation Case Studies: Three Eval Services

Case 1: The RAG Eval Platform Startup started with the hypothesis: "Series A-B LLM startups deploying RAG need specialized evaluation frameworks. They'll pay $5K-10K/month." They conducted 10 interviews and found strong problem recognition (80% articulated RAG evaluation challenges), but pricing was soft. Average willingness-to-pay was $3K/month, not $7.5K. They ran a landing page test and got 2.1% conversion. Decision: Proceed with a pilot, but revise pricing down to $3.5K/month. They need lower costs or higher margins. Hired one engineer to build an MVP (instead of three), targeting functionality that could achieve $3.5K profitability per customer. After the first customer paid $5K/month for a pilot, they learned customers would pay more for managed service (we'll set up evaluation for you) than for a platform (you set it up yourself). They pivoted to managed service, which supported higher pricing and differentiated from existing platforms.

Case 2: The Enterprise Eval Consulting Practice started with: "Large financial institutions need specialized evaluation for regulated models. They'll pay $200K-500K per year for a consulting engagement." They conducted 5 initial interviews and got strong signals (one prospect already had an RFP for eval services). They ran one consulting pilot with a regional bank (4-month engagement, $150K). The pilot worked, but they discovered the real problem wasn't evaluation methodology—it was documentation and governance processes for regulatory compliance. They pivoted to positioning eval as part of a broader governance consulting practice. Their eval expertise became the technical foundation for compliance work, which customers would pay more for. Revenue grew 3x when they stopped selling "eval consulting" and started selling "Model Risk Management Consulting (with eval as the core technical component)."

Case 3: The Open-Source Eval Tool Startup started by building an open-source eval framework (no customer validation). After 6 months, they had 500 GitHub stars but zero customers. They conducted post-hoc interviews with active users and discovered the problem: the tool was great for research but enterprises couldn't use it due to lack of support, integrations, and customization. They pivoted to a managed SaaS wrapper around the open-source tool. Enterprise sales required deeper integrations with existing platforms (DataDog, Weights & Biases) than they initially supported. They spent 3 months building integrations before finding their first paying customer. The learning: open-source adoption doesn't mean commercial demand. They should have validated the SaaS business model before building the tool.