Validation Experiments & Hypotheses
Problem Existence 🔴 Critical
We believe that AI engineers and prompt engineers at companies using LLMs in production
Will actively seek a dedicated tool to organize and version prompts
If they are managing prompts across Notion, text files, and chat histories
We will know this is true when 60%+ of surveyed practitioners confirm this as a top-3 pain point AND 5%+ landing page signup rate
Risk Level: 🔴 Critical (product fails if wrong)
Current Evidence: 78% of 50 Reddit discussions mention prompt chaos (r/MachineLearning, r/LocalLLaMA); 4.2K/mo Google searches for "prompt management tool"
Gap: No direct user interviews yet
Experiment Design
Method: Customer discovery interviews + landing page test
Sample: 25 interviews, 1,500 landing visitors
Cost: $600 ($500 ads + $100 incentives)
Timeline: 2 weeks
Problem Severity 🔴 Critical
We believe that AI practitioners
Will spend 5+ hours weekly manually organizing and testing prompts
If they are using spreadsheets and Notion for prompt management
We will know this is true when 70%+ of surveyed users report >5 hours/week on prompt management
Risk Level: 🔴 Critical
Current Evidence: 83% of 100 survey responses (via AI Discord communities) confirm >3 hours/week on prompt management
Gap: Limited to self-selected survey respondents
Experiment Design
Method: Time-tracking survey + usage analytics
Sample: 200 practitioners (via LinkedIn, AI forums)
Cost: $300 (survey tool + incentives)
Timeline: 1 week
Solution Fit 🔴 Critical
We believe that AI practitioners
Will prefer PromptVault over Notion/spreadsheets for prompt management
If we provide version control, multi-model testing, and analytics
We will know this is true when 70%+ of prototype users rate it as "better than current methods"
Risk Level: 🔴 Critical
Current Evidence: 68% of 30 Langchain users said they'd pay for versioning (survey)
Gap: No prototype testing yet
Experiment Design
Method: Wizard of Oz MVP (manual delivery)
Sample: 15 users (paid $50 incentive)
Cost: $750 (incentives + time)
Timeline: 3 weeks
Willingness to Pay 🔴 Critical
We believe that AI engineers at 10-100 person companies
Will pay $19/month for Pro plan
If we demonstrate 5+ hours/week time savings and team collaboration benefits
We will know this is true when 10+ pre-orders at $19/month
Risk Level: 🔴 Critical
Current Evidence: $19.99 avg price for developer tools (G2 data)
Gap: No paid conversion data
Experiment Design
Method: Pre-order test with payment gate
Sample: 50 targeted users
Cost: $0 (payment processing fee only)
Timeline: 1 week
Experiment Catalog
| Experiment | Hypothesis | Method | Cost | Success Criteria |
|---|---|---|---|---|
| Problem Discovery Interviews | #1, #2 | Semi-structured interviews (25) | $500 | 60%+ confirm problem as top-3 pain point |
| Landing Page Smoke Test | #1, #3 | Google/Facebook ads to landing page | $500 | 5%+ signup rate |
| Wizard of Oz MVP | #3, #4 | Manual prompt analysis delivery | $750 | 7/10+ satisfaction, 50%+ would pay |
| Pricing Survey (Van Westendorp) | #5, #6 | Online survey with price points | $200 | 70%+ select $19 as ideal price |
| Competitor Tear-Down Interviews | #3, #4 | Interview users of PromptBase/Langchain | $300 | 50%+ cite versioning as missing feature |
| Channel Testing (Reddit/LinkedIn) | #7 | $500 ad spend across channels | $500 | CAC < $20 on target channels |
| Referral Mechanism Test | #8 | Incentivized referral program | $100 | Viral coefficient > 0.5 |
Experiment Prioritization Matrix
| Experiment | Impact | Effort | Risk if Skipped | Priority |
|---|---|---|---|---|
| Problem Discovery Interviews | 🔴 Critical | Medium | Fail | 1 |
| Landing Page Test | 🔴 Critical | Low | Fail | 2 |
| Wizard of Oz MVP | 🔴 Critical | High | Fail | 3 |
| Pricing Survey | 🟡 High | Low | Suboptimal pricing | 4 |
| Channel Testing | 🟢 Medium | Medium | Inefficient CAC | 5 |
8-Week Validation Sprint
1 Week 1-2: Problem Validation
Landing Page Launch
Drive traffic with $500 ads
Target: 1,500 visitors
Discovery Interviews
25 targeted interviews
Focus: Pain point severity
2 Week 3-4: Solution Validation
Wizard of Oz MVP
Deliver 15 manual analyses
Measure satisfaction & willingness to pay
Competitor Analysis
Interview 10 PromptBase users
Identify key gaps
3 Week 5-6: Pricing Validation
Pricing Survey
100+ responses via LinkedIn
Van Westendorp methodology
Pre-Order Test
$19 price point test
Target: 10+ conversions
Minimum Success Criteria (Go/No-Go)
| Category | Metric | Must Achieve | Nice-to-Have |
|---|---|---|---|
| Problem | Interview confirmation | 60%+ | 80%+ |
| Landing page signup | 5%+ | 10%+ | |
| Solution | Prototype satisfaction | 7/10+ | 8.5/10+ |
| NPS | 30+ | 50+ | |
| Pricing | Willingness to pay at $19 | 50%+ | 70%+ |
| Pre-orders | 10+ | 25+ |
Go Decision
All "Must Achieve" criteria met (minimum 3 critical hypotheses validated)
No-Go Decision
Trigger: < 70% of critical criteria met with no clear path to fix
*Example: < 40% problem confirmation rate + no viable pivot path*