Validation Experiments & Hypotheses
Hypothesis #1: Problem Existence 🔴 Critical
We believe that engineering teams at startups and mid-size companies
Will actively seek a solution to track third-party API changes
If they experience production incidents due to missed API changes
We will know this is true when we see 70%+ of surveyed developers confirm this is a top-3 pain point AND 10%+ landing page signup rate
Risk Level: 🔴 Critical (product fails if wrong)
Current Evidence:
- Supporting: Forum discussions, search volume, competitor traction
- Contradicting: None identified
- Gaps: No direct user interviews yet
Experiment Design:
- Method: Customer discovery interviews + landing page test
- Sample Size: 20 interviews, 1,000 landing page visitors
- Duration: 2 weeks
- Cost: $500 (ads) + 20 hours (interviews)
Success Metrics:
| Metric | Fail | Minimum | Success | Home Run |
|---|---|---|---|---|
| Problem confirmation rate | <40% | 40-60% | 60-80% | >80% |
| Landing page signup | <2% | 2-5% | 5-10% | >10% |
Next Steps if Validated: Proceed to solution validation
Next Steps if Invalidated: Pivot to adjacent problem or exit
Experiment Catalog
| Experiment | Hypothesis Tested | Method | Metrics | Timeline | Cost |
|---|---|---|---|---|---|
| Problem Discovery Interviews | #1 (Problem Existence) | Semi-structured interviews | Problem confirmation rate | 2 weeks | $1,000-$1,500 |
| Landing Page Smoke Test | #1 (Problem Existence) | Landing page with waitlist signup | Signup rate | 2 weeks | $500-$1,000 |
| Wizard of Oz MVP | #2 (Solution Fit) | Manual service delivery | User satisfaction, NPS | 4 weeks | Time only |
Experiment Prioritization Matrix
| Experiment | Hypothesis | Impact | Effort | Risk if Skipped | Priority |
|---|---|---|---|---|---|
| Discovery Interviews | #1 | 🔴 Critical | Medium | Fail | 1 |
| Landing Page Test | #1, #2 | 🔴 Critical | Low | Fail | 2 |
| Wizard of Oz MVP | #2, #3 | 🔴 Critical | High | Fail | 3 |
8-Week Validation Sprint
Week 1-2: Problem Validation
- Launch landing page
- Recruit interview participants
- Conduct interviews
- Run landing page ads
Week 3-4: Solution Validation
- Analyze interview data
- Build Wizard of Oz process
- Deliver to 10 users
Week 5-6: Pricing & Willingness to Pay
- Run pricing survey
- Collect post-delivery payments
- Analyze pricing data
Week 7-8: Synthesis & Decision
- Compile all experiment results
- Make Go/No-Go decision
- Plan Phase 2 (if Go)
Go/No-Go Criteria
| Category | Metric | Must Achieve | Nice-to-Have |
|---|---|---|---|
| Problem | Interview confirmation | 60%+ | 80%+ |
| Problem | Landing page signup | 5%+ | 10%+ |
| Solution | Prototype satisfaction | 7/10+ | 8.5/10+ |
Pivot Triggers & Contingency Plans
Trigger #1: Problem Doesn't Exist
Signal: <40% of users confirm problem
Action: Interview users about their actual top problems, identify adjacent pain points
Pivot Options: Different problem in same audience, same problem in different audience
Trigger #2: Solution Doesn't Resonate
Signal: <50% satisfaction with prototype
Action: Deep-dive on what's missing, what's confusing, what's not valuable
Pivot Options: Simplify scope, change format, add human touch