05: User Research & Validation Plan
Key Assumptions to Validate
Precision validation targets 18 high-risk assumptions across problem fit, solution desirability, and business viability. Prioritize top 10 for initial sprint.
| Category | Assumption | Risk | Method | Target Evidence |
|---|---|---|---|---|
| Problem | ||||
| Teams track 10+ third-party APIs per app | High | Interviews/surveys | 70%+ confirm ≥10 APIs | |
| Miss ≥1 breaking change quarterly | High | Interviews | 60%+ report prod incidents | |
| Changelogs scattered/missed 50%+ time | High | Surveys | Quotes + 50% frequency | |
| Manual checks take 4+ hrs/week | Med | Interviews | Avg 5 hrs documented | |
| Security/auth changes most feared | High | Survey ranking | Top 2/5 priorities | |
| Provider emails ignored 70%+ time | Med | Interviews | 80%+ "lost in inbox" | |
| Solution | ||||
| Auto-detection from package.json viable | High | Prototype test | 80%+ APIs detected accurately | |
| LLM change classification ≥85% accurate | Critical | Expert review | 85%+ true positives | |
| Slack alerts preferred channel | Med | Survey | 60%+ top choice | |
| Impact analysis saves 50% debug time | High | Wizard of Oz | Users confirm time savings | |
| Dashboard reduces check frequency 80% | Med | Usability test | NPS >40 on unified view | |
| GitHub integration essential for teams | High | Interviews | 70%+ "must-have" | |
| Business | ||||
| $49/mo acceptable for 50 APIs | Critical | Van Westendorp | 60%+ acceptable range | |
| Free tier → paid conversion 20% | High | Landing page | >10% trial signups | |
| CAC <$50 via dev communities | High | Ad tests | Proven <$50/lead | |
| Churn <10% post-validation | Med | Pre-orders | >90% retention intent | |
| DevOps decides purchase (not CTO) | Med | Interviews | 70%+ buyer persona match |
Customer Discovery Interview Guide (60-90 min)
Target: 25 interviews (15 primary: eng teams, 5 secondary: DevOps, 5 tertiary: founders). Recruit via LinkedIn ("API dependency pains? 30min chat + $50"), Reddit r/devops, Hacker News. Use Otter.ai for transcripts.
Part 1: Background (10 min)
- Tell me about your role and day-to-day API work.
- How many third-party APIs does your app/team use?
- Biggest infra challenges right now?
Part 2: Problem (20 min)
- Walk me through last API change incident (e.g., Stripe webhook).
- How often do you miss changes? Prod impact?
- What triggers manual checks (deploys, outages)?
- Feelings/time lost? Security fears?
- Current spend on monitoring/tools?
Part 3: Solutions (15 min)
- What tools for changelogs (RSS, Dependabot)? Likes/dislikes?
- Switched recently? Why? Switch triggers?
- Ideal monitoring tool must-haves?
Part 4: Our Solution (15 min)
- A tool auto-tracks API changes, alerts Slack, links to your code...
- Most valuable? Concerns (accuracy, scraping)?
- Price expectation ($49/mo team)? Approvers?
Part 5: Wrap (10 min)
- 1-10 pain score? Beta interest?
- Referrals?
Survey Designs
Screening Survey (Target: 300 responses via LinkedIn/Reddit polls)
- Role? [Eng lead/DevOps/Founder]
- APIs tracked? [<10 / 10-50 / 50+]
- Missed change in 6mo? [Yes/No + impact]
- Pain 1-10?
- Time/week manual checks?
- Budget for monitoring?
- Interview interest? ($50)
Validation Survey (Target: 150 from screen + ads)
- Current tools satisfaction (1-10)
- Rank pains: breaking/security/deprecations
- A/B headlines interest (1-10)
- Van Westendorp: Too cheap/expensive?
- Feature priority: alerts/dashboard/impact
- Demographics: team size, tech stack
Validation Experiments
Landing Page (Week 3)
- A/B Headlines: "Catch API Breaks Before Prod", "Unified API Changelog Alerts", "Auto-Track 50+ APIs w/ Slack Alerts"
- Waitlist + fake door ($49 CTA)
- $750 FB/LinkedIn ads (dev targeting)
- Success: 1k visits, 7% signup, <8% bounce
Prototype: Wizard of Oz (Recommended, Week 7)
- Google Form: Paste package.json → manual LLM scan → email report
- 10 users, $0 cost, measure NPS/accuracy
- Alt: Figma clickable dashboard ($300)
Fake Door/Pre-Order: Pricing tiers on LP → "Coming Q1" or Stripe pre-pay (refundable). Success: 12% click, 3% pay.
8-Week Timeline
Wk1-2: Interviews (20)
Surveys (300)
Problem Synthesis
Assumption Pivot
Wk3-4: LP A/B ($750 ads)
Validation Survey
Fake Door Test
Solution Feedback
Wk5-6: Pricing Interviews (15)
Van Westendorp
Pre-Orders (10)
WTP Pivot
Wk7-8: Wizard Oz (15 users), NPS, Go/No-Go
Go/No-Go Criteria
| Metric | Target | Actual | Pass? |
|---|---|---|---|
| Problem validation (interviews) | 75%+ confirm pains | ☐ | |
| LP signup rate | >7% | ☐ | |
| Price acceptance ($49) | 65%+ | ☐ | |
| Pre-orders | 10+ | ☐ | |
| WoZ NPS | >45 | ☐ |
Fail 2+ → Pivot or kill.
Research Synthesis Template
Problem: Top pains/quotes/unexpected/wrongs
Solution: Features cared/ignored/UX/integration
Pricing: Optimal/sensitivity/anchors/model
GTM: Channels/discovery/objections
Next: Share raw transcripts + dashboard in Notion for team review. Budget: $3k total (incentives/ads).