This week: why a powered experiment isn't automatically trustworthy, and a sharp debate on whether startup methodology has moved the needle at all (argued from both side). You'll also find 460 open roles, upcoming events. Let’s go!
Brought to you by
Convert—A/B tests & personalization for growth teams
Experimentation Jobs—Find your next role
🔎 Interesting things you might have missed
Powered ≠ Trustworthy
By Mårten Schultzberg (Spotify) — A powered experiment isn't automatically credible. If your MDE was optimistic, significant results will be inflated: the winner's curse. Ask whether the effect size is believable, not just whether you hit 80% power. READ
____
Glossary of A/B testing terms
By Convert — A good overview of the mosy important experimentation terminology, each entry written by a named practitioner. You can even submit your own favourite A/B testing term. LEARN
____
We have learned nothing about startup success
By Jerry Neumann — Startup survival rates are unchanged after 30 years of Lean Startup, customer development, and the Business Model Canvas. Once a method is widely adopted, it stops conferring competitive advantage: the Red Queen effect. A real science of entrepreneurship wouldn’t codify winning moves; it would generate new approaches faster than they become obsolete. READ
____
We haven’t learned nothing
By Jeroen Coelen — Neumann's data is flawed, too broad or too narrow, and most founders don't actually use the methods anyway. The real problem: entrepreneurship isn't a natural science. Founders are architects, not scientists, and the answer is design patterns, not physics. READ
____
Last week’s favourite:
Why CRO is so difficult
By Georgi Georgiev — Motivated users endure terrible UX to complete tasks. CRO only moves marginal users (~20%), putting a hard mathematical ceiling on what's achievable. Fix bugs first, stop enshitification, and reset expectations accordingly. READ
🚀 Job opportunities
Find 460 open roles on ExperimentationJobs.com. This week’s featured roles:
Experimentation Engineer (Front‑End) at Whitebread (Holborn, United Kingdom)
Product Manager – Data & Experimentation at Fox Television (Remote, USA)
Lead Insights & CRO at Travix (Amsterdam, Netherlands)
Principal, Data Scientist, Experimentation Sciences at Walmart (USA)
Senior Data Scientist, Campaigns at Bolt (Berlin or Tallinn)
📅 Upcoming events
A running list of upcoming events. Subscribe here. (👋= join me, 🎁= discount)
28 Apr: How are experimentation agencies adapting to AI in 2026? (online)
28 Apr: Berlin Experimentation Meetup #13 (Berlin, Germany)
28-29 Apr: Marketing Analytics Summit (Santa Barbara, United States)
1 May: Workshop on Experimentation for Decision-Making (New York, USA)
🎁6 May: Space Academy workshop (London, UK) use EXMIND12 for 12% off
🎁7 May: Space Academy by Experimentation Elite (London) use EXMIND12
12 May: Circus (London, UK)
20 May: Experimentation London #11 (London, UK)
22 May: TLC | NoHacks panel: Brand trust, AI, and what’s breaking with Wil Reynolds, Juliana Jackson, Sani Manić, and Kelly Wortham (online)
🔥26 May: Deadline entering entry for Experimentation Elite Awards
👋28 May: Optimization Meetup Utrecht (Utrecht, Netherlands)
🎁1 Jun: Accelerating Innovation with A/B Testing with Ronny Kohavi (online,
$1,999$1,749 USD)10 Jun: Best in Test Awards (online)
🎁15 Jun: Advanced Topics in Practical A/B Testing with Ronny Kohavi (online,
$1,199$999 USD)24 Sep: Experimentation Elite Awards (London, UK)
When you’re ready, here’s how I can help
If this newsletter sparked something and you want to talk it through, a few ways I help:
Scale experimentation
Strategy, setup, metrics, and ways of working for teams that want more impact.Careers and hiring
Support with your next role, or help finding and hiring strong experimenters.Quick sparring
A fresh outside perspective on your ideas, roadmap, or experimentation setup.
Interested?
Just hit reply and tell me what you’re thinking about.
Have a great week — and keep experimenting.
Thanks, Kevin
Some great references this week. Thanks!