This week: Spotify's experimentation bootcamp is now free, and research on how well humans actually predict A/B test results (spoiler: not well). Also, voting is open for two experimentation awards, including one where I've been nominated. I'd appreciate your vote.
Brought to you by
Convert—A/B tests & personalization for growth teams
Experimentation Jobs—Find your next role
🔎 Interesting things you might have missed
Experimentation Thought Leadership Awards 2026
By Kameleoon — Voting is open until May 15th across five categories: Thought Leader of the Year, UX Expert, Cultural & Community Architect, Data & Analytics Innovator, and AI Evangelist. Winners announced May 27th. VOTE
____
Submit your case: Experimentation Culture Awards 2026
By Ton Wesseling — Deadline May 31st. Four updated categories this year: Decision Intelligence, Learning Organization, AI in Experimentation, and Experimentation at Scale. Submit your own case or nominate someone else. VOTE
____
Statsig joins Amplitude
Amplitude is taking on Statsig's brand and customers, combining product analytics with experimentation and feature management. Statsig's team moves to OpenAI; both platforms continue operating during the transition. This moves makes sense. The Statsig product (and their customers) deserves more love than OpenAI can give it. READ
____
Spotify’s experimentation bootcamp is now free
By Mårten Schultzberg — 11 courses, 72 lessons built on a decade of running A/B tests at Spotify. Covers everything from hypothesis testing to sequential testing. Free, self-paced, no account needed. STUDY
____
How well can humans predict A/B test effects?
By Jakub Linowski — 1,391 guesses across 13 tests: typical error was 2.7pp, with a consistent positive bias. Crowds reduce noise but not bias. Experience barely helps. Use estimates for prioritisation, not as a substitute for testing. READ
____
Last week’s favourite:
Powered ≠ Trustworthy
By Mårten Schultzberg (Spotify) — A powered experiment isn't automatically credible. If your MDE was optimistic, significant results will be inflated: the winner's curse. Ask whether the effect size is believable, not just whether you hit 80% power. READ
🚀 Job opportunities
Find 605 open roles on ExperimentationJobs.com. This week’s featured roles:
Growth Marketing Manager, CRO & Lifecycle at Fieldwire (USA)
Senior Software Engineer – Experiment Platform at Moloco (Seattle, USA)
Product Manager Experimentation Platform at Ebay (Toronto, Canada)
Principal Product Manager – Experimentation at GoDaddy (USA)
Data Scientist – Experimentation & Causal Inference at Wizeline (Mexico)
📅 Upcoming events
A running list of upcoming events. Subscribe here. (👋= join me, 🎁= discount)
12 May: Circus (London, UK)
20 May: Experimentation London #11 (London, UK)
22 May: TLC | NoHacks panel: Brand trust, AI, and what’s breaking with Wil Reynolds, Juliana Jackson, Sani Manić, and Kelly Wortham (online)
🔥26 May: Deadline entering entry for Experimentation Elite Awards
👋28 May: Optimization Meetup Utrecht (Utrecht, Netherlands)
🎁1 Jun: Accelerating Innovation with A/B Testing with Ronny Kohavi (online,
$1,999$1,749 USD)10 Jun: Best in Test Awards (online)
🎁15 Jun: Advanced Topics in Practical A/B Testing with Ronny Kohavi (online,
$1,199$999 USD)🆕16 Jul: Experimentation Culture Awards (online)
24 Sep: Experimentation Elite Awards (London, UK)
😃 Something that made me smile
When you’re ready, here’s how I can help
If this newsletter sparked something and you want to talk it through, a few ways I help:
Scale experimentation
Strategy, setup, metrics, and ways of working for teams that want more impact.Careers and hiring
Support with your next role, or help finding and hiring strong experimenters.Quick sparring
A fresh outside perspective on your ideas, roadmap, or experimentation setup.
Interested?
Just hit reply and tell me what you’re thinking about.
Have a great week — and keep experimenting.
Thanks, Kevin