Last week: I’ve been busy with hosting the experimentation conference at Booking. Will do a longer deepdive on all presentation later, but overall it was a great success. This Thursday I’ll give a talk at the Optimization Meetup Utrecht. There are still some tickets available, so I might see you there.
ps. I’m working on a new logo for Experimentation Jobs. Scroll to the bottom to see some designs I’m considering.
Brought to you by
Convert—A/B tests & personalization for growth teams
Experimentation Jobs—Find your next role
🔎 Interesting things you might have missed
The real ROI of experimentation
By Johan Rydberg (Spotify/Confidence) — Spotify wins 12% of experiments but learns from 64%. Counting winners misses the point. The real ROI is three things: shipped wins, prevented harm from rollbacks, and learning velocity. Measure those instead. READ
____
Causal inference is different in business
By Alejandro Alvarez Perez — Not every decision deserves the same rigour. Match your causal inference to the gravity of the decision: constructive decisions need speed, final decisions need precision. Three rules: start with the problem, skip causal inference if you don’t need it, and 80/20 your effort across what actually drives the decision. READ
____
Speak at the Women in Experimentation Summit 2026
You have until May 27 to apply to speak at this online summit which is scheduled on September 9–10. 70% female speakers, coaching provided, and speakers get paid. First-timers welcome. APPLY
____
Tips for buying a new A/B testing tool
Ruben de Boer created three short videos packed with tips on how to approach selecting a new A/B testing vendor. Take your time and make a good, solid decision. LEARN
____
Experimentation Elite Awards: nominations close soon
There are three categories: Individual, Team, and Campaign or Experiment. You can self-nominate or put someone forward. Don't overthink it, if the work is good, submit it. SUBMIT
____
The bear hunt
By Mitch — Gaming experiments to move metrics you know will go up isn’t experimentation; it’s Build-A-Bear. The real work is hunting bears that scare you, failing, and learning anyway. Short-term glory means long-term stagnation. READ
____
Last week’s favourite:
A review of experiments with synthetic users
By Jim Lewis & Jeff Sauro — 12 peer-reviewed papers, show one thing: synthetic users correlate with humans but fail on variance, subgroups, and depth. Use it for directional signals, not critical decisions. READ
🎧 Podcast of the week
Looking forward to get my hands on Eric Reis new book: Incorruptible. It explains why so many good companies drift away from their core and what leaders can do to prevent this from happening. Great stories by Eric on Lenny’s podcast.
If you like this episode, make sure to subscribe to the podcast. Or discover other podcasts via the Experimental Mind curated podcast feed.
🚀 Job opportunities
Find 794 open roles on ExperimentationJobs.com. This week’s featured roles:
Senior Front-end Developer (Experimentation) at Creative CX (London, United Kingdom)
Manager, Landing Page Optimization at Airwallex (San Francisco, USA)
Experimentation Chapter Lead at easyJet (London, United Kingdom)
Experimentation Lead, Sign-Up and Cancellation Optimization at Paramount (USA)
Analytics & CRO Consultant at Havas Group (Milan, Italy)
📅 Upcoming events
A running list of upcoming events. Subscribe here. (👋= join me, 🎁= discount)
🔥26 May: Deadline entering entry for Experimentation Elite Awards
👋28 May: Optimization Meetup Utrecht (Utrecht, Netherlands)
🎁1 Jun: Accelerating Innovation with A/B Testing with Ronny Kohavi (online,
$1,999$1,749 USD)👋10 Jun: Best in Test Awards (online)
🎁15 Jun: Advanced Topics in Practical A/B Testing with Ronny Kohavi (online,
$1,199$999 USD)🆕17 Jun: TLC Panel: Accessibility, AI Discoverability, & SEO with Shirley van Haalem, Alisa Scharf, and Sani Manic (online)
23 Jun: Causal Summit (online)
🔥26 Jun: Deadline shortlist submission Experimentation Heroes
👋16 Jul: Experimentation Culture Awards (online)
🆕9-10 Sep: Women in Experimentation Summit (online)
24 Sep: Experimentation Elite Awards (London, UK)
5 Nov: Experimentation Heroes (Amsterdam, Netherlands)
🎨 What do you think?
I’m working on new branding for Experimentation Jobs and would love your feedback. Have a look at these 6 design directions in greyscale.
Just hit reply and let me know your thoughts. Any feedback helps.
When you’re ready, here’s how I can help
If this newsletter sparked something and you want to talk it through, a few ways I help:
Scale experimentation
Strategy, setup, metrics, and ways of working for teams that want more impact.Careers and hiring
Support with your next role, or help finding and hiring strong experimenters.Quick sparring
A fresh outside perspective on your ideas, roadmap, or experimentation setup.
Interested?
Just hit reply and tell me what you’re thinking about.
Have a great week — and keep experimenting.
Thanks, Kevin