Experimental Mind #249
Your weekly overview of interesting reads, events and jobs for the experimental mind.
Hi folks! Last edition before many enjoy Christmas. Enjoy and see you next week with edition number 250.
➡️ The Bet Test: Spotting Problems in Bayesian A/B Test Analysis
Despite the fact that most experimentation practitioners have a healthy skepticism of large lifts, it is unfortunately common practice to use Bayesian A/B testing tools that don’t share that healthy skepticism.
Link to article from Eppo
Thanks to Convert, Sitespect, Zoho PageSense and Eppo for their support.
🔎 Interesting things you might have missed
Can you predict the outcome of actual live experiments
A couple of months ago Jakub Linowski, Lukas Vermeer, and Ron Kohavi started the A/B Patterns Reproducibility project and the first two experiments are live. You can try to predict the outcome and posisbly name a charity related to experimentation that will be awarded $1,000. Link
____
Craigslist of CRO agencies
Craig Sullivan is collecting a long list of CRO agencies (134 and counting). h/t to Alun Lucas for calling it the new ‘Craigslist’. Link
____
Our obsession with the word ‘convert’
The list contains many agencies with ‘convert/conversion in it’s name. It made me wonder: is it time for the CRO industry to rethink its obsession with the word 'convert'? Link
____
How to write great problem statements
Nailing down the right problem, makes solving for it easier. Jeroen Coolen shares some examples and his approach. Link
____
Shaping experimentation cultures at Vista
Jean de Bressy interviewed Lukas Vermeer about experimentation at Vista. Link
____
Scaling CUPED at Meta
There are many articles about the stats around CUPED, a technique to reduce variance, this one goes more into the challenges of scaling this to a large organization of software developers and decision makers: Meta. Link
____
Last week’s most clicked item:
Draft experiment plans with these ChatGPT prompts
David Bland has created a package of ChatGPT prompts to help you draft experiment plans for testing ideas. The prompts guides you through extracting and prioritizing assumptions, mapping them, and creating a 90-day experiment plan. Link
🎧 Podcast of the week
Ryan and Che from Eppo discuss three trends in experimentation.
🚀 Job opportunities
Find 100+ open roles on ExperimentationJobs.com. This week’s featured roles:
Senior Technical Consultant (Experimentation) at Creative CX (remote, UK)
Lead Product Designer (Growth/Experimentation) at Cleo (London, UK)
Product Manager, Experiments at Haus (USA)
Digital Experimentation Analytics Manager at Salesforce (USA)
CRO Executive at THG (Manchester, United Kingdom)
📅 Upcoming events
A running list of upcoming events.
15 Jan: Experimentation North (Leeds, UK)
21 Jan: CRO Belgium meetup: Experimentation culture (Ghent, Belgium)
🆕22 Jan: Experimentation London Meetup #3 (London, UK)
30 Jan: EXL Meetup (Sao Paulo, Brazil)
30 Jan: EXL Meetup (Rio, Brazil)
27-31 Jan: SUPERWEEK (Hungary)
26-28 Feb: Experimentation Island (St. Simons Island, Georgia, USA)
I’m working on an overview of all experimentation conferences & events in 2025.
😃 Something that made me smile
📣 How was this edition?
Let me know:
If you have made it to here and you are enjoying this newsletter, I would love to read a short testimonial from you.
Read what others are saying and add your own.
Have a great week — and keep experimenting.
Thanks, Kevin