Experimental Mind #220
Your weekly overview of interesting reads, events and jobs for the experimental mind.
Hi folks!
Did you know that the t-test originated in the Guinness brewery? You can read it by following the links in this newsletter. Including other reads you might have missed, a career interview with Stefanie Grimmling and something that made me smile.
Let’s go.
➡️ Best A/B testing tools (2024): Alternatives to Google Optimize
List of popular A/B testing tools with common features highlights.
Link to the article from Zoho PageSense
Thanks to Convert, Sitespect, Exatom and Zoho PageSense for their support.
🔎 Interesting things you might have missed
The Cult of Stat Sig
Eppo’s Giorgio Martini explains how the cult around statsig was born, how it can harm decision-making and how we can excape from it.
Link to article
____
How the Guinness Brewery Invented the Most Important Statistical Method
”It was the birthplace of the t-test, one of the most important statistical techniques in all of science. When scientists declare their findings “statistically significant,” they very often use a t-test to make that determination. How does this work, and why did it originate in beer brewing, of all places?”
Link to article
____
Relevant and non-relevant users in A/B-testing
Lars Hirsch states that it is common to make errors around identifing the relevant set of participants in the experiment: users who were eligible to be impacted by the experiment.
Link to article
____
A/B Testing Gets an Upgrade for the Digital Age
Researchers from Stanford Graduate School of Business are exploring new ways of doing experimentation.
Link to article
____
[Paper] False Positives in A/B Tests
Ronny Kohavi and Nanyu Chen have their paper accepted to KDD 2024. You can still give feedback to this final draft version until June 12.
Link to paper
____
New course: Accelerating Innovation with A/B Testing
BTW: Ronny Kohavi starts a new course on June 24.
Enroll (and get 500 dollar off)
____
Experimenation Events Budget Guide
Kelly Wortham from TLC put this resource together to help you understand how much bduget you would need to attend popular events in our industry.
Link to overview
____
Split gets acuired by Harness
Split is joining Harness, an end-to-end platform for software delivery.
Link to article
💬 My Experimentation Career Journey
Today’s interview is with Stefanie Grimmling. Stefanie is a member of the Management Team & Senior Managing Consultant at konversionsKRAFT. And she will be presenting at the growth marketing SUMMIT in Frankfurt am Main, later this month.
It’s important to be clear about what you believe in and how you think about trying new things. You should have a strong set of ideas about how to focus on users and how to experiment well, in addition to knowing how to do the experiments themselves. — Stefanie Grimmling
Read the full interview with Stefanie
🚀 Job opportunities
Find 100+ open roles on ExperimentationJobs.com. This week’s featured roles:
CRO Specialist at Sofatutor (Germany/Global)
Engineering Manager – Experimentation at LaunchDarkly (USA)
Experimentation Analyst at The Financial Times (London, United Kingdom)
CRO Data Analyst at Hostinger (Lithuania)
CRO Lead at CommerceCore (Vilnius, Lithuania)
Experimentation Jobs has been shortlisted for an Experimentation Elite Award + nominated for the Experimentation Culture Award.🎉
📅 Upcoming events
A running list of upcoming events.
4 Jun: CRO Belgium (Ghent, Belgium)
4-7 Jun: Marketing Analytics Summit (Phoenix, USA)
6 Jun: Women in Experimentation Summit (virtual)
6 Jun: Digital Marketing Live (Amsterdam, Netherlands)
6-7 Jun: NeuroPsychoEconomics (Milan, Italy)
8 Jun: MeasureCamp (Copenhagen, Denmark)
🎁11-12 Jun: Experimentation Elite (Birmingham, United Kingdom)
Use code ’EXPJOBS10‘ to get 10% off12 Jun: A/B Test Talks with Statsig (Berlin, Germany)
15 Jun: MeasureCamp (Paris, France)
🎁19 Jun: Growth Marketing SUMMIT (Frankfurt am Main)
🎁 24 Jun: Accelerating Innovation with A/B Testing with Ronny Kohavi (virtual,
$1,999$1,499 USD)
Full list of experimentation conferences & events in 2024
👋 On June 19th I’ll be attending growth marketing SUMMIT, Europe's premier real life conference for digital growth, experimentation and optimization in Frankfurt, Germany. Use code ’experimental-mind
‘ to get 10% off and let’s meet there.
💬 Quote to think about
“Everyone's running experiments, but only some of them have control groups and randomization.” — Sean Taylor
😃 Something that made me smile
![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1983dce2-c481-4709-83de-e8ce34058c34_1427x1469.png)
📣 Testimonials
If you have made it to here and you are enjoying this newsletter, I would love to read a short testimonial from you.
Read what others are saying and add your own.
Have a great week — and keep experimenting.
Thanks, Kevin
The story that Guiness prohibited employees from publishing papers, and thus William Gosset published it under the name Student, is well known.
Imagine if they did allow it, but asked him to name it the Guinness t-test. We would be giving them free publicity for over a century