Experimental Mind #234

Your weekly overview of interesting reads, events and jobs for the experimental mind.

Sep 09, 2024

Hi folks!

In today’s edition:

8 reads you might have missed
Podcast of the week: Analytics Power Hour with Matt Gershoff
Stewart Ehoff reflecting on his career in experimentation
5 fresh job opportunities
10 events happening in the next 60 days
Something that made me smile
Your feedback

➡️ How to Fix Sample Ratio Mismatch in Your A/B Tests
Sample ratio mismatch is a common issue in A/B testing. This blog explains what it is, how to identify it in your experiments, and which steps you can take to address it.
Link to article from SiteSpect

Thanks to Convert, Sitespect and Zoho PageSense for their support.

🔎 Interesting things you might have missed

Growth Strategy Online Course
Latest CXL course where Daphne Tideman will learn you about growth and its role within an organization, how to find product-market fit, and how to identify and implement your north star metric. You'll also discover how to create and work with your own growth model. Finally, you'll learn to identify and prioritize growth channels effectively.
Signup for the course

____

From Agile to Radical: business model
Jan Bosch asks himself why SaaS companies run tons of A/B tests and car companies run so few, if any? The answer: it’s the business model.
Link to article

____

What is A/B testing and why is it important?
Nice overview of the importance of A/B testing by Yuzheng Sun, Data Scientist at Statsig. Conclusion: humans are bad at attributions and are subject to lots of biases + humans are bad at predicting the outcome of their ideas.
Link to article

____

CUPED for switchback tests
Switchback experiments tend to need long run times. The experimentation team from Bolt developed a two-stage method of variance reduction using CUPED for switchback tests, which solves this problem.
Link to article

____

It doesn't matter how you define MVP
Different people have different ideas of what a Minimal Viable Product means. Next time you and your team are planning launching a MVP, first align on what you want to achieve.
Link to article

____

The frequentist vs Bayesian split in online experimentation
Georgi Georgiev reflects the frequentist vs bayesian split in online experimentation and puts in context of recent developments in the experimentation space.
Link to article

____

No Hacks Season 2 Promo
On September 18th Slobodan Manić will be back with season two of the No Hacks podcast. In this neat little promo you can see the lineup.

____

Last week’s most clicked item:

How DoorDash is pushing experimentation boundaries with interleaving designs
Interleaving can boost the sensitivity of experiments. And allow for quicker and more accurate detection of changes than in traditional A/B testing. The DoorDash team explains how this works and how they recommend others to implement it.
Link to article

🎧 Podcast of the week

This week’s recommendation is one from the Analytics Power Hour podcast where Matt Gershoff makes the case for adopting a just in time, just enough data mindset.

If you like this episode, make sure to subscribe to the Analytics Power Hour podcast. Or discover other podcasts via the Experimental Mind curated podcast feed.

💬 My Experimentation Career Journey

Today’s interview is with Stewart Ehoff, Head of Experimentation at RS Group.

The best thing you can demonstrate when interviewing or searching for new roles is curiosity. If you are inately curious, and have a thirst to learn, the hard skills that can seem complex or daunting like statistics can be taught and learned over time.
— Stewart Ehoff

Read the full interview with Stewart

🚀 Job opportunities

Find 100+ open roles on ExperimentationJobs.com. This week’s featured roles:

Senior Strategy Consultant (Experimentation) at Creative CX (UK)
Manager, Web Experimentation at Hootsuite (USA)
Website Optimization Marketing Manager at Globalization Partners (USA)
Conversion Rate Optimization (CRO) Manager at Health E-Commerce (USA)
Growth CRO Manager at Yotpo (Tel Aviv, Israel)

📅 Upcoming events

A running list of upcoming events.

18-19 Sep: Nonprofit Innovation & Optimization (Indianapolis, USA)
21 Sep: MeasureCamp (London, UK)
23 Sep: Accelerating Innovation with A/B Testing (virtual)
$500 discount for Experimental Mind readers
24-26 Sep: ConvEx (virtual)
9 Oct: Advanced Topics in Practical A/B Testing (Austin, USA)
10-11 Oct: Experimentation Live + Unite (Austin, USA)
17 Oct: Conversion Jam (Stockholm, Sweden)
18 Oct: Monetate Elevate (London, UK)
18-19 Oct: CODE@MIT (Boston, USA)
🆕7 Nov: DDMA Experimentation Heroes (Amsterdam, NL)

Check out more experimentation conferences & events in 2024

💬 Quote to think about

"I don’t know how many A/B tests we’ve run at this point. It’s probably more than 10,000. We’ve tested our way into getting more people to pay us, into getting more people to use Duolingo, more people to recommend it to their friends, etcetera. Without having run those, we would not be a successful company. I’m certain of that."

— Luis von Ahn, CEO of Duolingo

(h/t Carlos Trujillo)