Experimental Mind #251

Your weekly overview of interesting reads, events and jobs for the experimental mind.

Jan 20, 2025

Hi folks!

In today’s edition I can finally share another career interview. Nils Stotz reflect on his career and his recent move to Zalando.

And of course other nice reads, a podcast, fresh jobs, events and a short video about small data. Enjoy!

➡️ 4 Elements to Consider When Interpreting A/B Test Results
Do you have a clear approach for managing A/B test results? Without it you might not get of the benefits from your experimentation program. There are four essential elements for evaluating A/B test results.
Link to article from Sitespect

Thanks to Convert, Sitespect and Eppo for their support.

🔎 Interesting things you might have missed

New experimentation and causal inference tech blog
Alejandro Alvarez Pérez, lead Data Scientist at Adevinta, launched his experimentation and causal inference tech blog. The first post is about mastering the Poisson distribution: Intuition and Foundations. Link

____

Reforge has acquired Monterey AI
Product teams must understand the customer, but this is increasingly difficult as qualitative data grows fragmented across tools owned by various functions. Reforge is now trying to solve this with the acquisition of Monterey AI. Link

____

Results throwdown A/B Patterns reproducibility project
Remember the throwdown for the square vs rounded corners study, as part of the A/B Patterns Reproducibility Project? Gilbert Tjia predicted 0.58% and was the closest to the actual estimate of 0.56% (not significant). Link

____

How to think about Bets, Success Metrics, and Roadmapping
John Cutler is organising a virtual session on Jan 21st where he shares his ideas around thinking in bets, success metrics, and roadmapping. Register

____

Boost your career with an experimental mindset
Looking to take your career to the next step? Erin Shrimpton put together a short course on LinkedIn that looks exactly right for people reading this newsletter. Link

____

Last week’s most clicked item:

The Last Strategy Framework You'll Ever Need
Do think strategy is difficult? John Cutler bring some clarity by bringing different strategy frameworks back to their core. Link

🎧 Podcast of the week

In this episode Kristina Gibson, former Director of Product at Booking.com, explores when to rely on A/B tests compared to taking bigger strategic bets.

If you like this episode, make sure to subscribe to the podcast. Or discover other podcasts via the Experimental Mind curated podcast feed.

💬 My Experimentation Career Journey

Today’s interview is with Nils Stotz, leading the experimentation platform product team at Zalando.

Experimentation needs to be implemented in organizations through a well-thought change management process. I think this will always be the case, even in 10 years— Nils Stotz

Read the full interview with Nils