Experimental Mind #255

Your weekly overview of interesting reads, events and jobs for the experimental mind.

Feb 17, 2025

Hi folks! Thank you for reading. If something in this email triggers or inspires you, let me know. Just hit reply.

➡️ WIE x Convert Summit 2025
Want more public speaking experience with as least amount of stress as possible? Apply for a speaker slot for the second Women In Experimentation and Convert.com online summit. Do it before March 1st. Event is on July 9th.
Apply here supported by Convert

Thanks to Convert and Sitespect for their support.

🔎 Interesting things you might have missed

Deliveroo experimentation principles
Experimentation is central to innovation at Deliveroo, enabling data-driven decision-making, risk mitigation, and continuous product refinement through a mature and scalable experimentation platform. Beyond technical advancements, a strong experimentation culture ensures impactful testing by embedding shared principles that democratise experimentation across the organisation. Deliveroo has four. Link

____

The Duolingo Handbook
Duolingo released their culture handbook. With at the center five principles: Take the Long View, Raise the Bar, Ship It!, Show Don’t Tell and Make It Fun. This is a company that values experimenation.

Duolingo is the sum of thousands of experiments. The faster we can run experiments—successful or not—the faster we can improve the app and advance our mission. Over time, each of these changes builds on one another, creating a cycle of compounding growth.

Download

____

Directory of Experimentation & CRO Agencies
Craig Sullivan is building the first directory of CRO and experimentation agencies. You can help in making this resource better. Link

____

Instead of testing for SRM correct for it
It is important you test your experiment for potential sample ratio mismatch (SRM). But getting a SRM warning is frustrating. It tells you something might be off and you will need to investigate. Zach Flynn suggests instead of only testing for SRM, you can correct it. Link

____

Outdated distinction: product management vs R&D
Jan Bosch pushes for cross-functional teams that experiment quickly, using iterative development, A/B testing, and reinforcement learning, to build what truly delivers value while keeping products high-quality and adaptable. Link

____

Measuring the Intangible: Setting OKRs for Cultural and Behavioral Change
Jeff Gothelf explains that even goals related to cultural shifts or team dynamics can be measured by tracking behavioral changes. Instead of relying only on surveys, organisations should identify observable actions, such as increased collaboration or customer engagement, to assess progress effectively. Link

____

Last week’s most clicked item:

Use Decision Theory to choose significance levels for experiments
Zach Flynn suggests to use decision theory, specifically Expected Utility Maximisation, to set significance levels based on how much risk we're willing to tolerate when deciding whether to act on experimental results. Link

🎧 Podcast of the week

Melanie Kyrklund from Specsavers about their transistion from centralised experimentation to a Centre of Expertise around exeperimentation.

If you like this episode, make sure to subscribe to the podcast. Or discover other podcasts via the Experimental Mind curated podcast feed.

🚀 Job opportunities

Find 100+ open roles on ExperimentationJobs.com. This week’s featured roles:

CRO Lead at Fella Health (Europe, Latam and Asia)
Senior Product Director, Data, Martech & Experimentation at Viator (Multiple locations, Europe)
Senior Data Scientist – Experimentation at Lowe’s (Bengaluru, India)
Data Scientist – Insights (Organic Growth) at Preply (Barcelona, Spain)
Senior Conversion Rate Optimization (CRO) Manager at Playermaker (Tel Aviv, Israel)

📅 Upcoming events

A running list of upcoming events.

🆕20 Feb: A/B Testing Myths with Ronny Kohavi (virtual, 45 mins, free)
🎁3 Mar: Accelerating Innovation with AB Testing (virtual, get $500 off)
🎁26-28 Feb: Experimentation Island (USA, get $500 off)
13 Mar: DiDo #47: Collaboration in product teams (Utrecht, Netherlands)
🎁17 Mar: Advanced Topics in Practical A/B Testing (virtual, get $100 off)

🎁Also, don't miss GMS 2025 on June 18, 2025, in Frankfurt! As Europe's #1 growth conference, it's the perfect platform to learn, network, and grow. Use my promo code KEVIN100 to get €100 off your ticket and be part of this event! For more details, visit the official website: www.growthmindedsuperheroes.com

💬 Quote to think about

“If an outcome itself isn’t directly measurable, such as an attitude, it will inevitably lead to observable behaviors that can be measured.” — Jeff Gothelf