Experimental Mind: Kelly Wortham, Eric Ries, Measuring Discovery...
Your overview of interesting reads, events and jobs for the experimental mind: #194
I have one goal with this newsletter: simply create the best overview of what is happening in the experimentation space. If you find one thing valuable in this newsletter, please share it with a friend or colleague. Thank you.
➡️ Dec 4 a new cohort starts: Accelerating Innovation with AB Testing
Ronny Kohavi teaches the best class on A/B testing concepts. Even if you have been following his work for years, attending this class will bring many new insights. It offers a clear explanation of complex topics and the opportunity to interact with many peers. Enroll today and save 500 dollar (or join the shorter Essentials class)
💬 Kelly Wortham: My Experimentation Career Journey
Today’s career interview is with Kelly Wortham, founder of the Test & Learn Community, and the newly launched ForwardDigital.org consulting firm.
Keep engaging. Connect with your peers. Ask (and answer) questions. That’s how we grow and learn. The digital communities of analytics and experimentation are truly wonderful.
Read the full interview with Kelly
🔎 Interesting things you might have missed
Podcast interview with Eric Ries (creator of the Lean Startup methodology)
Eric Ries acknowledges that some people dislike the MVP concept, but he argues that this criticism often stems from misunderstandings about what an MVP actually is. He emphasizes that an MVP is not about building a low-quality or barebones product; rather, it is about testing a hypothesis as efficiently as possible. The key is to build the minimum set of features that allows you to (in)validate your hypothesis. LISTENThe promise of experimentation in development: what works and what doesn’t
The Heads of Experimentation from the United Nations Development Program (UNDP) share what they have learned “on the ground in such diverse contexts from Guatemala to Fiji to Zambia. We also reflect on the role experimentation plays in accelerating SDG progress and scaling solutions.” READHow do you measure product discovery?
Jeff Gothelf argues that product discovery should be measured to justify investments and ensure it is happening, deployed well, impacting work, and improving products. He suggests starting with basic activity metrics, then measuring insights and impact on products, and finally looking at customer behavior changes. READHow to choose the right metrics for your experimentation platform
”… there isn’t a single metric that can be used to measure the effectiveness of an Experimentation Platform perpetually. Instead, we recommend a multi-metric approach where depending on the maturity of your experimentation program and the business priorities, you can choose which metric to optimize for on a given period.” READThe true value of advertising via ad incrementality
Instacart uses randomized control trials to accurately measure ad incrementality, randomly assigning users to see ads or be held out to quantify the incremental value of advertisements on their platform. This allows Instacart to optimize ad effectiveness for individual advertisers and the overall system. READAddressing the challenges of SRM
“At DoorDash, we have pursued several approaches to reduce the platform’s rate of SRM, including innovating on methodology for how we diagnose SRM, improving our real-time observability and alerting, and focusing on education and awareness.“ READ
Bhavik Patel on Are pricing experiments unethical?
David Nelson, Tom Storr and Phillip Black started The Experimentation Group
Now live: Eppo Reports
New book release: Applied Causal Inference
[REMINDER] State of Experimentation Survey
Sven Schmit also shared a survey with me. It is collaboration between academics and industry to gain insight into the current state of experimentation across sectors, with results presented at CODE@MIT and made available to the experimentation community while protecting respondent anonymity. Please participate and help shape the future of experimentation. PARTICIPATE
🚀 Job opportunities
Looking for a new challenge in experimentation? Find 100+ experimentation related jobs on ExperimentationJobs.com. These jobs are from all over the world, on-site, fully remote or hybrid. Take a look and start pivoting your career.
This week's featured roles:
🆕Data Analist/Scientist at Online Dialogue (Utrecht, Netherlands)
CRO Account Manager at Croud (Shrewsbury, United Kingdom)
🆕Data & Product Analyst at Tagvenue (Poland)
🆕International Growth Lead at Eventbrite (London, United Kingdom)
🆕Engineering Manager | Experimentation at DeepL (Köln, Germany)
Senior CRO Analist at ClickValue (Amsterdam, Netherlands)
Are you hiring? Submit here
📅 Upcoming events
I will be attending Conversion Hotel next month. Will I see you there?
31 Oct: DDMA Experimentation Heroes (Amsterdam, the Netherlands)
🎁6 Nov: Innovation & AB Testing - the Essentials (virtual,
$999$899 USD)10 Nov: Conference on Digital Experimentation (Boston, USA)
17-20 Nov: Society for Judgement and Decision Making (virtual)
17-19 Nov: Conversion Hotel (Texel, Netherlands)
🎁4 Dec: Accelerating Innovation with A/B Testing with Ronny Kohavi (virtual,
$1,999$1,499 USD)7 Dec: Experimentation Elite (London)
See all experimentation related conferences & events 2023.
💬 Quote of the week
“As a company grows, everything needs to scale, including the size of your failed experiments. If the size of your failures isn’t growing, you’re not going to be inventing at a size that can actually move the needle. Amazon will be experimenting at the right scale for a company of our size if we occasionally have multi-billion dollar failures.” — Jeff Bezos, CEO Amazon
😅 This made me smile
👍 Thanks for reading
If you are enjoying the Experimental Mind newsletter please share this email with a friend or colleague. Looking for inspiration how to best do that? Take a look at what others are saying. Have a great week — and keep experimenting.