Experimental Mind #248

Your weekly overview of interesting reads, events and jobs for the experimental mind.

Dec 16, 2024

Hi folks! Two more weeks and 2024 is over. In this edition articles about: targeting/triggering, risk mitigation, confidence intervals, and drafting experiment plans with ChatGPT.

➡️ How Targeting and Triggering Work Together
Strategically combining targeting and triggering in your A/B tests produces reliable results by focusing tests on the right audience at the right time.
Link to article from Convert

Thanks to Convert, Sitespect, Zoho PageSense and Eppo for their support.

❔Question of the week

🔎 Interesting things you might have missed

Want your company to get better at experimentation?
Online experimentation has driven innovation in tech companies and is now accessible to most organizations, yet many limit its use due to reliance on data scientists. To scale experimentation we should be empowering broader teams (product, marketing, and sales) to independently design and run tests. Link

____

Think of risk mitigation as a ladder

Level 0: No experiment (no risk mitigation)
Level 1: Lack of evidence for negative impact (avoiding terrible decisions)
Level 2: Evidence for limited negative impact (bounding negative impact)
Level 3: Evidence for improvement (proving positive impact)

From Spotify’s blog: Experiments with Smaller Samples

____

What a confidence is and isn’t
Can you calculate a confidence interval and then claim that, with a probability of 95%, the value of interest lies within this interval? Well, actually not. But does this matter? This article got me thinking (again) about this not-so-intuitive concept. Link

____

[Dutch] A/B testen kritisch onder de loep
NS International heeft samen met de UVA een kritische blik geworpen op hun A/B-testen. Deze samenwerking met experts uit de academische wereld, leidde tot diepere inzichten en betrouwbaardere resultaten. Mooi voorbeeld dat andere organisaties kan inspireren om de betrouwbaarheid van hun eigen experimenten te verhogen. Link

____

Draft experiment plans with these ChatGPT prompts
David Bland has created a package of ChatGPT prompts to help you draft experiment plans for testing ideas. The prompts guides you through extracting and prioritizing assumptions, mapping them, and creating a 90-day experiment plan. Link

____

Last week’s most clicked item:

Measuring Product Impact Without A/B Testing
How Discord used the synthetic control method to measure the impact of their voice messages feature. Synthetic controls are a powerful addition to your causal inference toolkit when traditional A/B tests aren't feasible—due to logistical or ethical constraints. Link