The missing piece
Your weekly overview of interesting reads, events and jobs for the experimental mind.
This week: a deeper look at effect distributions and why your win rate alone tells you almost nothing — plus AI in A/B tools, a quasi-experiment from BBC Studios, and the validation mindset trap. You'll also find 1k+ open roles, upcoming events, and a new cartoon that made me smile.
Brought to you by
Convert—A/B tests & personalization for growth teams
Experimentation Jobs—Where experimenters find their next role
🔎 Interesting things you might have missed
Missing piece in experimentation programs: the effect distribution
When you run an A/B test, you never observe the true effect of your change. You observe a measured effect, which is your true effect plus random noise. That noise matters a lot.
If your true effect is small (let’s say a 0.5% lift), but your test is underpowered, your measured result will bounce around wildly. Sometimes it’ll look like +3%, sometimes -1%. If you only ship the ones that reach statistical significance, you’re systematically selecting the lucky high estimates. That’s the winner’s curse: the effects you ship are inflated versions of reality.
The 5% problem
At a 95% confidence threshold, 5% of tests on completely ineffective changes will still look significant by chance. So if your program produces exactly 5% winners, and you haven’t thought about the effect distribution, you can’t distinguish between “we have a few real wins” and “all our ideas are worthless and we’re just seeing noise.” Both look identical in your results dashboard.
What the effect distribution tells you
If you zoom out and look at the shape of true effects across many experiments, you get something much more useful than individual p-values:
You can see whether your ideas are generating any real signal at all, or just noise. You can estimate what a realistic lift looks like for your product, which should inform your power calculations, not your historical observed effects (which are inflated). And you can compare effect distributions across different types of experiments, checkout flow tests vs. recommendation tests, to see where the real upside is concentrated, and direct resources accordingly.
The resource allocation angle
But most teams never get to this. If you estimate separate effect distributions for different experiment categories, you can see which areas have high variance in true effects (meaning real upside potential) versus areas where everything clusters near zero (meaning you’re probably optimising something already well-optimised). That’s a principled way to decide where to invest your experimentation capacity. No gut feel, no HiPPO decision-making.
The bottom line
Most experimentation programs treat each test as an isolated decision. Win or lose, ship or don’t ship, move on. The effect distribution reframes this: your experiments are a program, and the program has statistical properties you can measure and improve. A 5% win rate with small, noisy effects is a very different program than a 5% win rate with large, clean effects, and you can’t tell the difference without thinking at the program level.
Read the full article by Tyler Buffington from Datadog: The Effect Distribution: The Missing Piece in Experimentation Programs
____
AI in A/B testing tools
The Convert team looked at the AI capabilities across 14 A/B testing tools. Of the 59 AI features audited, 58% are only chat wrappers, not very differentiating. What can’t be replaced by an LLM subscription are the features that have domain specific models trained on the platform’s own data (37% of features). Only 5% of features can be called fully agentic. Here AI runs the entire test cycle without a human. READ
____
When standard A/B testing is not possible
Danny Li from BBC Studios wanted to see the impact of a new recommendation feature but an A/B test was not possible. The the team used a difference-in-differences quasi-experiment to assess ML-driven recommendations against manually curated panels. A good, real-world example of how to tackle such a situation. READ
____
Missing piece in experimentation programs: the effect distribution
By Tyler Buffington (Datadog) — If only 5% of your experiments are significant, you might be running tests on ideas with zero true effect. Understanding the distribution of true effects across your program — not just individual results — reveals whether your wins are real, where to invest, and what each experiment is actually worth. READ
____
The curse of the validation mindset
The Lean Startup’s hypothesis-validation loop trains earl-stage founders the wrong muscle, according Jeroen Coelen. They need abductive reasoning (generating new explanations from surprising observations) not inductive confirmation of ideas they already believe. His advice: spend most of your time selling to real people. READ
____
Missed last week’s summary?
🚀 Job opportunities
Find 1k+ open roles on ExperimentationJobs.com. This week’s featured roles:
Lead UX Consultant / Team Lead at Creative CX (London, United Kingdom)
Senior Manager, Product Data Science & Experimentation at Tripadvisor (Poland)
CRO Manager at Sunny Cars (Haarlem, Netherlands)
Experimentation Manager Contract at VML (London, United Kingdom)
Director, CX Optimization & Experimentation at Range (USA)
📅 Upcoming events
A running list of upcoming events. Subscribe here. (👋= join me, 🎁= discount)
👋16 Jul: Experimentation Culture Awards (online)
29 Jul: Experimentation London #12 (London, UK)
31 Jul: TLC: 5 Stories to Rewire Your Statistical Intuition with Ishan Goel (online)
14 Aug: TLC: Bridging The Great Disconnect with Ole Gregersen (online)
🆕27 Aug: TLC Happy Hour (Chicago, USA)
🆕👋2-4 Sep: Future of Experimentation (Porto, Portugal
9-10 Sep: Women in Experimentation Summit (online)
👋24 Sep: Experimentation Elite Awards (London, UK)
👋5 Nov: Experimentation Heroes (Amsterdam, Netherlands)
😃 Something that made me smile
Very timely with so much World Cup football/soccer on TV. Niklas suggested this one, thanks!
When you’re ready, here’s how I can help
If this newsletter sparked something and you want to talk it through, a few ways I help:
Scale experimentation
Strategy, setup, metrics, and ways of working for teams that want more impact.Careers and hiring
Support with your next role, or help finding and hiring strong experimenters.Quick sparring
A fresh outside perspective on your ideas, roadmap, or experimentation setup.
Interested?
Just hit reply and tell me what you’re thinking about.
Have a great week and keep experimenting.
Thanks, Kevin