This week: how to value evidence-based decision making? A new paper has an answer. Also: LLM evals meet A/B testing, and why strong ideas don't need to ship immediately.
You'll also find 914 open roles, upcoming events, and a cartoon that made me smile.
Brought to you by
Convert—A/B tests & personalization for growth teams
Experimentation Jobs—Find your next role
🔎 Interesting things you might have missed
[Paper] Estimating the value of Evidence-Based Decision Making
By Abadie, Agarwal, Imbens et al. (MIT/Stanford/Amazon) — How much is running experiments actually worth? This paper builds a framework to calculate it. Using empirical Bayes on 4,8k+ Upworthy A/B tests, they show that standard p-value decision rules leave ~27–30% of attainable value on the table. READ
____
[Paper] Two-sided online review systems drive desirable customer behaviors
By Marder et al. — Eight experiments confirm it: when Airbnb or Uber can rate you back, you clean more, tip more, and talk more politely. Simply because your score is at stake, not because you’re kind. People fear being judged by someone who controls your future access. And it only kicks in when you plan to use the platform again. READ
My reflection: this research reminds me of the Black Mirror episode Nosedive. Here they depict a world where every social interaction is rated, and people perform relentless niceness to protect their score. This paper is the empirical evidence that we are already there. Not sure I like this state of things.
____
Stop building innovation labs
By Jack Strachan — The first wave of public sector labs worked because of the conditions around them: protected authority, proximity to decisions, evidence through building. When those conditions disappeared, so did the impact. The lab was never the innovation. Experimenters need to understand: a faster feedback loop only matters if it’s attached to something consequential. READ
____
Better experiments with LLM evals — a funnel, not a fork
By Matilda Ankargren and Mårten Schultzberg (Spotify) — LLM evals verify quality before an experiment; A/B tests validate real user impact after. Use evals to raise your experiment hit rate, then run evals on A/B data to calibrate the judge. Each cycle makes both smarter. READ
____
The website specification
By Joost de Valk — Great reference for what a good website should do, regardless of the stack. Every topic has a status: required, recommended, optional or avoid. CHECK
____
Why I don't write every day
By Mike Fisher — Not all work benefits from daily output. Strong ideas don't fade, they accumulate tension until writing (or shipping) becomes inevitable. In the context of experimentation: not every hypothesis deserves immediate execution. Time is a filter, and too early convergence is not good. READ
____
Last week’s favourite:
Tips for buying a new A/B testing tool
Ruben de Boer created three short videos packed with tips on how to approach selecting a new A/B testing vendor. Take your time and make a good, solid decision. LEARN
🚀 Job opportunities
Find 914 open roles on ExperimentationJobs.com. This week’s featured roles:
Senior Front-end Developer (Experimentation) at Creative CX (London, United Kingdom)
Senior Conversion Rate Specialist at AJ Bell (London, United Kingdom)
Experimentation & Cro Lead at Sas Group (Solna, Sweden)
Product Manager & CRO at Nacar (Barcelona, Spain)
Product Manager - Web Personalization at Bloomreach (Czechia)
📅 Upcoming events
A running list of upcoming events. Subscribe here. (👋= join me, 🎁= discount)
🎁1 Jun: Accelerating Innovation with A/B Testing with Ronny Kohavi (online,
$1,999$1,749 USD)👋10 Jun: Best in Test Awards (online)
🎁15 Jun: Advanced Topics in Practical A/B Testing with Ronny Kohavi (online,
$1,199$999 USD)17 Jun: TLC Panel: Accessibility, AI Discoverability, & SEO with Shirley van Haalem, Alisa Scharf, and Sani Manic (online)
23 Jun: Causal Summit (online)
🔥26 Jun: Deadline shortlist submission Experimentation Heroes
👋16 Jul: Experimentation Culture Awards (online)
9-10 Sep: Women in Experimentation Summit (online)
24 Sep: Experimentation Elite Awards (London, UK)
5 Nov: Experimentation Heroes (Amsterdam, Netherlands)
😃 Something that made me smile
When you’re ready, here’s how I can help
If this newsletter sparked something and you want to talk it through, a few ways I help:
Scale experimentation
Strategy, setup, metrics, and ways of working for teams that want more impact.Careers and hiring
Support with your next role, or help finding and hiring strong experimenters.Quick sparring
A fresh outside perspective on your ideas, roadmap, or experimentation setup.
Interested?
Just hit reply and tell me what you’re thinking about.
Have a great week — and keep experimenting.
Thanks, Kevin
Psst -I think the link is missing for "Two-sided online review systems drive desirable customer behaviors" . And since all the others are so good, I would you send it to me?