On May 19, 2025, we hosted the second Experimentation Conference at Booking.com at our Amsterdam campus. Building on the success of the 2024 event, this invite-only conference brought together 126 leading practitioners working on large-scale experimentation programs in 40 companies, with new voices, different topics and just the right mix of technical depth and organizational reflection.
Across the day, we explored a wide range of topics: from meta-analyses and experimentation pitfalls in marketplaces, to the nitty-gritty of platform decisions and statistical design choices. Below is a short recap of the key sessions, grouped by the conference’s three main themes:
Meta-analysis: learning across multiple experiments
SUTVA violations: when standard A/B testing is not enough
Experimentation platforms: building a scalable user-friendly platform for diverse users
Meta-analysis: learning across multiple experiments
Looking at many experiments can help you judge their impact more accurately.
From Illusion to Innovation — The Path to Reliable A/B Testing
This session was led by Eline van Baal and Hans Nauta from NS International. They shared how unexpected results in experiments on sustainability messaging and checkout design led them to embrace a more critical, replication-focused approach to experimentation. Working together with Professor Kobe Millet from Vrije Universiteit Amsterdam, they developed a re-testing framework to reduce false positives and improve the reliability of their conclusions.
Key takeaway: if you want to make sure your results are reliable, rerun your experiments
Aggregating Experiment Impact at Scale
Deeksha Gupta and Nils Skotara from Booking.com presented how they built a centralized impact hub to aggregate test results across teams and business units. They discussed how they addressed the “winner’s curse” — the tendency for reported test impacts to be overly optimistic — by estimating and adjusting for the false discovery rate and the overestimation error, helping the organization get a clearer view of overall experimentation impact.
Key takeaway: Successful variants (winners) are more likely to be shipped, creating a selection bias problem: some of them are false positives, and even true winners have an overestimated effect. Adjusting for the false discovery rate can help.
SUTVA violations: when standard A/B testing is not enough
Standard A/B testing, also called randomized controlled trials (RCT) need the Stable Unit Treatment Value Assumption (SUTVA). However, in real-world applications, this assumption is often violated, for example when there is interference between control and treatment.
When A/B Tests Lie: Spotting Interference in Vinted Marketplace
Giedrius Blazys from Vinted explained how standard A/B tests can mislead in marketplaces where users compete for limited goods or attention. For example, when desirable items are upsorted on Vinted in the treatment group, users in the treatment group buy them more likely, making them unavailable in the control group. He illustrated how test uplift decays — and can even reverse — as the percentage of treated users increases. Using real-world data and agent-based simulations, he shared how Vinted detects SUTVA violations, and the lightweight protocol they use to track interference through treatment scaling.
Key takeaway: When your measured uplift becomes smaller with increasing treatment allocation, it is a signal that you have interference bias — and should think about correcting for it.
Developing Switchback Experiments in Glovo
Iulian Vasilisca from Delivery Hero / Glovo talked about how Glovo corrects for marketplace interference by using time-clustered switchback experiments. Instead of splitting users into two separate groups, all users are repeatedly switched between control and treatment conditions over time. He offered a practical walkthrough of how these experiments are implemented in their platform — from assignment and logging, to statistical design and result analysis.
Key takeaway: When there’s interaction between different users at a specific point in time, don’t expose them to different variants at the same time, but rather sequentially in time.
Experimentation platforms: building a scalable user-friendly platform for diverse users
Experimentation platforms have to cater to a wide range of users with different skills and knowledge, and support a wide range of use cases. This session explored how central experimentation teams can build for such diverse requirements.
Buy What Accelerates You, Build What Differentiates You
Nils Stotz and Marcel Toben from Zalando explored how they distinguish between experimentation components that can be bought and those that should be built in-house. Drawing from their experience at Europe’s largest online fashion platform, they argued for a hybrid model: using third-party platforms for efficiency, while customizing domain-specific tooling to reflect the strategic diversity of teams across logistics, payments, and personalization.
Key takeaway: Outsource general functionality needs, but keep functionalities for your ‘secret sauce’ in-house.
Your Experiments Are a Waste of Time
Garret O’Connell from Bolt opened with a provocative statement: too many experiments are a waste of time and fail because of poorly chosen Minimum Detectable Effects (MDE). He introduced a straightforward framework for setting MDEs that align with business goals and real-world constraints, based on thinking about experiments as bets to balance impact and risk. He demonstrated how Bolt embedded this thinking into their platform to help teams set up statistically sound tests without needing deep expertise.
Key takeaway: Thinking about experiments as bets can help you set better MDEs.
Spotify’s Journey Navigating the Trade-offs Involved in Large-Scale Experimentation
Lizzie Eardley and Caro Grahlmann from Spotify shared their experience evolving the company’s experimentation infrastructure to balance speed, rigor, and scale. They discussed how they navigated competing priorities — like computational cost vs. user experience, and statistical rigor vs. team flexibility — while supporting thousands of experiments annually. Their talk was an honest reflection on trade-offs, iteration, and growth.
Key takeaway: Even though statistical rigor is important for high-quality experimentation results, real-world A/B testing requires some practical flexibility, because mistakes happen, and things change.
Final Thoughts
Beyond the talks, the conference offered a maybe less tangible but equally important value: real connection. Whether over breakfast, in hallway conversations, or during the borrel after the final talk, people shared stories, asked honest questions, and sparked new ideas. The field of experimentation is growing fast, but it’s also maturing. This event showed that well-run experiments need both strong math and strong collaboration.
This article was originally published on the Booking.com Medium blog.
I've always had an issue with this one. "Key takeaway: if you want to make sure your results are reliable, rerun your experiments".. Yes, replication is gold standard. But you're inflating false positive rate in a single replication. So, let's say you had a test result that came back with a negative signal and you really believed in the feature because of research or whatever, so you re-test assuming it was a false positive. This time, result is neutral, but, of course, no stat sig. Which result? So you treat again... Now, time and samples are confounding and we're not even really replicating. It may be minor, or it could be a huge impact to reaction to your treatment. 🤷♀️
Fantastic summary. Thanks!