Reflections Conversion Hotel 2025 conference
Last weekend I was at the Conversion Hotel conference. This was the twelfth edition, and I’ve joined eight so far, including one as a speaker. As I’ve done in 2023 and 2024 I will again share some my personal take aways and some general reflections. For all notes of the sessions visit this doc.
If you work in CRO, Experimentation or a related field in Europe, this is the conference you should visit next year. The dates are November 20 to 22, 2026. For people in the US, Kelly Wortham and Ton Wesseling are organising the second edition of Experimentation Island in February 2026.
Personal take aways
All presentations were good, most were great and a few were outstanding. These were my highlights.
Unsustainable Cost of AI by Iqbal Ali
Large AI systems use a lot of energy. Iqbal showed that we can often get the same work done with simpler models that require far less. It starts with clear workflows, validation steps and trying smaller local models before reaching for the big ones.
Experiment Quality by Melanie Mueller
Booking.com has been experimenting for twenty years and now runs around fifteen thousand experiments per year. But are they all good experiments? Melanie shared her five year journey of measuring and improving experiment quality from around ten percent to seventy five percent.
Optimizing for AI agents by Slobodan ‘Sani’ Manic
With the rise of AI, many expect strong adoption of AI agents that will do online shopping for us. Sani showed that many websites are not ready for this shift. CRO teams need to start optimising for these agents, which has a welcome side effect: it also improves websites for humans.
Science vs CRO by Marcella Sullivan
Marcella explained the difference between scientific research, which demands high levels of rigor, and experimentation in a business context, which aims for practical decision making. In companies we are not searching for universal truths. The level of rigor should match the situation and the decision at hand.
Typocalypse by Oliver Schöndorfer
This talk was part comedy, part theatre and full of practical lessons about the importance of good typography and clear information hierarchy. Contrast, hierarchy, font size and colour choices can make digital products more accessible for millions of people.
Reflections
But Conversion Hotel is much more than the presentations on stage. What makes it so special? A few reflections.
AI everywhere – Even though several talks focused on the downsides of AI, almost all experimentation tool vendors showed AI powered versions of their products. The one that impressed me most for now was Kameleoon’s prompt based experimentation feature. You can create a variant with a simple prompt and ship it as a test within minutes.
Unique takeaways – Everyone walks away with something different. It depends on where you are in your life and career. It is a bit like rereading a book. Even the second time you can find new insights.
Attention to detail – This is one of the few conferences where people buy a ticket without knowing the line up. The main reason is trust. People know host Ton refines every detail and delivers a great experience year after year.
All about connections – The conference is designed for real conversations. You never feel rushed. There is plenty of time to meet new people, reconnect with familiar faces and have meaningful discussions during breakfast, lunch, dinner or even on the beach.
Unconference – Attendees also become speakers during the many unconference sessions. These are small, intimate gatherings spread throughout the hotel that allow for deeper and more focused discussions.
If you have never been, make sure you go next year. And if you are planning to, let me know. I would love to meet you there.
How was this edition?
Let me know by clicking one of these options: Excellent | Great | Good | OK | Meh
Thanks, Kevin