Experimental Mind #272

Your weekly overview of interesting reads, events and jobs for the experimental mind.

Jun 23, 2025

Last week I attended the GMS conference in Frankfurt. Again a great experience especially because I met so many new and old friends. Ton Wesseling created this AI generated overview of all keynotes presentations. You can even chat with the raw input. These tools will not replace going to conference, but can definetly make it easier to do something with the insights.

This week: from an AI trained on Ronny Kohavi’s work to a new tool that audits scientific research for best practices. Plus: GoDaddy’s approach to experimentation, new AI-era job roles, and a call for abstracts from MIT CODE. Enjoy.

Thanks to Convert and Sitespect for their continued support.

🔎 Interesting things you might have missed

A/B testing RAG agent
Kameran Kolahi took Ronny Kohavi's work (book, courses, papers and articles) and turned it into ‘A/B AI’, a trusted, reliable source for deep A/B testing questions and reports. Try it out or first watch this demo.

____

CODE 2025 is now accepting abstracts
The MIT Conference on Digital Experimentation (MIT CODE) is accepting abstracts for their event happening mid November. Deadline to get your abstract in: August 4. >>>

____

How A/B testing transformed product development at GoDaddy
”The most impactful experiments don’t just move metrics, they advance our understanding of the customer and shape product strategy. A scattered, one-off experiment might drive a short-term win, but experiments aligned with product themes and clearly defined customer problems lead to stronger product experiences.”
>>>

____

A.I. might take your job: here are 22 new ones it could give you
A New York Times Magazine piece outlines three upcoming job categories in the era of AI: roles centred around trust (like AI ethicists and auditors), integration (embedding AI into business), and creative direction roles (guiding AI’s creative output). >>>

____

Building a world-class economic research team inside tech
Lessons from building an Economic Research team at Glassdoor: “a lot of hard-won lessons about how to build and scale a small economic research team in tech — and in the process turning Glassdoor into a household name among media and academics on the topics of pay transparency, company culture, and hiring trends.” >>>

____

Introducing Papercheck: check for best practices in scientific articles
Daniel Lakens and his team introduce Papercheck, a new tool using LLMs and code to automatically audit research repositories, flagging issues like imprecise p‑values, missing effect sizes, absent readme files, and inadequate documentation of data and code. >>>

____

Last week’s most clicked item:

Is this the next big thing? Synthetic A/B Testing
Jonny Longden sparked some interesting ideas around synthetic testing. This is definitely coming and already useful for automating visual QA, performance checks, and early usability signals. The problem I see is that these synthetic agents still reflect past data. Live A/B tests show how real people behave today, in current contexts and conditions. >>>

🚀 Job opportunities

Find 100+ open roles on ExperimentationJobs.com. This week’s featured roles:

Senior CRO Manager at Growth Shop (Canada)
Senior Data Scientist - Experimentation Platform at Skyscanner (London or Barcelona)
Conversion Rate Optimization (CRO) Lead at Lawhive (London, United Kingdom)
Principal Data Scientist, Experimentation at Zillow (USA)
Software Engineer, Experimentation at Vercel (Germany/Netherlands/UK)

📅 Upcoming events

A running list of upcoming events. Will I see you in Amsterdam or at the virtual Women in Experimentation summit?

🆕25 Jun: Vancouver Round Table (Vancouver, Canada)
🆕27 Jun: Nudgestock (London, UK)
1 Jul: Berlin Experimentation Meetup #9 (Berlin, Germany
3 Jul: EXL Meetup (Amsterdam, Netherlands)
🎁7 Jul: Accelerating Innovation with AB Testing (virtual, get $500 off)
9 Jul: Women in Experimentation Summit 2025 (virtual)
9 Jul: CRO & Growth Meetup (Paris, France)
🆕24 Jul: Testing Business Ideas with ChatGPT (virtual, get $50 off)

🎧 Podcast of the week

As a fan of Scotland, I often listen to the Scotcast podcast. In this episode Professor Richard Susskind shares his predictions on what the rise of artificial intelligence will mean for our work, the economy and the future of humankind. Recommended.