Experimental Mind #213

Your weekly overview of interesting reads, events and jobs for the experimental mind.

Apr 15, 2024

Hi folks!

This is a packed edition. Someone recorded a video making the case against A/B testing. Guess who responded with 7 pages of detailed counter arguments?

Other links: AI Playbook, benchmark report, admiting uncertainity, interview with Oriol Vila Masip, updated events and jobs, a quote about Ferrari’s and Fiats and you can help draw the longest flipbook animation ever. Enjoy!

➡️ AI Playbook for Research, CRO and Experimentation
Start your AI journey with this playbook created by industry experts. Full of prompts, frameworks and tools for every aspect of your experimentation program.
Dowload your free copy here

Thanks to Convert, Sitespect, Exatom and Zoho PageSense for their support.

🔎 Interesting things you might have missed

The Case Against A/B Testing
Solomon Kahn lists 18 common problems he sees in A/B testing in the real world. Watch this 30 minute video and (only then) read the next article.

I’m serious, have you watched it? This is a ‘good list’ of common arguments against A/B testing. You will hear them more, so be ready to understand them. Most points are not things against A/B testing, but againt bad A/B testing. And yes, there is a bunch of bad A/B testing happening in the world, but that does not mean we should stop it completely. Instead, we need to all better understand the imperfections and where possible fix them. Which brings me to …

___

The Imperfections of A/B Testing
Ronny Kohavi (of course) wrote a detailed respons (7 pages) with arguments in two categories: Misunderstandings and poor reasons + some valid concerns.
Link to article

___

Experimentation Program Benchmark Report 2024
Speero surveyed 206 companies about the state of their experimentation program and came up with 18 key findings “you should know if you want to accelerate your program speed and increase its impact.”
Get the report

___

Better Product Decisions with Guardrail Metrics
How Spotify is helping teams to take a step toward more sophisticated experimentation and decision-making by complementing your success metrics with guardrail metrics.
Link to article

___

GrabX Decision Engine
This is an internal open-source package that offers a comprehensive framework for designing and analysing experiments conducted on online experiment platforms.
Link to article

___

Why is it so hard to admit there’s uncertainty in our work?
Because it is uncomfortable. We humans need certainty and predictability. Jeff Gothelf shares a way to try a more “uncertain” approach.
Link to article

___

Two podcast recommendations:

Both are added to the Experimental Mind curated podcast feed.

💬 My Experimentation Career Journey

Today’s interview is with Oriol Vila Masip, Experimentation Lead at Adevinta.

I would strongly recommend joining a company that has experience, resources and reputation regarding experimentation. It is crucial to be surrounded by talented people from whom you can learn. — Oriol Vila Masip

Read the full interview with Oriol

🚀 Job opportunities

Find 100+ open roles on ExperimentationJobs.com. This week’s featured roles:

Senior Product Manager, Subscriptions Growth at Monzo (United Kingdom)
AB Test and Conversion Rate Optimization Specialist at H&M (Warsaw, Poland)
CRO Specialist at Alpina Group (IJsselstein, Netherlands)
Sr. Full Stack Engineer – Product Learning Platform at X (San Francisco, USA)
Lead Data Scientist – Experimentation (A/B Testing) at Target (Minneapolis, USA)

📅 Upcoming events

A running list of upcoming events.

🎁18 Apr: Emerce Conversion & Analytics (Amsterdam, NL). Get 15% off
20 Apr: MeasureCamp (Amsterdam, Netherlands)
23-24 Apr: Conversion Rate Conference (virtual)
13-17 May: CRO trends that need to go away (virtual)
15-17 May: Measuring Behavior (Aberdeen, Scotland)
23 May: Experimentation Culture Awards (virtual)
25 May: MeasureCamp (Madrid, Spain)
4-7 Jun: Marketing Analytics Summit (Phoenix, USA)
6 Jun: Women in Experimentation Summit (virtual)
6 Jun: Digital Marketing Live (Amsterdam, Netherlands)
6-7 Jun: NeuroPsychoEconomics (Milan, Italy)
8 Jun: MeasureCamp (Copenhagen, Denmark)
🆕11-12 Jun: Experimentation Elite (Birmingham, United Kingdom)
🆕15 Jun: MeasureCamp (Paris, France)
🆕🎁19 Jun: Growth Marketing SUMMIT (Frankfurt am Main)

Full list of experimentation conferences & events in 2024

👋 On June 19th I’ll be attending growth marketing SUMMIT, Europe's premier real life conference for digital growth, experimentation and optimization in Frankfurt, Germany. Use code ’experimental-mind‘ to get 10% off and let’s meet there.

💬 Quote to think about

“Fail Ferrari fast and FIAT cheap.” — Alberto Savoia (source)

😃 Something that made me smile

Russell Samora created this “experiment in generation loss; how will the original drawing mutate as it is traced by more and more people?”. You can also help draw the longest flipbook animation ever.

Experimental Mind