Experimental Mind: it's not about ROI

Now over to the newsletter with a selection of articles, jobs, events and quotes. Handpicked by me, so you don’t have to.

🔎 What I've been reading & listening

This week I added two episodes to the curated list of podcasts:

1. André Morys of konversionsKRAFT on the 1,000 Experiments Club

2. Rohan Katyal (Yelp) on Building An Experimentation Program (Experimentation Masters Podcast)

Make sure to subscribe.

Short paper by the Online Dialogue team stating that too much focus is on driving direct Return-on-Investment (ROI) from experimentation.

... this awareness must lead to budget and resources for a stand-alone experimentation Center of Excellence (CoE) outside of the marketing department, that different (not ROI focused) KPIs are needed when scaling up experimentation ...

They also link to a dataset showing that most agencies and vendors sell their work and tools by showcasing conversion uplifts.

There is some serious investment (millions of dollars) being done in experimentation tools like Eppo and Statsig.

They are trying to solve these main challenges:

1. Experiments are disconnected from the engineering process, adding complexity and risk

2. Setting up valid experiments requires too much pre-existing expertise, burning time and trust

3. Results aren’t easily actionable, and leave teams lost, frustrated, or winning experiments un-launched

There are some big differences between an amateur and an expert professional analyst and in this article, I’ll summarize 10 of them for you.

The differences: software skills, handling lots of data with ease, immunity to data science bias, understanding the career, refusing to be a data charlatan, resistance to confirmation bias, realistic expectations of data, knowing how to add value, thinking differently about time, and nuanced view of excellence

Not really surprising findings, but now based on an experimental design:

The researchers gathered 154 volunteers with a mean age of 29.6 years old. They randomly divided them into an intervention group, which was asked to stop using social media for one week (with a focus, in particular, on Facebook, Twitter, Instagram, and TikTok), and a control group which was given no instructions.

🚀 Job opportunities

I am working on a big improvement of the job board. Don't peek.

📅 Upcoming events

Join this great virtual event for free.

All upcoming events in the next months:

• 7 July: Experimentation Culture Awards (virtual)

• 🆕 26 July: CRAP Talks #18 (London)

• 4-6 Aug: Digital Elite Camp (Estonia)

• 22 Aug: Accelerating Innovation with A/B Testing (virtual)

See the full overview of events to find other events and conferences for you and your team.

💬 Quote of the week

"There are three principal means of acquiring knowledge: observation of nature, reflection, and experimentation. Observation collects facts; reflection combines them; experimentation verifies the result of that combination." — Denis Diderot

😉 Fun of the week

I had great fun playing this interactive game:

William B. Davis @willy_maps

The closest 10 airports https://t.co/82Rpwkot43 #builtwithmapbox #turfjs https://t.co/RlerHyaomE

☕ Thanks for reading

If you’re enjoying the Experimental Mind newsletter and want to say thank you? Buy me a coffee (or a beer). Or share this newsletter with a friend or colleague.

Have a great week — and keep experimenting.

Kevin