Experimental Mind: False Positives, Overlapping Experiments, Anecdotal Evidence, Task Playbooks ...

Oct 11, 2021

📰 What I've been reading this week

This is the third post in a multi-part series on how Netflix uses A/B tests to inform decisions and continuously innovate on our products.

Timothy Chan explains how to best deal with overlapping experiments.

Interaction effects are often overblown

3. Designing experiments with new ways of working

John Cutler's advice:

John Cutler @johncutlefish

Unless your experiment with new ways of working is *mostly* to the left, there is a high likelihood it will fail If it "works", then you'll have some leverage. The next experiment is trying to make it stick https://t.co/3K04shytLY

The honest answer to this question is usually: “I need a new story that’s more immediate, more vivid and most of all, more culturally aligned than the one I have now.” ... An anecdote is not evidence. But we often treat it that way.

This is the opposite of a full course, playbooks are bit-sized explainers on how to accomplish a specific task.

Just-in-time learning is focused on meeting the learner’s need when it arises, rather than pre-scheduled education sessions that occur regardless of the immediacy or scope of need.

🚀 Job opportunities

This week's featured jobs:

Product Manager, Experimentation at Adyen (Amsterdam, NL)
VP, Digital Experimentation at Chase (Wilmington DE, US)
Data Product Owner, Customer Experience at Vistaprint (remote EU)
Data Scientist at Bellingcat (remote EU)
Full Stack Developer Experimentation at Booking.com (Amsterdam)

Or have a look at the 13 other opportunities on the job board.

📅 Upcoming events

This is a running list of upcoming events:

Future Data (Oct 13, online)
🆕 Mind the Product conference (Oct 21-22, London)
DDMA Dutch CRO Awards (Nov 4, Amsterdam) see nominees
MIT Conference on Digital Experimentation (Nov 4-5, online)
🆕 CJAM Session with Nathalie Nahai & Bart Schutz (Nov 12, online)
🆕 CJAM Session with Dana di Tomaso & Collin Tate Crowell (Nov 18)
The Conference formerly known as Conversion Hotel (Nov 19-21, NL)
🆕 CJAM Session with Nir Eyal & Hazjier Pourkhalkhali (Nov 23, online)
🆕 CJAM Session with Terry Pierelli & Erin Weigel (Dec 2, online)

💬 Quote of the week

"Progress is more important than perfection.” — Simon Sinek

😉 Fun of the week

What if companies/Apps decided to sell you cookies? Yassine Chouk imagined the results for you.

🔁 Any feedback?

Do you have a minute to give me some feedback on this newsletter?

🙏 Thanks for reading

If you're enjoying the Experimental Mind newsletter you can buy me a coffee or beer. Thanks to the 5 people who already did it. Much appreciated.

And share this email with a friend or colleague. I would love this community of experimenters to grow. You can point them here to subscribe.

Have a great week — and keep experimenting.

Kevin

Experimental Mind