Experimental Mind: how Netflix uses A/B testing

Sep 27, 2021

📰 What I've been reading this week

The second post in a multi-part series on how Netflix uses A/B tests to inform decisions and continuously innovate on our products.

Gagan Biyani explains why traditional MVP's can lead founders astray. And he outlines a 3-step framework for developing a Minimum Viable Test:

Tips from Bhavik Patel on how to cope with a losing experiment:

My one rule for running experiments is: “remain emotionally detached”.

Learn the build vs. buy framework to consider opportunity costs and make an informed decision on whether to buy or build a custom solution.

Post from Eric Weber on creating value from data (products):

... what *value* are you creating for your customers and how do you know you are creating that value? Answering this question is hugely important.

Good post from Benn Stancil about the dismantling of Business Intelligence tools, what they actually are and should be.

BI tools should aspire to do one thing, and do it completely: They should be the universal tool for people to consume and make sense of data. If you—an analyst, an executive, or any person in between—have a question about data, your BI tool should have the answer

🚀 Job opportunities

This week's featured jobs:

Data Product Owner, Customer Experience at Vistaprint, Remote EU
Experimentation Insights Director at Nike, Beaverton, US
Data Analyst Experimentation at bol.com, Netherlands
Data Scientist at Bellingcat, Remote EU
Experimentation & Causal Inference Intern at Netflix, Los Gatos US

But wait ... there are another 22 opportunities on the job board. Let me know of you are hiring. You can also add your vacancy to the board.

Sophia Yang walks you through some of the popular variance reduction methods in online experimentation.

📅 Upcoming events

This is a running list of upcoming events:

MeasureSummit 2021 (Sep 28-Oct 1, online)
Experiment Nation: The Conference (Sep 29, online)
NEW Future Data (Oct 13, online)
DDMA Dutch CRO Awards (Nov 4, Amsterdam)
MIT Conference on Digital Experimentation (Nov 4-5, online)
The Conference formerly known as Conversion Hotel (Nov 19-21, NL)

💬 Quote of the week

"When a measure becomes a target, it ceases to be a good measure.” — Charles Goodhart

😉 Fun of the week

This image of Angela Merkel with birds is doing it's round on the web. I have applied it to launching an experimentation.

🙏 Thanks for reading

If you're enjoying the Experimental Mind newsletter you can support me and my family with a donation to the study fund for our two kids. Much appreciated.

And share this email with a friend or colleague. I would love this community of experimenters to grow. You can point them here to subscribe.

Have a great week — and keep experimenting.

Kevin

ps. if you have any feedback simply reply or click the thumbs up/down.

Experimental Mind