Part 1 in a multipart series about decision making and experimentation at Netflix.
At Netflix, we believe there’s a better way to make decisions about how to improve the experience we deliver to our members: we use A/B tests. Experimentation scales. Instead of small groups of executives or experts contributing to a decision, experimentation gives all our members the opportunity to vote, with their actions, on how to continue to evolve their joyful Netflix experience.
ps. Netflix is looking for an Experimentation & Causal Inference Intern
Gagan Biyani explains why traditional MVP's can lead founders astray. And he outlines a 3-step framework for developing a Minimum Viable Test:
Find your value proposition
List your risky assumptions
Test the atomic unit
The Wall Street Journal published a powerful multi-part series on the company this week, drawing from internal documents on everything from the company’s secretive practice of whitelisting celebrities to its knowledge that Instagram is taking a serious toll on the mental health of teen girls.
The question is: what will they do about it, now Facebook knows this?
A/B testing is rarely used for comparison between whole builds. That’s because each build integrates multiple feature changes and it is hard to figure out which features cause regressions, if any. However, the Microsoft Teams Experimentation team did make an attempt to use A/B testing as an integration testing tool for builds comparison.
Bhavik Patel suggests we start calling it 'Lighthouse Metric', not NSM.
Learn the build vs. buy framework to consider opportunity costs and make an informed decision on whether to buy or build a custom solution.
Good post from Benn Stancil about the dismantling of Business Intelligence tools, what they actually are and should be.
BI tools should aspire to do one thing, and do it completely: They should be the universal tool for people to consume and make sense of data. If you—an analyst, an executive, or any person in between—have a question about data, your BI tool should have the answer
In this study the authors look at 50 types of dark patterns across the desktop web, mobile web, and mobile app modalities of 105 services. Most used dark patterns are privacy related, see table below.
Sophia Yang walks you through some of the popular variance reduction methods in online experimentation.
Richard Thaler's book 'Nudge' was first published in 2008. With the publication of a new, radically updated edition, Thaler tries to persuade Stephen Dubner that nudging is more relevant today than ever.
Jan Bosch explains why:
A/B testing is relevant in any context where a mathematical optimum can’t be established easily. As such, it’s relevant for virtually any company and system. With the increased connectivity of systems and a continuously growing expectation of improvements to systems throughout their entire economic life, it’s clear that A/B testing is a capability that every industry, company and team needs to build up.
Jason Widup on how to get a systematic plan to run effective marketing experiments and learn from them.
This week's featured jobs:
And don't forget to check out the other 18 opportunities on the job board. Are you hiring? Simply add the vacancy to the board.
Google asked Ipsos to interview thousands of people about their personal privacy and data. Three tips for defining your data strategy:
Make it meaningful
Make it memorable
Make it manageable
I think too many of us spend too much time thinking like preachers, prosecutors, and politicians ... We need to think more like a scientists.
This is a running list of upcoming events:
The key to good problem solving is a willingness to try things, experiment thoughtfully, and do the work. — James Clear
Why do we track so many metrics? (by Work Chronicles)
Thanks for reading. If you're enjoying the Experimental Mind newsletter you can support me and my family with a donation to the study fund for our two kids. Much appreciated.
And share this email with a friend or colleague. I would love this community of experimenters to grow. You can point them here to subscribe.
Have a great week — and keep experimenting.
Kevin
ps. if you have any feedback simply reply or click the thumbs up/down.
How to outperform conventional A/B testing when scaling up personalized messages and services.
When Richard Thaler published Nudge in 2008 (with co-author Cass Sunstein), the world was just starting to believe in his brand of behavioral economics. How did nudge theory hold up in the face of a global financial meltdown, a pandemic, and other existential crises? With the publication of a new, radically updated edition, Thaler tries to persuade Stephen Dubner that nudging is more relevant today than ever.
