Experimental Mind: Netflix's Culture of Learning

🔎 What I've been reading last week

Earlier posts in this series covered the why, what and how of A/B testing, all of which are necessary to reap the benefits of experimentation for product development. But without a little magic, these basics are still not enough. The secret sauce that turns the raw ingredients of experimentation into supercharged product innovation is culture.

After reading all blog posts in this series, it's clear that Netflix is light years ahead of many companies, with regards to experimentation and the culture around decision making. In case you missed earlier blog posts, here is the full overview. Read, share and discuss how you make this happen in your org.

1. Decision making at Netflix

2. What is an A/B Test?

3. Interpreting A/B test results: false positives and statistical significance

4. Interpreting A/B test results: false negatives and power

5. Building confidence in a decision

6. Experimentation is a major focus of data science across Netflix

7. Netflix: A Culture of Learning

Microsoft also has a world–class scalable experimentation platform. In this blog post they explain how it has been built:

1. Develop a long-term vision and measure progress

2. Keep a pulse on (internal) customers

3. Create a culture of collaboration

A paper that looks at the effectiveness of alternative methods to measure the of Facebook ads:

... broadly speaking, our results suggest that state-of-the-art observational methods are unable to recover the causal effect of online advertising at Facebook. We conclude that observational methods for estimating ad effectiveness may not work until advertising platforms log auction-specific features for modeling.

Pinterest created Offline Replay Experimentation, a framework to simulate the performance of new ideas based on historical data.

While a simulation approach is less accurate than online experimentation, it complements online experimentation by eliminating waiting time for data collection, allowing for a large number of simultaneous arms, avoiding negative user impact, and saving engineering costs.

🚀 Job opportunities

New opportunities keep opening up. Experimentation is a fascinating field to work in. Checkout the 33 open roles.

These are the latest featured roles:

• 🆕 Performance Marketing Specialist at Picnic (Amsterdam)

• 🆕 Principal Analyst Experimentation at Vista (Remote EU)

• Data Scientist Experimentation at Just East (London)

• Experimentation Lead Software engineer at Vista (remote EU)

• Digital Experimentation Manager at Philips (Amsterdam)

• Head of Product Experimentation at Zalando (Berlin)

• CRO Lead at New Balance (Amsterdam, the Netherlands

WIth so many companies investing in experimentation capabilities and building teams, it's a challenge to find the right people. Posting your open roles to the job board is a great way to get your amazing job in front of the right people. It also helps to support the Experimental Mind newsletter.

📅 Upcoming events

This is a running list of upcoming events:

• Quantum Leap (Feb 7-9, Austin, TX, US)

• MeasureCamp North America (Feb 26, online)

• 🆕 Experimentation Elite (Apr 27, London)

I've started to create the full overview of events for 2022, let me know in case you are missing an event.

💬 Quote of the week

"All life is an experiment. The more experiments you make the better. — Ralph Waldo Emerson

😉 Fun of the week

Everyone says they want to be data-driven. Not everyone wants to do the necessary work (via Timo Dechau).

🙏 Thanks for reading

If you're enjoying the Experimental Mind newsletter, you can buy me a coffee or beer (in case you rather use Paypal, you can send it to 'kevin@experimentalmind.com'. Thanks to the 21 people who have already done it. Much appreciated.

Or simply share this newsletter with a friend or colleague (just like Luka did). Point them here to subscribe.

Have a great week — and keep experimenting.

Kevin