Experimental Mind: Decision Making, Lighthouse Metric, Dark Patterns, Thinking Modes ...

Sep 13, 2021

What I've been reading this week

Part 1 in a multipart series about decision making and experimentation at Netflix.

At Netflix, we believe there’s a better way to make decisions about how to improve the experience we deliver to our members: we use A/B tests. Experimentation scales. Instead of small groups of executives or experts contributing to a decision, experimentation gives all our members the opportunity to vote, with their actions, on how to continue to evolve their joyful Netflix experience.

ps. Netflix is looking for an Experimentation & Causal Inference Intern

Chetan Sharma will start a series of blogposts that "... are aimed at executives who are hesitant to ramp up more experimentation, and product leaders who want to get more buy-in to run experiments". In this first post he talks about metric strategies and why they are crucial in boosting the number of experiments.

A/B testing is rarely used for comparison between whole builds. That’s because each build integrates multiple feature changes and it is hard to figure out which features cause regressions, if any. However, the Microsoft Teams Experimentation team did make an attempt to use A/B testing as an integration testing tool for builds comparison.

I don't like articles that have 'death of ...' titles, but Jason Hreha does have a point here. Two actually:

This blogpost explains why and how Wish is evaluating A/B tests with percentile metrics.

Bhavik Patel suggests we start calling it 'Lighthouse Metric', not NSM.

On how to measure the success of an AI intervention in quantifiable outcome.

Statistical significance, confidence interval and statistical power (although that is more relevant before the start of your experiment).

In this study the authors look at 50 types of dark patterns across the desktop web, mobile web, and mobile app modalities of 105 services. Most used dark patterns are privacy related, see table below.

According to Alex Birkett it's:

This paper won the best paper award at the Software Engineering and Advanced Applications (SEAA) conference. Congratulations Aleksander, Pavel and Lukas.

Jan Bosch explains why:

A/B testing is relevant in any context where a mathematical optimum can’t be established easily. As such, it’s relevant for virtually any company and system. With the increased connectivity of systems and a continuously growing expectation of improvements to systems throughout their entire economic life, it’s clear that A/B testing is a capability that every industry, company and team needs to build up.

Jason Widup on how to get a systematic plan to run effective marketing experiments and learn from them.

Google asked Ipsos to interview thousands of people about their personal privacy and data. Three tips for defining your data strategy:

I think too many of us spend too much time thinking like preachers, prosecutors, and politicians ... We need to think more like a scientist.

Job opportunities

This week's featured jobs:

Technical Director Experimentation at EA Sports, Redwood/Remote US
Sr. Product Manager Experimentation at Hula, Remote US
Research Scientist Manager Experimentation at Facebook, London
Head of Experimentation at Zalando, Berlin
CRO Specialist at KPN, Amsterdam

Don't forget to check out the other 15 opportunities on the job board. Are you hiring? Simply add the vacancy to the board.

Upcoming events

Experimentation Works (Sep 16, online)
MeasureSummit 2021 (Sep 28-Oct 1, online)
Experiment Nation: The Conference (Sep 29, online)
MIT Conference on Digital Experimentation (Nov 4-5, online)
The Conference formerly known as Conversion Hotel (Nov 19-21, NL)

Quote of the week

If you're not experimenting, your baseline assumption is that whatever you're doing now is perfect — Richard Mathera

Fun of the week

As always, if you're enjoying this email , I'd love it if you shared it with a friend or two.

You can send them here to sign up. I try to make it one of the best emails you get each week, and I hope you're enjoying it. And should you come across anything interesting this week, send it my way! I love finding new things to read through members of this newsletter.

Have a great week — and keep experimenting.

Kevin

Experimental Mind