Experimental Mind: what makes a metric good?

🔎 What I've been reading

From the Microsoft Experimentation team:

Good metrics enable good decisions. What makes a metric good? In this blog post we introduce the STEDII (Sensitivity, Trustworthiness, Efficiency, Debuggability, Interpretability, and Inclusivity) framework to define and evaluate the good properties of a metric and of an A/B test analysis in general.

Thinking of starting a career as Data Analyst, but think you lack the knowledge and experience? Cassie Kozyrkov makes the case that you're already are one.

Bart Schutz applies the five ethical principles for influencing and changing behaviour at scale to a very concrete example from industry.

Two papers from the CMA's Behavioural Hub discussing and summarising evidence on online choice architecture (OCA) and how it potentially causes harm.

The ghost-ads methodology works with modern online ad platforms to reduce the cost of experimentation and improve measurement precision. ... Today, several advertising platforms offer ghost ad methodologies to measure ad effectiveness.

In Ronny Kohavi's class on 'Accelerating Innovation with A/B Testing', Luis Trindade presented Farfetch's approach to building and/or buying experimentation tooling.

📢 In case you missed Ronny Kohavi's class, you can join cohort #4 and he will teach you everything he knows about A/B testing in 5 two hour sessions. Subscribe via this link and you will receive a $50 Amazon gift card.

HBR article by Dean Karlan and Michael Luca:

To better leverage data, leaders need to understand the types of problems data can help solve as well as the difference between those problems that can be solved with improved prediction and those that can be solved with a better understanding of causation.

Sample (Size) Rate Mismatch #SRM feature was added to our platform and instantly 30% of our customers had it! Looking deeper it’s 6-7% of all experiments.

Who will follow?

🚀 Job opportunities

Ready to lead a global experimentation program? As CRO Specialist at Randstad Global you will help countries validate insights, identify key areas for improvement and hypotheses for testing.

All featured roles:

• Digital Analyst at RTL (Hilversum, NL)

• Webanalist at Kaartje2go (Zwolle, NL)

• Conversie Optimalisatie Specialist at Kaartje2go (Zwolle)

• Conversie Optimalisatie Lead at Kaartje2go (Zwolle, NL)

• CRO Specialist at Randstad Global (Amsterdam, NL)

Also checkout the job board for more open roles in the field of experimentation, CRO and analytics.

📅 Upcoming events

Upcoming events in the next months:

• 21 April: Emerce Conversion (Amsterdam)

• 27 April: Experimentation Elite (London)

• 14 May, MeasureCamp (London)

• 18-20 May: Measuring Behavior (Krakow)

• 19 May: MeasureBowling (12+ cities)

• 21 May: MeasureCamp (Amsterdam)

• 🆕 31 May: Accelerating Innovation with A/B Testing (virtual)

• 8-9 June: Advances with Field Experiments (Chicago)

• 9-10 June: NeuroPsychoEconomics (Lille)

• 15 June: Growth Marketing SUMMIT (Frankfurt am Main)

• 16-18 June: Marketing Science Conference (virtual)

• 23-24 June: Outperform Conference (Amsterdam)

• 25 June: MeasureCamp (Paris)

See the full overview of events to find 15+ other events and conferences for you and your team.

💬 Quote of the week

“Fight as if you are right, and listen as if you are wrong." — Karl Weick

😉 Fun of the week

Via Manuel Da Costa:

☕ Thanks for reading

If you’re enjoying the Experimental Mind newsletter and want to say thank you? Buy me a coffee (or a beer). Or share this newsletter with a friend or colleague.

Have a great week — and keep experimenting.

Kevin