Experimental Mind: Picking Metrics, Intuition Busters, Ruling Out, Causal Inference...

🔎 What I've been reading

Picking the right metrics for an experiment is not easy. The Statsig team is giving some good advice, including clear do's and don'ts.

Ronny Kohavi, Alex Deng and Lukas Vermeer have written a paper on Common Misunderstandings in Online Controlled Experiments: A/B Testing Intuition Busters. They are still open for review and comments.

Matt Gershoff explains two important concepts from A/B testing.

There are two things we are trying to rule out when we run AB Tests: confounding & sampling variability/error.

Netflix relies heavily on A/B testing to measure causal effects. In this blog post we get a peek into the recent held internal Causal Inference and Experimentation Summit.

The weeklong conference brought speakers from across the content, product, and member experience teams to learn about methodological developments and applications in estimating causal effects. We covered a wide range of topics including difference-in-difference estimation, double machine learning, Bayesian AB testing, and causal inference in recommender systems among many others.

🚀 Job opportunities

All featured roles:

• Conversie Optimalisatie Specialist at Kaartje2go (Zwolle, NL)

• Conversie Optimalisatie Lead at Kaartje2go (Zwolle, NL)

• CRO Specialist at Randstad Global (Amsterdam, NL)

• Digital Analyst at RTL (Hilversum, NL)

• Webanalist at Kaartje2go (Zwolle, NL)

Also checkout the job board for more open roles in the field of experimentation, CRO and analytics.

📅 Upcoming events

I'll be one of the jury members at the 2022 Experimentation Culture Awards. Join the award session and listen to inspiration stories.

All upcoming events in the next months:

• ⏩ 8-9 June: Advances with Field Experiments (Chicago)

• ⏩ 9 June: DiDo#39 Ga voor (CRO) goud! (Utrecht)

• ⏩ 9-10 June: NeuroPsychoEconomics (Lille)

• 15 June: Growth Marketing SUMMIT (Frankfurt am Main)

• 16-18 June: Marketing Science Conference (virtual)

• 25 June: MeasureCamp (Paris)

• 7 July: Experimentation Culture Awards (virtual)

• 4-6 Aug: Digital Elite Camp (Estonia)

• 🆕 22 Aug: Accelerating Innovation with A/B Testing (virtual)

Second batch of tickets will release soon:

• 🎫 18-20 Nov: Conversion Hotel (Texel, Netherlands)

See the full overview of events to find other events and conferences for you and your team.

💬 Quote of the week

“The old is dead, and I don’t know what the new is. The only way to find the new is to start different things and see if there’s something that can come out of experimentation.” — David Lynch

😉 Fun of the week

The latest Marketoonist on The Innovation Funnel:

... this Darwinian process of bringing ideas to life doesn’t necessarily lead to survival of the fittest ideas. If we’re not careful, the innovation funnel leads to survival of the safest ideas.  

☕ Thanks for reading

If you’re enjoying the Experimental Mind newsletter and want to say thank you? Buy me a coffee (or a beer). Or share this newsletter with a friend or colleague.

Have a great week — and keep experimenting.

Kevin