Experimental Mind: iteration, awards, autocorrelation, overtracking, CNN+ ...

🔎 What I've been reading

This is the first letter by new CEO Andy Jassy and it's all about iteration:

People often assume that the game-changing inventions they admire just pop out of somebody’s head, a light bulb goes off, a team executes to that idea, and presto—you have a new invention that’s a breakaway success for a long time. That’s rarely, if ever, how it happens.

One of the lesser known facts about innovative companies like Amazon is that they are relentlessly debating, re-defining, tinkering, iterating, and experimenting to take the seed of a big idea and make it into something that resonates with customers and meaningfully changes their customer experience over a long period of time.

Submit your case now for the 2022 Experimentation Culture Awards. Deadline is May 14th.

The point of this story is to illustrate that the Dunning-Kruger effect has nothing to do with human psychology. It is a statistical artifact — an example of autocorrelation hiding in plain sight.

Tracking users we cannot treat (called overtracking) affects the variance of our metrics and dilutes the treatment effect, making its detection harder.

Alberto Savoia explains how CNN could have tested this concept earlier.

If you build It, they will come ... NOT. After just a few weeks, CNN+ shut down. This idea would have been SOOOOO EASY to pretotype before pouring in $100M (or more.)

Interview with 2021 Nobel Prize recipient in Economics, Guido Imbens.

Testing is a tool for measuring the effect of a change in isolation. Launch tests are an important use of that tool, but only one use.

... we run 500 hundred simulated A/A tests every day using data accumulated from the last seven days. Each simulated A/A test randomly splits users into control and treatment groups and calculates the metrics of the two experiment groups in the same ways as the production metric calculation pipeline.

Cassie Kozyrkov shares the next 3 out of 10 differences between amateurs and professionals.

Kaartje2go deed het. En Arnoud Huberts legde op Emerce Conversion uit hoe en waarom. Kaartje2go is ook op zoek naar drie nieuwe toppers: zie hieronder.

🚀 Job opportunities

All featured roles:

• CRO Specialist at Randstad Global (Amsterdam, NL)

• Webanalist at Kaartje2go (Zwolle, NL)

• Conversie Optimalisatie Specialist at Kaartje2go (Zwolle)

• Conversie Optimalisatie Lead at Kaartje2go (Zwolle, NL)

• Digital Analyst at RTL (Hilversum, NL)

Also checkout the job board for more open roles in the field of experimentation, CRO and analytics.

📅 Upcoming events

This week:

• 27 April: Experimentation Elite (London)

• 28 April: Test Business Ideas by David J Bland (virtual)

Next months:

• 14 May, MeasureCamp (London)

• 18-20 May: Measuring Behavior (Krakow)

• 19 May: MeasureBowling (12+ cities)

• 21 May: MeasureCamp (Amsterdam)

• 31 May: Accelerating Innovation with A/B Testing (virtual)

• 8-9 June: Advances with Field Experiments (Chicago)

• 9-10 June: NeuroPsychoEconomics (Lille)

• 15 June: Growth Marketing SUMMIT (Frankfurt am Main)

• 16-18 June: Marketing Science Conference (virtual)

• 23-24 June: Outperform Conference (Amsterdam)

• 25 June: MeasureCamp (Paris)

• 🆕 7 July: Experimentation Culture Awards (virtual)

See the full overview of events to find 15+ other events and conferences for you and your team.

💬 Quote of the week

“We want people who like to experiment and tinker, and who realize launch is the starting line, not the finish line." — Amazon's 2021 Letter to Shareholders

😉 Fun of the week

The latest Marketoonist on product-market fit.

☕ Thanks for reading

If you’re enjoying the Experimental Mind newsletter and want to say thank you? Buy me a coffee (or a beer). Or share this newsletter with a friend or colleague.

Have a great week — and keep experimenting.

Kevin