Experimental Mind: shorter tests

📞 Schedule a call with me

Would you like to talk about an experimentation related topic or simply catch up? I have now hooked up my calendar to Cal.com which makes it really easy for you to schedule a call with me. Try it out. We can talk about whatever you like.

🔎 Interesting reads you might have missed

Georgi Georgiev: running shorter tests is key to improving the efficiency of experimentation as it translates to smaller direct losses from testing inferior experiences and also less unrealized revenue due to late implementation of superior ones.

In this article Georgi Georgiev covers the method required to calculate statistical significance for non-binomial metrics such as average revenue per user, average order value, and others.

Jakub Linowski shares some insight from datasets that show that indeed, on average, people can predict A/B tests.

From the Etsy Engineering team: In order to understand the overall business impact of the changes made across all teams, we need to start thinking of experiments as being part of a larger common system, and look at their results in a more holistic fashion.

Many business don't seem to understand the real impact digitalisation will have on them. Jan Bosch shares many fallancies he encounters, inclduing this one: "Experimentation has no role in product development":

The use of minimal viable products and A/B experimentation is typically non-existent and discouraged in traditional companies. The specification is used as a basis for all development activity and there’s no reason to question it. This brings us to the notion of knowable versus unknowable. Some things are simply unknowable until we try them out. The response of customers to new products or new features is one of these.

To test the impact of these changes, the UK Behavioural Insights Team is conducting a large-scale field RCT with 850,000 metered Water customers. 

🚀 Job opportunities

This week's featured roles:

• 🆕 Data Scientist at Wolt (Stockholm, Sweden)

• CRO Consultant at Conversion (London, United Kingdom)

• Experimentation Platform Advocate at Apple (Raleigh, USA)

• Sr. CRO Manager at IU Internationale Hochschule (remote or Hamburg, Germany)

• CRO Specialist at viaBovag (Utrecht, Netherlands)

More on ExperimentationJobs.com! Are you hiring? Submit here

📅 Upcoming events

• 30 Jan: Accelerating Innovation with A/B Testing (virtual)

• 31 Jan – 4 Feb: SUPERWEEK (Hungary)

See all experimentation related conferences & events 2023.

😄 Fun of the week

Elspeth Kirkman @Karminker

Me, hearing about confirmation bias: I knew it https://t.co/H4kxhXXdTM

👍 Thank you for reading

If you’re enjoying the Experimental Mind newsletter please share this email with a friend or colleague. Looking for inspiration how to best do that? Take a look at what others are saying.

Have a great week — and keep experimenting.