Experimental Mind: ask teams to improve metrics, not to ship features
Hi there! You're receiving this email because you signed up to Behavioral Insights, a weekly newsletter by Kevin Anderson at the intersection of experimentation, analytics and behavioral science. Thank you for being here.
Experimental Mind.
👋 Hi, in this week's Experimental Mind newsletter 8 articles to read and 23 job opportunities to explore. Enjoy.
1. Ask teams to improve metrics, not to ship features
Chetan Sharma will start a series of blogposts that "... are aimed at executives who are hesitant to ramp up more experimentation, and product leaders who want to get more buy-in to run experiments". In this first post he talks about metric strategies and why they are crucial in boosting the number of experiments. Read the article
2. The death of behavioral economics
I don't like articles that have 'death of ...' titles, but Jason Hreha does have a point here. Two actually: core findings are failing to replicate & interventions are surprisingly weak in practice. Read the article
3. How Wish A/B tests percentiles
This blogpost explains why and how Wish is evaluating A/B tests with percentile metrics. Read the article
4. Effective field-testing strategies for AI models
On how to measure the success of an AI intervention in quantifiable outcome. Read the article
5. Things to report in an A/B test analysis
Statistical significance, confidence interval and statistical power (although that is more relevant before the start of your experiment). Read the article
6. Five pillars to build an experimentation program
According to Alex Birkett it's:
Trustworthy Data
Human Resources
Leadership and Strategic Alignment
Experimentation Technology
Education and Cultural Buy-In
Read the article
7. Paper 'It takes a Flywheel to Fly: Kickstarting and Growing the A/B testing Momentum at Scale'
This paper won the best paper award at the Software Engineering and Advanced Applications (SEAA) conference. Congratulations Aleksander, Pavel and Lukas. Read the paper
8. Event: Experimentation Works
This event offers two promising talks from Lukas Vermeer and Simon Elsworth. Lukas will talk about 'Kickstarting and keeping the A/B testing momentum’ and Simon about 'In experimentation, what does ‘career progression’ actually mean?’. Join me at this free event on Thursday 16 September at 16:00 CEST. Sign up here
Interesting job opportunities 🔥
This week's featured jobs:
Head of Experimentation at Zalando, Berlin
CRO Specialist at KPN, Amsterdam
Sr. Experimentation Manager at Zillow, Seattle/Remote
Or check out the other 20 opportunities on the job board. You can add your vacancy as well.
Tweet of the week:
As always, if you're enjoying this newsletter, I'd love it if you shared it with a friend or two. You can send them here to sign up.
I try to make it one of the best emails you get each week, and I hope you're enjoying it.
And should you come across anything interesting this week, send it my way! I love finding new things to read through members of this newsletter.
Until next week, keep experimenting.
— Kevin