Experimental Mind #190: absence of proof
Your overview of interesting reads, events and jobs for the experimental mind.
Hi, enjoy this edition full of interesting reads you might have missed, job opportunities, upcoming events, a quote to think about and something that made me smile.
*Fullstory: 4 Best Practices for Designing for Tabbed Browsing
Opening links in new tabs can harm the user experience. Learn how to determine when a site link should open in a user’s existing tab, and when it makes sense to open a clicked link in a new tab. [Link]
🔎 Interesting reads you might have missed
Jonathan Fulton wrote an A/B Testing 101: What I wish I knew about AB testing when I started my career. He covers a bit of everything: randomisation, feature-flagging, metrics, statistics, etc.
Itamar Gillad shares a chapter from his new book on how organisations can adopt Evidence-Guided Development.
Ronny Kohavi shares an example from his days at Amazon where a feature which showed to have impact during experiments, got quite some pushback from another team. Can I A/B test that? Yes, with few exceptions.
Benjamin Elbers, Data Scientist at Spotify explains what to do when you can not include a control group in certain situations: how do they measure the feature impact? “One possible answer is to run an encouragement design. The core idea of an encouragement design is to assign the treatment to the entire population that we’re testing on, but randomize the encouragement to use the feature.”
Timothy Chan asked Ronny Kohavi some common experimentation related questions in this Ask Me Anything style interview.
“Variance reduction methods like CUPED are designed to work for experiments that target mean or average treatment effects. They are not designed to work with quantile treatment effects, such as the median or 90th percentile impact of an experiment.” Zach Flynn from Udemy shows how they now also reduced the variance of quantile treatment effects.
Lucia van den Brink asks Georgi Georgiev some good questions about sequential statistics.
The Adevinta / Marktplaats team built Fisher: “a python package that enables our Data Scientists to run straightforward hypothesis testing and to produce comprehensive reports with very few lines of code. Fisher supports the entire analysis workflow: from fetching experiment meta(data), to applying standard data cleansing procedures and transformation, to generating the final report. It integrates neatly into our teams’ ways of working by connecting to Jira, Github and different databases, rendering the entire analysis workflow close to fully automated.”
Seth Godin on the absence of proof:
… There’s more proof in the world than ever before, not less. It’s no longer, “I’ll believe it when I see it,” but instead, “I am confident enough to change my mind and informed enough to do the math and understand the concepts.” We have proof about mathematics, about black holes and about the efficacy of vaccines. We have statistical proof of what people click on and how humans respond in a double-blind test.
This doesn’t make belief less important. In fact, as proof shows up in one area, it simply gives us the opportunity to bring belief somewhere else, somewhere we can put it to good use.
🚀 Job opportunities
Looking for a new challenge in experimentation? Find 100+ experimentation related jobs on ExperimentationJobs.com. These jobs are from all over the world, on-site, fully remote or hybrid. Take a look and start pivoting your career.
This week's featured roles:
Senior CRO Analist at ClickValue (Amsterdam, Netherlands)
Senior Conversion Optimization Manager at Zolar (Germany)
Analytics & Optimization Consultant at Valtech_ (Copengagen, Denmark)
Staff Product Analyst – Experimentation at Eventbrite (Spain)
Digital Experimentation & Optimization Team Lead at Zooplus (Madrid, Spain)
Are you hiring? Submit here
📅 Upcoming events
12-14 Sep: ConvEx ’23 by VWO (virtual)
18-22 Sep: Experiment Nation Conference (virtual)
19-21 Sep: Nonprofit Innovation & Optimization Summit (Dallas, USA)
21-22 Sep: Advances with Field Experiments (Chicago)
See all experimentation related conferences & events 2023.
💬 Quote of the week
“Research, experimentation, and learning are very troubling notions for the executive who wishes the product org to be output-focused, efficient, and compliant.” — Itama Gillad
😅 This made me smile
👍 Thanks for reading
If you are enjoying the Experimental Mind newsletter please share this email with a friend or colleague. Looking for inspiration how to best do that? Take a look at what others are saying. Have a great week — and keep experimenting.