Behavioral Insights: what data can’t do & Semmelweis relfex
Hi there! You're receiving this email because you signed up to Behavioral Insights, a weekly newsletter by Kevin Anderson at the intersection of experimentation, analytics and behavioral science. Thank you for being here.
Behavioral Insights.
👋 Hi, lot's of interesting reads in this week's Behavioral Insights. Ranging from discussing what data can't do, Uber's data infrastructure to optimizing consent opt-in rates. Pick what looks interesting and let me know what you think.
1. What data can’t do ⛔
Article from the New Yorker covering the contents of two new books about data and statistics: 'Counting: How We Use Numbers to Decide What Matters' by Deborah Stone, which warns of the risks of relying too heavily on numbers, and 'The Data Detective' by Tim Harford, which shows ways of avoiding the pitfalls of a world driven by data. Remember: “To count well, we need humility to know what can’t or shouldn’t be counted.” Read the article
2. Measuring what matters 🔢
5 points to keep in mind when deciding on what to measure and track. Metrics should be: actionable, comparable, understandable, relevant, and of course, measurable. Read the article
3. Be data-conscious, not data driven 💬
In this blogpost, the Financial Times analytics team is making the claim we should be "aiming to be a values-driven but data conscious organization ... The first step we can make toward this is to stop using that phrase in our conversations." Read the article
4. Don't over-rely on A/B testing ✨
It can lead to deprioritizing disruptive innovation and only focusing on short term metrics. Read the article
5. How the New York Times A/B tests their headlines 📰
Tom Cleveland has been scraping data from the NYT front page and their official API. In this post he will tell you: how many articles are A/B tested?, how many headlines are tested for each article?, does it help?, and how different are the headlines? Read the article
6. Applied causal inference at Lyft 🚖
Keynote by Sean Taylor from Lyft at the Causal Data Science Meeting 2020 on how they apply causal inference techniques to drive impact at Lyft. Watch presentation
7. Uber’s journey toward better data culture from first principles 🚕
Uber realized they needed a holistic approach with data. In this article the principles underlying the infrastructure are being described. Interesting ideas on how to solve for data quality issues. Read the article
8. Experimentation at Tubi 📺
I had never heard of Tubi (not available in the EU), but this streaming platform is doing some interesting things around experimentation. "In this post, we’ll cover the three components we built to support experimentation and decision-making: (1) the experiment engine at the heart of the system, (2) the UI to democratize experimentation across the entire company, and (3) the automated analysis and experiment QA methodology. We’ll finish with some of the biggest lessons and surprises we learned along the way." Read the article
9. Alerting in Microsoft’s experimentation platform ⚠️
Article on how Microsoft uses alerts to help them identify and rectify issues. Read the article
10. Semmelweis reflex explained 🧼
"We are generally quite reluctant to change the way we think and behave: old habits die hard. This tendency is elegantly described by the concept of the Semmelweis reflex, the instinctive rejection of new and unwelcome ideas. The story behind this concept is a tragic example of the biases in human thinking." Read the article
11. Impact of iOS14 on how Facebook measures ad results 💥
"Facebook quietly posted a blog noting that it would no longer support holdout tests to measure conversion lift or enable A/B tests that include a control group for self-serve conversion lift studies." Read the article
12. Optimizing GDPR opt-in rates 📈
The analytics team from DHL shares how they optimized for consent opt-in rate. But, I am not sure if visitors also know what they have chosen. That's the hard part to know when trying to optimize these numbers. Read the article
😀 Fun of the week 🍪
Trying to reject cookies IRL. Really funny video.
As always, if you're enjoying this newsletter, I'd love it if you shared it with a friend or two. You can send them here to sign up.
I try to make it one of the best emails you get each week, and I hope you're enjoying it.
And should you come across anything interesting this week, send it my way! I love finding new things to read through members of this newsletter.
Until next week, keep experimenting.
— Kevin