Experimental Mind #273
Your weekly overview of interesting reads, events and jobs for the experimental mind.
This week: Amazon links metro ads to sales, Meta organizes decisions at scale, and Conversionista shows no one cares if your stats are Bayesian or frequentist: just the impact. Plus: expert generalists, betting mindsets, a possible Monetate–SiteSpect deal, and a new career spotlight from someone who breaks things for a living. Go!
➡️ Introducing The New SiteSpect UX
SiteSpect launched a sleek new UX: faster, cleaner, and built for self-service. With improved navigation, smarter test creation, and centralized tools, it's designed to make experimentation easier for everyone.
Link to article from Sitespect
Thanks to Convert and Sitespect for their continued support.
🔎 Interesting things you might have missed
A first-of-its-kind experiment to measure the impact of out-of-home advertising
Amazon researchers executed a geo-randomized field experiment on metro and commuter rail ads, surveying locals to determine who saw the exposure and measuring sales uplift across neighborhoods. They found that high-ad-exposure areas saw a clear sales increase over a four-week campaign, demonstrating a causal link between out-of-home advertising and consumer behavior. >>>
____
Bayesian or Frequentist engine for A/B test? Who cares?
Conversionista ran a study with 39 e‑commerce decision‑makers and found it doesn’t matter whether results are shown in Bayesian or frequentist formats. Choices to implement tests were driven almost entirely by the magnitude of improvement, not the statistics used. >>>
____
Meta’s centralized approach to decision record-keeping
Meta has a standardized dataset categorized by decision units, each representing a specific point in time when a decision is made regarding whether to ship or not ship a particular feature or product. This approach enables easier analysis of past experiments, identification of trends, and more informed decision-making across the organization. >>>
____
Expert Generalists
Expert generalists are people with broad curiosity and deep skills in a few areas, who collaborate across domains with humility and a focus on real customer needs. Their ability to connect dots and adapt makes them especially valuable in complex, fast-moving environments. Sounds like an experimental mind to me. >>>
____
The effectiveness equation by Google
Google released a report offering strategies to optimise impact measurement and build a stronger business case for marketing budget. >>>
____
Can you trust your data from ad platforms?
A new study reveals a key flaw in A/B testing on platforms like Meta and Google: algorithms assign different user groups to variants, skewing results. This means test outcomes may reflect audience differences, not ad effectiveness. Be cautious. >>>
____
What bookmakers can teach us about predicting the future
Every decision is a bet, just like bookmakers, we weigh probabilities under uncertainty. Adopting this mindset helps sharpen assumptions, improve calibration, and make smarter, more informed choices. >>>
____
Monetate buys SiteSpect (?)
Monetate secured funding from o15 Capital Partners to refinance debt and acquire SiteSpect. There is no reference on the company websites (yet). >>>
____
Last week’s most clicked item:
A.I. might take your job: here are 22 new ones it could give you
A New York Times Magazine piece outlines three upcoming job categories in the era of AI: roles centred around trust (like AI ethicists and auditors), integration (embedding AI into business), and creative direction roles (guiding AI’s creative output). >>>
💬 My Experimentation Career Journey
Today’s interview is with Matthew Hearn, a QA/UAT Tester with 10+ years’ experience and currently looking for a new opportunity.
I test everything from design tweaks to logic changes—basically, I try to “break things” before customers do. If something’s off, I’ll catch it. — Matthew Hearn
Read the full interview with Matthew
🚀 Job opportunities
Find 100+ open roles on ExperimentationJobs.com. This week’s featured roles:
Senior CRO Manager at Growth Shop (Canada)
Product Growth Engineer at Fyxer.ai (London, United Kingdom)
Senior Software Engineer, Experimentation Platform at GetYourGuide (Berlin, Germany)
Experimentation Lead at News UK (London, United Kingdom)
Senior Growth Marketing Manager at Buffer (Remote)
📅 Upcoming events
A running list of upcoming events. Will I see you in Amsterdam or at the virtual Women in Experimentation summit?
1 Jul: Berlin Experimentation Meetup #9 (Berlin, Germany
👋3 Jul: EXL Meetup (Amsterdam, Netherlands)
🆕6 Jul: Experimentation London #6 (London, UK
🎁7 Jul: Accelerating Innovation with AB Testing (virtual, get $500 off)
👋9 Jul: Women in Experimentation Summit 2025 (virtual)
9 Jul: CRO & Growth Meetup (Paris, France)
🎁24 Jul: Testing Business Ideas with ChatGPT (virtual, get $50 off)
🎧 Podcast of the week
John Blackman, a 91-year-old retired electrical engineer, shares how he used Claude and Replit to build a complex application for his church’s community service events—with no prior software development experience.
If you like this episode, make sure to subscribe to the podcast. Or discover other podcasts via the Experimental Mind curated podcast feed.
💬 Quote to think about
I think he might be right.
All the best people in experimentation and A/B testing have either:
A) travelled extensively around the world + love to travel
B) live in a different country than the one they were born in
Different cultures and countries just forces you to look at experimentation, optimization and testing with a different view of the world.
There is no option C.
— Hristian Kambourov
📣Shoutout of the week
Thanks Chris.
Add your own shoutout or read what others are saying.
👂How was this edition?
Let me know by clicking one of these options: Excellent | Great | Good | OK | Meh
Or even better, simply hit reply. I read every email.
Have a great week — and keep experimenting.
Thanks, Kevin