[Summer Series] Ronny Kohavi: My Experimentation Career Journey
Your overview of interesting reads, events and jobs for the experimental mind.
Hi, we will end the Summer Series with one of the most renowned experimentation experts: Ronny Kohavi. Read on and learn about his career in experimentation so far. Next time the newsletter will be back to its normal format.
*Analyzing Experiments with GA4: The Good, The Bad, and The Practical
Convert brings together an expert panel of practitioners for a talk about their best, most tried & true tips when it comes to analyzing experiment data in GA4. Join this free, Zoom session on August 28th.
💡Ronny Kohavi: My Experimentation Career Journey
I love helping companies become more data-driven by using the scientific method, and I have done that successfully for over 20 years at Microsoft (where I headed ExP, the Experimentation Platform), at Amazon, and in consulting engagements (but failed to do that at Airbnb, where Brian officially disclosed recently that he will not do A/B tests in most scenarios, but instead ship on “pride.”)
I co-authored the book Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing with Diane Tang and Ya Xu. The book sold over 20,000 copies in English, and was translated to Chinese, Japanese, Korean, and Russian.
I received the Individual Lifetime Achievement Award for Experimentation Culture in 2000. My papers have over 55,000 citations according to Google Scholar, and several of my papers are in the most cited A/B testing papers.
What is your current experimentation role and what do you do?
I consult in A/B testing and teach a 10-hour interactive Zoom class with Maven: Accelerating Innovation with AB Testing. If you’re interested in my class, here is a promo code: EXPMINDBENDER for the first 10 people to use it.
How did you enter the experimentation space? What was your first experimentation related role?
When I was at Amazon, back in 2003, we ran controlled experiments with our platform called Weblab, which was one of the teams I led.
I remember looking at the statistics of our success/failure rate and realizing that more than 50% of our ideas failed to move the metrics they were designed to improve. That was a big epiphany for me; very humbling.
How did you start to learn experimentation?
I learned some from statisticians who worked with me, and from reading multiple statistical books. I then co-authored one of the early papers on experimentation in 2007 (http://bit.ly/ExPHiPPO), which was one of the earlier mentions of HiPPO, the Highest Paid Person’s Opinion: http://bitly.com/HIPPOExplained.
In the full interview you will also learn what Ronny does to keep up with the ever-changing industry. He also shares his recommendations to someone who is looking to join the experimentation industry.
🚀 Job opportunities
Looking for a new challenge in experimentation? Find 100+ experimentation related jobs on ExperimentationJobs.com. These jobs are from all over the world, on-site, fully remote or hybrid. Take a look and start pivoting your career.
This week's featured roles:
Sr CRO Strategist, Technology and Services at Three Ships (Charlotte/Raleigh, USA)
Principal Product Manager, Analytics & Experimentation Lead at Skyscanner (Edinburgh, United Kingdom)
Experimentation Manager at DocuSign (San Francisco, USA)
Senior Product Manager, Experimentation Platform at BILL (USA)
Growth Marketing Manager, Conversion Rate Optimization at Hinge Health (USA)
Are you hiring? Submit here
👍 Thanks for reading
If you are enjoying the Experimental Mind newsletter please share this email with a friend or colleague. Looking for inspiration how to best do that? Take a look at what others are saying. Have a great week — and keep experimenting.