Experimental Mind: Evaluate LLMs, Spend Patterns, Proxy Metrics...
Your overview of interesting reads, events and jobs for the experimental mind: #192
Hi, before we dive into the latest articles you might have missed, can you help me understand who is reading this newsletter by answering this poll? Thank you.
➡️ Convert: Helping Organizations Run More Tests Across Diverse Channels
Experimentation is an amplifier — improving what’s already working, spotting ideas that aren’t worth rolling out, and unearthing innovation to revamp what’s effective but gradually regressing to the mean. Your experimentation tool should be able to support testing across all these growth motions. [read full article]
💬 Timothy Chan: My Experimentation Career Journey
I’m excited to see experimentation lead the charge on data-driven insights and decision-making into product development. While experimentation is well-established in marketing, and is starting to permeate tech products, what excites me most is to see other industries adopting experimentation.
Read the full interview with Timothy
🔎 Interesting reads you might have missed
How to Evaluate LLMs: A Complete Metric Framework
”At Microsoft, the Experimentation Platform has worked closely with multiple teams to launch and evaluate LLM products over the past several months. We learned and developed best practices on how to design AB tests and metrics to evaluate such features accurately and holistically. In this article, we are sharing the standard set of metrics that are leveraged by the teams, focusing on estimating costs, assessing customer risk and quantifying the added user value.” LINK
Experimenting with Spend Patterns
Jeannette Hallerman from Vista explains how Vista manages spend patterns to both optimize return on marketing investment and maximize test learnings: “One challenge in this experimentation culture is choosing between optimizing spend effectiveness (maximizing return on investment) and optimizing measurement effectiveness (maximizing learnings from the test).“ LINK
[Paper] Choosing a Proxy Metric from Past Experiments
”We introduce a new statistical framework to both define and construct an optimal proxy metric for use in a homogeneous population of randomized experiments.” LINK
Have you ever run the same A/B test twice?
Eline van Baal: “We discovered that many test results even swung in the opposite direction when re-testing. When we broke down the results into segments, the inconsistencies became even more pronounced.” LINK
Experimenters that changed their career
These people recently made a career change: Jurjen Jongejan, Linda Bouwman, Bhavik Patel, Jacob Jarnvall, Max Bradley, Jaiprakash R and Julie Stevenson. LINK
🚀 Job opportunities
Looking for a new challenge in experimentation? Find 100+ experimentation related jobs on ExperimentationJobs.com. These jobs are from all over the world, on-site, fully remote or hybrid. Take a look and start pivoting your career.
This week's featured roles:
CRO Account Manager at Croud (Shrewsbury, United Kingdom)
Product Manager, Product Ops Team at Eneba (Lithuania)
Retention Growth Manager at Ledger (Paris, France)
Marketing Manager, CRO and Website Experience at OpenTable (USA)
Senior CRO Analist at ClickValue (Amsterdam, Netherlands)
Are you hiring? Submit here
📅 Upcoming events
16-17 Oct: CXL Live (Austin, USA)
17 Oct GrowthHackers Conference (virtual)
26 Oct: Conversion Jam (Oslo/Stockholm)
See all experimentation related conferences & events 2023.
👍 Thanks for reading
If you are enjoying the Experimental Mind newsletter please share this email with a friend or colleague. Looking for inspiration how to best do that? Take a look at what others are saying. Have a great week — and keep experimenting.