Experimental Mind: GitLab, Change, Data PM's, DoorDash, Risky Bets ...

🔎 What I've been reading

This documentation describes GitLab's Growth engineering process for running experiments. Extensive read.

Great LinkedIn post by Matt Gershoff sharing his top 10 non-controversies in A/B testing. Number 10: Two Tailed vs One Tailed Tests.

Eric Weber on the upcoming role of Data Product Manager, also for experimentation:

... the ability to have a real product experimentation platform is transformational for many companies. An experimentation platform enables companies to identify causal relationships between product changes and user behavior (or other metrics). This space is *extremely* competitive right now, and I’m a huge fan of many startups that have launched here. For data product managers, experimentation platform requires a vast array of context: from experimental design, to cohort allocation, to logging, to guardrails metrics, to experiment readouts.

Working in the experimentation space can sometimes feel like an uphill battle. Reminder from Seth Godin: persist. This is how change happens.

Focused, persistent community action is how systems change. And systems concretize and enforce cultural norms.

If you care, keep talking. Keep acting. Stay focused. And don’t get bored.

DoorDash's centralized library for statistical analysis of experiments.

We needed a way to enforce standardization and facilitate knowledge sharing to improve the velocity of our experiment analysis.

The Statsig Pulse results page offers a snapshot of all the metric movements driven by an experiment. Sometimes, a brief scan of the color coded score card is enough to validate that all metrics…

I'll be one of the jury members at the 2022 Experimentation Culture Awards. Join the award session and listen to inspiration stories.

🚀 Job opportunities

All featured roles:

• Conversie Optimalisatie Specialist at Kaartje2go (Zwolle, NL)

• Conversie Optimalisatie Lead at Kaartje2go (Zwolle, NL)

• CRO Specialist at Randstad Global (Amsterdam, NL)

• Digital Analyst at RTL (Hilversum, NL)

• Webanalist at Kaartje2go (Zwolle, NL)

Also checkout the job board for more open roles in the field of experimentation, CRO and analytics.

📅 Upcoming events

Final call for joining Ronny Kohavi’s new class, where he will teach you everything he knows about A/B testing in 5 two hour sessions. First session is May 31. Subscribe via this link to receive a $50 Amazon gift card.

All upcoming events in the next months:

• ⏩ 31 May: Accelerating Innovation with A/B Testing (virtual)

• 🆕 3 June: TLC: Get Psyched for UX (virtual)

• 8-9 June: Advances with Field Experiments (Chicago)

• 🆕 9 June: DiDo#39 Ga voor (CRO) goud! (Utrecht)

• 9-10 June: NeuroPsychoEconomics (Lille)

• 15 June: Growth Marketing SUMMIT (Frankfurt am Main)

• 16-18 June: Marketing Science Conference (virtual)

• 25 June: MeasureCamp (Paris)

• 7 July: Experimentation Culture Awards (virtual)

• 🆕 4-6 Aug: Digital Elite Camp (Estonia)

Second batch of tickets will release soon:

• 🎫 18-20 Nov: Conversion Hotel (Texel, Netherlands)

See the full overview of events to find other events and conferences for you and your team.

💬 Quote of the week

Amazon's Risky Bet, according to BusinessWeek in 2006.

Jeff Bezos @JeffBezos

I have this old 2006 BusinessWeek framed as a reminder. The “risky bet” that Wall Street disliked was AWS, which generated revenue of more than $62 billion last year. https://t.co/ccF0nCOkhu

☕ Thanks for reading

If you’re enjoying the Experimental Mind newsletter and want to say thank you? Buy me a coffee (or a beer). Or share this newsletter with a friend or colleague.

Have a great week — and keep experimenting.

Kevin