Experimental Mind: Top-Down Projects

🔎 Interesting reads you might have missed

Itamar Gilad shares how to deal with product ideas landing from the top. Or are you in the lucky situation that this is never happening in your company?

John Cutler describes a better way to do goal setting:

Goal cascades meet internal needs, not customer needs. They meet the requirement for a planning process at the expense of catalyzing congruence across the organization. ... Ditch the cascade. Create models. Set goals wherever it makes sense to set goals.

Many companies want to 10x their experimentation velocity. The people from Statsig describe 5 techniques that help you do this:

1. Feature Rollouts: auto-measure new feature impact with an A/B test

2. Parameters: remove experiment variants in code to iterate faster

3. Layers: remove hardcoded experiment references from code

4. CUPED: use statistical techniques to get results faster

5. Holdouts: measure cumulative impact/progress without grunt work

Generating meaningful results through CRO and experimentation is tough. While almost anyone can achieve the odd one-off conversion rate uplift, producing wins with any consistency is another challenge entirely. What’s more, many CRO practitioners find that even when they do achieve the kinds of results they’re after, their winning variations fail to perform when served […]

Google recently (slightly) updated their documentation on how to minimise the impact of A/B tests on search ranking. Not new, but a good refresher.

This paper aims to make practitioners of experimentation more aware of imperfections in technology industry RCTs, which can be hidden throughout the engineering stack or in the design process.

The result of these imperfections can lead to a bias in the estimated causal effect, a loss in statistical power, an attenuation of the effect, or even a need to reframe the question that can be answered.

John Ostrowski recommended me the interview with Marty Cagan on Lenny's podcast. They talk about what good product teams do different from feature teams. To me a crucial ingredient in building a culture of experimentation.

🚀 Job opportunities

[DUTCH] Als Data Tracking & Measurement Specialist bij de Bijenkorf vertaal je de data behoeften via tracking implementaties vanuit alle digitale kanalen naar relevante klantdata en maakt dit inzichtelijk via analytics.

Find all open roles on the job board.

📅 Upcoming events

All upcoming events in the next months:

• 20 Sep: Growthhackers Conference (virtual)

• 26-30 Sep: Experiment Nation Conference (virtual)

• 27 Sep: CRAP Talks #19 (London, United Kingdom)

• 13 Oct: Digital Analytics Summit (Amsterdam, NL)

• 🆕 13 Oct: DiDo#40 Results are not Learnings (Utrecht, NL)

• 20-21 Oct: Conference on Digital Experimentation (Boston)

• 25-26 Oct: CXL Live (Austin, Texas)

• 31 Oct: Accelerating Innovation with A/B Testing (virtual)

• 3 Nov: Dutch CRO Awards (Amsterdam, NL)

• 18-20 Nov: Conversion Hotel (Texel, the Netherlands)

Also checkout the full overview of events for next months.

💬 Quote of the week

"... there is no Spotify model at Spotify" — an Agile Coach at Spotify (source)

😉 Fun of the week

How many sprint meetings can we have? (credits: Work Chronicles)

👍 Thank you for reading

If you’re enjoying the Experimental Mind newsletter please share this email with a friend or colleague. Looking for inspiration how to best do that? Take a look at what others are saying.