Experimental Mind: Headspace, Comparing Tools, Cinnamon, No Evidence, Excel Errors...

🔎 What I've been reading this week

How Headspace evaluated their personalised recommendation system, Content Customizer, via A/B testing.

... it is essential to evaluate the effectiveness of the recommendation system as a whole before deploying it to production. Online Controlled Experiments help us to assess our system’s impact with statistical evidence.

In his Accelerating Innovation with A/B Testing class, Ronny Kohavi invited 3 vendors to present their solution. Another four vendors also shared a slidedeck with answers on specific questions around features and pricing. Nice starting point for when you are looking into specific tooling.

I really enjoyed listening to this podcast where Lukas Vermeer talks about kickstarting experimentation within an organization. And I am not saying that because he's my manager. 😅 

Emily Thompson writes

... before we completely dump “Data Scientist” as a role in industry. I thought it might be worth breaking down where the confusion stems from in the first place.

... Not really, the real finding is cooler: Folks were told to smell a vial of cinnamon & 50% were told it was a creativity drug. The placebo boosted creativity! Why? People needed permission to be creative & reasons to feel competent

Convert created this A/B testing market share explorer tool that gives a clear picture of the A/B testing landscape. You can see the distribution of market share across many countries. Two things I don't like about it:

1. It's unclear how this data is collected. (I asked)

2. Pie charts... Ranked horizontal bar charts would be better

... it doesn’t work in real life. In real life, there is no such thing as a state of “no evidence” and it’s impossible to even give the phrase a consistent meaning.

Ouch ...

The UK COVID contact tracing effort saved to an old Excel file format (.XLS) rather than the new one (.XLSX). The smaller row limits resulted in names being dropped & they were never contacted by tracers.

🚀 Job opportunities

Does working for companies like Vista, RTL, ANWB, VodafoneZiggo, IKEA, Netflix, Apple, Adidas, Gitlab, Stripe, Just Eat, Skyscanner, Lego, and Spotify, sounds like fun? The job board currently has 17 open roles. Take a look.

These are the latest featured jobs:

• [Dutch] Conversie Manager at Online Dialogue (Utrecht, Netherlands)

• Principal Analyst, Experimentation at Vista (remote EU)

• Senior Experimentation Strategist at Speero (remote)

• Product Analytics & Data Lead at Stitch Money (remote NL/Cape Town)

Is your company hiring? Post your open roles to the job board. Readers of this newsletter can post for free, by using this coupon code at the last step: SUBSCRIBER200. In case of any questions, just drop me an email.

📅 Upcoming events

This is a running list of upcoming events:

• SUPERWEEK (Jan 31-Feb 4, 2022, Hungary)

• MeasureCamp North America (Feb 26, online)

Do you know of an event that should be on this list? Let me know.

💬 Quote of the week

"It's not an experiment if you know it's going to work." — Jeff Bezos

😉 Fun of the week

What do you want for Christmas? Please be realistic. [via Magda Baciu]

🔁 Any feedback?

I like to hear what you think of this newsletter, Share your feedback, it takes less than a minute.

🙏 Thanks for reading

If you're enjoying the Experimental Mind newsletter, you can buy me a coffee or beer (in case you rather use Paypal, you can send it to 'kevin@experimentalmind.com'. Thanks to the 17 people who have already done it. Much appreciated.

And share this email with a friend or colleague. I would love for this community of experimenters to grow. You can point them here to subscribe.

Have a great week — and keep experimenting.

Kevin