Behavioral Insights: optimal stopping, peeking & the rise of the mafia
Behavioral Insights.
In this weeks email, I'll discuss three articles about experimentation: optimal stopping, peeking and an academic paper that links the worlds first experiment to the rise of the Sicilian mafia ...
1. End experiments early with optimal stopping
There are many benefits to ending experiments early when possible: you can run more tests and you can prevent or benefit from possible losers or winners earlier. But there are also many risks when you stop your test to early. In this article, Netflix's experimentation team shows a way you can stop experiments earlier. With these kinds of features, Netflix can accelerate their current experimentation workflow, expedite product innovation, and ultimately bring the best experience and delight to customers. [read more]
2. The fourth Ghost of Experimentation: Peeking
When conducting A/B tests, It is tempting to check your results whilst the test is running to see how it is performing, this is also known as ‘peeking’, but in doing so you are giving this ghost more opportunities to fool you into thinking an ineffective treatment has had an impact. In the latest article in a series on Statistical Ghosts, Skyscanner's experimentation team shows ways to ‘peek safely'. [read more]
3. Origins of the Sicilian mafia
In the late eighteenth century James Lind, a British naval officer and surgeon, performed, according to many, the first controlled therapeutic trial of his time. He discovered that citrus fruits cured scurvy. This breakthrough caused a large increase in the
demand for oranges and lemons. This article shows that mafia presence is strongly related to the production of oranges and lemons. Demand for Sicilian lemons booms > Mafia emerges to protect lemon profits. [read more]
Happy experimenting!
— Kevin
ps. This Thursday I'll be speaking at Marketing Insights Event (MIE). Use the promo code 'MIE2019' to get a discount.