I've always had an issue with this one. "Key takeaway: if you want to make sure your results are reliable, rerun your experiments".. Yes, replication is gold standard. But you're inflating false positive rate in a single replication. So, let's say you had a test result that came back with a negative signal and you really believed in the feature because of research or whatever, so you re-test assuming it was a false positive. This time, result is neutral, but, of course, no stat sig. Which result? So you treat again... Now, time and samples are confounding and we're not even really replicating. It may be minor, or it could be a huge impact to reaction to your treatment. 🤷♀️
I've always had an issue with this one. "Key takeaway: if you want to make sure your results are reliable, rerun your experiments".. Yes, replication is gold standard. But you're inflating false positive rate in a single replication. So, let's say you had a test result that came back with a negative signal and you really believed in the feature because of research or whatever, so you re-test assuming it was a false positive. This time, result is neutral, but, of course, no stat sig. Which result? So you treat again... Now, time and samples are confounding and we're not even really replicating. It may be minor, or it could be a huge impact to reaction to your treatment. 🤷♀️
Agreed, it all depends on how you use replication! Good applications of replication are:
- using replication for impact measurement
- using replication to assess the quality of your experimentation, and adjust it for future experiments.
Bad application of replication: Change the decision about shipping an experiment based on replication.
Fantastic summary. Thanks!
Are the talks available online?
No, the talks are not recorded/published. Of course, you can always reach out to the individual presenters for more information.