This is such an important book that I hope it goes straight to the top of your summer reading list:
Buy it at the US Amazon store or wherever you buy books.
In chapter after chapter, Andrew Leigh gives detailed accounts of randomly controlled trials destroying commonly held beliefs. What we think should work often doesn’t. RCTs have influenced and shaped our world more than we think, yet we are just starting to insist on them.
Build an Experimental Culture
Google, Amazon, Facebook, Capital One - these companies are constantly running hundreds of experiments with randomized control groups. Scott Cook, chairman of Intuit, talks about replacing executive decisionmaking with randomly controlled trials.
There are three steps here:
Be aware of the value of RCTs.
Create a culture that values RCTs and knows when to use them.
Build strong muscles for designing RCTs and evaluating their data.
Step three is probably the hardest. Actually interpreting statistical data is beyond the capabilities of most statisticians. They are easily fooled by biases (as I showed in my Apple watch post recently). In fact, a sharp researcher recently discovered flaws in the way randomization was done in thousands of experiments, forcing many of those researchers to re-evaluate their data. It’s harder than you think.
I think it’s valuable to learn to do mini-trials that are quick and give a first-order answer that you can use to move forward with while continuing to refine your questions and designing trials. RCTs should be part of any agile approach to government, institutions, and companies. Even families and individuals can use them.
Do you? Have you ever done an RCT to answer a question?