Goff on the RCT Agenda

Brian Goff’s piece on the cost disease is one of my most-read guest posts. Now he’s written some thoughtful commentary on my recent piece on the RCT agenda. Enjoy!

---

Bryan,

Your post regarding the ideological presuppositions of RCTs is very insightful and important (looking forward to the book). It seems as though you and I are thinking about a lot of the same things. I’ve had similar thoughts about the broader set of applied micro-econometric, quasi-experimental methods (QEM, “Mostly Harmless” a la Angrist) that are so popular (DID, regression discontinuity, propensity matching, …). On one level, they are all worthwhile and interesting statistical methods, but they definitely seem to be employed in the uni-directional ideological way that you indicate with few exceptions (a couple that come to mind are articles by Kevin Grier and coauthors using “synthetic controls”: Venezuela and South America.)  In many ways, RCTs, QEM, or the more general fixation on “efficiency” by economists seems almost always to fall prey to the desire to be “knob twisters” as Buchanan & Tollison liked to call them.

Even on purely statistical/economic grounds, both RCTs and the QEM have been promoted as if there were no tradeoffs to them.  The QEM stuff sometimes eliminates large parts of the variation central to the question at hand without the researchers being aware of it.  John Cochrane has had some stinging criticisms of this practice on his Grumpy Economist blog such as in this post on article about inflation expectations and spending  (along with linking to a not so favorable review by Francis Diebold of Angrist's "Mostly Harmless Econometrics" -- although Diebold seemed to under-appreciate how influential this stuff has been.) Cochrane refers to his own experience of sitting through countless seminars with DID method using “100 fixed effects …”

All statistical approaches face tradeoffs, just like everything else. RCTs and QEMs that try to imitate RCTs have value, but their narrowness is a real limitation.  The lack of external validity or scalability is a substantial problem.  (I've often joked about the archetypical RCT,  DID or natural experiment based on something like a policy choice of the Sheboygan City Council that supposedly shows us how we need to adjust national policy.)  These issues get treated like quality adjustments in price indexes. Widely acknowledged, then widely ignored. The "external validity" tests are frequently, at best, unconvincing, and are, at worst, a joke.  It is ironic that in her TED talk, speaking of the virtues of RCT and the example of incentives for nurses, Duflo acknowledges that the buy-in of hospital administrators matters, however, that conclusion wasn’t established through a RCT.

Banerjee (and Duflo in her TED talk) and other RCT-only advocates (including in biomedicine) as well as Angrist and QEM-only advocates speak as if the their methods are free of trade-offs. It is sometimes hard to know what is arrogance, what is ideology, and what is a combination of the two.  Banerjee’s statement’s about the “worthless” observational studies no doubt has people like William Easterly in his crosshairs, but such a strong statement is just silly and certainly doesn’t take into account some enormously important advances in knowledge that came about from (relatively simple) observational studies. Examples include the link between lung cancer and smoking or the link between the HIV virus and AIDS (ironically, an eminent UC Berkeley molecular biologist wrote a book back in the 90s called Inventing the AIDS Virus in which he dismissed the link largely because it had not been established in lab, only with observational evidence). Observational studies, even when conducted very well, have their limitations also, but to forward the idea that they are nearly worthless is both pretentious and harmful.
   

Coupled with all of this is the emergence of a nearly "theory-free" kind of economics. We just need RCTs and QEMs, and then we can have purely "evidenced-based" decision-making. Lars Hansen takes that whole idea to task, which is quite revealing given that he's one of the most eminent math stat minds in our profession.  Beyond providing a way of thinking of the statistical evidence or generating questions for statistical evidence, organized thinking (theory) is important because it is so difficult to arrive at definitive statistical answers to many important questions. I spent most of my career doing applied statistical studies, but the experience of listening to organized thinking about a wide variety of problems and then hearing Buchanan, Tullock, Tollison, Crain, Shughart, Grier, and others challenge those ideas during weekly Public Choice seminars was invaluable.

This is not science.