I was in Google Ads for 3 years and got to know the Economics team quite well. They turned out in force to see Joseph Stiglitz, and completely boycotted Luigi Zingales, so that left-liberal bias was definitely there.
Maybe they came out for Nessim Nicholas Taleb; there were 150 people there so I can't be sure!
As for RCT's: I think is somewhat like saying "lefty academics use computers, so computers are bad." The bias you're describing is in the academic establishment, and RCT's are just a tool they happen to use. Eliminate RCTs and they'd advocate for government by some other means.
Then we must agree to disagree. To me he is clearly attacking how RCTs are used. Said another way: he isn’t attacking fire or matches, he’s attacking arsonists
He's just someone who rejects traditional left v. right thinking. "Pro-market is not the same as pro-business." Anyone with an open mind should be curious, if nothing else.
Well, broadly speaking he’s right of center. Your point that he ain’t hard right only reinforces Albert’s (I know: shocking, shocking) point that the Google team is leftist biased.
A rigorous logical induction of causal laws in the social sciences is impossible. The two basic reasons why imitation of the methods of the natural sciences doesn't work are that (1) the most relevant causal factors of interest, namely the mental states of individual human beings, are not available for direct observation by social scientists, let alone precise quantification via comparison against a standard; and (2) natural scientists, unlike social scientists, can inject themselves into the chain of causality to systematically vary one causal factor at a time in the system of interest while leaving all the others constant to see what consequences follow from the singular change. Simply put, social scientists are in no position to do the logical equivalent of controlled laboratory experiments on real societies.
Note also that the system of interest needs to be well-defined, which is why natural scientists derive their theories from highly simplified systems, while treating uncontrolled observations of complex systems as needing to be explained by pre-existing theories (i.e. *abductive* reasoning), not as confirming new theories. Even the randonomized control trial is not a true gold standard of empirical science, since RCT results are valid only for the population as originally sampled. Once the relevant characteristics of a population change, or once one examine a different population, causal factors that only affected a small subset of the original population might loom large in another context.
The best social scientists can hope for is to start with self-evident truths about human beings; that is, facts that are embedded in our very ability to comprehend the world and engage in meaningful conversations about it to others, and thus can't be disputed without self-contradiction. You don't need an experiment to know that human beings are purposeful, for example. Knowing that, an economist can deduce that everything that follows from the logic of purposeful action must be applicable to human societies.
From a socialist perspective the frustrating thing about a logic of purposeful action is that its conclusions are of the form "if causal factor x occurs, then all other things being equal the effect y will follow." Such qualitative results don't give you the timing and magnitude of effect y, nor does it enable you to precisely predict what will happen when all other things aren't equal, and other changes besides x are occurring simultaneously (as is always the case in real societies). People who crave the job of central planner don't find honest economics of any use to their charade of being benefactors to society, so they fake it and pretend that the methods of the natural sciences will empower them to engage in benevolent social engineering. An honest economist, on the other hand, refrains from such a conceit of certainty and pretense of doing "natural experiments" on society, and instead simply reminds everyone that we already know even without having to collect statistical/historical data (and without making up arbitrary assumptions that can't be justified as self-evident truths, another common sin among dishonest economists) that certain x's will invariably lead to results that everyone agrees is worse than not doing x.
You see very similar behavior in the industry sphere, too, where businesses or process owners will become dedicated to a particular way of doing things and demand answers to make that way work better. Pointing out that they would be better off doing it a different way, or that their way of doing things can't work the way they would like at all, that never goes over well.
If RCTs generate high quality medical interventions, why would they not generate high quality economic interventions?
The unit of observation for medical RCTs is an individual person. In most cases the "unintended consequences" of a medical intervention are limited to a single person, and thus RCTs provide a sense of the total effect.
In RCTs of economic interventions the unit of observation is often one person, household, or business, but the total effect will often spread far beyond that unit. The relevant system is the whole economy, not just the person affected directly by the intervention. Unintended consequences may not be observable in each observation and are less likely to be foreseen or even understood as consequences of the initial intervention.
Medical RCTs would not be helpful if we made the unit of observation seperate body parts instead of the whole person. Some pill might lower blood pressure but kill brain activity.
Medical RCTs don't (just) test Drug A vs Drug B. They test Drug A vs a control of doing nothing (usually via placebo).
Caplan is pointing out that Economic RCTs aren't doing that, they are testing Government Intervention A vs Government Intervention B. Where is the test of smaller class size vs not funding government schools? The closest you get is testing public schools vs public charter schools.
Due respect, you are using the acronym RCTs in a very different sense than does Caplan in his piece. You are using it solely in the technical sense of performing a randomized control trial, and claiming that Caplan is opposed to (small letters) randomized control trials.
But if you read the piece, that is NOT what he is saying at all. He is indicting how RCTs are used, the fact that only pro-government-intervention RCTs are published, etc.
To put it in non- economics terms, you are claiming that Caplan is objecting to fire, when in fact Caplan is only complaining how a particular tribe are arsonists in their use of fire.
Isn't it possible that (ironically) this gets the causality wrong? It not that RCTs encourage statist policies, but rather that a bias towards statism leads people to use RCTs (and other types of empiricism) in support of statism.
An example fresh in my mind of a study discouraging statism: research that shows that *affordable* housing is really just *old* housing. Thus strongly implying that if we want more affordable housing what we need is *not* programs to build affordable more affordable housing, but allowing more housing to be built so that existing housing will age and become affordable.
It's possible I've misunderstood what you are saying, but if so, I suspect I am not the only one...
The links to the footnotes in the first chapter DID NOT WORK. Either this should be fixed (in future articles) or the footnotes should not be linked at all.
> They’ll always either find that (a) the treatment “worked” — so government should scale it up; or (b) the treatment “failed” — so government should keep running RCTs until they find a treatment that works.
Isn't this just because arguing for more or less government action is a normative claim outside the scope of the RCT?
"this didn't work, maybe something else works" is a lot less controversial to claim than: "this didn't work, so stop trying"
"More or less government action" is no more a normative claim than "this treatment worked better or worse than the control". Both are assuming some particular end, e.g. improved math scores, is desirable. The question is how to get there, but the big question about whether or not the state can get there is largely ignored in favor of assuming the state can, and then trying to figure out how.
What about the famous oregon health insurance study, which was an RCT? I think it was conducted, among others, by the great economist Amy Finkelstein. In the sense that it proved the moral hazard effect (and subsequent raise in spending) of universal health insurance, which is something that most free marketers use as an argument to advocate for private insurance and or health saving accounts?
This post asserts that you can't treat whole countries like guinea-pigs, so you can't test big questions with RCTs. That may be true for, say monetary policy, which applies at a national scale. But I'm not sure I agree that you can only test markets versus governments at the level of whole countries. Let's say you have coffee farmers in a developing country that accept a price established by the state monopsony buyer of coffee in advance of the harvest. Couldn't you have an RCT that let some farmer cooperatives receive market prices, and others the government set price, and analyse the effects? You might say these results don't settle the big and general question of markets versus governments. But that's true of all RCTs. They cast a bright but narrow beam of light. So the incentives for nurses that work in Nairobi don't translate to Mexico or even other parts of Kenya for multiple devil in the detail reasons. I don't see a clear distinction between big and small questions, and thus which questions are amenable to RCTs and which not.
After over 40 years of research in the field when abhijit banerjee was tasked to create an anti poverty program in India he basically created a negative income tax (more or less). Isn't he admitting that Milton Friedman was correct?
> “natural experiments on markets versus government.” But top journals these days will never let you get away with that. The sample is too small, the randomization too questionable, and the question too big. Way too big.
Sure. 300,000 years of sustainable govt poverty vs. 300 years of vastly larger and vastly larger increase of capitalist prosperity. Coincidence? Maybe. Sure. Maybe these "scientists" could do an RCT on rocks and feathers in water to discover which group is more likely to sink. Science exists in a causal, non-coincidental universe, contra views of QM math which sneak in and then conclude a random metaphysics. Further, causality is the thing/action relation, not the prior event/later event relation. Action is the action of definite things with definite properties. A things actions are that things actions. Action cannot contradict the nature of the thing. Feathers cant act like rocks. Govt cant act like capitalism. They are different kinds of things with, thus, different kinds of actions. The minds of businessmen are not the guns of govt. Production is not force. Marx plus RCT "science" is a pig w//lipstick. Contra the claims of impossibly, value-free social "science," economic activity is value-causing activity. Economics must study values. But the hidden altruism of govt economies is the sacrifice of values. Production and the science of production require values. Rejecting values is rejecting social science. See "What Is Capitalism" by Ayn Rand for more philosophy of economics.
Nice post.
It's the same throughout the academic pyramids, now all rock-solid leftist, though fashions in method and styles may vary.
I was in Google Ads for 3 years and got to know the Economics team quite well. They turned out in force to see Joseph Stiglitz, and completely boycotted Luigi Zingales, so that left-liberal bias was definitely there.
Maybe they came out for Nessim Nicholas Taleb; there were 150 people there so I can't be sure!
As for RCT's: I think is somewhat like saying "lefty academics use computers, so computers are bad." The bias you're describing is in the academic establishment, and RCT's are just a tool they happen to use. Eliminate RCTs and they'd advocate for government by some other means.
I think you miss a big part of the point. Caplan is not indicting RCTs per se; he is indicting the RCTers and how they are using (and not using) RCTs.
I didn't miss the point. I just think he's attacking the TOOLS of the left, and the tools are pretty useful no matter what your point of view is.
Then we must agree to disagree. To me he is clearly attacking how RCTs are used. Said another way: he isn’t attacking fire or matches, he’s attacking arsonists
Caplan is a big advocate of RCTs.
>they'd advocate for government by some other means
God, tradition, equality.
And Luigi Zingales is hardly of the right. On his podcast, he tends to look to fall prey to the nirvana fallacy more than I would like.
Who said he's "of the right" ? Not me.
He's just someone who rejects traditional left v. right thinking. "Pro-market is not the same as pro-business." Anyone with an open mind should be curious, if nothing else.
Well, broadly speaking he’s right of center. Your point that he ain’t hard right only reinforces Albert’s (I know: shocking, shocking) point that the Google team is leftist biased.
A rigorous logical induction of causal laws in the social sciences is impossible. The two basic reasons why imitation of the methods of the natural sciences doesn't work are that (1) the most relevant causal factors of interest, namely the mental states of individual human beings, are not available for direct observation by social scientists, let alone precise quantification via comparison against a standard; and (2) natural scientists, unlike social scientists, can inject themselves into the chain of causality to systematically vary one causal factor at a time in the system of interest while leaving all the others constant to see what consequences follow from the singular change. Simply put, social scientists are in no position to do the logical equivalent of controlled laboratory experiments on real societies.
Note also that the system of interest needs to be well-defined, which is why natural scientists derive their theories from highly simplified systems, while treating uncontrolled observations of complex systems as needing to be explained by pre-existing theories (i.e. *abductive* reasoning), not as confirming new theories. Even the randonomized control trial is not a true gold standard of empirical science, since RCT results are valid only for the population as originally sampled. Once the relevant characteristics of a population change, or once one examine a different population, causal factors that only affected a small subset of the original population might loom large in another context.
The best social scientists can hope for is to start with self-evident truths about human beings; that is, facts that are embedded in our very ability to comprehend the world and engage in meaningful conversations about it to others, and thus can't be disputed without self-contradiction. You don't need an experiment to know that human beings are purposeful, for example. Knowing that, an economist can deduce that everything that follows from the logic of purposeful action must be applicable to human societies.
From a socialist perspective the frustrating thing about a logic of purposeful action is that its conclusions are of the form "if causal factor x occurs, then all other things being equal the effect y will follow." Such qualitative results don't give you the timing and magnitude of effect y, nor does it enable you to precisely predict what will happen when all other things aren't equal, and other changes besides x are occurring simultaneously (as is always the case in real societies). People who crave the job of central planner don't find honest economics of any use to their charade of being benefactors to society, so they fake it and pretend that the methods of the natural sciences will empower them to engage in benevolent social engineering. An honest economist, on the other hand, refrains from such a conceit of certainty and pretense of doing "natural experiments" on society, and instead simply reminds everyone that we already know even without having to collect statistical/historical data (and without making up arbitrary assumptions that can't be justified as self-evident truths, another common sin among dishonest economists) that certain x's will invariably lead to results that everyone agrees is worse than not doing x.
You see very similar behavior in the industry sphere, too, where businesses or process owners will become dedicated to a particular way of doing things and demand answers to make that way work better. Pointing out that they would be better off doing it a different way, or that their way of doing things can't work the way they would like at all, that never goes over well.
If RCTs generate high quality medical interventions, why would they not generate high quality economic interventions?
The unit of observation for medical RCTs is an individual person. In most cases the "unintended consequences" of a medical intervention are limited to a single person, and thus RCTs provide a sense of the total effect.
In RCTs of economic interventions the unit of observation is often one person, household, or business, but the total effect will often spread far beyond that unit. The relevant system is the whole economy, not just the person affected directly by the intervention. Unintended consequences may not be observable in each observation and are less likely to be foreseen or even understood as consequences of the initial intervention.
Medical RCTs would not be helpful if we made the unit of observation seperate body parts instead of the whole person. Some pill might lower blood pressure but kill brain activity.
Medical RCTs don't (just) test Drug A vs Drug B. They test Drug A vs a control of doing nothing (usually via placebo).
Caplan is pointing out that Economic RCTs aren't doing that, they are testing Government Intervention A vs Government Intervention B. Where is the test of smaller class size vs not funding government schools? The closest you get is testing public schools vs public charter schools.
You deploy Coase's wonderful expression "the total effect" deftly here, illuminating the folly of RCT scientism.
Due respect, you are using the acronym RCTs in a very different sense than does Caplan in his piece. You are using it solely in the technical sense of performing a randomized control trial, and claiming that Caplan is opposed to (small letters) randomized control trials.
But if you read the piece, that is NOT what he is saying at all. He is indicting how RCTs are used, the fact that only pro-government-intervention RCTs are published, etc.
To put it in non- economics terms, you are claiming that Caplan is objecting to fire, when in fact Caplan is only complaining how a particular tribe are arsonists in their use of fire.
I am curious whether you believe that RCTs are capable of testing the outcomes of free markets?
If so how would you do that?
Would it perhaps be a second control group with the other control group being “current policy?”
I don’t usually like to post fan-boy comments, but this piece is particularly excellent.
Shades of Bastiat here; though the more open question is how much of the pro-government resultant bias is intended versus unintended.
Isn't it possible that (ironically) this gets the causality wrong? It not that RCTs encourage statist policies, but rather that a bias towards statism leads people to use RCTs (and other types of empiricism) in support of statism.
An example fresh in my mind of a study discouraging statism: research that shows that *affordable* housing is really just *old* housing. Thus strongly implying that if we want more affordable housing what we need is *not* programs to build affordable more affordable housing, but allowing more housing to be built so that existing housing will age and become affordable.
It's possible I've misunderstood what you are saying, but if so, I suspect I am not the only one...
That is a good example of doing it right, but I have seen apopletic responses to that suggestion from afforadable housing advocates.
I have suggested that houses prices in SF can be fixed with 3 simple things:
An end to zoning, a supply of bulldozers, and 75 years.
The links to the footnotes in the first chapter DID NOT WORK. Either this should be fixed (in future articles) or the footnotes should not be linked at all.
> If you oppose free-market policies, you’re might be pleased
Should be "you might".
I think these randominstas might be thinking like Robin Hanson's efficient economist's pledge:
https://www.overcomingbias.com/p/efficient-economists-pledgehtml
Government officials are more likely to actually listen to them in attempting to accomplish their goals better.
But this is the logical consequence of a positivist outlook on economics. The primacy of a priori theory must be stressed to counter the RCT agenda.
> They’ll always either find that (a) the treatment “worked” — so government should scale it up; or (b) the treatment “failed” — so government should keep running RCTs until they find a treatment that works.
Isn't this just because arguing for more or less government action is a normative claim outside the scope of the RCT?
"this didn't work, maybe something else works" is a lot less controversial to claim than: "this didn't work, so stop trying"
"More or less government action" is no more a normative claim than "this treatment worked better or worse than the control". Both are assuming some particular end, e.g. improved math scores, is desirable. The question is how to get there, but the big question about whether or not the state can get there is largely ignored in favor of assuming the state can, and then trying to figure out how.
What about the famous oregon health insurance study, which was an RCT? I think it was conducted, among others, by the great economist Amy Finkelstein. In the sense that it proved the moral hazard effect (and subsequent raise in spending) of universal health insurance, which is something that most free marketers use as an argument to advocate for private insurance and or health saving accounts?
This post asserts that you can't treat whole countries like guinea-pigs, so you can't test big questions with RCTs. That may be true for, say monetary policy, which applies at a national scale. But I'm not sure I agree that you can only test markets versus governments at the level of whole countries. Let's say you have coffee farmers in a developing country that accept a price established by the state monopsony buyer of coffee in advance of the harvest. Couldn't you have an RCT that let some farmer cooperatives receive market prices, and others the government set price, and analyse the effects? You might say these results don't settle the big and general question of markets versus governments. But that's true of all RCTs. They cast a bright but narrow beam of light. So the incentives for nurses that work in Nairobi don't translate to Mexico or even other parts of Kenya for multiple devil in the detail reasons. I don't see a clear distinction between big and small questions, and thus which questions are amenable to RCTs and which not.
After over 40 years of research in the field when abhijit banerjee was tasked to create an anti poverty program in India he basically created a negative income tax (more or less). Isn't he admitting that Milton Friedman was correct?
> “natural experiments on markets versus government.” But top journals these days will never let you get away with that. The sample is too small, the randomization too questionable, and the question too big. Way too big.
Sure. 300,000 years of sustainable govt poverty vs. 300 years of vastly larger and vastly larger increase of capitalist prosperity. Coincidence? Maybe. Sure. Maybe these "scientists" could do an RCT on rocks and feathers in water to discover which group is more likely to sink. Science exists in a causal, non-coincidental universe, contra views of QM math which sneak in and then conclude a random metaphysics. Further, causality is the thing/action relation, not the prior event/later event relation. Action is the action of definite things with definite properties. A things actions are that things actions. Action cannot contradict the nature of the thing. Feathers cant act like rocks. Govt cant act like capitalism. They are different kinds of things with, thus, different kinds of actions. The minds of businessmen are not the guns of govt. Production is not force. Marx plus RCT "science" is a pig w//lipstick. Contra the claims of impossibly, value-free social "science," economic activity is value-causing activity. Economics must study values. But the hidden altruism of govt economies is the sacrifice of values. Production and the science of production require values. Rejecting values is rejecting social science. See "What Is Capitalism" by Ayn Rand for more philosophy of economics.