Dylan Matthews
By Dylan Matthews
I’m a policy reporter. My job is to explain to my readers whether smaller class sizes help students learn, whether tax cuts boost economic growth and whether housing programmes help families escape poverty.
In a perfect world, what I do would be a kind of science reporting. Just as my colleagues at the health desk often explain which medicines are effective and which are a bust, I’d ideally be able to describe what sociologists, economists and political scientists have discovered about which policies work.
With a few exceptions, however, I can’t really do that — at least not with the precision my health colleagues often can. More important, neither can policymakers in Congress and in many regulatory agencies. The Food and Drug Administration has more information available in deciding whether to approve a treatment that a few thousand people would receive than Congress does in considering a bill that would affect every American.
Each year, hundreds of carefully controlled, double-blind studies are done to learn whether a given pill is better than a placebo or whether a new surgery leads to quicker recoveries. Many of these studies are funded by a single agency, the National Institutes of Health. By contrast, in a typical year, no such studies are conducted to evaluate social policy proposals.
That’s not because such studies are impossible. In 1962, researchers in a small Michigan school district randomly selected 58 3- and 4-year-olds to enroll in a preschool programme, then spent decades comparing them with a control group of 65 kids who didn’t go to preschool. Those who enrolled learned more and made more money as adults. In 1976, the Chicago Housing Authority randomly placed public-housing residents in apartments in the city or in the suburbs, and then tracked the two groups. Those in the suburbs did better on every metric, from household income to their children’s rates of college attendance.
These studies had a big impact. The Chicago study, for example, is the main research cited in proposals to provide housing vouchers to poor families to break up pockets of “concentrated disadvantage.”
But such studies are very rare. They’re expensive, which discourages universities and school districts (such as the one in Michigan conducting the preschool study) from doing them. And often, as in the Chicago case, they come about only because a court orders them.
Because of this rarity, it’s easy to pick nits. The preschool study involved only children with low IQs, critics noted. Maybe the results would have been different with children of average or above-average intelligence. The Chicago housing study was conducted during a time when crime was much worse than today. Maybe with safer inner cities, you wouldn’t see similar gains from sending families to the suburbs.
Researchers have spent gallons of ink arguing over such caveats, trying to figure out what can and can’t be inferred from the meagre pool of good data with which they’re forced to work. At no point does the straightforward solution present itself: run another study.
“Rigorous ways to evaluate whether programmes are working exist,” then-White House budget director Peter Orszag said in 2009. “But too often such evaluations don’t happen. . . . This has to change.”
As Orszag said, it’s not that researchers don’t know how to evaluate programmes or don’t want to. Indeed, researchers who focus on international development have been doing so in recent years, with very promising results. Economists such as MIT’s Esther Duflo and Abhijit Banerjee and Yale’s Dean Karlan, along with their research groups, the Jameel Poverty Action Lab (JPAL) and Innovations for Poverty Action (IPA), have run dozens of randomised experiments in developing countries to see which forms of aid work and which are worthless or counterproductive. The goal, as the JPAL puts it, is to “reduce poverty by ensuring that policy is based on scientific evidence, and research is translated into action.”
With funding from individuals and foundations, but for the most part not governments, they have learned, for instance, that spreading information about the benefits of education keeps students in class; remedial tutoring doesn’t. Giving away bed nets reduces malaria infections; charging even a small amount for them is much less effective.
The confidence with which development researchers can make these judgments is in stark contrast to the Talmudic reading of a handful of studies that characterises debate about social policy in the United States. And that confidence means policymakers pay attention. Aid organisations have heaped praise on Duflo and company, with Usaid chief Rajiv Shah declaring that “the whole movement that Esther and her colleagues at MIT and around the world have really spearheaded is so important in rethinking how we make aid work.” The World Bank has teamed up with the JPAL to design better poverty-reduction programmes.
WP-BLOOMBERG
By Dylan Matthews
I’m a policy reporter. My job is to explain to my readers whether smaller class sizes help students learn, whether tax cuts boost economic growth and whether housing programmes help families escape poverty.
In a perfect world, what I do would be a kind of science reporting. Just as my colleagues at the health desk often explain which medicines are effective and which are a bust, I’d ideally be able to describe what sociologists, economists and political scientists have discovered about which policies work.
With a few exceptions, however, I can’t really do that — at least not with the precision my health colleagues often can. More important, neither can policymakers in Congress and in many regulatory agencies. The Food and Drug Administration has more information available in deciding whether to approve a treatment that a few thousand people would receive than Congress does in considering a bill that would affect every American.
Each year, hundreds of carefully controlled, double-blind studies are done to learn whether a given pill is better than a placebo or whether a new surgery leads to quicker recoveries. Many of these studies are funded by a single agency, the National Institutes of Health. By contrast, in a typical year, no such studies are conducted to evaluate social policy proposals.
That’s not because such studies are impossible. In 1962, researchers in a small Michigan school district randomly selected 58 3- and 4-year-olds to enroll in a preschool programme, then spent decades comparing them with a control group of 65 kids who didn’t go to preschool. Those who enrolled learned more and made more money as adults. In 1976, the Chicago Housing Authority randomly placed public-housing residents in apartments in the city or in the suburbs, and then tracked the two groups. Those in the suburbs did better on every metric, from household income to their children’s rates of college attendance.
These studies had a big impact. The Chicago study, for example, is the main research cited in proposals to provide housing vouchers to poor families to break up pockets of “concentrated disadvantage.”
But such studies are very rare. They’re expensive, which discourages universities and school districts (such as the one in Michigan conducting the preschool study) from doing them. And often, as in the Chicago case, they come about only because a court orders them.
Because of this rarity, it’s easy to pick nits. The preschool study involved only children with low IQs, critics noted. Maybe the results would have been different with children of average or above-average intelligence. The Chicago housing study was conducted during a time when crime was much worse than today. Maybe with safer inner cities, you wouldn’t see similar gains from sending families to the suburbs.
Researchers have spent gallons of ink arguing over such caveats, trying to figure out what can and can’t be inferred from the meagre pool of good data with which they’re forced to work. At no point does the straightforward solution present itself: run another study.
“Rigorous ways to evaluate whether programmes are working exist,” then-White House budget director Peter Orszag said in 2009. “But too often such evaluations don’t happen. . . . This has to change.”
As Orszag said, it’s not that researchers don’t know how to evaluate programmes or don’t want to. Indeed, researchers who focus on international development have been doing so in recent years, with very promising results. Economists such as MIT’s Esther Duflo and Abhijit Banerjee and Yale’s Dean Karlan, along with their research groups, the Jameel Poverty Action Lab (JPAL) and Innovations for Poverty Action (IPA), have run dozens of randomised experiments in developing countries to see which forms of aid work and which are worthless or counterproductive. The goal, as the JPAL puts it, is to “reduce poverty by ensuring that policy is based on scientific evidence, and research is translated into action.”
With funding from individuals and foundations, but for the most part not governments, they have learned, for instance, that spreading information about the benefits of education keeps students in class; remedial tutoring doesn’t. Giving away bed nets reduces malaria infections; charging even a small amount for them is much less effective.
The confidence with which development researchers can make these judgments is in stark contrast to the Talmudic reading of a handful of studies that characterises debate about social policy in the United States. And that confidence means policymakers pay attention. Aid organisations have heaped praise on Duflo and company, with Usaid chief Rajiv Shah declaring that “the whole movement that Esther and her colleagues at MIT and around the world have really spearheaded is so important in rethinking how we make aid work.” The World Bank has teamed up with the JPAL to design better poverty-reduction programmes.
WP-BLOOMBERG
