Right in the Short Run

Steve Levitt writes a reply to Noam Scheiber’s hit piece on Levitt and other eminent Economists. It is a great read, here is an excerpt:

It seems to me that Scheiber is tied up in knots to the point that he no longer knows what he believes. He seems to instinctively like clever research, but feels such guilt about it that he is compelled to denounce it. The incredibly important point that he misses is that often being clever is the way one cracks an important problem. He can denigrate the questions I have made progress on tackling, but it seems to me that understanding how crime responds to punishment, why crime fell in the 1990s, why Blacks are lagging Whites so badly educationally and economically, whether firms profit maximize, whether campaign spending affects election outcomes, and whether elected officials follow the will of the electorate are pretty reasonable topics for an economist to study. Sure, my approach to each of these was different than what everyone else was doing, but the questions I have asked have (usually) had both serious policy questions and economic issues at their heart, and I delivered some answers when others were not.

Joshua Angrist also responds:

Second, in his rush to tar some up-and-comers with the “cute-o-nomics” brush, Scheiber misses a central feature of the clean-identification research agenda, best explained by example.  One of the enduring scientific and policy questions in Labor economics is the sensitivity of hours worked to changes in pay (this matters for tax policy, for example).  The best evidence labor economists have on the relation between wages and hours worked comes from a small experiment (by Ernst Fehr and Lorenz Goette) involving the wages of bicycle messengers in Switzerland.  The second best comes from a study of stadium vendors by Gerald Oettinger.  Who cares about the riders of Veloblitz or snack sellers at Camden Yards?  We care because economics is predicated on the notion that a few simple principles explain behavior in many settings.  These studies produce results that are convincing and may well be general, though, as always in science, it will take replication to know for sure.

This is a response to chandrasekhar’s post, Cuteness.

There is plenty of material to criticize in Noam Scheiber’s recent TNR article, but one aspect of it annoyed me in particular. Scheiber claims that economics had a mid-life crisis in the 1980’s, and in response economists started thinking in an entirely different way…

By the ’80s, however, the data-crunchers had come down with a crisis f confidence. In one famous episode, the eminent economist H. Gregg Lewis reviewed several studies on unions. What he found was alarming: some papers reported that unions strongly increased wages; others reported exactly the opposite. The difference, in most cases, was simply the assumptions the authors had made.

Critiques like this tipped the discipline into a prolonged bout of soul-searching. The old approach had been sweeping in its ambition. But what good were ambitious goals if the best you could do was “on the one hand/on the other hand”style equivocation or, worse, plain jibberish? “People didn’t believe the estimates being produced,” recalls David Card, then a rising star at Princeton. “They felt the evidence in economics was not very credible.” Economists had long aspired to science. Suddenly they faced a harrowing thought: What if they were no better at pinning down truth than the average critical
studies major?

Yes, this all happened in the 1980’s. In the article, the time period has no relation to the change in methodology — it was the general sense of disappointment in the old ways that made people start looking for “clean ID”:

Having glimpsed this nihilistic vision, many economists ran screaming in the opposite direction. They concluded that the path to knowledge lay in solid answers to modest questions. Henceforth, the emphasis would be on “clean identification,” on sorting out what caused what.

The early practitioners of this approach–Angrist, Krueger, Card–had well-earned reputations as crafty researchers. But, by and large, all three men used their creativity to chip away at important questions. It was only in the late ’90s that the signs of overreach became apparent. To some professors at top departments, clean identification became a fetish. “Almost every student, myself included, had the terrible experience of getting up in front of the [professors] for whom identification is the Holy Grail, and getting cut to shreds when your identification strategy doesn’t pass muster,” recalls a recent Harvard Ph.D.

Actually, it’s a good thing that grad students fear getting cut to shreds if they have a bad identification stategy. When your results depend on your natural experiment being an actual natural experiment, establishing causality rests on good identification. Duh! Why should I bother trusting the conclusion of a paper with bad ID?

The problem is that there are only so many big questions that misgraded tests or arbitrary boundaries can shed light on. If you’re wedded to these techniques, eventually they lead you in obscure directions. “People think about the question less than the method,” says Berkeley professor Raj Chetty, one of the most sought-after Harvard graduates in recent years (and a notable exception to this trend). “They’re not thinking: What important question should I answer?’ So you get weird papers, like sanitation facilities in Native American reservations.”

I suspect Chetty did not intend for his criticism to have as far reaching implications for the discipline as Scheiber claims it does. But that’s beside the point. My sense is that Scheiber has completely brushed past a major cause of the change in the style of Economic research over the past few decades: computers.

After all, how likely is it that most economists would suddenly come to the same realization and change their methods in the same way? Sounds like a coordination problem! The increase in computing power and the ubiquity of the PC do a lot to explain these changes without the psychoanalysis.

PC’s everywhere mean data everywhere. It has become much simpler to keep records, and as a result governments, NGO’s, businesses, and people record tons of information electronically. Remember the Freakonomics bit about the average member of crack gangs making less than a McDonald’s worker? Legend has it that Sudhir Venkatesh, a PhD sociology student at U Chicago, obtained the data for the paper by infiltrating himself into a crack gang and winning over the leader. The gang ended up giving him an Excel spreadsheet file with their finances. Computers allowed the crack gang to easily keep records; these records were easily transferable and already in a format that was ideal for regression analysis.* (Correction at the end)

The processing power of computers has also increased substantially, doubling about every 18 months (It’s called Moore’s Law). In a decade, then, computers become about 100 times faster! If you’re thinking that the instrumental variables regressions that economists use for this “clean ID” stuff probably require lots of mathematical calculations, taking millions of clock cycles, then we’re on the same page.

I’ve worked with some of the data that Scheiber derides as “cute”, and it ends up being huge — tens of thousands of observations spanning several-hundred megabyte files. Even if you could find a hard drive big enough to hold that data back in 1985, and even if you could find enough memory to hold the matrix in temporary storage, no desktop computer from that decade could invert the matrix in a reasonable amount of time.

Yes, there may be other reasons that Economics has embraced the instrumental variables regression, with its holy grail of clean identification. But let’s acknowledge the advance in technology that occurred alongside the growth of these methods, and in my view, is at least partly responsible for the state of the discipline today.

* Oops! This is totally wrong. The gang’s books were physical books! A better example would be the paper on the parking tickets of UN diplomats. There were millions of dollars worth of fines and they were all tracked digitally by the New York City government. In a time without computers, it’s hard to imagine the government keeping such meticulous records or making them so easily available.