In an op-ed in today's Wall Street Journal, George Mason University economist Daniel Klein purported to show that liberals are less informed about economics than conservatives based on a Zogby poll he conducted with Zogby researcher Zeljka Buturovic. Klein and Buturovic looked at respondents' answers to eight questions about economics and their political leanings. Here's how Klein explained it in the Journal:

Rather than focusing on whether respondents answered a question correctly, we instead looked at whether they answered incorrectly. A response was counted as incorrect only if it was flatly unenlightened.

Consider one of the economic propositions in the December 2008 poll: “Restrictions on housing development make housing less affordable.” People were asked if they: 1) strongly agree; 2) somewhat agree; 3) somewhat disagree; 4) strongly disagree; 5) are not sure.

Basic economics acknowledges that whatever redeeming features a restriction may have, it increases the cost of production and exchange, making goods and services less affordable. There may be exceptions to the general case, but they would be atypical.

Therefore, we counted as incorrect responses of “somewhat disagree” and “strongly disagree.” This treatment gives leeway for those who think the question is ambiguous or half right and half wrong. They would likely answer “not sure,” which we do not count as incorrect.

However, as FiveThirtyEight's Nate Silver writes:

Here's what Klein and Buturovic did. They took a survey using one of Zogby's internet panels, which is by far the worst polling instrument that they could have selected. The panel was not weighted and was not in balance. For example, McCain led Obama 49-43 among respondents to the survey, even though roughly the opposite outcome was observed in the actual election -- and only about 4 percent of the respondents were Hispanic and only 39 percent were female. Then they asked 16 “questions of basic economics”, as the Journal's sub-head describes them, and arbitrarily included eight of them in their analysis but threw the other eight out.

Silver also writes that there are numerous problems with the questions. After taking apart several of them, he concludes:

So basically, what you're left with a number of questions in which people respond out of their ideological reference points because the questions are ambiguous, substanceless, or confusing. Klein is blaming the victims, as it were.

There would have been much better ways to construct a study like this one. For instance, questions could have been developed from standardized tests of high school students, like the AP Economics exam, or from surveys of academic economists. Such studies might well support Klein's thesis. But between the poorly-considered questions and the poor choice of survey partner, this amounts to junk science.

Over at the New Republic, Jonathan Chait also took down the survey, writing:

[A]ll these questions measure not economic knowledge per se but agreement with the neoclassical model. There are no questions about, say, whether tax cuts at current levels tend to increase or decrease tax revenues. This was a test obviously designed to portray conservatives as better informed.

Chait concludes: “The only thing this study demonstrates is the ideological hackery of its authors.”