Conservative author Brian C. Anderson cited a misleading study by the American Enterprise Institute (AEI) to support his assertions of liberal bias in the media. Anderson claimed that the study was “a good one” from “a pretty thick chapter on examples of liberal bias going back for years and years” from his new book South Park Conservatives: The Revolt Against Liberal Media Bias (Regnery, April 2005) during his April 18 appearance on Fox News' The O'Reilly Factor. In fact, the paper, “Is Newspaper Coverage of Economic Events Politically Biased?” by John R. Lott Jr. and Kevin A. Hassett, employs questionable social science methodology and draws exaggerated conclusions from its underlying data.

AEI study on media bias flawed

Lott and Hassett's paper has yet to appear in any peer-reviewed academic journal and is unlikely ever to do so. Neither author of the AEI study apparently has a background in communication studies, and they are woefully ignorant of the status of academic research in this area. The authors state that "[s]ystematic statistical studies of media coverage are much rarer," then discuss “five studies that do exist.” In fact, peer-reviewed journals have published literally hundreds of systematic studies published that bear in some way on the question of media bias; a meta-analysis in the Journal of Communication in 2000 analyzed 59 of those most directly addressing the question and concluded, “On the whole, no significant biases were found for the newspaper industry. Biases in newsmagazines were virtually zero as well. Meta-analysis of studies of television network news showed small, measurable, but probably insubstantial coverage and statement biases.”

Like much of Lott's research, the AEI paper supports a controversial thesis using multiple and progressively more esoteric statistical techniques, rendering the argument difficult for anyone without advanced statistical training (and many who do have such training) to understand. This technique allows the authors -- as well as those, like Anderson, those who cite the study -- to exaggerate what the data actually show.

Consider sentences like this one: “Indeed, all of those results imply a large bias in coverage ranging from 16.3 to 24.1 percentage points, though the results for the top 10 newspapers are statistically significant at only the .20 [p-value] level for a two-tailed t-test.” As anyone trained in statistics knows, significance at the .20 level is no significance at all, since it means that one out of every five times the observed result would have appeared at random. The commonly accepted level for “real” significance is the .05 level, though scholars regularly report results that reach the .10 level of significance (discussions of statistical significance and the meaning of p-values can be found here and here). Lott and Hassett even report as meaningful results that reach “significance” at the .30 level, which any scholar would dismiss out of hand, though they ignored results at similar significance levels that countered their “liberal bias” thesis by showing coverage favoring Republican presidents.

The authors also seem ignorant of standard communication research practice for performing content analyses, since they do not address the issue of inter-coder reliability -- i.e., whether the various different coders analyzing the texts are producing mutually consistent results. Lott and Hassett apparently performed no inter-coder reliability tests; without such a test, no peer-reviewed journal would accept a paper based on content analysis. Their only hint at inter-coder reliability is a statement in an appendix that “there were no disagreements [between coders] in how headlines were being classified,” a statement that anyone who has performed content analysis of news coverage would find simply unbelievable (a discussion of inter-coder reliability can be found here).

In addition, as others have noted, Lott and Hassett fail to account for the long-term economic trends that explain much of the way the economy has been perceived and reported during the past 15 years. For instance, though they control for the unemployment rate during a given quarter, as well as the unemployment rate of the previous quarter, the salient feature of the economy under President Clinton was steadily decreasing unemployment over several years. This long-term trend, and not simply the size of the difference in unemployment between one quarter and the next, created the impression of a good economy. Similarly, the public and the press understood the loss of jobs during the administration of President George W. Bush not just in the context of quarter-to-quarter changes, but in the contrast it presented with the previous administration.

The authors examined only newspaper headlines -- a reasonable choice given the magnitude of the task of analyzing full newspaper stories over a period of years. But having made that choice, they then refer repeatedly -- more than 150 times, in fact -- to “coverage,” “media coverage,” or “press coverage,” and the “bias” they claim to have found therein. In fact, they have not analyzed “coverage” at all, only headlines.

Lott has been embroiled in scholarly scandal before, most notably when he was unable to produce data from a survey he claimed to have conducted that produced remarkable findings, and when he was found to have created an alias -- “Mary Rosh” -- on internet chat sites to defend his work (more details here). Lott also made false statements about the 2000 presidential election in an October 2004 appearance on CNN's Lou Dobbs Tonight.

Who is Brian C. Anderson?

Anderson, O'Reilly's guest touting the study, works as a senior editor at City Journal, a quarterly magazine of the conservative Manhattan Institute, which has received millions of dollars from conservative foundations. City Journal itself was funded by the conservative John M. Olin Foundation starting in 1990.

In his writings, Anderson argues that conservatives are winning the media war. In an April 18 op-ed in the Los Angeles Times, he wrote: “Sure, talk radio is partisan, sometimes overheated. But it's also a source of argument and information. Together with Fox News and the blogosphere, it has given the right a chance to break through the liberal monoculture and be heard.” In a January 28 Los Angeles Times commentary arguing that conservatism is on the rise among college students, Anderson also praised the efforts of discredited Students for Academic Freedom president David Horowitz. In South Park Conservatives, Anderson presents his ideas about the media and increasing youth conservatism. Anderson has also promoted other conservative causes. In addition, in the July 18, 2002, New York Sun, Anderson claimed that Supreme Court cases such as Brown v. Board of Education have allowed liberals to consolidate power in the hands of judges to promote liberals' political aims:

Since Brown, the Living Constitution has been used to justify, amazingly, racial discrimination by the government. It has also led to completely new procedural rights for the criminally accused that revolutionized the way the nation dealt with crime and helped fuel the crime explosion that began in the mid-1960s. Obscenity has gained constitutional protection while restrictions of political speech have been deemed consistent with the First Amendment. Partial birth abortion has been sanctified in a way no framer ever intended and no part of the Constitution supports.

From the April 18 edition of Fox News' The O'Reilly Factor:

O'REILLY: But your thesis is challenged right off the bat by The New York Times, the Los Angeles Times, and the network news producers who say there is no liberal bias; it's never existed. How do you answer that?

ANDERSON: Well, I think it's just an absurd claim. My book has a pretty thick chapter on examples of liberal bias going back for years and years. A good one is two researchers from the American Enterprise Institute just did a look at newspaper headlines in the mainstream media and A.P. reports going back, I think, to the early '80s on how they reported economic data. And any time there was a Republican president, the -- the reports were much more negative, whatever the underlying economic data actually were.