In recent days, news outlets including CNN cited a study of several major media outlets, "A Measure of Media Bias" (pdf) by political scientist Timothy J. Groseclose of UCLA and economist Jeffrey D. Milyo of the University of Missouri-Columbia, purporting to demonstrate that America's news content has “a strong liberal bias.” But the UCLA-led study employed a measure of “bias” so problematic that its findings are next to useless. In addition, the authors -- apparently new to media content analysis -- seem unaware of the substantial scholarly work that exists on the topic, yet they do cite a number of right-wing sources to provide support for their claims.

Given the study's conclusions (that the media is replete with liberal bias) and the study's failure to acknowledge its authors' conservative pedigree, it is not surprising that a number of conservative news outlets picked up the story, as did a few mainstream outlets. Conservative MSNBC host Tucker Carlson interviewed Milyo about the study on the December 19 edition of MSNBC's The Situation with Tucker Carlson. The study was also cited by anchor Jack Cafferty during the December 20 edition of CNN's The Situation Room; on the December 19 editions of Fox News' Fox & Friends and Special Report with Brit Hume; in a December 19 article in The Commercial Appeal of Memphis, Tennessee; and in a December 20 Investor's Business Daily editorial by Edward R. Stephanopoulos. CBS News' Public Eye weblog also featured a post about the study.

None of the outlets that reported on the study mentioned that the authors have previously received funding from the three premier conservative think tanks in the United States: the American Enterprise Institute for Public Policy Research (AEI), The Heritage Foundation, and the Hoover Institution on War, Revolution and Peace. Groseclose was a Hoover Institution 2000-2001 national fellow; Milyo, according to his CV (pdf), received a $40,500 grant from AEI; and, according to The Philanthropy Roundtable, Groseclose and Milyo were named by Heritage as Salvatori fellows in 1997. In 1996, Groseclose and Milyo co-authored a piece for the right-wing magazine The American Spectator, titled “Lost Shepherd,” criticizing the then-recently defeated member of Congress Karen Shepherd (D-UT) and defending her successor, Enid Greene (R-UT); when the piece was published, Greene was in the midst of a campaign contribution scandal and later agreed to pay a civil penalty after the Federal Election Commission found (pdf) that she violated campaign finance laws.

Study riddled with flaws

In "A Measure of Media Bias" (pdf), Groseclose and Milyo attempted to “measure media bias by estimating ideological scores for several major media outlets” based on the frequency with which various think tanks and advocacy organizations were cited approvingly by the media and by members of Congress over a 10-year period. In order to assess media “bias,” Groseclose and Milyo assembled the ideological scores given to members of Congress by the liberal group Americans for Democratic Action; examined the floor speeches of selected members to catalog which think tanks and policy organizations were cited by those members; used those citations as the basis for an ideological score assigned to each think tank (organizations cited by liberal members were scored as more liberal, whereas organizations cited by conservative members were scored as more conservative); then performed a content analysis of newspapers and TV programs to catalog which think tanks and policy organizations were quoted. If a news organization quoted a think tank mentioned by conservative members of Congress, then it was said to have a conservative “bias.” As Groseclose and Milyo put it:

As a simplified example, imagine that there were only two think tanks, and suppose that the New York Times cited the first think tank twice as often as the second. Our method asks: What is the estimated ADA score of a member of Congress who exhibits the same frequency (2:1) in his or her speeches? This is the score that our method would assign the New York Times.

In other words, the study rests on a presumption that can only be described as bizarre: If a member of Congress cites a think tank approvingly, and if that think tank is also cited by a news organization, then the news organization has a “bias” making it an ideological mirror of the member of Congress who cited the think tank. This, as Groseclose and Milyo define it, is what constitutes “media bias.”

When Carlson asked him to explain the study, Milyo misrepresented his own study. Milyo noted that the study did not look at editorials, then said, “Of course, but that's how bias sneaks into news coverage. The reporter doesn't say, 'I think this.' He says, 'According to our expert, say, Barbra Streisand, this is true.' Right? It's the choice of the experts that allows the opinion to get in.” But Milyo's example of Streisand -- as though a news organization would actually cite her as an “expert” -- is flawed, considering that the study examined only mentions of think tanks and advocacy organizations (not of individual experts). Milyo ended his interview by telling Carlson, “My wife's a big fan [of Carlson].”

Definition of bias categorized ACLU as conservative

Any quantitative study of this sort must take a complex idea -- in this case, “bias” -- and operationalize it into something that can be measured. But given its rather odd operationalization of “bias,” it is perhaps unsurprising that the study's scheme leads to some categorizations no observer -- on the right or the left -- could take seriously, including the following:

• National Rifle Association of America (NRA) scored a 45.9, making it “conservative” -- but just barely.
• RAND Corporation, a nonprofit research organization (motto: “OBJECTIVE ANALYSIS. EFFECTIVE SOLUTIONS.”) with strong ties to the Defense Department, scored a 60.4, making it a “liberal” group.
• Council on Foreign Relations, whose tagline is “A Nonpartisan Resource for Information and Analysis” (its current president is a former Bush administration official; its board includes prominent Democrats and Republicans from the foreign policy establishment) scored a 60.2, making it a “liberal” group.
• American Civil Liberties Union (ACLU), bête noire of the right, scored a 49.8, putting it just on the “conservative” side of the ledger.
• Center for Responsive Politics, a group whose primary purpose is the maintenance of databases on political contributions, scored a 66.9, making it highly “liberal.”
• Center for Strategic and Budgetary Assessments, a defense policy think tank whose board of directors is currently chaired by former Representative Dave McCurdy (D-OK), scored a 33.9, making it more “conservative” than AEI and than the National Taxpayers Union.

We leave to the reader the judgment on whether anyone could take seriously a coding scheme in which RAND is considered substantially more “liberal” than the ACLU. But this is not the only problem with Groseclose and Milyo's study; they lump together advocacy groups and think tanks that perform dramatically different functions. For instance, according to their data, the National Association for the Advancement of Colored People (NAACP) is the third most-quoted group on the list. But stories about race relations that include a quote from an NAACP representative are unlikely to be “balanced” with quotes from another group on their list. Their quotes will often be balanced by quotes from an individual, depending on the nature of the story; however, because there are no pro-racism groups of any legitimacy (or on Groseclose and Milyo's list), such stories will be coded as having a “liberal bias.” On the other hand, a quote from an NRA spokesperson can and often will be balanced with one from another organization on Groseclose and Milyo's list, Handgun Control, Inc. (Nonetheless, this reference is somewhat confusing, since Handgun Control was renamed the Brady Campaign to Prevent Gun Violence on June 14, 2001, and there is no reference to the Brady Campaign in the study or clarification of the name change; therefore, it is impossible to determine from reading the study if Groseclose and Milyo's score reflects post-2001 citations by legislators and the media of the group under its new name.)

It is not hard to imagine perfectly balanced news stories that Groseclose and Milyo would score as biased in one direction or the other, given the study's methodology. For instance, an article that quoted a member of Congress taking one side of an issue, and then quoted a think tank scholar taking the other side, would be coded as “biased” in the direction of whichever side was represented by the think tank scholar. Since Groseclose and Milyo's measure of “bias” is restricted to citations of think tank and advocacy groups, this kind of miscategorization is inevitable.

Groseclose and Milyo's discussion of the idea of bias assumes that if a reporter quotes a source, then the opinion expressed by that source is an accurate measure of the reporter's beliefs -- an assumption that most, if not all, reporters across the ideological spectrum would find utterly ridiculous. A Pentagon reporter must often quote Defense Secretary Donald H. Rumsfeld; however, the reporter's inclusion of a Rumsfeld quotation does not indicate that Rumsfeld's opinion mirrors the personal opinion of the reporter.

Upon seeing how their coding scheme categorized different groups, the authors might have reconsidered the wisdom of their operationalization of “bias.” But apparently they did not. Their odd categorizations led to some startling conclusions, including the result stating that The Wall Street Journal has more “liberal bias” than any news outlet they surveyed. Although they are concerned only with the Journal's news pages and not its highly conservative editorial page, the Journal is respected on both the right and the left, and it would be shocking to hear even the most rabid right-winger assert that the Journal is America's most liberal news outlet. (Click here to read a statement by a spokesman for The Wall Street Journal's publisher, Dow Jones & Company, in response to Groseclose and Milyo's study.)

The authors also display a remarkable ignorance of previous work on the subject of media bias. In their section titled “Some Previous Studies of Media Bias,” they name only three studies that address the issue at more than a theoretical level. All three studies are, to put it kindly, questionable:

1) One study concluded that, since conservatives say in surveys that the media are biased, the media are probably biased.

2) Another study examined the geographic distribution of subscriptions to newsmagazines (perhaps the only extant study utilizing a method of assessing bias more indirect than Groseclose and Milyo's own) and concluded that, since there are more subscriptions in more heavily Democratic areas, the magazines probably have a liberal bias.

3) Yet another study, which Media Matters for America has addressed previously, was co-authored by AEI resident scholar John R. Lott Jr.

Citations of scholarly media studies absent

Although the authors seem completely unaware of it, in reality there have been dozens of rigorous quantitative studies on media bias and hundreds of studies that address the issue in some way. One place the authors might have looked had they chosen to conduct an actual literature review would have been a 2000 meta-analysis published in the Journal of Communication (the flagship journal of the International Communication Association, the premier association of media scholars). The abstract of the study, titled "Media bias in presidential elections: a meta-analysis," reads as follows:

A meta-analysis considered 59 quantitative studies containing data concerned with partisan media bias in presidential election campaigns since 1948. Types of bias considered were gatekeeping bias, which is the preference for selecting stories from one party or the other; coverage bias, which considers the relative amounts of coverage each party receives; and statement bias, which focuses on the favorability of coverage toward one party or the other. On the whole, no significant biases were found for the newspaper industry. Biases in newsmagazines were virtually zero as well. However, meta-analysis of studies of television network news showed small, measurable, but probably insubstantial coverage and statement biases.

Standard scholarly practice dictates the assembly of a literature review as part of any published study, and meta-analyses, as they gather together the findings of multiple studies, are particularly critical to literature reviews. That Groseclose and Milyo overlooked not only the Journal of Communication meta-analysis, but also the 59 studies it surveyed, raises questions about the seriousness with which they conducted this study.

Indeed, they seem to be unaware that an academic discipline of media studies even exists. Their bibliography includes works by right-wing media critics such as Media Research Center founder and president L. Brent Bozell III and Accuracy in Media founder Reed Irvine (now deceased), as well as an article from the right-wing website WorldNetDaily. But Groseclose and Milyo failed to cite a single entry from any of the dozens of respected scholarly journals of communication and media studies in which media bias is a relatively frequent topic of inquiry -- nothing from Journal of Communication, Communication Research, Journalism and Mass Communication Quarterly, Journal of Broadcasting & Electronic Media, Political Communication, or any other media studies journal.

UCLA's December 14 press release announcing Groseclose and Milyo's study quoted Groseclose as follows: “A media person would have never done this study. It takes a Congress scholar even to think of using ADA scores as a measure. And I don't think many media scholars would have considered comparing news stories to congressional speeches.” Groseclose is only too correct, and he might have gone on to say that a media scholar would have at the very least been familiar with the relevant literature. As to whether the use of congressional speeches and ADA scores has yielded some new insight, Groseclose's self-congratulation seems less than warranted.

Charge of liberal bias unsubstantiated

The authors' ignorance comes through in ways large and small; for instance, in one regression model, they include a variable coding each think tank as having an address on or off K Street, “the famous street for lobbying firms” -- as though its address indicates the nature of an organization. While it is true that some lobbying firms are located on K Street, many are not; in any case, when it comes to think tanks and policy groups, whether the organization's offices are located on K Street (as opposed to L Street or M Street) is unrelated to position on the political or ideological spectrum and is, therefore, a completely meaningless indicator. Groseclose and Milyo may be interested to learn that not all advertising firms are located on New York's Madison Avenue, and some businesses on Madison Avenue are not advertising firms.

Finally, of particular note is the way the study's authors toss about the word “bias” indiscriminately. We at Media Matters for America are particularly careful to make no accusations of bias, since saying a journalist or news outlet has a “bias” assumes that the one making the charge knows what lies within another's heart or mind. For this reason, most claims that the media are “biased” are problematic at best. But Professors Groseclose and Milyo have made charges of bias that are among the least substantiated we have encountered, even as they assessed what is at most a small piece of a much larger question. Even if their study were not riddled with methodological red flags and results that lack what scholars call “face validity” (or what is more commonly known as the “laugh test”), the notion that “bias” can be assessed by matching think tank citations of news organizations and members of Congress seems questionable in the extreme.