UPDATED: Fox Rushes To Dismiss Poll Critical Of GOP

A recent CBS poll found that the American public isn't very happy with anyone in Washington over the default crisis talks: 48 percent disapprove of how President Obama is handling the negotiations (43 percent approve) and 58 percent disapprove of Congressional Democrats' handling of the talks (31 percent approve). But Congressional Republicans fared the worst: 71 percent of respondents said they disapprove of the GOP's handling of the debt limit talks, while only 21 percent approve.

Unsurprisingly, Fox did not take this news well, claiming the poll's methodology was somehow flawed, despite the fact that they had praised a poll with similar methodology the day before. Several hosts took to the air to attack the poll, claiming it had a bias that fits the “narrative” being built “to prop up this president.” The attacks began last night on The O'Reilly Factor (via Nexis):

O'REILLY: But first, the “Top Story” tonight. Let's bring in Fox News senior political analyst Brit Hume from Washington. Are you surprised by the CBS News poll showing only 21 percent of Americans think the GOP is doing a good job in the debt debate?

BRIT HUME, FOX NEWS SENIOR POLITICAL ANALYST: No I'm not. I am a little bit surprised at that number 21 percent. I think Republicans as you suggested earlier are losing the PR war on this. I don't think the number is probably that bad. This is a poll of all adults. Not -- not -- not registered voters. Not even -- and certainly not likely voters. So there's a limit to how much it can tell you.

It also had only a small number of Republicans in the sample, about 25 percent, perhaps. So the poll is probably skewed. Nonetheless, Republicans -

(CROSSTALK)

O'REILLY: Why -- why would they -- let me just stop you there. Why would they do that, why bother taking a poll with only 25 percent of Republicans in the sample. But why bother?

HUME: Well, why bother taking -- well, in a political issue, I never have understood the value of polls of all adults.

(CROSSTALK)

O'REILLY: Yes, I mean, all the likely voters is -

(CROSSTALK)

HUME: I just don't think they tell you very much.

O'REILLY: -- but the 25 percent of Republicans, of course, you're going to skew it down if you don't -- but the rest of the poll is 75 percent non-Republican.

Hume admitted that “the poll is on the right track, to the extent that I think the Republicans are losing ... this isn't going well for Republicans.”

Today, the co-hosts of Fox & Friends also claimed the poll was unreliable, saying during one segment:

ALISYN CAMEROTA (guest host): So meanwhile, let's see how the Americans -- how Americans are feeling about who is winning this debt debate. Is it the Republicans or is it the Democrats? There have been other previous polls that have suggested that if the debt deal doesn't pass, the Republicans would be held more responsible -- for whatever reason, that the White House is winning the game of spin. So CBS News yesterday released a sort of stunning poll in which they found that, sure, the president would be held responsible, 48 percent disapprove of the way he's handling it, but 71 percent of respondents say that it's actually the GOP that they would hold responsible, and they disapprove of how the GOP is handling this.

STEVE DOOCY (co-host): And you're exactly right, Ali. It is a stunning poll, because CBS did it, and rather than talk to likely voters, which they did not do, or registered voters, they simply talked to adults, which is extraordinary. If you're going to talk to somebody about stuff that involves politics, talk to likely voters. They didn't do that. But it's CBS -- it's part of the mainstream media. Anyway, it's all part of the narrative that [Fox News contributor] Bernie Goldberg says they are building to prop up this president.

This text was aired on-screen while Doocy was speaking:

liberal storyline

Fox & Friends then aired a video clip of Goldberg saying: “The storyline is simply this -- Barack Obama is the reasonable one. He is the one who is responsible -- fiscally responsible. ... This narrative, this storyline, wasn't created by the media. Barack Obama painted that picture of himself, and his pals in the media predictably picked it up and just ran with it.”

So, to review, the charges here are that 1) the CBS poll is unfair because it only polled “adults” rather than “registered voters” or “likely voters” and 2) the CBS had an unfairly low number of Republicans in its sample size.

Is it, in fact, unusual to poll “adults” rather than registered or likely voters? Let's take a look at some recent polls, from a variety of polling organizations, and then see what an expert had to say about it.

Here's the methodology of a Pew Research poll about the debt limit, released July 18 (emphasis added):

The analysis in this report is based on telephone interviews conducted July 15-17, 2011, among a national sample of 764 adults 18 years of age or older living in the continental United States (458 respondents were interviewed on a landline telephone, and 306 were interviewed on a cell phone, including 128 who had no landline telephone).

And this is from the methodology of a recent AP/GfK poll:

The Associated Press-GfK Poll was conducted from June 16 through June 20, 2011, by GfK Roper Public Affairs & Corporate Communications - a division of GfK Custom Research North America. This telephone poll is based on a nationally-representative probability sample of 1,001 general population adults age 18 or older.

From a recent poll conducted by NBC News and The Wall Street Journal, a fellow News Corp. outlet (emphasis added):

wsj

And finally, from a recent Gallup poll, also about the deficit and default crisis:

Results for this Gallup poll are based on telephone interviews conducted July 7-10, 2011, with a random sample of 1,016 adults, aged 18 and older, living in all 50 U.S. states and the District of Columbia.

That last one is especially relevant, because Fox & Friends actually cited that very poll yesterday. They twisted the poll's results -- trying to falsely claim most Americans don't want a balanced approach to solving the default crisis -- but nonetheless, Fox News contributor Dana Perino said during the segment, “I still think that the Gallup poll, for most people, is kind of the gold -- Good Housekeeping seal of approval when it comes to polls.”

What do polling experts say about this methodology? In an email to Media Matters, polling expert and blogger for The New York Times' FiveThirtyEight politics blog Nate Silver said: “Polling adults is pretty much the norm this far in advance of an election, since it's hard to predict who will and won't vote until quite late in the campaign. Especially when you're polling on a policy issue rather than a 'horse race' question.”

As for the critique that the poll was “skewed” to downsize Republicans, CBS does weight their poll results, like most polling firms -- but they don't use political party affiliation, just other demographic criteria such as age, sex, race, and education. Here's how CBS explains their methodology (emphasis original):

Do Our Respondents Look Like The American Public?

At the end of our surveys, we find sometimes that we have questioned too many people from one group or another. Older people, for example, tend to be at home to answer the phone more than younger people, so there is often a greater percentage of older people in our surveys than exists in the American public.

When that happens, we take great pains to adjust our data so that I [sic] accurately reflects the whole population. That process is called “weighting.” We make sure that our final figures match U.S. Census Bureau breakdowns on age, sex, race, education, and region of the country. We also “weight” to adjust for the fact that people who share a phone with others have less chance to be contacted than people who live alone and have their own phones, and that households with more than one telephone number have more chances to be called than households with only one phone number.

So when we add up all the answers to our questions, we know that no one's opinion counts for more than it should. When you see one of our poll results on TV or in the newspaper, you know that it does not show the opinions of only one or two groups of Americans.

Sounds like the CBS poll might not be so skewed, after all.

UPDATE: Gary Langer, president of Langer Research Associates and ABC News pollster, told Media Matters that Fox's criticisms of the CBS poll were “spurious.” Langer said, " 'Likely voters' are only relevant in pre-election polling focused on candidate preferences ... The rest of the time, living in a democracy, everyone is entitled to an opinion." He added, “If the fundamental purpose [of a poll] ... is to understand public attitudes, that's best reflected in a survey of all adults.”

Regarding O'Reilly and Hume's criticisms that the CBS poll was “skewed” because the poll had “only 25 percent of Republicans in the sample,” Langer said, “That happens to be the reality for the political allegiance in our country.” He pointed to the most recent ABC News / Washington Post poll, which found that 26 percent of respondents identified as Republican. He added that the average percentage of self-identified Republicans across 10 previous national polls was 24 percent, and in 2009, the average across 11 polls was 23 percent.

Langer added: “We live in a time in which individuals who have a dog in the fight will criticize survey results that they perceive as not working to their advantage. There's nothing new about that -- there's also nothing remotely substantive or well-informed about it.”

Watch: