On the November 27 edition of MSNBC's Hardball, during the recurring “Big Number” feature, host Chris Matthews said, “Tonight, our Big Number is the number five. That's the number of Republican presidential candidates that [Sen.] Hillary Clinton [D-NY] trails in the November matchups.” Matthews said the information came from “a new Zogby poll.” However, Matthews did not note that the poll was an online Zogby Interactive poll in which participants were chosen from a database of volunteers. Matthews omitted this fact despite statements by the American Association for Public Opinion Research (AAPOR) and Democratic pollster Mark Blumenthal -- who appeared earlier in the day on MSNBC -- that such polls are unreliable.

A November 26 release detailing the findings cited by Matthews clearly identified the Zogby Interactive poll as an “online survey.” The release included a link to the "Interactive Poll/Survey Methodology," which answers the question “Who Participates in Zogby Interactive Polls?” by noting that the participants “are selected at random from a database of hundreds of thousands of individuals.” The database is composed of people who register themselves, as the same answer noted:

Zogby has assembled a database of individuals who have registered to take part in online polls through solicitations on the company's Web site, as well as other Web sites that span the political spectrum -- liberal, conservative, and middle of the road; politically active and apolitical; easy to reach and hard to find. Many individuals who participate in Zogby's telephone surveys also submit e-mail addresses so they may take part in online polls.

In the same answer, Zogby responded to “criticisms that interactive political polls include 'self-selected' political junkies skewing polls for fun and one-upmanship” by asserting that "[r]espondents of Zogby Interactive polls do not choose to take part in a poll, rather they are selected at random from a database of hundreds of thousands of individuals, much like the database of millions across the country who have telephones."

However, during his November 27 appearance on MSNBC Live, Blumenthal expressed doubt about the Zogby Interactive methodology. Blumenthal noted that the poll's findings differed from other recent telephone polls and added: "[W]hat's different about it is it was done online for people who had volunteered to be interviewed online. That particular method by that pollster wasn't all that accurate in 2006. So, I would just -- I'd be more cautious about the online surveys." Indeed, Wall Street Journal Online columnist Carl Bialik noted on November 16, 2006, that Zogby Interactive's 2006 Senate election “predictions missed by an average of 8.6 percentage points in those polls -- at least twice the average miss of four other polling operations I examined.”

In a November 26 post on his blog Political Arithmetik, University of Wisconsin-Madison professor Charles H. Franklin, former president of the Society for Political Methodology, wrote that the Zogby Interactive poll cited by Matthews “has produced some odd results” and mentioned that the online poll did not rely on “a normal random sample of the population.” Referring to a November 26 Reuters article headlined “New poll shows Clinton trails top 2008 Republicans,” Franklin further wrote that “based on the large outliers the Clinton results produce, I'd hold off on the Reuters headline until I saw some confirmation from other polls.”

On its website, the AAPOR states that "[e]ven if opt-in surveys" -- like the Zogby Interactive poll -- “are based on probability samples drawn from very large pools of volunteers, their results still suffer from unknown biases”:

When we draw a sample at random -- that is, when every member of the target population has a known probability of being selected -- we can use the sample to make projective, quantitative estimates about the population. A sample selected at random has known mathematical properties that allow for the computation of sampling error.

Surveys based on self-selected volunteers do not have that sort of known relationship to the target population and are subject to unknown, non-measurable biases. Even if opt-in surveys are based on probability samples drawn from very large pools of volunteers, their results still suffer from unknown biases stemming from the fact that the pool has no knowable relationships with the full target population.

Similarly, in response to a previous Zogby Interactive survey, Blumenthal wrote in an April 26, 2006, blog post:

Why is it important that the survey was conducted online?

1) This survey is not based on a “scientific” random sample -- The press release posted on the web site of the trade group that paid for the poll makes the claim that it is a “scientific poll” of “likely voters.” As we have discussed here previously, we use the term scientific to describe a poll based on a random probability sample, one in which all members of a population (in this case, all likely voters) have an equal or known chance of being selected at random.

In this case only individuals that had previously joined the Zogby panel of potential respondents had that opportunity. As this article on the Zogby's web site explains, their online samples are selected from “a database of individuals who have registered to take part in online polls through solicitations on the company's Web site, as well as other Web sites that span the political spectrum.” In other words, most of the members of the panel saw a banner ad on a web site and volunteered to participate. You can volunteer too -- just use this link.

Zogby claims that “many individuals who have participated in Zogby's telephone surveys also have submitted e-mail addresses so they may take part in online polls.” Such recruitment might help make Zogby's panel a bit more representative, but it certainly does not trans[f]orm it into a random sample. Moreover, he tells us nothing about the percentage of such recruits in his panel or the percentage of telephone respondents that typically submit email addresses. Despite Zogby's bluster, this claim does not come close to making his “database” a projective random sample of the U.S. population.

From the discussion on the 10 a.m. ET hour of the November 27 edition of MSNBC Live, which featured Blumenthal and Iowa-based pollster J. Ann Selzer, whose firm conducts the quadrennial Iowa Poll for The Des Moines Register:

HALL: Here to help us make sense of some of the numbers that are floating around out there, Mark Blumenthal, editor and publisher of Pollster.com, and Ann Selzer, she's a pollster and president of Selzer and Company. Thanks for joining us.

BLUMENTHAL: [unintelligible] to be here.

HALL: All right, Mark, I want to start with you.

SELZER: Good morning.

HALL: Good morning. We were talking in our newsroom. There are two recent polls out showing very different results --

BLUMENTHAL: Right.

HALL: -- when it comes to Hillary Clinton versus the top Republican candidates. You got one poll showing that she'd be beat by these Republican candidates. The other shows that she would come out ahead.

HALL: How do you make sense if you are a person at home and you stumble across one of these reading them online or maybe in the paper, and they're so different?

BLUMENTHAL: Well, that's always the trick, and I'd say go to Pollster.com, and we'll help you figure it out.

HALL: OK, outside of that.

BLUMENTHAL: The Zogby poll -- the one that looks very different from the poll from Gallup -- also looks very different from all the other recent surveys, and what's different about it is it was done online for people who had volunteered to be interviewed online. That particular method by that pollster wasn't all that accurate in 2006. So, I would just -- I'd be more cautious about the online surveys.

HALL: And, Ann, is that a concern how the pollsters get the information? Is, let's say, a phone or online not as reliable?

SELZER: Well, online polls are notoriously unreliable. And with one poll you get one answer, you do another poll the same way the next day, you can get a very different answer. So, telephone has shown to be the most reliable, that is you get the same answers if you do it the same way, consistently.

From the 5 p.m. ET hour of the November 27 edition of MSNBC's Hardball with Chris Matthews:

MATTHEWS: Time now for the Hardball “Big Number” that tells a big story. Tonight, our Big Number is five. That's the number of Republican presidential candidates that Hillary Clinton trails in the November matchups. According to a new Zogby poll, Rudy Giuliani, Mitt Romney, Fred Thompson, John McCain, and, believe it or not, Mike Huckabee, that's five, count 'em, five Republicans all now beating, yes, Hillary Clinton in the matchups for next November, and it's tonight's “Big Number.”

Now to the roundtable. Matt Continetti is with The Weekly Standard. Jonathan Carp-- Capehart of The Washington Post; he's on the editorial board. And Julie Mason of the Houston Chronicle. Julie, thank you for smiling.