Matthews cited Zogby Interactive poll without noting criticism of methodology
SUMMARY: On Hardball, citing "a new Zogby poll," Chris Matthews stated: "Tonight, our Big Number is the number five. That's the number of Republican presidential candidates that [Sen.] Hillary Clinton trails in the November matchups." However, Matthews did not note that the poll was an online Zogby Interactive poll in which participants were chosen from a database of volunteers. Matthews omitted this fact despite statements by the American Association for Public Opinion Research and Democratic pollster Mark Blumenthal -- who appeared earlier in the day on MSNBC -- that such polls are unreliable.
On the November 27 edition of MSNBC's Hardball, during the recurring "Big Number" feature, host Chris Matthews said, "Tonight, our Big Number is the number five. That's the number of Republican presidential candidates that [Sen.] Hillary Clinton [D-NY] trails in the November matchups." Matthews said the information came from "a new Zogby poll." However, Matthews did not note that the poll was an online Zogby Interactive poll in which participants were chosen from a database of volunteers. Matthews omitted this fact despite statements by the American Association for Public Opinion Research (AAPOR) and Democratic pollster Mark Blumenthal -- who appeared earlier in the day on MSNBC -- that such polls are unreliable.
A November 26 release detailing the findings cited by Matthews clearly identified the Zogby Interactive poll as an "online survey." The release included a link to the "Interactive Poll/Survey Methodology," which answers the question "Who Participates in Zogby Interactive Polls?" by noting that the participants "are selected at random from a database of hundreds of thousands of individuals." The database is composed of people who register themselves, as the same answer noted:
Zogby has assembled a database of individuals who have registered to take part in online polls through solicitations on the company's Web site, as well as other Web sites that span the political spectrum -- liberal, conservative, and middle of the road; politically active and apolitical; easy to reach and hard to find. Many individuals who participate in Zogby's telephone surveys also submit e-mail addresses so they may take part in online polls.
In the same answer, Zogby responded to "criticisms that interactive political polls include 'self-selected' political junkies skewing polls for fun and one-upmanship" by asserting that "[r]espondents of Zogby Interactive polls do not choose to take part in a poll, rather they are selected at random from a database of hundreds of thousands of individuals, much like the database of millions across the country who have telephones."
However, during his November 27 appearance on MSNBC Live, Blumenthal expressed doubt about the Zogby Interactive methodology. Blumenthal noted that the poll's findings differed from other recent telephone polls and added: "[W]hat's different about it is it was done online for people who had volunteered to be interviewed online. That particular method by that pollster wasn't all that accurate in 2006. So, I would just -- I'd be more cautious about the online surveys." Indeed, Wall Street Journal Online columnist Carl Bialik noted on November 16, 2006, that Zogby Interactive's 2006 Senate election "predictions missed by an average of 8.6 percentage points in those polls -- at least twice the average miss of four other polling operations I examined."
In a November 26 post on his blog Political Arithmetik, University of Wisconsin-Madison professor Charles H. Franklin, former president of the Society for Political Methodology, wrote that the Zogby Interactive poll cited by Matthews "has produced some odd results" and mentioned that the online poll did not rely on "a normal random sample of the population." Referring to a November 26 Reuters article headlined "New poll shows Clinton trails top 2008 Republicans," Franklin further wrote that "based on the large outliers the Clinton results produce, I'd hold off on the Reuters headline until I saw some confirmation from other polls."
On its website, the AAPOR states that "[e]ven if opt-in surveys" -- like the Zogby Interactive poll -- "are based on probability samples drawn from very large pools of volunteers, their results still suffer from unknown biases":
When we draw a sample at random -- that is, when every member of the target population has a known probability of being selected -- we can use the sample to make projective, quantitative estimates about the population. A sample selected at random has known mathematical properties that allow for the computation of sampling error.
Surveys based on self-selected volunteers do not have that sort of known relationship to the target population and are subject to unknown, non-measurable biases. Even if opt-in surveys are based on probability samples drawn from very large pools of volunteers, their results still suffer from unknown biases stemming from the fact that the pool has no knowable relationships with the full target population.
Similarly, in response to a previous Zogby Interactive survey, Blumenthal wrote in an April 26, 2006, blog post:
Why is it important that the survey was conducted online?
1) This survey is not based on a "scientific" random sample -- The press release posted on the web site of the trade group that paid for the poll makes the claim that it is a "scientific poll" of "likely voters." As we have discussed here previously, we use the term scientific to describe a poll based on a random probability sample, one in which all members of a population (in this case, all likely voters) have an equal or known chance of being selected at random.
In this case only individuals that had previously joined the Zogby panel of potential respondents had that opportunity. As this article on the Zogby's web site explains, their online samples are selected from "a database of individuals who have registered to take part in online polls through solicitations on the company's Web site, as well as other Web sites that span the political spectrum." In other words, most of the members of the panel saw a banner ad on a web site and volunteered to participate. You can volunteer too -- just use this link.
Zogby claims that "many individuals who have participated in Zogby's telephone surveys also have submitted e-mail addresses so they may take part in online polls." Such recruitment might help make Zogby's panel a bit more representative, but it certainly does not trans[f]orm it into a random sample. Moreover, he tells us nothing about the percentage of such recruits in his panel or the percentage of telephone respondents that typically submit email addresses. Despite Zogby's bluster, this claim does not come close to making his "database" a projective random sample of the U.S. population.
From the discussion on the 10 a.m. ET hour of the November 27 edition of MSNBC Live, which featured Blumenthal and Iowa-based pollster J. Ann Selzer, whose firm conducts the quadrennial Iowa Poll for The Des Moines Register:
HALL: Here to help us make sense of some of the numbers that are floating around out there, Mark Blumenthal, editor and publisher of Pollster.com, and Ann Selzer, she's a pollster and president of Selzer and Company. Thanks for joining us.
BLUMENTHAL: [unintelligible] to be here.
HALL: All right, Mark, I want to start with you.
SELZER: Good morning.
HALL: Good morning. We were talking in our newsroom. There are two recent polls out showing very different results --
BLUMENTHAL: Right.
HALL: -- when it comes to Hillary Clinton versus the top Republican candidates. You got one poll showing that she'd be beat by these Republican candidates. The other shows that she would come out ahead.
HALL: How do you make sense if you are a person at home and you stumble across one of these reading them online or maybe in the paper, and they're so different?
BLUMENTHAL: Well, that's always the trick, and I'd say go to Pollster.com, and we'll help you figure it out.
HALL: OK, outside of that.
BLUMENTHAL: The Zogby poll -- the one that looks very different from the poll from Gallup -- also looks very different from all the other recent surveys, and what's different about it is it was done online for people who had volunteered to be interviewed online. That particular method by that pollster wasn't all that accurate in 2006. So, I would just -- I'd be more cautious about the online surveys.
HALL: And, Ann, is that a concern how the pollsters get the information? Is, let's say, a phone or online not as reliable?
SELZER: Well, online polls are notoriously unreliable. And with one poll you get one answer, you do another poll the same way the next day, you can get a very different answer. So, telephone has shown to be the most reliable, that is you get the same answers if you do it the same way, consistently.
From the 5 p.m. ET hour of the November 27 edition of MSNBC's Hardball with Chris Matthews:
MATTHEWS: Time now for the Hardball "Big Number" that tells a big story. Tonight, our Big Number is five. That's the number of Republican presidential candidates that Hillary Clinton trails in the November matchups. According to a new Zogby poll, Rudy Giuliani, Mitt Romney, Fred Thompson, John McCain, and, believe it or not, Mike Huckabee, that's five, count 'em, five Republicans all now beating, yes, Hillary Clinton in the matchups for next November, and it's tonight's "Big Number."
Now to the roundtable. Matt Continetti is with The Weekly Standard. Jonathan Carp-- Capehart of The Washington Post; he's on the editorial board. And Julie Mason of the Houston Chronicle. Julie, thank you for smiling.
Wow. I can't believe this poll is actually making news. The methodology is not suspect -- it's simply not valid or reliable since it lacks a random probability sample which allows for the likelihood that anyone with a phone or email would be polled. This is basic stuff folks and we're all fools if we fall for this.
Then why has Hillary's team been requesting these same polls for years?
Dogrun, I'd be willing to bet that a presidential candidate checks all of the pollss.
Here's another take on the Zogby polls.
so the gallup poll you referenced is two days old and shows hillary with a lead over all the republicans. just further proof that the news is what the media says it is.
Two things I like about the Zogby poll;
1. supports the idea that those willing to give out their email addresses to strangers, and those who actually participate in on-line polls, are more likely to be Republicans.
2. If the poll is cited enough,and convinces enough people that a Hillary nomination will result in another GOP Prez., we may get somebody better than Hillary as the Dem. Nominee.
The poll is BS, Scarborough even discussed why it was not 100% valid yesterday.
Brilliant Sue! "Validity" and "Reliability" are terms that are not pedestrian and open to interpretation. Those of us who took statistics and market research classes in college know this.
"Validity" refers to does it measure what it sets out to measure. "Reliability" refers to its ability to be replicated with similar results time and time again. That's why so many poll results are similar and the probability sample is important.
I hate online polls -- and I have done my fair share of them for spin purposes as a PR Director.
I always let clients and others know that they are polls for publicity ONLY and are not valid or reliable but good for "news generation purposes."
Matthews omitted this fact despite statements by the American Association for Public Opinion Research and Democratic pollster Mark Blumenthal -- who appeared earlier in the day on MSNBC -- that such polls are unreliable.
Chris Matthews is also unreliable.
On second thought - he can be relied on to get things wrong.
I heard Sean Hannity at lunch today, and he's very excited about this poll.
In case anybody needed more proof of its bogusosity.
Off topic, but related by my mention of elegant wordsmith Hannity; Hannity noted the other day that Barack Obama is "literally taking the gloves off" in regards to Hillary.
Sorry, pet peeve of mine, people who don't get the difference between literal and figurative and need to highlight it by saying "literally". I'll stop now.
YES YES YES!!!!!!
I agree completely. People seem to think sticking "literally" into a sentence somehow provides emphasis to their figurative statement. I actually heard a weatherman say that it was literally raining cats and dogs. I saw no furry animals falling from the sky.
It literally drives me up the wall.
Along the same line: I was at the drive-through at Taco Bell yesterday, and the little speaker had two stickers on it which said "Order Hear".
There are two that drive me batty:
And you see allegedly educated people making these mistakes all the time.....
You guys need to go easy on the undocumented workers. English is their SECOND language so they may not be as well read or privileged as the MMFA commenters. They are making an honest effort though and I must admit that I've accidentally written "it's" when it should have been "its". I've also had trouble in the past with "who" vs. "whom". I'd give an ESL student a pass on using "hear" instead of "here".
I'm not talking about undocumented workers - I've seen these mistakes almost every day of my 24-year career as a technical writer, and the people who wrote the stuff I've seen these errors in were native-born Americans educated at many of our best engineering schools.
This polling methodology or it's administration seems just as valid as any other - it's a random sampling of liberals, conservatives and independents........just because they register and submit their email addresses to be polled doesn't make their choices any less valid or suspect than a random sampling.
In fact, because they are more active politically, apparently, they are far more likely to be informed and likely to vote......so a case could be made it's a far more valid poll than others.
The simple fact here is that MMFA just doesn't like the results.
Tommy, I think the poll is not the issue, and tells us something we allready know. America is a divided nation and this will be a close election. We can argue and look at polls till we are blue in the face but we are split 50/50. Its scary but true.
I agree with you, but I don't think that was where MMFA's emphasis was.
Tommy,
Good to have you back. If you want to see some fireworks, head to the Pat Buchannan thread ;).
Thanks Fried, Pat always provides fireworks, have read a few of them.
Appropriateness of sampling is not a matter of personal opinion; it's a matter of statistical inference and scientific method. Ask a statistician about the exact issue in this case and she or he will tell you you're wrong about this, Tommy.
All polls are inherently suspect, Zogby is a very reputable polling outfit.
That's like saying "all science is suspect." Studies and polls, like polls, vary in quality; they aren't all equally "suspect." And your blanket statement simply deflects from, rather than negates, the matter of fact that I pointed out.
Please note that Zogby conducts some polls without using a self-selected sample online. (Gee, I wonder why?) To the extent that Zogby or anyone else employs better methodology in a given poll, a reputation is deserved.
sorry for the typo--"studies, like polls" etc.
The point is if this poll had Hillary beating the other 5, it wouldn't be anywhere near here and there would be no misinformation. It's here because of the poll's results exclusively, not their methodology at all.
This polling methodology or it's administration...
This is a classic example of allegedly educated people using the wrong word.
Just so you know in the future, Tommy, the possessive form of "it" is spelled "its". "It's" means "it is".
Tommy, again "validity" is not open to interpretation. It is a social science term and a "random probabality sample" is key to determining validity AND reliability. The "opt in" methodology takes away the "probabilty" that anyone (albeit with a phone) could be contacted and asked their opinion. C'mon man . . . this is basic stuff.
So per this pol Hillary is trailing 5 GOP candidates. This would seem to be a cause for relief from the Goppers.
So which, if any GOP candidate does not beat Hillary in this pol? Julie, Mitt, McCain, Paul, and Huckabie, are all poised per this pol to stomp all over her. Whos left without much of a chance from the GOP?
I figured this item was going to show up today. The real subject of this posting is not Matthews reporting but the unreliability of the poll. Response from Zogby:
Mark Penn: Buckling Under the Pressure of an Unfavorable Poll
All is fair in love and war, the centuries–old proverb states. Politics is not included, but given the way the game is played in modern–day America, maybe it should be. That’s the sense I had again this morning watching Mark Penn, the chief political strategist for Democrat Hillary Clinton, denigrate our latest Zogby Interactive survey simply because it showed his client in a bad light (Link to Latest Poll Number). Penn made the contention on the MSNBC morning news program hosted by Joe Scarborough (Link to Video)
Zogby on "Morning Joe"; Interactive Poll on the Money - VIDEO
Penn mischaracterized this latest online Zogby poll as our first interactive survey ever – a bizarre contention, since we have been developing and perfecting our Internet polling methodology for nearly a decade (Zogby Intreractive Methodology), and since Penn’s company has been quietly requesting the results of such polls from Zogby for years. We always comply as part of our pledge to give public Zogby polling results to any and every candidate and campaign that asks for them. What is interesting is that no other campaign has made as many requests for Zogby polling data over the years than Penn has made on behalf of Clinton.
Because Mark Penn is a quality pollster himself, we chalk up his contention that our poll is “meaningless” as a knee–jerk reaction by a campaign under pressure coming down the stretch. Several other polls – Zogby surveys and others – have shown her national lead and her leads in early–voting states like Iowa and New Hampshire have shrunk. This is not unusual. These presidential contests usually tighten as the primaries and caucuses approach.
Fritz WenzelDirector of CommunicationsZogby International
(11/27/2007)
The point here is that corporate pigs like Matthews is purposefully, consciously, lying in order to assert his agenda to destroy Clinton. Every other poll, read that again, every other poll has Clinton ahead of them all by 4 to 6 points: all of them including the Gallup poll which CAME OUT ON THE SAME DAY AS ZOGBY'S. And Matthews knew it. As did all the other corporate/ pundit pimps who IGNORED the Gallup poll and ejaculated on themselves as they pushed this Zogby poll which all understand to be not an actual poll like Gallop. Matthews knew this and lied anyway. This is what this buffoon does. And he knows it. It's just another example of the arrogance of 'power'.
As far as comparing the sample of the on-line poll vs. phone polling, I would say neither are ideal.
Nobody answers their phone anymore if they don't know the caller or block these surveys from calling, so the sample is skewed towards the only people who answer them - I'd guess seniors.
And the online polls are skewed towards people who register to be polled on-line, probably younger tech-savvy people.
And, for both, people lie. Polls are suspect altogether.
Justice and Truth in the USA - Fact Check:
I just polled everyone at the Truth house -
Hillary did really really bad. Nobody likes her.
- poll methodology
The poll I cited was a stratefied random sample of respondents within the Truth house - and is accurate within +- 1%
In other words, you've left your mother's basement and asked her for her opinion and you posted the results of your "scientific" poll, correct?
Let's be fair....I'm sure he polled his Power Rangers, too.
John Zogby is a fine pollster...
His interactive poll is just one snapshot, and Zogby knows full well its limitations.
Based on same-day polling in 2004, Zogby called the election for John Kerry (311 electoral votes to 213, with 20 too close to call)....This is another important piece of data in the overwhelming evidence that the Republicans stole the 2004 election.
Another: there is an 8.8% difference between the exit polls and the official count in Ohio. They stole it. We are heading rapidly to fascism.
OK Fungus, you say they stole it based on some exit polling. Any more FACTS to back this statement up?
When you reply, please pull the tin-foil helmet off.
I just polled everyone at the Truth house -
Not a valid sample, because they only have eight functioning brain cells among them....
I polled my house. You are an idiot
Even when accidentally the Corporate Media publishes the truth, that accident arises from their instinct and effort to lie. Many posters have noted that the lead here is not the 99% of polling available which shows Hillary leading both Dems and Repugnants - but rather the strange Zogby BECAUSE it shows her trailing those Repugnants.
Too bad that Senator Clinton will be the nominee - I prefer at least Edwards, Kucinich, or Obama - but the effort by this Repugnant / Media axis to drive the stake into her heart which never penetrated President Clinton's.
Just for the record, somehow I got "enter" ahead of the concluding word:
"APPALLS!"
Mark Blumenthal asked for the results of the Poll. If they were good, the Clinton camp would have reported it as supporting them.
Could MMFA demonstrate any more that they are part of Clinton Inc?
Could you demonstate any better you are a complete moron?
How nice of Chris to comment on the one female member of the panel's "smiling."
He really needs to get his hormone levels checked.
I've never trusted online polls. Is this poll accurate? Maybe, maybe not. Even if it is, most of the comparisons fall within the margin of error, don't they? The bigger picture here is that the Media Bobblewhores are slobbering all over it because it gives them their "horse race"...which is all they care about.
I don't trust those polls either.
Whenever I'm surfing for porn, illegal viagra or mindless drivel from FreeRepublic.com those annoying pop-ups ask if I'd like to take part in a poll.
I never do, I go right for the good stuff.
The last 3 exit polls in Germany...
were off by 3/10ths of one-percent.
Germany votes on paper ballots only. As soon as the polls close, exit polls are immediately released. This removes the pressure to spit out a quick result--like in the land of democracy--America.
Not exit polls like those done in America, where the exit poll numbers are "adjusted" to conform to the official count. (Real, meaningful exit poll numbers.)
This country is a banana republic. The road to fascism. There are a number of groups out there that are monitoring elections and trying to preserve democracy.
These groups may have stopped the Cons from stealing the 2006 election. Make no mistake, the Cons are ready and able to steal 2008.
Hm. Numbers.
Ike Eisenhower, genuine American hero, brilliant general in anybody's book, great administrator -- I'll leave the greatness of his Presidency to historians who are WAY smarter than I -- was SHOCKED! SHOCKED to learn that 50% of Americans were below average in intelligence.
Gotta do something about that.
And a few years ago, James Dobson of Focus On The Family actually said that teenage unwed mother birth rates tend to go down above age 25.....
Stop concentrating on Hillary Clinton. She has no chance of winning anything.
Your free advice would be a bargain at twice the price. Hillary is probably the best Republican candidate out there, and would beat any of the other GOP empty suits. If the Dems can get a decent candidate in, Hillary could have a fight on her hands.
When I saw that even Mike Huckabee would beat Hillary Clinton, I knew immediately there was something fishy about this poll. I always thought the Zogby poll was some kind of weird, now this documents that it is.
Zogby's site says nobody buys more poll results from him than Hilary's camp, yet they were out doing 'damage control' on this poll before the ink was dry. If they weren't scared of the numbers, they wouldn't respond. Poor Hillary, from 'inevitable' to 'also ran.'
The Piaps is in trouble, even if the pant suit is asbestos.
Ed, I'm noticing a lot of Republicans posting here with strange advice and predictions about the Dem candidates.
Don't you think your energy might be put to better use somewhere else? Like, for example, trying to dig up a GOP candidate who isn't completely hopeless, rather than worrying about which Democrat is going to beat them?
Your entire party may appear as an "also ran" if you don't get to work.
Ed Schultz used his "progressive" radio show to hype this as a Zogby "international" with the Republican talking points on November 27th. He made no mention of the CNN poll, done by proven, tested methods, that showed the exact opposite results, with HRC beating all of the Republicans, generally by a greater margin than Obama. He had Zogby himself on for part of the program. I've e-mailed my local station and am about to start contacting those who advertise on the program. LOL: One is a bookstore where Hillary appeared to autograph "Living History". I've listened on and off today, and he doesn't seem to have mentioned the Chicago Sun Times story of Obama helping to get $1,000,000 to a former boss without disclosing any conflict of interests.
Matthews is a joke, and the only one who doesn't seem to know it is him. The far more accurate "Gallup" poll came out just two days ago, and it shows Hillary Clinton beating every GOP froint runner beseveral points, (she is beating Giuliani by 5). Of course, Matthews fails to mention that. Fat Chris, is so in the "tank" for Rudy Giulaini, it's pathetic....but hardly surprisng given his status as "Cheif Media Leg Humper For The GOP".