On June 12, Hardball host Chris Matthews aired two separate on-screen graphics, based on an NBC News/Wall Street Journal poll conducted June 6-9, that falsely suggested that Sen. John McCain's lead of 44 percent to 38 percent over Sen. Barack Obama among white suburban women is statistically significant. Specifically, the charts provided only the margin of error for the survey as a whole -- 3.1 percentage points -- and not the margin of error of 9.34 percentage points for the crosstab of white suburban women.

Huffington Post reporter Seth Colter Walls wrote on June 12 that after Huffington Post emailed “NBC's political unit asking for a numeric breakdown of 'suburban women' in their poll,” “MSNBC has now provided The Huffington Post with more information on its 'suburban women' finding showing a 44-38 McCain lead over Obama. 'This is within the margin of error of 9.34 percent based on a sample size of 110 within the larger poll,' an MSNBC source wrote over email. (That's three times the margin of error for the entire poll.) This means McCain's 44 percent figure of support among suburban women could actually be as low as 35 percent, while Obama's 38 percent figure could rise as high as 47 percent -- assuming a 95 percent confidence interval (for the stat wonks in the house). Alternatively, McCain could be leading Obama 53-29. While those distant outcomes are less likely true than NBC-WSJ's 44-38 finding, that broad variance raises questions about the statistical usefulness of this one particular crosstab, as opposed to the rest of the NBC-WSJ poll on the whole.”

From Walls' June 12 Huffington Post article:

Since that poll was released Wednesday, MSNBC has been reporting the “suburban women” finding often, sometimes hour-by-hour -- perhaps because it reinforces residual doubts about Obama's viability in the 'burbs. The question led Chris Matthews' “Hardball” program at 5pm on Wednesday with a graphic that read “Woman Trouble?”

But how solid was the NBC-WSJ poll's conclusion about those voters? With only 1,000 total respondents in the poll, and no guarantee that the sub-group of “suburban women” was balanced nationally -- meaning that these suburban women polled were drawn from a balanced cross section from America's vast suburbia -- a group of polling experts from across the ideological spectrum told The Huffington Post they viewed the findings with some suspicion.

“I am skeptical about results for smaller subgroups like 'suburban white women,'” said Emory University Professor of Political Science Alan Abramowitz. “There is more random 'noise' with smaller subgroups. How many of these 'suburban white women' were there in the NBC poll out of the 1,000 total registered voters? Figure about 750-800 whites, close to 400 white women, so maybe 150-200 suburban white women. A shift of a small number of voters would change the outcome.”

Republican pollster Tony Fabrizio, who you might expect to celebrate the poll, also cast doubt on its findings regarding a preference among suburban women for McCain -- if for no other reason than that any one poll's margin of error increases as the reduced sample size of a “subgroup” becomes smaller and smaller. “If you have a thousand samples, maybe your margin of error for your overall sample is 3.1 percent [the margin cited in the NBC-WSJ poll].” Assuming approximately half of the poll's respondents were women, he said, “your margin of error [when considering them alone] goes up to 4.5 percent. Then take females and segment them among rural, urban and suburban [subgroups]. ... You've doubled your margin of error in that group.” [See update below: MSNBC revealed the margin of error tripled for its “suburban women” subgroup.] Fabrizio also said that disrupting the national distribution of a sample by looking at subgroups can throw any analysis out of whack by the same proportion, adding, “there are other vagaries that can go on to influence the sub-sample, too.”

Democratic pollster Anna Greenberg agreed with Fabrizio, saying “I'm not sure I believe” the NBC-WSJ numbers on suburban women. She also said it was “bizarre” to single out the suburban women numbers as a statistically significant finding, given the more robust evidence that Obama was competitive with McCain among white women overall. “They missed the big story among women, in my view,” she said. “It's not like white women are a 'gimme' for Democrats. Obama is doing significantly better among college-educated white women [than Sen. John Kerry in 2004].”

[...]

MSNBC has now provided The Huffington Post with more information on its “suburban women” finding showing a 44-38 McCain lead over Obama. “This is within the margin of error of 9.34 percent based on a sample size of 110 within the larger poll,” an MSNBC source wrote over email. (That's three times the margin of error for the entire poll.) This means McCain's 44 percent figure of support among suburban women could actually be as low as 35 percent, while Obama's 38 percent figure could rise as high as 47 percent -- assuming a 95 percent confidence interval (for the stat wonks in the house). Alternatively, McCain could be leading Obama 53-29. While those distant outcomes are less likely true than NBC-WSJ's 44-38 finding, that broad variance raises questions about the statistical usefulness of this one particular crosstab, as opposed to the rest of the NBC-WSJ poll on the whole.* [Added later]

From the June 12 edition of MSNBC's Hardball with Chris Matthews:

MATTHEWS: And the latest NBC News/Wall Street Journal poll shows that Obama -- Barack Obama -- trails McCain among suburban women voters by 6 points and among white women -- or rather white men, by 20 points. How does Obama connect with white men and keep suburban women from going for McCain? We've got an expert to join us who's been with him in the trenches, Illinois Senator Dick Durbin, his national co-chair.

[...]

MATTHEWS: We're back with the round table and the politics fix. Let's take a look at a new poll here, the NBC/Wall Street Journal poll. It shows that John McCain is leading Barack Obama among white suburban women by six points. But among all white women, Obama's ahead. That six-point spread there is a situation some people, [Financial Times managing editor] Chrystia Freeland, think can be easily reconciled by a simple exposition of the difference between him and McCain on abortion rights.

FREELAND: I think that is a very, very important point, and that's going be a real strength for Barack Obama. You know, the other thing, Chris, that I think is really important to remember is it's less than a week since Hillary Clinton suspended her campaign. And while they were in the heat of the nomination battle, it was very hard for Barack Obama directly to go out for the female vote. He was, after all, up against the first credible female contender for the White House.

So, I think we'll need to wait a few more weeks, post-Hillary, to see how well he does on a level playing field, as it were, with women.

MATTHEWS: Jonathan, it's an interesting thing, we're here talking about ethnic differences and politics. But it is a cutting edge story, and it's fair enough we've got to do it. There is a resistance out there among white voters to the Democratic Party. Let's face it, they have resisted McCain -- I'm sorry, they resisted Kerry. They resisted Gore. White men as a group tend to be Republicans now. They're sportsmen, gun owners, conservative on issues of crime and foreign policy. But doesn't he have to do better than losing by 20 points on that score?

JONATHAN CAPEHART (Washington Post editorial writer): Yeah, he has to. And how he -- exactly how he does that, at this point, is a little beyond me. I mean, he was having trouble with sort of overall blue-collar white voters when he was going toe-to-toe with Senator Clinton. And I think that's why it's going to be very important, you know, once Senator Clinton comes back from vacation and recharges and rejuvenates, that she and her allies -- [Pennsylvania Gov. Ed] Rendell, [Ohio Gov. Ted] Strickland, and all those guys -- get out there and, you know, sing the gospel, if you will, that Senator Obama is the one they should be looking at and not Senator McCain.

MATTHEWS: OK. Cory [sic], I only have 10, 20 seconds.

JIM WARREN (Chicago Tribune managing editor): Chris --

MATTHEWS: Jim, what does he do? Can he do it?

WARREN: Well, yeah, on a couple things. First of all, early data suggests that 60, 70 percent of the Clinton women are going to go his way. And if you're talking about the suburban females, educated, affluent -- they tend to be anti-war. They tend to be pro-environmentalism. They tend to be pro-health care --

MATTHEWS: Right.

WARREN: -- a lot of aging baby boomers taking care of aging moms and dads. They're going to be inclined to Obama.