Polls, lies, damned lies and statistics

Throughout the world of political reporters and pundits, the blogosphere and social media — left, right and center — the conventional wisdom was impressive: Obama’s negative ads on Bain Capital and outsourcing were working in the 12 battleground states where they were exclusively running.

First, two weeks ago, came the Wall Street Journal/NBC poll, which triggered an avalanche of reporting reinforcing the themes that the Obama negative ads were hurting Romney and helping Obama big time. The national results were a statistical dead heat (47 percent Obama, 44 percent Romney). But in the 12 battlegrounds — Colorado, Florida, Iowa, Michigan, Nevada, New Hampshire, New Mexico, North Carolina, Ohio, Pennsylvania, Virginia and Wisconsin — where Obama has spent tens of millions on negative ads since April, Obama was ahead 50 percent to 42.

The Journal/NBC poll also showed that in the battleground states, Romney suffered a net increase of 11 percent in negative attitudes toward his business experience — from late May (36 percent favorable to 36 percent unfavorable) to late June (30 percent favorable, minus 6 percent, to 41 percent unfavorable, plus 5 percent). The cause reported by the Journal and NBC: the Obama negative ads.

But not so fast.

Voters were asked in the national survey whether they would feel more or less positive toward a businessman who managed a company that specialized in restructuring and selling companies. To anyone watching Obama’s TV ads — which were seen by most of the country on either TV or the Internet — those words could only describe Bain Capital. The answer between January 2012, before the negative ads ran, and June, after they ran, was no change at all (from minus 6 percent in January to minus 5 percent in June).

In addition, I could find no disclosure in the reporting of the Journal/NBC poll of the subsample total number of voters surveyed in the 12 battleground states. The total surveyed nationally was just 819 registered voters — with a fairly large plus-minus margin of error of 3.4 percent. It is therefore possible that the number of voters surveyed in the subsample among 12 battlegrounds was a fraction of 819 — meaning a considerably higher margin of error, which could make the 50-42 percent “lead” for Obama in the battlegrounds statistically meaningless.

USA Today’s Susan Page reported that among the 1,200 voters surveyed in the 12 battlegrounds, 8 percent of that total, or 96 voters surveyed, said they “changed” their minds because of the ads, from one candidate to the other. Of those who changed, 76 percent (of the 8 percent, or 73 voters across 12 states) said they switched to Obama, and 16 percent (23 voters across 12 states) said they switched to Romney. Page described this data as evidence that Obama’s negative ads were the “clear” winner.

That’s not so clear to me.

First, the subsample is very small — 8 percent of the total sample of 1,500, or 96 people spread out over 12 states — even though the 5:1 change ratio favoring Obama is statistically significant.

Second, to conclude the ads “caused” the change, it is required to have a control group, i.e., the same number of voters surveyed who have not seen the ads. For example, if the results among that control group show the same or similar percentage of changes favoring Obama, that would negate the inference that the ads “caused” the change. Without a control group, you simply cannot be sure of the cause.

Third, in April, a USA Today/Gallup poll in battleground states showed Obama ahead 51 percent to 42. But in this late-June sample, the results were Obama, 47 percent, to Romney at 45 percent. In other words, after Obama spent an estimated $68 million or more over Romney in the battlegrounds on negative ads during this time period, Obama lost a net 7 percent.