In a speech to the House of Commons in 1941, Sir Winston Churchill declared, “Nothing is more dangerous in wartime than to live in the temperamental atmosphere of a Gallup poll, always feeling one’s pulse and taking one’s temperature.”
While we may not be in a formally declared state of war, the realm of politics these days is in a constant state of battle. And now more than ever, campaigns, pundits and the media have the ability to engage in an interminable cycle of pulse-checking and temperature-taking. From the original Gallup polls to today’s fire hose of public data, the public’s ability to consume data about its own opinions has increased dramatically.
Yet as the amount of data available has grown, the knowledge we have about the sources of that data has not. The appetite for information about political races at all levels has exploded, and the speed demands of the internet age mean it is better to blog now, worry later.
Enter DailyKos and the Research 2000 (R2K) scandal this week. In recent months, R2K’s polling on behalf of DailyKos had been criticized for offering data that was suspiciously favorable to Democratic narratives. They showed Bill Halter leading Blue Dog Democrat Blanche Lincoln in Arkansas – a result that was not to be. They showed Tom Campbell surging against Carly Fiorina in the California Senate primary just as the opposite situation was unfolding.
Their research also was used to make the case that large quantities of Republicans hold “fringe” beliefs, and that, among other things, Republican leaders like John Boehner were very widely disliked by the electorate.
On Tuesday, Markos Moulitsas sent out shockwaves with a post alleging that R2K’s research was fraudulent.
I have no idea if the data produced by R2K was fabricated entirely, manipulated after the fact or produced properly in good faith and just happened to yield results that raise suspicion. Plenty of bright analysis has been done by others casting doubt on the results, and I’ll leave the final verdict to the courts (R2K and DailyKos are now locked in a legal battle over the scandal).
I don’t know why, on request, R2K didn’t “show their work” by sending their raw interview data to DailyKos, a simple move that should be standard practice and something that probably could have put an end to this controversy right off the bat.
But what I do know is that there is at least one positive outcome of this situation: consumers of polling data got a much needed wake-up call.
There’s a perception that polling numbers are simply the results of interviewing a certain number of people. If a poll shows 50 percent of people believe a statement, it must mean that exactly half the people the pollster called said they believed it. There’s a sort of blind trust placed in polling data, and that data ends up driving huge political narratives and strategic decisions in campaigns about where to allocate funds, who to reach out to, and what to say.
What most non-industry consumers of polling data don’t understand is the amount of creative control a pollster has over the outcomes the poll shows.
Many polls are of “likely voters,” and each pollster’s definition of what a “likely voter” is can have a huge role in what the survey shows. Pollsters also usually “weight” their data after the fact, making it match back up with certain demographic realities.
In most cases, weighting or screening of respondents isn’t done with intent to skew data one way or another. On the contrary, it is intended to create more useful results. Yet when pollsters refuse to disclose the way in which they model the data, the public as a whole has no good way to judge whether or not a poll is worth looking at.
It isn’t just about disclosure. It is entirely possible that a pollster would conduct a poll, use mediocre methods, fully disclose them, and the public at large still wouldn’t notice that something strange was afoot. Blogs like Pollster.com and FiveThirtyEight are helping to educate political observers about what to look for in good polling. (Disclosure: I am also a contributor to Pollster.com.)
But when a hot new poll yields headline-making results, it can be all too easy for the media to ignore those doubts. It is clearer than ever that the burden falls on the media to truly vet the pollster and make sure their findings have legitimacy.
There’s a chance that this whole situation will cast doubt on all public polling data, and that a charge of data fabrication will become an unfortunate way to try to discredit data unfavorable to a certain party’s narrative.
But the good guys shouldn’t worry.
While this most recent scandal has certainly rocked the polling industry, I am hopeful that the end result will be a demand for quality over quantity when it comes to survey research.
We may never stop being fascinated by the constant temperature taking that survey research facilitates. But with any luck, this controversy will encourage a more critical look at the ways some choose to check the public pulse.
Kristen Soltis is the Director of Policy Research at The Winston Group, a Washington, D.C.-based strategic consulting and opinion research firm. Soltis is a contributor at The Huffington Post, Pollster.com and The Next Right and has also provided political commentary for The American Spectator, BBC Radio and Ireland’s RTE.