Is Nate Silver’s value at risk?

Photo of Sean M. Davis
Sean M. Davis
COO, Media Trackers
  • See All Articles
  • Subscribe to RSS
  • Bio

      Sean M. Davis

      Sean M. Davis is the COO of Media Trackers. He received an MBA in finance from The Wharton School in 2010 and previously served as CFO of The Daily Caller.

The answer is: not all that much beyond what is already contained in state polls. Why? Because the FiveThirtyEight model is a complete slave to state polls. When state polls are accurate, FiveThirtyEight looks amazing. But when state polls are incorrect, FiveThirtyEight does quite poorly. That’s why my very simple model and Silver’s very fancy model produce remarkably similar results — they rely on the same data. Garbage in, garbage out.

So what happens if state polls are incorrect?

In 2008, the RealClearPolitics polling average was incorrect in two states — Indiana and North Carolina. Silver botched Indiana but correctly called North Carolina.

In 2010, it was much worse. State polling averages were wrong in Alaska (they said Joe Miller would be elected), wrong in Colorado (they said Ken Buck would be elected), and embarrassingly wrong in Nevada (they said Harry Reid would be involuntarily retired). FiveThirtyEight incorrectly forecast the winner in each of those states, perfectly reflecting the inaccurate information contained in the state polls.

Thus, of the five major state races in which polls were wrong over the last four years, Silver only got one right. I’m no baseball scout, but batting .200 when it counts won’t get you into the big leagues, let alone the All-Star game.

Silver has made a big deal this election cycle about how state polls are generally more accurate at forecasting the winner of the Electoral College and the popular vote than are national polls. That may well be true, although a Monte Carlo simulation of the final week’s worth of Florida polls in 2000 suggests otherwise.

But assuming it is true, how much actual data in the era of modern polling do we actually have? Maybe three presidential elections’ worth, going back to 2000. Or, if you want to be really generous, maybe eight if you go back to 1980.

Wall Street had exponentially more data when it incorrectly bet on the housing market than we have today when it comes to presidential election polling data. Although we pundits may think a handful of elections qualifies as a robust data set, political polling data simply pales in comparison to the wealth of data we have on the stock market, economic output, or life expectancy, going back to the example of the insurance actuary. But is the science settled on how Stock X will perform tomorrow, or what precise economic growth we’ll see next quarter, or exactly how long each of us will live? Of course not.

This takes us back to Nassim Taleb’s key insight: despite our best efforts, we humans are just not that good at predicting the future. The main assumption underlying Nate Silver’s Obama bet this year is that the state polls will be correct. Maybe they will be, even though three states were wrong in 2010, two states were wrong in 2008, one state was wrong in 2004 (WI), and a very important state in 2000 was incorrectly called by most pollsters.

Nate Silver’s model could very well forecast every state correctly next week, assuming the polls accurately reflect the true voting population. But if they’re wrong, it’ll be Nate Silver whose value is at risk. If that happens, I have a great title for his next book: “The Snake and the Oil.”

Sean M. Davis is the COO of Media Trackers. He received an MBA in finance from The Wharton School in 2010 and previously served as CFO of The Daily Caller.