Big Data’s Big Moment

Photo of Robert Cresanti
Robert Cresanti

It’s no surprise that we’re collectively taken aback by the accuracy of Nate Silver’s predictions, along with the predictions of the few other researchers who follow similar models. After all, so many months of punditry, analyses, and polls were instantly overshadowed by the application of data functions to the Electoral College.

Quite simply, it’s big data’s big moment.

But rather than breathlessly celebrating this achievement, we should take a step back and realize that Nate Silver’s prediction isn’t an anomaly. Rather, it’s a straightforward example of what big data can, and does, achieve on a daily basis. At SAP, we understand this – we provide big data tools to clients throughout the public sector, and during the campaign, we conducted sentiment analyses of the big data resulting from social media platforms to help provide another level of insight into the public’s thinking.

In truth, the name “big data” is a bit of a mischaracterization. The concept isn’t just about the amount of data itself – it’s about how we aggregate, analyze, and apply that data to the real world – just as Nate Silver did.

The ability to harness the wealth of information in the world – and to apply it in a way that gives us better predictability and heightened efficiencies – offers untold possibilities. And in the U.S., where we are confronting rapid population growth and limited resources, the importance of this cannot be denied.

We must learn to better take advantage of big data – not just because the insights could offer us new benefits, but because we also must avoid information overload by managing the wealth of data created each year.

According to a recent report from TechAmerica (which was co-authored by a colleague within SAP), in 2011, 1.8 zezabytes of information were created globally, and that amount is expected to double every year. This is the equivalent of 200 billion, two-hour high-definition movies, which one person could watch for 47 million years straight. This impacts both companies and governments, and in the absence of appropriate management tools, it threatens to overwhelm them.

But it also presents opportunities. Mr. Silver doesn’t make public his processes, but one can assume that, like most big data calculations, it involves incorporating many different data sets – for his purposes, for example, he would look not just at the results of polls, but also of the historical voter turnout in each state. He must evaluate the whole picture in order to achieve an accurate outcome.

That’s exactly how big data can, and does, affect our lives now. And with greater adoption of real-time (rather than retroactive) big data tools, the effects in the future will be even more dramatic.

For example, local police departments can use real-time big data tools to develop predictive models about when and where crimes are likely to occur by identifying patterns that might otherwise have gone unnoticed. That creates a system where police are not just solving crimes – they are preventing them.

Or, using real-time big data tools, regional power providers can measure ongoing power consumption from individual houses, identifying periods of peak demand in order to moderate energy use or warning when the grid might be subject to overload, averting a blackout.

And big data can improve medical researchers’ ability to aggregate the results of clinical trials, leading to more-thorough medical understanding and, potentially, more-targeted treatments. And once those new treatments, whether they be pharmaceuticals or medical devices, come to the market, both manufacturers and physicians will be better able to monitor patients in case of any emerging safety concerns.

Big data’s analytical and predictability capabilities were made clear through Mr. Silver’s election predictions, but its potential impact on businesses, the government, and the public can impact us daily – not just every four years.