The great promise of Big Data is that it can provide us with information that enables us to understand ourselves better, make better decisions and improve our quality of life.

More data is being created from more places today than ever before. Tweets, clicks, YouTube videos, retailer loyalty cards, cell phones, even sensors on buildings are producing tons of data daily. Trends in public sector data transparency are adding even more valuable data to the mix. Due to the enormity of the volume and variety of the sources, these data are often referred to as ‘Big Data’. A recent New York Times article defined Big Data as ‘shorthand for advancing trends in technology that open the door to a new approach to understanding the world and making decisions.’ The great promise of Big Data is that it can provide us with information that enables us to understand ourselves better, make better decisions and improve our quality of life. The great challenge of Big Data is, of course, that it deprives us of privacy and allows for public and private sector intrusion in our lives. These benefits, if managed well however, are significant and have the potential to make cities much better places:

1) End of Data-Free Decision-making . Many people often criticize government for making decisions without full consideration or understanding of the facts. Big Data can really change that. If harnessed right, it can allow us to make better decisions about people and places. On the people side, data collection systems have evolved rapidly over the last decade with more sophisticated and varied sources for capturing information including 311 calls, educational performance and health care. On the place side, more buildings, roads and machines today have sensors that are providing data 24/7 about volume of usage, energy consumption, etc., what people often refer to as the Internet of Things. This proliferation of data can take the current obsession with evidence based decision-making to a whole new level; potentially tying an array of decisions to real-time, current and substantial data sets.

2) Opportunities for Accelerating Technology for Civic Change . The growing trend of the public sector to make more of its data open to the public has led to an explosion of innovation and is redefining how citizens participate and interact with their government. To date, ‘civic tech’, a focus of my blog post on January 26, has focused on improving civic life generally, from real-time bus schedules to virtual land use planning. However, it’s not hard to imagine how civic tech, intentionally applied to the lives of low-income people and communities, could be transformational – from changing the relationship between police and neighborhoods to enabling online appointment scheduling and enrollment for public benefits that now force people to take off work or suffer face-to-face humiliations.

3) Unknown predictive possibilities . The predictive power of Big Data is being explored and will be a big part of improving fields as diverse as health care, economic development and education. We are already seeing the potential to use big data to predict student performance in states like North Carolina where education leaders are using high-tech data analytics to examine grades, attendance, course failures, declines in grade point average, and disciplinary incidents of elementary school students to predict who might be at risk of falling of track and even failing to graduate high school. Big data’s predictive power is also of increasing interest to police forces that face greater limitations due to municipal budget constraints. New efforts are emerging in cities to map the location and time of crimes to better understand where police need to be and when to provide the greatest benefit to communities. This predictive element of big data can dramatically improve effectiveness and drive efficiency and this will be one of the most exciting areas for cities to focus on especially as more and more of the world’s population continues to move to metropolitan areas.

The greatest promise of Big Data is its potential to become Humanity’s Dashboard, as Rick Smolan noted in the New York Times. Can we use it to continually understand ourselves better; to know where our public dollars are actually working or where our human and financial resources should be concentrated today to make the biggest difference for tomorrow? How we harness Big Data for common good will be a big question for the next decade.