International | Open government data

Out of the box

The open-data revolution has not lived up to expectations. But it is only getting started

ONE night last year Jim Rich came home from a basketball game in Texas with an excruciatingly painful leg. His wife Rosemary, a nurse, feared he was suffering something worse than a bad bruise. She entered his symptoms into iTriage, an app, and found references to compartment syndrome, which can cause paralysis and gangrene. That convinced her husband to go to hospital immediately, says Mrs Rich. He had emergency surgery a few hours later.

The app that helped save Mr Rich’s leg is one of many that incorporate government data—in this case, supplied by four health agencies. Six years ago America became the first country to make all data collected by its government “open by default”, except for personal information and that related to national security. Almost 200,000 datasets from 170 outfits have been posted on the data.gov website. Nearly 70 other countries have also made their data available: mostly rich, well-governed ones, but also a few that are not, such as India (see chart). Open Knowledge, a London-based group, reckons that over 1m datasets have been published on open-data portals using its CKAN software, developed in 2010.

The deluge of transport timetables, crime logs, pollution readings, property-tax records and the like has been a boon. It has allowed governments to serve citizens better, powered innovative startups and improved people’s lives. But it is not yet clear whether it will effect a transformation. For that to happen, the enthusiasm that drove so much information online has to mature into cool-headed pragmatism.

There’s money in free stuff

A few firms have already grown fat on open data. Zillow, a property website, and Garmin, an $7 billion firm that makes navigation software and hardware, were built on free government data. But small firms are likely to be the biggest beneficiaries, says Laure Lucchesi of Etalab, an advisory body set up by the French government. Somebody considering opening a coffee shop in New York can now check the city’s Business Atlas for suitable sites, for example. The interactive map shows which spots are busiest with pedestrians, or have growing numbers of young residents or new restaurants.

In hundreds of cities, commuters use free smartphone apps to check bus arrival times or work out how to get from A to B. Some apps, like Citymapper, also give costs—and calorie-burn counts—for each alternative. Hospital quality audits and individual doctors’ results are powering apps that help patients decide where to seek care. Check That Bike! can be used to determine whether a bicycle has been stolen. GPS, America’s military navigation system, can be found in everything from dating apps to microchips embedded in dog collars.

Many cities increasingly rely on residents to use their phones to report potholes, broken street lights and so on. Earlier this year volunteers for OpenStreetMap, a sort of Wikipedia of maps, helped French authorities build a national address database. The heroic task involved matching 25m addresses, spread across the records of six government agencies, with their geographical co-ordinates.

Government has been made leaner. After the British government published the records of all its contracts in 2010, one official spotted the same analytic reports among several departments’ purchases; the duplicates cost at least £4m ($6m). Slovakia and Ukraine saw an increase in the average number of firms bidding on government tenders. San Francisco’s authorities estimate that a fall in phone queries after it made transport data public in 2012 has already saved more than $1m.

Making data public can also fight corruption. Last year IMCO, a Mexican think-tank, found over 1,400 teachers apparently born on the same day in 1912, prompting a purge of the “ghosts” from payrolls. British and Nigerian officials have used property and company registers published by several governments to investigate money-laundering. OpenCorporates, a website, lists the owners of more than 90m companies by pooling registers from over 100 jurisdictions; its users include watchdogs, law firms and tax authorities.

This is all to be celebrated. Given the astonishing scale of the data deluge, though, it is reasonable to ask why more has not been achieved. There are four answers to that. First, the data that have been made available are often useless. Second, the data engineers and entrepreneurs who might be able to turn it all into useful, profitable products find it hard to navigate. Third, too few people are capable of mining data for insights or putting the results to good use. Finally, it has been hard to overcome anxieties about privacy.

Perhaps four-fifths of the data that have been released are not particularly useful, reckons Joel Gurin of the Centre for Open Data Enterprise, a think-tank in Washington. America’s cache includes a list of the last words of those executed by the state of Texas, for example. The rest is often missing some of the “metadata”—descriptive tags without which the raw information can be meaningless. In some developing countries, official data are so shoddy that businesses prefer to collect information themselves.

Searching open-data portals is often an arduous task. Working out which source is most useful is tricky when dozens have the same information. Registers that were built for administrative purposes only have not yet been redesigned into databases that can be sorted, analysed and matched with other data. Some sets are not kept up-to-date in the portals—a common problem for government-spending records, for example.

Gavin Starks of the Open Data Institute, a non-profit, points to a shortage of data-handling skills among officials, activists and journalists. It is not just that only a few people possess specialist skills, he says; many lack the ability to interpret figures. Mr Starks cites the example of British MPs, most of whom could not solve a simple maths test involving two probabilities.

The thorniest problem for open data now is privacy. Governments rushing to release individual-level data such as tax, medical or education records are “walking into a massive minefield”, warns Martin Tisne of the Omidyar Network, a philanthropic outfit. Such data are among the most valuable: they can boost, for example, precision medicine, which tailors each patient’s treatment. But a privacy scandal can cause a backlash against all open data. A public outcry recently forced Britain’s National Health Service to rethink plans for making anonymised patient-level data available for reuse.

Open-data activists have joined forces with bureaucrats and entrepreneurs to sort out all these problems. Their solutions are starting to work, and growing amounts of data are being put to good use.

At first, officials rushed to put data online without worrying too much about its quality. That is changing. Advocates such as Open Knowledge list the most valuable datasets and the features that make data truly open—such as an open licence or a machine-readable format. Many governments use these lists when they decide which data to release.

Matchmaking events that connect data custodians with analysts, coders and other geeks are becoming more common. Data-users learn which are the best sources; officials learn how to make them useful. Open-data “hackathons” now attract hundreds of volunteers and budding entrepreneurs apiece. They were held in over 200 cities on February 21st, which enthusiasts have declared International Open Data Day. Investors are flocking to such events in growing numbers, increasing the chances that bright ideas turn into successful businesses. America’s health “datapalooza”, launched by the country’s health department in 2010, drew 45 people then; two years later, 230 companies competed for 100 slots to present their inventions to a crowd of over 1,500 people.

Some events simply ask civic-minded geeks for their best ideas. (Britain’s government pleadingly called one competition “Show Us a Better Way”.) Others focus on specific problems. This year American government agencies ran competitions for ideas to help health-care providers tackle obesity, generate visuals of nutrient pollution of water sources and make it easy to explore millions of 19th-century newspaper pages, among others.

A crowd of non-profits are trying to improve people’s data-handling skills. They are publishing handbooks, organising training and coming up with tools that generate easy-to-understand visual data summaries. BudgIT, a startup that distils unwieldy budget documents into simple infographics, is training Nigerian politicians to use its online tools to track the completion of projects included in state budgets. Such efforts are sometimes funded by non-profits and the development agencies of rich countries.

Keeping secrets

The privacy problem is trickier. Some researchers and activists argue that almost all data can be cleverly scrambled to make them anonymous. Others doubt that this can ever be foolproof: as more data are published, it will become easier to mix and match from different sources to figure out who is who. In the end, reckons Mr Gurin, it will come down to a judgment about how much privacy is worth risking for the sake of better medical treatments, for example. Different countries are likely to come to different conclusions: in Sweden, for example, everyone’s income-tax records are online, including names.

It is impossible to predict where the open-data revolution will lead. In 1983 Ronald Reagan made America’s GPS data open to the world after a Soviet missile brought down a South Korean airliner that had strayed into Soviet airspace. Back then, no one could have guessed that this would, one day, help drivers find their way, singles find love and distraught pet-owners find their runaway companions.

This article appeared in the International section of the print edition under the headline "Out of the box"

How to fight back

From the November 21st 2015 edition

Discover stories from this section and more in the list of contents

Explore the edition

Discover more

Narendra Modi’s secret weapon: India’s diaspora

Migrants help campaign for the prime minister at home and lobby for the country abroad

Why young men and women are drifting apart

Diverging worldviews could affect politics, families and more


We’re hiring a global correspondent

An opportunity to join our editorial staff in London