We consider open data as part of a broader trend

We consider open data as part of a broader trend towards “open government”, where open data is combined with social media, mobile technology and other feedback mechanisms to transform the relationships that governments have with citizens and provide better and more relevant public services (such as we could broadly term “citizen-centric” open data). Open data also has the power to improve the lives of individuals through innovation in the private or third sector based on publicly available data sets, resulting in valuable services and economic growth (which we could call “consumer-centric” open data).

Our analysis of these global trends and our experience has emphasized that the US government has the right to think of transparency and open data as an evolutionary process that needs to be flexibly adapted to changing priorities. Public sector leaders and citizens have changed the way they use the available information:

– Citizen user feedback has declined over time, but there is growing external interest in more specialized data sets.

– Performance dashboards gave public employees a good start to enable better governance and comparison between agencies, and they now have an appetite for more detailed analytical capabilities to perform sophisticated analytics and drive change at agencies.

Where the publication of raw data sets has been useful to specialist developers, our experience has not always reached its full potential. Experience so far shows that data is most useful when released in a consumable way, and ease of use (for both the specialized developer audience as well as the non-specialized citizen audience) should be a key consideration in making open data real. Under-utilization of the data that is already in the public domain can be for two reasons: the data is not always in a standardized format and the information market may be underdeveloped. Instead, information must be easily accessible, well-organized, intuitive to use and understand, and data sets must be complete. In order to realize the full potential of open data – for both citizens and entrepreneurs (and to minimize the burden on public services), we think it is important to:

– Short-term publication compromise, but focus on strict data hygiene standards for currently collected data: The data is only as good as their quality. Inaccurate data sets can be misleading and damaging at best. Data quality is central to the open data agenda – but we agree that the cost of completely “clean” data can be prohibitive and that a compromise must be found between quality, cost and timeliness.

– Reduce, reuse, recycle: The burden on public service providers can be minimized by prioritizing the publication of data sets that have already been collected – for example, service quality management information and results rather than collecting new data sets. Providing context will also be important for the publication of performance results data so that they can fully understand this information and to avoid any misinterpretation of this data by the public.

– Make it valuable to citizens: Open data has been most empowering and successful both in the US and internationally when it has been real to citizens, ie. where individuals can relate to the information given and can effect real change. Successive examples show that open data provided in a local (e.g. city), neighborhood, and even street level can be most effective.

– Standardize for easy comparison and interoperability: standardized datasets in uniform formats allow citizens / users / developers to easily compare and analyze information – this is especially important for organizations such as police or local authorities where citizens may want to benchmark organizations against their peers.

– Make data into intelligence: Many public sector organizations are data rich, but marching the different data sets to create usable, analytical intelligence to provide insight (either for managers, commissioners or the public) can be more challenging. Widespread adoption of analytical and predictable modeling techniques that analyze and visualize information would provide easier consumer insight for those inside and outside organizations.

– Make targeted investments to turn “data” into a “service”: The government should make limited and targeted investments to turn a few crucial data sets into “services” by making them easily consumed – good examples are mash-ups like the crime cards. This is especially important for the citizen-centric open data agenda and will help establish a market and stimulate the demand for further publication of data sets.

As part of the prioritization process and as discussed in our overall perspective, we think the government has the right to think of transparency and open data as an evolutionary process that responds to changing needs (from users, developers and public services) over time. Based on our experience, we have developed a framework for accessing open data in this way and adapting to user needs as they become more sophisticated and can be helpful in developing an organization’s open data strategy and identifying the steps to prioritize.

In some circumstances, collecting and processing a raw data set can be prohibitively expensive – so it may be worth assessing the viability of encouraging the private sector to pay and commercialize the data. In such circumstances, the government should consider making it an obligation for the buyer to publish the data set in its raw format. If the government is able to make targeted investments to turn the data into a service, there may be an opportunity to sell those services to the public and generate a revenue stream. Although beyond the reach of the consultation, the government may consider charging a fee for access to anonymous data sets if direct commercial gain can result – e.g. Pharmaceutical companies and health data.

The proposals should also apply to non-public sector providers of public services such as charities or private sector companies, in order for the transparency agenda to be meaningful and truly accountable. It would also help build the case for mixed delivery of public services. We urge the government to implement measures to ensure consistency between the data collected and their presentation to enable cross-provider comparison.

One approach may be to include data publishing as an assessment criterion in the tenders to encourage providers. As a further step, the government may want to consider the possibility of making data collection and consistent presentation a contractual obligation. If this was pursued, it is worth considering transitional arrangements for current contracts until the contract is completed.

Although the legal and regulatory framework exists, it will be important to further consider guidelines and standards for the protection of personal data under the open data agenda. These should include clear guidance on what constitutes personal data in this context (for example, taking into account the many decision announcements under the Freedom of Information Act) and standards for anonymization and pseudonymization (which has great potential in health information). Individuals must be protected on a level playing field, taking into account how data sets could potentially be merged or reconstituted across government so that standards must be consistent and robust.

In order to improve accountability and public service provision, publishing existing management information should be given priority – which should not place an additional burden on organizations as they collect and report this information as a matter of course. However, we recognize that different levels of management information are currently being collected; from operational data, to service measures, and then output and result information. We recommend prioritizing the publication of goals, outputs and service measures. Most operational data is unlikely to be meaningful to the public and will need to contextualize to ensure that they are not misinterpreted.

An effective way to ensure that open data standards are integrated would be to make them a contractual obligation. In addition, we agree with the government’s goal of ensuring that the public sector is an intelligent and demanding customer. As part of this, it would be helpful to create focus groups that allow vendors and other stakeholders to contribute to open data standards in a productive way.