Even as we enter the winter months, advances in IT continue to heat up as we anticipate the potential benefits and challenges of emerging technologies. Top of mind is the future promise of artificial intelligence (AI) to revolutionize government, but less discussed is the possibility of a freeze in AI productivity – a result of insufficient data sets or security concerns that prevent data sharing.
Increasingly, government agencies are leveraging AI-based approaches to data analysis in order to predict trends, optimize processes, and speed up decision-making. The success of their efforts depends on the amount and diversity of data that is accessible to AI-based algorithms. Without adequate data to train on, the functionality and accuracy of these algorithms is compromised.
Government agencies often want to share their data, but they may be stymied by siloed organizational structures, fear of losing control over their data and algorithms, or security and privacy concerns. For example, while 91 percent of the public sector geointelligence (GEOINT) stakeholders surveyed by MeriTalk in February 2020 believe AI has the potential to greatly improve GEOINT effectiveness – impacting national security, emergency response, and urban planning – more than half also say security is their biggest challenge as they look to expand AI over the next decade.
Agencies must address these data and algorithm sharing challenges, or they face the danger of an AI winter – starving algorithms with too little data for accurate and meaningful results.
The Digital Data Marketplace Facilitates Secure Sharing
Fortunately, digital data marketplaces (DDMs) – also known as data marketplaces – can enable agencies to overcome data security and privacy concerns, as well as intra-agency silos.
The basic role of the DDM is to organize and facilitate interactions between data suppliers and algorithm developers to explore, select and agree to create, execute, and complete data science transactions. DDMs enable sovereign organizations, which require absolute control of their data assets, to securely offer and make assets available in order to achieve mutual benefits that no organization could achieve on its own.
DDM infrastructure is programmable, community-owned, and secure. Governance models regulate how the members of the marketplace interact. As a result, data providers and data consumers can share, buy, or sell data and algorithms privately and securely without violating government regulations, such as the General Data Protection Regulation.
DDM members create agreements regarding how they want to collaborate and execute transactions for data science workflows, enabling algorithms to train on one or multiple data sources. Agreements between parties are digitized as smart contracts that orchestrate and authorize all necessary steps needed to access and use the data. The contract also controls whether the results – the trained model –of the data science workflow can be moved out of the shared infrastructure.
Multiple Data-Sharing Mechanisms Support Most Government Use Cases
A DDM that supports multiple data-sharing mechanisms can support most government use cases. For example, a DDM should support the following:
- A distributed model, for data providers that want to keep data on premises due to confidentiality or intellectual property concerns. A private cage at the DDM can host an AI training stack and run federated, privacy-preserving algorithms that require higher power density requirements
- A federated model, for data and algorithm providers that prefer their assets remain inaccessible in each other’s locations. The data and algorithms are brought together at a neutral exchange infrastructure cages inside the DDM’s data centers for data trading and algorithm use. Raw data and algorithms are never taken outside the shared cage
- A centralized model, for data providers that are comfortable sharing low-risk or non-confidential assets. Data is stored in a persistent manner in a private cage at the DDM, then moved into public cloud infrastructures used by data science organizations for sharing or model training purposes
A robust DDM enables efficient data and algorithm sharing that meets nearly any privacy or security requirement. As a result, government agencies are assured of the quantity of data they need to achieve meaningful results from AI-enabled data analysis. Instead of an AI winter, they will realize an AI spring of new growth in intelligence and informed decision-making. To learn more, explore the Equinix AI Data Marketplace Solution.