Photo by BoliviaInteligente.
Barry Johnson and Aaron R. Williams
This is the latest blog in a series of posts that focuses on the importance of the federal statistical system and the challenges it faces to ensure its products remain timely, relevant, and trusted.
As we noted in our prior posts, the federal statistical system (FSS) needs to increase its use of commercial, administrative, and survey data, in combination, to offset the high costs and low response rates associated with surveys and to meet user demands for more granular and more timely data. The system must do this without sacrificing the high quality and reliability that sets official statistics apart from other data sources. In the past, FSS expert staff used highly customized computer systems to collect, test, and correct bespoke survey data, to produce ‘gold standard’ statistics. Updating these legacy processes to accommodate expanded use of alternative data sources could be costly, but transitioning to responsible AI-based tools can scale staff expertise to better leverage the value of alternative data sources.
What is AI?
The U.S. Office of Management and Budget uses a broad definition of AI that includes any artificial system that performs tasks under varying and unpredictable circumstances without significant human oversight, or that can learn from experience and improve performance when exposed to data. Included, for example, are artificial systems that solve tasks requiring human-like perception, cognition, planning, learning, communication, or even physical action.
How can AI be incorporated to improve federal statistics?
AI systems use a broad range of techniques that include machine learning, reinforcement learning, transfer learning, natural language processing, and generative AI. These tools can contribute to the development of official statistics in at least five ways that cover the full data lifecycle: collect, clean, analyze, and disseminate. Of course effectiveness may vary, so any tool used for official statistics should be carefully and thoroughly evaluated before being put into production. In particular, AI models should be carefully reviewed for bias as they are known to amplify societal biases that may be reflected in training data.
- Transforming text into usable data
Natural language processing (NLP) involves the ability of a computer system to analyze, interpret, and generate human language in a way that is both meaningful and useful. This approach can help federal statistics tap into new data sources and make existing processes more efficient. For example, the 2021 Canadian Census, used a NLP algorithm to assign codes to write-in survey responses that could not be processed using their standardized code look-up system, primarily due to misspellings or non-standard terminology. In the past, these would have been resolved by hand. Using NLP, almost 7 million responses across 31 questions were corrected and then successfully coded, saving approximately $4M and several weeks of processing time. NLP can also be used to extract data from unstructured text. As an example, narrative comments from survey respondents or written interviewer observations, that can then be used for analyses.
- Identifying and correcting data errors
AI based methods are rapidly revolutionizing approaches to improving data quality. One approach uses supervised machine learning models, those that are trained on already corrected data, to detect and correct errors in raw data. For example, Statistics of Income at the IRS used this approach to improve industry codes that taxpayers provide on Form 1040. NLP can be combined with these models to standardize spelling and terminology in a dataset, an approach the National Center for Health Statistics uses in its cause of death coder.
Unsupervised models, as the name implies, are not trained on perfected data, rather they are applied directly to a raw dataset to identify relationships and patterns among variables. This can be an effective way to identify anomalies, enabling the data producer to detect previously unknown types of errors and those related to emerging trends, that otherwise might have gone unnoticed using legacy data cleaning methods. An example is the increase in cryptocurrency transactions and investments may lead to new types of data anomalies. Hybrid, or semi-supervised methods are also showing promise for anomaly detection, especially when distributions are highly skewed.
- Imputing missing data
Missing data pose some of the same challenges as errors to data producers. For example, a survey respondent may fail to answer every question or may choose to provide answers within a range (e.g. “my age is between 50 and 65”) rather than a precise answer. In both cases, data producers often impute a value to aid analyses. Supervised and unsupervised machine learning methods can be used to assist with this problem, where they may provide more nuanced and precise approximations of true values than some legacy imputation methods.
- Linking records from disparate sources
As the Census Bureau notes, the number of available administrative and commercial data files has grown exponentially and present the FSS with opportunities to link data from these often disparate sources to produce official statistics. The supervised, unsupervised and hybrid machine learning approaches discussed above can support dynamic record linkage, to enable simultaneous and accurate linkage of multiple, large data sets even when there are no training data. Successful applications can reduce false negatives and false positives relative to older linkage methods. Similar to legacy linkage methods, AI-based methods must strictly adhere to privacy protection requirements associated with each candidate data set. These approaches may also improve agencies’ ability to identify and eliminate duplicate records in datasets.
- Developing and disseminating data products
AI can help make government data findable, accessible, interoperable, and reusable (FAIR). Generative AI can help agencies develop new, timely reports, and make existing reports more understandable and available in more languages. New tools like Model Context Protocols, and old tools like APIs, can help 3rd-parties, ranging from dashboards and Google Search to chatbots like claude.ai, ChatGPT, Microsoft Copilot, etc. to directly access government data. In another example, the planned National Secure Data Service’s Data Concierge Service, if implemented, could act as a front-door to the decentralized FSS, providing a conversational interface to help users more easily access detailed data or find answers to basic questions. Finally, AI can support creation of new data access modes. For example synthetic microdata (artificially generated data that mimic real-world data) produced using nonparametric machine-learning tools, coupled with a privacy preserving validation process can expand user access to data while protecting the privacy of the individuals behind the data.
WARNING! AI Must Be Used Responsibly!
While AI systems hold much promise for improving federal statistics, they can pose considerable risks if they are not responsibly managed through good governance protocols. The steps needed to ensure AI does not cause more harm than good can vary, but appropriate AI Governance before, during and after deployment is essential and should include:
- Clearly defining the purpose and objective of the AI system.
- Ensuring the explainability of the AI system by documenting the specific data used, methodology, and guardrails.
- Establishing (and meeting) appropriate benchmarks for accuracy, reliability, and if needed bias mitigation, before deploying an AI system.
- Continuously monitoring and testing the AI system for adherence to established standards once deployed.
- Securing and protecting the privacy and confidentiality of individuals whose data is accessed by the AI system during storage, processing, and dissemination.
It also important to note that agencies also face challenges from bad actors who would use AI in ways that potentially undermine the public trust in official statistics, a topic we address in our next post.
Given the potential benefits and challenges, close partnership between data scientists, statisticians, and subject matter experts is therefore foundational for developing responsible AI systems. In a noisier world where statistical agencies need to work hard to build user trust, the expertise of the professionals who live and breathe the data used to produce statistics that shape our understanding of the world is more important than ever.