Statistical frameworks for reliable machine learning predictions and inference

Abstract

Machine Learning (ML) models often achieve high accuracy, but fail to meet the reliability and robustness standards required for Official Statistics (OS). Neural networks, in particular, function as black-box predictors prone to overconfidence, offering no direct method to measure true uncertainty in predictions and estimates of population parameters. Non-rigorous approaches include treating ML predictions as gold standard data and heuristic notions of uncertainty, like softmax scores in classification problems, as valid measures of confidence. This can easily lead to unreliable uncertainty quantification. This paper handles two distinct problems: (1) quantifying prediction-level uncertainty for new observations and (2) quantifying noise-free uncertainty for estimates of population parameters. We propose handling the former via conformal prediction (CP) and the latter using prediction-powered inference (PPI). Both are model-agnostic statistical frameworks for uncertainty quantification. Finally, we present real-world use cases for OS, applying both techniques.

1. Introduction

The production of Official Statistics is increasingly reliant on machine learning,^1,2 especially to handle data imputation and non-traditional data sources. However, while flexible and often accurate, Machine Learning (ML) models, and especially Deep Neural Networks (DNNs), often fail to meet important statistical requirements necessary for rigorous prediction and inference, Uncertainty Quantification (UQ) in model predictions and inference on population parameters. A well-documented issue with DNNs is their tendency to produce overconfident estimates for probability distributions, especially with out-of-distribution (OOD) data. ³ This is problematic, as DNNs can be just as confident in incorrect predictions as they are in correct ones. As a result, if we solely rely on heuristic notions of uncertainty, such as softmax scores in classification problems that approximate a probability distribution over classes but do not quantify the true uncertainty, ⁴ we risk incorrectly estimating it.

In Official Statistics (OS), two main types of uncertainty are of particular interest: (1) prediction uncertainty, i.e., the uncertainty in the model’s prediction for a previously unseen data point, and, perhaps more importantly, (2) inference uncertainty, which pertains to the estimation of a population parameter from sample data. We propose tackling prediction uncertainty using Conformal Prediction (CP) techniques, ⁵ and parameter uncertainty via Prediction-powered Inference (PPI). ⁶ Both utilize predictions from a pre-trained model applied to labeled and unlabeled datasets.

CP is a model-agnostic, distribution-free framework for quantifying prediction uncertainty. It relies on labeled data to construct uncertainty measures, yielding prediction sets for classification and prediction intervals for regression tasks. Similarly, PPI uses minimal distributional assumptions to derive robust statistical properties from model predictions, but it also produces statistically valid, noise-free confidence intervals for population parameters such as means, quantiles, and regression coefficients. Recent work has sought to derive parameter confidence intervals from CP’s prediction intervals (e.g., conformal confidence regions ⁷ ), which appear valid for finite samples without strong noise assumptions. However, this analysis relies on PPI, given its established methodology and the likely verification of its noise assumptions with most datasets.

2. Related work

The literature on integrating machine learning into Official Statistics has grown substantially in recent years. In their manifesto, Puts, Salgado, and Daas ⁸ outline the broad vision of integrating ML in OS, focusing on opportunities and challenges that need to be addressed. Specifically, they call for a balanced approach to preserve the statistical rigor necessary for policy-relevant outputs. Dumpert et al. ⁹ propose a comprehensive quality concept for using ML in OS. They extend traditional quality frameworks by incorporating dimensions such as accuracy, reproducibility, explainability, timeliness, and cost-effectiveness, focusing on operational contexts as they provide guidelines for the systematic evaluation and monitoring of ML algorithms in statistical production processes. In a complementary vein, Molladavoudi and Yung ¹⁰ focus on the quality dimension of trustworthy machine learning. Their work handles model explainability and uncertainty quantification, as well as practical insights to embed these dimensions within existing quality assurance frameworks at national statistical offices. Van Delden, Burger, and Puts ¹¹ articulate ten propositions on the role of ML in OS, aiming to provide strategic recommendations on methodological considerations and operational challenges. Additionally, Nunes and Ashofteh ¹² review the operational dimensions of big data and machine learning operations, with a particular focus on the adoption of MLOps frameworks. Finally, Breidt and Opsomer ¹³ demonstrate how traditional model-assisted survey estimation can be enhanced with modern prediction techniques, showing how variance reduction and improved efficiency can be achieved when integrating machine learning predictions into established statistical estimators.

A broad literature on uncertainty quantification (UQ) complements recent work on conformal prediction (CP) and prediction-powered inference (PPI). Classical UQ methods include Bayesian inference, via Bayesian neural networks (BNNs) and full posterior or MCMC-based predictive distributions,^14–16 resampling and simulation techniques such as the bootstrap^17,18 and parametric Monte Carlo, ¹⁹ and ensemble or committe methods including bagging, ²⁰ random forests, ²¹ deep ensembles, ²² and MC-dropout. ²³ These approaches provide rich representations of epistemic and aleatoric uncertainty and are effective when model structure and priors are credible, but their validity typically depends on correct model specification, prior assumptions, or large-sample asymptotics. In contrast, CP yields model-agnostic, distribution-free finite-sample marginal coverage under exchangeability,^5,24 making it robust to misspecification and out-of-distribution inputs, though its guarantees are marginal rather than conditional and can be conservative under heteroskedasticity. ²⁵ PPI uses predictions to improve efficiency in estimating population parameters while retaining frequentist validity or confidence intervals under mild regularity conditions, ⁶ in contrast to fully Bayesian posterior-based inference.

Uncertainty quantification methods differ substantially in their computational costs. Full Bayesian approaches, such as BNNs with MCMC integration, offer detailed posterior uncertainty but are computationally intensive and difficult to scale to large models due to costly sampling and integration steps. Approximate Bayesian techniques such as MC dropout and deep ensembles reduce this cost but still require multiple forward passes or model training procedures, increasing inference time and memory requirements with the number of samples or ensemble members. In contrast, CP is model-agnostic and computationally lightweight: once a predictor is trained, as it primarily involves calibration on a held-out set rather than retraining or sampling, yielding valid coverage under mild exchangeability assumptions with minimal overhead. PPI similarly builds on fixed model predictions to produce confidence intervals for population parameters, with the main cost tied to aggregating predictions across labeled and unlabeled data rather than sampling or hyperparameter tuning. In general, Bayesian and ensemble-based UQ methods aim to model uncertainty and can yield detailed estimates when their assumptions are well justified, whereas CP and PPI aim to guarantee validity with minimal reliance on correct model specification, which is crucial when considering the ever-increasing use of black-box large models to analyze unstructured data, especially from textual sources.

3. Methodology

Here, we briefly introduce the concept of Uncertainty Quantification (UQ) and later define the main characteristics and properties of CP and PPI, as well as illustrate their algorithms.

3.1. The premise of uncertainty quantification

UQ is a broadly explored field in predictive inference and machine learning.^26,27 Its high-level goal is to assess, represent, and mitigate uncertainty in prediction and inference to achieve robust, trustworthy results and improve model interpretability. At its core, the general premise of UQ is to systematically characterize and quantify different sources of uncertainty, ensuring that both aleatoric (inherent data randomness, irreducible by nature) and epistemic (model-related, reducible with better modelling) uncertainties are accounted for Der Kiureghian and Ditlevsen. ²⁸ To illustrate this in Bayesian terms, we can write:

$$\begin{array}{r}p(y|x,D)=\int p(y|x,\theta )p\left(\theta |D\right)d\theta ,\end{array}$$

where $p(y|x,D)$ represents the likelihood of the outcome given the model parameters $\theta$ , and $p(\theta |D)$ is the posterior distribution over the parameters given the observed data $D$ . In this formulation, aleatoric uncertainty is captured in the conditional likelihood $p(y|x,\theta )$ , while epistemic uncertainty is encoded in the posterior distribution over parameters $p(\theta |D)$ .

Softmax “probabilities”. In classification tasks, DNNs typically use a projection (classification) layer that maps a $d$ -dimensional input to a $k$ -dimensional space, where $k$ is the number of classes. The $softmax$ activation is often applied to the projection layer’s output:

$$\begin{array}{r}softmax\left(z_i\right)=\frac{e^{z_i}}{\sum _{j=1}^k{e}^{zj}},\end{array}$$

where $z$ is the $k$ -dimensional output of the projection layer, and $z_i$ is the $i$ th dimension of the $\mathbf{z}$ vector. The $softmax$ function transforms the vector elements into values between $0$ and $1$ , summing to $1$ .

Prediction vs confidence intervals. Prediction intervals (PIs) and confidence intervals (CIs) capture conceptually different types of uncertainty. A PI is designed to represent the range within which a future observation $Y_{n+1}$ is expected to fall with a pre-defined confidence level $\alpha$ . Prediction intervals account for both the uncertainty in estimating the underlying model and the inherent variability in the data. For example, in a linear regression setting, a $1-\alpha$ prediction interval for a new observation given covariates $X_{n+1}$ can be expressed as:

$$\begin{array}{r}\hat{f}\left(X_{n+1}\right)\pm z_{1-\alpha /2}\sqrt{{\hat{\sigma }}^2+\mathrm{Var}\left(\hat{f}\left(X_{n+1}\right)\right)},\end{array}$$

where $\hat{f}(X_{n+1})$ is the point prediction, ${\hat{\sigma }}^2$ is the sample variance of the residuals, and $t_{1-\alpha /2}$ is the quantile from the $z$ -distribution. On the other hand, a CI quantifies the uncertainty in the estimation of a fixed model parameter, reflecting primarily epistemic (noise-free) uncertainty. For example, a $1-\alpha$ confidence interval for the population mean $\mu$ based on a sample $\{Y_i{\}}_{i=1}^n$ is given by:

$$\begin{array}{r}\overline{Y}\pm z_{1-\alpha /2}\frac{s}{\sqrt{n}},\end{array}$$

where $\overline{Y}$ is the sample mean, s is the sample standard deviation, and $n$ is the sample size. Therefore, CIs provide a measure of confidence in the estimated parameters, and PIs incorporate the additional uncertainty associated with predicting a new, individual outcome.

3.2. Conformal prediction

Conformal prediction is a statistical framework for UQ that derives prediction sets or intervals from a model’s predictions. Its appeal lies in its model-agnostic nature and its ability to provide finite-sample, distribution-free coverage guarantees under the mild assumption of data exchangeability. It uses a labeled calibration set and a notion of conformity, quantified through nonconformity scores, heuristic notions of uncertainty, to assess how well a new observation “conforms” to the observed data. Data exchangeability needs to be satisfied between data in the calibration set and new data points. When the assumption of data exchangeability is not satisfied, standard conformal methods no longer reliably achieve the nominal coverage level, and distribution-free guarantees break as the calibration ranking no longer reflects the distribution of new test points. Detailed theoretical foundations are discussed in Angelopoulos et al. ²⁹ From now on, we will focus on classification tasks (prediction sets), and while notation can vary, conclusions are valid for regression problems (prediction intervals).

Coverage and adaptivity. The fundamental property that needs to be satisfied by frameworks that aim to measure prediction uncertainty is coverage, i.e., the statistical guarantee that the true value of a prediction $Y_{n+1}$ is included with high probability in the set $C(X_{n+1})$ generated by the process. Coverage can be marginal, if it is guaranteed on average over the whole input space, or conditional, if it is guaranteed on average for each input. ³⁰ With a finite-sample correction, marginal coverage can be expressed as:

$$\begin{array}{r}1-\alpha \le P[Y_{n+1}\in C\left(X_{n+1}\right)]\le 1-\alpha +\frac{1}{n+1},\end{array}$$

while conditional coverage can be expressed as:

$$\begin{array}{rl}& 1-\alpha \le P[Y_{n+1}\in C\left(X_{n+1}\right)|X_{n+1}=x]\\ & \le 1-\alpha +\frac{1}{n+1},\mathrm{\forall }x\in X.\end{array}$$

Under the assumption of data exchangeability, CP ensures that the $C(X_{n+1})$ covers the true outcome with a probability of at least $1-\alpha$ . CP can only formally guarantee marginal coverage, but can get very close to conditional coverage if best practices are followed. ²⁴

Achieving marginal coverage is necessary but not sufficient for a procedure to be useful, as techniques could provide coverage guarantees but prediction sets or intervals so wide that they become irrelevant for the analysis. Adaptivity is the ability to adjust to local variability in the data. We expect smaller sets for easy-to-classify examples, and larger sets for difficult-to-classify examples. In general, adaptive procedures lead to larger but more informative prediction sets. Several diagnostics can be used to evaluate adaptivity, including label-stratified coverage (LSC), which evaluates the coverage across all classes.

Basic algorithm. The most common implementation of CP is the split conformal method. ²⁵ A pre-trained model $\hat{f}$ is used to predict the labels of a labeled dataset, called calibration set $D=\{(X_i,Y_i){\}}_{i=1}^n$ . For each calibration point, a nonconformity score is computed via a score function $s(X,Y)\in \mathbb{R}$ measuring the error between the prediction and true labels. An example of a score function for an adaptive CP procedure is:

$$s\left(X_i\right)=\sum _{j=1}^k\hat{f}{\left(X_i\right)}_j,$$

(1)

where, after sorting the softmax scores in descending order, $k$ represents the rank of the true class. The score for the $i$ th input is the sum of all the softmax scores until the true class is reached. Then, the empirical $1-\alpha$ quantile is computed for these scores. Following (1):

$$\begin{array}{r}\hat{q}=\frac{\lceil (n+1)(1-\alpha )\rceil }{n}.\end{array}$$

Prediction sets for new observations are then created by including all classes until the cumulative softmax score distribution reaches $\hat{q}$ .

3.3. Prediction-powered inference

Prediction-powered inference (PPI) combines machine-learning predictions with a smaller set of trusted, “gold standard” labeled observations to perform statistically valid inference on population parameters. It corrects bias in imputation estimators by estimating a rectifier, which quantifies the error between model predictions and gold standard outcomes. Through the construction of a confidence interval around the rectifier, PPI adjusts the imputed estimator to produce tighter, valid prediction-powered CIs. Additionally, PPI supports hypothesis testing.

Algorithm. The first step in the PPI procedure is to identify a problem-specific rectifier that measures the bias of the imputed estimator. In the case of mean estimation:

$$\begin{array}{r}\begin{array}{r}{\theta }^{*}=\mathbb{E}\left[Y\right]\\ \text{Population parameter}\end{array}\begin{array}{r}\mathrm{\Delta }=\mathbb{E}[f(X)-Y]\\ \text{Rectifier}\end{array}\end{array}$$

where $f(X)$ is the model’s prediction. The second step is to use the labeled data $D_{\text{gold}}=\{(X_i,Y_i){\}}_{i=1}^n$ to compute the empirical rectifier:

$$\begin{array}{r}\hat{\mathrm{\Delta }}\pm t_{1-\alpha /2,n-1}\frac{{\hat{\sigma }}_{f-Y}}{\sqrt{n}},\end{array}$$

where ${\hat{\sigma }}_{f-Y}$ is the sample standard deviation of $f(X_i)-Y_i$ . In parallel, we compute an imputed estimator for $\theta *$ using the abundant unlabeled data $\{X_i{\}}_{i=1}^N$ , with $N\gg n$ . In the case of mean estimation:

$$\begin{array}{r}{\tilde{\theta }}_f=\frac{1}{N}\sum _{i=1}^{N}f\left(X_i\right).\end{array}$$

While this estimator has low variance due to large $N$ , it is generally biased because it solely relies on predictions. Therefore, we adjust the imputed estimator using the rectifier to form the prediction-powered estimator. The final prediction-powered CI for $\theta *$ incorporates uncertainty from both components: the imputed estimator and the empirical rectifier. A simple version with normal approximation is:

$${{\hat{\theta }}_{PP}={\tilde{\theta }}_f-\hat{\mathrm{\Delta }}\to C{I}_{PP}=\hat{\theta }}_{PP}\pm z_{1-\alpha /2}\sqrt{\frac{{\hat{\sigma }}_f^2}{N}+\frac{{\hat{\sigma }}_{f-Y}^2}{n}},$$

(2)

where ${\hat{\sigma }}_f^2$ is the variance of $f(X)$ computed from the unlabeled data, and ${\hat{\sigma }}_{f-Y}^2$ is the variance of $f(X_i)-Y_i$ from the gold standard data. If $N\gg n$ , the width of the confidence interval will almost entirely depend on ${\hat{\sigma }}_f^2$ , meaning that a better model will achieve tighter CIs.

4. Applications

In this section, we show two Official Statistics use cases that handle non-traditional data sources where we apply CP and PPI techniques: (a) Predicting arrival ports from ship trajectories and (b) estimating the proportion of hate speech on social media.

4.1. Predicting arrival ports – conformal prediction

The Italian Institute of Statistics (Istat) is training deep learning models to predict arrival ports from ship trajectories of Automatic Identification System (AIS) signals to integrate survey-based official maritime statistics. ³¹ Formally, the problem consists of training a model $\hat{f}$ to find the best approximation of:

$$\begin{array}{rl}& p\left(Y|{\left\{X_t\right\}}_{t=1}^T\right),\text{with}{X}_t\\ & :=[{\text{latitude}}_t,{\text{longitude}}_t,{\text{speed}}_t,{\text{direction}}_t],\end{array}$$

where $Y$ is the arrival port, $X_t$ is the individual AIS signal at time $t$ , and $T$ is the total length of the ship trajectory. The idea is to get a valid prediction set of ports for each trajectory.

Methodology. We train an attention-based bidirectional long short-term memory (AT-BiLSTM) network^32,33 that receives ship trajectories as input and outputs a softmax score distribution over the label space of arrival ports. In other words, it assigns a score to each arrival port for every trajectory. The model is trained on 42,339 labeled trajectories from the first quarter of 2022, calibrated on 4,137 labeled trajectories, and tested on 4,597 labeled trajectories.

$$\begin{array}{rl}& \begin{array}{r}{D}_{\text{train}}={\{X_i,Y_i\}}_{i=1}^{\mathrm{42,339}}\\ \text{Training Set}\end{array}\begin{array}{r}{D_{\text{calib}}=\{X_i,Y_i\}}_{i=1}^{\mathrm{4,137}}\\ \text{Calibration Set}\end{array}\\ & \begin{array}{r}{D_{\text{test}}=\{X_i,Y_i\}}_{i=1}^{\mathrm{4,597}}\\ \text{Test Set}\end{array}\end{array}$$

The label space has $K=86$ dimensions, that is, we have $86$ unique ports in our dataset. The model is trained with a weighted cross-entropy loss function, and achieves a macro F1-score of $0.829$ on the test set, and $0.840$ on the calibration set.

Model overconfidence. To illustrate the problem of overconfidence, we can analyze the distribution of softmax scores assigned to the true class in incorrect predictions. Looking at Figure 1(right), we notice how the distribution is severely skewed towards very low values. Indeed, half of the mass lies below 0.18. This indicates that the model tends to make mistakes very confidently. Figure 1(left) also shows an individual example of this. If we used these scores as a statistically valid measure of uncertainty, we would severely underestimate it.

Figure 1.

Example of an overconfident incorrect prediction (left) and overall true-class softmax scores distribution over incorrect predictions (right).

Conformal prediction. A more rigorous approach is to apply split conformal prediction. To do this, we need to predict the labels for the calibration set $D_{\text{calib}}$ using our model. For each trajectory $X_i$ in $D_{\text{calib}}$ , we extract a softmax distribution over classes.

$$\begin{array}{r}\begin{array}{r}{z}_i=\hat{f}\left(X_i\right),z_i\in {\mathbb{R}}^{86}\\ \text{Output for}i\text{th trajectory}\end{array}\begin{array}{r}Z=\{\{z_{ij}{\}}_{j=1}^{86}{\}}_{i=1}^{\mathrm{4,137}}\\ \text{Softmax distributions}\end{array}\end{array}$$

where $z_{ij}\in \mathbb{R}$ is the softmax score from the $j$ th class extracted from the model’s prediction on the $i$ th trajectory. Now, we define an adaptive score function and extract the empirical quantile $\hat{q}$ using (1) and choosing $\alpha =0.1$ . Then, we can build prediction sets for the test set $D_{\text{test}}$ . In Figure 2(left), we show a sanity check for correct coverage, repeating the CP procedure for 5,000 different calibration and test set combinations as described in Angelopoulos and Bates. ²⁴ As expected, the distribution is bell-shaped around the theoretical value of $0.9$ . Figure 2(right) shows the distribution of the set sizes, i.e., the number of labels in the prediction set, extracted by running the procedure on the test set. While most of the mass lies in the 1–10 size interval, there is a significant amount of predictions whose sets are so wide that they are hardly informative.

Figure 2.

Empirical coverage distribution on the test set over $R=\mathrm{5,000}$ trials (left) and set-size distribution for the test set (right). The minimum set size is one.

To evaluate the adaptivity of our procedure, we can extract label-specific coverages, that is, we compute the coverage obtained by the model on each of the possible $86$ labels. Figure 3 shows the results of this label-stratified procedure. As we can see, $1-\alpha$ is obtained (or almost obtained) for most classes, with the lowest label-stratified coverage being just below 50%.

Figure 3.

Results of the label-stratified coverage (LSC) check. Each bar represents a label (arrival port), and each bar in grey represents coverage greater than or equal to $1-\alpha$ . The dashed red line shows the $1-\alpha$ threshold, $0.9$ , while the dotted blue line represents the scaled frequency distribution of labels in the dataset. Labels are sorted by most frequent to least frequent.

4.2. Estimating hate speech on X – prediction-powered inference

Here, the goal is to estimate the proportion of hate speech on X (formerly Twitter) over the total number of Italian Tweets related to migration and ethnic minorities. ³⁴ Given a vast corpus of $N=\mathrm{20,400,000}$ posts on X across five years (2018-2022), we want to estimate the true proportion of hateful content. In addition to this vast unlabeled corpus, we have a labeled dataset of Italian X posts from the EVALITA 2020 campaign, ³⁵ containing 8,100 data points, and another manually labeled dataset sampled from the corpus containing $681$ data points.

Methodology. To build a hate speech classifier, we fine-tune a multilingual robustly optimized BERT (XLM-R) model, ³⁶ particularly the large version with 561 million parameters (The code used for fine-tuning can be found at https://github.com/istat-methodology/fine-tuning-pipelines). We split ${D_{HS}\{X_i,Y_i\}}_{i=1}^{\mathrm{8,100}}$ into a training set and a test set with a $0.2$ ratio, and we obtain a macro F1 score of $0.81$ on the test set.

Prediction-powered inference. We can use PPI to create a confidence interval around the estimated proportion of hateful speech. First, we define the prediction-powered estimator as:

$$\begin{array}{r}{\hat{\theta }}_{PP}\leftarrow {\tilde{\theta }}_f-\hat{\mathrm{\Delta }}:=\frac{1}{N}\sum _{i=1}^{N}f\left({\tilde{X}}_i\right)-\frac{1}{n}\sum _{i=1}^{n}f\left(X_i\right)-Y_i.\end{array}$$

Then, we need to compute the sample variance of the imputed estimate and empirical rectifier:

$$\begin{array}{r}\begin{array}{r}{\hat{\sigma }}_f^2={\tilde{\theta }}_f(1-{\tilde{\theta }}_f)\\ \text{Sample variance of}\\ \text{imputed estimate}\end{array}\begin{array}{r}{\hat{\sigma }}_{f-Y}^2=\frac{1}{n}\sum _{i=1}^n{[f\left(X_i\right)-Y_i-\mathrm{\Delta }]}^2\\ \text{Sample variance of}\\ \text{empirical rectifier}\end{array}\end{array}$$

At this point, choosing $\alpha =0.05$ and assuming normality, we can compute the prediction-powered CI for the estimated proportion using $D_{\text{gold}}=\{X_i,Y_i{\}}_{i=1}^{681}$ as:

$$\begin{array}{rl}C{I}_{PP}& =[{\hat{\theta }}_{PP}-z_{1-\alpha /2}\sqrt{\frac{{\hat{\sigma }}_f^2}{N}+\frac{{\hat{\sigma }}_{f-Y}^2}{n}};{\hat{\theta }}_{PP}\\ & +z_{1-\alpha /2}\sqrt{\frac{{\hat{\sigma }}_f^2}{N}+\frac{{\hat{\sigma }}_{f-Y}^2}{n}}]=[0.065;0.131]\end{array}$$

Additionally, we could propagate the prediction-powered confidence interval throughout the whole period, obtaining a confidence interval for the time series (Figure 4). PPI could also easily let us test whether the difference between the proportions of hate speech in two periods of time is statistically significant.

Figure 4.

Prediction-powered confidence interval for the 2018 daily series of the proportion of hate speech on X.

Limitations. Recalling (2), when building the confidence interval for the time series, we are basically utilizing the same interval width over the whole series, since the interval width depends mostly on the variability of the empirical rectifier, if $N\gg n$ , which is estimated independently of time.

5. Conclusion

In this study, we have shown that rigorous uncertainty quantification in machine learning, essential for official statistical production, can be achieved by combining model-agnostic frameworks with high-throughput prediction systems. Our work presents two complementary methodologies: conformal prediction (CP) for quantifying prediction-level uncertainty, and prediction-powered inference (PPI) for obtaining statistically valid, noise-free confidence intervals for population parameters. CP allows us to construct prediction sets and intervals that adapt to local data variability while providing finite-sample coverage guarantees, ensuring that even out-of-distribution data are accompanied by reliable uncertainty estimates. In parallel, PPI exploits both the vast amounts of unlabeled data and a much smaller set of gold standard observations to correct biases inherent in naïve imputation approaches, resulting in more precise inference for population parameters of interest. Because the dominant component of the computational budget lies in fitting the underlying predictive model, while the execution of the uncertainty-quantification procedures considered here, namely conformal prediction (CP) and prediction-powered inference (PPI), introduces only marginal additional overhead, these methods are, at least in principle, operationally feasible for large-scale deployment within Official Statistics. In particular, once a model has been trained, CP and PPI require only lightweight post-processing steps (such as nonparametric calibration or residual-based corrections), whose computational complexity grows linearly with the calibration sample and is negligible relative to model training. This computational profile is especially advantageous in production environments where indicators must be generated repeatedly or at high frequency, since a single trained model can be reused to deliver uncertainty assessments across multiple outputs without materially increasing runtime or resource consumption.

The real-world use cases we explored – predicting arrival ports in maritime statistics and estimating the prevalence of hate speech on social media – illustrate how these frameworks can be integrated into Official Statistics. By a statistically rigorous uncertainty quantification, these methods can increase the interpretability and trustworthiness of machine learning outputs, as well as provide guidance and additional diagnostics during model development.

While CP and PPI naturally have limitations, like CP’s reliance on the assumption of data exchangeability, already tackled by recent variants, ³⁷ and require careful calibration as well as additional data, they prove to be especially flexible and scalable compared to most uncertainty quantification techniques.