Abstract
I would like to thank Dr Bell, Dr Haslett and Dr Lahiri for taking the time to share their thoughtful and constructive reactions to the article. The topic raises several challenging issues and the literature can be confusing, and I am grateful to these three outstanding experts for taking the time to think carefully about the issues and contribute towards a clearer shared understanding.
Dr Bell’s discussion further explores the issue of age bias’ in the US context, in the hypothetical case that old census data were used as poverty predictors rather than current administrative data. The results are mixed: Old census data lead to a deterioration in performance in estimates of poverty for some age groups when using the 1990 census, but the results are less clear when using the 2000 census. For both the 1990 census and 2000 census estimates, using both old census data and current administrative data leads to more accurate predictions. While these findings for the US case are provocative, they may differ in other countries. The consequences of age bias in the census data (relative to current administration data) vary depending on how fast regional patterns of poverty are changing, the nature of the administrative data and the estimation method. Dr Bell’s exercise utilized an area-level model, where the target area is the state, which links auxiliary data directly to the survey data in the model through state-level identifiers. This differs from a typical unit-level model with household covariates, which is more susceptible to age bias in auxiliary data because of the assumption that the auxiliary data in the survey and census follow a common distribution. Still, the conclusion that combining administrative and old census data improves estimates seems likely to apply to the use of geospatial data as well. Further research could be useful to quantify the benefits of adding old census data to current geospatial data in different contexts.
Dr Lahiri’s discussion also makes several excellent points, starting with the indispensability of sound modelling principles and robust estimation methodologies to generate dependable small-area estimates. Dr Lahiri also makes the important point that the quality of geospatial data across space may lead to biased estimates. This is particularly concerning when using crowd-sourced geospatial information, such as Open Streetmap, which appears to undercount buildings in more remote areas. But it may also be a concern with satellite imagery for short time periods and in areas and seasons when cloud cover is more prevalent. This raises a set of useful questions for further research, including the properties of missing geospatial data due to cloud cover, how this varies across space, time and satellites, and the strengths and weaknesses of different imputation methods to address it. More frequent satellite data collection, such as the daily global imaging carried out by Planet, or features derived from synthetic aperture radar, can also play an important role in addressing concerns over spatial variation in geospatial data quality.
Dr Lahiri’s discussion raises the important point that the integration of geospatial data at the primary sampling unit (PSU) or sub-area level holds the potential for mitigating selection bias in unit-level modelling. I would like to underscore this point: There are several contexts in which first-stage sampling weights may not fully accurately represent the inverse probability of PSU selection, for example, due to civil conflict in parts of target areas that makes enumeration impossible. In these cases, the use of sub-area auxiliary data may offer significant advantages over area-level models, as the use of sub-area auxiliary data leads to more accurate predictions when sub-areas are not missing at random. While the Chen et al. (2024)[1] paper referenced by Dr Lahiri offers intriguing evidence and an important step forward, more evidence is needed on the robustness of different methods to selection bias, different types of measurement error and other forms of model misspecification.
I also greatly appreciate Dr Haslett’s detailed and comprehensive discussion and agree with many of the points he raises. Areas of agreement include, among others:
-‘For production of such a large collection of Small Area Estimates (SAEs), such an automated process would be the dream of Official Statisticians internationally.’ -‘As stated in the abstract, contextual information is best used in models at the finest level available.’ -‘Data sources vary in quality.’ -‘Using complex deep learning (P13) as a black box to derive variables to use as predictors in models has particular pitfalls.’ -‘One type of model does not fit all. Methodology needs to depend on type of data available and variable of interest.’ -‘It is a misconception that models need either to be fitted to the entire country ignoring region or fitted separately to parts of it. These are the extremes, but in between are a plethora of options.’ -‘The correlation between the two sets of SAEs is so high in Bangladesh that it warrants further study.’ -‘Chi et al. (2022) supports use of the Relative Wealth Index based on satellite data as an alternative or even as a substitute for more established measures, but others are more wary.’ -‘Sophistication of technique should not be the core aim.’
As an aside, none of the three discussions raised any technical concerns about bias due to the use of unit-level models that use only aggregate predictors for predicting non-linear functions of welfare. This suggests that they do not see any inherent problems with this approach for combining survey and geospatial data to generate small area estimates of poverty. However, these models do tend to be estimated on surveys with lower effective sample sizes, since aggregate predictors exhibit less variation than indicators measured at the household level. Therefore, it is important in these cases to utilize an estimation method that appropriately accounts for sample weights when estimating model parameters. Improper weighting methods, while not affecting the consistency of estimated parameters, can lead to significant bias when effective sample sizes are small.
I do see a few issues a bit differently than Dr Haslett, which may partly explain differences in our degree of optimism regarding the integration of survey and geospatial data. One point of difference is the discussion on differences between different methodologies, particularly with respect to the Elbers Lanjouw and Lanjouw (ELL) and the Empirical Best Predictor (EBP) methods. Dr Haslett points out: In terms of mixed models, the principal difference between ELL and EBP is that ELL has no area-level random effect and EBP has no cluster-level random effect.’ While I agree that this has traditionally been an important difference between ELL and EBP, ELL in practice can also be estimated with random effects specified at the area level instead of the cluster level. A key difference between ELL and EBP is that ELL traditionally does not use a mixed model. ELL is derived from an unconditional means model (which econometricians typically refer to as a random-effects model’) that yields purely synthetic predictions. Partly as a consequence, current software for implementing EBP assumes that both the random area effect and idiosyncratic term are normally distributed. ELL, however, allows for non-normal error terms by sampling from the empirical distribution of the error terms when generating point estimates. Another difference between ELL and EBP in typical applications is that ELL typically includes the estimation of an alpha model’ to attempt to correct for heteroscedasticity. Finally, the procedure for estimating uncertainty also differs between the two methods: ELL estimates variances using essentially a multiple imputation approach, which accounts for uncertainty in the estimated model regression coefficients. EBP, on the other hand, estimates mean squared errors using a parametric bootstrap approach that treats the estimated regression coefficients as fixed.
Regarding the use of ELL versus EBP for geospatial data in particular, the article tried to convey the point made in Masaki et al. (2022)[2] that EBP becomes more attractive relative to ELL when the predictors are aggregate geospatial data rather than household-level census data. When the predictors are aggregated, the variance of the estimated area random effect increases relative to the variance of the idiosyncratic error term, due to less effective variation in the sample predictors within area. This increases the estimated shrinkage factor, leading the EBP model to give more weight to the survey data in the geospatial case than when using a household-level census. One would expect that all else being equal, this would give EBP a greater advantage relative to ELL, in terms of accuracy, when using aggregated geospatial data as predictors than when using household-level census data.
Dr Haslett also makes several insightful points on how to evaluate the accuracy of estimates. I agree that local estimates need to be more accurate than direct survey estimates to be any use at all, and what is fundamentally more important depends how much information there is in the satellite data and how well the chosen model utilizes that information’. As noted in the paper, the increase in accuracy of model-based estimates relative to direct estimates depends on several factors: the outcome indicator of interest, the context, the estimation method, the model, the nature of the sample data (including the level at which the satellite data is linked to it) and the quality of the satellite data indicators. The paper argues that where this has been tested in realistic settings to predict indicators related to poverty, results have generally been good. The improvement in the accuracy of the estimates, when measured against their correlation with census-based truth’, can be substantial. This can in turn provide meaningful reductions in uncertainty at no cost to coverage rates. For headcount poverty, so far, the improvement in the efficiency of small-area poverty estimates obtained using geospatial predictors, using direct estimates in places where it has been tested, is roughly comparable to multiplying the size of the survey by a factor in the range of 2.5 to 5. It, therefore, seems plausible to think that applying this type of data integration routinely could generate hundreds of millions of dollars worth of more precise data at little cost.
The problem is that geospatial small-area estimation does not work well in all contexts and for all indicators. One way to approach this problem is to work towards identifying methods that are robust, in the sense that they produce predictions that are always at least as accurate as the direct survey estimates. Tree-based machine learning approaches such as extreme gradient boosting may be promising in this regard because they make less restrictive functional form assumptions than linear models, though they may not perform as well when training data are scarce. A second and more feasible approach would be to identify diagnostics that use the sample data to estimate how much combining satellite data and survey data improves accuracy relative to direct estimates from survey data. That important research agenda is still ongoing, and this is a crucial part of the puzzle that needs more progress before scaling up implementation. Nonetheless, it is encouraging that results from several contexts indicate substantial improvement over direct estimates when using geospatial data to generate small-area estimates of poverty and wealth in realistic settings.
I appreciate the discussion on accuracy measures and agree that it is best to consider several. As a minor note, I see relative standard error more as a measure of precision than one of accuracy. But an important related point is that average relative error is a poor measure for evaluating predictions of proportions, because it overweights small discrepancies in non-poor areas. Put another way, an estimate that the poverty rate is 2% in an area with a true poverty rate of 1% should not be considered as inaccurate as an estimate that the poverty rate is 50% in an area with a true poverty rate of 25%. Correlation (Pearson and Spearman), mean absolute error and root mean squared error are more meaningful accuracy measures for proportions than relative bias.
Another minor point that may warrant clarification relates to section 3. I generally agree that, for many existing indicators, Linking new satellite-based target variables to current poverty measures is difficult’. However, this depends a lot on how the new satellite-based measures were developed, specifically on the training data and the outcome indicator. A good example is that the Meta relative wealth index may not capture aspects of poverty, as typically defined. A key reason for this is that the relative wealth index estimates an asset index, rather than consumption or income-based poverty. The decision to estimate an asset index was driven by the data access policies associated with demographic and health surveys, which do not contain income or consumption data but generally make Enumeration Area (EA)-level geocoordinates publicly available to researchers. Greater availability of jittered geocoordinates in the household income and budget surveys used for poverty measurement would make it much easier to link new satellite-based variables with official poverty measures.
A key section of Dr Haslett’s comments explores the interesting issue of what different levels of R2, in terms of correspondence between different estimates, imply for misclassification of areas with respect to a particular threshold. The paper reports R2 values for several studies partly because they are a useful summary indicator of the extent to which two sets of SAE estimates are consistent, but also because either R2 or its square root is reported in virtually every study. Furthermore, in my experience, comparing Pearson R2 across methods in practice tends to give the same qualitative answer as comparing Spearman rank correlations and mean absolute errors. Dr Haslett, however, makes the excellent point that achieving high predictive accuracy, whether measured through correlations, rank correlations or mean absolute error, is ultimately a means to an end rather than an end in itself.
A key conclusion of the informative simulation study that Dr Haslett shows is that even comparatively high values of correlation correspond to rather high probabilities of different classification above and below the cutoff for the two sets of SAEs’. This supports the provocative point that regrettably, the conclusion from Table 1 in the main manuscript must be that the agreement between the two sets of SAEs is insufficient to justify the use of sample survey plus satellite data as a standard substitute for more standard SAEs based on survey and census data’ and the bottom line point that the real cost is not the cost of data collection and model fitting, but the considerably greater cost and consequences if scarce resources are not well allocated’.
I can think of three points in response. The first is that it is not clear that the right social welfare function depends solely on achieving correct classification with respect to an arbitrary threshold. It also matters how large the classification errors are. When social welfare functions are the sum of individual welfare function and utility is concave in welfare, inclusion errors are worse when a very wealthy area is misidentified as poor than a case where an area barely above the cut-off is. Conceptually, it seems that the correct criterion to use to evaluate models is the loss in social welfare due to mistargeting a hypothetical transfer programme, perhaps evaluated using the type of social welfare functions typically used in the public economics literature, rather than either R2 or misclassification rates.
A second point is that both methods and geospatial data for prediction are improving over time, meaning that Table 1 is not necessarily fully representative of more recent practice. Recent work using XGboost or EBP models applied to well-defined spatial features that include information on building footprints, which is an important source of predictor variables. The resulting correlations with census-based estimates or actual values for monetary poverty, wealth indices and non-monetary poverty typically lie in the range of 0.8–0.9, which is somewhat higher than the average value reported in Table 1.
The final point is that model-based estimates using geospatial data should be judged against plausible counterfactuals. We can assume that resources have to be allocated to households or areas in some way, and the question is how to obtain the best information to do that to maximize the chosen concept (or concepts) of social welfare. The plausible alternatives to geospatial small-area estimates are typically either (a) imprecise and inaccurate direct estimates at the target area level, (b) precise direct estimates at a higher regional level that treat areas uniformly within regions or (c) the use of estimates derived from an old census. If using a survey and geospatial data leads to better targeting outcome than these competing options, the logical conclusion would be that sample surveys plus satellite data are the best option. As noted above, however, there is important work to do to understand when this is the case and how we would know that without access to a census.
Finally, the concluding section of the discussion makes several interesting points. Chief among them is that satellite data are best for predicting variables directly related to ground cover, almost certainly useful as candidate auxiliary variables for predicting certain types of expenditure poverty, but not so useful for food security based on household food availability or for providing information on variation between households within an area, and not particularly useful for predicting health-related SAE measures: stunting, underweight and wasting (and especially diarrhoea) in children under five. In general, publicly available satellite indicators such as buildings and land classification are good at picking up spatial variation in population density, which is, in turn, correlated with many socioeconomic indicators. Poverty and wealth, for example, tend to show strong systematic relationships with population density. But not every indicator will. My feeling is that the jury is still out on which socioeconomic indicators can be predicted with which satellite indicators. Satellite imagery, for example, may be very well-suited for tracking dynamic changes in food security indicators, if one appropriately accounts for appropriate lags in timing between the growing season and post-harvest season. But while there are drought indicators, I do not yet know of any publicly available geospatial indicators that measure food security. Stunting and wasting may also be correlated with population density, as well as food security. Performance depends not only on the indicator but also on the training data, context and satellite indicators. Nonetheless, I agree that there is now good evidence that geospatial data are almost certainly useful as candidate auxiliary variables for predicting certain types of expenditure poverty’. Statisticians, economists and others interested in economic measurement face an exciting and important research agenda to try to figure out when to use it and how to maximize its usefulness, for poverty, wealth, population and all socioeconomic indicators relevant to economic policy.
