Using Administrative Data to Explore the Effect of Survey Nonresponse in the UK Employment Retention and Advancement Demonstration

Abstract

Background:

Even a well-designed randomized control trial (RCT) study can produce ambiguous results. This article highlights a case in which full sample results from a large-scale RCT in the United Kingdom differ from results for a subsample of survey respondents.

Objectives:

Our objective is to ascertain the source of the discrepancy in inferences across data sources and, in doing so, to highlight important threats to the reliability of the causal conclusions derived from even the strongest research designs.

Research design:

The study analyzes administrative data to shed light on the source of the differences between the estimates. We explore the extent to which heterogeneous treatment impacts and survey nonresponse might explain these differences. We suggest checks which assess the external validity of survey measured impacts, which in turn provides an opportunity to test the effectiveness of different weighting schemes to remove bias. The subjects included 6,787 individuals who participated in a large-scale social policy experiment.

Results:

Our results were not definitive but suggest nonresponse bias is the main source of the inconsistent findings.

Conclusions:

The results caution against overconfidence in drawing conclusions from RCTs and highlight the need for great care to be taken in data collection and analysis. Particularly, given the modest size of impacts expected in most RCTs, small discrepancies in data sources can alter the results. Survey data remain important as a source of information on outcomes not recorded in administrative data. However, linking survey and administrative data is strongly recommended whenever possible.

Keywords

measurement methodology nonresponse bias

Get full access to this article

View all access options for this article.

References

Abowd

J. M.

Stinson

(2013). Estimating measurement error in annual job earnings: A comparison of survey and administrative data. The Review of Economics and Statistics, 95, 1451–1467.

Ashton

Portanti

(2011). Tracking procedures on the employment, retention and advancement survey. Office for National Statistics Survey Methodology, Bulletin No. 68: 12–22.

Banerjee

A. V.

Duflo

Glennerster

Kinnan

(2015). The miracle of microfinance? Evidence from a randomized evaluation. American Economic Journal: Applied Economics, 7, 22–53.

Barnow

Greenberg

(2015). Do estimated impacts on earnings depend on the source of the data used to measure them? Evidence from previous social experiments. Evaluation Review, 39, 179–228.

d’Haultfoeuille

(2010). A new instrumental method for dealing with endogenous selection. Journal of Econometrics, 154, 1–15.

Dorsett

Smeaton

Speckesser

(2013). The effect of making a voluntary labour market programme compulsory: Evidence from a UK experiment. Fiscal Studies, 34, 467–489.

Groves

R. M.

(2006). Nonresponse rates and nonresponse bias in household surveys. Public Opinion Quarterly, 70, 646–675.

Groves

R. M.

Peytcheva

(2008). The impact of nonresponse rates on nonresponse bias. Public Opinion Quarterly, 72, 167–189.

Heffetz

Rabin

(2013). Conclusions regarding cross-group differences in happiness depend on difficulty of reaching respondents. American Economic Review, 103, 3001–3021.

10.

Hendra

Riccio

Dorsett

Greenberg

Knight

Phillips

… Smith

(2011). Breaking the low-pay, no-pay cycle: Final evidence from the UK Employment Retention and Advancement (ERA) demonstration (Research Report 765). Department for Work and Pensions.

11.

Kapteyn

Ypma

J. Y.

(2007). Measurement error and misclassification: A comparison of survey and administrative data. Journal of Labor Economics, 25, 513–551.

12.

Kornfeld

Bloom

H. S.

(1999). Measuring program impacts on earnings and employment: Do unemployment insurance wage reports from employers agree with surveys of individuals? Journal of Labor Economics, 17, 1.

13.

Kreuter

Couper

Lyberg

(2010). The use of paradata to monitor and manage survey data collection. In Proceedings of the joint statistical meetings, American Statistical Association (pp. 282–296).

14.

Lin

I. F.

Schaeffer

N. C.

(1995). Using survey participants to estimate the impact of nonparticipation. Public Opinion Quarterly, 59, 236–258.

15.

Lundquist

Hsueh

Lowenstein

Fauceetta

K. G. D.

Michalopoulos

Knox

(2014). A family-strengthening program for low-income families: Final impacts from the supporting healthy marriage evaluation. New York, NY: MDRC.

16.

Moore

(in press). Decomposing differences in impacts on survey- and administrative-measured earnings from a job-training voucher experiment. Evaluation Review.

17.

Nuñez

Verma

Yang

(2015). Building self-sufficiency for housing voucher recipients: Interim findings from the work rewards demonstration in New York city. New York, NY: MDRC.

18.

Puma

M. J.

Olsen

R. B.

Bell

S. H.

Price

(2009). What to do when data are missing in group randomized controlled trials. NCEE 2009–0049. Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.

19.

Ramalho

Smith

(2013). Discrete choice non-response. The Review of Economic Studies, 80, 343–364.

20.

Riccio

Dechausay

Miller

Nuñez

Verma

Yang

(2013). Conditional cash transfers in New York city: The continuing story of the opportunity NYC-family rewards demonstration. New York, NY: MDRC.

21.

Schochet

Burghardt

McConnell

(2006). National job corps study and longer-term follow-up study: Impact and benefit-cost findings using survey and summary earnings records data. Princeton, NJ: Mathematica Policy Research.

22.

Yang

Hendra

(in press). The importance of using multiple data sources in policy assessments: Lessons from two conditional cash transfer programs in New York city. Evaluation Review.