Abstract
Measurement error can be very difficult to assess and reduce. While great strides have been made in the field of survey methods research in recent years, many ongoing federal surveys were initiated decades ago, before testing methods were fully developed. However, the longer a survey is in use, the more established the time series becomes, and any change to a questionnaire risks a break in that time series. This article documents how a major federal survey – the health insurance module of the Current Population Survey (CPS) – was redesigned over the course of 15 years through a systematic series of small, iterative tests, both qualitative and quantitative. This overview summarizes those tests and results, and illustrates how particular questionnaire design features were identified as problematic, and how improvements were developed and evaluated. While the particular topic is health insurance, the general approach (a coordinated series of small tests), along with the specific tests and methods employed, are not uniquely applicable to health insurance. Furthermore, the particular questionnaire design features of the CPS health module that were found to be most problematic are used in many other major surveys on a range of topic areas.
References
1.
Agency for Healthcare Research and Quality. 2015. “Medical Expenditure Panel Survey, MEPS Background.” Available at: http://meps.ahrq.gov/mepsweb/communication/household_participant_back.jsp (accessed on October 5, 2015).
2.
Beatty, P. 2010. “Considerations Regarding the Use of Global Survey Questions.” Paper presented at the Consumer Expenditures Survey Methods Workshop, Hyattsville, MD, December 8–9, 2010. Available at: http://www.bls.gov/cex/methwrkshp_pap_beatty.pdf (accessed on February 5, 2016).
3.
Beatty, P. and S. Schechter. 1998. “Questionnaire Evaluation and Testing in Support of the Behavioral Risk Factor Surveillance System (BRFSS), 1992–98.” Office of Research and Methodology, NCHS, Working paper series, 26: 12–17.
4.
Bee, C.A. and A. Cantu. 2013. “Using Timer Data to Evaluate the Respondent Burden of the 2013 CPS ASEC Content Test.” Proceedings of the Federal Committee on Statistical Methodology Research Conference, November 4–6, 2013, Washington, DC. Available at: http://fcsm.sites.usa.gov/files/2014/07/H3_Bee_2013FCSM.pdf (accessed on February 5, 2016).
5.
Belli, R.F., N. Schwarz, E. Singer, and J. Talarico. 2000. “Decomposition Can Harm the Accuracy of Behavioural Frequency Reports.” Applied Cognitive Psychology 14: 295–308.
6.
Bhandari, S. 2004. “People with Health Insurance: A Comparison of Estimates from Two Surveys.” Survey of Income and Program Participation (SIPP) Working Paper No. 243. Washington, D.C.: U.S. Census Bureau. Available at: https://www.census.gov/sipp/workpapr/wp243.pdf (accessed on February 2, 2016).
7.
Blair, E.A. and G.K. Ganesh. 1991. “Characteristics of Interval-Based Estimates of Autobiographical Frequencies.” Applied Cognitive Psychology 5: 237–250. Doi: http://dx.doi.org/10.1002/acp.2350050306.
8.
Blewett, L.A. and M.E. Davern. 2006. “Meeting the Need for State-Level Estimates of Health Insurance Coverage: Use of State and Federal Survey Data.” Health Services Research 41: 946–975. Doi: http://dx.doi.org/10.1111/j.1475-6773.2006.00543.x.
9.
Blumberg, S. 2014. “National Health Interview Survey Since 1957.” Federal Statistics on Health Insurance Coverage: Technical Meeting on Methods Used in Household Surveys, August 18, 2014, Washington, DC. Available at: http://www.census.gov/newsroom/releases/pdf/20140818_gw_final.pdf (accessed on February 5, 2016).
10.
Blumberg, S.J. and M.L. Cynamon. 1999. “Misreporting Medicaid Enrollment: Results of Three Studies Linking Telephone Surveys to State Administrative Records.” Proceedings of the Seventh Conference on Health Survey Research Methods. 189–195. Available at: http://www.cdc.gov/nchs/data/hsrmc/hsrmc_7th_proceedings_1999.pdf. (accessed on February 5, 2016).
11.
Blumberg, S.J. and J.V. Luke. 2011. “Wireless Substitution: Early Release of Estimates from the National Health Interview Survey, July-December 2010.” National Center for Health Statistics. Available at: (http://www.cdc.gov/nchs/nhis.htm) (accessed on February 5, 2016).
12.
Blumberg, S.J., L. Osborn, J.V. Luke, L. Olson, and M.R. Frankel. 2004. Estimating the Prevalence of Uninsured Children: An Evaluation of Data from the National Survey of Children with Special Health Care Needs, 2001. Vital Health Statistics 2 (136). Hyattsville, MD: National Center for Health Statistics.
13.
Boudreaux, M.H., B. Fried, J. Turner, and K.T. Call. 2013. “SHADAC Analysis of the Survey of Health Insurance and Program Participation.” State Health Assistance Data Center. Available at: http://www.shadac.org/files/shadac/publications/SHIPP_final_report.pdf (accessed on August 4, 2014).
14.
Congressional Budget Office. 2003. “How Many People Lack Health Insurance and for How Long?” A CBO Report. The Congress of the United States. Available at: https://www.cbo.gov/publication/14426 (accessed on February 5, 2016).
15.
Crespi, I. and J.W. Swineheart. 1982. “Some Effects of Sequenced Questions Using Different Time Intervals on Behavioral Self-Reports: A Field Experiment.” Paper presented at the American Association for Public Opinion Research. Hunt Valley, MD, May 1982.
16.
Czajka, J. and K. Lewis. 1999. “Using Universal Survey Data to Analyze Children’s Health Insurance Coverage: An Assessment of Issues.” Washington, DC: Mathematica Policy Research, Inc.
17.
Davern, M. 2009. “Unstable Ground: Comparing Income, Poverty & Health Insurance Estimates from Major National Surveys.” Paper presented at the Academy Health Annual Research Meeting, June 29, 2009, Chicago.
18.
Davern, M., K.T. Call, J. Ziegenfuss, G. Davidson, T. Beebe, and L. Blewett. 2008. “Validating Health Insurance Coverage Survey Estimates: A Comparison of Self-Reported Coverage and Administrative Data Records.” Public Opinion Quarterly 72: 241–259.
19.
Farley-Short, P. 2001. “Counting and Characterizing the Uninsured.” University of Michigan, Ann Arbor, MI: Economic Research Initiative on the Uninsured Working Paper Series. Available at: http://www.umich.edu/~eriu/pdf/wp2.pdf. (accessed on August 25, 2005).
20.
Fowler, F.J. 2004. “The Case for More Split-Sample Experiments in Developing Survey Instruments.” In Methods for Testing and Evaluating Survey Questionnaires , edited by S. Presser, J. Rothgeb, M. Couper, J. Lessler, E. Martin, J. Martin, and E. Singer. 173–188. Hoboken, NJ: Wiley.
21.
Grieco, E.M. and D.M. Armstrong. 2014. “Assessing the ‘Year of Naturalization’ Data in the American Community Survey: Characteristics of Naturalized Foreign Born Who Report – and Don’t Report – the Year They Obtained Citizenship.” Paper presented at the Applied Demography Conference, April 30 – May 3, 2014, San Antonio, TX.
22.
Hess, J., J. Moore, J. Pascale, J. Rothgeb, and C. Keeley. 2001. “The Effects of Person-level vs. Household-level Questionnaire Design on Survey Estimates and Data Quality.” Public Opinion Quarterly 65: 574–584.
23.
Hess, J. and E. Singer. 1995. “The Role of Respondent Debriefing Questions in Questionnaire Development”. In Proceedings of the Section on Survey Research Methods, August 13–17, 1995. 1075–1080. Washington, DC: American Statistical Association. Available at: http://www.amstat.org/sections/srms/proceedings/papers/1995_187.pdf (accessed on February 5, 2016).
24.
Klerman, J.A., M. Davern, K.T. Call, V. Lynch, and J. Ringel. 2009. “Understanding the Current Population Survey’s Insurance Estimates and the Medicaid ‘Undercount’” Health Affairs web exclusive. Doi: http://10.1377/hlthaff.28.6.w991.
25.
Klerman, J.A., J.S. Ringel, and B. Roth. 2005. “Under-Reporting of Medicaid and Welfare in the Current Population Survey”. RAND Working Paper WR-169-3. Santa Monica, CA: RAND.
26.
Lewis, K., M. Ellwood, and J. Czajka. 1998. Counting the Uninsured: A Review of the Literature . Washington, DC: The Urban Institute.
27.
Loftus, E.F., M.R. Klinger, K.D. Smith, and J. Fiedler. 1990. “A Tale of Two Questions: Benefits of Asking More Than One Question.” Public Opinion Quarterly 54: 330–345.
28.
Loomis, L. 2000. “Report on Cognitive Interview Research Results for Questions on Welfare Reform Benefits and Government Health Insurance for the March 2001 Income Supplement to the CPS.” Washington, DC: Center for Survey Methods Research, Statistical Research Division, U.S. Census Bureau.
29.
Lynch, V. 2006. “Causes of Error in Survey Reports About Who in the Household Gets Welfare.” Unpublished Paper, Joint Program in Survey Methodology. College Park, MD.
30.
Martin, E.A., R.E. Fay, and E.A. Krejsa. 2002. “Analysis of Questionnaire Errors in Survey Measurements of Census Coverage.” In Proceedings of the American Statistical Association Survey Research Methods Section, August 10–15, New York, NY. 2260–2265. Alexandria: VA American Statistical Association.
31.
National Research Council. 2006. Food Insecurity and Hunger in the United States: An Assessment of the Measure. Panel to Review the U.S. Department of Agriculture’s Measurement of Food Insecurity and Hunger , edited by G.S. Wunderlich and J.L. Norwood. Committee on National Statistics, Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press.
32.
Nelson D.E., E. Powell-Griner, M. Town, M.G. Kovar. 2003. A Comparison of National Estimates From the National Health Interview Survey and the Behavioral Risk Factor Surveillance System. American Journal of Public Health 93:1335–1341.
33.
Office of the Assistant Secretary for Planning and Evaluation, Health and Human Services. 2005. “Understanding Estimates of the Uninsured: Putting the Differences in Context.” Updated September 5. Available at: http://aspe.hhs.gov/health/reports/05/uninsured-understanding-ib/#estimates (accessed on April 9, 2016).
34.
Pascale, J. 2001a. “Methodological Issues in Measuring the Uninsured.” Proceedings of the Seventh Health Survey Research Methods Conference, Williamsburg, VA. 167–173. Available at: http://www.cdc.gov/nchs/data/hsrmc/hsrmc_7th_proceedings_1999.pdf (accessed on February 5, 2016).
35.
Pascale, J. 2001b. “The Role of Questionnaire Design in Medicaid Estimates: Results from an Experiment.” Talk presented to the Washington Statistical Society, March 21, 2001.
36.
Pascale, J. 2004. “Medicaid and Medicare Reporting in Surveys: An Experiment on Order Effects and Program Definitions.” Proceedings of the American Association for Public Opinion Research, American Statistical Association, May 13–16, 2004. pp. 4976–4983. Available at: http://www.amstat.org/sections/SRMS/Proceedings/ (accessed on April 2, 2015).
37.
Pascale, J. 2008. “Measurement Error in Health Insurance Reporting.” Inquiry 45(4): 422–437. doi: 10.5034/inquiryjrnl_45.04.422 Available at: http://inq.sagepub.com/content/45/4/422.full.pdf+html (accessed on February 4, 2015).
38.
Pascale, J. 2009a. “Findings from a Pretest of a New Approach to Measuring Health Insurance in the Current Population Survey.” Paper prepared for the Federal Committee on Statistical Methodology Research Conference, November 2–4, 2009. Available at: https://fcsm.sites.usa.gov/files/2014/05/2009FCSM_Pascale_VIII-A.pdf (accessed on February 5, 2016).
39.
Pascale, J. 2009b. “Survey Measurement of Health Insurance Coverage: Cognitive Testing Results of Experimental Questions on Integrated Current and Calendar Year Coverage.” Available through “Q-BANK,” a database of questions and reports on cognitive testing maintained by the National Center for Health Statistics. Available at: http://wwwn.cdc.gov/qbank/report/Pascale_Census_2009_HealthInsurance.pdf (accessed on February 5, 2016).
40.
Pascale, J. 2009c. “Health Insurance Measurement: A Synthesis of Cognitive Testing Results.” Paper presented at the Questionnaire Evaluation Standards (QUEST) meeting, May 18–20, 2009, Bergen, Norway.
41.
Pascale, J. 2015. “Moderniziing a Major Federal Government Survey: A Review of the Redesign of the Current Population Survey Health Insurance Questions. Study Series, Survey Methodology #2015–03. Available at: https://www.census.gov/srd/papers/pdf/SSM2015-03.pdf (accessed April 2016).
42.
Pascale, J., M. Boudreaux, and R. King. 2015. “Understanding the New Current Population Survey Health Insurance Questions.” Health Services Research 51: 240–261. Doi: http://dx.doi.org/10.1111/1475-6773.12312.
43.
Pascale, J., J. Rodean, J. Leeman, C. Cosenza, and A. Schoua-Glusberg. 2013. “Preparing to Measure Health Coverage in Federal Surveys Post-Reform: Lessons from Massachusetts.” Inquiry: The Journal of Health Care Organization, Provision, and Financing 50: 106–123. Doi: http://dx.doi.org/10.1177/0046958013513679.
44.
Pascale, J. 2011. “Findings from a Split-Ballot Experiment on a New Approach to Measuring Health Insurance in the Current Population Survey.” Report prepared for the Center for Survey Measurement, US Census Bureau.
45.
Pascale, J., M.I. Roemer, and D.M. Resnick. 2009. “Medicaid Underreporting in the CPS: Results from a Record Check Study.” Public Opinion Quarterly 73: 497–520. Available at http://inq.sagepub.com/content/45/4/422.full.pdf+html (accessed on April 2, 2015).
46.
Redline, C. 2013. “Clarifying Categorical Concepts in a Web Survey.” In “Topics in Survey Measurement and Public Opinion.” Special issue, Public Opinion Quarterly 77: 89–105.
47.
Research Project to Understand the Medicaid Undercount. 2008. “Phase II Research Results: Examining Discrepancies between the National Medicaid Statistical Information System (MSIS) and the Current Population Survey (CPS) Annual Social and Economic Supplement (ASEC).” Available at: https://www.census.gov/did/www/snacc/docs/SNACC_Phase_II_Full_Report.pdf (accessed on October 5, 2015).
48.
Resnick, D. 2013. “Microsimulation Support for Tax, Transfer & Health Insurance Policy Analysis Summary.” Submitted to David S. Johnson & Charles Nelson, U.S. Department of Commerce, U.S. Census Bureau, and Don Oellerich, U.S. Department of Health and Human Services, ASPE, on October 3, 2013.
49.
Resnick, D., S. Love, C. Taeuber, and J. Staveley. 2004. “Analysis of ACS Food Stamp Program Participation Underestimate.” Paper presented at the 2004 Joint Statistical Meeting, August 8–12, 2004, Toronto.
50.
Ringel, J.S. and J.A. Klerman. 2005. “Today or Last Year? How Do Interviewees Answer the CPS Health Insurance Questions?” RAND Labor and Population working paper series WR-288. Santa Monica, CA: RAND.
51.
Roman, A.M., A. Hauser, and A. Lischko. 2002. “Measurement of the Insured Population: The Massachusetts Experience.” Paper presented at the 2002 Annual Meetings of the American Association for Public Opinion Research, May 16–19, 2002, St. Pete’s Beach, FL.
52.
Rosenbach, M. and K. Lewis. 1998. “Estimates of Health Insurance Coverage in the Community Tracking Study and the Current Population Survey.” Document no. PR98-54. Washington, DC: Mathematica Policy Research, Inc.
53.
Schaeffer, N.C. and S. Presser. 2003. “The Science of Asking Questions.” Annual Review of Sociology 29: 65–88. Doi: http://dx.doi.org/10.1146/annurev.soc.29.110702.110112.
54.
Sudman, S., A. Fin, and L. Lannon. 1984. “The Use of Bounded Recall Procedures in Single Interviews.” Public Opinion Quarterly 48: 520–524.
55.
Swartz, K. 1986. “Interpreting the Estimates from Four National Surveys of the Number of People Without Health Insurance.” Journal of Economic and Social Measurement 14: 233–242.
56.
State Health Access Data Assistance Center (SHADAC). 2013. Comparing Federal Government Surveys that Count the Uninsured . State Health Access Data Center.
57.
U.S. Census Bureau. 2012. “America’s Families and Living Arrangements: 2012: Average Number of People: AVG1. Average Number of People per Household, by Race and Hispanic Origin/1, Marital Status, Age, and Education of Householder: 2010.” Available at: https://www.census.gov/hhes/families/data/cps2012AVG.html (accessed on October 5, 2015).
58.
U.S. Census Bureau. 2014. “Health Insurance in the United States: 2014 (P60-253), Detailed Tables, Table HI01. Health Insurance Coverage Status and Type of Coverage by Selected Characteristics: 2014.” Available at: http://www.census.gov/hhes/www/hlthins/data/index.html (accessed on October 5, 2015).
59.
U.S. Census Bureau. 2015a. “Measuring Health Insurance Coverage with the Current Population Survey: A History of Improvement.” Available at: http://www.census.gov/content/dam/Census/library/infographics/measuring_health_insurance.pdf (accessed on October 2, 2015).
60.
U.S. Census Bureau. 2015b. “About Health Insurance.” Available at: https://www.census.gov/hhes/www/hlthins/about/ (accessed on October 5, 2015).
61.
Willson, S. 2005. Cognitive Interviewing Evaluation of the National Immunization Survey Insurance Module: Results of Fieldwork and Laboratory Interviews . Unpublished report. Hyattsville, MD. National Center for Health Statistics.