Does participation in one wave of a survey have an effect on respondents’ answers to questions in subsequent waves? In this article, we investigate the presence and magnitude of “panel conditioning” effects in one of the most frequently used data sets in the social sciences: the General Social Survey (GSS). Using longitudinal records from the 2006, 2008, and 2010 surveys, we find convincing evidence that at least some GSS items suffer from this form of bias. To rule out the possibility of contamination due to selective attrition and/or unobserved heterogeneity, we strategically exploit a series of between-person comparisons across time-in-survey groups. This methodology, which can be implemented whenever researchers have access to at least three waves of rotating panel data, is described in some detail so as to facilitate future applications in data sets with similar design elements.
BailarBarbara A.1975. “Effects of Rotation Group Bias on Estimates from Panel Surveys.” Journal of the American Statistical Association70:23–30.
2.
BailarBarbara A.1989. “Information Needs, Surveys, and Measurement Errors.” Pp. 1–24 in Panel Surveys, edited by KasprzykD.DuncanG. J.KaltonG.SinghM. P.. New York: John Wiley.
3.
BenjaminiYoavHochbergYosef. 1995. “Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing.” Journal of the Royal Statistical Society, Series B (Methodological)57:289–300.
4.
CantorDavid. 2008. “A Review and Summary of Studies on Panel Conditioning.” Pp. 123–38 in Handbook of Longitudinal Research: Design, Measurement, and Analysis, edited by MenardS.. Burlington, MA: Academic Press.
5.
CasellaGeorgeBergerRoger L.. 2001. Statistical Inference. New York: Duxbury Press.
6.
ClintonJoshua D.2001. “Panel Bias from Attrition and Conditioning: A Case Study of the Knowledge Networks Panel.” Unpublished manuscript. Department of Political Science, Stanford University. Retrieved April 5, 2006 (http://www.princeton.edu/∼clinton/WorkingPapers/C_WP2001.pdf).
7.
CorderLarry S.HorvitzDaniel G.. 1989. “Panel Effects in the National Medical Care Utilization and Expenditure Survey.” Pp. 304–18 in Panel Surveys, edited by KasprzykD.DuncanG. J.KaltonG.SinghM. P.. New York: John Wiley.
8.
DasMarcelToepoelVeravan SoestArthur. 2011. “Nonparametric Tests of Panel Conditioning and Attrition Bias in Panel Surveys.” Sociological Methods & Research40:32–56.
9.
DillmanDon A.2000. Mail and Internet Surveys: The Tailored Design Method. New York: John Wiley.
10.
DuanNaiuhuaAlegriaMargaritaCaninoGlorisaMcGuireThomas G.TakeuchiDavid. 2007. “Survey Conditioning in Self-reported Mental Health Service Use: Randomized Comparison of Alternative Instrument Formats.” Health Services Research42:890–907.
11.
FeldmanJack M.LynchJohn G.. 1988. “Self-generated Validity and Other Effects of Measurement on Belief, Attitude, Intention, and Behavior.” Journal of Applied Psychology73:421–35.
12.
GelmanAndrewHillJenniferYajmiaMasanao. 2012. “Why We (Usually) Don’t Have to Worry about Multiple Comparisons.” Journal of Research on Educational Effectiveness5:189–211.
13.
GodinGastonSheeranPaschalConnerMarkGermainMarc. 2008. “Asking Questions Changes Behavior: Mere Measurement Effects on Frequency of Blood Donation.” Health Psychology27:179–84.
14.
Halpern-MannersAndrewWarrenJohn Robert. 2012. “Panel Conditioning in the Current Population Survey: Implications for Labor Force Statistics.” Demography49:1499–519.
15.
KesslerRonald C.WittchenHans-UlrichAbelsonJamie A.McGonagleKatharineSchwarzNorbertKendlerKenneth S.KnäuperBärbelZhaoShanyang. 1998. “Methodological Studies of the Composite International Diagnostic Interview (CIDI) in the US National Comorbidity Survey (NCS).” International Journal of Methods in Psychiatric Research7:33–55.
16.
KreuterFraukeMcCullochSusanPresserStanleyTourangeauRoger. 2011. “The Effects of Asking Filter Questions in Interleafed Versus Grouped Format.” Sociological Methods & Research40:88–104.
17.
KrosnickJon A.1991. “Response Strategies for Coping with the Cognitive Demands of Attitude Measures in Surveys.” Applied Cognitive Psychology5:213–36.
18.
KrosnickJon A.HolbrookAllyson L.BerentMatthew K.CarsonRichard T.Michael HanemannW.KoppRaymond J.MitchellRobert CameronPresserStanleyRuudPaul A.Kerry SmithV.MoodyWendy R.GreenMelanie C.ConawayMichael. 2002. “The Impact of ‘No Opinion’ Response Options on Data Quality: Non-attitude Reduction or an Invitation to Satisfice?” Public Opinion Quarterly66:371–403.
19.
KrumparIvar. 2013. “Determinants of Social Desirability Bias in Sensitive Surveys: A Literature Review.” Quality & Quantity47:2025–47.
20.
MathiowetzNancy A.LairTamra J.. 1994. “Getting Better? Changes or Errors in the Measurement of Functional Limitations.” Journal of Economic & Social Measurement20:237–62.
21.
McCormickMaryann K.ButlerDerrick M.SinghRajendra P.. 1992. “Investigating Time in Sample Effect for the Survey of Income and Program Participation.” Pp. 554–59 in Proceedings of the Survey Research Methods Section of the American Statistical Association.
PearsonThomas A.ManolioTeri A.. 2008. “How to Interpret a Genome-wide Association Study.” JAMA299:1775–877.
24.
RasinskiKenneth A.WillisGordon B.BaldwinAlison K.YehWenchiLeeLisa. 1999. “Methods of Data Collection, Perceptions of Risks and Losses, and Motivation to Give Truthful Answers to Sensitive Survey Questions.” Applied Cognitive Psychology13:465–84.
25.
SchaefferNora Cate. 2000. “Asking Questions about Threatening Topics: A Selective Overview.” Pp. 105–22 in The Science of Self-report: Implications for Research and Practice, edited by StoneA.TurkkanJ. S.BachrachC. A.JobeJ. B.KurtzmanH. S.CainV.. Hillsdale, NJ: Lawrence Erlbaum.
26.
ShermanSteven J.1980. “On the Self-erasing Nature of Errors of Prediction.” Journal of Personality and Social Psychology39:211–21.
27.
SmithTom W.2008. “Repeated Cross-sectional Research: The General Social Surveys.” Pp. 33–48 in Handbook of Longitudinal Research: Design, Measurement, and Analysis, edited by MenardS.. London, England: Elsevier.
28.
SmithTom W.KimJibumKochAchimParkAlison. 2007. The GSS Model of Social-science Research. GSS Project Report No. 27. Chicago, IL: National Opinion Research Center, University of Chicago.
29.
SmithTom W.SonJaesok. 2010. An Analysis of Panel Attrition and Panel Change on the 2006-2008 General Social Survey Panel. GSS Methodological Report No. 118. Chicago, IL: National Opinion Research Center, University of Chicago.
30.
SolonGary. 1986. “Effects of Rotation Group Bias on Estimation of Unemployment.” Journal of Business & Economic Statistics4:105–09.
31.
SpangenbergEric R.GreenwaldAnthony G.SprottDavid E.. 2008. “Will You Read This Article’s Abstract? Theories of the Question-behavior Effect.” Journal of Consumer Psychology18:102–06.
32.
SturgisPatrickAllumNickBrunton-SmithIan. 2009. “Attitudes over Time: The Psychology of Panel Conditioning.” Pp. 113–26 in Methodology of Longitudinal Surveys, edited by LynnP.. New York: John Wiley.
33.
ToepoelVeraDasMarcelvan SoestArthur. 2009. “Relating Question Type to Panel Conditioning: Comparing Trained and Fresh Respondents.” Survey Research Methods3:73–80.
34.
TohRex S.LeeEunkyuHuMichael Y.. 2006. “Social Desirability Bias in Diary Panels is Evident in Panelists’ Behavioral Frequency.” Psychological Reports99:322–34.
35.
TorcheFlorenciaWarrenJohn RobertHalpern-MannersAndrewValenzuelaEduardo. 2012. “Panel Conditioning in a Longitudinal Study of Chilean Adolescents’ Substance Use: Evidence from an Experiment.” Social Forces90:891–918.
36.
TourangeauRogerRipsLance J.RasinskiKenneth. 2000. The Psychology of Survey Response. New York: Cambridge University Press.
37.
TourangeauRogerYanTing. 2007. “Sensitive Questions in Surveys.” Psychological Bulletin133:859–83.
38.
van der ZouwenJohannesvan TilburgTheo. 2001. “Reactivity in Panel Studies and its Consequences for Testing Causal Hypotheses.” Sociological Methods & Research30:35–56.
39.
WangKevinCantorDavidSafirAdam. 2000. “Panel Conditioning in a Random Digit Dial Survey.” Pp. 822–27 in Proceedings of the Survey Research Methods Section of the American Statistical Association.
40.
WarrenJohn RobertHalpern-MannersAndrew. 2012. “Panel Conditioning Effects in Longitudinal Social Science Surveys.” Sociological Methods & Research41:491–534.
41.
WatertonJenniferLievesleyDenise. 1989. “Evidence of Conditioning Effects in the British Social Attitudes Panel Survey.” Pp. 319–39 in Panel Surveys, edited by KasprzykD.DuncanG. J.KaltonG.SinghM. P.. New York: John Wiley.
42.
WillisGordon B.SirkenMonreau G.NathanGad. 1994. “The Cognitive Aspects of Responses to Sensitive Survey Questions.” Working Paper Series, No. 9, National Center for Health Statistics, Hyattsville, MD.
43.
ZwaneAlix PetersonZinmanJonathanVan DusenEricParienteWilliamNullClairMiguelEdwardKremerMichaelKarlanDean S.HornbeckRichardGinéXavierDufloEstherDevotoFlorenciaCreponBrunoBanerjeeAbhijit. 2011. “Being Surveyed Can Change Later Behavior and Related Parameter Estimates.” Proceedings of the National Academy of Sciences108:1821–26.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
