In face-to-face surveys, interviewer observations are a cost-effective source of paradata for nonresponse adjustment of survey estimates and responsive survey designs. Unfortunately, recent studies have suggested that the accuracy of these observations can vary substantially among interviewers, even after controlling for household-, area-, and interviewer-level characteristics, limiting their utility. No study has identified sources of this unexplained variance in observation accuracy. Motivated by theoretical expectations from the observer bias literature, this study analyzed more than 45,000 open-ended justifications provided by interviewers in the U.S. National Survey of Family Growth (NSFG) for their observations on two key features of all sampled NSFG households: presence of children and expected probability of household response. The study finds that variability among interviewers in the cues used to record these observations (evident from the open-ended justifications) explains much of the previously unexplained variance in observation accuracy.
AmbadyNaliniHallahanMarkConnerBrett. 1999. “Accuracy of Judgments of Sexual Orientation from Thin Slices of Behavior.” Journal of Personality and Social Psychology77:538–47.
2.
AmbadyNaliniHallahanMarkRosenthalRobert. 1995. “On Judging and Being Judged Accurately in Zero-Acquaintance Situations.” Journal of Personality and Social Psychology69:518–29.
3.
BabbieEarl R.2001. The Practice of Social Research. 9th ed. Belmont, CA: Wadsworth/Thomson Learning.
4.
BaderMichael D. M.AilshireJennifer A.MorenoffJeffrey D.HouseJames S.. 2010. “Measurement of the Local Food Environment: A Comparison of Existing Data Sources.” American Journal of Epidemiology171:609–17.
5.
BernardJessie. 1945. “Observation and Generalization in Cultural Anthropology.” Journal of Sociology50:284–91.
6.
Casas-CorderoCarolinaKreuterFraukeWangYueyanBabeySusan. 2013. “Assessing the Measurement Error Properties of Interviewer Observations of Neighbourhood Characteristics.” Journal of the Royal Statistical Society (Series A)176:227–50.
7.
CaughyMargaret O.O’CampoPatricia J.PattersonJacqueline. 2001. “A Brief Observational Measure for Urban Neighborhoods.” Health & Place7:225–36.
8.
CohenM. P.1998. “Determining Sample Sizes for Surveys with Data Analyzed by Hierarchical Linear Models.” Journal of Official Statistics14:267–75.
9.
DahlhamerJames. 2012. “New Observation Questions.” Presented at the 2013 NHIS Centralized Refresher Training and Conference, December 3–6, 2012, Hyattsville, MD.
10.
DasT. Hari. 1983. “Qualitative Research in Organizational Behaviour.” Journal of Management Studies20:301–14.
11.
EckmanStephanieSinibaldiJenniferMöntmann-HertzAleksa. 2013. “Can Interviewers Effectively Rate the Likelihood of Cases to Cooperate?” Public Opinion Quarterly77:561–73.
12.
FeldmanJacob J.HymanHerbertHartCecil W.. 1951. “A Field Study of Interviewer Effects on the Quality of Survey Data.” Public Opinion Quarterly15:734–61.
13.
FunderDavid C.1987. “Errors and Mistakes: Evaluating the Accuracy of Social Judgment.” Psychological Bulletin101:75–90.
14.
FunderDavid C.1995. “On the Accuracy of Personality Judgment: A Realistic Approach.” Psychological Review102:652–70.
15.
GrahamRobert J.1984. “Anthropology and O.R.: The Place of Observation in Management Science Process.” The Journal of the Operational Research Society35:527–36.
16.
GrovesRobert M.HeeringaSteven G.. 2006. “Responsive Design for Household Surveys: Tools for Actively Controlling Survey Errors and Costs.” Journal of the Royal Statistical Society (Series A)169:439–57.
17.
HarrisKarij J.JeromeNorge W.FawcettStephen B.. 1997. “Rapid Assessment Procedures: A Review and Critique.” Human Organization56:375–8.
18.
HillMark O.1973. “Diversity and Evenness: A Unifying Notation and Its Consequences.” Ecology54:427–32.
19.
HosmerDavid W.LemeshowStanleySturdivantRodney X.. 2013. Applied Logistic Regression. 3rd ed. New York, NY: John Wiley.
20.
JerolmackColinKhanShamus. 2014. “Talk Is Cheap: Ethnography and the Attitudinal Fallacy.” Sociological Methods and Research43:178–209.
21.
JohnsonAllenSackettRoss. 1998. “Direct Systematic Observation of Behavior.” Pp. 301–32 in Handbook of Methods in Cultural Anthropology, chapter 9, edited by BernardH. R.Beverly Hills, CA: AltaMira Press.
22.
JonesEdward E.RiggsJanet M.QuattroneGeorge. 1979. “Observer Bias in the Attitude Attribution Paradigm: Effect of Time and Information Order.” Journal of Personality and Social Psychology37:1230–8.
23.
JonesMaliaPebleyAnne R.SastryNarayan. 2011. “Eyes on the Block: Measuring Urban Physical Disorder through In-Person Observation.” Social Science Research40:523–37.
24.
KazdinAlan E.1977. “Artifact, Bias, and Complexity of Assessment: The ABCs of Reliability.” Journal of Applied Behavior Analysis10:141–50.
25.
KimYoonsangChoiYoung-KuEmerySherry. 2013. “Logistic Regression with Multiple Random Effects: A Simulation Study of Estimation Methods and Statistical Packages.” The American Statistician67:171–82.
26.
KreuterFrauke, ed. 2013. Improving Surveys with Paradata: Analytic Uses of Process Information. Hoboken, NJ: Wiley.
27.
KreuterFraukeOlsonKristen M.WagnerJamesYanTingEzzati-RiceTrena M.Casas-CorderoCarolinaLemayMichaelPeytchevAndyGrovesRobert M.RaghunathanTrivellore E.. 2010. “Using Proxy Measures and Other Correlates of Survey Outcomes to Adjust for Nonresponse: Examples from Multiple Surveys.” Journal of Royal Statistical Society (Series A)173:389–407.
28.
LandisJ. RichardKochGary G.. 1977. “The Measurement of Observer Agreement for Categorical Data.” Biometrics33:159–74.
29.
LepkowskiJames M.MosherWilliam D.GrovesRobert M.WestBrady T.WagnerJamesGuHayou. 2013. “Responsive Design, Weighting, and Variance Estimation in the 2006-2010 National Survey of Family Growth.” National Center for Health Statistics, Vital and Health Statistics2.
30.
MandersonLenoreAabyPeter. 1992a. “An Epidemic in the Field? Rapid Assessment Procedures and Health Research.” Social Science Medicine35:839–50.
31.
MandersonLenoreAabyPeter. 1992b. “Can Rapid Anthropological Procedures Be Applied to Tropical Diseases?” Health Policy and Planning7:46–55.
32.
McCallGeorge J.1984. “Systematic Field Observation.” Annual Review of Sociology10:263–82.
33.
MillenDavid R.2000. “Rapid Ethnography: Time Deepening Strategies for HCI Field Research.” Proceedings on DIS00: Designing Interactive Systems: Processes, Practices, Methods, and Techniques, August 17-19, 2000, Brooklyn, NY, 280–6.
34.
MostSteven B.SchollBrian J.CliffordErin R.SimonsDaniel J.. 2005. “What You See Is What You Get: Sustained Inattentional Blindness and the Capture of Awareness.” Psychological Review112:217–42.
35.
PattersonMiles L.FosterJeffrey L.BellmerCraig D.. 2001. “Another Look at Accuracy and Confidence in Social Judgments.” Journal of Nonverbal Behavior25:207–19.
36.
PattersonMiles L.StockbridgeErica. 1998. “Effects of Cognitive Demand and Judgment Strategy on Person Perception Accuracy.” Journal of Nonverbal Behavior22:253–63.
37.
PattonMichael Q.1999. “Enhancing the Quality and Credibility of Qualitative Analysis.” Health Services Research34:1189–208.
38.
PetticrewMarkSempleSeanHiltonShonaCreelyKaen S.EadieDouglasRitchieDeborahFerrellCatherineChristopherYvetteHurleyFintan. 2007. “Covert Observation in Practice: Lessons from the Evaluation of the Prohibition of Smoking in Public Places in Scotland.” BMC Public Health7:204.
39.
ReppAlan C.NieminenGayla S.OlingerEllenBruscaRita. 1998. “Direct Observation: Factors Affecting the Accuracy of Observers.” Exceptional Children55:29–36.
40.
SeidlerJohn. 1974. “On Using Informants: A Technique for Collecting Quantitative Data and Controlling Measurement Error in Organization Analysis.” American Sociological Review39:816–31.
41.
SimmonsSandra F.BabineauSarahGarciaEmilySchnelleJohn F.. 2002. “Quality Assessment in Nursing Homes by Systematic Direct Observation: Feeding Assistance.” Journal of Gerontology: Medical Sciences57A: M665–71.
42.
SimonsDaniel J.JensenMelinda S.. 2009. “The Effects of Individual Differences and Task Difficulty on Inattentional Blindness.” Psychonomic Bulletin and Review16:398–403.
43.
SinibaldiJenniferDurrantGabriele B.KreuterFrauke. 2013. “Evaluating the Measurement Error of Interviewer Observed Paradata.” Public Opinion Quarterly77:173–93.
44.
SinibaldiJenniferEckmanStephanie. 2015. “Using Call-level Interviewer Observations to Improve Response Propensity Models.” Public Opinion Quarterly79:976–93.
45.
SinibaldiJenniferTrappmannMarkKreuterFrauke. 2014. “Which Is the Better Investment for Nonresponse Adjustment: Purchasing Commercial Auxiliary Data or Collecting Interviewer Observations?” Public Opinion Quarterly78:440–73.
46.
SmithHeather J.ArcherDaneCostanzoMark. 1991. “‘Just a Hunch’: Accuracy and Awareness in Person Perception.” Journal of Nonverbal Behavior15:3–17.
47.
StähliMichèle E.2010. “Examples and Experiences from the Swiss Interviewer Training on Observable Data (Neighborhood Characteristics) for ESS 2010 (R5).” Paper presented at the NC Meeting, March 31 to April 1, 2011, Mannheim, Germany.
48.
TverskyAmosKahnemanDaniel. 1974. “Judgment under Uncertainty: Heuristics and Biases.” Science185:1124–31.
49.
WagnerJamesWestBrady T.KirgisNicoleLepkowskiJames M.AxinnWilliam G.Kruger-NdiayeShonda. 2012. “Use of Paradata in a Responsive Design Framework to Manage a Field Data Collection.” Journal of Official Statistics28:477–99.
50.
WagnerJamesWestBrady T.GuyerHeidiBurtonPaulKelleyJenniferCouperMick P.MosherWilliam D.. 2017. “The Effects of a Mid-Data Collection Change in Financial Incentives on Total Survey Error in the National Survey of Family Growth.” Pp. 155–177 In Total Survey Error in Practice, chapter 8, edited by BiemerPaul P.LeeuwEdith deEckmanStephanieEdwardsBradKreuterFraukeLybergLars E.Clyde TuckerN.WestBrady T.. New York, NY: John Wiley.
51.
WestBrady T.2011. The Error Properties of Interviewer Observations and Their Implications for Nonresponse Adjustment of Survey Estimates. Doctoral dissertation, University of Michigan, Ann Arbor.
52.
WestBrady T. 2013a. “An Examination of the Quality and Utility of Interviewer Observations in the National Survey of Family Growth (NSFG).” Journal of the Royal Statistical Society (Series A)176:211–25.
53.
WestBrady T.2013b. “The Effects of Error in Paradata on Weighting Class Adjustments: A Simulation Study.” Pp. 361–88 in Improving Surveys with Paradata: Making Use of Survey Process Information, Chapter 15, edited byKreuterF.. Hoboken, NJ: Wiley.
54.
WestBrady T.KreuterFrauke. 2011. “Observational Strategies Associated with Increased Accuracy of Interviewer Observations: Evidence from the National Survey of Family Growth.” Proceedings of the Survey Research Methods Section of the American Statistical Association (AAPOR 2011), American Statistical Association, Alexandria, VA.
55.
WestBrady T.KreuterFrauke. 2013. “Factors Impacting the Accuracy of Interviewer Observations: Evidence from the National Survey of Family Growth (NSFG).” Public Opinion Quarterly77:522–48.
56.
WestBrady T.KreuterFrauke. 2015. “A Practical Technique for Improving the Accuracy of Interviewer Observations of Respondent Characteristics.” Field Methods27:144–62.
57.
WestBrady T.LittleRoderick J.A.. 2013. “Nonresponse Adjustment of Survey Estimates based on Auxiliary Variables Subject to Error.” Journal of the Royal Statistical Society (Series C) (Applied Statistics)62:213–31.
58.
WestBrady T.KreuterFraukeTrappmannMark. 2014. “Is the Collection of Interviewer Observations Worthwhile in an Economic Panel Survey? New Evidence from the German Labor Market and Social Security (PASS) Study.” Journal of Survey Statistics and Methodology2:159–81.
59.
WestBrady T.SinibaldiJennifer. 2013. “The Quality of Paradata: A Literature Review.” Pp. 339–359 In Improving Surveys with Paradata, chapter 14, edited byKreuterFrauke. New York, NY: John Wiley.
60.
ZenkShannon N.SchulzAmy J.MentzGracielaHouseJames S.GravleeClarence C.MirandaPatricia Y.MillerPatriciaKannanSrimanthi. 2007. “Inter-Rater and Test-Retest Reliability: Methods and Results for the Neighborhood Observational Checklist.” Health & Place13:452–65.
61.
ZhangDaowenLinXihong. 2008. “Variance Component Testing in Generalized Linear Mixed Models for Longitudinal/Clustered Data and Other Related Topics.” Pp. 19–36 in Random Effect and Latent Variable Model Selection. edited by, DunsonDavid B.New York, NY: Springer.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
