Telephone survey interviewers need to be able to accurately record answers to questions. While straightforward for closed questions, this task can be complicated for open questions. We examine interviewer recording accuracy rates from a national landline random digit dial telephone survey. We find that accuracy rates are over 90% for numeric response and interviewer-code, single-response items but are astonishingly low (49%) for a multiple-answer, nominal, interviewer-code item. Accuracy rates for narrative open questions were around 90% for themes but only about 70% for themes and elaborations. Interviewer behaviors (e.g., probing, feedback) are generally associated with lower accuracy rates. Implications for questionnaire design, interviewer training, and coding procedures are discussed.
BradburnN.SudmanS.WansinkB.. 2004. Asking questions: The definitive guide to questionnaire design for market research, political polls, and social and health questionnaires. San Francisco: Jossey-Bass.
3.
CouperM. P.KennedyC.ConradF. G.TourangeauR.. 2011. Designing input fields for non-narrative open-ended responses in web surveys. Journal of Official Statistics27:65–85.
4.
DielmanL.CouperM. P.. 1995. Data quality in a CAPI survey: Keying errors. Journal of Official Statistics11:141–46.
5.
DijkstraW.OngenaY.. 2006. Question–answer sequences in survey interviews. Quality & Quantity40:983–1011.
6.
DillmanD. A.SmythJ. D.ChristianL. M.. 2014. Internet, phone, mail, and mixed-mode surveys: The tailored design method. Hoboken, NJ: John Wiley.
GrovesR. M.MagilavyL. J.. 1980. Estimates of interviewer variance in telephone surveys. Proceedings of the American Statistical Association, Survey Research Methods Section, pp. 622–27. www.asasrms.org.
10.
GrovesR. M.NichollsW. L., II. 1986. The status of computer-assisted telephone interviewing: Part II—Data quality issues. Journal of Official Statistics2:117–34.
11.
GwartneyP. A.2007. The telephone interviewer’s handbook: How to conduct standardized conversations. San Francisco: Jossey-Bass.
12.
HicksW. D.EdwardsB.TourangeauK.McBrideB.Harris-KojetinL. D.MossA. J.. 2010. Using CARI tools to understand measurement error. Public Opinion Quarterly74:985–1003.
13.
JapecL.2008. Interviewer error and interviewer burden. In Advances in telephone survey methodology, eds. LepkowskiJ. M.TuckerC.BrickJ. M.de LeeuwE. D.JapecL.LavrakasP. J.LinkM. W.SangsterR. L., 187–211. Hoboken, NJ: John Wiley.
14.
KennedyJ. M.LengacherJ. E.DemerathL.. 1990. Interviewer entry error in CATI interviews. Paper presented at the International Conference on Measurement Errors in Surveys, Tucson, AZ, November11–14.
LepkowskiJ. M.SandoskyS. A.CouperM. P.ChardoulS.CarnL.ScottL. J.. 1995. A Comparison of recording errors between CATI and paper-and-pencil data collection methods. Proceedings of the American Statistical Association, Survey Research Methods Section, pp. 521–26. www.asasrms.org.
17.
LepkowskiJ. M.SadoskyS. A.WeissP. S.. 1998. Mode, behavior, and data recording error. In Computer assisted survey information collection, eds. CouperM. P.BakerR. P.BethlehemJ.ClarkC. Z. F.MartinJ.NichollsW. L.IIO’ReillyJ. M., 367–88. New York: John Wiley.
18.
MitchellS.StrobleM.FahrneyK.NguyenM.BibbB.ThissenM. R.StephensonW.. 2008. Using computer audio-recorded interviewing to assess interviewer coding error. Proceedings of the American Statistical Association, Survey Research Methods Section, pp. 4414–21. www.asasrms.org.
RustemeyerA.1977. Measuring interviewer performance in mock interviews. Proceedings of the American Statistical Association, Social Statistics Section, pp. 341–46. www.asasrms.org.
SchwarzN.HipplerH. J.DeutschB.StrackF.. 1985. Response scales: Effects of category range on reported behavior and comparative judgments. Public Opinion Quarterly49:388–95.
23.
SchwarzN.StrackF.HipplerH. J.BishopG.. 1991. The impact of administration mode on response effects in survey measurement. Applied Cognitive Psychology5:193–212.
24.
SmythJ. D.DillmanD. A.ChristianL. M.McBrideM.. 2009. Open-ended questions in web surveys: Can increasing the size of answer boxes and providing extra verbal instructions improve response quality?Public Opinion Quarterly73:325–37.
25.
StroblM.FahrneyK.NguyenM.BibbB.ThissenM. R.StephensonW.MitchellS.. 2008. Using computer audio-recorded interviewing to assess interviewer coding error. Paper presented at the American Association for Public Opinion Research, New Orleans, LA, May15–18.
26.
SudmanS.BradburnN. M.. 1982. Asking questions: A practical guide to questionnaire design. San Francisco: Jossey-Bass.
27.
TarnaiJ.KennedyJ.ScudderD.. 1998. Organizational effects of CATI in small to medium survey centers. In Computer assisted survey information collection, eds. CouperM. P.BakerR. P.BethlehemJ.ClarkC. Z. F.MartinJ.NichollsW. L.IIO’ReillyJ. M., 85–104. New York: John Wiley.
28.
ThissenR. M.2014. Computer audio-recorded interviewing as a tool for survey research. Social Science Computer Review32:90–104.
29.
TourangeauR.RipsL.RasinskiK.. 2000. The psychology of survey response. Cambridge: Cambridge University Press.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
