In survey data, four potential sources of measurement error can jeopardize the results: the respondents, the interviewers, the questions, and the data collection method. In the past two decades, a shift has occurred in the way survey data are collected; telephone surveys and, to a lesser degree, mail surveys are now more extensively used. This has stimulated empirical research on the influence of the data collection method on data quality. Most of these mode comparisons used univariate criteria. In this study we concentrate on the potential influence of the data collection method on two substantive structural models. A controlled field experiment was conducted in which a mail, a telephone, and a face-to-face survey were compared. Using multigroup comparisons, two substantive structural equation models (one for loneliness and one for general well-being) were compared across the different data collection methods. The different data collection methods turned out to produce significantly different covariance matrices. Subsequent analyses showed that structural models, based on these covariance matrices, also differed.
Get full access to this article
View all access options for this article.
References
1.
Baumgartner, Robert M.
and Thomas A. Heberlein. 1984. “Recent Research on Mailed Questionnaire Response Rate.” Pp. 65-76 in Making Effective Use of Mailed Questionnaires, edited by D. C. Lockhardt. San Francisco: Jossey-Bass.
2.
Bentler, Peter M.
and Douglas G. Bonett. 1980. “Significance Tests and Goodness-of-Fit in the Analysis of Covariance Structures.”Psychological Bulletin88:588-600.
3.
Bollen, Kenneth E.
1989. Structural Equations with Latent Variables. New York: Wiley.
4.
Bradburn, Norman M.
1969. The Structure of Well-being. Chicago: Aldine.
5.
Browne, Michael W.1984. “Asymptotically Distribution-Free Methods for the Analysis of Covariance Structures.”The British Journal of Mathematical and Statistical Psychology71:62-83.
6.
Burt, Ronald S.
, Michael G. Fischer, and Kenneth P. Christman. 1979. “Structures of Well-Being: Sufficient Conditions for Identification as Restricted Covariance Models.”Sociological Methods & Research8:111-120.
7.
Burt, Ronald S.
, James A. Wiley, Michael J. Minor, and James R. Murray. 1978. “Structure of Well-being: Form, Content, and Stability Over Time.”Sociological Methods & Research6:365-407.
8.
Byrne, Barbara M.
1989. A Primer of Lisrel. New York: Springer.
9.
Converse, Jean M.
and Stanley Presser. 1986. Survey Questions: Handcrafting the Standardized Questionnaire. Beverly Hills, CA: Sage.
10.
Cook, Thomas D.
and Donald T. Campbell. 1979. Quasi-Experimentation: Design & Analysis Issues. Chicago: Rand McNally.
11.
De Jong-Gierveld, Jenny1987. “Developing and Testing a Model of Loneliness.”Journal of Personality and Social Psychology53:119-128.
12.
De Jong-Gierveld, Jenny
and Frans Kamphuis. 1985. “The Development of a Rasch-Type Loneliness Scale.”Applied Psychological Measurement9:289-299.
13.
De Leeuw, Edith D. 1991. The Influence of Data Collection Procedure on Psychometric Reliability and Scaling Properties. Response Effects in Surveys, Technical Report no. 5. Amsterdam: Vrije Universiteit, Department of Social Research Methodology.
14.
De Leeuw, Edith D.
1992. Data Quality in Mail, Telephone, and Face-to-Face Surveys. Amsterdam: TT-Publikaties.
15.
De Leeuw, Edith D.
and Joop J. Hox. 1988. “The Effects of Response-Stimulating Factors on Response Rates and Data Quality in Mail Surveys: A Test of Dillman's Total Design Method.”Journal of Official Statistics4:241-250.
16.
De Leeuw, Edith D.
and Joop J. Hox. 1993. “Mode Effects in Structural Modeling; A Lisrel Multi-Group Comparison of Mail, Telephone, and Face-to-Face Survey Data.” Pp. 119-144 in Methodische Grundlagen und Anwendungen von Strukturgleichungsmodellen, edited by J. Reinecke and G. Krekeler. Mannheim, Germany: FRG e.V.
17.
Dillman, Don A.
1978. Mail and Telephone Surveys: The Total Design Method. New York: Wiley.
18.
Dillman, Don A.1991. “The Design and Administration of Mail Surveys.”Annual Review of Sociology17:225-249.
19.
Dillman, Don A.
and John Tarnai. 1988. “Administrative Issues in Mixed Mode Surveys.” Pp. 509-528 in Telephone Survey Methodology, edited by Robert M. Groves, Paul P. Biemer, Lars E. Lyberg, James T. Massey, William L. Nicholls II, and Joseph Waksberg. New York: Wiley.
20.
Edgington, Eugene S.
1987. Randomization Tests. New York: Marcel Dekker.
21.
Forsyth, Barbara H.
and Judith T. Lessler. 1991. “Cognitive Laboratory Methods: A Taxonomy.” Pp. 393-418 in Measurement Errors in Surveys, edited by P. P. Biemer, R. M. Groves, L. E. Lyberg, N. A. Mathiowetz, and S. Sudman. New York: Wiley.
22.
Fruchtgott, Ernest
. 1984. “Replicate, Again and Again.”American Psychologist33:1315-1316.
23.
Good, Philip
. 1994. Permutation Tests. New York: Springer.
24.
Goyder, John
. 1985. “Face-to-Face Interviews and Mailed Questionnaires: The Net Difference in Response Rate.”Public Opinion Quarterly49:234-252.
25.
Goyder, John
. 1987. The Silent Minority: Nonrespondents on Sample Surveys. Cambridge: Polity Press.
26.
Groves, Robert M.
1989. Survey Errors and Survey Costs. New York: Wiley.
27.
Groves, Robert M.
and Robert L. Kahn. 1979. Surveys by Telephone: A National Comparison with Personal Interviews. New York: Academic Press.
28.
Hedges, Larry V.
and Ingram Olkin. 1985. Statistical Methods for Meta-Analysis. London: Academic Press.
29.
Hox, Joop J. 1986. Het Gegruik van Hulptheoriën bij Operationaliseren (Using auxiliary theories for operationalization: A study of the construct of subjective well-being). Unpublished doctoral dissertation, University of Amsterdam, Amsterdam, the Netherlands.
30.
Hox, Joop J.
and Edith D. De Leeuw. 1994. “A Comparison of Nonresponse in Mail, Telephone, and Face-to-Face Surveys; Applying Multilevel Modeling to Meta-Analysis.”Quality and Quantity28:329-344.
31.
Hox, Joop J.
, Edith D. De Leeuw, and Ita G. G. Kreft. 1991. “The Effect of Interviewer and Respondent Characteristics on the Quality of Survey Data: A Multilevel Model.” Pp. 439-461 in Measurement Errors in Surveys, edited by P. P. Biemer, R. M. Groves, L. E. Lyberg, N. A. Mathiowetz, and S. Sudman. New York: Wiley.
32.
Jöreskog, Karl G.
and Dag Sörbom. 1989. Lisrel 7: A Guide to the Program and Applications. 2nd ed.Chicago: SPSS.
33.
Keenan, Brian
and Marilyn Mauch. 1986. Developing and Using Questionnaires. Washington, DC: U.S. General Accounting Office, Program Evaluation and Methodology Division.
34.
Kerssemakers, Frans A. M.
1985. “Telefonisch Enquêteren” (Telephone interviewing). Pp. 211-230 in CBS-Select 3. Voorburg/Heerlen, the Netherlands: Statistics Netherlands.
35.
Krosnick, Jon A.
, and Duane F. Alwin. 1988. “The Form-Resistant Correlation Hypothesis.”Public Opinion Quarterly52:526-538.
36.
Louwen, Frank
. 1992. “Bereidwillig maar niet bereikbaar of bereikbaar maar niet bereidwillig” (Willing but unreachable, or reachable but unwilling). Onderzoek10:5-9.
37.
Lubin, Ardie
. 1957. “Replicability as a Publication Criterion.”American Psychologist12:519-520.
38.
Lyberg, Lars
and Daniel Kasprzyk. 1991. “Data Collection Methods and Measurement Error: An Overview.” Pp. 237-257 in Measurement Errors in Surveys, edited by P. P. Biemer, R. M. Groves, L. E. Lyberg, N. A. Mathiowetz, and S. Sudman. New York: Wiley.
39.
Noreen, Eric W.
1989. Computer Intensive Methods for Testing Hypotheses: An Introduction. New York: Wiley.
40.
Oldendick, Robert W.
, George F. Bishop, Susan B. Sorenson, and Alfred J. Tuchfarber. 1988. “A Comparison of the Kish and Birthday Methods for Respondent Selection in Telephone Surveys.”Journal of Official Statistics4:307-318.
41.
Rosenberg, Morris
. 1979. Conceiving the Self. New York: Basic Books.
42.
Sackman, Harold
. 1974. Delphi Assessment: Expert Opinion, Forecasting and Group Process. Santa Monica, CA: RAND.
43.
Schuman, Howard
, and Stanley Presser. 1981. Questions and Answers in Attitude Surveys. New York: Academic Press.
44.
Schweizer, Karl
. 1989. “Eine Analyse der Konzepte, Bedingungen und Zielsetzungen von Replikationen” (An analysis of the concepts, conditions, and aims of replications). Archiv für Psychologie141:85-97.
45.
Snijkers, Ger J.M.E.1992. “Computergestuurd Enquêteren: Telefonisch of Persoonlijk?” (Computer-assisted interviewing: By telephone or in person?). Kwantitatieve Methoden39:53-69.
46.
Sudman, Seymour
and Norman M. Bradburn. 1982. Asking Questions: A Practical Guide to Questionnaire Design. San Francisco: Jossey-Bass.
47.
Trewin, Dennis
and Geoff Lee. 1988. “International Comparison of Telephone Coverage.” Pp. 9-24 in Telephone Survey Methodology, edited by R. M. Groves, P. P. Biemer, L. E. Lyberg, J. T. Massey, W. L. Nicholls, II., and J. Waksberg. New York: Wiley.
