Inspired by the research of Frank Andrews on the reliability and validity of survey questions, a large-scale research project was conducted in the Netherlands. The project was comprised of two different stages. For this project, more than 600 survey questions were included in different surveys according to a multitrait-multimethod design. The resulting data were analyzed in two steps. In the first step, estimates of validity and reliability were obtained for each question. The second step was a meta-analysis of the variation in data quality found in the first step. This variation was related to question-specific characteristics, response scale characteristics, context characteristics, and design characteristics. The article describes how the results of this study can be of practical use. In addition, the authors compare them to results of similar studies in the United States, Austria, and other Western, Central, and Eastern European countries.
Get full access to this article
View all access options for this article.
References
1.
Alwin, D. F.
1974. “Approaches to the Interpretation of Relationships in the Multitrait-Multimethod Matrix.” Pp. 79-105 in Sociological Methodology, edited by H. L. Costner. San Francisco: Jossey-Bass.
2.
Alwin, D. F.
and J. A. Krosnick. 1991. “The Reliability of Survey Attitude Measurement: The Influence of Question and Respondent Attributes.”Sociological Methods & Research20:139-181.
3.
Andrews, F. M.1984. “Construct Validity and Error Components of Survey Measures: A Structural Modelling Approach.”Public Opinion Quarterly48:409-422.
4.
Andrews, F. M.
, J. N. Morgan, J. A. Sonquist, and L. Klem. 1973. Multiple Classification Analysis. Ann Arbor, MI: Institute for Social Research.
5.
Bentler, P. M.
and D. G. Bonett. 1980. “Significance Tests and Goodness of Fit in the Analysis of Covariance Structures.”Psychological Bulletin88:87-104.
6.
Billiet, J.
, G. Loosveldt, and L. Waterplas. 1986. Het Survey-Interview Onderzocht: Effecten van het Ontwerp en Gebruik van Vragenlijsten op de Kwaliteit van de Antwoorden. Louvain, Belgium: Sociologisch Onderzoeksinstituut KU Leuven.
7.
Bollen, K. A.
1989. Structural Equations With Latent Variables. New York: John Wiley.
8.
Bradburn, N. M.
1983. “Response Effects.” Pp. 289-328 in Handbook of Survey Research, edited by P. H. Rossi, J. D. Wright, and A. B. Anderson. New York: Academic Press.
9.
Campbell, D. T.
and D. W. Fiske. 1959. “Convergent and Discriminant Validation by the Multimethod-Multitrait Matrix.”Psychological Bulletin56:833-853.
10.
Coenders, G. 1996. “Structural Equation Modelling of Ordinally Measured Survey Data.” Unpublished Ph.D. thesis, Universitat Ramon Llull, Barcelona, Spain.
11.
Cox, E. P.1980. “The Optimal Number of Response Alternatives for a Scale: A Review.”Journal of Marketing Research17:407-422.
12.
De Pijper, W. M.
and W. E. Saris. 1986. The Formulation of Interviews Using the Program INTERV. Amsterdam, Netherlands: Sociometric Research Foundation.
13.
Dijkstra, W.
and J. Van der Zouwen. 1982. Response Behaviour in the Survey Interview. London: Academic Press.
14.
Dillman, D. A.
1978. Mail and Telephone Surveys. New York: John Wiley.
15.
Groves, R. M.
1989. Survey Errors and Survey Costs. New York: John Wiley.
16.
Heise, D. R.
and G. W. Bohrnstedt. 1970. “Validity, Invalidity, and Reliability.” Pp. 104-129 in Sociological Methodology, edited by E. F. Borgatta and G. W. Bohrnstedt. San Francisco: Jossey-Bass.
17.
Hoogendoorn, A.
1994. Note on and Program for the Derivation of Standard Errors for MCA Coefficients. Amsterdam, Netherlands: University of Amsterdam, NIMMO.
18.
Jöreskog, K. G.1971. “Statistical Analysis of Sets of Congeneric Tests.”Psychometrika36:109-133.
19.
Jöreskog, K. G.
and D. Sörbom. 1988. PRELIS: A Program for Multivariate Data Screening and Data Summarization (2nd ed.). Mooresville, IN: Scientific Software.
20.
Jöreskog, K. G.
and D. Sörbom. 1989. LISREL VII: Users Reference Guide. Mooresville, IN: Scientific Software.
21.
Költringer, R.
1995a. “Categorization and Measurement Quality: A Population Study Using Artificial Multitrait-Multimethod Data.” In The Multitrait-Multimethod Approach to Evaluate Measurement Instruments, chap. 6, edited by W. E. Saris and A. Münnich. Budapest, Hungary: Eötvös University Press.
22.
Költringer, R.
1995b. “Measurement Quality in Austrian Personal Interview Surveys.” In The Multitrait-Multimethod Approach to Evaluate Measurement Instruments, chap. 12, edited by W. E. Saris and A. Münnich. Budapest, Hungary: Eötvös University Press.
23.
Lord, F.
and M. R. Novick. 1968. Statistical Theories of Mental Test Scores. Reading, MA: Addison-Wesley.
24.
Molenaar, N. J.
1986. Formuleringseffecten in Survey-Interviews (Wording Effects in Survey Interviews). Amsterdam, Netherlands: VU-uitgeverij.
25.
O'Brien R. M.
and P. Homer. 1987. “Correction for Coarsely Categorized Measures: LISREL's Polyserial and Polychoric Correlation Coefficients.”Psychometrika44:443-460.
26.
Rindskopf, D.1984. “Structural Equation Models: Empirical Identification, Heywood Cases, and Related Problems.”Sociological Methods & Research13:109-119.
27.
Rodgers, W. L.
, F. M. Andrews, and A. R. Herzog. 1992. “Quality of Survey Measures: A Structural Modeling Approach.”Journal of Official Statistics8:251-275.
28.
Saris, W. E.
1982. “Different Questions, Different Variables.” Pp. 78-96 in A Second Generation of Multivariate Analysis, Vol. 2: Measurement and Evaluation, edited by C. Fornell. New York: Praeger.
29.
Saris, W. E.
1990a. “The Choice of a Model for Evaluation of Measurement Instruments.” In Evaluation of Measurement Instruments by Meta-Analysis of Multitrait-Multimethod Studies, chap. 5, edited by W. E. Saris and A. van Meurs. Amsterdam, Netherlands: North-Holland.
30.
Saris, W. E.
1990b. “The Choice of a Research Design for MTMM Studies.” In Evaluation of Measurement Instruments by Meta-Analysis of Multitrait-Multimethod Studies, chap. 8, edited by W. E. Saris and A. van Meurs. Amsterdam, Netherlands: North-Holland.
31.
Saris, W. E.
1990c. “Summary of the Discussion.” In Evaluation of Measurement Instruments by Meta-Analysis of Multitrait-Multimethod Studies, chap. 12, edited by W. E. Saris and A. van Meurs. Amsterdam, Netherlands: North-Holland.
32.
Saris, W. E.
1991. Computer-Assisted Interviewing. Newbury Park, CA: Sage.
33.
Saris, W. E.
and F. M. Andrews. 1991. “Evaluation of Measurement Instruments Using a Structural Modelling Approach.” Pp. 575-599 in Measurement Errors in Surveys, edited by P. P. Biemer, R. M. Groves, L. E. Lyberg, N. Mathiowetz, and S. Sudman. New York: John Wiley.
34.
Saris, W. E.
and A. Münnich. 1995. The Multitrait-Multimethod Approach to Evaluate Measurement Instruments. Budapest, Hungary: Eötvös University Press.
35.
Saris, W. E.
, A. Satorra, and D. Sörbom. 1987. “The Detection and Correction of Specification Errors in Structural Equation Models.” Pp. 105-129 in Sociological Methodology, edited by C. C. Clogg. Washington, DC: American Sociological Association.
36.
Saris, W. E.
and A. C. Scherpenzeel. 1997. “Methodological Procedures for Comparative Research.” Forthcoming in Life Satisfaction in Western and Eastern Europe, edited by W. E. Saris, R. Veenhoven, A. C. Scherpenzeel, and B. Bunting. Budapest, Hungary: Eötvös University Press.
37.
Satorra, A.
1990. “Robustness Issues in the Analysis of MTMM and RMM Models.” Pp. 93-118 in Evaluation of Measurement Instruments by Meta-Analysis of Multitrait-Multimethod Studies, edited by W. E. Saris and A. van Meurs. Amsterdam, Netherlands: North-Holland.
38.
Scherpenzeel, A. C.
1995. A Question of Quality: Evaluating Survey Questions by Multitrait-Multimethod Studies. Amsterdam, Netherlands: University of Amsterdam, Statistics and Methodology Department.
39.
Scherpenzeel, A. C.
and W. E. Saris. 1993. “The Evaluation of Measurement Instruments by Meta-Analysis of Multitrait-Multimethod Studies.”Bulletin de Methodologie Sociologique39:3-19.
40.
Schuman, H.
and S. Presser. 1981. Questions and Answers in Attitude Surveys: Experiments on Question Form, Wording and Context. New York: Academic Press.
41.
van Driel, O. P.1978. “On Various Cases of Improper Solutions in Maximum Likelihood Factor Analysis.”Psychometrika43:225-243.
42.
van Meurs, A.
and W. E. Saris. 1990. “Memory Effects in MTMM Studies.” In Evaluation of Measurement Instruments by Meta-Analysis of Multitrait-Multimethod Studies, chap. 6, edited by W. E. Saris and A. van Meurs. Amsterdam, Netherlands: North-Holland.
43.
van Wijk, T. 1996. “Quality and Correlation: An Evaluation of Measurement Instruments by Meta-Analysis of Multitrait-Multimethod Studies.” Unpublished master's thesis, University of Amsterdam.
