Two distinctly different quantitative approaches are used to evaluate measurement instruments: the split-ballot experiment and the multitrait-multimethod (MTMM) approach. The first approach is typically used to indicate whether variation in the method causes differences in the response distribution; the second approach evaluates the reliability and validity of different methods. The new approach, suggested in this paper, combines the more attractive features of both methods. The strength of the split-ballot experiment is its use of independent random samples from the same population to provide information about differences in response distributions. This is also possible with the new approach, but this approach provides more detailed information about the reasons
Get full access to this article
View all access options for this article.
References
1.
AllisonP. D.1987. “Estimation of Linear Models with Incomplete Data.” Pp. 71–103 in Sociological Methodology, Vol 17, edited by CloggC. C.. Washington, DC: American Sociological Association.
2.
AlthauserR. P.HeberleinT. A.ScottR. A.. 1971. “A Causal Assessment of Validity: The Augmented Multitrait-Multimethod Matrix.” Pp. 151–69 in Causal Models in the Social Sciences, edited by BlalockH. M.Jr.Chicago: Aldine.
3.
AlwinD.1974. “An Analytic Comparison of Four Approaches to the Interpretation of Relationships in the Multitrait-Multimethod Matrix.” Pp. 79–105 in Sociological Methodology, edited by CostnerH. L.. San Francisco: Jossey-Bass.
4.
AndrewsF. M.1984. “Construct Validity and Error Components of Survey Measures. A Structural Modeling Approach.”Public Opinion Quarterly48:409–42.
5.
ArmingerG.SobelM.E.. 1990. “Pseudo-Maximum Likelihood Estimation of Mean and Covariance Structures with Missing Data.”Journal of the American Statistical Association85:195–203.
6.
BagozziR. P.YiY.. 1991. “Multitrait-Multimethod Matrices in Consumer Research.”Journal of Consumer Research17:426–39.
7.
BillietJ.LoosveldtG.WaterplasL.. 1986. “Het Survey-Interview Onderzocht: Effecten van het Ontwerp en Gebruik van Vragenlijsten op de Kwaliteit van de Antwoorden” (Research on surveys: effects of the design and use of questionnaires on the quality of the responses). Leuven, Belgium: Sociologisch Onderzoeksinstituut KU Leuven.
8.
BrannickM. T.SpectorP. E.. 1990. “Estimation Problems in the Block-Diagonal Model of the Multitrait-Multimethod Matrix.”Applied Psychological Measurement14:325–39.
9.
BrowneM. W.1984. “The Decomposition of Multitrait-Multimethod Matrices.”British Journal of Mathematical and Statistical Psychology37:1–21.
10.
BuntingB.AdamsonG.MulhallP. K.. 2002. “A Monte Carlo Examination of an MTMM Model with Planned Incomplete Data Structures.”Structural Equation Modeling9:369–89.
11.
CampbellD. T.FiskeD. W.. 1959. “Convergent and Discriminant Validation by the Multitrait Multimethod Matrices.”Psychological Bulletin56: 81–105.
12.
CampbellD. T.O'ConnellE. J.1967. “Method Factors in Multitrait-Multimethod Matrices: Multiplicative Rather Than Additive?”Multivariate Behavioral Research2:409–26.
13.
CoendersG.SarisW. E.. 1995. “Categorization and Quality: The Choice Between Pearson and Polychoric Correlations.” Pp. 125–44 in The Multitrait-Multimethod Approach to Evaluate Measurement Instruments, edited by SarisW. E.MünnichA.Budapest: Eötvös University Press.
14.
CoendersG.SatorraA.SarisW. E.. 1997. “Alternative approaches to Structural Modeling of Ordinal Data: A Monte Carlo Study.”Structural Equation Modeling4:261–82.
15.
CoendersG.SatorraA.SarisW. E.. 1998. “Relationship Between a Restricted Correlated Uniqueness Model and a Direct Product Model for Multitrait-Multimethod Data.” Pp. 151–72 in Advances in Methodology, Data Analysis and Statistics, Metodoloki Zvezki, Vol 14. Ljubljana, Slovenia.
16.
CoendersG.SatorraA.SarisW. E.. 2000. “Testing Nested Additive, Multiplicative and General Multitrait-Multimethod Models.”Structural Equation Modeling7:219–50.
17.
CortenI.SarisW. E.CoendersG.van der VeldW.AlbersC.CornelisC.. 2002. “The Fit of Different Models for Multitrait-Multimethod Experiments.”Structural Equation Modeling9:213–32.
18.
CudeckR.1988. “Multiplicative Models and MTMM Matrices.”Journal of Educational Statistics13, 131–47.
19.
EidM.2000. “Multitrait-Multimethod Model with Minimal Assumptions.”Psychometrika65:241–61.
20.
European Social Survey. 2002. European Social Survey Round 1: Report of the First Year. London: Natcen.
21.
GraysonD.MarshH. W.. 1994. “Identification with Deficient Rank Loading Matrices in Confirmatory Analysis: Multitrait-Multimethod Models.”Psychometrika59:121–34.
22.
JöreskogK. G.1971. “Simultaneous Factor Analysis in Several Populations.”Psychometrika34:409–26.
23.
JöreskogK. G.1990. “New Developments in LISREL: Analysis of Ordinal Variables Using Polychoric Correlations and Weighted Least Squares.”Quality and Quantity24:387–404.
24.
KennyD. A.1976. “An Empirical Application of Confirmatory Factor Analysis to the Multitrait-Multimethod Matrix.”Journal of Experimental Social Psychology12:247–52.
25.
KennyD. A.KashyD. A.1992. “Analysis of the Multitrait-Multimethod Matrix by Confirmatory Factor Analysis.”Psychological Bulletin112:165–72.
26.
KogovšekT.FerligojA.CoendersG.SarisW. E.. 2002. “Estimating Reliability and Validity of Personal Support Measures: Full Information ML Estimation with Planned Incomplete Data.”Social Networks24:1–20.
27.
KöltringerR.1995. “Measurement Quality in Austria Personal Interviews.” Pp. 207–25 in The Multitrait-Multimethod Approach to Evaluate Measurement Instruments, edited by SarisW.E.MünnichA.Budapest: Eötvös University Press.
28.
MarshH. W.1989. “Confirmatory Factor Analysis of Multitrait-Multimethod Data: Many Problems and Few Solutions.”Applied Psychological Measurement13:335–61.
29.
MarshH. W.BaileyM.. 1991. “Confirmatory Factor Analyses of Multitrait-Multimethod Data: Comparison of the Behavior of Alternative Models.”Applied Psychological Measurement15:47–70.
30.
van MeursA.SarisW. E.. 1990. “Memory Effects in MTMM Studies.” Pp. 134–46 in Evaluation of Measurement Instruments by Meta-analysis of Multitrait-Multimethod Studies, edited by SarisW. E.van MeursA.. Amsterdam: North Holland.
31.
MuthénB.1984. “A General Structural Equation Model with Dichotomous, Ordered Categorical, and Continuous Latent Variable Indicators.”Psychometrika49:115–32.
32.
MuthénB.KaplanD.HollisM.. 1987. “On Structural Equation Modeling with Data That Are Not Missing Completely at Random.”Psychometrika52:431–62.
33.
OlssonU.1979. “Maximum-Likelihood Estimation of the Polychoric Correlation Coefficient.”Psychometrika44:443–60.
34.
RodgersW. L.AndrewsF. M.HerzogA. R.. 1992. “Quality of Survey Measures: A Structural Modeling Approach.”Journal of Official Statistics8:251–75.
35.
SarisW. E.1990. “The Choice of a Model for Evaluation of Measurement Instruments.” Pp. 118–29 in Evaluation of Measurement Instruments by Meta-analysis of Multitrait Multimethod Matrices, edited by SarisW. E.van MeursA.. Amsterdam: North Holland.
36.
SarisW. E.1998. “A New Approach for Evaluation of Measurement Instruments: The Split-Ballot MTMM Design.” Presented at the International Conference on Methodology and Statistics, Preddvor, Slovenia, September 20–22, 1998.
37.
SarisW. E.AndrewsF. M.. 1991. “Evaluation of Measurement Instruments Using a Structural Modeling Approach.” Pp. 575–99 in Measurement Errors in Surveys, edited by BiemerP. P. et.al. New York: Wiley.
38.
SarisW. E.GallhoferI. N.. Forthcoming. “Estimation of the Effects of Measurement Characteristics on the Quality of Survey Questions.”.
39.
SarisW. E.KrosnickJ.. Forthcoming. “Comparing Questions with Agree/Disagree Response Options to Questions with Construct Specific Response Options.”.
40.
SarisW. E.MunnichA.. 1995. The Multitrait-Multimethod Approach to Evaluate Measurement Instruments. Budapest: Eötvös University Press.
41.
SarisW. E.Van WijkT.ScherpenzeelA.. 1998. “Validity and Reliability of Subjective Social Indicators: The Effect of Different Measures of Association.”Social Indicators Research45:173–99.
42.
SatorraA.1992. “Asymptotic Robust Inferences in the Analysis of Mean and Covariance Structures.” Pp. 249–78 in Sociological Methodology, Vol. 22, edited by MarsdenP. V.. Cambridge, MA: Basil Blackwell.
43.
SatorraA.1993. “Asymptotic Robust Inferences in Multi-Sample Analysis of Augmented-Moment Structures.” Pp. 211–29 in Multivariate Analysis: Future Directions, Vol. 2, edited by CuadrasC. M.RaoC. R.. Amsterdam: Elsevier.
44.
SatorraA.2001. “Goodness of Fit Testing of Structural Equation Models with Multiple Group Data and Nonnormality.” Chap. 12 in Structural Equation Modeling: Present and Future, edited by CudeckR.du ToitS.SörbomD.Lincolnwood, IL: Scientific Software International. SSI.
45.
ScherpenzeelA. C.1995. “A Question of Quality. Evaluating Survey Questions by Multitrait-Multimethod Studies.” Ph.D. dissertation, University of Amsterdam, Leidschendam, the Netherlands.
46.
ScherpenzeelA.SarisW. E.. 1997. “The Validity and Reliability of Survey Questions: A Meta-analysis of MTMM Studies.”Sociological Methods and Research25:341–83.
47.
SchumanH.PresserS., 1981. Questions and Answers in Attitude Surveys: Experiments on Question Form, Order, and Context. New York: Academic Press.
WothkeW.1996. “Models for Multitrait-Multimethod Matrix Analysis.” Pp. 7–56 in Advanced Structural Equation Modeling. Issues and Techniques, edited by MarcoulidesG. C.SchumackerR. E.. Mahwah, NJ: Lawrence Erlbaum.
