This study investigated the performance of five small-sample statistics (Lord's, Kristof's, McNemar's, Forsyth and Feldt's, and Braden's) that test whether two variables measure the same trait except for mea surement error. The conservative Type I error rates of the Lord and Kristof procedures and the liberal error rates of the McNemar, Forsyth and Feldt, and Braden procedures were corrected by determining appropriate critical values. Power comparisons were then made at the fixed α levels. In general, the McNemar statistic was shown to be the most powerful. Finally, the ef fects of non-normality were investigated, and it was demonstrated that the Braden technique became very liberal, whereas the other statistics tended to be some what liberal at the .01 significance level and reasona bly robust at the .05 level. Index terms: Disatten uated measures, Measurement error, Monte carlo simulation, Non-normality, Small-sample statistics, Type I error and power rates.
Get full access to this article
View all access options for this article.
References
1.
Blair, R.C., & Higgins, J.J. (1985). Comparison of the power of the paired samples t test to that of Wilcoxon's signed-ranks test under various population shapes . Psychological Bulletin,97, 119-128.
2.
Bobko, P., & Rieck, A. (1980). Large sample estimators for standard errors of functions of correlation coefficients. Applied Psychological Measurement, 4, 385-398.
3.
Braden, J.P. (1986). Testing correlations between similar measures in small samples. Educational and Psychological Measurement , 46, 143-148.
4.
Cureton, E.E. (1936). On certain estimated correlation functions and their standard errors. Journal of Experimental Education, 4, 252-264.
5.
D'Agostino, R.B. (1971). An omnibus test of normality for moderate and large size samples. Biometrika, 58, 341-347.
6.
Dixon, W.J., & Massey, F.J. (1969). Introduction to statistical analysis (3rd ed.). New York: McGraw-Hill.
DuBois, P.H. (1965). An introduction to psychological statistics. New York: Harper & Row.
9.
Edgell, S.E., & Noon, S.M. (1984). Effect of violation of normality on the t test of the correlation coefficient. Psychological Bulletin,95, 567-583.
10.
Forsyth, R.A., & Feldt, L.S. (1969). An investigation of empirical sampling distributions of correlation coefficients corrected for attenuation. Educational and Psychological Measurement, 29, 61-71.
11.
Forsyth, R.A., & Feldt, L.S. (1970). Some theoretical and empirical results related to McNemar's test that the population correlation coefficient corrected for attenuation equals 1.0. American Educational Research Journal , 7, 197-207.
12.
Games, P.A. (1983). Curvilinear transformations of the dependent variable . Psychological Bulletin,93, 382-387.
13.
Games, P.A. (1984). Data transformation, power, and skew: A rebuttal to Levine and Dunlap. Psychological Bulletin,95, 345-347.
14.
Ghiselli, E.E., Campbell, J.P., & Zedeck, S. (1981). Measurement theory for the behavioral sciences . San Francisco: Freeman.
15.
Havlicek, L.L., & Peterson, N.L. (1977). Effect of the violation of assumptions upon significance levels of the Pearson r. Psychological Bulletin,84, 373-377.
16.
Jöreskog, K.G. (1971). Statistical analysis of sets of congeneric tests . Psychometrika, 36, 109-133.
17.
Kelley, T.L. (1947). Fundamentals of statistics. Cam-bridge MA: Harvard University Press.
18.
Knapp, T.H., & Swoyer, V.H. (1967). Some empirical results concerning the power of Bartlett's test of the significance of a correlation matrix. American Education Research Journal, 4, 13-17.
19.
Kristof, W. (1973). Testing a linear relation between true scores of two measures. Psychometrika, 38, 101-111.
20.
Lehman, R.S. (1977). Computer simulation and modeling: An introduction . Hillsdale NJ: Erlbaum.
21.
Levine, D.W., & Dunlap, W.P. (1982). Power of the F test with skewed data: Should one transform or not?Psychological Bulletin,92, 272-280.
22.
Levine, D.W., & Dunlap, W.P. (1983). Data transformation, power, and skew: A rejoinder to Games. Psychological Bulletin,93, 596-599.
23.
Lord, F.M. ( 1957). A significance test for the hypothesis that two variables measure the same trait except for errors of measurement. Psychometrika, 22, 207-220.
24.
Lord, F.M. (1973). Testing if two measuring procedures measure the same dimension. Psychological Bulletin,79, 71-72.
25.
McNemar, Q. (1958). Attenuation and interaction. Psychometrika, 23, 259-266.
26.
Nunnally, J.C. (1978). Psychometric theory (2nd ed.). New York: McGraw-Hill.
27.
Rae, G. ( 1982). A Monte Carlo comparison of small sample procedures for testing the hypothesis that two variables measure the same trait except for errors of measurement. British Journal of Mathematical and Statistical Psychology, 35, 228-232.
28.
Rasmussen, J.L. (1985). Data transformation maximizing homoscedasticity and within-group normality. Behavior Research Methods, Instruments, and Computers, 17, 411-412.
29.
Shen, E. (1924). The standard error of certain estimated coefficients of correlation. Journal of Educational Psychology, 15, 462-465.
30.
Villegas, G. (1964). Confidence region for a linear relation. Annals of Mathematical Statistics, 35, 780-788.
31.
Zwick, R. (1986). Rank and normal scores alternatives to Hotelling's T. Multivariate Behavioral Research, 21, 169-186.
