For decades, the most common method used to detect differences between groups in comparative studies has been the use of statistical significance tests. Unfortunately, many current evaluation students are still being taught the use of such significance tests without being warned about their limitations and the need to supplement them with other estimates of treatment effects necessary to interpret between-group differences correctly. In this paper I describe briefly the problem and what new evaluators may do to better understand this issue and improve their evaluation practice, including references to sources where they can delve more deeply into this subject than is possible in this brief paper.
Get full access to this article
View all access options for this article.
References
1.
Algina, J., & Cracker, L. (1986). Introduction to modem and classical test theory. Orlando, FL: Harcourt, Brace, Jonanovich.
2.
American Psychological Association. (1994). Publication manual of the American Psychological Association (4th ed.). Washington, DC: Author.
3.
Carter, D. S. (1979). Comparison of different shrinkage formulas for estimating population multiple correlation coefficients. Educational and Psychological Measurement, 39, 261-266.
4.
Cohen, J. A. (1988). Statistical power analysis for the behavioralsciences (2nd ed.). Hillsdale, NJ: Erlbaum.
5.
Fisher, R. A. (1932). Statistical methods for research workers. Edinburgh: Oliver and Boyd.
6.
Fowler, R. J. (1985). Point estimates and confidence intervals in measures of association. Psychological Bulletin, 98, 160-165.
7.
Glass, G.V., McGaw, B, & Smith, M.L. (1981). Meta-analysisin social research. Beverly Hills, CA: Sage.
8.
Herzberg, P.A. (1969). The parameters of cross-validation. Psychometrika Monograph Supplement, 16, 1-67.
9.
Howell, D. C. (1982). Statistical Methods for Psychology. Boston: Duxbury.
10.
Jackson, S., & Brashers, D.E. (1994). Random factors in ANOVA (Sage University Paper series on Quantitative Applications in the Social Sciences, series no. 07-098). Thousand Oaks, CA: Sage.
11.
Johnson, D. W., Johnson, R. T., Dudley, B., & Magnuson, D. (1995). Training elementary school students to manage conflict. The Journal of Social Psychology, 135(6), 673-686.
12.
Kelley, T. L. (1935). An unbiased correlation ration measure. Proceedings of the National Academy of Sciences, 21, 554-559.
13.
Knapp, T. R. (1978). Canonical correlation: A general parametric significance testing system. Psychological Bulletin, 85, 410-416.
14.
Linden, M., Habib, T., & Radojevic, N. (1996). Using biofeedback procedures to regulate ADHD-related disruptive behavior. Biofeedback and Self-Regulation, 21(1), 35-49.
15.
Lord, F.M. (1950). Efficiency of prediction when a regression equation from one sample is used in a new sample. (Research Bulletin 50-110). Princeton, NJ: Educational Testing Services.
16.
Maxwell, S.E., & Delaney, H.D. (1990). Designing experiments and analyzing data: A model comparison perspective. Belmont, CA: Wadsworth.
17.
Maxwell, S. E., Camp, C. J., & Avery, R. D. (1981). Measures of strength of association: A comparative examination. Journal of Applied Psychology, 66, 525-534.
McWilliam, R. A., & Bailey, D. B. (1994). Predictors of service-delivery models in center-based early intervention. Exception Children, 61(1), 56-71.
20.
Meehl, P. E. (1978). Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology. Journal of Consulting and Clinical Psychology, 46, 806-834.
21.
Norisus, M. N. (1991). SPSS/PC+ Studentware. Chicago: SPSS Inc.
22.
Pedhazur, E.J. (1982). Multiple regression in behavioral science research. Fort Worth, TX: Holt, Rhinehart, and Winston
23.
Rosnow, R. L., & Rosenthal, R. (1989). Statistical procedures and the justification of knowledge in psychological science. American Psychologist, 44, 1276-1284.
24.
Serr, R., Lavay, B., Young, D., & Green, G. (1994) Dexterity and bench assembly work productivity in adults with mild mental retardation. Education and Training in Mental Retardation and Developmental Disabilities, 29 (2), 165-171.
25.
Shaver, J. P. (1993). What significance testing is, and what it is not. Journal of Experimental Education, 61, 293-316.
26.
Snyder, P., & Lawson, S. (1993). Evaluating results using corrected and uncorrected effect size estimates. Journal of Experimental Education, 61, 334-339.
27.
Stevens, J. (1992). Applied multivariate statistics for the social sciences (2nd ed.). Hillsdale, NJ: Earlbaum.
28.
Thompson, B. (1990). Finding a correction for the sampling error in multivariate measures of relationship: A Monte Carlo study. Educational and Psychological Measurement, 50, 15-31.
29.
Thompson, B. (1993). The use of statistical significance tests in research: Bootstrap and other alternatives. Journal of Experimental Education, 61, 361-377.
30.
Werry, R. J. (1931). A new formula for predicting the shrinkage of the coefficient of multiple correlation. Annals of Mathematical Statistics, 2, 440-457.
