The present article provides an overview of two result interpretation aids commonly used in group experimental design research: (a) statistical significance testing and (b) practical significance (i.e., effect sizes). The limitations of each are described and an alternative index is discussed: clinical significance. Clinical significance, which focuses on meaningful change, differs from both statistical and practical significance. Basic concepts associated with clinical significance, including reliable change estimates and criteria for determining clinical significance, are addressed. A heuristic example is presented to demonstrate various levels of clinical significance.
Get full access to this article
View all access options for this article.
References
1.
Abramowitz, J.S. (1998). Does cognitive-behavioral therapy cure obsessive-compulsive disorder ? A meta-analytic evaluation of clinical significance. Behavior Therapy, 29, 339-355.
2.
Achenbach, T.M. ( 1991). Manual for the Child Behavior Checklist. Burlington, VT: University of Vermont Department of Psychiatry.
3.
American Psychological Association. (2001). Publication manual of the American Psychological Association (5th ed.). Washington, DC: Author.
4.
Ankuta, G.Y. & Abeles, N. (1993). Client satisfaction, clinical significance, and meaningful change in psychotherapy. Professional Psychology: Research and Practice, 24, 70-74.
5.
Bagozzi, R.P., Fornell, C., & Larcker, D.F. (1981). Canonical correlation analysis as a special case of a structural relations model. Multivariate Behavioral Research , 16, 437-454.
6.
Beck, A.T., Steer, R.A., & Brown, G.K. (1996). Manual for the Beck Depression Inventory-II. San Antonio, TX: Psychological Corporation .
7.
Berger, J.O., & Sellke, T. (1987). Testing a point null hypothesis: The irreconcilability of p values and evidence. Journal of the American Statistical Association, 82, 112-122.
8.
Beutler, L.E., & Harwood, T.M. (2000). Prescriptive psychotherapy: A practical guide to systematic treatment selection. New York: Oxford University Press.
9.
Boring, E.G. (1919). Mathematical vs scientific importance. Psychological Bulletin, 16, 335-338.
10.
Carver, R. (1978). The case against statistical significance testing . Harvard Educational Review, 48, 378-399.
11.
Cohen, J. (1968). Multiple regression as a general data-analytic system. Psychological Bulletin, 70, 426-423.
12.
Cohen, J. (1969). Statistical power analysis for the behavioral sciences. New York: Academic Press .
13.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.
14.
Cohen, J. (1994). The earth is round (p < .05). American Psychologist, 49, 997-1003.
15.
Cross T.P., McDonald E., & Lyons, H. (1997). Evaluating the outcome of children's mental health services (2nd ed.). Boston: Judge Baker Children's Center.
16.
Fidler, F., & Thompson, B. (2001). Computing correct confidence intervals for ANOVA fixed-and random-effects effect sizes. Educational and Psychological Measurement, 61, 575-604.
17.
Fish, L. (1988). Why multivariate methods are usually vital. Measurement and Evaluation in Counseling and Development, 21, 130-137.
18.
Follette, W.C., & Callaghan, G.M. (1996). The importance of the principle of clinical significance-defining significant to whom and for what purpose: A response to Tingey, Lambert, Burlingame, and Hansen. Psychotherapy Research, 6, 133-143.
19.
Gelso, C.J. (1979). Research in counseling: Methodological and professional issues. The Counseling Psychologist, 8, 7-35.
20.
Hedges, L.V. & Olkin, I. (1985). Statistical methods for meta-analysis. New York: Academic Press.
21.
Heppner, P.P., Kivlighan, D.M. & Wampold, B.E., (1999). Research design in counseling (2nd ed.). Belmont, CA: Brooks Cole .
22.
Hill, C.R., & Thompson, B. (2004). Computing and interpreting effect sizes. In J.C. Smart (Ed.), Higher education: Handbook of theory and research (Vol. 19, pp. 175-196). New York: Kluwer.
23.
Howard, K.I., Kopta, S.M., Krause, M.S., & Orlinsky, D.E. (1986). The dose-response relationship in psychotherapy. American Psychologist, 49, 151-164.
24.
Howard, K.I., Moras, K., Brill, P.L., Martinovich, Z. & Lutz, W. (1996). Evaluation of psychotherapy: Efficacy, effectiveness, and patient progress. American Psychologist, 51, 1059-1064.
25.
Hsu, L.M. (1996). On the identification of clinically significant client changes: Reinterpretation of Jacobson's cut scores. Journal of Psychopathology and Behavioral Assessment, 18, 371-385.
26.
Huberty, C.J. (1993). Historical origins of statistical testing practices: The treatment of Fischer versus Neyman-Pearson views in textbooks. Journal of Experimental Education, 61, 317-333.
27.
Jacobson, N.S., & Revenstorf, D. (1988). Statistics for assessing the clinical significance of psychotherapy techniques: Issues, problems, and new developments. Behavioral Assessment, 10, 133-145.
28.
Jacobson, N.S., Follette, W.C., & Revenstorf, D. (1984). Psychotherapy outcome research: Methods for reporting variability and evaluating clinical significance . Behavior Therapy, 15, 336-352.
29.
Jacobson, N.S., Follette, W.C., & Revenstorf, D. (1986). Toward a standard definition of clinically significant change. Behavior Therapy, 17, 308-311.
30.
Jacobson, N.S., Roberts, L.J., Berns, S.B., & McGlinchey, J.B. (1999). Methods for defining and determining the clinical significance of treatment effects: Description, application, and alternatives. Journal of Consulting and Clinical Psychology , 67, 300-307.
31.
Kazdin, A.E. (1977). Assessing the clinical or applied importance of behavior change through social validation. Behavior Modifications , 1, 427-452.
32.
Kazdin, A.E. (1999). The meanings and measurement of clinical significance . Journal of Consulting and Clinical Psychology, 67, 332-339.
33.
Kazdin, A.E. (2001). Almost clinically significant (p < .01): Current measures may only approach clinical significance . Clinical Psychology: Science and Practice, 8, 455-462.
34.
Kazdin, A.E. (2003). Clinical significance: measuring whether interventions make a difference. In A. E. Kazdin (Ed.), Methodological issues & strategies in clinical research (pp. 672-691). Washington, DC: American Psychological Association .
35.
Kazdin, A.E., & Wilson, G.T. (1978). Evaluation of behavior therapy: Issues, evidence, and research strategies. Cambridge, MA: Ballinger.
36.
Kendall, P.C., & Grove, W.M. (1988). Normative comparisons in therapy outcome. Behavioral Assessment, 10, 147-158
37.
Kendall, P.C., Marrs-Garcia, A., Nath, S.R., & Sheldrick, R.C. (1999). Normative comparisons for the evaluation of clinical significance. Journal of Consulting and Clinical Psychology, 67, 285-299.
38.
Kirk, R. (1996). Practical significance: A concept whose time has come. Educational and Psychological Measurement, 56, 746-759.
39.
Kirk, R.E. (2003). The importance of effect magnitude. In S. F Davis (Ed.), Handbook of research methods in experimental psychology (pp. 83-105). Oxford, United Kingdom: Black-well.
40.
Knapp, T.R. (1978). Canonical correlation analysis: A general parametric significance testing system. Psychological Bulletin,85, 410-416.
41.
Lambert, M.J. & Bergin, A.E. (1994). The effectiveness of psychotherapy. In A. E. Bergin & S. L. Garfield (Eds.), Handbook of psychotherapy and behavior change (4th ed., pp. 143-189). New York: Wiley.
42.
Lambert, M.J., Bergin, A.E. & Collins, J.L. ( 1977). Therapist-induced deterioration in psychotherapy . In A. S. Gurman & A. M. Razin (Eds.), Effective psychotherapy: A handbook of research (pp. 452-481). New York: Pergamon Press.
43.
Lunnen, K.M., & Ogles, B.M. (1998). A multi-perspective, multivariable evaluation of reliable change. Journal of Consulting and Clinical Psychology , 66, 400--410.
44.
Martinovich, Z., Saunders, S., & Howard, K.I. (1996). Some comments on "Assessing clinical significance ." Psychotherapy Research, 6, 124-132.
45.
Mittag, K.C., & Thompson, B. (2000). A national survey of AERA members' perceptions of statistical significance tests and other statistical issues. Educational Researcher, 29(4), 21-27.
46.
Mohr, D.C. (1995). Negative outcome in psychotherapy: A critical review. Clinical Psychology-Science & Practice, 2, 1-27.
47.
Moon, T.R., Callahan, C.M., & Tomlinson, C.A. , (2003, April 28). Effects of state testing programs on elementary schools with high concentrations of student poverty-good news or bad news?Current Issues in Education [On-line], 6(8). Retrieved from http://cie.ed.asu.edu/volume6/number8/
48.
Neitzel, M.T., & Trull, T.J. (1988). Meta-analytic approaches to social comparisons: A method for measuring clinical significance. Behavior Assessment , 10, 159-169.
Ogles, B.M., Lunnen, K.M., & Bonesteel, K. (2001). Clinical significance: History, application and current practice. Clinical Psychology Review, 21, 421-446.
51.
Pederson, S. (2003). Effect sizes and "what if" analyses as supplements to statistical significance tests. Journal of Early Intervention , 25, 310-319.
Rosenthal, R. (1979). The "file drawer problem" and tolerance for null results. Psychological Bulletin, 86, 638-641.
54.
Rosnow, R.L., & Rosenthal, R. (1989) Statistical procedures and the justification of knowledge in psychological science. American Psychologist , 44, 1276-1284.
55.
Schmidt, F.L. & Hunter, J.E. (1997). Eight common but false objections to the discontinuation of significance testing in the analysis of research data. In L.L. Harlow, S.A & Muliak, & J.H. Steiger (Eds.), What if there were no significance tests? (pp. 37-64). Mahwah, NJ: Erlbaum.
56.
Shavelson, R.J., & Webb, N.M. (1991). Generalizability theory: A primer. Newbury Park, NJ: Sage.
57.
Shaver, J.P. (1991). Quantitative reviewing of research. In J.P. Shaver (Ed.), Handbook of research on social studies teaching and learning (pp. 83-97). New York: Macmillan
58.
Sheldrick, R.C., Kendall, P.C., & Heimberg, R.G. (2001). The clinical significance of treatments: A comparison of three treatments for conduct disordered children . Clinical Psychology, 8, 418-430.
59.
Snyder, P. (2000). Guidelines for reporting results of group quantitative investigations. Journal of Early Intervention, 23, 145-150.
60.
Snyder P. & Lawson, S. (1993). Evaluating results using corrected and uncorrected effect size estimates. Journal of Experimental Education, 64, 334-339.
61.
Speer D.C. (1998). Mental health outcome evaluation. San Diego: Academic Press.
62.
Speer, D.C. & Greenbaum, P. ( 1995). A comparison of five methods for computing significant individual client change and measurement rates: An individual growth curve approach. Journal of Consulting and Clinical Psychology, 63, 1044-1048.
63.
Strupp, H.H. & Hadley, S.W. (1977). A tripartite model of mental health and therapeutic outcome: With special reference to negative effects in psychotherapy. American Psychologist, 32, 187-196.
64.
Tyron, W.W., & Bernstein, D. (2003). Understanding measurement. In J. C. Thomas & M. Hersen (Eds.), Understanding research in clinical and counseling psychology (pp. 27-68). Mahwah, NJ: Lawrence Erlbaum.
65.
Thompson, B. (1993). The use of statistical significance tests in research: Bootstrap and other alternatives. Journal of Experimental Education, 61, 361-377.
66.
Thompson, B. (1996). AERA editorial policies regarding statistical significance testing: Three suggested reforms. Educational Researcher, 25(2), 26-30.
67.
Thompson, B. (1999). Journal editorial policies regarding statistical significance tests: Heat is to fire as p is to importance. Educational Psychology Review, 11, 157-169.
68.
Thompson, B. (2002). "Statistical", "practical", and "clinical": How many kinds of significance do counselors need to consider?Journal of Counseling and Development, 80, 64-71.
69.
Thompson, B. (Ed.). (2003). Score reliability: Contemporary thinking on reliability issues. Newbury Park, CA: Sage.
70.
Thompson, B. (in press). Research synthesis: Effect sizes. In J. Green, G. Camilli , & P. Elmore (Eds.), Complementary methods for research in education (3rd ed). Washington, DC: American Educational Research Association.
71.
Thompson, B., & Vacha-Haase, T. (2000). Psycho-metrics is datametrics: The test is not reliable. Educational & Psychological Measurement. 60, 174-195.
Trull, T.J., Neitzel, M.T., & Main, A. (1988). The use of meta-analysis to assess clinical significance of behavior therapy for agoraphobia. Behavior Therapy, 19, 527-538.
74.
Vacha-Haase, T., Henson, R.K., & Caruso, J. (2002). Reliability generalization: Moving toward improved understanding and use of score reliability. Educational and Psychological Measurement, 62, 562-569.
Weisz, J.R., Donenberg, G.R., Han, S.S., & Weiss, B. (1995). Bridging the gap between laboratory and clinic in child and adolescent psychotherapy. Journal of Clinical and Consulting Psychology, 63, 688-701.
77.
Wilkinson, L., & APA Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594-604. (Reprint available through the APA Home page: www.apa.org/journals/amp/amp548594.html )
78.
Wiger, D.E., & Solberg, K.B. (2001). Tracking mental health outcomes: A therapist's guide to measuring client progress, analyzing data, and improving your practice . New York: Wiley.
79.
Wise, E.A. (2004). Methods for analyzing psychotherapy outcomes: A review of clinical significance, reliable change, and recommendations for future directions. Journal of Personality Assessment, 82, 50-59.
80.
Zuckerman, M., Hodgins, H.S.Zuckerman, A. & Rosenthal, R. (1993). Contemporary issues in the analysis of data: A survey of 551 psychologists. Psychological Science, 4, 49-53.
