Objective: Presentation of effect sizes that can be interpreted in terms of clinical or practical significance is currently urged whenever statistical significance (a ‘p-value’) is reported in research journals. However, which effect size and how to interpret it are not yet clearly delineated. The present focus is on effect sizes indicating strength of correlation, that is, effect sizes that describe the strength of monotonic association between two random variables X and Y in a population.
Methods: A logical structure of measures of association is traced, showing the interrelationships among the many measures of association. Advantages and disadvantages of each are discussed.
Conclusions: Suggestions are made for the future use of measures of association in research to facilitate considerations of clinical significance, emphasizing distribution-free effect sizes such as the Spearman correlation coefficient and Kendall’s coefficient of concordance for ordinal versus ordinal associations, weighted and intraclass kappa for binary versus binary associations and risk difference (RD) for binary versus ordinal association.
Get full access to this article
View all access options for this article.
References
1.
Cohen J.The earth is round (p < 0.05) . American Psychologist1995; 49: 997-1003 .
2.
Dar R , Serlin RC , Omer H.Misuse of statistical tests in three decades of psychotherapy research . Journal of Consulting and Clinical Research1994; 62: 75-82 .
3.
Harris RJ.Significance tests have their place . Psychological Science1997; 8(1): 8-11 .
4.
Hunter JE.Needed: a ban on the significance test . Psychological Science1997; 8(1): 3-7 .
5.
Schmidt FL.Statistical significance testing and cumulative knowledge in psychology: implications for training of researchers . Psychological Methods1996; 1(2): 115-129 .
6.
Shrout PE.Should significance tests be banned? Introduction to a special section exploring the pros and cons . Psychological Science1997; 8(1): 1-2 .
7.
Wilkinson L.The task force on statistical inference. Statistical methods in psychology journals: guidelines and explanations . American Psychologist1999; 54: 594-604 .
8.
Rennie D.How to report randomized controlled trials: the CONSORT statement . Journal of the American Medical Association1996; 276(8): 649-649 .
9.
Altman DG , Schulz KF , Hoher D , Egger M , Davidoff F , Elbourne D , Gotzsche PC , Lang T , Consort Group . The revised CONSORT statement for reporting randomized trials: explanation and elaboration . Annals of Internal Medicine2001; 134(8): 663-694 .
10.
Begg C , Cho M , Eastwood S , Horton R , Homer D , Olkin I , Pitkin R , Rennie D , Schultz KF , Simel D , Stroup DF.Improving the quality of reporting of randomized controlled trials: the CONSORT statement . Journal of the American Medical Association1999; 276: 637-639 .
11.
Grissom RJ , Kim JJ.Effect sizes for research. Lawrence Erlbaum Associates , 2005.
12.
Kraemer HC , Morgan GA , Leech NL , Gilner JA , Vaske JJ , Harmon RJ.Measures of clinical significance . Journal of the American Academy of Child and Adolescent Psychiatry2003; 42(12): 1524-1529 .
13.
Kraemer HC , Kupfer DJ.Size of treatment effects and their importance to clinical research and practice . Biological Psychiatry2006; 59(11): 990-996 .
14.
Fisher RA.Frequency distribution of the values of the correlation coefficient in samples from an indefinitely large population . Biometrika1915; 10: 507-521 .
15.
Fisher RA.On the ‘probable error’ of a coefficient of correlation deduced from a small sample . Metron1921; 1: 1-32 .
16.
Pearson K.On a new method of determining the correlation between a measured character A and a character B, of which only the percentage of cases wherin B exceeds (or falls short of) a given intensity is recorded for each grade of A . Biometrika1909; 7: 96-105 .
17.
Kraemer HC.Reconsidering the odds ratio as a measure of 2 × 2 association in a population . Statistics in Medicine2003; 23(2): 257-270 .
18.
Sackett DL.Down with odds ratios!Evidence-Based Medicine1996; 1: 164-166 .
Last JM.A dictionary of epidemiology. Oxford University Press , 1995.
22.
Krantz DH.The null hypothesis testing controversy in psychology . Journal of the American Statistical Association1999; 44(448): 1372-1381 .
23.
Meehl PE.Theory testing in psychology and physics: a methodological paradox . Philosophy of Science1967; 34: 103-115 .
24.
Jones LV , Tukey JW.A sensible formulation of the significance test . Psychological Methods2000; 5(4): 411-414 .
25.
Efron B.Bootstrap methods: another look at the jackknife . The Annals of Statistics1979; 7: 1-26 .
26.
Efron B , Gong G.A leisurely look at the bootstrap, the jackknife, and cross-validation . The American Statistician1983; 37: 36-48 .
27.
Efron B.Bootstrap confidence intervals: good or bad?Psychological Bulletin1988; 104(2): 293-296 .
28.
David FN.Tables of the ordinates and probability integral of the distribution of the correlation coefficient in small samples. Cambridge University Press , 1938.
29.
Fisher RA.Statistical methods for research workers, second edition. Oliver & Boyd , 1928.
30.
Edgell SE , Noon SM.Effect of violation of normality on the t-test of the correlation . Psychological Bulletin1984; 95: 576-583 .
31.
Kraemer HC.Robustness of the distribution theory of the product-moment correlation coefficient . Journal of Educational Statistics1980; 5(2): 115-128 .
32.
Kowalski CJ.On the effects on non-normality on the distribution of the sample correlation coefficient . The Journal of the Royal Statistical Society1972; 21: 1-12 .
33.
Bartko JJ.The intraclass correlation coefficient as a measure of reliability . Psychological Reports1966; 19: 3-11 .
Shrout PE , Fleiss JL.Intraclass correlations: uses in assessing rater reliability . Psychological Bulletin1979; 86: 420-428 .
36.
Algina J.Comment on Bartko’s ‘On various intraclass correlation reliability coefficients’ . Psychological Bulletin1978; 85(1): 135-138 .
37.
Tate RF.Correlation between a discrete and a continuous variable. Point-biserial correlation . Annals of Mathematical Statistics1954; 25: 603-607 .
38.
Tate RF.Applications of correlation models for biserial data . American Statistical Association Journal1955; 50: 1078-1095 .
39.
Olkin I , Tate RF.Multivariate correlation models with mixed discrete and continuous . Annals of Mathematical Statistics1961; 32: 448-465 .
40.
Mayer LS.Estimating a correlation coefficient when one variable is not directly observed . Journal of the American Statistical Association1973; 68: 420-421 .
41.
Olsson U , Drasgow F , Dorans NJ.The polyserial correlation coefficient . Psychometrika1982; 47: 337-347 .
42.
Lee S , Poon W.Maximum likelihood estimation of polyserial correlations . Psychometrika1986; 51: 113-121 .
43.
Bedrick EJ , Breslin FC.Estimating the polyserial correlation coefficient . Psychometrika1996; 61(3): 427-443 .
44.
Pearson K.On the correlation of characters not quantitatively measurable . Philosophical Transactions of the Royal Society of London1901; 195A: 1-47 .
45.
Pearson K.On the probable error of a coefficient of correlation as found from a fourfold table . Biometrika1913; 9: 22-27 .
46.
Kraemer HC.What is the ‘right’ statistical measure of twin concordance (or diagnostic reliability and validity)?Archives of General Psychiatry1997; 54: 1121-1124 .
47.
Lyons MJ , True WR , Eisen SA , Goldberg J , Meyer JM , Faraone SV , Eaves LF , Tsuang MT.Differential heritability of adult and juvenile antisocial traits . Archives of General Psychiatry1995; 52: 906-915 .
48.
Lyons MF , Faraone SV , Tsuang MT , Goldberg J , Ramakrishnan V , Eaves LJ , Meyer JM , True WR , Eisen SA.Another view on the ‘right’ statistical measure of twin concordance . Archives of General Psychiatry1997; 54: 1126-1128 .
49.
Cohen J.Statistical power analysis for the behavioral sciences. Lawrence Erlbaum Associates , 1988.
50.
Kraemer HC.A Simple effect size indicator for two-group comparisons?: a comment on requivalent . Psychological Methods2006; 10(4): 413-419 .
51.
Rosenthal R , Rubin DB.requivalent: a simple effect size indicator . Psychological Methods2003; 8(4): 492-496 .
52.
Spearman C.The proof and measurement of association between two things . American Journal of Psychology1904; 15: 72-101 .
53.
Spearman C.A footrule for measuring correlation . British Journal of Psychology1906; 2: 89-108 .
54.
Fieller EC , Hartley HO , Pearson ES.Tests for rank correlation coefficients. I . Biometrika1957; 44: 470-481 .
55.
Fieller EC , Pearson ES.Tests for rank correlation coefficients. II . Biometrika1961; 48: 29-40 .
56.
Kraemer HC.On estimation and hypothesis testing problems for correlation coefficients . Psychometrika1975; 40(4): 473-485 .
57.
Zar JH.Significance testing of the Spearman rank correlation . Journal of the American Statistical Association1972; 67: 578-585 .
Kendall MG.Rank correlation methods. Hafner Publishing Company , 1962.
60.
Kendall M , Gibbons JD.Rank correlation methods, fifth edition. Oxford University Press , 1990.
61.
Gibbons JD.Nonparametric statistics: an introduction. Sage Publications , 1993.
62.
Kraemer HC.The small sample non-null properties of Kendall’s coefficient of concordance for normal populations . Journal of the American Statistical Association1976; 71: 608-613 .
63.
Kraemer HC.A measure of 2 × 2 association with stable variance and approximately normal small sample distribution: planning cost-effective studies . Biometrics1985; 42: 359-370 .
64.
Fleiss JL.Statistical methods for rates and proportions. John Wiley & Sons , 1981.
65.
Kraemer HC , Periyakoil VS , Noda A.Tutorial in biostatistics: kappa coefficients in medical research . Statistics in Medicine2002; 21: 2109-2129 .
66.
Kraemer HC.Measurement of reliability for categorical data in medical research . Statistical Methods in Medical Research1992; 1: 183-199 .
67.
Kraemer HC.Ramifications of a population model for k as a coefficient of reliability . Psychometrika1979; 44(4): 461-472 .
68.
Kendall MG.A new measure of rank correlation . Biometrika1938; 30: 91-93 .
69.
Arndt S , Turvey C , Andreason NC.Correlating and predicting psychiatric symptom ratings: Spearman’s r versus Kendall’s tau correlation . Journal of Psychiatric Research1999; 33(2): 97-104 .
70.
Brogden HE.A new coefficient: Application to biserial correlation and to estimation of selective efficiency . Psychometrika1949; 14: 169-182 .
71.
Lord FM.Biserial estimates of correlation . Psychometrika1963; 28: 81-85 .
Bedrick EJ.On the large sample distributions of modified sample biserial . Psychometrika1990; 55: 217-228 .
74.
Cohen J.Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit . Psychological Bulletin1968; 70: 213-229 .
75.
Bloch DA , Kraemer HC.2 × 2 kappa coefficients: measures of agreement or association . Biometrics1989; 45: 269-287 .
76.
Kraemer HC , Kazdin AE , Offord DR , Kessler RC , Jensen PS , Kupfer DJ.Measuring the potency of a risk factor for clinical or policy significance . Psychological Methods1999; 4(3): 257-271 .
77.
Kraemer HC.Evaluating medical tests: objective and quantitative guidelines. Sage Publications , 1992.
78.
McNeil BJ , Keeler E , Adelstein SJ.Primer on certain elements of medical decision making . The New England Journal of Medicine1975; 293: 211-215 .
79.
Swets JA , Pickett RM.Evaluation of diagnostic systems: methods from signal detection theory. Academic Press , 1982.
80.
Brownie C.Estimating Pr(X < Y)in categorized data using ‘ROC’ analysis . Biometrics1988; 44: 615-621 .
81.
Grissom RJ.Probability of the superior outcome of one treatment over another . Journal of Applied Psychology1994; 79: 314-316 .
82.
Hanley JA , McNeil BJ.The meaning and use of the area under a receiver operating characteristic (ROC) curve . Radiology1982; 143: 29-36 .
83.
McGraw KO , Wong SP.A common language effect size statistic . Psychological Bulletin1992; 111: 361-365 .
84.
Acion L , Peterson JJ , Temple S , Arndt S.Probabilistic index: an intuitive non-parametric approach to measuring the size of treatment effects . Statistics in Medicine2006; 25(4): 591-602 .
85.
Hsu LM.Biases of success rate differences shown in binomial effect size displays . Psychological Bulletin2004; 9(2): 183-197 .
86.
Altman DG , Andersen K.Calculating the number needed to treat for trials where the outcome is time to an event . British Medical Journal1999; 319: 1492-1495 .
87.
Cook RJ , Sackett DL.The number needed to treat: a clinically useful measure of treatment effect . British Medical Journal1995; 310: 452-454 .
88.
Altman DG.Confidence intervals for the number needed to treat . British Medical Journal1998; 317(7168): 1309-1312 .
89.
Newcombe RG.Confidence intervals for the number needed to treat-absolute risk reduction is less likely to be misunderstood . British Medical Journal1999; 318: 1765-1765 .
90.
Duncan BW , Olkin I.Bias of estimates of the number needed to treat . Statistics in Medicine2005; 24: 1837-1848 .
91.
Efron B.Computer-intensive methods in statistical regression. Technical Report #174, Division of Biostatistics, Stanford University , April 1995.
92.
Brown MB , Benedetti JK.Sampling behavior of tests for correlation in two-way contigency . Journal of the American Statistical Association1977; 72: 309-315 .
93.
Rosenthal I.Distribution of the sample version of the measure of association, Gamma . Journal of the American Statistical Association1966; 61: 440-453 .
94.
Ruben H.Non-central chi-square and gamma revisited . Communications in Statistics1974; 3: 607-633 .
95.
Newcombe RG.A deficiency of the odds ratio as a measure of effect size . Statistics in Medicine, inpress.
96.
Kirk DB.On the numerical approximation of the bivariate normal (tetrachoric) correlation coefficient . Psychometrika1973; 38(2): 259-268 .
97.
Cornfield J.A statistical problem arising from retrospective studies . In Neyman J , ed. Proceedings of the Third Berkeley Symposium. University of California Press, 1956: 135-135 .
98.
Cornfield J.A method of estimating comparative rates from clinical data. Applications to cancer of the lung, breast and cervix . Journal of the National Cancer Institute1951; 11: 1269-1275 .
99.
Mantel N , Haenszel W.Statistical aspects of the analysis of data from retrospective studies of disease . Journal of the National Cancer Institute1959; 22: 719-748 .
