This article describes a computer-assisted classroom demonstration illustrating the consequences of excluding nonsignificant findings from interpreting published literature. This demonstration, based on tenets of the Central Limit Theorem, simulates research interpretation when the full range of results are available compared with the subsample of significant results only. Results demonstrated that (a) exclusion of nonsignificant findings positively biases research interpretation and (b) smaller sample sizes are prone to greater bias when nonsignificant results are excluded from research interpretation.
Get full access to this article
View all access options for this article.
References
1.
BerlinJ. A.BeggC. B.LouisT. A. (1989). An assessment of publication bias using a sample of published clinical trials. Journal of the American Statistical Association, 84, 381–392.
2.
BozarthJ. D.RobertsR. R. (1972). Signifying significant significance, American Psychologist, 27, 774–775.
3.
CohenJ. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.
4.
CohenJ. (1994). The earth is round (p.05). American Psychologist, 49, 997–1003.
5.
DearK. B. G.BeggC. (1992). An approach for assessing publication bias prior to performing a meta-analysis. Statistical Science, 7, 237–245.
6.
EasterbrookP. J.BerlinJ. A.GopalanR.MatthewsD. R. (1991). Publication bias in clinical research. The Lancet, 337, 867–872.
7.
ForsytheG. E.MalcolmM. A.MolerC. B. (1977). Computer methods for mathematical computations.Englewood Cliffs, NJ: Prentice Hall.
8.
GrazianoA. M.RaulinM. L. (1989). Research methods: A process of inquiry.New York: Harper & Row.
9.
HedgesL. V. (1984). Estimation of effect size under nonrandom sampling: The effects of censoring studies yielding statistically insignificant mean differences. Journal of Educational Statistics, 9, 61–85.
10.
HopkinsK. D.GlassG. V. (1978). Basic statistics for the behavioral sciences.Englewood Cliffs, NJ: Prentice Hall.
11.
Matlab 4.0 [Computer program]. (1995). Natick, MA: Math Works.
12.
OakesM. (1986). Statistical inference: A commentary for the social and behavioral sciences.New York: Wiley.
13.
RayW. J. (1993). Methods toward a science of behavior (4th ed.). Monterey, CA: Brooks/Cole.
14.
ShaverJ. P. (1993). What statistical significance testing is, and what it is not. Journal of Experimental Education, 61, 293–316.
15.
Sigmastat 1.0 [Computer program]. (1994). San Rafael, CA: Jandel Scientific.
16.
SimesR. J. (1986). Publication bias: The case for an international registry of clinical trials. Journal of Clinical Oncology, 4, 1529–1541.
17.
Statistics Toolbox [Computer program]. (1996). Natick, MA: Math Works.
18.
SterlingT. C. (1959). Publication decisions and their possible effects on inferences drawn from tests of significance or vice versa. Journal of the American Statistical Association, 54, 30–34.
19.
TverskyA.KahnemanD. (1971). Belief in the law of small numbers. Psychological Bulletin, 76, 105–110.
