Researchers rarely mention statistical power in Teaching of Psychology teaching activity studies. Insufficiently powered tests promote uncertainty in the decision to accept or reject the tested null hypothesis and influence the interpretation of results. We analyzed the a priori power of statistical tests from 197 teaching activity effectiveness studies published from 1974 through 2006. We found that two thirds of the studies were powerful enough to detect only large effects. We compared observed sample sizes with expert recommendations and found that studies typically used sample sizes that were too small. We discuss limitations of underpowered statistical tests in evaluating teaching activity effectiveness and make design-related recommendations for improving power.
Get full access to this article
View all access options for this article.
References
1.
AsrafR. M.BrewerJ. K. (2004). Conducting tests of hypotheses: The need for an adequate sample size. The Australian Educational Researcher, 31, 79–94.
2.
BailarJ. C.IIIMostellerF. (1988). Guidelines for statistical reporting in medical journals: Amplifications and explanations. Annals of Internal Medicine, 108, 266–273.
3.
BorensteinM.RothsteinH.CohenJ. (2001). Power and precision [Computer software]. Englewood, NJ: Biostat, Inc.
4.
CohenJ. (1962). The statistical power of abnormal-social psychological research: A review. Journal of Abnormal and Social Psychology, 65, 145–153.
5.
CohenJ. (1969). Statistical power analysis for the behavioral sciences. New York: Academic.
6.
CohenJ. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.
7.
CohenJ. (1990). Things I have learned (so far). American Psychologist, 45, 1304–1312.
8.
CohenJ. (1992a). A power primer. Psychological Bulletin, 112, 155–159.
9.
CohenJ. (1992b). Statistical power analysis. Current Directions in Psychological Science, 1, 98–101.
10.
DalyJ. A.HexamerA. (1983). Statistical power in research in English education. Research in the Teaching of English, 17, 157–164.
11.
FaulF.ErdfelderE.LangA.-G.BuchnerA. (2007). G Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39, 175–191.
12.
HallahanM.RosenthalR. (1996). Statistical power: Concepts, procedures, and applications. Behaviour Research and Therapy, 34, 489–499.
13.
HarlowL. L.BurkholderG. J.MorrowJ. A. (2006). Engaging students in learning: An application with quantitative psychology. Teaching of Psychology, 33, 231–235.
14.
KraemerH. C.MintzJ.NodaA.TinklenbergJ.YesavageJ. A. (2006). Caution regarding the use of pilot studies to guide power calculations for study proposals. Archives of General Psychiatry, 63, 484–489.
15.
LakinJ. L.WichmanA. L. (2005). Applying social psychological concepts outside the classroom. Teaching of Psychology, 32, 110–113.
16.
LoSchiavoF. M.RobertsK. L. (2005). Testing pseudo-scientific claims in research methods courses. Teaching of Psychology, 32, 177–180.
17.
MadiganR.BrosamerJ. (1990). Improving the writing skills of students in introductory psychology. Teaching of Psychology, 17, 27–30.
18.
OsborneJ. W.ChristensenW. R.IIGunterJ. S. (2001, April). Educational psychology from a statistician's perspective: A review of the quantitative quality of our field. Paper presented at the 2001 meeting of the American Educational Research Association. Retrieved May 15, 2006, from http://www4.ncsu.edu/jwosbor2/otherfiles/MyResearch/POWER.pdf.
19.
PenickJ. E.BrewerJ. K. (1972). The power of statistical tests in science teaching research. Journal of Research in Science Teaching, 9, 377–379.
20.
RosnowR. L.RosenthalR. (1989). Statistical procedures and the justification of knowledge in psychological science. American Psychologist, 44, 1276–1284.
21.
RossiJ. S. (1990). Statistical power of psychological research: What have we gained in 20 years. Journal of Consulting and Clinical Psychology, 58, 646–656.
22.
SagarinB. J.Lawler-SagarinK. A. (2005). Critically evaluating competing theories: An exercise based on the Kitty Genovese murder. Teaching of Psychology, 32, 167–169.
23.
SedlmeierP.GigerenzerG. (1989). Do studies of statistical power have an effect on the power of studies. Psychological Bulletin, 105, 309–316.
24.
ThomasJ. H.McDanielC. R. (2004). Effectiveness of a required course in career planning for psychology majors. Teaching of Psychology, 31, 22–27.
25.
TomchoT. J.FoelsR. (2008). Assessing effective teaching of psychology: A meta-analytic integration of learning outcomes. Teaching of Psychology, 35, 286–296.
26.
TomchoT. J.FoelsR.RiceD.JohnsonJ.MosesT. P.WarnerD. J.WetherbeeR.AmalfiT. (2008). A review of ToP teaching strategies: Links to students' scientific inquiry skills development. Teaching of Psychology, 35, 147–159.
27.
WaschullS. B. (2001). The online delivery of psychology courses: Attrition, performance, and evaluation. Teaching of Psychology, 28, 143–147.
28.
WilkinsonL.the Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594–604.
29.
WoolleyT. W. (1983). A comprehensive power-analytic investigation of research in medical education. Journal of Medical Education, 58, 710–715.
30.
WoolleyT. W.DawsonG. O. (1983). A follow-up power analysis of the statistical tests used in the Journal of Research in Science Teaching. Journal of Research in Science Teaching, 20, 673–681.
