This article discusses the interrelated issues of statistical power, sampling, and effect sizes when conducting rigorous quantitative research. Technical and practical connections are made between these concepts and various inferential tests. To increase power and generate effect sizes that merit practical or clinical notice, not only must the research aims and associated design be well devised, reflecting best scientific practice, state of the art sampling procedures should be applied with sufficiently large and representative number of participants. Applications to research conducted in the counseling profession are included.
AguinisH.BeatyJ. C.BoikR. J.PierceC. A. (2005). Effect size and power in assessing moderating effects of categorical variables using multiple regression: A 30 year review. Journal of Applied Psychology, 90, 94–107.
2.
American Psychological Association. (2010). Publication manual of the American Psychological Association. (6th ed.). Washington, DC: Author.
3.
BabbieE. (2010). The practice of social research. (12th ed.). Belmont, CA: Wadsworth.
Coalition for Evidence-based Policy. (2003). Identifying and implementing educational practices supported by rigorous evidence: A user friendly guide. Washington, DC: U.S. Department of Education Institute of Education Sciences National Center for Education Evaluation and Regional Assistance.
6.
CochranW. G. (2007). Sampling techniques. (3rd ed.). New York, NY: Wiley.
7.
CohenB. H.LeaB. R. (2004). Essentials of statistics for the social and behavioral sciences. New York, NY: John Wiley.
8.
CohenJ. (1988). Statistical power analysis for the behavioral sciences. (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
9.
CohenJ. (1992). A power primer. Psychological Bulletin, 112, 155–159.
10.
CohnL. D.BeckerB. J. (2003). How meta-analysis increases statistical power. Psychological Methods, 8, 243–253.
11.
CooperH. (2010). Research synthesis and meta-analysis: A step-by-step approach. Thousand Oaks, CA: Sage.
CreswellJ. W. (2009). Research design: Qualitative, quantitative, and mixed methods approaches. (3rd ed.). Thousand Oaks, CA: Sage.
14.
DaveyA.SavlaJ. (2010). Statistical power analysis with missing data: A structural equation modeling approach. New York, NY: Routledge.
15.
FerronJ.SentovichC. (2002). Statistical power of randomization tests used with multiple-baseline designs. The Journal of Experimental Education, 70, 165–178.
16.
FidlerF.CummingG. (2008). The new stats: Attitudes for the 21st century. In OsborneJ. W. (Ed.), Best practices in quantitative methods (pp. 1–12). Thousand Oaks, CA: Sage.
GreenS. B.SalkindN. J. (2008). Using SPSS for Windows and Macintosh: Analyzing and understanding data. (5th ed.). Upper Saddle River, NJ: Prentice-Hall.
19.
GrissomR. J.KimJ. J. (2005). Effect sizes for research: A broad practical approach. Mahwah, NJ: Erlbaum.
20.
GuoJ-H.LuhW-M. (2009). On sample size calculation for 2×2 fixed-effect ANOVA when variances are unknown and possibly unequal. British Journal of Mathematical and Statistical Psychology, 62, 417–425.
21.
HillC. J.BloomH. S.BlackA. R.LipseyM. W. (2008). Empirical benchmarks for interpreting effect sizes in research. Child Development Perspectives, 2, 172–177.
22.
HouserR. (2008). Counseling and educational research: Evaluation and application. (2nd ed.). Thousand Oaks, CA: Sage.
23.
HowellD. (2008). Best practices in the analysis of variance. In OsborneJ. W. (Ed.), Best practices in quantitative methods (pp. 341–357). Thousand Oaks, CA: Sage.
HuckS. W. (2009). Statistical misconceptions. New York, NY: Routledge.
26.
JoB. (2002). Statistical power in randomized interventions studies with noncompliance. Psychological Methods, 7, 178–193.
27.
JuddC. M.McClellandG. H.RyanC. S. (2009). Data analysis: A model comparison approach. (2nd ed.). New York, NY: Routledge.
28.
KilleenP. R. (2005). An alternative to null-hypothesis significance tests. Psychological Science, 16, 345–353.
29.
KlineR. B. (2004). Beyond significance testing: Reforming data analysis methods in behavioral research. Washington, DC: American Psychological Association.
30.
KonstantopoulosS. (2008). An introduction to meta-analysis. In OsborneJ. W. (Ed.), Best practices in quantitative methods (pp. 177–194). Thousand Oaks, CA: Sage.
LipseyM. W.WilsonD. B. (2001). Practical meta-analysis. Thousand Oaks, CA: Sage.
33.
MaxwellS. E.DelaneyH. D. (2004). Designing experiments and analyzing data: A model comparison perspective. (2nd ed.). Mahwah, NJ: Erlbaum.
34.
MertensD. M. (2010). Research and evaluation in education and psychology. (3rd ed.). Los Angeles, CA: Sage.
35.
MeyersL. S.GamstG.GuarinoA. J. (2006). Applied multivariate research: Design and interpretation. Thousand Oaks, CA: Sage.
36.
MilesJ. (2003). A framework for power analysis using structural equation modeling procedure. BMC Medical Research Methodology, 3, 1–11.
37.
MillisS. R. (2003). Statistical practices: The seven deadly sins. Child Neuropsychology, 9, 221–233.
38.
MossP. A.PhillipsD. C.EricksonF. D.FlodenR. E.LatherP. A.SchneiderB. L. (2009). Learning from our differences: A dialogue across perspectives on quality in education research. Educational Researcher, 38, 501–517. doi: 10.3102/0013189X09348351.
39.
MuncerS. J.CraigieM.HolmesJ. (2003). Meta analysis and power: Some suggestions for the use of power in research synthesis. Understanding Statistics, 2, 1–12.
40.
NakagawaS.FosterT. M. (2004). The case against retrospective statistical power analyses with an introduction to power analysis. Acta Ethologica, 7, 103–108.
41.
OsborneJ. W. (Ed.). (2008). Best practices in quantitative methods. Thousand Oaks, CA: Sage.
RosnowR. L.RosenthalR. (1996). Computing contrasts, effect sizes, and counternulls on other people’s published data: General procedures for research consumers. Psychological Methods, 1, 331–340.
44.
RowlandD.ThortonJ. (2002). Transforming ideas into research. In NiebauerJ. (Ed.), The clinical research survival guide (pp. 43–78). London, UK: REMEDICA Publishing.
45.
SheskinD. (2007). Handbook of parametric and nonparametric statistical procedures. (4th ed). New York, NY: CRC Press.
46.
ShiehG. (2003). A comparative study of power and sample size calculations for multivariate general linear models. Multivariate Behavioral Research, 38, 285–307.
47.
SinkC. A.MvududuN. H. (2009). Meta-analysis. In ErfordB. T. (Ed.), Encyclopedia of counseling (pp. 334–337). Alexandria, VA: American Counseling Association.
48.
SinkC. A.StrohH. R. (2006). Practical significance: The use of effect sizes in school counseling research. Professional School Counseling, 9, 401–411.
StuartE. A.RubinD. B. (2008). Best practices in quasi-experimental designs: Matching methods for causal inference. In OsborneJ. W. (Ed.), Best practices in quantitative methods (pp. 155–176). Thousand Oaks, CA: Sage.
51.
ThalheimerW.CookS. (2002, August). How to calculate effect sizes from published research articles: A simplified methodology. Retrieved fromhttp://work-learning.com/effect_sizes.htm
52.
ThompsonB. (2006a). Research synthesis: Effect sizes. In GreenJ.CamilliG.ElmoreP. B. (Eds.), Handbook of complementary methods in education research (pp. 583–603). Washington, DC: American Educational Research Association.
53.
ThompsonB. (2006b). Role of effect sizes in contemporary research in counseling. Counseling and Values, 50, 176–186.
54.
ThompsonB. (2007). Effect sizes, confidence intervals, and confidence intervals for effect sizes. Psychology in the Schools, 44, 423–432.
55.
ThompsonB. (2008). Computing and interpreting effect sizes, confidence intervals, and confidence intervals for effect sizes. In OsborneJ. W. (Ed.), Best practices in quantitative methods (pp. 246–262). Thousand Oaks, CA: Sage.
TrustyJ.ThompsonB.PetrocelliJ. V. (2004). Practical guide for reporting effect size in quantitative research. Journal of Counseling & Development, 82, 107–110.
58.
Vacha-HaaseT.ThompsonB. (2004). How to estimate and interpret various effect sizes. Journal of Counseling Psychology, 51, 473–481.
59.
VenterA.MaxwellS. E.BoligE. (2002). Power is randomized group comparisons: The value of adding a single intermediate time point to a traditional pretest–posttest design. Psychological Methods, 7, 194–209.
60.
WheelerD. W.SeagramA. T.BeckerL.W.KinleyE. R.MlinekD. D. (2008). The academic chair’s handbook. (2nd ed.). San Francisco, CA: Jossey-Bass.
