This paper shows how Monte Carlo simulation can be used for sample size, power or precision calculations when planning medical research studies. Standard study designs can lead to the use of analysis methods for which power formulae do not exist. This may be because complex modelling techniques with optimal statistical properties are used but power formulae have not yet been derived or because analysis models are employed that divert from the population model due to lack of availability of more appropriate analysis tools. Our presentation concentrates on the conceptual steps involved in carrying out power or precision calculations by simulation. We demonstrate these steps in three examples concerned with (i) drop out in longitudinal studies, (ii) measurement error in observational studies and (iii) causal effect estimation in randomised controlled trials with non-compliance. We conclude that the Monte Carlo simulation approach is an important general tool in the methodological arsenal for assessing power and precision.
HalpernSDKarlawishJHTBerlinJA. The continuing unethical conduct of underpowered clinical trials. J Am Med Assoc2002; 288: 358–362.
2.
ShoukriMMAsyaliMHDonnerA. Sample size requirement for the design of reliability study: review and new results. Stat Methods Med Res2004; 13: 251–271.
3.
VenterAMaxwellSEBoligE. Power in randomized group comparisons: the value of adding a single intermediate time point to a traditional pretest-posttest design. Psychol Methods2002; 7: 194–209.
4.
FaulFErdfelderELangA-G. G*Power 3: a flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behav Res Methods2007; 39: 175–191.
5.
RaudenbushSW. Statistical analysis and optimal design for cluster randomized trials. Psychol Methods1997; 2: 173–185.
6.
RaudenbushSWLiuX. Effects of study duration, frequency of observation, and sample size on power in studies of group differences in polynomial change. Psychol Methods2001; 6: 387–401.
7.
ChristopherM. Monte Carlo simulation, Thousand Oaks, CA: Sage Publications, 1997.
8.
RubinsteinRYKroeseDP. Simulation and the Monte Carlo method, Hoboken, NJ: John Wiley & Sons, 2008.
9.
ShonkwilerRWMendivilF. Explorations in Monte Carlo methods, New York: Springer, 2009.
10.
GastaňagaVMMcLarenCEDelfinoRJ. Power calculations for generalized linear models in observational longitudinal studies: a simulation approach in SAS. Comput Methods Program Biomed2006; 84: 27–33.
11.
TuXMZhangJKowalskiJ. Power analyses for longitudinal study designs with missing data. Stat Med2007; 26: 2958–2981.
12.
WilliamsMSEbelEDWagnerBA. Monte Carlo approaches for determining power and sample size in low-prevalence applications. Preventive Veterinary Med2007; 82: 151–158.
13.
RichardsonDB. Power calculations for survival analyses via Monte Carlo estimation. Am J Ind Med2003; 44: 532–539.
14.
MuthénLKMuthénBO. How to use a Monte Carlo study to decide on sample size and determine power. Struct Equ Modell2002; 9: 599–620.
15.
BrowneWJLahiMGParkerRMA. A guide to sample size calculation for random effects models via simulation and the MLPowSim software package, Manual, University of Bristol, 2009.
16.
FievesonAH. Power by simulation. Stata J2002; 2: 107–114.
HarrisAHSCronkiteRMoosR. Physical activity, exercise coping, and depression in a 10-year cohort study of depressed patients. J Affective Disorders2006; 93: 79–85.
19.
HedekerDGibbonsRDWaternauxC. Sample size estimation for longitudinal designs with attrition: comparing time-related contrasts between two groups. J Edu Behav Stat1999; 24: 70–93.
20.
RoyABhaumikDKAryalS. Sample size determination for hierarchical longitudinal designs with differential attrition rates. Biometrics2007; 63: 699–707.
21.
BasagañaXSpiegelmanD. Power and sample size calculations for longitudinal studies estimating a main effect of a time-varying exposure. Stat Methods Med Res2010; 19: 1–17.
22.
RubinDB. Multiple imputation for nonresponse in surveys, New York: Wiley, 1987.
23.
SchaferJLGrahamJW. Missing data: our view of the state of the art. Psychol Methods2002; 7: 147–177.
24.
CohenJ. Statistical power analysis for the behavioral sciences, 2nd edn. Hillsdale, NJ: Lawrence Erlbaum Associates, 1988.
25.
WhitePDGoldsmithKAJohnsonAL. Comparison of adaptive pacing therapy, cognitive behaviour therapy, graded exercise therapy, and specialist medical care for chronic fatigue syndrome (PACE): a randomised trial. Lancet2011; 377: 823–836.
26.
CohenJ. The cost of dichotomization. Appl Psychol Meas1983; 7: 249–254.
27.
GreenlandS. Avoiding power loss associated with categorization and ordinal scores in dose-response and trend analysis. Epidemiology1995; 6: 450–454.
28.
MacCallumRCZhangSPreacherKJ. On the practice of dichotomization of quantitative variables. Psychol Methods2002; 7: 19–40.
29.
StreinerDL. Breaking up is hard to do: the heartbreak of dichotomizing continuous data. Can J Psychiatry2002; 47: 262–266.
30.
TaylorABWestSGAikenLS. Loss of power in logistic, ordinal logistic, and probit regression when an outcome variable is coarsely categorized. Edu Psychol Meas2006; 66: 228–239.
31.
AltmanDGRoystonP. The cost of dichotomizing continuous variables. Br MedJ2006; 332: 1080–1080.
32.
CarrollRJRuppertDStefanskiLA. Measurement error in nonlinear models, Boca Raton, FL: Chapman and Hall/CRC, 2006.
33.
ChenHCohenPChenS. Biased odds ratios for dichotomization of age. Stat Med2007; 26: 3487–3497.
34.
DunnGMaracyMDowrickC. Estimating psychological treatment effects from a randomized controlled trial with both non-compliance and loss to follow-up. Br J Psychiatry2003; 183: 323–331.
35.
LittleRJAYauLHY. Statistical techniques for analyzing data from prevention trials: treatment of no-shows using Rubin’s causal model. Psychol Methods1998; 3: 147–159.
36.
BellamySLLiuJYTen HaveTR. An introduction to causal modeling in clinical trials. Clin Trials2007; 4: 58–73.
37.
WhiteIR. Uses and limitations of randomization-based efficacy estimators. Stat Methods Med Res2005; 14: 327–347.
38.
AngristJDImbensGWRubinDB. Identification of causal effects using instrumental variables. J Am Stat Assoc1996; 91: 444–455.
39.
FrangakisCERubinDB. Principal stratification in causal inference. Biometrics2002; 58: 21–29.
40.
LittleRJALongQLinX. A comparison of methods for estimating the causal effect of a treatment in randomized clinical trials subject to noncompliance. Biometrics2009; 65: 640–649.
41.
HuiJRubinDB. Principal stratification for causal inference with extended partial compliance. J Am Stat Assoc2008; 103: 101–111.
42.
JoB. Statistical power in randomized intervention studies with non-compliance. Psychol Methods2002; 7: 178–193.
43.
RubinDB. Causal inference using potential outcomes: design, modeling, decisions. J Am Stat Assoc2005; 100: 322–331.
44.
BloomHS. Accounting for no-shows in experimental evaluation designs. Eval Rev1984; 8: 225–246.
JoBMuthénBModeling of intervention effects with noncompliance: a latent variable approach for randomized trials. In: MarcoulidesGASchumackerRE (eds). New developments and techniques in structural equation modeling, Hillsdale, NJ: Lawrence Erlbaum Associates, 2001, pp. 57–87.
47.
JoB. Estimation of intervention effects with noncompliance: alternative model specifications. J Educ Behav Stat2002; 27: 385–409.
HardinJWHilbeJM. Generalized linear models and extensions, 2nd edn. College Station, TX: Stata Press, 2007.
50.
SerlinR. Testing for robustness in Monte Carlo studies. Psychol Methods2000; 5: 230–240.
51.
TurnerELDobsonJLPocockSJ. Categorisation of continuous risk factors in epidemiological publications: a survey of current practice. Epidemiol Perspect Innov2010; 7: 9–9.
52.
YinPFanX. Assessing the reliability of Beck Depression Inventory scores: reliability generalization across studies. Educ Psychol Meas2000; 60: 201–223.
53.
WangFGelfandAE. A simulation based approach to Bayesian sample size determination for performance under a given model and for separating models. Stat Sci2002; 17: 193–208.
54.
InoueLYTBerryDAParmigianiG. Relationship between Bayesian and frequentist sample size determination. Am Stat2005; 59: 79–87.
55.
AdcockCJ. Sample size determination: a review. The Statistician1997; 46: 261–283.
56.
PezeshkH. Bayesian techniques for sample size determination in clinical trials: a short review. Stat Methods Med Res2003; 12: 489–504.
57.
BrownWBHersonJAtkinsonEN. Projection from previous studies: a Bayesian and frequentist compromise. Control Clin Trials1987; 8: 29–44.
58.
FengZGrizzleJE. Correlated binomial variates: properties of estimator of intraclass correlation and its effect on sample size calculation. Stat Med1997; 11: 1607–1614.
59.
JosephLDu BergerRBelisleP. Bayesian and mixed Bayesian/likelihood criteria for sample size determination. Stat Med1997; 16: 769–781.
60.
LeeSJZelenM. Clinical trials and sample size considerations: another perspective. Stat Sci2000; 15: 95–110.
61.
SpiegelhalterDJAbramsKRMylesJP. Bayesian approaches to clinical trials and health-care evaluation, Chichester: Wiley, 2004.
62.
TurnerRPrevostTThompsonSG. Allowing for imprecision of the intracluster correlation coefficient in the design of cluster randomized trials. Stat Med2004; 23: 1195–1214.
63.
CampbellMJDonnerAKlarN. Developments in cluster randomized trials and in statistics in medicine. Stat Med2007; 26: 2–19.
64.
HadeEMMurrayDMPen PennellML. Intraclass correlation estimates for cancer screening outcomes: estimates and applications in the design of group-randomized cancer screening studies. J Natl Cancer Inst Monogr2010; 40: 97–103.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
