There is substantial evidence that poorly designed and reported research can mislead decision making in clinical care. This review investigates the methodological quality of Swedish trials of a wide array of psychological and social interventions.
Method:
The review includes 302 articles published in peer-reviewed journals during 1990–2014, which evaluated a psychological or social intervention; included a randomized or nonrandomized control group with a pre–post- or pre-follow-up design; and targeted individuals in order to prevent or rehabilitate from physical, psychological, or social problems. The results are based on the information reported in the articles.
Results:
Overall, methodological quality increased with time. However, more recent studies contained methodological deficits. For example, few had used blinded data collectors, reported power analyses, or searched for adverse events.
Conclusion:
The results indicate a potential risk of biased results in both older and newer trials. Possible implications for practice and research are discussed.
AbdiH. (2007). Bonferroni and Šidák corrections for multiple comparisons. In SalkindN. J. (Ed.), Encyclopedia of measurement and statistics (pp. 103–107). Thousand Oaks, CA: Sage.
2.
Armijo-OlivoS.da CostaB. R.CummingsG. G.HaC.FuentesJ.SaltajiH.EggerM. (2015). PEDro or cochrane to assess the quality of clinical trials? A meta-epidemiological study. PLoS One, 10, e0132634. doi:10.1371/journal.pone.0132634
3.
Armijo-OlivoS.FuentesJ.OspinaM.SaltajiH.HartlingL. (2013). Inconsistency in the items included in tools used in general health research and physical therapy to evaluate the methodological quality of randomized controlled trials: A descriptive analysis. BMC Medical Research Methodology, 13, 116. doi:10.1186/1471-2288-13-116
4.
Armijo-OlivoS.OspinaM.da CostaB.EggerM.SaltajiH.FuentesJ.…CummingG. G. (2014). Poor reliability between cochrane reviewers and blinded external reviewers when applying the cochrane risk of bias tool in physical therapy trials. PLoS One, 9, e96920. doi:10.1371/journal.pone.0096920
5.
Armijo-OlivoS.WarrenS.MageeD. (2009). Intention to treat analysis, compliance, drop-outs and how to deal with missing data in clinical research: A review. Physical Therapy Reviewers, 14, 3649. doi:10.1179/174328809X405928
6.
BoettcherJ.RozentalA.AnderssonG.CarlbringP. (2014). Side effects in Internet-based interventions for Social Anxiety Disorder. Internet Interventions, 1, 3–11. doi:http://dx.doi.org/10.1016/j.invent.2014.02.002
7.
BurnsM. K.KlingbeilD. A.YsseldykeJ. E.Petersen-BrownS. (2012). Trends in methodological rigor in intervention research published in school psychology journals. Psychology in the Schools, 49, 843–851. doi:10.1002/pits.21637
8.
ChiavettaN. M.MartinsA. R.HenriquesI. C.FregniF. (2014). Differences in methodological quality between positive and negative published clinical trials. Journal of Advanced Nursing, 70, 2389–2403. doi:10.1111/jan.12380
9.
ClarkL.SchmidtU.TharmanathanP.AdamsonJ.HewittC.TorgersonD. (2013). Poor reporting quality of key randomization and allocation concealment details is still prevalent among published RCTs in 2011: A review. Journal of Evaluation in Clinical Practice, 19, 703–707. doi:10.1111/jep.12031
10.
Des JarlaisD. C.LylesC.CrepazN., & Transparent Reporting of Evaluations with Nonrandomized Designs Group. (2004). Improving the reporting quality of nonrandomized evaluations of behavioral and public health interventions: The TREND statement. American Journal of Public Health, 94, 361–366.
11.
DevereauxP. J.ChoiP. T.El-DikaS.BhandariM.MontoriV. M.SchunemannH.…GuyattG. H. (2004). An observational study found that authors of randomized controlled trials frequently use concealment of randomization and blinding, despite the failure to report these methods. Journal of Clinical Epidemiology, 57, 1232–1236. doi:10.1016/j.jclinepi.2004.03.017
12.
DwanK.AltmanD. G.ArnaizJ. A.BloomJ.ChanA.-W.EugeniaC.…WilliamsonP. R. (2008). Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS One, 3, e3081. doi:10.1371/journal.pone.0003081
13.
FlayB. R.BiglanA.BoruchR. F.CastroF. G.GottfredsonD.KellamS.…JiP. (2005). Standards of evidence: Criteria for efficacy, effectiveness and dissemination. Prevention Science, 6, 151–175. doi:10.1007/s11121-005-5553-y
14.
HaynesR. H.DevereauP. J.GuyattG. H. (2002). Physicians’ and patients choices in evidence based practice: Evidence does not make decisions, people do. British Medical Journal, 324, 1350–1351.
15.
HorigianV. E.RobbinsM. S.DominguezR.UchaJ.CarmenL. R. (2010). Principles for defining adverse events in behavioral intervention research: Lessons from a family focused adolescent drug abuse trial. Clinical Trials, 7, 58–68. doi:10.1177/1740774509356575
16.
HrobjartssonA.PildalJ.ChanA.-W.HaahrM. T.AltmanD. G.GøtzscheP. C. (2009). Reporting on blinding in trial protocols and corresponding publications was often inadequate but rarely contradictory. Journal of Clinical Epidemiology, 62, 967–973. doi:10.1016/j.jclinepi.2009.04.003
17.
JadadA. R.MooreR. A.CarrollD.JenkinsonC.ReynoldsD. J.GavaghanD. J.McQuayH. J. (1996). Assessing the quality of reports of randomized clinical trials: Is blinding necessary. Controlled Clinical Trials, 17, 1–12. doi:10.1016/0197-2456(95)00134-4
18.
JonssonU.AlaieI.ParlingT.ArnbergF. K. (2014). Reporting of harms in randomized controlled trials of psychological interventions for mental and behavioral disorders: A review of current practice. Contemporary Clinical Trials, 38, 1–8. doi:10.1016/j.cct.2014.02.005
KatikireddiS. V.EganM.PetticrewM. (2015). How do systematic reviews incorporate risk of bias assessments into the synthesis of evidence? A methodological study. Journal of Epidemiology & Community Health, 69, 189–195. doi:10.1136/jech-2014-204711
21.
KennedyS. S.MercerJ.MohrW.HuffineC. W. (2002). Snake oil, ethics, and the first amendment: What’s a profession to do?American Journal of Orthopsychiatry, 72, 5–15. doi:http://dx.doi.org/10.1037/0002-9432.72.1.5
22.
KuderG. F.RichardsonM. W. (1937). The theory of the estimation of test reliability. Psychometrika, 2, 151–160.
23.
LilienfeldS. O. (2007). Psychological treatments that cause harm. Perspectives on Psychological Science, 2, 53–70. doi:10.1111/j.1745-6916.2007.00029.x
24.
LipseyM. W.CullenF. T. (2007). The effectiveness of correctional rehabilitation: A review of systematic reviews. Annual Review of Law and Social Science, 3, 297–320. doi:10.1146/annurev.lawsocsci.3.081806.112833
LyonsK. (2003). Doctoral studies in social work: exploring European developments. Social Work Education, 22, 555–564. doi: 10.1080/0261547032000142661
27.
MacKinnonD. P. (2008). Introduction to statistical mediation analysis. New York, NY: Lawrence Erlbaum.
28.
MasonC. H.PerreaultW. D. (1991). Collinearity, power, and interpretation of multiple regression analysis. Journal of Marketing Research, 28, 268–280. doi:10.2307/3172863
29.
MhaskarR.DjulbegovicB.MagazinA.SoaresH. P.KumarA. (2012). Published methodological quality of randomized controlled trials does not reflect the actual quality assessed in protocols. Journal of Clinical Epidemiology, 65, 602–609. doi:10.1016/j.jclinepi.2011.10.016
30.
MoherD.HopewellS.SchulzK. F.MontoriV.GøtzscheP. C.DevereauxP. J.…AltmanD. G. (2010). CONSORT 2010 explanation and elaboration: Updated guidelines for reporting parallel group randomised trials. British Medical Journal, 340, c869. doi:http://dx.doi.org/10.1136/bmj.c869
31.
MoherD.JonesA.LepageL., & Consolidated Standards for Reporting of Trials Group. (2001). Use of the CONSORT statement and quality of reports of randomized trials: A comparative before-and-after evaluation. Journal of the American Medical Association, 285, 1992–1995. doi:10.1001/jama.285.15.1992
32.
MoherD.PhamB.JonesA.CookD. J.JadadA. R.MoherM.…KlassenT. P. (1998). Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses?Lancet Oncology, 352, 609–613. doi:10.1016/S0140-6736(98)01085-X
33.
MurrayD. M. (1998). Design and analysis of group-randomized trials. New York, NY: Oxford University Press.
34.
OakleyA. (1998). Experimentation and social interventions: a forgotten but important history. Education and debate, 317, 1239–1242. doi:http://dx.doi.org/10.1136/bmj.317.7167.1239
35.
OlssonT. M.SundellK. (2016). Research that guides practice: outcome research in Swedish Ph.D. theses across seven disciplines 1997–2012. Prevention Science, 17, 525–532.
36.
RobinsonW. S. (1950). Ecological correlations and the behavior of individuals. American Sociological Review, 15, 351–357. doi:10.2307/2087176. JSTOR 2087176
37.
RosenbaumP. R.RubinD. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70, 41–55. doi:10.1093/biomet/70.1.41
38.
RubinD. B. (1976). Inference and missing data. Biometrika, 63, 581–592. doi:10.1093/biomet/63.3.581
39.
SchulzK. F.ChalmersI.HayesR. J.AltmanD. G. (1995). Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. Journal of the American Medical Association, 273, 408–412. doi:10.1001/jama.1995.03520290060030
40.
ShadishW. R.ClarkM. H.SteinerP. M. (2008). Can nonrandomized experiments yield accurate answers? A randomized experiment comparing random and nonrandom assignments. Journal of the American Statistical Association, 103, 1334–1344. doi:10.1198/016214508000000733
41.
ShadishW. R.CookT. D.CampellD. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin Company.
42.
ShamliyanT.KaneR. L.JansenS. (2012). Systematic reviews synthesized evidence without consistent quality assessment of primary studies examining epidemiology of chronic diseases. Journal of Clinical Epidemiology, 65, 610–618. doi:10.1016/j.jclinepi.2011.10.017
43.
SkivingtonK.McCartneyG.ThomsonH.BondL. (2010). Challenges in evaluating Welfare to Work policy interventions: Would an RCT design have been the answer to all our problems?BMC Public Health, 17, 254. doi:10.1186/1471-2458-10-254
44.
SoaresH. P.DanielsS.KumarA.ClarkeM.ScottC.SwannS.DjulbegovicB. (2004). Bad reporting does not mean bad methods for randomised trials: Observational study of randomised controlled trials performed by the Radiation Therapy Oncology Group. British Medical Journal, 328, 22–24. doi:10.1136/bmj.328.7430.22
45.
Socialstyrelsen. (2009). Socialtjänstens öppna verksamheter för barn och unga—en nationell inventering av metoder [Outpatient care in child welfare—A national inventory of interventions]. Stockholm, Sweden: National Board of Health and Welfare.
46.
SundellK.ÅhsbergE. (2014). Vetenskaplig kvalitet i publicerade effektutvärderingar. 169 svenska utvärderingar av psykologiska och sociala interventioner [Methodological quality in published outcome evaluations. 169 Swedish trials of psychological and social interventions]. Stockholm, Sweden: National Board of Health and Welfare.
47.
SundellK.SoydanH.TengvaldK.AnttilaS. (2010). Advancing social work practice from opinion-based to evidence-based: Sweden’s Institute for evidence-based social work practice. Research on Social Work Practice, 20, 714–722. doi:10.1177/1049731509347887
TanakaM.JamiesonE.WathenN.MacMillanH. L. (2010). Methodological standards for randomised controlled trials of interventions for preventing recurrence of child physical abuse and neglect. Child Abuse Review, 19, 21–38. doi:10.1002/car.1068
50.
TengvaldK.SundellK. (2013). Mot bättre vetande. Centrum för utvärdering av socialt arbete och Institutet för utveckling av metoder i socialt arbete, 1992 till 2009 [Towards better judgment. The Center for Evaluation of Social Services and the Institute for Evidence-Based Social Work Practice, 1992-2009]. Stockholm, Sweden: National Board of Health and Welfare.
51.
ThorndikeE. L.WoodworthR. S. (1901). The influence of improvement in one mental function upon the efficiency of other functions. Psychological Review, 8, 247–261.
52.
TukeyJ. W. (1977). Exploratory data analysis. Reading, MA: Addison-Wesley.
53.
TurnerL.ShamseerL.AltmanD. G.SchulzK. F.MoherD. (2012). Does use of the CONSORT statement impact the completeness of reporting of randomised controlled trials published in medical journals? A Cochrane review. Systematic Reviews, 1, 60. doi:10.1186/2046-4053-1-60
54.
WeisburdD.LumC. M.PetrosinoA. (2001). Does research design affect study outcomes in criminal justice?Annals of the American Academy, 578, 50–70. doi:10.1177/000271620157800104
55.
WilcoxR. R. (2002). Understanding the practical advantages of modern ANOVA methods. Journal of Clinical Child and Adolescent Psychology, 31, 399–412. doi:10.1207/S15374424JCCP3103_12
56.
WinchW. H. (1908). The transfer of improvement in memory in school-children. British Journal of Psychology, 2, 284–293.
57.
WolffN. (2000). Using randomized controlled trials to evaluate socially complex services: Problems, challenges and recommendations. The Journal of Mental Health Policy and Economics, 3, 97–109. doi:10.1002/1099-176X(200006)3:2<97::AID-MHP77>3.0.CO;2-S
58.
WoodL.EggerM.GluudL. L.SchulzK. F.JüniP.AltmanD. G.…SterneJ. A. (2000). Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: Meta-epidemiological study. British Medical Journal, 336, 601–605. doi:10.1136/bmj.39465.451748.AD
59.
WrightC. C.SimJ. (2003). Intention to treat approach to data from randomized controlled trials: A sensitivity analysis. Journal of Clinical Epidemiology, 56, 533–842. doi:10.1016/S0895-4356(03)00155-0
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
