Meta-analysis is a statistical method that combines quantitative findings from previous studies. It has been increasingly used to obtain more credible results in a wide range of scientific fields. Combining the results of relevant studies allows researchers to leverage study similarities while modeling potential sources of between-study heterogeneity. This article provides a review of the core methodologies of meta-analysis that we consider most relevant to sociological research. After developing the foundation of the fixed- and random-effects models of meta-analysis models, this article illustrates the utility of the method with regression coefficients reported from two sets of social science studies. We explain the various steps of the process including constructing the meta-sample from primary studies, estimating the fixed- and random-effects models, analyzing the source of heterogeneity across studies, and assessing publication bias. We conclude with a discussion of steps that could be taken to strengthen the development of meta-analysis in sociological research, which will eventually increase the credibility of sociological inquiry via a knowledge-cumulative process.
AloeAriel M.BeckerBetsy Jane. 2012. “An Effect Size for Regression Predictors in Meta-analysis.” Journal of Educational and Behavioral Statistics37(2):278–97.
2.
AmatoPaul R.KeithBruce. 1991. “Consequences of Parental Divorce for Adult Well-being: A Meta-analysis.” Journal of Marriage and the Family53(1):43–58.
3.
AmatoPaul R.GilbrethJoan G.. 1999. “Nonresident Fathers and Children’s Well-Being: A Meta-Analysis.” Journal of Marriage and the Family61(3):557–73.
4.
ArthurWinfredJrBennettWinstonHuffcuttAllen I.. 2001. Conducting Meta-analysis Using SAS. Mahwah, NJ: Lawrence Erlbaum Associates.
5.
BeckerBetsy JaneMeng-Jia Wu. 2007. “The Synthesis of Regression Slopes in Meta-analysis.” Statistical Science22(3):414–29.
6.
BerkeyCatherine S.HoaglinDavid C.MostellerFrederickColditzGraham A.. 1995. “A Random-effects Regression Model for Meta-analysis.” Statistics in Medicine14(4):395–411.
7.
BoffettaPaoloBobakMartinBorsch-SupanAxelBrennerHermannErikssonStureGrodsteinFranJansenEugeneJenabMazdaJuergesHendrikKampmanEllen, and others. 2014. “The Consortium on Health and Ageing: Network of Cohorts in Europe and the United States (CHANCES) Project—Design, Population and Data Harmonization of a Large-scale, International Study.” European Journal of Epidemiology29(12):929–36.
8.
BorensteinMichaelHedgesLarry V.HigginsJulian PTRothsteinHannah R.. 2009. Introduction to Meta-analysis. Chichester, England: John Wiley.
9.
BoruchRobert F.2005. “Place Randomized Trials in Education, Criminology, Welfare and Health.” The Annals of the American Academy of Political and Social Science599:6–18.
10.
BraniganAmelia R.McCallumKenneth J.FreeseJeremy. 2013. “Variation in the Heritability of Educational Attainment: An International Meta-analysis.” Social Forces92(1):109–40.
11.
BrockwellSarah E.GordonIan R.. 2001. “A Comparison of Statistical Methods for Meta-analysis.” Statistics in Medicine20(6):825–40.
12.
ChanAn-WenAltmanDouglas G.. 2005. “Identifying Outcome Reporting Bias in Randomized Trials on PubMed: Review of Publications and Survey of Authors.” BMJ330(7494):753–6.
13.
ChanMeowLan EvelynArveyRichard D.. 2012. “Meta-analysis and the Development of Knowledge.” Perspectives on Psychological Science7(1):79–92.
14.
CheungMike W. L.2014. “Modeling Dependent Effect Sizes with Three-level Meta-analyses: A Structural Equation Modeling Approach.” Psychological Methods19(2):211–29.
15.
CheungMike W. L.2015. Meta-analysis: A Structural Equation Modeling Approach. Hoboken, NJ: John Wiley.
16.
ChinnSusan. 2000. “A Simple Method for Converting an Odds Ratio to Effect Size for Use in Meta-analysis.” Statistics in Medicine19(22):3127–31.
17.
CialdiniRobert B.TrostMelanie R.. 1998. “Social Influence: Social Norms, Conformity and Compliance.” Pp. 151–92 in The Handbook of Social Psychology. Vol. 2, edited by GilbertD. T.FiskeS. T.LindzeyG.. Boston, MA: McGraw-Hill.
18.
ColemanJames S.CampellE.HubsonC.McPartlandJ.MoodA.WeinfeldF.YorkR.. 1966. Equality of Educational Opportunity. Washington, DC: U.S. Government Printing Office.
19.
CooperHarris H.HedgesLarry V.. 2009. “Research Synthesis as a Scientific Process.” Pp. 3–18 in The handbook of research synthesis and meta-analysis, edited by CooperH. H.HedgesL. V.ValentineJ. C.. New York: Russell Sage Foundation.
20.
CopasJohnShiJian Qing. 2000. “Meta-analysis, Funnel Plots and Sensitivity Analysis.” Biostatistics1(3):247–62.
21.
CorrellShelley J.BenardStephenPaikIn. 2007. “Getting a Job: Is There a Motherhood Penalty?” American Journal of Sociology112(5):1297–338.
22.
DebrayThomas PAMoonsKarel GMValkenhoefGert vanEfthimiouOrestisHummelNoemiGroenwoldRolf HHReitsmaJohannes B., and GetReal Methods Review Group. 2015. “Get Real in Individual Participant Data (IPD) Meta-analysis: A Review of the Methodology.” Research Synthesis Methods6(4):293–309.
23.
DeeksJonathan J.AltmanDouglas G.BradburnMichael J.. 2008. “Statistical Methods for Examining Heterogeneity and Combining Results from Several Studies in Meta-analysis.” Pp. 285–312 in Systematic Reviews in Health Care: Meta-analysis in Context, Second Edition, edited by EggerM.SmithG. D.AltmanD. G.. London: BMJ Publishing Group.
24.
DoucouliagosChrisStanleyTom D.. 2013. “Are All Economic Facts Greatly Exaggerated? Theory Competition and Selectivity.” Journal of Economic Surveys27(2):316–39.
25.
DoucouliagosHristosPaldamMartin. 2006. “Aid Effectiveness on Accumulation: A Meta Study.” Kyklos59(2):227–54.
26.
DuvalSueTweedieRichard. 2000. “Trim and Fill: A Simple Funnel-plot–Based Method of Testing and Adjusting for Publication Bias in Meta-analysis.” Biometrics56(2):455–63.
27.
EggerMatthiasSmithGeorge DaveySchneiderMartinMinderChristoph. 1997. “Bias in Meta-analysis Detected by a Simple, Graphical Test.” BMJ315(7109):629–34.
28.
FirebaughGlenn. 2008. Seven Rules for Social Research. Princeton, NJ: Princeton University Press.
29.
GerberAlan S.MalhotraNeil. 2008. “Publication Bias in Empirical Sociological Research: Do Arbitrary Significance Levels Distort Published Results?” Sociological Methods & Research37(1):3–30.
30.
GlassGene V.1976. “Primary, Secondary, and Meta-analysis of Research.” Educational Researcher5(10):3–8.
31.
HarrisR.BradburnM.DeeksJ.HarbordRogerAltmanD.SterneJ.. 2008. “Metan: Fixed- and Random-effects Meta-analysis.” Stata Journal8(1):3–28.
32.
HedgesLarry V.1983. “A Random Effects Model for Effect Sizes.” Psychological Bulletin93(2):388–95.
33.
HedgesLarry V.TiptonElizabethJohnsonMatthew C.. 2010. “Robust Variance Estimation in Meta-regression with Dependent Effect Size Estimates.” Research Synthesis Methods1(1):39–65.
34.
HedgesLarry V.VeveaJack L.. 1998. “Fixed- and Random-effects Models in Meta-analysis.” Psychological Methods3(4):486–504.
35.
HedgesLarry V.LaineRichard D.GreenwaldRob. 1994. “An Exchange: Part I: Does Money Matter? A Meta-analysis of Studies of the Effects of Differential School Inputs on Student Outcomes.” Educational Researcher23(3):5–14.
36.
HigginsJulianThompsonSimon G.. 2002. “Quantifying Heterogeneity in a Meta-analysis.” Statistics in Medicine21(11):1539–58.
37.
HigginsJulian P. T.ThompsonSimon G.DeeksJonathan J.AltmanDouglas G.. 2003. “Measuring Inconsistency in Meta-analyses.” BMJ327(7414):557–60.
38.
HoaglinDavid C.2016. “Misunderstandings about Q and ‘Cochran’s Q Test’in Meta-analysis.” Statistics in Medicine35(4):485–95.
39.
Huedo-MedinaTania B.Sánchez-MecaJulioMarín-MartínezFulgencioBotellaJuan. 2006. “Assessing Heterogeneity in Meta-analysis: Q Statistic or I2 Index?” Psychological Methods11(2):193–206.
40.
JacksonDanBowdenJackBakerRose. 2010. “How Does the DerSimonian and Laird Procedure for Random Effects Meta-analysis Compare with Its More Efficient but Harder to Compute Counterparts?” Journal of Statistical Planning and Inference140(4):961–70.
41.
JacksonDanRileyRichardWhiteIan R.. 2011. “Multivariate Meta-analysis: Potential and Promise.” Statistics in Medicine30(20):2481–98.
42.
JohnsonElizabeth I.EasterlingBeth. 2012. “Understanding Unique Effects of Parental Incarceration on Children: Challenges, Progress, and Recommendations.” Journal of Marriage and Family74(2):342–56.
43.
KandelDenise B.1978. “Homophily, Selection, and Socialization in Adolescent Friendships.” American Journal of Sociology84(2):427–36.
44.
KonstantopoulosSpyrosHedgesLarry V.. 2004. “Meta-analysis.” Pp. 281–97 in The Sage Handbook of Quantitative Methodology for the Social Sciences, edited by KaplanD.. Thousand Oaks, CA: Sage.
45.
LewisSteffClarkeMike. 2001. “Forest Plots: Trying to See the Wood and the Trees.” BMJ322(7300):1479–80.
46.
LinLifengChuHaitao. 2018. “Quantifying Publication Bias in Meta-analysis.” Biometrics74(3):785–94.
47.
LinLifengChuHaitaoMuradMohammad HassanHongChuanQuZhiyongColeStephen R.ChenYong. 2018. “Empirical Comparison of Publication Bias Tests in Meta-analysis.” Journal of General Internal Medicine33(8):1260–67.
48.
LipseyMark W.2009. “Identifying Interesting Variables and Analysis Opportunities.” Pp. 147–58 in The Handbook of Research Synthesis and Meta-analysis, edited by CooperH. H.HedgesL. V.ValentineJ. C.. New York: Russell Sage Foundation.
ManskiCharles F.1993. “Identification of Endogenous Social Effects: The Reflection Problem.” The Review of Economic Studies60(3):531–42.
51.
McCartneyKathleenRosenthalRobert. 2000. “Effect Size, Practical Importance, and Social Policy for Children.” Child Development71(1):173–80.
52.
McManusR. J.WilsonSueDelaneyB. C.FitzmauriceD. A.HydeC. J.TobiasR. S.JowettSueHobbsF. D. R.. 1998. “Review of the Usefulness of Contacting Other Experts When Conducting a Literature Search for Systematic Reviews.” BMJ317(7172):1562–63.
53.
MoffittTerrie E.CaspiA.RutterM.SilvaP. A.. 2001. Sex Differences in Antisocial Behaviour: Conduct Disorder, Delinquency, and Violence in the Dunedin Longitudinal Study. Cambridge, England: Cambridge University Press.
54.
MulderMonique BorgerhoffBowlesSamuelHertzTomBellAdrianBeiseJanClarkGregFazzioIlaGurvenMichaelHillKimHooperPaul L.. 2009. “Intergenerational Wealth Transmission and the Dynamics of Inequality in Small-scale Societies.” Science326(5953):682–88.
55.
PetersJaime L.SuttonAlex J.JonesDavid R.AbramsKeith R.RushtonLesley. 2007. “Performance of the Trim and Fill Method in the Presence of Publication Bias and Between-study Heterogeneity.” Statistics in Medicine26(25):4544–62.
56.
PoldinOlegValeevaDiliaraYudkevichMaria. 2016. “Which Peers Matter: How Social Ties Affect Peer-group Effects.” Research in Higher Education57(4):448–68.
57.
ReedJeffreyBaxterPam. 2009. “Using Reference Databases.” Pp. 73–102 in Handbook Of Research Synthesis and Meta-analysis, edited by CooperH. H.HedgesL. V.ValentineJ. C.. New York: Russell Sage Foundation.
58.
RoelfsDavid J.ShorEranFalzonLouiseDavidsonKarina W.SchwartzJoseph E.. 2013. “Meta-analysis for Sociology—A Measure-driven Approach.” Bulletin of Sociological Methodology117(1):75–92.
59.
RoksaJosipaPotterDaniel. 2011. “Parenting and Academic Achievement: Intergenerational Transmission of Educational Advantage.” Sociology of Education84(4):299–321.
60.
RosenthalRobert. 1979. “The File Drawer Problem and Tolerance for Null Results.” Psychological Bulletin86(3):638–41.
61.
RothsteinHannah R.HopewellSally. 2009. “Grey Literature.” Pp. 103–25 in The Handbook of Research Synthesis and Meta-analysis, edited by CooperH.HedgesL. V.ValentineJ. C.. New York: Russell Sage Foundation.
62.
RothsteinHannah R.SuttonAlexander J.BorensteinMichael. 2006. Publication Bias in Meta-analysis: Prevention, Assessment and Adjustments. Hoboken, NJ: John Wiley.
63.
SacerdoteBruce. 2014. “Experimental and Quasi-experimental Analysis of Peer Effects: Two Steps Forward?” Annual Review of Economics6(1):253–72.
64.
ShorEranRoelfsDavid J.CurreliMistyClemowLynnBurgMatthew M.SchwartzJoseph E.. 2012. “Widowhood and Mortality: A Meta-analysis and Meta-regression.” Demography49(2):575–606.
65.
ShorEranRoelfsDavid J.YogevTamar. 2013. “The Strength of Family Ties: A Meta-analysis and Meta-regression of Self-reported Social Support and Mortality.” Social Networks35(4):626–38.
66.
StanleyTom D.2001. “Wheat from Chaff: Meta-analysis as Quantitative Literature Review.” The Journal of Economic Perspectives15(3):131–50.
67.
StanleyTom D.DoucouliagosHristos. 2015. “Neither Fixed nor Random: Weighted Least Squares Meta-analysis.” Statistics in Medicine34(13):2116–27.
68.
StanleyTom D.DoucouliagosHristosGilesMargaretHeckemeyerJost H.JohnstonRobert J.LarochePatriceNelsonJon P.PaldamMartinPootJacquesPughGeoff. 2013. “Meta-analysis of Economics Research Reporting Guidelines.” Journal of Economic Surveys27(2):390–94.
69.
SterneJonathan A. C.EggerMatthias. 2005. “Regression Methods to Detect Publication and Other Bias in Meta-analysis.” Pp. 99–110 in Publication Bias in Meta-Analysis: Prevention, Assessment and Adjustments, edited by RothsteinH. R.SuttonA. J.BorensteinM.Chichester, England: John Wiley & Sons.
70.
SuttonAlex J.DuvalS. J.TweedieR. L.AbramsKeith R.JonesDavid R.. 2000. “Empirical Assessment of Effect of Publication Bias on Meta-analyses.” BMJ320(7249):1574–77.
71.
SuttonAlexander J.HigginsJulian. 2008. “Recent Developments in Meta-analysis.” Statistics in Medicine27(5):625–50.
72.
ThomasJeremy N.OlsonDaniel VA. 2010. “Testing the Strictness Thesis and Competing Theories of Congregational Growth.” Journal for the Scientific Study of Religion49(4):619–39.
73.
ThompsonSimon G.HigginsJulian. 2002. “How Should Meta-regression Analyses Be Undertaken and Interpreted?” Statistics in Medicine21(11):1559–73.
74.
ValentineJeffrey C.2009. “Judging the Quality of Primary Research.” Pp. 129–46 in The Handbook of Research Synthesis and Meta-analysis, edited by CooperH. H.HedgesL. V.ValentineJ. C.. New York: Russell Sage Foundation.
75.
Van de BongardtDaphneReitzEllenSandfortTheoDekovićMaja. 2015. “A Meta-analysis of the Relations between Three Types of Peer Norms and Adolescent Sexual Behavior.” Personality and Social Psychology Review19(3):203–34.
76.
Van Den NoortgateWimOnghenaPatrick. 2005. “Parametric and Nonparametric Bootstrap Methods for Meta-analysis.” Behavior Research Methods37(1):11–22.
77.
Van HouwelingenHans C.ArendsLidia R.StijnenTheo. 2002. “Advanced Methods in Meta-analysis: Multivariate Approach and Meta-regression.” Statistics in Medicine21(4):589–624.
78.
VeronikiAreti AngelikiJacksonDanViechtbauerWolfgangBenderRalfBowdenJackKnappGuidoKussOliverHigginsJulian PTLanganDeanSalantiGeorgia. 2016. “Methods to Estimate the Between-study Variance and Its Uncertainty in Meta-analysis.” Research Synthesis Methods7(1):55–79.
79.
ViechtbauerWolfgang. 2007. “Confidence Intervals for the Amount of Heterogeneity in Meta-analysis.” Statistics in Medicine26(1):37–52.
WilsonDavid B.2009. “Systematic Coding.” Pp. 159–76 in The Handbook of Research Synthesis and Meta-analysis, edited by CooperH. H.HedgesL. V.ValentineJ. C.. New York: Russell Sage Foundation.
82.
YoungCristobal. 2009. “Model Uncertainty in Sociological Research: An Application to Religion and Economic Growth.” American Sociological Review74(3):380–97.
83.
YoungCristobalHolsteenKatherine. 2017. “Model Uncertainty and Robustness: A Computational Framework for Multimodel Analysis.” Sociological Methods & Research46(1):3–40.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
