Meta-analysis has proved increasingly popular in management and organization
studies as a way of combining existing empirical quantitative research to
generate a statistical estimate of how strongly variables are associated. Whilst
a number of studies identify technical, procedural and practical limitations of
meta-analyses, none have yet tackled the meta-theoretical flaws
in this approach. We deploy critical realist meta-theory to argue that the
individual quantitative studies, upon which meta-analysis relies, lack
explanatory power because they are rooted in quasi-empiricist meta-theory. This
problem, we argue, is carried over in meta-analyses. We then propose a ‘critical
realist synthesis’ as a potential alternative to the use of meta-analysis in
organization studies and social science more widely.
AckroydSKarlssonJC (2014) Critical realism, research
techniques, and research designs. In: EdwardsPKO’MahoneyJVincentS (eds) Studying Organizations Using Critical Realism A
Practical Guide. Oxford:
Oxford University Press,
21–45.
2.
AguinisHEdwardsJRBradleyKJ (2016) Improving our understanding
of moderation and mediation in strategic management
research. Organizational Research Methods. Epub
ahead of print 27January2016. DOI: 10.1177/1094428115627498.
3.
BhaskarR (1975) A Realist Theory of
Science. Leeds: Leeds
Books Limited.
4.
BhaskarR (1978) On the possibility of social
scientific knowledge and the limits of naturalism.
Journal for the Theory of Social Behaviour8(1):
1–28.
5.
BhaskarR (1998) The Possibility of
Naturalism. London:
Routledge.
6.
BornmannLMutzR (2015) Growth rates of modern
science: A bibliometric analysis based on the number of publications and
cited references. Journal of the Association for
Information Science and Technology6(11):
2215–2222.
7.
CollierA (1994) Critical Realism: An Introduction to
Roy Bhaskar’s Philosophy. London:
Verso.
8.
CollinsonMReesCEdwardsPK. (1998) Involving Employees in Total
Quality Management: Employee Attitudes and Organisational Context in
Unionised Environments. London:
Department of Trade and
Industry.
9.
CowtonCJ (1998) The use of secondary data in
business ethics research. Journal of Business
Ethics17(4):
423–434.
10.
CoyneJCHagedoornMThombsB (2011) Most published and
unpublished dissertations should be excluded from meta-analyses: Comment on
Moyer et al. Psycho-Oncology20(2):
224–225.
11.
DanermarkBEkstromMJakobsenL (2002) Explaining Society: An Introduction
to Critical Realism in the Social Sciences.
Abingdon:
Routledge.
12.
DirpalG (2015) Human resource management practices
and performance link: Applying critical realist meta-theory.
Unpublished doctoral dissertation, University of the West of
England.
13.
DonaldsonL (1996) For Positivist Organization
theoryLondon:
SAGE.
14.
DonaldsonL (2003) Organization theory as a
positive science. In: KnudsenCTsoukasH (eds) The Oxford Handbook of Organization
Theory. Oxford:
Oxford University Press,
39–62.
15.
DonaldsonL (2005) Vita contemplativa following
the scientific method: How I became a committed functionalist and
positivist. Organization Studies26(7):
1071–1088.
16.
DownwardP (2016) Regression analysis: A
review. In: LeeFCroninB (eds) Handbook of Research Methods and Applications in
Heterodox Economics. London:
Elgar,
210–256.
17.
EdenD (2002) From the editors.
Replication, MA and scientific progress, and AMJ’s publication
policy. Academy of Management Journal45(5):
841–846.
18.
EdwardsPKO’MahoneyJVincentS (eds) (2014) Studying Organizations Using
Critical Realism: A Practical Guide.
Oxford: Oxford University
Press.
19.
Elder-VassD (2010) The Causal Power of Social
Structures: Emergence, Structure and Agency.
Cambridge: Cambridge
University Press.
20.
EllisPD (2010) The Essential Guide to Effect Sizes:
Statistical Power, Meta-analysis, and the Interpretation of Research
Results. Cambridge:
Cambridge University
Press.
21.
FiskeDW (1982) Convergent-discriminant
validation in measurements and research strategies.
New Directions for Methodology of Social & Behavioral
Science12: 77–92.
22.
FleetwoodS (2009) The ontology of things,
properties and powers. Journal of Critical
Realism8(3):
343–366.
23.
FleetwoodS (2012) Laws and tendencies in
Marxist political economy. Capital &
Class36(2):
235–262.
24.
FleetwoodS (2014) Bhaskar and critical
realism. In: AdlerPDu GayPMorganG. (eds) The Oxford Handbook of Sociology, Social Theory
and Organisation Studies: Contemporary Currents.
Oxford: Oxford University
Press,
182–219.
25.
FleetwoodS (2016) The critical realist
conception of open and closed systems. Journal of
Economic Methodology. Epub ahead of print 10August2016. DOI: 10.1080/1350178X.2016.1218532.
26.
FleetwoodSHeskethA (2006) HRM-performance research:
Under-theorized and lacking explanatory power. The
International Journal of Human Resource Management17(12):
1977–1993.
27.
FleetwoodSHeskethA (2010) Explaining the Performance of Human
Resource Management. Cambridge:
Cambridge University
Press.
28.
FrankeGR (2001) Applications of meta-analysis
for marketing and public policy: A review. Journal
of Public Policy & Marketing20(2):
186–200.
29.
GielenACKerkhofsMJVan OursJC (2010) How performance related pay
affects productivity and employment. Journal of
Population Economics23(1):
291–301.
HodgkinsonGPRousseauDM (2009) Bridging the rigour–relevance
gap in management research: It’s already happening!Journal of Management Studies46(3):
534–546.
32.
HoughSLHallBW (1994) Comparison of the Glass and
Hunter-Schmidt meta-analytic techniques. The Journal
of Educational Research87(5):
292–296.
33.
JohnsonPDuberleyJ (2000) Understanding Management Research:
An Introduction to Epistemology.
London:
SAGE.
34.
LawsonT (1995) A realist perspective on
contemporary ‘economic theory’. Journal of Economic
Issues29(1):
1–32.
35.
LawsonT (1997) Economics and Reality.
London:
Routledge.
LawsonT (2009) Applied economics, contrast
explanation and asymmetric information. Cambridge
Journal of Economics33(3):
405–419.
38.
LewisP (1999) Metaphor and critical
realism. In: FleetwoodS (ed.) Critical Realism in Economics: Development and
Debate. London:
Routledge,
83–102.
39.
LindenBPriestleyJ (2009) Critical reading of
meta-analyses. British Journal of Cardiac
Nursing4(7):
335–337.
40.
MillerGA (1987) Meta-analysis and the
culture-free hypothesis. Organization
Studies8(4):309–326.
41.
O’MahoneyJ (2011) Embracing essentialism: A
realist critique of resistance to discursive power.
Organization19(6):
723–774.
42.
PawsonR (2002) Evidence-based policy: The
promise of realist synthesis. Evaluation8(3):
340–358.
43.
PawsonR (2004) Evidence based policy: A
realist perspective. In: CarterBNewC (eds) Making Realism Work: Realist Social Theory and
Empirical Research. London:
Routledge,
26–50.
44.
PawsonR (2005) Evidence based policy: A
realist perspective. In: CarterBNewC (eds) Making Realism Work: Realist Social Theory and
Empirical Research. London:
Routledge,
1–49.
45.
PawsonRGreenhalghTHarveyG. (2005) Realist review: A new method
of systematic review designed for complex policy
interventions. Journal of Health Services Research
& Policy10(suppl. 1):
21–34.
46.
PorporaDV (2015) Reconstructing Sociology: The
Critical Realist Approach.
Cambridge: Cambridge University
Press.
47.
ReichlCLeiterMPSpinathFM (2014) Work–nonwork conflict and
burnout: A meta-analysis. Human Relations67(8):
979–1005.
48.
RossiJS (1987) How often are our statistics
wrong? A statistics class exercise. Teaching of
Psychology14(2):
98–101.
49.
RothPLPurvisKLBobkoP (2012) A meta-analysis of gender
group differences for measures of job performance in field
studies. Journal of Management38(2):
719–739.
50.
RousseauDMManningJDenyerD (2008) Evidence in management and
organizational science: Assembling the field’s full weight of scientific
knowledge through syntheses. The Academy of
Management Annals2(1):
475–515.
51.
SayerA (2010) Method in Social Science: Revised
2nd Edition. Abingdon:
Routledge.
52.
ShiromAGilboaSSFriedY. (2008) Gender, age and tenure as
moderators of work-related stressors’ relationships with job performance: A
meta-analysis. Human Relations61(10):
1371–1398.
53.
Sims-SchoutenWRileyS (2014) Employing a form of critical
realist discourse analysis for identity research: An example from women’s
talk of motherhood, childcare and employment. In:
EdwardsPO’MahoneyJVincentS (eds) Using Critical Realism to Study
Organisations. Oxford:
Oxford University Press,
46–65.
54.
SmithCElgerT (2014) Critical realism and
interviewing subjects. In: EdwardsPO’MahoneyJVincentS (eds) Using Critical Realism to Study
Organisations. Oxford:
Oxford University Press,
109–131.
55.
SubramonyM (2009) A meta-analytic investigation
of the relationship between HRM bundles and firm
performance. Human Resource Management48(5):
745–768.
56.
ThomasRDaviesA (2005) Theorizing the micro-politics
of resistance: New public management and managerial identities in the UK
public services. Organization Studies26(5):
683–706.
57.
TranfieldDDenyerDSmartP (2003) Towards a methodology for
developing evidence-informed management knowledge by means of systematic
review. British Journal of Management14(13):
207–222.
58.
VincentS (2011) The emotional labour process:
An essay on the economy of feelings. Human
Relations64(10):
1369–1392.
59.
WeedM (2005) Research synthesis in sport
management: Dealing with ‘Chaos in the Brickyard’.
European Sport Management Quarterly5(1):
77–90.
