Many refinements of statistical design have been offered to solve the replication problem identified by the Open Science Collaboration and Camerer and colleagues. There are, however, two distinct kinds of experimentation: Fisher design and theory designed. Therefore, there are two kinds of replication. Only for the Fisher design does replication reproduce conditions of prior experiments in order to compare new with prior results, and only there has a replication problem been demonstrated. In contrast, replications for theory-designed experiments test experimental results against theoretical predictions, and only for theory-designed experiments can replication be extended broadly across the scope of a theory. The authors analyze the logic of the two types of experiments as well as hybrids that mix qualities of both.
AbbottAndrew. 2007. “Notes on Replication.”Sociological Methods and Research36(2):210–18.
2.
AckermanJoshua M.NoceraChristopher C.BraghJohn A.2010. “Incidental Haptic Sensations Influence Social Judgements and Decisions.”Science328(5986):1712–15.
3.
Archimedes. [232 bce] 1897. The Works of Archimedes. New York: Dover.
4.
AtmanspacherHarald. 2016. “Introductory Remarks to Part III.” Pp. 201–204 in Reproducibility: Principles, Problems, Practices and Prospects, edited by AtmanspacherH.MaasenS.Hoboken, NJ: John Wiley.
5.
AviezerHillelTropeYaacovTodorovAlexander. 2012. “Body Cues, Not Facial Expressions Discriminate between Positive and Negative Emotions.”Science338(6111):1225–29.
6.
BabbieEarl R.2013. The Practice of Social Research. Belmont, CA: Wadsworth Cengage Learning.
BalafoutasLoukasSutterMatthias. 2012. “Affirmative Action Policies Promote Women and Do Not Harm Efficiency in the Laboratory.”Science335(6068):579–82.
BoringEdwin G.1954. “The Nature and History of Experimental Control.”American Journal of Psychology67(4):573–89.
11.
CamererColin F.DreberAnnaForsellEskilHoTeck-HuaHuberJürgenJohannessonMagnusKirchlerMichael, et al. 2016. “Evaluating Replicability of Laboratory Experiments in Economics.”Science351(6280):1433–36.
12.
CampbellDonald T.StanleyJulian C.1963. Experimental and Quasi-experimental Designs for Research. Chicago: Rand McNally.
13.
ColemanRenita. 2019. Designing Experiments for the Social Sciences. Thousand Oaks, CA: Sage.
14.
DerexMaximeBeuginMarie-PaulineGodelleBernardRaymondMichel. 2013. “Experimental Evidence for the Influence of Group Size on Cultural Complexity.”Nature503(7476):485–87.
EarpBrian D.TrafimowDavid. 2015. “Replication, Falsification, and the Crisis of Confidence in Social Psychology.”Frontiers in Psychology6:621.
17.
EinsteinAlbertInfeldLeopold. 1938. The Evolution of Physics. New York: Simon & Schuster.
18.
EmanuelsonPamelaWillerDavid. 2018. “Social Structures in Transition: Application of Two Theories to Chiefdoms.”Socius4:1–14.
19.
FaradayMichael. 1859. Experimental Research in Chemistry and Physics. London: Taylor & Francis.
20.
FirebaughGlenn. 2007. “Replication Data Sets and Favored-Hypothesis Bias.”Sociological Methods and Research36(2):200–209.
21.
FisherRonald A.1935. The Design of Experiments. London: Oliver & Boyd.
22.
FreeseJeremy. 2007. “Replication Standards for Quantitative Social Science.”Sociological Methods and Research36(2):153–72.
23.
FriedmanDanielSunderShyman. 1994. Experimental Methods: A Primer for Economists. Cambridge, UK: Cambridge University Press.
24.
GalileoGalilei. [1665]1954. Dialogues Concerning Two New Sciences. New York: Dover.
25.
GervaisWillNorenzayanAra. 2012. “Analytic Thinking Promotes Religious Disbelief.”Science336(6080):493–96.
26.
GneezyUriKeenanElizabeth A.GneezyAyelet. 2012. “Avoiding Overhead Aversion in Charity.” Science346(6209):632–35.
27.
HauserOliverRandDavidPeysakhovichAlexanderNowakMartin. 2014. “Cooperating with the Future.”Nature511(7508):220–23.
28.
HedströmPeterBearmanPeter. 2009. “What Is Analytic Sociology All About?” Pp. 3–24 in The Oxford Handbook of Analytical Sociology, edited by HedströmP.BearmanP.Oxford, UK: Oxford University Press.
29.
JacksonMichelleCoxD. R.2013. “The Principles of Experimental Design and Their Application in Sociology.”Annual Review of Sociology39:27–49.
30.
JassenMarco A.HolahanRobertLeeAllenOstromElinor. 2010. “Lab Experiments for the Study of Social-Economic Systems.”Science328(5978):613–17.
31.
KarpickeJeffrey D.BluntJanell R.2011. “Retrieval Practice Produces More Learning Than Elaborate Studying with Concept Mapping.”Science331(6018):772–75.
32.
KiddDavidCastanoEmanuele. 2013. “Reading Literary Fiction Improves Theory of Mind.”Science342(6156):377–80.
33.
KovacsÁgnesTéglásErnőEndressAnsgar Denis. “The Social Sense: Susceptibility to Other’s Beliefs in Human Infants and Adults.”Science330(6012):1830–34.
34.
LeeSpike W. S.SchwarzNorbert. 2010. “Washing Away Postdecisional Dissonance.”Science328(5979):709.
35.
LiebersonStanleyLynnFreda B.2002. “Barking up the Wrong Branch: Scientific Alternatives to the Current Model of Sociological Science.”Annual Review Sociology28:1–19.
36.
LiuJunWillerDavidEmanuelsonPamela. 2011. “Network Structure and Power Distribution: Explanations from Elementary Theory.”Social Research2(1):134–66.
37.
LiuJunWillerDavidEmanuelsonPamela. 2013. “Coercive Structures: Theory and Testing.”Chinese Journal of Sociology33:176–92.
38.
LiuJunYanZhao. 2014. “Replication Studies of NET.”Social Science in Ning Xia3:58–63.
39.
LucasJeffreyMorrellKevinPosardMarek. 2013. “Considerations on the ‘Replication Problem’ in Sociology.”American Sociologist44(2):217–32.
40.
MarxKarl. [1867] 1967. Capital. New York: International.
41.
MertonRobert K. [1948] 1968. SocialTheory and Social Structure. New York: Free Press.
42.
MertonRobert K.1967. “On Sociological Theories of the Middle Range.” Pp. 39–72 in On Theoretical Sociology. New York: Free Press.
43.
MillJohn Stuart. [1843] 1967. A System of Logic. London: Longmans, Green.
44.
MorewedgeCarey K.HuhYoung EunVosgerauJoachim. 2010. “Thought for Food: Imagined Consumption Reduces Actual Consumption.”Science330(6010):1530–33.
45.
MuthukrishnaMichaelHenrichJoseph. 2019. “A Problem in Theory.”Nature: Human Behavior3:221–29.
46.
NewtonIsaac. [1729] 1966. Mathematical Principles of Natural Philosophy and His System of the World. Berkeley: University of California Press.
47.
NishiAkihiroShiradoHirokazuRandDavidChristakisNicholas. 2015. “Inequality and Visibility of Wealth in Experimental Social Networks.”Nature526(7573):426–31.
48.
Open Science Collaboration. 2015. “Estimating the Replication of Psychological Science.”Science349(6251):1–8.
49.
PopperKarl. [1934] 1959. The Logic of Scientific Discovery. London: Hutchinson.
50.
PopperKarl. 1957. The Poverty of Historicism. New York: Routledge.
RosenthalRobert. 1979. “The File Drawer Problem and Tolerance for Null Results.”Psychological Bulletin86(3):638–41.
57.
ShahAnuj K.MullainathanSendhilShafirEldar. 2012. “Some Consequences of Having Too Little.”Science338(6107):682–85.
58.
SimpsonBrentWillerDavid. 2002. “Applications of Elementary Theory to the Social Structures of Antiquity.” Pp. 231–54 in Structure, Culture and History: Recent Issues in Social Theory, edited by ChewS. C.KnottnerusJ. D.Lanham, MD: Rowman & Littlefield.
59.
SmithVernon. 1989. “Theory, Experiment and Economics.”Journal of Economic Perspectives3(1):151–69.
60.
SparrowBetsyLiuJennyWegnerDaniel M.2011. “Google Effects on Memory: Cognitive Consequences of Having Information at Our Fingertips.”Science333(6043):776–78.
61.
StarbuckWilliam H.2016. “60th Anniversary Essay: How Journals Could Improve Practices in Social Sciences.”Administrative Science Quarterly61(2):165–83.
62.
SteinleFredrich. 2016. “Stability and Replication of Experimental Results: A Historical Perspective” Pp. 39–64 in Reproducibility: Principles, Problems, Practices and Prospects, edited by AtmanspacherH.MaasenS.Hoboken, NJ: John Wiley.
63.
ThyeShane. 2014. “Logical and Philosophical Foundations of Experimental Research in the Social Sciences.” Pp. 53–82 in Laboratory Experiments in the Social Sciences, edited by WebsterM.Jr.SellJ.Amsterdam: Elsevier.
64.
ThyeShaneWillerDavidMarkovskyBarry. 2006. “From Status to Power: New Models at the Intersection of Two Theories.”Social Forces84(3):1471–96.
65.
van BavelJay J.Mende-SiedleckiPeterBradyWilliam J.ReineroDiego J.2015. “Contextual Sensitivity in Scientific Replication.”Proceedings of the National Academy of Sciences113(23):6454–59.
66.
WalkerHenryWillerDavid. 2007. “Experiments and the Science of Sociology.” Pp. 25–55 in Laboratory Experiments in the Social Sciences, edited by WebsterM.SellJ.San Diego, CA: Elsevier.
67.
WalkerHenry A.WillerDavid. 2006. “Experimental Methods (Social Psychology).” In Encyclopedia of Sociology, edited by RitzerG.Malden, MA: Blackwell.
68.
WalkerHenryWillerDavid. 2014. “Legitimating Collective Action and Countervailing Power.”Social Forces92(3):1217–39.
69.
WalkerLisa Slattery. 2014. “Developing Your Experiment.” Pp. 127–44 in Laboratory Experiments in the Social Sciences, edited by Webster, Jr.M.Jr.SellJ.Amsterdam: Elsevier.
70.
WatsonJames A.1969. The Double Helix. New York: Mentor.
71.
WeberMax. [1918] 1968. Economy and Society. Berkeley: University of California Press.
72.
WeberMax. [1920] 1950. “The Social Causes for the Decay of Ancient Civilization.”Journal of General Education1:75–88.
73.
WillerDavid. 1984. “Analysis and Composition as Theoretic Procedures.”Journal of Mathematical Sociology10:241–70.
74.
WillerDavidEmanuelsonPamelaLovagliaMichael J.SimpsonBrentThyeShane R.WalkerHenry A.CorraMamadi, et al. 2014. “Elementary Theory: 25 Years of Expanding Scope and Increasing Precision.”Advances in Group Processes31:175–217.
75.
WillerDavidLovagliaMichaelMarkovskyBarry. 1997. “Power and Influence: A Theoretical Bridge.”Social Forces76(2):571–604.
76.
WillerDavidSzmatkaJacek. 1993. “Cross-National Experimental Investigations of Elementary Theory: Implications for the Generality of Theory, and the Autonomy of Social Structure.”Advances in Group Process10:37–81.
77.
WillerDavidWalkerHenry. 2007. Building Experiments: Testing Social Theory. Stanford, CA: Stanford University Press.
78.
WillerDavidWebsterMurray. 1970. “Theoretical Concepts and Observables.”American Sociological Review35(4):748–57.
79.
WillerDavidWillerJudith. 1973. Systematic Empiricism: Critique of a Pseudoscience. Englewood Cliffs, NJ: Prentice Hall.
80.
WilsonT.ReinhardDavid A.WestgateErin C.GilbertDaniel T.EllerbeckNicoleHahnCherylBrownCasey L., et al. 2014. “Just Think: The Challenge of the Disengaged Mind.” Science345(6192):75–77.
81.
ZhaoYanLiuJun. 2015“The Effects of Hierarchical Power and Structural Power on Resource Allocation.”China Studies2.
82.
ZhaoYanLiuJun. 2016. “Resource Distribution under Hierarchical Power and Structural Power Conditions.”China Studies23:112–28.
