In this article, we examine whether cross-national studies disclose enough information for independent researchers to evaluate the validity and reliability of the findings (evaluation transparency) or to perform a direct replication (replicability transparency). The first contribution is theoretical. We develop a heuristic theoretical model including the actors, factors, and processes that influence the transparency of cross-national studies and provide an overview of the measures currently taken to improve research transparency. The second contribution is empirical, in which we analyze the level of transparency in published cross-national studies. Specifically, using a random sample of 305 comparative studies published in one of 29 peer-reviewed social sciences journals (from 1986 to 2016), we show that, even though all the articles include some methodological information, the great majority lack sufficient information for evaluation and replication. Lastly, we develop and propose a set of transparency guidelines tailored for reporting cross-national survey research.
BakerMonya. 2016. “Is There a Reproducibility Crisis?” Nature533:452–54.
5.
BerryJohn W.PoortingaYpe W.SegallMarshall H.DasenPierre R.. 1992. Cross-cultural Psychology: Research and Applications. 2nd ed. New York: Cambridge University Press.
6.
BethlehemJelkeSchoutenBarry. 2010. “Nonresponse Error: Detection and Correction.” Pp. 558–78 in The SAGE Handbook of Survey Methodology, edited byWolfChristofJoyeDominiqueSmithTom W.Yang-chihFu. Los Angeles, CA: Sage.
7.
BillietJaakKochAchimPhilippensMichel. 2007. “Understanding and Improving Response Rates.” Pp. 113–38 in Measuring Attitudes Cross-Nationally: Lessons from the European Social Survey, edited byJowellRogerRobertsCarolineFitzgeraldRoryEvaGillian. Los Angeles, CA: Sage.
8.
BreznauNate. 2016. “Secondary Observer Effects: Idiosyncratic Errors in Small-N Secondary Data Analysis.” International Journal of Social Research Methodology19:301–18.
9.
BrooksClemManzaJeff. 2006. “Social Policy Responsiveness in Developed Democracies.” American Sociological Review71:474–94.
10.
CarseyThomas M.2014. “Making DA-RT a Reality.” Political Science and Politics47:72–77.
11.
Committee on Publication Ethics. 2011. Code of Conduct and Best Practice Guidelines for Journal Editors. London, England: Author.
12.
DafoeAllan. 2014. “Science Deserves Better: The Imperative to Share Complete Replication Files.” Political Science & Politics47:60–66.
FanelliDaniele. 2009. “How Many Scientists Fabricate and Falsify Research? A Systematic Review and Meta-analysis of Survey Data.” PLoS One4:e5738.
15.
FirebaughGlenn. 2007. “Replication Data Sets and Favored-hypothesis Bias. Comment on Jeremy Freese (2007) and Gary King (2007).” Sociological Methods & Research36:200–09.
16.
Fortin-RittbergerJessicaHowellDavidQuinlanStephenTodosijevicBojan. 2016. “Supplementing Cross-national Survey Data with Contextual Data.” Pp. 670–80 in The Sage Handbook of Survey Methodology, edited byWolfChristofJoyeDominiqueSmithTom W.FuYang-chih. Los Angeles, CA: Sage.
17.
FreelonDeen. 2010. “ReCal: Intercoder Reliability Calculation as a Web Service.” International Journal of Internet Science5:20–33.
18.
FreeseJeremy. 2007. “Replication Standards for Quantitative Social Science: Why not Sociology?” Sociological Methods & Research36:153–72.
19.
GelmanAndrewLokenEric. 2014. “The Statistical Crisis in Science. Data-dependent Analysis—A ‘Garden of Forking Paths’— Explains Why Many Statistically Significant Comparisons Don’t Hold up.” American Scientist102:460–65.
HäderSabineLynnPeter. 2007. “How Representative Can a Multi-nation Survey Be?” Pp. 33–52 in Measuring Attitudes Cross-nationally: Lessons from the European Social Survey, edited byJowellRogerRobertsCarolineFitzgeraldRoryEvaGillian. Los Angeles, CA: Sage.
22.
HarknessJanet A.1991. “In Pursuit of Quality: Issues for Cross-national Survey Research.” International Journal of Social Research Methodology2:125–40.
23.
HarknessJanet A.BraunMichaelEdwardsBradJohnsonTimothy P.LybergLars E.MohlerPeter P.PennellBeth-EllenSmithTom W.. 2010. Survey Methods in Multinational, Multiregional, and Multicultural Contexts. Hoboken, NJ: Wiley.
24.
HarknessJanet A.van de VijverFons J. R.MohlerPeter P.. 2003. Cross-cultural Survey Methods. New York: Wiley.
25.
HöfflerJan H.2017. “ReplicationWiki—Improving Transparency in Social Sciences Research.” D-Lib Magazine34:1–6.
26.
HuizengaHilde M.WetzelsRuudvan RavenzwaaijDonWagenmakersEric-Jan. 2012. “Four Empirical Tests of Unconscious Thought Theory.” Organizational Behavior and Human Decision Processes117:332–40.
27.
IoannidisJohn P. A.2005. “Why Most Published Research Findings Are False.” PLoS Medicine2:696–701.
28.
IshiyamaJohn. 2014. “Replication, Research Transparency, and Journal Publications: Individualism, Community Models, and the Future of Replication Studies.” PS: Political Science & Politics47:78–83.
29.
JohnLeslie K.LoewensteinGeorgePrelecDrazen. 2012. “Measuring the Prevalence of Questionable Research Practices with Incentives for Truth Telling.” Psychological Science23:524–32.
30.
KingGary. 1995. “Replication, Replication.” PS: Political Science & Politics28:444–52.
31.
LavallePierreBeaumontJean-Francois. 2010. “Weighting: Principles and Practicalities.” Pp. 460–76 in The Sage Handbook of Survey Methodology, edited byWolfChristofJoyeDominiqueSmithTom W.FuYang-chih. Los Angeles, CA: Sage.
32.
LavrakasPaul J.2008. Encyclopedia of Survey Research Methods. Thousand Oaks, CA: Sage.
33.
LucasJeffrey W.MorrellKevinPosardMarek. 2013. “Considerations on the ‘Replication Problem’ in Sociology.” The American Sociologist44:217–32.
34.
LupiaArthurElmanColin. 2014. “Openness in Political Science: Data Access and Research Transparency.” PS: Political Science & Politics47:19–42.
35.
LynnPeter. 2003. “Developing Quality Standards for Cross-national Survey Research: Five Approaches.” Social Research Methodology6:323–36.
36.
LynnPeterJapecLilliLybergLars. 2007. “What’s So Special about Cross-national Surveys?” Pp. 7–20 in Conducting Cross-national and Cross-cultural Surveys, edited byHarknessJanet. Mannheim, Germany: ZUMA.
37.
MaynardMarcTimms-FerraraLois. 2011. “Methodological Disclosure Issues and Opinion Data.” Journal of Economic and Social Measurement36:19–32.
38.
MertonRobert K.1973. The Sociology of Science: Theoretical and Empirical Investigations. Chicago: University of Chicago Press.
39.
MoreyRichard D.ChambersChristopher D.EtchellsPeter J.HarrisChristine R.HoekstraRinkLakensDaniëlLewandowskyStephanMoreyCandice CokerNewmanDaniel P.SchönbrodtFelix D.VanpaemelWolfWagenmakersEric-JanZwaanRolf A.. 2016. “The Peer Reviewers’ Openness Initiative: Incentivizing Open Research Practices through Peer Review.” Royal Society Open Science3:150547.
40.
MunafòMarcus R.BishopDorothy V. M.ButtonKatherine S.ChambersChristopher D.Du SertNathalie PercieSimonsohnUriWagenmakersEric-JanWareJennifer J.IoannidisJohn P. A.. 2017. “A Manifesto for Reproducible Science.” Nature Human Behaviour1:1–9.
41.
NelsonLeif D.SimmonsJosephSimonsohnUri. 2017. “Psychology’s Renaissance.” Annual Review of Psychology69:17.1–17.24.
42.
Open Science Collaboration. 2015. “Estimating the Reproducibility of Psychological Science.” Science349:aac4716.
43.
RubinDonald B.1976. “Inference and Missing Data.” Biometrika63:581–92.
44.
SiegriedGablerHäderSabine. 2010. “Special Challenges of Sampling for Comparative Surveys.” Pp. 346–56 in The Sage Handbook of Survey Methodology, edited byWolfChristofJoyeDominiqueSmithTom W.FuYang-chih. Los Angeles, CA: Sage.
45.
SilberzahnRaphaelUhlmannEric L.MartinDaniel P.AnselmiPasqualeAustFrederikAwtreyEli C.BahníkŠtěpánBaiF.BannardC.BonnierE.CarlssonR.CheungF.ChristensenG.ClayR.CraigM. A.Dalla RosaA.DamL.EvansM. H.Flores CervantesI.FongN.Gamez-DjokicM.GlenzA.Gordon-McKeonS.HeatonT. J.HederosK.HeeneM.Hofelich MohrA. J.HögdenF.HuiK.JohannessonM.KalodimosJ.KaszubowskiE.KennedyD. M.LeiR.LindsayT. A.LiveraniS.MadanC. R.MoldenD.MollemanE.MoreyR. D.MulderL. B.NijstadB. R.PopeN. G.PopeB.PrenoveauJ. M.RinkF.RobustoE.RoderiqueH.SandbergA.SchlüterE.SchönbrodtF. D.ShermanM. F.SommerS. A.SotakK.SpainS.SpörleinC.StaffordT.StefanuttiL.TauberS.UllrichJ.VianelloM.WagenmakersE.-J.WitkowiakM.YoonS.NosekB. A.. 2018. “Many Analysts, One Dataset: Making Transparent How Variations in Analytical Choices Affect Results.”Advances in Methods and Practices in Psychological Science1:337–56.
46.
SimmonsJosephNelsonLeif D.SimonsohnUri. 2011. “False-positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allow Presenting Anything as Significant.” Psychological Science22:1359–66.
47.
SmithTom W.FuYang-chih. 2010. “The Globalization of Surveys.” Pp. 680–92 in The Sage Handbook of Survey Methodology, edited byWolfChristofJoyeDominiqueSmithTom W.FuYang-chih. Los Angeles, CA: Sage.
48.
SpiessMartin. 2010. “Dealing with Missing Values.” Pp. 597–612 in The Sage Handbook of Survey Methodology, edited byWolfChristofJoyeDominiqueSmithTom W.FuYang-chih. Los Angeles, CA: Sage.
49.
SteneckNicholas H.2006. “Fostering Integrity in Research: Definitions, Current Knowledge, and Future Directions.” Science and Engineering Ethics12:53–74.
50.
StoopIneke A. L.2016. “Unit Nonresponse.” Pp. 409–24 in The Sage Handbook of Survey Methodology, edited byWolfChristofJoyeDominiqueSmithTom W.FuYang-chih. Los Angeles, CA: Sage.
51.
TilleYvesMateiAlina. 2010. “Basics of Sampling for Survey Research.” Pp. 311–28 in The Sage Handbook of Survey Methodology, edited byWolfChristofJoyeDominiqueSmithTom W.FuYang-chih. Los Angeles, CA: Sage.
52.
Van de VijverFons J. R.LeungKwok. 1997. Methods and Data Analysis for Cross-cultural Research. Thousand Oaks, CA: Sage.
53.
WichertsJelte M.VeldkampCoosje L.AugusteijnHilde E. M.BakkerMarjanvan AertRobbie C. M.van AssenMarcel A. L. M.. 2016. “Degrees of Freedom in Planning, Running, Analyzing, and Reporting Psychological Studies: A Checklist to Avoid p-Hacking.” Frontiers in Psychology7:1–12.
54.
WolfChristofJoyeDominiqueSmithTom W.FuYang-chih. 2016. The Sage Handbook of Survey Methodology. Los Angeles, CA: Sage.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
