As the availability of online data grows rapidly, researchers are confronted with a pressing question: How should social scientists collect Internet data for research? This study focuses on one of the most commonly used data collection techniques: web scraping. Going beyond canned approaches by leveraging a general framework of data communication, this study illustrates how online information can be systematically queried and fetched for reproducible research. To generalize our approaches, we additionally explore the variations in site security and architecture that analysts may encounter during the scraping process before they are given access to the desired data. The approaches we introduce do not rely on any proprietary software and can be easily implemented on any computing platform with programming languages such as Python or R. The methodological discussion in this study is meant to be applicable to current web-based research efforts. We include three examples with complete Python implementation. We also present an integrated workflow that enables researchers to produce analytical data sets that are traceable and thus verifiable for analysis or replication. Lastly, options related to the validity and efficiency of data are discussed, and we highlight the ongoing debate surrounding the ethics of online data collection, ultimately advocating for the fair use of online data.
AbbottA. (2016). Processual sociology. Chicago, IL: University of Chicago Press.
2.
AndersonC. J.BahníkŠ.Barnett-CowanM.BoscoF. A.ChandlerJ.ChartierC. R.ZuniK. (2016). Response to comment on “estimating the reproducibility of psychological science.”Science, 351, 1037. doi:10.1126/science.aad9163
3.
BainbridgeW. S. (2007). Computational sociology. In RitzerG. (Ed.), The Blackwell encyclopedia of sociology. doi:10.1111/b.9781405124331.2007.x
4.
BarbaL. A. (2018). Terminologies for reproducible research. Computing Research Repository, abs/1802.03311 (arXiv:1802.03311).
BlankG. (2017). The digital divide among Twitter users and its implications for social research. Social Science Computer Review, 35, 679–697. doi:10.1177/0894439316671698
7.
BonzaniniM. (2016). Mastering social media mining with Python. Birmingham, England: Packt.
8.
BoydD.CrawfordK. (2012). Critical questions for big data. Information, Communication & Society, 15, 662–679.
9.
CamererC. F.DreberA.ForsellE.HoT. H.HuberJ.JohannessonM.WuH. (2016). Evaluating replicability of laboratory experiments in economics. Science, 351, 1433–1436. doi:10.1126/science.aaf0918
10.
ChristensenG. S.MiguelE. (2016, December). Transparency, reproducibility, and the credibility of economics research (Working Paper No. 22989). National Bureau of Economic Research. doi:10.3386/w22989
11.
CrawfordK.MiltnerK.GrayM. L. (2014). Critiquing big data: Politics, ethics, epistemology. International Journal of Communication, 8, 1663–1672.
FryeC.PluschM.LiebermanH. (2003). Static and dynamic semantics of the web. In W. Wahlster (Ed.), Spinning the semantic web (pp. 376–401). Cambridge, MA: MIT Press.
14.
GilbertD. T.KingG.PettigrewS.WilsonT. D. (2016). Comment on “estimating the reproducibility of psychological science.” Science, 351, 1037–1037. doi:10.1126/science.aad7243
15.
HerndonJ.O’ReillyR. (2016). Data sharing policies in social sciences academic journals: Evolving expectations of data sharing as a form of scholarly communication. In KellemL. M.ThompsonK. (Eds.), Databrarianship: The academic data librarian in theory and practice (pp. 219–243). Chicago, IL: American Library Association.
16.
IgnatowG.MihalceaR. (2017). An introduction to text mining: research design, data collection, and analysis. Thousand Oaks, CA: Sage.
17.
IliadisA.RussoF. (2016). Critical data studies: An introduction. Big Data & Society. doi:10.1177/2053951716674238
18.
JanetzkoD. (2017). The role of APIs in data sampling from social media. In SloanL.Quan-HaaseA. (Eds.), The Sage handbook of social media research methods (Chapter 10). Thousand Oaks, CA: Sage.
LandersR. N.BrussoR. C.CavanaughK. J.CollmusA. B. (2016). A primer on theory-driven web scraping: Automatic extraction of big data from the Internet for use in psychological research. Psychological Methods, 21, 475–492.
22.
LazerD.RadfordJ. (2017). Data ex Machina: Introduction to big data. Annual Review of Sociology, 43, 19–39. doi:10.1146/annurev-soc-060116-053457
23.
LessigL. (2002). The future of ideas: The fate of the commons in a connected world. New York, NY: Vintage Press.
24.
McFarlandD. A.LewisK.GoldbergA. (2016). Sociology in the era of big data: The ascent of forensic social science. The American Sociologist, 47, 12–35.
25.
McGrathR. (2019). Twython (Version: 3.7.0): Pure Python wrapper for the Twitter API [Computer software]. Retrieved fromhttps://github.com/ryanmcgrath/twython
26.
MetcalfJ.CrawfordK. (2016). Where are human subjects in big data research? The emerging ethics divide. Big Data & Society, 3. doi:10.1177/2053951716650211
27.
MitchellR. (2018). Web scraping with Python: Collecting more data from the modern web (2nd ed.). Sebastopol, CA: O’Reilly Media.
28.
MoranT. P. (2005). The sociology of teaching graduate statistics. Teaching Sociology, 33, 263–284. doi:10.1177/0092055X0503300303
29.
MunzertS.RubbaC.MeissnerP.NyhuisD. (2015). Automated data collection with R: A practical guide to web scraping and text mining. Hoboken, NJ: Wiley.
National Academies of Sciences, Engineering, and Medicine. (2019). Reproducibility and replicability in science. Washington, DC: The National Academies Press.
32.
PengR. D. (2009). Reproducible research and biostatistics. Biostatistics, 10, 405–408. doi:10.1093/bio-statistics/kxp014
33.
PengR. D. (2011). Reproducible research in computational science. Science, 334, 1226–1227. doi:10.1126/science.1213847
XinY.CaiT. (2019). Paying money for freedom: Effects of monetary compensation on sentencing for criminal traffic accident offenses in China. Journal of Quantitative Criminology. doi:10.1007/s10940-019-09409-w
40.
ZimmerM. (2010). “But the data is already public”: On the ethics of research in Facebook. Ethics and Information Technology, 12, 313–325. doi:10.1007/s10676-010-9227-5
