Web scraping, a technique for extracting data from web pages, has been in use for decades, yet its utilization in the field of migration, mobility, and migrant integration studies has been limited. The field faces notorious limitations regarding data access and availability, particularly in low-income settings. Web scraping has the potential to provide new datasets for further qualitative and quantitative analysis. Web scraping requires no financial resources, is agnostic to epistemic divides in the field, reduces researcher bias, and increases transparency and replicability of data collection. As large providers of digital data such as Facebook or Twitter increasingly restrict access to their data for researchers, web scraping will become more important in the future and deserves its place in the toolbox of migration and mobility scholars. This short and nontechnical methods note introduces the fundamental concepts of web scraping, provides guidance on how to learn the technique, showcases practical applications of web scraping in the study of migrant populations, and discusses potential future use cases.
AlencarA.2019. “Digital Place-Making Practices and Daily Struggles of Venezuelan Refugees in Brazil.” In The SAGE Handbook of Media and Migration, edited by SmetsK.LeursK.GeorgiouM.WittebornS.GajjalaR., 503–514. London: Sage.
2.
Arcila CalderónCarlosBlanco-HerreroDavidValdez ApoloMaría Belén. 2020. “Rechazo Y Discurso De Odio En Twitter: Análisis De Contenido De Los Tuits Sobre Migrantes Y Refugiados En Español / Rejection and Hate Speech in Twitter: Content Analysis of Tweets about Migrants and Refugees in Spanish.” Revista Española de Investigaciones Sociológicas172 (1): 21–39.
3.
AzzolinaStefanoRazzaManuelSartianoKevinWeitschekEmanuel. 2021. “Price Discrimination in the Online Airline Market: An Empirical Study.” Journal of Theoretical and Applied Electronic Commerce Research16(6): 2282–2303.
4.
BilsborrowRichard E.HugoG.OberaiA. S.. 1997. International Migration Statistics: Guidelines for Improving Data Collection Systems.1. publ. Geneva: ILO.
5.
BlackJ.DeardenK.SingletonA.LaczkoFrank. 2017. Global Overview of the Available Data on Migrant Deaths and Disappearances: Fatal Journeys, Volume 3. Geneva: International Organization for Migration.
6.
BoegershausenJohannesDattaHannesBorahAbhishekStephenAndrew T.. 2022. “Fields of Gold: Scraping Web Data for Marketing Insights.” Journal of Marketing86(5): 1–20.
7.
BöhmeMarcus H.GrögerAndréStöhrTobias. 2020. “Searching for a Better Life: Predicting International Migration with Online Search Keywords.” Journal of Development Economics142: 102347.
8.
CesareNinaLeeHedwigMcCormickTylerSpiroEmmaZagheniEmilio. 2018. “Promises and Pitfalls of Using Digital Traces for Demographic Research.” Demography55 (5): 1979–99.
9.
CzymaraChristian S.Dochow-SondershausStephanDrouhotLucas G.SimsekMügeSpörleinChristoph. 2022. “Catalyst of Hate? Ethnic Insulting on YouTube in the Aftermath of Terror Attacks in France, Germany and the United Kingdom 2014–2017.” Journal of Ethnic and Migration Studies49 (2): 535–553.
10.
Dedecek GertzHelena. 2022. “Collecting Migrants’ Facebook Posts: Accounting for Ethical Measures in a Text-as-Data Approach.” Frontiers in Sociology7: 932908.
11.
DekkerRianneEngbersenGodfriedKlaverJeanineVonkHanna. 2018. “Smart Refugees: How Syrian Asylum Migrants Use Social Media Information in Migration Decision-Making.” Social Media + Society4(1): 205630511876443.
12.
DeVitoNicholas J.RichardsGeorgia C.InglesbyPeter. 2020. “How We Learnt to Stop Worrying and Love Web Scraping.” Nature585(7826): 621–22.
DrouhotLucas G.DeutschmannEmanuelZuccottiCarolina V.ZagheniEmilio. 2022. “Computational Approaches to Migration and Integration Research: Promises and Challenges.” Journal of Ethnic and Migration Studies49 (2): 389–407.
15.
EdelmanBenjamin.2012. “Using Internet Data for Economic Research.” Journal of Economic Perspectives26(2): 189–206.
16.
EdelmannAchimWolffTomMontagneDanielleBailChristopher A.. 2020. “Computational Social Science and Sociology.” Annual Review of Sociology.46(1): 61–81.
17.
EkmanMattias.2019. “Anti-Immigration and Racist Discourse in Social Media.” European Journal of Communication34(6): 606–18.
18.
EU Data Protection Supervisor. 2019. Formal Consultation on EASO’s Social Media Monitoring Reports (Case 2018–1083).Brussels.
19.
GollaS.MüllerD.. 2020a. Web Scraping Social Media: Legitimate Research or a Breach of Contract? PRIF. PRIF Blog.
20.
GollaS.MüllerD.. 2020b. Web Scraping Social Media: Pitfalls of Copyright and Data Protection Law:PRIF. PRIF Blog.
21.
HanS.AndersonC. K. 2021. “Web scraping for hospitality research: Overview, opportunities, and implications.” Cornell Hospitality Quarterly62(1): 89–104.
22.
HangartnerDominikGennaroGloriaAlasiriSaryBahrichNicholasBornhoftAlexandraBoucherJosephBuse DemirciBuket, et al.2021. “Empathy-Based Counterspeech can Reduce Racist Hate Speech in a Social Media Field Experiment.” Proceedings of the National Academy of Sciences of the United States of America118, 50.
23.
HeidenreichTobiasEberlJakob-MoritzLindFabienneBoomgaardenHajo. 2020. “Political Migration Discourses on Social Media: A Comparative Perspective on Visibility and Sentiment Across Political Facebook Accounts in Europe.” Journal of Ethnic and Migration Studies46(7): 1261–80.
24.
HillenJ.2019. “Web Scraping for Food Price Research.” British Food Journal121(12): 3350–3361.
25.
KrotovVlad.JohnsonLeigh. 2022. “Big Web Data: Challenges Related to Data, Technology, Legality, and Ethics.” Business Horizons66 (34): 481–491.
26.
LanHaiShaDexuanSrirenganathan MalarvizhiAnushaLiuYiLiYunMeisterNadineLiuQianWangZifuYangJingchaoYangChaowei Phil. 2021. “COVID-Scraper: An Open-Source Toolset for Automatically Scraping and Processing Global Multi-Scale Spatiotemporal COVID-19 Records.” IEEE Access: Practical Innovations, Open Solutions9: 84783–98.
27.
LandersRichard N.BrussoRobert C.CavanaughKatelyn J.CollmusAndrew B.. 2016. “A Primer on Theory-Driven Web Scraping: Automatic Extraction of Big Data from the Internet for Use in Psychological Research.” Psychological Methods21(4): 475–92.
LazerDavidPentlandAlexAdamicLadaAralSinanBarabasiAlbert-LaszloBrewerDevonChristakisNicholas, et al.2009. “Social Science. Computational Social Science.” Science323(5915): 721–23.
30.
LeursKoen.WittebornSaskia. 2021. “Digital Migration Studies.” In Research Handbook on International Migration and Digital Technology, edited by McAuliffeMarie, 15–29. Cheltenham: Edward Elgar Publishing.
31.
LiuX.2022. “China’s Anti-Unfair Competition Law on Data Crawling.” Advances in Economics, Business and Management Research219 (1): 406–413.
32.
MacanovicAna.2022. “Text Mining for Social Science - the State and the Future of Computational Text Analysis in Sociology.” Social Science Research108: 102784.
33.
MancosuM.VegettiF.. 2020. “What You Can Scrape and What Is Right to Scrape: A Proposal for a Tool to Collect Public Facebook Data.” Social Media + Society6(3): 1–11.
34.
MarresNoortjeWeltevredeEsther. 2013. “SCRAPING the SOCIAL?” Journal of Cultural Economy6(3): 313–35.
35.
MasakiT.MadsonB. “Data Gaps in Microdata in the Context of Forced Displacement.” Working Paper Prepared for the Workshop on Filling the Remaining Data Gaps in the Forced Displacement Context held on May 31 and June 1, 2023, at the World Bank Headquarters in Washington, DC (FORTHCOMING).
36.
MüllerKarsten.SchwarzCarlo. 2022. “From Hashtag to Hate Crime: Twitter and Anti-Minority Sentiment.” American Economic Journal: Applied Economics(FORTHCOMING).
37.
NalbandianLucia.2022. “An Eye for an ‘I:’ a Critical Assessment of Artificial Intelligence Tools in Migration and Asylum Management.” CMS10(1): 32.
38.
RecchiEttoreDeutschmannEmanuelGabrielliLorenzoKholmatovaNodira. 2021. “The Global Visa Cost Divide: How and Why the Price for Travel Permits Varies Worldwide.” Political Geography86: 102350.
39.
RettbergJill Walker.GajjalaRadhika. 2016. “Terrorists or Cowards: Negative Portrayals of Male Syrian Refugees in Social Media.” Feminist Media Studies16(1): 178–81.
40.
RinkenSebastian.OrtegaJosé Luis. 2022. “Leveraging the Web for Migration Studies: Data Sources and Data Extraction.” In Migration Research in a Digitized World, edited by PötzschkeSteffenRinkenSebastian, 129–48. IMISCOE Research Series.Cham: Springer International Publishing.
41.
SalahAlbert AliCanca.CansuErmanBaris. 2022. “Ethical and Legal Concerns on Data Science for Large Scale Human Mobility.” In Data Science for Migration and Mobility, edited by SalahAlbert A.KorkmazEmre E.BircanTuba, Oxford University Press.
42.
SalahAlbert AliKorkmazEmre ErenBircanTuba, eds. Data Science for Migration and Mobility. Oxford: Oxford University Press.
SandbergMarieRossiLucaGalisVasilisBak JørgensenMartin. 2022. Research Methodologies and Ethical Challenges in Digital Migration Studies.Cham: Springer International Publishing.
45.
SanliturkEbruZagheniEmilioDańkoMaciej J.TheileTomAkbaritabarAliakbar. 2023. “Global Patterns of Migration of Scholars with Economic Development.” Proceedings of the National Academy of Sciences of the United States of America120(4): e2217937120.
46.
SellarsA.2018. “Twenty Years of Web Scraping and the Computer Fraud and Abuse Act”. Boston University Journal of Science and Technology24(2): 372–415.
47.
SenIndiraFlöckFabianWellerKatrinWeißBerndWagnerClaudia. 2021. “A Total Error Framework for Digital Traces of Human Behavior on Online Platforms.” The Public Opinion Quarterly85(S1): 399–422.
48.
SingrodiaVidhiMitraAnirbanPaulSubrata. 2019. “A Review on Web Scrapping and Its Applications.” In 2019 International Conference on Computer Communication and Informatics (ICCCI), 1–6: IEEE.
49.
SîrbuAlinaAndrienkoGennadyAndrienkoNataliaBoldriniChiaraContiMarcoGiannottiFoscaGuidottiRiccardo, et al.2021. “Human Migration: The Big Data Perspective.” International Journal of Data Science and Analytics11(4): 341–60.
50.
SpörleinChristoph.SchlueterElmar. 2021. “Ethnic Insults in YouTube Comments: Social Contagion and Selection Effects During the German “Refugee Crisis.” European Sociological Review37(3): 411–28.
51.
SpyratosSpyridonVespeMicheleNataleFabrizioWeberIngmarZagheni.EmilioRangoMarzia. 2019. “Quantifying International Human Mobility Patterns Using Facebook Network Data.” PLOS ONE14(10): e0224134.
TjadenJasper.2021. “Measuring Migration 2.0: A Review of Digital Data Sources.” Comparative Migration Studies9(1): 1–20.
54.
TjadenJasper. 2023. “Migration of elites across the centuries.” https://jaspertjaden.com/project/migration-of-european-elites-across-the-centuries/. 1(1). 1-24.
55.
TrezzaD.2023. “To Scrape or Not to Scrape, This is Dilemma. The Post-API Scenario and Implications on Digital Research.” Frontiers in Sociology8: 1145038.
56.
Valdez ApoloMaría BelénArcila CalderónCarlosAmoresJavier Jiménez. 2019. “El Discurso Del Odio Hacia Migrantes Y Refugiados a Través Del Tono Y Los Marcos De Los Mensajes En Twitter.” RAEIC6(12): 361–384.
57.
WillekensFransMasseyDouglas S.RaymerJamesBeaucheminCris. 2016. “International Migration Under the Microscope.” Science352(6288): 897–99.
58.
ZagheniEmilioGarimellaVenkata Rama KiranWeberIngmarStateBogdan. 2014. “Inferring International and Internal Migration Patterns from Twitter Data.” In Proceedings of the 23rd International Conference on World Wide Web, edited by ChungChin-WanBroderAndreiShimKyuseokSuelTorsten, 439–44. New York, NY: ACM.
59.
ZagheniEmilioWeberIngmarGummadiKrishna. 2017. “Leveraging Facebook's Advertising Platform to Monitor Stocks of Migrants.” Population and Development Review43(4): 721–34.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
