Data collection from journalistic news apps without prerequired coding experience using MacroDroid smartphone automation software to simulate user interactions

Abstract

The share of people accessing news with their smartphones has steadily increased (Newman et al., 2022). One popular way to access journalistic content on smartphones is through specialized, standalone apps from news providers. However, despite extensive use and gatekeeping potential, research on the specific composition of news offered through such news apps for mobile devices is scarce (Palacios et al., 2016; Wolf & Schnauber, 2015).

This research gap is also caused by methodological challenges: several tools are readily available to automatically scrape content from websites and social media pages (e.g., RSS-feeds, HTML/CSS-selectors, news archives, application programming interfaces [APIs]). In contrast, data scraping from smartphone-based news apps is less straightforward: such apps rarely offer APIs, and they are often written in Java or Kotlin (Ardito et al., 2020), as they mostly run on Android (statcounter GlobalStats, 2022). This paper documents a data collection technique that is accessible to a wide range of mobile communication scholars, using the off-the-shelf smartphone automation app MacroDroid (Higgins, 2012). MacroDroid can be downloaded and installed on Android-based mobile devices directly from the Google Play Store. The app offers a graphical user interface (GUI) and hence requires no actual programming.

The two procedures described in this paper simulate a human smartphone user who logs on to a news app and scrolls down its public feed at recurring time intervals to collect data on preselected news menus within the app. MacroDroid achieves this by automatically collecting over time the positioning of news articles as presented by journalists within mobile news apps. This way, scholars can capture news agendas that journalists might specifically provide for mobile devices and compare them to the corresponding content on desktop-based news websites or within social media to analyze whether gatekeeping processes differ across various distribution channels (see also Hase et al., 2022; Wehden, 2023). MacroDroid might further be used, for example, to automatically capture screenshots of group discussions within messenger apps like WhatsApp¹ or content from hard-to-access apps based on visuals like Tik Tok.

Smartphone automation software, such as Android-based MacroDroid, is an “application, which automates repetitive tasks” such as “[a]utomatically turning dark mode on at night and off in the morning” (MacroDroidDev, 2021). These pre-programmed automation tasks are termed macros. Macros are organized as modular systems with three components (MacroDroidDev, 2021) that are selected from drop-down menus within the GUI and are then arranged in a specific order:

Each macro is started when a specific event occurs (termed trigger; e.g., specific points in time, incoming calls).

When triggered, each macro performs several pre-defined actions; for example, (simulated) user interactions (UIs) like clicks/scroll motions, launching other apps, taking camera pictures. UIs within a specific target app (e.g., a news app) can be anchored through XY-coordinates based on the smartphone's screen resolution or by directly identifying the click target (e.g., a share icon) through a built-in procedure. To attain a bug-free runtime, actions should be separated by short waiting periods; for example, to allow the target app to launch and properly load content.

Macros may optionally include specific constraints (e.g., time-based).

One way to automate data retrieval is by implementing a macro that is taking scrolling screenshots of the target (news) app's starting screen (i.e., long screenshots of the target (news) app, containing, for example, headlines, teasers, and images that can be subsequentially examined in a manual content analysis), triggered in recurring time intervals. To get such a scrolling screenshot macro running, a Samsung, Huawei, LG, OnePlus or MIUI smartphone operating on Android 9 or higher is required. The former is needed because these smartphones are equipped with screenshot apps (e.g., Samsung Scroll Capture) that allow taking overlay scrolling screenshots while running third-party apps (Otachi, 2019). The latter is required because MacroDroid only allows taking automated screenshots without root access since the implementation of Android 9 (Higgins, 2021), which requires programming skills and is not recommended for safety reasons (Nazar et al., 2012). Figure 1 illustrates the schematic program flow of the scrolling screenshot macro implemented in MacroDroid. A step-by-step tutorial and a video demo of the macro can be found in the online appendix.²

Figure 1.

Schematic program flow chart of the scrolling screenshot macro.

Scrolling screenshots provide a good starting point for manual content analysis as they preserve the richness of the data material by including images. However, they are unsuitable for subsequent automated data analysis that relies on machine readable data types. To gather such data, MacroDroid automation can obtain ordered lists of article URLs that mirror article positioning within the app at specific points in time. These URLs can later be used to scrape article content through corresponding website versions (e.g., Feiks, 2019, pp. 86–97).

To do so, an URL-list macro is implemented within MacroDroid that simulates a user scrolling through the news app from top to bottom, chronologically opening each article. The respective article's URL is then saved to a text-file stored on the smartphone's hard drive using the article sharing function. The macro is triggered in recurring time intervals. A schematic program flow chart of the URL-list macro is displayed in Figure 2. A step-by-step tutorial and a video demo are available in the online appendix.²

Figure 2.

Schematic program flow chart of the URL-list macro.

This form of URL-list retrieval is layout-dependent since XY-coordinates are used to scroll to the next article and the sizes of articles’ thumbnails depend on the number of lines in each headline/teaser. Therefore, the tool must be calibrated based on screen resolution and the target app's layout. Nonetheless, sometimes articles can be missed or captured twice. Therefore, smaller scroll motions are recommended as duplicates can be erased during data cleaning. The loop that simulates scrolling to the next article and opening it can be terminated after a predefined number of iterations or based on the current clipboard content (i.e., the last URL that was captured).

When automatedly collecting data from mobile apps with MacroDroid, the respective smartphones employed for data collection should be used exclusively for this purpose to allow for an undisturbed data retrieval that relies on simulated UIs. As it might be inconvenient to collect data on smartphones and later manually transfer it to a computer, this process can be smoothened by automating cloud data storage with MacroDroid.

Supplemental Material

sj-docx-1-mmc-10.1177_20501579231202321 - Supplemental material for Data collection from journalistic news apps without prerequired coding experience using MacroDroid smartphone automation software to simulate user interactions

Supplemental material, sj-docx-1-mmc-10.1177_20501579231202321 for Data collection from journalistic news apps without prerequired coding experience using MacroDroid smartphone automation software to simulate user interactions by Lars-Ole Wehden in Mobile Media & Communication

Supplemental Material

sj-docx-2-mmc-10.1177_20501579231202321 - Supplemental material for Data collection from journalistic news apps without prerequired coding experience using MacroDroid smartphone automation software to simulate user interactions

Supplemental material, sj-docx-2-mmc-10.1177_20501579231202321 for Data collection from journalistic news apps without prerequired coding experience using MacroDroid smartphone automation software to simulate user interactions by Lars-Ole Wehden in Mobile Media & Communication

Footnotes

Acknowledgments

The author would like to thank Nina Springer, chair of the research unit ‘Journalism Studies I’ within the Department of Communication at the University of Münster, for her valuable feedback on an earlier version of the manuscript.

Declaration of conflicting interests

The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Lars-Ole Wehden

Notes

Author biography

Lars-Ole Wehden is a post-doctoral researcher at the Chair of Journalism Studies within the Department of Communication at the University of Münster, Germany. He holds a BA in Communication and Political Science and an MA in Media and Political Communication from the Free University of Berlin. His doctoral thesis, completed at the University of Münster, focused on analyzing gatekeeping processes in journalism within the realm of social media. Additionally, his research interests extend to modern developments in journalism, including immersive storytelling in virtual reality (VR) and the integration of scientific references within journalistic content.

References

Ardito

Coppola

Malnati

Torchiano

(2020). Effectiveness of Kotlin vs. Java in Android app development tasks. Information and Software Technology, 127(November), 106374. https://doi.org/10.1016/j.infsof.2020.106374

Feiks

(2019). Empirische Sozialforschung mit Python: Daten automatisiert sammeln, auswerten, aufbereiten [Empirical social research with Python: Collecting, evaluating, and preparing data automatically]. Springer VS.

Hase

Boczek

Scharkow

(2022). Adapting to affordances and audiences? A cross-platform, multi-modal analysis of the platformization of news on Facebook, Instagram, TikTok, and Twitter. Digital Journalism, 11(8), 1499–1520. https://doi.org/10.1080/21670811.2022.2128389

Higgins

(2012). MacroDroid. https://www.macrodroid.com/

Higgins

(2021). No root screenshots. MacroDroid Forum. https://www.macrodroidforum.com/index.php?threads/no-root-screenshots.79/

Krotov

Johnson

Silva

(2020). Tutorial: Legality and ethics of web scraping. Communications of the Association for Information Systems, 47(December), 539–563. https://doi.org/10.17705/1CAIS.04724

MacroDroidDev. (2021). Overview. MacroDroid Wiki. https://macrodroidforum.com/wiki/index.php/Special:History/Overview

Nazar

Seeger

M. M.

Baier

(2012). Rooting android: Extending the ADB by an auto-connecting WiFi-accessible service. In Laud

(Ed.), Information security technology for applications. NordSec 2011, Lecture notes in computer science, vol. 7161 (pp. 189–204). Springer.

Newman

Fletcher

Robertson

C. T.

Eddy

Nielsen

R. K.

(2022). Reuters Institute Digital News Report 2022. Reuters Institute for the Study of Journalism.

10.

Otachi

(2019). How to capture a scrolling screenshot on Android. Online Tech Tips. https://www.online-tech-tips.com/smartphones/how-to-capture-a-scrolling-screenshot-on-android/

11.

Palacios

Barbosa

Firmino da Silva

da Cunha

(2016). Mobile journalism and innovation: A study on content formats of autochthonous news apps for tablets. In Aguado

J. M.

Feijóo

Martínez

I. J.

(Eds.), Emerging perspectives on the mobile content evolution (pp. 239–262). IGI Global.

12.

statcounter GlobalStats. (2022). Mobile operating system market share worldwide. https://gs.statcounter.com/os-market-share/mobile/worldwide

13.

Wehden

L.-O.

(2023). Journalistische Gatekeeper in den Sozialen Medien: Eine empirische Analyse der Nachrichtenverbreitung durch Journalist*innen anhand von Ereignis- und Meldungsmerkmalen [Journalistic gatekeepers in social media: An empirical analysis of news dissemination by journalists based on event and message characteristics]. Springer Fachmedien. https://doi.org/10.1007/978-3-658-40257-0

14.

Wolf

Schnauber

(2015). News consumption in the mobile era: The role of mobile devices and traditional journalism’s content within the user’s information repertoire. Digital Journalism, 3(5), 759–776. https://doi.org/10.1080/21670811.2014.942497

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.15 MB

0.16 MB