Abstract
As the use of digital news services increases, service providers recommend news articles based on readers’ interests and preferences. In this study, we analyzed the differences in news diversity, the number of political news articles, and the political bias of personalized news compared to non-personalized news, focusing on Naver News and Kakao News, Korea’s representative news aggregators. Additionally, we investigated whether readers’ political orientation affects the political bias of personalized news. Through a survey company, we recruited 299 adults aged 20 to 59 who had used Internet news portals for at least 10 min daily over the past month. An online survey was conducted to measure demographic variables, news usage, political orientation, and an online quasi-experiment was carried out to examine the news articles provided to them through Internet news portals in both logged-in and logged-out states. The results indicated that personalized news has lower content diversity, higher source diversity, and provides fewer political news articles compared to non-personalized news. In the case of Naver News, personalized news was more neutral in terms of coverage and statement bias than non-personalized news. Furthermore, news readers’ political orientation did not have a statistically significant effect on the coverage and statement bias of personalized news. Before the introduction of personalized news, Internet news portals in South Korea were suspected of being politically biased. The results of this study suggested that personalized algorithms did not provide more politically biased news articles than conventional news delivery systems and did not cause filter bubbles.
Plain Language Summary
This study investigated the differences in news diversity, the number of political news articles, and the political bias of personalized news compared to non-personalized news. Furthermore, this study analyzed whether readers’ political orientation affects the political bias of personalized news. The researchers conducted an online experiment and the results of 299 respondents were obtained using Naver News(197) and Kakao News(102). They found that personalized news has lower content diversity, higher source diversity, and provides fewer political news articles compared to non-personalized news. The results showed that news readers’ political orientation did not significantly affect the political bias of personalized news.
Introduction
More than a decade ago, news editors determined both the topics and timing of news delivery. However, digital news is provided over the Internet and recommendation algorithms now play a role similar to that of news editors; algorithms have become the gatekeepers of news. Traditional news media strive to provide unbiased news that is regularly checked and made answerable (Shoemaker, 1991). However, ensuring the absence of bias in digital news services is more challenging because they recommend news for each reader through recommendation algorithms. As of April 2020, 73% of news consumers in the U.S. and 79% in the U.K. used the Internet to access news (Newman et al., 2020) and digital news providers used recommendation algorithms to select news articles (Kwak et al., 2021). Hence, scholars are examining the impact of personalized news on democracy in an era where news is predominantly accessed via the Internet (Beam & Kosicki, 2014; Brundidge, 2010).
When the recommended news reflects readers’ interests or preferences, the range of information exposed may become narrow, resulting in filter bubbles (Pariser, 2011) and echo chambers (Sunstein, 2009).Studies have attempted to empirically verify the filter bubble and echo chamber phenomena (Bakshy et al., 2015; Feuz et al., 2011; Flaxman et al., 2016; Haim et al., 2018; Hannak et al., 2013; Nikolov et al., 2015) because the recommendation algorithm of digital news services significantly influence readers’ political orientation and public opinion (Beam & Kosicki, 2014; Fletcher & Nielsen, 2018; Hannak et al., 2013). In South Korea, politicians have raised concerns about the political bias of Internet news portals, such as Naver and Kakao (Choi, 2017). Choi (2017) pointed out that while concerns about political bias in South Korea’s Internet news portals existed, objective empirical evidence was lacking. He reported that the political bias of Internet news portals is not fixed, but rather varies depending on specific events, sometimes providing more liberal news articles and other times more conservative ones. However, his study was based on data collected during a period when news articles were recommended by human editors, before Naver and Kakao fully implemented personalized algorithms.
Few studies have quantitatively verified how personalized news differs from non-personalized news in terms of diversity (Möller et al., 2018). If the diversity of personalized news is low, only a limited range of information can be provided to the readers, leading them to form uninformed opinions (Fletcher & Nielsen, 2018; Haim et al., 2018). Therefore, this study statistically analyzed whether the personalized news recommended by the algorithm was significantly different from non-personalized news in terms of diversity.
Politically biased news can prevent news readers from making informed decisions about political issues (Beam & Kosicki, 2014) and, in severe cases, contribute to societal division (Bryanov et al., 2020). If recommendation algorithms provide readers only with politically aligned news, their confirmation bias may be reinforced, thereby negatively impacting political discourse and leading to polarization (Iyengar & Hahn, 2009). Therefore, it is important to determine whether the news recommended by the recommendation algorithm is politically biased. In addition, it is necessary to understand how the news recommended by the algorithms varies, depending on the reader’s political orientation.
So far, there have been no studies regarding the exposure of personalized and non-personalized news readers to political news stories and whether the recommended news articles are politically biased. Recent studies have found that news articles selected directly by users tend to be more politically biased than those recommended by algorithms on search engines or social media (Cardenal et al., 2019; Robertson et al., 2023). However, no study has examined whether personalized or non-personalized news delivers more politically biased articles. Therefore, this study investigated whether personalized news provides more political news stories than non-personalized news and whether it provides politically biased news stories. In addition, we analyzed whether readers’ political orientations influence the news article recommendations, that is, whether the recommendation algorithms recommend politically biased news articles according to people’s political orientations.
Unlike in many other countries, people in South Korea predominantly access news articles through Internet news portals (Newman et al., 2019). Furthermore, South Koreans tend to prefer domestic Internet news portals over global platforms like Google (Kwak et al., 2021). Given that South Korean politicians have long raised concerns about the political bias of Internet news portals (Choi, 2017), these platforms may have considered these concerns when designing their personalization algorithms. While extensive research has focused on personalized news in the context of Google’s search engine and news aggregators (Haim et al., 2018; Fletcher & Nielsen, 2018; Nechushtai & Lewis, 2019; Robertson et al., 2023), studies on other Internet news portals remain limited. Therefore, this study examines the case of personalized news in South Korea, specifically analyzing the news diversity and political bias of news articles recommended by domestic Internet news portals.
Based on the above, this study establishes the following five research questions:
Literature Review
Personalized News Service in South Korea: Naver News and Kakao News
In South Korea, major Internet news portals such as Naver and Kakao have dominated the digital news market over the past decade (Newman et al., 2019). As of 2019, 66% of news users in South Korea accessed news through Naver, while 34% used Kakao (Newman et al., 2019). According to a survey conducted in Korea, 70.9% of news readers have used personalized online news (KISDI, 2019). In particular, in South Korea, personalized news services of Internet portals are widely used. While Western countries, such as the U.S., Canada, Finland, Norway, and Australia do not widely use news aggregators, Asian countries, such as South Korea, India, Indonesia, and Thailand, generally use news from the Internet news aggregator portals (Newman et al., 2019).
On February 17, 2017, Naver applied its algorithm system, AiRS, to its mobile news service (The Korea Bizwire, 2017). News articles recommended by AiRS is provided only when the reader is logged in. Personalized news is presented at the top of the main news screen. Kakao, another Korean Internet portal, developed a news recommendation system, RUBICS, in June 2015 (The Korea Bizwire, 2017). RUBICS recommends more suitable customized news when readers are logged in.
Before using personalized algorithms, both Naver and Kakao were suspected of a biased engagement in providing news articles to readers. Hence, in response to criticism, Internet news portals opted to leave news editorial control to AI (The Atlantic, 2017). However, ever since the two portals started using personalized news algorithms, the adequacy of the news articles in terms of diversity and political bias has not been verified. Therefore, this study comprehensively analyzed the diversity and political bias of the news articles recommended by Internet news portals using recommendation algorithms.
Diversity of Personalized News
Recommending news through algorithms can help overcome information overload; nevertheless, it can reduce news diversity (Fletcher & Nielsen, 2018). This can lead to news readers receiving limited information and, as a result, forming narrow opinions (Fletcher & Nielsen, 2018; Haim et al., 2018), leading to filter bubbles (Pariser, 2011) or echo chambers (Sunstein, 2009).
Several studies have attempted to empirically verify the filter bubble and echo chamber phenomena (Bakshy et al., 2015; Feuz et al., 2011; Flaxman et al., 2016; Haim et al., 2018; Hannak et al., 2013; Nikolov et al., 2015). Some studies have explained that partially limited information is provided by recommendation algorithms (Bakshy et al., 2015; Feuz et al., 2011; Hannak et al., 2013). However, certain studies deny the existence of these phenomena (Flaxman et al., 2016; Haim et al., 2018). Rather, they prove that recommendation algorithms increase news diversity by recommending news related to various news topics and sources (Beam & Kosicki, 2014; Fletcher & Nielsen, 2018; Möller et al., 2018; Nechushtai & Lewis, 2019; Schroeder & Kralemann, 2005).
A digital news company, operating recommendation algorithms, does not disclose how its algorithms select news articles (Hannak et al., 2013). Pasquale (2015) suggested that these algorithms should be regarded as “black boxes” because it is difficult to understand their parameters for determining personalized news. Since digital news providers do not release the methodology behind their algorithms (Pasquale, 2015), in order to infer how these recommendations are made, it is important to verify the differences between personalized and non-personalized news (Beam & Kosicki, 2014). Personalized news refers to a service that applies personalization algorithms to provide distinct news articles to individual users based on their interests and preferences, in contrast to those offered to others (Haim et al., 2018; Hannak et al., 2013). Non-personalized news refers to a service that provides the same news articles to all users.
Haim et al. (2018) classified news diversity into content diversity and source diversity. Content diversity refers to the diversity of the topics covered by the news. Source diversity refers to the pluralism of the news press. To analyze content diversity and source diversity provided by Google News’ recommendation algorithm, they examined the difference between content diversity and source diversity offered by personalized and non-personalized news. The results indicated that personalized news readers were provided more news about their preferred topics than non-personalized news readers. However, there was no significant difference in source diversity between the two. Möller et al. (2018) conducted a study on de Volkskrant, a digital news service in the Netherlands, and found that the news selected by the algorithm was more diverse in terms of topics (content diversity) than the news selected by human editors or popular news. Beam and Kosicki (2014) verified that readers of personalized news were provided news from various sources and on diverse topics. Hence, they argued that recommendation algorithms do not reduce news diversity. However, these studies only surveyed readers’ subjective perceptions and did not directly analyze the provided news articles.
Although there have been studies that descriptively compare the differences in diversity between personalized and non-personalized news (Fletcher & Nielsen, 2018; Haim et al., 2018; Hannak et al., 2013; Möller et al., 2018), there has been no study to investigate whether the difference is statistically significant. Therefore, in this study, we measured the diversity of personalized and non-personalized news, using formulas developed in previous studies (McDonald & Dimmick, 2003), and analyzed whether the differences are statistically significant. Like previous studies, this study, too, classified news diversity into content diversity and source diversity (Haim et al., 2018). Content diversity refers to the variety of topics covered in the news articles, while source diversity refers to the plurality of the news organizations producing them (Voakes et al., 1996; Thurman, 2011).
Political News Recommendation by Recommendation Algorithms
People mainly understand political issues through cognitive shortcuts; therefore, they make political decisions based on easily accessible media content (Conover & Feldman, 1989; Mendelsohn, 1996). For example, voters who are more exposed to the media, evaluate candidates based on news coverage rather than party identification (Mendelsohn, 1996). Patterson (1993) found that news coverage of a front-runner influences voters’ perceptions of the viability of the presidential candidate. Additionally, several studies have argued that exposure to political news or information during elections can affect readers’ candidate judgment (Abrams & Brody, 1998; Cho, 2005; Gattermann et al., 2016; Gattermann, 2020).
Therefore, if a digital news service provides more political news to a specific group, that group may be more influenced by the news when making political decisions. By providing more political news to personalized news readers, the news service providers can have a greater influence on readers’ political decision-making.
Haim et al. (2018) stated that Google News provides an equal amount of political news to both interested and non-interested readers. However, whether a greater number of personalized news readers are exposed to political news stories than non-personalized news readers and whether the number of recommended political news articles is a result of their political orientation has not been studied. Therefore, this study analyzed whether there is a difference in the amount of political news provided to personalized and non-personalized news readers of Naver News and Kakao news. Furthermore, the study examined if the political tendency of personalized news readers affected the number of political news recommended and consumed.
The Political Bias of Personalized News
As mentioned above, political news can significantly influence an individual’s political decisions (Conover & Feldman, 1989; Zeldes et al., 2008); therefore, politically balanced news reports are significant for the development of a democracy (Bagdikian, 1971). Several scholars have argued that news readers should not be exposed to overly biased news during public discourse and political decision-making (Beam & Kosicki, 2014; Bennett & Iyengar, 2008; Iyengar & Hahn, 2009). Biased news can limit readers’ ability to make informed political decisions (Beam & Kosicki, 2014). If algorithm-recommended news has a certain political bias, whether liberal or conservative, it can lead to polarization of political decisions (Bryanov et al., 2020). Therefore, it is important to determine whether the news recommended by the algorithm is politically biased.
Until now, studies have measured whether newspaper articles (Fico & Freedman, 2004; Fico et al., 2006; Groseclose & Milyo, 2005) and television broadcasts (D’Alessio & Allen, 2000; Diddi et al., 2014; Schiffer, 2006; Zeldes et al., 2008) are politically biased. Some studies have found that news articles recommended by personalization algorithms on search engines and social media tend to be less politically biased than those selected directly by users (Cardenal et al., 2019; Robertson et al., 2023). Cardenal et al. (2019) found that direct navigation increases selective exposure whereas Google reduces it. Additionally, they argued that the relationship between origins to news and selective exposure is strongly moderated by ideology, suggesting that search engines and social media are not politically neutral. Robertson et al. (2023), in their study analyzing news articles provided through Google Search and those directly selected by users during the 2018 and 2020 U.S. election periods, found that exposure to and engagement with partisan or unreliable news articles on Google Search is primarily driven by users’ own choices rather than algorithmic recommendations. These findings support the argument that partisan audiences tend to actively avoid counter-attitudinal political information (Arceneaux et al., 2012).
However, there have been limited studies that statistically analyzed and quantified the political bias of news articles recommended by personalization algorithms, compared to those provided uniformly to the general public without personalization on news aggregators. Therefore, in this study, we measured the political bias of personalized and non-personalized news and examined the differences between the two. If news readers’ political orientation impacts the algorithm’s news recommendation (i.e., if the algorithm recommends only the news suited to readers’ political orientation), readers would not be exposed to various political opinions and be limited to information bias toward their preferred political party. Since individuals with strong political preferences tend to prefer content that aligns with their political views (Mummolo, 2016), and algorithms select news that the readers prefer (Haim et al., 2018), news aggregators may try to provide readers with their preferred political news. If the algorithm continues to provide only the preferred news, it would gradually strengthen confirmation bias (Iyengar & Hahn, 2009; Knobloch-Westerwick et al., 2020). Therefore, it is necessary to examine how the algorithm-recommended news varies depending on readers’ political orientation.
Studies have been conducted on how news articles recommended by algorithm changes according to the political orientation of the news readers (Beam & Kosicki, 2014; Fletcher & Nielsen, 2018; Flaxman et al., 2016; Nechushtai & Lewis, 2019). These studies argued that personalized news did not show a significant difference according to readers’ political orientation (Fletcher & Nielsen, 2018; Flaxman et al., 2016; Nechushtai & Lewis, 2019). However, another study found that the recommended news differed according to readers’ political orientation (Beam & Kosicki, 2014). Fletcher and Nielsen (2018) found that readers who found and consumed news through a search engine accessed more diverse topics. In addition, their news was less significantly influenced by left- or right-leaning news sources. Similarly, Flaxman et al. (2016) analyzed the web histories of 50,000 U.S. citizens and revealed that the more often search engines were used, the greater the exposure to news articles from media outlets with opposing political orientations. Nechushtai and Lewis (2019) reported that Google News recommended similar news to left- and right-leaning readers. Nevertheless, Beam and Kosicki (2014) stated that Republicans who use personalized news through Internet portals were more likely to receive news consistent with their political perspective.
However, Beam and Kosicki (2014), Fletcher and Nielsen (2018), Nechushtai and Lewis (2019), and Flaxman et al. (2016) analyzed the political bias of the source (news press) and not of the provided news. Therefore, this study statistically investigated whether there was any relationship between the political orientation of personalized news readers and the political bias of the recommended news. Different from the above-mentioned studies, this study directly measured the political bias of the news articles and analyzed whether readers’ political orientation affected them.
Methods
Experiment Design and Participants
To statistically test the research questions, we conducted an online experiment and adopted multiple time-series designs, a type of quasi-experimental design. We divided participants into two groups and requested each group to measure the news provided by news services in a logged-out setting (control group) and a logged-in setting (experimental group), which utilized recommendation algorithms. In this procedure, the experimental treatment, which urges the readers to use a news recommendation algorithm was manipulated only for the experimental group. We found that the same news were recommended when all the participants used their news service in a logged-out setting (control group). Hence, this study was able to compare the control group and the experimental group. In this study, personalized news refers to a service that applies personalization algorithms to provide distinct news articles to individual users based on their interests and preferences, in contrast to those offered to others (Haim et al., 2018; Hannak et al., 2013). Non-personalized news refers to a service that provides the same news articles to all users. Therefore, we define news provided in the logged-out setting as “non-personalized news” and those provided in the logged-in setting as “personalized news.”
In this experiment, all the participants were “male and female adults, aged 20 to 59 years, who had experienced an algorithm news recommendation service, provided either by Naver or Kakao, for an average of more than 10 min per day on their mobile over the past month (baseline date: September 30, 2019).” Considering the quota sampling for genders, ages, and services ([Naver and Kakao]), we recruited 340 participants for the mobile experiments and online surveys conducted through Macromill Embrain, a South Korea-based research company. The experiment and online survey results of 299 (Naver News: 197, Kakao News: 102) respondents were used for the final analysis, excluding those who provided unreliable responses in the experiment process (participation less than three times) and who did not accurately present the login-logout service usage records for each experiment session (A screenshot of a mobile news screen with a list of news articles).
The gender ratio among Internet users in South Korea is 51.2% (male) and 48.8% (female; National Information Society Agency, 2019), while the user ratio for Naver and Kakao is 66% (Naver) and 34% (Kakao; Newman et al., 2019). Among the 299 participants, the gender ratio was 51.2%:48.8% (153:146), and the Naver-to-Kakao user ratio was 65.9%:34.1% (197:102), closely mirroring the distribution of Internet users. Therefore, the participants in this study can be considered representative. Table 1 indicates the descriptive statistics of all the participants.
Demographics and Descriptive Statistics.
Experiment Procedures
Participants were fully informed about the purpose and nature of this study, and their voluntary consent was obtained before completing an anonymous online survey. No sensitive or personally identifiable information was collected, and the study adhered to ethical research practices. We conducted an online survey of the participants where we examined whether they used algorithm-based news services, demographics, daily news usage, and political orientation. Information measured by the participants was collected over a total of six times; news articles from non-personalized news services when logged-out and news articles from algorithm-based personalized news services when logged-in were measured three times each.
The experiment was conducted online (mobile) for 2 weeks, from October 7 to October 20, 2019. The participants were requested to use the news aggregator service, both when logged out and logged in, through the mobile application of either Naver News or Kakao News assigned to them, three times a day (8–9 A., 12 AM–1 PM, and 7–8 PM) on designated dates (October 7, 8, 15, and 16, 2019). At each designated date and time, each participant was asked to capture the title, outlet, content, time, and the main screen of the news list (with seven news articles) displayed on top of the home page of the news portal (Figure 1). Since the news lists at the top of the mobile screen usually draw more attention, it allowed the participants to capture the top seven news items, as referred to by Pan et al. (2007), Haim et al. (2018), Hannak et al. (2013), and Kwak et al. (2021). Each of these seven news articles was analyzed to verify the research questions. Naver and Kakao update news articles several times a day; however, they do not disclose the exact number of updates per day. Therefore, we experimented, using three sampling points when news readers got the most news, i.e., on the way to work, after work, and at lunchtime (Kwak et al., 2021). As a result, the participants performed the tasks six times throughout the day (thrice each for logged-out and logged-in). The participants received text notifications 10 min before the start and the end of each experiment and were rewarded for the improved quality of the experiment and survey results.
Mobile screenshot of algorithmic personalization.
Analysis Method
To assess RQ 1, we conducted a t-test to determine whether there were statistically significant differences in content diversity and source diversity between personalized news (logged-in setting) and non-personalized news (logged-out setting). For RQ 2, we performed a t-test to assess whether the number of political news articles differed significantly between personalized and non-personalized news. To investigate RQ 3, we conducted a hierarchical multiple regression analysis, with users’ political orientation as the independent variable and the number of political news articles provided to them as the dependent variable. To test RQ 4, we conducted a t-test to examine whether there were differences in coverage bias and statement bias between personalized and non-personalized news. For RQ 5, we performed a hierarchical multiple regression analysis. The dependent variables were the coverage bias and statement bias in the set of political news articles delivered to users via personalized news, while the independent variable was users’ political orientation.
Measurement Method
Content diversity refers to the variety of topics covered in the news articles (Voakes et al., 1996; Thurman, 2011). To measure content diversity, news topics were categorized based on the news titles submitted by the participants. There were six news areas—political, economic, social, international, IT and scientific, and sports and culture, and five categories—political figures and events, business and finance, international affairs, sports, and entertainment, based on Tewksbury (2003) and Naver News and Kakao News’ method of classifying news articles.
Source diversity refers to the plurality of the news organizations producing the news articles (Voakes et al., 1996; Thurman, 2011). To measure source diversity, the media types were classified based on the media outlets of the news articles. There were a total of 60 media outlets that were classified into nine news platforms—comprehensive program channels, terrestrial broadcast, 24-hr cable news channel, national daily newspaper, economic daily newspaper, regional daily newspaper, news agency, online-only news media, and others, based on Kwak et al. (2018). Based on the topics and source classifications presented above, content diversity and source diversity for each participant was measured by Simpson’s standardized diversity index (Dz) (McDonald & Dimmick, 2003). Simpson’s Dz is less affected by the number of categories than Simpson’s D, which was widely used to measure diversity and was used by Masini et al. (2018) to measure news diversity. Simpson’s Dz is as follows:
For content diversity, “pi” refers to the ratio of news articles in a specific topic classification to the number of news articles received by experimental participants. “k” means the number of topic categorizations; here six. For source diversity, “pi” refers to the ratio of the number of news articles in a specific source classification to the number of news articles provided to the experimental participants and “k” means the number of source classifications; here nine. Simpson’s Dz has a value between zero and one. When diversity is at its lowest, that is, all the news articles are in one category, it has a value of zero, whereas when diversity is at its highest, that is, there is an equal distribution of news articles across all categories, it has a value of one.
The number of political news articles in personalized and non-personalized news was measured based on the political news articles in the news topic classification. News readers’ political orientation was measured on a scale of 9 with 1: very liberal, 5: neutral, and 9: very conservative.
The political bias of the news articles was measured in two ways—coverage bias and statement bias. Coverage bias, for each article, was measured in four categories derived from Fico et al. (2006), Fico and Freedman (2004), and Fico and Cote (1997). First, by assessing the prominence and space given to quoted or paraphrased statements in news articles, bias was measured as +1 if partisans for the liberal party were given more coverage than those of the conservative party, and −1 if partisans for conservative parties had more coverage. Second, we measured whether the statements of either parties’ supporters were cited in the first paragraph of the article. According to the prior study, the statements of liberal party supporters were considered +1, and those of conservative party supporters were considered −1. Therefore, it was rated +1 if only liberal supporting statements were quoted and −1 if only conservative supporting statements were quoted. It was rated 0 if either both or neither supporters were quoted. Third, we assessed whether both sides’ statements were covered in the second to fifth paragraphs by applying the same measurement method described above. Lastly, we measured whether the claims of the supporters were cited in the sixth to the last paragraph. The total scores were between −4 and +4. Scores closer to +4 indicated a liberal party bias and scores closer to −4 indicated a conservative party bias.
Statement bias was measured using the method developed by Schiffer (2006). If a news article had a positive tone, irrespective of the party, it was rated +3. It was rated +2 for a neutral tone and +1 for a negative tone. After measuring both liberal and conservative parties in the same manner, the conservative party’s score was subtracted from the liberal party’s score. The total had a value between +2 and −2, that is, the closer it was to +2, the higher the liberal party bias, and the closer it was to −2 the higher the conservative party bias. For example, if a news article had a positive tone for the liberal party but had a neutral tone for the conservative party, the liberal party scored +3 and the conservative party scored +2. Therefore, the total was +1 [+3 (−) +2].
In addition, the control variables for the regression analysis were measured. In an online survey, we estimated the age, gender, education, income, marriage, and news usage of all the participants. Age was measured on a four-point scale, divided by 10 years (1 = 20–29 years, 2 = 30–39 years, 3 = 40–49 years, and 4 = 50–59 years). Gender was coded as 1 = men and 2 = women. Education was recorded on a four-point scale, from middle school to graduate school (1 = middle school graduation, 2 = high school graduation, 3 = university or bachelor’s degree, and 4 = graduate school or master’s degree). Monthly income was coded on a seven-point scale (1 = less than 1 million KRW, 2 = between 1 and 2 million KRW, 3 = between 2 and 3 million KRW, 4 = between 3 and 4 million KRW, 5 = between 4 and 5 million KRW, 6 = between 5 and 6 million KRW, and 7 = more than 6 million KRW). Marital status was measured as 1 = married and 2 = unmarried. News usage was measured on an eight-point scale as the average daily use of digital news service (1 = between 10 and 20 min, 2 = between 20 and 30 min, 3 = between 30 and 40 min, 4 = between 40 and 50 min, 5 = between 50 min to 1 hr, 6 = between 1 hr and 1 hr 10 min, 7 = between 1 hr 10 min to 1 hr 20 min, and 8 = more than 1 hr 20 min). As mentioned earlier, readers with less than 10 min of news usage were excluded from the study.
Intercoder Reliability
News topics and political bias were measured by two authors; one with a PhD and the other with a master’s degree in the media field. After coding was completed, its reliability was verified. News characteristics and the four items of coverage bias, which were nominal variables, were calculated by Cohen’s Kappa coefficient. Statement bias, a continuous variable, was calculated by Cronbach’s alpha coefficient. Cohen’s Kappa coefficient of the news characteristics was .989 and of the coverage bias’s four items were .971, .955, .926, and .949, respectively. Cronbach’s alpha coefficient of the statement bias was .889, which is generally regarded as a reliable level. For items that were differently measured, the coders and the researcher discussed and reached an agreement to determine the final measurement value.
Research Results
Statistical Analysis
During the experiment, a total of 11,678 news articles (5,845 personalized and 5,833 non-personalized) were recommended by Naver News and Kakao News. Of these, 3,149 (27.0%) were political news articles. Table 2 presents the statistics on news topics and news sources (media outlets).
News Topics and Sources.
In October 2019, news articles covered various political topics in Korea. President Moon Jae-in, a former Democratic Party of Korea leader, appointed the new Minister of Justice to reform the prosecution service and introduce the Corruption Investigation Office for High-ranking Officials. In addition, a variety of political news were reported. Table 3 indicates the classification of political news articles recommended to the participants during the study.
Political News.
Diversity of Personalized News (RQ 1)
To find out the difference in news diversity (both content and source) between personalized and non-personalized news, the difference between content diversity and source diversity of two different news was verified by a t-test.
As a result, Naver News, Kakao News, and Naver and Kakao News combined, all showed the same differences in news diversity. Personalized news had a lower content diversity (Naver: t = −2.056, p < .05; Kakao: t = −2.398, p < .05; Naver + Kakao: t = −2.972, p < .005) and a higher source diversity than non-personalized news (Naver: t = 4.018, p < .005; Kakao: t = 1.932, p < .1; Naver + Kakao: t = 4.451, p < .005) (Table 4).
News Diversity Between Personalized and Non-Personalized News.
Note. Group (a) is personalized news (logged in), Group (b) is non-personalized news (logged out).
p < .005. *p < .05. †p < .1.
Since personalized news reflects readers’ interests and preferences (Beam & Kosicki, 2014), it lowers content diversity. This result contradicts Möller et al. (2018) who found, in a study of digital news services in the Netherlands, that the news from recommendation algorithm provided a wider variety of news topics. However, the result of this study is consistent with Haim et al. (2018) who found that algorithms provided more news on topics of readers’ interest than any other topic.
Meanwhile, personalized news had better source diversity than non-personalized news, which was consistent with Beam and Kosicki (2014) and Fletch and Nielson (2018) that personalized news readers received news articles from varied sources.
Political News Recommended by Algorithms (RQs 2 and 3)
A t-test was used to compare the number of political news articles in personalized and non-personalized news. The results indicate that in the case of Naver News and Naver and Kakao News combined (Naver: t = −3.078, p < .005; Naver + Kakao: t = −3.160, p < .005), personalized news had significantly less political news articles than non-personalized news (Table 5). This indicated that recommendation algorithms were suitable for news readers’ preferences, with fewer politics-related news. Since algorithms selected and recommended news based on reader interests and preferences (Beam & Kosicki, 2014), the result showed that Naver News readers, using personalized news services, wanted to receive less political news than those availing non-personalized news services.
Number of Political News Between Personalized and Non-Personalized News.
Note. Group (a) is personalized news (logged in), Group (b) is non-personalized news (logged out).
p < .005.
To find out news readers’ political tendencies and their impact on the number of political news articles recommended to them, hierarchical multiple regression analysis was used. In Model 1, targeting ‘Naver News’ readers, we found the relationship between their demographics, such as age, gender, education, income, marital status, and news usage, and the number of political news recommended by algorithms. In Model 2, after controlling independent variables in Model 1, we studied how readers’ political orientation changed the number of political news. Models 3 and 4 conducted the respective analysis for ‘Kakao News’ readers, while Models 5 and 6 conducted the respective analysis for Naver and Kakao News combined.
In Model 5, none of the independent variables significantly affected the number of political news items. As a result of adding readers’ political orientation as an independent variable in Model 6, only political orientation had a significant impact (β = .073, p < .05) (Table 6).
Influence of News Readers’ Political Orientation on the Number of Political News.
Note. For sex, male = coded 1, female = coded 2; as for marital status, married = coded 1, unmarried = coded 2.
p < .005. *p < .05.
It was observed that more political news was presented to conservative news readers. Since algorithms provide news tailored to readers’ interests and preferences (Beam & Kosicki, 2014), we inferred that conservatives in Korea were more interested in political news than liberals. This may be because the news sources (media outlets) primarily accessed by Korean readers tend to provide relatively conservative news articles (Ha & Shin, 2016; Kwak, 2011; Shin, 2016). Popular media outlets in South Korea, such as the Chosun Ilbo, JoongAng Ilbo, and Dong-A Ilbo, are conservative in nature; therefore, a majority of the news provided through digital news services is related to conservative politics (Kwak, 2011). As a result, conservative supporters are more likely to prefer political news that aligns with their political opinions. Thus, we assumed that algorithms provided them with their preferred political news topics.
The Political Bias of Personalized News (RQs 4 and 5)
To validate RQ 4, we compared coverage and statement bias between personalized and non-personalized news (Table 7). Concerning Naver News, there was a statistically significant difference in coverage and statement bias between personalized and non-personalized news. News services with algorithms provided more liberal news articles in the context of coverage bias (t = 9.384, p < .005). However, it provided more conservative news articles in the context of statement bias (t = −4.892, p < .005). The average(mean) of coverage bias between personalized and non-personalized news was −.044 and −.613, respectively; thus, algorithms recommended more neutral news articles in favor of coverage bias. The average(mean) of statement bias between personalized and non-personalized news was .008 and .069, respectively; thus, algorithms recommended more neutral news articles in favor of statement bias.
Comparison of Political News Bias Between Personalized and Non-Personalized News.
Note. Group (a) is personalized news (logged in), Group (b) is non-personalized news (logged out).
p < .005.
In Kakao News, there was no significant difference between personalized and non-personalized news regarding coverage and statement bias (coverage bias: t = −.600, p > .05; statement bias: t = 1.760, p > .05). In a combined analysis of Naver News and Kakao News readers, personalized news provided more liberal news articles than non-personalized news in favor of coverage bias (t = 7.705, p < .005) and more conservative news articles in favor of statement bias (t = −3.497, p < .005). This matched the results of the Naver News readers. The average(mean) of the coverage bias of personalized and non-personalized news was −.058 and −.407, respectively; thus, algorithms recommended more neutral news articles in favor of coverage bias. The average(mean) of the statement bias of personalized and non-personalized was −.026 and .005, respectively; thus, the algorithm recommended more neutral news articles in favor of statement bias. However, both personalized and non-personalized news articles measured close to zero, suggesting that both delivered overall neutral articles.
To verify RQ 5, hierarchical multiple regression analysis was performed. The dependent variables were based on coverage and statement bias of the set of political news articles recommended by personalized news to the readers. The independent variable was political orientation, and the control variables included age, sex, education, income, marital status, and news usage (Table 8).
Influence of News Readers’ Political Orientation on Political News and Bias.
p < .005. *p < .05.
Statistical analysis showed that some variables affected the coverage bias in political news. In Kakao News, a higher number of conservative news articles were recommended to married readers (β = −.440, p < .05). The analysis of both Naver News and Kakao News readers revealed that a higher number of liberal news articles were recommended to those with lower news readership (β = −.044, p < .05). However, readers’ political orientation did not significantly influence the coverage (β = −.001, p > .05) or statement bias (β = .001, p > .05) of personalized news. Since algorithms recommend news articles tailored to reader preferences and interests (Beam & Kosicki, 2014), it can be assumed that it would provide news tailored to readers’ political orientation as well. However, according to the study results, personalized news did not recommend news articles based on individuals’ political orientations. The results were consistent with previous studies that claimed that news recommended by algorithms did not differ depending on readers’ political tendencies (Fletcher & Nielsen, 2018; Flaxman et al., 2016; Nechushtai & Lewis, 2019). In addition, these results suggest that the portal algorithm, similar to those used by Naver and Kakao, does not contribute to confirmation bias in readers.
Conclusions and Discussion
This study analyzed how personalized news, recommended by algorithms (Newman et al., 2019), differs in terms of news diversity, number of political news, and political bias compared to news recommended by editors. In addition, it investigated whether readers’ political orientations affect the political bias of personalized news.
In a comparative analysis of news diversity, personalized news showed lower content diversity and higher source diversity. According to Möller et al. (2018), digital news services in the Netherlands provided news articles on a variety of topics through recommendation algorithms. However, the results of this study, focusing on Naver News and Kakao News in South Korea, supported the contrary. This was because algorithms provide news appropriate to readers’ interests and preferences (Beam & Kosicki, 2014); therefore, they provide more news articles on a specific topic. This reduces content diversity. In fact, Haim et al. (2018) stated in a study of Google News that the news aggregator service provided more news on topics that readers were interested in than other topics.
Personalized news had a higher source diversity than non-personalized news. This finding was consistent with Beam and Kosicki (2014) and Fletcher and Nielson (2018) who claimed that the news chosen by the algorithm provides articles from diverse sources. Fletcher and Nielson (2018) attributed the reason to automated serendipity, a phenomenon in which the automated news selection of recommendation algorithms included news from media companies, which news readers do not usually access. The results of this study indicate that the recommendation algorithms of the news aggregator services in South Korea recommend a narrow range of topics to provide more suitable news articles to readers; however, it recommends news articles from various sources.
The study found that there was a difference in the number of political news recommendations between personalized and non-personalized news, whereby, personalized news services provided fewer political news. Haim et al. (2018) suggested that algorithms provided news topics suitable to readers’ interests. Therefore, it is presumed that personalized news provided fewer political news articles than non-personalized news because readers are more interested in and prefer topics other than politics.
Nevertheless, news readers’ political orientation impacted the number of political news. Algorithms recommended a higher number of conservative political news to the readers. This may be due to the fact that major media outlets in Korea primarily provide conservative news articles, which align with the preferences of conservative supporters (Ha & Shin, 2016; Kwak, 2011; Shin, 2016). The major popular media outlets in South Korea include Chosun Ilbo, JoongAng Ilbo, and Dong-A Ilbo, which have conservative tendencies (Ha & Shin, 2016; Shin, 2016). Thus, for conservative news readers, the political news received through digital news services would most probably belong to their preferred media outlets. Therefore, they have a higher preference for political news from digital news services.
Naver News and Kakao News had different results for the political bias of personalized and non-personalized news. In Naver News, personalized news was more liberal in terms of coverage bias and more conservative in terms of statement bias. However, when comparing the political bias of personalized and non-personalized news, personalized news tended to recommend news articles with less political bias.
Personalized news articles in Kakao News did not show statistically significant differences in coverage and statement bias. However, for both personalized and non-personalized news services, the average (mean) of news articles’ coverage and statement bias were close to zero; thus, providing relatively unbiased news. Hence, Kakao News was relatively politically unbiased in both personalized and non-personalized news.
In South Korea, politicians raised the concern that Internet news portals, such as Naver News and Kakao News, provided politically biased news to people (Choi, 2017; The Atlantic, 2017). As an answer to this accusation, Internet news portals introduced recommendation algorithms (The Atlantic, 2017). However, this study found that Kakao News recommended relatively unbiased news articles regarding both coverage and statement bias. Naver News showed a tendency for both coverage and statement bias; however, it recommended more politically unbiased news articles by news services using the algorithms. In conclusion, the algorithms used by news aggregator services do not provide a higher number of politically biased news or cause greater filter bubbles than traditional news systems. Several existing researchers claimed that readers’ exposure to politically biased news could lead to polarization (Bryanov et al., 2020) and restrict high-quality political decision-making (Beam & Kosicki, 2014; Bennett & Iyengar, 2008; Iyengar & Hahn, 2009). However, the results of this study revealed that Naver News alleviated such problems by introducing recommendation algorithms.
In terms of political bias, the results of this study differed from previous research. This study found that, in South Korea, personalized news contained more liberal news articles than non-personalized news in terms of coverage bias, whereas it included more conservative news articles in terms of statement bias. In contrast, Fletcher and Nielsen (2018) found that Google Search in the U.S., U.K., Germany, and Spain provided news articles in a more politically balanced manner, and Robertson et al. (2023) stated that news articles provided by Google Search in the U.S. were less politically biased than news articles selected directly by users. By measuring political bias from two perspectives—coverage bias and statement bias—this study produced more nuanced findings compared to studies conducted in other countries.
However, news readers’ political orientation did not have statistically significant effects on the political bias (both coverage and statement bias) provided by the algorithms. The results were in line with previous studies (Fletcher & Nielsen, 2018; Flaxman et al., 2016; Nechushtai & Lewis, 2019), which proved that the news recommended by the algorithm did not vary depending on readers’ political tendencies. However, this contrasts with Beam and Kosicki (2014), who found that Republican users of personalized news portals were more likely to receive news articles from sources aligned with their political views, and Flaxman et al. (2016), who reported that U.S. citizens with higher search engine usage were increasingly exposed to media outlets with political orientations opposite to their own. The reason why the results of this study differ from those of the two studies is likely because, in South Korea, there have long been concerns about the filter bubble and political bias in Internet news portals (Choi, 2017; The Atlantic, 2017). As a result, Internet news portal providers have improved their algorithms to prevent the delivery of biased news based on users’ political orientations. If news readers’ political orientation influenced news recommendations, there would be the risk of readers not receiving diverse information and developing narrow political views (Bryanov et al., 2020). However, the results indicated that South Korea’s major news aggregators, such as Naver News and Kakao News, did not pose a problem of confirmation bias.
In conclusion, the findings on source diversity in this study are consistent with studies conducted in other countries, supporting its external validity. However, the results regarding content diversity and political bias diverged from those observed in other countries. As mentioned earlier, unlike in other countries, people in South Korea rely heavily on Internet news portals such as Naver and Kakao for their news consumption (Newman et al., 2019). The findings of this study suggest that the political bias and filter bubble phenomenon in algorithmically recommended personalized news in South Korea may differ from those observed in other countries.
Implications and Limitations
This study suggests the following academic implications. First, it finds that the algorithm-recommended news has lower content diversity and higher source diversity. In addition, we statistically find evidence supporting the differences in content diversity and source diversity between personalized and non-personalized news, both indicating significant results.
Secondly, this study shows that the amount of personalized political news provided to the news readers is related to readers’ political orientation. In particular, more political news is provided to politically conservative news readers. The relationship between personalized political news and news readers’ political orientation has not been covered in previous studies. Although these results arise from the conservative media environment in Korea (Ha & Shin, 2016; Kwak, 2011; Shin, 2016), there is a concern that it may strengthen confirmation bias. However, as RQ 5 indicates, news readers’ political tendencies do not statistically influence the political bias of news statements, and concerns about confirmation bias do not exist concerning algorithm-recommended political news.
Third, this study demonstrates that personalized news presents more neutral news articles than non-personalized news. In addition, we find that readers’ political orientation does not significantly influence the political bias of the news articles. While previous studies have found that news articles recommended by personalization algorithms on search engines and social media tend to be less politically biased than those selected directly by users (Cardenal et al., 2019; Robertson et al., 2023), there have been limited studies that statistically analyzed and quantified the political bias of news articles recommended by personalization algorithms, compared to those provided uniformly to the general public without personalization on news aggregators. Contrary to previous studies, this study directly measures the political bias of personalized and non-personalized news and examined the differences between the two.
This study suggests the following practical implications. First, personalized news on Internet portals have less diverse content (topics) and more diverse sources (media outlets). Thus, a multidimensional analysis is needed to assess the exact diversity provided by news services and to determine the existence of a filter bubble. Second, the findings quantitatively show that Korea’s Internet news portals provide politically neutral news articles through personalized news systems.
Third, Korea’s Internet news portals do not provide politically biased people with biased news to strengthen their political bias. Combining the results of RQs 3 and 5, personalized news provides more political news to conservative people, yet does not provide them with more conservatively biased news articles. Personalized news algorithms present individual readers with news articles on their preferred topics; however, they are not politically biased. These findings provide important points for discussions among policymakers and politicians regarding news on Korea’s Internet news portals.
Despite these implications, this study has certain limitations. First, this study analyzed a small sample in a short investigation period. Analysis of larger samples over a longer period must be conducted in future research for deeper insights. The research methods presented in this study would provide a meaningful methodology for analyzing the diversity and political bias of news articles recommended by algorithms.
Second, this study only included news aggregators and Internet news portals. It did not analyze other digital news services, such as social media and mobile applications of the press. This was relevant because a majority of people in Korea use news aggregators on Internet portals (Newman et al., 2020). However, a comparative analysis of other news providers is recommended.
Third, this study analyzed news articles provided by Naver News and Kakao News, Korea’s leading digital news services, which makes it difficult to compare the results with the findings of other countries. Further study of digital news service providers is important to generalize the results of this study.
Fourth, this study measured participants’ political orientation but did not assess their political interest or political knowledge. Although the sample in this study can be considered representative in terms of gender and service user ratio, its representativeness regarding political interest and political knowledge remains uncertain. Future research needs to analyze samples that are representative in these aspects.
Footnotes
Ethical Considerations
Formal ethics approval from the Institutional Review Board was not required under the regulations of Pukyong National University, as the research involved only an anonymous online survey, did not collect sensitive or personally identifiable information, and posed minimal risk to participants. All participants were fully informed about the purpose and procedures of the study and gave their informed consent voluntarily before completing the survey. The study was conducted in accordance with ethical research practices, including the APA Ethical Principles of Psychologists and Code of Conduct (2017).
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported by the MSIT (Ministry of Science and ICT), Korea, under the ITRC (Information Technology Research Center) support program (IITP-2023-2020-0-01749) supervised by the IITP (Institute of Information & Communications Technology Planning & Evaluation).
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Data Availability Statement
The data used in this study are available from the corresponding author upon reasonable request.