Sage Journals: Discover world-class research

Abstract

Online user comments (UCs) as the most popular type of online audience participation nowadays form a popular and important field of research. The widespread examination of UCs across different disciplines leads to a variety of terms and constructs and thus a missing clarity about the discussed topics. With this computational scoping review, we uncovered six relevant, overarching topics and their development in the field. Due to the combination of an automatic text analysis via structural topic modeling and a qualitative evaluation, we were able to describe the current state of UC research and found an inherently interdisciplinary body of literature. We observed an inter- and intradisciplinary fragmentation and call for a systematization of the used terms, constructs, and examined topics.

Keywords

Computational methods mixed methods online discourse social media user comments user recommendation

Online user comments (UCs) are nowadays the most popular form of online audience participation (Weber, 2014). Studies from various disciplines (e.g. communication science, psychology, political science, sociology) cover a wide range of topics and perspectives (democratic norms, Freelon, 2015; extremist behavior, Almoqbel et al., 2019; product perceptions, Yıldırım, 2013). Due to this widespread examination of UCs in different disciplines, a variety of terms and constructs, such as news or reader comments (Abdul-Mageed, 2008; Springer and Kümpel, 2018) in communication science, user reviews (Hu et al., 2019; Ye et al., 2009) in economics and self-help or social support groups (Wright, 2016) in health care and psychology have evolved. This fragmentation may hinder the progress of research on UCs and, due to missing clarity about the discussed topics in related fields, restricts the ability of researchers to effectively use the existing literature. While even these specific types of UCs are rarely reviewed in a systematic and aggregated way (Malinen, 2015; Naab and Sehl, 2017), we argue that the concepts in different disciplines are interconnected and future studies might benefit from an interdisciplinary discourse on the topic. Therefore, we offer a computational scoping review of this large, interdisciplinary and heterogeneous field, which to the best of our knowledge remains lacking. Through mapping the interdisciplinary field of UCs, we prepare existing literature for further processing and reveal research lines and fruitful future research questions. The results from this comprehensive literature review will be discussed regarding implications for a prospective research agenda and communication science’s role in the field.

User comments

The comment feature has become the most common way for users to contribute to various kinds of online content (Ernst et al., 2017; Hermida and Thurman, 2008; Weber, 2014). UCs are part of the broader concept of user-generated content (UGC; McCluskey and Hmielowski, 2012), which refers to phenomena like consumer reviews, blogs, blog aggregators, social networking sites (SNSs), web forums or wikis (Xiang and Gretzel, 2010). The variety of UGC is one reason why the field of UCs has been viewed as diffuse (McCluskey and Hmielowski, 2012) and differing terms and concepts arose. Due to this variety in UCs and the resulting heterogeneity within the existing literature, it is difficult to provide a generally valid definition of UCs. Therefore, we discuss definitions from previous literature and highlight similarities and differences. Based on this assessment, we develop a broad working definition for UCs that can guide our study. Thereby, we aim to capture the existing complexity and heterogeneity in the field by focusing on three of the most present concepts of UCs, namely, news comments (Weber, 2014), comments on SNS (Ernst et al., 2017), and user reviews (Tang et al., 2013). Even though all these forms of UCs are distinct and sometimes based on different technical features, we will argue in the following that they are all intertwined, thus, a broad view might be helpful to cross-connect between different subjects (e.g. product sales and public discourse) and to reinforce an interdisciplinary discourse on UC. This superordinate perspective will enable us to find overarching patterns and underlying core topics in the big frame of UCs. However, we are well aware that in individual studies other definitions might be more useful to answer specific research questions and that a working definition cannot be exhaustive.

UCs on news websites (“news comments” or “readers comments”) are considered highly relevant to society. This great significance is primarily justified by the prominent position of UCs, which are usually located directly below or at the end of news articles on news websites, thus potentially having the same reach as the commented journalistic items (Graham and Wright, 2015; Springer et al., 2015). In addition, the comment function allows users to switch from the passive receiver role to the active speaker role (Manosevitch and Walker, 2009; Springer et al., 2015). Both aspects are recognized as desirable for democratic processes (Manosevitch and Walker, 2009; Springer et al., 2015; Springer and Kümpel, 2018). Usually, news comments are informal and consist of short and unstructured responses of users “to a news item, expressing sentiment/opinion, a question, a rumor, or a call to action” (Liu et al., 2015: 769). Ziegele and Quiring (2013) define UCs

as a computer-mediated, written, public, and asynchronous subcategory of MSIC [media-stimulated interpersonal communication] that is published in the immediate context of a news item on the websites of mass-media organizations [. . .] or on their respective presences within other online communication services [. . .]. (p. 131)

They add that a sequence of UCs has to be called online discussion. Although on one hand their definition is rather narrow—UCs can only occur in the news and in a mass media environment—Ziegele and Quiring (2013), on the other hand, broaden the view by including various other online platforms, especially SNSs, in their definition in addition to traditional news websites.

The integration of other online platform types into the characterization of UCs is long overdue at this point. Since the emergence of the Internet, there have been forms of UCs that are not related to news. The earliest platforms that allowed commenting are so-called “web-based online services with features that enable members to communicate with each other” (Malinen, 2015: 228), including listservs, bulletin boards, and chatrooms. These early types of SNS were characterized by features such as publicness, anonymity, asynchronicity, organization by topics, and moderation (Witschge, 2008). Nowadays, these definite characterizations no longer apply, as major shifts have taken place due to the rapid development in the field of SNSs. While publicness is still a constituting characteristic of UCs, SNS, such as Facebook or Twitter, go beyond sheer public communication (Carr and Hayes, 2015). Anonymity in UCs is still a much-discussed topic (Graf et al., 2017), but no longer necessarily given (Halpern and Gibbs, 2013) as platforms like Facebook might enforce a real-name policy. Furthermore, interaction in SNS might be immediate and delayed at the same time (Carr and Hayes, 2015) and is not strictly organized by topics. Most importantly, scholars argue that UCs are not necessarily connected to the content they are published beneath, nor are they always favorable for democratic discussions or constructive by any means, especially on SNS (Ernst et al., 2017). Instead, UCs “may be considered as a standard feature of the web 2.0. Comments made mainly by unknown users appear below the various kind of online content (e.g. news articles, posts on SNSs, videos)” (Ernst et al., 2017: 4). Here, the scope of UC research broadened again by explicitly incorporating various types of online content in the very definition of UCs.

The view on UCs further expanded by including user-generated reviews, which can be found below various web 2.0 contents (Tang et al., 2013). Reviews are often seen as the electronic form of word-of-mouth (Baka, 2016; Ye et al., 2009) and are found to strongly influence sales of a variety of goods (e.g. Chevalier and Mayzlin, 2006). User reviews can be defined as “peer-generated evaluations posted on company or third-party websites” (Schuff and Mudambi, 2012: 186). Scholars add that they are positive or negative statements which are available online and made by potential, actual, or former customers about a product or company (Baka, 2016; Yıldırım, 2013). Yıldırım (2013) identifies reviews both as interpersonal interaction and as UCs since they “give instructions to other users and help what [they] should do” (p. 361). Furthermore, reviews also provide feedback to brands, websites, and customer services. On contrary, Duan et al. (2008) describe reviews as not just influencing product sales, but also as the outcome of product sales, potentially created independently to online content. Nevertheless, as reviews are published in an online context and may function as interpersonal communication, they can be classified as UCs (see Ernst et al., 2017).

Lately, automated practices of commenting undermine the idea of interpersonal interaction in UCs. For example, Mishne et al. (2005) find that up to two thirds of all comments are infiltrated with spam. The authors highlight effective methods to prevent this kind of automated commenting: registration, captchas, prevention of HTML, blacklists, and throttling (Mishne et al., 2005). These and other strategies are well established nowadays and part of most comment sections (Friess and Eilders, 2015; Ziegele, 2016), but spam, bots, and other automated communicators still pose societal and technical challenges for platforms and researchers.

In the previous paragraphs, various aspects of UCs were addressed and a plethora of different problems and definitions was revealed. A detailed examination of the individual aspects is certainly beneficial, but out of the scope of the present study. The field of UCs is a complex and extremely diverse area and even within certain contexts, such as news comments or user reviews, different perspectives exist. This heterogeneity of the existing work is useful to address individual research questions and describe different platforms and communication behavior. However, the objective of our work is to map as much of this diversity as possible to cross-connect between topically quite different but nevertheless intertwined subjects and help to reinforce an interdisciplinary discourse on UC. Accordingly, in this work, we highlight the similarities in different UC research areas and propose a functional working definition of UCs that reduces the complexity, serves as a guideline for our study, and offers a suitable basis for the systematic search terms.

Based on the aforementioned literature, we recall four aspects that in our opinion capture the intertwined nature of the existing types of UC, always being aware of the diversity and heterogeneity of UCs and the different underlying technical structures. First, UCs are computer-mediated. This is undisputed in the literature, no matter if it is about news comments (Ziegele and Quiring, 2013), comments on SNS (Ernst et al., 2017), or user reviews (Baka, 2016). To distinguish UCs from private online communication they have to be public, which is widely acknowledged as well (Baka, 2016; Witschge, 2008; Ziegele and Quiring, 2013). Most importantly, UCs are a form of interpersonal communication (Yıldırım, 2013; Ziegele and Quiring, 2013) that is made public in connection with some kind of online content (Ernst et al., 2017). Both aspects are controversial in the literature, as especially the presence of interpersonal communication is questioned in user reviews (Duan et al., 2008), while the connection to online content in the field of news comments is often limited to news content (Ziegele and Quiring, 2013). However, more recent definitions widened the scope of UC research by accentuating that user reviews can very well serve as interpersonal communication (Yıldırım, 2013) and news articles are only one form of online content among others that are followed by UCs (Ernst et al., 2017). Considering these key common elements from previous definitions, we propose in our working definition to describe a UC as a computer-mediated, public, and interpersonal form of communication, which is published in connection with online content.

Interdisciplinary research on user comments

While there was a limited interest in the field of communication science for the topic 20 years ago (Tomasello, 2001), the research on concepts related to UCs is now increasingly prominent. Scholars stated in different literature reviews that the most important online communities are discussion forums or bulletin boards (mostly dedicated to a certain topic, such as health or online learning), blogs, as well as SNS (Malinen, 2015; Naab and Sehl, 2017), and certain journals have a leading role in publishing about UGC (e.g. Journal of Computer-Mediated Communication and New Media & Society; Naab and Sehl, 2017). However, all these studies focused on UGC and did not pay special attention to UCs themselves. One active subfield of UCs in communication research has been news comments and their role in democratic societies. On one hand, empirical evidence showed that both, journalists and the public, generally did not value users’ comments very highly, as they often were malicious and abusive (Hermida and Thurman, 2008; Noci et al., 2012; Reich, 2011). On the other hand, UCs are the most popular form of public online participation (Reich, 2011; Weber, 2014) and were viewed as desirable, as they could contribute to democratically valuable and vivid interpersonal discourse on topics of public interest (Freelon, 2010).

In economics, UCs are viewed as one important information source for consumers’ purchase decision as they compensate for perceived risk and information asymmetries (Cheng and Zhou, 2010). There is broad evidence that UCs have a direct impact on sales for a variety of products and services, specifically hotel bookings (Ye et al., 2009), movies (Ghose and Ipeirotis, 2010), video games (Zhu and Zhang, 2006), or restaurants (Zhang et al., 2010). While the valence of the reviews influences the purchase decision, whereby negative reviews seem to have a greater effect than positive ones, the quality of the UCs is important too—high quality reviews are more likely to have an impact on the consume decision (Cheng and Zhou, 2010). Furthermore, UCs have an impact on the perceived media bias and third-person effect in online media (Houston et al., 2011) and are raising the persuasiveness of online articles in the short term, but not in the long term (Heinbach et al., 2018). Nevertheless, the precise effects are inconclusive due to varying approaches and differing components in studies on UCs.

Online forums, support groups and SNSs are particularly popular because of the inexpensive and convenient provision of help they offer (DeHoff et al., 2016; Eysenbach et al., 2004). Therefore, UCs play an important role in health research and psychology. Furthermore, UCs provide the opportunity to gain information about illnesses and effects of drugs or medication due to self-reported patient data as well as the chance to give feedback to doctors and their medical staff (Verhoef et al., 2014). In the field of psychology, the reflection on effects of UCs ranges from the negative impact that cyberbullying on SNSs has on children (Alsehaima and Alanazi, 2018) to the occurrence of the bystander effect in UCs and how to deal with it (Naab et al., 2018). Mental health and well-being also play an important role here (DeForte et al., 2020) as well as the effect of UCs on users’ perception of, for example, the tentativeness of research findings (Feinkohl et al., 2016).

Many more research fields and topics related to UCs exist that could not be discussed in detail here. As a result of technical innovations and developments, the field is constantly growing in its quantity and diversity, however, a comprehensive overview of the field of UCs is missing until now.

Present study

A first step in mapping a research field is to describe how researchers have published in it up until now (Meier et al., 2020). To illustrate the evolution of UC research and to reveal the most prolific UC research outlets, we will analyze the distribution of publications over time and the outlets in which studies regarding UCs are published. The disciplines in which research on UCs are published allow us to map the most relevant fields of research and give us an idea of which concepts might be most present regarding UCs. This results in our first research question:

RQ1. To what extent are researchers publishing about UCs (a) over time and (b) across outlets and (c) disciplines?

Furthermore, we aim to systematically identify the underlying core topics that have received attention in interdisciplinary research on UCs. This is most relevant due to the use of diverse terms and concepts in research on UCs and will help to combine the knowledge generated in different disciplines regarding these topics. Studying the development of these topics over time may provide us with insights into whether and how certain issues dominate the field and whether others have fallen or risen in prestige. Therefore, our other research question is as follows:

RQ2. What are (a) the main topics studied in UC research and how are (b) these topics distributed over time?

Method

Scoping review methodology

A scoping review can map a complex research field (Arksey and O’Malley, 2005) and is viewed as particularly helpful for research fields that are highly heterogeneous in nature (Mays et al., 2001) and fast growing. The UC field concerns a relatively broad issue that hasn’t been defined clearly in the literature yet, while also being part of a growing research field (Günther and Domahidi, 2017). Thus, a review approach with manual search and coding would be very resource-intensive and outdated upon completion. Therefore, we used computational methods that can deal with a large number of studies and that have demonstrated their potential in recently published studies on similar problems (Günther and Domahidi, 2017).

Sample

We systematically developed search terms (see Online Appendix 1) consisting of several names for UCs and connected constructs (e.g. news comments, online comments, user reviews) as well as platforms that allow for comments and which provide a wide range of coverage. To consider the different disciplines, we selected the ten largest SNSs, question-and-answer, e-commerce, rating platforms, forums and news sites (see Online Appendix 1 for further details). The original search string was enhanced and manually validated iteratively in multiple steps. The final search string was then applied to articles’ titles in 11 relevant databases (see Online Appendix 2) via the meta-database EBSCO Host from 1 January 2000 until 4 November 2019. We chose the year 2000 as a limit because UC sections became more present afterwards (Reich, 2011; Tomasello, 2001). To examine the extent, range, and nature of research activity, we included academic and non-academic journals, overviews, books, dissertations, and conference materials in our search. All documents had to be written in English to ensure comparability for analyses.

After downloading the data from EBSCO (n = 1969), we excluded all duplicates and entries with missing abstract, title, journal, and/or author data, resulting in our final sample of n = 1240 potentially relevant documents. All our analyses were based on articles’ abstracts and metadata (e.g. title, year, journal).

Analytical approach

Publication behavior

Based on the Social Science Citation Index (SSCI) categorization or, if they could not be found in the SSCI, on journals’ self-descriptions and Internet-search based, we assigned disciplines to each outlet in our sample. Sub-categories (e.g. psychiatry/psychology and experimental psychology) were summarized (e.g. psychology; see Online Appendix 3). When more than two main disciplines were listed for a journal in SSCI or the journals’ self-description, we coded them as multidisciplinary (n = 446). Dissertations, conference papers, and books were coded according to their self-descriptions (n = 76).

Topic modeling

To process our sample, we used topic modeling, a statistical technique for analyzing large corpora of documents without reading or annotating the original texts (Blei, 2012). Given our sample’s characteristics, we used structural topic modeling (STM), which is designed to address differences in topical content, resulting from external variables, such as time of writing or field of publication (Roberts et al., 2014). We used publication time and discipline as metadata in our model. Before analysis, common preprocessing steps, such as removal of punctuation, numbers, and stop words were implemented, followed by word stemming and term frequency–inverse document frequency (TF–IDF) weighting (Manning et al., 2008; Roberts et al., 2014). We estimated an STM based on the text in articles’ abstracts using the STM package in R (Roberts et al., 2014). To select the number of topics, we estimated 21 topic models from k = 5 to k = 25 and selected the 13-topic model as the most useful. We then estimated a set of 20 separate 13-topic STMs with different initial parameters and selected the model from this set which maximizes our corresponding topic-word vectors’ semantic coherence and exclusivity (Roberts et al., 2014). All data management, cleaning, and analysis were performed using R (R Core Team, 2018). We manually checked abstracts, titles, and top words for 20 randomly selected documents per topic after the automated analysis to make sure that the topics are meaningful.

Manual analysis

To reduce the complexity of further analyses, we merged and selected topics that fit together based on our qualitative assessment of the degree of thematic overlap between the selected topics based on individual topics’ top words and the associated abstracts. This resulted in seven more general topics (Table 1) based on common research themes. Scientific Characteristics, created out of the individual topics Science and Methods, reflected mainly designs and methods of scientific papers and were excluded from the following analyses. Note that the final topics are a distribution of terms and each document is a mixture of different topics, therefore they are not mutually exclusive from each other and every study on UCs will consist of several of the found topics. To ease interpretation, the most prominently observed topic will be assigned for each document.

Table 1.

Initial topics with top words and merged topics.

Top 10 words (word stem)	Proportion	Labeled topics	Merged topics
review, onlin, consum, product, help, effect, purchas, custom, studi, market	.127	Recommendation	User recommendation
review, negat, posit, hotel, onlin, effect, respons, influence, rate, custom	.101	Hospitality
user, rate, recommend, generat, base, use, can, restaur, app, new	.054	User traits
social, media, post, site, share, facebook, network, communic, use, other	.054	Social media	SNSs
student, languag, video, use, onlin, discours, youtub, articl, author, use	.076	Interactivity	SNSs
emot, onlin, patient, cultur, physician, negat, feedback, profession, websit, care	.046	Health care	Health care and lifestyle
health, report, public, includ, comment, articl, intervent, issu, identifi, standard	.070	Lifestyle	Health care and lifestyle
studi, use, research, data, content, analysi, find, design, paper, one	.087	Science	Scientific Characteristics
review, sentiment, propos, method, featur, model, opinion, product, base, extract	.094	Methods	Scientific Characteristics
review, inform, qualiti, evalu, sourc, credibl, system, use, decis, result	.071	Online trust	Online trust
comment, news, onlin, reader, articl, effect, public, studi, user, particip	.112	Deliberation	Online discourse
engag, critic, onlin, contribut, messag, argument, format, digit, behavior, theory	.052	Online discussion	Online discourse
factor, inform, present, relat, provid, busi, satisfact, adopt, servic, framework	.056	Discourse architecture	Discourse architecture

Results

Publication behavior in the field

Our results revealed that the number of publications on UCs has increased over time (see Online Appendix 4). While only n = 6 papers were published at the beginning of the millennium, more than 30 times as many (n = 205) were published 18 years later. Note that we observed a decline in 2019, which most likely was the result of finishing data collection in early November 2019. Most of the 566 unique publication outlets in our sample provided less than five papers (n = 513). Dissertations on UCs in the humanities (n = 48) and social sciences were the largest outlet on UCs in the sample. Interestingly, the most dominant journals mainly were associated with economics, while genuine communication science journals were not represented among the 10 outlets with the largest shares.

Overall, economics published most studies in the field of UCs (n = 247), followed by communication science (n = 111), computer science (n = 87), health and medicine (n = 85), and psychology (n = 71). Out of 23 disciplines in the data set, only nine occurred more than 20 times, namely, the aforementioned and the following: library science (n = 31), education (n = 30), and engineering (n = 23). Interestingly, communication science was the second most important discipline. Note that multidisciplinary sciences had the largest share, with n = 447 publications.

The increase in the number of publications in most disciplines (Figure 1) began after 2010. Before that, only economics and the multidisciplinary sciences surpassed a single-digit number of publications. Among the five most relevant disciplines, health care, psychology, and communication science have stagnated or declined in recent years, while computer science, economics, and multidisciplinary sciences continued to grow in the number of publications.

Figure 1.

Number of publications over year and disciplines (with at least 20 documents overall).

Most important topics in the field

Here, we discuss our final topics (Table 1) and include examples of recent and thematically appropriate publications to illustrate each respective topic (see Online Appendix 5 for full references of these studies).

User recommendation

User recommendation was the most prominent topic in our sample (n = 410). Many papers on this topic have concentrated on the description of UCs in specific environments, for example, Yelp, TripAdvisor, health-related groups or forums (Fitchett and Hoogendoorn, 2019; Liu et al., 2019; Meyer et al., 2019; Stoleriu et al., 2019). In conjunction with this, user recommendations’ characteristics were discussed, for example, credibility (Craciun and Moore, 2019), values (Ham et al., 2019; Zhang et al., 2018), distrust (Ahmad and Sun, 2018), and private self-awareness (Sohn et al., 2019). However, researchers also tried to extract opinions (Jia, 2018; Kim and Noh, 2019; Tang, 2017) and meanings (Bicquelet, 2017), out of user-generated data. Furthermore, UCs’ impact on consumers’ decision-making processes (Hernandez-Ortega, 2019; Pekgün et al., 2018; Wang et al., 2018), social norms in the context of newspapers (Chung, 2019), (dis)encouragement from engagement (Kim and Park, 2019), the need to seek news after reading comments (Jahng, 2018), emotional and social contagion (Kwon and Gruzd, 2017), behavioral change (Lin et al., 2019), and the perception of online science articles (Winter and Krämer, 2016) was discussed. Overall, the vast majority of studies under user recommendation focused on assessing user recommendations’ importance and providing guidance for users, media, companies, or policy makers on how to deal with UCs and their inherent characteristics.

In addition, more technically oriented research regarding recommender systems played a role in our sample. Classic recommendation techniques, such as content-based approaches (Chen and Chen, 2015) and especially collaborative filtering (Mirza et al., 2018; Pradhan, 2017; Xing et al., 2019; Xu et al., 2018), were present, while studies on social recommendations were rare.

Online discourse

The second largest topic in our sample (n = 224) focused on UCs as an essential part of online discussions by assessing different quality categories (e.g. civility and rationality) of the discussion, mainly based on the concept of deliberative democracy.

Research on civility spans a wide field and the effects from uncivil or hateful comments were examined in our sample, for example, the amount of money donated to refugees, which decreased after consuming hateful comments (Ziegele et al., 2018), while Santinele Martino and Andrejek (2019) found incredulous acting, pathologizing, shaming, and sometimes-violent reactions toward trans-abled people in UCs. Some studies evaluated how to react to negative online reviews as a physician (Jonke, 2019; Widmer et al., 2019) or hotel manager (Olimpia, 2019). Incivility in comments also affected the perception of journalistic content by decreasing the perceived quality of news items (Weber et al., 2019). Hsu et al. (2019) offered a method to extract negative comments automatically. Koban et al. (2018) investigated personality traits as predictors for civil/uncivil commenting. The presented studies all focused on the lack of civility in online discussions, but the described effects affect various aspects of online communication. Furthermore, different incivility levels were discussed, ranging from impolite comments to hate speech (Ziegele et al., 2018), while some studies evaluated how to react to this negativity (Jonke, 2019; Olimpia, 2019).

Argument quality and rationality also were discussed in different contexts, such as how to extract arguments and opinions from user reviews (Kim and Kim, 2019; Korfiatis et al., 2019), enhance online consumer reviews’ helpfulness (Srivastava and Kalro, 2019), or encourage engagement in online discussions (Kwon et al., 2019). In addition, the concept of identity in online discussions (Meredith and Richardson, 2019), perceived descriptive norms (Liu and Shi, 2019), or the relevance of online discourses’ design (Vendemia et al., 2019; Wang et al., 2019) also played a role.

Social networking sites

Social interaction and communication were the overarching motives in this topic (n = 177), with a strong emphasis on a broad variety of communication platforms that have increasing user bases and provide the opportunity for a large audience to comment: YouTube (Döring and Mohseni, 2019; Marcon and Caulfield, 2017; Weijs et al., 2019), Facebook (Linker et al., 2018), Twitter (Smith et al., 2018; Trilling, 2015), Reddit (Flesch, 2019), Yelp (Nakayama and Wan, 2018), TripAdvisor (van Laer et al., 2019), and blogs (Dowd, 2017), among other forms. Comparative studies between different platforms were also present (Aerts et al., 2017b; Ben-David and Soffer, 2019; Thomas-Meyer et al., 2017).

Similarly, a wide range of phenomena on these platforms was reported: echo chambers in the climate-change discourse (Walter et al., 2018), the importance of comments and ratings in decision making on SNSs (Carbonell and Brand, 2018), shared airline passengers’ perceptions of service quality (Brochado et al., 2019), the detection and extraction of hidden networks in e-governmental websites (Alguliyev et al., 2019), hate speech (Döring and Mohseni, 2019), and linguistic phenomena (Flesch, 2019).

Health care and lifestyle

Online discussions on health issues have received much research attention over the past few years (see Figure 2; n = 154). For example, researchers studied stigma and/or negativity in comments on weight-related news (Brooker et al., 2018), surgeons (Donnally et al., 2018), breastfeeding (Len-Rios, Bhandari and Medvedeva, 2014), on LGBT people confronted with hate speech (Baider, 2018), and tested whether one-sided UCs affect opinions and intentions about home birth (Witteman et al., 2016), as well as narrative formats and UCs’ role in preventive health measures (Stavrositu and Kim, 2015). Apart from the aforementioned research, studies were conducted on weight-loss apps (Frie et al., 2017), rape myths on Facebook (Clay, 2019), and online professional textual feedback’s value (Tang et al., 2015).

Figure 2.

Number of publications over years and topics.

In addition, a wide range of lifestyle themes was examined. One subtopic was tourism and hospitality with research on the importance of UCs for hotels (Fernandes and Fernandes, 2018; Fong et al., 2017; Ghosh, 2018), restaurants/fast food chains (Beuscart et al., 2016; Huang, 2017; Šerić and Praničević, 2018; Sun et al., 2019; Yang et al., 2017), airlines (Siering et al., 2018), and the tourism and hospitality sector in general (Agarwal, 2015). Furthermore, studies on fashion (Camacho-Otero et al., 2019), art and censorship (Mokwena and Banda, 2019), online shopping (Hu, 2019), and mobile apps (Palomba et al., 2018) could also be found in our sample.

Online trust

Some studies on the topic of online trust (n = 52) focused on UCs and reviews’ trustworthiness, for example, Valdivia et al. (2019) presented a method for coping with inconsistencies in user reviews and how this leads to more trustworthy insights. Inconsistencies in UCs and reviews often were based on biases (Sikora and Chauhan, 2012), such as the popularity effect (Goes et al., 2014), the contagion effect from preceding comments (Aerts et al., 2017a), and the use of emotional arousal and emoticons (Candi et al., 2017; Manganari and Dimara, 2017).

Aside from the abstract use of the term online trust, studies focused on trust/distrust in public online discussions, for example, on immunizations (Tustin et al., 2018) and climate skepticism (Kaiser, 2017).

A third subtopic was a very general consideration of UCs’ potential effects on product rankings (Lee et al., 2019), product sales (Li et al., 2019), hotel bookings (Liu and Ji, 2019), consumer decisions (Roozen and Raedts, 2018; Willemsen et al., 2011; Xiao and Li, 2019), and opinion changes (Hong and Cameron, 2018).

Discourse architecture

Studies on discourse architectures (n = 36) lately focused on the investigation of singular design features’ direct impact. Heng et al. (2018) showed that UCs’ length impacts the perceived helpfulness of user reviews, while Naab et al. (2018) investigated the flagging of uncivil UC sections. Wan et al. (2018) analyzed different design features (e.g. sorting, reading limits, releasing strategies) that could influence user reviews in e-commerce environments. In addition, we found comparative research between website architectures, for example, between physician-review websites (Kordzadeh, 2019).

Changes over time

Regarding the second part of RQ2, our results revealed that the development of all six discussed topics started to increase sharply after 2010 (see Figure 2). Particularly for the topic with the largest share in our sample, user recommendation, only 36 publications could be found before this date. Between 2016 and 2019, the number of annual publications exceeded this by far (n₂₀₁₆ = 47; n₂₀₁₇ = 48; n₂₀₁₈ = 70; n₂₀₁₉ = 65). The same was true for discourse architecture and online trust, each of which also had around 10% of its publications published before 2010 (discourse architecture: n = 4; online trust: n = 6). Overall, we saw a clear rise in research on UCs since 2000, with a sharp increase after 2010.

Discussion

UCs were examined in this article as a form of interpersonal public online communication in which people post comments below diverse content from communicators, such as journalists, politicians, companies, or other users. A vast amount of research has been conducted on these so-called “below the line” (Graham and Wright, 2015) comments. However, due to manifold research interests and approaches, studies on UCs are dispersed across disciplines and subfields, which help to reduce to complexity of the field and to answer the variety of research questions. To offer superordinate insights into the variety and diversity of UC research and to foster the possibility to engage more in cross-disciplinary discussions, we analyzed the core topics and publication behavior in the field of UCs by combining a classic scoping review methodology and innovative computational methods. However, the reviews findings can only illustrate the publication behavior as represented by our sample. While we created, tested and improved the extensive list of search terms that we included to reflect the field’s interdisciplinary nature, no list will be complete and it is probable that our search terms will not cover all research on the topic. Moreover, the population of relevant studies is unknown, databases can have systematic errors and researchers’ access to the database content is limited (in the case of EBSCO Host, for example, by the subscription purchased by each university). Thus, even a systematic approach has inevitable limitations and not all studies, journals and disciplines might be represented equally well by this research. To be able to process a large sample in a language-dependent content analysis, we could include only English publications. Furthermore, due to our broad scope, we didn’t distinguish between biological and non-biological commenters. However, there are clear indications that automated practices of commenting are an important topic to consider in future research (Mishne et al., 2005).

Our results reveal that the interest in UCs has increased strongly in the past 10 years. While this growth in the number of publications can be found across different fields of communication (Günther and Domahidi, 2017), this might be attributed to the increased use and dissemination of UC sections in various online contexts as well (Reich, 2011). Our broad perspective on UCs allows us to consider different disciplines and reflect on communication science’s role in the field. We found dominance of economic-oriented research on UCs, mostly concerning user reviews, followed by research from communication science. The publication output from communication science on the topic was consistently high in our sample, but did not grow over time, unlike the numbers of publications in economics or health sciences. However, we observed a strong increase in multidisciplinary publications to which communication science probably contribute a relevant part due to the field’s interdisciplinary nature (Leydesdorff and Probst, 2009). As we followed the SSCI categorization to reveal disciplines’ impact on the research field, we only assessed journals’, not researchers’, disciplinary affiliations. Future studies could refine our approach.

With the combination of an automatic text analysis via structural topic modeling and a qualitative evaluation, we found six relevant main topics in our sample: user recommendation, online discourse, SNS, health care and lifestyle, online trust, and discourse architecture, whereof user recommendation clearly dominates the field. Research on recommendations and recommender systems became relevant in the mid-1990s (Adomavicius and Tuzhilin, 2005) and is linked to a variety of disciplines (Tang et al., 2013). Two main foci on user recommendations exist in the literature: the systems in which these user recommendations are published and the user recommendations’ content itself. Under the user recommendation topic, studies that describe UCs’ characteristics were very common in our sample. This implies that, regardless of disciplines, the actual content of UCs is considered relevant and a variety of approaches are tested to access them as a valuable resource. Nevertheless, the used characteristics such as quality (Weber et al., 2019) or usefulness (Hernandez-Ortega, 2019) are vague and differ inter- and intradisciplinary, which reflects the existing ambiguity on UCs. We observe a similar focus for online discourse that often refers to the quality of user contributions. We propose to build on the existing literature in both cases in order to strengthen and clarify constructs that are applicable across disciplines. Such interdisciplinary valid constructs might be characteristics of UCs, such as civility, factuality, objectivity, or accuracy. Research on recommendation systems was less present in our sample, even though we found several studies in further literature, especially on how they can be improved (Xu et al., 2018). Due to the interdependence between the environment in which comments are made and the comments themselves, it seems reasonable to combine these two separate research foci in future research to obtain fruitful insights into the topic.

SNS, as one of the most relevant themes in communication science research (Cappella, 2017), is represented in our sample mainly by studies that use data from different SNS platforms (e.g. YouTube, Facebook, Twitter, Reddit). While studies on individual platforms are most common (Momeni et al., 2013), only a few comparative studies on various platforms (Carrascosa et al., 2018; Jiang and Wilson, 2018) and from combined (social) media data sources exist (Ben-David and Soffer, 2019). Generally, we could not find any in-depth evaluations on the possible frameworks for SNS research on UCs.

Several topics in our sample (online discourse, online trust, and discourse architecture) analyzed the terms of discourse in online environments. Research in the named topics provides a broad-based number of insights on many of these aspects. Studies on online discourse or online trust focus on particular aspects, such as quality dimensions (Ziegele et al., 2018) or the trustworthiness of online discourses (Tustin et al., 2018). Studies on health care and lifestyle applied these questions on one concrete topic, such as malicious and stigmatizing comments on weight-related news (Brooker et al., 2018). In addition, research on discourse architecture reflected these aspects mostly from a technical perspective such as the flagging of uncivil comments in comment sections (Naab et al., 2018). These insights call for a joint reflection on the interplay between the mentioned parts of online discourse. Friess and Eilders (2015) offer a first attempt of an integrated framework in the field but are limited to online deliberation. Future research should pick up this important impulse. However, an analysis via topic modeling only reveals superordinate topics and that more detailed insights into possible subtopics can be best achieved through qualitative approaches in future studies.

UCs “are a research subject of continuous and fast technical, economic, and social change” (Naab and Sehl, 2017: 1268). Therefore, it is most likely that the drawn image in this review will gradually change over time. However, the presented snapshot in this scoping review reveals that UCs are a research object that is compatible with many areas of communication research and is even adopted in a broad variety of other research fields. Due to this interdisciplinarity and the missing integration of various approaches, we invite other researchers to build on the presented insights in this heterogeneous field, for example, by getting involved in cross-disciplinary discussions. This could start, for example, with a typology about the different types of UCs. Furthermore, future research could use our detected topics to further specify and systematize the discovered subfields. Communication science, as one of the main actors in the field and an inherently interdisciplinary domain, should take a leading role in shaping the field of UCs for future research.

Supplemental Material

sj-pdf-1-nms-10.1177_1461444821994491 – Supplemental material for The growing field of interdisciplinary research on user comments: A computational scoping review

Supplemental material, sj-pdf-1-nms-10.1177_1461444821994491 for The growing field of interdisciplinary research on user comments: A computational scoping review by Max Schindler and Emese Domahidi in New Media & Society

Footnotes

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Max Schindler

Emese Domahidi

Supplemental material

Supplemental material for this article is available online.

Author biographies

Max Schindler, MA, is a PhD candidate at the Department of Communication Science with focus on Computational Communication Science, Ilmenau University of Technology.

Emese Domahidi, PhD, is a junior professor at the Department of Communication Science with focus on Computational Communication Science, Ilmenau University of Technology. Her research focuses on (cognitive) biases during the reception of digital media, and social consequences of online media use.

References

Abdul-Mageed

(2008) Online news sites and journalism 2.0: reader comments on Al Jazeera Arabic. Triplec: Communication, Capitalism & Critique. Open Access Journal for a Global Sustainable Information Society 6(2): 59–76.

Adomavicius

Tuzhilin

(2005) Toward the next generation of recommender systems: a survey of the state-of-the-art and possible extensions. IEEE Transactions on Knowledge and Data Engineering 17(6): 734–749.

Almoqbel

Wohn

Hayes

, et al. (2019) Understanding Facebook news post comment reading and reacting behavior through political extremism and cultural orientation. Computers in Human Behavior 100: 118–126.

Alsehaima

Alanazi

(2018) Psychological and social risks to children of using the internet: literature review. Journal of Child & Adolescent Behaviour 6(380): 2.

Arksey

O’Malley

(2005) Scoping studies: towards a methodological framework. International Journal of Social Research Methodology 8(1): 19–32.

Baka

(2016) The becoming of user-generated reviews: looking at the past to understand the future of managing reputation in the travel sector. Tourism Management 53: 148–162.

Ben-David

Soffer

(2019) User comments across platforms and journalistic genres. Information, Communication & Society 22(12): 1810–1829.

Blei

(2012) Probabilistic topic models. Communications of the ACM 55(4): 77–84.

Brooker

Barnett

Vines

, et al. (2018) Doing stigma: online commenting around weight- related news media. New Media & Society 20(9): 3201–3222.

10.

Cappella

(2017) Vectors into the future of mass and interpersonal communication research: big data, social media, and computational social science. Human Communication Research 43(4): 545–558.

11.

Carr

Hayes

(2015) Social media: defining, developing, and divining. Atlantic Journal of Communication 23(1): 46–65.

12.

Carrascosa

Mendez

Triga

(2018) EU Cohesion Policy in the Media: A Computational Text Analysis of Online News, User Comments and Social Media [Report]. COHESIFY. Available at: https://strathprints.strath.ac.uk/70279/ (accessed 9 September 2020).

13.

Cheng

Zhou

(2010) Study on effect of eWOM: a literature review and suggestions for future research. In: 2010 international conference on management and service science, Wuhan, China, 24–26 August, pp. 1–4. New York: IEEE.

14.

Chevalier

Mayzlin

(2006) The effect of word of mouth on sales: online book reviews. Journal of Marketing Research 43(3): 345–354.

15.

DeForte

Huang

Bourgeois

, et al. (2020) The association between app-administered depression assessments and suicidal ideation in user comments: retrospective observational study. JMIR Mhealth and Uhealth 8(8): e18392.

16.

DeHoff

Staten

Rodgers

, et al. (2016) The role of online social support in supporting and educating parents of young children with special health care needs in the United States: a scoping review. Journal of Medical Internet Research 18(12): e333.

17.

Duan

Whinston

(2008) Do online reviews matter?—An empirical investigation of panel data. Decision Support Systems 45(4): 1007–1016.

18.

Ernst

Schmitt

Rieger

, et al. (2017) Hate beneath the counter speech? A qualitative content analysis of user comments on YouTube Related to counter speech videos. Journal for Deradicalization 10: 1–49.

19.

Eysenbach

Powell

Englesakis

, et al. (2004) Health related virtual communities and electronic support groups: systematic review of the effects of online peer to peer interactions. BMJ 328(7449): 1166.

20.

Feinkohl

Flemming

Cress

, et al. (2016) The impact of personality factors and preceding user comments on the processing of research findings on deep brain stimulation: a randomized controlled experiment in a simulated online forum. Journal of Medical Internet Research 18(3): e59.

21.

Freelon

(2010) Analyzing online political discussion using three models of democratic communication. New Media & Society 12(7): 1172–1190.

22.

Freelon

(2015) Discourse architecture, ideology, and democratic norms in online political discussion. New Media & Society 17(5): 772–791.

23.

Friess

Eilders

(2015) A systematic review of online deliberation research. Policy & Internet 7(3): 319–339.

24.

Ghose

Ipeirotis

(2010) Estimating the helpfulness and economic impact of product reviews: mining text and reviewer characteristics. IEEE Transactions on Knowledge and Data Engineering 23(10): 1498–1512.

25.

Graf

Erba

Harn

(2017) The role of civility and anonymity on perceptions of online comments. Mass Communication and Society 20(4): 526–549.

26.

Graham

Wright

(2015) A tale of two stories from “Below the Line”: comment fields at the Guardian. The International Journal of Press/Politics 20(3): 317–338.

27.

Günther

Domahidi

(2017) What communication scholars write about: an analysis of 80 years of research in high-impact journals. International Journal of Communication 11: 3051–3071.

28.

Halpern

Gibbs

(2013) Social media as a catalyst for online deliberation? Exploring the affordances of Facebook and YouTube for political expression. Computers in Human Behavior 29(3): 1159–1168.

29.

Heinbach

Ziegele

Quiring

(2018) Sleeper effect from below: long-term effects of source credibility and user comments on the persuasiveness of news articles. New Media & Society 20(12): 4765–4786.

30.

Hermida

Thurman

(2008) A clash of cultures. Journalism Practice 2(3): 343–356.

31.

Hernandez-Ortega

(2019) Not so positive, please! Effects of online consumer reviews on evaluations during the decision-making process. Internet Research 29(4): 606–637.

32.

Houston

Hansen

Nisbett

(2011) Influence of user comments on perceptions of media bias and third-person effect in online news. Electronic News 5(2): 79–92.

33.

Zhang

Gao

, et al. (2019) What do hotel customers complain about? Text analysis using structural topic model. Tourism Management 72: 417–426.

34.

Jiang

Wilson

(2018) Linguistic signals under misinformation and fact-checking: evidence from user comments on social media. Proceedings of the ACM on Human-Computer Interaction 2: 1–23.

35.

Leydesdorff

Probst

(2009) The delineation of an interdisciplinary specialty in terms of a journal set: the case of communication studies. Journal of the American Society for Information Science and Technology 60(8): 1709–1718.

36.

Liu

Zhou

Zhao

(2015) Understanding news 2.0: a framework for explaining the number of comments from readers on online news. Information & Management 52(7): 764–776.

37.

McCluskey

Hmielowski

(2012) Opinion expression during social conflict: comparing online reader comments and letters to the editor. Journalism: Theory, Practice & Criticism 13(3): 303–319.

38.

Malinen

(2015) Understanding user participation in online communities: a systematic literature review of empirical studies. Computers in Human Behavior 46: 228–238.

39.

Manning

Raghavan

Schütze

(2008) Introduction to Information Retrieval. Cambridge: Cambridge University Press.

40.

Manosevitch

Walker

(2009) Reader comments to online opinion journalism: a space of public deliberation. In: International symposium on online journalism, Austin, TX, 17–18 April.

41.

Mays

Roberts

Popay

(2001) Synthesising research evidence. In: Fullop

Allen

Clarke

, et al. (eds) Studying the Organisation and Delivery of Health Services: Research Methods. London: Routledge, pp. 188–220.

42.

Meier

Domahidi

Günther

(2020) Computer-mediated communication and mental health: a computational scoping. In: Rice

(ed) The Oxford Handbook of Digital Technology and Society. Oxford: Oxford University Press, pp. 79–110.

43.

Mishne

Carmel

Lempel

(2005) Blocking blog spam with language model disagreement. In: AIRWeb, Chiba, Japan, 10 May.

44.

Momeni

Tao

Haslhofer

, et al. (2013) Identification of useful user comments in social media: a case study on flickr commons. In: Proceedings of the 13th ACM/IEEE-CS joint conference on digital libraries, Indianapolis, IN, 22–26 July, pp. 1–10. New York: ACM.

45.

Naab

Sehl

(2017) Studies of user-generated content: a systematic review. Journalism 18(10): 1256–1273.

46.

Naab

Kalch

Meitz

(2018) Flagging uncivil user comments: effects of intervention information, type of victim, and response comments on bystander behavior. New Media & Society 20(2): 777–795.

47.

Noci

Domingo

Masip

, et al. (2012) Comments in news, democracy booster or journalistic nightmare: assessing the quality and dynamics of citizen debates in Catalan online newspapers. In: International symposium on online journalism, Austin, TX, 20–21 April. Available at: https://isoj.org/wp-content/uploads/2016/10/ISOJ_Journal_V2_N1_2012_Spring.pdf#page=46 (accessed 9 September 2020).

48.

R Core Team (2018) R: A Language and Environment for Statistical Computing. Vienna: R Foundation for Statistical Computing. Available at: https://www.R-project.org/

49.

Reich

(2011) User comments: the transformation of participatory space. In: Singer

Hermida

Domingo

, et al (eds) Participatory Journalism. Chichester: Wiley-Blackwell, pp. 96–117.

50.

Roberts

Stewart

Tingley

(2014) Stm: R package for structural topic models. Journal of Statistical Software 10(2): 1–40.

51.

Schuff

Mudambi

(2012) What makes a helpful online review? A study of customer reviews on Amazon. com. Social Science Electronic Publishing 34(1): 185–200.

52.

Springer

Kümpel

(2018) User-generated (dis) content. In: Nuernbergk

Neuberger

(eds) Journalismus im Internet. Wiesbaden: Springer, pp. 241–271.

53.

Springer

Engelmann

Pfaffinger

(2015) User comments: motives and inhibitors to write and read. Information, Communication & Society 18(7): 798–815.

54.

Tang

Liu

(2013) Social recommendation: a review. Social Network Analysis and Mining 3(4): 1113–1133.

55.

Tomasello

(2001) The status of Internet-based research in five leading communication journals, 1994–1999. Journalism & Mass Communication Quarterly 78(4): 659–674.

56.

Tustin

Crowcroft

Gesink

, et al. (2018) User-driven comments on a Facebook advertisement recruiting Canadian parents in a study on immunization: content analysis. JMIR Public Health and Surveillance 4(3): e10090.

57.

Verhoef

van de Belt

Engelen

, et al. (2014) Social media and rating sites as tools to understanding quality of care: a scoping review. Journal of Medical Internet Research 16(2): e56.

58.

Weber

(2014) Discussions in the comments section: factors influencing participation and interactivity in online newspapers’ reader comments. New Media & Society 16(6): 941–957.

59.

Weber

Prochazka

Schweiger

(2019) Why user comments affect the perceived quality of journalistic content: the role of judgment processes. Journal of Media Psychology: Theories, Methods, and Applications 31(1): 24–34.

60.

Witschge

(2008) Examining online public discourse in context: a mixed method approach. JAVNOST-The Public 15(2): 75–91.

61.

Wright

(2016) Communication in health-related online social support groups/communities: a review of research on predictors of participation, applications of social support theory, and health outcomes. Review of Communication Research 4: 65–87.

62.

Xiang

Gretzel

(2010) Role of social media in online travel information search. Tourism Management 31(2): 179–188.

63.

Dutta

Datta

, et al. (2018) Identifying functional aspects from user reviews for functionality-based mobile app recommendation. Journal of the Association for Information Science and Technology 69(2): 242–255.

64.

Yıldırım

(2013) The effects of user comments on e-Trust: an application on consumer electronics. Journal of Economics, Business and Management 1(4): 360–364.

65.

Law

(2009) The impact of online user reviews on hotel room sales. International Journal of Hospitality Management 28(1): 180–182.

66.

Zhang

Law

, et al. (2010) The impact of e-word-of-mouth on the online popularity of restaurants: a comparison of consumer reviews and editor reviews. International Journal of Hospitality Management 29(4): 694–700.

67.

Zhu

Zhang

(2006) The influence of online consumer reviews on the demand for experience goods: the case of video games. In: ICIS 2006 proceedings, p. 25. Available at: https://aisel.aisnet.org/icis2006/25 (accessed 9 September 2020).

68.

Ziegele

(2016) Nutzerkommentare als Anschlusskommunikation. Wiesbaden: Springer Fachmedien.

69.

Ziegele

Quiring

(2013) Conceptualizing online discussion value: a multidimensional framework for analyzing user comments on mass-media websites. Annals of the International Communication Association 37(1): 125–153.

70.

Ziegele

Koehler

Weber

(2018) Socially destructive? Effects of negative and hateful user comments on readers’ donation behavior toward refugees and homeless persons. Journal of Broadcasting & Electronic Media 62(4): 636–653.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.74 MB