Detecting agro : Korean trolling and clickbaiting behaviour in online environments

Abstract

This article presents one of the first approaches to provide the understanding of agro (one of the unique eye-attracting cues) headlines and thumbnails in online video sharing platform, YouTube. We annotated 1881 headlines and thumbnails, based on agro and the type of agro. Then, we experimented with machine learning models to classify agro data from the non-agro data. With a bidirectional long short-term memory (Bi-LSTM) model, we achieved 84.35% of accuracy in detecting agro headlines and 82.80% of accuracy in detecting agro thumbnails. We believe that the automatic detection of agro headlines can allow users to have better experience in browsing through and getting the content that they want online.

Keywords

Agro Bi-LSTM clickbait Korea

1. Introduction

Present-day online platforms have brought manifold changes in the way people create, consume and share media content; the facileness of creating and sharing digital contents online has allowed people to become not only consumers, but also producers of media content [1]. This paradigm shift from passive-consumer to consumer-creator economy has accelerated the creation of innovative media content, unprecedented in its diversity and numbers. Naturally, online platforms evolved into a new type of marketplace, in which the economy is run by the interplay between the people and the media resources that surround them. As a corollary, media contents turned more and more provocative to capture the eyes of the distracted users. This situation of the user attention being directly correlated with revenue gave birth to the problem of agro in online environment.

Agro is a unique Korean term derived from the word, ‘aggravation’ [2]. It originated in the context of massive multiplayer online role-playing games (MMORPG), in which people used the word to refer to players who intentionally do something provocative to divert the attention of the counterpart player [2]. Now, the term is generally used by the Korean society to refer to contents that deliberately use eye-catching (provocative, peculiar, etc.) language or images to capture the attention of the audience [2] (Table 1). When used in the apt way, it can elicit appreciation and attention from the users (e.g. good marketing). However, when used in a deceptive way, agro can severely undermine user experience (e.g. clickbait, trolling).

Table 1.

Guidelines for data annotations.

Headline				Thumbnail
Content		Delivery		Delivery
Content	No-content	Provocative	Cliff-hanger	Provocative	Contains text
Main/partial plot,famous line,partial scene in the movie	None of thecharacteristicsof content	Slang, provocative,hyperbolic, emotional, sexualexpressions	Text inquestion form	Violent,shocking, sexual,emotional	Text insidethe image

In the academia, agro has been quoted in studies of Korean neologism [3 –7], and online Korean fan community behaviour [8]. It has been more ardently discussed in Korean news articles, in which journalists have continuously pointed out the adverse social effects of agro [9]. They often pinpoint videos that have been re-created with subjective interpretation and provocative editing concerning celebrities [10]. Representative examples are the YouTube videos regarding one Korean actress, Jane Doe (alias), who recently came into the controversy of gas lighting her boyfriend. This issue triggered countless agro videos on different online media platforms. Videos with deceptive thumbnails (e.g. using part of a frame in a video that can be interpreted as sexual or provocative) and provoking subtitles/headlines covered the Korean media (Figure 1). More examples can be found in online news sites. For example, Wikitree,¹ an online news site, posted an article with the headline, ‘“I Do It Every Once a Week.” … Jane Doe (alias) Confesses Everything After her Divorce with John Doe (alias) [Video]’, insinuating a sexual tone with the absence of an object. However, the content of the article is actually about how Jane Doe (alias) lets her daughter visit her father every once a week [11], which betrays the initial expectation of the headline. This type of agro, which crosses the line of marketing, creates the problem of undermining users’ perceived trust [9], regarding online contents, by building up the expectation and under delivering.

Figure 1.

Example of agro headline and thumbnail in YouTube; The headline of this YouTube video is, ‘Am I a pushover if I still sympathize with her? A reexamination of how this celebrity toys with her hand, presuming the camera was off’. The subtitles in the thumbnail translate to, ‘You can never see this scene again, surprisingly hard to believe how this celebrity toys with her hand, if it weren’t captured on camera’ from top to bottom.

Even worse, such contents, at times, have been known to find their way towards under-aged users, aggravating the social problem of agro online [9].

Thus, it is important to address the problem of agro and find a way to detect agro to create and maintain a healthier online browsing environment. On that account, this study aims to build a classifier that can detect agro headlines and thumbnails in online platforms. We focus on one of the globally diffused video sharing platforms in Korea, YouTube. Our research question of this article is as follows:

RQ. Can we classify agro titles in YouTube, through the analysis of the thumbnails and titles?

2. Literature review

Despite the commonplace appearance of the term agro in the Korean press and social media, there have yet been no prior studies that explored the concept of agro itself in the academia. Hence, our article aims to construct an inaugural definition of agro by examining the bordering terms and concepts to agro. In this section, we will try to explain agro by exploring the psychological mechanism behind the concept and juxtaposing similar terms and concepts to help further our understanding of agro. First, we will begin with a discussion of ’stimulus driven attention mechanism’ to explain the core mechanism behind agro. Then, we will explain different types of ‘attention stimuli in online environments’, to see which attributes attract users’ attention in online environments. After that, we will explore two akin concepts to agro (i.e. ‘clickbait’ and ‘trolling’), to further our understanding of agro.

2.1. Psychological mechanism behind agro

2.1.1. Stimulus-driven attention mechanism

Human’s evolutionary success relies heavily on the efficiency of organisms to detect and respond to biologically important events [12]. Hence, it is natural for our nervous system to automatically orient its resources to processing these types of events as soon as they are perceived [12]. This orienting response is said to be elicited by two main classes of biologically important stimulation: (1) novel stimuli and (2) signal stimuli [13 –16]. Novel stimuli is stimuli unknown or unexpected in a particular environment, whereas signal stimuli is stimuli known, or even expected, and is critical for the individual, such as food, mating partners, danger or emotional cues [13 –16].

When such novel or signal stimuli is perceived by our nervous system, we form our attentional responses through an automatic mechanism that is unconscious and stimulus driven. We call this ‘stimulus-driven attention’ or bottom-up process; it is an attentional mechanism that selects information because some attributes of the stimuli capture our attention, independently of our cognitive goals or beliefs [17]. This is different from voluntary (goal-directed) attention, which is triggered and developed in a top-down process; this type of goal-directed attention selects information because it fulfils some goal-defined criterion that we already had in our head (i.e. driven by our central nervous system) [17].

Oftentimes, we find ourselves involuntarily, and automatically responding to salient stimuli (i.e. very intense, voluminous or sudden) [18], due to our stimulus-driven attention mechanism. According to several prior research [18], the stimuli that activate our attention are defined as being, ‘either a sense-impression, very intense, voluminous, or sudden; or … an instinctive stimulus, a perception which, by reason of its nature rather than its mere force, appeals to some of our congenital impulses’ [18]. These stimuli differ from one animal to another, but some stimuli are known to have stronger instinctive attention-grabbing effect on most of the animals [18]. These include, ‘strange things, moving things, wild animals, bright things, pretty things, metallic things, blows, blood, and so on’ [18].

Understanding stimulus-driven attention mechanism is important to fathoming agro because agro is based on the exploitation of our involuntary attentional resources to hook us into content that may be far from, or even irrelevant to what we were looking for.

2.1.2. Attentional stimuli in online environments

Now we have addressed the stimulus-driven attention mechanism, we will delve into examining the attentional stimuli that have been proven to be effective in capturing users’ attention in online environments.

With the surge of shared information on the web and on social networks, efforts to grab users’ attention have greatly increased in the last decade [19]. However, consumers’ attentional resources are limited and have been referred to as ‘the scarcest resource in today’s business’ [20]. To this trend, researchers have continually searched for different stimuli to maximise the attention of their consumers.

In terms of language as a technique for capturing someone’s attention, researchers have discovered that stimuli, such as language intensity, sentiment, length and domain specificity in a headline can significantly increase users’ attraction to an online article [21]. In particular, the sentimental levels of headlines showed a notable relationship between the popularity of the news and the dynamics of the posted comments on that particular news [22]. Specific writing styles, such as listicles, cliff-hangers and human interest stories were also revealed to drive user engagement [23].

In terms of image as a technique for capturing someone’s attention, fearful, shocking, sexual and emotional cues in images have manifested high correlation with user attentional effects [24 –27]. This is due to the characteristics of human brain activities, which have specialised to attend to stimuli related to survival; fearful and shocking cues assist humans in perceiving threat around their surroundings [27], and sexual cues help men find one’s sexual partner for reproduction [24]. Plus, emotionally significant stimuli have shown to potentiate attention effects, specifically during later stages of cognitive processing [26].

Attentional stimuli are important in understanding agro because they characterise the attributes that agro contents employ to maximise people’s attentional resources. In forms of text, many agro content may take advantage of intense language, emotional language, sexual language, listicles and cliff-hangers, whereas in forms of image, they may take advantage of fearful, shocking, sexual or emotional stimuli to capture the attention of the users.

2.2. Similar concepts to agro

2.2.1. Clickbait

The most comparable concept to agro in many English-speaking countries is clickbait. Clickbait refers to ‘certain types of web content that is designed to entice users into clicking a link to a particular page’ [28]. While these baits may deceive the users into clicking, clickbaits do not live up to the expectation since they usually focus on the secondary aspect rather than on the key aspect, or even a relevant part of the story [29 –31]. Completeness, fairness and accuracy are absent in clickbaits [29].

Agro shares some similarities with clickbait. First, they often play on the secondary aspect rather than on the key aspect to capture the attention of the user. Second, their main purpose is to generate clicks by hooking the attention of the user. On the other hand, what sets agro apart from clickbaits is that not all agro content are absent in their completeness, fairness or accuracy. For example, agro content can be complete, fair or accurate, but at the same time, very provocative in its way of delivery. Moreover, agro does not necessarily need to be deceptive; agro can be designed to be deceptive but deception is not necessarily a requirement for a content to be agro. Hence, we can understand agro as a broader, umbrella term that can include clickbait, and is used to refer to all the content that is designed to be attention-grabbing.

2.2.2. Trolling

Another similar concept that we can discuss is trolling. Trolling refers to ‘an online behaviour, intended to provoke a reaction, aggravate conversation or lure others into fruitless argumentation’ [32 –34]. Hardaker and colleagues reviewed the definitions of trolling by drawing upon an extensive archive of data collected over a 9-year period from a single forum [35]. They revealed that the definitions of a troll typically contain four characteristics [35]:

Deception (disguising one’s motive);

Aggression (attempting to provoke others);

Disruption (disturbing the interaction);

Success (trolls are considered to be successful if they have accomplished to provoke others).

Shachaf and colleagues described trolling as an attention-seeking behaviour [36], whereas some scholars suggested personality traits as reasons for trolling [32, 33].

The overlapping characteristic of agro and trolling is that they are both attention-seeking and possibly provocative. Although the purpose of agro is not wholly to provoke others, their aim is to capture others’ attention, which can entail provocation. This is because people tend to automatically react to such provocative stimuli.

2.2.3. Machine learning approaches to detect clickbait and trolling

In 2014, clickbait became one of the hottest issues in Facebook, which stirred up some academic studies to detect clickbait in social media environment. Facebook tried to detect clickbaits based on the click-to-share ratio and the amount of time spent on these stories [37]. Moreover, Potthast and colleagues attempted to detect potential clickbait tweets in Twitter using common words triggering clickbaits [28], while Anand and colleagues employed deep learning techniques for detecting clickbaits [38]. Recently, several diverse approaches for detecting clickbait have also been conducted from the presence of individual words (lexical features) [30,39,40], to more complex language and grammatical structures [30,39], to the genre or subject matter itself [39] and to the content-agnostic user comments [41].

As for studies regarding machine learning approaches related to trolling, Dlala and colleagues attempted to detect troll users by classifying users subsequently banned by the moderators [42], defined as ‘Future-Banned Users’ (trolls), from the civil users, defined as ‘Never-Banned Users’. A number of features, including post content, user activity, reactions of the community and moderator’s actions were considered and employed. Mihaylov and colleagues conducted a similar study using community rating, topic consistency, order of comments, answers and time of the day to predict troll users in an online newspaper community [43]. Moreover, Al Marouf [5] tried to create a profile of personality traits with user’s textual data, while Fornacciari [44] tried to detect and predict troll users in social media through a sentiment analysis.

As seen, there has been a number of machine learning approaches related to detecting both clickbaits and trolling online. However, no studies to our knowledge have tried to employ machine learning approaches to detect agro in online environments. Hence, our study is unique in that it presents the first machine learning approach to detect and classify agro contents online.

2.3. Definition of agro

Based on this review of the bordering concepts and background mechanisms regarding agro, we define agro as follows: agro is an attention-seeking behaviour that plays on our stimulus-driven attention mechanism, which exploits salient stimuli to direct our attention. These stimuli include intense language, emotional language, sexual language, listicles, cliff-hangers, fearful image, shocking image, sexual image, emotional images and more. Agro is a concept that is similar to clickbait in that they often play on the secondary aspect rather than on the key aspect to capture the attention of the user. Moreover, both of their main purposes are to generate clicks. On the other hand, agro is different from clickbait in that it is not necessarily void of completeness, fairness, accuracy or deception. Agro is also a concept that shares similarities with trolling in that they are both attention-seeking, and possibly provocative. With this definition of agro, we now present the first machine learning approach to classify agro titles in Korean movie channels in YouTube, through the analysis of the headlines and thumbnails.

3. Agro data

3.1. Data collection

To sample our corpus, we used YouTube application programming interface to collect 10,000 headlines and thumbnails from Korean movie review channels in YouTube. These channels generally summarise the movie by skipping through the important plot with a complementary voice-over. We targeted the Korean movie channels in YouTube because they are noted for having agro headlines and thumbnails.

3.2. Annotation

Three annotators, who were enthusiastic about movies, were gathered via one of the private universities in South Korea for the agro labelling task. After explaining the annotators about the definition and examples of agro headlines and thumbnails, we asked them to label the data based on the annotation guideline as follows:

The headlines were labelled based on (1) content and (2) delivery. The thumbnails were labelled based on (1) delivery. Because the annotators had to know the story-line of the movie to be able to label the headlines based on content, we advised them to only classify the headlines of the movies that they have seen before. After the first annotation, we finalised the labels by voting the major label (i.e. 2 (yes): 1 (no) → finalised as (yes)). No labels had equal distribution of votes. The collected agro examples are presented in Figures 2 –7.

Figure 2.

(Movie: Midnight Sun). Headline: If you don’t piss yourself after seeing this movie, I will shed tears for you instead.

Figure 3.

(Movie: 21). Headline: A novel method of how an IQ 210 genius engineering student becomes top 0.1% millionaire.

Figure 4.

(Movie: Messiah). Headline: The chaos that will occur when Jesus resurrects after 2000 years.

Figure 5.

(Movie: Dracula). Headline: Netflix series that you cannot watch alone in the middle of the night.

Figure 6.

(Movie: The Notebook). Headline: Beautiful and pure love story that you cannot see without shedding your tears.

Figure 7.

(Movie: My Super Ex-Girlfriend). Headline: What happens when you dare do-it with a vigorous girlfriend.

3.2.1. Headlines

1.Content. Since the primary purpose of headlines should be to inform the audience about the content of the story [29], we labelled the title as content when it had any reference to the plot, part of a scene, or a famous line from the movie. Titles without any references to the mentioned above were labelled as no-content.

(a) Content

(b) No-content

2.Delivery. As for the delivery of the headlines, we adapted the standards from the studies of advertising and viral journalism and modified them to fit our study of agro.

(a) Provocative. A marketing technique that has long been used to increase the news worthiness of an event, is the use of hyperboles or the use of superlatives and comparisons of superiority [45]. Sometimes, slangs, capital letters and exclamation or admiration signs are used to maximise the expressiveness of the words [46]. In addition, utilisation of sexual or emotion-triggering content in advertisements has shown to create higher level of excitement in the content [47 –50]. Hence, we decided to label the title as provocative when it contained any of the following:

• Slang;

• Hyperbole;

• Emotional language;

• Sexual language.

Several examples of provocative headlines can be seen in the headlines of Figures 2 (slang and hyperbole), 3 (hyperbole), 5 (hyperbole), 6 (hyperbole and emotional), and 7 (sexual language).

(b) Cliff-hanger. Headlines posed as questions have been used since print marketing to arouse curiosity [51]. Online advertising and online news media often use hypothetical questions, rhetorical questions, leading questions, tag questions accompanied by self-referencing cues [52]. Such cues tap into the curiosity that is inherent in humans [52], and persuade the users to click to see the content by exploiting the knowledge gap [53]. Such headlines, as argued by prior research [54], are akin to cliff-hangers. Hence, we decided to label the title as cliff-hangers when it contained headlines posed as questions. Some examples of this type of cliff-hanger headlines can be seen in the headlines of Figures 3 and 7.

We labelled the headlines as agro, when the delivery of the headline had either provocative or cliff-hanger characteristics. The headlines without either of them were labelled non-agro.

3.2.2. Thumbnails

In case of thumbnails, when the image was provocative (e.g. violent, shocking, sexual or emotional) or if the image had a text inside, we labelled the data as agro. We labelled those without any of the mentioned above as non-agro. Several examples of agro thumbnails are presented in Figures 2 –4, 6 and 7. The subtitle in Figure 3 writes, ‘pardon …?’ (IQ210), (MIT professor), – how much is your income? lol from top to bottom. The subtitle in Figure 4 writes, (Is he real Jesus?), (Nah, that dude is a conman) from left to right. The subtitle in Figure 6 writes, Tissue is needed, I love you and Beautiful and pure love story that you cannot see without shedding your tears from top to bottom. The subtitle in Figure 7 writes, My girlfriend is too strong … and I don’t think I can go to work tomorrow … from top to bottom.

With this definition of agro, we tried to detect agro titles and thumbnails in Korean movie review channels in YouTube through machine learning models. Then, we also tried to see if the model can differentiate agro content from agro no-content.

3.3. Corpus description

Since we asked the participants to label the data only if they have seen the movie, we were left with 1881 headlines and thumbnails after the annotation. Among 1881 headlines, 70% (1323) was ago and 30% (558) was non-agro. Within the agro headlines, 42% (871) was cliff-hanger, 46% (788) was provocative and 17% (336) was both cliff-hanger and provocative. Within the agro headlines, 52% (986) was content and 18% (337) was no-content. The average number of tokens was eight (mean: 8.29, SD: 2.52, max: 23, min: 2) for agro headlines. The average number of tokens was six (mean: 6.83, SD: 2.28, max: 18, min: 2) for non-agro headlines. As for the thumbnails, 73% (1366) was agro and 28% (515) was non-agro.

Figure 8, and Tables 2 and 3 show the summary of the collected datasets in this study.

Figure 8.

Agro data distribution.

Table 2.

Summary of data distributions.

	Thumbnail, agro	Thumbnail, non-agro
Headline, non-agro	379 (20.14%)	179 (9.51%)
Headline, agro	987 (52.47%)	336 (17.86%)

Table 3.

Summary of headline data distributions.

	Agro content	No-content
Non-agro	425 (22.59%)	133 (7.07%)
Agro	986 (52.41%)	337 (17.91%)

4. Experiments and results

To examine the reliability of the labelled dataset in detecting agro content online, we experimented with natural language processing (NLP) tasks and deep learning models. As for the programming language and environments, we used Python 3.6.0 and PyTorch. To verify the annotated agro headlines, we experimented with bidirectional long short-term memory (Bi-LSTM) and convolutional neural network (CNN) [55 –57]. As for the thumbnails, we experimented with CNN. For both headline and thumbnail classification tasks, we employed SMOTE (Synthetic Minority Oversampling Technique) to balance the data. As for the baseline classifiers, we employed support vector machine [58], XGBoost [59] and random forest (RF) [60].

4.1. Agro classification

4.1.1. Headline

To classify the labelled headlines, we experimented with Bi-LSTM and CNN. Since more than 70% of the annotated headlines and thumbnails in the dataset were agro, we employed SMOTE to address the data imbalance issue [61]. Then, we randomly divided the collected headlines into training (1354, 72%), validation (150, 8%) and testing (377, 20%) sets. We tokenised each headline with the OKT (Open Korean Text) tokeniser from KoNLPy. ² The maximum word counts of the headlines and total vocabulary size were 21 and 1441, respectively.

Bi-LSTM. The tokenised words of the headlines were input to the embedding layer with 128 units. The representation of the input data was then sent to the Bi-LSTM layer with 64 units. The final output of the Bi-LSTM was calculated through sigmoid function. Both RMSprop optimiser and binary cross-entropy loss were used. Ten epochs were employed in the training sessions with 16 batch size. The training took 21.90 s.

CNN. To employ a CNN-based classifier, we created a sequence of the tokenised words by embedding a layer with 128 units. Input length was padded to the max length of the headlines. The sequence was then input to the CNN layer with 64 units. Then, the max pooling layer extracted features from the given data. The final output was then computed with sigmoid function to classify whether or not the given comment is an agro headline. Adam optimiser and binary cross-entropy loss was used. Ten epochs were employed in the training sessions with 16 batch size. The training took 21.20 s.

XGBoost. Tokenised words of the same embedding mentioned above were used as inputs. Five-fold cross-validation procedures were used. GridSearchCV was employed to find the optimal hyper-parameters. The best hyper-parameters were presented as follows: learning rate: 0.1, max depth: 9 and number of estimators: 180. The training took 0.63 s.

RF. Tokenised words of the same embedding as mentioned above were used as inputs. Five-fold cross-validation procedures were used. GridSearchCV was employed to find the optimal hyper-parameters. The best hyper-parameters were presented as follows: max depth: 16 and number of estimators: 256. The training took 0.21 s.

Table 4 presents the agro headline classification results with four evaluation metrics. The Bi-LSTM outperformed other models with 88.57% and 72.16% of precision on agro and non-agro, respectively. The Bi-LSTM also outperformed other models with 90.18% and 68.63% of recall on agro and non-agro, respectively. In general, the Bi-LSTM showed the greatest levels in all evaluation metrics.

Table 4.

Results of the binary classification task of agro and non-agro headlines.

Classifier	Class	Precision (%)	Recall (%)	F₁-score (%)	Accuracy (%)
CNN	Agro	86.27	89.09	87.66	81.70
	Non-agro	67.74	61.76	64.62
Bi-LSTM	Agro	88.57	90.18	89.37	84.35
	Non-agro	72.16	68.63	70.35
XGBoost	Agro	82.31	82.91	82.61	74.54
	Non-agro	53.00	51.96	52.48
RF	Agro	77.51	92.73	84.44	75.07
	Non-agro	58.33	27.45	37.33

CNN: convolutional neural network; Bi-LSTM: bidirectional long short-term memory; RF: random forest.

4.1.2. Thumbnail

To classify the labelled thumbnails, we experimented with CNN. SMOTE was used to balance the data. Then, we randomly divided the collected images into training (1354, 72%), validation (150, 8%) and testing (377, 20%) sets.

CNN. Tokenised words of the same embedding mentioned above were used as inputs. The sequence was then input to the CNN layer with 64 units. The max pooling layer was used to extract features from the given data. The final output was computed with sigmoid function to classify whether or not the given image is an agro thumbnail. Ten epochs were employed in the training sessions with 32 batch size. The training took 298.31 s.

We measured the micro and macro F₁-score for the binary agro classification of the thumbnails (Table 5). A CNN-based model showed 79.84% of micro F₁-score and 75.61% of macro F₁-score.

Table 5.

Results of the binary classification task of agro and non-agro thumbnails.

Classifier	Class	Precision (%)	Recall (%)	F₁-score (%)	Accuracy (%)
CNN	agro	89.32	87.76	88.54	82.80
	non-agro	63.92	67.39	65.61

CNN: convolutional neural network.

4.2. Agro type classification

4.2.1. Headline

After the agro classification task, we further experimented with classifying the type of agro headlines in terms of delivery (cliff-hanger and provocative) and content (content and no-content). We used only the agro headlines and employed the same aforementioned models. The maximum word counts of the headlines were 33 and the total vocabulary size was 1186. There were (952, 72%) training, (106, 8%) validation and (265, 20%) testing set. Tables 6 –8 show the summary of agro type classification.

Table 6.

Results of the binary classification task of provocative and non-provocative headlines.

Classifier	Class	Precision (%)	Recall (%)	F₁-score (%)	Accuracy (%)
CNN	Provocative	78.41	83.13	80.70	75.09
	Non-provocative	68.54	61.62	64.89
Bi-LSTM	Provocative	76.06	86.14	80.79	74.34
	Non-provocative	70.13	54.55	61.36
XGBoost	Agro	74.14	77.71	75.88	69.06
	Non-agro	59.34	54.55	56.84
RF	Agro	72.83	80.72	76.57	69.06
	Non-agro	60.49	49.49	54.44

CNN: convolutional neural network; Bi-LSTM: bidirectional long short-term memory; RF: random forest.

Table 7.

Cliff short for cliff-hanger. Results of the binary classification task of cliff-hanger and non-cliff-hanger headlines.

Classifier	Class	Precision (%)	Recall (%)	F₁-score (%)	Accuracy (%)
CNN	Cliff	91.36	90.80	91.08	89.06
	Non-cliff	85.44	86.27	85.85
Bi-LSTM	Cliff	85.71	92.02	88.76	85.66
	Non-cliff	85.56	75.49	80.21
XGBoost	Cliff	83.12	78.53	80.76	76.98
	Non-cliff	68.47	74.51	71.36
RF	Cliff	76.28	73.01	74.61	69.43
	Non-cliff	59.63	63.73	61.61

CNN: convolutional neural network; Bi-LSTM: bidirectional long short-term memory; RF: random forest.

Table 8.

Results of the binary classification task of content and no-content headlines.

Classifier	Class	Precision (%)	Recall (%)	F₁-score (%)	Accuracy (%)
CNN	Content	85.43	88.08	86.73	80.38
	No-content	65.15	59.72	62.32
Bi-LSTM	Content	85.93	88.60	87.24	81.13
	No-content	66.67	61.11	63.77
XGBoost	Content	76.62	79.79	78.17	67.55
	No-content	39.06	34.72	36.76
RF	Content	75.83	82.90	79.21	68.90
	No-content	38.89	29.17	33.33

CNN: convolutional neural network; Bi-LSTM: bidirectional long short-term memory; RF: random forest.

4.2.2. Provocative

As for the classification of provocative and non-provocative headlines, the deep learning models generally showed a higher performance than the baseline classifiers. This is due to the ability of the deep learning models to learn more complex features. The CNN performed better in predicting non-provocative in terms of 61.62% of recall and 64.89% of F₁-score, whereas the Bi-LSTM performed better in predicting provocative in terms of 86.14% of recall and 80.79% of F₁-score. As for the precision, CNN performed higher in predicting provocative (78.41%), while Bi-LSTM performed better in predicting non-provocative (70.13%).

4.2.3. Cliff-hanger

In case of the classification of cliff-hanger and non-cliff-hanger headlines, the CNN showed the greatest levels in all evaluation metrics, except the recall of cliff (90.80%) and precision of non-cliff (85.44%). As for the classification of content headlines and no-content headlines, the Bi-LSTM performed better in all evaluation metrics, except for the precision in no-content (66.67%). The implemented models are publicly available.³

5. Concluding remarks

This article presents one of the beginning machine learning approaches for agro detection using its headlines and thumbnails in one of the globally employed video sharing platforms, YouTube. We have annotated 1881 headlines and thumbnails based on agro and the type of agro. Then, we experimented with machine learning models to classify agro data from the non-agro data. With a Bi-LSTM model, we achieved 84.35% accuracy in detecting agro headlines and thumbnails. Moreover, baseline models, including XGBoost and RF performed over 74% in average in classifying agro from non-agro data. This implies that our labelled data are a reliable dataset for predicting and detecting agro online. We believe that the automatic detection of agro headlines will help users improve their experience in browsing through and getting the content that they want online.

Based on the findings of the current study, the following academic implications and contributions can be presented. First, our research is significant in that it is the first study to address agro in the online environment. We shed light on a unique eye-catching technique that online South Korean users use, called agro. A number of previous studies have focused on marketing based clickbait when it comes to dragging attention from the users. Our study is unique in that we introduce a broader concept of agro, which can be used as an umbrella term to encompass techniques, including clickbait, trolling and more. Understanding agro and detecting agro will help reduce fallacious advertising online, creating a healthier browsing environment. Second, we contribute a reliable labelled dataset that is specific to South Korea. Labelling dataset is a strenuous task that requires time and money. Our agro dataset, which was labelled rigorously through a 2-week time period, will serve as a basis for future studies on agro.

We also discuss some practical implications of our study. A classifier to detect agro will help improve the browsing experience of the users; they will be able to search, browse and fetch the appropriate information online. However, there remain some limitations in our study. First, we only have used a small number of dataset. Future studies should aim to include more data to further corroborate the findings. Second, the data are Korean-specific, meaning that the implications of the current study can be difficult to generalise to other languages. Future research should aim to include more diverse language to generalise the findings.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship and/or publication of this article: This research was supported by the MSIT (Ministry of Science, ICT), Korea, under the High-Potential Individuals Global Training Program (IITP-2021-0-02104) supervised by the IITP (Institute for Information & Communications Technology Planning & Evaluation). This research was also supported by National Research Foundation (NRF) of Korea Grant funded by the Korean Government (MSIT; No. 2021R1A4A3022102).

ORCID iD

Eunil Park

Notes

References

Jenkins Dea Henry. Convergence culture. London: SAGE, 2008.

Lee

. Agro. Gyeonghyang News, https://m.khan.co.kr/opinion/column/article/201905222051015 (2019, accessed 21 November 2021).

Pyo

. A study on morphological patterns and generative mechanism of Korean neologisms combined with English morphemes. Korean J Ling 2017; 42(1): 97–120.

Choi

. The nativization of foreign elements through hybrid neologisms – focusing on dependent foreign elements. J Soc Korean Lang Liter 2019; 78: 177–214.

Marouf A

Ajwad

Ashrafi

. Looking behind the mask: a framework for detecting character assassination via troll comments on social media using psycholinguistic tools. In: 2019 international conference on electrical, computer and communication engineering (ECCE), Cox's Bazar, Bangladesh, 7–9 February 2019, pp. 1–5. New York: IEEE.

Choi

. A study of the education method of the Korean neologism – focused on derivative words of neologism in 2014. J Korean Lang Liter Educ 2015; 58: 253–280.

Han

. Analysis of usage characteristics of new words based on persistency and burstiness in news articles. J Digit Cont Soc 2019; 20(1): 51–58.

Jung

Kim

. An exploratory study on modalities and harmful effects of ‘Chinmokjil (Socializing Behavior)’. J Digit Cont Soc 2018; 19(8): 1471–1480.

Lee

. The age of competition 2021, contents have crossed the line. Korea Entertainment, Sports News, https://www.koreaes.com/news/articleView.html?idxno=358482 (2020, accessed 21 November 2021).

10.

Yoon

. [n-focus] rumor of rumors…stimulating fake entertainment news on YouTube and SNS 2021. MoneyToday, https://news.mt.co.kr/photonews/graphicNewsList.html?id=2021042507008241178 (2021, accessed 21 November 2021).

11.

Yoon

. ‘I do it every once a week’. Cho Yoonhee confesses everything after her divorce with Lee Donggun (Video). WikiTree, https://www.wikitree.co.kr/articles/667927 (2021, accessed 21 November 2021).

12.

Carretié

Hinojosa

Martín-Loeches

. Automatic attention to emotional stimuli: neural correlates. Hum Brain Map 2004; 22(4): 290–299.

13.

Graham

Hackley

. Passive and active attention to input. In: Jennings

Coles

MGH

(eds) Handbook of cognitive psychophysiology: Central and autonomic nervous system approaches. Chichester: John Wiley & Sons, 1991, pp. 251—356.

14.

Öhman

Hamm

Hugdahl

. Cognition and the autonomic nervous system: orienting, anticipation, and conditioning. In: Cacioppo

(ed.) Handbook of psychophysiology. Cambridge: Cambridge University Press, 2000, pp. 533–575.

15.

Siddle

. Elicitation and habituation of the orienting response, in orienting and habituation. Persp Hum Res 1983; 1: 109–182.

16.

Sokolov

. vospriyatiye i uslovny refleks [Perception and the conditioned reflex]. Moscow: M. Gu English Translation Published by Pergamon Press, 1963.

17.

Yantis

. Stimulus-driven attentional capture. Curr Dir Psychol Sci 1993; 2(5): 156–161.

18.

Spencer

. The principles of psychology. Princeton, NJ: Princeton University Press, 2021.

19.

Gelper

van der Lans

van Bruggen

. Competition for attention in online social networks: implications for seeding strategies. Manag Sci 2021; 67(2): 1026–1047.

20.

Pieters

Wedel

. Attention capture and transfer in advertising: brand, pictorial, and text-size effects. J Market 2004; 68(2): 36–50.

21.

She

Zhang

. The impact of headline features on the attraction of online financial articles. Int J Web Inf Syst 2019; 15(5): 510–534.

22.

Dos

Rieis JCS

de Souza

de Melo

POSV

. Breaking the news: first impressions matter on online news. In: Ninth international AAAI conference on web and social media, Oxford, 26–29 May 2015, pp. 357–366. Palo Alto, CA: AAAI.

23.

Kilgo

Harlow

García-Perdomo

. A new sensation? An international exploration of sensationalism and social media recommendations in online news publications. Journalism 2018; 19(11): 1497–1516.

24.

Nummenmaa

Hietanen

Santtila

. Gender and visibility of sexual cues influence eye movements while viewing faces and bodies. Arch Sex Behav 2012; 41(6): 1439–1451.

25.

Parry

Jones

Stern

. ‘shockvertising’: an exploratory investigation into attitudinal variations and emotional reactions to shock advertising. J Consum Behav 2013; 12(2): 112–121.

26.

Schupp

Stockburger

Codispoti

. Selective visual attention to emotion. J Neurosci 2007; 27(5): 1082–1089.

27.

Forbes

Purkis

Lipp

. Better safe than sorry: simplistic fear-relevant stimuli capture attention. Cogn Emot 2011; 25(5): 794–804.

28.

Potthast

Köpsel

Stein

. Clickbait detection. In: European conference on information retrieval, Padua, 20–23 March 2016, pp. 810–817. Cham: Springer.

29.

Orosa

Garcia

. Use of clickbait in the online news media of the 28 EU member countries. Rev Latina Comun Soc 2017; 72: 1261–1277.

30.

Chakraborty

Paranjape

Kakarla

. Stop clickbait: detecting and preventing clickbaits in online news media. In: 2016 IEEE/ACM international conference on advances in social networks analysis and mining (ASONAM), Davis, CA, 18–21 August 2016, pp. 9–16. New York: IEEE.

31.

Muddiman

Scacco

. Clickbait content may not be click-worthy. Austin, TX: Center for Media Engagement, 2019.

32.

Binns

. Don’t feed the trolls! managing troublemakers in magazines’ online communities. J Pract 2012; 6(4): 547–562.

33.

Bishop

. The psychology of trolling and lurking: the role of defriending and gamification for increasing participation in online communities using seductive narratives. In: Li

(ed.) Virtual community participation and motivation: Cross-disciplinary theories. Hershey, PA: IGI Global, 2012, pp. 160–176.

34.

Cheng

Danescu-Niculescu-Mizil

Leskovec

. Antisocial behavior in online discussion communities. In: Ninth international AAAI conference on web and social media, Oxford, 26–29 May 2015, pp. 61–70. Palo Alto, CA: AAAI.

35.

Hardaker

. Trolling in asynchronous computer-mediated communication: from user discussions to academic definitions. J Polit Res 2010; 6: 215–224.

36.

Shachaf

Hara

. Beyond vandalism: Wikipedia trolls. J Inf Sci 2010; 36(3): 357–370.

37.

El-Arini

Tang

. News feed fyi: click-baiting. Facebook, https://about.fb.com/news/2014/08/news-feed-fyi-click-baiting/

38.

Anand

Chakraborty

Park

. We used neural networks to detect clickbaits: you won’t believe what happened next! In: European conference on information retrieval, Aberdeen, 8–13 April 2017, pp. 541–547. Cham: Springer.

39.

Chen

Conroy

Rubin

. Misleading online content: recognizing clickbait as ‘false news’. In: Proceedings of the 2015 ACM on workshop on multimodal deception detection, Seattle, WA, 13 November 2015, pp. 15–19. New York: ACM.

40.

Vorakitphan

Leu

Fan

. Clickbait detection based on word embedding models. In: International conference on innovative mobile and internet services in ubiquitous computing, Sydney, NSW, Australia, 3–5 July 2019, pp. 557–564. Cham: Springer.

41.

Shang

Zhang

Wang

. Towards reliable online clickbait video detection: a content-agnostic approach. Knowl-based Syst 2019; 182: 104851.

42.

Dlala

Attiaoui

Martin

. Trolls identification within an uncertain framework. In: 2014 IEEE 26th international conference on tools with artificial intelligence, Limassol, 10–12 November 2014, pp. 1011–1015. New York: IEEE.

43.

Mihaylov

Georgiev

Nakov

. Finding opinion manipulation trolls in news community forums. In: Proceedings of the nineteenth conference on computational natural language learning, Beijing, China, 30–31 July 2015, pp. 310–314. Stroudsburg, PA: Association for Computational Linguistics.

44.

Fornacciari

Mordonini

Poggi

. A holistic system for troll detection on twitter. Comput Hum Behav 2018; 89: 258–268.

45.

Molek-Kozakowska

. Towards a pragma-linguistic framework for the study of sensationalism in news headlines. Discourse Commun 2013; 7(2): 173–197.

46.

Bazaco

. Clickbait as a strategy of viral journalism: conceptualisation and methods. Rev Latina Comun Soc 2019; 74: 94–115.

47.

Severn

Belch

. The effects of sexual and non-sexual advertising appeals and information level on cognitive processing and communication effectiveness. J Advertising 1990; 19(1): 14–22.

48.

LaTour

Pitts

Snook-Luther

. Female nudity, arousal, and ad response: an experimental investigation. J Advertising 1990; 19(4): 51–62.

49.

Dobele

Lindgreen

Beverland

. Why pass on viral messages? Because they connect emotionally. Bus Horiz 2007; 50(4): 291–304.

50.

Rodić

Koivisto

. Best practices in viral marketing. Helsinki: Aalto University School of Economics, 2012.

51.

Howard

. The prevalence of question use and question strategies in print advertising. Curr Iss Res Advertising 1988; 11(1–2): 89–112.

52.

Lockwood

. Academic clickbait: articles with positively-framed titles, interesting phrasing, and no wordplay get more attention online. The Winnower 2016; 3, https://thewinnower.com/papers/4892-academic-clickbait-articles-with-positively-framed-titles-interesting-phrasing-and-no-wordplay-get-more-attention-online

53.

Scacco

Muddiman

. Using controlled and field experiments to create and test digital news quizzes. Thousand Oaks, CA: SAGE, 2019.

54.

Blom

Hansen

. Click bait: forward-reference as lure in online news headlines. J Pragmat 2015; 76: 87–100.

55.

Schuster

Paliwal

. Bidirectional recurrent neural networks. IEEE T Signal Proces 1997; 45(11): 2673–2681.

56.

Kalchbrenner

Grefenstette

Blunsom

et al. A convolutional neural network for modelling sentences. In: Proceedings of the 52nd annual meeting of the association for computational linguistics, Baltimore, MD, 22–27 June 2014, pp. 212–217. Stroudsburg, PA: Association for Computational Linguistics.

57.

Kim

Lee

Park

et al. A deep learning model for detecting mental illness from user content on social media. Sci Rep 2020; 10(1): 1–6.

58.

Hwang

Kim

Park

et al. Who will be your next customer: a machine learning approach to customer return visits in airline services. J Bus Res 2020; 121: 121–126.

59.

Chen

Guestrin

. Xgboost: a scalable tree boosting system. In: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, San Francisco. CA, 13–17 August 2016, pp. 785–794. New York: ACM.

60.

Breiman

. Random forests. Mach Learn 2001; 45(1): 5–32.

61.

Kim

et al. Integrating deep learning and machine translation for understanding unrefined languages. CMC-Comput Mater Continua 2022; 70(1): 669–678.