Sage Journals: Discover world-class research

Abstract

Human behavior is impacted by emotion, mood, personality, needs and subjective well-being. Emotion and mood are human affective states while personality, needs and subjective well-being are influences on those affective states. Ontologies are a method of representing real-world knowledge, such as human affective states and their influences, in a format that a computer can process. They allow researchers to build systems that harness affective states. By unifying terms and meanings, ontologies enable these systems to communicate and share knowledge with each other. In this paper, we survey existing ontologies on affective states and their influences. We also provide the psychological background of affective states, their influences and representational models. The paper discusses a total of 20 ontologies on emotion, one ontology on mood, one ontology on needs, and 11 general purpose ontologies and lexicons. Based on the analysis of existing ontologies, we summarize and discuss the current state of the art in the field.

Keywords

Ontology affective state survey affective computing emotion

1. Introduction

Ontologies have become more and more popular in fields such as web technologies or data integration and extraction [68]. An ontology can be seen as a catalog that shows entities in a specific field and the relationships between them. It represents structural knowledge for any domain and defines a common vocabulary to be shared. In addition, it defines data and data structures to be used in applications in the same field [57]. An ontology is defined as “an explicit specification of a conceptualization” [35]. An ontology consists of classes, properties and individuals that define a particular domain [54].

Classes are the focal point of ontologies, and they describe the concepts in a domain. A class represents a group of different individuals that share similar characteristics. An instance of a class is called an individual. Object properties describe the semantic relationship between individuals [57]. People and systems communicate with each other from different backgrounds and contexts, using varying words and concepts [83]. A well-designed ontology provides standardized definitions and vocabularies in a particular domain, allowing a flow of communication [89]. Figure 1 shows a representation of an ontology and its components. For instance, person is a class, Jon is an individual, and studiesIn is an object property.

Fig. 1.

Representation of ontology components. Oval shapes represent classes, rounded rectangles represent individuals, and arrows indicate object properties.

Ontologies are created using a machine-processable language such as the Web Ontology Language (OWL), an international standard for the design and exchange of ontologies. The Web Ontology Language uses a set of classes, sub-classes and properties which are organized into a hierarchical structure by property axioms [39]. New ontologies may be developed from the foundation of pre-existing ones and potentially be designed with future re-use in mind. Ontologies can be designed at the top level (a general ontology) and then customized according to the domain or application. They can also be designed for a particular application or system, in which case they are called application ontologies. Ontologies can be re-used as a whole or partially, depending on the project needs [10,37]. By not having to create an entire ontology from scratch, time is saved, and the quality and maintainability of the new ontology is improved. Moreover, by reusing existing work, knowledge can be mapped from one domain to the domain of another ontology [27].

Not only does using an ontology allow for human affective states and their influences to be represented in an understandable computer format, it also improves the understanding and communication between people. The structure of the ontology reveals the definition of human affective states, influences and the relationships between them. It enables the sharing of human knowledge in a digital format.

Human behavior is formed by affective states and their influences. While the human interpretation of the relationship between affective states and influences is far from perfect, it is superior to the interpretation by a computer. Therefore, representing human affective states and their influences in a semantic way enables the communication between humans and systems. Moreover, it inspires the development of applications that automatically detect and predict behaviors and meanings. Ontologies provide a unified vocabulary for each concept in a domain, so that the interpretation of messages shared between computer applications is universal.

For example, an ontology that defines emotion, causes, and events to predict student emotion in an e-learning session can be used to predict the emotions of students with regard to answering test questions [25]. Another example is an avatar that is capable of showing appropriate facial expressions and gestures [32]. The expressions and gestures were selected based on an ontology that represents emotions associated with facial expressions and gestures.

Fig. 2.

The left side shows ways in which humans express their emotions (Domain). The context impacts the expressed emotions. On the right side, psychological models of emotion are shown (Representation Model).

In [18] a survey was carried out about ontologies for human behavior recognition. The emphasis of the survey is on context ontologies to track human activities. It presents general and domain-specific ontologies. In our paper, however, we aim to give an overview of existing ontologies on human affective states and their influences.

The remainder of this paper is organized as follows. In Section 2, we introduce the psychological theories used to build the existing ontologies for human affective states and their influences. Section 3 describes the lexicons used in the existing ontologies. Section 4 surveys current ontologies for human affective states and their influences as well as other related ontologies. Our conclusions and an outlook are provided in Section 5.

2. Background in affective states and their influences

We argue that psychological theories represent the primary point of ontology design in the domain of human affective states and their influences. These theories form the basis for the existing ontologies that are discussed in Section 4. In Section 2.1, affective states (emotion and mood) are presented. The influences, which are personality, subjective well-being and needs are introduced in Section 2.2. Finally, Section 2.3 describes the relationship between affective states and their influences.

2.1. Affective states

Emotion is the result of a person’s exposure to an internal or external stimulus and is expressed by changes in facial expression, gesture, voice or physiological parameters [74]. Emotion plays an important role in a person’s decision-making process. As such, emotion detection is an important step toward the understanding of human beings. Computationally, an emotion can be represented either in a discrete (categorical), dimensional or componential (appraisal) way [41]. Figure 2 gives an overview of how emotions are expressed and shows representational models.

In the discrete model, emotions are classified by words and grouped into families that share similar characteristics. The most common ones are called basic emotions (archetypal), and they can be found in many cultures. These emotions include happiness, surprise, fear, sadness, anger and disgust [3]. Additionally, neutrality [3], contempt [62], anticipation, trust, and love [62] can be considered. In the work of Izard, the emotions contempt, distress, guilt, interest, and shame were added to the basic emotions [43].

Another discrete emotion classification was proposed by Douglas-Cowie et al., who listed 48 emotion categories and arranged them into 10 groups. They include negative forceful, negative/positive thoughts, caring, positive lively, re-active, agitation, negative not in control, negative passive and positive quiet [21].

Plutchik grouped eight basic emotions in a wheel, placing similar emotions together and opposing ones at 180 degrees apart. The model is called Plutchik’s wheel of emotions. The contrasting pairs consist of joy versus sadness; anger versus fear; acceptance versus disgust; and surprise versus expectancy. The model also includes advanced emotions made up of combined basic ones. In addition, each emotion in the model represents a basic level of intensity [62].

Furthermore, Drummond used a vocabulary of 10 emotions: happiness, caring, depression, inadequateness, fear, confusion, hurt, anger, loneliness, and remorse. They were divided into the three categories: strong, medium, and light. For example, the emotion of happiness consists of being thrilled in the strong category, cheerful in the medium category, and cool in the light category.1

¹
http://tomdrummond.com/leading-and-caring-forchildren/emotion-vocabulary/

In the dimensional model of emotion, an emotion is represented by a number applied to each dimension. For instance, the Circumplex Model by Russell has the two dimensions valence and arousal. Valence is correlated with the degree of pleasantness or unpleasantness of an emotion while arousal refers to the amount of physiological change in the person’s body [69]. Figure 3 shows the Circumplex Model of affect with valence (pleasantness) on the horizontal and arousal (activation) on the vertical axis. As an example, happy is represented by positive valence along with high arousal, whereas relaxed is represented by positive valence along with low arousal (deactivation). In a similar fashion, Whissel models emotion as a 2D space whose dimensions are evaluation and activation [85].

Fig. 3.
A graphical representation of the Circumplex Model of affect. The horizontal axis represents the valence dimension, and the vertical axis represents arousal.

Mehrabian created the Pleasure–Arousal–Dominance (PAD) model of emotional states where dominance was added as a third dimension. It is the feeling of being in control of a situation versus the feeling of being controlled [53]. Osgood et al. use the names evaluation, activity and potency [60]. Cowie et al. use evaluation, activation and power [16]. A fourth dimension named unpredictability was added by Fontaine [28]. It denotes a person’s reaction to a stimulus based on their familiarity with the situation. Other choices of dimensions can be seen for example in the model of Watson and Tellegen who proposed the dimensions of negative affect (NA) and positive affect (PA) [84], or in the model of Feidakis et al. who use intensity, frequency and duration as emotion dimensions [26].

The basis of Componential appraisal models is the observation that emotions occur in humans based on their evaluation of events. This type of model highlights the cognitive background of emotions. Appraisal theory ties human emotions to the way they interpret events. It states that a person uses fixed criteria to evaluate a situation and to produce suitable emotions. People appraise the situation based on their familiarity with the event (novelty), whether or not it is relevant to their goals, their ability to cope with the consequences of the event (agency), and if it is well-matched to standards and social values (norms) [73].

The OCC (Ortony, Clore, and Collins) appraisal model reasons about agents, beliefs, objects and events. This model is popular in computer science systems that draw conclusions from emotions [59]. The OCC model defines a finite set that allows for the characterization of emotions. Moreover, it delivers a semi-formal descriptive language of emotion types. The model classifies 22 emotions into three main categories: consequences of events (e.g., joy and pity), actions of agents (e.g., pride and reproach), and aspects of objects (e.g., love and hate). These three main categories are further classified into subgroups. For instance, if the evoked emotion differs whether the consequence of an event is focused on the person or on others.

Mood is an emotional state that affects the experience and behavior of a person. It has a lower intensity but a longer duration than emotion [74]. Mood affects a person’s judgment: people in a happy mood tend to draw optimistic conclusions while people in a bad mood are likely to make pessimistic judgments [47]. Although emotion and mood are both feelings that people experience, there are differences between them. Emotions are caused by a specific situation, they last for a short duration and have a high intensity. Moods on the other hand have no clear causes, last longer and have lower intensity [42]. Mood can be represented by using discrete or dimensional models [46].

Sentiment is another human affective state defined by a person’s opinion or feeling towards something. Sentiment is expressed with words such as like, dislike, good or bad. For example, people use social media to express their sentiments about products, movies, etc. [40].
2.2. The influences of affect

Personality is defined as an individual pattern of affect, behavior, cognition and goals over time and space [67]. Personality reflects a person’s attitude and characteristics. The two most popular personality theories are the Myers–Briggs Type Indicator (MBTI) [86], and the Big Five [14]. The MBTI is mainly used in the training world, for example to decide for the appropriate carer. Big Five is dominant theory for academic research [7].

As the name implies, the Big Five theory represents personality in five dimensions. An outgoing, energetic person is described by high Extroversion. A friendly and cooperative person is described by the Agreeableness trait. Conscientiousness means that someone is responsible, dependable and organized. A sensitive and nervous person has Neurotic traits, and a social, intellectual person has a large value in the Openness dimension [52].

In the MBTI, each personality fits into only one of 16 types. These types are based on four features of personality, each one combined with its opposite: Extroversion (E) vs Introversion (I), Sensing (S) vs Intuition (N), Thinking (T) vs Feeling (F) and Judgment (J) vs Perception (P) [86]. Because there are two features within each of the four dimensions, there are 16 possible combinations. It is noteworthy that although the MBTI is a very widespread test of personality, many psychologists do not support it and claim that no significant conclusions can be drawn from it. There is no evidence that every individual can be described with its 16 categories.

Subjective well-being refers to how people judge and evaluate their life. The term is a container for diverse types of evaluations. Life satisfaction, for example, is considered a cognitive component because it is based on evaluative beliefs. Positive and negative affect are another component of subjective well-being, reflecting the level of pleasant and unpleasant feelings that people experience in their lives [75].

Human Needs are necessities for the development of physical and mental growth of individuals. Human needs are the underlying layers that trigger emotions and feelings, which later empower and direct human behavior [81]. Need categories are classified and represented by the internal aspect within individuals and the external aspect of a particular community, including social, cultural, economic and political aspects. In the Self-Determination Theory (SDT) [70], a macro theory focusing on the individual’s inner feelings, human needs are categorized into three basic psychological needs: Autonomy, Competence and Relatedness.

In Human Motivation Theory, Abraham Maslow presents a pyramid of five need categories arranged in hierarchical levels based on their importance to human beings. The five categories by decreasing importance are survival, security and safety, social, self-esteem and self-actualization. This model has been updated to adapt two new dimensions under the self-actualization category: cognition and aesthetic needs. Also, the theory explored the self-transcendent needs as the need to help others as a further category on the top of the pyramid [48].

In the Human Scale Development Model proposed by Max-Neef, the fundamental need categories for individuals and communities are formulated in a universal and interactional structure [50]. The model distinguishes between universal needs and the satisfiers, or strategies to meet these needs. The needs are finite and constant across all human cultures, while the satisfiers are changeable over time and differ between cultures. The model defines the needs and satisfiers in a matrix with two dimensions; the need dimension in axiological categories consist of: subsistence, protection, affection, understanding, participation, idleness, identity, creation and freedom. The satisfiers in existential categories are represented in the form of being, having, doing and interacting.

2.3. The relationships between the affective states and their influences

Emotion and mood can impact and influence each other. Moods influence which emotions will be experienced and repeated experience of emotions contributes to mood. For example, a negative mood can be triggered when a person interacts with an object or situation that elicits frustration [12]. An individual’s Big Five personality traits have an impact on their emotions. People can show different emotional responses to the same situation, and in some cases, personality is responsible for this difference. As an example, when a person with an extroverted personality is offered help by a stranger, the person may be happy about the help. On the other hand, if the person had an introverted personality, they might react with fear instead [23]. Subjective well-being can also be influenced by personality. Extroversion is the most significant predictor of positive affect, while Neuroticism is the most significant predictor of negative affect and life satisfaction [33].

An interesting point is that subjective well-being influences a person’s mood and emotion. When people make a positive judgment about their life, they will experience good emotions and moods. When a person experiences bad emotions or moods, then this is because they feels unhappy about their life expectations [19]. Feelings and emotions indicate the state of satisfaction of a person’s needs [66].

3. Emotion related lexicons and language

This section describes the emotion dictionaries that are used later in Section 4.2. Emotion dictionaries classify words into emotional dimensions, emotional categories, or both. In addition, they group emotion words into sets of synonyms.

Emotion Markup language (EmotionML) [76] is a general-purpose emotion annotation and representation language that provides a standard emotion representation format. It consists of the emotion vocabularies and their features. Figure 4 illustrates EmotionML syntax in an annotated text encoded in XML: the emotion category is afraid (emotion vocabulary) with intensity 0.4 (emotion feature).

Fig. 4.

EmotionML syntax in an emotional text with annotations encoded in XML.

Since the data is annotated in a standard way, the interpretation of the message between systems is the same. EmotionML uses Ekman’s discrete basic emotions and the PAD dimensional model to represent emotions and their features. The language can be applied in different contexts, such as data annotation and emotion recognition. The annotation can be applied to text, static images, speech recordings and video. Figure 5 demonstrates a case where an emotion is recognized from face and voice.

Fig. 5.

EmotionML syntax for an emotion that was detected from face and voice.

WordNet [55] is an online lexicon for the English language. WordNet distributes the lexicon into five categories: nouns, verbs, adjectives, adverbs and function words. It clusters words together based on their meanings and defines semantic relations between words, as well as grouping them into sets of synonyms called synsets. WordNet currently contains 155,287 words, organized into 117,659 synsets.2 ²

https://wordnet.princeton.edu/

SentiWordNet [4] is an enhanced lexical resource for supporting sentiment classification and opinion-mining applications. It assigns three scores to each synset in WordNet: positive, negative and neutral. This annotation indicates how positive, negative and neutral the terms in each synset are. For example, a sentence with a positive word such as happy will have the following scores: 1 (positive), 0 (negative), 0 (neutral).

To include concepts of affect, WordNet-Affect [79] was developed as an extension that labels synsets with emotion, mood and behavior. WordNet-Affect creates an additional hierarchy in WordNet with emotion labeling. The hierarchy of WordNet-Affect categorizes emotion words into classes such as positive emotion, negative emotion and neutral emotion.3 ³

https://www.gsi.dit.upm.es/ontologies/wnaffect/

MultiWordNet [61] is an extension of WordNet with a multilingual lexical database. It is available in Italian, Spanish, Portuguese, Hebrew, Romanian and Latin. The important relationship between words is synonymy. A group of synonyms identifies a concept.

SenticNet 3 [13] is a concept-level opinion lexicon for sentiment analysis. It includes polarity for words and multi-word expressions. The polarity can be a number in the range between $- 1$ and 1, or it can be a flag (positive or negative). SenticNet 3 contains 30,000 common and common-sense concepts. It is different from other sentiment analysis resources such as WordNet-Affect, because it associates semantics and sentics with common and common-sense knowledge. Common-sense knowledge can help to determine the polarity of a concept in a multi-word expression sentence. This improves subsequent text-based sentiment analysis.4 ⁴

http://sentic.net/

HowNet [20] is an online bilingual English and Chinese ontology. It describes the semantic relations between concepts and their attributes. A semantic relation can be expressed as synonym, antonym, etc. The top level classification in HowNet includes entity, event, attribute and attribute value. HowNet uses 80,000 words and phrases to build the ontology.5 ⁵

http://www.keenage.com/html/e_index.html

The semantic relation between concepts is language-dependent. The nature of the Chinese language is unlike English, so the semantic relation between Chinese concepts is different from English concepts.

In the field of the Semantic Web, there are many lexicons available that represent data with different formats. For example, the amount of parts of speech can differ between lexicons. Moreover, it is difficult to link them with existing ontologies. The Lexicon Model for Ontologies (LEMON) [51] supports the sharing of terminological and lexicon resources on the Semantic Web, and connects them with existing ontologies. LEMON was built based on semantics by reference. The principle consists of the two layers lexical and semantic. The lexical layer describes the morphology and syntax of a word, and considers the suffix and the prefix. Semantic layers describe the meaning of a word, and the core classes in the ontology allow to define a lexicon with a specific language and topic. Compared to WordNet, LEMON is richer in word formats and representations.

4. Existing ontologies in affective states and their influences

This section presents the existing affective state ontologies and their influences. In Section 4.1, we discuss general purpose ontologies that were re-used by more specific emotion ontologies. Re-Use can be a starting point for the creation of a new ontology and it can increase domain knowledge [57]. In Section 4.2, existing emotion ontologies are presented. There exist more ontologies that target emotion than there are ontologies for mood and other influences. In Section 4.3 then presents a mood ontology and Section 4.4 introduces a need ontology.

4.1. Re-used ontologies

This section introduces general ontologies that are being re-used for the creation of the emotion ontologies that are discussed in Section 4.2.

CONtext ONtology (CONON) [36] is used for modeling context in pervasive computing environments. The purpose of CONON is reasoning and representation in context-aware applications. CONON defines an upper ontology (general) that represents the context and situations, which can be extended by adding a domain-specific ontology. For example, CONON contains classes about a person, their location (indoor space and outdoor space) and their activity. These classes describe the contextual environment that surrounds the person. In addition, the ontology includes a class for computing devices such as applications and networks.

Friend Of A Friend (FOAF)6

⁶
http://www.foaf-project.org/
is an ontology that is used to describe a person, their activities and their relations to other people. It consists of classes that represent a person (first name, family name), their gender, age, education, organization, homepage, information about organizational project(s) they are involved in, culture, etc. On top of this, it includes an MBTI personality classification. It allows a user to find people of European culture, or people who know a certain person in a machine-readable way. Figure 6 shows a part of the FOAF ontology that contains a person’s name, email address, homepage and another person they know.

Fig. 6.
Example from FOAF Ontology.

The Provenance Ontology [56] was built to establish trust in published scientific content. The three classes Agent, Activities and Entities provide the starting point. Agents could be people, an organization, or software- that produce activity on the data (entity). The activity can be data processing, like transforming data into a different format. By using the Provenance Ontology, the history and the life cycle of a document can be obtained. The ontology also allows systems in its domain to exchange data due to a uniform terminology.
4.2. Emotion ontologies

Daily human communication carries many emotions [3] which can be expressed through text, facial expression, voice and body language. Emotion can also be influenced by the contextual environment. A person expresses the same emotion in different situations (contexts), but with different intensity. Emotion ontologies can also describe general concepts. We categorize existing emotion ontologies into the five domains as they are shown in Fig. 2.

4.2.1. Text domain

A lot of work has been put into building ontologies that analyze and detect emotion from text. People express their emotions with words in formal and informal ways. Text in social media has different characteristics: users often use slang terms and abbreviations. Additionally, users of textual media may express their emotions via emoticons.

Table 1
Summary of emotion ontologies in the text domain. In addition to the reference, the names of the ontologies are listed if available

Ontology name/Prefix Goal Emotion model Ontology re-use Lexicon/Language

Emotions Ontology [71] Analyze unstructured data Discrete

Emotive Ontology [80] Detect and analyze emotion in informal text from social media Discrete WordNet, Dictionary.com, Thesaurus.com, Oxford Dictionary, Merriam–Webster Dictionary

[1] Give the student the right feedback in e-learning Discrete

Ontology of Emotions and Feelings [49] Automatically annotate emotion in text Discrete

[64] Analyze emotion in text Discrete and Dimensional EmotionML

[44] Define emotion words and their intensity Discrete and Dimensional Japanese Emotion Expression Dictionary, EmotionML

[87] Analyze emotion in text Discrete HowNet

Onyx [72] Annotate emotion in user generated content Discrete and Dimensional Lemon, Provenance Ontology EmotionML, WorNet-Affect

SO [65] Represent the structure and the semantics of emoticons Discrete

VSO [11] Detect sentiment from visual content Discrete SentiWordNet, SentiStrength

Ontology name/Prefix	Goal	Emotion model	Ontology re-use	Lexicon/Language
Emotions Ontology [71]	Analyze unstructured data	Discrete
Emotive Ontology [80]	Detect and analyze emotion in informal text from social media	Discrete		WordNet, Dictionary.com, Thesaurus.com, Oxford Dictionary, Merriam–Webster Dictionary
[1]	Give the student the right feedback in e-learning	Discrete
Ontology of Emotions and Feelings [49]	Automatically annotate emotion in text	Discrete
[64]	Analyze emotion in text	Discrete and Dimensional		EmotionML
[44]	Define emotion words and their intensity	Discrete and Dimensional		Japanese Emotion Expression Dictionary, EmotionML
[87]	Analyze emotion in text	Discrete		HowNet
Onyx [72]	Annotate emotion in user generated content	Discrete and Dimensional	Lemon, Provenance Ontology	EmotionML, WorNet-Affect
SO [65]	Represent the structure and the semantics of emoticons	Discrete
VSO [11]	Detect sentiment from visual content	Discrete		SentiWordNet, SentiStrength

Many ontologies were built to analyze text in social media. Some of the ontologies in the text domain were created for a particular purpose, focusing on international languages like English, Chinese, Japanese, French and Italian. Table 1 summarizes emotion ontologies in the text domain. The table contains columns displaying the ontology name or prefix, the goal, the emotion model that is used, other ontologies that were re-used, and the lexicons that provided the terminology. The used emotion models can be discrete, dimensional, or based on the OCC model. It should be noted that some ontologies were built from scratch and are not based on existing ontologies or lexicons. This leads to some cells being empty in Table 1.

A system was built to analyze the unstructured informal text inside posts about electronic products to understand online consumer behavior in the market [71]. The aim of the system was text-mining in social media. In order to be able to analyze consumer behavior on social media, the Emotions Ontology was created. One of the main classes in the ontology is Sentiment, with two subclasses Happiness and sadness. For example, under happiness there exist the subclasses enjoy, fun, eager and smiling, while under sadness, there are dislike, disappointed, bad and worst. Additionally, the ontology contains classes about products such as computer, and household. The system is comprised of four modules: the ontology management module, which contains the created ontology, a user query processing module, an information foundation module, and a query analysis engine module. The ontology models the product with the associated emotions based on the social media posts. Then a user can utilize the system to make a query about the product with a specific emotion.

The Emotive Ontology [80] was built to detect and analyze emotion in informal text from social media as well. The approach detects a range of eight high-level emotions: anger, confusion, disgust, fear, happiness, sadness, shame and surprise. Each emotion in the ontology is named according to prior work (Izard, Ekman, Plutchik, Drummond), and commonly encountered emotions within Twitter messages are added. The Emotive Ontology is also capable of expressing the intensity of emotions. During the creation of the Emotive Ontology, many dictionaries and word dataset were looked at, such as WordNet, Dictionary.com, Thesaurus.com, the Oxford English online dictionary and the Merriam-Webster online dictionary. Since its goal is to detect emotion from informal slang text, websites that contain slang expressions were also examined. Examples are the Leicestershire Slang Page, the Dictionary of Slang, and the Online Slang Dictionary. To make sure that an emotion set that is as large as possible is covered, existing emotional lexicons were integrated into the Emotive Ontology. Natural Language Processing (NLP) and part of speech tagging were used as pre-processing steps for emotion detection. The ontology was tested and evaluated on a dataset taken from Twitter.

An ontology that helps to give students appropriate feedback in e-learning sessions was proposed by Arguedas et al. [1] The ontology is divided into the two main classes Emotion Awareness and Affective Feedback. In emotion awareness, the emotion is analyzed, and in affective feedback, the appropriate feedback that a teacher would give to a student is determined. The emotion awareness class includes the different types of emotions in a categorical model, moods (bored, concentrated, motivated, unsafe) and behaviors that students experience in e-learning environments. The emotion is detected during collaborative virtual learning processes, including textual conversations, debates and wikis.

In [49] the Ontology of Emotions and Feelings that automatically annotates emotion in texts and determines their intensity is introduced. This French ontology classifies 950 words (600 are verbs and 350 are nouns) into 38 semantic classes according to their meanings. Words in the lexicon are emotionally labeled as positive, negative and neutral. NaviTexte, a software designed for text navigation, was used to apply the ontology. It understands and applies knowledge to a specific text [15]. The goal of the ontology is to automatically annotate emotions in texts and to automatically navigate through the text.

An ontology of emotion objects is introduced in [64]. Emotion objects are collected from a large, Japanese blog corpus. An emotive expression lexicon for Japanese language is used to distinguish emotion words. The ontology is created using an EmotionML annotation scheme, that was modified to meet the needs of the Japanese language. The ontology classes represent emotion according to Nakamura’s classification which is “a collection of over two thousand expressions describing emotional states collected manually from a wide range of literature” [63]. The emotion in the ontology is represented in a dimensional model. The ontology also contains classes for number of characters, part of speech, and semantic categories. In the latter class, emotion objects are categorized into groups such as human activities and abstract objects.

To define emotion words and their intensity in Japanese, a Japanese emotion ontology was proposed [44]. Emotion words were taken from websites such as Twitter. The intensity calculation is based on how many times an emotion word appears in a document. The words are categorized into ten emotions: joy, anger, sadness, fear, shame, like, disgust, exciting, comforted and surprise. The authors adapt their ontology into other emotion classifications which are positive, negative and neutral. The Pleasure–Arousal–Dominance theory is adapted as well. The authors used OWL and EmotionML to describe the ontology. One of the proposed applications for the ontology is a character generator. The system can receive voice inputs. The audio is then translated into text and analyzed by the emotion ontology. The output is a character with facial animations.

To analyze Chinese text, a Chinese emotion ontology was created [87]. It was semi-automatically created using HowNet. The ontology contains 113 emotion categories and was created by first extracting affective events from the dictionary. Then, emotions are manually assigned to the semantic role of the events, producing the Emotion Prediction Hierarchy. Finally, the Emotion Prediction Hierarchy is transformed into the emotion ontology. This step involves assigning verbs extracted from the dictionary to the Emotion Prediction Hierarchy.

For advanced emotion analysis, the Onyx Ontology was presented [72]. It offers a comprehensive set of tools for any kind of emotion analysis. Onyx re-uses the Provenance Ontology and the LEMON model. In the Provenance Ontology, the activity is emotion analysis, which means turning plain data into semantic emotion information. The Onyx ontology thus has a class called Emotion Analysis that is responsible for representing the information source, e.g., a website, the algorithm, used as well as the emotion model. The Emotion Set class contains information about a group of emotions found in a text, including the person expressing the emotion, the domain, information about the original text and the sentences that contain the emotion. The Emotion class of the ontology contains information about the emotion model, appraisal, action tendency and emotion intensity. To support the annotation process, WorNet-Affect and EmotionML were used. Two different testing scenarios were created in order to evaluate the ontology. The first is to make queries against the ontology, and second is translating EmotionML resources to Onyx and vice versa.

Users of social media express their emotions using emoticons. The Smiley Ontology (SO) represents the structure and the semantic meaning of emoticons [65], which allows an application to understand and utilize emoticons. Moreover, it allows applications to exchange emoticons with the correct interpretation. The ontology design is based on Smiley Layer Cake.7 ⁷

http://www.slideshare.net/milstan/beyond-social-semantic-web

The model consists of three layers. The bottom layer that deals with the message between the sender and the receiver (Underlying Emotion). The second layer is the Structure of the emoticon, representing the emoticons that the message contains, such as text, face or object. The top layer (Visual Appearance) describes the appearance of the emoticon, such as its color and whether it is animated or not. The core class of the Smiley Ontology is the Emoticon class, which represents the concept of an emoticon. Emoticons can be visually represented as a sequence of characters, a picture, or both. Each system has its own set of pictures that represent emoticons. The ontology has a class named Emoticon System that contains all possible pictures for the emoticons generated from a social software tool.

The Visual Sentiment Ontology (VSO) [11] was proposed to detect sentiment from visual content. The psychological foundation for the ontology is Plutchik’s wheel of emotion. The initial step to build the ontology is to retrieve images and videos from Flickr and YouTube respectively. The tags associated with the images and videos were analyzed. Then the top 100 tags for different emotions were obtained. For each tag, the sentiment value is computed according to the two linguistic models SentiWordNet and SentiStrength [82]. Due to this use of linguistic models on the image tags, we decided to categorize the VSO into the text domain. The assigned sentiment value for each emotion word ranges from $- 1$ (negative) to $+ 1$ (positive). The words obtained from the tags are classified into nouns and adjectives. From the collected data, Adjective Noun Pairs (ANP) like “misty night” or “colourful clouds” were obtained. The top level of the VSO shows the relationship between the emotions, ANP, and sentiment values. The VSO contains more than 3,000 ANP. One application of VSO is the prediction of the sentiment expressed in a Twitter image.

4.2.2. Facial expression domain

It has been shown that emotional facial expressions make up 55% of our communication [3]. Emotion is expressed in humans by facial movement. For example, when a person is surprised, they open their mouth and raise the eyebrows. Table 2 summarizes emotion ontologies in the facial expressions domain.

Table 2
Summary of emotion ontologies in the facial expression domain

Ontology name/Prefix Goal Emotion model Ontology re-use Lexicon/Language

[32] Model emotional facial expressions in virtual environments Discrete and Dimensional

[38] Represent non-verbal behavior in virtual environments Discrete

OLA [25] Predict students’ emotions during e-learning OCC

Ontology name/Prefix	Goal	Emotion model
[32]	Model emotional facial expressions in virtual environments	Discrete and Dimensional
[38]	Represent non-verbal behavior in virtual environments	Discrete
OLA [25]	Predict students’ emotions during e-learning	OCC

An emotion ontology was created to support the modeling of emotional facial animation expression in virtual humans within MPEG-4 [32]. Human actions are translated into a virtual world with avatars by using an ontology. A virtual world (environment) is a computer graphic-based environment that generates the impression that users are in a different place than their actual location [24]. The ontology allows storing, indexing and retrieving the right information about facial animation for a given emotion. The ontology define the relationship between facial animation concepts standardized in MPEG-4 and emotions. It contains the classes Face, Face Animation, Face Expression, Emotion and Emotion Model (Ekman and Plutchik). In addition, a class for parameters of facial animations is present. To use the ontology and extract the correct information for a specific emotion, the Racer Query Language interface for OWL is used [5]. This query language is close to natural language. For example, we can query using following question: What is the facial animation that expresses a depressed emotion?

Another emotions ontology was proposed within a framework called Nonverbal Toolkit for the cooperation of heterogeneous modules that gather, analyze and present non-verbal communication cues [38]. The aim of this framework is to gather non-verbal behavior in the real world and represent it in a virtual environment, such as an avatar in second life.8 ⁸

http://secondlife.com

To ease the communication and the exchange between the modules, an ontology was developed. The ontology defines shared vocabulary that can be understood and used by all modules. It represents emotion by using a categorical approach (Ekman Model) as well as complex emotions, which are a mix of more than one simple emotion. For a good representation of the non-verbal communication cue level in the virtual environment, emotion intensity is also specified. Moreover, the affective states are partially derived from personality traits. Human personality affects the expressed emotion and its intensity.

Table 3

Summary of individual ontologies that do not fit into the domains of the other tables

Ontology name/Prefix	Goal	Model	Domain
OntoEmotion [29]	Convert unmarked input text to emotional voice	Discrete and Dimensional	Emotion Voice Domain
[31]	Represents body gestures in a virtual environment	Dimensional	Emotion Body Expression Domain
COMUS [77]	Retrieve music in a sematic way	Discrete	Mood
FHN [17]	Express the relationship between various needs and their satisfiers	Manfred Max-Neef	Need

Table 4

Summary of emotion ontologies in the contextual environment domain

Ontology name/Prefix	Goal	Emotion model	Ontology re-use	Lexicon/Language
[9]	Represent the affective states for context aware applications	Discrete	CONON
BIO_EMOTION [88]	Recognize emotion based on the user’s biomedical factors and environment	Discrete and Dimensional

In the e-learning domain, the Ontology for Predicting Students Learners’ Affect (OLA) [25] was introduced. The ontology is based on the OCC model of emotions. An interactive application was designed to estimate student emotion when interacting with a quiz about Java programming. The application monitors and records student action when answering questions and saves them in the student’s log file. When a student does not answer a question correctly, the event status becomes confirmed and appreciation is set to disliking. On the other hand, when the student answers a question correctly, appraisal become desirable. So, the log file data is the input for the ontology to compute the OCC model variables that predict student emotion by using ontology inference. Students may express different emotions while answering a single question. Therefore, the students’ emotions were studied in three different situations: When the students see the question for the first time, when the students choose the answer, and when the correct answer is displayed.

4.2.3. Voice domain

Emotion can be detected from voice by analyzing the change in voice tone, volume, rate, pitch, and the pauses between words [29]. In voice emotion extraction, the EmoSpeech system was built to convert unmarked input text to emotional voice. The developed emotion ontology (OntoEmotion) is organized in a taxonomy that covers the basic emotions to the most specific emotional categories [29]. OntoEmotion is presented in English and Spanish. The emotion class in the ontology uses the categorical model. The ontology represents the specific words each language offers for denoting emotion in its class named Word. To classify words in English and Spanish, the class Word has two subclasses named English Word and Spanish Word. Another class in the ontology defines emotion synonyms. The emotion concepts are linked to the three emotional dimensions of Evaluation, Activation and Power. This is in accordance with Osgood et al. as mentioned in Section 2. The EmoSpeech system uses EmoTag, which is a tool for automated mark-up of texts with emotional labels. These values are the input for the ontology. The ontology then classifies the input into an emotional concept. Next, the text is read aloud with the emotion assigned by EmoTag. For instance, EmoTag was applied on fairy tale stories in English and Spanish to annotate emotions [30].

Another application was designed to extract rich emotional semantics of tagged Italian artistic resources through an ontology method [6]. To select the tags that contain emotional content, several Semantic Web and natural language processing tools were incorporated such as multilingual lexicons (MultiWordNet) and affective lexicons (WordNet-Affect, and SentiWordNet). The software uses OntoEmotion because it has a taxonomic structure that reflects psychological models of emotions and is implemented by using Semantic Web technologies. However, the ontology was enhanced by adding a new subclass named Italian Word to the root concept Word.

Table 3 summarizes individual ontologies from the following domains: Emotion Voice Domain (OntoEmotion), Emotion Body Expression Domain, mood (COMUS), and need (FHN). Since there is only a small number of ontologies in these domains, we decided to create a table for miscellaneous domains.

4.2.4. Body expressions domain

Gestures and expressions of the human body can convey emotion. In [31] an ontology of body expressions to represent body gestures in virtual humans within MPEG-4 is presented. Animations are annotated with emotional information by using Whissel’s wheel of emotion. Because of the complexity of bodily expression, gestures were associated with emotions. In the ontology, seven gestures were considered. For example, hand clapping is associated with joy and excitement. To use the ontology, a query with natural language was used.

4.2.5. Contextual environment domain

Analyzing emotions within a given context gives insights to the relationship between an emotion and its cause. Table 4 summarizes emotion ontologies in the contextual environment domain.

Table 5
Summary of emotion ontologies in the general domain

Ontology name/Prefix Goal Emotion model Ontology re-use Lexicon/Language

HEO [34] Annotate emotion in multimedia data Discrete and Dimensional FOAF

[58] Describe emotional cues Discrete and Dimensional

SHEO [2] Identify complex emotions in text Discrete and Dimensional HEO

Ontology name/Prefix	Goal	Emotion model	Ontology re-use
HEO [34]	Annotate emotion in multimedia data	Discrete and Dimensional	FOAF
[58]	Describe emotional cues	Discrete and Dimensional
SHEO [2]	Identify complex emotions in text	Discrete and Dimensional	HEO

An ontology to represent the affective states in context-aware applications was generated in [9]. It expresses the relationship between affective states and other contextual elements such as time and location. The ontology is built based on the existing ontology CONON. Its Emotion class (state) uses Ekman’s basic emotions. In addition, the class named Secondary contains emotion that is related to the targeted scenario. The ontology was applied to a visit to an art museum, where a person moves from one room to another while their emotions were monitored. The secondary emotions are thus relaxed and stressed. The ontology represents the most powerful emotion (dominant). Object properties are used to express the relation between emotions and context elements.

The ontology in [8] was built based on the previous ontology [9]. However, emotion was defined by three possible data type properties: positive, negative and neutral.

The BIO_EMOTION ontology recognizes emotion based on the user’s electroencephalographic (EEG) and bio-signal features, as well as the situation and environmental factors [88]. It supports reasoning about the user’s emotional state. The focus of the ontology is the mapping between low-level biometric features and high-level human emotion. It defines inference rules by using corresponding relationships between EEG and emotion. The BIO_EMOTION ontology consists of 84 classes and 38 properties. The Emotion class defines user affective states. Emotions are represented by Ekman’s discrete model and the dimensional circumplex model. User context is represented by the Situation class through location, time and event. Demographic information like name, age, and gender is integrated into the ontology. Additionally, a class to represent bio signals is provided. The machine learning software WEKA was used to generate IF-THEN statements in the reasoning process.9 ⁹

http://www.cs.waikato.ac.nz/ml/weka/

To evaluate the ontology, the DEAP dataset is used, which is a database for emotion analysis using physiological signals [45].

4.2.6. General domain

General ontologies, also called upper level ontologies, have been proposed to recognize emotion. They define general concepts that are the common in a domain. Such ontologies can be extended according to the developer’s purpose by defining domain-specific classes, or they can be linked to an existing domain-specific ontology. Table 5 summarizes emotion ontologies in the general domain.

A high-level ontology named the Human Emotions Ontology (HEO) was developed in order to annotate emotion in multimedia data [34]. The main class in the ontology is Emotion which is expressed in dimensional and categorical models. An emotion has an intensity, appraisals and action tendencies, and it can be expressed through face, text, voice and gesture. Additionally, the ontology contains classes for the multimedia content and the annotator of the media. The Annotator class has two subclasses: Human or Machine (automatically annotated). Since the emotion is expressed by a person, HEO re-uses the Friend Of A Friend (FOAF) ontology. A subclass Observed Person of class person was created in FOAF and connected to the Emotion class of HEO. Moreover, some object properties were added in FOAF that are relevant to emotion such as age, culture, language and education.

The Semantic Human Emotion Ontology (SHEO) [2] was built based on HEO to identify complex emotions that are composed of two or more simple emotions. For instance, the complex emotion of contempt is a combination of the two basic emotions anger and disgust. Software was designed to use the ontology for analyzing simple and complex emotions in text as well as emotion in images.

In the general context, the Ontology of Emotional Cues to describe emotional cues at different levels of abstraction was proposed [58]. The concepts are gathered into three modules to detect emotion. Emotion is represented by a categorical and a dimensional model. An emotional cue can be simple or complex. This is represented in the emotional cue module. A simple cue can be expressed through facial expressions, gesture and speech. Complex emotional cues are a combination of two or more simple cues. The media module describes the properties of the emotional cues.

4.3. Mood ontologies

Mood can affect a person’s daily life choices. Building ontologies that can match a person’s mood with their desires allows for greater satisfaction. Oftentimes, people choose the music they listen to based on their current mood.

A recommendation-based music system was built with the Context-Based Music Recommendation Ontology (COMUS) to retrieve music in a semantic way [77]. The ontology reasons about the user’s mood, situation and preferences. It consists of classes concerning the Person, their Mood and the Music. The COMUS Ontology is connected to many ontologies such as FOAF. Some classes in COMUS, such as the person class that includes personal information, are similar to the FOAF ontology. The Person class defines general personal properties such as name, age, gender and hobby. Additionally, the ontology represents the user contexts event, time and location. The Mood class use a discrete model. Each mood class has a sub-class of mood similarity. For example, aggressive is similar to hostile, angry, etc. The other classes are domain-specific and related to music. Users can ask the music recommender system to find songs based on their mood. The system then delivers the appropriate music to them with the help of the ontology.

4.4. Need ontologies

Understanding and conceptualizing human needs helps to achieve human satisfaction. Building ontologies that define human needs through vocabulary and relationships allows to build systems that can automatically interpret and serve human needs.

The Fundamental Human Needs Ontology (FHN) was introduced to express the relationship between various needs and their satisfiers [17]. The ontology is based on the need model by Max-Neef. It represents the Agent and his or her Role, their Needs and Satisfiers. A person can play different roles, and each role requires different satisfiers. For example, when a person is at home their needs are satisfied differently from when they are at work.

5. Conclusions

This paper surveys existing ontologies in human affective states (emotion and mood) and their influences (personality, needs and subjective well-being). Emotion can be expressed in many ways such as in text, voice, facial expressions, and gestures. Great efforts have been made to build ontologies to detect and annotate emotions, while there are only few that target mood and human influence. We believe that investigating ontologies for mood, needs and personality in a similar way as it has been done for emotion is an interesting avenue for future research.

Subjective well-being corresponds to human life satisfaction, which in turn leads to positive emotion and good moods. It is thus an important influence factor on human affective states. After exploring existing ontologies, and to the best of our knowledge, we did not find ontologies regarding subjective well-being. Regarding personality, the Friend Of A Friend ontology includes MBTI personality traits. Even though MBTI is a popular personality theory, it is only used in the training world for businesses, while the Big Five personality model is preferred in academic research. It thus seems surprising that the Big Five model has not been considered in the design of any of the surveyed ontologies. We believe that creating ontologies for subjective well-being and the Big Five personality traits are interesting future fields of research.

Existing emotion ontologies display many similarities regarding the language, their classes and the Psychological theories that have been adapted. The OWL language is used in many ontologies such as in [44], and [32]. Also, it can be seen that the Ekman theory was adapted in many ontologies such as [32,80], and [38]. In contrast to Ekman’s emotion categories, the OCC model is also a popular model in the computer science area. It takes events and their causes into consideration. Hence, it is a valuable model to track the shift of human emotion when an event takes place. The dimensional emotion model, which is also employed frequently, plots emotions with multiple dimensions. As a result, it distinguishes variances in these dimensions for different people with similar emotions. For example, two persons with the same emotion “happy”, can have different numbers for each dimensions.

The existing ontologies were built to serve a variety of different languages like English, Chinese, Spanish, Japanese, French and Italian. An interesting observation is the existence of multilingual ontologies such as OntoEmotion. Multilingual ontologies can be built by mapping ontologies in different languages onto each other. Ontology mapping can be seen as a re-using process, which is one of the benefits of building ontologies. Mappings can be managed by using a multilingual dictionary such as WordNet. Several projects have adapted WordNet for the use with different languages. In the mapping process, the main step is finding semantic relationships between concepts in different ontologies. Creating a multilingual ontology allows for the building of multilingual applications that overcome the language barrier. This will open up an application to a wider range of users [22,78]. Areas like the medical domain and many others can benefit from multilingual ontologies. Physicians can make sure that they are referring to the same concept in multiple languages. Social media applications can also benefit from a multilingual ontology, because their users speak different languages.

References

Arguedas,

Xhafa,

Daradoumis, An ontology about emotion awareness and affective feedback in elearning, in: 2015 International Conference on Intelligent Networking and Collaborative Systems, INCoS 2015, Taipei, Taiwan, September 2–4, 2015,

Xhafa and

Barolli, eds, IEEE, 2015, pp. 156–163. doi:10.1109/INCoS.2015.78.

S.A.

Arias Tapia,

H.F.

Gómez Alvarado,

Barbosa Corbacho,

Ratte,

Torres-Diaz,

P.V.

Torres-Carrion and

J.M.

Garcia, A contribution to the method of automatic identification of human emotions by using semantic structures, in: Proceedings of ICL2014 – 2014 International Conference on Interactive Collaborative Learning, IEEE, 2014, pp. 60–70. doi:10.1109/ICL.2014.7017748.

Arumugam and

Purushothaman, Emotion classification using facial expression, International Journal of Advanced Computer Science and Applications (IJACSA) 2(7) (2011), 92–98. doi:10.14569/IJACSA.2011.020714.

Baccianella,

Esuli and

Sebastiani, SentiWordNet 3.0: An enhanced lexical resource for sentiment analysis and opinion mining, in: Proceedings of the International Conference on Language Resources and Evaluation, LREC 2010, Valletta, Malta, 17–23 May 2010,

Calzolari,

Choukri,

Maegaard

Mariani

Odijk

Piperidis

Rosner and

Tapias, eds, European Language Resources Association, 2010, http://www.lrec-conf.org/proceedings/lrec2010/summaries/769.html .

C.J.O.

Baker,

Su,

Butler and

Haarslev, Ontoligent interactive query tool, in: Canadian Semantic Web, CSWWS 2006, First Canadian Semantic Web Working Symposium, Quebec, Canada, June 2006,

M.T.

Kone and

Lemire, eds, Semantic Web and Beyond: Computing for Human Experience, Vol. 2, Springer, 2006, pp. 155–169. doi:10.1007/978-0-387-34347-1_11.

Baldoni,

Baroglio,

Patti and

Rena, From tags to emotions: Ontology-driven sentiment analysis in the social semantic web, Intelligenza Artificiale 6(1) (2012), 41–54. doi:10.3233/IA-2012-0028.

Bayne, The “big five” versus the Myers–Briggs, The Psychologist 7(1) (1994), 14–16, https://thepsychologist.bps.org.uk/volume-7/edition-1 .

K.-I.

Benta,

Cremene and

Todica, Towards an affective aware home, in: Ambient Assistive Health and Wellness Management in the Heart of the City, 7th International Conference on Smart Homes and Health Telematics, ICOST 2009, Tours, France, July 1–3, 2009, Proceedings,

Mokhtari,

Khalil,

Bauchet,

Zhang and

C.D.

Nugent, eds, Lecture Notes in Computer Science, Vol. 5597, Springer, 2009, pp. 74–81. doi:10.1007/978-3-642-02868-7_10.

K.-I.

Benta,

Rarau and

Cremene, Ontology based affective context representation, in: Proceedings of the 2007 Euro American Conference on Telematics and Information Systems, EATIS 2007, Faro, Portugal, May 14–17, 2007,

R.P.

Chagas do Nascimento,

Berqia,

Serendero and

Carrillo Zambrano, eds, 2007, p. 46. doi:10.1145/1352694.1352741.

10.

Blomqvist, The use of Semantic Web technologies for decision support – A survey, Semantic Web 5(3) (2014), 177–201. doi:10.3233/SW-2012-0084.

11.

Borth,

Ji,

Chen,

T.M.

Breuel and

S.-F.

Chang, Large-scale visual sentiment ontology and detectors using adjective noun pairs, in: ACM Multimedia Conference, MM ’13, Barcelona, Spain, October 21–25, 2013,

Jaimes,

Sebe,

Boujemaa,

Gatica-Perez,

D.A.

Shamma,

Worring and

Zimmermann, eds, ACM, 2013, pp. 223–232. doi:10.1145/2502081.2502282.

12.

Brave and

Nass, Emotion in human–computer interaction, in: The Human–Computer Interaction Handbook: Fundamentals, Evolving Technologies and Emerging Applications,

J.A.

Jacko and

Sears, eds, L. Erlbaum Associates Inc., Hillsdale, NJ, USA, 2003, pp. 81–96, chapter 4, http://dl.acm.org/citation.cfm?id=772081.

13.

Cambria,

Olsher and

Rajagopal, SenticNet 3: A common and common-sense knowledge base for cognition-driven sentiment analysis, in: Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence, Québec City, Québec, Canada, July 27–31, 2014,

C.E.

Brodley and

Stone, eds, AAAI Press, 2014, pp. 1515–1521, http://www.aaai.org/ocs/index.php/AAAI/AAAI14/paper/view/8479 .

14.

P.T.

Costa and

R.R.

MacCrae, Revised NEO Personality Inventory (NEO PI-R) and NEO Five-Factor Inventory (NEO FFI): Professional Manual, Psychological Assessment Resources, 1992, https://books.google.com/books?id=H_jXtgAACAAJ .

15.

Couto and

J.-L.

Minel, Navitexte, a text navigation tool, in: AI*IA 2007: Artificial Intelligence and Human-Oriented Computing, 10th Congress of the Italian Association for Artificial Intelligence, Rome, Italy, September 10–13, 2007, Proceedings,

Basili and

M.T.

Pazienza, eds, Lecture Notes in Computer Science, Vol. 4733, Springer, 2007, pp. 720–729. doi:10.1007/978-3-540-74782-6_62.

16.

Cowie,

Douglas-Cowie,

Tsapatsoulis,

Votsis,

Kollias,

Fellenz and

J.G.

Taylor, Emotion recognition in human–computer interaction, IEEE Signal Processing Magazine 18(1) (2001), 32–80. doi:10.1109/79.911197.

17.

S.D.

Dsouza, Cloud-based ontology solution for conceptualizing human needs, Master’s thesis, Université d’Ottawa/University of Ottawa, 2015.

18.

Díaz Rodríguez,

M.P.

Cuéllar,

Lilius and

Delgado Calvo-Flores, A survey on ontologies for human behavior recognition, ACM Computing Surveys 46(4) (2013), 43:1–43:33. doi:10.1145/2523819.

19.

Diener and

M.Y.

Chan, Happy people live longer: Subjective well-being contributes to health and longevity, Applied Psychology: Health and Well-Being 3(1) (2011), 1–43. doi:10.1111/j.1758-0854.2010.01045.x.

20.

Dong,

Dong and

Hao, HowNet and its computation of meaning, in: COLING 2010, 23rd International Conference on Computational Linguistics, Demonstrations Volume, Beijing, China, 23–27 August 2010, Association for Computational Linguistics, 2010, pp. 53–56, http://aclweb.org/anthology-new/C/C10/C10-3014.pdf .

21.

Douglas-Cowie,

Cowie,

Sneddon,

Cox,

Lowry,

McRorie,

J.-C.

Martin,

Devillers,

Abrilian,

Batliner,

Amir and

Karpouzis, The HUMAINE database: Addressing the collection and annotation of naturalistic and induced emotional data, in: Affective Computing and Intelligent Interaction, Second International Conference, ACII 2007, Lisbon, Portugal, September 12–14, 2007, Proceedings,

Paiva,

Prada and

R.W.

Picard, eds, Lecture Notes in Computer Science, Vol. 4738, Springer, 2007, pp. 488–500. doi:10.1007/978-3-540-74889-2_43.

22.

Dragoni, Multilingual ontology mapping in practice: A support system for domain experts, in: The Semantic Web – ISWC 2015 – 14th International Semantic Web Conference, Bethlehem, PA, USA, October 11–15, 2015, Proceedings, Part II,

Arenas,

Ó.

Corcho,

Simperl,

Strohmaier,

d’Aquin,

Srinivas,

P.T.

Groth,

Dumontier,

Heflin,

Thirunarayan and

Staab, eds, Lecture Notes in Computer Science, Vol. 9367, Springer, 2015, pp. 169–185. doi:10.1007/978-3-319-25010-6_10.

23.

Egges,

Kshirsagar and

Magnenat-Thalmann, A model for personality and emotion simulation, in: Knowledge-Based Intelligent Information and Engineering Systems, 7th International Conference, KES 2003, Oxford, UK, September 3–5, 2003, Proceedings, Part I,

Palade,

R.J.

Howlett and

L.C.

Jain, eds, Lecture Notes in Computer Science, Vol. 2773, Springer, Oxford, UK, 2003, pp. 453–461. doi:10.1007/978-3-540-45224-9_63.

24.

S.R.

Ellis, What are virtual environments?, IEEE Computer Graphics and Applications 14(1) (1994), 17–22. doi:10.1109/38.250914.

25.

Eyharabide,

Amandi,

Courgeon,

Clavel,

Zakaria and

J.-C.

Martin, An ontology for predicting students’ emotions during a quiz. Comparison with self-reported emotions, in: 2011 IEEE Workshop on Affective Computational Intelligence, WACI 2011, Paris, France, April 14, 2011, IEEE, 2011, pp. 76–83. doi:10.1109/WACI.2011.5953153.

26.

Feidakis,

Daradoumis and

Caballe, Emotion measurement in intelligent tutoring systems: What, when and how to measure, in: Intelligent Networking and Collaborative Systems (INCoS), 2011 Third International Conference on, IEEE, 2011, pp. 807–812. doi:10.1109/INCoS.2011.82.

27.

J.V.

Fonou-Dombeu and

Huisma, Adaptive search and selection of domain ontologies for reuse on the Semantic Web, Journal of Emerging Technologies in Web Intelligence 5(3) (2013), 230–239. doi:10.4304/jetwi.5.3.230-239.

28.

J.R.

Fontaine,

K.R.

Scherer,

E.B.

Roesch and

P.C.

Ellsworth, The world of emotions is not two-dimensional, Psychological Science 18(12) (2007), 1050–1057. doi:10.1111/j.1467-9280.2007.02024.x.

29.

Francisco,

Gervás and

Peinado, Ontological reasoning to configure emotional voice synthesis, in: Web Reasoning and Rule Systems, First International Conference, RR 2007, Innsbruck, Austria, June 7–8, 2007, Proceedings,

Marchiori,

J.Z.

Pan and

de Sainte Marie, eds, Lecture Notes in Computer Science, Vol. 4524, Springer, 2007, pp. 88–102. doi:10.1007/978-3-540-72982-2_7.

30.

Francisco and

Hervás, EmoTag: Automated mark up of affective information in texts, in: EUROLAN 2007 Summer School, Alexandru Ioan Cuza University of Iaşi, Proceedings of the Doctoral Consortium, Iaşi, July 30–August 2, 2007,

Forăscu,

Postolache,

Puşcaşu and

Vertan, eds, 2007, pp. 5–12, http://eurolan.info.uaic.ro/2007/DC/DC_final-vol.pdf .

31.

García-Rojas,

Vexo,

Thalmann,

Raouzaiou,

Karpouzis and

S.D.

Kollias, Emotional body expression parameters in virtual human ontology, in: Proceedings of 1st International Symposium on Shapes and Semantics, 2006, pp. 63–70.

32.

García-Rojas,

Vexo,

Thalmann,

Raouzaiou,

Karpouzis,

S.D.

Kollias,

Moccozet and

Magnenat-Thalmann, Emotional face expression profiles supported by virtual human ontology, Journal of Visualization and Computer Animation 17(3–4) (2006), 259–269. doi:10.1002/cav.130.

33.

J.L.

González Gutiérrez,

Moreno Jiménez,

Garrosa Hernández and

Peñacoba Puente, Personality and subjective well-being: Big five correlates and demographic variables, Personality and Individual Differences 38(7) (2005), 1561–1569. doi:10.1016/j.paid.2004.09.015.

34.

Grassi, Developing HEO human emotions ontology, in: Biometric ID Management and Multimodal Communication, Joint COST 2101 and 2102 International Conference, BioID_MultiComm 2009, Madrid, Spain, September 16–18, 2009, Proceedings,

Fiérrez-Aguilar,

Ortega-Garcia,

Esposito,

Drygajlo and

Faúndez-Zanuy, eds, Lecture Notes in Computer Science, Vol. 5707, Springer, 2009, pp. 244–251. doi:10.1007/978-3-642-04391-8_32.

35.

T.R.

Gruber, A translation approach to portable ontology specifications, Knowledge Acquisition 5(2) (1993), 199–220. doi:10.1006/knac.1993.1008.

36.

Gu,

X.H.

Wang,

H.K.

Pung and

D.Q.

Zhang, An ontology-based context model in intelligent environments, in: Proceedings of 2004 Communication Networks and Distributed Systems Modeling and Simulation Conference (CNDS ’04), Vol. 2004, 2004, pp. 270–275.

37.

Hoekstra, The knowledge reengineering bottleneck, Semantic Web 1(1–2) (2010), 111–115. doi:10.3233/SW-2010-0004.

38.

Honold,

Schüssel,

Panayotova and

Weber, The Nonverbal Toolkit: Towards a framework for automatic integration of nonverbal communication into virtual environments, in: 2012 Eighth International Conference on Intelligent Environments, Guanajuato, México, June 26–29, 2012, IEEE, 2012, pp. 243–250. doi:10.1109/IE.2012.13.

39.

Horrocks, Ontologies and the semantic web, Communications of the ACM 51(12) (2008), 58–67. doi:10.1145/1409360.1409377.

40.

E.H.

Hovy, What are sentiment, affect, and emotion? Applying the methodology of Michael Zock to sentiment analysis, in: Language Production, Cognition, and the Lexicon,

Gala,

Rapp and

Bel-Enguix, eds, Text, Speech and Language Technology, Vol. 48, Springer, 2015, pp. 13–24. doi:10.1007/978-3-319-08043-7_2.

41.

Hudlicka and

Gunes, Benefits and limitations of continuous representations of emotions in affective computing: Introduction to the special issue, International Journal of Synthetic Emotions 3(1) (2012), http://www.igi-global.com/Files/Ancillary/64d30392-4125-4708-9e7f-9599f99a6657_IJSE3(1)Preface.pdf.

42.

Hume, Emotions and moods, in: Organizational Behavior,

S.P.

Robbins and

T.A.

Judge, eds, Pearson, London, UK, 2012, pp. 258–297.

43.

C.E.

Izard, Emotion theory and research: Highlights, unanswered questions, and emerging issues, Annual Review of Psychology 60 (2009), 1–25. doi:10.1146/annurev.psych.60.110707.163539.

44.

Kaneko and

Okada, Building of Japanese emotion ontology from knowledge on the web for realistic interactive CG characters, in: Proceedings, 2013 Seventh International Conference on Complex, Intelligent, and Software Intensive Systems, CISIS 2013, Asia University, Taichung, Taiwan, 3–5 July 2013,

Barolli,

Xhafa,

H.-C.

Chen,

A.F.

Skarmeta Gómez and

Hussain, eds, IEEE, 2013, pp. 735–740. doi:10.1109/CISIS.2013.132.

45.

Koelstra,

Mühl,

Soleymani,

J.-S.

Lee,

Yazdani,

Ebrahimi,

Pun,

Nijholt and

Patras, DEAP: A database for emotion analysis using physiological signals, IEEE Transactions on Affective Computing 3(1) (2012), 18–31. doi:10.1109/T-AFFC.2011.15.

46.

Laurier,

Sordo,

Serrà and

Herrera, Music mood representations from social tags, in: Proceedings of the 10th International Society for Music Information Retrieval Conference, ISMIR 2009, Kobe International Conference Center, Kobe, Japan, October 26–30, 2009,

Hirata,

Tzanetakis and

Yoshii, eds, International Society for Music Information Retrieval, 2009, pp. 381–386, http://ismir2009.ismir.net/proceedings/OS5-4.pdf .

47.

J.S.

Lerner,

Li,

Valdesolo and

K.S.

Kassam, Emotion and decision making, Annual Review of Psychology 66 (2015), 799–823. doi:10.1146/annurev-psych-010213-115043.

48.

A.H.

Maslow, Motivation and Personality, 2nd edn, Harper, 1970, http://www.maslow.com/contents/motivationandpersonality1954.htm .

49.

Y.Y.

Mathieu, Annotation of emotions and feelings in texts, in: Affective Computing and Intelligent Interaction, First International Conference, ACII 2005, Beijing, China, October 22–24, 2005, Proceedings,

Tao,

Tan and

R.W.

Picard, eds, Lecture Notes in Computer Science, Vol. 3784, Springer, 2005, pp. 350–357. doi:10.1007/11573548_45.

50.

Max-Neef, Development and human needs, in: Real-Life Economics: Understanding Wealth Creation,

Ekins and

Max-Neef, eds, Routledge, 1992, pp. 197–213.

51.

J.P.

McCrae,

Spohr and

Cimiano, Linking lexical resources and ontologies on the Semantic Web with Lemon. in: The Semantic Web: Research and Applications – 8th Extended Semantic Web Conference, ESWC 2011, Heraklion, Crete, Greece, May 29–June 2, 2011, Proceedings, Part I,

Antoniou,

Grobelnik,

Paslaru Bontas Simperl,

Parsia,

Plexousakis,

De Leenheer and

J.Z.

Pan, eds, Lecture Notes in Computer Science, Vol. 6643, Springer, 2011, pp. 245–259. doi:10.1007/978-3-642-21034-1_17.

52.

R.R.

McCrae and

O.P.

John, An introduction to the five-factor model and its applications, Journal of Personality 60(2) (1992), 175–215. doi:10.1111/j.1467-6494.1992.tb00970.x.

53.

Mehrabian, Pleasure-arousal-dominance: A general framework for describing and measuring individual differences in temperament, Current Psychology 14(4) (1996), 261–292. doi:10.1007/BF02686918.

54.

Meroño-Peñuela,

Ashkpour,

van Erp,

Mandemakers,

Breure,

Scharnhorst,

Schlobach and

van Harmelen, Semantic technologies for historical research: A survey, Semantic Web 6(6) (2015), 539–564. doi:10.3233/SW-140158.

55.

G.A.

Miller, WordNet: A lexical database for English, Communications of the ACM 38(11) (1995), 39–41. doi:10.1145/219717.219748.

56.

Moreau,

Clifford,

Freire,

Futrelle,

Gil,

P.T.

Groth,

Kwasnikowska,

Miles,

Missier,

Myers,

Plale,

Simmhan,

E.G.

Stephan and

Van den Bussche, The open provenance model core specification (v1.1), Future Generation Computer Systems 27(6) (2011), 743–756. doi:10.1016/j.future.2010.07.005.

57.

N.F.

Noy,

D.L.

McGuinness et al., Ontology development 101: A guide to creating your first ontology, Knowledge Systems Laboratory Technical Report KSL-01-05 and Medical Informatics Technical Report SMI-2001-0880, Stanford University, 2001, http://www.ksl.stanford.edu/people/dlm/papers/ontology-tutorial-noy-mcguinness-abstract.html.

58.

Obrenovic,

Garay,

J.M.

López,

Fajardo and

Cearreta, An ontology for description of emotional cues, in: Affective Computing and Intelligent Interaction, First International Conference, ACII 2005, Beijing, China, October 22–24, 2005, Proceedings,

Tao,

Tan and

R.W.

Picard, eds, Lecture Notes in Computer Science, Vol. 3784, Springer, 2005, pp. 505–512. doi:10.1007/11573548_65.

59.

Ortony,

G.L.

Clore and

Collins, The Cognitive Structure of Emotions, Cambridge University Press, 1990, http://www.cambridge.org/catalogue/catalogue.asp?isbn=0521386640 .

60.

C.E.

Osgood, The nature and measurement of meaning, Psychological Bulletin 49(3) (1952), 197–237. doi:10.1037/h0055737.

61.

Pianta,

Bentivogli and

Girardi, Developing an aligned multilingual database, in: Proceedings of 1st International Conference on Global WordNet (GWC2002), 2002.

62.

Plutchik, The nature of emotions, American Scientist 89(4) (2001), 344. doi:10.1511/2001.4.344.

63.

Ptaszynski,

Rzepka,

Araki and

Momouchi, Automatically annotating a five-billion-word corpus of Japanese blogs for affect and sentiment analysis, in: Proceedings of the 3rd Workshop in Computational Approaches to Subjectivity and Sentiment Analysis, WASSA@ACL 2012, Jeju Island, Republic of Korea, July 12, 2012,

Balahur,

Montoyo,

Martínez-Barco and

Boldrini, eds, Association for Computational Linguistics, 2012, pp. 89–98. doi:10.1016/j.csl.2013.04.010.

64.

Ptaszynski,

Rzepka,

Araki and

Momouchi, A robust ontology of emotion objects, in: Proceedings of the Eighteenth Annual Meeting of the Association for Natural Language Processing (NLP-2012), 2012, pp. 719–722, http://arakilab.media.eng.hokudai.ac.jp/~araki/2011/2011-D-14.pdf .

65.

Radulovic and

Milikic, Smiley ontology, in: Proceedings of the 1st International Workshop on Social Networks Interoperability (SNI 2009) in Conjuction with the 4th Asian Semantic Web Conference 2009, Shanghai, China, December, 2009, 2009.

66.

Rauschmayer, Linking emotions to needs: A comment on mindsets, rationality and emotion in multi-criteria decision analysis, Journal of Multi-Criteria Decision Analysis 13(4) (2005), 187–190. doi:10.1002/mcda.387.

67.

Revelle and

K.R.

Scherer, Personality and emotion, in: The Oxford Companion to Emotion and the Affective Sciences,

Sander and

K.R.

Scherer, eds, Oxford University Press, 2009, pp. 304–306.

68.

Roussey,

Pinet,

M.A.

Kang and

Corcho, An introduction to ontologies and ontology engineering, in: Ontologies in Urban Development Projects,

Falquet,

Métral,

Teller and

Tweed, eds, Advanced Information and Knowledge Processing, Vol. 1, Springer, London, 2011, pp. 9–38. doi:10.1007/978-0-85729-724-2_2.

69.

J.A.

Russell, A circumplex model of affect, Journal of Personality and Social Psychology 39(6) (1980), 1161–1178. doi:10.1037/h0077714.

70.

R.M.

Ryan and

E.L.

Deci, Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being, American Psychologist 55(1) (2000), 68–78. doi:10.1037/0003-066X.55.1.68.

71.

K.M.

Sam and

C.R.

Chatwin, Ontology-based text-mining model for social network analysis, in: 2012 IEEE 6th International Conference on Management of Innovation & Technology, IEEE, 2012, pp. 226–231. doi:10.1109/ICMIT.2012.6225809.

72.

J.F.

Sánchez-Rada and

C.A.

Iglesias, Onyx: Describing emotions on the Web of Data, in: Proceedings of the First International Workshop on Emotion and Sentiment in Social and Expressive Media: Approaches and Perspectives from AI (ESSEM 2013) a Workshop of the XIII International Conference of the Italian Association for Artificial Intelligence (AI*IA 2013), Turin, Italy, December 3, 2013,

Battaglino,

Bosco,

Cambria,

Damiano,

Patti and

Rosso, eds, CEUR Workshop Proceedings, Vol. 1096, CEUR-WS.org, 2013, pp. 71–82, http://ceur-ws.org/Vol-1096/paper6.pdf .

73.

K.R.

Scherer, Appraisal theory, in: Handbook of Cognition and Emotion,

Dalgleish and

M.J.

Power, eds, John Wiley & Sons, 1999, chapter 30, pp. 637–663. doi:10.1002/0470013494.ch30.

74.

K.R.

Scherer, Psychological models of emotion, in: The Neuropsychology of Emotion,

J.C.

Borod, ed., Series in Affective Science, Oxford University Press, 2000, pp. 137–162.

75.

Schimmack, The structure of subjective well-being, in: The Science of Subjective Well-Being,

Eid and

R.J.

Larsen, eds, The Guildford Press, 2008, pp. 97–123.

76.

Schröder,

Baggia,

Burkhardt,

Pelachaud,

Peter and

Zovato, EmotionML – An upcoming standard for representing emotions and related states, in: Affective Computing and Intelligent Interaction – 4th International Conference, ACII 2011, Memphis, TN, USA, October 9–12, 2011, Proceedings, Part I,

S.K.

D’Mello,

A.C.

Graesser,

B.W.

Schuller and

J.-C.

Martin, eds, Lecture Notes in Computer Science, Vol. 6974, Springer, 2011, pp. 316–325. doi:10.1007/978-3-642-24600-5_35.

77.

Song,

Kim,

Rho and

Hwang, Music ontology for mood and situation reasoning to support music retrieval and recommendation, in: Third International Conference on the Digital Society (ICDS 2009), Cancun, Mexico, February 1–7, 2009, IEEE Computer Society, 2009, pp. 304–309. doi:10.1109/ICDS.2009.50.

78.

Stock and

Cialone, An approach to the management of multiple aligned multilingual ontologies for a geospatial Earth observation system, in: GeoSpatial Semantics – 4th International Conference, GeoS 2011, Brest, France, May 12–13, 2011, Proceedings,

Claramunt,

Levashkin and

Bertolotto, eds, Lecture Notes in Computer Science, Vol. 6631, Springer, 2011, pp. 52–69. doi:10.1007/978-3-642-20630-6_4.

79.

Strapparava and

Valitutti, WordNet Affect: An affective extension of WordNet, in: Proceedings of the Fourth International Conference on Language Resources and Evaluation, LREC 2004, Lisbon, Portugal, May 26–28, 2004, European Language Resources Association, 2004, pp. 26–28, http://www.lrec-conf.org/proceedings/lrec2004/pdf/369.pdf .

80.

M.D.

Sykora,

Jackson,

O’Brien and

Elayan, Emotive ontology: Extracting fine-grained emotions from terse, informal messages, IADIS International Journal on Computer Science and Information Systems 8(2) (2013), 106–118, http://www.iadisportal.org/ijcsis/papers/2013160208.pdf .

81.

Tay and

Diener, Needs and subjective well-being around the world, Journal of Personality and Social Psychology 101(2) (2011), 354–365. doi:10.1037/a0023779.

82.

Thelwall,

Buckley,

Paltoglou,

Cai and

Kappas, Sentiment strength detection in short informal text, Journal of the American Society for Information Science and Technology 61(12) (2010), 2544–2558. doi:10.1002/asi.21416.

83.

Uschold and

Gruninger, Ontologies: Principles, methods and applications, The Knowledge Engineering Review 11(2) (1996), 93–136. doi:10.1017/S0269888900007797.

84.

Watson,

Wiese,

Vaidya and

Tellegen, The two general activation systems of affect: Structural findings, evolutionary considerations, and psychobiological evidence, Journal of Personality and Social Psychology 76(5) (1999), 820–838. doi:10.1037/0022-3514.76.5.820.

85.

C.M.

Whissell, The dictionary of affect in language, in: Emotion: Theory, Research, and Experience, Volume 4: The Measurement of Emotions,

Plutchik and

Kellerman, eds, Academic Press, 1989, chapter 5, pp. 113–131. doi:10.1016/B978-0-12-558704-4.50011-6.

86.

Wright Saunders, Katharine and Isabel: Mother’s Light, Daughter’s Journey, Nicholas Brealey Publishing, 1991, https://books.google.ca/books?id=hYlWAAAAYAAJ .

87.

Yan,

D.B.

Bracewell,

Ren and

Kuroiwa, The creation of a Chinese emotion ontology based on HowNet, Engineering Letters 16(1) (2008), 166–171, http://www.engineeringletters.com/issues_v16/issue_1/EL_16_1_24.pdf .

88.

Zhang,

Hu,

Chen and

Moore, Ontology-based context modeling for emotion recognition in an intelligent web, World Wide Web 16(4) (2013), 497–513. doi:10.1007/s11280-012-0181-5.

89.

Zini and

Sterling, Designing ontologies for agents, in: 1999 Joint Conference on Declarative Programming, AGP’99, L’Aquila, Italy, September 6–9, 1999,

M.C.

Meo and

Vilares Ferro, eds, 1999, pp. 29–42.

Survey on ontologies for affective states and their influences

Abstract

Keywords

1. Introduction

2.1. Affective states

2.3. The relationships between the affective states and their influences

3. Emotion related lexicons and language

4.1. Re-used ontologies

4.2.1. Text domain

4.2.4. Body expressions domain

4.2.5. Contextual environment domain

4.3. Mood ontologies

4.4. Need ontologies

5. Conclusions

References