A unified valence scale based on diagnosis of facial expressions

Abstract

Affect-adaptive systems detect the emotional user state, assess it against the current situation, and adjust interaction accordingly. Tools for real-time emotional state detection, like the Emotient FACET engine (Littlewort et al., 2011), are based on the analysis of facial expressions. When developing affect-adaptive systems, output from the diagnostic engine must be mapped onto theoretical models of emotion. The Circumplex Model of Affect (Russell, 1980) describes emotion on two dimensions: valence and arousal. However, FACET offers three classifiers for valence: positive, neutral, and negative valence. The present study aimed at developing an algorithm that converts these into a unified valence scale. We used FACET to analyze valence-labeled images from the AffectNet database. In a multiple regression analysis, FACET classifier values predicted database valence and explained 38% of the variance. By inserting classifier values into the regression equation, a unified valence scale can be calculated that matches dimensional models of emotion. This research forms the groundwork for adaptation of the emotional user state based on the FACET engine. A future affect-adaptive system can now use the FACET engine to detect the emotional user state on a unified valence dimension, which allows for distinct classification and interpretation of emotions.

Get full access to this article

View all access options for this article.

References

Bernin

Müller

Ghose

Luck

von Grecos

(2017). Towards More Robust Automatic Facial Expression Recognition in Smart Environments. Proceedings of the 10th International Conference on Pervasive Technologies Related to Assistive Environments -PETRA ‘17, Article https://doi.org/10.1145/3056540.3056546, 37–44.

Cohen

(1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.). Lawrence Erlbaum Associates.

Dupré

Krumhuber

E. G.

Küster

McKeown

G. J.

(2020). A performance comparison of eight commercially available automatic classifiers for facial affect recognition. PloS One, 15(4), e0231968. https://doi.org/10.1371/journal.pone.0231968

Egger

Ley

Hanke

(2019). Emotion Recognition from Physiological Signal Analysis: A Review. Electronic Notes in Theoretical Computer Science, 343, 35–55. https://doi.org/10.1016/j.entcs.2019.04.009

Ekman

Friesen

W. V.

(1978). PsycTESTS Dataset. American Psychological Association (APA). https://doi.org/10.1037/t27734-000

Fuchs

Hochgeschurz

Schmitz-Hübsch

Thiele

(2020). Adapting Interaction to Address Critical User States of High Workload and Incorrect Attentional Focus – An Evaluation of Five Adaptation Strategies. International Conference on Human-Computer Interaction, 335–352. https://doi.org/10.1007/978-3-030-50439-7_23

Gasper

Clore

(2002). Attending to the big picture: Mood and global versus local processing of visual information. Psychological Science, 13(1), 34–40. https://doi.org/10.1111/1467-9280.00406

Hemmerich

(2022). Multiple Lineare Regression: Voraussetzungen. https://statistikguru.de/spss/multiple-lineareregression/voraussetzungen.html

Hudlicka

(2003). To feel or not to feel: The role of affect in human–computer interaction. International Journal of Human-Computer Studies, 59(1-2), 1–32. https://doi.org/10.1016/S1071-5819(03)00047-8

10.

iMotions Biometric Research Platform (Version 8.2) [Computer software]. (2021). IMotions. Copenhagen, Denmark. https://imotions.com/

11.

Jones

C. M.

Jonsson

I. M.

(2005). Automatic recognition of affective cues in the speech of car drivers to allow appropriate responses. Proceedings of the 17th Australia Conference on Computer-Human Interaction: Citizens Online: Considerations for Today and the Future, 1–10.

12.

Littlewort

Whitehill

Fasel

Frank

Movellan

Bartlett

(2011). The computer expression recognition toolbox (CERT). Face and Gesture, 298–305. https://doi.org/10.1109/FG.2011.5771414

13.

Mollahosseini

Hasani

Mahoor

M. H.

(2017). Affectnet: A database for facial expression, valence, and arousal computing in the wild. IEEE Transactions on Affective Computing(10), Article 1, 18–31.

14.

Russell

J. A.

(1980). A circumplex model of affect. Journal of Personality and Social Psychology, 39(6), 1161– 1178. https://doi.org/10.1037/h0077714

15.

Stöckli

Schulte-Mecklenbeck

Borer

Samson

A. C.

(2018). Facial expression analysis with AFFDEX and FACET: A validation study. Behavior Research Methods(50), Article 4, 1446– 1460.

16.

Tijs

Brokken

IJsselsteijn

(2008). Creating an Emotionally Adaptive Game. In Stevens

S. M.

Saldamarco

S.J.

(Eds.), Lecture Notes in Computer Science: Vol. 5309. Entertainment Computing -ICEC 2008 (Vol. 5309, pp. 122–133). Springer Berlin Heidelberg. https://doi.org/10.1007/978-3-54089222-9_14

17.

Wickens

C. D.

(2005). Attentional tunneling and task management. Proceedings of the 13th International Symposium on Aviation Psychology, 620–625.

18.

Woolf

B. P.

Burleson

Arroyo

Dragon

Cooper

Picard

R. W.

(2009). Affect-aware tutors: Recognising and responding to student affect. International Journal of Learning Technology(4), Article 3-4, 129–164.