The principle of compositionality, an important postulation in language and cognition research, posits that the meaning of a complex expression is determined by the meaning of its constituting parts and the operation performed on those parts. Here, we provide strong evidence that this principle plays a significant role also in interpreting facial expressions. In three studies in which perceivers interpreted sequences of two emotional facial expression images, we show that the composite meaning of facial expressions results from the meaning of its constituting expressions and an algebraic operation performed on those expressions. Our study offers a systematic account as to how the meaning of facial expressions (single and sequences) are being formed and perceived. In a broader context, our results raise the possibility that the principle of compositionality may apply to human communication modalities beyond spoken language, whereby a minimal number of components are expanded to a much greater number of meanings.
AmbadarZ.SchoolerJ. W.CohnJ. F. (2005). Deciphering the enigmatic face: The importance of facial dynamics in interpreting subtle facial expressions. Psychological Science, 16, 403–410. https://doi.org/10.1111/j.0956-7976.2005.01548.x
2.
BarsalouL. W. (2017). Cognitively plausible theories of concept composition. In Compositionality and concepts in linguistics and psychology (pp. 9–30). Cham: Springer International Publishing.
3.
DeBruineL. M.JonesB. C.UngerL.LittleA. C.FeinbergD. R. (2007). Dissociating averageness and attractiveness: Attractive faces are not always average. Journal of Experimental Psychology: Human Perception and Performance, 33, 1420. https://doi.org/10.1037/0096-1523.33.6.1420
4.
DuS.TaoY.MartinezA. M. (2014). Compound facial expressions of emotion. Proceedings of the National Academy of Sciences, 111(5), E1454–E1462. https://doi.org/10.1073/pnas.1322355111
5.
DubéS. P. (1997). Visual bases for the perception of facial expressions: A look at some dynamic aspects. Concordia University.
6.
DummettM. (1981). Frege: Philosophy of language. Harvard University Press.
EkmanP.FriesenW. V. (1969). The repertoire of nonverbal behavior: Categories, origins, usage, and coding. Semiotica, 1, 49–98. https://doi.org/10.1515/semi.1969.1.1.49
9.
EkmanP.FriesenW. V. (1971). Constants across cultures in the face and emotion. Journal of Personality and Social Psychology, 17, 124. https://doi.org/10.1037/h0030377
10.
EkmanP.FriesenW. V. (1976). Measuring facial movement. Environmental Psychology and Nonverbal Behavior, 1, 56–75. https://doi.org/10.1007/BF01115465
11.
FischerJ.WhitneyD. (2014). Serial dependence in visual perception. Nature Neuroscience, 17, 738–743. https://doi.org/10.1038/nn.3689
12.
FredricksonB. L. (1991). Anticipated endings: An explanation for selective social interaction. ProQuest Information & Learning.
13.
FredricksonB. L. (2000). Extracting meaning from past affective experiences: The importance of peaks, ends, and specific emotions. Cognition & Emotion, 14, 577–606. https://doi.org/10.1080/026999300402808
14.
FredricksonB. L.KahnemanD. (1993). Duration neglect in retrospective evaluations of affective episodes. Journal of Personality and Social Psychology, 65, 45. https://doi.org/10.1037/0022-3514.65.1.45
15.
GiesbrechtE. (2009, July). In search of semantic compositionality in vector spaces. In International Conference on Conceptual Structures (pp. 173–184). Springer, Berlin: Heidelberg.
16.
HabermanJ.HarpT.WhitneyD. (2009). Averaging facial expression over time. Journal of Vision, 9, 1–1. https://doi.org/10.1167/9.11.1
17.
HamptonJ. (1997). Conceptual combination. In LambertsK.ShanksD. R. (Eds.), Knowledge, concepts and categories (pp. 133–159). The MIT Press.
18.
HirstG. (1992). Semantic interpretation and the resolution of ambiguity. Cambridge University Press.
19.
HoeksemaJ. (2000). Compositionality of meaning. Morphology: A Handbook on Inflection and Word-Formation. Walter de Gruyter, Berlin, 851–857.
20.
HolmS. (1979). A simple sequentially rejective multiple test Procedure. scandinavian Journal of Statistics, 6(2), 65–70. http://www.jstor.org/stable/4615733
21.
IzardC. E.HaynesO. M. (1988). On the form and universality of the contempt expression: A challenge to Ekman and Friesen’s claim of discovery. Motivation and Emotion, 12, 1–16. https://doi.org/10.1007/BF00992469
22.
JackR. E.GarrodO. G.SchynsP. G. (2014). Dynamic facial expressions of emotion transmit an evolving hierarchy of signals over time. Current Biology, 24, 187–192. https://doi.org/10.1016/j.cub.2013.11.064
23.
JackR. E.SunW.DelisI.GarrodO. G.SchynsP. G. (2016). Four not six: revealing culturally common facial expressions of emotion. Journal of Experimental Psychology: General, 145, 708. https://doi.org/10.1037/xge0000162
24.
JanssenT. M. V. (2001). Frege, contextuality and compositionality. Journal of Logic, Language and Information, 10(1), 115–136. https://doi.org/10.1023/A:1026542332224
25.
KahnemanD. (2000). Experienced utility and objective happiness: A moment-based approach. In KahnemanD.TverskyA. (Eds.), Choices, values and frames (pp. 673–692). New York: Cambridge University Press.
26.
KamachiM.BruceV.MukaidaS.GyobaJ.YoshikawaS.AkamatsuS. (2013). Dynamic properties influence the perception of facial expressions. Perception, 42, 1266–1278. https://doi.org/10.1068/p3131n
27.
KeltnerD.CordaroD. T. (2017). Understanding multimodal emotional expressions: recent advances in basic emotino theory. In The science of facial expression (pp. 57--76).
28.
LabrooA. A.DharR.SchwarzN. (2008). Of frog wines and frowning watches: semantic priming, perceptual fluency, and brand evaluation. Journal of consumer research, 34, 819–831. https://doi.org/10.1086/523290
29.
LangnerO.DotschR.BijlstraG.WigboldusD. H.HawkS. T.van KnippenbergA. (2010). Presentation and validation of the Radboud faces database. Cognition and Emotion, 24, 1377–1388. https://doi.org/10.1080/02699930903485076
MikolovT.ChenK.CorradoG.DeanJ. (2013). Efficient estimation of word representations in vector space. CoRR abs/1301.3781. http://arxiv.org/abs/1301.3781.
PenningtonJ.SocherR.ManningC. D. (2014, October). Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) (pp. 1532–1543).
36.
PerrettD.LeeK.Penton-VoakI.RowlandD.YoshikawaS.BurtD., …, AkamatsuS. (1998). Effects of sexual dimorphism on facial attractiveness. Nature, 394(6696), 884–887. https://doi.org/10.1038/29772
37.
PerrettD.MayK.YoshikawaS. (1994). Facial shape and judgements of female attractiveness. Nature, 368(6468), 239–242. https://doi.org/10.1038/368239a0
38.
ReinlM.BartelsA. (2015). Perception of temporal asymmetries in dynamic facial expressions. Frontiers in psychology, 6, 1107.
39.
RhodesG.HickfordC.JefferyL. (2000). Sex-typicality and attractiveness: Are supermale and superfemale faces super-attractive?British Journal of Psychology, 91, 125–140. https://doi.org/10.1348/000712600161718
RhodesG.LeeK.PalermoR.WeissM.YoshikawaS.ClissaP.JefferyL. (2005). Attractiveness of own-race, other-race, and mixed-race faces. Perception, 34(3), 319--340.
42.
RussellJ. A. (1991). The contempt expression and the relativity thesis. Motivation and Emotion, 15, 149–168. https://doi.org/10.1007/BF00995675
43.
RussellJ. A.FehrB. (1987). Relativity in the perception of emotion in facial expressions. Journal of Experimental Psychology: General, 116, 223. https://doi.org/10.1037/0096-3445.116.3.223
44.
SchererK. R. (1984). On the nature and function of emotion: A component process approach. Approaches to emotion, 2293(317), 31.
45.
SchererK. R.EllgringH. (2007). Are facial expressions of emotion produced by categorical affect programs or dynamically driven by appraisal?Emotion (Washington, D.C.), 7, 113. https://doi.org/10.1037/1528-3542.7.1.113
46.
SmithB. (1987). “Philosophy 266: Representation, formality, and the foundations of computation in Pelletier F.J, 1994, The principle of semantic compositionality.
47.
SoferC.DotschR.OikawaM.OikawaH.WigboldusD. H.& TodorovA. (2017). For your local eyes only: culture-specific face typicality influences perceptions of trustworthiness. Perception, 46, 914–928. https://doi.org/10.1177/0301006617691786
48.
SoferC.DotschR.WigboldusD. H.& TodorovA. (2015). What is typical is good the influence of face typicality on perceived trustworthiness. Psychological Science, 26, 39–47. https://doi.org/10.1177/0956797614554955
49.
TanakaJ.GilesM.KremenS.SimonV. (1998). Mapping attractor fields in face space: The atypicality bias in face recognition. Cognition, 68, 199–220. https://doi.org/10.1016/S0010-0277(98)00048-1
50.
ThayerS. (1980). The effect of facial expression sequence upon judgments of emotion. Journal of Social Psychology, 111(2), 305.
51.
TiddemanB.BurtM.PerrettD. (2001). Psychmorph version 5; prototyping and transforming facial textures for perception research. Computer Graphics and Applications, IEEE, 21, 42–50. https://doi.org/10.1109/38.946630
52.
VareyC.KahnemanD. (1992). Experiences extended across time: evaluation of moments and episodes. Journal of Behavioral Decision Making, 5, 169–185. https://doi.org/10.1002/bdm.3960050303
53.
WebsterM. A.MacLinO. H. (1999). Figural aftereffects in the perception of faces. Psychonomic Bulletin & Review, 6, 647–653. https://doi.org/10.3758/BF03212974
54.
YeomansK. A.GolderP. A. (1982). The Guttman-Kaiser Criterion as a Predictor of the Number of Common Factors. Journal of the Royal Statistical Society: Series D (The Statistician), 31(3), 221--229.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
