A Distributed Model of Face and Body Integration

Abstract

Separated face- and body-responsive brain networks have been identified that show strong responses when observers view faces and bodies. It has been proposed that face and body processing may be initially separated in the lateral occipitotemporal cortex and then combined into a whole person representation in the anterior temporal cortex, or elsewhere in the brain. However, in contrast to this proposal, our recent study identified a common coding of face and body orientation (ie, facing direction) in the lateral occipitotemporal cortex, demonstrating an integration of face and body information at an early stage of face and body processing. These results, in combination with findings that show integration of face and body identity in the lateral occipitotemporal, parahippocampal and superior parietal cortex, and face and body emotional expression in the posterior superior temporal sulcus and medial prefrontal cortex, suggest that face and body integration may be more distributed than previously considered. I propose a new model of face and body integration, where areas at the intersection of face- and body-responsive regions play a role in integrating specific properties of faces and bodies, and distributed regions across the brain contribute to high-level, abstract integration of shared face and body properties.

COMMENT ON: Foster C, Zhao M, Bolkart T, Black MJ, Bartels A, Bülthoff I. The neural coding of face and body orientation in occipitotemporal cortex. Neuroimage. 2022 Feb 1;246:118783. doi: 10.1016/j.neuroimage.2021.118783. Epub 2021 Dec 5. PMID: 34879251.

Visual neuroscience has identified networks of brain regions across occipitotemporal cortex that show strong responses to particular stimulus categories, including faces, bodies, scenes and words. ¹ Among these categories, the separation of face- and body-responsive brain regions is somewhat surprising as faces and bodies are part of a single object – a person. Thus, this leads to the question of why separated face- and body-responsive networks exist. One possibility is that separated networks may somehow allow occipitotemporal cortex to optimally perform computations that allow us to encode and extract high-level face and body properties (eg, identities, categories, direction of attention, emotional expressions) from varied low-level visual input. However, as we determine many of these properties using information from both the face and body, we must integrate face and body information somewhere in the brain.

In recent years, several potential models of face and body integration in the brain have been proposed.^2-4 Common to all of these models is a proposal that integration is hierarchical, with faces and bodies initially processed more separately in posterior brain regions and then gradually being integrated in more anterior brain regions. Figure 1 illustrates the locations of the main face-responsive regions (Figure 1: red) and body-responsive regions (Figure 1: blue) in occipitotemporal cortex. One model of face and body integration proposed that faces and bodies are processed in a parts-based manner in the occipital face area (OFA) and extrastriate body area (EBA), as whole faces and bodies in the fusiform face area (FFA) and fusiform body area (FBA), and then as whole persons in the anterior temporal cortex. ² A second model extended this proposal to include some person-responsive voxels in the fusiform gyrus and multimodal person integration in the superior temporal sulcus. ³ Finally, a review paper outlined 4 different models of face and body integration, ranging from face and body networks remaining completely separated in occipitotemporal cortex and integrated elsewhere in the brain, to face and body networks being fully integrated networks with a hierarchical organisation from lateral occipitotemporal cortex through to the anterior temporal cortex. ⁴ Thus, these models show that different possibilities exist for the location of face and body integration in the brain, but it is generally considered that faces and bodies are initially processed more separately, before being integrated and then processed as whole persons.

Figure 1.

Locations of face- and body-responsive regions and coordinates showing integration of face and body properties. Face-responsive regions are shown in red, body-responsive regions are shown in blue and overlapping face- and body-responsive regions are shown in purple. These regions are based on mean coordinates and volumes for the pSTS-body, ⁵ and all other regions. ⁶ Spheres show coordinates of face and body orientation integration in green, ⁶ face and body identity integration in yellow ⁷ and face, body and voice expression integration in cyan. ⁸

Evidence for Early Integration of Face and Body Properties

Our recent study ⁶ suggests that this assumption of hierarchical coding may be incorrect, as we find evidence of integration of face and body information at an early stage of face and body processing in the lateral occipitotemporal cortex. We recorded brain responses using functional magnetic resonance imaging (fMRI) while participants viewed images of faces and bodies from different orientations (ie, facing directions). We then used a multivoxel pattern analysis cross-classification approach to test whether patterns of brain responses distinguishing the different face orientations could generalise to responses evoked by the body orientations, and vice versa. Such generalisation would suggest that these regions encode orientation information using a common coding, and would therefore suggest that there is an integration of orientation information from the face and body. Our whole-brain searchlight results showed that we could decode orientation across face and body stimuli bilaterally in a region at the intersection of the face-responsive OFA and the body responsive EBA (Figure 1: green spheres). Thus, our results demonstrate that face and body orientation information is integrated at a very early stage of face and body processing.

In addition to this early integration of face and body processing, our results also showed that there is further processing of face orientation beyond the lateral occipitotemporal cortex. We found that the FFA encoded face orientation, but not body orientation, which is consistent with previous findings of face-specific orientation coding in the FFA. ⁹ Potentially, such further processing could aid us in determining direction of attention when the head is turned to orient in a different direction on the body. However, further work, testing neural responses to stimuli where the head is turned away from the direction of body orientation, will be required to determine if this is the case.

Findings from other studies are also in accordance with early integration of face and body information. The posterior lateral face patch, as well as the middle lateral face patch, in the macaque monkey showed responses to the position a face would be expected to be located when this face was only implied by contextual cues from the presence of a body. ¹⁰ This suggests that body information is integrated into the responses of this early face-responsive area. Furthermore, effective connectivity of face and body networks determined by microstimulation of face- and body-responsive patches identified some voxels at the intersection of face- and body-responsive patches, including the posterior lateral face patch and the middle superior temporal sulcus body patch, that were activated by both microstimulation of face and body patches. ¹¹ This finding provides further support for integration of face and body information at the intersection of face- and body-responsive regions, including the most posterior regions. Lastly, human transcranial magnetic stimulation (TMS) has also provided evidence for early interactions between face and body processing. TMS applied separately to the right OFA and right EBA impaired both face and body processing at an early time period after stimulus onset. ¹² Thus, this finding suggests there is shared processing of faces and bodies that is disrupted by stimulation of these early face- and body-responsive human brain regions.

Integration of Other Shared Face and Body Properties

Where in the brain are other shared properties of faces and bodies integrated? We tested how shared identity, detected from the face and body shown alone, is encoded in the brain using a similar searchlight cross-classification approach. ⁷ Participants were trained to recognise individuals from images of the face and body, and then their brain activity was recorded using fMRI as they viewed these face and body images. Searchlight results identified patterns of responses that could generalise across face and body identities in several regions, including a region in the right lateral occipitotemporal cortex close to our coordinates that showed face and body orientation integration, but also more dispersed locations including the right parahippocampal cortex and right superior parietal cortex (Figure 1: yellow spheres). Thus, face and body identity integration appears to involve more dispersed brain regions as compared to face and body orientation integration.

Why might identity integration involve dispersed regions? One possibility is that these dispersed regions encode different aspects of face and body identity. Identity coding in the right lateral occipitotemporal cortex might be driven by shared shape properties of face and body identities, and might be orientation-specific. In previous work, we found that we could decode stimulus weight (ie, fatter vs thinner) across face and body stimuli from the EBA, ¹³ suggesting there may be a shared coding of this category in lateral occipitotemporal cortex that could be driven by shared shape-related aspects or abstract coding of these stimuli. Stimuli in this study were all shown in the same orientation, and although stimuli in our identity decoding were from different orientations, it is possible that successful decoding across face and body stimuli could be driven by successful decoding within the same orientation. Orientation-specific decoding in lateral occipitotemporal cortex would be consistent with our orientation decoding results, ⁶ as well as a general model that proposes that posterior category-responsive regions encode view-specific representations. ¹⁴ In contrast to view-specific coding in the lateral occipitotemporal cortex, previous work has identified view-invariant coding of face identity in the fusiform and anterior temporal cortex,^15,16 and we also identified view-invariant coding of body identity in these regions. ⁷ Thus, potentially the right parahippocampal cortex, located in the anterior temporal cortex, may encode identity in a more abstract, view-invariant manner. Similarly, the intraparietal sulcus, close to our coordinates in the superior parietal cortex, has been associated with abstract encoding of face and object identity, ¹⁷ suggesting that face and body identity may also be encoded in an abstract manner in the parietal cortex. Altogether, these insights suggest that different aspects of identity-related information may be encoded in different regions, but further research is required to understand the exact coding used by these distributed regions.

A third shared face and body property, for which shared coding has been tested, is emotional expression. Participants viewed movies of faces and bodies (shown separately) displaying emotional expressions, as well as voices expressing the same emotions. ⁸ Peelen et al ⁸ then performed a searchlight analysis to identify brain regions that encoded the perceived emotion (eg, happiness, sadness, anger) across face, body and voice stimuli. They identified 2 regions showing abstract integration of emotional expressions, the left posterior superior temporal sulcus (pSTS) and the right medial prefrontal cortex (Figure 1: cyan spheres). Notably, the coordinates of expression integration in the pSTS are in-between the location of the pSTS-face region, a region known be involved in encoding facial expressions, ¹⁸ and the body-responsive EBA. As decoding of emotional expression was across voice stimuli, in addition to face and body stimuli, it is possible that the left hemisphere specificity of the pSTS results may have been influenced by language lateralisation in the left hemisphere. Decoding of emotional expression across stimuli from the medial prefrontal cortex shows that emotional expression integration, like identity integration, occurs across distributed brain regions. Emotional expression coding in the medial prefrontal cortex has been shown to be highly abstract, as coding has also been shown to generalise to emotions that are inferred by situation context. ¹⁹ Altogether, these results show distributed coding of emotional expression across the brain, and some evidence that different aspects of emotional expressions may be encoded in these different regions.

A Distributed Model of Face and Body Integration

The experimental findings outlined above demonstrate that several regions are involved in face and body integration, and that different brain regions are involved in the integration of different face and body properties (eg, orientations, identities, emotional expressions). In Figure 2, I give an overview of the brain regions in occipitotemporal cortex that show integration of shared face and body properties, in relation to the known functional properties of face- and body-responsive occipitotemporal regions.^18,20 This model highlights the importance of areas at the intersection of face- and body-responsive regions, with the intersection of the OFA and EBA involved in orientation and shape integration, and the intersection of the EBA and pSTS-face in expression integration. Thus, intersecting areas appear to integrate specific face and body properties that may first be processed separately for faces and bodies in the 2 nearby face- and body-responsive regions. Future studies could investigate this proposal by testing whether neuropsychological patients with damage to these intersecting regions show deficits in the integration of specific face and body properties, or whether TMS to these intersecting regions causes disruptions in behavioural tasks that require an integration of face and body processing.

Figure 2.

An overview model of the brain regions involved in visual processing and integration of faces and bodies in occipitotemporal cortex, including their known functional response properties. Face-responsive brain regions are shown in red, body-responsive regions are shown in blue and regions showing integration of face and body properties are shown in purple.

Further distributed regions, beyond the occipitotemporal regions outlined in Figure 2, have also been found to integrate face and body properties, and may do so in a high-level abstract manner. In the studies outlined above, the right superior parietal cortex was found to encode face and body identity information, and the right medial prefrontal cortex was found to encode face, body and voice emotional expression information,^7,8 and both of these regions have been also associated with abstract coding of these respective properties.^17,19 Moreover, it is likely that there may be further regions, both within and outside of occipitotemporal cortex, that are involved in the integration of other shared face and body properties that have not yet been tested experimentally. In summary, the proposed model outlines a new perspective on face and body integration that is distributed and property-specific, but further research is needed to uncover the full extent of brain regions involved in face and body integration, and the specific coding of information in these regions.