Sage Journals: Discover world-class research

Abstract

Autism is a neurodevelopmental condition that impacts individuals’ communication and social interaction skills. Autistic children often have smaller N170 amplitudes in response to faces than neurotypical children. Autistic children also avoid the salient areas of the face. Technology-based interventions have been developed to teach autistic children how to recognise facial expressions, but the results have exhibited considerable variability across studies. The current study explored the effectiveness of an iPad app designed to support autistic children in recognising facial expressions by examining how participants process facial information through event-related potentials (ERP) and eye-tracking recordings. ERPs and eye tracking were recorded from 20 neurotypical and 15 autistic children aged between 6 and 12 years. The results replicated previous work, with the autistic group having smaller N170 and Vertex Positive Potential amplitudes and more scan time off the face when compared to non-autistic children. Following the intervention, some changes were observed in facial feature scanning among autistic participants, characterised by increased time spent on the face and decreased fixations. These findings add to the work, indicating that eye tracking may be a valuable biomarker for intervention outcomes in autism. Further research into N170 as a biomarker is needed.

Keywords

autism facial expressions eye tracking vertex positive potential N170

Introduction

Autism spectrum disorder (autism, hereafter) is a neurodevelopmental condition characterised by challenges with social communication and interaction, restricted interests, and repetitive behaviours.^1–3 Differences in facial emotion recognition and processing, often linked to social communication challenges, have been proposed to stem from altered attention to facial features, particularly the eye region, among autistic individuals.^4–9 Researchers employ diverse approaches to investigate how individuals perceive and process facial emotions. Electroencephalographic (EEG) techniques, such as event-related potential (ERP) recordings associated with facial feature processing, focusing on ERPs such as the N170 and Vertex Positive Potential (VPP)^9–12 while eye-tracking studies record gaze patterns during facial stimuli viewing.^4,8,13–16

Studies combining EEG and eye-tracking paradigms have sought to examine how attentional and behavioural processes correlate to one another across tasks. For example, Wagner, Hirsch, Vogel-Farley, Redcay and Nelson⁷ identified an atypical processing pattern from face-specific ERPs to faces and gaze behaviour in autistic versus neurotypical adolescents. Additionally, some studies concurrently record ERP and eye-tracking data to investigate both overt and covert attention while viewing faces.^17–19 These studies highlight the objective and reliable nature of combining the measures.^17,19 However, there is still little research which uses these methods to explore intervention response with autistic populations.^20,21 Integrating eye-tracking and EEG within a translational cognitive neuroscience framework is a scientifically novel methodology recommended for understanding the neural and gaze patterns associated with behaviours and measure intervention effectiveness.²⁰ The following study explored the effectiveness of an intervention task designed to improve facial emotion recognition through this innovative framework.

Teach Yourself to Learn Emotions (TYLES)

The TYLES task is described in detail elsewhere²² thus, it will only be briefly summarised here. TYLES was designed as a game-like iPad app, aiming to gradually teach participants to remove tiles from the most salient regions, as this offers the best strategy for recognising expressed emotions.²³ Participants strategically revealed facial regions, primarily the eyes and mouth, by removing tiles one at a time. Their goal was to correctly identify emotions (eg, happy, sad, neutral, surprise, anger, disgust) while minimising the number of tiles removed. Following the successful identification of an emotion, participants received positive feedback.

We hypothesised that the N170 and VPP amplitudes would be smaller in the autistic group and that they increase following intervention. In addition, we predicted that autistic participants would spend more time looking at the salient areas of the face after the intervention. Finally, it was hypothesised that the increased time spent visually scanning the more salient areas of a face correlates with the amplitude increase of the N170 and VPP following the intervention.

Methods

Participants

Participants were recruited through social media advertising, local independent school newsletters, and allied health clinics in the Hunter New England Region of New South Wales, Australia. To be included in the study, participants had to be between 6 and 12 years old and have normal vision. The autistic participants had their diagnosis confirmed by their external clinician. Participants were excluded if they had a history of neurological or endocrine conditions, wore glasses or contact lenses, or had experienced a traumatic brain injury with sustained unconsciousness. Written informed consent to participate in this study was provided by the participants’ legal guardians (Table 1).

Table 1.

Sample Demographics.

	Autistic	Non-Autistic	Statistical Comparison
Biological Sex			X² (1, 35) = 14.04, p < .001
Female	1	14
Male	14	6
Ethnicity/Race			X² (1, 35) = 3.48, p = .18
Asian	0	1
Indigenous Australian and/or Torres Strait Islander	2	0
White/Caucasian	13	19

A total of 35 participants were included in the study: 20 neurotypical (non-autistic) and 15 autistic children. The average age was 8.40 years (SD = 1.91). In the autistic group, the average age was 9.00 years (SD = 1.81), while in the non-autistic group, it was 7.95 (SD = 1.90). There was no significant difference in age between groups (U = 104.00, p = .13). Among the autistic participants, eight reported having co-occurring diagnoses; seven were diagnosed with Attention-Deficit Hyperactivity Disorder, and three were diagnosed with Generalised Anxiety Disorder. Three participants had additional diagnoses of Sensory Processing Disorder, Oppositional Defiance Disorder, Obsessive-Compulsive Disorder, Intellectual Disability and Post-Traumatic Stress Disorder. Among the non-autistic participants, two were diagnosed with physical conditions: one with hypermobility and one with Henoch Schoenlein Nephritis/Nephrotic Syndrome.

Stimuli

Electroencephalogram and Eye-Tracking

The facial stimuli for the EEG and eye-tracking comprised 60 colour images of children aged between two and eight years, selected from the from the Child Affective Facial Expression (CAFÉ) set by LoBue and Thrasher.²⁴ To ensure high-quality stimuli, images were chosen based on a documented high inter-rater agreement (81% to 98%) for expression categorisation within the CAFÉ database.²⁴ Each facial expression (neutral, happy, angry) was equally represented, featuring male and female subjects.

In addition to facial expressions, object images were included, with an equal distribution of cars, trees, and houses sourced from the Shutterstock website. These images underwent digital adjustments to maintain consistent size, brightness, and contrast. The 120 total trials for faces and objects represented a compromise to keep the recording time short and to not overburden the participants, potentially risking non-compliance. The stimuli presentation for both EEG and eye-tracking was developed using the SR Research Experiment Builder software.²⁵

Procedure

All autistic participants underwent two sessions of EEG and eye-tracking recordings, before (Time 1) and after two-weeks of the TYLES intervention (Time 2). Participants were asked to use TYLES daily for up to 30 min. Further details about the intervention and user engagement are published elsewhere.²²

Neurotypical children, which were being used as a baseline, only participated in one session of EEG and eye-tracking recordings, and did not receive the intervention. Each recording lasted one hour, with short breaks as required and a snack break between EEG and eye-tracking sessions. The order of EEG and eye-tracking recordings for autistic children was counterbalanced.

Electroencephalogram

EEG recordings were conducted on a laptop computer (Dell Latitude E5570 Intel(R) Core ™ i7-6820HQ CPU @ 2.70 GHz, 2701 Mhz) using the Neuroscan ACQUIRE (Version 4.5.1) software under dimmed lights while the participants were comfortably seated facing the computer monitor (27-inch, screen resolution 2560 × 1440, refresh rate 59 Hz of an Intel(R) Core™ i7-9700 CPU @ 3.00 Ghz 3.00 Ghz Dell computer) at an approximate distance of 60 cm. Participants’ head circumferences were measured, and they were fitted with a 32-channel QuikCap either small or medium in size²⁶; amplified using a SynAmps2 amplifier. The nose served as reference, and the mastoids as active channels. Vertical and horizontal electrodes were placed above and below the left eye and approximately 1 cm aside from each eye to record the eye blinks and saccades. According to the manufacturer specifications, electrodes were filled with recording gel using a blunt needle to maintain impedances below 5 kΩ.

Throughout the EEG recording, participants were instructed to sit still and attentive, focusing on the screen. The recording consisted of a 500 millisecond (ms) stimulus presentation followed by a 100 ms blank screen interval. There were two blocks of recordings, which were approximately seven minutes and 20 s long each. EEGs were continuously recorded at a 1000 Hz sampling rate, filtered through a 30 Hz low pass filter and DC high pass filter. All 32 channels were recorded with a laptop computer, while stimuli were generated by a second laptop computer. Baseline correction was performed by subtracting the average of the pre-stimulus recording 100 milliseconds before the stimulus presentation.

Eye-Tracking

The display screen (27-inch, screen resolution 2560 × 1440, refresh rate 59 Hz of an intel(R) core (TM) i7-9700 CPU @ 3.00 Ghz, 3.00 Ghz Dell computer) presented the facial stimuli for visual scan path recording and the data was observed concurrently on a separate monitor. The participants were comfortably seated in an adjustable chair, 60 cm from the screen, with a horizontal visual field of 29° and vertical of 22.15°. An SR Eye Link camera was centred below the monitor using a desktop mount, configured in remote viewing mode, and tracking the right eye. A ‘tracking’ sticker was placed in the middle of the participant's forehead ensured alignment and permitted the exclusion of a headrest. Each participant went through calibration and validation procedures to confirm accurate eye fixation position and valid eye movement tracking.

Once calibration and validation were confirmed, the visual scanning task was presented. Stimuli were presented for a 6000 milli-second duration each, aligned with procedures previously employed by McCabe, Rich, Loughland, Schall and Campbell²⁷ and Campbell, McCabe, Leadbeater, Schall, Loughland and Rich.²⁸ The order of emotional expressions was pseudo-randomized to prevent immediate repetition and to ensure that participants did not see the same emotion displayed on a face consecutively (eg, angry followed by angry). There was also an equal presentation of male and female faces. Following each stimulus, a fixation cross was presented in the middle of the screen to maintain calibration until the camera detected a valid visual gaze threshold. Once the visual gaze was confirmed, the next image was manually presented. If issues arose with the eye-tracking integrity, recalibration occurred, and the task resumed from the previous stopping point.

Data Analyses

Event Related Potential (Pre-Processing)

Artefact rejection exceeding ±100 uV was conducted as a first pass for extreme outliers, followed by the ICA-based ocular artefact removal using Neuroscan EDIT Version 4.5.1.²⁶ The ERP analysis software guided semi-automated ocular artifact correction within the parameters 400 ms at a capacity of 10% over threshold after manual review of 20 adequate blink responses. Data were epoched by stimulus (faces and objects using the period of −100 ms-500 ms) and baseline corrected from −100 ms-0 ms. Mean amplitude calculations for N170 and VPP were performed for the 150 ms–180 ms post-stimulus onset interval. Each file was grand averaged by condition (ie, face or object). A difference wave function (face-object) was performed to represent the net effect of face versus object responses.

Data from two autistic participants were excluded during the artifact rejection process due to significant interference. However, one of the excluded autistic participant's post-intervention data was included where usable. Additionally, one neurotypical participant's dataset was lost during the data collection process. Homogeneity of variance between both studied groups was confirmed by conducting the Levene's test in SPSS Version 27.²⁹ Difference scores between the object and face grand-averaged amplitudes (object – face) for the P7/P8 (N170) and Cz (VPP) electrodes were created.

Visual Scanning (Pre-Processing)

Eye-tracking recordings were prepared using SR Research Data Viewer software. Firstly, the areas of interest (AOI) were defined, including the face, mouth, left eye, right eye, nose, whole face, and whole picture. The most salient areas (left eye, right eye, mouth, and nose) were summed to create a new variable (IA_Feature). The proportional fixation time for each AOI was calculated by dividing the looking time on each trial by the total looking time on the whole picture.²⁸

The data output reports were exported from SR Research Data Viewer and data quality checks for missing fields, duplicates, and formatting issues were conducted. A technical glitch during recording led to the exclusion of two participants’ baseline data: one non-autistic and one autistic participant. In addition, one non-autistic participant's data was lost. No data was excluded based on time off face due to the small sample size and known challenges autistic individuals have with looking at faces.

Combined Analyses

To test the hypothesis, combined analyses were conducted using the ERP and eye-tracking data. Difference scores were calculated for the ERP and eye tracking data, separately, by subtracting Time 1 results from Time 2 results. Spearman's Rank Correlation was conducted to examine the relationships between measures.

Statistical Analyses

All statistical analyses were performed using SPSS Version 27.²⁹ The viewing tasks primarily violated normality with skew, and the Levene Test showed that the variances for most created interest areas (eyes, mouth, all features [eyes, mouth, and nose], whole picture) were not equal, except for the whole face and nose. Table 2 below shows the Levene statistic for each interest area.

Table 2.

Levene Statistic for Homogeneity of Variances.

	Levene Statistic	p
Individual Interest Areas
Eyes	5.97 (1, 2818)	<.001
Mouth	92.76 (1, 2771)	<.001
Nose	1.18 (1, 2818)	.23
Whole Face	0.61 (1, 2818)	.44
All Features	14.08 (1, 2818)	<.001
Whole Picture	21.48 (1, 2818)	<.001

Between-group comparisons were analysed using non-parametric Mann-Whitney U and Chi-Square Tests. Paired Sample T-Tests were used when assumptions of normality were met; otherwise, Wilcoxon Signed Rank Tests were performed. The alpha level was set at 0.05 two-sided for all analyses.

Results

Combined Analyses

The Spearman Rank Correlation did not confirm a significant association between the N170 (P8/P7) or VPP (Cz) ERPs and fixation times on the salient facial areas (eyes or mouth).

Event Related Potential

Analyses were conducted to explore variations between the non-autistic and autistic (Time 1) groups. The VPP ERP amplitude in response to face stimuli was smaller in the autistic group than in the non-autistic group at Cz (F(1, 30) = 4.45, p = .04; Cohen's f = 0.33). No other differences between groups were found for objects or faces at the P7 or P8 (N170) electrodes. Figure 1 highlights the differences between groups at the Cz (VPP) electrode.

Figure 1.

Amplitudes at Cz for the Vertex Positive Potential (VPP) of Autistic (ASD; Time 1) and Non-Autistic (TD) Groups in Response to Objects and Faces Stimuli.

Further analyses were conducted to explore variations between the non-autistic and autistic (Time 1) groups using the difference scores (faces - object). Significant differences between groups were found at the P7 (N170) electrode (F(1, 30) = 4.69, p = .04; Cohen's f = 0.34) and Cz (VPP) electrode (F(1, 30) = 4.77, p = .04; Cohen's f = 0.34). Autistic participants had smaller negative difference scores at Cz (VPP) compared to the non-autistic group, but larger positive difference scores at P7 (N170). Figure 2 below displays the significant difference between non-autistic (TD) and autistic (ASD) at P7 and P8. There was no significant group difference at the P8 (N170) electrode.

Figure 2.

N170 ERPs for P7 and P8 Comparing Non-Autistic (TD) and Autistic Groups (ASD; Time 1).

EEG data were available from 10 autistic children after completing the intervention. Non-completion was due to not attending Time 2 assessment, refusal to wear the EEG cap or excessive fear when applying the recording gel with a blunted needle into the electrode discs on the EEG cap. Paired sample t-tests were conducted to compare the difference scores. There was no significant difference between Time 1 and Time 2 amplitudes for any electrode. Figure 3 shows the non-significant differences between Time 1 and Time 2 for the VPP.

Figure 3.

Differences From Time 1 to Time 2 Across Faces and Objects for the VPP Showed as Amplitude at Cz (ms).

Eye-Tracking

Table 3 displays the Median, and Interquartile range between groups’ fixation durations. The non-autistic group had significantly longer fixation durations on the eyes compared to the autistic group (U = 416 601.50, p < .001), mouth (U = 385 019.00, p < .001), and whole face (U = 433 332.50, p < .05), whereas the autistic group had significantly longer fixation durations off-face (U = 392 005.50, p < .001).

Table 3.

Fixation Duration (ms) Across Interest Areas and Groups.

	Median	Interquartile Range
Individual Interest Areas
Eyes
Autistic	803.00	1551.00
Non-Autistic	1079.00	1787.25
Mouth
Autistic	0.00	816.00
Non-Autistic	406.00	1220.00
Nose
Autistic	362.00	772.00
Non-Autistic	366.00	809.75
Whole Face
Autistic	1105.00	742.75
Non-Autistic	1172.50	1442.50
All Features
Autistic	2029.75	2635.25
Non-Autistic	2764.50	2321.50
Off Face
Autistic	360.00	1137.75
Non-Autistic	173.00	815.50

Note. Pre-Intervention (Time 1) Autistic Participants versus Non-Autistic Participants.

Furthermore, significant between-group differences were found for total fixations on the mouth and off-face. The autistic group had a higher number of fixations on the mouth (U = 385 019.00, p < .001) and off-face (U = 392 005.50, p < .001) than the non-autistic group. In addition, the autistic group made more total fixations overall than the non-autistic group as indicated by the total fixations (U = 405 164.50, p < .001). Table 4 presents the Median, and Interquartile range for the number of fixations on each facial area.

Table 4.

Number of Fixations on Interest Areas Between Groups.

	Median	Interquartile Range
Total Fixations
Autistic	12.00	7.00
Non-Autistic	11.00	6.00
Eyes
Autistic	2.00	3.00
Non-Autistic	2.00	3.00
Mouth
Autistic	1.00	2.00
Non-Autistic	1.00	2.00
Nose
Autistic	1.00	2.00
Non-Autistic	1.00	2.00
Whole Face
Autistic	4.00	3.00
Non-Autistic	3.00	3.00
All Features
Autistic	5.00	5.00
Non-Autistic	5.00	4.00
Off Face
Autistic	1.00	4.00
Non-Autistic	1.00	2.00

Note. Pre-Intervention (Time 1) Autistic Participants versus Non-Autistic Participants.

We then explored fixation duration differences between Time 1 and Time 2 for the autistic group. Time spent fixating on the nose (Z = −3.27, p < .01), whole face (Z = −3.98, p < .001) and all features (Z = −1.92, p = .05) increased at Time 2. In contrast, time spent off the face significantly decreased (Z = −3.61, p < .001). Figure 4 illustrates the averaged difference scores within the autistic group, which showed a visible decrease in fixation time on the mouth and whole picture following the intervention, while fixation time on other areas increased.

Figure 4.

Fixation Duration (ms) Average Difference Scores ± SE of the Fixation Duration (ms) on Interest Areas (Face, Mouth, Nose, Eyes, Whole Picture and Features) in the Pre- and Post-Intervention Assessments for the Autistic Group.

The total number of fixations also significantly decreased (Z = −6.33, p < .001). Table 5 presents the median and interquartile range for fixation duration and number of fixations from Time 1 to Time 2.

Table 5.

Fixation Duration (ms) and Total Fixations from Time 1 to Time 2 in the Autistic Group.

	Median	Interquartile Range
Individual Interest Areas
Eyes
Time 1	803.00	1551.00
Time 2	923.00	1772.50
Mouth
Time 1	0.00	816.00
Time 2	0.00	666.00
Nose
Time 1	362.00	772.00
Time 2	398.00	916.00
Whole Face
Time 1	1105.00	1105.00
Time 2	1310.00	1616.75
All Features
Time 1	2029.75	2635.25
Time 2	2295.00	2590.75
Off Face
Time 1	360.00	1137.75
Time 2	126.00	841.50
Total Fixations
Time 1	12.00	7.00
Time 2	10.00	7.00

Further exploratory analysis was conducted on the differences between Time 1 and Time 2 in fixation duration and total fixations across the three expressions presented in the eye-tracking task. There was a significant increase in the time participants spent fixating on the nose area for neutral expressions (Pre: m = 246.00, IQR = 0.00, 1105.00 vs Post: m = 438.50, IQR = 0.00, 879.50; Z = −1.93, p = .05), However, this increase was not observed for happy or angry expressions. Participants significantly increased their fixation time on the whole face for all three expressions: Angry (Pre: m = 1048.50, IQR = 556.50, 1758.00 vs Post: m = 1263.00, IQR = 672.50, 2130.25; Z = −2.12, p < .05), Happy (Pre: m = 1158.00, IQR =601.50, 1853.25 vs Post: m = 1364.50, IQR = 650.75, 2317.00; Z = −2.45, p < .05), and Neutral (Pre: m = 1084.00, IQR = 455.25, 1938.25 vs Post: m = 1325.00, IQR = 631.25, 2294.25; Z = −2.26, p < .05). A significant decrease in fixation time off the face was observed for the angry expression (Pre: m = 357.50, IQR = 0.00, 787.00 vs Post: m = 35.00, IQR = 0.00, 613.00; Z = −3.30, p < .01), but not happy or neutral expressions. There was a significant decrease in total fixations across all three emotions: Angry (Pre: m = 12.00, IQR = 8.00, 16.00 vs Post: m = 11.00, IQR = 7.00, 14.00; Z = −4.08, p < .001), Happy (Pre: m = 12.00, IQR = 8.00, 15.00 vs Post: m = 10.00, IQR = 6.00, 13.00; Z = −2.85, p < .01), and Neutral (Pre: m = 11.00, IQR = 8.00, 14.00 vs Post: m = 10.00, IQR = 6.00, 13.00; Z = −4.07, p < .001).

Discussion

This study investigated the effectiveness of the TYLES intervention in altering neural and behavioural responses to facial emotional expressions by observing eye gaze and neural responses in autistic participants before and after using the task. Specifically, the hypotheses predicted that emotion recognition skills would improve in the autistic group at Time 2, as measured by larger N170 and VPP amplitudes and increased fixation on salient facial features.

Combined Analyses

It was hypothesised that there would be a correlation between the time spent scanning the salient facial areas and the N170/VPP response in the autistic group. However, this hypothesis was not confirmed. This result suggests that the intervention may not have effectively supported autistic children in extracting information from the salient areas, leading to a more robust brain response. This finding aligns with previous studies conducted with autistic individuals, which also found that they did not spend as much time scanning the salient facial areas as neurotypical individuals.^7,30 Further research is needed to understand the relationship between face processing and attention to salient facial areas in autistic people.

Event-Related Potential

The study findings replicated group differences observed in other studies by showing a smaller N170 amplitude in the autistic group.^4,12,31 The non-autistic group had a larger N170 amplitude to faces, suggesting more significant processing of facial features when compared to the autistic group. The post-intervention N170 and VPP amplitudes of the autistic participants showed no significant changes compared to their pre-intervention results. The utilisation of ERPs to discern distinctions in brain activity between autistic and neurotypical individuals is a common practice, and it has been proposed as a potential biomarker to explore the outcomes of interventions due to their accessibility and non-invasive nature.^4,11 However, studies using ERPs to measure changes in the N170 following social skill interventions are relatively novel and as such, have yielded mixed results. Some studies have identified no changes in N170 amplitude or latency,^32,33 whereas others have suggested that N170 latency may be an important measure of intervention effectiveness.³⁴ Even in studies applying evidence-based treatments like pivotal response treatment, Early Start Denver Model, and Program for the Education and Enrichment of Relational Skills (PEERS^®) to teach social communication skills, outcomes concerning the N170 have been inconsistent.^35–38

This variability might be attributed to the time required to develop neural pathways related to face expertise. Modifying N170 and VPP responses may take longer compared to other brain responses. Consequently, while ERPs have provided insights into treatment outcomes and subtle differences in face and emotion processing, the current study's results align with the habit formation concept.^4,12 Currently, there are no established guidelines regarding the duration of skill-building interventions. Future studies that employ ERPs for measuring treatment effects should contemplate utilizing extended intervention periods to capture potentially delayed changes in neural responses.

Eye-Tracking

The eye-tracking data from the study confirmed that autistic participants spent less time looking at the salient areas of the face than neurotypical participants. However, it is worth noting that this is consistent with previous research and supports the eye avoidance hypothesis.^{4,8,14,39–41} The shorter eye fixation time in this study strongly indicates eye avoidance in autism. This is further supported by early studies showing that autistic people across all ages relied on the lower part of the face to recognise facial expressions (Simmons et al., 2009). Subsequent studies have confirmed these findings and have also shown that autistic youth often avert their gaze away from the eyes of others.^42,43 The study's outcomes mirror trends observed in other eye-tracking investigations, such as the work by Pelphrey, Sasson, Reznick, Paul, Goldman and Piven,⁴⁴ which similarly demonstrated that individuals with autism often direct their gaze toward alternative facial features like the nose or mouth, possibly perceived as less intimidating. Future qualitative studies could aim to understand the thoughts and emotions associated with this behaviour to delve deeper into the phenomenon of eye avoidance in autistic youth.

Previous research has found that autistic people may have challenges scanning negative facial expressions,^44–46 probably because they find these expressions threatening or overwhelming. However, the current study found that the participants spent less time looking away from the faces when scanning negative emotion expressions. This suggests that the participants may have become more comfortable looking at these expressions over time. It is important to note that not all autistic individuals have difficulty scanning negative facial expressions. Some recognise all basic facial expressions similarly to their neurotypical peers.^47,48 Therefore, further research is needed in order to account for the variability among individuals on the autistic spectrum, particularly since the broadening of diagnostic criteria for autism in the DSM-5.⁴⁹

Although there have been few studies on the topic, recent research has shown that eye-tracking can be a valuable and reliable way to measure the effectiveness of social skill interventions.^50–53 For example, one study found that children exhibited differences in attention to salient facial features following pivotal response intervention (Bradshaw et al, 2019), while another found that children who received slowness therapy, which aims to slow down speech, had longer gaze durations on the salient areas of the face.⁵¹ Although these studies did not use the same intervention as the current study, they suggest that eye-tracking could be a valuable tool for evaluating the effectiveness of social skill interventions.

A potential limitation of this study is re-test effect. Repeated exposure to the eye-tracking task could be partially attributable to the re-test effect rather than the intervention itself. To address this limitation, future research should consider employing a randomised controlled trial. However, a general study limitation was the small sample size due to COVID-19 restrictions as well as meeting the deadline for study completion. While eye tracking suggests behavioural changes following intervention, the N170 and VPP amplitudes did not change significantly. It is likely that the sample size did not provide sufficient power to detect potential ERP amplitude changes. It is also possible the reduced N170 and VPP amplitudes act more as an autism trait than a state marker. Another study limitation is the number of co-morbidities in our autistic sample which could not be statistically controlled for due to the small sample size. Furthermore, the current study's relatively short intervention period limits the generalisability of the findings. It is possible that a longer intervention duration could find more robust and sustained effects. Future research should explore the impact of extended intervention periods with the RCT framework.

This study demonstrated a novel methodology using ERP and eye tracking to investigate the effectiveness of the TYLES task. The results indicate that the N170 and VPP responses remained unchanged following the brief intervention. However, participants’ facial scanning did show changes, with participants spending more time on the face, although focusing more on the nose rather than the more salient areas of the face. Interestingly, although the participants successfully recognised emotions, their pattern of facial feature scanning differed. Thus, unresolved questions remain regarding the effectiveness of facial emotion recognition interventions in improving skills for autistic people.

Footnotes

Acknowledgements

The authors would like to thank all the young people and their families who gave up their time to participate in this study, especially with the added stress of a global pandemic. The authors would also like to thank Gavin Cooper for his help with creating ERP figures and Paula Gómez Álvarez for reviewing the manuscript.

ORCID iDs

Natalie G. Wall

Oliver Smith

Linda Campbell

Ulrich Schall

Ethics Approval

The Human Ethics Review Committee at the University of Newcastle approved the study (approval: H-2016-0098) on March 07, 2019. Legal guardians gave written consent for review and signature before starting the assessments.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by Hunter Medical Research Institute (Happy, Healthy Kids; No. G1801008) and an Australian Government Research Training Program Fee Offset and Scholarship.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Availability of Data and Materials

Study data are archived at the University of Newcastle and are available from the corresponding author on reasonable request with institutional ethics approval.

References

Lord

Brugha

Charman

, et al. Autism spectrum disorder. Nat Rev Dis Primers. 2020;6(1):5. doi:https://doi.org/10.1038/s41572-019-0138-4

American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders. 5th ed. American Psychiatric Association; 2013.

American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders. 5th ed, Text Revision. American Psychiatric Association; 2022.

Black

Chen

NTM

Iyer

, et al. Mechanisms of facial emotion recognition in autism spectrum disorders: Insights from eye tracking and electroencephalography. Neurosci Biobehav Rev. 2017;80(1):488–515. doi:https://doi.org/10.1016/j.neubiorev.2017.06.016

Hileman

Henderson

Mundy

Newell

Jaime

. Developmental and individual differences on the P1 and N170 ERP components in children with and without autism. Dev Neuropsychol. 2011;36(2):214–236. doi:https://doi.org/10.1080/87565641.2010.549870

McPartland

Dawson

Webb

Panagiotides

Carver

. Event-related brain potentials reveal anomalies in temporal processing of faces in autism spectrum disorder. J Child Psychol Psychiatry. 2004;45(7):1235–1245. doi:https://doi.org/10.1111/j.1469-7610.2004.00318.x

Wagner

Hirsch

Vogel-Farley

Redcay

Nelson

. Eye-Tracking, autonomic, and electrophysiological correlates of emotional face processing in adolescents with autism Spectrum disorder. Journal article. J Autism Dev Disord. 2013;43(1):188–199. doi:https://doi.org/10.1007/s10803-012-1565-1

Zhao

. Atypical gaze patterns to facial feature areas in autism spectrum disorders reveal age and culture effects: A meta-analysis of eye-tracking studies. Autism Res. 2021;14(12):2625–2639.

Parker

Crowley

Naples

, et al. The N170 event-related potential reflects delayed neural response to faces when visual attention is directed to the eyes in youths with ASD. Autism Res. 2021;14(7):1347–1356. doi:https://doi.org/10.1002/aur.2505

10.

Oakley

Loth

Jones

Chatham

Murphy

. Advances in the identification and validation of autism biomarkers. Comment Nat Rev Drug Discovery. 2022;21(10):697–698. doi:https://doi.org/10.1038/d41573-022-00141-y

11.

Kang

Keifer

Levy

Foss-Feig

McPartland

Lerner

. Atypicality of the N170 event-related potential in autism spectrum disorder: A meta-analysis. Biol Psychiatry: Cognit Neurosci Neuroimag. 2018;3(8):657–666. doi:https://doi.org/10.1016/j.bpsc.2017.11.003

12.

Monteiro

Simões

Andrade

Castelo Branco

. Processing of facial expressions in autism: A systematic review of EEG/ERP evidence. Review J Autism Dev Disord. 2017;4(4):255–276. doi:https://doi.org/10.1007/s40489-017-0112-6

13.

Boraston

Blakemore

S-J

. The application of eye-tracking technology in the study of autism. J Physiol. 2007;581(3):893–898. doi:https://doi.org/10.1113/jphysiol.2007.133587

14.

Frazier

Strauss

Klingemier

, et al. A meta-analysis of gaze differences to social and nonsocial information between individuals with and without autism. J Am Acad Child Adolesc Psychiatry. 2017;56(7):546–555. doi:https://doi.org/10.1016/j.jaac.2017.05.005

15.

Guillon

Hadjikhani

Baduel

Rogé

. Visual social attention in autism spectrum disorder: Insights from eye tracking studies. Neurosci Biobehav Rev. 2014;42:279–297. doi:https://doi.org/10.1016/j.neubiorev.2014.03.013

16.

Kouo

Egel

. The effectiveness of interventions in teaching emotion recognition to children with autism Spectrum disorder. Review J Autism Dev Disord. 2016;3(3):254–265. doi:https://doi.org/10.1007/s40489-016-0081-1

17.

Vettori

Dzhelyova

Van der Donck

, et al. Combined frequency-tagging EEG and eye tracking reveal reduced social bias in boys with autism spectrum disorder. Cortex. 2020;125:135–148.

18.

Vettori

Dzhelyova

Van der Donck

, et al. Frequency-tagging electroencephalography of superimposed social and non-social visual stimulation streams reveals reduced saliency of faces in autism spectrum disorder. Front Psychiatry. 2020;11. doi:https://doi.org/10.3389/fpsyt.2020.00332

19.

Vettori

Van der Donck

Nys

, et al. Combined frequency-tagging EEG and eye-tracking measures provide no support for the “excess mouth/diminished eye attention” hypothesis in autism. Mol Autism. 2020;11(1):1–22.

20.

Tan

Liu

. A trend on autism Spectrum disorder research: Eye tracking-EEG correlative analytics. IEEE Trans Cognit Dev Syst. 2022;14(3):1232–1244. doi:https://doi.org/10.1109/tcds.2021.3102646

21.

Parellada

Andreu-Bernabeu

Burdeus

, et al. In search of biomarkers to guide interventions in autism spectrum disorder: A systematic review. Am J Psychiatry. 2023;180(1):23–40.

22.

Wall

Wallis

Smith

Campbell

Loughland

Schall

. Teach yourself to learn (facial) expressions: an acceptability and usability study of an app teaching facial emotion recognition skills to autistic children. Int J Hum–Comput Inter. 2025. doi:https://doi.org/10.1080/10447318.2025.2474491

23.

Wegrzyn

Vogt

Kireclioglu

Schneider

Kissler

. Mapping the emotional face. How individual face parts contribute to successful emotion recognition. PLOS ONE. 2017;12(5):e0177239. doi:https://doi.org/10.1371/journal.pone.0177239

24.

LoBue

Thrasher

. The child affective facial expression (CAFE) set: Validity and reliability from untrained adults. Front Psychol. 2014;5:15–32. doi:https://doi.org/10.3389/fpsyg.2014.01532

25.

Experiment Builder. Version 2.2.299. SR Research Ltd; 2019.

26.

Compumedics Neuroscan. Compumedics; https://compumedicsneuroscan.com/

27.

McCabe

Rich

Loughland

Schall

Campbell

. Visual scanpath abnormalities in 22q11.2 deletion syndrome: Is this a face specific deficit? Psychiatry Res. 2011;189(2):292–298. doi:https://doi.org/10.1016/j.psychres.2011.06.012

28.

Campbell

McCabe

Leadbeater

Schall

Loughland

Rich

. Visual scanning of faces in 22q11.2 deletion syndrome: Attention to the mouth or the eyes? Psychiatry Res. 2010;177(1-2):211–215. doi:https://doi.org/10.1016/j.psychres.2009.06.007

29.

IBM SPSS Statistics for Windows. Version 28.0. IBM SPSS; Released 2020.

30.

Parker

Kamps

. Effects of task analysis and self-monitoring for children with autism in multiple social settings. Empirical study; quantitative study. Focus Autism Other Dev Disabil. 2011;26(3):131–142. doi:https://doi.org/10.1177/1088357610376945

31.

Jeste

Nelson

. Event related potentials in the understanding of autism Spectrum disorders: An analytical review. J Autism Dev Disord. 2008;39(3):495. doi:https://doi.org/10.1007/s10803-008-0652-9

32.

Key

Corbett

. The unfulfilled promise of the N170 as a social biomarker. Biol Psychiatry: Cognit Neurosci Neuroimag. 2020;5(3):342–353. doi:https://doi.org/10.1016/j.bpsc.2019.08.011

33.

Faja

Webb

Jones

, et al. The effects of face expertise training on the behavioral performance and brain activity of adults with high functioning autism spectrum disorders. J Autism Dev Disord. 2012;42(2):278–293. doi:https://doi.org/10.1007/s10803-011-1243-8

34.

Shu

Zhang

Xue

, et al. Intervention effect of group sensory integration training on social responsiveness and N170 event-related potential of children with autism. Behav Sci (Basel). 2024;14(3):202. doi:https://doi.org/10.3390/bs14030202

35.

Kala

Rolison

Trevisan

, et al. Brief report: Preliminary evidence of the N170 as a biomarker of response to treatment in autism spectrum disorder. Front Psychiatry. 2021;12:709382.

36.

Dawson

Jones

Merkle

, et al. Early behavioral intervention is associated with normalized brain activity in young children with autism. J Am Acad Child Adolesc Psychiatry. 2012;51(11):1150–1159. doi:https://doi.org/10.1016/j.jaac.2012.08.018

37.

Baker

Veytsman

Choy

Blacher

Stavropoulos

KKM

. Investigating changes in reward-related neural correlates after PEERS intervention in adolescents with ASD: Preliminary evidence of a “precision medicine” approach. Front Psychiatry. 2021;12. doi:https://doi.org/10.3389/fpsyt.2021.742280

38.

Haendel

Barrington

Magnus

, et al. Changes in electroencephalogram coherence in adolescents with autism Spectrum disorder after a social skills intervention. Autism Res. 2021;14(4):787–803. doi:https://doi.org/10.1002/aur.2459

39.

Chita-Tegmark

. Social attention in ASD: A review and meta-analysis of eye-tracking studies. Res Dev Disabil. 2016;48:79–93. doi:https://doi.org/10.1016/j.ridd.2015.10.011

40.

Hao

Wang

Xie

Wang

Liu

. How do autism spectrum disorders process human face? A meta-analysis of eye-tracking studies. Adv Psychol Sci. 2018;26(1):26.

41.

Stuart

Whitehouse

Palermo

Bothe

Badcock

. Eye gaze in autism spectrum disorder: a review of neural evidence for the eye avoidance hypothesis. J Autism Dev Disord. 2023;53(5):1884–1905. doi:https://doi.org/10.1007/s10803-022-05443-z

42.

Hutt

Ounsted

. The biological significance of gaze aversion with particular reference to the syndrome of infantile autism. Behav Sci. 1966;11(5):346–356. doi:https://doi.org/10.1002/bs.3830110504

43.

Tanaka

Sung

. The “eye avoidance” hypothesis of autism face processing. J Autism Dev Disord. 2016;46(5):1538–1552. doi:https://doi.org/10.1007/s10803-013-1976-7

44.

Pelphrey

Sasson

Reznick

Paul

Goldman

Piven

. Visual scanning of faces in autism. Journal article. J Autism Dev Disord. 2002;32(4):249–261. doi:https://doi.org/10.1023/a:1016374617369

45.

Ashwin

Chapman

Colle

Baron-Cohen

. Impaired recognition of negative basic emotions in autism: A test of the amygdala theory. Soc Neurosci. 2006;1(3-4):349–363. doi:https://doi.org/10.1080/17470910601040772

46.

Humphreys

Minshew

Leonard

Behrmann

. A fine-grained analysis of facial expression processing in high-functioning adults with autism. Neuropsychologia. 2007;45(4):685–695. doi:https://doi.org/10.1016/j.neuropsychologia.2006.08.003

47.

Castelli

. Understanding emotions from standardized facial expressions in autism and normal development. Autism. 2005;9(4):428–449. doi:https://doi.org/10.1177/1362361305056082

48.

Loveland

Tunali–Kotoski

Chen

, et al. Emotion recognition in autism: Verbal and nonverbal information. Dev Psychopathol. 1997;9(3):579–593. doi:https://doi.org/10.1017/s0954579497001351

49.

American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders. 5th ed. American Psychiatric Association; 2013.

50.

Bradshaw

Shic

Holden

, et al. The use of eye tracking as a biomarker of treatment outcome in a pilot randomized clinical trial for young children with autism. Randomized controlled trial research support, non-U.S. Gov't. Autism Res. 2019;12(5):779–793. doi:https://doi.org/10.1002/aur.2093

51.

Gepner

Charrier

Arciszewski

Tardif

. Slowness therapy for children with autism spectrum disorder: A blind longitudinal randomized controlled study. J Autism Dev Disord. 2022;52(7):3102–3115. doi:https://doi.org/10.1007/s10803-021-05183-6

52.

Greene

Parish-Morris

Sullivan

, et al. Dynamic eye tracking as a predictor and outcome measure of social skills intervention in adolescents and adults with autism Spectrum disorder. J Autism Dev Disord. 2021;51(4):1173–1187. doi:https://doi.org/10.1007/s10803-020-04594-1

53.

Fletcher-Watson

Hampton

. The potential of eye-tracking as a sensitive measure of behavioural change in response to intervention. Sci Rep. 2018;8(1):14715.

Using EEG and Eye Tracking to Evaluate an Emotion Recognition iPad App for Autistic Children

Abstract

Keywords

Introduction

Teach Yourself to Learn Emotions (TYLES)

Methods

Participants

Stimuli

Electroencephalogram and Eye-Tracking

Procedure

Electroencephalogram

Eye-Tracking

Data Analyses

Event Related Potential (Pre-Processing)

Visual Scanning (Pre-Processing)

Combined Analyses

Statistical Analyses

Results

Combined Analyses

Event Related Potential

Eye-Tracking

Discussion

Combined Analyses

Event-Related Potential

Eye-Tracking

Footnotes

Acknowledgements

ORCID iDs

Ethics Approval

Funding

Declaration of Conflicting Interests

Availability of Data and Materials

References