Restricted accessResearch articleFirst published online 2022-5
Audiovisual Temporal Processing in Children and Adolescents With Schizophrenia and Children and Adolescents With Autism: Evidence From Simultaneity-Judgment Tasks and Eye-Tracking Data
Impaired audiovisual temporal integration, manifested as an abnormally widened temporal-binding window (TBW) for integrating sensory information, is found in both autism spectrum disorder (ASD) and schizophrenia (SCZ) and contributes to aberrant perceptual experiences and impaired social communication. We conducted two experiments using age-comparable samples of participants with early-onset SCZ and participants with ASD. Sophisticated paradigms, including a unisensory temporal-order-judgment task (TOJ), an audiovisual-simultaneity-judgment task (SJ), and an eye-tracking task were used. Results showed generalized deficits in temporal processing in SCZ ranging from unisensory to multisensory modalities and from nonspeech to speech stimuli. In contrast, the widened TBW in ASD mainly affected speech stimuli processing. Applying the eye-tracking task with ecologically valid linguistic stimuli, we found that both participants with SCZ and participants with ASD exhibited reduced sensitivity of detecting audiovisual speech asynchrony. This impaired audiovisual speech integration correlated with negative symptoms. Although both ASD and SCZ have impaired multisensory temporal integration, ASD impairs speech-related processing, and SCZ is associated with generalized deficits.
American Psychiatric Association. (2000). Diagnostic and statistical manual of mental disorders (4th ed., text rev).
2.
American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (5th ed.).
3.
AndreasenN. C. (1989). The Scale for the Assessment of Negative Symptoms (SANS): Conceptual and theoretical foundations. British Journal of Psychiatry, 155(Suppl. 7), 49–52. https://doi.org/10.1192/S0007125000291496
4.
AuyeungB.Baron-CohenS.WheelwrightS.AllisonC. (2008). The Autism Spectrum Quotient: Children’s Version (AQ-Child). Journal of Autism and Developmental Disorders, 38(7), 1230–1240. https://doi.org/10.1007/s10803-007-0504-z
5.
BrockJ.BrownC. C.BoucherJ.RipponG. (2002). The temporal binding deficit hypothesis of autism. Development and Psychopathology, 14(2), 209–224. https://doi.org/10.1017/S0954579402002018
6.
CapaR. L.DuvalC. Z.BlaisonD.GierschA. (2014). Patients with schizophrenia selectively impaired in temporal order judgments. Schizophrenia Research, 156(1), 51–55. https://doi.org/10.1016/j.schres.2014.04.001
7.
CenC. Q.LiangY. Y.ChenQ. R.ChenK. Y.DengH. Z.ChenB. Y.ZouX. B. (2017). Investigating the validation of the Chinese Mandarin version of the Social Responsiveness Scale in a Mainland China child population. BMC Psychiatry, 17, Article 51. https://doi.org/10.1186/s12888-016-1185-y
8.
ChenT.HsiehM. H.LinY.ChanP. S.ChengC. (2020). Mismatch negativity to different deviant changes in autism spectrum disorders: A meta-analysis. Clinical Neurophysiology, 131(3), 766–777. https://doi.org/10.1016/j.clinph.2019.10.031
9.
Chita-TegmarkM. (2016). Social attention in ASD: A review and meta-analysis of eye-tracking studies. Research in Developmental Disabilities, 48, 79–93. https://doi.org/10.1016/j.ridd.2015.10.011
10.
de Boer-SchellekensL.StekelenburgJ. J.MaesJ. P.Van GoolA. R.VroomenJ. (2014). Sound improves diminished visual temporal sensitivity in schizophrenia. Acta Psychologica, 147, 136–142. https://doi.org/10.1016/j.actpsy.2013.06.013
11.
FeldmanJ. I.DunhamK.CassidyM.WallaceM. T.LiuY.WoynaroskiT. G. (2018). Audiovisual multisensory integration in individuals with autism spectrum disorder: A systematic review and meta-analysis. Neuroscience and Biobehavioral Reviews, 95, 220–234. https://doi.org/10.1016/j.neubiorev.2018.09.020
12.
FerriF.NikolovaY. S.PerrucciM. G.CostantiniM.FerrettiA.GattaV.HuangZ.EddenR.YueQ.D’AuroraM.SibilleE.StuppiaL.RomaniG. L.NorthoffG. (2017). A neural “tuning curve” for multisensory experience and cognitive-perceptual schizotypy. Schizophrenia Bulletin, 43(4), 801–813. https://doi.org/10.1093/schbul/sbw174
13.
FerriF.VenskusA.FotiaF.CookeJ.RomeiV. (2018). Higher proneness to multisensory illusions is driven by reduced temporal sensitivity in people with high schizotypal traits. Consciousness and Cognition, 65, 263–270. https://doi.org/10.1016/j.concog.2018.09.006
14.
FletcherP. C.FrithC. D. (2009). Perceiving is believing: A Bayesian approach to explaining the positive symptoms of schizophrenia. Nature Reviews Neuroscience, 10(1), 48–58. https://doi.org/10.1038/nrn2536
15.
Foss-FeigJ. H.SchauderK. B.KeyA. P.WallaceM. T.StoneW. L. (2017). Audition-specific temporal processing deficits associated with language function in children with autism spectrum disorder. Autism Research, 10(11), 1845–1856. https://doi.org/10.1002/aur.1820
16.
FoucherJ. R.LacambreM.PhamB. T.GierschA.ElliottM. A. (2007). Low time resolution in schizophrenia. Lengthened windows of simultaneity for visual, auditory and bimodal stimuli. Schizophrenia Research, 97(1–3), 118–127. https://doi.org/10.1016/j.schres.2007.08.013
GierschA.LalanneL.CorvesC.SeubertJ.ShiZ.FoucherJ.ElliottM. A. (2009). Extended visual simultaneity thresholds in patients with schizophrenia. Schizophrenia Bulletin, 35(4), 816–825. https://doi.org/10.1093/schbul/sbn016
19.
GrossmanR. B.SteinhartE.MitchellT.McilvaneW. (2015). “Look who’s talking!” Gaze patterns for implicit and explicit audio-visual speech synchrony detection in children with high-functioning autism. Autism Research, 8(3), 307–316. https://doi.org/10.1002/aur.1447
20.
HaßK.SinkeC.ReeseT.RoyM.WiswedeD.DilloW.OranjeB.SzycikG. R. (2017). Enlarged temporal integration window in schizophrenia indicated by the double-flash illusion. Cognitive Neuropsychiatry, 22(2), 145–158. https://doi.org/10.1080/13546805.2017.1287693
21.
HappéF.FrithU. (2006). The weak coherence account: Detail-focused cognitive style in autism spectrum disorders. Journal of Autism and Developmental Disorders, 36(1), 5–25. https://doi.org/10.1007/s10803-005-0039-0
22.
HayR. A.RoachB. J.SrihariV. H.WoodsS. W.FordJ. M.MathalonD. H. (2015). Equivalent mismatch negativity deficits across deviant types in early illness schizophrenia-spectrum patients. Biological Psychology, 105, 130–137. https://doi.org/10.1016/j.biopsycho.2015.01.004
23.
HillockA. R.PowersA. R.WallaceM. T. (2011). Binding of sights and sounds: Age-related changes in multisensory temporal processing. Neuropsychologia, 49(3), 461–467. https://doi.org/10.1016/j.neuropsychologia.2010.11.041
24.
Hillock-DunnA.WallaceM. T. (2012). Developmental changes in the multisensory temporal binding window persist into adolescence. Developmental Science, 15(5), 688–696. https://doi.org/10.1111/j.1467-7687.2012.01171.x
25.
JavittD. C.SweetR. A. (2015). Auditory dysfunction in schizophrenia: Integrating clinical and basic features. Nature Reviews Neuroscience, 16(9), 535–550. https://doi.org/10.1038/nrn4002
26.
KawakamiS.UonoS.OtsukaS.YoshimuraS.ZhaoS.ToichiM. (2020). Atypical multisensory integration and the temporal binding window in autism spectrum disorder. Journal of Autism and Developmental Disorders, 50, 3944–3956. https://doi.org/10.1007/s10803-020-04452-0
27.
KawakamiS.UonoS.OtsukaS.ZhaoS.ToichiM. (2020). Everything has its time: Narrow temporal windows are associated with high levels of autistic traits via weaknesses in multisensory integration. Journal of Autism and Developmental Disorders, 50(5), 1561–1571. https://doi.org/10.1007/s10803-018-3762-z
28.
KayS. R.FiszbeinA.OplerL. A. (1987). The positive and negative syndrome scale (PANSS) for schizophrenia. Schizophrenia Bulletin, 13(2), 261–276. https://doi.org/10.1093/schbul/13.2.261
29.
KwakyeL. D.Foss-FeigJ. H.CascioC. J.StoneW. L.WallaceM. T. (2011). Altered auditory and multisensory temporal processing in autism spectrum disorders. Frontiers in Integrative Neuroscience, 4, Article 129. https://doi.org/10.3389/fnint.2010.00129
30.
LalanneL.Van AsscheM.GierschA. (2012). When predictive mechanisms go wrong: Disordered visual synchrony thresholds in schizophrenia. Schizophrenia Bulletin, 38(3), 506–513. https://doi.org/10.1093/schbul/sbq107
31.
LiuJ.WongK. K. Y.DongF.RaineA.TuvbladC. (2019). The Schizotypal Personality Questionnaire–Child (SPQ-C): Psychometric properties and relations to behavioral problems with multi-informant ratings. Psychiatry Research, 275, 204–211. https://doi.org/10.1016/j.psychres.2019.03.006
32.
LiuY.-X.LiuJ.WangY.-F. (2011). Reliability and validity of Chinese version of the Mini International Neuropsychiatric Interview for Children and Adolescents (child version). Chinese Mental Health Journal, 25(1), 8–13.
33.
MartinB.GierschA.HuronC.van WassenhoveV. (2013). Temporal event structure and timing in schizophrenia: Preserved binding in a longer “now.” Neuropsychologia, 51(2), 358–371. https://doi.org/10.1016/j.neuropsychologia.2012.07.002
34.
MeilleurA.FosterN. E. V.CollS. M.BrambatiS. M.HydeK. L. (2020). Unisensory and multisensory temporal processing in autism and dyslexia: A systematic review and meta-analysis. Neuroscience and Biobehavioral Reviews, 116, 44–63. https://doi.org/10.1016/j.neubiorev.2020.06.013
35.
MottronL.DawsonM.SoulièresI.HubertB.BurackJ. (2006). Enhanced perceptual functioning in autism: An update, and eight principles of autistic perception. Journal of Autism and Developmental Disorders, 36(1), 27–43. https://doi.org/10.1007/s10803-005-0040-7
36.
NäätänenR.PaavilainenP.RinneT.AlhoK. (2007). The mismatch negativity (MMN) in basic research of central auditory processing: A review. Clinical Neurophysiology, 118(12), 2544–2590. https://doi.org/10.1016/j.clinph.2007.04.026
37.
NikolaidesA.MiessS.AuveraI.MüllerR.KlosterkötterJ.RuhrmannS. (2016). Restricted attention to social cues in schizophrenia patients. European Archives of Psychiatry and Clinical Neuroscience, 266(7), 649–661. https://doi.org/10.1007/s00406-016-0705-6
38.
NoelJ. P.De NiearM.Van Der BurgE.WallaceM. T. (2016). Audiovisual simultaneity judgment and rapid recalibration throughout the lifespan. PLOS ONE, 11(8), Article 0161698. https://doi.org/10.1371/journal.pone.0161698
39.
NoelJ. P.De NiearM. A.LazzaraN. S.WallaceM. T. (2018). Uncoupling between multisensory temporal function and nonverbal turn-taking in autism spectrum disorder. IEEE Transactions on Cognitive and Developmental Systems, 10(4), 973–982. https://doi.org/10.1109/TCDS.2017.2778141
40.
NoelJ. P.De NiearM. A.StevensonR.AlaisD.WallaceM. T. (2017). Atypical rapid audio-visual temporal recalibration in autism spectrum disorders. Autism Research, 10(1), 121–129. https://doi.org/10.1002/aur.1633
41.
NoelJ. P.StevensonR. A.WallaceM. T. (2018). Atypical audiovisual temporal function in autism and schizophrenia: Similar phenotype, different cause. European Journal of Neuroscience, 47(10), 1230–1241. https://doi.org/10.1111/ejn.13911
42.
OliverL. D.Moxon-EmreI.LaiM.GrennanL.VoineskosA. N.AmeisS. H. (2021). Social cognitive performance in schizophrenia spectrum disorders compared with autism spectrum disorder: A systematic review, meta-analysis, and meta-regression. JAMA Psychiatry, 78(3), 281–292. https://doi.org/10.1001/jamapsychiatry.2020.3908
43.
PattenE.WatsonL. R.BaranekG. T. (2014). Temporal synchrony detection and associations with language in young children with ASD. Autism Research and Treatment, 2014, Article 678346. https://doi.org/10.1155/2014/678346
44.
PellicanoE.BurrD. (2012). When the world becomes “too real”: A Bayesian explanation of autistic perception. Trends in Cognitive Sciences, 16(10), 504–510. https://doi.org/10.1016/j.tics.2012.08.009
45.
PowersA. R.HillockA. R.WallaceM. T. (2009). Perceptual training narrows the temporal window of multisensory binding. The Journal of Neuroscience, 29(39), 12265–12274. https://doi.org/10.1523/JNEUROSCI.3501-09.2009
46.
RighiG.TenenbaumE. J.McCormickC.BlossomM.AmsoD.SheinkopfS. J. (2018). Sensitivity to audio-visual synchrony and its relation to language abilities in children with and without ASD. Autism Research, 11(4), 645–653. https://doi.org/10.1002/aur.1918
47.
SandersL. L. O.De MillasW.HeinzA.KathmannN.SterzerP. (2013). Apparent motion perception in patients with paranoid schizophrenia. European Archives of Psychiatry and Clinical Neuroscience, 263(3), 233–239. https://doi.org/10.1007/s00406-012-0344-5
48.
SchmidtH.McFarlandJ.AhmedM.McDonaldC.ElliottM. A. (2011). Low-level temporal coding impairments in psychosis: Preliminary findings and recommendations for further studies. Journal of Abnormal Psychology, 120(2), 476–482. https://doi.org/10.1037/a0023387
49.
SinhaP.KjelgaardM. M.GandhiT. K.TsouridesK.CardinauxA. L.PantazisD.DiamondS. D.HeldR. M. (2014). Autism as a disorder of prediction. Proceedings of the National Academy of Sciences of the United States of America, 111(42), 15220–15225. https://doi.org/10.1073/pnas.1416797111
50.
SmithE.ZhangS.BennettoL. (2017). Temporal synchrony and audiovisual integration of speech and object stimuli in autism. Research in Autism Spectrum Disorders, 39, 11–19. https://doi.org/10.1016/j.rasd.2017.04.001
51.
SterzerP.AdamsR. A.FletcherP.FrithC.LawrieS. M.MuckliL.PetrovicP.UhlhaasP.VossM.CorlettP. R. (2018). The predictive coding account of psychosis. Biological Psychiatry, 84(9), 634–643. https://doi.org/10.1016/j.biopsych.2018.05.015
52.
StevensonR. A.BaumS. H.KruegerJ.NewhouseP. A.WallaceM. T. (2018). Links between temporal acuity and multisensory integration across life span. Journal of Experimental Psychology: Human Perception and Performance, 44(1), 106–116. https://doi.org/10.1037/xhp0000424
53.
StevensonR. A.ParkS.CochranC.McIntoshL. G.NoelJ. P.BarenseM. D.FerberS.WallaceM. T. (2017). The associations between multisensory temporal processing and symptoms of schizophrenia. Schizophrenia Research, 179, 97–103. https://doi.org/10.1016/j.schres.2016.09.035
54.
StevensonR. A.SegersM.NcubeB. L.BlackK. R.BebkoJ. M.FerberS.BarenseM. D. (2018). The cascading influence of multisensory processing on speech perception in autism. Autism, 22(5), 609–624. https://doi.org/10.1177/1362361317704413
55.
StevensonR. A.SiemannJ. K.SchneiderB. C.EberlyH. E.WoynaroskiT. G.CamarataS. M.WallaceM. T. (2014). Multisensory temporal integration in autism spectrum disorders. The Journal of Neuroscience, 34(3), 691–697. https://doi.org/10.1523/JNEUROSCI.3615-13.2014
56.
StevensonR. A.ToulminJ. K.YoumA.BesneyR. M. A.SchulzS. E.BarenseM. D.FerberS. (2017). Increases in the autistic trait of attention to detail are associated with decreased multisensory temporal adaptation. Scientific Reports, 7(1), Article 14354. https://doi.org/10.1038/s41598-017-14632-1
57.
SuY. E.WangH.GengY. G.SunL.DuY. S.FanF.SuL. Y. (2015). Parent ratings of ADHD symptoms in Chinese urban schoolchildren: Assessment with the Chinese ADHD rating Scale–IV: Home version. Journal of Attention Disorders, 19(12), 1022–1033. https://doi.org/10.1177/1087054712461177
58.
ThoenesS.OberfeldD. (2017). Meta-analysis of time perception and temporal processing in schizophrenia: Differential effects on precision and accuracy. Clinical Psychology Review, 54, 44–64. https://doi.org/10.1016/j.cpr.2017.03.007
59.
ThyeM. D.BednarzH. M.HerringshawA. J.SartinE. B.KanaR. K. (2018). The impact of atypical sensory processing on social impairments in autism spectrum disorder. Developmental Cognitive Neuroscience, 29, 151–167. https://doi.org/10.1016/j.dcn.2017.04.010
60.
TsengH. H.BossongM. G.ModinosG.ChenK. M.McGuireP.AllenP. (2015). A systematic review of multisensory cognitive-affective integration in schizophrenia. Neuroscience and Biobehavioral Reviews, 55, 444–452. https://doi.org/10.1016/j.neubiorev.2015.04.019
61.
TuriM.KaraminisT.PellicanoE.BurrD. (2016). No rapid audiovisual recalibration in adults on the autism spectrum. Scientific Reports, 6, Article 21756. https://doi.org/10.1038/srep21756
van de CruysS.EversK.van der HallenR.van EylenL.BoetsB.de-WitL.WagemansJ. (2014). Precise minds in uncertain worlds: Predictive coding in autism. Psychological Review, 121(4), 649–675. https://doi.org/10.1037/a0037665
van LaarhovenT.StekelenburgJ. J.VroomenJ. (2019). Increased sub-clinical levels of autistic traits are associated with reduced multisensory integration of audiovisual speech. Scientific Reports, 9(1), Article 9535. https://doi.org/10.1038/s41598-019-46084-0
66.
WallaceM. T.StevensonR. A. (2014). The construct of the multisensory temporal binding window and its dysregulation in developmental disabilities. Neuropsychologia, 64, 105–123. https://doi.org/10.1016/j.neuropsychologia.2014.08.005
67.
YaguchiA.HidakaS. (2018). Distinct autistic traits are differentially associated with the width of the multisensory temporal binding window. Multisensory Research, 31(6), 523–536. https://doi.org/10.1163/22134808-00002612
68.
ZhouH.CaiX.WeiglM.BangP.CheungE. F. C.ChanR. C. K. (2018). Multisensory temporal binding window in autism spectrum disorders and schizophrenia spectrum disorders: A systematic review and meta-analysis. Neuroscience and Biobehavioral Reviews, 86, 66–76. https://doi.org/10.1016/j.neubiorev.2017.12.013
69.
ZhouH.CheungE. F. C.ChanR. C. K. (2020). Audiovisual temporal integration: Cognitive processing, neural mechanisms, developmental trajectory and potential interventions. Neuropsychologia, 140, Article 107396. https://doi.org/10.1016/j.neuropsychologia.2020.107396
70.
ZhouH.ShiL.YangH.CheungE. F. C.ChanR. C. K. (2020). Audiovisual temporal integration and rapid temporal recalibration in adolescents and adults: Age-related changes and its correlation with autistic traits. Autism Research, 13(4), 615–626. https://doi.org/10.1002/aur.2249
71.
ZhouH.YangH.ShiL.LuiS. S. Y.CheungE. F. C.ChanR. C. K. (2021). Correlations between audiovisual temporal processing and sensory responsiveness in adolescents with autistic traits. Journal of Autism and Developmental Disorders, 51(7), 2450–2460. https://doi.org/10.1007/s10803-020-04724-9
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
