Anecdotal evidence has suggested that rater-based measures (e.g., parent report) may have strong across-trait/within-individual covariance that detracts from trait-specific measurement precision; rater measurement-related bias may help explain poor correlation within Autism Spectrum Disorder (ASD) samples between rater-based and performance-based measures of the same trait. We used a multi-trait, multi-method approach to examine method-associated bias within an ASD sample (n = 83). We examined performance/rater-instrument pairs for attention, inhibition, working memory, motor coordination, and core ASD features. Rater-based scores showed an overall greater methodology bias (57% of variance in score explained by method), while performance-based scores showed a weaker methodology bias (22%). The degree of inter-individual variance explained by method alone substantiates an anecdotal concern associated with the use of rater measures in ASD.
Brookman-FrazeeL.StadnickN.ChlebowskiC.Baker-EriczenM.GangerW. (2018). Characterizing psychiatric comorbidity in children with autism spectrum disorder receiving publicly funded mental health services. Autism, 22(8), 938–952. https://doi.org/10.1177/1362361317712650
2.
CampbellD. T.FiskeD. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56(2), 81–105. https://www.ncbi.nlm.nih.gov/pubmed/13634291
3.
ConnorsC. K. (2008). Connors 3. Multi-Health Systems.
4.
ConstantinoJ. N.GruberC. (2005). Social Responsiveness Scale. Western Psychological Services.
5.
ConstantinoJ. N.GruberC. P. (2012). Social Responsiveness Scale–Second Edition (SRS-2). Western Psychological Services.
6.
CronbachL. J.MeehlP. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52(4), 281–302. https://doi.org/10.1037/h0040957
7.
de GiambattistaC.VenturaP.TrerotoliP.MargariF.MargariL. (2021). Sex differences in autism spectrum disorder: Focus on high functioning children and adolescents. Frontiers in Psychiatry, 12, Article 539835. https://doi.org/10.3389/fpsyt.2021.539835
8.
DencklaM. B. (1985). Revised neurological examination for subtle signs. Psychopharmacology Bulletin, 21, 773–779.
9.
EhlersS.GillbergC.WingL. (1999). A screening questionnaire for Asperger syndrome and other high-functioning autism spectrum disorders in school age children. Journal of Autism and Developmental Disorders, 29(2), 129–141. https://doi.org/10.1023/a:1023040610384
10.
EwenJ. B.ShapiroB. K. (2005). Disorders of attention or learning in neurodevelopmental disorders. Seminars in Pediatric Neurology, 12(4), 229–241. https://doi.org/10.1016/j.spen.2005.12.005
11.
FriedrichF. J.RaderS. D. (1997). Component process analysis in experimental and clinical neuropsychology. In MaruishM. E.MosesJ. A. (Eds.), Clinical neuropsychology: Theoretical foundations for practitioners (pp. 59–79). Lawrence Erlbaum.
12.
GioiaG. A.IsquithP. K.RetzlaffP. D.EspyK. A. (2002). Confirmatory factor analysis of the Behavior Rating Inventory of Executive Function (BRIEF) in a clinical sample. Child Neuropsychology, 8(4), 249–257. https://doi.org/10.1076/chin.8.4.249.13513
13.
HavdahlK. A.Hus BalV.HuertaM.PicklesA.OyenA. S.StoltenbergC.LordC.BishopS. L. (2016). Multidimensional influences on autism symptom measures: Implications for use in etiological research. Journal of the American Academy of Child and Adolescent Psychiatry, 55(12), 1054–1063.e1053. https://doi.org/10.1016/j.jaac.2016.09.490
14.
KaufmanJ.BirmaherB.AxelsonD.PerepletchikovaF.BrentD.RyanN. (2013). Kiddie Schedule for affective disorders and schizophrenia for school-aged children: Lifetime version (Kiddie-SADS-PL 2013 working draft). Western Psychiatric Institute and Clinic and Yale University.
15.
KorkmanM.KirkU.KempS. (2007). NEPSY-II: A developmental neuropsychological assessment. The Psychological Corporation.
16.
LeungR. C.VoganV. M.PowellT. L.AnagnostouE.TaylorM. J. (2016). The role of executive functions in social impairment in autism spectrum disorder. Child Neuropsychology, 22(3), 336–344. https://doi.org/10.1080/09297049.2015.1005066
17.
LordC.RisiS.LambrechtL.CookE. H.JrLeventhalB. L.DiLavoreP. C.PicklesA.RutterM. (2000). The autism diagnostic observation schedule-generic: A standard measure of social and communication deficits associated with the spectrum of autism. Journal of Autism and Developmental Disorders, 30(3), 205–223.
18.
LordC.RutterM.DiLavoreP. C.RisiS.GothamK.BishopS. L. (2012). Autism Diagnostic Observation Schedule second edition (ADOS-2) manual (Part 1): Modules 1–4. Western Psychological Services.
19.
LordC.RutterM.Le CouteurA. (1994). Autism Diagnostic Interview-revised: A revised version of a diagnostic interview for caregivers of individuals with possible pervasive developmental disorders. Journal of Autism and Developmental Disorders, 24, 659–685.
20.
McAuliffeD.PillaiA. S.TiedemannA.MostofskyS. H.EwenJ. B. (2017). Dyspraxia in ASD: Impaired coordination of movement elements. Autism Research, 10(4), 648–652. https://doi.org/10.1002/aur.1693
21.
McAuliffeD.ZhaoY.PillaiA. S.AmentK.AdamekJ.CaffoB. S.MostofskyS. H.EwenJ. B. (2020). Learning of skilled movements via imitation in ASD. Autism Research, 13(5), 777–784. https://doi.org/10.1002/aur.2253
22.
NunnallyJ. C.BernsteinI. H. (2010). Psychometric theory (3rd ed.). Tata McGraw-Hill.
23.
PoldrackR. A.YarkoniT. (2016). From brain maps to cognitive ontologies: Informatics and the search for mental structure. Annual Review of Psychology, 67, 587–612. https://doi.org/10.1146/annurev-psych-122414-033729
24.
RongY.YangC. J.JinY.WangY. (2021). Prevalence of attention-deficit/hyperactivity disorder in individuals with autism spectrum disorder: A meta-analysis. Research in Autism Spectrum Disorders, 83, 101759.
25.
RutterM.BaileyA.LordC. (2003). The social communication questionnaire: Manual. Western Psychological Services.
26.
ShalliceT.CooperR. P. (2011). The organization of mind. Oxford University Press.
27.
SimonoffE.PicklesA.CharmanT.ChandlerS.LoucasT.BairdG. (2008). Psychiatric disorders in children with autism spectrum disorders: Prevalence, comorbidity, and associated factors in a population-derived sample. Journal of the American Academy of Child and Adolescent Psychiatry, 47(8), 921–929. https://doi.org/10.1097/CHI.0b013e318179964f
28.
SoltaniA.SchworerE. K.AltayeM.FidlerD. J.BeebeD. W.WileyS.HoffmanE. K.VothK.EsbensenA. J. (2023). Psychometric properties of inhibitory control measures among youth with Down syndrome. Journal of Intellectual Disability Research, 67, 753–769. https://doi.org/10.1111/jir.13043
TangB.LevineM. A.AdamekJ. H.WodkaE. L.CaffoB. S.EwenJ. B. (2023). Evaluating causal psychological models: A study of language theories of autism using a large sample. Frontiers in Psychology, 14, Article 1060525. https://doi.org/10.3389/fpsyg.2023.1060525
31.
WaterhouseL.LondonE.GillbergC. (2016). ASD validity. Review Journal of Autism and Developmental Disorders, 3, 302–329. https://doi.org/doi:10.1007/s40489-016-0085-x
32.
WechslerD. (2003). Wechsler Intelligence Scale for children, fourth edition. The Psychological Corporation.
33.
WechslerD. (2014). Wechsler Intelligence Scale for children, fifth edition. The Psychological Corporation.
WigginsL. D.BargerB.MoodyE.SokeG.PandeyJ.LevyS. (2019). Brief teport: The ADOS calibrated severity score best measures autism diagnostic symptom severity in pre-school children. Journal of Autism and Developmental Disorders, 49(7), 2999–3006. https://doi.org/10.1007/s10803-017-3072-x
36.
WilsonB. N.CrawfordS. G.GreenD.RobertsG.AylottA.KaplanB. J. (2009). Psychometric properties of the revised Developmental Coordination Disorder Questionnaire. Journal of Physical and Occupational Therapy in Pediatrics, 29, 182–202. https://doi.org/10.1080/01942630902784761
37.
WodkaE. L.PutsN. A.MahoneE. M.EddenR. A.TommerdahlM.MostofskyS. H. (2016). The role of attention in somatosensory processing: A multi-trait, multi-method analysis. Journal of Autism and Developmental Disorders, 46(10), 3232–3241. https://doi.org/10.1007/s10803-016-2866-6
