Abstract
There is a critical need for validated screening tools for autism spectrum disorder in very young children so families can access tailored intervention services as early as possible. Few screeners exist for children between the recommended screening ages of 18–24 months. This study examined the utility of a new autism-specific parent-report screening tool, the Early Screening for Autism and Communication Disorders for children 12–36 months. Field-testing was conducted from five sites with 471 children screened for communication delays in primary care or referred for familial risk or concern for autism spectrum disorder. The Early Screening for Autism and Communication Disorders was evaluated in three age groups: 12–17, 18–23, and 24–36 months. A best-estimate diagnosis of autism spectrum disorder, developmental delay, or typical development was made. Receiver operating characteristic curves were examined for all 46 items and the 30 items that best discriminated autism spectrum disorder from the non-spectrum groups. Area under the curve estimates for the total were greater than 0.90 across age groups. Cutoffs were established for each age group with sensitivity between 0.86 and 0.92 and specificity between 0.74 and 0.85. Results provide preliminary support for the validity of the Early Screening for Autism and Communication Disorders as an autism-specific screener in children 12–36 months with elevated risk of communication delay or autism spectrum disorder.
Lay abstract
There is a critical need for accurate screening tools for autism spectrum disorder in very young children so families can access tailored intervention services as early as possible. However, there are few screeners designed for children 18–24 months. Developing screeners that pick up on the signs of autism spectrum disorder in very young children has proved even more challenging. In this study, we examined a new autism-specific parent-report screening tool, the Early Screening for Autism and Communication Disorders for children between 12 and 36 months of age. Field-testing was done in five sites with 471 children screened for communication delays in primary care or referred for familial risk or concern for autism spectrum disorder. The Early Screening for Autism and Communication Disorders was tested in three age groups: 12–17, 18–23, and 24–36 months. A best-estimate diagnosis of autism spectrum disorder, developmental delay, or typical development was made. Analyses examined all 46 items and identified 30 items that best discriminated autism spectrum disorder from the non-spectrum groups. Cutoffs were established for each age group with good sensitivity and specificity. Results provide preliminary support for the accuracy of the Early Screening for Autism and Communication Disorders as an autism-specific screener in children 12–36 months with elevated risk of communication delay or autism spectrum disorder.
Keywords
Advances in research have documented reliable diagnoses of autism spectrum disorder (ASD) can be made between 14 and 24 months of age (Guthrie et al., 2013; Ozonoff et al., 2015; Pierce et al., 2019; Zwaigenbaum, Bauman, Stone, et al., 2015). Since 2007, the American Academy of Pediatrics (AAP) has recommended screening all children for ASD at 18 and 24 months (Johnson & Myers, 2007) and recently reaffirmed these recommendations (Hyman et al., 2020). Yet, the median age of diagnosis in the United States has hovered at 4–5 years for more than a decade (Maenner et al., 2020). Mounting research has shed light on limitations of available autism screeners (Carbone et al., 2020; Guthrie et al., 2019; Stenberg et al., 2014; Sturner, Howard, Bergmann, Stewart, & Afarian, 2017; Yuen et al., 2018), one factor contributing to late diagnosis. There is a critical need for screeners to improve early detection. This article reports on field-testing of a new autism screener for children 12–36 months with elevated risk of communication delay or ASD.
Limitations of existing screening approaches and tools
The Modified Checklist for Autism in Toddlers (M-CHAT) or M-CHAT-Revised (M-CHAT-R; Robins et al., 2001, 2014) is the most widely studied and utilized autism screener for toddlers. Reported sensitivity and specificity for this screener may be overestimated because, for many studies, only children receiving positive screens completed diagnostic evaluations (Zwaigenbaum, Bauman, Fein, et al., 2015). Studies by Robins and colleagues have reported estimated sensitivities over 0.80 (Chlebowski et al., 2013; Kleinman et al., 2008; Robins et al., 2001, 2014) based on screening children referred for suspected ASD, receiving early intervention (EI), or with familial risk as well as in primary care. However, more recent large studies conducted in primary care have shown limitations in the performance of the M-CHAT/M-CHAT-R (Carbone et al., 2020; Guthrie et al., 2019; Stenberg et al., 2014, 2021; Sturner, Howard, Bergmann, Morrel, et al., 2017; Sturner, Howard, Bergmann, Stewart, & Afarian, 2017).
Robins et al. (2014) concluded that the M-CHAT should be used with the Follow-Up interview (i.e., M-CHAT/F) to reduce false positives and avoid unnecessary referrals, as positive predictive value (PPV) increased from 0.36 to 0.74 with the Follow-Up interview. However, in a review of electronic medical records of universal screening using the M-CHAT/F between 16 and 24 months in a large pediatric practice (
Research characterizing children accurately identified by the M-CHAT/M-CHAT-R has indicated that developmental level and age are important factors. Robins et al. (2014) reported that their ASD sample had below average developmental levels, with standard scores approximately two standard deviations below the mean. Results of a meta-analysis of 19 studies comparing outcomes of 1,658 children with ASD ascertained using the M-CHAT indicated that developmental level was significantly higher in infant sibling samples (
Research on the M-CHAT/M-CHAT-R in primary care suggests it performs better in children 24–36 months compared to children under 24 months (Guthrie et al., 2019; Pandey et al., 2008; Sturner, Howard, Bergmann, Stewart, & Afarian, 2017; Toh et al., 2018; Yuen et al., 2018). For example, Guthrie et al. (2019) reported a sensitivity of 0.35 for children 16–20 months, compared to 0.49 for children 21–26 months. With regard to developmental level, in another population study, Stenberg et al. (2021) reported that at 18 months of age, the M-CHAT identified only 28.8% of children with ASD and identified 81.3% of children with intellectual disability without ASD. Collectively, these findings indicate that, in general population samples, the M-CHAT may be missing far more children with ASD than it is detecting at 18–24 months and identifying more children with developmental delays (DDs) without ASD, highlighting the need for improved screeners at younger ages and for primary care settings.
The new AAP clinical report (Hyman et al., 2020) identified two promising autism screening tools for children 18–30 months of age, the Parents Observations of Social Interactions (POSI; Smith et al., 2013) and the Early Screening for Autism and Communication Disorders (ESAC). The POSI is a brief questionnaire been studied with preliminary community-referred samples, showing favorable comparison on sensitivity, but lower performance on specificity than the M-CHAT (Salisbury et al., 2018; Smith et al., 2013). Additional validation in larger samples is needed to understand psychometric properties and clinical utility of the POSI. The ESAC is a parent-report tool developed with an initial set of 46 items to capture the heterogeneity of delayed social communication skills and early signs of autism symptoms. Preliminary research on the development of ESAC items revealed a group difference with a large effect (
Accurately screening for ASD younger than 18 months is even more challenging. Research on the Early Screening of Autistic Traits Questionnaire (Dietz et al., 2006), designed to screen at 14–15 months in primary care, indicated a detection rate of 0.6/1,000. Similarly, the First Year Inventory for 12-month-olds (Reznick et al., 2007) demonstrated a sensitivity of 0.44 (Turner-Brown et al., 2013). While these screeners show promise, the low accuracy does not yet support clinical use.
Most children with ASD exhibit observable symptoms by 12–24 months (Bacon et al., 2018; Pierce et al., 2019; Zwaigenbaum, Bauman, Stone, et al., 2015), suggesting the potential of screening under 24 months. Existing tools may have limited utility capturing the heterogeneity of early behavioral manifestations of ASD, including symptoms in restricted, repetitive behaviors (RRBs) observed at or before 12–24 months (Dow et al., 2017, 2020; Elison et al., 2014; Kim & Lord, 2010; Richler et al., 2007; Watt et al., 2008; Wolff et al., 2014). The accuracy of autism screeners may be improved by incorporating questions capturing the range of ASD symptoms in young children.
Purpose of this study
The ESAC addresses the need for a more accurate and comprehensive parent-report screening tool by incorporating questions about early social communication and the presence of RRBs, based on core diagnostic features of ASD (American Psychiatric Association [APA], 2013). This study reports the performance of the ESAC during field-testing with children 12–36 months of age recruited from five sites using varied ascertainment sources. We recruited children with parental or professional concerns for ASD or DD, familial-risk, or positive primary care screening for communication delay yielding a diverse sample of children with elevated risk for ASD. The first aim of this study was to examine item-level performance in order to identify the items that best differentiated these children with ASD and without ASD. The second aim was to examine the preliminary psychometric properties of the best set of ESAC items and performance across three age groups in this elevated risk sample: 12–17, 18–23, and 24–36 months.
Methods
Procedures for sample selection and recruitment
Participants were 471 toddlers 12–36 months of age, recruited from five sites: Florida State University (FSU;
Parents were asked to complete the ESAC prior to or at the time of the developmental or diagnostic evaluation, before most families received the diagnosis. Only nine ESACs were completed more than a month after the autism diagnostic evaluation. Some children were seen longitudinally, resulting in multiple ESACs. The earliest ESAC within each age group was chosen, yielding 596 ESACs (12–17 months:
Sample demographic and developmental characteristics are summarized in Table 1. Sex was reported for all but three children. Race and ethnicity were available for 73% (
Sample characteristics by diagnostic group.
ASD: autism spectrum disorder; DD: developmentally delayed; TD: typically developing; NS: non-spectrum; NR: not reported; ADOS: Autism Diagnostic Observation Schedule; SA: Social Affect; CSS: calibrated severity score; RRB: restrictive/repetitive behavior; MSEL: Mullen Scales of Early Learning; VR: Visual Recognition; FM: Fine Motor; RL: Receptive Language; EL: Expressive Language; ELC: Early Learning Composite; DQ: developmental quotient; NS groups include DD and TD children. MSEL
MSEL subscale and composite scores were not reported for
Measures
ESAC
The ESAC was initially developed as an autism screener to follow up broadband screeners like the ITC, an AAP-recommended screener (Hyman et al., 2020; Johnson & Myers, 2007) for communication delays including ASD. The ESAC field-test version included 46 items generated by three authors (A.M.W., J.W., and C.L.) covering ASD symptom domains (APA, 2013) to measure lack of typical social communication development and presence of RRBs. The majority of items (35) were rated on a 3-point scale as occurring
MSEL
The MSEL is a standardized assessment of developmental level. Standard
ADOS
The ADOS is a semi-structured, standardized assessment that measures symptoms of ASD and provides domain and total scores. Children received different modules based on age and verbal ability. To allow comparison of scores across modules, calibrated severity scores (CSSs) were derived (Esler et al., 2015; Gotham et al., 2009; Hus et al., 2014).
Statistical analyses
Receiver operating characteristic (ROC) curves, based on logistic regressions, were estimated to generate area under the curve (AUC) values and examine performance of each item in discriminating between the ASD and NS groups.
For items with derived sums, ROC analysis results were used to determine item-level cutoffs that would convert scores into the same 3-point rating scale. Criteria for assigning a “2” included a specificity value as close to 0.90 as possible and sensitivity ⩾0.60. The cutoff for a “1” was based on optimal sensitivity and specificity (i.e., as close to 0.80 as possible). Cutoff values for these items were examined separately across age groups.
Using the best performing items, ROC curves were estimated with the sum of retained items creating an ESAC total to generate AUC and sensitivity/specificity values. Curves were estimated separately across age groups to compare optimal cutoffs and estimate PPVs and negative predictive values (NPVs). Internal consistency of the ESAC was estimated using coefficient alpha. Test–retest reliability was approximated for the subsample with two ESAC administrations (
Community involvement
There was no community involvement in this study.
Results
Site comparisons of children with ASD
The 121 children recruited from primary care samples (FSU, UCSD) were significantly younger than the 62 recruited from referred samples (NIMH, UM, UCD;
Item-level analyses
Item-level results are presented in Table 2. Twenty items had AUC values less than 0.65. We chose to retain four of these items because they captured the heterogeneity of clinically relevant RRBs (Items 32, 41, 44, 46; endorsing these items likely indicates clinical concern for ASD but frequency of endorsement is low). A total of 30 items were retained. Cutoffs, sensitivity, and specificity values for the seven continuous items are presented in Table 3. Because these items had different ranges (see Table 2), the values that met the criteria outlined above for a “1” and “2” varied across items and age. Different cutoff values were identified across age groups for three items (21, 31, and 37); cutoffs did not vary by age for the remaining items.
Area under the curve and item-level information across diagnostic groups.
AUC: area under curve; ASD: autism spectrum disorder.
Bolded items were retained for the final item set.
Cutoff values for retained continuous items based on receiver operating characteristic analysis.
Psychometric properties of final ESAC item set
ROC curves showed high AUC values, indicating the ESAC performed consistently well across age groups (Table 4); however, optimal cutoffs differed by age group. Within the 12- to 17-month subsample a cutoff of 20 yielded balanced sensitivity (0.86) and specificity (0.84). The cutoff of 18 yielded the strongest sensitivity (0.87–0.88) and specificity (0.84–0.85) estimates for both the 18- to 23-month and 24- to 36-month subsamples. Estimated PPVs were highest for the oldest age group and NPVs were comparable across ages (Table 4).
ESAC total score receiver operating characteristic analysis results.
AUC: area under curve; CI: confidence interval; PPV: positive predictive value; NPV: negative predictive value.
Developmental characteristics by screening status
ESAC Totals and developmental characteristics by screening status across age groups are reported in Table 5. A majority of children with ASD screened positive on the ESAC across age groups. Children with false negatives demonstrated a pattern of higher MSEL scores on average, compared to the children with true positives. Across age groups, there were only 28 false negatives of the 596 screens, and most of these were from the primary care sample. Children with true negatives demonstrated a pattern of higher MSEL scores on average than children with false positives. Across age groups, 52%–77% of the children with false positives had a classification of DD. Furthermore, the average time between screening and diagnosis based on the ADOS decreased with age (12–17 months:
Developmental characteristics by screening status.
SD: standard deviation; ESAC: Early Screening for Autism and Communication Disorders; ADOS: Autism Diagnostic Observation Schedule; SA: Social Affect; CSS: calibrated severity score; RRB: restrictive/repetitive behavior; MSEL: Mullen Scales of Early Learning; VR: Visual Recognition; FM: Fine Motor; RL: Receptive Language; EL: Expressive Language; ELC: Early Learning Composite.
MSEL
Internal consistency and test–retest reliability
Coefficient alphas for the final ESAC item set ranged from 0.92 to 0.95, indicating good internal consistency. For the subsample with greater than or equal to two administrations (
Discussion
This study examined ESAC results from preliminary field-testing of children with elevated risk for ASD recruited from primary care and referred samples. ROC curves were examined to determine the items that best discriminated children with and without ASD. AUCs exceeded 0.90 for total scores of the best 30 items, and preliminary cutoffs were established for the 30-item ESAC across age groups with promising sensitivity and specificity. A cutoff score was recommended at 20 for 12–17 months and 18 for 18–36 months. Results provide initial support for validity of the ESAC as an autism screener for children 12–36 months with elevated risk, accurately identifying over 80% of children with ASD in this sample. The ESAC had strong psychometric properties in the AAP-recommended screening age range of 18–24 months, as well as adequate sensitivity down to 12 months, which is particularly promising given challenges of accurately screening for ASD under 24 months.
The ESAC was more accurate in a sample with a higher developmental level by 18 points on the MSEL Early Learning Composite (ELC), than children with ASD identified through community referrals as well as universal screening with the M-CHAT (see Micheletti et al., 2019). Likewise, MSEL average
This study extends the findings of Wetherby et al. (2008) and Pierce et al. (2011) by documenting the accuracy of the ESAC as an autism-specific parent-report screener to follow up a broadband screen. Over 75% of our sample were recruited using the ITC in primary care at two sites (FSU and UCSD) and were younger, on average, at age of screening at these sites. Sensitivity and specificity of the ESAC were better than psychometric properties reported in early validation studies of the M-CHAT (Kleinman et al., 2008; Robins et al., 2001), which included children screened in primary care and those referred through EI. Importantly, our sample included a large number of children screened under 24 months. Most M-CHATs in Robins and colleagues’ (2001) early study were administered to children at 18- or 24-month well-child visits, though they did not report the mean age at screening. Furthermore, only children who were detected by their M-CHAT score or by the primary care provider concerns were invited for a diagnostic evaluation; therefore, the actual sensitivity is likely to be lower than what was reported, given that children who screened negative and did not have provider concerns were assumed to not have ASD in specificity calculations. Studies of wide-scale population use of the M-CHAT with more systematic follow-up of screen negatives have yielded poorer psychometric properties (Carbone et al., 2020; Guthrie et al., 2019). A large-scale study of the ESAC with an independent sample recruited from primary care is underway and is critical for further validation.
Our findings highlight the importance of examining performance across different age ranges from 12 to 36 months to retain adequate psychometric properties while still detecting children who present with “subtler” differences in social communication and RRBs (Sturner, Howard, Bergmann, Morrel, et al., 2017; Sturner, Howard, Bergmann, Stewart, & Afarian, 2017). Consistent with research on other screeners, the ESAC’s PPV increased with age. However, it was encouraging that the ESAC maintained strong psychometric properties across the age ranges considering the greater amount of time between screening and diagnosis in the youngest screening group. Interestingly, among the continuous items, some cutoff scores were observed to change with age and some did not. Different cutoffs were needed for early developing social communication skills, including gestures, play with objects, and pretend play with toys. Conversely, no adjustments to cutoff scores were needed for items that tapped into features specific to ASD like sensory responses, repetitive behaviors, insistence on sameness, and intense interests.
Finally, this study demonstrates the accuracy of parent report to screen young children. Parent concern, by itself, may be less accurate for children at younger ages (Wetherby et al., 2008). Retrospective and prospective studies show that about 75% of parents of children with ASD have concerns by 24 months of age, 50% at 18 months, and 30% at 12 months (Chawarska, Paul et al., 2007; Wetherby et al., 2008). Results of this study support that parents are accurate at reporting what their child does and does not do, whether they are concerned or not. Furthermore, for the subsample with two ESACs, a strong association was observed between total scores, supporting the stability of ESAC scores over time.
Strengths, limitations, and future directions
Strengths of this study include a large sample size of children recruited from five sites and prospectively followed and evaluated using gold-standard measures for ASD. Our sample was large enough to examine three separate age intervals, with ample sampling of positive and negative screens and a variety of ascertainment methods. Despite these strengths, our field-test trial has limitations. Because our sample had elevated risk for ASD, the reported accuracy of the ESAC may be higher than in a low-risk primary care sample. Group differences in developmental level may have contributed to the high discrimination among groups. The TD group had above-average developmental skills, and despite being higher than previous screening studies, the developmental level of the ASD group was lower than the DD and TD groups. Thus, it will be important for follow-up studies to examine contributions of developmental differences to the ESAC’s accuracy and specifically test the ESAC’s predictive utility in children with low and high developmental level.
Demographic information was not available for a substantial portion of children and this sample may be limited in racial, ethnic, and socioeconomic diversity. In addition, the full diagnostic battery was not completed for all children in the NS group. Given the small sample sizes at some sites, the performance of the ESAC could not be examined separately across sites and ascertainment method. Finally, in order to increase sample size and resulting power, multiple ESACs (if available) from the same child were included across (but not within) age groups. While assumptions of independence were not violated, in the older age groups, about a third of the ESACs were repeats, which may bias its accuracy. However, given AAP recommendation that children be screened for ASD at 18- and 24-month well-child visits, it is important that accuracy be established for initial and repeated administrations of screening tools.
Future directions include examining the final 30-item set of the ESAC across new independent samples of children at 12–36 months and the performance of the ESAC as an autism screener in a low-risk primary care sample. Further examination of psychometric features is needed, including replication with new samples, test–retest reliability, and a factor analysis to further establish validity. Research is needed to study parent-report screeners with follow-up observational measures (e.g., Dow et al., 2020) to accelerate referrals for diagnosis and eligibility for early intervention.
Conclusion
Results of this study provide preliminary support for the validity of the ESAC as an autism screener for children 12–36 months of age with elevated risk and add to research documenting the accuracy of parent report to screen young children. Use of a parent-report screening tool like the ESAC minimizes the time required of healthcare providers, maximizes the role of the family, and provides reasonably accurate information about whether to refer for further evaluation. These findings offer promise for this new screener to more accurately identify more children at younger ages and contribute to reducing the age of diagnosis and access to intervention.
Footnotes
Declaration of conflicting interests
The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: A.M.W. is co-author of the Communication and Symbolic Behavior Scales and receives royalties but not from this study. Catherine Lord is author of the Autism Diagnostic Observation Schedule–Second Edition (ADOS-2). C.L. and W.G. are authors of the ADOS Toddler Module (ADOS-T). They receive royalties from use of the ADOS-2/ADOS-T, but not from this study. The remaining authors have no financial relationships relevant to this article to disclose.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported in part by the Eunice Kennedy Shriver National Institute of Child Health & Human Development grants RO1HD078410 and RO1HD065272 (PI: A.M.W.), the National Institute on Deafness and Other Communication Disorders grant R01DC007462 (PI: A.M.W.), and the Centers for Disease Control and Prevention Cooperative Agreement U01DD000304 (PI: A.M.W.), in part by National Institute of Mental Health completed as part of protocol 06-M-0065, NCT00271622 under ZIAMH002868. The content is solely the responsibility of the authors and does not necessarily represent the official views of these federal agencies.
