Abstract
Parents typically recognize developmental problems associated with autism in their child by 2 years of age [1–4]. Although symptoms are recognizable in infancy, the average age of diagnosis has been reported as ranging from 3.6 years to almost 7 years of age [2], [5], [6]. Indications of the potential efficacy of early intervention [7] highlight the importance of early diagnosis. Early diagnosis in infants and preschool children is dependant upon the identification of the early features of autism.
There are a number of symptoms unique to young children with autism [8]. A range of studies which compared a broad range of symptoms in infants and preschool aged children with autism to children with developmental delay without autism (see 9–19]. However, fewer studies identified specific early features in the area of stereotyped behaviours and routines [9], [13], [14]. This relative absence or low frequency in preschool children of some diagnostic features that are seen in older children may indicate a developmental process in the emergence of symptomatology, with some features requiring a greater level of maturation [20].
Early features of autism: differentiating behaviours relating to standard diagnostic criteria
dev delay, developmental delay; typical dev, typically developing; pop repres, population representative; unk, unknown, information not provided in article for autism sample. †retrospective interview.
The use of screening questionnaires by primary care physicians and early childhood professionals currently offers the best method for the early identification of children with autism. A dual-level approach to the assessment and diagnosis of autism has been recommended, with level 1 requiring the routine screening of preschool children for developmental disorders by primary care providers [21]. Knowledge of the early features of autism and the stability of early diagnosis [8] has now paved the way for the development of autism screening tools for preschool children.
Screening instruments
Existing instruments which have been assessed for their capacity to differentiate autism from other disorders include the Checklist for Autism in Toddlers [22], the Social Communication Questionnaire [23], the Screening Tool for Autism in Two-year olds (STAT) [24], and the Developmental Behaviour Checklist [25].
The parent completed Social Communication Questionnaire (SCQ) consists of 40 questions based on the Autism Diagnostic Interview-Revised (ADI-R) [13]. It has been shown to differentiate between autism and mental retardation (area under the curve = 0.916) [23]. The capacity of the SCQ to differentiate young children under 4 years of age with autism from those with developmental delay has not been established.
The Developmental Behaviour Checklist (Primary carer version; DBC-P) [26], [27] is a 96 item parent or carer completed checklist designed to measure behavioural and emotional disturbance in children and adolescents with intellectual disability. The DBC-P has been evaluated in terms of its ability to distinguish between children and adolescents with autism and those with intellectual disability without autism [25]. Twenty-nine items of the DBC-P best discriminated between those with autism and those with intellectual disability without autism. It was concluded that the DBC-P is a sensitive screening tool for autism in intellectually disabled children and adolescents (aged 4–18 years), with an area under the curve (AUC) of 0.80, sensitivity of 0.86, and specificity of 0.69.
The Checklist for Autism in Toddlers (CHAT) [22] was developed to identify children at risk of autism at 18 months of age. The CHAT has been designed to be used by general practitioners or health visitors at a child's 18 month checkup. It consists of two sections; the first contains nine yes/no questions which the clinician asks the parent. The second section consists of clinician observations of five child behaviours or skills. It has been shown to be able to distinguish between children with autism, developmental delay, and normal children [22], and autism and other developmental delays [28].
A long-term follow-up study has established a sensitivity of 0.38, specificity of 0.98, and positive predictive value of 0.047 [29]. Repeating the CHAT 1 month after the first administration increased the positive predictive value to 0.75, but while the specificity was high (100%), sensitivity was low (0.18).
The M-CHAT is a modified version of the CHAT and consists of 23 yes/no items completed by parents [30]. It retains the original 9 CHAT items, along with 14 additional items generated through a literature search and clinical experience. The M-CHAT has been used to screen 1122 children at their 18 month well-baby checkup (18–25 months of age) and a high-risk sample of 171 children in receipt of early intervention services (aged 18–30 months) [30]. Six items were found to best differentiate the children with autism from those without autism, with a sensitivity of 0.87, specificity of 0.99, positive predictive power of 0.80 and negative predictive power of 0.99. There is currently no research on the ability of the M-CHAT to discriminate children with developmental delay with autism from those with developmental delay without autism.
The STAT [24] consists of 12 items assessing imitation, play, and communication. The STAT is an interactional assessment of children aged 24–35 months, takes approximately 20 minutes to complete, and is administered by a trained clinician. A sample of 40 children (7 with autism) aged 27–35 months was used to establish the scoring algorithm [24]. A further sample of 33 children (12 with autism) aged 24–35 months was used to validate the screening algorithm. The algorithm correctly identified 100% of the children with autism and 91% of the nonautistic children in the development sample. Application of the algorithm to the second sample resulted in correct identification of 83% of the autism sample and 86% of the nonautistic sample. Although this study provides preliminary support for the efficacy of the STAT, small sample sizes warrant caution in the interpretation of the results and necessitate replication with larger samples.
It is usually not feasible to screen entire populations of children, therefore screening could be focused on ‘atrisk’ populations of preschool children with developmental delay and language delay. A feasible screening methodology would need to be implemented within the system of primary health care professionals who conduct regular health checks on young children, such as paediatricians, maternal and child health nurses, and general practitioners. An ideal screening questionnaire would be brief and completed by parents. In the field of autism, it is essential that a screening tool be able to differentiate children with autism from those with developmental delay without autism.
In terms of these criteria, the CHAT [22] and the STAT [24] require professional administration and their psychometric properties require further definition. The DBC-P and the SCQ are completed by parents. However, it has not yet been demonstrated that either of these questionnaires is capable of differentiating between children younger than 4 years with autism from those with developmental delay or intellectual disability without autism. This study therefore aimed to evaluate the DBC-P as a screening tool for autism in young children with developmental delay aged 18–48 months.
Method
Design
Items of the DBC-P which differentiated a sample of children aged 18–48 months with autism and developmental delay from a group of children with developmental delay who did not have autism were identified. An autism screening algorithm was produced and sensitivity and specificity of various cut-off points were defined.
Participants
The children with autism were recruited from five experienced regional autism assessment teams in Victoria and New South Wales (Australia). Forty-one of the autism subjects (68.3%) were diagnosed by one of two clinicians who have a high interrater reliability in terms of diagnosis (Cohen kappa = 0.95, calculated on a sample of 107 cases of autistic disorder) [25]. The remaining cases were diagnosed by clinicians experienced in the assessment and diagnosis of young children with autism. Independent case file reviews were conducted by the authors to confirm that DSM-IV [31] criteria for Autistic Disorder were met for each subject.
The children with developmental delay without autism were recruited through early intervention services in Victoria and New South Wales, Australia. Each case was reviewed in order to confirm that the child was developmentally delayed and that autism was neither diagnosed nor suspected. For cases where there was any doubt about a diagnosis, an independent review was conducted by a child psychiatrist experienced in developmental delay and autism. If doubts remained, the case was not included in the study sample.
A diagnosis of developmental delay was established by contacting health professionals involved in the assessment of each subject to obtain reports of any developmental assessments. A variety of assessment tools were used, such as the Bayley Scales of Infant Development [32], McCarthy Scales of Children's Abilities [33], Merrill-Palmer Scale [34], Gesell Developmental Schedule [35], [36], Leiter International Performance Scale-Revised [37], and the Psychoeducational Profile-Revised [38]. Developmental delay, that is global cognitive delay or significant language delay, established by a standardized assessment (that is, a standard score 2 of more standard deviations below the mean on an assessment tool) or diagnosed by an experienced health professional, was a requirement for inclusion in either group. Children with physical disabilities only were not included in the sample.
Measures
Developmental Behaviour Checklist (Primary carer version, DBC-P)
The DBC-P [26], [27] is a 96 item parent completed checklist designed to assess behavioural and emotional problems in children aged 4–18 years with intellectual disability. The items are scored on a 0–2 rating scale, where 0 = ‘not true as far as you know’, 1 = ‘somewhat or sometimes true’, and 2 = ‘very true or often true’. Parents are asked to rate the items in terms of their child's behaviour in the past 6 months. A total score can be calculated along with scores on 5 factor analytically derived subscales [27].
The DBC-P has high reliability between parents (ICC = 0.80) [27]. Internal consistency is also high (〈 = 0.941). High correlations between the DBC-P Total Behaviour Problem Score and other, professionally administered, measures of behavioural disturbance in children with intellectual disability have been found, providing evidence of concurrent validity [27].
The DBC-P was completed by the parents of the all children. Although standardized norms for the DBC-P have not been established for children aged less than 4 years, clinical experience of the usefulness of this instrument in this age range by the authors led to its use in this study. As normative comparisons were not required for the purposes of this study, the lack of normative data in this age range was not an issue.
Analyses
As the DBC-P was developed for use in children and adolescents aged 4–18 years, it was predicted that some items would be either irrelevant or inappropriate for the age group involved in this study. The first stage of data analysis therefore involved determining which items of the DBC-P were not relevant for the study population. This was accomplished by establishing the frequencies of each item for the entire sample. Those items which were endorsed by less than 75% of the parents, that is those items that were given a score of zero (‘not true as far as you know’) in 75% or more cases, were excluded from further analyses.
Univariate logistic regressions were performed to establish which of the remaining items of the DBC-P differentiated the autism and control groups. Confirmatory factor analysis was performed with those items from the univariate logistic regressions which significantly differentiated the two groups. Factor loadings were then used to develop the DBC-P early autism screening algorithm (DBC-Early Screen).
Receiver operating characteristics (ROC) analysis was used to evaluate the overall performance of DBC-Early Screen as a screening tool for autism and to determine cut-off points. The area under the ROC curve provides a global assessment of the performance of the test or diagnostic accuracy. Sensitivity (proportion of true positives correctly identified by the test) and specificity (proportion of true negatives correctly identified by the test) were both calculated [39]. Positive predictive values (proportion of subjects with positive test results who are correctly diagnosed) and negative predictive values (proportion of subjects with negative test results who are correctly diagnosed) were also calculated [40].
Results
Sample characteristics
The autism and developmental delay group consisted of 60 children, with a DSM-IV [31] diagnosis of autistic disorder. The developmental delay control group consisted of 60 children, none of whom had a diagnosis of autism. All of the children in both groups had developmental delay. The mean age of the autism group was 40.31 months (range = 22–48 months, SD = 5.96), and the mean age of the developmentally delayed group was 35.92 months (range = 19–47 months, SD = 7.57). The autism and developmental delay group was significantly older than the control group, t (111.88) = −3.53, p = 0.001. Eighty-two percent of the autism group were male, and 67% of the developmental delay group.
Suitability of DBC-P items
Frequencies were run on the total sample for 95 items of the DBC-P. There were 30 items scored ‘not true as far as you know’ (a score of zero) by 75% or more of parents. These items were excluded from further analyses because they were considered developmentally inappropriate for the age of the sample. They included items which required a degree of language competence, toileting skills, and symptoms (e.g. delusions and hallucinations) which are rarely seen in infants and toddlers. The remaining 65 DBC-P items were included in the next stage of analyses.
Univariate logistic regressions
The 65 DBC-P items were each individually entered into a series of univariate logistic regressions. A conservative significance level of less than or equal to.01 was chosen in order to take into account the effects of multiple comparisons. Thirty DBC-P items were found to be significant at the level of p ≤ 0.01. These items are listed in Table 2.
Significantly differentiating DBC-P items: estimated coefficients (β), wald statistics, p values, and odds ratios
Development of the screening algorithm
Confirmatory factor analysis was used to extract a single common factor from the 30 items of the DBC-P which were found to be significant at the p ≤ 0.01 level in the univariate logistic regression analyses. The factor analysis also included the binary group membership (‘autism’) variable. Loadings of the DBC-P items were permitted to load freely on the common factor, while the loading of the group membership variable was constrained to 1.00. The effect of this constraint is to align the factor with autism. Polychoric correlations were calculated between the three response point (0, 1, 2) DBC-P items and with the binary variable autism. Polychoric correlations assume that a normally distributed variable underlies each observed variable [41]. The use of polychoric correlations in the factor analysis ensured that any differences in the proportion of respondents endorsing each of the items did not affect the magnitude of the correlation coefficient [42], [43].
Mplus 2.01 [41] was used to calculate parameter estimates for the model using the weighted least square estimator with mean and variance-adjusted chi-square test statistic. The adequacy of model fit was assessed by the chi-square statistic (χ2), the ratio of the chi-square statistic to the degrees of freedom (χ2/DF), the Comparative Fit Index (CFI), the Non-Normed Fit Index (NNFI) [44] and the Root Mean Square Error of Approximation (RMSEA) [45]. Values ofχ2/DF below 2 or 3 are regarded as denoting adequate fit [46]. Values larger than 0.90 are desirable on the CFI and NNFI. Browne and Cudeck [45] have suggested that values of the RMSEA below 0.05 are indicative of acceptable model fit.
The following indices reflect the fit of the model to the data:χ2(62) = 142.661,χ2/DF = 2.29, NNFI = 0.92, CFI = 0.88, RMSEA = 0.105. These generally showed that the model provided a satisfactory fit, but as the sole aim of this analysis was to determine the loadings of the items on the autism factor in order to develop a screening algorithm, ways in which it might be improved are not relevant.
Item loadings are shown in Table 3. These loadings can be interpreted as the correlation of DBC-P items with a dimension that can be described as liability to autism. All loadings were substantial in size and significantly greater than zero, reflecting the basis upon which items were chosen for inclusion in the confirmatory factor analysis.
Confirmatory factor analysis: DBC-P item loadings
Using an item loading cut point of 0.60 or greater, 17 DBC-P items were selected to create a DBC-P autism screening algorithm (DBCEarly Screen). These 17 items and their loadings are described in Table 3. A cut point of 0.60 produced a sufficiently short screening tool, but retained items which had face validity and were considered clinically important in the diagnosis of autism in young children.
Using the item loadings in a weighted screening algorithm produced an autism screening score which ranged from 0 to 23.02. An unweighted screening score was also generated by summing the selected items. This produced a screening score ranging from 0 to 34. Receiver Operating Characteristics (ROC) curves were generated for both the weighted and unweighted screening algorithms in order to evaluate the performance of each and assign cut-off scores for screening.
An ROC curve was generated for the weighted 17-item DBC-Early Screen score. An Area Under the Curve (AUC) of 0.874 (SE = 0.032, 95% CI = 0.810–0.938) was obtained, indicating good overall performance of the weighted screening algorithm.
An ROC curve was also generated for the unweighted 17-item DBC-Early Screen score. An AUC of 0.871 (SE = 0.033, 95% CI = 0.806–0.936) was obtained, indicating good overall performance of the unweighted screening algorithm.
The range of potential cut points were studied for both the weighted and unweighted screening algorithms in order to optimize both sensitivity and specificity. Higher sensitivity was favoured, whilst attempting to maintaining specificity as close to 0.70 as possible. Cut-off points were selected for both the weighted and unweighted 17-item DBC-Early Screen scores. For the weighted DBC-Early Screen score, a cut-off point of greater than or equal to 6.995 was selected and greater than or equal to 10.500 for the unweighted DBC-Early Screen score. Sensitivity, specificity, correct classification rate, predictive value of a positive test, and predictive value of a negative test were calculated for both the weighted and unweighted DBC-Early Screen scores using these cut-off points (see Table 4).
Sensitivity, specificity, correct classification rate, predictive value of positive test, and predictive value of negative test for the weighted (cut-off of ≥6.995) and unweighted (cut-off of ≥10.500) 17-item DBC-Early Screen scores
Discussion
Thirty items of the DBC-P were found to differentiate the infants and preschool children with autism and developmental delay from those with developmental delay without autism (see Table 2). Consistent with other studies on the early features of autism in infants and preschool children, this study identified a number of features in the areas of social interaction and communication which differentiated the children with autism and developmental delay from those with developmental delay without autism. In contrast to a number of studies, this study did find behaviours in the area of restricted, repetitive and stereotyped patterns of behaviour, interests and activities that significantly differentiated the groups of young children.
This inconsistency may be attributable to age differences in the samples studied. Three of the studies which did not find differences in the area of stereotyped behaviours studied children who were less than 24 months of age [11], [12], [19] (see 9] and 36–59 months [13], [14]. Therefore it appears likely, as has been suggested, that such behaviours develop in the third to fourth years [10]. To determine the presence of a developmental sequence in the emergence of autism symptoms further longitudinal studies are needed beginning with sufficiently large samples of children aged less than 24 months.
Previous research using the DBC-P to differentiate between children and adolescents (aged 4–18 years) with intellectual disability without autism and those with intellectual disability with autism resulted in 54 items which significantly differentiated the two groups [25]. Of the 30 items found to significantly differentiate the two groups in the current study, only five items were not amongst the 54 items which were found to differentiate the older age group. These included: (i) switches lights on and off, pours water over and over; or similar repetitive activity; (ii) noisy or boisterous; (iii) whines or complains a lot; (iv) becomes overexcited; and (v) excessively distressed if separated from familiar person. Conversely, 29 items which were found to differentiate the older groups with and without autism, did not differentiate the younger age groups.
The current study used a factor loading cut-off of 0.60 for inclusion of items in the screening algorithm, resulting in 17 items making up the screening score. The Brereton et al. [25] study used a cut-off of 0.70, resulting in a 26 item screening algorithm. A further three items (with factor loadings = 0.62) were also added to the algorithm which were considered core symptoms of autism, resulting in a screening algorithm of 29 items of the DBC-P. The screening algorithm of the current study shares 14 of its 17 items with the screening algorithm for 4–18 years olds. It has three items which are not in the algorithm for use with older children, namely: (i) switches lights on and off, pours water over and over; or similar repetitive activity; (ii) stubborn, disobedient or uncooperative; and (iii) doesn't show affection. The algorithm for use with 4–18 years olds has an additional 15 items which are not in the screening algorithm for younger children and infants.
Whilst identical analytical techniques were used between the current study and the study by Brereton et al. [25] it is clear that although there are similarities in items which differentiate 18–48 month old children with autism and developmental delay from those with developmental delay without autism and 4–18 years old children and adolescents with and without autism, there are also sufficient differences to indicate that the presentation of autism in infants and young children is indeed different to that of older children and adolescents. Of particular note is the wider range of behaviours which differentiate the older children and adolescents with autism from those with intellectual disability without autism, and the presence of more behaviours in the area of restricted, repetitive and stereotyped patterns of behaviour, interests and activities. This wider range of behaviours may be a reflection of an increasing repertoire of behaviour as children age and develop.
This study indicates that the DBC-P is a potentially useful screening tool for autism in children aged 18–48 months with developmental delay. Both the weighted and unweighted DBC-Early Screen scores performed well. The sensitivity, specificity, and positive predictive values obtained are acceptable for developmental screening [47–49] for both the weighted and unweighted autism screening algorithms. Cut-off points were chosen in order to optimize both sensitivity and specificity, but choice of cut-off point depends upon the purpose of the screening. The main purpose would usually be screening at risk populations for referral to scarce specialist diagnostic services. Therefore, higher sensitivity is necessary, whilst attempting to maintain specificity as close to 0.70 as possible.
There was no difference between the psychometric performance of the weighted and unweighted DBCEarly Screen scores. The unweighted screening score has clear practical advantages over the weighted screen score in terms of scoring. Therefore an unweighted score of 11 or more using the 17 items of the DBC-P which comprise the early autism screen will provide the best indication of the need to refer for specialist assessment in most circumstances.
The requirements for a screening tool have been defined as good specificity and sensitivity, simplicity, low cost, acceptability, and safety [50]. The DBC Early Screen has been demonstrated to have good sensitivity and specificity, it is quick and easy for a parent to complete, it is brief and inexpensive to produce, and is acceptable to both parents and clinicians. There are no risks to the patient in completing the checklist.
A screening tool is not diagnostic. The purpose of this screening tool is to identify those infants and children with developmental delay who are displaying symptoms which indicate the likelihood of autism. Conversely, a negative screen result does not rule out the possibility of autism. Screening tools should never be used in isolation from clinical observation and judgement. The planned role of this autism screening tool is to assist early childhood primary care professionals with the identification of developmentally delayed young children who may benefit from a referral to a specialist autism assessment service.
This study has some limitations. The early autism screening algorithm was developed using data from children with developmental delay. It is possible that the screening tool can identify children without cognitive delays at risk for autism, however, this has not been tested. It is possible that high functioning children with autism are more difficult to identify at an early age than those with developmental delay in addition to autism, but their delayed or deviant language development should bring them to the attention of early childhood services. Children with Asperger's disorder are less likely to present early because they do not have developmental or language delay [31]. The DBC-Early Screen was not designed to screen for Asperger's disorder.
The autism and developmental delay samples are representative of young children with autism diagnosed by Australian autism assessment clinics, therefore it is unlikely that there was any specific bias in the autism sample. It is possible that there were non-specific biases in the control group of children with developmental delay without autism. The parents of children in this group all received an invitation to participate in a study of developmental delay from the early intervention service their child attended. Parents were not directly contacted and asked to participate by the authors of the study. There may have been a bias in the way in which the study was introduced to parents by early intervention staff and in the motivation for parents to participate in the project.
It is important to note that the properties of screening tool efficacy (sensitivity, specificity, AUC) are generally inflated estimates, and that cross validation studies of instruments are required. With the exception of the CHAT study [29], evaluations of autism screening tools have been done with the parents of children who have already taken part in the assessment process and received a diagnosis of autism. In addition to cross validation studies of autism screening tools it is therefore also essential that more prospective studies of screening tools are undertaken, in order to better evaluate how these tools perform in the ‘real world’. A large scale community field trial evaluation of the DBC Early Screen is currently underway.
This study confirms that there are identifiable features of autism prior to 4 years of age in young children with developmental delay. The utility and efficacy of a parent completed checklist as a brief population autism screening tool for children aged 18–48 months requires further confirmation in a larger study of children referred to community health services with developmental delay. The value of this screening tool (the DBC-Early Screen) in screening populations of young children without developmental delay also requires further study.
Footnotes
Acknowledgements
Funding support – Financial Markets Foundation for Children, Forest Hill Early Childhood Foundation. Assessment services, early intervention services, and families who generously supported this study. Monash Autism Programme (Avril Brereton and Megan Morling), and Travancore Child and Family Centre. The statistical support of A/Prof Andrew Mackinnon is gratefully acknowledged.