Abstract
Diagnosis of Autism Spectrum Conditions (ASC) can be an extended procedure since ASC tend to both vary greatly across individual symptoms and diagnostic pathways with serious challenges to opportune access and diagnosis in low resource settings. We adapted the Q-CHAT-25 for use in a routine health check-ups programme at Chilean primary health clinics by developing a 10-item version of this questionnaire recruiting n = 287 (F: 112/M: 175) participants (Controls: n = 125, F: 58/M: 67; Developmental Delay: n = 149, F: 53/M: 96; Autism Spectrum Conditions: n = 13, F: 1/M: 12). Our findings show that the Q-CHAT-10 can be successfully applied in health-check programmes. The results for the Q-CHAT-10 show high internal consistency (Cronbach’s α: 0.85) and good overall performance, significantly correlating (r = 0.79, p < 0.0001) with the Q-CHAT-25. The Q-CHAT-10 had a sensitivity of 92.86% and a specificity of 76.86% in the Developmental Delay sample. The positive predictive value was 48% with a positive likelihood ratio of 4.01 and a negative likelihood ratio of 0.09 with a post-test probability of disease of 19%. This study provides evidence that the implementation of Autism Spectrum Condition screening programmes using the Q-CHAT-10 is a cost-effective measure that improves diagnosis of Autism Spectrum Conditions in those participating in conditional cash transfer programmes in low- and middle-income countries or low resource setting in high income countries.
Lay abstract
Getting a diagnosis of autism can take long, because autism is different across people, but also because it depends on the way it gets diagnosed. This is especially important in poorer countries or in the case of poor people living in wealthier countries that have significant groups of disadvantaged communities. We adapted a 10-item version of the Q-CHAT-25 questionnaire for use in routine health check-ups programme in Chile and recruited 287 participants under the age of three divided into three groups: Controls (125), Developmental Delay (149) and Autism Spectrum Condition (13). Our results show that a short questionnaire for autism screening can be successfully applied in a health-check programme in poor resource settings. Our results show that our questionnaire had good overall performance, not different to its longer version, the Q-CHAT-25. Our questionnaire was autism specific, with good sensitivity and reliability, and is suitable to be used in a screening setting. This study provides evidence that the implementation of Autism Spectrum Condition screening programmes using the Q-CHAT-10 provides value for money and improves diagnosis of Autism Spectrum Condition in those participating in routine health check-up programmes in developing countries or poor areas of wealthy countries.
Keywords
Introduction
Autism (henceforth Autism Spectrum Conditions – ASC) is a heterogeneous cluster of neurodevelopmental conditions, in which the nature and severity of their characteristics exists within a wide spectrum of symptoms, presenting with persistent challenges in social communication and social interaction, and restricted or repetitive patterns of behaviour, interests or activities across multiple contexts (Baron-Cohen, 2017; Lai et al., 2014). So far ASC has only been behaviourally defined and no clear biological, neurological or genetic marker can be used as diagnostic instrument (Baron-Cohen, 2017; Ruzich et al., 2015). While prospective population screening prevalence studies suggest that approximately 1%–2% of the population is affected by ASC globally, few prevalence studies have been based in low- and middle-income countries (LMICs) or in low resource settings in high-income countries (HICs) (Elsabbagh et al., 2012; Samms-Vaughan, 2014; Wallace et al., 2012). The consequence of this has been a limited characterisation of ASC in Latin America and the Caribbean, and more broadly in the global south, with only five countries reporting autism prevalence estimates in South America (Dekkers et al., 2015; Elsabbagh et al., 2012; Ribeiro et al., 2017; Samms-Vaughan, 2014).
The early identification and early treatment of children with autism are consensually regarded as two of the most important factors for improving lifetime outcomes for individuals presenting with the condition (Dawson et al., 2010; Franz & Dawson, 2019; Warren et al., 2011). The age at which ASC symptoms first appear varies and can be challenging to detect in very young children, while the changes in symptom profiles that occur across early years make it difficult for clinicians to reach an early diagnosis without adequate health system screening programmes (Baron-Cohen, 2017; Ruzich et al., 2015; Sun et al., 2015). Diagnosis of ASC can be an extended procedure since ASC tend to both vary across the spectrum and depend on the referral mechanism and diagnostic pathways of different national, regional and local health systems across the world (Elsabbagh et al., 2012; Samms-Vaughan, 2014). Serious challenges to opportune access and diagnostic pathways in low resource settings are also common, such as the constraints seen in countries affected by austerity after the 2007 subprime crisis or who might suffer from post-COVID-19 economic hardship (Poovathinal et al., 2016; Ribeiro et al., 2017; Samms-Vaughan, 2014; Sun et al., 2015; Wallace et al., 2012). Another important challenge is that universal autism screening has so far not been validated as a cost-effective investment of resources in HIC such as the United States and United Kingdom, with the exception of ‘high-risk’ individuals, as shows evidence from Canada (Mandell & Mandy, 2015; Yuen et al., 2018). In low resource settings and in countries or areas where health systems are still primarily focusing on other pressing matters (such as child mortality) and where health services have been designed to care for acute life-threatening challenges (such as infectious diseases and malnutrition), expensive and resource-intensive screening for autism has been generally deemed a low policy priority (Mandell & Mandy, 2015; Wallace et al., 2012). Although important to consider when designing screening programmes in low resource settings, this criticism does not appear to consider the potential positive effect of universal screening on reducing health inequalities (Mandell & Mandy, 2015).
The lack of screening pathways across LMIC and HIC pose serious concerns about the opportune and adequate access to healthcare and specialist services of autistic children and the serious unmet health needs they might face, also considering the significant delay between the point of first contact, to an eventual diagnosis and subsequent early interventions (Estes et al., 2015; Franz & Dawson, 2019; Wallace et al., 2012; Warren et al., 2011). The development and implementation of research focused on undetected ASC in individuals in low resource settings is therefore crucial since many children in these underserved contexts might benefit from screening, early diagnosis and adequate healthcare and support, improving their overall well-being and long-term outcomes (Cidav et al., 2017; Franz & Dawson, 2019; Mandell & Mandy, 2015; Warren et al., 2011). Another key issue to consider when assessing screening programmes facing standard cost-effective analysis benchmarks in health technology assessment agencies is that the challenges of autism screening are diverse and include issues such as (a) an evidence base mostly stemming from cross-sectional studies or clinical research samples, (b) over-reporting of effectiveness of interventions or screening outcomes linked to selective sampling strategies, (c) failure to account for healthcare system inefficiencies and (d) potential policy consequences of screening programmes when implemented on a population level, such as the development and scaling-up of support services for children once diagnosed and the policy implications for resource allocation of different programmes (Yuen et al., 2018; Zwaigenbaum & Maguire, 2019). It is for these reasons that tackling and expanding the evidence of early detection of neurodevelopmental conditions such as autism is a crucial step to address the unmet need for mental health services in children, reducing the effects that social determinants of health might have in ASC long-term outcomes (Estes et al., 2015; Mandell & Mandy, 2015; Wallace et al., 2012).
Chile is a South American country that, according to the World Bank, has recently transitioned from middle- to high-income country status, which encompasses economies whose gross national income (GNI) per capita is above US$12,536, and has often been seen as a regional health system innovator (Bossert & Leisewitz, 2016; Roman-Urrestarazu et al., 2018; World Bank, 2018). Even though the transition to HIC status happened in 2013, and Chile’s pre-pandemic per capita GNI in 2018 was US$23,750, Chile has also persistently ranked as one of the most unequal countries in the world with a Gini coefficient of 0.46 in 2017 according to the Organisation for Economic Co-operation and Development (OECD), with important differences in health outcomes according to socioeconomic status (OECD, 2018). In effect, social inequalities in Chile are expressed as inequity in access and use of healthcare services, higher unmet need and longer waiting times, with this consequently reflected in poorer long-term outcomes and lower health status for those on lower incomes (Pan American Health Organization (PAHO), 2015). For example, infant mortality for children born to mothers with less than 3 years of schooling is 3.4 times higher than that of children whose mother has more than 13 years of education (PAHO, 2015). With this in mind, Chile was one of the first Latin American countries to adopt a comprehensive social security system, creating a state-run, single payer ‘Servicio Nacional de Salud – SNS’ in the 1950s, based on the British National Health Service – NHS (Bossert & Leisewitz, 2016; Roman-Urrestarazu et al., 2018). Since the 1980s, Chile transitioned to a mixed social health insurance system, keeping its strong focus on early childhood as a healthcare priority and developing after the 2000s a clear target in children’s bio-psychosocial development (Bedregal et al., 2010, 2016). One of these policies is the ‘Chile Crece Contigo – ChCC’ Bio-Psychosocial Development Programme which is a conditional cash transfer (CCT) programme structured around a management model that organizes different social programmes, welfare, social services, healthcare, and education to support the development of Chilean children according to their specific needs since birth (Bedregal et al., 2010, 2016; Torres et al., 2018). Rolled out in 2006, it began a process of rapid deployment with national coverage by 2008 (Torres et al., 2018). The backbone of this programme is the health checks programme that is centrally subsidized in the form of CCT to low-income families (Bedregal et al., 2010, 2016; de Andrade et al., 2015). The main idea behind CCT is that families with up-to-date children’s health check-ups receive around US$30 per month per child from birth until the age of 6 (Bedregal et al., 2010, 2016). This constitutes in many Chilean households an important 6% income boost, considering that 50% of Chilean workers earned less than US$515 per month in 2018 (Instituto Nacional de Estadistica, 2018).
The uptake of the CCT health-checks programme in Chile has been remarkably high, with more than 85% of all Chilean children between 12 and 24 months of age actively participating in the programme (Bedregal et al., 2016; Milman et al., 2018; Torres et al., 2018). The health checks include activities aimed at the prevention, detection and treatment of different developmental and health conditions while providing support to vulnerable mothers and families (Bedregal et al., 2016). The health checks are applied in the framework of a multidisciplinary team (MDT) and consist of different modules such as clinical history taking, vaccines, physical examination, screening and guidance on general developmental and psychosocial aspects of care for children (Bedregal et al., 2016; Milman et al., 2018). One of the pathways that has been included in ChCC is the assessment of psychomotor development which has been measured by a local scale, namely the ‘Escala de Evaluación del Desarrollo Psicomotor de 0–24 meses – (EEDP)’ by S. Rodríguez et al., which is applied at a 18 months check-up to children participating in the programme (Rodriguez et al., 1979). It measures five different developmental dimensions including motor, language, social and coordination skills (Rodriguez et al., 1979). Those deemed to present with neurodevelopmental delay are referred to specialist clinics for evaluation in secondary care (Rodriguez et al., 1979).
The presence of a programme with such high uptake for population screening of developmental delay (DD) provides an ideal ecological framework for research into screening for ASC in Chile and low resource settings. Our goal was to assess how a CCT programme, and particularly ChCC, might improve ASC screening programmes by looking at three different stratification levels: (a) general population participating in routine health checks as part of the ChCC programme, (b) children referred to specialist services due to DD as measured by the EEDP scale and (c) a validation sample of children already diagnosed with ASC and that are part of the a secondary care specialist service case load. Our research aim was to evaluate how a CCT using a primary and secondary care model in a LMIC or low resource in a HIC setting might operationalize early diagnosis screening instruments, while fulfilling the targets and goals that the ChCC has for these three subgroups of children. The chosen screening tool was the Quantitative Checklist for Autism in Toddlers (Q-CHAT-25), which is a revised version of the original CHAT questionnaire (Allison et al., 2012; Baron-Cohen et al., 1992; Magiati et al., 2015). The Q-CHAT-25 allows parents and caregivers to quantify autistic traits in children 18–30 months of age and to classify toddlers that might be on track to present with a developmental trajectory for ASC from those neurotypically developing children (Allison et al., 2012, 2008) The long 25-item Q-CHAT has been validated across a wide range of different settings and is proven to have excellent power to discriminate children with ASC from general population toddlers at 18–48 months (Allison et al., 2012; Harris et al., 2014; Mohammadian et al., 2015). Use of a quantitative measure allows the screening instrument to detect more subtle manifestations of ASC and place them inside a spectrum of continuous scores (Allison et al., 2012, 2008). The Q-CHAT-25 has good test–retest reliability and adequate internal consistency, with its scores shown to be normally distributed (Allison et al., 2012, 2008; Gutiérrez-Ruiz et al., 2019; Magiati et al., 2015). Together with this, the widespread use in different countries and cultural settings makes the Q-CHAT-25 a very strong candidate as ASC screening instrument (Allison et al., 2012; Baron-Cohen et al., 1992; Magiati et al., 2015).
In order for the Q-CHAT-25 to be used as part of the ChCC programme, it should ideally be reduced to its 10-question version, as recommended by Allison et al., while also being culturally appropriate and locally validated (Allison et al., 2012). Although the full-length versions can be also be implemented, one of the aims of this study is to adapt the long 25-item Q-CHAT for use in the time-constrained health check-ups programme at Chilean primary health clinics by developing a 10-item version of this questionnaire, adapted to the local reality and maximizing contact time of the ChCC programme. We also decided to use the Q-CHAT-25 over the CHAT in order to shift from a categorical to a dimensional screening framework of early diagnosis in ASC in a low-resource context using a World Bank and United Nations Development Programme (UNDP)-validated CCT programme (Allison et al., 2012; Fort et al., 2007; Martínez-Aguilar et al., 2017).
Our main objective with this was to adapt Allison et al. (2012) methodology to a low resource setting and identify which 10 items from the Q-CHAT would show the same levels of sensitivity and specificity as the full-length version of this instrument in (a) ASC case, (b) DD, and (c) Controls samples, in order to be applied as a screening measures integrated to the CCT programme ChCC in both the general population and at a secondary referral pathway. We hypothesize that within this health check framework an ASC screening tool is ideally suited to detect and refer potential cases to diagnostics pathways and would fill the ASC diagnostic gap for Chilean healthcare professionals in primary care. They often require making quick clinical decisions in real time, given important economic constraints, about whether to refer patients to specialist services for ASC, without sacrificing the excellent psychometric properties of the instrument (Allison et al., 2012; Gutiérrez-Ruiz et al., 2019). We hypothesize as well that implementing this measure as an add-on to the existing screening programme would be cost-effective and potentially create health system savings and expedite the diagnostic process for autistic children in Chile. This programme could also be implemented in other high- or middle-income countries facing resource constraints and who have implemented CCT and health check programmes for children. This study’s ultimate goal is therefore to enable health authorities to establish an ASC screening programmes, but also to facilitate the scale-up of autism detection pathways using the framework of CCT programmes such as those widely promoted across many international development agencies (Allison et al., 2012; Fort et al., 2007; Martínez-Aguilar et al., 2017).
Methods
Setting and selection of study participants
This study was carried out at the Paediatric Neuropsychiatry Service, Hospital Clínico San Borja Arriarán (HCSBA), Santiago de Chile. HCSBA is part of the Chilean public hospital network and includes the boroughs of Santiago and Estación Central in its catchment area, while also being a national referral centre for paediatric neurology. According to the Chilean 2017 Census, the total population in this health trust catchment area was N = 551,536 with N = 11,783 children aged between 18 and 30 months living in these two boroughs of metropolitan Santiago (Instituto Nacional de Estadistica, 2017). In order to understand HCSBA service users’ socioeconomic background, it is important to understand Chile’s health system payment structure, where 76.3% of the population are covered by the state-run social health insurer or National Health Fund (Fondo Nacional de Salud – FONASA), 18.2% are covered by for profit private insurance companies and the remaining 2.95% of the population have institutional coverage, such as the armed forces (Roman-Urrestarazu et al., 2018). Membership to the public insurance scheme has been used extensively as a proxy for income since it is linked to wage-deducted contributions, with privately insured users preferring hospitals in the private network and rarely attending public clinics (Roman-Urrestarazu et al., 2018). There are currently four FONASA categories divided by income bracket: FONASA A, which caters for people without any stable income, homeless or in reception of welfare; FONASA B, which is intended for people with monthly earnings below US$319; FONASA C, for those between US$320 and US$464; and FONASA D for those earning above US$465. FONASA A and B have 100% health cover and no co-payments and C and D incur in 10% and 20% of co-payments, respectively, with the option to purchase pay as you go services in the private sector. The population in HCSBA catchment area tends to be enrolled in FONASA. For example, between July 2017 and July 2018, there were N = 3779 children between 18 and 30 months of age that had a consultation in HCSBA, belonging to the following insurance categories: FONASA A: 1496 (39.6%), FONASA B: 951 (25.2%), FONASA C: 534 (14.13%), FONASA D: 758 (20%), Private Insurance: 40 (1.06%).
We chose a population-based framework to study children aged between 18 and 30 months at three different levels: (a) general population individuals participating in the ChCC routine health-checks programme at 18 months (Controls), (b) children referred to specialist services at HCSBA after participating in the 18 months health-check schedule due to DD evaluated through the EEDP for the at-risk (0.84–0.70) or DD (⩽0.69) cut-offs, and (c) children diagnosed with ASC according to Diagnostic and Statistical Manual of Mental Disorders (5th ed.; DSM-V criteria; ASC cases) and part of the HCSBA case load (Allison et al., 2012). There were N = 2218 contacts of children between July 2017 and July 2018 attending the health-check programme and we recruited our sample from that group. Only ASC cases diagnosed at a recognized clinic by a recognized specialist or clinical psychologist using DSM-V criteria were included in our ASC case group. For the DD group, we included 179 children referred to the HCSBA in a 6-month period (between 1 July and 31 December 2017), of which 19 were not able to be interviewed due to either not attending their appointments or refusing to answer the questionnaire. Of the 160 remaining participants, we excluded 10 children: 5 for not meeting our age criteria and falling outside of our 18–30 month bracket, 1 for poor engagement during the interview and 4 because of diagnostic issues. These were due to one meeting diagnostic criterion for Rett syndrome, one meeting criterion for Down syndrome, one for having a severe premature birth with an intracranial bleeding and one who was deemed to meet normal developmental milestones and not present with DD. The parents of the remaining 150 children between the ages of 18 and 30 months completed the Q-CHAT-25 and provided sociodemographic information. We report FONASA category as a proxy for socioeconomic status to those children in secondary care (DD and ASC group).
Questionnaires and coding procedures
The Q-CHAT contains 25 questions concentrating on behaviours that reveal autistic traits in very early childhood (Allison et al., 2012, 2008). Each item has five response possibilities based on the frequency to which the child shows the behaviour (Allison et al., 2012). The highest frequency of an autistic trait scores 4, and the lowest frequency of an autistic trait scores 0. Half the items are reverse scored. Following Allison et al., and to increase the consistency of the method used to determine the best 10 items, we converted the Likert-type rating scale to a binary scoring system recoding a score of 0 or 1 to 0, and a score of 2, 3 or 4 to score 1 (Allison et al., 2012). We included questionnaires with up to two missing answers and took a conservative approach and assumed missing values were zero. We adapted the Q-CHAT-25 to the Spanish spoken in Chile having as reference the original questionnaire available at the Autism Research Centre, University of Cambridge, and then compared our translation to other Spanish versions available there (Allison et al., 2008). Children were screened with the Q-CHAT between July 2017 and July 2018 with follow-up assessment using Autism Diagnostic Observation Schedule (ADOS-2) up to August 2018.
Validation of the Q-CHAT-25 item selection in ASC cases versus controls
In order to adapt the Q-CHAT-25 to the Chilean CCT programme maximizing administration time, we first proceeded to validate the 10 most discriminant questions from the Q-CHAT-25 by comparing our ASC cases to our Controls in a derivation and control sample after face-to-face application of the questionnaire (Allison et al., 2012; Magiati et al., 2015). Participants from the ASC Case and Control groups were randomized and allocated to the derivation and validation samples (Allison et al., 2012). The best 10 items from each measure were calculated from the derivation samples by estimating a discrimination index (DI) for every item (Allison et al., 2012). We calculated this by subtracting the proportion of participants who scored 1, defined as having an autism trait positive response in the Control group on each item, from the proportion of participants who scored 1 in the ASC group (Allison et al., 2012; Magiati et al., 2015). Good items on a measure have been described as having a DI between 0.3 and 0.7 (Gutiérrez-Ruiz et al., 2019; Magiati et al., 2015; Mohammadian et al., 2015). The 10 questions with the highest DI were chosen from the long 25-item Q-CHAT. Receiver operating characteristic (ROC) curves containing the 10 most discriminating items for each measure were produced on the validation samples (Allison et al., 2012; Gutiérrez-Ruiz et al., 2019; Magiati et al., 2015), with ROC curves plotted for sensitivity and 1-specificity of all possible scores on the measure. The presence of a diagnosis of ASC was the dependent variable on the Q-CHAT score and the independent predictor variable (Allison et al., 2012). The area under the curve (AUC) is a measure of the overall predictive validity, where an AUC = 0.50 indicates random prediction of the independent variable (Allison et al., 2012; Hanley & McNeil, 1982). An AUC of >0.90 indicates excellent validity (Allison et al., 2012; Hanley & McNeil, 1982; Myerson et al., 2001). The AUC was calculated for each 10-item measure, and compared with the AUC for the full version (Allison et al., 2012; Hanley & McNeil, 1982; Myerson et al., 2001). A one-way analysis of variance (ANOVA) was conducted to compare the 10-item questionnaire between ASC Cases, DD and Controls. Internal consistency (Cronbach’s alpha) was calculated for each group and for the whole sample. Correlations were examined between total scores on the short and long forms of the Q-CHAT (Allison et al., 2012).
Using the Q-CHAT-10 as screening in the Controls and DD groups
After selecting the 10 most discriminant items of the Q-CHAT-25, we then validated the proposed Q-CHAT-10 with a case cut-off point of 3 for validity of case selection in the DD after collecting the questionnaires in face-to-face contact (Allison et al., 2012). We report in the DD group sensitivity, specificity, positive likelihood ratio (PLR), negative likelihood ratio, condition prevalence, positive predictive value (PPV), negative predictive value (NPV) and accuracy. In order to also assess possible cases in the Controls group, and as part of our ethics proposal, we also used the Q-CHAT-10 screening key, clinical assessment and ADOS referral in this group for case ascertainment. Using the same cut-off point of 3 or above, we subsequently referred children in the Controls group using the same diagnostic pathway as children with DD. All referred participants were first assessed by a Paediatric Neurologist and if suspicious of ASC subsequently referred again for an ADOS-2 evaluation at HCSBA as a cross-over measure used both for consistency of the selected cut-off point, referral validity and diagnostic accuracy of the clinicians performing clinical assessments (Gotham et al., 2009; Molloy et al., 2011). A random sample of children from the ASC case group also was evaluated with ADOS-2, in order to produce and ad hoc validation of the clinical diagnostic process at HCSBA. We then correlated Q-CHAT-10 scores with ADOS-2 final scores. The ADOS-2 interview was carried out by four different researchers, one from the Autism Research Centre, University of Cambridge and three staff members of the Neuropediatric Service HCSBA. This project received ethical approval from the Servicio de Salud Metropolitano Central Clinical Ethics Committee.
Results
The final number of included participants was 287 (F: 112 (39%); M: 175 (61%)) which is 2.44% of the total census population (N = 11,783) for the catchment area of HCSBA and 18.82% of the total number of pupils who had either 1 or 2 contacts (N = 2218), and that were screened between July 2017 and July 2018 for their 18-month ChCC health checks. The socioeconomic status of those in secondary care (DD and ASC groups) was (a) ASC group (N = 13): FONASA A: 3 (23.1%), B: 4 (30.7%), C: 3 (23.1%), D: 1 (7.7%), Private Insurance: 0, Missing: 2 (15.4%), and for the (b) DD group (N = 149): FONASA A: 43 (28.9%), B: 21 (14.1%), C: 27 (18.1%), D: 36 (24.2%), Private Insurance: 3 (2.0%), Missing: 17 (11.4%). We reached this number after excluding one female participant after being diagnosed with Angelman’s syndrome after answering the questionnaire, and another female participant due to data quality issues (more than seven missing answers in Q-CHAT-25). In our final sample, we found that 15 subjects had missing responses, where only 3 participants had two missing questions, the rest only presenting one missing item. Altogether, there were 13 (M: 12/F: 1, 4.5%) autistic children in our ASC sample, 149 (M: 96/F: 53, 51.8%) children in the DD group, and 125 (M: 67/F: 58, 43.6%) Controls who completed the Q-CHAT. In the validation of the Q-CHAT-10 items, we had 62 Controls and 7 ASC participants forming the derivation (N = 69) case sample and 63 Controls and 6 ASC participants the validation (N = 69) case sample. For the description of the ASC and Controls validation and derivation sample, please see Table 1 and Figure 1 for the study design and participant selection.
Participant characteristics.
ASC: Autism Spectrum Conditions; DD: Developmental Delay.
Study design and participant selection.
Q-CHAT-25 10-item selection in validation and derivation sample
Results from item analysis are presented in Tables 2 and 3. The AUC for both the Q-CHAT-10 (0.952) and the Q-CHAT-25 (0.948) was higher than 0.9 in the Cases–Controls comparisons. The AUC value for the Q-CHAT-10 version was slightly higher than the 25-item version. ROC curves are displayed in Figure 2. The coordinates of the curve indicating the scores at different sensitivity thresholds can be seen in Table 2.
Item analysis showing discrimination index for quantitative checklist for autism in toddlers (Q-CHAT)-25 item version.
Most discriminating 10 items for Quantitative Checklist for Autism in Toddlers (Q-CHAT)-25 item version.
ROC and AUC characteristics for Q-CHAT-25 and Q-CHAT-10.
There was a significant difference in Q-CHAT-10 scores between ASC (M = 7.46, SD: 2.85, 95% confidence interval (CI): 5.74–9.18), DD (M = 2.26, SD: 2.45, 95% CI: 1.87–2.66) and Controls (M = 0.94, SD: 1.7, 95% CI: 0.64–1.25), one-way ANOVA (F (2284) = 56.4, p < 0.0001), with a large difference of magnitude in the means (eta squared = 0.284). Cronbach’s alpha for the Q-CHAT-10 in the whole sample (N = 287) was 0.85, showing high internal consistency. Cronbach’s alpha for Controls was 0.79 (N = 125), for DD it was 0.79 (N = 149) and for ASC it was 0.84 (N = 13). The Q-CHAT-25 was significantly correlated with the shorter 10-item version (r = 0.79, p < 0.0001).
Q-CHAT-10 screening
During this final stage, 16 participants (12.8%, M: 9/F: 7, mean age: 18.5 months) from the Controls group dropped out of the study (2 with scores of ⩾3), without attending a follow-up. In this abridged Controls sample (N = 109) and using a cut-off point of 3 in the Q-CHAT-10, we obtained N = 99 participants scoring below 3 (mean Q-CHAT-10 score: 0.44, 95% CI: 0.32–0.57) who were deemed to be neurotypical. We referred 10 participants who scored 3 points or above for further assessment by a Paediatric Neurologist at HCSBA. From this sample, six participants (mean Q-CHAT-10 score: 4.66, 95% CI: 0.13–7.21) after a face-to-face clinical assessment were deemed not to meet ASC criteria by a Paediatric Neurologist. We referred four (M: 2/F: 2, 0.032% of sample) participants (mean Q-CHAT-10 score: 6.75, 95% CI: 1.49–12.00) for an ADOS-2 assessment, all of which were diagnosed with ASC and had an average score of 22.25 (95% CI: 13.69–30.81).
In the DD sample (N = 149), we found that N = 54 or 36.24% (mean Q-CHAT-10 score: 4.98, 95% CI: 4.43–5.53) of participants had a score of 3 or above in the Q-CHAT-10, while N = 95 participants scored below (mean Q-CHAT-10 score: 0.72, 95% CI: 0.57–0.88) this cut-off point. Participants were then sequentially screened in a face-to-face assessment by Paediatric Neurologist, as per the DD pathway in place in Chile, with an ultimate referral to an ADOS-2 assessment at HCSBA. From this sample, 27 participants were clinically screened and deemed neurotypical. We then proceeded to invite 20 participants in the DD sample to an ADOS-2 (M: 18.5, 95% CI: 15.25–21.75) assessment for the purpose of validating the referral pathway and diagnostic protocol, while 6 of them were clinically diagnosed with ASC at HCSBA after an MDT assessment led by a Paediatric Neurologist following the standard HCSBA ASC diagnostic protocol. These six participants were diagnosed by these means since they had a pending ADOS-2 assessment at the time of finalizing this project. The mean Q-CHAT-10 score for the referred N = 26 participants was 6.15 (95% CI: 5.30–7.01). One subject had both a Q-CHAT-10 score of 3 or above and was referred for an ADOS-2 assessment after being screened but did not meet the ADOS-2 threshold for ASC. In the ASC cases sample (N = 13), we decided to validate the HCSBA diagnostic process by randomly referring 10 participants for an ADOS evaluation all of whom met ASC diagnostic criteria and scored on average 17.80 (95% CI: 13.96–21.64) points. The total number of participants interviewed in our study with the ADOS-2 was N = 37 (M: 31/F: 6) participants (12.89% of sample), with 4 Controls (0.032% of sample), 10 ASC (76.92% of sample), and 23 DD participants (15.43% of sample). This means we were able to confirm with an ADOS-2 test 36 out of the 45 children (80%) diagnosed with ASC in our sample.
The results for the Q-CHAT-10 screening for the DD group showed a sensitivity of 92.86% and a specificity of 79.86% with a PPV of 48.15% (pre-test odds = 0.19). For all results, please see Table 4.
Sensitivity and specificity of Q-CHAT-10 in Developmental Delay group N = 149.
CI: confidence interval; DD: Developmental Delay.
Discussion and conclusion
CCTs are an established policy across a range of LMIC and low resource settings and have been endorsed by development agencies such as the World Bank, UNDP and Inter-American Development Bank (IDB) to improve population health outcomes (Rawlings & Rubio, 2005; Sewall, 2008; Stampini & Tornarolli, 2012). Most of these programmes take on from early predecessors in HIC, such as the successful US Head Start, rolled out during President Lyndon B. Johnson’s administration and the United Kingdom’s Sure Start programmes which was implemented by Gordon Brown during Tony Blair’s tenure as Prime Minister (Cattan et al., 2019; Kline & Walters, 2016). In Latin America, these programmes were pioneered in the late 1990s first in Brazil with ‘Bolsa Familia’ and then with ‘Oportunidades’ in Mexico, so by 2011, CCT had spread to 18 countries in the region and covered as many as 135 million beneficiaries across Latin America and the Caribbean (Cecchini & Atuesta, 2017; Sewall, 2008; Stampini & Tornarolli, 2012). Since then, the success of CCT has contributed to their adoption in other world regions, increasing global familiarity with their distinctive approach to development, while improving crucial long-term health outcomes such as increased uptake of immunization rates and coverage and improvements in child morbidity (Cahyadi et al., 2018; Das et al., 2005; Rawlings & Rubio, 2005).
One of the reasons that make CCT attractive in low resource settings is the bundling of interventions and the possibility to include different add-ons to the different screening programmes that constitute this health contact platform, maximizing easy and pragmatic use of scarce resources that might be tailored to the health priorities of different countries, population subgroups and areas where they are implemented. In line with this, our findings show that an add-on questionnaire of 10 items from the Q-CHAT-25 can be successfully applied in low resource settings as part of the ChCC CCT programme to children attending their 18-month health check-ups and those referred for further assessment by presenting DD. Our results also show that this is an efficient use of resources and a cost-effective measure to add an ASC screening tool to an already existing DD screening programme without increasing the administration time of the screening in the ChCC programme. It is important to note that although the ChCC programme currently focuses on DD, autism is not a part of it, so our results support the inclusion of an autism-specific screening platform.
The results for the 10 selected item Q-CHAT showed high internal consistency (Cronbach’s α: 0.85) and good overall performance, significantly correlating (r = 0.79, p < 0.0001) with its longer version the Q-CHAT-25. The AUC for both the Q-CHAT-10 (0.952) and the Q-CHAT-25 (0.948) was higher than 0.9 in the Cases versus Controls comparisons. Furthermore, the items selected for our short 10-item version were the same as in the work of Allison et al. except for item 21 (D′ = 0.70), which states ‘Does your child spontaneously look at your face to check your reaction when faced with something unfamiliar?’ and with the exclusion of ‘Would you describe our child’s first words as (typical)’ (Allison et al., 2012). We also found that the Q-CHAT-10 in the DD has good overall performance in the selected setting with good sensitivity and specificity, which are in line with what had been informed previously by Allison et al., with a sensitivity in our DD sample of 92.86% versus Allison et al. group’s 91%, and a specificity in our sample of 77% versus 89% in her sample (Allison et al., 2012). This is also the case in our PPV, which was 48.15% in our DD group and was slightly lower to what has been reported previously (58%) (Allison et al., 2012). Another finding of this work was the PLR which was 4.01 meaning a moderate increase in the post-test probability of disease (19%) as per epidemiological nomenclature, suggesting that for every false positive detected, there are four true positives.
In the Controls group, the performance of the Q-CHAT-10 was less optimal, and although we found a sensitivity of 100% (since we found no false negatives at the end of the study) (95% CI: 39.76%–100%), and a specificity of 94.34% (95% CI: 88.09%–97.89%) for a PLR of 17.67 (95% CI: 8.12%–38.43%), assuming a prevalence of 1.76% for ASC (Roman-Urrestarazu et al., in press); the PPV here was almost half the one found in the DD group with 24.04% (95% CI: 12.70%–40.78%) for an accuracy of 94.44% (95% CI: 88.37%–97.91%). We are continuously screening study participants in the HCSBA service users list for possible ASC diagnosis to accurately assess our NPV and false-negative rate.
These findings support our hypothesis that the target group for ASC screening in the ChCC CCT programme should be the DD group, which supports what Yuen et al. proposed for ‘high-risk’ screening in Canada (Yuen et al., 2018). We could also hypothesize that since the DD group presents with complex developmental symptoms that overlap with ASC, this might explain the relatively high rate of false positives we found and the moderate increase in post-test probabilities. As a screening test, the Q-CHAT-10 performs considerably well in the DD group, considering that from our results if a child scores negative (has a score below 3), we can be reassured that a child does not have the disorder (NPV = 97.89%), and that this initial screening correctly identifies 77% of those who do not have ASC (the specificity), which considering the age of our participants makes the Q-CHAT-10 a solid and robust screening tool. In support of this is the negative likelihood ratio of 0.09 found in our DD sample, which gives us the change in the odds of having a diagnosis in patients with a negative test, which in turn is a 10-fold decrease in the odds of having a condition in a patient with a negative test result.
When compared with the M-CHAT, which is the tool currently used in Chile and recommended by the Ministry of Health since 2011 in their national clinical guidelines for early detection of ASC, our findings suggest that using a shorter screening questionnaire, placed inside a CCT and with a testing time of around 5 min, is a better option for the early diagnosis of ASC in a DD group. The Q-CHAT-10 optimized administration and effectiveness by reducing contact time with service users of the ChCC programme while being an accessible tool that any health professional could handle and apply (Coelho-Medeiros et al., 2017, 2019; MINSAL, 2011). We also think that the M-CHAT should be superseded with a shorter, DSM-V based tool for ASC, such as the Q-CHAT-10, especially considering that current guidance does not consider the inclusion of DSM-V criteria for ASC. Another issue is that although in the Chilean validation of the M-CHAT (Coelho-Medeiros et al., 2019), a higher internal consistency was reported, with a Cronbach’s alpha of 0.889, and a very high sensitivity (100%) and specificity (83.3%); we must mention that this group recruited subjects for their case sample from what was defined as a ‘high ASC suspicion group’, which included a pre-assessment and referral of cases by specialists such as paediatricians, paediatric neurologists and child and adolescent psychiatrists (Coelho-Medeiros et al., 2019). On the contrary, in order to reduce the possibility of selection bias, we utilize an already existing screening programme (ChCC) in a predefined group of children, from a defined catchment area, participating in routine health check-ups and based on our referral pathway exclusively on Q-CHAT scores. This is the reason why we consider our study design more robust to the cofounding problem that this specialist pre-assessment might introduce, since it utilizes expensive resources not usually available in low resource settings.
While challenges to the ChCC programme persist mostly linked to funding constrains, improved health outcomes due to adequate and timely screening are evident, such as the reduction of DD prevalence from 25.1% to 11.7% in Chilean children aged between 3 and 4 years between 2006 and 2017 (Bedregal et al., 2016; Margozzini & Passi, 2018; Subsecretaría de Salud Pública, 2017; Torres et al., 2018). This research shows the potential impact that CCT programmes could have in improving long-term health outcomes in an often neglected area of children’s health such as neurodevelopment. When one considers that health priorities in socially deprived areas have been mostly focused on reducing infant mortality, communicable disease and obesity and have often overlooked mental health and psychiatric conditions as crucial determinants of the well-being of children, our results provide supporting evidence to tackle a crucial public health challenge such as ASC. We also think that there should be a more broad discussion about including early interventions as part of the autism screening pathway in Chile, since identifying young children with ASC in a low resource context might have the effect of identifying problems without offering families a solution (Mandell & Mandy, 2015). It is also important to have in mind that HCSBA is also a paediatric neurology national referral centre and might not necessarily represent the reality across hospitals in the Chilean public network. Currently, HCSBA offers a comprehensive care package that includes children’s psychologist, speech and language therapist and occupational therapist as well as paediatric neurologist and psychiatrist for those diagnosed with autism. Other centres in Chile might not have the same experience and resources to develop early interventions, which might present as a challenge if one would like to scale an ASC pathway to other parts of the country. We believe that the first step for scaling ASC services would be to improve national screening, and subsequently implementing a national intervention pathway, such as the one already in place for DD, that linked primary and secondary care in specially designed early stimulation rooms. These rooms are located in health clinics, community centres and nurseries and are spaces where children jointly with their parents or caregivers carry out educational and comprehensive stimulation activities that reinforce various aspects of child development under the supervision of health professionals such as occupational and speech and language therapists. We believe they are the ideal setting for an ASC-specific early intervention programme.
We should also mention some shortcomings in our study. In our Controls sample using the Q-CHAT, data were collected prospectively between 18 and 30 months of age, so it is likely that there might be children within this sample that had low scores on our assessment who might have subsequently received a diagnosis which as mentioned before might affect our NPV. We must also acknowledge that the FONASA and insurance data from the DD and ASC group, had between 11.4% and 15.4% of missingness. We think it is important to note that collecting income data can present with challenges in low resource settings, since usually welfare and health insurance payments are linked to wages and reported income, as is the case in Chile, with some participants avoiding disclosure for that reason (Burger et al., 2016; Moore et al., 2000). We tried to the best of our resources to provide an overview of the FONASA categories in HCSBA for the dates we carried our project. We believe this provides a good overview of the sociodemographic profile of the population where we carried our study. Another issue is that that the gender breakdown in the case derivation (N = 7: F/M 0:7) and validation (N = 6: F/M 1:5) sample lacks female participants in one case and only has one in the other. Considering that we were screening in general population Controls, and then in a DD sample, this derivation sample falls in line with previous research with small samples or who didn’t provide a derivation/validation analysis for this subgroup, with our overall sample size (N = 287) being larger than those studies (Devescovi et al., 2020; Park et al., 2018; Raza et al., 2019; Wong et al., 2014). We do acknowledge that his might be a source of bias, but the results stemming from this analysis replicate the ones reported in Allison et al. (2012, 2008). Considering the difficulty in recruiting participants of this age, we believe we have addressed this issue in our case ascertainment by performing over 80% of ADOS-2 interviews to those diagnosed with autism. Following that line, the other shortcoming we should mention was that not all ASC cases were evaluated with an ADOS-2 interview, and some were diagnosed exclusively with a clinical assessment. Considering the low resource setting where this study took place in a public hospital in Latin America, we think that a benchmark of more than four-fifths of case ascertainment with the ADOS-2 interview is a good outcome and robustly support the clinical diagnostic process at HCSBA.
Another one of the shortcomings of this study is that up until now there is still not a validated pathway for early interventions in ASC for young children below 30 months of age, and therefore most studies that focus on the clinical outcomes of children identified with ASC through screening have had mixed results, specially so in clinical populations (Sappok et al., 2015; Siu & US Preventive Services Task Force (USPSTF), 2016; Zwaigenbaum & Maguire, 2019). Although there are studies suggesting treatment benefit in older children identified through family, clinician, or teacher concerns, the USPSTF found inadequate evidence on the efficacy of treatment of cases of ASC detected through screening or among very young children (Siu & USPSTF, 2016). Treatment studies have been generally very small, few are randomized trials, most include children who were older than would be identified through screening, and they tend to be carried in population samples clinically referred to research centres rather than screen-detected patients (Levy et al., 2020; Siu & USPSTF, 2016). Even with this evidence some have said that there is a clear benefit of early detection and intervention at the same time young children are developing crucial milestones in language and social abilities (Franz & Dawson, 2019). Early intervention optimizes health outcomes, increases independence and reduces long-term costs to both families and health systems (Cidav et al., 2017; Estes et al., 2015).
Globally, there is increased recognition of the importance of early detection and interventions as a critical public health focus from international organizations such as the World Health Organization (WHO), which clearly delineates the role of potential early interventions in high- and middle-income countries with low resources, but also in LMIC (WHO, 2013). Considering the advantages that Chile has in benefitting from a solid public health network, that although underfinanced remains responsive, caution should be advised when assuming that countries with a lower development index and fragile health systems could implement this programme. What our results could inform is the implementation of ASC screening to already existing CCT programmes in middle- or high-income countries, who might have large population subgroups facing social deprivation, and where inequality might be associated with serious unmet health needs of children. Linked to this is another important issue that should be discussed, which is the economic impact of increasing the uptake of diagnostic services for ASC service users and how the Chilean health system is prepared to deal with increased demand for paediatric mental health services, which is something that other countries might also face. Considering the scope of our work, we believe this should be a new area of research in the ChCC. We ascertain that the Q-CHAT-10 is a starting point and should be promoted in Chile and Latin America as the to go to screening tool since we have demonstrated that a CCT programme is the ideal framework for its deployment. We must also consider that so far, no national prevalence estimate exists for ASC in Chile, making any screening process difficult to be evaluated in its full scope since we cannot estimate the amount of people that might benefit or would require interventions.
To the best of our knowledge our work is the first to implement an ASC screening procedure in a running CCT programme having as its central aim improving ASC diagnostic pathways in an HIC setting facing considerable resource constraints. This study provides evidence that the implementation of an ASC screening programme using the Q-CHAT-10 is a cost-effective measure that improves diagnosis of ASC in both the general population participating in the ChCC programme and those with DD. By linking the ChCC programme to an ASC diagnostic pathway, we were able to maximize access to a set of universal services enshrined in the ChCC programme and aligned to the United Nations Sustainable Development Goals, such as fostering child development and improving the delivery of health services and welfare targeted at children stemming from socially disadvantaged families as well as specialized provisions for vulnerable children.
Footnotes
Author contributions
A.R-U. and C.Y. collected data, analysed the data and wrote the manuscript and edited the different drafts. C.L-G. collected data and edited the different drafts. C.E. collected data and edited the different drafts. C.A. collected data and edited the different drafts. C.B. provided methodological guidance and edited the different drafts. M.T. was senior researcher and provided methodological guidance and overall project management and edited the different drafts. S.B-C. was senior researcher and provided methodological guidance and overall project management and edited the different drafts.
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship and/or publication of this article: A.R-U. received funding from the Gillings Fellowship in Global Public Health and Autism Research, Grant Award YOG054 to the Cambridge Institute of Public Health (PI Carol Brayne). S.B-C. received funding from Innovative Medicines Initiative 2 Joint Undertaking (JU) under grant agreement No. 777394. The JU receives support from the European Union’s Horizon 2020 research and innovation programme and EFPIA and AUTISM SPEAKS, Autistica, SFARI. S.B-C. also received funding from the Autism Research Trust, Autistica, the MRC, the Wellcome Trust and the NIHR Cambridge Biomedical Research Centre. The research was supported by the National Institute for Health Research (NIHR) Collaboration for Leadership in Applied Health Research and Care East of England at Cambridgeshire and Peterborough NHS Foundation Trust.
Ethics approval
This study was approved by the Ethics Committee CEC-SSMC the 28 June 2017.
