Abstract
Introduction
Developmental delay in children is common and present in up to 13% of children. 1 Early intervention is beneficial for cognitive, social, and emotional development in children with developmental disability. 2 Identifying children at risk has important social, academic, and economic implications. 3
In primary care practice, developmental delay is generally underidentified without standardized developmental screening.4-7 A 2006 American Academy of Pediatrics Policy Statement recommended standardized developmental screening for all children at age 9, 18, and 24 months. 4 The Infant Developmental Inventory (IDI) 8 and the Ages and Stages Questionnaire–Third Edition (ASQ) 9 are 2 recommended, caregiver-completed screening instruments. 4
An increasing number of primary care practices are choosing developmental screening instruments to comply with the American Academy of Pediatrics recommendations. 10 Previous studies have not evaluated the utility, satisfaction, and efficiency of the tools in the primary care setting.
The purpose of this study was to compare the clinical utility and efficiency of the IDI versus the ASQ when used at 9-month well-child visits in paper format. Our study aims were to determine the following:
Percentage of correct screening questionnaire completion by the caregiver and correct interpretation, documentation, and billing by the clinician with use of the IDI versus the ASQ.
Percentage of children who do not pass screening as well as the time to complete the screen with use of the IDI versus the ASQ.
We hypothesized that the use of the ASQ compared with the IDI instrument at 9-month well-child visits would improve the percentage of accurate developmental screening questionnaire completion and interpretation because all questions in the ASQ developmental areas need to be answered in order for the screen to be scored. A previous investigation examining the use of the IDI in our study population also demonstrated that the IDI was completed correctly less than 50% of the time. 11 In addition, we hypothesized that use of the ASQ would not lead to a significant increase in total visit time.
Methods
Study Setting
The project was conducted between February 2012 and October 2012 at 2 freestanding primary care clinics affiliated with an academic medical center in Rochester, Minnesota. At one care clinic, a family medicine care team (4 physicians and 3 nurse practitioners) participated, and at the other clinic, a pediatric care team (2 physicians and 1 nurse practitioner) participated. The institutional review board of the academic center deemed this study a quality improvement project exempt from any approval requirement. Participating nurses and clinicians were instructed, with 60 minutes of training, on how to appropriately complete and score the screening instruments. Prior to the study, the IDI was used and interpreted by the clinician at all 9-month well-child visits. As part of standard care at the study sites, patients were provided an appointment time 20 minutes before their scheduled time to see the clinician for the purpose of completing previsit questionnaires.
Study Sample
The caregivers of children between 8 and 11 months of age presenting for a 9-month well-child visit with the appropriate care teams at each site were eligible to participate.
Study Instruments
Infant Developmental Inventory
Derived from the Child Development Inventories, and part of the Child Developmental Review, the IDI is designed for infants 0 to 18 months old.8 The IDI has a reported sensitivity of 0.85 and specificity of 0.77. 12
The IDI was scored using published instructions in the screening manual.8,11 “Probable pass” score was created by investigators to categorize incorrectly completed IDI screens which contained at least 2 check marks above the 70% line in each of the 5 developmental domains. 11
Ages and Stages Questionnaire–3
The Ages and Stages Questionnaire–3 (ASQ) is used to identify developmental delay in children aged 1 month to 5.5 years. The screen takes 10 to 15 minutes to complete 9 and can be scored in 2 to 3 minutes. 13 The ASQ has a reported sensitivity of 86% and specificity of 85% 13 and has been recently validated in low-risk children. 14 The ASQ-3 was scored as directed using published instructions. 9
Study Protocol
On checking in, caregivers were given either the IDI or ASQ based on assigned alternating days of the week. The patient was brought to an exam room by a licensed practical nurse who scored the screening questionnaire if it was completed prior to completion of the rooming process. Otherwise, the clinician scored the screen before reviewing the result with the caregiver.
If a child failed the screen, the clinician was instructed to make a referral to Early Intervention or pediatric subspecialty services. For children with borderline screen scores or whose parents declined referral for failing scores, follow-up developmental screening was scheduled 1 to 3 months later at either a primary care clinic visit or by completion of a mailed ASQ screen. The clinician would also contact the developmental screening coordinator (DSC), a registered nurse whose time was dedicated to the management of developmental screening within the primary care practice, who would then manage the referral and follow-up process
All developmental screening questionnaires were collected by the DSC, who recorded the child’s sex, race, and insurance status. In addition, the total time of visit (from patient check-in to the clinician leaving the exam room at the end of the clinic visit), care team location, clinician type, and follow-up plan after screening were recorded.
The DSC would also review and record whether the questionnaire administration and interpretation was documented and billed for by the clinician in the medical record. “Complete” documentation was defined as the clinician recording both the administration and result of screening. The interpretation of the questionnaire result (pass or probable pass, borderline, fail) was compared between the DSC and the clinician. “Correct” interpretation meant that the clinician recorded the same outcome as the DSC. If the clinician did not record the result of the screen, “incomplete documentation” was marked as the interpretation outcome.
Caregiver Survey
Caregivers were provided surveys, assessing satisfaction with screening and completion time, at the time of check-in and were asked to complete the questionnaire at the end of the visit. The surveys were collected by nursing staff for investigators.
Statistical Analysis
Comparisons between the groups administered the IDI versus the ASQ screens were evaluated using Fisher’s exact test or χ2 test for categorical variables, the 2-sample t test for continuous variables, and the Wilcoxon rank sum test for caregiver reports by survey. A weighted κ statistic was calculated to estimate the chance-corrected agreement between the clinician and DSC. All calculated P values were 2-sided, and P < .05 was considered statistically significant.
Results
A total of 78 children between the age of 264 and 321 days were included in the study. The demographic characteristics of the 41 patients screened with the IDI and 37 patients screened with the ASQ are summarized in Table 1. No significant differences were detected in gender, age, insurance type, care team, or clinician type. A greater proportion of caregivers of Caucasian children were administered the ASQ compared with the IDI (P = .030).
Summary of Patient and Clinician Characteristics.
Abbreviations: ASQ, Ages and Stages Questionnaire; IDI, Infant Developmental Inventory.
Comparisons between the 2 groups are based on the 2-sample t test for patient age and the χ2 test for all other variables.
Table 2 summarizes the IDI and ASQ screening questionnaire results and outcomes after screening. Out of the 37 children administered the ASQ, the clinician’s interpretation was documented for 33. Of these 33 interpretations, 12 (36.4%) had borderline or failing scores compared with 5 (12.2%) of the 41 children administered the IDI (P = .014).
Screening Questionnaire Findings, by the Type of Questionnaire Administered.
Abbreviations: ASQ, Ages and Stages Questionnaire; IDI, Infant Developmental Inventory.
Comparisons between the 2 groups are based on the χ2 test or Fisher’s exact test as appropriate. Length of visit was compared using the 2-sample t test.
Of of 33 children administered the ASQ and with documented scores, 12 (36.4%) did not receive passing scores, while 5 (12.2%) of the 41 children administered the IDI did not receive passing scores (P = .014).
The developmental screening coordinator only received the clinician scoring summary sheet for the screen and was unable to review the entire screen.
In our study, caregivers correctly completed the ASQ more often as compared with the IDI. Out of 41 IDI screens, 12 (29.3%) were completed incorrectly while no ASQ screens were completed incorrectly (P < .001). The documentation by the clinician of both administration and result of the screen was completed correctly in all 41 IDI screens compared with 33/37 (89.2%) of ASQ screens (P = .046). No significant differences were detected in clinician billing rates when comparing the IDI to the ASQ, but billing was only completed after 35 (44.9%) of all screens. The total length of the visit with the use of IDI was not significantly different from visits using the ASQ.
Agreement in screen interpretation between clinicians administering the ASQ and the DSC was 71.8% (23/32; weighted κ statistic = 0.54). Agreement between clinicians administering the IDI and the DSC was 67.6% (25/37; weighted κ statistic = 0.18).
Responses to caregiver satisfaction surveys (51/78, 65.4%) are presented in Table 3. Based on the responding caregivers, the IDI takes less time to complete than the ASQ [P < .001]; the greatest percentage of caregivers (35.5%) took 10 to 14 minutes to complete the ASQ and 3 to 5 minutes to complete the IDI (55.0%) [P <. 001].
Summary of Responses to the Caregiver Questionnaire, Overall and by Questionnaire Group.
Abbreviations: ASQ, Ages and Stages Questionnaire; IDI, Infant Developmental Inventory.
Comparisons between the 2 groups were evaluated based on the Wilcoxon rank sum test.
Discussion
Our study demonstrates that the IDI and ASQ are both feasible to administer at point of care in primary care practice. Compared with the IDI, the ASQ is more likely to be completed correctly but takes longer to finish and more often requires follow-up for abnormal results.
In our study, there was a significant difference in the percentage of children who did not pass the ASQ compared to the IDI. The failure rate with use of the IDI is consistent with published psychometric properties of the instrument 8 as well as results found in a previous study evaluating the IDI in our study population. 11 Previous studies assessing the ASQ in primary care practices have reported failure rates ranging from 11% to 40%.15-17A recently published study raised concerns about the validity of the ASQ when used in the general pediatric population. 18 In this study of 587 children, the sensitivity and specificity of the ASQ were 41% and 82%, respectively, when compared with the Bayley Developmental Scales. It is plausible that the sensitivity and specificity of the ASQ is lower than the published level in low-risk populations, and this occurrence could explain why more children did not pass this screen in our study. With our small sample size and the fact that we did not compare our tools to a gold standard outcome for development delay, it is not possible to draw any conclusions regarding the validity of the instruments from our study data.
Many of the children in this study who did not pass their developmental screens had borderline scores. Unfortunately, there are currently no recommendations regarding follow-up or referral for children receiving borderline scores and no previous studies specifically evaluating outcomes of children with borderline screening results. One potential approach is close follow-up (1-3 months in this study), at which time the developmental screen is repeated. Additional studies are necessary to evaluate long-term developmental and functional outcomes (qualification for Early Intervention services) for children who do not pass developmental screens.
We found that caregivers correctly completed the ASQ more often than the IDI and that there was a discrepancy between clinician and DSC interpretation in about 30% of screens. These findings suggest that effective education on administration and interpretation of paper format screening instruments is an important component of developmental screening implementation. Alternatively, having specifically trained personnel, such as the DSC, assigned to score, interpret, and follow up on abnormal screening questionnaires may help improve the accuracy of the screening process. Developmental screening using electronic instruments may also lead to improved screening rates without affecting the flow of clinic visits. 19
There could be multiple explanations for the discrepant ASQ and IDI correct completion rates. The ASQ cannot be easily interpreted without correct completion, as all of the questions need to be answered to obtain an evaluation of each developmental domain. In contrast, the IDI, in the paper form, is a checklist, and it is plausible that clinicians take liberty on how much of the checklist to complete before making their interpretation of the screen result. Additionally, clinicians in our study used the IDI as part of routine practice before the study and may have become comfortable with their typical use of the screen, even if divergent from published administration and interpretation instructions. This fact could also potentially explain why the IDI was documented more often in the medical record than the ASQ because clinicians are more familiar with use of the tool.
To investigate the efficiency of the IDI and ASQ in clinical practice, this study describes the time needed for caregivers to complete questionnaires in the context of the total visit time. Caregivers subjectively reported that the ASQ took more time to complete than the IDI, but no statistically significant difference was found in the total objective length of visit time when comparing the IDI and ASQ in our study. Because total length of visit was recorded from patient check-in to clinician leaving the room, our study protocol could not adequately assess the many other factors that may affect the time of a visit, including time spent in the waiting room, time dedicated to developmental screening discussion, and complexity of the patient’s needs. Further studies should use objective time measures to assess the effect of developmental screening on length and flow of clinic visits.
Strengths of our prospective study include a prestudy orientation and education session for all participating staff as well as a standard check-in and rooming process. In addition, our study design allowed for evaluation of the usefulness and efficiency of the screening instruments as they are used within the primary care practice site setting. Limitations of our study include a small sample size, no direct comparison of the IDI and ASQ in each individual patient, and no gold standard measurement for developmental delay in study subjects. There was also lack of data assessing long-term developmental follow-up for children in the study.
Conclusion
The IDI and ASQ are both practical to administer at point of care. For primary care clinicians, the choice of instrument likely depends on the baseline presence of developmental disorders in their patient population as well as their capability of follow-up and referral for abnormal results.
Footnotes
Acknowledgements
The authors thank Whitney L. Landsteiner, RN, for her aid in data collection. In addition, the authors would like to thank the family medicine and pediatric care team nurses and desk staff.
Authors’ Note
The contents of this article are solely the responsibility of the authors and do not necessarily represent the official views of the National Institutes of Health.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This article was supported by National Institutes of Health/National Center for Research Resources Center for Translational Science Activities Grant UL1 RR024150. The Mayo Clinic–Rochester Department of Pediatric and Adolescent Medicine provided discretionary funding for the statistical analysis.