Abstract
PURPOSE:
Development of an easy to use tool for the assessment of Developmental Coordination Disorder that is applicable to Indian children.
METHODS:
The steps for instrument development were followed stringently. In order to ensure the robustness of the tool, psychometric properties were determined with 127 children attending various schools. The Winsteps program was used to analyze data for a Rasch model analysis.
RESULTS:
The Kaiser-Meyer-Olkin Measure of Sampling Adequacy (KMO) analysis showed an acceptable limit. The real item separation reliability of 4.84 demonstrates that the items of this tool create a well-defined variable.
CONCLUSION:
This study met the objectives that it attempted to. The final tool consists of two versions- the Assessment Battery for Children with coordination Disorder-Short Version (ABCD-SV) consisting of 12 items, and the ABCD-FV consisting of 20 items. ABCD fulfills criterion A and B of the Diagnostic and Statistical Manual of Mental Disorders (DSM-IV-TR) criteria for Developmental Coordination Disorder (DCD).
Keywords
Background
Although covert disabilities of childhood are minor in isolation, they may have a profound effect on academic performance and overall psychosocial development in children. One of the plethora of minor childhood disabilities is Developmental Coordination Disorder (DCD) [1]. It has been demonstrated that children with DCD have marked impairments in motor coordination, which interfere with their participation at home and academic activities, as specified in the Diagnostic and Statistical Manual of Mental Disorders (DSM-IV TR) [2].
DSM-IV-TR has laid down four criteria for the diagnosis of DCD wherein two are inclusive (Criteria A and B) and the other two are exclusive (meeting Criteria C or D would eliminate diagnosis). Criterion A stipulates that: “Performance in daily life activities that require motor coordination is substantially below that expected by age and intelligent quotient (IQ). This may be manifested by marked delays in achieving motor milestones, dropping things, ‘clumsiness’, and poor performance in sports or poor handwriting.”
Criterion B states that “The disturbances in Criterion A significantly interferes with academic achievement or activities of daily living and must be measured within the context of the situation.” DCD pertains to the unaccountable (not explained by any medical diagnosis) failure of children to acquire age-appropriate motor skills. DCD has been recognized as an entity in the West for several decades. However, it is as yet not a well-known condition in India. This is evident from the paucity of literature from India [3, 4].
DCD has been shown to have effects on the child’s overall development, physical activity, and academic performance. Studies have shown that children with DCD are often obese. As these children are not high athletic performers, they tend to get lesser amounts of physical activity eventually leading to obesity [5, 6]. Therefore, there is a need for the early and accurate diagnosis of DCD. The standardized tools commonly used to identify DCD in children are subsets of the Bruininks-Oseretsky Test of Motor Proficiency (BOTMP), and the Movement Assessment Battery for Children (MABC-2). However, neither of these tools are specific to DCD. The cultural context of the activities themselves are inadequately explored for Indian children. Moreover, these tests are proprietary and costs are prohibitive for low resource settings such as public schools in India and similar low and middle income countries [7].
In order to screen children for DCD, both A and B criteria of DSM-IV-TR must be objectively evaluated under standardized conditions. Consequently, the objectives of this study were as follows:
Development and preliminary evaluation of psychometric properties of a
tool to aid in diagnosis of DCD that is applicable to Indian children. To evaluate the ease of use (ability for a novice administrator to
perform the test using the user manual and easy availability of test materials) of the
tool in practice.
Materials and methods
Ethical clearance was granted by the Manipal University Ethical Committee. To achieve the above objectives, the study was divided into two phases.
Phase I
Development of a performance tool for assessment of DCD and preliminary testing of psychometric properties. In order to meet this objective, a 4-step procedure was followed.
Step 1: Item generation and development of scoring system
Literature on DCD was extensively searched and domains of difficulty were identified. These domains were made as comprehensive as possible in order to form an exhaustive list of possible dysfunctions.
From the literature, the following domains were extracted: in-hand manipulation, sensory-motor coordination, and motor diversity [8]. In-hand manipulation included manipulation of various items within the hand for academics and activities of daily living. The sensory-motor domain included gross motor, fine motor, co-curricular, and self-care activities. Motor diversity included gross motor activities. After the domains were differentiated, test items were generated by a minimum of two professionals working in the domain-specific discipline and familiar with normal child development (4 physiotherapists, 4 occupational therapists, 2 speech and language pathologists, and 4 teachers). All of the professionals who participated in the study had at least 5 years of experience with children and/or pediatric disorders including DCD. The physiotherapists generated items for gross motor function in the sensory-motor and motor diversity domains, occupational therapists for the in-hand manipulation and sensory-motor domains; speech language pathologists for the sensory-motor coordination domain, and teachers in all domains. Participating experts were cautioned to select items that a child of the target age group is typically expected to perform. Thus, all items generated were activities that children typically perform.
After item generation, all individuals involved in this process decided on the scoring criteria using a rating scale [9] which allows the scorer to indicate the degree or frequency of behavior, skill, or strategies displayed by the test taker. The tool and scoring were decided after a consensus meeting. The initial scoring criteria was on a 10 point Likert scale ranging from 1–10 with higher scores indicating better performance. Time taken to complete the tasks, errors, quality of movement, and proximal stability were considered as indicators for scoring. Each of these constructs was given equal weight.
Step 2: Face validation
The completed tool was given to two professionals (PT/OT) who were
not in the item generation group for face validation and obtaining first version of ABCD
(ABCD-1
Step 3: Construct validity
In order to ensure the robustness of a tool, construct validity must be determined. For this phase of the study, 60 typically developing children without any disease or dysfunction that may potentially affect their motor performance (as per report of parent and teachers) were recruited from various schools. Children who participated in the face validation were not included in this phase. The procedure followed for participant selection was as follows: the Deputy Director of Public Instruction (DDPI) Udupi, who is the official in charge of all mainstream schools, was approached for permission to conduct the study in Udupi District. A list of schools in Udupi district were identified and schools were randomly selected. The head teachers of the selected schools were approached. Informed consent was obtained from parents and the child’s verbal assent was taken before the tool was administered to the child. The data from 60 children were analyzed.
Step 4: Preliminary testing of psychometric properties
To establish normative values for the tool, a school in Ujjain, Madhya Pradesh (a location culturally distinct from Udupi) was included. Permission was obtained as mentioned in step 3. One hundred and twenty-seven children [10] between 6–15 years of age were recruited and tested.
In order to assess concurrent validity, the MABC-2 was administered to forty of the same children. Comparison of scores from items that were similar in construct were expected to reveal whether the items on ABCD were valid: e.g. scores on the subset of in-hand manipulation on ABCD was compared with scores on manual dexterity on MABC-2. In order to ensure test-retest reliability of ABCD, 20 typically developing children and 20 children having motor difficulties, as identified by the class teacher using DCDQ, were recruited. All 40 children were administered the ABCD during the same time of day on two different occasions with an interval of seven days between tests. To decrease the experimenter bias, the examiner was blinded to the child’s score from the first assessment.
Following the second administration of ABCD, those children identified as having motor difficulty on ABCD were given a motor remediation program which was incorporated into the regular curriculum by the physical education or craft teacher for a duration of six weeks. Following this program, the ABCD was administered again in order to assess the responsiveness of the tool to changes in motor skill.
Phase II
The final tool with materials required, administration instructions, scoring, and interpretation was included in a user manual. The purpose of this phase was to evaluate if a novice administrator would be able to get consistent scores by referring to the manual without formal training. We consider this as an important aspect for applicability of the tool. The expectation is that the tool may be used by teachers or healthcare professionals to screen children with DCD for early detection and intervention.
A novice administrator who is a physiotherapist (fourth author) undertook this phase of the study. With the help of the manual, he put together a test kit and administered the test twice, with an interval of five days, on five children between the ages of 12 and 14 years. Of these, two children were identified as having motor difficulties by the concerned teacher.
Data analysis
Phase I: Tool development
Item generation
Descriptive methodology was employed.
Face validity
Descriptive analysis was performed.
Construct validity
Tool evaluation was performed using both traditional classical test theory (CTT) and item response theory (IRT) methods.
CTT analysis: Exploratory factor analysis was conducted using the
Statistical Package for the Social Sciences (SPSS 16.0). Reliability and validity of the
tool was assessed using traditional CTT approach. The construct validity was assessed by
principal component analysis (PCA). Sampling adequacy for PCA was determined if the
Kaiser-Meyer-Olkin Measure of Sampling Adequacy (KMO) for overall data was
IRT analysis: In order to examine the validity of ABCD tool as a performance tool for criteria A and B of the DSM-IV-TR, Rasch model measurement analysis was performed using the Rating Scale Model (RSM) in WINSTEP 3.74 software [12]. The criteria of scale functioning examined under Rasch were:
The reliability of measure location was assessed by separation
indices, item and person reliability. For the instrument to be sensitive to
distinguish between high and low functioning performers, the acceptable value of
separation indices is 2.0 with person reliability The effectiveness of the 5-point response scale was confirmed
through Rasch Likert scale category functioning analysis. Ten observations per
category is considered to be ideal for assessing category functioning. Category
frequency, Step measure (Rasch-Andrich threshold), average measure and category
fit statistics were used as diagnostic tools. Recommendations for ideally
functioning categories include the following:
The step measure parameter defining the boundaries between
categories should increase monotonically. Average person measure for each step should be higher than
the average person measures of previous step in the person item map. A score with categories which are mutually exclusive and
hierarchical in nature is said to be a well-functioning score. Likewise, a person
item map represents the average number of persons who scored at each level. In a
well-functioning scale, this is expected to be orderly. Two fit indices (infit and outfit mean square (MNSQ) statistics)
were used to investigate whether all items contributed adequately to the scale
construct. Infit MNSQ is more sensitive to the unexpected response of persons
whose abilities are near item difficulty, while outfit is more sensitive to
unexpected outlying observations. A MNSQ value in the range of 0.6–1.4 is
considered as ideal. A MNSQ value greater than 1.4 indicates that the item fails
to define the same construct as other items do. MNSQ values lower than 0.6 may be
an indication of item redundancy. Outfit means square should be less than 2 [14]. The person-item map was generated to visually inspect the targeting
of item difficulty to the expected age-appropriate ability of the child. Optimal
targeting occurs when a set of items in a tool covers the full range of scores in
the population.
Preliminary testing of psychometric properties
Concurrent validity with MABC-2 was assessed using Pearson’s
product-moment correlation. Test-retest reliability was confirmed through intra-class
correlation coefficient (ICC); a two factor mixed effect model was used to assess the
test-retest reliability of ABCD. ICC of
Responsiveness was determined through the adjusted effect size computed by dividing the difference between the scores by the pooled standard deviation [16].
Phase II
Analysis of the second phase of the study was done descriptively by compiling information gained through an informal interview.
Results
Phase I: Tool development
Item generation
Nineteen items were initially generated by the experts. Each item was rated on a 5 (0–4) point Likert scale wherein zero equated to the weakest performance and 4 to the best performance. Following the consensus meeting, it was decided to change the scoring criteria to a 10-point scale (1–10), since it was argued that a five-point scale may not have the potential to discriminate adequately. The weighting criteria remained unchanged.
Factor loadings on pattern matrix for Assessment Battery for Children with
Developmental Coordination Disorder-Full Version (ABCD-FV)
Factor loadings on pattern matrix for Assessment Battery for Children with Developmental Coordination Disorder-Full Version (ABCD-FV)
Extraction method: Principal Component Analysis; Rotation Method: Oblimin With Kaiser Normalization.
Factor loadings of items on forced rotated factor analysis for Assessment Battery for Children with Developmental Coordination Disorder-Short Version (ABCD-SV)
Extraction method: Principal Component Analysis; Rotation Method: Oblimin With Kaiser Normalization.
Assessment Battery for Children with Developmental Coordination Disorder full version and short version
Note: items marked with * form ABCD-SV.
Reliability and separation indices of Assessment Battery for Children with Developmental Coordination Disorder (ABCD) tool
Rating scale diagnostics of Assessment Battery for Children with Developmental Coordination Disorder Short Version
Item fit statistics of Assessment Battery for Children with Developmental Coordination Disorder Short Version and Assessment Battery for Children with Developmental Coordination Disorder Full Version
Note: Items Eat and Read of ABCD-FV exceeds 1.4 which could be improved or dropped.
The 19 test items when subjected to face validation yielded 16
items; after deletion of three items for the clarification of the operational definition
of the score, thus obtaining ABCD-1
The scores from the initial 20 children that were administered the
test in this phase were analyzed and it was noticed that there was no differentiation
below a score of 5. Accuracy, quality, and compensation were all considered in the
scoring. Therefore, the scores were collapsed to 5, again with modified operational
definitions. ABCD-1
Construct validity
Principal Component Analysis: After testing the assumption for PCA
(linearity of variables and presence of outliers within 3 SD), PCA was run on the 20
item scale with oblique rotation. Inspection of correlation matrix showed that all
variables had one correlation coefficient
IRT analysis: Rasch derived item and person separation indices and
item reliability of both versions of ABCD are depicted in Table 4. Acceptable values of item reliability
(
Item map of Assessment Battery for Children with Coordination Disorder-Short
Version (ABCD-SV).
Item map of Assessment Battery for Children with Coordination Disorder-Full
Version (ABCD-FV). In the above figure, (the Wright Map) items are indicated by
their short name, and each individual person’s performance is represented by (#)
or (.).
Moreover, infit and outfit MNSQ statistics were acceptable for all the categories. The average measure increased monotonically across rating scale categories and there were no disordered threshold values (Table 5). Ordering of the threshold could also be observed in the corresponding category probability curve. For instance, intersection of category 1 and 2 is located to the left side of that of categories 2 and 3, hence indicating no disordering of threshold values.
For ABCD-SV, mean square infit ranged from 0.69 to 1.09 and outfit mean square value ranged from 0.72 to 1.16 for all items. For ABCD-FV, infit mean square ranged from 0.71 to 1.50 and outfit mean square value ranged from 0.71 to 1.51 for all items (Table 6).
The person item map shows the distribution of the children’s ability score (left side of the Fig. 1) and item difficulty (right side of the Fig. 1) for each item of ABCD-SV (Fig. 1) and ABCD-FV (Fig. 2). Children with higher ability and items with more difficulty were located at the top of the map.
Concurrent validity: tool items were valid against test items in corresponding domains of MABC-2 (Table 7).
Correlation with Movement Assessment Battery for Children (MABC-2)
Correlation with Movement Assessment Battery for Children (MABC-2)
Test-retest values of Assessment Battery for Children with Developmental
Coordination Disorder (
Test-retest reliability: All the items had ICC values
Responsiveness: Effect size was
Most standard tests are proprietary in nature and are expensive to procure. One objective of this study was to develop a test that can easily be replicated by anyone using the test manual which is descriptive and gives dimensions of all test items and scoring criteria. In order to assess the feasibility of procuring items easily and administering the test using only the test manual, we conducted this phase of the study. The results for this phase were compiled based on informal feedback received from the novice test administrator. The novice administrator reported that he was able to administer the test to the children after studying the manual and was able to achieve more than 70% concurrence in scoring between the two test administrations. This suggests that the manual written for ABCD is self-explanatory and easy to follow and that test items are available without difficulty. Consequently, ABCD can be compiled easily without proprietary material, thereby making it easily accessible globally. The test manual can be obtained from the authors freely upon request. The overall impression from the administrator was that although time consuming and without special training or pediatric expertise, it is feasible to administer ABCD to children attending mainstream school. The scores obtained by the novice and one of the other authors’ revealed 60–75% concurrences. Due to the very small sample size (5 children), no statistics were employed for this section. It was observed by both test administrators that children were interested in completing the test since the items were enjoyable and therefore boredom did not seem to be a concern.
Discussion
Two versions (ABCD-FV and ABCD-SV) of the final tool were developed in this study. Both of the tools were comprised of three constructs with 20 items in ABCD-FV and 12 items in ABCD-SV. These constructs are related to in-hand manipulation, motor diversity, and sensory-motor coordination.
The total possible score for ABCD-FV is 80 and ABCD-SV is 48. The motor proficiency of children increases with age. Hence, the discriminative factor would be time. Children’s qualitative performance must be taken along with timing. The authors recommend that scores be interpreted based on percentiles which have been computed for the children involved in this study. Due to various contextual factors including cultural representativeness and prior exposure, the authors also suggest that normative values should be computed for populations that are different from ours. Interpretation must be made only on the percentile comparison of the total score.
A child who is suspected of DCD through behavioral observation or the
Developmental Coordination Disorder Questionnaire (DCDQ) [17], may be administered the ABCD-SV as a screening tool. If a child scores less
than the 20
This tool is only intended to evaluate two of the criteria (A and B) along with parental reports. Diagnosis of DCD requires fulfillment of the other two criteria as well. Individual item percentile scores are given to aid the tester in identifying particular difficulties that the child may have. These are not meant to be used for diagnosing DCD because difficulty in one domain does not fulfil diagnostic criteria for DCD. When this study started, the DSM-IV-TR was used as the guiding factor for tool development. However, the diagnostic criteria described in the DSM-V does not conflict with the purpose and constructs of this tool.
This study presents the initial results of ABCD tool development. Due to the relatively small prevalence of DCD, further validation will require a large population and is expected to take considerable time. The applicability of this tool to populations other than school children in India must also be evaluated since certain differences in culture and habit may play a role in this factor.
The test is a compilation of common activities. Hence, individual items may be used in remediation of special difficulties; for instance, if a child has difficulty in the gross motor items, training using the test items may help in achieving the skill. This is expected to minimize the misdiagnosis of children with DCD.
A part of this study is intended to gauge an impression of the use of this tool among teachers and therapists. There was significant difference in scores post intervention. There was no measurable improvement in the academic performance as the duration of the intervention was small. Teachers were interviewed about the feasibility of incorporating the testing and intervention in the school curriculum. Teachers commented that testing is time consuming, however, intervention could easily be embedded in the school curriculum.
Factors such as time and availability of testing materials must be studied to conclusively establish whether the tool is a stable measure of performance. This aspect should be studied further over a large population. The administration of this tool is time consuming, since it requires following the prescribed protocol for evaluation. Nevertheless, the ABCD would make an important addition to aid in the identification of children with DCD so that remediation programming may be initiated early.
This tool was developed using stringent research methods and valid statistical measures. However, this study is not without its limitations. The sample size that was used in construct validation is the minimum recommended. We suggest that further analysis must be performed for full validation. Some of the items on the tool may be mediated with prior experience. Academic skills such as reading and writing have a hidden construct of academic exposure; these items cannot be interpreted in isolation. The prevalence of DCD is low, and hence only 10 children were flagged as possible DCD cases with the ABCD. Of these, only 7 children fulfilled all criteria for DCD diagnosis. Therefore, the results of this study are limited to preliminary indications. Further validation of the tool will be required to provide more conclusive evidence. The ABCD scores must be correlated with DCDQ scores over large numbers of children in the future. Longitudinal validation is also recommended. The observations from phase II are informal and they must be validated using formal research methods.
Conclusion
This study met the objectives that it attempted to. The final tool consists of two versions ABCD-SV and ABCD-FV consisting of 12 and 20 items respectively which fulfills criterion A and B of DSM-IV-TR and DSM-V criteria for DCD. If children are identified as deficient in motor performance in the SV, the authors recommend that they should be examined further for DCD.
Footnotes
Acknowledgments
The authors would like to acknowledge the assistance given by Mrs. Sushma Ganasan, various school administrators, teachers, parents, and children.
Conflict of interest
The authors have no conflict of interest to report.
