Sage Journals: Discover world-class research

Abstract

Background and aims

Praxis, the ability to plan and execute a series of gestures or motor sequences, is impaired in individuals with autism spectrum disorder. In this paper, we present the first study of praxis during a gesture imitation task in a unique population of children with autism spectrum disorder: deaf children who have been exposed to American Sign Language from birth by their Deaf parents. Lifelong exposure to sign language in deaf individuals entails practice with gesture imitation. We ask if deaf, signing children with autism spectrum disorder present with praxis impairments similar to those reported in the literature for hearing children with autism spectrum disorder not exposed to sign.

Methods

Thirty deaf children of Deaf parents (16 typically developing and 14 with autism spectrum disorder), matched for chronological and mental age, were tested on a simple gesture imitation task. Children were asked to imitate 24 gestures performed by an adult model on a laptop computer that varied along dimensions of movement type (16 trials) and palm orientation direction (eight trials). Data were coded for nine praxis parameters (five formation parameters and four manner parameters) and three timing measures.

Results

Results showed that the deaf children with autism spectrum disorder produced more errors than the typically developing deaf children on six of the nine praxis dimensions (hand orientation, final orientation, modulation, directness, pace, and overflow) and were more impaired on manner parameters than formation parameters, suggesting underlying deficits in motor control/coordination leading to dyspraxia. Praxis scores were strongly related to severity of autism spectrum disorder symptoms and receptive sign language scores.

Conclusions

We thus find evidence that praxis is impaired in deaf, signing children with autism spectrum disorder, despite lifelong exposure to a gestural language and extensive practice with gesture imitation.

Implications

Our findings suggest that interventions targeted toward enhancing praxis and coordination of finger and hand movements could help facilitate language development in children with autism spectrum disorder.

Keywords

Gesture imitation deafness autism spectrum disorders dyspraxia

Gesture¹ is an essential feature of human communication systems for both hearing people (who gesture as they speak) and for deaf people (for whom manual gestures are linguistic symbols). Deaf children acquiring signed languages must become adept at imitating gestures in order to learn linguistic symbols. This entails perceiving the gesture produced by another person; transposing those movements onto one’s own body; and reproducing the gesture, including aspects of motor planning, coordination, and execution. Gestural communication is clearly advanced in children who are exposed to signed languages such as American Sign Language (ASL), insofar as they communicate primarily in the visual–gestural modality. In contrast, gestural communication is universally impaired in children with autism spectrum disorder (ASD), whether or not such gestures are self-initiated (Mundy & Sigman, 2015) or in response to commands of others (Mostofsky et al., 2006). Difficulties in gestural imitation originate in toddlers and young children with ASD (Charman et al., 1997; Rogers, Bennetto, McEvoy, & Pennington, 1996; Stone, Ousley, & Littleford, 1997) and are thought to continue into adulthood (Avikainen, Wohlschläger, Liuhanen, Hanninen, & Hari, 2003; Bernier, Dawson, Webb, & Murias, 2007; Hobson & Lee, 1999; Rogers et al., 1996).

Young children with ASD often show deficits in prelinguistic pointing gestures (Özçaliskan, Adamson, & Dimitrova, 2015) and in gestures stemming from joint attention (Mundy, Sigman, & Kasari, 1990) between 21 and 45 months of age compared to typically developing (TD) controls matched for chronological, mental, and language age. Older children with ASD between the ages of five and 18 show significant difficulties in imitating both meaningless and meaningful (i.e. communicative) gestures compared to age-matched TD children and children with developmental delays, with greater impairment when producing meaningless gestures (Stone et al., 1997). Imitation deficits in children with ASD are thought to be related to children’s general motor ability levels (Vanvuchelen, Roeyers, & De Weerdt, 2007) and have been proposed to be part of a more generalized impairment in praxis (Dewey, Cantell, & Crawford, 2007; Mostofsky et al., 2006). In this study, we investigate whether deaf, signing children with ASD, between the ages of five and 14, also show gestural impairments compared to TD deaf, signing children, or if their extensive experiences with gestural language during infancy and early childhood might attenuate such challenges.

Praxis in ASD and relation to other skills

Praxis is the ability to plan and execute a series of gestures or motor sequences in imitation, on verbal command, and during tool use (Dewey, 1993; Mostofsky et al., 2006). Praxis skills are important both for learning new motor skills such as tying one’s shoes, as well as for communicating during social interactions. TD children improve their praxis skills over development and reach adult-like performance by 12 years of age (Dewey, 1993). There is considerable evidence for impaired praxis performance in children with ASD compared to age-matched TD children between the ages of five and 18 (Dewey et al., 2007; Mostofsky et al., 2006; Vanvuchelen et al., 2007). Specifically, children with ASD show greater errors during praxis in imitation, on verbal command, and during tool use compared to age-matched TD children using different standardized praxis measures (Dowell, Mahone, & Mostofsky, 2009; Mostofsky et al., 2006; Srinivasan, Lynch, Bubela, Gifford, & Bhat, 2013) as well as during performance of nonstandardized actions or gestures (Dewey et al., 2007; Vanvuchelen et al., 2007). Some of the commonly reported praxis errors in children with ASD include spatial errors (e.g. incorrect positioning of the moving limb in space), reversal errors (e.g. waving with palm facing inward), as well as timing errors (e.g. delayed reaction times) (Dewey et al., 2007; Mostofsky et al., 2006; Ohta, 1987).

Signs are typically analyzed as being composed of a hand configuration (a handshape and palm orientation), a location (either on the body or in the signing space), and a movement (Stokoe, 1960). In this paper, we distinguish between errors of formation or manner. Formation errors encompass hand configuration (the general characteristics of the gesture in terms of orientation to the partner) and location (placement of one’s own limb or body in space). In contrast, manner errors encompass the movement qualities of the gestures performed. Formation errors in children can be attributed to perceptual causes, i.e. perception of self and other or perception of one’s own limb or body in space (Smith & Bryson, 2007). It has been hypothesized that impaired self–other mapping may affect an autistic child’s ability to perform imitative actions (Rogers & Pennington, 1991; Williams, Whiten, & Singh, 2004). By contrast, manner errors are clearly linked with impairments in motor control and coordination, which are often reported in children with ASD (Bhat, Landa, & Galloway, 2011; Vanvuchelen et al., 2007). In this study, we group various types of praxis errors into formation and manner categories in order to determine which may be more impaired in children with ASD and thus to determine if such errors are more attributable to perceptual or motor causes.

Praxis performance in children with ASD correlates with other skills such as gesture recognition, receptive language, and autism severity, but not intelligence (Dowell et al., 2009; Dziuk et al., 2007; Gizzonio et al., 2015; Stieglitz Ham et al., 2011). Specifically, praxis performance measured on the Florida Apraxia Battery correlated with social communication scores obtained from the Autism Diagnostic Observation Schedule (ADOS; Lord et al., 2012) (Dowell et al., 2009). Studies conducted by this group and others have reported an association between gesture recognition (i.e. receptive communication) and gesture performance (i.e. expressive communication) in older children with ASD (Dowell et al., 2009; Stieglitz Ham et al., 2011). Clearly, gesture perception interacts with and is integrally linked to gesture production. Taken together, these studies suggest that praxis and imitation impairments may interact with and possibly contribute to some of the social and receptive communication deficits in hearing children with ASD.

Praxis in signers with ASD

Children with ASD acquiring signed languages are an interesting population to investigate in terms of praxis performance because the ability of such children to move their limbs to imitate accurately is paramount for their language production. Two prior studies examining the relationship between praxis performance and sign language production in hearing children with ASD have found a relationship between language and motor skills (Seal & Bonvillian, 1997; Soorya, 2003). Seal and Bonvillian (1997) found that sign vocabulary size and accuracy significantly correlated with praxis and fine motor scores in children with severe ASD who were learning to sign, while Soorya (2003) found that children with ASD had more praxis errors and poorer motor performance than age-matched TD children; she also found a strong relationship between praxis performance and sign production in hearing children with ASD.

Though these studies provide some evidence supporting the influence of praxis performance on sign language acquisition and production in hearing children with ASD, they tell us little about the impact of motor impairments in children with ASD who are acquiring a sign language from birth, such as deaf children. Such children constitute an interesting population in which to study the development of praxis and its impact on ASD because signed languages employ a complex series of communicative manual and facial movements that are quite different from the movements of the vocal tract entailed in spoken languages. Furthermore, studying deaf children who are exposed to a sign language from birth by their Deaf² parents (i.e. native signers) would be informative because such children likely have far more exposure to and practice with gesture imitation than their hearing counterparts. We, therefore, see native-signing children as a unique population in which to investigate praxis performance in ASD. Studies of such children could shed light on the relationship between motor skills and language generally and provide insight into the creation of possible motor interventions for children with ASD.

To our knowledge there is only one published report of praxis performance in native sign language users. Bhat, Srinivasan, Woxholdt and Shield (2016) compared the ability of 22 deaf children (11 TD and 11 with ASD) to execute sequences of complex handshapes entailed in fingerspelling (manual representations of English letters). They found that the deaf children with ASD produced more errors in pace, sequence precision, accuracy, and body part use than the TD deaf children and also took longer to fingerspell each word. Additionally, the deaf children with ASD had lower receptive language scores than the TD deaf children and their language scores were strongly correlated with their praxis performance and ASD severity. Taken together, these findings suggest that praxis is impaired in native sign language users with ASD and that the degree of motor impairment is strongly related to language development. However, the impairments detected may have been due, in part, to the complexity of the fingerspelling task itself: children had to first recognize visual linguistic symbols (English letters) printed on a card (in effect, a literacy task), match each letter to the corresponding handshape in their mental lexicon, retrieve that representation from the lexicon, and translate the representation into a hand gesture representing each letter in sequence. As this is a multistep, complex process, it is possible that the poorer praxis performance by children with ASD could have been caused by the task’s demands rather than purely by an impairment in praxis skills per se.

In this paper, we ask if deaf children with ASD exhibit impairments in praxis performance on a nonlinguistic gesture imitation task that lacks the linguistic decoding and phonological retrieval demands of the fingerspelling task described in Bhat et al. (2016). If praxis is broadly impaired in ASD, then we would expect group differences on this task similar to the findings of Bhat et al. (2016). If group differences are not found, however, then we could conclude that praxis is only impaired when children are faced with complex, multistep tasks, but not in simple gesture imitation.

Method

Participants

The children who participated in the study were part of a sample that has now been described in a number of published studies (Bhat et al., 2016; Shield, Cooley, & Meier, 2017; Shield, Meier, & Tager-Flusberg, 2015; Shield, Pyers, Martin, & Tager-Flusberg, 2016). Each of these papers has reported on slightly different numbers from this sample because children were given a battery of cognitive and linguistic tasks and not all children completed all tasks. For this study, 14 ASD deaf children (11 males and three females) and 16 TD deaf children (six males and 10 females) between five and 14 years of age completed the protocol described in this paper.

As with our previous studies, none of the children had received a cochlear implant, and all had been exposed to ASL as their primary language since birth, either because one or both of their parents were themselves Deaf and used ASL as a primary means of communication at home (29 of the 30 children) or because the parents themselves were raised with ASL as their first language by Deaf parents, though they were themselves hearing (one child). In addition, five participants were excluded from the study because they were nonverbal and did not respond to the testing measures. Recruitment was conducted using an online video in ASL posted on YouTube and Facebook as well as through schools for the deaf. The first author, a hearing male fluent in ASL, conducted all research visits at the child’s home or school. Written parental consent was obtained for all subjects prior to testing, and a duly constituted Institutional Review Board prospectively approved the study procedures.

We used the Social Communication Questionnaire (SCQ) (Rutter, Bailey, & Lord, 2003) to screen all children for ASD. All TD children scored well below the clinical cutoff score of 15 (M = 2.44, SD = 2.97, range = 0–10) and were significantly different from the ASD group (M = 13.71, SD = 7.46, range = 4–31) (t(28) = 5.58, p < 0.0001). There are currently no instruments designed to identify ASD in deaf children, and the current gold standard, the ADOS-2 (Lord et al., 2012) is explicitly not meant for use with deaf children (Mood & Shield, 2014). However, given the current absence of appropriate instruments, we confirmed ASD diagnosis in the ASD group by adapting the ADOS-2 into ASL (for a detailed description of the modifications made to the ADOS-2, see Shield et al., 2015). Thirteen of 14 deaf children with ASD scored above threshold for ASD classification on the ADOS-2 (ADOS-2 algorithm scores: M = 12.50, SD = 5.21, range 1–20). For the one child who scored below the cutoff threshold (ADOS-2 algorithm score = 1), we relied on the clinical judgment of a native-signing licensed clinical psychologist with training in diagnosing ASD. After reviewing this child’s videotaped behavioral data, the clinical psychologist determined that this child met clinical criteria for ASD, even with a below-threshold ADOS-2 score. Thus, we have included this child in our analysis.

Matching

The groups were not different in terms of chronological age (t(28) = .44, p = .66, ns) or nonverbal intelligence (t(28) = 1.57, p = .13, ns) as assessed using the Test of Nonverbal Intelligence, Fourth Edition (TONI-4; Brown, Sherbenou, & Johnsen, 2010). We also tested the children for receptive sign language ability using the ASL Receptive Skills Test (ASL RST) (Enns, Zimmer, Boudreault, Rabu, & Broszeit, 2013). The ASL RST is a standardized ASL comprehension measure for children between three and 13 years of age (Allen & Enns, 2013). During this test, children viewed signed sentences of increasing complexity on a laptop computer and pointed to one of four pictures that best matched the preceding signed sentence. The TD deaf children had significantly higher raw scores on the ASL RST than the deaf children with ASD, t(28) = 5.55, p < .0001. Therefore, the groups were not matched for language ability. For a summary of the characteristics of the two groups, see Table 1.

Table 1.

Group characteristics (SD in parentheses).

	Chronological age	TONI-4 standard score	ASL RST raw score
Deaf with ASD (N = 14, 3 F)	9.48 (2.85)	98.07 (8.86)	15.50 (11.02)
TD deaf (N = 16, 10 F)	9.10 (1.80)	103.38 (9.50)	32.75 (5.43)

ASD: autism spectrum disorder; ASL RST: American Sign Language Receptive Skills Test; SD: standard deviation; TD: typically developing; TONI-4: Test of Nonverbal Intelligence, Fourth Edition.

Protocol

Gesture imitation task

A task was designed to test children’s ability to imitate simple gestures. All video stimuli involved an adult model dressed in black executing the gestures in front of a green screen. A model unfamiliar to participants was used in order to achieve uniform stimulus presentation. The task consisted of 16 movement trials, in which the model performed a single arm movement, and eight palm orientation trials, in which the model held a static hand configuration. For the movement trials, each gesture consisted of a single arm movement produced by the model using her right hand in one of four directions: up (four trials), down (four trials), inward toward the signer (four trials), or outward away from the signer (four trials).

In addition, half of the trials were filmed from a front perspective (as if the child were facing the model head-on), while the other half of the trials were filmed from a side perspective (a 90° shift) (see Figure 1). Stimuli also varied in their rate of execution: the 12 “normal” gesture stimuli lasted an average of 1.2 s, while the two “fast” trials lasted an average of 0.85 s, and the two “slow” trials lasted an average of 2.74 s. This speed variable was included to see if children spontaneously matched the rate of gesture execution. The model held her hand in a B-handshape (with thumb retracted and the four fingers extended) for all movement trials (see Figure 1).

Figure 1.

A sample front-perspective (left) and side-perspective stimulus (right). In these examples the movement that the model is about to execute starts close to her body and proceeds outward.

The eight palm orientation trials consisted of the model holding a static handshape. The stimuli differed in the direction that the model’s palm faced (either up, down, inward toward the model, or outward away from the model) and in the model’s hand configuration. Half of the trials were produced with an O-handshape, while the other half were produced with the more complex R-handshape, with the middle finger crossed over the index finger (see Figure 2). Again, half of the trials were from a front perspective, while the other half of the trials were filmed from a 90° side perspective.

Figure 2.

Examples of the palm orientation items. On left, the O-handshape facing outward; on right, the R-handshape facing inward.

The children were seated in front of a laptop computer and each gesture was played one at a time, with time allotted for the children to imitate what they had seen; the next trial was not shown until the child had imitated the prior gesture and was advanced via remote control by the investigator, who sat behind the child. The camera was set up at approximately a 45° angle behind the child, so that both the child and the laptop screen were visible at the same time. Video instructions were provided in ASL prior to the task and two practice items were given prior to testing to make sure that children understood what they were to do. The order of presentation was balanced across subjects; two different orders of presentation were created and the two orders were randomly assigned to participants. Although the order of presentation varied within the movement and palm orientation trials, movement trials were always shown first. For a list of all trial stimuli, see Appendix 1.

Coding

A trained coder blind to the diagnostic classification of children watched the recorded videos and scored for praxis and timing errors for each of the 24 items in the gesture imitation task (see Table 2 for details). Each action was assigned a score of 0 (correct) or 1 (incorrect) within each praxis error category. Praxis error types were classified into two categories: errors of formation (i.e. hand selection, body orientation, hand orientation, movement direction, and final orientation) reflect inaccuracies in the general characteristics of the gesture (i.e. which hand to use, which direction to turn the body, which direction to face the palm, which direction to move the arm, and where to end the gesture), and errors of manner (i.e. modulation, directness, pace, and overflow) reflect inaccuracies in the movement qualities of the gesture (i.e. with the same amount of force, along the same path, at the same rate, and with the same number of movements or limbs involved). As mentioned earlier, manner errors are associated with the child’s ability to control/coordinate arm movements (Bhat, Landa, & Galloway, 2011; Vanvuchelen et al., 2007) whereas formation errors are associated with the child’s perception of self versus other as well as their own limb and body in space (Smith & Bryson, 2007; Williams et al., 2004).

Table 2.

Coding scheme for the gesture imitation task.

Coding scheme	Description
Praxis errors
Formation errors
Hand selection	Child did not use same hand to imitate as that used by the model to gesture, i.e. if the child used his left hand to imitate the gesture of the right-handed model
Body orientation	Child’s arm and/or body oriented differently than that of the adult model
Hand orientation	Child incorrectly oriented their hand in space compared to the model. For example, child’s hand faced up when imitating model’s downward-facing hand
Movement direction^a	Child’s movement direction differed from that of the adult. For example, adult moved their hand from up to down and child reversed the direction and moved their hand from down to up
Final orientation	Child’s final placement and orientation of body parts were incorrect compared to that of the model
Manner errors
Modulation^a	Child’s movement was insufficient or exaggerated in terms of effort/force exerted, or range of motion of joints compared to the model
Directness	Child used a circuitous/indirect path to complete their motion, demonstrated hesitation during the action, or completed the movement too swiftly with fewer steps compared to model
Pace^a	Child moved faster or slower that the adult
Overflow	Child performed extra movements at the end of the movement sequence, or involved both hands in the movement, or repeated the movement multiple times
Timing measures
Reaction time	Time period between adult’s initiation of the gesture and child’s initiation of imitation
Time to best effort	Time period between child’s initiation of action to the completion of action
Difference in duration^a	Difference between duration of the child’s movement and duration of adult’s movement. (note: negative values indicated that the child performed the movement more quickly than the model)
Control measure
Looking	Child failed to look at the model’s demonstration before performing the action

These errors were not scored for the eight palm orientation items since they were static postures.

We also included three measures regarding the timing of the child’s imitations: reaction time, time to best effort, and difference in duration, indicating the difference in temporal duration between the adult’s gesture and the child’s imitation. Finally, we included a control measure, looking errors, which indicated if the child was looking away during the stimulus and thus did not observe the gesture he was supposed to imitate.

Scores for each praxis error type were summed and transformed into error rates by dividing by the number of items. Total scores for formation and manner categories were also calculated. For the timing measures, times were averaged across trials for each parameter.

Interrater reliability was established by having three different raters independently code data of four participants. The scoring results were then compared among the raters, and discrepancies were resolved through consensus coding. After reaching consensus for the initial four participants, the primary coder blindly recoded these four participants and then compared her codes to the consensus codes that had been agreed upon with the other two coders. Once it was determined that the coding system was reliable (i.e. when the primary coder was able to blindly arrive at the same codes as the consensus codes), the primary coder continued with the remaining participants’ scores.

Some of the trials could not be scored for one or more parameters because the camera angle obscured the participant’s gestures. One hundred and ninety-four data points could not be scored out of a total of 7200 data points (2.7% of all data). These data points were treated as nonresponses and were not calculated in mean scores.

Statistical analysis

We checked our data for assumptions of parametric statistics. To compare group differences in praxis errors, we used multivariate analysis of variance (MANOVA). Since we coded for slightly different errors for movement and palm orientation items, we conducted separate MANOVAs on movement items and palm orientation items. In the first pass of analysis, using MANOVA, we evaluated the effect of group on our two dependent variables of total formation error score and total manner error score while adding “looking score” as a covariate in the analysis. If the group effect was significant, then we conducted follow-up MANOVAs to identify specific error categories (within errors of formation and manner) that were different between the two groups. We conducted similar but separate MANOVAs to assess the time parameters for the movement and palm orientation items. Finally, we calculated Pearson product-moment correlations to detect relationships between variables.

Results

Analyses by diagnostic group

Movement items

The omnibus MANOVA indicated a significant main effect of group (Pillai’s trace = 0.556, F(2, 26) = 16.27, p < 0.001). Specifically, after controlling for the effects of the looking score, the two groups differed significantly from each other both on total formation error scores (F(1, 27) = 9.91, p = 0.004) and total manner error scores (F(1, 27) = 29.51, p < 0.001). Our follow-up analyses on formation errors suggested that deaf TD and ASD groups differed significantly only on hand orientation (F(1, 27) = 7.15, p = 0.013) and final orientation (F(1, 27) = 17.43, p < 0.001) errors. In terms of manner errors, the deaf children with ASD made significantly more modulation (F(1, 27) = 17.05, p < 0.001), directness (F(1, 27) = 18.71, p < 0.001), pace (F(1, 27) = 17.30, p < 0.001), and overflow (F(1, 27) = 11.20, p = 0.002) errors than the TD deaf group. Mean error scores on movement items for the two groups are reported in Figure 3.

Figure 3.

Mean number of errors made by each group on praxis measures for the 16 movement items. ASD: autism spectrum disorder; TD: typically developing.

We also calculated Pearson product-moment correlations across all subjects in order to determine if a relationship existed between the errors on the movement trials and participant characteristics such as chronological age, nonverbal intelligence (as indicated by TONI score), ASD symptoms (as indicated by total SCQ score), and receptive language level (as indicated by ASL RST scores). For these correlations, we looked at total formation score, total manner score, and total praxis error score (sum of total formation and manner scores). Receptive language was very strongly negatively related to total formation (p < .01), manner (p < .001), and praxis (p < .001) error scores. ASD symptoms were strongly related to manner (p < .001) and total praxis (p < .001) scores and somewhat less strongly related to total formation score (p < .05). Nonverbal intelligence was moderately related to total formation (p < .05) and total praxis (p < .05) error scores but not to total manner score, and chronological age was unrelated to any praxis error scores. All values of r are indicated in Table 3.

Table 3.

Correlations between errors and participant characteristics.

	Formation score	Manner score	Total praxis error score
Age	−0.13	−0.03	−0.08
NVIQ	−0.41*	−0.31	−0.40*
ASD	0.41*	0.76***	0.70***
ASL	−0.52**	−0.73***	−0.74***

ASD: autism spectrum disorder; ASL: American Sign Language; NVIQ: non-verbal intelligence quotient.

=p < .05, **=p<.01, ***=p < .001.

As for the timing variables (reaction time, time to best effort, and difference in duration), no group differences were found for any of the variables. Thus, deaf children with ASD and TD deaf children did not differ in how quickly they imitated or how long their imitated gestures lasted. Both groups started imitating relatively quickly, about 1.25 s after the model initiated her movements. Both groups’ actions lasted approximately the same amount of time, between 1 and 1.5 s, and both groups executed their movements slightly faster than the adult model. We calculated Pearson product-moment correlations across all subjects in order to determine if a relationship existed between the timing parameters and participant characteristics; no significant relationships were detected.

Palm orientation items

The omnibus MANOVA indicated a significant main effect of group (Pillai’s trace = 0.632, F(2, 18) = 15.48, p < 0.001). Specifically, after controlling for the effects of the looking score, the two groups differed significantly from each other on both total formation error scores (F(1, 19) = 6.45, p = 0.02) and total manner error scores (F(1, 19) = 20.48, p < 0.001). Our follow-up analyses on formation errors suggested that the deaf TD and ASD groups differed significantly only on final orientation (F(1, 19) = 20.73, p < 0.001) errors. In terms of manner errors, the deaf children with ASD made significantly more directness (F(1, 19) = 8.12, p = 0.01) and overflow (F(1, 19) = 20.51, p < 0.001) errors compared to the TD deaf group. Mean error scores on palm orientation items for the two groups are reported in Figure 4.

Figure 4.

Mean number of errors by group on the eight palm orientation trials. ASD: autism spectrum disorder; TD: typically developing.

We calculated Pearson product-moment correlations across all subjects in order to determine if a relationship existed between error rates on the palm orientation parameters and participant characteristics (age, nonverbal intelligence, autism symptoms, and receptive language level). Receptive language was strongly negatively related to total manner (p < .001) and total praxis error scores (p < .001), and severity of ASD symptoms was strongly related to total manner (p < .001) and total praxis error scores (p < .01), while nonverbal intelligence was weakly negatively related to total praxis error score (p < .05). Chronological age was unrelated to any of the praxis measures (see Table 4).

Table 4.

Correlations between palm orientation errors and participant characteristics.

	Formation score	Manner score	Total praxis error score
Age	0.23	0.06	0.21
NVIQ	−0.39	−0.36	−0.47*
ASD	0.30	0.72***	0.59**
ASL	−0.40	−0.79***	−0.70***

ASD: autism spectrum disorder; ASL: American Sign Language; NVIQ: non-verbal intelligence quotient.

=p < .05, **=p < .01, ***=p < .001.

As for the timing of the palm orientation items, no group differences were found for reaction time or time to best effort. We calculated Pearson product-moment correlations across all subjects in order to determine if a relationship existed between the timing parameters and participant characteristics. Only ASD severity was found to be moderately related to reaction time, r(19) = 0.52, p < .05.

Discussion

In this study, for the first time, we have investigated praxis errors during gesture imitation in a unique population, deaf children with ASD who are natively exposed to a sign language by their Deaf parents. As in the one previous report on praxis performance in this population on a more complex sign production task (Bhat et al., 2016), we found that deaf children with ASD performed more poorly than TD deaf children even on a simple nonlinguistic gesture imitation task. Specifically, children with ASD committed more praxis errors in terms of hand orientation, final orientation, modulation, directness, pace, and overflow compared to their age- and IQ-matched TD peers. Deaf children with ASD exhibit both formation and manner errors, though our data suggest that the manner parameters presented more challenges than the formation parameters. We found no group differences for body orientation, movement direction, or hand selection, indicating that children with ASD were as good as TD children at imitating the direction to turn the body, the direction of movement of the gesture, and the selection of the correct limb. However, deaf children with ASD performed more poorly than TD children on all manner parameters coded, for both movement and palm orientation items. This finding suggests that children with ASD may be better at matching the grosser aspects of gesture imitation such as determining which limb to use and where to place and move it than they are at controlling the rate and path of the movements executed. Thus, as with previous work, we again find differences in praxis performance between TD children and children with ASD, even when those children have been exposed to a visual–gestural language from birth by their Deaf parents. Such children have ample opportunity for practice with gesture imitation, arguably far more practice than hearing children typically enjoy. Moreover, we find such differences even when the task is made nonlinguistic and as simple as possible, thus providing evidence that the praxis errors in pace, sequence precision, accuracy, and body part use described previously by Bhat et al. (2016) are not solely due to task complexity, literacy, or the linguistic nature of the stimuli.

Deaf children with ASD produced multiple manner errors and fewer formation errors. Furthermore, their ASD symptoms were more strongly correlated with manner errors than with formation errors. These findings are consistent with the vast literature confirming motor impairments in children with ASD (Ament et al., 2015; Bhat et al., 2011; Green et al., 2009; Jansiewicz et al., 2006; McPhillips, Finlay, Bejerot, & Hanley, 2014). Manner errors demonstrate poor motor control and coordination, specifically in terms of movement modulation, directness, overflow, and pace. If motor coordination is affected in children with ASD it will lead to poor action scaling, resulting in errors in modulation, directness, and overflow. For example, a child may perform an action larger or smaller than that of the model, use a circuitous path, or perform additional movement cycles; the latter two error types will also result in pace errors as children will take longer to complete their movements. Hearing children with ASD, both high functioning and low functioning, have greater impairments in gesture production related to motor difficulties, but not with the conceptualizing of gestures or comorbid behavioral impairments compared to age- and IQ-matched TD children (Vanvuchelen et al., 2007). Our finding of strong correlations between ASD symptoms and manner/total errors also support the notion that motor difficulties are indeed central to ASD (Dziuk et al., 2007; Hilton, Zhang, Whilte, Klohr, & Constantino, 2012). Consistent with our findings, another study found that praxis performance in children with ASD was a strong predictor of ASD features (Dziuk et al., 2007). In fact, a study involving concordant and discordant twins with ASD showed that all children affected with ASD had significant motor impairments compared to unaffected siblings and that the motor impairments of children with ASD correlated with severity of ASD symptoms. These authors concluded that motor impairments are a core characteristic of ASD and not an endophenotype. We also note that while the deaf children with ASD produced fewer formation errors than manner errors, they still produced more formation errors than the TD deaf children. As the formation errors were limited to hand orientation, we believe that these may reflect deficits in self–other mapping (Williams et al., 2004) and not in proprioception. Individuals with ASD are known to show greater reliance on proprioceptive control than visual control when performing arm movements including better motor learning and generalization (Izawa et al., 2012).

Despite these group differences, we find little evidence of differences in terms of the timing of the imitation of gestures. We find a modest relationship between ASD severity and reaction time on palm orientation items only (the higher the SCQ score, the slower children were to initiate the gestures, resulting in a longer reaction time). Our results reveal that children with more severe ASD symptoms may be slower at imitating gestures, but the overall lack of group differences in reaction time demonstrates that this relationship is not robust and may not be characteristic of all (or even most) individuals with ASD. Indeed, a recent meta-analysis of 32 studies of reaction time in 964 individuals with ASD compared with 1032 controls found no evidence of reaction time slowing in ASD (Ferraro, 2016). However, movement slowness in children with ASD has been reported across multiple tasks, including reaching (Mari, Castiello, Marks, Marraffa, & Prior, 2003), handwriting (Fuentes, Mostofsky, & Bastian, 2009), button pressing (Biscaldi et al., 2014), and repetitive hand movements (Biscaldi et al., 2014; Jansiewicz et al., 2006). It could be the case that children with ASD react at the same rate as TD children but then execute movements somewhat more slowly overall than TD children, or that the complexity of the task could contribute to overall slowing. In our previous work (Bhat et al., 2016), we found that deaf children with ASD took longer to fingerspell each target word than TD deaf children. The task described in Bhat et al. (2016) involved the production of a series of handshapes, while the current study involved a single movement or hand configuration. Previous work has shown that imitation skills in individuals with ASD were more impaired for multistep actions compared to single-step actions, suggesting possible deficits in cognitive skills such as working memory (Rogers et al., 1996). Our task involved minimal working memory demands since every gesture consisted of a single hand configuration and/or movement.

Our data give some indication of what could be responsible for these motor impairments. Both ASD severity and receptive language skills were strongly related to praxis performance, while nonverbal intelligence was only weakly related, and chronological age was not at all related to praxis performance. Thus, the more severe the ASD symptoms of a child, and the lower his receptive language level, the more praxis errors we can expect he will make. Motor impairments, while not being diagnostic of ASD, continue to be demonstrated in children with ASD—even on a simple gesture imitation task in a sample of children who have been imitating gestures their whole lives. We replicate the tight relationship between motor skills and sign language comprehension that we have demonstrated before in deaf children on a sign production task (Bhat et al., 2016). This finding adds to previous work on hearing children with ASD showing a tight relationship between praxis performance and communicative skills (Dowell et al., 2009) and provides more evidence that impairments to the motor system could have detrimental effects on language acquisition. Our findings with deaf children with ASD are furthermore expected to generalize to hearing children with ASD due to the biological (i.e. brain regions involved) and linguistic (i.e. phonological, morphological, and syntactic levels of structure) equivalence of spoken and signed languages. Regardless of modality, the relationship between language skills and comprehension appears similar in sign and speech. It is interesting to note the clear dissociation between praxis performance and mental and chronological age; neither of these characteristics appears to be a factor in motor impairment. This finding also confirms prior work showing no relationship between praxis scores and intelligence (Vanvuchelen et al., 2007).

Our work has implications for clinical practice. Praxis performance continues to be a challenging area for some children with ASD, especially the elements of manner such as modulation, directness, pace, and overflow. Clinicians might want to focus on activities that encourage children with ASD to work specifically on these skills. In addition to improving motor skills, our work suggests that motor performance has implications for language development. Although they are not part of the diagnostic criteria for ASD (American Psychiatric Association, 2013), motor impairments are a significant challenge for many children with ASD, and clinicians should know that physical therapy with children with ASD could have a number of beneficial effects. In the future, studies should investigate the effects of motor interventions not just on praxis skills but also on language skills and severity of ASD symptoms, given the strong connections between these areas of development.

The study had several limitations. We cannot know, for example, if the deaf children with ASD in our study have benefited from their lifelong exposure to a gestural language in comparison to hearing, speaking children with ASD who do not have such exposure. Instead, we have compared signing children with ASD to TD signing children. It is entirely possible that the deaf children with ASD in our sample in fact do exhibit better praxis performance than nonsigning children with ASD, and that their sign language exposure has benefited the development of their motor skills. Future work should investigate this important question. If deaf children with ASD show enhanced praxis performance compared to hearing, nonsigning children with ASD, then such a finding would suggest that sign language exposure or a focus on gestural communication in therapy could be beneficial for the linguistic and motor development of hearing children with ASD.

It is important to note that the groups were not matched for receptive language, and the children with ASD exhibited lower receptive language abilities than the TD children. While it would be ideal to match for language, the rarity of this population—deaf children with ASD of Deaf parents—made this unfeasible. Finding a TD language match for the ASD group would have required testing TD children under the age of five. Thus, while we acknowledge the limitation presented by the different language levels of the two groups, we believe that matching the groups for chronological and mental age represents a significant accomplishment. In future work, a language-matched TD group could help further clarify if differences in praxis performance are due to language differences.

Another limitation is that the groups were not matched for gender, and there were more females in the TD group (N = 10) than in the ASD group (N = 3). Gender differences in gesture production have been documented in the literature, with boys lagging behind girls in the production of their first gesture–speech combinations by about three months (Özçaliskan & Goldin-Meadow, 2010), and female infants producing more fine motor movements and specific imitative gestures just 3–96 hours after birth (Nagy, Kompagne, Orvos, & Pal, 2007). It is possible that the differences we have found between the TD and ASD groups could be due, at least in part, to such gender differences, and follow-up studies should attempt to recruit similar numbers of boys and girls in both groups. This was not feasible for the current study because of the very rare nature of the population (deaf children of Deaf parents with and without ASD).

In addition, a relatively small sample size could hinder the statistical power of our analyses. Given the rarity of the population being studied, we believe that a sample of 30 native-signing deaf children represents a sizable contribution. However, our conclusions must be considered as tentative because of the small sample size.

Methodologically we encountered some difficulty in the setup of the task since children were tested in a variety of environments at homes and schools across the United States. Thus, unlike a laboratory setup, the camera angle differed somewhat across subjects, which led to some loss of data (2.7% of all trials) due to obscured views.

Conclusion

We have presented a follow-up study to our previous work demonstrating impairments in the praxis performance of a unique population of children, deaf children with ASD who communicate primarily in a visual–gestural language. We add to this prior report by showing that the impairments in praxis performance are not only present in a sign language production task, with its additional demands on memory, but are also found in a gesture imitation task, which has no such demands. Furthermore, we show that the praxis impairments are found in the child’s movements regardless of whether or not the movements or palm orientations have linguistic value. In our previous work, the movements we had coded were linguistically meaningful; in the current work, the gestures the children were imitating were meaningless gestures and hand configurations. Still, the motor impairments remain, despite the simplicity of the task, which did not include any complex movements, but rather simple movement primes. Manner errors in deaf children with ASD strongly correlated with their ASD symptoms, suggesting that deficits in motor control/coordination contribute to the gestural impairments of ASD in general, and should be considered a core characteristic of ASD. Even more telling is our finding that lifelong exposure to sign language is not enough to overcome the motor challenges faced by children with ASD. We believe that deaf children with ASD could benefit from motor-based interventions targeting praxis impairments, not just to promote their motor skills but also their cognition and language.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: A. Shield’s work was supported by a grant from the National Institute on Deafness and Other Communication Disorders of the National Institutes of Health (1F32-DC0011219) and a Research Enhancement Grant from the Autism Science Foundation (14-04). A. Bhat’s work was supported by an Institutional Development Award (IDeA) from the National Institute of General Medical Sciences of the National Institutes of Health (U54-GM104941; PI: Binder-Macleod).

Notes

Appendix 1

References

Allen

T. E.

Enns

(2013) A psychometric study of the ASL Receptive Skills Test when administered to deaf 3-, 4-, and 5-year-old children. Sign Language Studies 14(1): 58–79. doi: 10.1353/sls.2013.0027.

Ament

Mejia

Buhlman

Erklin

Caffo

Mostofsky

Wodka

(2015) Evidence for specificity of motor impairments in catching and balance in children with autism. Journal of Autism and Developmental Disorders 45(3): 742–751. doi: 10.1007/s10803-014-2229-0.

American Psychiatric Association (2013) Diagnostic and statistical manual of mental disorders, fifth edition, Arlington, VA: American Psychiatric Publishing.

Avikainen

Wohlschläger

Liuhanen

Hänninen

Hari

(2003) Impaired mirror-image imitation in Asperger and high-functioning autistic subjects. Current Biology 13(4): 339–341. doi: 10.1016/S0960-9822(03)00087-3.

Bernier

Dawson

Webb

Murias

(2007) EEG mu rhythm and imitation impairments in individuals with autism spectrum disorder. Brain and Cognition 64(3): 228–237. doi: 10.1016/j.bandc.2007.03.004.

Bhat

A. N.

Landa

R. J.

Galloway

J. C.

(2011) Current perspectives on motor functioning in infants, children, and adults with autism spectrum disorders. Physical Therapy 91(7): 1116–1129. doi: 10.2522/ptj.20100294.

Bhat

A. N.

Srinivasan

S. M.

Woxholdt

Shield

(2016) Differences in praxis performance and receptive language during fingerspelling between deaf children with and without autism spectrum disorder. Autism. . Advance online publication. doi: 10.1177/1362361316672179.

Biscaldi

Rauh

Irion

Jung

N. H.

Mall

Fleischhaker

Klein

(2014) Deficits in motor abilities and developmental fractionation of imitation performance in high-functioning autism spectrum disorders. European Child & Adolescent Psychiatry 23(7): 599–610. doi: 10.1007/s00787-013-0475-x.

Brown

Sherbenou

R. J.

Johnsen

S. K.

(2010) Test of nonverbal intelligence, fourth edition, Austin, TX: Pro-Ed.

10.

Charman

Swettenham

Baron-Cohen

Cox

Baird

Drew

(1997) Infants with autism: An investigation of empathy, pretend play, joint attention, and imitation. Developmental Psychology 33: 781–789.

11.

Dewey

(1993) Error analysis of limb and orofacial praxis in children with developmental motor deficits. Brain and Cognition 23(2): 203–221. doi: 10.1006/brcg.1993.1055.

12.

Dewey

Cantell

Crawford

S. G.

(2007) Motor and gestural performance in children with autism spectrum disorders, developmental coordination disorder, and/or attention deficit hyperactivity disorder. Journal of the International Neuropsychological Society 13(2): 246–256. doi: 10.1017/S1355617707070270.

13.

Dowell

L. R.

Mahone

E. M.

Mostofsky

S. H.

(2009) Associations of postural knowledge and basic motor skill with dyspraxia in autism: Implication for abnormalities in distributed connectivity and motor learning. Neuropsychology 23(5): 563–570. doi: 10.1037/a0015640.

14.

Dziuk

M. A.

Gidley Larson

J. C.

Apostu

Mahone

E. M.

Denckla

M. B.

Mostofsky

S. H.

(2007) Dyspraxia in autism: Association with motor, social, and communicative deficits. Developmental Medicine and Child Neurology 49(10): 734–739. doi: 10.1111/j.1469-8749.2007.00734.x.

15.

Enns

C. J.

Zimmer

Boudreault

Rabu

Broszeit

(2013) American Sign Language: Receptive skills test, Winnipeg, Canada: Northern Signs Research, Inc.

16.

Ferraro

F. R.

(2016) No evidence of reaction time slowing in autism spectrum disorder. Autism 20(1): 116–122. doi: 10.1177/1362361314559986.

17.

Fuentes

C. T.

Mostofsky

S. H.

Bastian

A. J.

(2009) Children with autism show specific handwriting impairments. Neurology 73(19): 1532–1537. doi: 10.1212/WNL.0b013e3181c0d48c.

18.

Gizzonio

Avanzini

Campi

Orivoli

Piccolo

Cantalupo

Fabbri-Destro

(2015) Failure in pantomime action execution correlates with the severity of social behavior deficits in children with autism: A praxis study. Journal of Autism and Developmental Disorders 45(10): 3085–3097. doi: 10.1007/s10803-015-2461-2.

19.

Goldin-Meadow

Brentari

(2016) Gesture, sign and language: The coming of age of sign language and gesture studies. Behavioral and Brain Sciences 40: 1–82. doi: 10.1017/S0140525X15001247.

20.

Green

Charman

Pickles

Chandler

Loucas

Simonoff

Baird

(2009) Impairment in movement skills of children with autistic spectrum disorders. Developmental Medicine & Child Neurology 51(4): 311–316. doi: 10.1111/j.1469-8749.2008.03242.x.

21.

Hilton

C. L.

Zhang

Whilte

M. R.

Klohr

C. L.

Constantino

(2012) Motor impairment in sibling pairs concordant and discordant for autism spectrum disorders. Autism 16(4): 430–441. doi: 10.1177/1362361311423018.

22.

Hobson

R. P.

Lee

(1999) Imitation and identification in autism. Journal of Child Psychology and Psychiatry 40: 649–659.

23.

Izawa

Pekny

S. E.

Marko

M. K.

Haswell

C. C.

Shadmehr

Mostofsky

S. H.

(2012) Motor learning relies on integrated sensory inputs in ADHD, but over-selectively on proprioception in autism spectrum conditions: Distinct patterns of motor memory in autism. Autism Research 5(2): 124–136. doi: 10.1002/aur.1222.

24.

Jansiewicz

E. M.

Goldberg

M. C.

Newschaffer

C. J.

Denckla

M. B.

Landa

Mostofsky

S. H.

(2006) Motor signs distinguish children with high functioning autism and Asperger’s syndrome from controls. Journal of Autism and Developmental Disorders 36(5): 613–621. doi: 10.1007/s10803-006-0109-y.

25.

Lord

Rutter

DiLavore

P. C.

Risi

Gotham

Bishop

S. L.

(2012) Autism Diagnostic Observation Schedule, Second Edition (ADOS-2), Torrance, CA: Western Psychological Services.

26.

Mari

Castiello

Marks

Marraffa

Prior

(2003) The reach-to-grasp movement in children with autism spectrum disorder. Philosophical Transactions of the Royal Society B: Biological Sciences 358(1430): 393–403. doi: 10.1098/rstb.2002.1205.

27.

McPhillips

Finlay

Bejerot

Hanley

(2014) Motor deficits in children with autism spectrum disorder: A cross-syndrome study: Motor deficits in children with ASD. Autism Research 7(6): 664–676. doi: 10.1002/aur.1408.

28.

Mood

Shield

(2014) Clinical use of the Autism Diagnostic Observation Schedule-Second Edition with children who are deaf. Seminars in Speech and Language 35: 288–300. doi: 10.1055/s-0034-1389101.

29.

Mostofsky

S. H.

Dubey

Jerath

V. K.

Jansiewicz

E. M.

Goldberg

M. C.

Denckla

M. B.

(2006) Developmental dyspraxia is not limited to imitation in children with autism spectrum disorders. Journal of the International Neuropsychological Society 12(3): 314–326.

30.

Mundy

Sigman

(2015) Joint attention, social competence, and developmental psychopathology. In: Cicchetti

Cohen

D. J.

(eds) Developmental psychopathology, Hoboken, NJ: John Wiley & Sons, Inc., pp. 293–332. Retrieved from http://doi.wiley.com/10.1002/9780470939383.ch9.

31.

Mundy

Sigman

Kasari

(1990) A longitudinal study of joint attention and language development in autistic children. Journal of Autism and Developmental Disorders 20: 115–128.

32.

Nagy

Kompagne

Orvos

Pal

(2007) Gender-related differences in neonatal imitation. Infant and Child Development 16(3): 267–276. doi: 10.1002/icd.497.

33.

Ohta

(1987) Cognitive disorders of infantile autism: A study employing the WISC, spatial relationship conceptualization, and gesture imitations. Journal of Autism and Developmental Disorders 17: 45–62.

34.

Özçaliskan

Adamson

L. B.

Dimitrova

(2015) Early deictic but not other gestures predict later vocabulary in both typical development and autism. Autism 20: 754–763. doi: 10.1177/1362361315605921.

35.

Özçaliskan

Goldin-Meadow

(2010) Sex differences in language first appear in gesture: Sex differences in language first appear in gesture. Developmental Science 13(5): 752–760. doi: 10.1111/j.1467-7687.2009.00933.x.

36.

Rogers

S. J.

Bennetto

McEvoy

Pennington

B. F.

(1996) Imitation and pantomime in high-functioning adolescents with autism spectrum disorders. Child Development 67(5): 2060–2073. doi: 10.2307/1131609.

37.

Rogers

S. J.

Pennington

B. F.

(1991) A theoretical approach to the deficits in infantile autism. Development and Psychopathology 3: 137–162.

38.

Rutter

Bailey

Lord

(2003) Social communication questionnaire, Los Angeles, CA: Western Psychological Services.

39.

Seal

B. C.

Bonvillian

J. D.

(1997) Sign language and motor functioning in students with autistic disorder. Journal of Autism and Developmental Disorders 27: 437–466.

40.

Shield

Cooley

Meier

R. P.

(2017) Sign language echolalia in deaf children with autism spectrum disorder. Journal of Speech Language and Hearing Research 60(6): 1622. doi: 10.1044/2016_JSLHR-L-16-0292.

41.

Shield

Meier

R. P.

Tager-Flusberg

(2015) The use of sign language pronouns by native-signing children with autism. Journal of Autism and Developmental Disorders 45: 2128–2145.

42.

Shield, A., Pyers, J., Martin, A., & Tager-Flusberg, H. (2016). Relations between language and cognition in native-signing children with autism spectrum disorder. Autism Research. 9, 1304–1315. doi: 10.1002/aur.1621.

43.

Smith

Bryson

(2007) Gesture imitation in autism: II. Symbolic gestures and pantomimed object use. Cognitive Neuropsychology 24: 679–700.

44.

Soorya

L. V.

(2003) Evaluation of motor proficiency and apraxia in autism: Effects on sign language acquisition (Unpublished doctoral dissertation), Binghamton, NY: State University of New York.

45.

Srinivasan

S. M.

Lynch

K. A.

Bubela

D. J.

Gifford

T. D.

Bhat

A. N.

(2013) Effect of interactions between a child and a robot on the imitation and praxis performance of typically developing children and a child with autism: A preliminary study. Perceptual and Motor Skills 116(3): 885–904. doi: 10.2466/15.10.PMS.116.3.885-904.

46.

Stieglitz Ham

Bartolo

Corley

Rajendran

Szabo

Swanson

(2011) Exploring the relationship between gestural recognition and imitation: evidence of dyspraxia in autism spectrum disorders. Journal of Autism and Developmental Disorders 41(1): 1–12. doi: 10.1007/s10803-010-1011-1.

47.

Stokoe

W. C.

(1960) Dictionary of American Sign Language on linguistic principles, Burtonsville, MD: Linstok Press.

48.

Stone

W. L.

Ousley

O. Y.

Littleford

C. D.

(1997) Motor imitation in young children with autism: What’s the object? Journal of Abnormal Child Psychology 25(6): 475–485.

49.

Vanvuchelen

Roeyers

De Weerdt

(2007) Nature of motor imitation problems in school-aged boys with autism: A motor or a cognitive problem? Autism 11(3): 225–240.

50.

Williams

J. H.

Whiten

Singh

(2004) A systematic review of action imitation in autistic spectrum disorder. Journal of Autism and Developmental Disorders 34: 285–299.

Impaired praxis in gesture imitation by deaf children with autism spectrum disorder

Abstract

Background and aims

Methods

Results

Conclusions

Implications

Keywords

Praxis in ASD and relation to other skills

Praxis in signers with ASD

Method

Participants

Matching

Protocol

Gesture imitation task

Coding

Statistical analysis

Results

Analyses by diagnostic group

Movement items

Palm orientation items

Discussion

Conclusion

Footnotes

Declaration of conflicting interests

Funding

Notes

Appendix 1

References