Analysis of gait synchrony and balance in neurodevelopmental disorders using computer vision techniques

Abstract

Gait tasks are commonly administered during motor assessments of children with neurodevelopmental disorders (NDDs). Gait analyses are often conducted in laboratory settings using costly and cumbersome experiments. In this paper, we propose a computational pipeline using computer vision techniques as an ecological and precise method to quantify gait in children with NDDs with challenging behaviors. We analyzed videos of 15 probands (PB) and 12 typically developing (TD) siblings, engaged in a preferred-pace walking task, using pose estimation software to track points of interest on their bodies over time. Analyzing the extracted information revealed that PB children had significantly less whole-body gait synchrony and poorer balance compared to their TD siblings. Our work offers a cost-effective method while preserving the validity of its results. This remote approach increases access to more diverse and distant cohorts and thus lowers barriers to research participation, further enriching our understanding of motor outcomes in NDDs.

Introduction

Neurodevelopmental disorders (NDDs) are a group of disorders characterized by abnormalities in cognitive, motor, and social domains. Among these abnormalities, motor impairments are often the first signs of an increased risk of NDDs in young children, ¹ with the most salient sign being an atypical gait. Typical gait is characterized by the rhythmical movement and smooth synchronization of upper and lower limbs in a cyclical pattern that results in the displacement of the individual’s center of gravity in space. ² A number of studies have analyzed coordination and symmetry of spatiotemporal gait characteristics in children with NDDs. For example, studies on children with cerebral palsy have examined ipsilateral ³ and contralateral ⁴ coordination in the upper and lower limbs, as well as stride-to-stride variability and symmetry. ⁵ Altogether, prior work suggests that gait synchrony is important for high functional mobility, efficient performance of daily living tasks, and improvements in quality of life.^6–8 Although a precise definition of synchrony has proven elusive, prior studies support the importance of defining various measures of gait synchrony and developing methods to accurately measure such parameters for clinical studies of NDDs. These measurement methods have improved significantly with the availability of high-performance kinematic methods. For instance, researchers have utilized the Vicon motion capture system, ground-reaction force plates, ⁹ force-sensitive insoles, ¹⁰ and GAITRite® instrumented walkway system ¹¹ to measure various aspects of synchrony in different populations.

Besides synchrony, dynamic balance (referred as “balance” hereafter for brevity) is another central feature of gait and defined as the ability to maintain one’s center of mass with respect to one’s base of support while moving. Most typically developing (TD) children master gait balance by the age of seven. ¹² While balance is commonly evaluated in children using instruments, such as the Pediatric Balance Scale, ¹² that rely on the clinician’s interpretive expertise, recent studies have utilized kinematic technologies, such as ground-reaction force plates,^13–15 the GAITRite® system,^16,17 and the Wii Balance Board,^18,19 to propose more precise methods to measure balance.

A potential obstacle for the aforementioned kinematic methods is the requirement for the participants or patients to be physically present in the lab space. This limitation often hinders the applicability and ecological relevance of these methods, especially for individuals with cognitive and behavioral challenges. In recent years, machine learning (ML) and computer vision have shown promise in resolving the above issues and advancing gait research. ²⁰ With high-volume data collection technologies and sophisticated data analysis tools, computational approaches have become increasingly relevant to gait analysis. Compared to conventional kinematic methods, computational methods are more cost-effective and less cumbersome. Moreover, computer vision approaches are physically non-obtrusive and less demanding for clinical populations, such as individuals with NDDs.

Accordingly, the present study followed three aims: (1) To determine the feasibility of using modern video processing technologies to extract various profiles of gait synchrony and balance from recorded videos using easy point-and-shoot cameras. Based on previous work showing the applicability of deep learning techniques in gait research, ²¹ we hypothesized that such techniques would enable us to quantify gait with high accuracy and specificity, without asking participants—especially children with anxiety, sensory hypersensitivity, and difficulty with imitation and attention—to be physically present in a laboratory. This aim would help evaluate whether our computational pipeline is suitable for off-site or remote video-based gait assessment. (2) To investigate the clinical usefulness of our gait analysis method using a convenience cohort of children with a 16p11.2 mutation and their TD siblings. ML-based pose estimation techniques have been previously used to study gait, ²¹ but not in clinical populations. Herein, we focused on quantifying and comparing gait synchrony and balance in 16p11.2 probands and a control group. Our hypothesis stated that synchrony and balance would be markedly different between these groups, with the probands having less synchronous and balanced gait. This aim would determine whether such quantifications of gait synchrony and balance could potentially be used as phenotypical descriptors in clinical studies. (3) To introduce a new measure which captures whole-body synchrony that is more robust to handedness, compared to measures in prior studies ⁴ (see Discussion section for more details). Our hypothesis was that using this measure would capture whole-body gait synchrony more robustly when comparing proband and control groups. This last aim would provide a new measure of gait synchrony to enhance current motor assessment methods.

Method

Participants

Children included in this study were recruited as part of a large family research meeting focusing on children with a 16p11.2 mutation (proband; PB) and their non-affected, typically developing (TD) siblings. ¹⁷ Children with 16p11.2 present mild-to-moderate neurodevelopmental impairments across language, social, and motor domains. ²² All participants’ caregivers provided written informed consent, and the experimental protocol conformed to the ethical guidelines of the Institutional Review Board of the study’s home institution. Children were administered a battery of locomotion tasks and the Pediatric Evaluation of Disability Inventory–Computer Adaptive Test (PEDI-CAT). ²³

Given the aims of the present study, we excluded from our analysis children who did not perform the task independently, exhibited irrelevant gait movements (e.g., hopping, skipping), and/or whose body was occluded by another person during the task. Hence, video recordings of n = 15 PB children (mean age: 9.90 years, seven girls) and n = 12 non-affected TD siblings (mean age: 9.67 years, four girls) engaged in the locomotion tasks were used in the final analysis (Table 1).

Table 1.

Descriptive statistics of study participants.

	PB (n = 15)	TD (n = 12)	p value
Female %	53%	41%
Age (years), M(SD.)	9.90 (3.27)	9.67 (4.00)	0.87
Age (years), range	4.35–15.11	4.18–15.36

Materials

To study gait synchrony and balance, we analyzed data collected from the “preferred pace walk test” (PPWT), in which children were instructed to walk along a 10-m walkway at their natural pace. All videos used in the current study were originally recorded using a single point-and-shoot camera that was set up at roughly 15 feet from the beginning of a GAITRite® walkway used in the original study. The camera was pointed at ∼30° to the sagittal planes of the children’s bodies so that limb and body midline movements could be captured. Short video segments 3–9 s long (M = 5.2 s, SD = 1.4) of each child’s PPWT were manually extracted and cropped to feature only the child performing the task. This was a necessary step so as to improve the accuracy of video processing and to eliminate the effect of extraneous objects or persons in the frame.

The pre-processed videos were then fed into Detectron2, ²⁴ a deep learning-based object detection software package. Detectron2 uses a pre-trained model to extract and track 17 points of interest (POIs; Figure 1(a)) on each child’s body (Figures 1(b) and (c)). Next, for each video frame, the POIs were used to calculate five angles ${\theta }_1,{\theta }_2, \dots , {\theta }_5$ illustrated in Figure 1(d), generating five angular temporal signals (ATS) for each video (Figure 1(e)). These signals were manually cleaned by clipping the beginning and end of the clean portion of each ATS and then replacing each outlier (>3 SDs away from the mean) with the last non-outlier point of the ATS. 15 to 50 points (M = 31.7, SD = 12.6) from the beginning and 10 to 120 points (M = 29.2, SD = 27.3) from the end of the ATSs were clipped, eliminating 11.6% of the PB and 18.5% of TD groups’ ATS points. Finally, each ATS was normalized by subtracting its mean from each of its values.

Figure 1.

Extraction of points of interest (POIs) and calculation of gait synchrony and balance measures from video recordings. (a) POIs on a body. White circles on the body silhouette denote articulated figure POIs, and blue circles denote neck and torso coordinates, calculated as the midpoint between the shoulders and hips, respectively; (b) frames of a sample input video recording; (c) a sample frame annotated with extracted POIs; (d) POI temporal signals extracted from the sample video; (e) all five angular temporal signals (ATSs) for the sample video; (f) FFT frequency-domain profiles of ATSs.

To quantify gait synchrony using the above ATSs, the Fast Fourier Transform (FFT) was applied to the first four ATSs of each participant, and the results were used to create a frequency-domain profile of each ATS (Figure 1(f)). The main swinging frequencies of each child’s arms and legs were calculated as the frequencies $f_1^{max}$ , ... $f_4^{max}$ at which the FFT spectra attained their maxima. These main frequencies were then used to calculate their variance ${\sigma }_{f^{max}}^2=\frac{1}{3}{\displaystyle \sum }_{i=1}^4{\left(f_i^{max}-{\mu }_{f^{max}}\right)}^2$ where ${\mu }_{f^{max}}=\frac{1}{4}{\displaystyle \sum }_{i=1}^4{f}_i^{max}$ . The smaller the value of ${\sigma }_{f^{max}}^2$ , the more synchronous the limbs were during PPWT. In order to assess the reliability of our pipeline, we performed a sanity check on our gait synchrony measure ${\sigma }_{f^{max}}^2$ against two similar measures (MARP and σ_CRP) reported in Meyns et al. ⁴

Finally, the variance ${\sigma }_{{\theta }_5}^2$ of the ATS corresponding to ${\theta }_5$ (i.e., the angle between the vertical axis and the line connecting the neck and torso) was calculated to capture the balance of the child while walking, such that the larger the value of ${\sigma }_{{\theta }_5}^2$ , the less stable the child’s gait. To assess the prediction power of ${\sigma }_{{\theta }_5}^2$ in conjunction with age for distinguishing PB versus TD children, we used a leave-one-out (LOO) evaluation mechanism to create and evaluate support vector machine (SVM) classification models ²⁵ with radial basis function (RBF) kernels. These classification models find maximum margin hyperplanes in high- or infinite-dimensional spaces which would best separate the two classes—in our case, PB and TD. These hyperplanes in high-dimensional spaces correspond to potentially highly non-linear classifiers in the original feature spaces, and thus, SVMs are among the most popular ML techniques for their practical versatility and theoretical soundness.

Data Analysis

Data analysis was performed in two parts for each of our gait synchrony and balance parameters. First, we performed a Mann–Whitney U test on each parameter between PB and TD groups on the whole population, as well as on girls-only and boys-only subgroups. Second, we performed a correlation analysis between each parameter and age to determine associations between children’s gait profiles and age.

Results

Overall, results of the Mann–Whitney U test showed significant differences in gait synchrony between the groups (Figure 2(a)). The TD children demonstrated significantly more synchrony among their extremities (upper/lower limbs) during gait compared to the PB children. Further, a sex-based analysis revealed that when girls and boys (Figures 2(c) and (e)) were analyzed separately, no significant differences were found between PB and TD children, indicating that within their sex groups, girls and boys demonstrate similar profiles. Additionally, Pearson correlation coefficients between ${\sigma }_{f^{max}}^2$ and age, across sex, revealed small correlations (Figures 2(b), (d), and (e)). All effect sizes measured by AUC ROC were medium (Table 2).

Figure 2.

Gait synchrony and balance in PB and TD groups. Differences in gait synchrony (a, c, e) and balance (g, i, k) between PB and TD groups across sex. (b, d, f) Scatterplots with regression lines show correlations between ${\text{σ}}_{{\text{f}}^{\text{max}}}^2$ and age across sex. (h, j, l) Scatterplots with regression lines show correlations between ${\sigma }_{{\theta }_5}^2$ and age across sex. (m, n, o) Classification boundaries and margins between PB and TD groups. Significance: * p < 0.05. ** p < 0.01.

Table 2.

Statistics of various gait synchrony and balance measures across sex.

	Boys and Girls (15 PB, 12 TD)				Girls (8 PB, 5 TD)				Boys (7 PB, 7 TD)
	Mdn	U	f	p	Mdn	U	f	p	Mdn	U	f	p
${\sigma }_f^{2}max$	1.19	54	0.70	0.04	1.69	12.5	0.69	0.15	0.72	15	0.69	0.12
${\sigma }_{{\theta }_5}^2$	1.22	30	0.83	0.00	1.09	6	0.85	0.02	1.27	10	0.80	0.04
$\mathrm{MAR}{\mathrm{P}}_{{\theta }_1,{\theta }_4}$	27.10	69	0.62	0.16	26.56	12	0.70	0.14	28.00	22	0.55	0.40
$\mathrm{MAR}{\mathrm{P}}_{{\theta }_2,{\theta }_3}$	26.25	51	0.72	0.03	26.31	12	0.70	0.14	25.44	13	0.73	0.08
${\sigma }_{\mathrm{CR}{\mathrm{P}}_{{\theta }_1,{\theta }_4}}$	37.84	66	0.63	0.13	36.29	12	0.70	0.14	37.94	23	0.53	0.45
${\sigma }_{\mathrm{CR}{\mathrm{P}}_{{\theta }_2,{\theta }_3}}$	36.08	37	0.79	0.01	36.08	8	0.80	0.05	35.96	10	0.80	0.04

Note. Each row shows the values of a statistic across various sex groups. First level of column headers shows the sex groups as well as the breakdown of PB and TD participants within each group. Second level of column headers shows the statistics measures for each sex group. Mdn is the median value, U is the Mann–Whitney U test statistic, f is its AUC ROC effect size, and p is its p-value.

A secondary inspection for outliers identified two PB children with perfect synchrony ( ${\sigma }_{f^{max}}^2=0$ ) and two TD children with relatively poor synchrony ( ${\sigma }_{f^{max}}^2>2)$ . Manual examination of their video recordings revealed that the two PB children displayed gait that was indistinguishable to untrained eyes from typical PPWT gait. In contrast, the two TD children with less synchrony were walking with minimal arm swing, making their PPWT gait look atypical.

Our balance analysis revealed significantly lower (i.e., better) balance scores in TD than in PB children (Figure 2(g)). This difference in balance remained significant for girls and boys when analyzed separately (Figures 2(i) and (k)), with a large effect size (Table 2).

We used a leave-one-out scheme to evaluate our SVM classifiers for all, girls-only, and boys-only subpopulations (Table 3). We obtained high predictive accuracy using the balance measure ${\sigma }_{{\theta }_5}^2$ and age, which suggests that these measures can potentially be used together as indicators of neurodevelopmental deficits. Classification boundaries between the PB and TD groups were computed by training SVM classifiers with RBF kernels on the whole dataset (Figure (2m)) and on each subgroup (Figures 2(n) and 2(o)). There was a general trend toward better balance among older participants, with a few exceptions leading to complex decision boundaries.

Table 3.

Leave-one-out accuracy measures for the SVM classifiers.

	Girls and boys	Girls	Boys
Precision	0.86	0.75	0.50
Recall	0.80	0.75	0.43
Specificity	0.45	0.33	0.57
Matthew’s correlation coefficient	0.63	0.35	0.00
F1 score	0.83	0.75	0.46

The results of the sanity check illustrated in Figure 3 show that measures proposed in previous work, such as ${\text{MARP}}_{{\theta }_2,{\theta }_3}$ and ${\sigma }_{{\text{CRP}}_{{\theta }_2,{\theta }_3}}$ (i.e., synchrony between the left arm and right leg), show significantly more synchronous gait in the TD group and are consistent with our ${\sigma }_{f^{max}}^2$ measure (first column of Figure 3 and Table 2). Additionally, we observed a sex effect (using p-value and effect size) for the ${\text{MARP}}_{{\theta }_2,{\theta }_3}$ measure, but not for ${\sigma }_{{\text{CRP}}_{{\theta }_2,{\theta }_3}}$ (last two columns of Figure 3 and Table 2).

Figure 3.

Sanity check results for MARP and σ_CRP. Significance: * p < 0.05. ** p < 0.01.

Discussion

The current study proposes a method to examine whole-body gait synchrony and balance in children using an ecologically friendly and non-intrusive approach. Indeed, synchrony is a feature of gait essential to smooth and stable locomotion,^26,27 which so far has received little attention in gait research. To fill this gap and test the clinical feasibility of our synchrony parameter, we created an end-to-end pipeline of data acquisition, pose estimation (using deep learning), as well as data cleaning and analysis. Children with a 16p11.2 mutation form a genetically homogenous population with a high prevalence of neurodevelopmental delays, particularly motor impairments, and thus represent a suitable group to test the clinical utility of our parameters and ultimately contribute to the characterization of their motor phenotype. As such, using a cohort of children with a 16p11.2 mutation and their typically developing siblings, we investigated the clinical usefulness of our pipeline by measuring and comparing their whole-body gait synchrony and balance.

To put our work in a broader context, we have developed a computational pipeline applicable across various paradigms and suitable for more naturalistic and less overwhelming laboratory spaces. Overall, our findings provide the much-needed quantifiable measures of gait synchrony and balance that can help refine motor phenotypes in children with NDDs. We argue that our low-cost setup and novel, robust measures enable us to better understand of the role of synchrony in motor development, particularly with respect to motor control and smoothness of movement in the face of environmental perturbations. ²⁸

In sum, our methods demonstrated less whole-body gait synchrony in PB children than in TD children during a locomotion task. In addition, small non-significant correlations between gait synchrony measures and age indicate that group differences were not attributed to delayed motor development in the age range under study. Only two outlying data points were identified in each group, and manual examination of their video recordings corroborated the calculations made by our computational pipeline. In particular, the two TD children with relatively lower synchrony scores completed the PPWT with their arms held stiffly by their sides. Research has shown that partly active and partly passive arm swing is necessary for optimal stability and efficiency during typical human gait. ²⁹ Stiff arm posturing observed in the TD children might be an indicator of discomfort during testing and reaffirms the need for more ecologically friendly research approaches.

Of note, no overall sex differences were found for our gait synchrony measure. This finding suggests that regardless of their genetic status, girls and boys in the current sample demonstrate similar gait synchrony profiles as their same-sex peers. However, girls as a whole (Mdn ${\sigma }_{f^{max}}^2$ = 1.69) appeared to have less synchronous gait compared to boys (Mdn ${\sigma }_{f^{max}}^2$ = 0.72). Few studies on gait in individuals with NDDs, such as Down syndrome ³⁰ and cerebral palsy, ³¹ are reporting on sex factors in kinematic parameters; and research on this issue in children remains scarce.

Similar to our results on gait synchrony, significant differences in balance were found between PB and TD children. Consistent with previous findings, ¹⁷ PB children exhibited significantly poorer balance during the locomotion task compared to their TD siblings. In contrast to the results from the gait synchrony analysis, group differences in balance remained significant across sex, suggesting that children in the PB group demonstrated poorer balance regardless of their sex. Finally, small and/or non-significant correlations between children’s balance measures and age suggest that group differences were not a result of delayed motor development.

Results of our sanity check for our synchrony parameter ${\sigma }_{f^{max}}^2$ confirm its utility against other established measures in the literature. ⁴ However, we argue that our synchrony parameter represents a more natural measure that involves symmetrical, whole-body movement dynamics, thus making it less susceptible to variations in handedness. Indeed, previous measures seemed unstable when overall handedness skewed toward one side.

While our findings highlight the promise of modern computer vision techniques in clinical gait assessment, they should be interpreted in the context of the following limitations. First, although our sample size was comparable to those of previous studies,^4,6 it was relatively small and not demographically varied. For example, we conducted a post-hoc power analysis for all tests with non-significant results and found these tests underpowered, mainly due to the small sample size, which prevented us from interpreting effect sizes. We advocate for studying larger, less biased, and more heterogenous populations. Second, we did not have access to scores of standardized motor assessments or cognitive tests. Future work could explore how our gait synchrony and balance parameters are related to these developmental measures. Third, our data collection protocol and computational pipeline were not set up to study the intra-subject variability which is common in NDDs. Although high intra-subject gait variability for individuals with 16p11.2 syndrome has not been known or reported in prior work, future studies should address this issue with targeted study designs using larger populations. Fourth, we checked the precision of the pose estimation results by visually verifying the Detectron2 outputs. However, more rigorous verification and validation of the pose estimation results should be performed by comparing them to those obtained using well-established motion capture technologies. Last, our experimental setup did not allow us to test robustness to variations in recording angle, which could be addressed by using a secondary camera at a different angle.

Conclusions

Modern computer vision techniques provide low-cost and precise methods for extracting video-based information from clinical populations in naturalistic, ecologically friendly environments. In this study, we presented a data acquisition and processing pipeline to extract and analyze measures of gait synchrony and balance in children with a 16p11.2 mutation and their typically developing siblings. Hence, we used a combination of a single point-and-shoot camera and modern video processing software to extract signals that are robust to recording conditions and noise. This approach provides both researchers and clinicians with an inexpensive, ecological, and high-accuracy method to study gait and movement in individuals, including children with NDDs, who may exhibit behavioral challenges affecting test administration. The proposed approach can be implemented remotely, for example, in the context of clinical trials and geographically distant communities. We posit that our pipeline can be used to analyze other types of movement disorders, which we plan on investigating in the future.

Longstanding disparities in research sampling caused by racism and socioeconomic and geographical barriers have been pervasive in science, and have only become more apparent in the midst of the COVID-19 pandemic. ³² We contend that a low-cost and remote method of data collection, especially from continuously overlooked participants, could alleviate such disparities and advance efforts to close the knowledge and treatment gaps resulting from biased samples. Our computational approach represents one possible population-based solution ³³ to increase enrollment and improve generalizability to participants of various social and genetic backgrounds. We plan on further validating our gait synchrony and balance results by studying larger cohorts of participants across different types of NDDs and more diverse demographic backgrounds.