Functional level of lesion scale: Validating fourteen years of research with the national spina bifida patient registry

Abstract

PURPOSE:

Functional level of lesion (FLOL) is a grading of the level of neurological function in patients with myelomeningocele and other forms of spina bifida. It has been widely used as an independent variable in studies of spina bifida, but its inter-rater reliability has not previously been tested. The purpose of this study was to measure inter-rater reliability of FLOL testing and compare testing performed by a non-medically trained research associate to testing performed by a pediatric rehabilitation medicine specialist.

METHODS:

Children in a multi-disciplinary spina bifida clinic underwent FLOL grading by a non-medically trained research associate. On the same day, these children were also graded by a pediatric rehabilitation medicine specialist. Cohen’s weighted kappa statistic was used to compare grading, with the rehabilitation medicine specialist considered the gold standard.

RESULTS:

A total of 71 patients participated. FLOL was graded for left and right leg for each participant, resulting in 142 measurements. Cohen’s weighted kappa was κ= 0.809, with a standard error of 0.034 and 95% confidence interval 0.723–0.875, indicating substantial agreement.

CONCLUSION:

FLOL as measured according to the instructions of the National Spina Bifida Patient Registry by a non-medically trained researcher is a reliable method to grade lower extremity function in spina bifida.

1 Introduction

Spina bifida remains the most common, permanently disabling birth defect in the United States, affecting approximately 166,000 Americans and occurring in eight out of every 10,000 live births [1]. Children born with spina bifida often have decreased function in their lower extremities, bowel and bladder dysfunction, hydrocephalus, Chiari II malformation, and scoliosis [2]. The level of spinal defect correlates not only with motor function and sensation in the lower extremities, but with the type of assistive devices necessary for mobility. For example, a patient with a thoracic level lesion will most likely have limited sensation below the hips and will require a wheelchair as their method of mobility; a patient with a low lumbar or sacral defect will typically experience loss of sensation below the ankle and require ankle foot orthotics to ambulate.

The anatomical location of the lesion provides a useful overall projection of expected neurologic function, but it does not always correlate with the individual’s functional level of lesion (FLOL). Also, the anatomical level is only an estimation of the expected level of deficit at birth and does not account for the disease process or how tethered spinal cord can impact functional decline over time.

Manual muscle testing (MMT) has historically been the most used method for documenting and finding changes in muscle strength. MMT evaluates the ability of the nervous system to adapt the muscle to the changing pressure of the examiner’s test and therefore must be performed by someone trained in muscle function [3]. There is subjective interpretation in MMT, and it requires that the examiner consider the position of the patient and be able to isolate the muscles being tested. While this information is critically important in finding subtle changes in strength or loss of function, it can be time consuming and difficult to do in a busy, multi-disciplinary clinic. There are also a variety of scales used in MMT, which can make interpreting results difficult for individuals who have not been trained or who do not dedicate sufficient time or diligence to proper testing.

Established in 2008, the National Spina Bifida Patient Registry (NSBPR) at the Centers for Disease Control and Prevention (CDC) has become the largest and most comprehensive spina bifida dataset. It includes variables ranging from basic demographic information to interventions performed (surgical and non-surgical procedures) and outcome information (bowel and bladder continence, mobility, skin problems). Due to the comprehensive nature of the NSBPR, it provides a structural framework for improving the standard and quality of care received by patients with spina bifida. Since its inception, the registry has used a 5-point categorical scale to evaluate FLOL in patients with spina bifida (Figure 1). Because spina bifida is a condition that includes neurological deficits that differ between patients, a variable to describe the level of neurological function is crucially important. This variable, FLOL, has become the predominant factor used to indicate neurological function in these patients. Furthermore, in many studies based on NSBPR data, FLOL has been shown to have an independent association with outcomes like bowel continence, ambulation, and presence of hydrocephalus [4–19].

Fig. 1

Level of Lesion [NSBPR Version 2.7.1 Annual Form].

The FLOL scale measures functional motor control and not ambulatory status. Instructions given in the NSBPR Version 2.7.1 Annual Form for assessing FLOL read as follows: “Any member of the clinic team assesses the patient for lowest level of independent movement which can be reproduced by the patient during a visit. This needs to be assessed annually as it may change over time. Clinics are instructed to indicate the level that represents the results for both the patient’s right and left sides. If the patient is uncooperative, ask the parent or adult with the patient to report on the patient’s lowest level of movement.” The simplicity of the instructions is intended to allow for non-medically trained research staff to perform reproducible and reliable measurements. FLOL has become a standard variable for information coming from the NSBPR; however, the inter-rater reliability of this method of assessing functional level has never been tested.

At the authors’ institution, Physical Medicine and Rehabilitation (PM&R) physicians who are board certified in pediatric rehabilitation medicine perform motor assessments on patients with spina bifida. Their unique training and methodical evaluation of function allows them to systematically compare clinical motor changes in patients. This training makes their evaluation a baseline standard for evaluation of FLOL. This study sought to determine the reliability of the FLOL scale used by the NSBPR as assessed in the majority of NSBPR clinics, i.e., by a research associate.

2 Methods

All children enrolled in the NSBPR seen in a multi-disciplinary spina bifida clinic between May and September of 2021 were independently evaluated using the FLOL scale by a PM&R physician and a non-medically skilled research coordinator. All clinical and demographic variables were prospectively collected using data-collection methodology as described for sites participating in the NSBPR [20]. These data collection procedures have been reviewed and approved by the authors’ Institutional Review Board.

2.1 Data analysis

Clinical and demographic variables were summarized using counts and descriptive statistics. To assess the inter-rater reliability of the FLOL scale, Cohen’s weighted kappa analysis was performed [21]. Results are presented as weighted kappa value, standard error, and 95% confidence interval (CI). Analysis was performed in Microsoft Excel.

3 Results

A total of 71 consecutive participants’ right and left lower extremities were evaluated, resulting in 142 data points. The majority were White, non-Hispanic, with more female than male participants (66% female, 34% male). Patient age ranged from 6 months to 20 years. Demographic information can be found in Table 1.

Table 1

Sample description

Variable	n	(%)
Patients	71	(100.0)
Age
Mean	10.8y
Median	11.6y
Range	0.5 to 20.3 y
Sex
Female	47	(66)
Male	24	(34)
Race/ethnicity
White, non-Hispanic	42	(59)
Black or African American	13	(18)
Asian	11	(16)
Hispanic	5	(7)
Diagnosis
Meningomyelocele	52	(73)
Lipoma of spinal cord	10	(14)
Split cord malformation	5	(7)
Fibrolipoma of filum terminale	2	(3)
Myelocystocele	2	(3)

The inter-rater reliability agreement of Cohen’s weighted kappa was κ= 0.809, with a standard error of 0.034, and 95% CI 0.723–0.875. Relevant reference ranges for interpretation of Cohen’s weighted kappa are “substantial agreement: 0.61–0.80” and “almost perfect agreement: 0.81–1.00” [22].

4 Discussion

Since 2008, the FLOL scale has been used among CDC NSBPR sites to provide functional motor information. It is the primary variable used to indicate the level of neurological function. This is critically important in spina bifida because the condition itself results in neurological dysfunction that differs between patients, varying from essentially normal to complete paraplegia with no motor or sensory function in the lower extremities.

Multiple studies have shown statistically significant association between FLOL and spina bifida-related outcomes. Children with more rostral FLOL (indicating more impaired neurologic function) are more severely neurologically affected and as such have been shown to be more likely to undergo treatment for hydrocephalus or Chiari 2 malformation and to undergo orthopedic or neurosurgical procedures [4, 23]. Patients with more rostral FLOL are less likely to have the ability to ambulate independently and in the community and are less likely to have the ability to independently transfer from a wheelchair [16, 19]. Studies of bowel and bladder continence and management show higher likelihood of incontinence and more frequent bladder or bowel management interventions in patients with more rostral FLOL [7–9, 24]. Pressure ulcers and skin breakdown are also more likely in patients with more rostral FLOL [6, 24]. Finally, adults with more rostral FLOL are more likely to be unemployed [10]. In summary, among studies using data from the NSBPR, FLOL is widely used as an independent variable and frequently has significant association with the outcome of interest. Therefore, establishing the inter-rater reliability of FLOL as a variable is crucially important.

The primary result of this study was a Cohen’s weighted kappa value of κ= 0.809 for FLOL, comparing grading performed by a pediatric rehabilitation medicine specialist (gold standard) to grading performed by a lay research associate. While there is some variability in who grades FLOL across centers within the NSPBR, in most cases, FLOL is assigned by research associates [20]. Thus, the study’s comparison between lay research associates and trained physicians is relevant. The kappa value observed in this study lies on the border of “substantial” and “almost perfect” agreement, indicating strong inter-rater reliability and validating the technique in use to measure FLOL at these centers [22].

There are limitations to this study. Inter-rater reliability was assessed between a lay research associate and a trained specialty physician. The lay researcher was trained to perform the grading and had two years of experience. These results may not be applicable for a less experienced researcher. Intra-rater reliability could not be assessed because performing multiple FLOL gradings on the same day is likely to lead to second measurements that are biased by memory of the first. Performing serial measurements on different days is also problematic because FLOL has been shown to change over time [16]. This assessment was performed in a pediatric spina bifida clinic. Therefore, these results may not be applicable for adult populations. However, many adolescents were included, and their assessment is likely very similar to adults.

5 Conclusion

This study shows that the NSBPR FLOL scale is a reliable method to assess lower limb motor function. It is a satisfactory tool for non-medically trained personnel to measure FLOL. It allows ease of assessment and provides a practical utility to clinicians.