Abstract
Background and objective
The 12-item Multiple Sclerosis Walking Scale (MSWS-12) is a patient-reported outcome instrument that quantifies the progressive loss of walking ability from the patient perspective. However, previous psychometric analyses indicated floor and ceiling effects across the multiple sclerosis severity spectrum. This study aimed to address floor effects by creating a gait module that can be used in conjunction with the MSWS-12 for better measurement of treatment benefit in the higher functioning multiple sclerosis population.
Methods
We used a step-wise mixed methods study design, with relapsing–remitting multiple sclerosis patients (wave 1, n=88; wave 2, n=30), combining qualitative (concept elicitation and cognitive debriefing interviews) and quantitative (Rasch Measurement Theory) data collection and analytical techniques and consultation interviews with three neurologists specializing in multiple sclerosis.
Results
Thirty-seven walking ability concepts were identified, and a five-domain conceptual framework was created. Draft items were generated and refined with patient and neurologist input. Draft items covered gait-related concepts such as dragging, shuffling, limping, tripping and falling. Rasch measurement theory psychometric analysis indicated administering MSWS-12 plus gait items improved measurement precision in targeted populations with better walking ability.
Conclusion
Study findings indicate that new gait items could improve sensitivity to detect clinical change in walking ability for higher functioning multiple sclerosis patients.
Introduction
Loss of walking ability is a hallmark of functional decline in multiple sclerosis (MS), 1 and is important to monitor as it signals changes in disease severity and progression. 2 Recent developments regarding diagnosis and treatment also warrant consideration. MS patients are being diagnosed earlier in the disease course 3 and are maintaining function, particularly walking ability, for longer periods of time. 4 Furthermore, emerging therapies, particularly remyelinating therapies, can potentially improve walking ability.5,6
In clinical trials, walking ability is often assessed using performance outcome measures, including the timed 25-foot walk and the six-minute walk test. 7 However, these measures have a limited connection to functional ability and the impact on patients’ day-to-day lives. Because including patient-reported outcome (PRO) measures in clinical trials is increasingly important from a development and regulatory perspective, 8 we need robust, well targeted PRO measures of walking ability to capture this information, particularly in patients with more mild walking dysfunction.
The Multiple Sclerosis Walking Scale 12-item (MSWS-12) is a PRO measure that is widely used to address this need. 7 MSWS-12 comprises 12 items describing the impact of MS on walking, identified from a larger pool of 141 items derived from patients, clinicians and literature about the ‘health impact of MS. 9 MSWS-12 was psychometrically validated using classic test theory methods. However, more recent research using modern test theory methods (Rasch Measurement Theory; RMT) detected disordering in the instrument’s response categories and poor targeting at the floor of the scale, 10 suggesting that MSWS-12 may be less sensitive to change in patients with more subtle deficits in walking ability. In addition, like many legacy PRO measures developed prior to regulatory guidance emphasizing conceptual clarity, 11 MSWS-12 was not developed based on a well-defined, comprehensive conceptual framework. 9
The objective of this research was to address the floor effect of the MSWS-12 and its sensitivity to change in walking ability in early stage relapsing–remitting multiple sclerosis (RRMS) patients. In alignment with regulatory and professional guidance for addressing measurement issues in existing PRO measures,12,13 we conducted patient-centered research to develop new items that measure symptoms and functional impact of walking limitations in early MS and potentially across the spectrum of MS severities.
Materials and methods
Study design overview
We performed patient-centered research to expand and refine our understanding of the concept of interest – walking ability in less disabled people with MS – to create a conceptual framework of treatment benefit and pilot test potential new items. The research comprised two waves of patient interviews and consultation with three clinical neurologists in MS at each stage. Interviews collected both qualitative (discussion) and quantitative (PRO response) data, which allowed us to conduct a mixed methods analysis (see Figure 1). In psychometric research, ‘mixed methods’ refers to the synthesis of qualitative and quantitative methods to identify, define and operationalize PRO instruments as measures of a given concept of interest in a specific context of use. 14
This Figure outlines the methods, analyses and main outocmes for each of the twowaves of this research.
Study population and recruitment process
Institutional review board approval was obtained, and participants provided written informed consent. Early RRMS patients were recruited through the study sponsor’s patient services department and through a social media site for MS patients. Eligible patients were diagnose d with RRMS within the last 2 years and had a patient determined disease steps (PDSS) 15 score of 0–1 (no to mild disability). This PDDS range coincides with the Expanded Disability Status Scale (EDSS) 0–2 levels, 16 in which recent research indicated limitations in the MSWS-12’s measurement range and precision (i.e. floor effects).
Patient interviews
Two waves of patient interviews were conducted. Wave 1 concept elicitation interviews followed a semistructured interview guide that fostered discussion around symptoms and impacts experienced in early MS. Patients also completed and discussed the MSWS-12. Data from wave 1 interviews allowed us to identify aspects of walking ability relevant to this sample and to create a conceptual framework to organize meaningful aspects of walking ability that can be used to evaluate treatment benefit in the context of clinical research. 8 From these data, we generated new items to address the floor effect in the MSWS-12.
Wave 2 interviews were cognitive debriefing interviews to establish relevance, clarity and ease of completion of the draft items that were generated after wave 1. Patients followed a ‘think aloud’ process, completing the items while explaining their thought process and noting any problems or ambiguities in these items. 17 Patients also completed the MSWS-12 and new items to provide quantitative data for RMT analysis.
All interviews were conducted over the telephone; the MSWS-12 and new draft items were displayed on patients’ computer screens, and item responses were captured by means of an online platform. Interviews were audio-recorded and transcribed.
Three neurologists specializing in MS (SC, MDG, KKR) were consulted after each of the waves to confirm the clinical relevance of the concepts elicited and to provide feedback on the emerging conceptual framework and items.
Data analysis
Wave 1 qualitative analysis: concept elicitation
Transcripts were analyzed thematically 18 through detailed line-by-line coding, 19 using ATLAS.ti software. 20 Coding was targeted to capture walking ability concepts. Codes and quotations were inductively categorized into overarching domains. Each code was compared with the rest of the data to create analytical domains and subdomains. Saturation, or the point in the data collection process when no new concept-relevant information emerges from additional qualitative data, 21 was assessed by ordering interviews chronologically, grouping them into quantiles and comparing concepts emerging by each sequential quantile.
Development of conceptual framework, concept mapping to MSWS-12 and item generation
Patient-generated walking ability concepts were inductively categorized into domains to create a conceptual framework of walking ability in MS. Concepts and domains were mapped against the MSWS-12 to identify concepts important to patients that were not in the existing instrument.
Item generation followed item construction principles,11,22–24 aiming to have an adequate range of items to cover the selected domain of walking ability. Item construction used as many of the patients’ own words as possible.
Wave 2 qualitative analysis: cognitive debriefing
This analysis aimed to identify wording ambiguities and assess relevance and acceptability in relation to each new item, response scale and set of instructions. Additional items suggested by wave 2 participants that could further expand the measurement of walking disability in early RRMS were also explored. 17
Quantitative data analysis
Small-scale RMT analysis was performed using MSWS-12 data from wave 1 and MSWS-12 plus complimentary items data from wave 2 using RUMM 2030. 25 RMT analysis was performed on the original PRO instrument scoring and on the revised scale structure proposed following preliminary analysis. 26 RMT analysis compares observed data against the stringent criteria of the Rasch model, aiming to assess the sample-to-scale targeting, item performance and person fit.27,28
Results
Study sample
A total of 118 patients with RRMS participated in this research study (n=88 wave 1; n=30 wave 2). Difficulties with walking were reported by 78% (n=69) of wave 1 patients at screening; all patients in wave 2 reported walking difficulty. Most patients were women (wave 1, 74%; wave 2, 80%), and the sample in both waves was evenly divided between PDDS 0 (50%) and 1 (50%) (see Table 1).
Sample characteristics.
aCounts not mutually exclusive; some patients reported more than one employment status.
RMT: Rasch measurement theory.
Wave 1: concept elicitation
Patients reported 37 unique concepts related to walking ability. Table 2 lists these concepts, while Table 3 provides exemplar quotes illustrating walking concepts. Patients described decreasing walking capacity in speed, distance or duration that varied based on the walking context, such as stairs or unfamilar terrain. They also discussed adapations they employed to deal with walking problems such as leaning on furniture or walls and concentrating when walking. Patients further described their walking ability in terms of gait problems, including shuffling, stumbling, tripping and falling. These higher functioning patients reported that while the walking issues they experienced on a day-to-day basis were relatively minor, they affected their quality of life. Most patients described proactively limiting activities such as shopping, walking with friends, hiking on rough terrain, or walking long distances based on concerns about their walking ability. Patients explained that their walking ability was usually severely impacted during flares; on a day-to-day basis, walking could be affected by weakness, neuromuscular symptoms (e.g. tremor, spasm, numbness), fatigue and fatigability, proprioceptive and vestibular problems and pain.
Conceptual framework of walking ability in early RRMS, mapping to MSWS-12.
✓ Indicates concept covered by MSWS-12.
aConcept elicited similar to MSWS-12 item 11 (affected how smoothly you walk).
RRMS: relapsing–remitting multiple sclerosis; MSWS-12: Multiple Sclerosis Walking Scale 12-item.
Examples of patient descriptions of walking concepts.
Saturation analysis found that 33 of 37 of the initially identified walking-related concepts arose within the first 40 interviews. The four concepts elicited in the remaining 48 interviews were similar to concepts derived from the first 40 interviews, indicating that interviews produced a comprehensive picture of walking-related concepts in higher functioning patients with RRMS (see Table 4).
Saturation of walking concepts (n=37).
Conceptual framework, concept mapping to MSWS-12 and item generation
To understand better the impact on walking ability in MS and to inform a conceptually clear measurement strategy for generating new items, concepts related to walking ability were inductively categorized into five conceptual domains (see Table 2):
Adaptation: methods patients employ to adapt to walking limitations; Balance and coordination: problems remaining upright and steady; Capacity: limitations on the ability to walk, run, or stand; Context: limitations on walking in certain environments; Gait: problems reported with the manner of walking.
Elicited concepts were then mapped to the MSWS-12 to determine its conceptual coverage and identify gaps. Of the 37 concepts elicited from higher functioning RRMS patients, 28 were not assessed by items of the MSWS-12 (see Table 2).
At this stage, item development was informed by consultation with clinicians specializing in MS and reference to the conceptual framework. First, clinicians reviewed the conceptual framework and walking concepts elicited from patients to determine their suitability for extending the measurement range of the MSWS-12 for patients with less severe disability. Of the 28 concepts not included in the MSWS-12, 13 were endorsed by two of the three clinical experts as additional items to extend the range of the MSWS-12: bending, crouching, raising leg/foot, drop foot, escalators, foot drag, leaning while walking, leg gives out, rising from sitting/low position, shuffling, tripping, walking duration and concentrating while walking.
Clinician recommendations were then examined considering gaps in MSWS-12 coverage and with reference to the walking conceptual framework. MSWS-12 does not offer comprehensive coverage in any of the five domains of walking except adaptation. While all five domains within the walking conceptual framework could be developed into distinct MS walking scales, developing gait items appeared to provide the best opportunity to address a gap in MSWS-12 coverage. This could also enhance measurement of treatment benefit in less disabled patients. MSWS-12 offers only one gait item, ‘affected how smoothly you walk’, most similar to descriptions of the ‘gait problems’ concept elicited in interviews. Given that two of three clinicians endorsed six gait concepts (drop foot, foot drag, leaning while walking, leg gives out, shuffling and tripping) as relevant for higher functioning patients, and an additional five concepts (drop foot, leg drag, foot drag, toe drag, limp) as more relevant for lower functioning patients, patient-elicited concepts in this domain offered the potential to develop items that could assess walking ability across the broad range of MS severity.
Gait items were developed in alignment with best practices for item development. Gait concepts were assessed in light of clinician feedback, patient comments, redundancies with existing MSWS-12 items, potential item formulation problems (e.g. ‘double-barreled’ items assessing more than one domain or describing more than one task) and potential to extend the measurement of the concept of interest. Nine draft gait items were developed, with response options and instructions formatted to complement the MSWS-12 (see items 1–9, Table 5).
MSWS-12 and gait items.
Wording in
MSWS-12: Multiple Sclerosis Walking Scale 12-item.
Wave 2: cognitive debriefing
Debriefing the nine draft gait items with 30 early RRMS patients demonstrated that overall these items were relevant, well understood and acceptable to patients. Some concerns were identified with items 02, 03 and 05. Item 05 (be clumsy) presented the most interpretation difficulty; half the patient sample reported that they considered upper limb clumsiness when responding to this item. Two patients stated that items 02 (legs give out) and 03 (legs lock up) were the same; an additional three patients stated they did not understand what was meant by ‘lock up’. Some patients suggested adding clarifying wording to item 02 (legs or knees give out) including ‘losing control’, ‘buckling’, or ‘weakness’. To clarify item 03 (legs lock up), two patients suggested specifying which part of the leg locks up. Patients were also asked whether any important questions were missing from the gait measure; one patient suggested adding an item on walking to the side instead of in a straight line.
Final gait items
Clinicians reviewed patient feedback on the draft gait items and suggested item revisions and potential additions. Based on patient and clinician feedback, three gait items were revised for better comprehension (item 02, legs or knees give out; item 03, legs lock up; item 05, be clumsy) and one new item (item 10, trouble walking in a straight line) was added (see Table 5).
Quantitative data analysis
Analysis of MSWS-12 data from wave 1 indicated that sample-to-scale targeting of the scale was suboptimal. Many items appear not relevant for the sample as there are no measurements on the right-hand side of the continuum (ceiling: worse walking ability); whereas measurement for some people on the left (floor: better walking ability) is not matched by any items of equivalent difficulty in relation to walking (see Figure 2).
Sample-to-scale targeting. This figure provides a direct comparison of walking ability within the sample and within the scale items. The upper histograms (pink blocks) represent the sample distribution and the lower histograms (blue blocks) the scale item threshold distribution plotted on the same interval metric continuum of walking ability. (a) Sample-to-scale targeting of the original MSWS-12 items and (b) the improvements to the match between sample and scale targeting introduced by merging the original MSWS-12 with the gait module items. Sample measurements falling off the 0–100 range of the scale indicate patients for whom the scale remains too easy.
Combining MSWS-12 and gait items resulted in better targeting to the early MS sample (Figure 2(b)). The person-item threshold plot of the combined scales indicates that all but three of the new gait items (legs lock, shuffle and limp) sit on the lower end of the continuum (better walking ability) and therefore improve MSWS-12 targeting on the floor of the scale. Adding new gait items increased relative coverage of the range of walking ability measured by the scale in this sample to 74% from 70%. Increased coverage improves a scale’s precision and sensitivity associated with measurement, as evidenced by the sum and average standard error scores associated with person estimates that decreased from 18.29 to 13.657 and 0.61 to 0.46 respectively.
No item misfit was identified; all items displayed fit residuals within recommended ranges and none of the items displayed significant chi-square correlations. However, the five-level response scale only worked as intended for 11 of 21 items, suggesting patients may have had difficulty discriminating among the five response options. In addition, 13 pairs of items displayed residual correlations above the recommended level (>0.3), suggesting possible dependency between items, which can impact true reliability. Despite this, the reliability of the combined item sets with and without extremes was good (PSI 0.90/0.90) and person misfit was low (n=3; 10%). This suggests good measurement precision in a good quality dataset. Figure 2 illustrates the improved sample-to-scale targeting of MSWS-12 plus gait items.
Discussion and conclusions
The patient’s perspective is a central part of clinical research as evidenced by the recently published 21st Century Cures Act 2016, which emphasizes patient-centered outcomes. 29 In addition, well defined and reliable PRO instruments have been considered essential by the US Food and Drug Administration for over a decade.11,12 Walking is a central concept of interest for MS clinical trials. While its objective measurement is vital, the patient’s real-world experience needs to be integrated to provide meaning and context to standardized walking assessments. A notable example is the AMPYRA (dalfampridine) label, in which the MSWS-12 was used to provide meaning to the results of the timed 25-foot walk. 30
In this study, we have begun to address the issues around floor effects previously identified in the MSWS-12, and thus improve the potential to detect therapeutic effects in clinical trials, research and ultimately, practice. Our study, which bridges the 12 original MSWS items with 10 new gait items, demonstrates one way to improve targeting and sensitivity to change across a wider range of MS severity. This multi-phase, mixed methods study (together with companion studies presented elsewhere)31,32 addresses the knowledge and measurement gaps around the symptom experience of higher functioning MS patients, presenting a clearer picture of the entire experience of patients affected by MS from early diagnosis onwards.
Our patient-centered research with 88 patients resulted in a rich pool of walking concepts relevant in early RRMS, from which a five-domain conceptual framework of walking ability in MS was generated. Because it had the best potential to address concepts not included in the MSWS-12, we chose the gait domain for item development. Ten candidate items were produced to enhance the measurement of walking ability in MS, particularly in higher functioning patients with better walking ability. Clinical neurologists helped ensure that item development focused on the most clinically relevant items to measure walking ability within this domain. Wave 2 interviews with 30 additional patients confirmed the relevance and understanding of the new items and provided evidence used to revise and refine the items.
Preliminary psychometric analysis of the MSWS-12 plus new items suggests improved targeting in this higher functioning RRMS sample, with reduced floor effects and greater precision in the ability to discriminate different levels of walking ability. Importantly, the item map of the combined scale indicates that all but three of the new gait items (legs lock up, shuffle and limp) sit on the lower end of the continuum (better walking ability), results that align with the clinical assessment of the difficulty or severity of these items. Thus when administered together with MSWS-12, gait items improve the MSWS-12’s scale targeting on the lower end of the scale, and importantly, results indicate our proposed gait module can be used as a standalone PRO instrument.
Findings should be interpreted with consideration of the study’s limitations. A new item (item 10, trouble walking in a straight line) was added after wave 2 interviews and clinician discussions; no performance data are available for this item. Recruitment methodology dictated that inclusion criteria were based on self-report information rather than a clinically confirmed diagnosis of RRMS. In addition, while preliminary psychometric results are promising, the small sample size and targeted mild disability population for the RMT analysis require that these results be interpreted with caution. Potential issues relating to thresholds and dependency should be revisited in larger, broader samples to decide on the best steps forward. Additional analysis in a larger, clinically defined sample would help confirm the validity and generalizability of these findings as well as the new items’ measurement properties. Finally, the MSWS-12 and gait item stems (‘In the past two weeks, how much has your MS …’) are simple and walking descriptions are brief, presenting low cognitive burden. However, given that the enhanced conceptual coverage in higher functioning patients is achieved by administering 10 items in addition to the MSWS-12 items, it may be worthwhile to explore the time needed to complete the gait items in future studies.
Created using best practices for PRO item development, the new gait items are potentially well suited to measure gait ability across the spectrum of MS, while specifically addressing measurement issues in less disabled patients. Greater sensitivity to change in this higher functioning population will allow more accurate measurement of walking disability progression and of reversal of disability when it occurs. Capturing this experience of walking ability enriches our understanding of functioning in the MS population represented now and enhances the likelihood of understanding the potential benefits of emerging therapies that may halt, or even reverse, early disease progression. Next steps include a full validation study to understand the psychometric properties of the overall item set across a broad MS population. Following that, additional modules covering the remaining four domains (adaptation, balance and coordination, capacity and context) could be developed using a similar mixed methods process.
Footnotes
Acknowledgements
The authors wish to acknowledge and thank the 118 patients who shared their MS stories with their research team, as well as project interviewers JoAnne Liebeler, Catherine Podeszwa and Sasha Spite; Biogen’s patient services team, and the MyMSTeam.
Conflict of interest
The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: The study was funded by Biogen. Shih-Yin Chen, Jennifer Petrillo and Carmen Castrillo-Viguera are employees of and stockholders in Biogen. Diego Cadavid was an employee of Biogen when the study was conducted. This work is not related to his current employment with Fulcrum Therapeutics. Sara Strzok, Sophie Cleanthous, Farrah Pompilus, Stefan Cano and Patrick Marquis are employees of Modus Outcomes, which received payment from Biogen Pharmaceuticals to conduct this research. Stanley Cohan receives research support from Biogen, Novartis, Mallinckrodt, Sanofi-Genzyme, Genentech and Opexa, is a paid consultant and/or serves on advisory boards for Biogen, Sanofi-Genzyme, Novartis and has received speaking honoraria, travel expenses and meals/lodging from Biogen, Novartis, Sanofi-Genzyme, Acorda and Genentech. Myla Goldman has received personal consultancy funds from EMD Serono, Genzyme and Novartis, and institutional consultancy and/or research funds from Acorda, Biogen Idec and Novartis Pharmaceuticals, and grant supported by the NMSS and NIH (K23NS062898). Kiren Kresa-Reahl speaker’s bureau honoraria include Biogen, Novartis, TEVA, EMDSerono, Mallinckrodt, Genzyme; she has provided consultant services to Biogen, Genentech and EMDSerono, as well as research support to Biogen, Novartis, Mallinckrodt, Genzyme and Genentech.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.