The MSOAC approach to developing performance outcomes to measure and monitor multiple sclerosis disability

Abstract

Background:

The Multiple Sclerosis Outcome Assessments Consortium (MSOAC) was formed by the National MS Society to develop improved measures of multiple sclerosis (MS)-related disability.

Objectives:

(1) To assess the current literature and available data on functional performance outcome measures (PerfOs) and (2) to determine suitability of using PerfOs to quantify MS disability in MS clinical trials.

Methods:

(1) Identify disability dimensions common in MS; (2) conduct a comprehensive literature review of measures for those dimensions; (3) develop an MS Clinical Data Interchange Standards Consortium (CDISC) data standard; (4) create a database of standardized, pooled clinical trial data; (5) analyze the pooled data to assess psychometric properties of candidate measures; and (6) work with regulatory agencies to use the measures as primary or secondary outcomes in MS clinical trials.

Conclusion:

Considerable data exist supporting measures of the functional domains ambulation, manual dexterity, vision, and cognition. A CDISC standard for MS (http://www.cdisc.org/therapeutic#MS) was published, allowing pooling of clinical trial data. MSOAC member organizations contributed clinical data from 16 trials, including 14,370 subjects. Data from placebo-arm subjects are available to qualified researchers. This integrated, standardized dataset is being analyzed to support qualification of disability endpoints by regulatory agencies.

Keywords

MS disability performance outcome measures data standards clinical trial database regulatory qualification

Introduction

The need for better measures of MS disability has been recognized for decades. In 1993, the National Multiple Sclerosis Society (NMSS) convened an international workshop on the topic.¹ One result was a task force, charged with recommending outcome assessment methods that might improve on the Kurtzke² Expanded Disability Status Scale (EDSS). The task force recommended quantitative neurological performance testing as opposed to clinical rating scales such as EDSS, largely because performance outcome measures (PerfOs) have superior psychometric properties. The task force recommended the Timed 25-Foot Walk (T25FW) as a measure of walking speed, the 9-Hole Peg Test (9HPT) as an upper extremity dexterity measure, and the Paced Auditory Serial Addition Test (PASAT; 3-second version) as a measure of cognitive processing speed.³ The task force also urged the academic community to develop a test for visual function, because high contrast letter acuity was not sensitive to change. The task force recommended inclusion of three PerfOs, together called the Multiple Sclerosis Functional Composite (MSFC) for inclusion in future trials. What followed was inclusion of MSFC in most prospectively designed clinical trials conducted by industry and academia and development of Low Contrast Letter Acuity (LCLA) as a more sensitive measure of MS-related visual impairment.⁴ Many placebo-controlled clinical trials demonstrated treatment effects on the MSFC score. However, complexities related to the reference population used to create standardized scores and difficulty assigning clinical meaningfulness to z-score changes limited use of the MSFC as a primary outcome measure for registration trials.^5,6

In view of perceived limitations of the MSFC approach, and with recognition of the continuing need for better clinical measures of MS-related disability, the Multiple Sclerosis Outcome Assessments Consortium (MSOAC) was established in 2012 to accelerate the development of therapies for MS.⁷ MSOAC established the concept of interest (COI) for meaningful treatment benefit as “MS disability,” or simply “disability,” characterized as neurological or neuropsychological impairments that result in limitations in activities and restrictions in participation or life roles, caused by MS, that are understood to be important by the person with MS. Frequent interactions with the European Medicines Agency (EMA) and the U.S. Food and Drug Administration (FDA) served to shape the consortium’s research plan and guide efforts to select PerfOs (https://www.fda.gov/downloads/drugs/guidances/ucm230597.pdf) and to determine suitability of using specific PerfOs to quantify MS disability in MS clinical trials. The context of use (COU) for the selected PerfO was use as primary or secondary endpoints in clinical trials of treatments intended to slow or stop the worsening of disability in MS.

MSOAC first defined a conceptual framework for disability measures in MS, drawing on the International Classification of Functioning, Disability and Health (ICF) core sets for MS.⁸ Early on, MSOAC members highlighted the need for a visual measure to include as part of a multi-dimensional outcome measure and expressed a preference for the Symbol Digit Modalities Test (SDMT) over PASAT as a measure of processing speed because of accumulating experience with both tests. Also, MSOAC members agreed to focus on dimensions of MS that lent themselves to simple, objective, and reliable measurement and not focus on crucial dimensions of MS (e.g. pain, fatigue) that were inherently patient self-reported. A systematic literature review was conducted to assess published evidence on measures for walking speed, manual dexterity, vision, and information processing speed. This literature provided support for the ability of performance measures for these domains to capture how people with MS feel and function.^9–12

Key to the MSOAC goal is analysis of the prospectively acquired data from multiple clinical trials. This paper details the methods used to establish the MSOAC database and the Statistical Analysis Plan (SAP) that is presently being applied to assess the clinical meaningfulness of different performance measures. Future papers will report on the results of these analyses.

Methods and initial results

Establishing a consortium

MSOAC is organized and managed by the Critical Path Institute (C-Path; https://c-path.org/programs/msoac/). With input from NMSS, C-Path established the membership agreements and engaged a wide spectrum of stakeholders, including persons with MS, advocacy organizations, clinical researchers, industry sponsors, regulators, and other governmental agencies, all working together with standard development organizations, contract research organizations, and data managers (Supplementary Table 1). C-Path supplied expertise for development of therapeutic area data standards and the remapping of legacy data to the Clinical Data Interchange Standards Consortium (CDISC) data standard accepted by the FDA and used by C-Path for analytic purposes. C-Path staff also provided regulatory expertise to guide each step through the FDA’s¹³ Drug Development Tool and EMA’s Novel Methodologies qualification processes for PerfO qualification (Table 1).

Table 1.

Glossary.

Acronym	Term
9HPT	9-Hole Peg Test—a brief, standardized, quantitative test of upper extremity function
ADaM	Analysis Data Tabulation Model—the CDISC model that defines standards for analysis datasets
BDI	Beck Depression Inventory—a 21-question multiple-choice self-report inventory to measure the severity of depression
CDASH	Clinical Data Acquisition Standards Harmonization—describes the recommended data collection fields for 16 domains, including demographics, adverse events, and other domains common to most therapeutic areas and clinical research phases (https://www.cdisc.org/standards/foundational/cdash)
CDE	Common Data Element—data element that is common to multiple datasets across different studies: https://www.commondataelements.ninds.nih.gov/MS.aspx#tab=Data_StandardsNLM/NIH
CDISC	Clinical Data Interchange Standards Consortium is a non-profit Standards Development Organization (SDO): https://www.cdisc.org
CFAST	Coalition For Accelerating Standards and Therapies—formed by CDISC and C-Path to focus on therapeutic area data standards and analysis standards: https://www.cdisc.org/partnerships/cfast
ClinRO	Clinician-reported outcome measure—a measurement based on a report that comes from a trained healthcare professional after observation of a patient’s health condition. Most ClinRO measures involve a clinical judgment or interpretation of the observable signs, behaviors, or other manifestations related to a disease or condition.¹⁴
COA	Clinical Outcome Assessment—assessment of a clinical outcome can be made through report by a clinician, a patient, a non-clinician observer or through a performance-based assessment. There are four types of COAs: clinician-reported outcome, observer-reported outcome, patient-reported outcome, and performance outcome.¹⁴
COI	Concept(s) of Interest (COI) for meaningful treatment benefit—a description of the meaningful aspect of patient experience that will represent the intended benefit of treatment (e.g. presence/severity of symptoms, limitations in performance of daily activities).¹⁴
COU	Context of Use—A statement that fully and clearly describes the way the medical product development tool is to be used and the medical product development-related purpose of the use.¹⁴
EDSS	Expanded Disability Status Scale—a clinician-reported outcome measure of disability in MS
FSS	Functional Systems Scores—a clinician-reported measure of pyramidal, cerebellar, brainstem, sensory, bowel and bladder, visual, and cerebral (or mental) activity
LCLA	Low Contrast Letter Acuity
MIC	Minimal Important Change—smallest change in score in the domain of interest which patients perceive as important.¹⁵
MID	Minimal Important Difference—the difference observed between groups that are known to differ on the construct of interest in an important way.¹⁵
MS	Multiple Sclerosis
MSFC	Multiple Sclerosis Functional Composite—a three-part, standardized, quantitative assessment instrument for assessing mobility, dexterity and cognition in clinical studies of MS
PASAT	Paced Auditory Serial Addition Test—a test used to assess capacity and rate of information processing and sustained and divided attention
PPMS	Primary Progressive Multiple Sclerosis
PerfO	Performance Outcome Measure—a measurement based on a task(s) performed by a patient according to instructions that is administered by a healthcare professional.¹⁴
PRO	Patient-Reported Outcome Measure—a measurement based on a report that comes directly from the patient (i.e. study subject) about the status of a patient’s health condition without amendment or interpretation of the patient’s response by a clinician or anyone else.¹⁴
Qualification	Qualification—a conclusion, based on a formal regulatory process that within the stated context of use, a medical product development tool can be relied upon to have a specific interpretation and application in medical product development and regulatory review.¹⁴
RRMS	Relapsing Remitting Multiple Sclerosis
SAP	Statistical Analysis Plan
SDMT	Symbol Digit Modalities Test
SDTM	Study Data Tabulation Model—provides a standardized, predefined collection of domains for clinical data submissions
SF-36	Short Form (36) Health Survey—a 36-item patient-reported survey of patient health
SPMS	Secondary Progressive Multiple Sclerosis
T25FW	Timed 25-Foot Walk—a quantitative mobility and leg function performance test based on a timed 25-walk
TAPSC	Therapeutic Area standards Program Steering Committee—an operations group in CFAST focused on therapeutic area data standards
WHO ICF	World Health Organization International Classification of Functioning, Disability and Health

In addition to contributing data, many MSOAC members participated in a Coordinating Committee, which served as the governing body. Working groups were established to focus on (1) Defining Disability, (2) Data Standards and Integration, (3) Clinical Outcome Assessments, (4) Regulatory, (5) Literature Review, (6) Statistics, and (7) Voice of the Patient (VOP).

Selecting domains of function from the ICF core sets for MS

In a series of in-person meetings and teleconferences, the Defining Disability Workgroup examined ICF domains for MS.⁸ Inclusion and exclusion criteria (Table 2) were developed and applied to the ICF domains and to the associated measures of those domains. The Workgroup used the specified COU to provide a contextual anchor for the selection process. An important component of this process was the Workgroup’s mapping of the ICF domains to activities of daily living that are limited by MS. Several rounds of reviews were needed to reduce the candidate domains to a smaller set of finalist domains. The Workgroup then utilized a numerical rating system to arrive at a consensus concerning the most appropriate domains to be considered. This final set was then discussed with the Coordinating Committee, which endorsed the recommendations. The Workgroup then proceeded to identify the most appropriate performance measures to assess each of the domains.

Table 2.

Inclusion and exclusion criteria for selecting ICF domains.

Inclusion criteria	Exclusion criteria
A domain must represent something related to MS that affects a significant proportion of people with MS.	The domain is not thought to relate to activities, functions, or roles that are important to people with MS in their everyday lives.
A domain must be something that can be measured objectively and that does not rely entirely on patient-reported symptoms.	The domain is not commonly affected in people with MS (e.g. hearing).
A domain must be something that can be measured easily, with minimal equipment, and in a reasonable amount of time.	The domain does not change over time or vary depending on MS severity.
A domain must be something that affects a real-life function that is meaningful to a person with MS.	The domain cannot be objectively assessed (e.g. fatigue or pain).
A domain should preferably be one for which accessible data exist from MS clinical trials.	The function related to the domain cannot be quantified or cannot be measured using practical test procedures (e.g. sexual function).

Domains selected from the core and comprehensive ICF domains are shown in Table 3. Domains that did not represent common MS symptoms were eliminated with the understanding that the domain may be affected in a limited number of MS patients. Because the workgroup was only focused on objectively measurable domains, domains that could only be assessed by patient reports were eliminated with the understanding that certain of these domains (e.g. fatigue, depression) represent significant issues in MS. Given the COU, that is, large clinical trials, certain domains (e.g. gait pattern functions) were considered of value but too complex to incorporate in such studies. Both memory and speed of information processing were considered for inclusion as measures of cognition. Evidence from the literature indicated that speed of information processing is involved in memory and has a stronger relationship to real-life activities such as employment. Therefore, speed of information processing was selected as the most useful cognitive domain. The final domains selected reflect a core set of real-life functions meaningful to MS patients for which data exist in clinical trial datasets and in the scientific literature.

Table 3.

Activities of daily living limited by disability in MS mapped to ICF domains.

Activities of daily living limited by disability in MS	Bodily function involved in the activity of daily living	ICF domain^a	Comments	Possible neuro performance measures
1. Remembering to take medications^8,16	Cognition: learning, retention, and recall of information	b144	One of the most frequently impaired cognitive functions in MS patients but complex and time-consuming to measure.	1. California Verbal Learning Test 2. Brief Visuospatial Reminder Test 3. 7/24 Spatial Recall Test
2. Keeping up with conversations^8,17–19	Cognition: speed and accuracy of processing information	b1600b164	Very practical to measure with a good deal of literature to support it. In addition, it is a function that MS patients complain about and that is related to activities and participation.	1. Symbol Digit Modalities Test 2. Paced Serial Additions Test
3. Seeing someone crossing the street^8,20,21	Vision: recognizing people and objects	b120	Basic to many daily activities and practical, sensitive tests are available.	Low Contrast Letter Acuity
4. Reading a newspaper⁸	Vision: reading	b120	Basic to many daily activities and practical, sensitive tests are available.	Low Contrast Letter Acuity
5. Walking quickly to be on time for an appointment^8,22–24	Ambulation: walking at different speeds	d450b730	Frequently affected in MS and easily measured in varied clinical settings.	Timed 25-Foot Walk
6. Using a knife and fork, writing, and using a computer keyboard^8,25–27	Coordination: fine hand use	d440 d445b760	Often affected in MS and can interfere with a wide variety of important daily functions	9-Hole Peg Test

References in column 1 document the importance of the bodily function to people living with MS.

ICF Brief Core and Comprehensive Domains: b120: seeing functions; b760: voluntary movement functions; b144: memory; d440: fine hand use; b164: higher level cognitive functions; d445: hand and arm use; b730: muscle power functions; d450: walking; b1600: pace of thought.

Literature review and extraction methods

A related activity to further define the COI of disability in MS focused on the four domains selected by the Defining Disability Workgroup: ambulation, arm dexterity, vision, and cognition. Research questions were developed (Table 4) that could be addressed through an extensive literature review. Search parameters were designed to identify articles on performance measures relevant to domains of interest. In addition to the T25FW, 9HPT, LCLA, and SDMT, alternate measures used in the four domains were included in the literature search as well as articles that would combine domains in a disability assessment.

Table 4.

Literature review research questions.

1. What are the most common symptoms or impairments caused by MS?

2. Which MS symptoms or impairments are the most challenging for people with MS?

3. Which symptoms or impairments are most likely to be altered by treatment and/or are predictive of future worsening?

4. What daily activities are compromised by each of the symptoms or impairments of MS?

5. What validated measures exist to evaluate MS symptoms or impairments?

6. What are the psychometric properties of these measures, including dimensionality, reliability (reproducibility, internal consistency, inter-rater agreement, etc.), validity, objectivity, sensitivity to differences and change, predictive validity, and clinical meaningfulness, among others?

7. How feasible (cost, complexity, timeliness, etc.) are these measures for use in large clinical trials?

8. How have these measures performed to date in the context of clinical trials?

9. How adequate is the published evidence supporting the utilization of these measures, based on standard criteria for level of evidence?

10. What is the evidence concerning what constitutes the size of a change or difference in each measure that is both perceptible to a person living with MS and that constitutes an important difference in day-to-day function?

The literature review was performed in three levels (Figure 1). Parameters and search terms were defined (Supplementary Table 2) in Level 1 and abstract filtering criteria (Supplementary Table 3) were applied during the Level 2 Review. In reviewing the initial search, the Literature Review Workgroup identified a number of key papers that had been missed because key words and abstracts did not always include the performance measure search terms. Alternative search criteria increased the number of abstracts identified to approximately 9000. Broadening the search criteria captured the missing articles but also identified many other articles unrelated to the scope of the project. The Literature Review Workgroup decided to use an enrichment technique that allowed the addition of subject matter experts (SMEs)-recommended papers that should definitely be in the review. This combined “enriched search” approach identified approximately 3000 papers.

Figure 1.

Overview of literature review results.

Based on the results from the literature review, the SDMT was selected as the measure of choice^9,28,29 for processing speed. The workgroup considered potential measures of vision and decided on LCLA, utilizing 1.25% and 2.5% contrast Sloan charts, based on its strong performance in recent clinical trials.¹¹ Walking was considered as essential for inclusion, and the workgroup decided on walking speed as the most appropriate measure based in large part on the extensive use of the T25FW in clinical trials as part of the MSFC.^12,30 Finally, the workgroup endorsed the inclusion of a measure of manual dexterity to assess upper extremity function including coordination. The 9HPT, also part of the MSFC, was endorsed as the most appropriate measure in this domain based on its successful use in numerous clinical trials.^10,30

Articles analyzed by the Literature Review Workgroup (see Data Extraction Table, Supplementary Table 4) were drawn on for recently published review articles, which summarized the utility and validity of each recommended measure.^9–12 Authors of the reviews determined which of the identified articles should be included in the reviews. Using the vision domain as an example, the actual search parameters were not specifically designed to assess vision in all its aspects in MS, nor even LCLA when used in other non-MS settings. In addition, background and technical references (e.g. information on optical coherence tomography (OCT), visual evoked potentials (VEP) etc.) were included in the vision publication that support the use of LCLA in MS but that were not part of the formal literature search. A similar approach was used for published reviews of the other domains.

Developing a CDISC therapeutic area data standard for MS

To allow aggregation of data from clinical trials, a common data standard for MS had to be developed and data from each trial remapped to that standard. The process for creating the first MS data standard was instituted through the Coalition for Accelerating Standards and Therapies (CFAST), an initiative formed by CDISC and C-Path to create and maintain data standards in therapeutic areas important to public health. In general, the process mirrors that of other Standards Development Organizations (SDOs), including the International Organization for Standardization (ISO), Health Level 7 (HL7), and Integrating the Healthcare Enterprise (IHE). In brief, the sequential steps include scoping/charter, modeling and producing a draft standard, initial review and comment disposition, final public review and comment, disposition, and publication. The comment disposition ensures that those who contribute to the development process know how the comments were resolved to produce the resulting consensus-based standard.

C-Path submitted the scoping proposal for approval to the Therapeutic Area Program Steering Committee (TAPSC) that is organized by CFAST. The scoping proposal included a brief description of the project, including background information and proposed deliverables. Following approval of the scoping proposal, a detailed project proposal was submitted to TAPSC. The charter contained detailed information on the proposed standard, including focus populations, proposed team members/roles and other resources, stakeholder engagement considerations, concepts in scope, and a gap analysis of these concepts versus existing CDISC standards. The TAPSC reviewed and approved the charter. The Data Standards and Integration Workgroup developed the concept model and drafted the data standard, which was subsequently subjected to two rounds of review, including a public comment process. Revisions were incorporated, and a separate group of data standard experts carried out the final review and approval.

The Workgroup drew on the information content of the “common data elements (CDEs)” for MS that were developed through National Institute of Neurological Disorders and Stroke (NINDS)-supported efforts to identify those biomedical concepts that would form the MS CDISC data standard. Though CDEs guide researchers with recommendations on what should be captured and ensure consistent definitions of the captured content, they do not stand alone as data standards. A complete data standard also specifies how the collected data are represented in a database. Data standard specifications must also account for the often complex relationships between individual data elements to ensure that reviewers can construct accurate analyses involving multiple data elements which may exist in more than one table in the database.

Some of the retained CDEs were also further refined into comprehensive concepts—represented visually as concept maps—that described their origins in study-related processes and their interrelationships to other data elements. One such concept map (“relapse”) is presented in Figure 2. Standard development workgroup discussions revealed that multiple pathways can lead to the conclusion that relapse has occurred in patients with MS, including variations in relapse criteria and criteria for determining severity. These criteria are typically, but not always, anchored on changes in EDSS score. The resulting data standard accommodates this variation and specifies where in the CDISC data model (SDTM) this information can be found (represented by yellow boxes). In the CDISC Therapeutic Area User Guide (TAUG) for MS v1.0, this concept map is followed by more explicit mock data examples showing how these data are represented and how they are linked to each other in a relational database (http://www.cdisc.org/therapeutic#MS; Figure 3).

Figure 2.

A concept map representing relapse in MS.

Figure 3.

Process used for development of CDISC data standards for MS, v1.0.

During the development of the MS data standard, it was recognized that the organization of data within SDTM would benefit from the creation of two additional SDTM domains: (1) Functional Tests (FT), which includes performance measures such as the T25FW and 9HPT, and (2) Ophthalmic Examinations (OE), which includes the LCLA findings. The SDTM domains used for the MSOAC database are shown in Table 5.

Table 5.

Study data tabulation model (SDTM) domains used for the MSOAC database.

SDTM domain	Abbreviation	Observation class	Contents
Clinical Events	CE	Events	MS symptoms and relapse events and other events
Concomitant/Prior Medications	CM	Interventions	Betaseron, dexamethasone, glatiramer acetate, interferon, methylprednisolone, prednisolone, prednisone, etc.
Demographics	DM	Special purpose	Age, gender, race, trial arm, country
Disposition	DS	Events	Informed consent, randomization, reason for early withdrawal
Findings About Clinical Events	FACE	Findings sub-class	Number of relapses, relapses requiring hospitalization or steroids, result of relapse diagnosis tests, etc.
Findings About Medical History	FAMH	Findings sub-class	Number of relapses 1, 2, or 3 years before study start or since MS diagnosis; experienced acute relapse
Functional Tests	FT	Findings	T25FW, 9HPT, PASAT, SDMT
Medical History	MH	Events	MS diagnosis and pre-study symptoms, general medical history
Ophthalmic Examinations	OE	Findings	Visual acuity (low and high contrast)
Physical Examination	PE	Findings	General physical exam
Questionnaires	QS	Findings	BDI-FS, BDI-II, EDSS, FS scores, MSNQ, Neurological Change Questionnaire, RAND-36, SF-36, SF-12
Reproductive System Findings	RP	Findings	Pregnancy test
Subject Characteristics	SC	Findings	Dominant hand
Subject Disease Milestones	SM	Special purpose	MS relapse events
Trial Disease Milestones	TM	Trial design	Definitions of MS relapse

The Functional Tests (FT) and Ophthalmic Examinations (OE) domains were developed as a result of the therapeutic area data standard for MS. PASAT and SDMT are cognitive function tests that are included in the FT domain.

Acquiring, standardizing, and pooling data from MS clinical trials

To facilitate sharing of clinical trial data, C-Path developed two legal agreements that govern MSOAC membership and data contributions. Following execution of the legal agreements, MSOAC acquired 16 datasets from consortium industry and academic members (Table 6) and remapped the data to the new CDISC data standard for MS (Figure 4). The standardized data consisting of control and treatment arms of clinical trials formed the MSOAC database. The database includes information on a range of performance measures from 14,370 study subjects. Baseline descriptive statistics for age, sex, race, treatment arms, and disease severity as assessed by EDSS are shown in Figure 5.

Table 6.

Source datasets in the MSOAC database.

CT.gov no.^a	Description	N	MS Type	EDSS	FSS	T25FW	9HPT	PASAT	SDMT	LCLA	SF-36	BDI-II
NCT00027300	AFFIRM	939	RRMS	√	√	√	√	√	NO	√	√	NO
NCT00030966	SENTINEL	1196	RRMS	√	√	√	√	√	NO	√	√	√
NCT00127530	MS-F203	301	ALL	√	NO	√	NO	NO	NO	NO	NO	NO
NCT00134563	TEMSO	1086	RRMS	√	√	√	√	√	NO	NO	√	NO
NCT00211887	COMBIRX	1008	RRMS	√	√	√	√	√	NO	√	√	NO
NCT00289978	FREEDOMS	1272	RRMS	√	√	√	√	√	NO	NO	NO	NO
NCT00297232	STRATA	1094	RRMS	√	√	NO	NO	NO	√	NO	NO	BDI-FS
NCT00340834	TRANSFORMS	1292	RRMS	√	√	√	√	√	NO	√	NO	NO
NCT00355134	FREEDOMS II	1083	RRMS	√	√	√	√	√	NO	√	NO	NO
NCT00483652	MS-F204	239	ALL	√	NO	√	NO	NO	NO	NO	NO	NO
NCT00530348	CARE-MS 1	563	RRMS	√	√	√	√	√	NO	√	√	NO
NCT00548405	CARE-MS 2	798	RRMS	√	√	√	√	√	NO	√	√	NO
NCT00869726	MAESTRO	610	SPMS	√	√	√	√	√	NO	NO	√	NO
NCT00906399	ADVANCE	1512	RRMS	√	√	√	√	√	√	√	SF-12	√
N/A^b	PROMISE	943	PPMS	√	√	√	√	√	NO	NO	√	NO
N/A^b	IMPACT	434	SPMS	√	√	√	√	√	NO	NO	√	√

The outcome measures that are included in the MSOAC database from each study are indicated by a check; No indicates that data from a particular measure are not included in the database. N is the number of subjects in a dataset; for the MS Type, “All” includes RRMS, SPMS, and PPMS.

CT.gov refers to the Clinical Trials.gov website where clinical trials are registered.

Study does not have a Clinical Trials.gov identifier.

Figure 4.

Steps in data mapping.

Figure 5.

Baseline descriptive statistics for the pooled subjects in the MSOAC Database.

As a resource for the research community, a database containing the placebo arms of MS clinical trials was also established (https://c-path.org/programs/msoac/). C-Path staff secured permission for the inclusion of ~2500 individual patient records that are part of the overall MSOAC database and developed the infrastructure to support the storage, security, access requests, data use agreements, and access approvals, including a standing Review Board. Baseline descriptive statistics of this placebo-arm database are shown in Figure 6.

Figure 6.

Baseline descriptive statistics for the pooled subjects in the placebo-arm database.

Analyzing data in the MSOAC database

MSOAC’s Statistics Workgroup developed the SAP, incorporating both regulatory feedback and recommendations from MSOAC members on which functional domains to examine, the analyses to be performed, and the optimal approach for incorporating the VOP. MSOAC members living with MS reviewed the initial plans for measuring clinically meaningful aspects of disability and identified gaps in the approach. A literature review provided insights on what aspects of disability are of most importance to people with MS and what performance measures adequately capture those concepts. Four performance measures were selected for detailed analysis, based on literature review and availability of PerfOs in the MSOAC database: the T25FW for ambulation, the 9HPT for manual dexterity, LCLA for vision (1.25% and 2.5% contrast), and both the SDMT and the PASAT for cognition. As detailed in the supplementary material, the following attributes were assessed for each measure: floor or ceiling effects, test–retest reliability, change over time, construct validity, convergent validity, extent of practice effects, known-group validity, sensitivity to change, and the minimum clinically important change in performance scores. Both the placebo arm and the treatment arm of the aggregated data were used for the statistical analyses. Results from the statistical analyses will be reported separately.

Conclusion

MSOAC was formed to develop more sensitive methods to measure whether a drug effectively reduces disability worsening in MS, based on the belief that acceptance of a more sensitive and precise, yet meaningful measure would accelerate progress in developing effective MS therapies. Therefore, the primary purpose for MSOAC was to qualify disability performance measures as primary or secondary endpoints for MS clinical trials submitted to the FDA and the EMA. Qualification of the SDMT as a measure of information processing speed is underway at the FDA, and qualification of all four performance measures (SDMT, T25FW, 9HPT, and LCLA) is in process at the EMA. In addition, given the preference for simple, reproducible performance tests, the consortium recognized that the same outcome measures could be useful within medical practice to grade MS severity and monitor patients over time, potentially harmonizing the metrics used in clinical trials and clinical practice. By including the same operationally defined, quantitative measures in clinical trials and healthcare settings, it should be possible to use “real world data” to augment clinical trials, test interventions in less controlled settings, and realize the potential of the “learning health system.”³¹

MSOAC is a global effort, with members from 5 advocacy organizations, 2 regulatory agencies and 1 other governmental agency, 12 pharmaceutical companies, 23 academic institutions, 4 consultant groups, and 4 non-profit organizations. By sharing data and expertise in teams of volunteers that reported to the MSOAC Coordinating Committee (i.e. the Data Standards and Integration Workgroup, Defining Disability Workgroup, Clinical Outcome Assessments Workgroup, Regulatory Advisory Workgroup, Literature Review Workgroup, Statistical Workgroup, and VOP Workgroup), consortium members have delivered the following: (1) the first TAUG for MS, which is freely available at http://www.cdisc.org/therapeutic#MS; (2) a standardized database of 14,370 trial subjects for use in qualification of new PerfOs; (3) a placebo-arm database for use by the research community; (4) an extensive review of the literature on performance measures relevant to MS;^9–12 and (5) analyses of performance measure data for submission to the FDA and the European Medicines Agency for qualification. An approach to assess clinical meaningfulness of differences in the four measures by directly engaging persons with MS is also underway. Termed the VOP, this effort will contribute evidence toward the clinical meaningfulness of walking speed, manual dexterity, visual acuity, and speed of information processing in the lives of people with MS.

The consortium approach is not without challenges. Sharing data proved difficult or impossible for several members. All but one participant were willing to provide the needed copyright permissions for incorporation of scales into the MS CDISC data standard. Stakeholders were initially divided on the research plan, including optimal approaches to establish clinical meaningfulness.

Generating a CDISC standard for MS was a milestone that allowed pooling of clinical trial data for MSOAC’s analysis and regulatory submission. The CDISC standard also provided a new tool to the entire MS community, which is of value now that all drug trial data submitted to the FDA³² and Pharmaceuticals and Medical Devices Agency (PMDA) must be in CDISC format. Another consortium objective that benefits the research community was the creation of a separate database containing the placebo-arm data from registration trials (https://c-path.org/programs/msoac/). Most importantly, MSOAC’s proposed outcome measure, once qualified by the EMA and the FDA, will be adopted by drug developers to demonstrate treatment benefit of therapies designed to slow progression of disability and promote improvement in MS. MSOAC illustrates the potential for pre-competitive, cooperative, consortium-driven progress in drug development tools that benefit both sponsors and the broader MS community.

Footnotes

Acknowledgements

The authors gratefully acknowledge the perspectives provided by Weyman Johnson, William Anthony, and Elizabeth Morrison-Banks, which served to focus the effort on clinical meaningfulness. In addition to the MSOAC members who co-authored this article, participants who have contributed to MSOAC projects through MSOAC workshops, teleconferences, and workgroups include the following (listed alphabetically by affiliation): Steven Greenberg (AbbVie), Jane Haley (AbbVie), Xiaolan Ye (AbbVie), Thomas Marshall (AbbVie), Andrew Blight (Acorda), Craig Sherburne (Alberta MS Research Foundation), Christina Casteris (Biogen), John Richert (Biogen), Gilmore O’Neill (Biogen), Jacob Elkins (Biogen), Tim Swan (Biogen), Jesse Cedarbaum (Bristol-Myers Squibb), Sanjay Keswani (Bristol-Myers Squibb), Tanuja Chitnis (Brigham and Women’s Hospital), Dan Ontaneda (Cleveland Clinic), June Halper (Consortium of Multiple Sclerosis Centers), Bob Stafford (Critical Path Institute), Bess LeRoy (Critical Path Institute), Stephen Joel Coons (Critical Path Institute), Maria Isaac (European Medicines Agency, Geoffrey Dunbar (EMD Serono)), Tanya Fischer (EMD Serono), Thorsten Eickenhorst (EMD Serono), Irina Antonijevic (Sanofi Genzyme), Stephen Lake (Sanofi Genzyme), David Margolin (Sanofi Genzyme), Jeff Palmer (Sanofi Genzyme), Phillipe Truffinet (Sanofi Genzyme), Paul Thompson (GlaxoSmithKline), Maria Davy (GlaxoSmithKline), Gill Webster (Innate Immunotherapeutics), Simon Wilkinson (Innate Immunotherapeutics), Giampaolo Brichetto (Italian Multiple Sclerosis Society), Paola Zaratin (Italian Multiple Sclerosis Society), Kathryn Fitzgerald (Johns Hopkins University), Peter Calabresi (Johns Hopkins University), Lauren Strober (Kessler Foundation Research Center), Wendy Kaye (McKing Consulting), Aaron Miller (Icahn School of Medicine at Mount Sinai), Timothy Coetzee (National MS Society), Karen Lee (MS Society of Canada), Susan Kohlhass (MS Society UK), Ursula Utz (National Institute of Neurological Disorders and Stroke), Frank Dahlke (Novartis), David Leppert (Novartis), Paul McGuire (Novartis), Jeremy Hobart (Plymouth Hospital), Shari Medendorp (Premier Research), Adam Jacobs (Premier Research), Bruno Musch (Roche/Genentech), Donna Masterman (Roche/Genentech), Algirdas Kakarieka (Roche/Genentech), Giancarlo Comi (Scientific Institute H.S. Raffaele), Lauren Krupp (NYU Langone Medical Center), Joshua Steinerman (Teva), Volker Knappertz (Teva), Maria Pia Sormani (University of Genoa), Brenda Banwell (University of Pennsylvania), Andrew Goodman (University of Rochester), Jerry Wolinsky (University of Texas), Sarrit Kovacs (US FDA), Marc Walton (US FDA), Wen-Hung Chen (US FDA), Michelle Campbell (US FDA), Elektra Papadopoulos (US FDA), William Dunn (US FDA), and Chris Polman (VU Medical Center). The MSOAC Directors gratefully acknowledge the encouragement and guidance provided by our FDA colleagues throughout the course of this work. The authors thank Alicia West, MSOAC’s project coordinator, for expert technical assistance.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: MSOAC is funded largely through the National Multiple Sclerosis Society grant (no. RG 4869-A-1) to the Critical Path Institute. Annual dues from sponsors supplement the NMSS grant.

References

Whitaker

McFarland

Rudge

et al . Outcomes assessment in multiple sclerosis clinical trials: A critical analysis. Mult Scler 1995; 1(1): 37–47.

Kurtzke

JF.

Rating neurologic impairment in multiple sclerosis: An Expanded Disability Status Scale (EDSS). Neurology 1983; 33(11): 1444–1452.

Rudick

Antel

Confavreux

et al . Clinical outcomes assessment in multiple sclerosis. Ann Neurol 1996; 40(3): 469–479.

Balcer

Baier

Cohen

et al . Contrast letter acuity as a visual component for the multiple sclerosis functional composite. Neurology 2003; 61(10): 1367–1373.

Ontaneda

LaRocca

Coetzee

et al . ; NMSS MSFC Task Force. Revisiting the multiple sclerosis functional composite: Proceedings from the National Multiple Sclerosis Society (NMSS) Task Force on Clinical Disability Measures. Mult Scler 2012; 18(8): 1074–1080.

Fox

Thompson

Baker

et al . Setting a research agenda for progressive multiple sclerosis: The International Collaborative on Progressive MS. Mult Scler 2012; 18(11): 1534–1540.

Rudick

Larocca

Hudson

et al . Multiple Sclerosis Outcome Assessments Consortium: Genesis and initial project plan. Mult Scler 2014; 20(1): 12–17.

Coenen

Cieza

Freeman

et al . ; Members of the Consensus Conference. The development of ICF Core Sets for multiple sclerosis: Results of the International Consensus Conference. J Neurol 2011; 258(8): 1477–1488.

Benedict

DeLuca

Phillips

et al . ; Multiple Sclerosis Outcome Assessments Consortium. Validity of the Symbol Digit Modalities Test as a cognition performance outcome measure for multiple sclerosis. Mult Scler 2017; 23: 721–733.

10.

Feys

Lamers

Francis

et al . ; Multiple Sclerosis Outcome Assessments Consortium. The Nine-Hole Peg Test as a manual dexterity performance measure for multiple sclerosis. Mult Scler 2017; 23: 711–720.

11.

Balcer

Raynowska

Nolan

et al . ; Multiple Sclerosis Outcome Assessments Consortium. Validity of low-contrast letter acuity as a visual performance outcome measure for multiple sclerosis. Mult Scler 2017; 23: 734–747.

12.

Motl

Cohen

Benedict

et al . ; Multiple Sclerosis Outcome Assessments Consortium. Validity of the timed 25-foot walk as an ambulatory performance outcome measure for multiple sclerosis. Mult Scler 2017; 23: 704–710.

13.

FDA. Clinical Outcome Assessment Qualification Program, https://www.fda.gov/Drugs/DevelopmentApprovalProcess/DrugDevelopmentToolsQualificationProgram/ucm284077.htm; European Medicines Agency. Scientific advice and protocol assistance—Qualification of novel methodologies for medicine development, http://www.ema.europa.eu/ema/index.jsp?curl=pages/regulation/document_listing/document_listing_000319.jsp

14.

The BEST (Biomarkers, EnpointS and other Tools) resource, https://www.ncbi.nlm.nih.gov/books/NBK338448/

15.

Mayo

NE.

Dictionary of quality of life and health outcomes measurement. 1st ed. Milwaukee, WI: International Society for Quality of Life Research, 2015.

16.

Rao

Leo

Ellington

et al . Cognitive dysfunction in multiple sclerosis. II. Impact on employment and social functioning. Neurology 1991; 41(5): 692–696.

17.

Strober

Christodoulou

Benedict

RHB

et al . Unemployment in multiple sclerosis: The contribution of personality and disease. Mult Scler 2012; 18(5): 647–653.

18.

Sosnoff

Balantrapu

Pilutti

et al . Cognitive processing speed is related to fall frequency in older adults with multiple sclerosis. Arch Phys Med Rehabil 2013; 94(8): 1567–1572.

19.

Strober

Rao

Lee

et al . Cognitive impairment in multiple sclerosis: An 18 year follow-up study. Mult Scler Relat Disord 2014; 3(4): 473–481.

20.

Balcer

Galetta

Polman

et al . Low-contrast acuity measures visual improvement in phase 3 trial of natalizumab in relapsing MS. J Neurol Sci 2012; 318(1–2): 119–124.

21.

Feaster

Bruce

JM.

Visual acuity is associated with performance on visual and non-visual neuropsychological tests in multiple sclerosis. Clin Neuropsychol 2011; 25(4): 640–651.

22.

Motl

McAuley

Wynn

et al . Physical activity, self-efficacy, and health-related quality of life in persons with multiple sclerosis: Analysis of associations between individual-level changes over one year. Qual Life Res 2013; 22(2): 253–261.

23.

Pilutti

Dlugonski

Sandroff

et al . Gait and six-minute walk performance in persons with multiple sclerosis. J Neurol Sci 2013; 334(1–2): 72–76.

24.

Sidovar

Limone

Lee

et al . Mapping the 12-item multiple sclerosis walking scale to the EuroQol 5-dimension index measure in North American multiple sclerosis patients. BMJ Open 2013; 3(5): e002798.

25.

Bosma

Kragt

Polman

et al . Walking speed, rather than Expanded Disability Status Scale, relates to long-term patient-reported impact in progressive MS. Mult Scler 2013; 19(3): 326–333.

26.

Costelloe

O’Rourke

McGuigan

et al . The longitudinal relationship between the patient-reported Multiple Sclerosis Impact Scale and the clinician-assessed Multiple Sclerosis Functional Composite. Mult Scler 2008; 14(2): 255–258.

27.

Polman

Rudick

RA.

The multiple sclerosis functional composite: A clinically meaningful measure of disability. Neurology 2010; 74(suppl. 3): S8–S15.

28.

Goverover

Strober

Chiaravalloti

et al . Factors that moderate activity limitation and participation restriction in people with multiple sclerosis. Am J Occup Ther 2015; 69(2): 6902260020p1–6902260020p9.

29.

Morrow

Drake

Zivadinov

et al . Predicting loss of employment over three years in multiple sclerosis: Clinically meaningful cognitive decline. Clin Neuropsychol 2010; 24(7): 1131–1145.

30.

Rudick

Polman

Cohen

et al . Assessing disability progression with the Multiple Sclerosis Functional Composite. Mult Scler 2009; 15(8): 984–997.

31.

Califf

Robb

Bindman

et al . Transforming evidence generation to support health and health care decisions. N Engl J Med 2016; 375(24): 2395–2400.

32.

FDA references for the CDISC data standards—Study Data for Submission to CDER and CBER, https://www.fda.gov/Drugs/DevelopmentApprovalProcess/FormsSubmissionRequirements/ElectronicSubmissions/ucm248635.htm; PMDA references for the CDISC data standards requirements, http://www.pmda.go.jp/files/000153708.pdf#page=6

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

1.59 MB

0.62 MB