Orthosis Comfort Score: Establishing initial evidence of reliability and validity in ankle foot orthosis users

Abstract

Background:

Comfort of an orthosis is an important characteristic that is likely to dictate use of and satisfaction with a device. However, instruments to assess only orthosis user comfort do not exist. The Prosthetic Socket Fit Comfort Score, developed previously for prosthesis users, may be adapted to serve this purpose.

Objectives:

This study’s purpose was to assess the validity and reliability of the Orthosis Comfort Score, a self-report instrument adapted from the Prosthetic Socket Fit Comfort Score.

Study design:

This is a prospective, observational study designed to establish initial evidence of validity and reliability for an outcome measure that assesses comfort.

Methods:

Ankle foot orthosis users completed the Orthosis Comfort Score and two validated patient satisfaction questionnaires. An orthotist documented an assessment of fit. Post-visit Orthosis Comfort Scores were documented after the appointment and 2–4 weeks later. Orthosis Comfort Scores were compared to the patient satisfaction questionnaires, assessment of fit and orthosis use (hours per week).

Results:

There were 46 study participants. Orthosis Comfort Scores had a moderate positive correlation with their orthotist’s assessment of fit, very strong positive correlations with patient satisfaction questionnaires and fair positive correlation with orthosis use (all correlations p < 0.05).

Conclusion:

This study demonstrates initial evidence for the validity and reliability of the Orthosis Comfort Score in ankle foot orthosis users.

Clinical relevance

The Orthosis Comfort Score is a simple patient-reported outcome measure that can be readily incorporated into clinical practice or research study to obtain a rapid assessment of comfort. It can be used to facilitate communication about device fit, evaluate comfort over time and/or assess changes in comfort with a new device.

Background

Orthoses are devices which interact closely with the body to improve overall function and well-being.^1,2 Patients’ experience of orthosis use, including perceived benefits, disadvantages and satisfaction, is strongly influenced by how comfortable the orthosis is to wear. ² Lack of comfort is a frequent complaint to health care professionals and one of the most common reasons patients decide to discontinue use of their orthosis.^2–4 To address this issue, orthotists frequently make adjustments to orthotic devices to improve comfort.

Currently, there are no specific outcome measures developed to assess orthosis comfort. Existing outcome measures are designed either to assess comfort in other patient populations or include comfort as a component of broader constructs such as patient satisfaction.^1,2,5 There is potential to adapt an existing outcome measure to explicitly assess comfort with an orthosis. However, before an outcome measure can be recommended for use in clinical practice, there needs to be evidence supporting its psychometric properties in the target population. ⁶

Hanspal et al. ⁵ developed the Prosthetic Socket Fit Comfort Score (SCS) to assess comfort in prosthetic device users. They adapted the frequently used numerical rating scale for pain to create a single question, patient-reported outcome measure of socket comfort. ⁵ The SCS has evidence for convergent validity with moderate to strong correlations with prosthesis fit and residual limb health. ⁵ The SCS also has strong evidence for reliability. ⁵ The SCS is simple, straightforward and quick to administer. It is routinely used clinically and for research regarding prosthetic device comfort.^7–12

Patient-reported outcome measures developed to assess satisfaction of orthosis users include the Client Satisfaction with Device subsection of the Orthotics and Prosthetic Users’ Survey (OPUS-CSD) and the Assistive Devices subscale of the Quebec User Evaluation of Satisfaction with Assistive Technology 2.0 (QUEST 2.0).^13–19 The OPUS-CSD is an eight-item questionnaire where each item is rated on a 4-point Likert-type scale from strongly disagree to strongly agree. Item 2 asks the level of agreement with the statement ‘my device is comfortable throughout the day’. The QUEST 2.0 is an eight-item questionnaire where each item is rated on a 5-point ordinal response scale from not satisfied at all to very satisfied, with space for comments. Item 7 of the QUEST 2.0 reads, ‘How satisfied are you with how comfortable your assistive device is?’

The OPUS-CSD and QUEST 2.0 have some evidence for reliability and validity to evaluate patient satisfaction in orthosis users.^13–19 Both include an item regarding comfort. A recent systematic review concluded that both questionnaires have high reliability, the OPUS-CSD has satisfactory unidimensionality and the QUEST 2.0 has adequate construct validity.^1,16,19 While these questionnaires appropriately assess comfort, their aim is to assess multiple factors that contribute to patient satisfaction. ¹⁶ Therefore, they are not specific to comfort. In addition, they likely take longer to complete, which may be challenging in a clinical setting.

Given the importance of comfort to orthosis users, it would seem useful to have an outcome measure that focused solely on comfort. The SCS has been widely used by prosthetists. Therefore, we chose to adapt the SCS to create a similar measure, the Orthosis Comfort Score (OCS).

This study’s aim is to provide evidence for the validity and reliability of the OCS. We hypothesized that the OCS will have evidence for convergent and predictive validity, reliability, as well as responsiveness to change in ankle foot orthosis (AFO) users. Specifically, we hypothesized that OCSs and scores from instruments and items that measure similar constructs would be strongly correlated (ρ > 0.8), OCSs and the orthotist assessment of fit would be moderately correlated (ρ > 0.5), OCSs would be moderately correlated (ρ > 0.5) with AFO use time, repeated OCSs obtained within a 2- to 4-week period would be moderately correlated (ρ > 0.5), OCSs obtained by paper and orally at the same visit would be very strongly correlated (ρ > 0.8) and change in OCSs would be significantly greater for people who received an AFO adjustment than in those who did not. We also hypothesized that it will take less time to complete the OCS than the QUEST 2.0 and OPUS-CSD.

Methods

The study was approved by the Ottawa Health Science Network Research Ethics Board (protocol ID 20170024-01H). Participation in the study was voluntary, and all participants provided written informed consent.

A prospective observational study design was used. Potential participants were identified and recruited through The Ottawa Hospital Rehabilitation Centre’s outpatient rehabilitation orthotics clinic and the Prosthetics and Orthotics Department from June 2017 to June 2018. Participants were included if they were at least 18 years old, able to read and write in English and currently had an AFO. Participants were excluded if they had complete sensory loss below the knee or if they had cognitive deficits preventing them from providing informed consent or completing questionnaires.

A target sample size of 50 participants was calculated for our primary outcome of convergent validity. The calculation was based on the assumption that we would find similar correlations between assessment of fit and the OCS as Hanspal et al., ⁵ who identified a Kendall’s tau correlation coefficient of 0.48 in a sample of 44 participants. A sample of 50 participants would give an estimated 98% power to identify a statistically significant correlation if the correlation coefficient was 0.5% and 80% power if the correlation coefficient was as low as 0.35.

Protocol

The question for the SCS was modified for orthosis users. ⁵ The OCS reads, ‘On a 0-10 scale, if 0 represents the most uncomfortable orthosis fit you can imagine and 10 represents the most comfortable orthosis fit, how would you score the comfort of your orthosis at the moment?’

Participants were scheduled for an appointment with their orthotist at the Prosthetics and Orthotics Department at The Ottawa Hospital Rehabilitation Centre. They completed four questionnaires prior to their appointment. These included a baseline data questionnaire (age, gender, diagnosis leading to prescription of AFO, time since AFO delivery in months and use of AFO in past week in days/week and hours/day), OPUS-CSD, QUEST 2.0 and pre-appointment OCS (OCS-pre). The latter three questionnaires were done in a randomized order to minimize the effects of familiarity with items being asked.

After completing the questionnaires, participants had an appointment with their orthotist, who was blinded to the questionnaire results. During the appointment, their orthotist rated the fit of their AFO based on history and physical assessment of the patient on a 5-point rating scale (1-poor, 2-unsatisfactory, 3-fair, 4-good or 5-excellent), as done by Hanspal et al. ⁵ Their orthotist then asked them the OCS question verbally (OCS-verbal) to assess alternate form equivalence. Following this, the orthotist would make any adjustments that they felt were clinically indicated. They recorded whether adjustments were made or not. At the end of their appointment, after having opportunity to try on their orthosis and walk within the department, participants completed a post-appointment OCS question (OCS-post) on paper.

Participants received a follow-up phone call 2–4 weeks after their appointment, where they were asked the OCS question (OCS-phone) and about the use of their AFO in the past week (days/week and hours/day).

Data analysis

The Spearman’s correlation (ρ), a non-parametric test, was used to determine correlations as the OCSs were not normally distributed and orthotist assessment of fit, OPUS-CSD and QUEST 2.0 are ordinal variables. To assess for convergent validity, we compared OCSs to the orthotist assessment of fit, OPUS-CSD and QUEST 2.0, as well as the specific items of the latter two which addressed comfort. To assess for predictive validity, we compared the OCSs to the use of the AFO (total number of hours in the past week).

To determine if OCSs were stable over time (test–retest reliability), the OCS at the end of their appointment (OCS-post) was compared to the OCS at their follow-up phone call 2–4 weeks later (OCS-phone) using Spearman’s correlation.

To assess for alternate form equivalence, Spearman’s correlation and descriptive statistics were used to determine if participants reported the same OCS when anonymously recording on paper (OCS-pre) and verbally stating to their orthotist (OCS-verbal).

To assess the responsiveness of the OCS, the participants were separated into two groups (Group 1 – adjustment to AFO performed and Group 2 – no adjustment to AFO). The difference between OCS-pre and OCS-post was calculated for each group and assessed with a chi-squared test. The standardized response mean, as described by Husted et al., ²⁰ was calculated to determine effect size.

The mean times to complete the OCS, CSD-OPUS and QUEST 2.0 were calculated as the time the questionnaire was started, subtracted from the time the questionnaire was completed, in minutes. If the questionnaire start time and end time were the same, the time to complete was rounded up to 1 minute. Otherwise, the time was rounded up or down to the nearest minute. Missing data were excluded from analysis. A repeated measures analysis of variance (ANOVA) with Greenhouse–Geisser correction followed by post hoc tests using the Bonferroni correction was used to compare means.

For all statistical tests, a p value of <0.05 was considered significant. Strength of correlation was interpreted using values described by Gust and D’journo, ²¹ where a very strong correlation is ρ > 0.8, moderate to strong correlation is from 0.5 > ρ > 0.8, fair correlation is from 0.2 > ρ > 0.5 and poor if ρ < 0.2. Statistical analyses were performed using SPSS software (IBM SPSS Statistics for Windows, Version 23.0, Armonk, NY: IBM Corp.).

Results

A total of 63 individuals were contacted regarding the study. In total, 5 did not meet the inclusion and exclusion criteria, 1 did not consent to the study and 11 withdrew, leaving 46 participants. There was 1 participant out of the 46 who was not able to complete the OCS phone component of the study as they had been admitted to hospital. This participant was excluded from the test–retest reliability calculation. Reasons for withdrawal were appointments that were missed, cancelled or rescheduled outside of the study dates.

A wide variety of patient-reported clinical conditions necessitated the use of the AFO, with the most common being foot drop and peripheral neuropathy. A total of 60.9% of the participants were male (n = 28). The mean age was 52.4 ± 17.0 years. Participants had used an AFO for a mean of 38.3 months (range: 1.5 weeks–29 years).

Convergent validity

There was a moderate positive correlation between the OCSs-pre and the orthotist assessments of fit. There were very strong positive correlations between the OCSs and the OPUS-CSD, QUEST 2.0 and their specific questions about comfort (Table 1).

Table 1.

Evidence for convergent validity of OCS by comparison to clinical assessment of fit, patient satisfaction score and individual item scores using Spearman’s rho. Interpretation is as per the values in Gust et al. ²¹

Comparator	Spearman’s rho (ρ)	p value	Strength of correlation
Orthotist Assessment of Fit	0.603	<0.001	Moderate
CSD-OPUS	0.816	<0.001	Very strong
QUEST 2.0	0.836	<0.001	Very strong
CSD-OPUS item 2	0.862	<0.001	Very strong
QUEST 2.0 item 7	0.850	<0.001	Very strong

OCS: Orthosis Comfort Score; CSD-OPUS: Client Satisfaction with Device subsection of the Orthotics and Prosthetic Users’ Survey; QUEST: Quebec User Evaluation of Satisfaction with Assistive Technology.

Predictive validity

Predictive validity was assessed by comparing OCSs to the hours of use of the AFO. A positive relationship of fair strength existed between the OCSs and amount of orthosis use in hours per week (ρ = 0.257, p = 0.016).

Reliability

Positive correlations of moderate strength were noted between OCSs at the end of participants’ appointments (OCS-post) and OCSs 2–4 weeks later (OCS-phone) at the time of their follow-up phone call (ρ = 0.643, p < 0.001), indicating moderate test–retest reliability.

Alternate form equivalence

The positive correlation between the OCS reported anonymously (OCS-pre) and the OCS reported to the orthotist (OCS-verbal) was very strong (ρ = 0.933, p < 0.001). The mean difference in scores when the OCS-pre was subtracted from the OCS-verbal was 0.217 ± 0.841 (range: –1 to 4). Together, these provide evidence of alternate form equivalence.

Responsiveness to change

In total, 35 participants had adjustments to their AFO during their appointment. For these individuals, there was a significant increase in the OCS of 1.76 ± 1.92 points (p < 0.001). The standardized response mean was 0.916, reflecting a large effect size, as described by Husted et al. ²⁰ Those who did not have adjustments to their AFO had no significant change in the OCS (0.1 ± 0.3 points, p = 0.16).

Time to complete

The OCS took participants 1.13 ± 0.33 minutes to complete. The CSD-OPUS took 1.87 ± 1.28 minutes and QUEST 2.0 took 2.76 ± 1.24 minutes (Figure 1). Repeated measures ANOVA showed a statistically significant difference (F(1.89, 64.22) = 27.135, p < 0.001), and post hoc tests showed significant differences between all means (OCS and CSD-OPUS, OCS and QUEST 2.0, CSD-OPUS and QUEST 2.0; all p < 0.001).

Figure 1.

Time to complete OCS in a clinical setting – average length of time in minutes required to complete the questionnaires (OCS, CSD-OPUS and QUEST 2.0), rounded to the nearest minute.

Discussion

Comfort is an important quality which influences patients’ use of orthotic devices; however, there is a paucity of tools in the literature that assess orthosis comfort. The SCS is a simple, validated outcome measure for prosthetic socket comfort. ⁵ This study adapted this tool for use with orthotic devices (OCS) and established initial validity evidence for its use.

The results of this study demonstrate initial evidence for convergent validity of the OCS. Prior studies have found that comfort is the most important item contributing to patient satisfaction on the QUEST 2.0. ¹⁶ Therefore, it is not surprising that the OCS was strongly correlated with it. Deviations from a perfect correlation (1.0) may be explained by the numerous other factors, including appearance and functionality, that contribute to patient satisfaction, as measured by the QUEST 2.0 and CSD-OPUS questionnaires. The OCS is strongly correlated with the specific items on patient satisfaction questionnaires that ask directly about comfort. The lack of perfect correlation may suggest that the way questions are phrased influences the way that the patient responds. For example, the OCS asks how comfortable the device is, while the QUEST 2.0 asks how satisfied the person is with the comfort of the device. These are slightly different concepts. A patient who finds that their AFO increases their function dramatically may be satisfied with a lower level of comfort and hence report a higher score for ‘satisfaction with comfort’ but a lower score for ‘level of comfort’. The moderate correlation between OCS and the orthotist assessment of fit suggests that comfort of the device is associated, to a certain extent, with the fit of the device. The strength of this association is similar to the correlation observed between socket comfort and prosthetist-assessed fit in the SCS development study. ⁵ There are other factors related to fit that may impact comfort, so the correlation would not always be expected to be strong. For example, an AFO that is designed to provide some correction of a deformity and fits perfectly might create minor discomfort for the patient as it stretches a tendon.

Initial evidence of predictive validity was demonstrated by a fair correlation between the OCS and the amount of time participants wore their AFO. This study included any AFO user, not just those advised to wear their brace most or all of the time. Certain AFOs are designed to be used only during certain tasks, therefore worn for shorter periods of time, for example, walking outdoors. This may explain why the strength of the association between comfort and AFO use was only fair. Further studies could analyse participants who are advised to wear their AFO most of the time to further clarify the relationship between hours of use and the OCS. As well, there are other reasons besides comfort that impact the decision to wear or not wear a brace which could be documented in future studies.

The moderate positive correlation between OCSs at the end of the participants’ appointment and 2–4 weeks later demonstrates initial evidence for test–retest reliability. A 2-week interval between assessments has been used in prior studies, and it balances a time interval short enough to ensure stability of the measurement, while long enough for participants to forget their prior answers. ¹⁹ Participants only have a short time to assess their AFO in a clinical visit, as well as only using the device on indoor, flat terrain. It is certainly possible that a device that initially felt comfortable to the participant no longer felt as comfortable after wearing it for longer periods of time or when using it for different tasks than level ground walking.

Participants who had adjustments made to their AFO had a significant increase in their OCSs, compared with no change if no adjustments were made. This provides initial evidence of the OCS’s responsiveness to change. The mean magnitude of this change was small and less than the minimal detectable change calculated for the SCS in a prior study. ²² In our study, the sub-group numbers were relatively small (n = 11 for no adjustments, n = 35 for adjustments). As well, we did not collect information regarding the reasons for appointments or adjustments. Future research could determine the minimal detectable change of the OCS.

Administrative equivalence is important to establish for outcome measures like the OCS, where in clinical scenarios, patients will be reporting their scores to their treating clinician. This study demonstrates administrative equivalence between a written questionnaire format and verbal report to their orthotist, which is consistent with prior evidence for numerical rating scales. ²³

The OCS is quick for patients to complete, taking on average just over 1 minute. Our study design had limited sensitivity for this measure as time was measured in minutes, from the time participants collected the questionnaire to the time they returned it. This included the time to walk from the desk to their seat and vice versa. As well, if participants completed their questionnaire in less than 1 minute, this was rounded up to 1 minute for data analysis, which occurred only for the OCS. Therefore, the time to complete the OCS was likely overestimated. Future studies could address this by measuring time for completion in seconds. These results suggest that the OCS is practical to incorporate into clinical practice, considering that the burden of time is one of the most commonly cited limitations for incorporation of outcome measures into clinical practice. ²⁴

In summary, this study found initial evidence for convergent and predictive validity, administrative equivalence, responsiveness to change and test–retest reliability for the OCS in AFO users. This is consistent with prior literature showing the validity of single-item outcome measures, including for footwear comfort and pain assessment.^25–27 It should be noted that the OCS, like the SCS, only assesses comfort at one particular time. For some research and clinical purposes, it may be useful to develop a measure that attempts to capture a person’s average comfort. ²⁸ The OCS should be further testing using methodology and standard sizes consistent with established standards for psychometric testing of outcome measures, such as the COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN), prior to more strongly recommending its use. ²⁹ This study included only AFO users, as AFOs are the most commonly provided custom orthotic devices. Further research should also look at a broader group of orthotic devices to ensure similar psychometric properties of the OCS.

Conclusion

The OCS is a short, simple tool which has initial evidence for validity, reliability and responsiveness to change. Clinicians and researchers may consider using the OCS as an outcome measure for orthosis users.