Development of the RightWheel manual wheelchair wheel rolling resistance clinical decision support system through iterative clinician interviews

Abstract

Introduction

Manual wheelchair users (MWU) frequently experience upper limb (UL) injury and pain. Clinical practice guidelines (CPG) provide guidance on how to reduce risk of UL injury and pain but the recommendations provide only general advice such as to minimizing repetitive strain by reducing rolling resistance (RR). RR is due to energy loss between wheels and ground during wheelchair propulsion and is a major contributor of repetitive strain for MWUs. Motivated by the recommendation to reduce RR, we developed a clinical decision support system (CDSS) to provide client-specific RR predictions across several wheelchair setups to allow clinicians and users to make informed decisions.

Methods

An iterative user-centered design process (mixed methods) recruited ATP certified occupational or physical therapists to suggest modifications, assess usability and usefulness, identify client use cases, and provide rear wheel and caster selection criteria.

Results

Six clinicians participated and suggested over 100 modifications. Usability (SUS = 83.8; modified QUIS = 7.5) and perceived usefulness (TAQ = 4.7) were acceptable. Client use cases and rear wheel and caster selection criteria were identified. All clinicians thought it would be a useful tool.

Conclusions

RightWheel online CDSS provides user-customized RR estimates for equipment options in an easy-to-understand format, and was deemed ready for pilot launch.

Keywords

assistive technology rolling resistance propulsion clinical practice guidelines knowledge translation dissemination human factors mobility device

Introduction

Upper limb (UL) injury and pain are frequently experienced by manual wheelchair users (MWU).^1,2 Clinical practice guidelines (CPG) for preservation of UL function were developed to provide guidance to clinicians on how to reduce risk of UL injury and pain for MWU. The CPG’s³ recommend minimizing repetitive strain, but only general guidelines are available, such as minimizing rolling resistance (RR) by using pneumatic versus airless insert tires, positioning the axle as far forward as possible to distribute more weight on the rear wheels, and optimizing propulsion biomechanics through proper hand position relative to the axle and appropriate elbow angle relative to the push rim.

Importantly, RR is the primary source of repetitive strain for MWU, and is caused by energy dissipation when the wheels roll on the ground.⁴ RR varies depending on factors such as the wheel type and size, load, and the surface propelled upon.⁵ RR is often tested at a system-level, which makes it difficult to understand how the RR for each rear wheel or caster varies according to changes in factors like loading, ground surface, and tire pressure. With a system-level test, the weight on the rear wheels and caster can be measured, but separating their contribution to RR is more difficult. A component-level RR test was developed^5,6 and validated,⁷ which provides RR forces for individual wheels and factors. From this, individualized system-level RR predictions for a specific user weight can be estimated. These estimates, in turn, can be generated for several prospective wheelchair setups and thereby be used to help MWUs and clinicians select the equipment setup that can reduce RR and thereby UL risk while meeting other criterion. These prospective estimates could serve as a valuable clinical decision support system (CDSS) that could help optimize the wheelchair prescription process.

Precision and accuracy of component-level RR was assessed and validated in two studies, comparing with treadmill RR system-level tests⁷ and with SmartWheel system-level RR tests. Both studies demonstrated the precision and accuracy of component-level RR compared with system-level RR tests. Component-level RR predictions are somewhat lower than system-level tests,⁷ with −1.1 N offset observed versus treadmill, and excellent reliability based on intraclass correlation coefficient (ICC) of 0.94 with 95% confidence interval [0.91–0.95].⁷ Component level RR has also been used to quantify the effects of corrosion, shock and abrasion for casters.⁸

Compared to system-level tests, component-level tests cannot assess differences in user propulsion technique and the effect of speed on component-level RR force is minimal (comparing 0.5 and 1.0 m/s), whereas system-level tests do report differences related to speed.⁹ Surfaces must be able to be applied to the drum.

Decision support system benefits

CDSS provide patient specific recommendations via computer-based systems and are often integrated into the electronic medical record.¹⁰ CDSS provide many benefits, including (1) clinical management to support adherence to CPG,^10,11 (2) diagnostic support by providing suggestions for clinicians,¹² (3) communication and education with patients,¹³ (4) improved workflow,¹⁴ and (5) alerts for follow up appointments.¹⁵ Clinicians value CDSS when they are easy to use and implement into clinical care, especially if they save time during the clinical encounter and patients are more satisfied with care after using CDSS.¹⁶ CDSS also adds value by increasing patient and clinician knowledge, engaging patients in sharing preferences and in decision making and reducing decisional conflict.¹⁶ Barriers and facilitators research suggest that the CDSS should support clinician assessment and decision making, not replace it, because there are many comorbidities and other considerations that the tool cannot incorporate.¹⁷ Currently, pressure mapping is the only CDSS used by clinicians for wheelchair provision, which provides visual feedback on high pressure points and pressure distribution to assist with cushion selection to prevent pressure injuries.

RightWheel concept and considerations

We conceptualized a CDSS that would incorporate component-level RR data in a user-friendly format, developed with clinicians’ input on necessary revisions, usability, and perceived usefulness. The objective was to incorporate as many of the important benefits of CDSS as possible in RightWheel, and usability and perceived usefulness were identified as important outcome measures because how ‘easy to use and implement in clinic care’ are criteria which will determine if the CDSS will or will not be used by clinicians. To accomplish this, an iterative user-centered design process was planned, which incorporates human factors and usability methodologies and considerations into the design process.¹⁸ User-centered design is often used for development of computer-user interfaces to improve the user experience,¹⁸ and is a widely accepted approach to product design.

Our goals were to develop the prototype RightWheel CDSS, and then iterate the RightWheel design features until the tool had acceptable usability and perceived usefulness, and adds value to the work of clinicians/wheelchair providers based on if it would be a useful tool and their intention to use RightWheel. The hypothesis is that RightWheel has acceptable usability based on system usability scale (SUS) $\geq 68$ , modified questionnaire for user interface satisfaction (QUIS) $\geq 6$ , and perceived usefulness based on technology acceptance questionnaire (TAQ) $\geq 4$ .

While iterating the RightWheel design, we will identify use cases where RightWheel can provide value to clients and understand and outline the rear wheel and caster selection criteria.

Methods

A three stage (0→2) development process (mixed methods) was used to develop RightWheel. The objectives and software platform for each stage are summarized in Figure 1. The development steps are summarized in Table 1.

Figure 1.

Development objectives and software by stage.

Table 1.

Development steps by stage.

	Stage 0	Stage 1	Stage 2
Development steps:
Define design criteria	✓	✓	✓
Define algorithms and equations	✓
Develop prototype and verify accuracy	✓	✓	✓
Gather feedback		✓	✓

The Stage 0 objective is to develop a proof-of-concept RR calculator, and completion criteria is a working prototype. The Stage 1 objective is to develop an online CDSS integrating a graphical user interface and completion criteria is a working online prototype which can be demonstrated to stakeholders.

The Stage 2 objectives are to develop an expanded online CDSS that is intuitive, easy-to-use and provides actionable information for clinicians, using an iterative development process and recruiting clinicians to provide feedback on ‘must-have’ and ‘nice-to-have’ changes and preferred images. The think-aloud method¹⁹ was used during clinician interviews because it has been shown to assist in the development of a usable CDSS.²⁰

The completion criteria for Stage 2 are (1) all feasible ‘must-have’ and ’nice-to-have’ changes are implemented, (2) RightWheel has acceptable usability and perceived usefulness, (3) a list of clients who would benefit from RightWheel are identified (use cases), (4) decision criteria for rear wheel and caster selection are summarized, and (5) RightWheel is deemed ready for a pilot launch study.

Participant recruitment (stage 2)

Institutional Review Board review determined that the focus groups/interviews for this study (STUDY20060124) met regulatory requirements for exempt research. The inclusion criteria for participants are occupational therapists, physical therapists, or assistive technology engineers who evaluate at least ten manual wheelchair clients per year in a seating clinic, with an ATP certification. There were no exclusion criteria. Recruitment was planned until redundant feedback is obtained or ten clinicians have been interviewed. Clinicians were recruited from individuals who had expressed interest in participating in development of the tool during a RR state of the science webinar, and clinicians who are known to the research team.

Outcome measures (stage 2)

Two standard usability surveys (SUS²¹ and modified QUIS²²) were selected as outcome measures because they are often used in development of technology, are validated, reliable, and have an established threshold for acceptability. SUS assesses usability using ten general questions scored on a scale of 1 to 5 and converted to a 100 point scale, with overall average reported. QUIS evaluates satisfaction of the user-computer interface, with twenty seven questions grouped into five categories (overall reaction, screen, technology and system information, learning, and system capabilities), is scored on scale of 0 to 9, and reports the average score. The modified QUIS included eighteen questions, and removed questions which were not relevant to RightWheel. Both surveys have been widely used and have pre-determined scores representing ‘acceptable’ threshold. A technology acceptance questionnaire (TAQ) was developed to measure perceived usefulness, and incorporates the identified CDSS benefits aligned within the technology acceptance model (TAM) framework. TAM was developed to increase use of information technology and promote technology acceptance, and is based on the theory of reasoned action, a social-psychological behavior theory²³ that intention to accept technology is determined by attitude, perceived usefulness, and perceived ease of use (Figure 2).^23,24 TAM has proven validity and reliability, has been used widely and increasingly is used to assess healthcare and mobile health.^23–25 Variations to the TAM framework have also been used to assess technology acceptance.^23,24 The rationale for criteria of TAQ ≥ 4 is that an average response of “agree” (i.e., 4) on a scale of 1 (strongly disagree) to 5 (strongly agree) indicates an acceptable threshold of perceived usefulness, leading to behavioral intention to use RightWheel.

Figure 2.

Technology acceptance model.

Interviews and surveys (stage 2)

Participation required up to 2 hours in an online meeting (Zoom, Teams), and completion of a pre-meeting clinician characteristics survey and three post-meeting online surveys (SUS,²¹ modified QUIS,²² TAQ). The TAQ survey was added mid-study and completed by the last two participants. The meeting time was scheduled at the convenience of the clinician. The online meeting followed the sequence of topics outlined in Table 2. Clinicians were asked about the rear wheel and caster selection process (questions in Table 3). During the think-aloud portion of the interview, participants suggested ‘must-have’ and ‘nice-to-have’ changes while using the CDSS. Then a structured series of interview questions were asked, including (as needed) clarifications for suggestions provided during the think-aloud portion of the interview. Questions were skipped if participants had already provided feedback which answered those specific questions. The full interview guide is in Supplemental Tables 1, 2 and 3.

Table 2.

Participant interview sequence for RightWheel development.

Activity	Time (min)	Outcomes
0) Clinician characteristics questionnaire (pre-interview)	5	Clinician characteristics
1) Describe rear wheel and caster selection process	10	Selection criteria
2) Training and demonstration (PowerPoint, using CDSS)	10
3) Think aloud¹⁹: Participant remotely controls CDSS	30	Suggested changes
4) Right-Wheel interview guide: ‘must-have’, ‘nice-to-have’ changes, overall impression, use cases.	30	Suggested changes and use cases
5) Usability/Usefulness questionnaires (post-interview)	5	SUS, modified QUIS, TAQ

Table 3.

Rear wheels and casters selection criteria questionnaire.

1	Think about the last time you prescribed a manual wheelchair, briefly tell me about the process.
2	How did rear wheel and caster selection fit in the process? Tell me about the steps in the process.
3	How did you involve clients and suppliers in the process?
4	What are the important criteria for selection of rear wheels and casters?
5	What are challenges to rear wheel and caster selection?
6	What technology do you use to inform decisions?
7	Who are the stakeholders? How might they use a CDSS?
8	Do many of your clients have an extra set of wheels, and if they had a flat, they could change it?
9	Can most people use pneumatic tires?

Image preferences for percent difference and equivalent weight were assessed by displaying four options for each image type in RightWheel (via use of drop-down menus) and asking the clinician their preference. To determine the most promising use cases, clinicians were asked the profile of users who would and would not benefit from the tool. All feasible recommended design changes were made to RightWheel, and as needed, clinicians were asked to review and confirm that their recommendations were appropriately implemented into the revised prototype during a short (∼15 min) follow-up online meeting. All feasible recommended design changes were incorporated prior to the next clinician interview.

At the end of stage 2, multiple images had been refined or newly generated based on clinician suggestions, and resultant force had been added to the results. To obtain input from all clinicians on these image options, an online preferences survey was developed which included questions about image preferences, awareness of resultant force, and value of resultant force estimates.

A rear wheel and caster database with equipment which is meaningful to clinicians was developed, based on equipment previously evaluated in our lab,^5,7 frequency of equipment observed in prior field study,²⁶ and clinician interview feedback. The nine rear wheel tires included high pressure pneumatic, low pressure pneumatic, knobby pneumatic, solid polyurethane, and airless insert, with 24” diameter (7) or 25” diameter (2). All were mounted on standard spoked wheels with one exception (solid polyurethane on mag wheel). The fourteen casters include soft roll casters (3, 4, 5, 6 inch diameter × 1.5” width), polyurethane (4, 5, 6, 8 inch diameter × 0.75 to 1.0” width) rollerblade (3” diameter × 1” width), pneumatic (6 or 8” diameter × 1.25” to 2” width), and semi-pneumatic (8” diameter × 1.75” width). Two surfaces representative of typical commercial and home environments were selected, linoleum (Forbo) and low pile carpet with ¼” felt padding.

Data and statistical analysis

All surveys were conducted online using Qualtrics. Meetings were recorded and transcribed, with clinicians de-identified using a participant number, and the recording deleted approximately 1 month after the interview. Both the researcher and illustrator participated in all interviews. Responses to rear wheel and caster selection criteria were tabulated and analyzed by question to extract common themes. Clinician quotes were selected that highlight important challenges and common viewpoints. Think-aloud¹⁹ ‘must-have’ and ‘nice-to-have’ suggested changes were categorized in four areas: layout, content, look/style, and images, by participant, with suggestions compiled in a spreadsheet by participant and category, and completion of changes tracked. A summary of the recommended changes implemented are reported, and suggested changes not implemented are summarized with rationale. Clinician image preferences (Likert-scale ratings) were tabulated. Use cases were summarized into a list of clients who would benefit from RightWheel, and type of clients who are not appropriate.

Clinician demographic characteristics are tabulated and reported. Usability and perceived usefulness participant scores mean and standard deviation (SD) are reported, with acceptable thresholds of SUS ≥ 68, modified QUIS ≥ 6 and TAQ ≥ 4.^21,22

Results

Stage 0: define design criteria

The design criteria were to develop a CDSS that quantifies and informs how rear wheel and caster selection changes RR forces, customized for client-specific weight, weight distribution, equipment and surface. The calculator functions are defined by user inputs and outputs (shown in Table 4), where the user can select one baseline set of equipment and three alternative options for comparison. The equipment options and surfaces included are based on published RR data from Ott et al. (six rear wheels (high and low pressure pneumatic, knobby pneumatic, solid polyurethane on mag wheel, and airless insert) and six casters (4, 5 and 8” diameter soft roll, polyurethane, semi-pneumatic and pneumatic) on two surfaces).⁵

Table 4.

RightWheel user inputs and outputs.

User selected inputs	User selection options	Numerical outputs	Graphical outputs
• User weight	• Rear wheels	• Equivalent weight	• Equivalent weight
• Wheelchair weight	• Casters	• Percent difference
• Weight distribution	• Two surfaces

Stage 0: Define algorithms and equations

The calculations are broken into multiple steps: rear wheel and caster loads, RR force, percent difference, and additional equivalent weight. RR force calculations use the slope and intercept from linear regression of load versus RR force test data and calculates the RR force for the specific rear wheel and caster loads to calculate system-level RR. Equivalent weight is calculated using RR slope and intercept to estimate the additional weight needed on the reference rear wheel and caster to equal the RR of the compared equipment.²⁷ The output displays the increase or decrease in RR, equivalent weight and percent differences, depending on the equipment compared.

Stage 0: Develop prototype and verify accuracy

The stage 0 prototype selected the appropriate slope and intercept data for each wheel/surface combination using ‘IF (AND’ logic. ‘IF’ logic was also used to select equivalent weight images. The stage 0 prototype was functional but not aesthetically pleasing or user friendly (Figure 3). Accuracy of excel output were compared with hand calculations for multiple rear wheels and casters at various weights to ensure that there are no calculation errors. The outcome of Stage 0 was a working prototype that could be used to complete calculations and demonstrate functionality with stakeholders.

Figure 3.

Stage 0 RightWheel prototype.

Stage 1: Define design criteria

The stage 1 design criteria build on the Stage 0 prototype, with criteria expanded to include a graphical user interface and other criteria which are summarized in Table 5. The images are displayed based on an algorithm that assigns a specific image for a pre-determined range of results. Images previously used to represent equivalent weight (kettlebell on a scale) were selected from a published RR fact sheet²⁷ and the images representing percent difference were developed based on a cell phone signal strength concept.

Table 5.

Overview of design criteria by development stage.

	Stage 0	Stage 1	Stage 2
Objective	• Proof of concept	• Online CDSS	• Online CDSS refined using user-centered iterative design process.
Content (Function)	• Defined in Table 4.	• Same as Stage 0	• Add RR force and resultant force.
Content (Function)	• Compare baseline to three equipment options	• Same as Stage 0	• Rear wheel and caster pictures, descriptions, and tradeoffs.
Content (Equipment Choices)	• Equipment options are based on published RR data	• Same as Stage 0	• Expand equipment options to represent equipment available in specialized seating clinics.
Layout, Look & feel	• None	• Graphical user interface	• Intuitive, easy-to-use user interface.
		• Simple design and layout	• Clean look with whitespace.
		• Information is easy to understand.	• Utilize drop down menus, radio buttons and sliders.
		• Drop-down menus for rear wheel, caster and surface.	• Utilize drop down menus, radio buttons and sliders.
Images	• Simple kettlebell image representing equivalent weight	• Impactful images for equivalent weight and percent difference.	• Develop custom illustrations for equivalent weight, percent difference and resultant force.
Images	• Simple kettlebell image representing equivalent weight		• Modify and/or add illustrations based on clinician feedback.

Stage 1: Develop prototype and verify accuracy

The online CDSS was developed for an existing WordPress website, utilizing add-on software, with the final prototype shown in Figure 4. Accuracy of calculations was verified by comparing results with the stage 0 prototype. The add-on software had three challenges. First, the add-on software had limitations that made programming and troubleshooting difficult and was quickly identified as a significant weakness by the investigators; second, the software had limited options for user interface look and layout; and third, there were monthly software use fees.

Figure 4.

Stage 1 RightWheel prototype.

Stage 1: Gather feedback

The prototype was demonstrated at a state-of-the-science online webinar in November 2021, and a poll with 49 participants (comprised of 53% clinicians, 12% suppliers, 12% manufacturers), found 82% agreeing RightWheel would be useful in their context. Of these participants, 15 responded with interest in assisting in our development of this tool.

Stage 2: Define design criteria

The Stage 2 design criteria builds on the Stage 1 prototype, with additional design criteria outlined in Table 5. R Studio Shiny app was selected because it provides an aesthetically pleasing graphical user interface and the open-source software has free and low-cost monthly online hosting options at shinyapps.io. Shiny has extensive user tutorials, guides, and examples of working program code, making the development process easier.

Stage 2: Develop prototype and verify accuracy

The initial stage 2 prototype placed all the content on a single page. Accuracy was verified by comparing results with the stage 0 prototype. The illustrator created four concept image options for equivalent weight, consisting of a wheelchair user with either a backpack, trailer, on a ramp, or with wind blowing, to visualize the difference in force required to propel (Figure 5). The person in the wheelchair was a neutral character, not the focus of the image. Four different concept images representing percent difference were also created, based on a speedometer concept, with cool to hot colors indicating lower or higher differences (Figure 6). Initially only three percent difference gauge versions (representing minimum, zero, and maximum) and three to five equivalent weight versions (representing low to high) for each image type were developed.

Figure 5.

Initial equivalent weight image options.

Figure 6.

Percent difference image options.

Stage 2: Gather feedback

Recruit clinicians

Six clinicians were recruited from specialized seating clinics with high throughput of services, all of which were in the Midwest. The clinicians were recruited by email from the list of webinar participants who expressed interest in assisting in the CDSS development or clinicians known by our research team. Clinician demographic characteristics and equipment preferences are summarized in Table 6 and Figure 7 respectively. The clinicians recruited were experienced, and regularly or often prescribe custom manual wheelchairs, were involved in selecting tires and casters, and considered RR as part of the equipment selection process. Most clinicians preferred pneumatic tires (5 of 6), performance wheels (5 of 6), and all preferred soft roll casters (6 of 6). Half of clinicians said they used online tools to guide decision making, ranging from several times a week (1), once a week (1), once a month (2), or a few times per year (1). Most said the barrier to using online tools is not having enough time during patient appointment (4), not enough time for other reasons (1), not comfortable using online tools (1), HIPPA/security of information concerns (1), and difficult to have online tool pulled up while working with client (1). Examples of online tools used by clinicians includes University of Pittsburgh RR flyers and resources (2), manufacturer websites (1), RESNA (2), supplier websites (1), research articles (1), Paralyzed Veterans of America (1).

Table 6.

Clinician characteristics.

	N	(%)
Wheelchair seating and mobility experience
>10 years	3	(50)
5 to 10 years	2	(33)
2 to 5 years	1	(17)
up to 2 years	0	(0)
How often do you prescribe custom manual wheelchairs?
Regularly (5× or more per month)	5	(83)
Often (2–4× per month)	1	(17)
Infrequently (1× per month)	0	(0)
What percentage of chairs do you influence selection of tires or casters?
Every time	3	(50)
Most of the time	3	(50)
Sometimes	0	(0)
To what degree do you feel informed about wheel and caster selection on RR?
Very informed	4	(66)
Aware	2	(34)
Limited awareness	0	(0)
To what degree do you take RR into consideration when selecting RW or casters?
High degree	4	(66)
Moderate degree	2	(34)
Low degree	0	(0)
If RR is not usually considered, why?Time limitations (2), less important than other considerations, lower priority (especially when power assist used), lack of information on RR available to guide decision making, must consider insurance coverage.

Figure 7.

Clinician Rear Wheel and Caster Preferences *Other considerations for selecting tires and rims: goals for use, client factors such as grip strength, funding **What makes rear wheels and/or casters preferred? Reduced RR (2), lighter weight, better shock absorption, terrain management, durability, rigor.

Suggested changes

Suggested modifications to the CDSS are summarized in Table 7 by participant and type of suggestion, and all feasible recommended changes were completed before the next clinician interview. Over 100 suggested modifications were implemented. The suggested changes (Table 8) ranged from simple items (size or color of text, placement of text) to moderate changes such as modifying images to better convey information, to complex changes such as adding tabs to provide RR estimates for brief, expanded and surface comparisons, moving equipment images and tradeoffs to a separate tab, and adding a weight distribution calculation estimate. Three clinicians provided a second set of feedback in a brief (<20 min) online meeting to confirm if the implemented changes matched their suggestions and intent and to identify any additional needed changes (noted as P2.2, P3.2, and P5.2 in Table 7). Images developed for the CDSS were refined based on feedback, and after all feedback and preferences were obtained, a full range of the preferred images were developed. The Stage 2 final RightWheel prototype integrating all clinician feedback has six individual pages (or tabs), with the brief comparison page shown in Figure 8 and estimated weight distribution page shown in Figure 9. Rear wheels and casters were added to the database, for a total of nine rear wheel and fourteen caster options on linoleum and carpeted surfaces.

Table 7.

Number of suggestions by category and participant.

	LayoutHow info is displayed	ContentWhat is displayed	Look/feelOverall display	ImagesHow to modify
P1	4	7	1	1
P2	4	13	0	3
P2.2	11	1	0	0
P3	6	6	1	1
P3.2	0	2	0	0
P4	0	17	1	0
P5	1	7	3	6
P5.2	0	0	0	0
P6	0	2	0	3
Total	26	55	6	14

Table 8.

Summary of suggested changes implemented.

Content	Layout
• Add equipment descriptions	• Provide expanded and brief comparisons
• Add more casters to equipment options	• List equipment in a separate tab
• Add references, video links and resources	• Improve display of results text
• Use larger text for easier to read screen	• Simplify and remove extraneous info
• Define surfaces and include a picture	• Refine layout of weight distribution tab
• Change formatting, color of help text
• Add, refine weight distribution tab
• Screen reader compatible

Images	Look/Feel
• Use ramp image to illustrate reduced RR	• Add Pitt colors/logo
• Add reference images for baseline option	• Accessibility for color blind individuals
• Add line to emphasize axle position
• Add more downhill images
• Include a larger backpack image

Figure 8.

Stage 2 final RightWheel prototype - brief comparison tab.

Figure 9.

Stage 2 final RightWheel prototype - estimated weight distribution tab.

Two accessibility considerations are incorporated: (1) legibility and visibility, including for colorblind individuals, and (2) screen reader compatibility. For optimal legibility and visibility, a white background with black text and larger font size are incorporated throughout.²⁸ A limited amount of blue text is used for contrast. Results are conveyed with text and images, and the images have cues which are understandable even if colors cannot be discerned, for example, the red needle indicating percent difference and resultant force has a black outline, and equivalent weight images do not require color to understand the information conveyed. RightWheel is screen reader compatible, and was evaluated using standard accessibility Voice Over Mac software, which demonstrated the ability to read text, drop down menu options, results text, and alt text for results images.

Suggestions that were not implemented are listed in Table 9 with the rationale and potential to include in future revisions. Adding a third surface representing a concrete, sidewalk or an uneven surface was suggested by multiple clinicians. However, developing and validating a concrete/sidewalk surface for use on the component-level RR test equipment was outside the scope of this development project, but could be completed in the future. RightWheel was programmed to accommodate a third surface, so adding an additional surface only requires test data for that additional surface.

Table 9.

Suggested changes not implemented.

Category	Suggestion	Reason not implemented
Content	• Third surface (outdoor sidewalk, plush/heavy carpet, or uneven surfaces/thresholds)	• Programmed for 3rd surface; develop simulated outdoor surface (future)
	• Make equipment option tab printable/downloadable	• Use screenshot instead
	• Equipment choices selected in the brief comparison tab would carry over to the expanded tab	• Software issue (not possible)
Images	• Wheelbase/seat dump image changes with input	• Feasible but lack of time (future)

Image preferences

The clinician preferences for equivalent weight (Figure 5) and percent difference (Figure 6) are summarized in Figure 10. For equivalent weight, there was no single preferred image. The backpack and trailer were preferred by most clinicians, and wind and ramp preferred by some. Based on the lack of a single preferred option for equivalent weight, that each image resonated with different clinicians, and because all images could be retained using an image selection button, all four equivalent weight image options were included in the final version. For images representing percent difference, the full gauge version 1 was preferred by all clinicians and was selected for use in RightWheel. At the end of the study, the full range of selected images were developed (minimum to maximum for gauges, low to high for equivalent weight).

Figure 10.

Image preferences (interview).

The clinician preferences survey image options are shown in Figures 11 and 12, with results in Figure 13 and Table 10. There was a strong preference for the red trailer and backpack images (equivalent weight), angled slider (resultant force), and triangle (weight distribution), all of which were incorporated in final version and displayed in Figures 6 and 7. For images conveying reduced RR, there was no clear preference, so the image with no backpack or trailer was retained. Clinicians valued resultant force and supported including that information in RightWheel (Table 10).

Figure 11.

Image options for weight distribution (triangle, square, rectangle) and resultant force (angled or flat slider).

Figure 12.

Downhill reduced rolling resistance image options.

Figure 13.

Image preferences (survey).

Table 10.

Value of resultant force (RF) information.

(n = 6)	Strongly agree	Agree	Neither agree nor disagree	Dis-agree	Strongly disagree
I am aware of RF and its relationship to UL injury	4	2
I value having an estimate of RF for my client	2	3	1
Adding RF is confusing			1	4	1
I would prefer if you only provided RR and not include RF		1	1	2	2

Usability and perceived usefulness

Usability and perceived usefulness exceeded acceptable thresholds with results by participant and mean (SD) summarized in Table 11 (full results in Supplemental Tables 4–6). Clinicians agreed ‘I plan to use this tool’ (mean = 5), and ‘I would recommend RightWheel to other clinicians’ (mean = 4.5) with full details in Supplemental Table 6.

Table 11.

SUS usability and usefulness results by participant.

		P1	P2	P3	P4	P5	P6	mean	(SD)
SUS	$\geq 68$	80	87.5	77.5	75	82.5	100	83.8	(9.1)
m QUIS	$\geq 6$	7.1	7.7	7.6	6.6	7.3	9.0	7.5	(0.8)
TAQ	$\geq 4$	—	—	—	—	4.4	5.0	4.7	(0.5)

When asked ‘would this be a useful tool for you’, all the clinicians thought it would be useful, often stating specific types of client use cases where it would be most useful. Clinician quotes include:

• “Yeah, it’s just a different method vs the resources that I have that show the differences between the rear tires. This takes into effect a little bit more of the frame configuration along with the casters and how all those play together. It’s pretty cool.”

• “To me, it would be a useful tool, because number one, I didn’t know there was so much difference in the different options.”

• “I’m rather impressed… This, there’s a lot happening on the screen, but it all makes sense.”

• “I love it. I think it’s very useful. I think it’s practical. I think you can use it in letters of justification.”

Client use cases

The clients who would benefit from use of RightWheel include manual wheelchair users who are (1) active, (2) have shoulder concerns or (3) use airless insert or big knobby tires. These may be individuals who are (4) open to optimizing, (5) interested in their care, (6) set in their ways, or (7) newer curious users. Clients that are thought not appropriate for RightWheel include (1) clients who are pushed who have either cognitive limitations or who are frail and elderly; (2) clients who are overwhelmed, including first time wheelchair users. The rationale for not using RightWheel with first time wheelchair users was that it is overwhelming due to the large number of decisions and because every decision and choice is new. Not all clinicians agreed with this, but many did agree that first time wheelchair users are often very overwhelmed.

Rear wheel and caster selection criteria

For rear wheels, the common theme was that clinicians prefer pneumatic tires, and the most important determinant is if the client can maintain air pressure (on their own or through family or other support). A second factor was the intended use/distance propelled related to how important reducing RR is for the client. A full list is shown in Table 12.

Table 12.

Rear wheel and caster selection criteria.

Rear wheel type	Caster type/size
• Pneumatic versus airless insert	• Style and type of materials
• Ability to maintain air pressure	• Maneuverability
• Client, family or other support	• Environment
• Hand function/grasp	• Terrain management
• Distances propelled /intended use	• Threshold size, cracks
• Anticipated terrain	• Concern: getting stuck
• Client weight	• Caster diameter/width
• Propulsion efficiency	• Wheelchair skills
• Long term effects	• Wheelie ability for obstacles
• Ride comfort	• Ride quality (vibration/shock damping)
• Vibration	• Durability
• Durability/concern about flats	• Power add-on options

For casters, all prefer soft roll casters, and the primary considerations are terrain (in terms of threshold size and cracks) and concerns about getting stuck, as well as client wheelchair skills and the ability to do wheelies, vibration, durability, and use of power add-on equipment. During this process, clinicians educate clients on options and discuss benefits and challenges related to equipment choices.

Wheelchair seat to floor height and seat slope are important for independent transfers and are impacted by rear wheel and caster diameter selection. Wheelchair setup for foot propelling, which is more common for geriatric clients, is also affected by rear wheel and caster diameter. Results for other interview questions are summarized in Table 13.

Table 13.

Summary of clinician responses for rear wheel and caster selection criteria.

What are the important criteria for selection of rear wheels and casters? Summarized in Table 12

• “Maintenance piece is the most challenging part, because that is really what’s going to hold people back from being able to use those tires.”

• “The only challenge I really have is convincing the client that pneumatic is going to be a better option than solid tires. Sometimes I might convince them that, hey, we’ll get them (i.e., pneumatic tires) in the beginning and if you hate it after a couple months, we can switch it back to something else.”

• For existing wheelchair users “some of them are very set in their ways and their chair, and how it’s configured, and aren’t open to different wheels, different casters.”

Who are the stakeholders? The client is a key stakeholder because they will be using the wheelchair, and a family member if they are putting it in/out of the vehicle and maintaining it. The clinician guides the process and is responsible for the final wheelchair fitting. The supplier ensures that the custom wheelchair can be assembled and provided. For the Veterans Administration, the manufacturer is consulted instead of a supplier. Other stakeholders include the payor (often an insurance company) and wheelchair designers.

Standard equipment versus upcharges? There are challenges to paying upcharges for equipment.

• “Some insurance companies don’t allow the patient to pay the difference.”

• “Suppliers are resistant to adding any upgrades to the chair, driven by cost.”

Do clients have extra wheels and tires? Active MWU’s often have spares from a prior chair, whereas new users do not.

• “Highly active or experienced MWU’s usually have spares (from a previous chair or purchased) or from Office of Vocational Rehabilitation (OVR) funding if needed for work, otherwise no spares.”

• “For a new user, very few have another set, but for the more experienced, more active users, many of them do.”

Can most clients use pneumatics? No, it depends on their functional capabilities and their willingness to do the maintenance. Some are very comfortable maintaining air pressure, but others have no idea how to add air or maintain tires.

• “It’s not just the ability to fill, it is being mindful in being consistent with monitoring - that is the biggest one for the pneumatic tires. I don’t really have that many people complain of getting flats that often.”

• “People are definitely very concerned about what happens if they do have a flat tire.”

What technology is used to inform decisions? There are limited technologies used in the clinic: pressure mapping for seat cushions, hands on demos of equipment, manufacturer websites, computer aided design/computer aided manufacturing (CAD/CAM) drawings to look at chair setup, and scales to measure weight on front and rear wheels.

Discussion

RightWheel development objectives were met. The CDSS was refined through an iterative development process, and the hypothesis that RightWheel would have acceptable usability and perceived usefulness was confirmed, along with clinician intention to use the tool in the future. Clinicians agreed the intended benefits are provided, and thought RightWheel would be a useful tool. The use cases where clients would benefit from its use were identified, and the rear wheel and caster selection criteria were compiled and summarized. RightWheel²⁹ is deemed ready to proceed to a pilot launch.

The iterative development process used to refine RightWheel incorporated >100 suggested modifications. The clinician’s recommendations were insightful and helped improve RightWheel to meet the objective of being intuitive and easy to use, demonstrating the value of a user-centered design process. Clinician image preferences had common themes for some but not for all images. Clinicians suggested ways to modify and improve images to make them more intuitive and easy-to-understand, changes that the development team would not have identified without their input. Although the iterative development process took time to recruit, interview and integrate feedback before the next interview, RightWheel was significantly improved by following this process. The clinicians who participated in this study were experienced and very aware of the importance of minimizing RR, which could be a source of bias. The lack of anonymity for surveys and interviews could have caused participants to be more positive in their feedback.

Usability and usefulness

Clinician usability scores may reflect perception differences of the questionnaire scale. Participant 4 had lower scores for both surveys, which may reflect more critical grading than other participants. Each clinician evaluated a different revised version of RightWheel, since all prior suggested changes were incorporated before the next clinician assessment. The number of suggestions and type of suggestions varied by clinician, with some providing more suggestions on content, and others more on layout and/or improving images, reflecting each clinician’s own preferences and expertise. The last clinician had fewer suggestions for improvement, indicating a plateau had been reached. Because the TAQ questionnaire was added mid-study, only two clinicians completed it. The two scores reflect agreement in perceived usefulness of RightWheel, (with 4 = agree, 5 = strongly agree), and overall average of 4.7, and responses to ‘I plan to use this tool' (average = 4.5) and ‘I would recommend to other clinicians’ (average = 5).

Use cases

Seven use cases types/descriptions were identified where RightWheel would be useful to the client. The rationale for ‘individuals set in their ways’ is that the visualization could help the client evaluate information differently and potentially make a different decision. There were also examples of cases where the tool would not be useful. For first time wheelchair users, some thought they would be an ideal case, by providing clear information for decision making and avoiding poor equipment selection that may be difficult to change in the future. Others thought there is so much going on, so many decisions, that there is no extra time available during that type of assessment appointment “for new wheelchair users, its overwhelming the number of decisions and choices.” One clinician stated they use two evaluation visits to address the situation of being overwhelmed, because it is so important to ensure they get the appropriate equipment from the beginning.

RightWheel was developed to provide clinicians an improved way to communicate the impact of RR on equipment choices to their clients. Other potential clinician uses of RightWheel includes training clinician-students, educating novice clinicians and ATP’s, and supporting clinicians in daily practice to communicate and educate clients. One clinician suggested they could use screenshots in their letter of medical necessity.

Rear wheel and caster selection criteria

The responses provided insight into how clinicians think through rear wheel and caster selection for their clients. There were a few key considerations (1) can the client maintain the air pressure in the tire; (2) what is the anticipated use, typical distance and terrain, and what equipment would be best for the client; and (3) how do caster and rear wheel size effect the overall wheelchair setup (i.e., floor to seat height, independent transfers and/or foot propelling). Challenges preventing use of pneumatic tires could also be viewed as product development opportunities. Can pneumatic tire air refilling be simplified or automated? Can the requirements for dexterity to attach the pump to the valve be reduced through improved design? Can lower RR tires be developed that do not require air? Is run-flat technology feasible for wheelchair tires? It would be helpful to understand the perceived versus actual risk of a flat tire, and how often flats occur for MWU’s.

Each wheelchair order form identifies standard versus upcharge items, but clinicians had commented “some insurance companies don’t allow the patient to pay the difference” and “suppliers are resistant to adding any upgrades to the chair, driven by cost.” If the optimal rear wheel or caster for the user requires upcharges, this could be a barrier to access. Soft roll casters often require an upcharge, which could limit their use. For example, soft roll casters provide benefits of somewhat lower RR, reduced vibration³⁰ and the increased width which may help avoid getting stuck for some terrain. For rear wheels, the standard option is often a 1 3/8” wide pneumatic or airless insert tire, with most others requiring an upcharge. Ott et al. measured 200 manual wheelchairs, primarily at adapted sporting events and found 54% high pressure pneumatic tires (>689 kPa), 18% low pressure pneumatic tires (<689 kPa), and 18.5% solid or airless insert tires, with two tire manufacturers representing the majority (39.5 and 35%) of all tires observed.²⁶ For these MWU’s, 61.5% had performance wheels and 28% had Lite Spoke wheels.²⁶ Both the tires and wheel types indicate, at least in this group of MWU’s, that upgrades to custom manual wheelchair equipment were common.

Equipment preferences

Clinician equipment preferences were aligned with low RR rear wheels and casters, with most preferring pneumatic tires and all preferring soft roll casters, which corresponds with published RR results. Comparing identical diameters, pneumatic tires have significantly lower RR compared to airless insert tires,^5,31 and soft roll casters have somewhat lower RR than polyurethane and provide some vibration reduction.³⁰

For clinicians who want to explain to clients about RR forces and equipment choices, RightWheel provides a system-level RR estimate for each client and illustrates differences between options. For example, all else being equal, larger diameter wheels have lower RR^4,32,33 and RightWheel illustrates this for both rear wheels and casters, for example when comparing 24 and 25” diameter rear wheels with identical tires. Wheel diameter is an important aspect of rear wheel and caster selection, which affects wheelchair setup including seat to floor height, ability for independent transfers and foot propelling.

RightWheel was developed to assist clinicians with appropriate provision by providing user-customized RR and quantifying differences between rear wheel and caster options, and is the first online tool available to clinicians to quantify repetitive strain from RR for individual MWUs.

Conclusion

Using an iterative, user-centered design process, we successfully developed and refined the CDSS RightWheel, and met all development goals, with acceptable usability and perceived usefulness. All clinicians thought that RightWheel would be a useful tool. RightWheel provides user-specific RR in an easy-to-understand format, quantifies differences between equipment options, and is ready for a pilot launch study. There are very few online tools used to assist in the wheelchair provision process, and RightWheel would be a unique tool in the clinic.

Clinicians described the rear wheel and caster selection criteria and challenges. Clinicians preferred pneumatic tires and soft roll casters, both of which provide low RR compared with other options. Some MWU’s who might benefit from pneumatic tires are not able to maintain air pressure either due to physical limitations and/or lack of support. For these clients, if changing the rear wheel to a pneumatic is not possible, modifying the caster can also reduce RR.

Many manual wheelchair users could benefit from use of RightWheel: clients who are active, have shoulder concerns, or use airless inserts or big knobby tires, and are either open to optimizing, interested in their care, set in their ways or newer curious users.

Limitations

A limited number of clinicians were interviewed, and most were highly experienced. Some of the clinicians were known by the researcher and/or our research group. If RightWheel is used for less experienced clinicians or for ATP training, it may require additional assessment of usability for these stakeholders. Other important stakeholders’ (MWUs, suppliers, and manufacturers) preferences and opinions were not surveyed but will be assessed in a future study. The selection of rear wheels and casters tested could be expanded, and an additional surface, such as a simulated outdoor surface, could be included. The weight distribution estimate function (developed at the request of a clinician) utilizes average anthropometric measurements for a limited number of client height and body mass index combinations to estimate client center of gravity, and has limited accuracy when comparing to actual clients. Understanding how this feature would be used by clinicians is needed to determine the necessary changes to improve accuracy.

Future work

A launch study is planned to follow this development phase, and will have clinicians use RightWheel, assess usability and usefulness over time, provide feedback on value and recommended changes, and confirm the type of users (use cases) who most benefit from RightWheel and obtain further feedback on most impactful images representing reduced RR. An evaluation by individuals who are colorblind or use screen readers could provide feedback and suggestions to improve accessibility. Increased RR from simulated long term use could be incorporated in future versions of RightWheel. A prior study measured increased RR for some casters after environmental exposure (corrosion, shock, and abrasion), which highlighted the importance of caster maintenance and appropriate replacement.⁸ The effect of wheel type on RR should be evaluated and considered for inclusion in RightWheel.

Supplemental Material

Supplemental Material - Development of RightWheel manual wheelchair wheel rolling resistance clinical decision support system through iterative clinician interviews

Supplemental Material for Development of RightWheel manual wheelchair wheel rolling resistance clinical decision support system through iterative clinician interviews by Holly Wilson-Jene, Rachel Cowan, Zachary Roy and Jon Pearlman in Journal of Rehabilitation and Assistive Technologies Engineering

Footnotes

Acknowledgements

We would like to thank the International Society of Wheelchair Professionals (ISWP) Standards Working Group for their continued support of this research, and Sofia Main for developing the stage 1 online prototype.

ORCID iDs

Holly Wilson-Jene

Rachel E Cowan

Jonathan Pearlman

Statements and declarations

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Improving Health and Function Through Use of Performance Standards in Wheelchair Selection Grant # 90REGE0001-02-00, U.S. Agency for International Development through Agreement Nos. APC-GM-00068, SPANS-037, APC-GM 0107, and FY19-A01-6024.

Conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Contributorship

JP, REC and HWJ conceived of the manuscript. HWJ developed and revised CDSS prototypes (stage 0, stage 2). ZR developed and revised custom illustrations for the CDSS. HWJ and ZR conducted the clinician interviews. HWJ wrote the first draft of the manuscript. All authors reviewed and edited the manuscript and approved the final version of the manuscript.

Supplemental Material

Supplemental material for this article is available online.

References

Sie

Waters

Adkins

, et al. Upper extremity pain in the post-rehabilitation spinal cord injured patient. Arch Phys Med Rehabil 1992; 73(1): 44–48.

Curtis

Drysdale

Lanza

, et al. Shoulder pain in wheelchair users with tetraplegia and paraplegia. Arch Phys Med Rehabil 1999; 80: 453–457.

Paralyzed Veterans of America Consortium for Spinal Cord Medicine . Preservation of upper limb function following spinal cord injury: a clinical practice guideline for health-care professionals. J Spinal Cord Med 2005; 28: 434–470.

Kauzlarich

Thacker

. Wheelchair tire rolling resistance and fatigue. J Rehabil Res Dev 1985; 22: 25–41.

Ott

Wilson-Jene

Koontz

, et al. Evaluation of rolling resistance in manual wheelchair wheels and casters using drum-based testing. Disabil Rehabil Assist Technol 2022; 17: 1–12.

Ott

Marcus

Henderson

, et al. Development and calibration of drum-based rolling resistance testing machine for manual wheelchair components. Technol Disabil 2021; 33: 123–136.

Wilson-Jene

Cowan

Pearlman

. A novel approach to directly measuring wheel and caster rolling resistance accurately predicts user-wheelchair system-level rolling resistance. J Rehabil Assist Technol Eng 2023; 10: 20556683231180877.

Wilson-Jene

Mhatre

Ott

, et al. Rolling resistance of casters increases significantly after two years of simulated use. J Rehabil Assist Technol Eng 2021; 8: 20556683211025149.

Soleymani

Cowan

. Relationship between rolling resistance, preferred speed, and manual wheelchair propulsion mechanics in non-disabled adults. Disabil Rehabil Assist Technol 2024; 19: 1980–1991.

10.

Sutton

Pincock

Baumgart

, et al. An overview of clinical decision support systems: benefits, risks and strategies for success. npj Digit Med 2020; 3: 17.

11.

Kwok

Dinh

, et al. Improving adherence to asthma clinical guidelines and discharge documentation from emergency departments: implementation of a dynamic and integrated electronic decision support system. Emerg Med Australas 2009; 21: 31–37.

12.

Friedman

Elstein

Wolf

, et al. Enhancement of clinicians’ diagnostic reasoning by computer-based consultation: a multisite study of 2 systems. JAMA 1999; 282: 1851–1856.

13.

Anderson

Godwin

Saleem

, et al. Accessibility, usability, and usefulness of a web-based clinical decision support tool to enhance provider-patient communication around Self-management TO Prevent (STOP) stroke. Health Inf J 2014; 20: 261–274.

14.

Flanagan

Arbuckle

Saleem

, et al. Development of a workflow integration survey (WIS) for implementing computerized clinical decision support. AMIA Annu Symp Proc 2011; 2011: 427–434.

15.

Shaikh

Berrong

Nettiksimmons

, et al. Impact of electronic health record clinical decision support on the management of pediatric obesity. Am J Med Qual 2015; 30: 72–80.

16.

Dobler

Sanchez

Gionfriddo

, et al. Impact of decision aids used during clinical encounters on clinician outcomes and consultation length: a systematic review. BMJ Qual Saf 2019; 28: 499–510.

17.

Agency for Healthcare Research and Quality . Challenges and barriers to clinical decision support (CDS) design and implementation experienced in the agency for healthcare research and quality CDS demonstrations, 2010.

18.

Loudenmilk

. User-centered design. O’Reiley Media, 2013. https://learning.oreilly.com/

19.

Jääskeläinen

. Think-aloud protocol. Handbook of translation studies 1. John Benjamins Publishing, 2010, pp. 371–374.

20.

Kilsdonk

Peute

Riezebos

, et al. Uncovering healthcare practitioners’ information processing using the think-aloud method: from paper-based guideline to clinical decision support system. Int J Med Inf 2016; 86: 10–19.

21.

U.S. Dept. of Health and Human Services. Research-based web design & usability guidelines: System Usability Scale (SUS). https://www.usability.gov/how-to-and-tools/methods/system-usability-scale.html (2006, accessed 23 May 2022).

22.

Chin

Diehl

Norman

. Development of an instrument measuring user satisfaction of the human-computer interface. In: Association for Computing Machinery (ACM) conference proceeding. Washington, DC, 1988, pp. 213–218.

23.

Holden

Karsh

. The technology acceptance model: it’s past and it’s future in health care. J Biomed Inf 2010; 43: 159–172.

24.

Yarbrough

Smith

. Technology acceptance among physicians: a new take on TAM. Med Care Res Rev 2007; 64: 650–672.

25.

Nadal

Sas

Doherty

. Technology acceptance in mobile health: scoping review of definitions, models and measurement. J Med Internet Res 2020; 22: e17256.

26.

Ott

Henderson

Wilson-Jene

, et al. A high prevalence of manual wheelchair rear-wheel misalignment could be leading to increased risk of repetitive strain injuries. Disabil Rehabil Assist Technol 2023; 18: 544–552.

27.

University of Pittsburgh . Rolling resistance fact sheet. https://www.wheelchairstandards.pitt.edu/fact-sheets (2020, accessed 13 June 2024).

28.

Handbook of human factors in medical device design (chapter 2, page 35). CRC Press Taylor & Francis Group, 2010.

29.

University of Pittsburgh . RightWheel. https://pittwheelchairstandards.shinyapps.io/RightWheel/, 2024.

30.

Misch

Liu

Sprigle

. Effect of wheels, casters and forks on vibration attenuation and propulsion cost of manual wheelchairs. IEEE Trans Neural Syst Rehabil Eng 2022; 30: 2661–2670.

31.

Kwarciak

Yarossi

Ramanujam

, et al. Evaluation of wheelchair tire rolling resistance using dynamometer-based coast-down tests. J Rehabil Res Dev 2009; 46: 931–938.

32.

Zepeda

Chan

Sawatzky

. The effect of caster wheel diameter and mass distribution on drag forces in manual wheelchairs. J Rehabil Res Dev 2016; 53: 893–900.

33.

Sauret

Bascou

de Saint Remy

, et al. Assessment of field rolling resistance of manual wheelchairs. J Rehabil Res Dev 2012; 49: 63–74.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.14 MB