How Long Does It Take for Patients to Complete PROMIS Scores? An Assessment of PROMIS CAT Questionnaires Administered at an Ambulatory Sports Medicine Clinic

Abstract

Background:

Challenges exist in routinely collecting patient-reported outcomes (PROs) from patients at a busy ambulatory clinic. A number of validated Patient-Reported Outcomes Measurement Information System (PROMIS) subdomains allow for efficient PRO administration.

Purpose:

To determine the time to completion (TTC) of 3 PROMIS computer adaptive test (CAT) scores. CAT questionnaires were administered at the ambulatory clinic with the following PROMIS subdomains: Pain Interference (PI), Depression, and Physical Function for lower extremity (PF) or for upper extremity (UE). The secondary purpose was to determine the influence of patient demographic factors on TTC.

Study Design:

Cross-sectional study; Level of evidence, 3.

Methods:

Patients were recruited from 3 fellowship-trained upper extremity and sports medicine orthopaedic surgery clinics. PROMIS CAT questionnaires were administered to consecutive patients during the study period (July 2017–September 2017). The start and completion times of each CAT were recorded. The primary outcome of interest was TTC of the questionnaires. Patients were stratified into age quartiles to determine the impact of age on TTC. Patient demographic information, such as sex, race, and ethnicity, was determined retroactively.

Results:

A total of 1178 questionnaire sets consisting of 3658 individual PROMIS forms were analyzed. The mean TTC was 3.29 minutes for all 4 forms in aggregate, with PROMIS PI, PF, UE, and Depression taking on average 1.05, 0.74, 0.96, and 0.57 minutes to complete, respectively. Patients from the oldest age quartile (mean ± SD, 70.3 ± 7.5 years) had a statistically significant longer TTC as compared with the second quartile (41.2 ± 4.7 years) (3.70 vs 2.87 minutes; P < .05). Asian patients had the longest PROMIS PF TTC, while white patients completed PF with the shortest TTC (1.28 vs 0.68 minutes; P < .05). Patients of unstated ethnicity had a longer TTC for PF as compared with their Hispanic/Latino and non-Hispanic/Latino counterparts (0.91 vs 0.30 and 0.70 minutes; P < .05).

Conclusion:

PROMIS CAT forms are efficient tools for collecting patient-reported outcomes in the ambulatory orthopaedic surgery clinic. Older patients, Asian patients, and patients of unstated ethnicity took longer to complete the forms.

Keywords

patient-reported outcomes PROMIS time to completion computer adaptive testing

There has been an increased emphasis on value-based care as clinicians, insurers, and government officials have been acknowledging the value of patient-reported outcomes (PROs), especially as they pertain to clinical outcomes and patient satisfaction. This recent emphasis has led to the development of many PROs for different types of pathology. PROs serve as a measure of care by measuring physical, mental, and social function. These scores can enhance patient-clinician communication, and they have the potential to improve the care provided to patients.¹ Although the use of PROs in orthopaedics has increased in recent years, questions remain regarding the logistics and utilization within clinical practice.

The variability of outcomes reporting has been well documented in several orthopaedic conditions.^7,8,17 In 2004, the National Institutes of Health in the United States put forth the Patient-Reported Outcomes Measurement Information System (PROMIS; US Department of Health and Human Services) to standardize PROs for a variety of medical conditions. PROMIS utilizes a larger database with validated items to output an accurate, reproducible, and validated score.^2,4 Additionally, given the multiple available formats, such as a computer-adaptive test (CAT) and short form of different question lengths, PROMIS forms were found to be more efficient than comparable “legacy” PROs.^13
–15 Multiple studies have demonstrated decreased time required for completion of PROMIS forms as compared with legacy scores.^3,11,13

Some PROMIS domains are clinically relevant to the field of orthopaedic surgery and sports medicine. However, the feasibility of incorporating comprehensive PROMIS scores is still not known. Successful adoption of routine PROMIS collection into the sports medicine clinic is contingent on timely administration of the forms. Therefore, the purpose of this study was to elucidate the time to completion (TTC) of the commonly used subdomains of PROMIS questionnaires—Pain Interference (PI), Physical Function -lower extremity (PF) or -upper extremity (UE), and Depression—to determine the influence of demographic factors on TTC.

Methods

Following institutional board review approval, patients examined by 3 fellowship-trained shoulder and elbow or sports medicine orthopaedic surgeons from a single health system (S.M., E.C.M., V.M.) were recruited for the study. Surveys were administered to patients presenting with a shoulder, elbow, hip, or knee complaint and the ability to use a tablet computer (iPad tablet; Apple). Patients were excluded if they were unable to read or write English or if they declined to participate. Patients unable to finish the survey in the waiting room were excluded to ensure that TTC was not falsely prolonged as the patient was called back to the clinic room or interacted with clinic staff and providers. Patients with an inadequate capturing of data (eg, TTCs of zero minutes to multiple hours) were assumed to have technical issues with the software system. TTCs >3 standard deviations from the mean or of zero minutes were subsequently considered outliers and excluded from analysis.

Participation in the study was voluntary. The survey was administered on a tablet with electronic data capture software (REDCap).⁶ The survey included a questionnaire regarding the general nature of the patient’s visit (eg, the location of pain and name of provider), followed by a CAT set consisting of the following PROMIS domains: UE for upper extremity function or PF for lower extremity function, PI, and Depression. All participants were surveyed in this order, with standard PROMIS adult forms.

Demographic information, including sex, age, and race, was retrospectively collected from the patient’s electronic medical record. The start and completion times of each survey were electronically recorded. All study data were collected and managed with REDCap, a secure web-based application designed to support data capture.⁶

Statistical Analysis

Continuous data were compared among groups with 1-way analyses of variance and independent t tests. P ≤ .05 denoted a statistically significant difference. Pearson correlations were performed to investigate the relationship between PROMIS TTC and patient age. Correlation coefficients were interpreted on the basis of absolute values, with 0.00 to 0.30 representing a negligible correlation, 0.31 to 0.50 a weak correlation, 0.51 to 0.70 a moderate correlation, 0.71 to 0.90 a strong correlation, and 0.91 to 1.00 a very strong correlation. All analyses were performed with Stata (v 14; StataCorp).

Results

A total of 1302 CAT questionnaire sets (PF/UE, PI, and Depression) were collected from 915 unique patients; 1178 sets consisting of 3658 individual PROMIS CAT questionnaires from 841 patients were included in the analysis (Figure 1 and Table 1). The mean age of the patients surveyed was 47.0 years; 618 were male (52.5%) and 560 were female (47.5%). The majority of patients were white (65%) and identified as non-Hispanic or Latino (82.5%).

Figure 1.

Questionnaire sets analyzed. TTC, time to completion.

TABLE 1

Patient Demographic Characteristics (N = 1178) ^a

Variable	Patients, n (%)
Age, y ^b	47.0 ± 18.7 (12-100)
Sex
Male	618 (52.5)
Female	560 (47.5)
Race
White	766 (65.0)
Black	228 (19.4)
Asian	52 (4.4)
Other/unknown	132 (11.2)
Ethnicity
Hispanic or Latino	21 (1.8)
Not Hispanic or Latino	972 (82.5)
Unknown	185 (15.7)

^a Data are based on 1178 questionnaire sets from 841 unique patients.

^b Mean ± SD (range).

The mean total questionnaire TTC was 3.29 minutes, with PROMIS PI, PF, UE, and Depression taking 1.05, 0.74, 0.96, and 0.57 minutes to complete, respectively (Table 2). Among patients who made multiple visits during the study period, TTC did not shorten with subsequent visits (P = .75). For PROMIS Depression, age had a negligible negative correlation with TTC (–0.05, P = .05). When TTC was analyzed as a function of age divided into quartiles (12-32, 32-48, 48-61, and 61-100 years), the second quartile had a statistically significant shorter mean TTC as compared with the fourth quartile (2.87 vs 3.70 minutes, P < .05) (Figure 2).

TABLE 2

Time to Completion ^a

	PROMIS Time to Completion, min
	Total	PI	PF	UE	Depression
Mean	3.29	1.05	0.74	0.96	0.57
Range	1-44	0-130	0-12	0-22	0-39
Sex
Male	3.42	0.94	0.97	0.87	0.54
Female	3.10	0.77	0.69	0.93	0.53
Race
White	3.15	0.82	0.68 ^b	0.91	0.51
Black	3.59	0.87	0.82 ^b	0.87	0.77
Asian	3.03	0.61	1.28 ^b	1.00	0.26
Other/unknown	3.33	1.00	0.69 ^b	0.56	0.73
Ethnicity
Hispanic/Latino	4.13	1.19	0.30 ^b	0.67	1.31
Non-Hispanic/Latino	3.24	0.86	0.70 ^b	0.92	0.53
Unknown	3.32	0.84	0.91 ^b	0.82	0.56
Age ^c	–0.04	–0.01	–0.07	–0.03	–0.05 ^b

^a Values are presented as means unless noted otherwise. PF, Physical Function for lower extremity; PI, Pain Interference; PROMIS, Patient-Reported Outcomes Measurement Information System; UE, Physical Function for upper extremity.

^b Denotes times to completion that are involved in a statistically significant outcome (P ≤ .05).

^c Values are expressed as correlation coefficients.

Figure 2.

Total time to completion by age quartile (mean ± SD age): first (21.7 ± 4.8 years), second (41.2 ± 4.7 years), third (54.9 ± 3.6 years), and fourth (70.3 ± 7.5 years).

For the PROMIS PF subdomain, white patients had the shortest TTC, while Asian patients had the longest (0.68 vs 1.28 minutes, P < .05). Those of unknown ethnicity had the longest TTC for PF (0.91 minutes), while non-Hispanic/Latino patients took longer to complete this questionnaire as compared with Hispanic/Latino patients (0.70 vs 0.30 minutes, P < .05). Sex did not have a significant impact on total TTC or any questionnaire item (P > .05).

In sum, patients presented with 23 unique diagnoses, with no significant differences in TTC. No significant difference in TTC was found when TTC was stratified by anatomic location, reason of visit, or different clinical encounters by the same patient (P > .05) (Figure 3).

Figure 3.

Total time to completion by anatomic location of reason for visit.

Discussion

In the busy ambulatory orthopaedic surgery setting, careful consideration must be taken to prevent an interruption in clinic flow. Our findings suggest that the total TTC for 3 PROMIS CAT scores was approximately 3 minutes. Each component score—PI, PF, UE, and Depression—was completed in a similar amount of time. Patient age, race, and ethnicity appeared to have an effect on TTC, while sex had no effect.

Our study found that PROMIS PF took 44 seconds to finish, while PI took 63 seconds to finish. Paulino Pereira and colleagues¹⁵ compared multiple questionnaires administered on a tablet device, including PROMIS, to measure quality of life, physical function, and pain among patients with metastatic spine disease. They found that PF and PI were the most reliable and were completed in a median of 42 (interquartile range [IQR], 35-60 seconds) and 24 seconds (IQR, 18-34 seconds), respectively. Stapleton and Degitz¹⁶ similarly found that PI took a mean 45 ± 37 seconds to complete by patients discharged from the emergency room who presented with the chief complaint of pain. The patients in our study were similarly able to complete the PROMIS domains on average in <1 minute, with only the PI score taking >1 minute to complete (1.05 minutes) but still falling within 1 SD of reports in the literature. Kortlever et al¹¹ reported that PI was completed in 30 seconds (IQR, 24-44) by patients with both atraumatic and traumatic upper extremity injuries. In our study, all 3 PROMIS CAT forms were cumulatively completed in <4 minutes, which is consistent with reports of PROMIS TTC in the literature.^{3,11,13,15,19} To date, no study has reported the TTC of multiple PROMIS CAT sets, which can present with logistical difficulties not inherent to a solitary questionnaire, namely with the software interface.

We hypothesized that as patients age, they would have increasing TTC attributed to decreased comfort with tablet computer technology as compared with younger generations.⁵ Our data confirm this hypothesis, as the eldest quartile of patients took the longest time to complete the total questionnaire set. It is important to note that although the oldest quartile took the most time, they were still able to complete the survey expeditiously. We did not find any significant differences in TTC with regard to sex. In our study, we found TTC of PROMIS PF to differ by race, with Asian patients having the longest mean TTC and Hispanic/Latino patients completing PF with the shortest mean time. Previous studies showed that PF has high internal consistency and validity across age and race/ethnicity groups in terms of scoring.^9,12,18 Although we did find differences in our groups, the majority of our patients were able to complete our questionnaire in <5 minutes. Differences among our groups may not be clinically meaningful in terms of collecting PROs within the clinic setting.

Ease of administration and user-friendliness are important when PROs are implemented within the clinic setting. Many studies in the literature have demonstrated that electronic collection results in a higher rate of question completion when compared with paper forms.¹⁰ The recent rise of connectivity and the familiarity of patients with tablet computers have made electronic survey completion easier. Therefore, implementing PRO collection systems with scores that can be completed in a timely fashion may facilitate a higher rate of outcome reporting in a busy ambulatory clinic. Although we did find minor differences in TTC, these are not likely significant enough to create obstacles in the ambulatory setting. Our patients were able to quickly finish the surveys and not significantly affect clinic workflow.

Limitations of the present study included our inability to confirm whether the patient or a health care proxy or family member completed the survey. It is possible that parents may have completed surveys for minors or that caregivers did so for older or non-English-speaking patients, and we were unable to control for this factor. This limitation is inherent in any PRO collection initiative. This may be a particular limitation amid populations with high proportions of non-English speakers or those without ability to use tablet technology. All participants were required to be fluent in English to be included in this study, but our conclusions with regard to race/ethnicity did not consider whether English was a secondary language for the patient. As a result, the PROMIS CAT is likely not an adequate form for collection of PROs among non-English-speaking patients.

In addition, we did not use PROMIS pediatric forms for our patients <18 years old, which may have affected the TTC of our youngest quartile. We also did not make an attempt to control for any medical issues, which may have affected TTC, especially among older patients with decreased cognitive abilities. Patients who were unable to complete the survey in the waiting room were excluded to ensure that TTC was not falsely elevated as patients were brought back to the clinic room and had their vitals taken. As a result, patients who would have had a higher TTC may have been excluded. Our findings may not be representative of the overall population; that is, our patient population may have been more adept with tablet technology, as a preponderance of our patients were younger. We did not collect data on how many patients were unable to complete the questionnaire because of unfamiliarly with tablet computers or poor English language skills, which would have been a useful adjunct to our study. In addition, our analysis of ethnic groups was limited by the demographic data listed in the electronic medical record and may not be extrapolated to other parts of the country. We also did not evaluate socioeconomic factors, such as education or income, which may have affected a patient’s TTC.

In conclusion, our results indicate that age, race, and ethnicity appeared to have a statistically significant effect on TTC, while sex did not. Although our study did reveal slight differences among groups in TTC, our cohort’s mean TTC of 3.29 minutes demonstrates that our patients were able to quickly and efficiently complete PROMIS. We recommend considering implementation of PROMIS scores as a routine PRO tool in the shoulder, elbow, and sports medicine population that can be utilized for patients, whether they undergo surgery or not.

Footnotes

One or more of the authors has declared the following potential conflict of interest or source of funding: S.M. is a paid consultant for DePuy and Exactech; has received educational support from Arthrex; and has received hospitality payments from DePuy Synthes, Zimmer Biomet Holdings, Biomet Orthopedics, Exactech, Tornier, Conventus Orthopaedics, and Arthrex. V.M. has received hospitality payments from Stryker and Pinnacle and educational support from Arthrex and Pinnacle. E.C.M. receives royalties from Springer, has received educational support from Pinnacle and Smith & Nephew, and has received hospitality payments from Smith & Nephew and Stryker.

Ethical approval for this study was obtained from the Henry Ford Health System Institutional Review Board (Nos. 10820 and 10821).

References

Basch

. Patient-reported outcomes—harnessing patients’ voices to improve clinical care. N Engl J Med. 2017;376(2):105–108.

Baumhauer

. Patient-reported outcomes—are they living up to their potential? N Engl J Med. 2017;377(1):6–9.

Beckmann

Hung

Bounsanga

Wylie

Granger

Tashjian

. Psychometric evaluation of the PROMIS Physical Function Computerized Adaptive Test in comparison to the American Shoulder and Elbow Surgeons score and Simple Shoulder Test in patients with rotator cuff disease. J Shoulder Elbow Surg. 2015;24(12):1961–1967.

Cella

Yount

Rothrock

. The Patient-Reported Outcomes Measurement Information System (PROMIS): progress of an NIH Roadmap cooperative group during its first two years. Med Care. 2007;45(5 suppl 1):S3–S11.

Czaja

Charness

Fisk

. Factors predicting the use of technology: findings from the Center for Research and Education on Aging and Technology Enhancement (CREATE). Psychol Aging. 2006;21(2):333–352.

Harris

Taylor

Thielke

Payne

Gonzalez

Conde

. Research Electronic Data Capture (REDCap)—a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009;42(2):377–381.

Hawthorne

. Review of Patient Satisfaction Measures. Canberra, Australia: Australian Government Department of Health and Ageing; 2006.

Heidegger

Saal

Nuebling

. Patient satisfaction with anaesthesia care: what is patient satisfaction, how should it be measured, and what is the evidence for assuring high patient satisfaction? Best Pract Res Clin Anaesthesiol. 2006;20(2):331–346.

Jensen

Potosky

Reeve

. Validation of the PROMIS physical function measures in a diverse US population-based cohort of cancer patients. Qual Life Res. 2015;24(10):2333–2344.

10.

Jensen

Rothrock

DeWitt

. The role of technical advances in the adoption and integration of patient-reported outcomes in clinical care. Med Care. 2015;53(2):153–159.

11.

Kortlever

Janssen

van Berckel

Ring

Vranceanu

. What is the most useful questionnaire for measurement of coping strategies in response to nociception? Clin Orthop Relat Res. 2015;473(11):3511–3518.

12.

Liu

Yuan

Wang

. Comparability of the Patient-Reported Outcomes Measurement Information System Pediatric short form symptom measures across culture: examination between Chinese and American children with cancer. Qual Life Res. 2016;25(10):2523–2533.

13.

Morgan

Kallen

Okike

Lee

Vrahas

. PROMIS Physical Function Computer Adaptive Test compared with other upper extremity outcome measures in the evaluation of proximal humerus fractures in patients older than 60 years. J Orthop Trauma. 2015;29(6):257–263.

14.

Oude Voshaar

Ten Klooster

Glas

. Validity and measurement precision of the PROMIS physical function item bank and a content validity-driven 20-item short form in rheumatoid arthritis compared with traditional measures. Rheumatology (Oxford). 2015;54(12):2221–2229.

15.

Paulino Pereira

Janssen

Raskin

. Most efficient questionnaires to measure quality of life, physical function, and pain in patients with metastatic spine disease: a cross-sectional prospective survey study. Spine J. 2017;17(7):953–961.

16.

Stapleton

Degitz

. An innovative data collection method for investigating unresolved pain after ED discharge: a pilot study. J Emerg Nurs. 2014;40(6):598–604.

17.

Stone

Jacobs

Luo

. High degree of variability in reporting of clinical and patient-reported outcomes after hip arthroscopy [published online September 1, 2017]. Am J Sports Med. doi:10.1177/0363546517724743

18.

Teresi

Ocepek-Welikson

Cook

. Measurement equivalence of the Patient Reported Outcomes Measurement Information System (PROMIS) pain interference short form items: application to ethnically diverse cancer and palliative care populations. Psychol Test Assess Model. 2016;58(2):309–352.

19.

Tyser

Beckmann

Franklin

. Evaluation of the PROMIS Physical Function Computer Adaptive Test in the upper extremity. J Hand Surg Am. 2014;39(10):2047–2051.