A Health-Related Quality-of-Life Measure for Use in Patients with HIV: A Validation Study

Abstract

In the United States, HIV has evolved from an acute disease to a chronic illness making health-related quality of life a pre-eminent goal for many persons living with HIV (PLWH). There have been a number of HIV-specific quality-of-life instruments developed, but little attention has been paid to the validation of standardized nondisease-specific quality-of-life instruments tailored to PLWH. The goal of this research was to validate the Patient-Reported Outcomes Measurement Information System (PROMIS)-29, a questionnaire that measures health-related quality of life in PLWH. A sample of 1306 PLWH completed an online anonymous survey assessing their symptom experience and health-related quality of life. A subsample of 209 participants completed another questionnaire 30 days later. The subscales of the PROMIS-29 showed high internal consistency reliability (range = 0.87–0.97). The PROMIS-29 detected differences in health-related quality of life in those persons who reported an AIDS diagnosis compared to those who did not report an AIDS diagnosis. The PROMIS-29 has demonstrated reliability, validity, and reproducibility for use in measuring health-related quality of life in PLWH.

Introduction

HIV has changed from an acute illness to a chronic disease.¹ With continual improvements in the treatment of HIV, persons living with HIV (PLWH) are living longer but experiencing more symptoms associated with the illness and its treatment as well as the symptoms associated with the normal aging process. More than 40% of adults with HIV are 50 years of age and older, and the relative proportion of older adults living with HIV is growing.² As the population of PLWH ages, there is a sharply increased risk of poorer everyday functioning and HIV-related disability.³

Health-related quality of life has been conceptualized as an important metric for understanding perceived well-being among persons living with chronic conditions.⁴ Health-related quality of life can be used as a criterion for assessing subjective self-reported health.⁵ This is especially relevant for PLWH who are often living with the illness as well as other comorbid conditions.⁶

Therefore, the impact of HIV on health-related quality of life, particularly in light of the changing face of the epidemic, warrants further study. One of the challenges of studying health-related quality of life in PLWH is the need for an appropriate tool for measuring this construct. While there are several tools developed for measuring health-related quality of life in HIV, most of these tools were developed specifically for HIV at a time when the illness was much more acute and patients were unlikely to live for long enough to develop many of the diseases associated with the normal aging process, such as cardiovascular disease, arthritis, and osteoporosis.⁷

The Patient-Reported Outcomes Measurement Information System^® (PROMIS) measures have not been validated in PLWH. These measures are very useful since the PROMIS measures are a common data element (CDE) and a National Institutes of Health (NIH)-supported CDE. We specifically focused this work on the PROMIS-29, a multi-item measure for assessing generic profile health-related quality-of-life measure.⁸ PROMIS is an NIH Roadmap initiative to develop instruments to measure patient-reported outcomes in respondents with a wide range of chronic illness and demographic characteristics.

The PROMIS-29 is quickly becoming a standard patient-reported research and practice measure and is recommended for initial outcome assessment.^9,10 Studies continue to support its construct validity and feasibility.^11,12 However, the PROMIS-29 has not been validated in PLWH and since the PROMIS-29 is a CDE and the number of persons in the United States who continue to live with HIV is vast, this is an important step in improving outcomes and more specifically health-related quality-of-life outcomes in PLWH. The purpose of this research is to report on the psychometric properties of the PROMIS-29 in PLWH. Ultimately, the goal of this work is to identify a generic health-related quality-of-life instrument, which can be used to assess the metric in PLWH in the United States and other developed countries.

Methods

Study participants were recruited from February to July 2016 through an online anonymous survey of PLWH to assess their symptom experience. Recruitment sites included POZ.com, Craigslist, and Facebook.com (the largest online social networking site).

The online survey collected information on health symptoms that impeded daily activities, and secondary measures, including demographics, substance use, homelessness, education, employment, antiretroviral therapy adherence, CD4 count, HIV viral load, scheduled/missed healthcare visits, and health-related quality of life (PROMIS-29). The purpose of the overall study was to understand the symptom experience and self-management strategies of PLWH. The entire survey had 118 questions and we used a subset of the survey questions included in the larger survey to conduct this analysis.

We also conducted a follow-up survey with a subset of 209 PLWH participants who completed the original survey and provided a contact e-mail address for follow-up. Participants for the follow-up survey were selected from participants who had fully completed the original survey, agreed to participate in the follow-up survey, and provided a valid e-mail address. These individuals were e-mailed a one-time follow-up survey, 1 month after the original survey. Participants who completed the follow-up survey were given a $10 gift card as a token of appreciation for their time. The follow-up survey included demographic questions, PROMIS-29, and the HIV symptom index.¹³

Study subjects

For inclusion in the study, participants needed to report that they were 18 years or older, living in the United States, diagnosed with HIV, and able to read and write in English. Pregnant women were excluded from this survey because the symptoms associated with pregnancy are transient and not reflective of the symptoms related to HIV disease and other comorbid conditions related to the disease. The Columbia University Medical Center Institutional Review Board approved all procedures and granted a waiver of the requirement to obtain written documentation of consent.

Instruments

PROMIS-29

The PROMIS-29 v 1.0 short form is a multi-dimensional 29-item generic measure of health and is intended for use across a variety of conditions. It includes seven domains: Physical Functioning, Anxiety, Depression, Fatigue, Sleep Disturbance, Satisfaction with Participation in Social Roles, and Pain Interference, with an additional Pain Intensity. There are four items in the first seven domains with responses ranging from 1 to 5 (i.e., 16 decrements each), and a single item in pain intensity is assessed using a single 11-point numeric rating scale anchored between no pain (0) and worse imaginable pain (10).¹⁴ Raw scores for each domain are calculated by summing the item scores while adjusting for missing item responses. Raw scores are transformed using the T score metric based on the item response theory calibrations, in which scores have a mean of 50 and standard deviation (SD) of 10 for the general population in the United States. T scores can be estimated using the scoring tables listed in the PROMIS manuals. A higher PROMIS T score implies more of the concept being measured; for instance, a higher PROMIS score on physical functioning indicates better functioning, whereas a higher score on depression indicates more severe depressive symptoms. We did not make any additions or changes to the original PROMIS-29 instrument.

Sociodemographic questionnaire

A self-reported sociodemographic questionnaire was developed to collect information on the participants' age, gender, race, ethnicity, sexual orientation, education, household income, and marital status.

The HIV Symptom Index

The HIV Symptom Index is a 20-item standard instrument routinely used for clinical care and research with PLWH to capture the prevalence and magnitude of HIV-related symptoms. The index was developed to identify and describe symptoms for the purpose of developing targeted interventions.¹³ Patients identify symptoms experienced and then rate each reported symptom as to the level of bothersomeness on a five-point Likert-type scale ranging from symptom not present (0) to bothers me a lot (4). The HIV Symptom Index has demonstrated construct validity with high test–retest reliability (intraclass correlation coefficient [ICC] = 0.92) and internal consistency (α = 0.79).¹³

Procedures

Psychometric test theory involves the construction and evaluation of clusters of questions, called scales, which are used to gather information about patient quality of life. All newly developed or revised quality-of-life scales and existing scales in new patient groups must undergo psychometric evaluation. We evaluated the following properties in our study sample: variability, internal consistency reliability, reproducibility, construct validity, and criterion validity. Each property and the appropriate analysis are described below.

The full range of item responses and of scale scores are reported in the data. Optimal variability is denoted by patient responses at both ends of the scale as well as in the middle. Scales that are skewed, whether positively or negatively, tend to be less responsive to change from treatment effects or disease progression. To ensure limits in the variability, the frequency of missing data should be limited and randomly distributed across participant responses.

Internal consistency reliability is a measure of the similarity of individual responses across several items, indicating the homogeneity of a scale and the extent to which the scale is free of random error. Cronbach's α coefficient provides an estimate of reliability based on all possible correlations between items collected at any time point.¹⁵ Cronbach's α scores range between 0.0 and 1.0 with the desired range of scores between 0.70 and 0.95.¹⁶

Validity refers to how well the scale measures the attribute it is intended to measure, such as pain or physical functioning. There are several components to validity and in this study we focused on the construct and criterion validity.

Construct validity examines the extent to which a scale is measuring what it claims to measure. We measured three subtypes of construct validity in our study: convergent validity, discriminant validity, and known-group validity. Convergent validity refers to the degree to which theoretically correlated measures are in fact correlated, while discriminant validity is used to evaluate the differences between uncorrelated and correlated subscales. One way to examine the convergent and discriminant validity is to assess the correlations among scale scores within the instrument based on known relationships. For example, scales measuring sleep disturbance are expected to correlate moderately with one another, while scales measuring physical functioning are expected to have weaker correlations with mental functioning scales because they measure different constructs. We used a multi-trait multi-method matrix¹⁷ with interscale correlations to assess the convergent and discriminant validity. We also assessed the known-groups validity, which tests for anticipated differences on specific scale scores between groups that are known to be clinically different. In the case of our study population, we assessed the differences between participants who reported ever having an AIDS diagnosis versus no AIDS diagnosis. PLWH with a history of an AIDS diagnosis were in poorer health and more symptomatic. We evaluated the ability of this instrument to distinguish between PLWH with an AIDS diagnosis and those with no previous AIDS diagnosis.

Criterion validity is the extent to which a measure is correlated with a validated outcome measure and it is usually split into concurrent validity and predictive validity. We used the correlation between the total HIV symptom index score¹³ and each subscale at baseline and follow-up survey to measure the predictive and concurrent validity, respectively.

Reproducibility (test–retest) measures the degree to which an instrument yields stable scores over a short period of time, assuming there is no clinical change. Reproducibility is generally measured by the ICC, which ranges between 0.0 and 1.00. An ICC of >0.75 indicates excellent reproducibility, while an ICC between 0.4 and 0.75 indicates good reproducibility.¹⁸

Results

A total of 2101 surveys were missing demographic data and/or key outcome variables (i.e., noncompleters) and were removed from the data set yielding a final sample of 1306 respondents. Participants completed the one-time survey in an average of 67.15 min (SD = 376.91) and no incentives were given for completion. As reported in Table 1, the majority of PLWH were male, white, had several years of post-high school education, had a household income <$40,000, and were single, separated, or divorced. The mean age of the participants was 48.5 years (SD = 11.7) with a range of 19–81 years.

Table 1.

Characteristics of Study Sample (N = 1306 PLWH)

Characteristics	N	%
Gender
Male	933	71.44
Female	359	27.49
Transgender male/transman/FTM	2	0.15
Transgender female/transwoman/MTF	8	0.61
Genderqueer	4	0.31
Race^a
White/Caucasian	830	63.55
Black/African American	398	30.47
Other	97	7.43
Ethnicity
Hispanic	157	12.02
Non-Hispanic	1149	87.98
Sexual orientation
Homosexual	799	61.18
Heterosexual	374	28.64
Bisexual	133	10.18
Education
Less than high school graduate	51	3.90
High school, technical school graduate	316	24.20
Some college	392	30.02
College	286	21.90
Graduate school/professional school	258	19.75
Annual household income
<$20,000	491	37.60
$20,000–$39,999	294	22.51
$40,000–$59,999	157	12.02
$60,000–$79,999	123	9.42
$80,000–$99,999	61	4.67
≥$100,000	100	7.66
Marital status
Married or in a steady relationship	389	29.80
single, separated, divorced, or widowed	884	67.70
Age (years) mean (SD)	48.5 (11.70)

Totals are greater than sample size since participants can select more than one race.

PLWH, persons living with HIV; SD, standard deviation.

The range, mean, median, and SD for each of the subscales' T score at baseline are reported in Table 2. Completion rates were identical for all subscales except for the Pain Interference scale. Variability was evaluated for each of the subscales. To score minimum or maximum on the subscale's T score, a respondent would have to report the lowest or highest functioning for every item included in the subscale. The full range of responses was observed for each of the PROMIS subscales.

Table 2.

Descriptive Statistics: Subscale T Scores for PLWH Who Completed the PROMIS-29 Quality-of-Life Questionnaire at Baseline

Subscale	N	Range	Mean	Median	SD	Floor ^a (%)	Ceiling ^b (%)
Physical Functioning	1306	22.9–56.9	48.97	48.00	8.62	0.23	49.31
Anxiety	1306	40.3–81.6	55.28	55.80	10.29	22.13	0.84
Depression	1306	41.0–79.4	55.29	55.70	10.22	24.35	2.07
Fatigue	1306	33.7–75.8	54.41	55.10	10.95	9.80	4.90
Sleep Disturbance	1306	32.0–73.3	53.70	54.30	9.06	3.60	2.60
Satisfaction with Participation in Social Roles	1306	29.0–64.1	50.11	49.80	11.27	7.89	29.25
Pain Interference	1293	41.6–75.6	52.88	55.60	10.20	38.05	3.56

Percent of subjects who have scored the lowest possible dimension score.

Percent of subjects who have scored the highest possible dimension score.

PROMIS, Patient-Reported Outcomes Measurement Information System.

Internal consistency, reliability, and construct validity

Internal consistency reliability, as measured by Cronbach's α coefficient, is reported on the diagonal (bold type) for each of the multi-item scales in Table 3. All the scales displayed acceptable Cronbach's α values (>0.7) with the scores ranging from 0.87 to 0.97. Interscale correlations, which measure convergent validity along with discriminant validity, a subtype of construct validity, are also reported in Table 3. The interscale correlations related to mental health (i.e., anxiety and depression) were highly correlated (0.75) and those that were related to mental health are moderately correlated, such as anxiety and sleep disturbance (0.43) and depression and fatigue (0.53). There were no subscales with correlations greater than 0.75, which would indicate high correlation and therefore redundancy.

Table 3.

Internal Scale Consistency Scores and Interscale Correlations for PROMIS Subscales (n = 1306)

	Physical Functioning	Anxiety	Depression	Fatigue	Sleep Disturbance	Satisfaction with Participation in Social Roles	Pain Interference
Physical Functioning	0.92
Anxiety	−0.34	0.92
Depression	−0.34	0.75	0.94
Fatigue	−0.54	0.53	0.53	0.95
Sleep Disturbance	−0.31	0.43	0.44	0.51	0.87
Satisfaction with Participation in Social Roles	0.64	−0.42	−0.45	−0.61	−0.38	0.97
Pain Interference	−0.69	0.38	0.36	0.56	0.38	−0.56	0.97

Known-groups validity, another subtype of construct validity, was evaluated by measuring differences in mean scale scores at baseline among groups expected to vary with regard to outcome: AIDS diagnosis and no AIDS diagnosis (Table 4). Statistically significant differences between AIDS diagnosis and no AIDS diagnosis were found for all subscales.

Table 4.

Mean Scale Scores at Baseline By AIDS Status for PROMIS-29 Subscales

	AIDS diagnosis (n = 466)		No previous AIDS diagnosis (n = 803)
T scores Subscale	Mean	SD	Mean	SD	p^a
Physical Functioning	46.24	8.66	50.68	8.10	<0.001
Anxiety	56.55	9.95	54.45	10.44	<0.001
Depression	56.61	9.76	54.42	10.39	<0.001
Fatigue	57.61	10.11	52.53	10.98	<0.001
Sleep Disturbance	54.69	8.56	52.99	9.30	0.0013
Satisfaction with Participation in Social Roles	46.94	10.99	52.08	10.99	<0.001
Pain Interference	55.88	10.33	50.95	9.61	<0.001

Significant at 0.01 level.

Reproducibility and criterion validity

Reproducibility results are reported for 209 individuals who completed two questionnaires 30 days or less apart, as shown in Table 5. ICCs are good (i.e., >0.40 and <0.75) for Physical Functioning, Sleep Disturbance, Satisfaction with Participation in Social Roles, and Pain Interference. ICCs were >0.60 for all other subscales. Table 5 also presents information on criterion validity, operationalized as both concurrent validity and predictive validity, measured as the correlation between each subscale and a validated tool measuring total HIV symptom index score at baseline and follow-up survey.

Table 5.

Reproducibility and Criterion Validity of PROMIS Subscales (n = 209)

Variable	ICC	Correlation between baseline score and total HIV symptom index score (predictive validity)	Correlation between follow-up score and total HIV symptom index score (concurrent validity)
Physical Functioning	0.71	−0.40	−0.49
Anxiety	0.78	0.61	0.67
Depression	0.81	0.52	0.58
Fatigue	0.81	0.62	0.67
Sleep Disturbance	0.70	0.44	0.42
Satisfaction with Participation in Social Roles	0.67	−0.53	−0.60
Pain interference	0.61	0.54	0.52

ICCs are only calculated for respondents with 2 weeks or less between questionnaires.

ICC, intraclass correlation coefficient.

Discussion

This study was conducted as part of a larger web-based survey to understand the symptom experience of PLWH. The validation study reported in this article was conducted to explore the psychometric properties of the PROMIS-29, a health-related quality-of-life instrument that includes subscales of Physical Functioning, Anxiety, Depression, Fatigue, Sleep Disturbance, Satisfaction with Participation in Social Roles, Pain Interference, and Pain Intensity, in a PLWH population. The results of this study support the validity, reliability, and reproducibility over 30 days of the PROMIS-29 for use in PLWH.

The response rates for individual items were excellent. Results indicate that the internal consistency reliability of the measure is very good, with high Cronbach's α values providing strong evidence of reliability and no indication of redundancy in scale items

Evaluation of reproducibility over 30 days produced good results when ICC was calculated with a subsample of participants who completed the PROMIS-29, 30 days apart demonstrating good reproducibility. Moreover, the results from the PROMIS-29 were moderately correlated with those from a validated HIV symptom index. This demonstrates good predictive and concurrent validity.

In a previous study in women with fibromyalgia, the PROMIS instruments had fair to high internal consistency (Cronbach's α = 0.58–0.94),¹⁹ yet a range lower than that was found in our work. In another study in people with rheumatoid arthritis, test–retest reliability ranged from 0.725 to 0.883, and Cronbach's alpha from 0.906 to 0.991.²⁰ Finally, in a study to validate the PROMIS in a scleroderma clinic, all correlations between PROMIS domains and respective legacy measures were large and in the hypothesized direction (ranged from 0.61 to 0.82).²¹ In comparison to earlier validation studies in other diseases, our findings provide very strong psychometric evidence for use of the PROMIS-29 as a measure of health-related quality of life in PLWH.

Construct validity was assessed by measuring differences in mean scale scores at baseline among persons who reported a previous AIDS diagnosis, compared to those who had never been diagnosed with AIDS. Statistically significant differences between groups were found, in the expected direction, for all subscales except sleep disturbance. These trends were expected based on the clinical evidence that PLWH with a previous AIDS diagnosis are more likely to report poorer health-related quality of life.

The construct validity of the measure was also supported by the degree to which interscale correlations corresponded to what was expected. The higher correlations among mental health items (anxiety and depression) than between the other subscale items are an additional indication supporting the construct validity. Overall, the evidence supports the construct validity of the PROMIS-29 in PLWH.

There are a number of limitations of this validation study. First, we are unable to validate that all our study participants were HIV positive. We asked our participants during the screening question as well as two points during the survey. In addition, most of our study sample was recruited from POZ.com, an online site for people living with and affected by HIV/AIDS. Moreover, a growing number of validity studies indicate higher reporting of sexual risk and substance-using behaviors with computer-based surveys compared to mail, phone, and in-person surveys,^22

–25 making it more likely that persons diagnosed with HIV would be willing to report their status.

A second limitation is that we used the convenience sampling method, which limits the generalizability of our findings. A large portion of our study participants self-identified as being male (71%) and homosexual (61%), however, this is reflective of the current demographics of the US HIV epidemic. Our study sample was greater than 30% black/African American and 12% Latino, which closely mirrors the current HIV prevalence in racial and ethnic minority groups in the United States.²⁶ Moreover, we had more than 2000 people who started the survey and only 1300 who completed it, further suggesting that our sample may be a self-selected group who may be more interested in their own health. Second, the test–retest interval was relatively brief and further research should test the stability of the PROMIS-29 over a longer period of time. Finally, to provide comprehensive evidence for concurrent validity of the PROMIS-29 in this population, further work correlating the PROMIS-29 with other standardized scales of quality of life for PLWH, such as the Multidimensional Quality of Life Questionnaire for PLWH,²⁷ is warranted.

In conclusion, this study has provided preliminary evidence in support of the validity, reliability, and reproducibility of the PROMIS-29 in PLWH. Notably, the PROMIS-29 offers the added benefit of being a standardized instrument that was developed for use across diseases and is a CDE required for use in some NIH-funded studies. As such, the validation of the PROMIS-29 in PLWH is a needed contribution to the extant HIV literature, in that this is a necessary instrument for measuring health-related quality of life in this study population.

In light of our findings, the authors recommend the use of PROMIS-29 as a measurement tool for assessing health-related quality of life in PLWH. Validation of the PROMIS-29 is an important step in ensuring that appropriate outcome measures are available for assessing health-related quality of life in PLWH, but this will also hopefully stimulate additional research and practice to improve health-related quality of life in PLWH. In addition, as management of HIV has shifted to be a part of general primary care, use of a scale across illnesses that include PLWH is an important step in fitting within the needs of our healthcare system.

Footnotes

Acknowledgments

This publication was supported by funding from the National Institute of Health/National Institute of Nursing Research (R01NR015737; PI: R.S.). The findings and conclusion in this article are those of the authors and do not necessarily represent the views of the NIH. The authors thank Lena Milian for her assistance with the data collection.

Author Disclosure Statement

No competing financial interests exist.

References

Deeks

, Lewin

, Havlir

. The end of AIDS: HIV infection as a chronic disease. Lancet, 2013; 382:1525–1533.

Centers for Disease Control. HIV Among People Aged 50 and Over. Atlanta, GA: U.S. Department of Health and Human Services, CDC, 2016.

Cahill

, Valadéz

. Growing older with HIV/AIDS: New public health challenges. Am J Public Health, 2013; 103:e7–e15.

Testa

, Simonson

. Assessment of quality-of-life outcomes. N Engl J Med, 1996; 334:835–840.

Perez

, Baño

, Ruz

MAL

, et al. Health-related quality of life of patients with HIV: Impact of sociodemographic, clinical and psychosocial factors. Qual Life Res, 2005; 14:1301–1310.

Mitchell

, Nguyen

, Isenberg

, Maragh-Bass

, Keruly

, Knowlton

. Psychosocial and service use correlates of health-related quality of life among a vulnerable population living with HIV/AIDS. AIDS Behav, 2016 Oct 27. [Epub ahead of print] DOI: 10.1007/s10461-016-1589-1.

, Lamping

. Assessment of quality of life in HIV disease. AIDS, 1994; 8(Suppl 1).

Cella

, Riley

, Stone

, et al. The Patient-Reported Outcomes Measurement Information System (PROMIS) developed and tested its first wave of adult self-reported health outcome item banks: 2005–2008. J Clin Epidemiol, 2010; 63:1179–1194.

Working Group on Health Outcomes for Older Persons with Multiple Chronic Conditions. Universal health outcome measures for older persons with multiple chronic conditions. J Am Geriatr Soc, 2012; 60:2333–2341.

10.

Forrest

, Bevans

, Tucker

, et al. Commentary: The Patient-Reported Outcome Measurement Information System (PROMIS®) for children and youth: Application to pediatric psychology. J Pediatr Psychol, 2012; 37:614–621.

11.

Hinchcliff

, Beaumont

, Thavarajah

, et al. Validity of two new patient reported outcome measures in systemic sclerosis: The PROMIS-29 Profile and the FACIT-Dyspnea. Arthritis Care Res, 2011; 63:1620–1628.

12.

Selewski

, Collier

, MacHardy

, et al. Promising insights into the health related quality of life for children with severe obesity. Health Qual Life Outcomes, 2013; 11:29.

13.

Justice

, Holmes

, Gifford

, et al. Development and validation of a self-completed HIV symptom index. J Clin Epidemiol, 2001; 54(Suppl 1):S77–S90.

14.

Viney

, Norman

, Brazier

, et al. An Australian discrete choice experiment to value EQ-5D health states. Health Econ, 2014; 23:729–742.

15.

Cronbach

. Coefficient alpha and the internal structure of tests. Psychometrika, 1951; 16:297–334.

16.

Tavakol

, Dennick

. Making sense of Cronbach's alpha. Int J Med Educ, 2011; 2:53–55.

17.

Campell

, Fiske

. Convergent and discriminant validation by the multitrait-multimethod matrix. Psychol Bull, 1959; 56:81–105.

18.

Rosner

. Fundamentals of Biostatistics. Belmont, CA: Duxbury Press, 1995.

19.

Merriwether

, Rakel

, Zimmerman

, et al. Reliability and construct validity of the Patient-Reported Outcomes Measurement Information System (PROMIS) instruments in women with fibromyalgia. Pain Med, 2016;pii: pnw187.

20.

Bartlett

, Orbai

A-M

, Duncan

, et al. Reliability and validity of selected PROMIS measures in people with rheumatoid arthritis. PLoS One, 2015; 10:e0138543.

21.

Khanna

, Maranian

, Rothrock

, et al. Feasibility and construct validity of PROMIS and legacy instruments in an academic scleroderma clinic—Analysis from the UCLA Scleroderma Quality of Life Study. Value Health, 2012; 15:128–134.

22.

Link

, Mokdad

. Alternative modes for health surveillance surveys: An experiment with web, mail, and telephone. Epidemiology, 2005; 16:701–704.

23.

Perlis

, Des Jarlais

, Friedman

, Arasteh

, Turner

. Audio-computerized self-interviewing versus face-to-face interviewing for research data collection at drug abuse treatment programs. Addiction, 2004; 99:885–896.

24.

Elford

, Bolding

, Davis

, Sherr

, Hart

. Web-based behavioural surveillance among men who have sex with men: A comparison of online and offline samples in London, UK. J Acquir Immune Defic Syndr, 2004; 35:421–426.

25.

Newman

, Des Jarlais

, Turner

, Gribble

, Cooley

, Paone

. The differential effects of face-to-face and computer interview modes. Am J Public Health, 2002; 92:294–297.

26.

Centers for Disease Control and Prevention. HIV Among Gay and Bisexual Men. 2016. Available at: www.cdc.gov/hiv/group/msm (Last accessed September 14, 2016 ).

27.

Remple

, Hilton

, Ratner

, Burdge

. Psychometric assessment of the Multidimensional Quality of Life Questionnaire for Persons with HIV/AIDS (MQOL-HIV) in a sample of HIV-infected women. Qual Life Res, 2004; 13:947–957.