Abstract
Background/Aims:
The Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE®) was developed to capture symptomatic adverse events from the patient perspective. We aim to describe statistical properties of PRO-CTCAE items and summary scores and to provide evidence for recommendations regarding PRO-CTCAE administration and reporting.
Methods:
Using data from the PRO-CTCAE validation study (NCT02158637), prevalence, means, and standard deviations of PRO-CTCAE items, composite scores, and mean and maximum scores across attributes (frequency, severity, and/or interference) of symptomatic adverse events were calculated. For each adverse event, correlations and agreement between attributes, correlations between attributes and composite scores, and correlations between composite, mean, and maximum scores were estimated.
Results:
PRO-CTCAE items were completed by 899 patients with various cancer types. Most patients reported experiencing one or more adverse events, with the most prevalent being fatigue (87.7%), sad/unhappy feelings (66.0%), anxiety (63.6%), pain (63.2%), insomnia (61.8%), and dry mouth (60.0%). Attributes were moderately to strongly correlated within an adverse event (r = 0.53 to 0.77, all p < 0.001) but not fully concordant (κweighted = 0.26 to 0.60, all p < 0.001), with interference demonstrating lowest mean scores and prevalence among attributes of the same adverse event. Attributes were moderately to strongly correlated with composite scores (r = 0.67 to 0.97, all p < 0.001). Composite scores were moderately to strongly correlated with mean and maximum scores for the same adverse event (r = 0.69 to 0.94, all p < 0.001). Correlations between composite scores of different adverse events varied widely (r = 0.04 to 0.68) but were moderate to strong for conceptually related adverse events.
Conclusions:
Results provide evidence for PRO-CTCAE administration and reporting recommendations that the full complement of attributes be administered for each adverse event, and that attributes as well as summary scores be reported.
Background/aims
The National Cancer Institute’s (NCI) Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE®) is a library of 124 items, measuring 78 symptomatic adverse events (AEs) from the patient perspective. The NCI contracted development of this measure as a companion to the Common Terminology Criteria for Adverse Events (CTCAE now in version 5) for use in cancer clinical trials. 1 PRO-CTCAE is a valid and patient-centric method for capturing symptomatic AEs, including those that may be missed or underreported by clinicians.2,3 Because of the unique structure for grading AEs according to the CTCAE, each PRO-CTCAE symptomatic AE is assessed using one to three self-report items, measuring the attributes of presence, amount, frequency, severity, and/or interference with usual or daily activities. For example, pain is measured using three items for pain frequency, severity, and interference with usual or daily activities. The standard recall period for PRO-CTCAE is the past 7 days. The composite score for each AE can be produced algorithmically based on individual PRO-CTCAE items; this composite scoring algorithm was designed with clinical investigator input and is structured and scaled to resemble how clinicians would grade the same AE. 4 The composite score is designed to aid analysis and interpretation of PRO-CTCAE by providing a single summary score of each patient-reported symptomatic AE. 4
This secondary analysis of prospectively collected data was designed to evaluate the statistical properties (including distributional characteristics, correlations, and agreement) of PRO-CTCAE items and summary scores using data from patients with a variety of cancer types, a majority of whom had recently received cancer treatment. Previous work with this dataset has established the validity and reliability of PRO-CTCAE items and response scales, reported how items correlate with anchors from other patient-reported instruments, established mode equivalence, compared recall periods, and developed a rubric for an overall AE composite score.3–8 In this study we aimed to address questions related to administering, analyzing, and reporting PRO-CTCAE, including: how attribute items relate to each other and the extent to which each attribute contributes unique information; how summary scores relate to individual items; and how symptomatic AEs relate to each other. To address these questions, we investigated the statistical properties of individual PRO-CTCAE item scores and summary scores, including: prevalence of AEs, as determined by attribute score > 0 and by composite score > 0; distributional characteristics of items and summary scores; correlations and agreement between attributes for each AE; correlations between each attribute and the composite score for each AE; correlations between the composite score and other summary scores for each AE; and correlations between AEs. 1 Knowledge of these statistical properties can provide empirical insights to inform trial design, analysis and results reporting in studies that include PRO-CTCAE.
Method
Data used in this study were originally collected at nine centers as part of a large validation study of the PRO-CTCAE (ClinicalTrials.gov: NCT02158637).3–9 English-speaking adult patients with a solid tumor or hematologic malignancy who were receiving or initiating chemo- and/or radiation therapy were eligible to participate. 9 This study uses baseline data collected at the first study visit. Most participating patients received prior cancer treatment or were on active therapy at the time of enrollment. Included in the current study are items from a set of 20 symptomatic AEs that were administered to all participants in the validation sample. These 20 symptom terms reflect symptoms that are commonly reported by patients with a variety of cancer sites and treatment types, specifically anxiety, arm or leg swelling, constipation, decreased appetite, dry mouth, fatigue, hair loss, headache, insomnia, diarrhea, mouth or throat sores, nausea, numbness/tingling in hands/feet, pain, concentration problems, taste problems, rash, sad/unhappy feelings, shortness of breath, vomiting, and feeling like nothing could cheer you up. These 20 AEs were measured by one to three attributes, for a total of 41 items. More information about the PRO-CTCAE is available at https://healthcaredelivery.cancer.gov/pro-ctcae/.
Frequency, severity, interference, and amount are measured on a 0 (Never/None/Not at all) to 4 (Almost constantly/Very severe/Very much) scale. Presence is measured on a 0/1 (yes/no) scale. Branching logic was employed so that if a participant indicated they were not experiencing a specific AE, that is, frequency was 0, questions about severity and interference were skipped; a zero value was imputed for the conditionally skipped items. Composite scores were determined based on AE attributes, as previously described. 4 Composite scores are scaled from 0 to 3, with higher scores representing worse symptoms. Mean and maximum scores across attributes for the same AE were calculated by patient for AEs with two or more attributes. Composite scores, mean scores, and maximum scores were computed for all respondents with complete data for the AE, meaning all items not conditionally skipped were answered.
Item scores and composite scores were descriptively summarized using means and standard deviations. Prevalence of score > 0 was summarized as frequency and percentage. Rash presence is a dichotomous item with no composite score and was descriptively summarized as frequency and percentage of item prevalence; rash was not included in correlation or agreement analyses. For each AE, Pearson correlation coefficients were calculated between attributes (frequency, severity, and/or interference), between composite scores and each attribute (amount, frequency, severity, and/or interference), and between composite score, mean score, and maximum score among patients with complete data for all administered attributes for the AE who had a score > 0 on one or more attribute. As a supplemental analysis, Pearson and Spearman correlation coefficients were calculated among patients with complete data for all administered attributes of the AE, without the limitation of a score > 0 on an attribute. As correlation coefficients are inflated by the inclusion of a large number of 0 scores, this analysis is presented in the supplement and for the primary analysis the sample was restricted to patients with scores > 0 on one or more AE attributes. Agreement between attributes by AE was assessed using weighted kappa coefficients, among patients with a score > 0 on one or more attribute. Coefficients > 0.40 were interpreted as moderate agreement. 10 In addition, Pearson and Spearman correlation coefficients were calculated between composite scores for AEs, and the magnitude of correlation was graphically summarized (Pearson correlation coefficients only). Correlations ≥ 0.50 were considered moderate to strong in magnitude.11,12
Results
Of the 940 patients enrolled in the validation study, 899 completed one or more PRO-CTCAE items of interest at baseline and were included in the analysis. At the time of survey, most patients (743/899, 82.6%) had an Eastern Cooperative Oncology Group Performance Status (ECOG PS) of 0–1 and most (737/883, 83.5%) had received treatment (chemotherapy, radiation therapy, and/or surgery) in the past 2 weeks (Table 1).
Demographics for N = 899 patients included in the analysis.
Table 2 reports attribute-level means, standard deviations, and prevalences (percentage of patients with score > 0) for each of the 20 symptomatic AEs examined in this study. Item means were generally low (across all attributes, 31.7% (13/41) of item means were ≥ 1). For AEs with multiple attributes, interference consistently had the lowest mean score and lowest prevalence among the attributes (Table 2). For AEs measured by frequency and/or severity as well as interference, only rarely were mean interference scores higher than the scores of other attributes. Pain and fatigue were the symptomatic AEs with the greatest proportion of participants who rated interference higher than they did frequency and/or severity, (97/899 [10.8%] and 119/899 [13.2%] of participants, respectively) (Table S1).
PRO-CTCAE attribute scores, composite, mean and maximum scores, and prevalence rates.
Dashes are given where an attribute does not exist for the AE.
Range of attributes, mean, and maximum score is 0–4. Range of composite score is 0–3.
AE overall sample size is the number of complete sets of administered attributes. Patients were excluded from summary if they were missing one or more attribute for the AE. A zero value was imputed for conditionally skipped items, thus these conditionally skipped items were not considered missing.
Amount of hair loss.
Presence of rash.
Composite score means ranged from 0.3 to 1.5, with only 5 AEs demonstrating mean composite scores of 1 or greater (Table 2). The prevalence of composite scores > 0 ranged from 17.8% (vomiting) to 87.7% (fatigue), with the most prevalent AEs being fatigue (767/875, 87.7%), sad/unhappy feelings (554/840, 66.0%), anxiety (532/837, 63.6%), pain (536/848, 63.2%), insomnia (527/853, 61.8%), and dry mouth (505/842, 60.0%) (Table 2). Results for mean and maximum scores were similar to those of composite scores (Table 2). Composite scores were more similar to mean scores than to maximum scores. For all AEs, the mean and standard deviation of the composite scores never exceeded those of the maximum scores.
Among patients experiencing an AE (i.e. respondents with at least one PRO-CTCAE attribute score > 0), attributes were all moderately to strongly positively correlated, with correlation coefficients ranging from 0.53 to 0.77 (p < 0.001 for all; Table 3). In AEs with all three attributes, correlations between the interference and severity items were consistently numerically stronger than those between the interference and frequency items (Table 3). Agreement between attributes was fair to moderate, with weighted kappa coefficients ranging from 0.26 to 0.60 (p < 0.001 for all; Table 4). In AEs with three attributes, agreement was consistently numerically highest between frequency and severity and lowest between frequency and interference. Individual attribute scores were moderately to strongly correlated with the composite score (r = 0.67 to 0.97), severity and interference attribute scores demonstrated stronger correlations with the composite score than did frequency attribute scores (Table 3). Composite scores were also moderately to strongly correlated with both mean and maximum scores (r = 0.69 to 0.94); compared to maximum scores, mean scores demonstrated stronger correlation with the composite score (Table 3). Mean and maximum scores also demonstrated strong correlations (r = 0.78 to 0.96) (Table 3). The supplemental analysis based on all patients (including those with all attribute scores equal to 0) yielded uniformly higher correlations, as expected. Pearson and Spearman correlations were similar (Tables S2, S3, and S4).
Correlations between attributes and summary scores (composite, mean, maximum) of adverse events among respondents with any attribute score > 0.
Some AEs are assessed using a single attribute (constipation, dry mouth, hair loss, diarrhea, taste problems). For these AEs, the only correlation computed is between the single attribute and the composite score. In these cases and others, dashes are given where an attribute or summary score does not exist for the AE. Rash presence is a dichotomous item with no composite score and is omitted from this table.
All participants with one or more attribute score > 0 were included in each analysis.
Pearson correlation coefficient. All correlation coefficients significantly differ from 0 (p < 0.001).
Correlation between composite score and hair loss amount.
Agreement among PRO-CTCAE attribute pairs in respondents with any attribute score > 0.
Dashes are given where an attribute does not exist for the AE.
All participants with one or more attribute scores > 0 were included in each analysis.
All Kappa coefficients significantly differ from 0 (p < 0.001).
Figure 1 visualizes correlations between composite scores, which ranged from 0.04 to 0.68 (Table S4). The strongest correlations were seen between conceptually related or mechanistically co-occurring AEs, specifically sad/unhappy feelings and feeling nothing could cheer you up (r = 0.68), anxiety and sad/unhappy feelings (r = 0.65), anxiety and feeling nothing could cheer you up (r = 0.56), nausea and vomiting (r = 0.60), decreased appetite and taste changes (r = 0.55), and fatigue and pain (r = 0.51) (Table S4). Comparable results were seen for correlations between mean scores and between maximum scores.
Correlations between composite scores. Sample size is the number of non-missing pairs, ranging from n = 805 to 842. Shade and size of circles show strength of correlation, with stronger correlations represented as larger, darker circles and weaker correlations represented as smaller, lighter circles. Table S4 presents values of correlation coefficients visualized in this figure. Rash is a dichotomous AE with no composite score. As such, it was not included in this analysis.
Conclusions
Prior work established the validity and reliability of PRO-CTCAE and developed a composite scoring algorithm to yield a single numerical score for AEs.3,4 In this report we used data from the PRO-CTCAE validation study to extend those efforts by examining the prevalence, distributional characteristics, correlations, and agreement of attribute and summary scores in a diverse cohort of cancer patients, most of whom were undergoing treatment or had received treatment prior to study enrollment. Results presented here derive from data used to validate the PRO-CTCAE item library and can be used as normative or reference values, with the understanding that data were gathered in the setting of a diverse cancer patient population receiving or initiating chemo- and/or radiation therapy. We have described relationships among AE attributes and relationships between AE attributes and summary scores. These analyses can be applied to inform trial design, analysis and results reporting, and to strengthen the interpretability of PRO-CTCAE as a measure of tolerability.
While summary metrics like the composite score allow for succinct reporting of symptomatic AEs, descriptive analysis of individual attributes in addition to the summary score is also warranted and can provide a granular understanding of the burden of symptomatic AEs, particularly for AEs that are important to understanding the tolerability of a treatment regimen being studied. This recommendation is in line with the United States Food and Drug Administration guidance on PRO reporting, namely that summary scores and individual item scores be reported in tables and visualizations. 13 The distinct attributes of an AE can be differentially impacted by treatment, a nuance that may be missed when summary scores alone are reported. 14 Researchers should note that prevalence (as determined by score > 0) differs depending on whether prevalence is based on the individual attributes of frequency, severity, interference, or a composite score. An important result to highlight is that while PRO-CTCAE attributes are correlated, they are not identical. This finding supports the recommendation to use the complete set of attributes for an AE (and to conserve respondent burden by using conditional branching logic) rather than administering only a single item for assessment, as each attribute contributes unique information about the symptomatic AE and its contribution to conclusions about tolerability. Clinicians and researchers may be hesitant to include multiple attributes due to concerns about patient burden; however, patient burden can be alleviated by judicious selection of symptomatic AEs for monitoring and thoughtful determination of the intervals for PRO-CTCAE assessments within a given trial design. 15 Selecting for surveillance only those AEs most relevant to the treatment, and scheduling assessments to align with the anticipated onset and offset of symptomatic AEs can help manage patient burden, as assessments that are perceived to be relevant by patients may be less burdensome. 16
Composite scores were strongly correlated with mean scores and maximum scores (all correlations ≥ 0.78); however each of these summary scores (composite score, mean score, maximum score) may have value in analysis and reporting. Further study is needed to understand the strengths and limitations of these different approaches with respect to individual- and group-level interpretability and change over time. Composite scores, mean scores, and maximum scores may each offer distinct advantages as summary measures. As previously described, composite scores were designed to align with CTCAE grades and to have a clinically meaningful interpretation. 4 For example, in contrast to mean scores, composite scores and maximum scores are always integers, which may be more interpretable. In contrast to mean scores which weight all attributes equally and maximum scores which assign greatest weight to whichever attribute has the highest score, composite scores reflect clinicians’ interpretation of each unique combination of attribute scores. For example, a patient reporting frequency = 4, severity = 1, and interference = 0 would be assigned a composite score = 1, mean score = 1.67, and maximum score = 4. Composite scores for different AEs were all positively correlated, although most correlations were weak and very few pairs were more than moderately correlated, supporting the distinctness of the AEs assessed by PRO-CTCAE. Low correlation between conceptually unrelated AEs was expected. Our results showing that some AEs were comparatively more strongly correlated is intriguing, but is not a robust exploration of symptom clustering, a concept worth further investigation.
Strengths of this study include the large and diverse sample (with respect to demographic and clinical characteristics, cancer type, and treatment modality). The sample included patients with varied previous treatments and was enriched for patients with impaired performance status, meaning we were able to include patients with substantial symptom burden.
A limitation of this study is that analyses addressed only 20 of the 78 symptomatic AEs included in the PRO-CTCAE item library; further study is needed to determine if comparable results are seen with other symptomatic AEs such as cough, urinary urgency and frequency, or fecal incontinence. However, the AEs included in this analysis reflect the most common, cross-cutting AEs reported by patients with a variety of cancer types enrolled in clinical trials. 17 The data used in this analysis were drawn from a study that was designed as a validation study, and as such, in this diverse sample, participants were asked about many AEs that were not necessarily relevant to their treatment experiences. As a result, the proportions of zero scores and low scores were greater than what might be seen in a more homogeneous patient population asked about a set of AEs targeting a specific disease or treatment. Thus, to confirm our observations, these findings should be replicated in different settings. This study made use of only one time point, as such, we have not investigated associations at multiple predefined points during treatment or change over time. Lastly, although PRO-CTCAE is now widely available in more than 50 languages, the validation study from which our data are drawn was conducted only with English-speaking patients.
This study describes the statistical properties of PRO-CTCAE in a large sample of patients undergoing cancer treatment. Knowledge of these properties can be applied in future research to guide study design, analysis, and results reporting. Importantly, these results support current recommendations to include the full complement of attributes for each AE and to report attributes as well as summary scores.
Supplemental Material
sj-docx-1-ctj-10.1177_17407745241286065 – Supplemental material for Statistical properties of items and summary scores from the Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE®) in a diverse cancer sample
Supplemental material, sj-docx-1-ctj-10.1177_17407745241286065 for Statistical properties of items and summary scores from the Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE®) in a diverse cancer sample by Carolyn Mead-Harvey, Ethan Basch, Lauren J Rogak, Blake T Langlais, Gita Thanarajasingam, Brenda F Ginos, Minji K Lee, Claire Yee, Sandra A Mitchell, Lori M Minasian, Tito R Mendoza, Antonia V Bennett, Deborah Schrag, Amylou C Dueck and Gina L Mazza in Clinical Trials
Footnotes
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the National Cancer Institute of the National Institutes of Health [Award No. U01CA233046, Analyzing and Interpreting PRO-CTCAE with CTCAE and Other Clinical Data to Characterize Drug Tolerability]; data collection for the Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE) validation study was supported by the National Cancer Institute [Contract HHSN261200800043C].
Disclaimer
The opinions expressed in this article are the authors’ own and do not reflect the view of the National Institutes of Health, the US Department of Health and Human Services, or the United States government.
Trial registration number
ClinicalTrials.gov: NCT02158637.
Supplemental material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
