Psychometric Validation and Meaningful Change Thresholds of the New Nasal Polyposis Symptom Diary

Abstract

Objectives:

The Nasal Polyposis Symptom Diary (NPSD) is a novel and short patient-reported outcome (PRO) tool specifically developed to assess important and relevant symptoms reported by patients with chronic rhinosinusitis with nasal polyposis (CRSwNP). We evaluated the psychometric properties of 4 predefined NPSD-derived scores intended to support symptom-improvement assessments of investigational therapies for inclusion in product labeling.

Methods:

Five hundred eighteen patients with severe CRSwNP from a Phase III clinical trial (NCT03401229) completed the NPSD, comprising 11 items: 8 symptom-specific, 2 symptom-impact, and 1 optional medication-compliance. The psychometric characteristics of 3 single-item symptom scores (Nasal Blockage Score [NBS], Nasal Congestion Score [NCS], and Difficulty with Sense of Smell Score [DSS]) and a Total Symptom Score (TSS, summary of the 8 symptom-specific items) were evaluated for reliability, validity, and ability to detect change. Within-patient meaningful change thresholds (MCTs) were established using anchor- and distribution-based methods. Comparative PROs included the 22-item Sino-Nasal Outcome Test (SNOT-22) and Patient Global Impression of Severity (PGI-S).

Results:

The TSS exhibited strong internal consistency (Cronbach α = .88) and test-retest reliability (intraclass correlation coefficient >.80). Correlation between the TSS and SNOT-22 total score indicated good convergent validity (r = .70). All 4 NPSD scores demonstrated known-groups validity (significant differences among subgroups of patients with predetermined disease severity levels based on PGI-S categories) and were sensitive to detect change in patients’ clinical status (significant differences among subgroups of patients with reported changes between 2 time-points in PGI-S and Patient Global Impression of Change scores). MCTs for improvement were established at 1.0 point for NBS, NCS, and DSS, and 4.0 points for TSS.

Conclusion:

These findings support the reliability, validity, and suitability of the 4 NPSD-derived scores for evaluating treatment effect on CRSwNP symptoms and their use in clinical trials with predetermined MCTs for improvement.

Keywords

patient-reported outcome measure instrument development psychometric properties meaningful change threshold chronic rhinosinusitis with nasal polyposis

Introduction

Chronic rhinosinusitis with nasal polyposis (CRSwNP), a chronic inflammatory disease of the sinonasal mucosa, is characterized by persistent endoscopic signs of nasal polyps (NPs; inflammatory stalked protrusions of nasal mucosa), mucopurulent discharge, and/or edema.^1,2 Prominent symptoms, including nasal blockage, obstruction, and congestion, as well as nasal discharge, facial pain/pressure, and impaired sense of smell, can also significantly affect patients’ health-related quality of life (HRQOL).^3,4 Therefore, symptom improvement is an important treatment goal and a key objective in clinical trials for CRSwNP therapies.^1,5

Patient-reported outcome (PRO) measures, increasingly used as endpoints in clinical studies, provide details on patient-perceived experience of medical conditions and treatments.⁶ PRO endpoint selection relies on their ability to measure spontaneous or treatment-induced changes dependably and sensitively over time. Non-disease-specific PRO measures, such as the Short Form 36 Health Survey (SF-36), and disease-specific PRO measures, such as the 22-item Sino-Nasal Outcome Test (SNOT-22), have been used to evaluate changes in health-related symptoms of chronic rhinosinusitis (CRS).⁷ However, these PRO measures were not specifically developed for patients with CRSwNP.⁷ Recall periods in existing tools (eg, SNOT-22) might be too long to capture daily symptom variation,^8
-10 potentially overlooking important day-to-day changes in symptoms induced by medical therapy.^11
-13 A literature review of existing instruments indicated a gap for a novel, brief, daily PRO questionnaire developed specifically in CRSwNP to track changes in symptoms and impacts during clinical trials.¹⁰

The Nasal Polyposis Symptom Diary (NPSD), a new, content-valid daily symptom questionnaire, was developed to capture PROs for use as efficacy endpoints to evaluate the clinical benefit of investigational therapy for patients with severe CRSwNP.¹⁰ The NPSD assesses symptoms, symptom impact on HRQOL, and medication compliance. In accordance with the US Food and Drug Administration guidance on PRO measures for use in medical product development to support labeling claims,¹⁴ qualitative research indicated that the NPSD was easy to use and measured key relevant and disturbing symptoms from the perspective of patients with severe CRSwNP.^10,15 We aimed to complete the NPSD evaluation by assessing its psychometric properties: reliability, construct validity, ability to detect change, and meaningful change thresholds (MCTs). This manuscript focuses on 4 NPSD-derived symptom scores measured in a Phase III trial: a summary score of the 8 symptom items (Total Symptom Score [TSS]) and 3 single-item symptom scores (Nasal Blockage Score [NBS], Nasal Congestion Score [NCS], and Difficulty with Sense of Smell Score [DSS]). These scores were selected based on their prevalence in published literature^16
-20 and their reported prevalence and relevance by patients with CRSwNP interviewed for the development of the NPSD.¹⁰

Methods

Data Collection

Patients with severe CRSwNP were from a randomized, double-blind, placebo-controlled, multicenter Phase III clinical trial (OSTRO, NCT03401229) that evaluated the efficacy and safety of benralizumab.¹⁵ The target population consisted of adults with severe, symptomatic NP despite adherence to intranasal corticosteroid (INCS) therapy. Eligible patients were randomized 1:1 to benralizumab 30 mg or placebo subcutaneously every 4 weeks for the first 3 doses and every 8 weeks thereafter up to Week 56. Screening and enrollment details are available in the Supplemental Methods.

The NPSD is a brief PRO developed to capture patient-reported experiences with key CRSwNP symptoms (8 items) and impact on HRQOL over the past 24 hours (2 items) (Figure 1).¹³ INCS compliance (yes/no), added to monitor medication adherence in OSTRO, is not part of this analysis. The TSS is the sum of the first 8 item scores that evaluate NP symptoms (scored on a 4-point scale [0—none, 1—mild, 2—moderate, 3—severe]), with a resulting score from 0 to 24. To summarize patients’ day-to-day experiences and avoid selecting days with extreme variability in symptom severity, aggregated biweekly baseline scores were calculated as the average of responses from Day −13 to Day 1 from all non-missing daily scores. Biweekly scores missing 8 or more daily responses were considered missing. Acceptability, or willingness by patients to complete the NPSD, was assessed via completion rates for two 14-day periods (at screening and at baseline), calculated as the number of completed diary entries divided by the number of expected entries. The range of patient responses on each baseline biweekly mean NPSD single-item score was evaluated for floor (≥25% of patients responding at the most severe end of the response scale) and ceiling (≥25% of patients responding at the least severe end of the response scale) effects.

Figure 1.

Nasal Polyposis Symptom Diary (NPSD) items and scores.

Additional PRO measures analyzed included the SNOT-22, a CRS-specific assessment that captures patient-reported physical problems, functional limitations, and emotional consequences of CRS,^8,21 SF-36 version 2 (SF-36v2), a self-report survey of functional health and well-being²²; Patient Global Impression of Severity (PGI-S) and Patient Global Impression of Change (PGI-C), which measure patient-perceived overall symptom severity and change in symptom severity in response to treatment, respectively; and the Asthma Control Questionnaire 6 items (ACQ-6, administered only to those with comorbid asthma) to assess 5 key asthma symptoms and bronchodilator use over the previous week.²³

Analyses

Psychometric analyses followed regulatory guidelines¹⁴ using blinded screening and baseline data from OSTRO pooled treatment arms. Reliability and validity analyses were performed on the modified full analysis set (mFAS), which included all randomized patients and those who failed screening due to less severe symptoms or HRQOL impact (Supplemental Methods). Ability to detect change from baseline to Week 16 and MCT analyses were based on the full analysis set (FAS) of randomized patients.

Reliability

Reliability was evaluated for internal consistency—extent to which an instrument yields consistent and reproducible estimates of an expected score²⁴—by examining baseline biweekly mean NPSD scores using Cronbach’s coefficient alpha values. Test-retest reliability—instrument stability over time—was assessed using intraclass correlation coefficients (ICCs) between biweekly mean NPSD scores from the screening and baseline periods in 2 groups of patients who had stable symptoms between the 2 periods.

Construct Validity

Convergent validity—demonstration of logical relationships between an instrument and other measures or patient characteristics¹⁴—was examined using Spearman-ranked correlation coefficients between baseline biweekly mean NPSD scores and baseline mean scores from established PRO (SNOT-22, SF-36v2, PGI-S, and ACQ-6) and clinical (endoscopic Nasal Polyp Score [NPS], and Lund-Mackay scores) measures; correlations were expected to be moderate (>.3 to ≤.5) to high (>.5) with related measures (SNOT-22, PGI-S, and endoscopic NPS) and moderate to low (≤.3) with less related measures (SF-36v2, ACQ-6, and Lund-Mackay).²⁵ Convergent validity was also examined across pairs of NPSD and SNOT-22 items, with expected correlations ≥.5 for conceptually related items.²⁶ Known-group validity looked for logical patterns of differences in baseline biweekly mean NPSD scores across subgroups of patients with known symptom severity based on baseline PGI-S score (5 categories), worst baseline endoscopic NPS (4 categories), or history of NP surgery (yes, no), using two-sample t-test or one-way analysis of variance (ANOVA).

Ability to Detect Change

The ability to detect change over time—magnitude and direction of changes in scores reflecting changes in patients’ health¹⁴—was assessed by comparing changes from baseline to Week 16 in biweekly mean NPSD scores across groups of patients who experienced changes during the same period in mean PGI-S and PGI-C scores using 1-way analysis of covariance (ANCOVA). Ability to detect change was also evaluated by Spearman correlations with changes in endoscopic NPS and SNOT-22 total score.

Determination of Clinically Meaningful Change Thresholds

Thresholds for meaningful within-patient changes were estimated using anchor- and distribution-based methods, supplemented with empirical cumulative distribution function (eCDF) and probability density function (PDF) plots. For the anchor-based approach, medians of changes from baseline to Week 16 in biweekly mean NPSD scores were calculated for different combinations of categories of change from baseline to Week 16 in PGI-S and PGI-C mean scores (Supplemental Figure S1) to select the anchor that best differentiated patients who improved meaningfully from those who did not. Spearman rank correlations between changes in NPSD and PGI-S and PGI-C scores verified each anchor’s appropriateness. Two distribution-based approaches—one-half of a standard deviation (½ SD) and 1 standard error of measurement (1 SEM)—assessed baseline NPSD scores variability. Estimates derived from anchor- and distribution-based methods, as well as eCDF and PDF plots, were examined for convergence to a single threshold value representing meaningful within-patient improvement.

Results

Patient Population Characteristics

Of 969 patients screened for eligibility in OSTRO, 518 were retained in the mFAS population, including randomized patients (FAS, n = 410) and patients who failed screening. Most patients from the mFAS population were male (65.8%; 341/518), middle-aged (mean [SD] age, 50.3 [13.4] years), resided outside the United States (76.4%; 396/518), and reported a history of asthma (65.8%; 311/473) and NP surgery (72.9%; 345/473).

Acceptability, Baseline Values, and Range of Responses

The NPSD completion rates were 83.8% at screening and 84.1% at baseline (mFAS), indicating good acceptability, and were similar to the baseline completion rates of other PRO measures (Table 1). Baseline mean and median values of NBS, NCS, and DSS were toward the upper end of the possible range of values (>2.4; range, 0-3), as was the case for the TSS (>15; range, 0-24; Table 1). Baseline scores for other PRO measures were generally in the middle of their range of values, except for PGI-S scores (mean 4.39; range, 0-5; Table 1), indicating that the mFAS population exhibited a wide range of symptom severity, despite elevated NPSD and PGI-S scores.

Table 1.

Summary Statistics of NPSD and Other PRO Scores at Baseline, mFAS (N = 518).

	Completion rate	Count	Possible range	Mean (SD)	Median (25th, 75th quartiles)	Minimum	Maximum
Biweekly NPSD scores^a
NBS	84.1	458	0-3	2.46 (0.63)	2.71 (2.00, 3.00)	0.14	3.00
NCS	84.1	458	0-3	2.47 (0.62)	2.71 (2.00, 3.00)	0.00	3.00
DSS	84.1	458	0-3	2.86 (0.39)	3.00 (3.00, 3.00)	0.85	3.00
TSS	84.1	458	0-24	15.51 (4.63)	15.35 (12.14, 19.07)	2.33	24.00
SNOT-22 total score	79.7	413	0-110	69.05 (19.45)	68.00 (54.00, 83.00)	30.00	110.00
SF-36 v2
Physical component	79.5	412	0-50	42.61 (8.85)	42.97 (36.43, 49.26)	22.15	63.53
Mental component	79.5	412	0-50	43.81 (11.41)	45.55 (35.69, 52.84)	13.76	66.48
PGI-S score	79.7	413	0-5	4.39 (0.73)	5.00 (4.00, 5.00)	1.00	5.00
ACQ-6 score^b	89.1	277	0-6	1.99 (1.24)	1.83 (1.17, 2.83)	0.00	5.83

Note. Greater scores represent more severe symptoms or disease except for SF-36v2, for which greater values indicate better health-related quality of life.

Biweekly means of daily symptom scores reported in patients with available records during the 2-week baseline period (Day −13 to Day 1), excluding patients with missing records from the calculations.

ACQ-6 administered only to patients with comorbid asthma (n = 311).

More than 25% of patients responded at the most severe end of the response scale for the NBS (38.2%), NCS (36.9%), and DSS (80.1%) (Supplemental Figure S2); such floor effects were expected given OSTRO recruitment criteria. Other items had more balanced distributions. No ceiling effects at the least severe end of the response scale were observed.

Reliability

The TSS demonstrated excellent internal consistency, with a Cronbach’s alpha of .88 (optimal values ≥.7).²⁷ The calculated ICCs between screening and baseline NPSD scores among 2 groups of patients with stable symptoms were consistently above .8, indicating excellent test-retest reliability (defined by ICC > .75²⁸; Table 2).

Table 2.

Test-Retest Reliability: Intraclass Correlation Coefficients (ICCs) Between NPSD Scores at Screening and Baseline Among Subgroups of Patients With Stable Symptoms Between These Periods, mFAS (N = 518).

	ICCs (95% CI)^a
	Stable endoscopic NPS^b	Stable SNOT-22 score^c
	n = 227	n = 131
NBS	0.831 (0.786, 0.867)	0.873 (0.824, 0.909)
NCS	0.828 (0.782, 0.865)	0.883 (0.838, 0.915)
DSS	0.910 (0.884, 0.930)	0.960 (0.944, 0.972)
TSS	0.918 (0.895, 0.936)	0.944 (0.921, 0.960)

ICCs calculated between NPSD scores at screening (Day −27 to Day −14) and baseline (Day −13 to Day 1).

Stability defined as having the same endoscopic NPS at Visit 1 (at Day −27 ± 7) and Visit 2 (at Day −7 ± 7).

Stability defined as having a change in SNOT-22 total score from Visit 1 (at Day −27 ± 7) to Visit 3 (Day 1) between −8.90 and 8.90, and having the same SNOT-22 nasal blockage symptom score at Visits 1 and 3.

Construct Validity

TSS was strongly (≥.5) correlated with SNOT-22 total score, moderately (0.3-0.5) and inversely correlated with SF-36v2 component scores (negative correlation coefficients consistent with greater SF-36v2 scores indicating better HRQOL vs greater NPSD scores indicating worse symptoms), and weakly (<.3) correlated with the endoscopic NPS, PGI-S, and ACQ-6 scores (Table 3).²⁵ NBS and NCS were moderately correlated with SNOT-22 total and PGI-S scores, supporting the validity to measure nasal symptom severity. The DSS correlated weakly with all other scores.

Table 3.

Construct Validity: Spearman Rank Correlation Coefficients Between Baseline NPSD and Other PRO and Clinical Measures, mFAS (N = 518).

Spearman rank correlation coefficients	Endoscopic NPS	SNOT-22 total score	SF-36v2 physical component^a	SF-36v2 mental component^a	PGI-S score	ACQ-6 score^b	Lund-Mackay score^c
Spearman rank correlation coefficients	n = 283	n = 413	n = 412	n = 412	n = 413	n = 277	n = 181
NBS	0.252	0.325	−0.177	−0.096	0.480	−0.008	0.073
NCS	0.292	0.354	−0.179	−0.127	0.452	−0.006	0.106
DSS	0.070	0.094	−0.096	−0.035	0.238	0.022	0.233
TSS	0.127	0.700	−0.439	−0.374	0.289	0.253	0.086

Note. Values with r < .3 represent no or weak correlations; bold values with .3 ≥ r < .5 represent moderate correlations (bold); bold values with r ≥ .5 represent strong correlations.²⁸

For SF-36v2 components, greater values indicate better health-related quality of life (explaining the inverse correlation represented by minus signs).

ACQ-6 was administered only to patients with comorbid asthma (n = 311).

Lund-Mackay scores were calculated only for patients with available nasal sinus CT scans (n = 181; range, 0-24).

Convergent validity was confirmed by strong correlations between NPSD and SNOT-22 scores that evaluated similar symptoms (nasal blockage, runny nose, nasal discharge, and difficulty with sense of smell/taste) or related symptoms (NPSD headache, facial pain, and facial pressure with SNOT-22 dizziness, ear pain, and facial pain/pressure) (Supplemental Table S1). Moderate correlations between NPSD runny nose, postnasal drip, headache, facial pain, and facial pressure and SNOT-22 reduced productivity, reduced concentration, frustrated/restless/irritable, sad, and embarrassed also indicated convergent validity.

All 4 NPSD scores increased significantly across groups of patients with increasing levels of symptom severity by PGI-S scores (very mild to moderate, severe, or very severe), demonstrating known-group validity (one-way ANOVA, P < .0001; Supplemental Table S2). NBS and NCS also increased significantly with increasing NP size (worst endoscopic NPS ≥ 2), whereas DSS and TSS increased minimally without reaching statistical significance. Notably, DSS was significantly associated with prior NP surgery.

Ability to Detect Change

The ability of NPSD scores to detect change over time was established through associations in changes in NPSD scores from baseline to Week 16 with changes from baseline to Week 16 in PGI-S scores, and with Week 16 PGI-C scores (one-way ANCOVA, P < .0001; Table 4). NBS scores improved by 1.00 point among patients whose PGI-S or PGI-C scores improved, whereas scores improved by only 0.25 and 0.38 points and remained unchanged among patients whose PGI-S or PGI-C scores did not change or worsened, respectively. The TSS improved by more than 5.8 points among patients whose PGI-S and PGI-C scores improved, whereas it improved minimally or worsened in patients whose PGI-S and PGI-C scores did not change or worsened.

Table 4.

Ability to Detect Change From Baseline to Week 16: Least Squares (LS) Mean (95% CI) Changes in NPSD Scores Across Subgroups of Patients With Changes in Scores of Other PRO Measures, FAS (N = 410).

	Least square mean (95% CI) changes in NPSD scores
	Categories of change in PGI-S score^a				Categories of PGI-C score^b
	Improvement	No change	Worsening	P-value^c	Improved	No change	Worsened	P-value^c
	n = 240	n = 113	n = 19	P-value^c	n = 235	n = 94	n = 43	P-value^c
NBS	−1.00 (−1.10, −0.90)	−0.25 (−0.39, −0.11)	−0.04 (−0.38, 0.30)	<.0001	−1.00 (−1.10, −0.91)	−0.38 (−0.53, −0.22)	0.04 (−0.18, 0.27)	<.0001
NCS	−0.99 (−1.09, −0.90)	−0.23 (−0.37, −0.10)	−0.01 (−0.34, 0.32)	<.0001	−0.99 (−1.08, −0.90)	−0.38 (−0.53, −0.24)	0.08 (−0.13, 0.30)	<.0001
DSS	−0.43 (−0.51, −0.35)	−0.07 (−0.19, 0.04)	0.06 (−0.22, 0.34)	<.0001	−0.43 (−0.51, −0.35)	−0.09 (−0.22, 0.04)	−0.04 (−0.22, 0.15)	<.0001
TSS	−5.88 (−6.40, −5.36)	−1.65 (−2.41, −0.90)	0.04 (−1.80, 1.88)	<.0001	−5.87 (−6.39, −5.35)	−2.61 (−3.43, −1.79)	0.64 (−0.57, 1.86)	<.0001

Note. Negative change values indicate symptom improvement and bold values indicate statistical significance.

Categories defined using changes in PGI-S scores from baseline to Week 16.

Categories defined using the PGI-C score at Week 16: improved category included patients who reported “Much better,” “Moderately better,” and “A little better” responses; no change category included patients who reported “About the same” responses; and worsened category included patients who reported “Much worse,” “Moderately Worse,” and “A little worse” responses.

P-values were calculated using one-way ANCOVA and were all statistically significant.

Strong correlations in change from baseline to Week 16 in NBS (r = .572), NCS (r = .582), and TSS (r = .691) scores and moderate correlations in change in DSS (r = .304) score with change in SNOT-22 total score confirmed the ability to detect change of these NPSD scores (Supplemental Table S3).²⁹ Changes in NBS, NCS, and TSS also correlated moderately (.3 ≥ r < .5) with changes in NP size (worst or total endoscopic NPS), which might have been limited over 16 weeks.

Determination of Clinically Meaningful Change Thresholds

Correlations between median changes from baseline to Week 16 in NPSD scores and PGI-S anchors were strong (NBS: 0.58; NCS: 0.60; and TSS: 0.59) to moderate (DSS: 0.34), exceeding the acceptable threshold of 0.3.²⁹ PGI-S was used as the primary anchor due to a greater correlation with NPSD score changes versus PGI-C (data not shown) and being less likely to have recall bias. The low ends of the 95% confidence intervals derived from the mean NPSD score changes for patients whose PGI-S scores did not change were −0.38, −0.36, −0.14, and −2.26 for the NBS, NCS, DSS, and TSS scores, respectively (Table 5). In comparison, the lowest median NPSD score changes in groups by PGI-S categories of improvements exceeded these lower confidence limits (NBS: −0.77; NCS: −0.74; DSS: −0.19; and TSS: −4.30; Table 5). These lowest median NPSD scores similarly exceeded baseline score variations thresholds identified from the distribution-based analyses (½ SD: 0.23, 0.25, 0.18, and 2.08; 1 SEM: 0.19, 0.20, 0.11, and 1.15 for NBS, NCS, DSS, and TSS, respectively). Clear differentiations were observed in eCDF and PDF curves between groups of patients who did not change PGI-S categories versus those who improved (data not shown).

Table 5.

Determination of Meaningful Change Threshold (MCT) in NPSD Scores: Summary of Anchor-Based Analyses, mFAS (N = 410).

	Median and mean (95% CI) NPSD changes^a among subgroups of patients with different combinations of PGI-S score change categories^b
	PGI-S change in ≥3 categories		PGI-S change in ≥1 category		PGI-S change in 1 or 2 categories		No change in PGI-S
	n = 58		n = 240		n = 182		n = 113
	Median	Mean (95% CI)	Median	Mean (95% CI)	Median	Mean (95% CI)	Median	Mean (95% CI)
NBS	−2.00	−1.78 (−2.00, −1.56)	−0.93	−1.00 (−1.11, −0.89)	−0.77	−0.75 (−0.85, −0.64)	0.00	−0.27 (−0.38, −0.16)
NCS	−1.95	−1.79 (−2.01, −1.57)	−0.93	−0.99 (−1.09, −0.88)	−0.74	−0.73 (−0.83, −0.64)	0.00	−0.25 (−0.36, −0.15)
DSS	−0.19	−0.75 (−0.98, −0.51)	0.00	−0.43 (−0.52, −0.34)	0.00	0.06 (−0.42, −0.24)	0.00	−0.08 (−0.14, −0.01)
TSS	−9.14	−9.35 (−10.68, −8.02)	−5.39	−5.91 (−6.51, −5.30)	−4.30	−4.81 (−5.41, −4.21)	−1.30	−1.62 (−2.26, −0.98)

Note. Negative change values indicate symptom improvement. Median and mean changes in NPSD scores were calculated among subgroups of patients with different combinations of PGI-S score categories of changes and compared with values from the subgroup of patients with no change in PGI-S score.

Changes in NPSD and PGI-S scores from baseline to Week 16.

Different combinations of PGI-S score categories of changes defined based on change from baseline to Week 16 in the PGI-S scores (5-point scale: 0—no symptoms, 1—very mild symptoms, 2—mild symptoms, 3—moderate symptoms, 4—severe symptoms, 5—very severe symptoms).

Final NPSD MCTs were established based on triangulating estimates from all streams of evidence and rounding conservatively to the nearest 0.5 unit: −1.0 for the NBS, NCS, and DSS, and −4.0 for the TSS (Figure 2), with negative values indicating symptom improvement. Recommended thresholds for NBS, NCS, and DSS exceeded the median change score for one 2-category shift on the PGI-S (from very severe at baseline to moderate at Week 16; −0.86, −0.93, 0.00, respectively), as well as for most 1-category shifts (from very severe to severe, severe to moderate, and moderate to mild; Supplemental Table S4). The recommended MCT for TSS also exceeded the median change score for the 2-category shift from moderate to no symptoms (−3.69), as well as two 1-category shifts (from very severe to severe [−3.51] and severe to moderate [−3.77]). These findings provided confidence that the MCT estimates reflect a magnitude of symptom improvement that is meaningful to patients.

Figure 2.

Triangulation approach to determine meaningful improvement threshold for Nasal Blockage Score (NBS).

Discussion

The NPSD is a fit-for-purpose daily PRO tool to measure variation in day-to-day experience among patients with CRSwNP in clinical trials. Following regulatory guidelines, interviews of patients with CRSwNP established the content validity of this novel PRO tool.¹⁰ The comprehensive psychometric analyses based on OSTRO data from over 500 screened patients demonstrated the reliability, construct validity, and ability to detect change in NPSD scores measuring single (NBS, NCS, and DSS) and combined (TSS) symptoms. Test-retest ICC values for NPSD scores ranged from 0.828 to 0.960, in line with values reported for other instruments such as the SNOT-22 test-retest reliability of 0.93 in patients with CRS.⁸ Using measures from more than 400 patients during the first 16 weeks of OSTRO, NPSD scores significantly detected changes in symptom severity, and meaningful improvement estimates were established at a median score change of −1 point for NBS (OSTRO co-primary endpoint), NCS, and DSS (key secondary endpoints), and −4 points for TSS (secondary endpoint). The conservative MCT estimates clearly identified and differentiated patients with symptom improvements from those experiencing no change, supporting the ability of these MCTs to identify meaningful improvements in patients’ reported symptoms.

In the qualitative analysis, two-thirds of patients considered nasal blockage and nasal congestion as distinct concepts,¹⁰ with “blockage” representing airway restriction and breathing difficulties, and “congestion” relating to sinus issues, mucus build-up, and runny nose.¹⁰ Based on patients’ symptom descriptions and a desire for patients to understand the questionnaire, NBS and NCS were included as separate items in the NPSD and separate single-item scores in this psychometric analysis. However, OSTRO data demonstrated that these 2 items were highly correlated. Consequently, including both NBS and NCS in the calculated TSS may have had a heavier bearing than less-related items. This is not unanticipated as PRO instrument development is an iterative process where older versions may be updated based on new evidence. Future exploration of a combined nasal blockage/congestion score as part of the TSS may be valuable. Alternatively, the importance of these 2 symptoms may justify keeping the 2 items separate to contribute extra weight to the TSS. The TSS and all 3 NPSD single-item scores were established as psychometrically acceptable. Future studies may further optimize the NPSD by exploring the complex relationship between these patient-reported symptoms.

The NPSD was developed for patients with CRSwNP to capture day-to-day experiences. Daily input reduces recall bias and detects small and rapid changes in symptoms, allowing a more precise recognition of onset of aspects such as therapeutic effect. The brevity and daily administration of the NPSD were proven suitable in a clinical study setting.¹⁵ This research provides the comprehensive evaluation of psychometric properties of single-item (NBS, NCS, and DSS) and combined measures (TSS) in patients with CRSwNP, with easily interpreted within-patient improvement thresholds estimated using anchor- and distribution-based methods. NPSD scores measuring the most common and relevant patient-reported symptoms were psychometrically robust.

Because of recruitment criteria for OSTRO, a potential weakness stems from some symptoms being rated severe by nearly all patients, including difficulty with sense of smell, and to a lesser extent, nasal blockage and congestion. As it is usual for recruitment criteria in severe patient trials to result in worst symptoms at baseline, we included screen-failed patients who had a broader range of symptom severity. Meaningful estimates for improvement were determined using OSTRO data; due to the baseline disease severity in these patients, applicability of the results may be limited in patients with less severe CRSwNP. Studies including patients with milder disease may allow for more comprehensive psychometric estimates.

In conclusion, the NPSD is a fit-for-purpose PRO tool, with well-determined MCTs, for measuring day-to-day variation in patient-reported experience of CRSwNP in clinical studies.

Supplemental Material

sj-docx-1-aor-10.1177_00034894231177769 – Supplemental material for Psychometric Validation and Meaningful Change Thresholds of the New Nasal Polyposis Symptom Diary

Supplemental material, sj-docx-1-aor-10.1177_00034894231177769 for Psychometric Validation and Meaningful Change Thresholds of the New Nasal Polyposis Symptom Diary by Vivian H. Shih, Ashley F. Slagle, Cristina Ivanescu, Giulio Flore, Oren Meyers, James Kreindler, Ubaldo J. Martin and Viktoria Werkström in Annals of Otology, Rhinology & Laryngology

Footnotes

Acknowledgements

The authors thank the patients who participated in the study and the study site personnel who executed it. Writing and editing assistance, including preparation of a draft manuscript under the direction and guidance of the authors, incorporation of author feedback, and manuscript submission, was provided by Emily Ruzich, PhD (IQVIA Consulting Services, Cambridge, MA, USA) and Catherine Champagne, PhD (Kay Square Scientific, Newtown Square, PA, USA). This support was funded by AstraZeneca. All authors have authorized the submission of their manuscript via third party and have approved the funding statement and declarations of conflicting interests.

Author Contributions

VHS, CI, GF, and OM participated in the design of the psychometric analysis plan, and analyzed and interpreted the data. AFS was consulted for interpretation of selected analysis results. All authors had access to the data, and participated in the development and review of the manuscript. All authors read and approved the final manuscript.

Declaration of Conflicting Interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: VHS, JK, and VW are employees and shareholders of AstraZeneca. AFS is an employee of Aspen Consulting, LLC, which received funding from AstraZeneca to complete the study. CI and GF are employees of IQVIA, which received funding from AstraZeneca to complete the study. OM was an employee of IQVIA at the time of the study. UJM was an employee and shareholder of AstraZeneca at the time of the study.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Funding for manuscript development was provided by AstraZeneca.

Ethics Approval

The OSTRO trial was conducted in accordance with the Declaration of Helsinki, International Conference on Harmonization of Technical Requirements for Registration of Pharmaceuticals for Human Use, Good Clinical Practice guidelines, and the ethics committee at each participating site. All patients provided written informed consent at the time of enrollment.

Data Availability

Data underlying the findings described in this manuscript may be requested in accordance with AstraZeneca’s data-sharing policy described at .

Supplemental Material

Supplemental material for this article is available online.

References

Fokkens

Lund

Hopkins

, et al. Executive summary of EPOS 2020 including integrated care pathways. Rhinology. 2020;58(2):82-111.

Koennecke

Klimek

Mullol

Gevaert

Wollenberg

. Subtyping of polyposis nasi: phenotypes, endotypes and comorbidities. Allergo J Int. 2018;27(2):56-65.

Alobid

Benítez

Bernal-Sprekelsen

, et al. Nasal polyposis and its impact on quality of life: comparison between the effects of medical and surgical treatments. Allergy. 2005;60(4):452-458.

Khan

Huynh

TMT

Vandeplas

, et al. The GALEN rhinosinusitis cohort: chronic rhinosinusitis with nasal polyps affects health-related quality of life. Rhinology. 2019;57(5):343-351.

Bachert

Bhattacharyya

Desrosiers

Khan

. Burden of disease in chronic rhinosinusitis with nasal polyps. J Asthma Allergy. 2021;14:127-134.

Mercieca-Bebber

King

Calvert

Stockler

Friedlander

. The importance of patient-reported outcomes in clinical trials and strategies for future optimization. Patient Relat Outcome Meas. 2018;9:353-367.

Rudmik

Hopkins

Peters

Smith

Schlosser

Soler

. Patient-reported outcome measures for adult chronic rhinosinusitis: a systematic review and quality assessment. J Allergy Clin Immunol. 2015;136(6):1532-1540.e2.

Hopkins

Gillett

Slack

Lund

Browne

. Psychometric validity of the 22-item sinonasal outcome test. Clin Otolaryngol. 2009;34(5):447-454.

Downie

Andersson

Rimmer

, et al. Symptoms of persistent allergic rhinitis during a full calendar year in house dust mite-sensitive subjects. Allergy. 2004;59(4):406-414.

10.

O’Quinn

Shih

Martin

, et al. Measuring the patient experience of chronic rhinosinusitis with nasal polyposis: qualitative development of a novel symptom diary. Int Forum Allergy Rhinol. 2022;12(8):996-1005.

11.

Lidder

Detwiller

Price

, et al. Evaluating metrics of responsiveness using patient-reported outcome measures in chronic rhinosinusitis. Int Forum Allergy Rhinol. 2017;7(2):128-134.

12.

Levy

Mace

Rudmik

Soler

Smith

. Low 22-item sinonasal outcome test scores in chronic rhinosinusitis: why do patients seek treatment? Laryngoscope. 2017;127(1):22-28.

13.

Chowdhury

Mace

Bodner

, et al. Does medical therapy improve SinoNasal outcomes test-22 domain scores? An analysis of clinically important differences. Laryngoscope. 2019;129(1):31-36.

14.

US Department of Health and Human Services. Guidance for industry. Patient-reported outcome measures: use in medical product development to support labeling claims. US Food and Drug Administration. 2009. Accessed July 18, 2022. https://www.fda.gov/downloads/drugs/guidances/ucm193282.pdf

15.

Bachert

Han

Desrosiers

, et al. Efficacy and safety of benralizumab in chronic rhinosinusitis with nasal polyps: a randomized, placebo-controlled trial. J Allergy Clin Immunol. 2022;149(4):1309-1317.e12.

16.

Dudvarski

Pendjer

Djukic

Janosevic

Mikic

. The analysis of clinical characteristics of the chronic rhinosinusitis: complicated and uncomplicated form. Eur Arch Otorhinolaryngol. 2008;265(8):923-927.

17.

Hox

Bobic

Callebaux

Jorissen

Hellings

. Nasal obstruction and smell impairment in nasal polyp disease: correlation between objective and subjective parameters. Rhinology. 2010;48(4):426-432.

18.

Dietz de Loos

Hopkins

Fokkens

. Symptoms in chronic rhinosinusitis with and without nasal polyps. Laryngoscope. 2013;123(1):57-63.

19.

Rix

Håkansson

Larsen

Frendø

von Buchwald

. Management of chronic rhinosinusitis with nasal polyps and coexisting asthma: a systematic review. Am J Rhinol Allergy. 2015;29(3):193-201.

20.

Bohman

Oscarsson

Holmberg

, et al. Relative frequencies of symptoms and risk factors among patients with chronic rhinosinusitis with nasal polyps using a case-control study. Acta Otolaryngol. 2018;138(1):46-49.

21.

Piccirillo

Merritt

Jr Richards

. Psychometric and clinimetric validity of the 20-Item Sino-Nasal Outcome Test (SNOT-20). Otolaryngol Head Neck Surg. 2002;126(1):41-47.

22.

Maruish

. User’s Manual for the SF-36V2 Health Survey. 3rd ed. QualityMetric Inc; 2011.

23.

Juniper

Svensson

Mörk

A-C

Ståhl

. Measurement properties and interpretation of three shortened versions of the asthma control questionnaire. Respir Med. 2005;99(5):553-558.

24.

Meyers

Gamst

Guarino

. Applied Multivariate Research: Design and Interpretation. Sage Publications, Inc; 2006.

25.

Cohen

. Statistical Power Analysis for the Behavioral Sciences. 2nd ed. Lawrence Erlbaum Associates; 1988.

26.

Cohen

. A power primer. Psychol Bull. 1992;112(1):155-159.

27.

Fayers

Machin

. Quality of Life. The Assessment, Analysis and Interpretation of Patient-Reported Outcomes. 2nd ed. Wiley; 2007.

28.

Fleiss

. The Design and Analysis of Clinical Experiments. Wiley Classics Library Edition. John Wiley & Sons, Inc; 1999.

29.

Coon

Cook

. Moving from significance to real-world meaning: methods for interpreting change in clinical outcome assessment scores. Qual Life Res. 2018;27(1):33-40.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.29 MB