Establishing survey methodology to capture long term population level patient reported outcomes (PROs) in a surgical population

Abstract

Understanding long-term patient outcomes (PROs) following surgery requires an efficacious survey methodology. Leveraging a statewide hernia surgery registry to establish a sampling frame, we conducted a 1-year post-operative survey using measures of patient-reported hernia recurrence (the Ventral Hernia Recurrence Inventory), pain (the PROMIS Pain Intensity 3a), and quality of life (the HerQLes scale). Our responsive design approach varied invitation and reminder contact modes and incentive offer across multiple design phases, with the goal of minimizing non-response bias and maximizing cost effectiveness. Outcomes included: contact and response rates (%); item non-response (%); and the association between reminders and incentive offer with response rates, respondent characteristics, and item non-response (%). Differences in demographic and clinical characteristics of respondents and non-respondents were investigated and adjusted using registry data. Of 7062 patients who received hernia surgery between January 2020 and March 2022, 6068 were sampled, 5645 were contacted (contact rate 93.0%), and 1816 responded to the survey (overall response rate 29.9%). Response rates by cohort were 42.3%, 32.5%, 25.2%, and 25.9%, with overall low item non-response. Response rates increased with number of reminders, but with diminishing returns over time; offer of postpaid incentive over no incentive did not significantly improve response rates or influence item non-response. Weighted respondents were comparable to the survey population. We illustrate a strategy to maximize response rate amongst surgical patients and evaluate the representativeness of long-term PROs using a sample-based registry, targeted multi-mode contact methods, and weighting adjustment methods.

Keywords

response rate nonresponse bias survey delivery method patient-reported outcome hernia surgery responsive survey design

Background

Collection of long-term patient reported outcomes (PROs) data following ventral hernia repair remains the Achilles heel of hernia repair quality improvement efforts. Long-term patient reported outcomes on recurrence and pain symptoms are integral to improve shared decision making between patients and their surgeons regarding pursuit of operative hernia repair,^1,2 yet such data require significant long-term follow-up and integration with claims data to collect quantitative PRO metrics. We are currently failing at this type of measurement.³ First, outcomes are not being measured for long enough: most outcomes data exist only in the short term (e.g., 30-days complications), despite the importance of the long-term patient experience for shared decision-making in preference-sensitive and quality-of-life surgeries.^4–7 Second, there are inherent disadvantages to the outcomes as measured currently: hernia recurrence, for example, is typically measured based on rates of re-repair from claims data,^8–10 without input from patients who don’t seek reoperations and who may have different experiences^5,10–13; or is measured using survey methodologies unequipped to evaluate the magnitude of nonresponse bias. The lack of long-term data for one of the most common ambulatory surgeries in Michigan,¹⁴ the United States,^15,16 and the world¹⁷ leaves surgeons at a disadvantage to deliver care in line with what matters to patients.¹

To meet this gap for long-term outcomes data with proper evaluations, the Core Optimization Hernia Registry (COHR) was founded on January 1, 2020, as a special project of the Michigan Surgical Quality Collaborative (MSQC).⁴ COHR builds upon a “culture” of clinical hernia registries globally,¹⁸ yet many other registries suffer from low rates of follow-up, retrospective and/or single institution design,^10,19 voluntary reporting requirements,^20,21 or a lack of generalizability to a US patient population.^1,4,21–23 COHR was established with two objectives: to expand statewide MSQC data to include prospective collection of hernia-specific process and outcome variables,⁴ and to introduce validated measures of PROs to study long-term hernia recurrence, pain, and quality of life. To achieve the latter, the MSQC-COHR launched an effort to capture validated and representative long-term PROs in a hernia repair population using responsive survey design and this paper describes these efforts.

The MSQC-COHR was a prime sampling frame within which to run a survey. The MSQC’s network of statewide clinical data collection provided a population-based cohort and the necessary infrastructure to standardize our process for replicability and maximize data quality. Further, the richness of auxiliary data on patients contained within the registry meant that, in analysis of survey results, we did not have to rely on typical techniques to manage blind non-response. As a result, our study enhances the literature of survey non-response amongst surgical patients and contributes to the methodology for long-term collection of PROs from surgical patients. Finally, our use of responsive survey design, where design characteristics were varied across multiple survey phases and key outcomes were constantly analyzed for successive adaptation,²⁴ was particularly relevant in the post-operative space where predictors of survey response and the effectiveness of various contact modes to achieve representativeness are understudied.^25–27

We describe our efforts to launch this survey initiative, conduct iterative refinement over time, and attain meaningful follow-up for patients in a statewide clinical hernia registry. We took particular interest in reducing non-response bias while also minimizing costs and ensuring internal and external validity of our survey.^28,29 In this work we present best survey practices for a surgical patient population and open the door for future efforts to study long-term PROs in other preference-sensitive surgeries, such as gynecology, orthopedics, and vascular.

Methods

Target population

Our target population consisted of all patients who underwent operative hernia repair in the State of Michigan. To study this population, we utilized data from the MSQC, a partnership between Blue Cross Blue Shield of Michigan/Blue Care Network, the American College of Surgeons (ACS), and hospitals within the state of Michigan.^30,31 MSQC operates a clinical quality improvement registry that utilizes the infrastructure of the ACS National Surgical Quality Improvement Program (NSQIP) to prospectively collect data from a diverse set of (as of 2023) 69 member hospitals in Michigan (https://msqc.org/), representing over 90% of all general and vascular surgical procedures performed across the state. These cases are then sampled using a proprietary algorithm to minimize selection bias while ensuring a population-based representative sample of all surgical cases performed in the state.^4,32 Registered nurses are trained as Surgical Clinical Quality Reviewers (SCQRs) to perform data abstraction of entire electronic medical records, including operative notes, which are regularly assessed to ensure validity and reliability.⁴ Data captured in the registry includes patient demographic information, perioperative information, and 30 and 90-days clinical and patient reported outcomes.³² Individual hospital and physician data cannot be identified in the MSQC registry.^30,31 Over 23,400 patients who underwent operative hernia repair in the State of Michigan since 2019 have been captured in the MSQC.

Beginning in 2020, COHR expanded statewide MSQC data to prospectively collect operative characteristics, patient comorbidities, and postoperative surgical complications linked to hernia-relevant Current Procedural Terminology (CPT) codes^30,31 as well as hernia-specific process and outcome quality variables such as hernia location, size of hernia, mesh use and type, and surgical approach⁴ (Appendix). The capture of these data elements provides a more complete picture of hernia repair within the State of Michigan. Since its inception, the COHR has captured nuanced data on over 15,400 hernia repair patients from across the entire MSQC network. Our study population thus consisted of all operative hernia repair patients in the State of Michigan captured in the MSQC-COHR since 2020, when hernia-specific data collection began.

For our study’s sampling frame, we targeted all patients captured in the MSQC-COHR database over 18 years of age and who underwent open, laparoscopic, or robotic ventral and incisional hernia repair as determined by CPT codes (Appendix) between January 2, 2020, and March 31, 2022 (hereafter “COHR patients”). We sequentially defined four operative cohorts in order to target with surveys all eligible MSQC-COHR patients at 1-year post-operation. Through this design, each of our cohorts represented the extant MSQC-COHR population, thus achieving a population-based representative sample of hernia patients in the State of Michigan.^29,33

Data quality in the context of measurement of PROs

Strategies for the measurement of PROs has been inconsistent across clinical specialties. In specialties where long-term follow-up is standard (e.g., cardiology, orthopedics, bariatrics), measurement has been integrated into clinical workflow.^2,34,35 However, in hernia, follow-up beyond the immediate recovery period is uncommon. Hernia patients who experience operative complications or recurrence may decide not to re-present,¹³ potentially due to obstacles to appearing in clinic like financial cost, distance to travel, and time required.^35,36 If patients do seek follow-up, they may not present to their initial operating surgeon or institution.²¹ In this context, surveys can remotely collect long-term PROs from surgical patients in a more cost-effective manner,^{25,26,35–38} even into the extended post-operative period when follow up is increasingly less likely despite an increasing risk of hernia recurrence over time.¹ Finally, survey methodology provides us with the means to evaluate the representation and measurement properties of estimates simultaneously under cost constraints.^26,35,36,38

Responsive survey design

Following the principles of responsive design described by Groves and Heeringa (2006), we identified pre-hoc multiple design features most likely to influence our key outcomes of non-response bias and cost-effectiveness: survey contact mode, number of reminders, and offer of incentive. Thus, across four operative cohorts drawn sequentially from the MSQC-COHR, we tested specific combinations of design characteristics in successive phases of our responsive survey design.²⁷ Across phases, we were able to evaluate the trade-offs of each strategy between comparable cohorts and iteratively adapt our survey approach.²⁴

Our first design feature of interest was survey contact mode. In the context of a large registry, an electronic mode such as email or short message service (SMS) is appealing for being less time and cost-intensive to administer (particularly with regard to full time equivalent (FTE) staffing costs), and for the potential to automatically incorporate results into patient charts.³⁹ This choice of survey contact mode was also encouraged by literature on the post-operative measurement of PROs, as prior studies have established both the effectiveness^25,35,37 and non-inferiority^26,39 of electronic modes to paper (“snail mail”) contact modes when distributed at 1-year post-operation to a surgical cohort. However, due to potential differences in contact information available in the registry, the benefits to participation of multimodal strategies, and evidence of the effectiveness of telephone calls for PRO collection amongst surgical patients,^25,35,37 we elected to test three modes of contact to maximize sample representativeness and decrease coverage error: email, SMS, and phone call.^1,40–42 In addition, individuals were invited to complete surveys by either (a) email, (b) email and SMS, or (c) telephone based on mode of contact available in the patient chart and based on the design features of the survey phase (Appendix). For (b), individuals received concurrent invitations through each modality. Electronic (email, SMS) invitations included an individually customized link to complete the survey via Qualtrics (Qualtrics, Provo, UT). For (c), phone invitations to complete a survey were delivered verbally by callers, and callers then read aloud the Qualtrics survey word-for-word.

Second, we tested across phases the number of maximum contact attempts (i.e., reminders), following prior literature.^13,39,43 Every individual who did not respond to the first invitation received at least one reminder, and all individuals with telephone numbers were eligible for phone call reminders in addition to the original mode of contact.⁴³ For all patients, the number of contact attempts and their cooperation (e.g., refusal) were tracked by team members.⁴⁴ Team members varied the time and day of the week at which they conducted phone calls to maximize capture of patients. Over our 26-months survey period, our staff consisted of one survey manager (JY), one lab manager (AH), one data manager (YC), one data analyst (AK), and six total interviewers. We considered cost effectiveness in terms of the FTE (i.e., staffing) cost of telephone contact mode and number of telephone reminders.

Our third survey design feature of interest was incentive provision. Incentive provision is widely understood to increase survey participation,⁴⁵ including for surgical patients,²⁵ and thus reduce nonresponse bias^33,46 that might arise when there are differences in survey responses between respondents and non-respondents as multiplied by the non-response rate.⁴⁵ We tested the effect of provision of a post-paid incentive on non-response bias and cost effectiveness in two phases.

Survey cohorts and design phases

Detailed information on each cohort and corresponding survey design phase appears in the Appendix. Cases in the Pilot cohort (P1) were drawn from eight participating pilot hospitals (the first to opt to participate in MSQC-COHR) across five sequential groups defined using date of operation. As part of Phase 1 of our responsive survey design, individuals were contacted according to the contact methods above. Non-respondents received 5-6 additional reminders via original method of invitation and, if phone number was available, two reminder phone calls over the next 5 months until response was achieved, or the cohort was retired (Table A2). Hereafter, respondents to the P1 effort are referred to as P1a.

Following responsive design principles, we designed a two-phase sampling approach for non-response.²⁴ Patients in the P1 cohort who failed to respond by the final contact wave were rolled into a sub-cohort (P1b) to test Sub-Phase 1b, which additionally offered a postpaid financial incentive (a $10.00 gift card to the patient’s choice of either Starbucks or Amazon) for survey completion. By focusing on a sub-sample of non-respondents from P1, we responsively tested a more cost-intensive survey approach across a smaller sample.²⁴ In October 2022, patients in P1b received a new invitation to complete the same survey with the addition of the offer of the incentive. P1b patients received two reminders of both invitation and incentive via email and/or SMS over a 3-week period; phone reminders were not conducted for the P1b cohort.

Three cohorts were subsequently drawn which captured patients from all MSQC-COHR hospitals. Our second cohort (Cohort 1, C1) captured patients who were at least 1-year post operation in July 2022 (Table A3). For Phase 2 of our responsive survey design, contact modes were the same as in Phase 1, but—following responsive design principles—the total number of reminders was reduced given evidence of diminishing returns to later contact waves. The third cohort (Cohort 2, C2) captured patients who were at least 1-year post-operation in November 2022 (Table A4). In Phase 3 of our responsive survey design, we chose to offer the same financial incentive of a $10.00 gift card to Starbucks or Amazon as piloted for P1b, but this time at first invitation. Individuals were contacted according to method available as in prior phases; however, following responsive design, phone-only individuals were not contacted as we determined this method to be the most cost-intensive with the lowest response rate. The fourth cohort (Cohort 3, C3) captured patients who were at least 1-year post-operation in April 2023 (Table A5). For Phase 4, we utilized the same survey design features as in Phase 3 minus the offer of an incentive and the release of a fifth reminder.

Survey questionnaire

The survey used one generic and two hernia-specific validated patient reported outcomes measures (PROMs)⁴⁷: (a) the Ventral Hernia Recurrence Inventory (VHRI)^1,36; (b) the Patient-Reported Outcome Measurement Information System (PROMIS) Pain Intensity 3a^23,48; and (c) the Hernia-Related Quality-of-Life (HerQLes) scale.⁴⁹ These specific instruments measure patient-reported hernia recurrence, pain, and abdominal function and quality of life outcomes to generate a more wholistic view of the patient experience after surgery.^1,10 With these three PROMs, the survey included a total of 18 questions (Table A6), of which one was potentially sensitive. The questionnaire included an informed consent statement in the introduction page that participants were required to acknowledge prior to proceeding. The questionnaire was available only in English and individuals with language barriers determined at the time of interview were considered to be non-respondents. More information on the survey questionnaire is available in the Appendix.

Case definitions

We followed case definitions in accordance with the American Association for Public Opinion Research (AAPOR) disposition guidelines.⁵⁰ More information can be found in the Appendix. All surveys that were non-deliverable were excluded from all analyses except delivery rate.

Statistical analysis

Across the entire study, we evaluated the following outcomes: (a) contact rate; (b) follow-up time; (c) response rate; and (d) item non-response. Follow-up time was calculated amongst respondents as the difference in months between an individual’s date of hernia operation and their date of survey response and was measured to determine how close to 1-year post-operation we were measuring PROs, in order to better understand the risk of hernia recurrence with time. Item non-response was defined as complete survey response (100% completion) or partial survey response (≥75% completion of all PROMs).

With regard to our responsive design evaluation, our analysis focused on key outcomes (non-response bias and cost effectiveness) that we anticipated would vary depending on three key design characteristics (contact mode, number of reminders, and incentive offer). We investigated the following outcomes related to survey contact mode: (a) contact information availability in the registry; (b) differences in age, sex, race, ethnicity, and hospital locality (urban/rural) by (i) mode of contact available (i.e., sample representativeness by mode) and by (ii) mode of response (amongst respondents who were contacted via both email and SMS); and (c) the presence of mode effects. For (b), by comparing differences in patient characteristics by contact mode available, we evaluated whether our choice of contact mode across phases affected sample composition and noncoverage. For (c), we were additionally interested whether the presence of an interviewer for the telephone mode influenced nonresponse on the potentially sensitive question, compared to self-administered modes (email and SMS).^51,52

We next studied the effect of number of reminders on (a) response rates (i.e., did more reminders increase response rate?), (b) characteristics of respondents (i.e., sample representativeness), and (c) item non-response. For (b), we evaluated differences between early and late respondents to investigate whether additional reminders captured individuals who responded later and were thus more similar to non-respondents.⁴⁵ Early respondents were defined as those respondents across all cohorts who responded to the initial survey invitation (contact wave 1). Late respondents were defined as those respondents across all cohorts who responded at any point after the first reminder (contact wave 2+). For (c), item non-response was evaluated to determine whether individuals who responded to later reminders submitted lower-quality data than earlier respondents.

Our last design feature was incentive offer. We evaluated the association between incentive offer and (a) response rate, (b) respondent characteristics, and (c) item non-response.^45,53 By comparing characteristics of respondents who were and were not offered incentives (i.e., respondents in Phases 1a vs 1b, and in Phases 3 vs 4), we evaluated in (b) whether incentive offer affected non-response bias. Finally, to address our key outcome of cost-effectiveness, we discussed the relative cost and FTE requirements (in terms of survey mode, number of reminders, and incentive offer) of each design phase.

In our evaluation of overall non-response, we compared the characteristics of respondents, non-respondents, the survey sample, and the MSQC-COHR population.³³ To evaluate differences, we used Chi-squared tests for categorical variables and Student’s t-test for continuous variables. The t-test of difference in proportions was also used to determine differences between groups. A p-value <0.05 was considered statistically significant in all analyses. To reduce non-response bias and improve external validity,²⁹ we developed a two-part weighting scheme. This scheme first consisted of non-response weights that adjusted for differences between survey respondents and the survey sample⁵⁴; these were constructed using binomial logistic regression in which respondent (vs non-respondent) status was regressed against relevant explanatory patient demographics, hernia characteristics, surgical characteristics, and patient comorbidities. Individuals (respondents and non-respondents) with missing explanatory variables (e.g., age, hernia size) were included in the weighting scheme, which coded for missingness. Our scheme secondly consisted of raking weights, which were calculated using iterative proportional fitting (Stata’s ipfweight)⁵⁵ to balance the weighted respondent sample totals across the known distribution of the existing MSQC-COHR population.^54,56 A set of covariates for the raking weights was defined pre-hoc based on correlation with nonresponse, noncoverage, and survey outcomes and then adjusted to further minimize differences between the weighted and unweighted samples^57,58: sex, age, race, surgical priority, and ASA Class. Major outliers were trimmed using the ipfweight maximum threshold function,⁵⁵ with an upper threshold value of 5.⁵⁹ For all estimates, we implemented Taylor-linearized variance estimation using Stata’s svy command.⁶⁰

This quality improvement study was exempt from regulation by the University of Michigan Institutional Review Board (HUM00091060). All statistical analysis was performed using Stata version 17.0 (StataCorp, Inc., College Station, TX). The STROBE guidelines were used to report this study.⁶¹

Results

Study population

We captured 7062 patients who received hernia repair in Michigan between January 2, 2020, and March 31, 2022. By cohort, the number of patients included were: P1, n = 555; P1b, n = 262 (eligible P1 non-respondents); C1, n = 2472; C2, n = 1768; and C3, n = 2267 (Table 1).

Contact rate

The overall contact rate was 93.0% (5645 out of 6068; Table 1). This varied by cohort, as design phases that omitted telephone contact (e.g., Cohort 3) had higher contact rates than design phases that included telephone-only individuals (e.g., Cohort 1). Across all cohorts, of the 419 individuals for whom surveys were non-deliverable, 259 were from the email/SMS group, 159 were from the phone only group, and 1 was from the email only group.

Table 1.

Sample size (n), disposition, and contact rate by survey cohort and overall.

Survey cohort	Sample size (n)	Respondents + refusals	Not yet contacted	AAPOR Disposition Code			Contact rate^a
Survey cohort	Sample size (n)	Respondents + refusals	Not yet contacted	“Non-response”	“Non-interview”	“Not eligible”	Contact rate^a
Pilot cohort (P1)	555	284	0	190	81	0	85.41% (474/555)
Cohort 1 (C1)	2472	1041	0	1107	320	4	86.89% (2148/2472)
Cohort 2 (C2)	1768	333	450	967	18	0	98.63% (1300/1318)
Cohort 3 (C3)	2267	446	544	1277	0	0	100.00% (1723/1723)
Overall	7062	2104	994	3541	419	4	93.02% (5645/6068)

^aContact rate calculated using AAPOR CON1 formula.

Follow-up time

Across all cohorts, median follow-up time was 16 months (IQR 14–18 months) (Table 2). Follow-up time was longest for P1b, a sub-cohort of P1 that received additional survey invitations; patients responded on average over 2 years after their operation date (median 26.7, IQR 24–29 months, maximum 34 months). In later cohorts (C-3), the median response time was <15 months (IQR 14–16 months).

Table 2.

Average follow-up time in months by survey cohort and overall.

	Survey cohort					Overall
	P1a	P1b	C1	C2	C3	Overall
Mean follow-up time, months (SD)	19.66 (2.18)	26.65 (3.89)	17.99 (3.97)	14.74 (1.56)	14.98 (1.58)	16.96 (3.62)

SD = standard deviation.

Survey response rates

Across all cohorts, survey data was obtained from 1816 of 6068 eligible individuals who received at least one survey request, with an overall response rate of 29.9% (Table 3). The largest portion of nonresponse across all cohorts was attributable to refusals (n = 4252) versus non-contact (n = 419). The response rate for P1a was 38.2% (212 out of 555). Of 343 non-respondents from P1, 262 were rolled into the pilot incentive sub-cohort (P1b), which had a response rate of 8.8% (23 out of 262). Thus, the overall response rate for P1 was 42.3% (235 out of 555). Response rates for cohort 1 (C1), cohort 2 (C2), and cohort 3 (C3) were 32.5% (803 out of 2472), 25.2% (332 out of 1318), and 25.9% (446 out of 1723), respectively.

Table 3.

Response rate (%) by survey cohort and overall.

	Survey cohort					Overall
	P1a	P1b	C1	C2	C3	Overall
Response rate^a	38.20% (212/555)	8.78% (23/262)	32.48% (803/2472)	25.19% (332/1318)	25.89% (446/1723)	29.93% (1816/6068)

^aResponse rate calculated using AAPOR RR2 formula.

Item non-response: Survey completion

Of those individuals who initiated their surveys, 1645 fully completed their surveys (100% completion rate = 90.6%), with another 41 individuals partially completing their surveys (≥75% completion rate = 92.8%). 130 individuals failed to achieve at least 75% completion. Across all cohorts, item response rate dropped with survey progression (Figure 1).

Figure 1.

Percent of all survey respondents achieving 100% completion of Patient Reported Outcomes Measures by question.

Mode of contact and mode of response

The majority of individuals (72.6%) listed both email and phone, while 26.4% listed telephone only and 1% listed email only (Table 4). Mode of contact appeared to affect sample representativeness, as individuals with both email and phone (mean age 53.8 years) and with email only (mean age 51.5 years) were on average younger than individuals with phone only (mean age 56.7 years; p < 0.05) (results not shown). Individuals who received surgery at rural hospitals were 6.4 percentage points more likely than individuals who received surgery at urban hospitals to list phone only (p < 0.05). Non-White individuals were 4.3 percentage points more likely than White individuals to list phone only (p = 0.006). There were no differences between Hispanic and non-Hispanic individuals (p = 0.285).

Table 4.

Description of contact method available in sample and corresponding contact and reminder modes.

Contact method available	% In the sample	Initial contact mode	Reminder mode
Only email contact information	0.99%	Email	Email
Only telephone numbers	26.42%	Telephone	Telephone
Email and telephone numbers	72.59%	Email and SMS	Email, SMS, telephone

With regard to mode of response, we considered only those individuals who had a primary choice, i.e., received invitations via both email and SMS and chose their method of response. The majority of these respondents (>54%) utilized email (results not shown). Across all cohorts, those who chose to respond via SMS were on average 2.9 years younger than those who chose to respond via email (p < 0.001). Women also demonstrated a stronger preference for SMS than men (p < 0.05). White individuals were more likely to respond via email while Black individuals were more likely to respond via SMS (p < 0.001). No differences were observed by Hispanic ethnicity. Patients who received surgery at rural hospitals demonstrated a stronger preference for email than those who received surgery at urban hospitals (p < 0.05). We did not find evidence of mode effects in patterns of response to potentially sensitive survey question 13 (results not shown).

Reminder effectiveness: Impact of reminders on response rates, respondent characteristics, and item non-response

Across all cohorts, 43.1% of respondents responded to the initial survey invitation, with a total of 66.0% of all respondents having responded by the end of the first reminder wave (Table 5). Figure 2 depicts the trend across invitation and reminders in number of respondents and response rate by cohort. Additional reminders were associated with increases in response rate over time.

Table 5.

Number (n) of survey respondents and response rate (%) per contact wave, by survey cohort and overall.

Contact wave	Survey cohort (n, response rate)								Total respondents
Contact wave	P1 respondents		C1 respondents		C2 respondents		C3 respondents		Total respondents
Invitation	90	16.22%	333	13.47%	139	10.55%	221	12.83%	783	43.12%
1st reminder	42	23.78%	213	22.09%	65	15.48%	96	18.40%	416	66.02%
2nd reminder	19	27.21%	77	25.20%	50	19.27%	64	22.11%	210	77.59%
3rd reminder	23	31.35%	44	26.98%	38	22.15%	39	24.38%	144	85.52%
4th reminder	15	34.05%	67	29.69%	28	24.28%	26	25.89%	136	93.01%
5th reminder	12	36.22%	n/a	n/a	12	25.19%	n/a	n/a	24	94.33%
6th reminder	9	37.84%	n/a	n/a	n/a	n/a	n/a	n/a	9	94.82%
Phone reminder^a	2	38.20%	69	32.48%	n/a	n/a	n/a	n/a	71	98.73%
P1b 1st reminder	16	41.08%	n/a	n/a	n/a	n/a	n/a	n/a	16	99.61%
P1b 2^nd reminder	7	42.34%	n/a	n/a	n/a	n/a	n/a	n/a	7	100.00%
Total	235	42.34%	803	32.48%	332	25.19%	446	25.89%	1816	100.00%

^aPhone reminders for P1 and C1 only.

Figure 2.

Total number of survey respondents and response rate by survey cohort by contact wave. Respondents to the 6^th reminder of P1a group 1 are grouped with 5^th reminder respondents from P1a groups 1-5. Respondents to either P1a phone reminder are grouped under “phone reminder”.

All cohorts captured over 40% of their respondents with invitation only (Table 6). Addition of the first reminder was associated with capture of over 60% of all respondents across all cohorts. Compared to late respondents, early respondents were more likely to be older, to have Medicaid insurance, to have received mesh, and to have a diagnosis of diabetes (all p < 0.05) (Table 7). Early respondents were less likely than late respondents to have a diagnosis of a chronic condition requiring steroids (p < 0.05). Number of reminders did not influence item non-response.

Table 6.

Distribution of early and late respondents by survey cohort and overall.

	Early respondents^a, n (%)	Late respondents^b, n (%)	Total number of respondents, n
Pilot cohort	90 (42.45%)	42 (62.26%)	103
Cohort 1	333 (41.47%)	213 (68.00%)	257
Cohort 2	139 (41.87%)	65 (61.45%)	128
Cohort 3	221 (49.55%)	96 (71.07%)	129
Overall	783 (43.12%)	416 (66.02%)	617

^aEarly respondents responded to contact wave 1 (invitation).

^bLate respondents responded to contact wave 2+.

Table 7.

Characteristics of early and late respondents (all survey cohorts).

	Early respondents^a, n (%)	Late respondents^b, n (%)	p value*
	(n = 783)	(n = 1033)	p value*
Patient demographics
Female sex	374 (47.77)	500 (48.40)	0.7902
Mean age, years (SD)	56.07 (13.58)	54.56 (14.10)	0.0217*
Race
White	662 (84.55)	852 (82.48)	0.241
Black	91 (11.62)	137 (13.26)	0.296
Other	30 (3.83)	44 (4.26)	0.646
Ethnicity (Hispanic)	15 (1.92)	23 (2.23)	0.6479
Insurance
Private	559 (71.39)	771 (74.64)	0.121
Medicare	186 (23.75)	221 (21.39)	0.232
Medicaid	31 (3.96)	24 (2.32)	0.043*
Uninsured	2 (0.26)	6 (0.58)	0.309
Other	5 (0.64)	11 (1.06)	0.342
Hernia-specific variables
Hernia location
Epigastric	142 (18.59)	231 (23.03)	0.023*
Umbilical	492 (64.40)	629 (62.71)	0.465
Infraumbilical	54 (7.07)	68 (6.78)	0.812
Suprapubic	16 (2.09)	16 (1.60)	0.444
Missing	60 (7.86) (n = 764)	59 (5.88) (n = 1003)	0.100
Previous hernia repair (Y/N)	122 (15.95) (n = 765)	187 (18.44) (n = 1014)	0.170
Hernia size
<2 cm	163 (30.81)	254 (34.99)	0.121
2–6 cm	283 (53.50)	351 (48.35)	0.072
>6 cm	83 (15.69) (n = 529)	121 (16.67) (n = 726)	0.642
Mesh use (Y/N)	627 (81.01)	764 (75.05)	0.003*
Mesh use (Y/N)	(n = 774)	(n = 1018)	0.003*
Myofascial release (Y/N)	50 (6.84) (n=731)	74 (7.76) (n=953)	0.474
Comorbidities
Mean BMI (SD)	32.59 (7.58) (n = 781)	32.50 (7.54) (n = 1030)	0.807
Smoker in year prior (Y/N)	117 (14.94)	155 (15.00)	0.971
Diabetes (Y/N)	143 (18.26)	144 (13.94)	0.012*
COPD (Y/N)	48 (6.13)	61 (5.91)	0.841
Sleep apnea (Y/N)	312 (39.85)	411 (39.79)	0.979
CHF (Y/N)	0 (0.00)	1 (0.10)	0.384
Hypertension (Y/N)	374 (47.77)	466 (45.11)	0.261
Chronic condition requiring steroids (Y/N)	24 (3.07)	51 (4.94)	0.047*
Current cancer (Y/N)	18 (2.30)	17 (1.65)	0.316
Current dialysis (Y/N)	5 (0.64)	5 (0.48)	0.659
DVT (Y/N)	39 (6.68) (n = 584)	46 (6.29) (n = 731)	0.778
Ventilator dependent (Y/N)	0 (0.00)	0 (0.00)	n/a
Ascites (Y/N)	4 (0.39)	3 (0.38)	0.989
ASA class
I	41 (5.24)	67 (6.49)	0.265
II	368 (47.00)	509 (49.27)	0.338
III	358 (45.72)	432 (41.82)	0.097
IV	16 (2.04)	25 (2.42)	0.589
V	0 (0.00)	0 (0.00)	n/a
Functional status (Y/N Independent)	781 (99.74)	1029 (99.61)	0.634
Surgical information
Surgical priority
Elective	31 (3.96)	47 (4.55)	0.158
Urgent	21 (2.68)	40 (3.87)	0.163
Emergent	731 (93.36)	946 (91.58)	0.539
Surgical approach
Open	450 (58.59)	617 (60.91)	0.333
Laparoscopic	97 (12.63)	113 (11.15)	0.337
Robotic	221 (28.78) (n = 768)	283 (27.94) (n = 1013)	0.697
Hospital characteristics
Hospital locality			0.352
Rural	155 (19.80)	223 (21.59)
Urban	628 (80.20)	810 (78.41)

All frequencies represented as n (%) unless otherwise specified. For variables with missingness, the n of the non-missing sample is included in the table cell. SD = standard deviation. Y/N = Yes/No.

*Statistically significant (p < 0.05) difference between early respondents and late respondents.

^aEarly respondents responded to contact wave 1 (invitation).

^bLate respondents responded to contact wave 2+.

Incentive offer: Impact on response rates, respondent characteristics, and item non-response

We experimented with the incentive offer under two conditions. In the first (sub-phase 1b), incentives were offered to the P1b cohort to minimize nonresponse bias by increasing the participation rate amongst 262 non-respondents from the P1 cohort.⁴⁶ This additional contact attempt with incentive offer was associated with 23 additional surveys, increasing the total response rate of the P1 cohort from 38.20% (212 out of 555) to 42.34% (235 out of 555). However, we found no statistical differences in the observed characteristics of P1a and P1b respondents to suggest any reduction in nonresponse bias (Table A8). In our second condition (phase 3), C2 was offered an incentive at the first attempted contact. Survey outcomes from C2 were then compared with outcomes from C3, which was not offered an incentive. We found no statistical difference in response rates between C2 and C3 (Figure 2), despite C2 having been offered an incentive, C3 having received fewer reminders than C2, and the C3 sample size being larger by 31%. There were no statistical differences in the observed characteristics of C2 and C3 respondents (Table A8), leading us to conclude that offer of an incentive again did not influence nonresponse bias. Under neither condition did we find differences in item non-response with incentive offer (results not shown).

Cost effectiveness: Impact of contact mode, reminders, and incentive offer on survey cost

We considered cost effectiveness as the balance of cost (consisting of contact mode, reminders, and incentive offer) relative to response rate per survey design phase (Table 8). During the study period, our team’s effort amounted to a combined 1-2 full time equivalents (FTE). Because telephone calls had the highest relative cost of the three tested modes due the staffing required, phases that utilized telephone surveys had relatively higher associated FTE-related costs than phases that did not conduct telephone surveys. Similarly, phases with more telephone reminders also had higher FTE effort requirements than phases with fewer reminders. Finally, phases that utilized post-paid incentive offers had higher costs than phases that did not offer incentives. In our assessment, Phase 3 was less cost effective than Phase 4 as we did not observe returns in response rate to the incentive offer. Though Phase 2 was more costly than Phase 4 due to its FTE requirement for telephone calls, it boasted a higher response rate (32.5% vs 25.9%). Though Phase 1 had the highest response rate, the large number of reminders tested was not feasible at scale. Thus, we concluded that Phase 2 was the most relatively cost-effective combination of survey design components.

Table 8.

Design phase, characteristics, and response rate (%).

Design phase	Survey cohort	Design characteristics			Response rate
Design phase	Survey cohort	Survey modes	Reminders	Incentive offer	Response rate
Phase 1	Pilot cohort (P1)	Email, SMS, telephone	Email, SMS, telephone	No	38.20%
Phase 1	Pilot cohort (P1)	Email, SMS, telephone	5-6 reminders	No	38.20%
Phase 1b	Pilot sub-cohort (P1b)	Email, SMS	Email, SMS	Yes	8.78%
Phase 1b	Pilot sub-cohort (P1b)	Email, SMS	2 reminders	Yes	8.78%
Phase 2	Cohort 1 (C1)	Email, SMS, telephone	Email, SMS, telephone	No	32.48%
Phase 2	Cohort 1 (C1)	Email, SMS, telephone	4 reminders	No	32.48%
Phase 3	Cohort 2 (C2)	Email, SMS	Email, SMS	Yes	25.19%
Phase 3	Cohort 2 (C2)	Email, SMS	5 reminders	Yes	25.19%
Phase 4	Cohort 3 (C3)	Email, SMS	Email, SMS	No	25.89%
Phase 4	Cohort 3 (C3)	Email, SMS	4 reminders	No	25.89%

SMS = Short message services.

Overall non-response: Differences between respondents and non-respondents

Table 9 describes the characteristics of respondents and (a) non-respondents, (b) the survey population, and (c) the entire MSQC-COHR population. Differences were observed between respondents and non-respondents, as respondents were more likely to be female, older (mean difference +1.56 years), and White; to have Medicare, to have had a prior hernia repair, to have hernias greater than 6 cm in size, to have received mesh or myofascial release; and to have had their surgery performed at a rural hospital (all p < 0.05). Differences were also observed between respondents and the MSQC-COHR population; particularly, respondents were more likely to have undergone elective surgery (p < 0.05). Respondents were less likely than the MSQC-COHR population to be of Hispanic ethnicity, to have Medicaid, to have smoked in the year prior, and to be of ASA Class IV (all p < 0.05).

Table 9.

Characteristics of MSQC-COHR population, survey sample, and survey non-respondents and respondents, unweighted.

	Total MSQC-COHR, n (%)	Total survey, n (%)	Non-respondents, n (%)	Respondents, n (%)	p value
	(N = 17485)	(N = 7062)	(N = 4252)	(N = 1816)	Respondents vs. total MSQC-COHR	Respondents vs. non-respondents
Patient demographics
Female sex	7952 (45.48)	3148 (44.58)	1896 (44.59)	874 (48.13)	0.0310*	0.011*
Mean age, years (SD)	54.61 (14.56)	54.55 (14.47)	53.63 (14.65)	55.21 (13.89)	0.0965	0.0001*
Race
White	14239 (81.44)	5726 (81.08)	3440 (80.90)	1514 (83.37)	0.0433*	0.0229*
Black	2247 (12.85)	952 (13.48)	579 (13.62)	228 (12.56)	0.7250	0.2660
Other	999 (5.71)	384 (5.44)	233 (5.48)	74 (4.07)	0.0037	0.0217*
Ethnicity (Hispanic)	519 (2.97)	198 (2.80)	137 (3.22)	38 (2.09)	0.0330*	0.0160*
Insurance
Private	12572 (71.90)	5163 (73.11)	3171 (74.58)	1330 (73.24)	0.2260	0.2747
Medicare	3753 (21.46)	1454 (20.59)	799 (18.79)	407 (22.41)	0.3487	0.0012*
Medicaid	774 (4.43)	309 (4.38)	201 (4.73)	55 (3.03)	0.0051*	0.0026*
Uninsured	149 (0.85)	49 (0.69)	26 (0.61)	8 (0.44)	0.0638	0.4161
Other	237 (1.36)	87 (1.23)	55 (1.29)	16 (0.88)	0.0874	0.1733
Hernia-specific variables
Hernia location
Epigastric	3144 (20.54)	1470 (21.38)	887 (21.46)	373 (21.11)	0.5748	0.7606
Umbilical	9753 (63.71)	4355 (63.34)	2606 (63.04)	1121 (63.44)	0.8232	0.7674
Infraumbil	991 (6.47)	491 (7.14)	302 (7.31)	122 (6.90)	0.4880	0.5712
Suprapubic	223 (1.46)	113 (1.64)	69 (1.67)	32 (1.81)	0.2512	0.7002
Missing	1197 (7.82)	447 (6.50)	270 (6.53)	119 (6.73)	0.1038	0.7737
Missing	(n = 15308)	(n = 6876)	(n = 4134)	(n = 1767)	0.1038	0.7737
Previous hernia repair (Y/N)	2510 (16.21) (n = 15486)	1091 (15.70) (n = 6950)	631 (15.08) (n = 4185)	309 (17.37) (n = 1779)	0.2100	0.0260*
Hernia size
<2 cm	3845 (34.76)	1579 (32.28)	961 (32.63)	417 (33.23)	0.2802	0.7046
2–6 cm	5733 (51.84)	2646 (54.10)	1617 (54.91)	634 (50.52)	0.3752	0.0090*
>6 cm	1482 (13.40)	666 (13.62)	367 (12.46)	204 (16.25)	0.0054*	0.0010*
>6 cm	(n = 11060)	(n = 4891)	(n = 2945)	(n = 1255)	0.0054*	0.0010*
Mesh use (Y/N)	11681 (74.93)	5260 (75.29)	3129 (74.39)	1391 (77.62)	0.0125*	0.0080*
Mesh use (Y/N)	(n = 15589)	(n = 6986)	(n = 4206)	(n = 1792)	0.0125*	0.0080*
Myofascial release (Y/N)	811 (5.63)	369 (5.61)	200 (5.06)	124 (7.36)	0.0041*	0.0010*
Myofascial release (Y/N)	(n = 14402)	(n = 6573)	(n = 3952)	(n = 1684)	0.0041*	0.0010*
Comorbidities
Mean BMI (SD)	32.70 (7.62)	32.72 (7.70)	32.74 (7.71)	32.54 (7.56)	0.3875	0.3510
Mean BMI (SD)	(n = 17453)	(n = 7047)	(n = 4243)	(n = 1811)	0.3875	0.3510
Smoker in year prior (Y/N)	3556 (20.34)	1429 (20.24)	888 (20.88)	272 (14.98)	<0.001*	<0.001*
Diabetes (Y/N)	2817 (16.11)	1147 (16.24)	681 (16.02)	287 (15.80)	0.7321	0.8303
COPD (Y/N)	1194 (6.83)	466 (6.60)	256 (6.02)	109 (6.00)	0.1797	0.9870
Sleep apnea (Y/N)	6675 (38.18)	2671 (37.82)	1540 (36.22)	723 (39.81)	0.1739	0.0080*
CHF (Y/N)	65 (0.37)	21 (0.30)	14 (0.33)	1 (0.06)	0.0310*	0.0490*
Hypertension (Y/N)	8211 (46.96)	3360 (47.58)	1993 (46.87)	840 (46.26)	0.5694	0.6590
Chronic condition requiring steroids (Y/N)	604 (3.45)	270 (3.82)	174 (4.09)	75 (4.13)	0.1342	0.9460
Current cancer (Y/N)	397 (2.27)	145 (2.05)	90 (2.12)	35 (1.93)	0.3512	0.6340
Current dialysis (Y/N)	171 (0.98)	80 (1.13)	62 (1.46)	10 (0.55)	0.0706	0.0030*
DVT (Y/N)	740 (5.95) (n = 12427)	280 (5.56) (n = 5032)	155 (5.16) (n = 3001)	85 (6.46) (n = 1315)	0.4589	0.0870
Ventilator dependent (Y/N)	17 (0.10)	3 (0.04)	1 (0.02)	0 (0.00)	0. 1776	0.5130
Ascites (Y/N)	122 (0.70)	48 (0.68)	31 (0.73)	7 (0.39)	0.1235	0.1200
ASA class
I	890 (5.09)	368 (5.21)	219 (5.15)	108 (5.95)	0.1152	0.2063
II	8423 (48.18)	3428 (48.56)	2125 (50.00)	877 (48.29)	0.9289	0.2225
III	7590 (43.42)	3041 (43.07)	1782 (41.93)	790 (43.50)	0.9478	0.2571
IV	572 (3.27)	221 (3.13)	123 (2.89)	41 (2.26)	0.0195*	0.1657
V	7 (0.04)	2 (0.03)	1 (0.02)	0 (0.00)	0.3940	0.5467
V	(n = 17482)	(n = 7060)	(n = 4250)	(n = 1816)	0.3940	0.5467
Functional status (Y/N Independent)	17312 (99.01)	7000 (99.12)	4212 (99.06)	1810 (99.67)	0.0052*	0.0121*
Surgical information
Surgical priority
Elective	15813 (90.44)	6442 (91.22)	3906 (91.86)	1677 (92.35)	0.0079*	0.5192
Emergent	935 (5.35)	351 (4.97)	185 (4.35)	78 (4.30)	0.0562	0.9302
Urgent	737 (4.22)	269 (3.81)	161 (3.79)	61 (3.36)	0.0799	0.4141
Surgical approach
Open	10351 (60.26)	4206 (60.59)	2557 (61.16)	1067 (59.91)	0.7739	0.3656
Lap	2022 (11.77)	761 (10.96)	463 (11.07)	210 (11.79)	0.9801	0.4213
Robotic	4803 (27.96)	1975 (28.45)	1161 (27.77)	504 (28.30)	0.7610	0.6763
Robotic	(n = 17176)	(n = 6942)	(n = 4181)	(n = 1781)	0.7610	0.6763
Hospital characteristics
Hospital locality					0.3401	0.045*
Rural	3475 (19.87)	1426 (20.19)	791 (18.60)	378 (20.81)
Urban	14010 (80.13)	5636 (79.81)	3461 (81.40)	1438 (79.19)

All frequencies represented as n (%) unless otherwise specified. For variables with missingness, the n of the non-missing sample is included in the table cell. Variables used for raking were sex, age, race, surgical priority, and ASA Class IV. SD = standard deviation. Y/N = Yes/No.

*Statistically significant (p < 0.05) difference.

After application of non-response adjustment (using all 28 covariates available)⁵⁴ and raking weights (using sex, race, age, surgical priority, and ASA class),^54,56 balance was achieved between respondents and the underlying MSQC-COHR population across all covariates (Table 10).

Table 10.

Characteristics of survey respondents with non-response and raking weighting compared to MSQC-COHR population and survey sample.

	Total MSQC-COHR, % (N = 17485)	Total survey, % (N = 7062)	Respondents, weighted, % [95%CI] (N = 1816)
Patient demographics
Female sex^a,b	45.48	44.58	46.33 [43.99, 48.69]
Mean age, years^a,b	54.61	54.55	54.21 [53.56, 54.87]
Race^a,b
White	81.44	81.08	81.71 [79.72, 83.54]
Black	12.85	13.48	12.79 [11.28, 14.47]
Other	5.71	5.44	5.50 [4.37, 6.89]
Ethnicity (Hispanic)^a	2.97	2.80	3.02 [2.18, 4.17]
Insurance^a
Private	71.90	73.11	74.01 [71.89, 76.03]
Medicare	21.46	20.59	20.07 [18.32, 21.95]
Medicaid	4.43	4.38	4.17 [3.19, 5.43]
Uninsured	0.85	0.69	0.60 [0.29, 1.21]
Other	1.36	1.23	1.15 [0.70, 1.87]
Hernia-specific variables
Hernia location^a
Epigastric	20.54	21.38	21.38 [19.47, 23.42]
Umbilical	63.71	63.34	62.90 [60.54, 65.20]
Infraumbilical	6.47	7.14	7.53 [6.29, 8.98]
Suprapubic	1.46	1.64	1.63 [1.15, 2.31]
Missing	7.82 (n = 15308)	6.50 (n = 6876)	6.56 [5.24, 8.32]
Previous hernia repair (Y/N)^a	16.21 (n = 15486)	15.70 (n = 6950)	15.71 [14.12, 17.44]
Hernia size^a
<2 cm	34.76	32.28	32.76 [30.16, 35.46]
2–6 cm	51.84	54.10	53.61 [50.78, 56.42]
>6 cm	13.40 (n = 11060)	13.62 (n = 4891)	13.63 [11.93, 15.54]
Mesh use (Y/N)^a	74.93 (n = 15589)	75.29 (n = 6986)	75.25 [73.07, 77.30]
Myofascial release (Y/N)^a	5.63 (n = 14402)	5.61 (n = 6573)	5.65 [4.73, 6.72]
Comorbidities
Mean BMI	32.70 (n = 17453)	32.72 (n = 7047)	32.58 [32.21, 32.95]
Smoker in year prior (Y/N)^a	20.34	20.24	19.18 [17.21, 21.31]
Diabetes (Y/N)	16.11	16.24	15.96 [14.29, 17.79]
COPD (Y/N)	6.83	6.60	6.01 [4.98, 7.24]
Sleep apnea (Y/N)	38.18	37.82	39.49 [37.21, 41.82]
CHF (Y/N)	0.37	0.30	0.06 [0, 0.40]
Hypertension (Y/N)	46.96	47.58	45.18 [42.85, 47.54]
Chronic condition requiring steroids (Y/N)	3.45	3.82	3.96 [3.15, 4.96]
Current cancer (Y/N)	2.27	2.05	1.67 [1.19, 2.33]
Current dialysis (Y/N)	0.98	1.13	0.60 [0.32, 1.13]
DVT (Y/N)	5.95 (n = 12427)	5.56 (n = 5032)	6.24 [5.04, 7.71]
Ventilator dependent (Y/N)	0.10	0.04	0
Ascites (Y/N)	0.70	0.68	0.43 [0.20, 0.91]
ASA class^b
I	5.09	5.21	6.07 [5.02, 7.32]
II	48.18	48.56	48.56 [46.21, 50.93]
III	43.42	43.07	42.79 [40.48, 45.14]
IV	3.27	3.13	2.57 [1.88, 3.52]
V	0.04	0.03
Functional status^a (Y/N Independent)	99.01	99.12	99.41 [98.51, 99.76]
Surgical information
Surgical priority^a,b
Elective	90.44	91.22	91.18 [89.63, 92.51]
Urgent	4.22	3.81	4.04 [3.13, 5.20]
Emergent	5.35	4.97	4.78 [3.83, 5.96]
Surgical approach^a
Open	60.26	60.59	60.95 [58.61, 63.23]
Laparoscopic	11.77	10.96	11.34 [9.93, 12.92]
Robotic	27.96 (n = 17176)	28.45 (n = 6942)	27.72 [25.65, 29.88]
Hospital characteristics
Hospital locality^a
Rural	19.87	20.19	19.32 [17.57, 21.19]
Urban	80.13	79.81	80.68 [78.81, 82.43]

For variables with missingness, the n of the non-missing sample is included in the table cell. CI = confidence interval. Y/N = Yes/No.

*Statistically significant (p < 0.05) difference between the weighted survey respondents and the total MSQC-COHR population.

^aIncluded in non-response weighting scheme.

^bIncluded in raking weighting scheme.

Discussion

Measurement of patient reported outcomes (PROs) is a critical long-term need for surgical patients in order to improve shared decision-making preoperatively and patient care postoperatively.^2,3,62 For hernia surgery, a preference-sensitive operation with low rates of morbidity and mortality, measurement of outcomes that center on the patient’s quality of life post-operatively stands to transform care.⁶³ To ensure that such transformation is evidence-based, the goal of this study was to design a survey of post-operative PROs using a responsive design approach. Building from prior efforts in other clinical specialties including orthopedics,^26,37,64,65 bariatrics,²⁵ cardiology,^66,67 and oncology,⁶⁸ we defined likely best practices for the measurement of PROs within the field of hernia. We then tested and iteratively refined these design characteristics throughout multiple survey design phases to achieve population-level representation of long-term PROs among individuals who received hernia repair.

Internal and external validity

Surveys are a well-established tool to measure PROs. Internal validity of the MSQC-COHR survey was ensured through use of validated PROMs²⁸: the Ventral Hernia Recurrence Inventory (VHRI),¹ the PROMIS Pain Intensity 3a,^23,48 and the HerQLes scale.⁴⁹ Other efforts to collect PROs from surgical patients have typically been single-surgeon or single-institution, or have independently conducted by participating institutions with wide variation in survey procedure, response rates, and small samples.^25,26,37,65 Instead, we maximized external validity of our survey through integration into an existing clinical registry that ensured representativeness of patients at the state level.

Patient ineligibility based on lack of contact information available within MSQC-COHR may limit generalizability. Individuals without home internet access or with landline telephones (who could not have received SMS text messages) were at a disadvantage for survey participation.⁶⁹ This may have produced noncoverage bias if our sampling frame did not represent our target population.^45,70 However, we tested multimode survey contact methods to reduce non-response error^28,29 and to capture the maximum number of eligible patients.⁷¹ In later survey phases, we discontinued contact of phone-only individuals given the required costs and the overall low contact and response rates with this population. However, as phone-only individuals in our sample were on average older and more likely to have received surgery at rural hospitals, some generalizability may have been lost. Future iterations of our survey will employ targeted telephone calls and other “mode-sensitivity” strategies to increase the probability of response among older, rural adults in the registry.^26,72,73

To examine potential non-response bias, we compared survey respondents to the survey population and to non-respondents and found statistical differences, though with small magnitudes. We found a slightly higher mean age among respondents compared to non-respondents in our sample. Though this diverges from general literature on survey completion, where additional years of age is negatively associated with survey completion,⁷⁴ prior surgical PROs literature has shown a positive association between years of age and survey response.^37,42,75 We also found that respondents were more likely than non-respondents to have Medicare insurance, which may be an artifact of the slightly higher ages in the respondent sample.²⁶

Respondents in our study were more likely than the survey sample to have undergone elective surgery and were less likely than non-respondents to be insured by Medicaid, both factors which may indicate greater access to care related to socioeconomic status (SES) among respondents. Individuals of higher SES are more likely to respond to surveys²⁶; individuals of lower SES may have worse health outcomes and thus be less likely to respond.⁷⁶ Undergoing elective surgery indicates a certain relationship with the healthcare system, and individuals with stronger ties to the surveying institution are more likely to respond.⁷⁶ Similarly, we observed that respondents were more likely than non-respondents to have undergone a previous hernia repair, likely indicating engagement with their care and greater personal benefit associated with investment in quality improvement efforts.³⁷

Respondents were more likely than non-respondents and the MSQC-COHR population to have larger hernias, which may be associated with higher preoperative pain, an established predictor of higher engagement with surveys.³⁷ Patients with larger hernias likely undergo more significant operations that may be associated with increased long-term pain. As a result, these patients with poorer long-term outcomes may engage more with surveys than patients without complaints.¹ Interestingly, respondents were also more likely than non-respondents and the MSQC-COHR population to have received mesh or myofascial release in their repair. An association between mesh use and hernia size⁷⁷ may drive this finding; however the association between mesh use and reduced risk of recurrence¹² may mean we underestimate overall recurrence in our sample due to non-response bias.

Comparison of early and late respondents is a particular strength of this paper, as survey literature suggests that individuals who require more contacts before survey response could be more similar to non-respondents,⁴⁵ and thus follow-up efforts like additional reminders can reduce non-response bias.^24,28,29 Though we observe some statistical differences between early and late respondents, differences are largely minor and distinct from those unadjusted differences between respondents and the survey population. This suggests that our employment of additional reminders and extended follow-up may not be effective in reducing non-response bias as we fail to capture individuals more representative of the underlying population. Future work to investigate non-response bias should include description of differences in exposure-outcome associations between early and late respondents: for example, if late respondents’ outcomes are measured weeks to months after the outcomes of early respondents, and late respondents are less likely to have received mesh, we may estimate a false association between mesh use and recurrence simply due to late respondents’ longer time window during which a hernia could recur.⁶⁸ It is for this reason that we calculate and monitor median follow-up time in this analysis.

To further improve our generalizability we employed a non-response and raking weighting scheme.²⁹ Weighting allows for correction of imbalances on factors that potentially confound the association between PROs and likelihood of response, e.g., patient age or surgical priority. The auxiliary variables necessary for post-survey adjustment for non-response were readily available within the MSQC-COHR, including detailed demographic and clinical data on non-respondents.^33,67 Our weighting scheme allowed for balance on all measured covariates between the study population and weighted respondents.

Replicability

A substantial goal of this project was to ensure that our survey process and results could be replicated for our registry into the future. We employed responsive survey design²⁴ to test, through multiple phases, survey mode, number of reminders, and incentive offer, all with a focus on cost effectiveness. Costs arose from two areas: staff resources and incentives. First, though non-FTE-dependent methods for survey collection (i.e., email/SMS distribution of electronic surveys) have been demonstrated as equally reliable and less cost-intensive than FTE-dependent methods (i.e., telephone interviews),^42,78 we identified that mixed non-FTE and FTE-dependent strategies (i.e., use of telephone modality) captured a more representative sample than non-FTE-dependent methods only. Thus, we concluded that focused use of staff resources, like targeted use of telephone calls, was a cost-effective approach. Second, though reminders demonstrated some returns to survey response, the marginal returns to additional reminders diminished over time as all cohorts captured over two thirds of their total respondents by the end of the second reminder. Ideally, the benefits associated with the use of reminders must be balanced with their operative costs.^25,43 By our final design phase, we identified four reminders as a cost-effective number.

Financial incentives were designed to minimize non-response bias among individuals who would have otherwise been non-respondents.⁴⁶ We tested their utility in two ways: towards likely non-respondents only, and across an entire cohort. Though counter to the majority of survey participation literature, we failed to observe key improvements in response rate with offer of a post-paid $10.00 incentive³³ or any differences in demographic or clinical patient types between respondents who were and were not offered incentives, indicating no likely reduction in non-response bias. At $10 per person, the incentive offer amounted to a major cost for our statewide effort but may not have been financially meaningful to patients. Thus, in future iterations of the survey we will discontinue the use of incentives and instead target patients’ salience and “intrinsic motivation” to participate through initial mention of the survey during clinical interactions, tailoring of the survey’s introductory language, and integration of the survey into the patient portal.^46,72 As a positive, we did not find a decrease in response quality with the offer of incentives,⁴⁶ as item non-response was not meaningfully different in incentive versus non-incentive cohorts.

Limitations

This study was conducted in a single US state (Michigan) and within a clinical quality initiative (CQI, i.e., MSQC-COHR), which provided key infrastructure to our survey effort. However, this approach also has limitations. MSQC-COHR captures the vast majority (>90%) of our target population, but the remaining 10% of hernia cases in the state may be meaningfully different. Further, the use of a proprietary algorithm to sample cases leaves us unable to consider potential effects on sampling variance. Our dependence on a statewide registry of surgical patients with linked demographic and clinical data and contact information may make the application of our practices less straight-forward in other contexts. Nonetheless, the growing movement to form state-wide CQIs for surgical specialties⁷⁹ will hopefully expand the ability of clinicians in other states to collect population-level post-operative survey data. Even without the support of a larger CQI, our survey validation approach could feasibly be adopted by single institutions to better monitor patient outcomes over time.

Another limitation of this study is the degree of missingness in our data across eight covariates, largely concentrated within the first year of the MSQC-COHR database during which not all hospitals in the state were yet fully capturing hernia-specific covariates from patient charts. To account for this missingness, we coded for missingness in our weighting models, but a lack of full data limits our ability to evaluate and account for non-response bias. Another limitation is our inability to account for unobservable patient variables, particularly those representative of socioeconomic status not collected by the MSQC,^44,80 thus limiting the power of our selection bias investigation and adjustment.^33,43 However, we do not anticipate that the lack of these variables introduces significant bias into our results given the extent of the auxiliary data we have. In our study we did not capture non-English speaking hernia patients and instead categorized them as non-responders. Preferred language is not collected in our dataset, and so we were unable to evaluate whether non-English speaking nonrespondents were systematically different from English-speaking respondents. Future survey efforts may incorporate additional languages, such as Spanish or Arabic, to improve generalizability to all races and ethnicities within our state. Finally, findings on the effect of an incentive in cohort P1b are confounded with the additional survey invitation, and so we have focused our analysis of incentives on response rate to the comparison of cohorts 2 and 3.

Moving beyond response rates

The overall response rate across all four cohorts was 29.9%. Prior studies of PRO surveys in surgical cohorts have yielded widely varying 1-year response rates, from around 30%^25,26 to upwards of 80%.¹⁰ For example, Parrish et al. (2020) report a 43.6% response rate amongst orthopedic patients at a single U.S. institution whose outcomes were collected at 1 year post-operation either during a clinic visit (via tablet) or at home (via email).³⁷ In a statewide email-based survey of U.S. bariatric patients at 1-year post-surgery, Alvarez et al. (2021) found significant hospital-level variation (21.1% to 77.3%) in response rate, which the authors attributed to differences in procedure: hospitals with higher response rates had encouraged patients to complete paper surveys in clinic.²⁵ However, both of these specialties—orthopedics and bariatrics—can leverage more established long-term patient follow-up than hernia. Within hernia, the Danish Ventral Hernia Database touts exceptionally high response rates (>80%),¹⁰ but such rates would be near-impossible to achieve in the US without a nationalized health system with permanent patient identification numbers.^18,21 Instead, our approach—targeted multi-modal survey distribution to patients within a registry—is more practical within the U.S. healthcare environment and provides a comparable response rate to 1-year surveys conducted in person.

Conclusion

Patient-reported outcomes (PROs) are integral to the understanding of long-term hernia repair success. In this study, we report a pragmatic and sustainable methodology for survey collection implementation of validated PROs at 1 year within a statewide hernia surgical population, improving upon prior studies that require in-person collection of PROs.² Through responsive survey design we tested and implemented key techniques to reduce the risk of non-response bias through our sampling strategy, multimode contact methods, validated PROMs, and survey weighting scheme,^33,69 and we transparently identified successes of our survey methods. Continued survey recruitment methods will be adapted to mitigate non-response bias. Our study offers a framework to successfully measure population-based long-term patient reported outcomes, within and beyond the field of hernia, so that we may improve upon the current information asymmetry between patients and surgeons, in which risk factors for recurrence, pain, and poor quality of life at 1 year and beyond are largely unknown.¹⁰

Footnotes

Authors’ note

AK is an MD-PhD candidate (Epidemiology) at the University of Michigan (UM). TSG is an Associate Research Scientist at the Survey Research Center/Institute for Social Research at UM. BF is a general surgery resident at UM. AH is a Research Area Specialist Senior at the Center for Healthcare Outcomes & Policy and a PhD student in Health Infrastructure and Learning Systems (HILS) at UM. JY is a Project Coordinator for the Michigan Surgical Quality Collaborative and an MPH candidate at UM. MR is a Clinical Assistant Professor (Health Management and Policy) at UM. DT is Professor of Surgery at UM.

Acknowledgements

Thanks to the members of the Core Optimization Hernia Registry team, particularly Kiran O’Connor, Deena Sukhon, and Graham Gilliam for their assistance in conducting phone surveys; and Yuxuan Chen (Michigan Surgical Quality Collaborative) for data management support.

ORCID iD

Abigail L. Kappelman

Ethical considerations

This quality improvement study was exempt from regulation by the University of Michigan Institutional Review Board (HUM00091060).

Author contributions

Survey design and rollout were conducted by AH, JY, and DT. AK merged and analyzed the data, developed the weighting scheme, interpreted all data, and wrote the manuscript. BF provided data support. TSG and MR provided survey design and analytic support. All authors read and approved the final manuscript.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by NINDS [grant number T32GM007863] (AK), NIMHD [grant number F30MD019520] (AK), NIH [grant number T32AG062403] (BF), and NIDDK [grant number R01DK128179]. The Michigan Surgical Quality Collaborative (MSQC) is funded as part of the Blue Cross Blue Shield of Michigan Value Partnership program. The content of this study is solely the responsibility of the authors and does not necessarily reflect the official views of Blue Cross Blue Shield of Michigan. No funder or sponsor had any role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; or decision to submit the manuscript for publication. AK had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Declaration of conflicting interests

The Authors declare that there is no conflict of interest. Dr Telem receives consulting fees from Medtronic.

Data Availability Statement

The data that support the findings of this study are available from the Michigan Surgical Quality Collaborative (MSQC) but restrictions apply to the availability of these data, which were used under agreement for the current study, and so are not publicly available. Data are however available from the authors upon reasonable request and with permission of Blue Cross Blue Shield of Michigan.*

Appendix

References

Baucom

Ousley

Feurer

, et al. Patient reported outcomes after incisional hernia repair—establishing the ventral hernia recurrence inventory. Am J Surg 2016; 212(1): 81–88.

Van Veenendaal

Poelman

Van Den Heuvel

, et al. Patient-reported outcomes after incisional hernia repair. Hernia 2021; 25(6): 1677–1684.

Sokas

Edelen

, et al. A review of PROM implementation in surgical practice. Ann Surg 2022; 275(1): 85–90.

Howard

Ehlers

Delaney

, et al. Leveraging a statewide quality collaborative to understand population-level hernia care. Am J Surg 2021; 222(5): 1010–1016.

Helgstrand

Thygesen

Bisgaard

, et al. Differential recurrence after laparoscopic incisional hernia repair: importance of a nationwide registry-based mesh surveillance. Br J Surg 2020; 107(9): 1130–1136.

Campbell

Henderson

Englesbe

, et al. Surgical site infection prevention: the importance of operative duration and blood transfusion—results of the first American college of surgeons–national surgical quality improvement program best practices initiative. J Am Coll Surg 2008; 207(6): 810–820.

East

Hill

Dames

, et al. Patient views around their hernia surgery: a worldwide online survey promoted through social media. Front Surg 2021; 8: 769938.

Stey

Russell

Sugar

, et al. Extending the value of the national surgical quality improvement program claims dataset to study long-term outcomes: rate of repeat ventral hernia repair. Surgery 2015; 157(6): 1157–1165.

Lindmark

Löwenmark

Strigård

, et al. Major complications and mortality after ventral hernia repair: an eleven-year Swedish nationwide cohort study. BMC Surg 2022; 22(1): 426.

10.

Christoffersen

Helgstrand

Rosenberg

, et al. Long-term recurrence and chronic pain after repair for small umbilical or epigastric hernias: a regional cohort study. Am J Surg 2015; 209(4): 725–732.

11.

Henriksen

Friis-Andersen

Jorgensen

, et al. Open versus laparoscopic incisional hernia repair: nationwide database study. BJS Open 2021; 5(1): zraa010.

12.

Katawazai

Wallin

Sandblom

. Long-term reoperation rate following primary ventral hernia repair: a register-based study. Hernia 2022; 26(6): 1551–1559.

13.

Helgstrand

Rosenberg

Kehlet

, et al. Reoperation versus clinical recurrence rate after ventral Hernia Repair. Ann Surg 2012; 256(6): 955–958.

14.

Sheetz

Corona

Cramm

, et al. Variation in ambulatory surgery utilization in Michigan. J Surg Res 2014; 189(2): 255–261.

15.

Karaca

McDermott

. High-volume invasive, therapeutic ambulatory surgeries performed in hospital-owned facilities, 2016. In: Healthcare Cost and Utilization Project (HCUP) Statistical Briefs. Agency for Healthcare Research and Quality (US); 2006. Available from: https://www.ncbi.nlm.nih.gov/books/NBK563613/

16.

McDermott

Liang

. Overview of major ambulatory surgeries performed in hospital-owned facilities, 2019. Healthcare Cost and Utilization Project (HCUP) Statistical Briefs. Agency for Healthcare Research and Quality (US), 2006. [cited 2023 Sep 15]. Available from: https://www.ncbi.nlm.nih.gov/books/NBK577044/

17.

Poulose

Shelton

Phillips

, et al. Epidemiology and cost of ventral hernia repair: making the case for hernia research. Hernia 2012; 16(2): 179–183.

18.

Schwab

Dietz

Menzel

, et al. Pitfalls in interpretation of large registry data on hernia repair. Hernia 2018; 22(6): 947–950.

19.

Kenawy

Underhill

Jacobs

, et al. Ten-year outcomes following ventral hernia repair: making the case for better post-market surveillance in the USA. Surg Endosc 2022, [cited 2023 Jun 20]; Available from.

20.

Köckerling

Hoffmann

Adolf

, et al. Potential influencing factors on the outcome in incisional hernia repair: a registry-based multivariable analysis of 22,895 patients. Hernia 2021; 25(1): 33–49.

21.

Kyle-Leinhase

Köckerling

Jørgensen

, et al. Comparison of hernia registries: the CORE project. Hernia 2018; 22(4): 561–575.

22.

Kehlet

Bay-Nielsen

Danish Hernia Database Collaboration . Nationwide quality improvement of groin hernia repair from the Danish Hernia Database of 87,840 patients from 1998 to 2005. Hernia 2008; 12(1): 1–7.

23.

Poulose

Roll

Murphy

, et al. Design and implementation of the Americas Hernia society quality collaborative (AHSQC): improving value in hernia care. Hernia 2016; 20(2): 177–189.

24.

Groves

Heeringa

. Responsive design for household surveys: tools for actively controlling survey errors and costs, 2006. [cited 2023 Nov 25]; Available from: https://deepblue.lib.umich.edu/handle/2027.42/71787

25.

Alvarez

Stricklen

Buda

, et al. Factors associated with completion of patient surveys 1 year after bariatric surgery. Surg Obes Relat Dis 2021; 17(3): 538–547.

26.

Schamber

Takemoto

Chenok

, et al. Barriers to completion of patient reported outcome measures. J Arthroplast 2013; 28(9): 1449–1453.

27.

Gummer

Christmann

Verhoeven

, et al. Using a responsive survey design to innovate self-administered mixed-mode surveys. J Roy Stat Soc Stat Soc 2022; 185(3): 916–932.

28.

Ponto

. Understanding and evaluating survey research. JADPRO 2015; 6(2): 168–171, Available from: https://www.advancedpractitioner.com/issues/volume-6,-number-2-(marapr-2015)/understanding-and-evaluating-survey-research.aspx

29.

Agaku

. Methodological considerations for the design and implementation of reliable and valid web surveys. Public Health Toxicol 2021; 1(1): 1–10.

30.

Campbell

Kubus

Henke

, et al. The Michigan surgical quality collaborative: a legacy of Shukri Khuri. Am J Surg 2009; 198(5): S49–S55.

31.

Nikolian

Regenbogen

. Statewide clinic registries: the Michigan surgical quality collaborative. Clin Colon Rectal Surg 2019; 32(01): 016–024.

32.

Howard

Ehlers

Delaney

, et al. Incidence and trends of decision regret following elective hernia repair. Surg Endosc 2022; 36(9): 6609–6616.

33.

Groves

. Nonresponse rates and nonresponse bias in household surveys. Public Opin Q 2006; 70(5): 646–675.

34.

Wohlfahrt

Zickmund

Slager

, et al. Provider perspectives on the feasibility and utility of routine patient-reported outcomes assessment in heart failure: a qualitative analysis. J Am Heart Assoc 2020; 9(2): e013047.

35.

Zolin

Krpata

Petro

, et al. Long-term clinical and patient-reported outcomes after transversus abdominis release with permanent synthetic mesh: a single center analysis of 1203 patients. Ann Surg 2023; 277(4): e900–e906.

36.

Tastaldi

Barros

PHF

Krpata

, et al. Hernia recurrence inventory: inguinal hernia recurrence can be accurately assessed using patient-reported outcomes. Hernia 2020; 24(1): 127–135.

37.

Parrish

Jenkins

Patel

, et al. Demographic and perioperative factors associated with patient-reported outcomes measurement information system (PROMIS) survey completion. Clin Spine Surg 2020; 33(10): E519–E524.

38.

Anthony

Long

Hynan

, et al. Surgical complications exert a lasting effect on disease-specific health-related quality of life for patients with colorectal cancer. Surgery 2003; 134(2): 119–125.

39.

Bojcic

Sue

Huon

, et al. Comparison of paper and electronic surveys for measuring patient-reported outcomes after anterior cruciate ligament reconstruction. TPJ 2014; 18(3): 22–26.

40.

Stern

Bilgen

Dillman

. The state of survey methodology: challenges, dilemmas, and new frontiers in the era of the tailored design. Field Methods 2014; 26(3): 284–301.

41.

Deiss

Chen

Sarin

, et al. Patient-reported outcomes 6 months after enhanced recovery after colorectal surgery. Perioper Med 2018; 7: 19.

42.

Wong

Brusseleers

Hall

, et al. Mixed-mode versus paper surveys for patient-reported outcomes after critical illness: a randomised controlled trial. Aust Crit Care 2022; 35(3): 286–293.

43.

Neve

Van Benthem

PPG

Stiggelbout

, et al. Response rate of patient reported outcomes: the delivery method matters. BMC Med Res Methodol 2021; 21(1): 220.

44.

Lee

Fredriksen-Goldsen

McClain

, et al. Are sexual minorities less likely to participate in surveys? An examination of proxy nonresponse measures and associated biases with sexual orientation in a population-based health survey. Field Methods 2018; 30(3): 208–224.

45.

Smith

Witte

Rocha

, et al. Effectiveness of incentives and follow-up on increasing survey response rates and participation in field studies. BMC Med Res Methodol 2019; 19(1): 230.

46.

Singer

. The use and effects of incentives in surveys. Ann Am Acad Polit Soc Sci 2013; 645(1): 112–141.

47.

Gram-Hanssen

Tolstrup

Zetner

, et al. Patient-reported outcome measures for patients undergoing inguinal hernia repair. Front Surg 2020; 7: 17.

48.

Revicki

Chen

Harnam

, et al. Development and psychometric analysis of the PROMIS pain behavior item bank. Pain 2009; 146(1): 158–169.

49.

Krpata

Schmotzer

Flocke

, et al. Design and initial implementation of HerQLes: a hernia-related quality-of-life survey to assess abdominal wall function. J Am Coll Surg 2012; 215(5): 635–642.

50.

The American Association for Public Opinion Research . Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys, 2023.

51.

Dillman

Sangster

Tarnai

, et al. Understanding differences in people’s answers to telephone and mail surveys. New Dir Eval 1996; 1996(70): 45–61.

52.

Heerwegh

Loosveldt

. Face-to-Face versus web surveying in a high-internet-coverage population: differences in response quality. Public Opin Q 2008; 72(5): 836–846.

53.

Siemiatycki

Campbell

. Nonresponse bias and early versus all responders in mail and telephone surveys. Am J Epidemiol 1984; 120(2): 291–301.

54.

Kolenikov

. Post-stratification or non-response adjustment? Survey Practice 2016; 9(3), Available from: https://www.surveypractice.org/article/2809-post-stratification-or-non-response-adjustment

55.

Bergmann

. IPFWEIGHT: stata module to create adjustment weights for surveys. Boston College Department of Economics, 2011. (Statistical Software Components).

56.

Kulas

Robinson

Smith

, et al. Post-stratification weighting in organizational surveys: a cross-disciplinary tutorial. Hum Resour Manag 2018; 57(2): 419–436.

57.

Battaglia

Hoaglin

Frankel

. Practical considerations in raking survey data. Surv Pract 2009; 2(5): 1–10, Available from: https://www.surveypractice.org/article/2953-practical-considerations-in-raking-survey-data

58.

Kolenikov

. Calibrating survey data using iterative proportional fitting (Raking). STATA J: Promoting communications on statistics and Stata 2014; 14(1): 22–59.

59.

DeBell

Krosnick

Lupia

. Methodology report and user’s guide for the 2008-2009 ANES panel study, 2010.

60.

StataCorp. Stata: release 17. Stata Press; 2021 [cited 2025 May 7]. Available from: https://www.stata.com/manuals13/svysvyset.pdf

61.

von Elm

Altman

Egger

, et al. The strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies. Lancet 2007; 370(9596): 1453–1457.

62.

Lee

THJ

Ulisney

Choudhuri

, et al. Understanding the patient perspective after ventral hernia repair. Hernia 2019; 23(5): 995–1001.

63.

Colavita

Tsirline

Belyansky

, et al. Prospective, long-term comparison of quality of life in laparoscopic versus open ventral Hernia repair. Ann Surg 2012; 256(5): 714–723.

64.

Acosta

Tang

Regal

, et al. Investigating the bias in orthopaedic patient-reported outcome measures by mode of administration: a meta-analysis. JAAOS Glob Res Rev 2020; 4(12): e2000194.

65.

Patel

Lee

, et al. Predictors of low patient-reported outcomes response rates in the California joint replacement registry. J Arthroplast 2015; 30(12): 2071–2075.

66.

Berghammer

Mattsson

Johansson

, et al. Comparison of participants and non-participants in patient-reported outcome surveys: the case of assessment of patterns of patient-reported outcomes in adults with congenital heart disease – international study. Cardiol Young 2017; 27(3): 427–434.

67.

Simsek

Manemann

Yost

, et al. Participation Bias in a survey of community patients with heart failure. Mayo Clin Proc 2020; 95(5): 911–919.

68.

Lie

Rueegg

Fosså

, et al. Limited evidence of non-response bias despite modest response rate in a nationwide survey of long-term cancer survivors—results from the NOR-CAYACS study. J Cancer Surviv 2019; 13(3): 353–363.

69.

Bradley

Kuriwaki

Isakov

, et al. Unrepresentative big surveys significantly overestimated US vaccine uptake. Nature 2021; 600(7890): 695–700.

70.

Lee

Brown

Grant

, et al. Exploring nonresponse Bias in a health survey using neighborhood characteristics. Am J Public Health 2009; 99(10): 1811–1817.

71.

McNeil

Evans

Johnson

, et al. Clinical-quality registries: their role in quality improvement. Med J Aust 2010; 192(5): 244–245.

72.

Coon

van Riper

Morton

, et al. Evaluating Nonresponse Bias in survey research conducted in the rural Midwest. Soc Nat Resour 2020; 33(8): 968–986.

73.

Jackson

Medway

Megra

. Can appended auxiliary data be used to tailor the offered response mode in cross-sectional studies? Evidence from an address-based sample. Journal of Survey Statistics and Methodology 2023; 11(1): 47–74.

74.

Herzog

Rodgers

. Age and response rates to interview sample surveys. J Gerontol 1988; 43(6): S200–S205.

75.

Millar

Elena

Gallicchio

, et al. The feasibility of web surveys for obtaining patient-reported outcomes from cancer survivors: a randomized experiment comparing survey modes and brochure enclosures. BMC Med Res Methodol 2019; 19(1): 208.

76.

Groves

Peytcheva

. The impact of nonresponse rates on nonresponse bias: a meta-analysis. Public Opin Q 2008; 72(2): 167–189.

77.

Howard

Ehlers

Delaney

, et al. Hospital-level variation in mesh use for ventral and incisional hernia repair. Surg Endosc 2023; 37(2): 1501–1507.

78.

Adogwa

Elsamadicy

Cheng

, et al.

Assessing patient reported outcomes measures via phone interviews versus patient self-survey in the clinic: are we measuring the same thing?

World Neurosurg 2015; 87: 230–234.

79.

Sheetz

Englesbe

. Expanding the quality collaborative model as a blueprint for higher-value care. JAMA Health Forum 2020; 1(5): e200413.

80.

Dunne

Martin

Bailey

, et al. Participation bias in a sexuality survey: psychological and behavioural characteristics of responders and non-responders. Int J Epidemiol 1997; 26(4): 844–854.

81.

Howard

Ehlers

Delaney

, et al. Sex disparities in the treatment and outcomes of ventral and incisional hernia repair. Surg Endosc 2023; 37(4): 3061–3068.

82.

Instrument: PROMIS pain intensity - short form 3a v1.0 | NIDA CTN common data elements. [cited 2023 Dec 27]. Available from: https://cde.nida.nih.gov/instrument/0a481bfb-a5e6-3c84-e050-bb89ad43314d

83.

PROMIS pain intensity scoring manual. 2021 [cited 2023 Dec 27]. Available from: https://www.healthmeasures.net/images/PROMIS/manuals/Scoring_Manuals_/PROMIS_Pain_Intensity_Scoring_Manual.pdf

84.

Krpata

Petro

Prabhu

, et al. Effect of Hernia mesh weights on postoperative patient-related and clinical outcomes after open Ventral Hernia Repair. JAMA Surg 2021; 156(12): 1085–1088.

85.

Petro

Zolin

Krpata

, et al. Patient-Reported outcomes of robotic vs laparoscopic ventral Hernia repair with intraperitoneal mesh: the PROVE-IT randomized clinical trial. JAMA Surg 2021; 156(1): 22–29.

86.

Renshaw

Gupta

Poulose

. Establishing the minimal clinically important difference for the Hernia-related quality of life survey (HerQLes). Am J Surg 2022; 223(2): 245–249.