Abstract
Objectives
Fecal immunochemical testing (FIT) is an effective screening tool for colorectal cancer. If an FIT is abnormal, a follow-up colonoscopy is necessary to remove polyps or find cancers. We sought to develop a usable risk prediction model to identify patients unlikely to complete a colonoscopy following an abnormal FIT test.
Methods
We recalibrated and then redeveloped a prediction model in federally qualified health centers (FQHCs), using a retrospective cohort of patients aged 50–75 with an abnormal FIT test and clinical data. Logistic and Cox regressions were used to recalibrate and then redevelop the model.
Results
The initial risk model used data from eight FQHCs (26 clinics) including 1723 patients. When we applied the model to a single large FQHC (34 clinics, 884 eligible patients), the model did not recalibrate successfully (c-statistic dropped more than 0.05, from 0.66 to 0.61). The model was redeveloped in the same FQHC in a cohort of 1401 patients with a c-statistic of 0.65.
Conclusions
The original model developed in a group of FQHCs did not adequately recalibrate in the single large FQHC. Health system, patient characteristics or data differences may have led to the inability to recalibrate the model. However, the redeveloped model provides an adequate model for the single FQHC.
Introduction
Colorectal cancer (CRC) screening is the third leading cause of cancer death in the United States. 1 CRC screening by annual fecal immunochemical test (FIT) is an accessible and cost-effective strategy to lower CRC incidence and mortality. Fecal testing is able to detect blood in the stool which may be indicative of polyps or cancers. 2 This mode of screening depends on a follow-up colonoscopy after an abnormal FIT result to remove polyps or expedite CRC diagnosis and treatment. However, up to 50% of patients with abnormal FIT fail to complete this essential follow-up component, as they face multilevel barriers including a lack of knowledge.3,4
Predicting and Addressing Colonoscopy Non-Adherence in Community Settings (PRECISE) is an NCI-funded study that aims to recalibrate and apply a risk prediction model to identify patients who are likely to forgo follow-up colonoscopy. 5 PRECISE tests the effectiveness of a patient navigation intervention among patients with a moderate or low probability of adhering to follow-up colonoscopy recommendations. Patients are stratified according to risk of nonadherence to follow-up colonoscopy and an adapted patient navigation program is delivered to a randomized subset of eligible patients.
Predictive analytics have been found to work in community settings. In Strategies and Opportunities to Increase CRC Screening in Priority Populations (STOP CRC), a risk prediction model was created to identify patients unlikely to complete a colonoscopy following an abnormal FIT test. 6 The original risk prediction model was developed using data from 26 federally qualified health center (FQHC) clinics in Oregon and California. This model showed adequate separation of patients across risk levels (bootstrap-corrected c-statistic > 0.63, R2 = 14.03), and included eight variables.
Application of risk prediction models to settings outside of where the model was created can be valuable to systems where analytic capacity is limited in creating their own models, yet it can be challenging due to variations in the health system or patient characteristics, clinic practices, and the ability to capture data in the electronic health record (EHR). Here, we conduct a retrospective cohort study to externally recalibrate a community-based risk prediction model for identifying patients unlikely to obtain a colonoscopy following an abnormal FIT test result. The recalibration was at SeaMar Community Health Centers in Washington State. The model would be redeveloped if the recalibration of the original model was not adequate.
Methods
Recalibration of the model
SeaMar Community Health Centers is a large FQHC in Washington State that served 217,000 patients in 2018. The recalibration of the STOP CRC risk prediction model began by replicating the data used in the STOP CRC model at SeaMar. The methods of the STOP CRC model are described elsewhere. 6
Briefly, the STOP CRC cohort included patients aged 50–75 with an abnormal FIT result and one year of follow-up time. The original STOP CRC model was created using the data from 2014 to 2016 from eight FQHCs, including 26 clinics and 1596 patients. 6 Available EHR codes were used in a Cox regression model to predict colonoscopy completion within 6 months of an abnormal FIT result. We fit a full model and used a step-down process, stopping when the model retained ≥ 90% of the variation explained by the full model. Discrimination was measured by a bootstrap-corrected c-statistic. Our final model, developed from data in the EHR, included 5-year age group, non-White race, insurance type, quintiles of Gini Index (income inequality), long-term anticoagulant use, receipt of a flu vaccine in the past year, frequency of missed clinic appointments, and clinic site. The model produced a bootstrap corrected c-statistic of 0.63. This model showed excellent calibration between observed and predicted risk. 6 The characteristics in the model were easily obtainable and believed to be replicable, except for “Health Center” of clinics in Oregon and California, which would be impossible to replicate.
To recalibrate the model in the PRECISE project, we assembled a retrospective cohort of patients aged 50–75 with an abnormal FIT result and a referral to colonoscopy (n = 884). This project recalibrated the original risk prediction model by comparing two modes with the original eight variables (except for Health Center). Cohort separation (discrimination) for low- and moderate-probability patients was calculated using the c-statistic (Table 1).7,8
Characteristics in the recalibration and redevelopment models.
The c-statistics have been bootstrap-corrected to account for shrinkage to provide a more realistic estimate of discrimination in future populations.
Based on the D-statistic, is a summary measure of discrimination based on the variation in explained risk.
Federal poverty level.
STOP CRC: Strategies and Opportunities to Increase CRC Screening in Priority Populations; Predicting and Addressing Colonoscopy Non-Adherence in Community Settings.
We extracted EHR data from SeaMar Health Center; specifically, we identified patients, aged 50–75, with abnormal FIT tests from January 2016 to June 2018. All eligible patients were included from 32 clinics across eight counties. Age, race, and insurance were obtained from administrative databases. Anticoagulant use was determined using a pharmacy order or diagnosis. Flu vaccination in the past year and prior missed appointments were obtained from the EHR. Patients were excluded if they had a colonoscopy up to 3 years prior to the abnormal FIT, or 6 days following the FIT result, as these procedures were likely not follow-up to an abnormal FIT. No missing data required imputation, as missing often determined lack of a condition (e.g. hemorrhoids or flu vaccination). All data and categories are described in the Appendix (see online supplementary material).
The abnormal FIT date anchored the 12-month follow-up time period. Patients were followed for up to 1 year to determine colonoscopy completion. We were unable to know if a patient was lost to follow-up, as the community clinic setting does not track enrollment. Completion was a binary outcome. Time to colonoscopy completion was not modeled due to inconsistencies in the procedure date documentation practice at SeaMar.
There were minor differences in the development data and the recalibration data. As Gini income inequality was not available, we substituted “federal poverty level.” Income categories included “greater than the federal poverty level,” “less than federal poverty level” and “unknown,” when missing. Because SeaMar was a single health center, the health center variable in the original model could not be used; instead, we used “county” where each of the 32 clinics was located to represent possible geographic variation across clinics.
We used logistic regression to predict the likelihood of completing a colonoscopy and used the c-statistic to evaluate the discrimination of the model. We assessed calibration by arranging patients into quintiles using the mean linear predictor value and comparing the observed and predicted risk values for each quintile.
Model redevelopment
Because of the reduction in performance (c-statistics 0.66 in original model vs. 0.61 re-calibrated model), the research team decided that the model needed to be redeveloped. We used the same cohort of SeaMar patients as in the recalibration model; that is, patients aged 50–75 with an abnormal FIT result from January 2016 to June 2018 who received care at any SeaMar medical clinic. Patients were excluded if they had a recent colonoscopy, defined as up to 3 years prior to the abnormal FIT or 6 days following the FIT result. The abnormal FIT date anchored the 12-month follow-up interval. The main outcome, completion of colonoscopy, was a binary outcome and was documented if the date was within 12 months of FIT result.
No missing data required imputation and this was treated as a discrete category when appropriate. Variables described above (age, race, insurance, anticoagulant use, flu vaccination, prior missed appointments, federal poverty level, county) were retained for use in this model. Additional predictors were added to the model and retained through a stepwise process to simplify the model. The newly added variables included language, homelessness, gender and mammogram, a comorbidity score (Gagne), marital status, number of encounters, hemorrhoids, a presence of blood in the stool, prior colonoscopy completion, and body mass index (BMI). Characteristics from the original model that were not retained because they added no value to the model included income inequality (Gini Index), anticoagulant use, and flu vaccination in the past year.
Logistic regression was used to predict the likelihood of completing a colonoscopy. The internal validation process was performed by generating a bootstrap estimate of the optimism of both full and reduced models, with a sample size of 500. Patients were arranged in quintiles by linear predictor value. Observed and predicted risk values were calculated for each quintile.
Results
Recalibration model
Patients with an abnormal FIT and colonoscopy referral were eligible at SeaMar for the model recalibration (n = 884). Colonoscopy was completed within 12 months for 52.5% (n = 464) of the patients with an abnormal FIT and colonoscopy referral. The final model included a 5-year age group, race, insurance type, federal poverty level, anticoagulant use, receipt of a flu vaccine in the past year, frequency of missed clinic appointments, and clinic county. The model produced a naïve c-statistic of 0.61, a bootstrap-corrected c-statistic of 0.55, and R-squared of 3.9% (Table 1). The calibration of the model is shown in Figure 1, where you can see the predicted risk separating more widely in the upper quintiles.
Calibaration of the recalibrated model.
We considered a c-statistic drop ≥ 0.05 as an inferior separation, requiring model refinement. The reduction in the bootstrap-corrected c-statistic was 0.08, thus requiring model refinement or redevelopment.
Model redevelopment
The redevelopment model final cohort included patients with an abnormal FIT (n = 1401). The patient population increased as the referral requirement was dropped as a condition for eligibility.
Colonoscopy was completed within 6 months for 50.9% (n = 713) of the patients with an abnormal FIT. The federal poverty level, anticoagulant use, flu vaccine in the past year, and prior missed appointments were not retained in the redeveloped model.
The final reduced redeveloped model included 12 variables: age, race, language, insurance, county, a composite variable for gender and mammogram screening, number of prior missed appointments, Gagne's comorbidity score, number of prior encounters, BMI, marital status, and prior screening (Table 2). The model produced a naïve c-statistic of .65, a bootstrap-corrected c-statistic of 0.61, and R-squared of 8.7% (Table 1). The calibration of this model can be seen in Figure 2, where you can see adequate calibration between the observed and predicted risks.
Calibration of the redeveloped model.
Patient characteristics in the redeveloped model.
The odds ratio reflects the effect of each 5-year increase in age, a grouped linear term.
Discussion
The STOP CRC model did not recalibrate in the SeaMar population, but a refined and redeveloped model was created that provides sufficient risk prediction. Although the redeveloped model was below the standards generally used to make decisions for individual patients, a c-statistic of 0.60 may still be more useful than using a single characteristic to identify patients who may benefit from navigation. The redeveloped model leverages readily available data in the EHR, and tailors the model to the SeaMar population.
The objective of this model was to predict patients’ probability of completing a follow-up colonoscopy after an abnormal FIT test. This model does identify predicted probability and will be used in the PRECISE analysis for determining the effectiveness of patient navigation among patients at varying probability levels.
To our knowledge, these models are the only models to assess the probability of adhering to follow-up to abnormal FIT testing. Risk factors for colon cancer are well known yet many patients at higher risk continue to not screen. 9 The models in this paper can aid in such screening programs, by prioritizing patients by their likelihood of completing screening.
Regardless of screening recommendations, medically underserved populations continue to screen at lower rates than the average population.10,11 Using risk-based population management tools, like this model, for targeting screening resources could help reduce disparities in preventive care.
This model is limited in that it was created for a single health system in the Pacific Northwest. The lack of recalibration from the STOP CRC model to the PRECISE model indicates that there are regional or system variations in patient and provider follow-up to abnormal FIT testing. Regional variation may include area characteristics such as the rurality of patients being seen in the diverse SeaMar system. While most colonoscopies have little to no patient out-of-pocket costs because Washington is a Medicaid expansion state, delays in access to colonoscopy are common among Medicaid enrollees. 12 Another limitation is that quality performance measures for states with Medicaid expansion and prioritized CRC screening may explain better screening rates than commercial coverage or Medicare, and therefore the model's insurance predictor coefficients may not transport to other states without those quality performance incentives.
Variations may also stem from facility-level variation in the referral communication, the referral process, provider communications with the patient and specialty providers serving FQHC patients. Nuance in health systems may limit generalizability and require systems to assess the applicability of models prior to. We hope this model is useful to guide health systems in the development of their own models yet recognize the limitations in underresourced health systems.
Furthermore, because of the lack of interoperability among health records in primary care and specialty care, this project relies on the data within the primary care health record as predictors of colonoscopy completion. EHR data is collected for billing purposes as well as for the management of patient health. This can create challenges in capturing the data needed for models. In this case, other variables not available in EHRs (i.e. social support and out-of-pocket costs), could improve model accuracy.
The redeveloped model may serve as an efficient way to direct patient navigation resources. The model helps identify patients who would likely benefit from navigation given their low probability of obtaining a colonoscopy without additional support. Because the model was redeveloped for SeaMar and has yet to undergo external validation in a distinct population, its clinical use should be limited to SeaMar.
Conclusion
Predictive models that carry over from other settings are rare. While the recalibration of the STOP CRC model was not successful, the redevelopment provides an adequate model for identifying patients in need of assistance to complete the colonoscopy. Predicting follow-up colonoscopy after an abnormal FIT allows clinics to target patient navigation resources to patients who need them most. Our ongoing PRECISE trial will apply this predictive model and conduct an innovative trial of a systems-level precision healthcare intervention tailored to the patient likelihood of completing a colonoscopy. The approach may represent a sustainable way to improve follow-up colonoscopy completion in community health center patients.
Supplemental Material
sj-docx-1-msc-10.1177_09691413231195568 - Supplemental material for The recalibration and redevelopment of a model to calculate patients’ probability of completing a colonoscopy following an abnormal fecal test
Supplemental material, sj-docx-1-msc-10.1177_09691413231195568 for The recalibration and redevelopment of a model to calculate patients’ probability of completing a colonoscopy following an abnormal fecal test by Amanda F Petrik, Eric S Johnson, Matthew Slaughter, Michael C Leo, Jamie Thompson, Rajasekhara R Mummadi, Ricardo Jimenez, Syed Akmal Hussain and Gloria Coronado in Journal of Medical Screening
Footnotes
Acknowledgments
The study team recognizes the work of colleagues Michael Kwon, Hannah Dinh, Victoria Gawlik, Jeffrey Gibbs, and Leslie Mosso. We also acknowledge the creation of the original STOP CRC model in partnership with OCHIN.
Authors’ contributions
AP, ES, and GC wrote the main manuscript text. MS, SH, and ML conducted analysis and wrote sections of the results. JT was the project director and added context to the manuscript including the results and discussion. RM was the clinical expert for all gastroenterology expertise and questions. RJ is the medical director of SeaMar and gave the context of the project in the SeaMar setting.
Availability of data and material
The data that support the findings of this study may be available from Kaiser Permanente Northwest but restrictions apply. Data may be available from the authors upon reasonable request and with the permission of SeaMar. Contact the corresponding author (Amanda Petrik) to request data availability.
Ethics approval and consent to participate
PRECISE has obtained approval from the Kaiser Permanente Northwest Institutional Review Board (IRB protocol number: 00000779), which has granted a waiver of informed consent, because the study involves minimal risks to patients. All methods were carried out in accordance with relevant guidelines and regulations.
Declaration of conflicting interests
Since 2020, Dr. Coronado has served as a scientific advisor for Exact Sciences and Dr. Coronado and Dr. Mummadi have been funded through a contract with Guardant Health and the Kaiser Permanente Center for Health Research.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the National Cancer Institute, (grant number R01CA218923). Amanda Petrik's time during manuscript preparation was partially supported by grant number K12HS026370 from the Agency for Healthcare Research and Quality.
Trial registration
National Clinical Trial (NCT) identifier: NCT03925883.
Supplemental material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
