Sage Journals: Discover world-class research

Abstract

Background:

Produce prescription projects (PPRs) allow healthcare professionals to “prescribe” fruits and vegetables for patients experiencing food insecurity and a diet-related chronic disease. Evaluation of healthcare outcomes, utilization, and costs data is prudent to understand the impact of PPRs. However, substantial challenges exist. The objective of this study is to understand facilitators, barriers, lessons learned, and emergent best practices for data derived from electronic health records (EHR) among PPRs.

Design and methods:

A multiple methods case study including four PPRs funded through a pilot grant to use EHR-derived data to measure healthcare outcomes, utilization, and costs of health care. Data sources included grant applications (n = 4), data use agreements (DUA; n = 4), memoranda of understandings (n = 4), pre/post healthcare outcomes and utilization data, and qualitative interviews/focus groups (n = 10). For analysis we used: descriptive statistics; paired t-tests for changes in values pre/post PPR; and thematic qualitative analysis to construct themes.

Results:

The four cases shared varied healthcare outcomes and utilization measures and submitted less data than was outlined in their respective DUA. Three salient themes emerged: PPR projects need: (1) sufficient time and resources to develop procedures to collect and share healthcare data; (2) tailored healthcare outcome measures to PPR design, outcomes of interest, and EHR capabilities; (3) technical support related to technology, data security and sharing.

Conclusions:

EHR data can provide insight on the impact of PPRs and related healthcare interventions on health outcomes and cost-effectiveness. Evaluation efforts must consider project capacity and ensure adequate resources to collect and securely share healthcare data.

Keywords

Food is medicine produce prescription case study health outcomes healthcare cost healthcare utilization food security nutrition security electronic health record evaluation

Significance for public health

Prevention and management of diet-related chronic disease is complex for households that experience food and nutrition insecurity. Many public health, clinical, and community organizations are working together to mitigate food and nutrition insecurity through interventions such as Food is Medicine (FIM). The suite of FIM interventions includes produce prescriptions, which allow healthcare professionals to “prescribe” fruits and vegetables for patients experiencing food insecurity and a diet-related chronic disease diagnosis. Rigorous evaluation of produce prescription projects (PPR) is lacking. Given the intersection with healthcare, evaluation of healthcare outcomes, utilization, and cost data through the electronic health record (EHR) appears to be practical, yet challenges exist. This multiple case study evaluation elucidates facilitators, barriers, lessons learned, and emergent best practices among active PPR projects collecting, sharing, and evaluating data derived from electronic health records (EHR).

Introduction/background

Food is medicine to promote food security and chronic disease prevention and management

Fruits and vegetables (FVs) are critical to prevention and management of diet-related chronic conditions, such as type 2 diabetes and cardiovascular disease.^1
–3 Adequate intake of FVs can be challenging for those residing in the United States (US) with low-income and food or nutrition insecurity.⁴ Food Is Medicine (FIM) approaches are being integrated into healthcare systems in the US to address chronic disease management, recognizing the critical role nutrition plays in health outcomes. FIM programs pair improved access to healthful foods, such as fruits and vegetables (FVs), with intervention strategies (e.g., nutrition education) to eligible patients in the healthcare setting. They involve partnerships between healthcare organizations and other groups, such as community-based organizations and local food growers/producers.^5,6

Produce prescription projects (PPRs) are an increasingly common type of FIM intervention focused on increasing FV intake.⁷ These projects allow healthcare professionals to “prescribe” FVs for patients experiencing food insecurity and a diet-related chronic disease diagnosis. Since 2010, hundreds of PPRs have been launched throughout the US.^7,8 These programs vary widely in priority audience, screening and eligibility procedures, prescription value, delivery mechanism (e.g., produce box, voucher), funders, implementation approaches, and evaluation components. Growing evidence suggests PPRs may increase FV purchasing⁹ and consumption¹⁰; reduce household food insecurity^11,12; improve clinical health outcomes including hemoglobin A1c,^13,14 diastolic blood pressure,¹⁵ and body mass index¹⁶; decrease cost to the healthcare system related to providing care¹⁷; and improve patient experiences.^18,19

USDA Gus Schumacher Nutrition Incentive Program (GusNIP) overview

The United States Department of Agriculture (USDA) National Institute of Food and Agriculture (NIFA) funds Produce Prescription Programs (PPRs) through the Gus Schumacher Nutrition Incentive Program (GusNIP). GusNIP is part of the USDA’s four pillar approach to nutrition security²⁰ that contribute to the White House Conference goals of “ending hunger and reducing diet-related diseases and disparities” through the National Strategy on Hunger, Nutrition, and Health.²¹ Since it was established in 2019, the GusNIP family of funds has provided $60,189,825.92 million in funding to 126 PPR projects across the US. Additionally, GusNIP funds a Nutrition Incentive Program Training, Technical Assistance, Evaluation, and Information Center (GusNIP NTAE Center) cooperative agreement with NIFA. The GusNIP NTAE Center (“NTAE”) primary awardee is the Gretchen Swanson Center for Nutrition, in partnership with Fair Food Network and University of California San Francisco. The NTAE created and leads the Nutrition Incentive Hub, a coalition of specialists who provide expertise about GusNIP programs.

GusNIP PPR projects are required to include a healthcare partner or be a healthcare entity themselves (e.g., hospital, federally qualified health center). GusNIP PPR projects are required to enroll individuals who are (1) eligible for income-qualifying benefits like Supplemental Nutrition Assistance Program (SNAP) or enrolled in Medicaid and (2) a member of a low-income household who has or is at risk of developing a diet-related health condition.²² PPR projects provide eligible participants with prescriptions for fresh FVs, which are typically redeemed at collaborating farmers markets, grocery and corner stores, and in healthcare settings (e.g., one morning per week food box distribution at a clinic).⁷ Finally, nutrition education and/or other supporting services, such as transportation, are commonly added to augment programs.^23,24

Evaluation of produce prescription projects with electronic health record data

The 2018 Farm Bill that established GusNIP requires that the NTAE conduct an overarching evaluation of GusNIP projects.²⁵ PPR evaluation includes health outcomes, utilization, and cost to the healthcare system related to providing care, among others.²⁶ Upon funding, all GusNIP PPR grantees agree to report on these outcomes. In addition to a survey that all GusNIP PPR grantees are required to collect with a subset of participants,²⁷ one potential source for these data are electronic health records (EHRs). EHRs can provide relevant data documented within the clinical care process. Several studies have leveraged EHRs to evaluate FIM interventions’ effects on biomarkers or healthcare utilization.^{9,14,16,28,29}

Many GusNIP grantees’ leverage EHR data to meet requirements for reporting on health outcomes, utilization, and cost to the healthcare system related to providing care. However, EHR data can be difficult for PPR projects to access and evaluate for myriad reasons. Notably, some GusNIP grantees are clinical organizations that have easier access to EHR data, while others are community-based organizations that must work with their clinical partners to obtain this access. GusNIP grantees have both formally and informally reported to the NTAE the following as challenges for collecting, accessing, and/or sharing EHR-derived healthcare data:

PPR project is not connected to EHR data and/or it is difficult to access within clinical workflows.

Partners do not have the time or capacity to perform additional tasks related to evaluation requirements (e.g., EHR programming, EHR data extraction).

Staff are unsure how to derive data from EHRs.

Uncertainty exists on how to initiate and implement data sharing agreements between community-based organizations and healthcare entities.

Institutional Review Board (IRB) processes to seek approval to conduct human subjects research (e.g., accessing EHR data) are difficult and time-consuming to navigate.

In 2022, 86% of PPR grantees reported plans to access EHR data for their project evaluation. By the end of these grantees’ funding cycles, almost none had successfully accessed nor utilized EHR data.³⁰ Due to these challenges, the NTAE offered all active PPR grantees the opportunity to apply for a small grant to participate in a case study about deriving EHR data for their PPR project evaluation. The small grant mechanism offered financial support to access health outcomes, utilization, and cost data from the EHR; specialized technical assistance from the NTAE; and occasions to share qualitative data about the facilitators and barriers faced when collecting, extracting, and sharing these EHR data. Therefore, the purpose of this case study was to answer the following research questions:

What is the quality (e.g., completeness) of data derived from EHRs, as originally planned through a data use agreement (DUA) and memorandum of understanding (MOU)?

What is the experience of GusNIP PPR grantees who aim to access healthcare outcomes, utilization, and cost data through EHR?

What barriers and facilitators exist for GusNIP PPR grantees to collect and share healthcare outcomes, utilization, and cost data derived from EHRs?

How can the NTAE support GusNIP PPR grantees in the processes involved in EHR-derived healthcare outcomes, utilization, and cost data?

Methods

Conceptual framework

Constructivism was used to frame this project, which is built on the premise of a social construction of reality. A key advantage of this approach is close collaboration between the researcher and the participant while enabling the participant to talk about his/her experiences. It is through this discourse that participants are able to describe their views of reality, and this enables the researcher to better understand the participants’ actions and experiences.³¹

Research design and methods

An instrumental, multiple methods, multiple case-study design was employed for this project, drawing upon quantitative and qualitative data. Due to its flexibility and rigor, this approach is valuable in understanding approaches to PPR evaluation in healthcare.³² The unit of analysis is defined as each GusNIP PPR grantee (n = 4) who participated. An instrumental approach is often used in situations where the case itself is of secondary interest; for instance, the case plays a supportive role in facilitating understanding of a broader question. Findings include a variety of perspectives from participating grantees; however, individual stories are not the project’s focus. Case study methodology allows researchers to view problems from multiple perspectives, thereby enriching the meaning of a singular perspective.³² Engaging multiple cases allows researchers to triangulate findings across different contexts to better understand similarities and differences.³³

For the quantitative component of the case study, each program had the flexibility to submit its own chosen data elements that aligned with health outcomes, utilization, and cost to the healthcare system, which was outlined in a DUA and MOU. To better understand the process experienced by these partners, qualitative interview and focus group data provided a richer, deeper level of insight.^34
–37 Since GusNIP PPRs vary in terms of partnerships, capacity, staff expertise, and other factors, it is important to understand the experience of multiple types of projects. This study was approved by the University of Nebraska Medical Center Institutional Review Board (#829-20-EX) as exempt.

Project selection

This case study is bound by both the geographic location of each participating GusNIP grantee and the 14-month small grants funding cycle (May 2022–August 2023). PPR grantees were selected to participate based on their application to a Request for Applications to a Small Grants Case Study Project. Applicants (i.e., GusNIP PPR grantees) that were selected for a small grant demonstrated that they, in collaboration with their healthcare partner, were able to (1) extract healthcare outcomes, utilization, and cost data for individuals enrolled in the PPR before, during, and after participation and (2) link cost and utilization data with clinical metrics from EHRs and GusNIP surveys. As outlined in the Request for Applications, these applicants also needed to be able to access similar healthcare outcomes, utilization, and cost data from a sample of non-PPR participants to serve as a matched comparison group (e.g., control). Awardees agreed to provide de-identified data to the NTAE. In all, five grantees applied and were selected for funding, though only four ultimately participated. To distinguish these five grantees from all other GusNIP PPR grantees, they will be called “Cases” throughout the remainder of this paper.

Data collection procedures

Multiple methods of data collection were employed to triangulate findings for a rigorous, complete picture of each Case experience.³³ First, the NTAE and each Case set up a DUA and MOU to partner in the research project. The MOUs were established to formally outline the specific roles and responsibilities to be undertaken by each party (NTAE and Case), while the DUAs clearly delineated which data was to be collected and securely shared between partners as part of this research project. DUAs delineated that data to be shared would include: healthcare outcomes (HbA1c; blood pressure; anxiety severity³⁸; depression severity),³⁹ utilization measures (cost of health care provided, charged or billed amounts; attended, no show, and canceled appointments), number of PPR prescriptions redeemed, pre/post survey data, and associated timeframe to submit data. Of note, no protected health information (e.g., identifying information or protected health information [PHI]) was shared between the Cases and NTAE for this study thus obtaining patient-level consent was not required.

Researchers quantified the number of exchanges needed between NTAE and Cases to secure DUA and MOUs. To do so, a comprehensive review of all correspondence between Cases, affiliated partners, and the NTAE regarding the execution of the required DUAs and MOUs was systematically categorized, including: contract dates initiated and finalized, number of individuals involved and their role, number of email interactions, number of contractual revisions needed before a final version was signed and fully executed, as well as observed key challenges and facilitators throughout the process. Additionally, NTAE-based researchers conducted a comprehensive document analysis of each Case small grants application narrative in comparison with their fully executed DUA and MOU with the NTAE to compare and contrast data submitted versus proposed in the DUA/MOU (of note, in all cases, less data was submitted than was approved in the DUA/MOU).

For healthcare utilization and outcomes, selected measures were collected and recorded by healthcare providers per standard of clinical care and extracted from the EHR. Additionally, corresponding dates were provided for some health values to align the measures with the baseline or post timeframes of the PPR intervention period. Each Case submitted their EHR-derived data to private folders using a secure SharePoint site set up by the NTAE.

Semi-structured qualitative interviews were conducted with each Case at the start of the small grants funding cycle (October 2022–January 2023). At the funding cycle mid-point (February–April 2023), grantees participated in one of two focus groups to discuss their progress. After the full funding cycle and data transfer to the NTAE were completed, each Case participated in a close out interview (August–October 2023). The same lead qualitative researcher conducted all interviews and focus groups to provide continuity and fidelity in the data collection methods.⁴⁰ Moderator guides and types of data collection (i.e., focus group versus interview) for each data collection point can be found in Table 1. Each Case had the option to invite more than one representative to their pre and post interview—which is why some were focus groups and some were interviews. The mid-point focus group was selected as the data collection method because collectively the Cases wanted the opportunity to have a structured conversation amongst themselves to learn from one another at this mid-point.

Table 1.

Moderator guide and data collection methods used for qualitative data collection.

Interview timepoint	Data collection method	Moderator guide questions
Pre (prior to project period) August to October 2022	Individual interviews (n = 4) Focus groups (n = 2) with two to four individuals each	● Walk me through your motivation and thought process when applying for this award ● What do you expect this project will be like for your GusNIP healthcare cost and utilization grant project? ● Tell me about the infrastructure/relationships/roles you have in place to facilitate this GusNIP Healthcare cost and utilization grant project ○ Probes: people, positionality, organizational collaboration, MOU ● Tell me about infrastructure/relationships/roles you will need to have in place in the near future to facilitate this GusNIP Healthcare cost and utilization grant project ● How can the NTAE facilitate your success of this GusNIP Healthcare cost and utilization grant project? ● Who else should I talk with at your organization?
Mid point (halfway through project period) February to April 2023	Focus group (n = 2) with three to six individuals each	● How are things going with your GusNIP Healthcare cost and utilization grant project? ○ Probes: challenges, successes, barriers, lessons learned, decisions about what data to report ● Tell me about the MOU, data sharing, and agreements process ○ Probes: sharing with NTAE ● Walk me through the technology involved in your process. Probes: EHR, sharing with NTAE ● As comprehensive as you can be, walk me through all of the people (and their positions) involved in this effort at your organization ● Moving forward, what do you expect the next part of your project to look like? ○ Probes: lessons learned, new strategies ● If you had to tell another grantee about collecting and sharing healthcare cost, utilization, and outcomes data for your GusNIP PPR project, what would you say? ● How can the NTAE facilitate your success of this GusNIP Healthcare cost and utilization grant project? ● If we could directly tell our funder (USDA NIFA) what collecting and sharing healthcare cost, outcomes, and utilization is like, what would you say? ● What have we not yet discussed in terms of this project that you’d like to share with me?
Post (after project completion: September to December 2023	Focus groups (n = 3) with tow to three individuals) Individual interviews (n = 3)	● Tell me about your role within your organization ● Tell me about your experience with the small grants project ● Tell me about the key personnel who were involved with small grants at your organization ○ Probes: data collection, administration, approvals, EHR, technical support ● Think back to 1 year ago before you started the small grants project—how does the experience compare to your initial thoughts before you began the project? ● If another produce prescription program came to you for advice on how to collect healthcare cost, utilization, and outcomes data—what would you tell them? ● Tell me about the data transfer with the NTAE ● The NTAE has had challenges supporting grantees with getting this type of EHR data—what are your recommendations as to how the NTAE could better serve grantees? ● From an organization perspective, what were key opportunities and challenges related to this project? ● Please share anything else that I haven’t asked about regarding the small grants project at your organization

EHR: electronic health record; NTAE: Nutrition Incentive Program Training, Technical Assistance, Evaluation, and Information Center; PPR: produce prescription program; USDA NIFA: United States Department of Agriculture, National Institute of Food and Agriculture (NIFA).

Analysis

EHR-derived data that was obtained from each Case was compared to the DUAs that were submitted to identify alignments and gaps. The missingness in healthcare utilization data (clinic visits, no show appointments, appointments canceled) and outcome measures (HbA1c, blood pressure, anxiety severity, depression severity) were compared for the pre and post periods within participants using descriptive statistics (frequencies, percentages). Additionally, descriptive statistics were conducted to quantify the number of exchanges needed between NTAE and Cases to secure DUAs and MOUs. Following data review, we identified that control group data was missing for the majority of cases. For the impact analysis, a one-group, pre/post design was used to compare averages (using paired t-tests) for continuous normally distributed outcomes and medians (using the Wilcoxon Signed Rank test) for continuous outcome variables that were not normally distributed. Analyses were conducted using STATA (version 18).

All interview audio recordings were professionally transcribed verbatim. The lead qualitative researcher checked each transcript for accuracy against the audio recordings. Next, researchers used Atlas.ti (Version 24.0.1) as a digital qualitative management tool to facilitate organization and analysis.⁴¹ The interview transcriptions were coded using qualitative content analysis methods,^42,43 which helped generate comparisons across cases to understand salient cross-case themes. The data were coded in various quotation increments depending on context of the quotation.⁴⁴ The first pass of coding involved inductive free coding, which was narrowed by collapsing and integrating codes to remove redundancy during the second pass which involved describing and defining each code. To enhance rigor, a sample of the transcripts were double coded, with the lead qualitative researcher (who collected all data) coding 100% of the transcripts and a second coder independently double coding 35% of the transcripts. The two coders met weekly to review discrepancies and resolved discrepancies with members of the larger research team. Code and concept maps⁴⁵ were developed to serve as visual network representation of the coded data and to facilitate emergent categories across the different transcripts, eventually leading to three main themes. Documents (e.g., proposals and MOUs) were analyzed using similar methods and coded separately from the transcribed interviews. Descriptive statistics were conducted to quantify the number of exchanges needed between NTAE and Cases to secure DUAs and MOUs.

Results

Overview of cases (n = 4)

Table 2 presents a summary of the four Cases who participated in this project. Case A is a large healthcare organization located in a large metropolitan area (Central US), Case B is a single federally qualified healthcare center (FQHC) located in a moderately-sized metropolitan area (West US), Case C is a community-based organization that partners with an FQHC in a rural area (Southwest US), and Case D is a large network of FQHCs located in a large metropolitan area (West US). Grantees served populations ranging from approximately 300 to over 3000 annually across one or more clinics. Often, the EHR projects focused on a single clinic or a specific subpopulation, resulting in different numbers of participants than the overall project.

Table 2.

Detailed description of each case, DUA and MOU language, budget use, and data provided and differences between DUA and shared dataset.

Case	Organization	PPR description	Key personnel involved	Proposed evaluation description	DUA language	Budget use	Shared dataset description	DUA and shared dataset differences
Case A	Large healthcare system with centers for outpatient care	At monthly visits over 6 months, participants receive either $60 for small households (<4) or $120 for larger households (≥4) to purchase FVs Participants also receive healthy foods for themselves and their families, nutrition education, grocery shopping tips, recipe ideas, and connections to wrap-around services such as SNAP, WIC, and housing support Eligibility requirements: Enrolled in Medicare, Medicaid, or Veteran’s Affairs insurance Screen positive for food insecurity Screen positive for diet-related chronic disease	Project manager for healthy food centers Informatics manager for reporting Data analyst	Pre/post HbA1c from EHR for patients with type 2 diabetes Data provided from PPR participants and control group not receiving PPR	Responsibilities: Extract selected utilization data (and cost/charge/billed amount if available) for individuals enrolled in the PPR before, during, and after participation Extract same selected utilization data (and cost/charge/billed amount if available) for a comparison group of individuals not enrolled in the PPR during the same time period Link utilization data (and cost data, if available) with clinical data and survey data Share data, de-identified with a numeric identifier Utilization measures: Number of visits to PPR distribution center Number of visits with dietitian or dietetic technician Clinical measures: HbA1c	Budget used to support FTE of 3 key (existing) staff members	Dataset with 100 EHR records from PPR participants Variables include HbA1c (up to seven values; three from pre and four from post), total number of clinic visits, and counts of visits for ancillary services (e.g., RDN) ● Missingness in the HbA1c measurements ● No survey data to link ● No data from a comparison group	● Plan for a two-group study design; able to conduct 1 group pre/post design Missing Elements: ● Comparison group data ● Cost/charged/billed amount ● Time data (for clinic and dietician visits) to understand pre or post times ● Number of visits to PPR distribution center ● PPR survey data ● 38% of the HbA1c measurements do not have both pre and post values
Case B	An FQHC with food hub component serving a diverse community through holistic care	Participants receive weekly boxes (valued at $14) of fresh produce for themselves and their families. Boxes include nutrition education, recipes, and tips for food sovereignty and are distributed by pick-up or delivery. Produce selection is tailored to the cultural palette of community members. Eligibility requirements: ● One or more chronic medical condition (patient or household member) ● Live in income defined zip codes	● Operations manager ● Research coordinator ● Research assistant	A formative/exploratory project	Responsibilities: ● Explore how to extract selected cost (or charge/billed amount) and utilization data for individuals enrolled in the PPR before, during, and after participation ● Explore how to link that cost (and/or charge/billed amount) and utilization data with clinical data (from electronic medical records) and participant-level core metrics survey data ● Share these data, de-identified with a numeric identifier Utilization measures: ● Patient appointment/no show rates Clinical measures: ● HbA1c ● BP	Budget used to support FTE of three keys (existing) staff members	● Aggregated dataset by month (Feb 2021–July 2023) ● Variables include HbA1c, blood pressure, hypertension diagnosis, and no-show rates ● Additional data for participants’ enrollment date, insurance information, billing codes and costs for services received ● Aggregated data cannot be linked to survey data ● Cost data can be linked to survey data	● Plan was to submit two time points for each participant, but monthly data submitted Missing elements: ● Aggregated data cannot be analyzed at the participant level ● Time data for billing codes and costs for services received to understand changes in costs before/after program ● PPR survey data was not provided
Case C	A non-profit organization and FQHC providing mental health services in a rural county	A holistic 6-month program, participants participate in nutrition education, cooking education classes, and mental health wellness. Participants receive a produce prescription in the form of a bi-weekly box (valued at $44) full of fresh FVs Eligibility requirements: ● Screen positive for food insecurity ● Behavioral health patients with no specific health outcome requirements	● Manager for community farm programs ● Clinical supervisor of behavioral health	A formative/exploratory project	Responsibilities: ● Extract selected utilization data for individuals enrolled in the PPR before, during, and after participation ● Link utilization data with clinical data and survey data ● Share these data, de-identified with a numeric identifier Utilization measures: ● Number of new clinic visits ● Number of follow up visits ● Number of appointment visits ● Number of canceled visits Clinical measures: ● PHQ-9 ● GAD-7	Budget provided to clinic staff to offset time for EHR data extraction	● Dataset with 26 records from PPR participants at two clinics ● Variables include all clinic visits attended (#), no-shows (#), cancellations (#), and PHQ-9 and GAD-7 responses ● Clinical data is linked to survey results	● One group pre/post test could be conducted as planned Missing elements: ● 15% of the GAD-7 and PHQ-9 data was missing ● 24% of the PPR survey data was missing, however could not be analyzed due to accuracy concerns related to inconsistent identification number alignment ● Number of new visits and number of follow-up visits not provided; total number of appointments provided instead
Case D	A FQHC providing primary medical, dental, and behavioral health care services at 17 health centers (4 of which participate in PPR) in an urban county	For 6 months, participants receive an electronic debit card redeemable for $40/month for fresh FVs at participating Kroger grocery stores Eligibility requirements: ● Enrolled in state-based Medicaid program, ● Screen positive for food insecurity, and ● Meet one of the following criteria: (1) have a diagnosis of type 2 diabetes, or (2) enrolled in the National Diabetes Prevention Program	● Director for quality, health equity, and innovation ● Program manager for community wellness ● Quality improvement program coordinator	Pre/follow up survey & pre/post HbA1c from EHR for patients with type 2 diabetes	Responsibilities: ● Extract selected cost (or charge/billed amount) and utilization data for individuals enrolled in the PPR before, during, and after participation ● Extract same selected cost and utilization data for a comparison group of individuals not enrolled in the PPR during the same time period ● Link cost and utilization data with clinical data and survey data ● Share these data, de-identified with a numeric identifier Utilization measures: ● Number of total clinic visits ● How many clinic visits (numbers and percentages) were medical visits with the patient’s PCP ● How many clinic visits, how many (numbers and percentages) were visits with ancillary services (RD, Health Educator, or Behavioral Health) ● Total number of times patient did not show for a medical visit with the patient’s PCP Clinical measures: ● HbA1c ● BP	Budget used to hire new/additional staff member to focus on EHR data extraction	● Robust dataset from 46 pilot PPR participants and 46 control participants ● Variables include BP, HbA1c, and total number of clinic visits from within 90 days before and 90 days after enrollment ● Missingness in the control group data ● Clinical data not matched with survey data	● Two-group study design planned, but one group pre/post implemented Missing elements: ● Charged and billed amount ● Clinic visits (total) missing for 20% of the PPR group ● Clinic visits (primary care only) ● PPR participant survey ● Control group data was provided, however, substantial missingness due to the cadence of clinic visits. 100% of the comparison group is missing pre and post data for HbA1c or BP ● 61% of PPR group has missing HbA1c data for pre and post ● 41.3% of PPR group has missing BP data

BP: blood pressure; DUA: data use agreement; EHR: electronic health record; FQHC: Federally Qualified Health Center; FTE: full time equivalent; FVs: fruits and vegetables; GAD-7: Generalized Anxiety Order, 7-item; MOU: memorandum of understanding; PCP: primary care providers; PHQ-7: Patient Health Questionnaire, 9-item; PPR: produce prescription program; RD: Registered dietician.

Of note, a fifth grantee applied and engaged in a “pre” interview with the research team. However, shortly after that interview, the staff member of that healthcare organization who was leading the effort resigned from their position. Following, the organization did not have sufficient staff capacity to engage in the project and withdrew their application. No DUA, MOU, funding distribution, or data transfer occurred. Therefore, the fifth grantee was not included in any tables or results in this paper.

Establishing DUAs and MOUs and submitting EHR data

Establishing DUAs and MOUs proved to be time and resource intensive among all four cases. Challenges centered around identifying the “right” person at any given organization to execute and sign the documents and identifying the type of data (e.g., metrics) to collect and share. Table 3 provides a quantified representation to understand the processes, time/resources and expertise needed, and experience with executing DUAs and MOUs with each grantee.

Table 3.

Frequency of counts related to the process of securing MOUs and related data sharing/safety agreements with each grantee.

	Case A: Large healthcare system with centers for outpatient care	Case B: FQHC with food hub & holistic care approach	Case C: Non-profit and FQHC providing mental health services	Case D: FQHC providing primary medical, dental, and behavioral health care services
Number of days to complete contracts	186	350	160	35
Number of individuals involved	10	4	5	4
Number of emails between case personnel and research team	106	52	25	23
Number of MOU drafts	7	3	3	1
Number of DUA drafts	10	1	1	1

DUA: data use agreement; FQHC: federally qualified health center; MOU: memorandum of understanding.

Descriptions of each Case and MOU/DUA language, budget use, and EHR data provided are shown in Table 2. The four Cases varied in priority populations, program delivery strategies, and the outcomes that were submitted from their EHR. Therefore, NTAE researchers chose not to combine data across these programs, and instead analyzed individually for each case as the data allowed. All four of the Cases submitted data that had missing elements including comparison data (e.g., control group), utilization data (e.g., missed versus attended appointments; charged or billed amounts), time data to compare outcomes during the baseline and post period, number of visits to the PPR distribution center to pick up fresh FV, and PPR survey data. For Case B, aggregated monthly data was submitted instead of participant-level data that could be used to analyze participant outcomes. This was related to limitations in the EHR technology available to Case B. As delineated in Table 2, only one Case (B) shared cost data in the form of billing charges. The other three Cases had no access to cost data.

For the health outcomes that were submitted, there was missingness when baseline and post data were matched for each participant; this missingness ranged from 0% to 100%, depending on the specific outcome. A description of the quantitative data shared is found in Table 4.

Table 4.

Description of EHR-based quantitative data shared from each case.

Case and Measures	n	Baseline value		Post value		Diff.	Test statistic; p value
	n	Median (IQR)	Mean (SD)	Median (IQR)	Mean (SD)	Diff.	Test statistic; p value
Case A measures
HbA1c^a	45	7.9 (2.5)		7.2 (1.8)		−0.7	Z = 1.94; p = 0.0518
Case C measures
Clinic visits—total^b,c	26		105.08 (39.88)		87.12 (48.46)	17.96	T = 2.57; p = 0.0166
No show appointments—total^a,c	26	12 (29)		5 (11)		−7	Z = 3.19; p = 0.0008
Appointments canceled^a,c	26	21.5 (20)		13 (20)		−8.5	Z = 2.96; p = 0.0021
Anxiety severity^a	22	5 (10)		2 (10)		−3	Z = 1.92; p = 0.0500
Depression severity^a	22	5.5 (10)		3 (5)		−2.5	Z = 2.88; p = 0.0030
Case D measures
Clinic visits—total^a,b	37	1 (1)		1 (1)		0	Z = 0.05, p = 0.9805
HbA1c^a	17	8.1 (1.7)		7.7 (2.5)		−0.4	Z = −0.71; p = 0.9540
Blood pressure (systolic)^a	18	131.5 (16)		122.5 (21)		−9	Z = 0.55; p = 0.6012
Blood pressure (diastolic)^a	18	73 (18)		76 (18)		3	Z =−0.24; p = 0.8234

Diff.: difference between baseline and post values.

Cases A, C, and D provided data. Case B did not provide analyzable data.

Outcome variable was not normally distributed, so Wilcoxon signed rank score test was used.

Outcome variable was normally distributed, so the paired t test was used.

Clinic visits, no show appointments, and appointments canceled measured in the last 6 months.

Changes in health outcomes

For health outcome variables, Case A (n = 45) used HbA1c and observed a non-statistically significant decline in median values (p = 0.052). Case B provided no health outcome data. Case C measured anxiety severity and depression severity as their main health outcomes. Among the 22 participants, median anxiety severity (p = 0.05) and depression severity scores significantly decreased (p = 0.003). Case D used HbA1c and blood pressure (SBP and DBP) values. Median HbA1c values (n = 17) declined after the program, but this difference was not statistically significant (p = 0.95). For SBP (n = 18), values improved slightly but were not statistically significant (p = 0.60). DBP values increased slightly, but were not statistically significantly different (p = 0.82).

Changes in healthcare utilization

Healthcare utilization was analyzed for Case A, using total clinic visits over 6-month PPR participation period, and no statistically significant changes were observed after the program (p = 0.98). The median number of visits both at baseline and after the program was one visit (IQR = 1). Case B provided no healthcare utilization data. Case C provided the total number of clinic visits (attended), the total number of no-show appointments, and the total number of appointments canceled for 26 participants. The average number of clinic visits decreased and (95% CI: 3.55, 32.37) was statistically significant (p = 0.017); the median number of no-show appointments (p < 0.001) and the median number of canceled appointments (p = 0.002) also decreased. Of note, Case D did not provide data to analyze for healthcare utilization.

Qualitative findings

Three salient themes emerged from the qualitative dataset. These included: (1) facilitators to collecting and sharing EHR-derived healthcare outcomes, utilization, and cost data; (2) challenges encountered with collecting and sharing EHR-derived healthcare outcomes, utilization, and cost data; (3) capacity building and future directions in collecting and sharing EHR-derived healthcare outcomes, utilization, and cost data. Categories that support each theme as segregated by each grantee are illustrated in Table 5.

Table 5.

Qualitative themes delineated by each case.

Themes	Case A	Case B	Case C	Case D
Theme #1—facilitators to collecting and sharing EHR-derived healthcare outcomes, utilization, and cost data
High motivation	✓	✓	✓	✓
Strong quality improvement and/or evaluation team familiar with EHR extraction	✓			✓
One dedicated staff member to extract data from EHR	✓	✓	✓	✓
Strong leadership support	✓	✓	✓	✓
Theme #2—challenges encountered with collecting and sharing EHR-derived healthcare outcomes, utilization, and cost data
Limitations in technology/need for manual extraction		✓	✓	✓
Determining best metrics to accurately evaluate PPR		✓	✓	✓
Missing data	✓	✓	✓	✓
Time-consuming process of deidentification and manually checking for PHI	✓	✓	✓	✓
Naiveté regarding working with EHRs		✓	✓	✓
Competing priorities between key partners			✓
Time/capacity		✓	✓	✓
Identifying appropriate leadership to execute MOU/DUA	✓
Theme #3—capacity building and future directions in collecting and sharing EHR-derived healthcare outcomes, utilization, and cost data
Appreciate flexibility/tailored planning		✓	✓	✓
Now understand what staff are needed for this type of program evaluation		✓	✓	✓
Built capacity for research team		✓	✓	✓
Ability to share outcomes with key partners, and inspire renewed interest in future research	✓	✓	✓	✓
Understand what kind of upfront communication with all partners is needed to establish shared priorities		✓	✓
New understanding of which leadership and team members are needed to execute contracts and provide templates for contracts	✓		✓
Now understand capacity, limitations, and needs of existing technology available		✓	✓	✓
Gained confidence in determining what research questions and metrics are best to evaluate unique programs		✓	✓

DUA: data use agreement; EHR: electronic health record; MOU: memorandum of understanding; PHI: private health information; PPR: produce prescription program.

Theme #1: Facilitators to collecting and sharing EHR-derived healthcare outcomes, utilization, and cost data

High motivation to learn how to and to “practice” collecting and sharing EHR-derived healthcare outcomes, utilization, and cost data was a key facilitator across Cases. Participants expressed the desire to better evaluate their programs, expand opportunities to receive funding, and share findings with participating parties, such as healthcare providers and local growers/food suppliers who provide fresh FV for the PPR projects. One grantee shared:

And so, I’m really proud of what we found. I think this is the first time we’re able to do it and the results are super exciting to share back with our providers. We did it at the provider meeting last week and then tomorrow, we’re going to get some feedback on what does this data mean to you? So really creating a culture of using data to affect services and care. And the other exciting thing on the other spectrum is tomorrow we’re having a farmer’s dinner with the farmers who (. . .) created or grew the produce, and they get to see these graphs, like how did their produce affect people’s health? And I think that’s really exciting for us. Case B

Motivation also stemmed from grantees striving to be compliant with GusNIP PPR grant funding requirements. Other key facilitators included strong leadership support, a dedicated (paid) staff member to extract the EHR data, and in two of the four Cases, a strong internal evaluation and quality improvement team.

Theme #2: Challenges encountered with collecting and sharing EHR-derived healthcare outcomes, utilization, and cost data

Participants also shared key challenges in collecting and sharing these data. Three of the four Cases discussed time as a key challenge, given the need to manually extract data from their EHRs. One Case explained:

I can’t just punch in names and then pull a data and then get a compile list. I had to go individually into somebody’s [chart], then I had to punch in the data for them, and then. . . I had to put in the particular dates, and then I had to pull the data and then transfer it [to the spreadsheet]. I had to go in individually for each client to do that (. . .) and sometimes the system isn’t always cooperating, so you have to finagle it a little bit. Case C

Participants from three of the four Cases also noted this was their first time working within their EHRs to extract healthcare outcomes, utilization, or cost data and discovered profound limitations with the technology. These challenges included an inability to automate longitudinal data extraction (e.g., software will only report most recent lab value) and to extract disaggregated data, and an inability for multiple programs and software to “talk to” one another. Participants also discussed concerns with missingness in their datasets and went to great lengths to triple check data for protected health information (PHI) prior to sharing with the NTAE. Another challenge (in two of four cases) was determining what type of healthcare outcome, utilization, and cost data would best answer their research question as tailored to their program implementation protocols, such as inclusion criteria for eligible patients.

Theme #3: Capacity building and future directions in collecting and sharing EHR-derived healthcare outcomes, utilization, and cost data

Participants acknowledged their naiveté in using EHR data for evaluation, cited this project was a “steep learning curve,” and conveyed feeling far more prepared should they engage in such an endeavor again. Clear communication, knowing what resources (e.g., technology and staff expertise) were available, and understanding limitations (e.g., technology) from the start of the project were discussed as helpful learnings for when they work with EHRs in the future. Participants discussed challenges with securing MOU, DUA, and IRB approvals and indicated the capacity needed for securing these agreements and approvals, especially given that this was done as a part of a small (e.g., $10,000) grant. All participants suggested needing flexibility and the ability to tailor their evaluations, which were permitted within this project. They suggested that the ability to choose what metrics “make sense” for their program, technology, and staff capacity was imperative to their success, self-efficacy, and ultimately their willingness to continue this type of evaluation. One participant shared:

So, what I’ve been enjoying, and it goes along with some of the advice, is with this small grant, I like the flexibility that we had to customize our searches based on what’s realistic and feasible for us. So, we did appreciate that because we were able to be creative and just really look into our systems and see, okay, this is possible. This is not possible. So, I think if they were to do something like this, I think giving organizations this flexibility for them to kind of provide this information based on what is feasible for them. Because every healthcare center has a different platform. Some may use Epic, a whole different, we use NextGen. They may use a different population database than we do. So, there’s different limitations and capacities with all these different platforms. So, I think we can have the same flexibility. I think it’s possible. And that’s how we were able to complete this program was because of how [the NTAE] were and us customizing the searches. Case D

Most participants were not aware what type of data they had access to (or did not have access to, in the case of cost data) and expressed needing an opportunity such as this small grant project to familiarize themselves with data availability. Participants discussed the need to have a dedicated, paid staff to conduct the EHR data extraction and emphasized the time it takes, even for skilled employees. The following participant, a quality improvement staff member, suggested:

I mean, funding plays a big role. I mean, in reality, we only had 46 patients, which maybe to some, it’s just considered a little bit of patients. It’s definitely less than a hundred, but these 46 patients just took up a lot of time. And also because of that manual work, and we didn’t have the capacity to pull data so easily from even our own platforms. So, definitely having a staff member that just is dedicated to doing this manual work and this data would be a good thing to invest in. And then we also have to keep in mind, we also have probably other deliverables and other work tasks. So, definitely a staff member to just be dedicated on this project and pulling data. And yeah, I think that’s my number one. Case D

Participants were overwhelmingly grateful for this opportunity and the ability to “practice” without punitive consequences should they be unable to collect data they originally planned to collect (e.g., their DUA did not match the final shared dataset and less data than was proposed in the DUA was shared). When asked how national evaluators (e.g., NTAE) could best support GusNIP PPR grantees in collecting and sharing EHR-derived datasets, participants from one focus group shared:

Speaker A:

I think just stay flexible. (. . .) I really appreciate (that) we’re on this journey together it’s exploration and that is helpful (. . .) That’s the most helpful.

Speaker B:

Yes, I completely agree. I like that we were able to kind of customize this project to fit our agency in our community.

Speaker C:

Same. I want to echo what was just said. I think it’s been really great having that reassurance from your team [NTAE] that everything will be okay and that this is a learning process for all of us. Cases A and B

Discussion

Together, these findings tell a story of each grantee’s experience with collecting, extracting, organizing, and sharing EHR-derived data for their healthcare outcomes, utilization, and cost evaluation. Community-based organizations and non-research medical institutions (e.g., FQHCs) are increasingly engaged in addressing social determinants of health under a clinical care model, such as FIM interventions. These projects require increasing engagement in clinical evaluation processes, despite well-established challenges to using EHR-based data for research.⁴⁶ Findings from this case study analysis can be used as an example of how EHR-based evaluations are operationalized within these non-research institutions.

A key facilitator for all four Cases in this study was high motivation to engage in this project. With a relatively small amount of funding (e.g., $10,000), these four grantees willingly applied to participate in this project, suggesting from the outset they were motivated to engage. Nevertheless, motivation proved to be key throughout the duration of the small grant funding cycle. Further unpacking motivation, we can draw on the Theory of Acceptability in Healthcare Interventions which posits that acceptability is a multi-faceted construct that centers on seven constructs: affective attitude, burden, perceived effectiveness, ethicality, intervention coherence, opportunity costs, and self-efficacy.⁴⁷ Participants verbalized their improved self-efficacy, perceived effectiveness (e.g., need), and burden (e.g., time) throughout the interviews and focus groups. Since motivation was such a key driver in each grantee case’s willingness to participate and persevere despite the challenges in this project, it is prudent to consider how to capitalize and leverage intrinsic motivation of key parties (e.g., CBO leadership and staff, clinical staff) to engage in such evaluation processes.

There are well-established challenges with data sharing within public health research—one literature review summarizes them as technical, motivational, economic, political, legal, or ethical challenges.⁴⁸ Findings from this study can be further mapped with these challenges⁴⁸—in that technology and economic challenges persisted across three of the four grantee cases. The only Case that did not struggle with technical and economic challenges was that of Case A, as this large medical institution had well-built infrastructure, expertise, and resources to collect such EHR-based data. Of note, legal challenges in executing the DUA and MOU documents with this Case were arguably the most challenging and required the most time, personnel, and “back and forth” between the NTAE and grantee. The only Case who was not a healthcare organization, but rather a CBO (Case C), experienced challenges related to data access—this Case relied on an external evaluator and experienced challenges related to new partnerships and period of growth regarding collaborative efforts across entities (e.g., clinic, CBO, external evaluator). Across all Cases, the small grant budget was used to offset FTE of a clinical staff member to extract the EHR-derived data, and it was strongly emphasized that one dedicated paid staff member (versus several different staff contributing) was the most effective way to extract this data. Those who extracted EHR data for this project said they intended to share the task across several colleagues, but in the end, the learning curve was so steep that it was more effective to just “do it myself.” However, they also noted if the sample size were to increase, manual data extraction by just one person would not be sustainable. Finally, though not overtly stated as a barrier, the legal and ethical challenges related to attempts to share PHI required grantees to spend an inordinate amount of time manually checking each dataset prior to uploading and sharing with the NTAE. This manual “triple check” explained by three of the four Cases likely contributed to overextension of the FTE committed to the project.

In all Cases, different entities (e.g., evaluators, clinic staff) and/or technologies (e.g., Qualtrics, EPIC) were used to collect and extract data. This approach required a tremendous amount of time to harmonize, especially when data were deidentified and individual-level records (e.g., survey and EHR data) needed to be linked. In only one Case (C), survey data was linked to clinical data. Of note, this Case requested a 4-month extension to conduct this lengthy process. Case D suggested they could have logistically connected these datasets, but they lacked time and personnel to do so. Two of the Cases (A and B) did not have the ability to connect survey-based data with EHR-extracted data for various reasons. Even within clinical organizations, there were often different platforms or programs where data was housed. Though trying to link de-identified data was not a concern because all clinic staff had access to identifiable data on all platforms/software programs within their organization, manually “searching” different programs/platforms for data was very time consuming and time-prohibitive.

All Cases, as well as the NTAE, recognized the need to build capacity for evaluations featuring EHR-derived data in the future. For three of the four Cases, this was the first time they had ever participated in an EHR-derived data extraction and sharing project. For the NTAE, this project provided an opportunity to establish DUA and MOU templates, a secure site for data transfer, and an understanding for how to best support GusNIP PPR grantees with evaluations in healthcare settings (e.g., provide flexible advice on suggested data to extract). Additionally, all grantees needed to establish or amend IRB approvals to access and share human subjects data (e.g., EHR data) as required for all research involving humans the United States. Some grantees had straightforward access to an IRB (e.g., the healthcare organization itself has its own IRB), while others had to collaborate with University-based collaborators to route the IRB protocol through an external University-based IRB.

The challenge of working across sectors and linking de-identified EHR datasets is documented in the literature.⁴⁹ These findings align with published literature regarding the need for improvements in technology, technical skills, and innovative approaches to integrating datasets.⁴⁹ In addition, there are opportunities for technical assistance and support from funders and evaluators that can mitigate aforementioned challenges. First, enhancing intrinsic motivation from key players involved (e.g., CBO, clinical organization) and harmonizing motivational factors and goals across parties from the project’s inception would improve communication and mitigate competing priorities. Second, it is prudent to underscore flexibility and tailoring of what kind of metrics should be collected given differences in PPR protocols, grantee data access and infrastructure, and types of data available. Providing support to decide what metrics adequately answer the research and evaluation questions specific to any given PPR requires those giving advice to understand the PPR in its entirety (e.g., patient inclusion criteria, relationship with clinical site, leadership support, technical infrastructure, etc.). Third, templates for DUAs and MOUs, scripts for email communication with leadership, opportunities for peer-to-peer learning, and examples of metrics (e.g., healthcare outcomes) can be helpful tools for organizations for whom these processes are new. Additionally, suggestions on what “type” of expertise should be on the evaluation team (e.g., quality improvement or data management specialists) to assist with tasks such as extracting data or who have specific expertise in the EHR technology. It is noteworthy that some of these highly specialized roles may not be available to smaller organizations, as we observed with Cases B, C, and to some extent, Case D. Strengths of this study include multiple methods of data collection to understand a full picture of each Case’s experience in collecting data derived from an EHR. The collaborative nature of the partnership between each Case and the NTAE allowed for bidirectional learning in a trusted and supportive space. A key limitation to this project is related to generalizability to other PPR grantees in that the selected Cases were highly motivated to participate in this evaluation and may not represent the level of motivation for other PPR grantees (e.g., selection bias). Another limitation is that none of these Cases opted to collect data on emergency department utilization, hospital admissions, or readmissions. Much “food is medicine” literature focuses on these types of programs as potentially effective in decreasing “big ticket” healthcare costs as a means to bolster support from payers.^50,51 The paucity in this type of data collected by the Cases in this study is largely because as FQHCs, they do not have access to the hospital-based EHR where this data would be tracked.

EHR data holds much promise but presents hurdles within FIM. Drawing upon facilitators and overcoming challenges outlined in this case study to EHR data extraction within FIM interventions will help to strengthen the evidence base and build capacity for healthcare delivery models to demonstrate public health impact.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The GusNIP NTAE Center is funded through a cooperative agreement and is supported by Gus Schumacher Nutrition Incentive Program grant no. 2019-70030-30415/project accession no. 1020863, grant no. 2023-70435-38766/project accession no. 1029638, and grant no. 2023-70414-40461/project accession no. 1031111 from the USDA National Institute of Food and Agriculture.

ORCID iDs

Sarah A Stotz

Hilary Seligman

Victoria A Zigmont

Juan Aguilera

Carmen Byker Shanks

References

Boeing

Bechthold

Bub

, et al. Critical review. Vegetables and fruit in the prevention of chronic diseases. Eur J Nutr 2012; 51(6): 637–663.

Fan

Zhang

, et al. Fruit and vegetable intake and risk of type 2 diabetes mellitus: meta-analysis of prospective cohort studies. BMJ Open 2014; 4(11): e005497.

Stanaway

Afshin

Ashbaugh

, et al. Health effects associated with vegetable consumption: a Burden of Proof study. Nat Med 2022; 28(10): 2066–2074.

Lee

Moore

Park

, et al. Adults meeting fruit and vegetable intake recommendations—United States, 2019. MMWR Morb Mortal Wkly Rep 2022; 71: 1–9. http://dx.doi.org/10.15585/mmwr.mm7101a1

Downer

Clippinger

Kummer

Food is medicine research action plan [Internet]., 2022. https://www.aspeninstitute.org/programs/food-and-society-program/food-is-medicine-project/

Mozaffarian

Blanck

Garfield

, et al. A food is medicine approach to achieve nutrition security and improve health. Nat Med 2022; 28(11): 2238–2240.

Wholesome Wave and DAISA Enterprises. Produce prescription programs US field scan report : 2010–2020, https://www.daisaenterprises.com/uploads/4/4/0/5/44054359/produce_prescription_programs_us_field_scan_report__june_2021_final.pdf (2021, accessed 17 January 2022).

GusNIP National Technical Assistance and Evaluation Center. GusNIP NTAE. Gus Schumacher Nutrition Incentive Program (GusNIP): impact findings Y3: September 1, 2021 to August 31, 2022., 2023.

Xie

Price

Curran

, et al. The impact of a produce prescription programme on healthy food purchasing and diabetes-related health outcomes. Public Health Nutr 2021; 24(12): 3945–3955.

10.

Bhat

Coyle

Trieu

, et al. Healthy food prescription programs and their impact on dietary behavior and cardiometabolic risk factors: a systematic review and meta-analysis. Adv Nutr 2021; (21): 1–13.

11.

Oronce

CIA

Miake-Lye

Begashaw

, et al. Interventions to address food insecurity among adults in Canada and the US. JAMA Health Forum 2021; 2(8): e212001.

12.

Ridberg

Bell

Merritt

, et al. Effect of a fruit and vegetable prescription program on children’s fruit and vegetable consumption. Prev Chronic Dis 2019; 16(6): 1–13.

13.

Bryce

Wolfson Bryce

Cohen Bryce

, et al. A pilot randomized controlled trial of a fruit and vegetable prescription program at a federally qualified health center in low income uncontrolled diabetics. Prev Med Rep 2021; 23: 101410.

14.

Veldheer

Scartozzi

Bordner

, et al. Impact of a prescription produce program on diabetes and cardiovascular risk outcomes. J Nutr Educ Behav 2021; 24: 18.

15.

Cook

Ward

Newman

, et al. Food security and clinical outcomes of the 2017 Georgia fruit and vegetable prescription program. J Nutr Educ Behav 2021; 53(9): 770–778.

16.

Cavanagh

Jurkowski

Bozlak

, et al. Veggie Rx: an outcome evaluation of a healthy food incentive programme. Public Health Nutr 2017; 20(14): 2636–2641.

17.

Lee

Mozaffarian

, et al. Cost-effectiveness of financial incentives for improving diet and health through medicare and medicaid: a micro simulation study. PLoS Med 2019; 16(3): 1–20.

18.

Schlosser

Smith

Joshi

, et al. “You guys really care about me. . .”: a qualitative exploration of a produce prescription program in safety net clinics. J Gen Intern Med 2019; 34(11): 2567–2574.

19.

Stotz

Lee

Thompson

JJ.

“It was an unexpected bond”: how an emerging participant-driven online social network may be enhancing an eLearning nutrition education & supplemental produce intervention. Digit Health 2021; 7: 1–9.

20.

United States Department of Agriculture. Food and Nutrition Security. https://www.nifa.usda.gov/topics/food-nutrition-security (accessed 24 March 2024).

21.

Biden-Harris Administration National Strategy on Hunger, Nutrition, and Health, 2022. https://odphp.health.gov/sites/default/files/2022-06/White%20House%20Tookit%20(2).pdf

22.

United States Department of Agriculture. Gus Schumacher Nutrition Incentive Program (GusNIP). National Institute of Food and Agriculture, https://www.nifa.usda.gov/grants/programs/hunger-food-security-programs/gus-schumacher-nutrition-incentive-program (2022, accessed 25 July 2023).

23.

Jones

VanWassenhove-Paetzold

Thomas

, et al. Impact of a fruit and vegetable prescription program on health outcomes and behaviors in young Navajo children. Curr Dev Nutr 2020; 4(8): 1–7.

24.

Stotz

Mitchell

Szczepaniak

, et al. A qualitative exploration of approaches applied by nutrition educators within nutrition incentive programs. J Nutr Educ Behav 2023; 55(3): 224–234.

25.

Agriculture Improvement Act of 2018, Pub. L. No. 115-334., 2019. https://www.federalregister.gov/documents/2020/08/21/2020-17356/implementation-of-the-agriculture-improvement-act-of-2018

26.

Budd Nugent

Byker Shanks

Seligman

, et al. Accelerating evaluation of financial incentives for fruits and vegetables: a case for shared measures. Int J Environ Res Public Health 2021; 18(22): 12140.

27.

USDA GusNIP National Techincal Assistance and Evaluation Center. Participant-level core metrics overview: produce prescription projects, 2022. https://www.nutritionincentivehub.org/resources/resources/reporting-evaluation/core-metrics-produce-prescription/participant-level-metrics

28.

Nguyen

Duan

Lee

, et al. Association of a medicare advantage posthospitalization home meal delivery benefit with rehospitalization and death. JAMA Health Forum 2023; 4(6): e231678.

29.

Ranjit

Aiyer

Toups

, et al. Clinical outcomes of a large-scale, partnership-based regional food prescription program: results of a quasi-experimental study. BMC Res Notes 2023; 16(1): 13.

30.

Long

Yaroch

Byker Shanks

, et al. Leveraging electronic health record data within food is medicine program evaluation: considerations and caveats. Adv Nutr 2024; 15(4): 100192.

31.

Baxter

Jack

Qualitative case study methodology: study design and implementation for novice researchers. Qual Rep 2008; 13(4): 544–559.

32.

Yin

RK.

Case study research: design and methods. 3rd ed. London: Sage Publications, 2003.

33.

Tracy

SJ.

Qualitative quality: eight “big-tent” criteria for excellent qualitative research. Qual Inq 2010; 16(10): 837–851.

34.

Buller

Woodall

Zimmerman

, et al. Formative research activities to provide web-based nutrition education to adults in the upper Rio Grande Valley. Fam Community Health 2001; 24(3): 1–12.

35.

Park

Nitzke

Kritsch

, et al. Internet-based interventions have potential to affect short-term mediators and indicators of dietary behavior of young adults. J Nutr Educ Behav 2008; 40(5): 288–297.

36.

Whittemore

Chao

Jang

, et al. Implementation of a school-based Internet obesity prevention program for adolescents. J Nutr Educ Behav 2013; 45(6): 586–594.

37.

Kattelmann

White

Greene

, et al. Development of young adults eating and active for health (YEAH) Internet-based intervention via a community-based participatory research model. J Nutr Educ Behav 2014; 46(2): S10–S25.

38.

Spitzer

Kroenke

Williams

JBW

, et al. A brief measure for assessing generalized anxiety disorder: the GAD-7. Arch Intern Med 2006; 166(10): 1092–1097.

39.

Kroenke

Spitzer

Williams

JB.

The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med 2001; 16(9): 606–613.

40.

Roulston

Considering quality in qualitative interviewing. Qual Res 2010; 10(2): 199–228.

41.

Paulus

Lester

Deptster

Digital tools for qualitative research. 1st ed. London: Sage Publications, 2014.

42.

Stemler

An overview of content analysis. Res Eval 2001; 7(17): 32.

43.

Hsieh

Shannon

SE.

Three approaches to qualitative content analysis. Qual Health Res 2005; 15(9): 1277–1288.

44.

Saldaña

The coding manual for qualitative researchers. 2nd ed. London: Sage Publications, 2012.

45.

Simons

Case study research in practice. New York: Sage Publications, 2009.

46.

Holmes

Beinlich

Boland

, et al. Why is the electronic health record so challenging for research and clinical care? Methods Inf Med 2021; 60(1–2): 32–48.

47.

Sekhon

Cartwright

Francis

JJ.

Acceptability of healthcare interventions: an overview of reviews and development of a theoretical framework. BMC Health Serv Res 2017; 17(1): 1–13.

48.

Van Panhuis

Paul

Emerson

, et al. A systematic review of barriers to data sharing in public health. BMC Public Health 2014; 14(1): 1–9.

49.

Walker

Hefner

DePuccio

, et al. Approaches for overcoming barriers to cross-sector data sharing. Am J Manag Care 2022; 28(1): 11–16.

50.

Food as medicine: translating the evidence. Nat Med 2023; 29(4): 753–754.

51.

Estrella

Scheidell

Khan

, et al. Cross-sectional analysis of food insecurity and frequent emergency department use. West J Emerg Med 2021; 22(4): 911.

The realities of data derived from electronic health records to evaluate health outcomes,utilization,and cost of produce prescription programs: A multiple case study evaluation