Abstract
The author has served as the program evaluator for the Southwest Center for Agricultural Health, Injury Prevention, and Education (SW Ag Center) since 2013, providing an opportunity to study a mature program. A longitudinal study was conducted to evaluate the usefulness of health outcome measures in determining program success. The historical progress of the SW Ag Center was traced using agriculture, forestry, and fishing (AgFF) health outcome data from the U.S. Bureau of Labor Statistics Survey of Occupational Injuries and Illnesses (BLS SOII). Observable signs of maturity, such as collaboration and capacity building with partners, were examined using an original process measure called “in-reach.” BLS SOII trends fluctuated over time, even though AgFF injury and illness rates declined regionally from 1995 to 2023. The evaluation revealed limitations in the BLS SOII health outcome measures, while in-reach indicated sustainability. Characteristics of health outcome measures better suited for AgFF operations are described.
Keywords
The goal of this article is to present a case study of a well-established program, highlighting the importance of health outcome measures in assessing its success. Notably, the federal program under evaluation was abruptly terminated after 30 years of continuous funding in March 2025. With this key context in mind, the article is divided into several sections. The first section describes the evaluation setting, including the funding agency, the National Institute for Occupational Safety and Health (NIOSH), and the program itself, the Southwest Center for Agricultural Health, Injury Prevention, and Education (SW Ag Center). The evaluation summary of the SW Ag Center during the author's tenure emphasizes major developments and insights gained over the years. The second section provides a detailed assessment, showcasing both traditional and innovative outcome measures, and discusses their findings in depth to explain their importance. The paper concludes with a comprehensive discussion of the strengths and weaknesses of current health outcome measures, along with clear recommendations to restore ongoing funding.
Context of the Evaluation
The National Institute for Occupational Safety and Health
NIOSH is an extramural program of the Centers for Disease Control and Prevention (CDC). Until March 2025, NIOSH received annual funding in 5-year cycles to support essential research and educational efforts aimed at protecting the health and safety of workers in agriculture, forestry, and fishing (AgFF). NIOSH allocated resources to 11 Agricultural Safety and Health Research Centers (Ag Centers) to advance its mission. However, in March 2025, funding was suddenly cut, staff were laid off, and federal support for health programs—including NIOSH and all Ag Centers—was halted as part of a government efficiency effort. Fortunately, funding was restored in late June 2025, and support for FY26 has been reestablished.
The Southwest Center for Agricultural Health, Injury Prevention, and Education
Established in late 1995 at the University of Texas Health Northeast, the SW Ag Center serves Arkansas, Louisiana, New Mexico, Oklahoma, and Texas by focusing on research and outreach initiatives to reduce injuries and fatalities among AgFF workers and their families. Aligning with NIOSH's priorities, the Center's mission is twofold: (1) to document significant decreases in illness and injuries and (2) to collaborate with industry partners to promote impactful research and build capacity through outreach and research-to-practice activities. The center works to foster strategic partnerships that enhance the health and safety of diverse AgFF populations across its five-state region. Since its founding, the SW Ag Center has maintained a full-time professional administrative staff at the University of Texas, Tyler Health Sciences Center. Each Ag Center was required to support three main components: (1) Administration, Planning, and Evaluation, (2) Outreach, and (3) Research projects. Outreach efforts are primarily carried out by disseminating evidence-based educational materials at regional and local AgFF venues, under the leadership of a dedicated outreach coordinator and a team of committed staff. The SW Ag Center typically supports 7–10 university-based research projects that cover basic science, translation, intervention, and surveillance, with most studies being observational in community settings. As of December 2025, outreach activities had only partially resumed due to the loss of the coordinator and much of the staff during the funding freeze. Ongoing uncertainties about funding beyond FY26 further complicated hiring efforts. Although research projects, except for a laboratory-based study, began resuming data collection, the SW Ag Center operated without a program coordinator, and insufficient staffing severely hindered implementation activities.
SW Ag Center Program Evaluation Methods
NIOSH required evaluation as a key administrative component. The author has served as the program evaluator for the SW Ag Center since 2013, covering roughly half of the Center's history and the last three of its six funding cycles. NIOSH did not provide detailed guidance on how to conduct evaluations nor mandate the use of logic models for program planning and assessment. However, with the evaluators’ expertise, the SW Ag Center trained leadership and researchers in program planning using logic models and required them to develop these models for funding applications. This process clarified the connections between NIOSH objectives, Center goals, and the intended outcome measures. Importantly, all research projects are aligned with the Center's outcomes (listed in Evaluation Methods below). As a result, the logic model framework provided a cohesive, hierarchical approach to program planning, implementation, and evaluation (Newbill et al., 2017).
Evaluation practices have evolved over the years in response to the Center's development and changes within NIOSH. Importantly, program evaluation has been ongoing and benefited from the stability of the evaluation team. Work conducted during the 2011–2016 cycle began with a social network analysis and a stakeholder perception survey (Helitzer & Newbill, 2016). In June 2017, NIOSH hosted a workshop on Contribution Analysis (CA), building on Mayne's work (Avalos, 2017; Downes et al., 2019), and the evaluation immediately adopted this approach for the 2016–2022 cycle, incorporating the Capability, Opportunity, Motivation (CA COM-B) model for behavior (B) change (see Mayne, 2011, 2017, 2018, 2019). The sixth year of the cycle responded to the COVID-19 pandemic, which began in January 2020 (Year 4 of the cycle). NIOSH recognized that the Ag Center researchers could not complete their projects due to nearly a year of lost work, but secured an additional year of funding. The evaluation for that cycle resulted in a detailed CA that included commentary on pandemic-related disruptions and marked NIOSH's first foray into evaluation practice (Newbill, 2023). Knowing that the current cycle would include the SW Ag Center's 30th anniversary, the evaluation aimed to explore the Center's historical trajectory by analyzing trends in AgFF surveillance health outcomes data for the region. This would help identify signs of a mature program. Evaluating health outcomes is challenging because most evaluations last 3–5 years, and research often involves small (<30) groups of workers. However, since Ag Centers are federal programs established and funded since 1995, routine and standardized documentation of work-related injuries and illnesses in AgFF is required. Consequently, it was reasonable to assume that relevant health outcomes data should be available and publicly accessible. Guided by this expectation and the historical perspective adopted, and using the twofold mission of the SW Ag Center as a guiding principle, the evaluation questions were:
What are the trends (1995–present) in the rate of AgFF non-fatal occupational injuries and illnesses in the SW Ag Center region compared to U.S. AgFF rates? Additionally, how do these trends differ among the five states in the region? What observable evidence exists for collaboration and capacity building with industry partners that indicate the long-term investment (i.e., maturity) of the SW Ag Center in the region?
Data Sources
Health outcomes are difficult to assess because most evaluations span 3–5 years and involve small (<30) groups of workers. However, Ag Centers are federal programs established and funded since 1995 that require routine, standardized documentation and reporting of work-related injuries and illnesses in the AgFF. Therefore, it is reasonable to assume that relevant health data are available and publicly accessible. To analyze trends in health outcomes, quantitative data were obtained from the U.S. Bureau of Labor Statistics (BLS) Survey of Occupational Injuries and Illnesses (SOII) Annual Survey Summary Numbers & Rates for profile years 1990–2023 (Bureau of Labor Statistics, 2026). The BLS SOII offers the only comprehensive, nationally consistent data on worker injuries and illnesses and is used exclusively by NIOSH. In 1970, Congress assigned the BLS the task of developing a comprehensive statistical system for work-related injuries, illnesses, and fatalities in private industry. The Occupational Safety and Health (OSH) Act established a division of responsibilities at the national level: data collection (BLS), regulatory enforcement (Occupational Safety and Health Administration [OSHA]), and safety and health research (CDC). In 1992, the BLS restructured the SOII to include data on the characteristics of injured or ill workers and the circumstances of specific nonfatal injury or illness cases involving lost workdays. According to the SOII Handbook of Methods (Bureau of Labor Statistics, 2023b), occupational injury is any injury, such as a cut, fracture, sprain, amputation, etc., resulting from a work-related event or a single instantaneous exposure in the work environment; occupational illness is any abnormal condition or disorder caused by exposure to employment-related factors, excluding injuries from instantaneous events or exposures. It includes both acute and chronic illnesses or diseases that may result from inhalation, absorption, ingestion, or direct contact.
The BLS SOII is a partnership between federal and state agencies, where employers’ reports are gathered and processed by state agencies in collaboration with the BLS (Bureau of Labor Statistics, 2023a). State agencies train employers on how to record and report their injury and illness incidents. AgFF operations are randomly chosen to participate in SOII, and employers are notified in writing a year in advance. Estimates from the SOII are based on a scientifically selected probability sample rather than a complete census of all employers. The primary source of the sample is the Quarterly Census of Employment and Wages (QCEW), which is a quarterly count of employment and wages reported by employers. State unemployment insurance (UI) programs are the main source of the QCEW data. The QCEW report includes employer name and address information for over 95% of jobs across the United States, including data at the county, Metropolitan Statistical Area (MSA), state, and national levels, sorted by industry.
Robust quality control procedures ensured the accuracy of the data collected. Specifically, QCEW conducts two surveys each year for about one-third of all operations with more than three employees. The Multiple Worksite Report (MWR) is sent to operations with multiple locations, while single-location operations receive the Annual Refiling Survey (ARS). A response rate of 70% is required for the ARS and 80% for the MWR. Other notable features of BLS data specific to the AgFF sector include (Bureau of Labor Statistics, 2023b):
Series breaks occurred in 2003, 2011, 2014, and 2023. These breaks reflect BLS's efforts to accurately track changes in industry and occupational categories, as well as emerging diseases and disorders, along with related updates in medical classifications and terminology (Drudi, 2015; Wiatrowski, 2014). The revisions have increased the specificity and detail of occupational injury and illness data over time. The datasets remain comparable (Bureau of Labor Statistics, 2020; Drudi, 2015). They remain the only reliable source of national data on workplace injuries and illnesses. NIOSH researchers use them to report trends and current conditions in occupational safety and health. SOII estimates the number and frequency (incidence rates) of workplace injuries and illnesses based on employer-recorded logs maintained throughout the year. These records reflect the employer's understanding of which cases are work-related under recordkeeping guidelines. AgFF data only included farms/operations with 11 or more employees. AgFF data included hunting after 2003.
To credibly demonstrate evidence of collaboration and capacity building attributable to the SW Ag Center's long-term regional involvement, a process measure was established. The Center regularly updates partner lists and an outreach activity log, including details on the partners’ roles, whether the engagement is new or ongoing, the type of outreach activity, when and where it occurred, the Center's role, and the platform (e.g., live or virtual). At the author's request, the partner status—“new” or “on-going”—was added to the activity log at the start of the new funding cycle (2022). Additionally, an original program implementation assessment was developed and included in the evaluation to formalize the process data. “In-reach” is an original term defined as when producers and/or consumers of SW Ag educational products, materials, and/or expertise voluntarily “reach in” to the Center for help.
The author established inclusion and exclusion criteria for in-reach. Communication was recorded as in-reach if it was unsolicited and led to a collaborative activity. The latter aims to capture “mature” interactions, arguing that simply asking for support is not remarkable, whereas the Center's capacity and expertise to act on a request to produce something relevant are significant. For example, the Center Director or principal investigator (PI) receives a request such as: “I have an idea for a study and would like to meet sometime to talk about the design and feasibility…” or “I was hoping to meet with you to discuss this initiative and perhaps partner on outreach.” These requests were not recorded as in-reach until some follow-up action was taken in collaboration with the SW Ag Center. The final determination was verified by the author as described below.
Data Collection Methods
BLS Survey of AgFF Injuries and Illnesses
The BLS SOII provides the only comprehensive, nationally consistent data on worker injuries and illnesses. It is the only source of data on AgFF occupational safety and health funded wholly or partly by NIOSH. A request for access to incidence rates of nonfatal occupational injuries and illnesses in AgFF for five states from 1990 to 2025, or whatever was available, was submitted via the online portal in February 2025. Within 2 days, I received an email from a BLS economist with detailed instructions on which datasets to access and how to access them. Additionally, important links to information on the SOII Handbook of Methods and SOII Concepts were provided, explaining the series breaks (Bureau of Labor Statistics, 2023b). The series breaks divided the 1990–2025 dataset into four segments: 1990–2001; 2002; 2003–2013; and 2014–2025. Moreover, each state (and the U.S. overall) had to be accessed separately, so data were compiled into a single Excel spreadsheet from 24 individual datasets. The final and most time-consuming step involved cleaning and verifying the data, specifically interpreting dashes as no data and 0.0 cells with footnotes as indicating either <15 cases or <50 cases. This variation required confirming across states and years that 0.0 consistently meant <15 cases for the data relevant to this study. Data checks were repeated during chart construction, especially when variations appeared over time and across states. The BLS SOII data was accessed in February and March 2025, before the government shutdown interrupted data releases. The datasets used in this study are available from the author.
In-Reach: Collaboration and Capacity Building
The author advised outreach staff on the inclusion criteria for in-reach, and with their cooperation and understanding, in-reach was officially added to the outreach activity log in 2023 and retrospectively updated for 2022. As mentioned earlier, partnership status was also included in the log. Both additions provided measures of collaboration and capacity-building, which are likely indicators of a mature program. The author verified log entries quarterly by triangulating data from PI routine reports and conducting random checks with an independent consultant. Additionally, an observer–participant role was assumed during key opportunities to gather information, such as leadership meetings, biannual advisory board meetings, and routine research PI update sessions. Anecdotal notes were taken to document responses to questions about details and specifics of experiences and funds received from agencies other than NIOSH.
Findings
A decision matrix was created to assess the construct “usefulness” (Table 1). Criteria were chosen based on established psychometric properties and practical features of measurement tools. A rating scale of low, moderate, and high was used to assess the BLS SOII and the in-reach collaboration/capacity-building data. Importantly, decisions for each criterion took into account AgFF mandates and the concerns of the SW Ag Center. The reasoning behind the decision matrix is described below.
Decision Matrix to Determine the Usefulness of Agriculture, Forestry, and Fishing Health Outcome Measures to Agricultural Health, Injury Prevention, and Education Centers.
In-Reach and Collaboration
Due to funding, 2 years of in-reach data were gathered. In 2023, in-reach accounted for 25% (eight of 32) of SW Ag Center outreach activities, which increased to 54% (20 of 37) in 2024. Table 2 presents in-reach activities over the 2 years by partner status (new or ongoing), event name, SW Ag role, location, and estimated attendance. Most requests focused on exhibits or presentations about AgFF issues, mainly from Texas-based agencies. However, several international, national, and regional organizations sought insights into topics the Center has long been concerned with (e.g., logging, forestry, cattle, and commercial fishing). Other requests addressed emerging national and regional issues, especially women in AgFF and mental health. Additionally, most in-reach partners were new (20 of 27, 74%), highlighting a need for capacity-building through collaboration.
Southwest Center for Agricultural Health, Injury Prevention, and Education In-Reach Activity Log 2022–2024.
In addition to requests for support from the SW Ag Center, Research PIs were contacted for help in their areas of AgFF expertise (e.g., commercial shrimping, logging, and dairy cattle workers). A detailed example and several highlights are provided below.
The CDC officially reported the presence of H5N1 avian influenza (bird flu) in dairy cattle in late March 2024, although cases first appeared in January. By May, nine states and 36 herds were affected, with one human case confirmed in Texas. The SW Ag Center sponsored a research principal investigator, David Douphrate, PhD. His expertise and contacts within the dairy industry expanded through a SW Ag Center pilot project (R21 grant) funded during the 2011–2016 cycle, which investigated ergonomics among dairy workers (Douphrate et al., 2012). This initiative led to a research project grant (R01) with the Center during the 2016–2022 cycle. The White House Office of Pandemic Preparedness & Response (OPPR), seeking advice from researchers experienced in AgFF research, specifically on dairy farms, contacted Dr. Douphrate. This connection resulted in a $3.8 million CDC grant for 1 year to evaluate the prevalence of avian influenza among dairy farm workers in Texas, Oklahoma, New Mexico, and California. The multidisciplinary research team is led by Dr. Douphrate, who has extensive experience in primary data collection on dairy farms and maintains strong relationships with milk producers across Western states.
Other notable examples of recent, ongoing collaboration and supplementary funding awarded to the SW Ag Center and its Research PIs include:
Texas Department of Agriculture partners with AgriSafe Network to promote mental health, suicide prevention, and stress relief related to extreme weather events. The Texas A&M Transportation Institute and the Texas A&M School of Public Health are running an awareness campaign to create a safety program for log truck drivers, with training taking place in Arkansas, Texas, and Louisiana. A free mobile health clinic, called Docside Clinic™, in Galveston, Texas, teamed up with a locally owned dock, a Federally Qualified Health Center (FQHC), a local food bank, the U.S. Coast Guard, and the University of Texas Health's Center for Violence Prevention and Occupational Health Residency Program. The clinic offered vaccinations, medicine, food, first-aid kits, wilderness first-aid guides, and hygiene items to fishermen in Port Arthur, along with mental health and substance use services (Guillot-Wright et al., 2022). The T.L.L. Temple Foundation, which supports rural communities in East Texas, sponsored an AgFF experiential site visit for medical students to Reed Farm. Public Health, Health, and Safety students (University of Texas, Tyler) had the chance to visit a working farm to learn about the specific needs of an aging workforce and to explore strategies for preventing illness and injury.
BLS Survey of AgFF Injuries and Illnesses Data
SOII data specific to AgFF were available for the years 1990–2023, with Ag Center data starting in 1995. The 5 years of data prior to the establishment of the Ag Centers serve as a baseline, enabling comparison of occupational injury and illness trends before and after. Only data from the five states in the SW Ag Center region, rather than the entire U.S. AgFF, were extracted from the BLS SOII Annual Survey Summary Numbers & Rates for profile years 1990–2023 (Bureau of Labor Statistics, 2026). The U.S. trendline is included for comparison in both figures.
AgFF Injuries. The trend in AgFF injury data varied significantly by state. To better visualize this trend, the five states are combined into a Region 5 composite and compared to the U.S. (Figure 1). Overall, the trend shows (1) a general decline in AgFF injuries since 1995, when data collection began for the region, and (2) the SW Ag Center region consistently remained below the U.S. AgFF rates, except for one year. In 2013, a notable increase caused the injury rate to surpass the U.S. average, mainly due to a sharp rise in Louisiana. To examine this spike, another data set was reviewed that tracks case and demographic rates for days-away-from-work cases (Bureau of Labor Statistics, 2026). Data was available only from 2011 to 2020. The rates of total injury and illness include cases with lost workdays, but event details do not specify how these data are further categorized. The event data identified 2013 as an unusually high injury year in Louisiana, characterized by contact with an object or equipment and being struck by an object; injuries included sprains and tears affecting the trunk, back, and upper extremities. The cause of injury was not reported. AgFF includes two occupations with higher injury risks: logging and commercial fishing/shrimping.
AgFF injuries for Southwest Center for Agricultural Health, Injury Prevention, and Education Region 5a compared to the United States.
AgFF Illnesses. Compared to injuries, there was much more variation among states, along with many missing data points and some states with fewer than 15 cases. The composite Region 5 trend provides the clearest picture. It shows illnesses fluctuating across the United States, except for a sharp rise from 2020 to 2022 (Figure 2). Arkansas and Oklahoma experienced the largest spikes. A smaller increase occurred nationwide at the same time. Illness categories in the BLS database do not include pathogens, but the “all other” category for sources of illness increased in 2020 for the United States and all Region 5 states, except NM, which saw a decrease. The alignment with U.S. data, the timing, and the sharpness of the surge suggest COVID-19 as the cause of the significant rise in illnesses, rather than agriculture-related events.
AgFF illnesses for Southwest Center for Agricultural Health, Injury Prevention, and Education Region 5a compared to the United States.
Examination of Targeted Health Outcomes
Recall that program-targeted outcomes are outlined in the SW Ag Center logic model as (1) reducing ag-related injuries and illnesses in the region, and (2) enhancing capacity through collaboration, outreach, and research-to-practice activities. The most significant evidence of impact comes from the process data for in-reach and collaboration described earlier. Additional evidence, summarized after a short explanation of the CA conducted during the previous funding cycle (Newbill, 2023), is detailed below. CA is a theory-based evaluation in which the evaluator develops a plausible theory of change (ToC) that explains how the program advances from input to impact, then collects and assesses evidence supporting the theory. If the evidence supports the ToC, it is reasonable to conclude the program contributed to the impact, and contribution claims are justified (Mayne, 2011, 2015, 2018, 2019). As noted, CA produces substantial data, so the evidence table is summarized in Table 3, with the full table available in Newbill (2022). Key findings from the CA include:
Stakeholder perception survey trend analysis revealed increased awareness of worker and workplace safety practices and a greater use of resources to enhance them. SW Ag community-based projects served as catalysts for improving access to health services and safety practices. Research PIs were encouraged to develop research-to-outreach products that facilitate practical use. With strong support from the SW Ag Center Content Strategist, most accomplished this.
Southwest Center for Agricultural Health, Injury Prevention, and Education Evidence Table for Contribution Analysis (Abbreviated).
The intervention statement was: The SW Ag Center intervention aims to improve the safety and health of AgFF workers through Project update (1) sponsoring original multi-disciplinary research, intervention, surveillance, and translation projects to identify and reduce risks relevant to AgFF and (2) raising awareness of safety and health practices among AgFF workers, business owners, and communities through an evidence-based outreach program. SW Ag Center supporting assumptions were: (1) (sufficient) funding for research and outreach initiatives; (2) willingness of AgFF stakeholders to (a) engage/participate and (b) implement/adopt recommended, evidence-based practices; (3) willingness of community-based champions to support Center research, outreach, and awareness campaigns; (4) capacity of SW Ag Center to contribute to the development of educational resources that are accessible and adoptable; (5) capacity of SW Ag Center to engage AgFF researchers in Center initiatives and in mentorship relationships; (6) ability of SW Ag Center to identify and scientifically consider emerging issues in AgFF; (7) ability of SW Ag Center to expand its sphere of influence through strategic partnerships.
The BLS SOII trends are less convincing, and there was significant variability across states in the region, with large year-to-year fluctuations. However, the BLS SOII trends do support these observations:
AgFF injury and illness rates declined in the SW Ag Center region from 1995 to 2023. Injury and illness rates in the SW Ag Center region stayed below the U.S. AgFF linear trend line for most of the period between 1995 and 2023.
Even partially attributing these observations to SW Ag Center activities is challenging. Analyzing Texas data might clarify this because the Center started tracking outreach efforts from 2016 to 2022. The overall evaluation for that period found that 49% (65 of 132) of outreach events occurred in Texas, compared to 5% (seven of 132) in the other five states (Newbill, 2022). The remaining 46% were national, international, or virtual events, with virtual events increasing in 2020 due to COVID-19 travel restrictions. Since the Center has been based in Tyler, Texas, United States, throughout its existence, it is reasonable to assume most outreach has happened in Texas since 1995. It makes sense to consider Texas as the best example of the Center's success because it managed the largest volume of activities. Hypothetically, if the Texas trend is lower than that of neighboring states, one possible explanation could be the impact of the Ag Center activities.
Previous figures show that the most reliable SOII data is the injury rate. Figure 3 shows the injury incidence for the region, with Texas as the linear trend. Texas injury data is not consistently lower than that of other states in the region. Instead, the Texas trend line crosses the rates for the other four states until 2014, when Texas and Oklahoma increased, Louisiana and Arkansas declined, and New Mexico stayed around Texas until 2020, when it dropped below the linear trend. (A similar pattern occurred in the illness data, but it has a lot of missing data and is not shown.) The observed data do not provide enough evidence to support the hypothesis that the SW Ag Center intervention effectively reduces AgFF injury rates.
AgFF injuries for Southwest Center for Agricultural Health, Injury Prevention, and Education Region 5a compared to Texas trendline
Before concluding that the SW Ag Center intervention has had a negligible effect (or indeterminate at best) on AgFF injury and illness rates, other factors must be considered. One clear factor is that comparing states is not advisable since AgFF occupations differ, and therefore, risks will also vary. Less obvious but more concerning are the limitations of BLS SOII data and its suitability as a measure of the SW Ag Center's impact on improving health outcomes.
BLS Data Limitations
Characteristics of the BLS SOII outcome data include well-known and ongoing limitations (Boden et al., 2010; Bureau of Labor Statistics, 2020; Gunter, 2016). These limitations relevant to AgFF illness and injury data are summarized here.
SOII estimates the number and frequency (incidence rates) of workplace injuries and illnesses based on employer-recordkeeping logs maintained during the year. These records reflect the employer's understanding of which cases are work-related according to recordkeeping guidelines. Unlike the QCEW survey, there are no quality-control or validation processes. Non-sampling errors are well-known, particularly an undercount of injuries and illnesses (Bureau of Labor Statistics, 2023b). Since 2009, the potential causes and extent of this undercount have been examined, with the latest evidence suggesting reporting, methodological, and timing issues (Boden et al., 2010).
Reporting issues involved various disincentives for both employers and workers to report workplace injuries. Employees might avoid reporting injuries because they fear retaliation from their employers or pressure from colleagues to remain silent when safety incentives are in place. Employers may doubt whether an injury is related to work, making them less likely to record the case promptly in the OSHA log or hesitant to acknowledge injuries due to concerns about increased workers’ compensation costs. Another reason for undercounting is the range of methodological and data-quality issues. These include differences in capture rates between single- and multi-unit establishments, variations in workers’ compensation and SOII rates, and misinterpretation of reporting requirements. Additionally, undercounts can result from timing issues. Remember that the sample unit for the BLS SOII is the QCEW, which is derived from state UI and workers’ compensation systems. The SOII has also been less effective at capturing cases where the injury date differs from the first day of lost work required for workers’ compensation.
In 2013, the Council of State and Territorial Epidemiologists (CSTE), in partnership with the BLS, NIOSH, and OSHA, formed a working group to review new developments in the surveillance of work-related injuries and illnesses and to update recommendations to address gaps in data (Council of State and Territorial Epidemiologists, 2013). Numerous recommendations were made under two main goals:
Use health data to fill gaps in employer reporting, aiming primarily to establish a multi-source surveillance system for work-related traumatic injuries that complements employer-based data and to expand state-based surveillance efforts to increase the variety of conditions, worker populations, and hazards monitored. Enhance the collection and use of employer-reported occupational injury and illness data by OSHA and BLS, aiming to expand and modernize data gathering for both agencies and improve communication and outreach to employers regarding recordkeeping.
Gunter (2016) provides the latest update on responses to these recommendations. BLS used the new funding to support research aimed at (1) conducting a multisource matching effort using SOII, workers’ compensation, and other records to identify a broader range of workplace injuries, (2) performing additional matching of SOII and workers’ compensation data, and (3) interviewing employers about their injury and illness recordkeeping practices. Wiatrowski (2014) suggested that some CSTE recommendations were impractical due to a lack of authority and/or resources, while others did not fall under the BLS mandate. The most recent study on the undercount is a fellowship report by Wuellner and Phipps (2017), which analyzed BLS SOII data from four states. Data quality issues still exist, with knowledge of and compliance with OSHA injury reporting requirements, recordkeeping practices, and state-based survey procedures for the BLS survey affecting the accuracy of BLS estimates of nonfatal occupational injuries and illnesses.
In summary, efforts since 2009 to improve AgFF health data quality have produced few, if any, real improvements in BLS data. Since the most recent publicly available study is nearly 10 years old, it can be concluded that key issues with undercounting remain unresolved. More importantly, suggested changes, if implemented, would still leave out many AgFF operations, such as farms and businesses with fewer than 11 employees and undocumented migrant workers who are not eligible for unemployment benefits under current laws. The 1970 OSH Act includes a recurring appropriations rider from OSHA that excludes agriculture operations employing 10 or fewer workers (Pollack & Gellerman Keimig, 1987, p. 19). Many AgFF workplaces relevant to the SW Ag Center region operate with fewer than 10 employees (e.g., shrimp fishing and logging) or hire undocumented workers (e.g., migrant farm workers). Additionally, the accuracy of the BLS SOII estimates tends to improve with better knowledge and adherence to OSHA/BLS recordkeeping requirements. It is reasonable to assume that employers of small farms and operations are unlikely to spend time learning OSHA recordkeeping.
Conclusions and Recommendations
The data on the implementation process of in-reach and its related collaboration, combined with evidence supporting contributions to program outcomes (Newbill, 2023), strongly backs the claim that the SW Ag Center's efforts have improved AgFF safety and health in the region. The new measure of in-reach is a strength of the study. It enhances not only the practical relevance of assessing a mature program but also overall evaluation practices.
Another strength of the study is its long-term scope. Analyzing 30 years of AgFF injuries and illnesses using the outcome data NIOSH relies on has never been done before. Although it is disappointing, it is encouraging to see that the BLS SOII data are too coarse to convincingly show the Ag Center's impact on health outcomes. Undercounts continue despite recommendations from CTSE (2013) and subsequent investigations through 2017. Furthermore, given the current depleted state of the BLS due to government restrictions, further reforms seem unlikely, and the SOII will remain the only source of national-level data on nonfatal injuries and illnesses.
If NIOSH Ag Centers are reestablished in any form, with or without federal funding, findings presented here suggest that more appropriate measures of AgFF health outcomes should be recommended. The current study indicates that such measures should at least (1) include all sizes of AgFF operations, especially those with fewer than 11 employees, which may include workers without UI (such as undocumented workers and those on H1 visas), and (2) promote better communication and outreach between OSHA and AgFF operations at the state level, focusing on developing and sharing effective training materials on proper recordkeeping practices.
Footnotes
Declaration of Conflicting Interests
The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Centers for Disease Control and Prevention (CDC) (Grant Number U54OH007541-23-02).
