Interest from Congress, executive branch leadership, and various other stakeholders for greater accountability in government continues to gain momentum today with government-wide efforts. However, measuring the impact of research programs has proven particularly difficult. Cause and effect linkages between research findings and changes to morbidity and mortality are difficult to prove. To address this challenge, the National Institute for Occupational Safety and Health program evaluators used a modified version of contribution analysis (CA) to evaluate two research programs. CA proved to be a useful framework for assessing research impact, and both programs received valuable, actionable feedback. Although there is room to further refine our approach, this was a promising step toward moving beyond bibiliometrics to more robust assessment of research impact.
AgarwalA.DurairajanayagamD.TatagariS.EstevesS. C.HarlevA.HenkelR.…RamasamyR. (2016). Bibliometrics: Tracking research impact by selecting the appropriate metrics. Asian Journal of Andrology, 18, 296–309. doi:10.4103/1008-682X.171582
2.
American Evaluation Association. (2013). An evaluation roadmap for a more effective government. Retrieved fromhttp://www.eval.org/p/cm/ld/fid=52
3.
BiggsJ. S.FarrellL.LawrenceG.JohnsonJ. K. (2014). A practical example of contribution analysis to a public health intervention. Evaluation, 20, 214–229.
4.
BornmannL. (2017). Measuring impact in research evaluations: A thorough discussion of methods for, effects of and problems with impact measurements. Higher Education, 73, 775–787.
CohenG.SchroederJ.NewsonR.KingL.RychetnikL.MilatA. J.…ChapmanS. (2015). Does health intervention research have real world policy and practice impacts: Testing a new impact assessment tool. Health Research Policy and Systems, 13, 3.
7.
DelahaisT.ToulemondeJ. (2012). Applying contribution analysis: Lessons from five years of practice. Evaluation, 18, 281–293.
8.
DybdalL.NielsenS. B.LemireS. (2011). Contribution analysis applied: Reflections on scope and methodology. The Canadian Journal of Program Evaluation, 25, 29–57.
9.
GertlerP.MartinezS.PremandP.RawlingsL.VermeerschC. (2011). Impact evaluation in practice. Washington, DC: The World Bank.
10.
GreenhalghT.RafteryJ.HanneyS.GloverM. (2016). Research impact: A narrative review. BMC Medicine, 14, 78.
11.
HeatonJ. (2008). Secondary analysis of qualitative data: An overview. Historical Social Research/Historische Sozialforschung, 33(3), 33–45.
12.
HowardJ. (2009). Informing public health policy and practice: The strategic management of research processes and organizations. Governance, 22, 203–216.
13.
Institute of Medicine and National Research Council. (2009). Evaluating occupational health and safety research programs: Framework and next steps. Washington, DC: National Academies Press.
14.
KokM. O.SchuitA. J. (2012). Contribution mapping: A method for mapping the contribution of research to enhance its impact. Health Research Policy and Systems, 10, 21.
15.
LemireS. T.NielsenS. B.DybdalL. (2012). Making contribution analysis work: A practical framework for handling influencing factors and alternative explanations. Evaluation, 18, 294–309.
16.
MaynardR.GoldsteinN.NightingaleD. S. (2016). Program and policy evaluations in practice: Highlights from the federal perspective. New Directions for Evaluation, 2016, 109–135.
17.
MayneJ. (2001). Addressing attribution through contribution analysis: Using performance measures sensibly. The Canadian Journal of Program Evaluation, 16, 1.
18.
MayneJ. (2011). Contribution analysis: Addressing cause and effect. In ForssK.MarraM.SchwartzR. (Eds.), Evaluating the complex: Attribution, contribution, and beyond. New Brunswick, NJ: Transaction Publishers.
PattonM. Q. (2012). A utilization-focused approach to contribution analysis. Evaluation, 18, 364–377.
24.
RallisS. F.RossmanG. B. (2000). Dialogue for learning: Evaluator as critical friend. New Directions for Evaluation, 2000, 81–92.
25.
SarliC. C.DubinskyE. K.HolmesK. L. (2010). Beyond citation analysis: A model for assessment of research impact. Journal of the Medical Library Association, 98(1), 17–23.
26.
U.S. Government Accountability Office. (2013). Managing for results: Executive branch should more fully implement the GPRA modernization act to address governance challenges (GAO-13-518). Retrieved fromhttp://www.gao.gov/assets/660/655541.pdf
27.
WilliamsV.EisemanE.LandreeE.AdamsonD. (2009). Demonstrating and communicating research impact: Preparing NIOSH programs for external review. Santa Monica, CA: RAND Corporation.
