Impact assessment seeks to evaluate the effects of a new system realized on target beneficiaries and is an essential process via which to tangibly demonstrate the operational and economic benefits of a research and development (R&D) program. This paper contributes a framework –Program-management Understanding, Measurement, and Assessment (PUMA) - for developing an impact assessment approach for planning and evaluating R&D programs. The intent of the framework is to help an R&D organization provide traceability from the identification of program needs to selecting and conducting R&D to implementation to defining and measuring results. The framework is demonstrated using an Office of Naval Research project.
Get full access to this article
View all access options for this article.
References
1.
BrinkerhoffR. (1988). An integrated evaluation model for HRD. Training and Development Journal, 42(2), 66–68.
2.
BrinkerhoffR. (2003). The success case method: Find out quickly what's working and what's not. San Francisco: Berrett-Koehler.
3.
BrinkerhoffR.GillS. (1994). The Learning alliance: Systems thinking in human resource development. San Francisco: Jossey-Bass.
4.
BushnellD. (1990). Input, process, output: A model for evaluating training. Training and Development Journal, 42(3), 41–43.
5.
CohnJ.StanneyK.M.MilhamL.Bell CarrollM.JonesD.SullivanJ.DarkenR. (2008). Training effectiveness evaluation: From theory to practice. In SchmorrowD.CohnJ.NicholsonD. (Eds.), The Handbook of Virtual Environment Training: Understanding, Predicting and Implementing Effective Training Solutions for Accelerated and Experiential Learning (Vol 3, Sec 2, pp. 157–172). Aldershot, Hampshire, UK: Ashgate Publishing.
6.
EricksonP. (1990). Evaluating training results. Training and Development Journal, 44(1), 57–59.
7.
FoxonM. (1994). A process approach to transfer of training. Part 2: Using action planning to facilitate the transfer of training. Australian Journal of Educational Technology, 10(1), 1–18.
8.
GilbertT. (1978). Human competence: Engineering worthy performance. New York: McGraw-Hill.
9.
HamblinA. (1974). Evaluation and control of training. London: McGraw-Hill.
10.
HoltonE. (2005). Holton's evaluation model: New evidence and construct elaboration. Advances in Developing Human Resources, 7(1), 37–54.
11.
KaufmanR.KellerJ. (1994). Levels of evaluation: Beyond Kirkpatrick. Human Resource Development Quarterly, 5(4), 371–380.
12.
KirkpatrickD.KirkpatrickJ. (2006). Evaluating training programs: The four levels. San Francisco: Berrett-Koehler Publishers.
13.
KotnourT. (2009). Transforming Organizations: Strategies and Methods. Taylor & Francis.
14.
KraigerK.FordJ.SalasE. (1993). Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training and evaluation. Journal of Applied Psychology, 78(2), 311–328.
15.
LincolnR.DunetD. (1995). Training efficiency and effectiveness model (TEEM). Performance and Instruction, 34(3), 40–47.
16.
PhillipsJ.PhillipsP. (2005). ROI at work. Alexandria, VA: American Society for Training and Development.
17.
PrenskillH.Russ-EftD. (2003) A framework for reframing HRD evaluation practice and research. In GilleyA.BieremaL.CallahanJ. (Eds.), Critical Issues in HRD pp.199–257. Cambridge, MA: Perseus.
18.
PreskillH.Russ-EftD. (2005). Building evaluation capacity: 72 activities for teaching and training. Thousand Oaks, CA: Sage.
19.
RicheyR. (1992). Designing instruction for the audit learning. London: Kogan Page.
20.
Russ-EftD.BoberM.de la TejaI.FoxonM.KoszalkaT. (2008). Evaluator competencies: Standards for the practice of evaluation in organizations. San Francisco: Jossey-Bass.
21.
SalasE.MilhanL.CannonC. (2003). Training evaluation in the military: Misconceptions, opportunities, and challenges. Military Psychology, 15(1), 3–16.
22.
SimpsonH.OserR. (2003). Evaluating large scale training simulations. Military Psychology, 15(1), 25–40.
23.
SwansonR.HoltonE. (1999). Results: How to assess performance, learning, and perceptions in organizations. San Francisco: Berrett-Koehler.
24.
SwansonR.SleezerC. (1987). Training effectiveness evaluation. Journal of European Industrial Training, 11(4), 7–16.
25.
WangG.SpitzerD. (2005). Human resource development. Advances in Developing Human Resources, 7(1), 5–15.
