Evaluation theory provides a rigorous foundation for the practice of military operation assessment. Government and industry assessors have used evaluation theory to improve the effectiveness of assessment across a wide range of fields. This article focuses on the relationship between evaluation theory and military assessment. We briefly survey the major evaluation approaches with a focus on connecting the theoretical models to practical security-related applications. These evaluation approaches include expertise-oriented, program-oriented, decision-oriented, and participant-oriented models. Within the overarching framework of these approaches, we consider alternative monitoring and evaluation designs in detail, including descriptive designs (case study, cross-sectional, time-series), quasi-experimental designs (interrupted time-series, comparison group, case study), and experimental designs (posttest-only, pre-post). Then, we discuss quantitative and qualitative methods for analyzing and reporting uncertainty with respect to each design alternative, with an emphasis on mixed-method approaches. Throughout the review, we make the relationship between evaluation theory and operation assessment practice explicit through examples, and we suggest more detailed references where appropriate.
ArnhartLMBKingML. Are we there yet? Implementing best practices in assessments. Mil Rev2018; 97: 20–29.
2.
FitzpatrickJLSandersJRWorthenBR. Program evaluation: Alternative approaches and practical guidelines. 4th ed.Boston: Pearson, 2011.
3.
WholeyJSHatryHPNewcomerKE (eds) Handbook of practical program evaluation. 3rd ed.San Francisco: John Wiley & Sons, 2010.
4.
EisnerEW. Educational connoisseurship and criticism: Their form and functions in educational evaluation. J Aesthet Educ1976; 10: 135.
5.
TylerRW. General statement on evaluation. J Educ Res1942; 35: 492–501. DOI:10.1080/00220671.1942.10881106.
6.
National Performance Review. Reaching public goals: Managing government for results. Technical report, Government Printing Office, Washington, DC, 1993.
7.
Office of Management, and Budget. Program evaluation: What constitutes strong evidence of program effectiveness? Technical report, Government Printing Office, Washington, DC, 2004.
8.
MarquisJPMcnerneyMJZimmermanSR, et al. Developing an assessment, monitoring, and evaluation framework for U.S. Department of Defense security cooperation. Technical report, RAND, Santa Monica, CA, 2016.
9.
PaulCYeatsJClarkeCP, et al. Assessing and evaluating department of defense efforts to inform, influence, and persuade: Desk reference. Santa Monica, CA: Rand Corporation, 2015.
10.
Office of the Under Secretary of Defense for Policy. DoD Instruction 5132.14: Assessment, Monitoring, and Evaluation Policy for the Security Cooperation Enterprise. Washington, DC: Government Printing Office, 2017.
11.
WilliamsAPMorrisJC. The development of theory-driven evaluation in the military: Theory on the front line. Am J Eval2009; 30: 62–79. DOI: 10.1177/1098214008329522.
WeissCHMarkMM. The oral history of evaluation part IV: The professional evolution of Carol Weiss. Am J Eval2006; 27: 475–483.
15.
CousinsJBEarlLM. The case for participatory evaluation. Educ Eval Policy Anal1992; 14: 397–418. DOI:10.3102/01623737014004397.
16.
MarkMMShotlandRL. Stakeholder-based evaluation and value judgments: The role of perceived power and legitimacy in the selection of stakeholder groups. Eval Rev1985; 9: 605–626.
17.
North Atlantic Treaty Organization. NATO operations assessment handbook version 3.0. Arlington, VA: Department of Defense, 2015.
18.
United States Joint Chiefs of Staff. Joint publication 5–0: Joint planning. Arlington, VA: Department of Defense, 2017.
19.
United States Joint Chiefs of Staff. Joint publication 3–0: Joint operations. Arlington, VA: Department of Defense, 2017.
20.
Joint Staff J-7. Commander’s handbook for assessment planning and execution. Arlington, VA: Department of Defense, 2011.
21.
United States Joint Chiefs of Staff. Joint doctrine note 1–15: Operation assessment. Arlington, VA: Department of Defense, 2015.
22.
United States Joint Chiefs of Staff. Operation assessment: Multi-service tactics, techniques, and procedures. Technical report, Department of Defense, Arlington, VA, 2015.
23.
Office of the Chief of Naval Operations. Navy warfare publication 5–01: Navy planning. Arlington, VA: Department of Defense, 2013.
24.
Department of the Army. Field Manual 6–0: Commander and staff organization and operations. Arlington, VA: Department of Defense, 2014.
25.
Department of the Army. Army doctrine publication 3–0 operations. Arlington, VA: Department of Defense, 2017.
26.
Military Operations Research Society. Advancing the professionalism of assessments. Technical report, Military Operations Research Society, Alexandria, VA, 2018.
27.
StakeRE. The art of case study research. Thousand Oaks, CA: Sage, 1995.
28.
YinRK. Case study research: Design and methods. London and Singapore: Sage, 2009.
29.
CookTDCampbellDTShadishW. Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin, 2002.
SaieCMAhnerDK. Investigating the dynamics of nation-building through a system of differential equations. J Oper Res Soc2018; 69: 619–629. DOI: 10.1057/s41274–017–0256-x.
35.
MontgomeryDC. Design and analysis of experiments. Hoboken, NJ: John Wiley & Sons, 2017.
36.
FersonSJoslynCAHeltonJC, et al. Summary from the epistemic uncertainty workshop: Consensus amid diversity. Reliab Eng Syst Safe 2004; 85: 355–369. DOI:10.1016/j.ress.2004.03.023.
37.
KnightFH. Risk, uncertainty and profit. New York: Hart, Schaffner and Marx, 1921.
38.
CamererCWeberM. Recent developments in modeling preferences: Uncertainty and ambiguity. J Risk Uncertain1992; 5: 325–370.
VardemanSB. What about the other intervals?Am Stat1992; 46: 193–197. DOI:10.1080/00031305.1992.10475882.
41.
MeekerWQHahnGJEscobarLA.Statistical intervals: A guide for practitioners and researchers. 2nd ed.Hoboken, NJ: John Wiley &; Sons (2017, accessed 6 March 2019).
42.
MushenESchrodenJ. Are we winning? A brief history of military operations assessment. Technical report, CNA, Arlington, VA, 2014.
43.
MathisonS. Encyclopedia of evaluation. Thousand Oaks, CA: Sage (2005, accessed 6 March 2019).
