The use of theory-driven evaluation is an emerging practice in the military—an aspect generally unknown in the civilian evaluation community. First developed during the 1991 Gulf War and applied in both the Balkans and Afghanistan, these techniques are now being examined in the North Atlantic Treaty Organisation (NATO) as a means to evaluate the effects of military operations in complex, asymmetric conflict environments. In spite of these practices, theory-driven evaluation in the military is still in the developmental stages. This article traces the development to date of theory-driven evaluation in NATO and assesses its strengths and weaknesses in the military context. We conclude that a cross-pollination of ideas between military and civilian evaluators is urgently needed to improve the quality and effectiveness of military evaluation.
Alberts, D., & Hayes, R. (2003). Power to the edge-command and control in the information age. Washington, DC: Command and Control Research Program. Retrieved September 10, 2008, from http://www.dodccrp.org/files/Alberts_Power.pdf.
2.
Alberts, D., & Hayes, R. (2007). Planning for complex endeavors. Washington, DC: Command and Control Research Program . Retrieved September 10, 2008, from http://www.dodccrp.org/files/Alberts_Planning.pdf.
3.
Bärtl, M. (2007, March). Practices, challenges, and issues to resolve in campaign assessment-operational HQ experiences from ISAF. Paper presented at NATO Allied Command Transformation Operational Analysis Conference, Norfolk, VA.
4.
Behn, R. (2003). Why measure performance? Different purposes require different measures. Public Administration Review, 63, 586-606.
5.
Bickman, L. (1987). The functions of program theory. In Bickman. L (Ed.), Using program theory in evaluation (pp. 5-19). New Directions for Program Evaluation, Vol. 33. San Francisco, CA: Jossey Bass .
6.
Bickman, L. (2000). Summing up program theory. In Bickman. L (Ed.), Program theory in evaluation: Challenges and opportunities (pp. 103-112). New Directions for Evaluation, Vol. 87. San Francisco, CA: Jossey Bass .
7.
Chen, H.-T. (1990). Theory-driven evaluations. Newbury Park, CA: Sage.
Chen, H.-T. (2005). Practical program evaluation. Assessing and improving planning, implementation, and effectiveness. Thousand Oaks, CA: Sage.
10.
Chen, H.T., & Rossi, P.H. (1981). The multi-goal, theory-driven approach to evaluation: A model linking basic and applied social science. In Freeman. H. E, & Solomon. M. A (Eds.), Evaluation Studies Review Annual, Vol. 6. Beverly Hills, CA: Sage.
11.
Chen, H.-T., & Rossi, P.H. (1983). Evaluating with sense: The theory-driven approach . Evaluation Review, 7, 283-302.
12.
Chen, H.-T., & Rossi, P.H. (1987). The theory-driven approach to validity. Evaluation and Program Planning, 10, 95-103.
13.
Chen, H.-T., & Rossi, P.H. (1992). Using theory to improve program and policy evaluations . New York, NY: Green-wood Press .
14.
Clausewitz, C. von. (1968). On war (Graham, J. J., Trans.). London, Great Britain: Penguin. (Original work published 1832).
15.
Clements, P., Chianca, T., & Sasaki, R. (2008). Reducing world poverty by improving evaluation of development aid. American Journal of Evaluation, 29, 195-214.
16.
Curry, H. (2004). The current battle damage assessment paradigm is obsolete. Air and Space Power Journal, 18, 13-17.
17.
Deptula, D. (2001a). Effects-based operations: A change in the nature of warfare. Arlington, VA: Defence and Airpower Series, Aerospace Education Foundation. Retrieved December 14, 2008, from http://www.airforcemagazine.com/MagazineArchive/Documents/2001/April%202001/0401effects.pdf .
18.
Deptula, D. (2001b). Firing for effects. Air Force , 84, 45-53. Retrieved September 10, 2008, from http://www.afa.org/magazine/April2001/0401effects.pdf .
19.
Diehl, J.E., & Sloan, C.E. (2005). Battle damage assessment: The ground truth. Joint Force Quarterly, 37, 59-64.
20.
Donaldson, S.I. (2007). Program theory-driven evaluation science: Strategies and applications. New York, NY: Taylor and Francis.
21.
Drucker, P.F. (1954). The practice of management. New York, NY: Harper and Row.
22.
Defence, Science and Technology Laboratory. ( 2005). Code of best practice for the use of measures of effectiveness (MOE) (Report No. Dstl/CR14304v1.1). UK Ministry of Defence .
23.
English, B., & Kaleveld, L. (2003). The politics of program logic. Evaluation Journal of Australasia, 3, 35-42.
24.
Evans, D. (2003). Operational analysis in support of HQ ISAF, Kabul, Afghanistan, 2002. In Woodcock. A, & Davis. D (Eds.), Analysis for governance and stability (pp. 198-226). Cornwallis, Canada: The Canadian Peacekeeping Press.
25.
Gargani, J. (2003, November). The history of theory-based evaluation: 1909 to 2003. Paper presented at the American Evaluation Association annual conference, Reno, NV.
26.
Gartner, S.S., & Myers, M.E. (1995). Body counts and ``success'' in the Vietnam and Korean wars. Journal of Interdisciplinary History, 25, 377-395.
27.
Hopkin, A.J. (2004). Operational analysis in support of HQ MND(SE), Basrah, Iraq, 2003. In Woodcock. A, & Rose. G (Eds.), Analysis for Stabilization and Counter-Terrorist Operations, Cornwallis , Canada: The Canadian Peacekeeping Press.
28.
ISAF mission. (2008, July). ISAF Mirror, 49. ISAF Public Affairs Office. Retrieved September 10, 2008, from http://www.nato.int/isaf/docu/mirror/2008/mirror_49_200807.pdf .
29.
Jobbagy, Z. (2005). Powered flight, strategic bombing and military coercion: Study on the origins of effects-based operations. Den Haag, The Netherlands: The Clingendael Centre for Strategic Studies. Retrieved December 14, 2008, from http://www.hcss.nl/en/publication/37/Powered-Flight,-Strategic-Bombing-and-Military-Coe.html .
30.
Lambert, N.J. (2002). Measuring the success of the NATO operation in Bosnia and Herzegovina 1995-2000. European Journal of Operational Research, 140, 459-481.
31.
Lane, R., & Sky, E. (2006). The role of provincial reconstruction teams in stabilization. Royal United Services Institute Journal, 151, 46-51.
32.
Lipsey, M.W., Crosse, S., Dunkle, J., Pollard, J., & Stobart, G. (1985). Evaluation: The state of the art and the sorry state of the science. In Cordray. D. S (Ed.), Utilizing prior research in evaluation planning (pp. 7-28). New Directions for Program Evaluation, Vol. 27. San Francisco: Jossey-Bass.
33.
Maley, W. (2007). Provincial reconstruction teams in Afghanistan-how they arrived and where they are going. NATO Review, 2007(3), NATO. Retrieved September 10, 2008, from http://www.nato.int/docu/review/2007/issue3/english/art2.html.
34.
Mattis, J.N. (2008). USJFOM commander's guidance for effects-based operations. Joint Force Quarterly, 51, 105-108.
35.
Moltke, H. von. (1993). On the art of war: Selected writings. (Hughes, D. J., & Bell, H. Trans.; Hughes, D. J., Ed.). New York, NY: Random House (Original work published in 1871).
36.
Murray, W., & Scales, R.H. (2003). The Iraq war: A military history. Cambridge, MA: Harvard University.
37.
Nato. (2006a). MC position on an effects-based approach to operations. North Atlantic Military Committee (signed 06 June 2006). (Document No. MCM 0052-2006/IMSWM-0147-2006 SD3).
Nato. (2007a). BiSC draft pre-doctrinal EBAO handbook version 4.2, October 4, 2007. Bi-Strategic Command Effects-Based Approach to Operations Working Group.
40.
Nato. (2007b). Bi-Strategic Command Discussion Paper on EBAO. (Document No. J5PLANS/2920-036/07). H.Q. Supreme Allied Powers Europe and H.Q. Supreme Allied Command Transformation .
Nato. (2007d). Effects-based assessment NATO analysis report. Limited objective experiment conducted as a portion of the multinational experiment series 5, 04-08 June 2007. H.Q. Supreme Allied Command Transformation, Norfolk, VA.
Neighbour, M., Bailey, P., Hawthorn, M., Lensing, C., Robson, H., Smith, S., et al. (2002). Providing operational analysis to a peace support operation: The Kosovo experience. Journal of the Operational Research Society, 53, 523-543.
47.
Organisation for Economic Cooperation and Development. (2003, July). Policy coherence: Vital for global development . Retrieved September, 10, 2008, from http://www.oecd.org/dataoecd/11/35/20202515.pdf.
48.
Perrin, B. (1998). Effective use and misuse of performance measurement . American Journal of Evaluation, 19, 367-379.
49.
Picciotto, R. (2005). The evaluation of policy coherence for development . Evaluation, 11, 311-330.
50.
Picciotto, R. (2007). The new environment for development evaluation . American Journal of Evaluation, 28, 509-521.
51.
Posavac, E.J., & Carey, R.G. (2006). Program evaluation: Methods and case studies (7th ed.). Upper Saddle River, NJ: Pearson.
52.
Pressman, J.L., & Wildavsky, A.B. (1984). Implementation. Berkeley, CA: University of California.
53.
Rauch, J.T. (2002). Assessing airpower's effects: Capabilities and limitations of real-time battle damage assessment. (Masters dissertation, School of Advanced Airpower, Air University, 2002). Retrieved September 10, 2008, from http://handle.dtic.mil/100.2/ADA420587 .
54.
Rogers, P.J. (2000). Program theory: Not whether programs work but how they work. In Stufflebeam. D. L, Madaus. G. F, & Kellaghan. T (Eds.), Evaluation models (pp. 209-232). Boston: Kluwer Academic.
55.
Rogers, P.J. (2007). Theory-based evaluation: Reflection ten years on. In Mathison, S. (Ed.), Enduring issues in evaluation: The 20th anniversary of the collaboration between NDE and AEA. New Directions for Evaluation, Vol. 114. San Francisco: Jossey Bass.
56.
Sartorius, R.H. (1991). The logical framework approach to project design and management. American Journal of Evaluation, 12, 139-147.
57.
Scheirer, M.A. (1987). Program theory and implementation theory: Implications for evaluators. In Bickman, L. (Ed.), Using program theory in evaluation (pp. 59-76). New Directions for Program Evaluation, Vol. 33. San Francisco: Jossey Bass.
58.
Schick, A. (1969). System politics and systems budgeting. Public Administration Review, 29, 137-151.
59.
Scriven, M. (1994). The fine line between evaluation and explanation . Evaluation Practice, 15, 75-77.
60.
Shaw, I., & Crompton, A. (2003). Theory, like mist on spectacles, obscures vision . Evaluation, 9, 192-204.
61.
Smith, E. (2003). Effects-based operations: Applying network centric warfare in peace, crisis and war. Washington, DC: Command and Control Research Program. Retrieved September 10, 2008, from http://www.dodccrp.org/files/Smith_EBO.PDF .
62.
Smith, E. (2006). Complexity, networking, and effects-based approaches to operations. Washington, DC: Command and Control Research Program. Retrieved September 10, 2008, from http://www.dodccrp.org/files/Smith_Complexity.pdf .
63.
Stufflebeam, D. L. (Ed.). (2001). Evaluation models. New directions for evaluation, 89. San Francisco: Jossey-Bass.
64.
Sullivan, H. & Stewart, M. (2006). Who owns the theory of change? Evaluation, 12, 179-199.
65.
Swedish Armed Forces Headquarters. (2007). Swedish EBAO development after the autumn experiment 2006. Enkoping, Sweden: Swedish Armed Forces.
66.
Unesco. (2007). Results-based programming, management and monitoring (RBM) at UNESCO: Guiding principles . Bureau of Strategic Planning, UNESCO, Paris, France. Retrieved September 10, 2008, from http://portal.unesco.org/fr/files/40194/11924654571BSP_RBM_guiding_principles_October_2007__2_.pdf/BSP+RBM+guiding+principles+October+2007+_2_.pdf.
67.
Weiss, C.H. (1997a). How can theory-based evaluations make greater headway? Evaluation Review, 21, 501-524.
68.
Weiss, C.H. (1997b). Theory-based evaluation: Past, present and future . In Rog, D. J., & Fournier, D. (Eds.), Progress and future directions in evaluation: Perspectives on theory, practice, and methods, New Directions for Evaluation, Vol. 76. San Francisco, CA: Jossey Bass.
69.
Wildavsky, A.B. (1966). The political economy of efficiency: Cost-benefit analysis, systems analysis, and program budgeting. Public Administration Review, 26, 292-310.
70.
Wildavsky, A.B. (1969). Rescuing policy analysis from PPBS. Public Administration Review, 29, 189-202.
