Using a set of scenarios derived from actual evaluation studies, this simulation study examines the reported influence of evaluation information on decision makers’ potential actions. Each scenario described a context where one of three types of evaluation information (large-scale study data, case study data, or anecdotal accounts) is presented and a specific decision needs to be made. Participants were asked to indicate which type of data presented would influence their decision making. Results from 131 participants indicate that participants were influenced by all types of information, yet large-scale and case study data are more influential relative to anecdotal accounts; certain types of evaluation data are more influential among certain groups of decision makers; and choosing to use one type of evaluation data over the other two depends on the independent influence of other types of evaluation data on the decision maker, as well as prior beliefs about program efficacy.
Alessi, S. M. (1988). Fidelity in the design of instructional simulations. Journal of Computer-Based Instruction, 15(2), 40-47.
2.
Alkin, M. C., Daillak, R., & White, B. (1979). Using evaluations. Beverly Hills, CA: Sage.
3.
Braskamp, L. A. (1980). Credibility of a local education program evaluation report: Author source and client characteristics. American Educational Research Journal, 15, 441-450.
4.
Brown, R. D., Braskamp, L. A., & Newman, D. L. (1978). Evaluator credibility and acceptance as a function of report styles: Do jargon and data make a difference? Evaluation Quarterly, 2, 331-341.
5.
Brown, R. D., Braskamp, L. A., & Newman, D. L. (1982). Studying evaluation utilization through simulations. Evaluation Review, 6, 114-126.
6.
Brown, R. D., & Newman, D. L. (1982). An investigation of the effects of different data presentation formats and order of arguments in a simulated adversary evaluation. Education Evaluation and Policy Analysis, 4, 197-204.
7.
Brown, R. D., Newman, D. L., & Rivers, L. S. (1985). Does the superintendent’s opinion affect school boards’ evaluation information needs? An empirical investigation. Urban Education, 20(2), 204-221.
8.
Cannon, H. M., & Burns, A. C. (1999). A framework for assessing the competencies reflected in simulation performance. Developments in Business Simulation & Experiential Exercises, 26, 40-44.
9.
Dickson, J. R., Whitely, R. R., & Faria, A. J. (1990). An empirical investigation of the internal validity of a marketing simulation game. Developments in Business Simulation & Experiential Exercises, 17, 47-52.
10.
Donaldson, S. I., & Christie, C. A. (2005). The 2004 Claremont debate: Lipsey vs. Scriven. Determining causality in program evaluation and applied research: Should experimental evidence be the gold standard? Journal of Multidisciplinary Evaluation, 3, 60-77.
11.
Einhorn, H. J., & Hogarth, R. M. (1986). Decision making under ambiguity. Journal of Business, 59(4), 225-250.
12.
Feinstein, A. H., & Cannon, H. M. (2002). Constructs of simulation evaluation. Simulation & Gaming, 33(4), 425-440.
13.
Ford, R. C., & Richardson, W. D. (1994). Ethical decision making: A review of the empirical literature. Journal of Business Ethics, 13(3), 205-221.
14.
Granville, A. (1977). Where do school decisions really come from? Sources of influence and the influence of evaluation. Unpublished doctoral dissertation, University of California, Los Angeles.
15.
Henry, G. T. (2000). Why not use? New Directions for Evaluation, 88, 85-98.
16.
Henry, G. T., & Gordon, C. S. (2004, November). Coming full circle: Assessing evaluation’s influence on program performance. Paper presented at the American Evaluation Association annual conference, Atlanta, GA.
17.
Henry, G. T., & Mark, M. M. (2003). Beyond use: Understanding evaluation’s influence on attitudes and actions. American Journal of Evaluation, 24, 293-314.
18.
Hodgkinson, G. P., Bown, N. J., Maule, A. J., Glaister, K. W., & Pearman, A. D. (1999). Breaking the frame: An analysis of strategic cognition and decision making under uncertainty. Strategic Management Journal, 20(10), 977-985.
19.
Kirkhart, K. E. (2000). Reconceptualizing evaluation use: An integrated theory of influence. New Directions for Evaluation, 88, 5-23.
20.
Mark, M. M. (2004, November). Building a better evidence-base for evaluation theory: Examining questions of feasibility and utility and specifying “how to.” Paper presented at the American Evaluation Association annual conference, Atlanta, GA.
21.
Mark, M. M., & Henry, G. T. (2004). The mechanisms and outcomes of evaluation influence. Evaluation, 10, 35-57.
22.
O’Neil, H. F., Allred, K., & Dennis, R. A. (1997). Validation of a computer simulation for assessment of interpersonal skills. In H. F. O’Neil, Jr. (Ed.), Workforce readiness: Competencies and assessment (pp. 229-254). Mahwah, NJ: Lawrence Erlbaum.
23.
Ory, J. C., & Braskamp, L. A. (1981). Faculty perceptions of the quality and usefulness of three types of evaluation information. Research in Higher Education, 15, 271-282.
24.
Pablo, A. L. (1994). Determinants of acquisition integration level: A decision making perspective. Academy of Management Journal, 37(4), 803-836.
Patton, M. Q., Grimes, P. S., Guthrie, K. M., Brennan, N. J., Franch, B. D., & Blyth, D. A. (1977). In search on impact: An analysis of the utilization of federal health evaluation research. In C. H. Weiss (Ed.), Using social research in public policy making (pp. 141-163). Lexington, MA: D. C. Heath.
27.
Prohaska, T. R., Keller, M. L., Leventhal, E. A., & Leventhal, H. (1987). Impact of symptoms and aging attribution on emotions and coping. Health Psychology, 6(6), 495-514.
28.
Rausser, G. C., & Johnson, S. R. (1975). On the limitation of simulation in model evaluation and decision analysis. Simulation & Games, 6(2), 115-150.
29.
Rosenstock, I. M., Strecher, V. J., & Becker, M. H. (1988). Social learning theory and the health belief model. Health Education Quarterly, 15(2), 175-183.
30.
Scientific culture and educational research. (2002). Educational Researcher, 31(8).
31.
Smith, H. W., Winer, J. L., & George, C. E. (1983). The relative efficacy of simulation experiments. Journal of Vocational Behavior, 22(1), 96-104.
32.
Tan, S. J., & Halpern, D. F. (2006). Applying the science of psychology to a public that distrusts science. In S. I. Donaldson, D. E. Berger, & K. Pezdek (Eds.), Applied psychology: New frontiers and rewarding careers (pp. 153-170). Mahwah, NJ: Lawrence Erlbaum.
33.
Ward, E. (1954). The theory of decision making. Psychological Bulletin, 51, 380-417.
34.
Weiss, C. H., Murphy-Graham, E., & Birkeland, S. (2005). An alternative route to policy influence: How evaluations affect D.A.R.E. American Journal of Evaluation, 26, 12-30.
