Evaluation theories can be tested in various ways. One approach, the experimental analogue study, is described and illustrated in this article. The approach is presented as a method worthy to use in the pursuit of what Alkin and others have called descriptive evaluation theory. Drawing on analogue studies conducted by the first author, we illustrate the potential benefits and limitations of analogue experiments for studying aspects of evaluation and for contributing to the development and refinement of evaluation theory. Specifically, we describe the results of two studies that examined stakeholder dialogue under different conditions of accountability frame, interpersonal motives, and epistemic motives. We present the studies’ main findings while highlighting the potential for analogue studies to investigate questions of interest concerning evaluation practice and theory. Potentials and pitfalls of the analogue study approach are discussed.
AbmaT. A. (2000). Stakeholder conflict: A case study. Evaluation and Program Planning, 23, 199–210.
2.
AbmaT. A. (2006). The practice and politics of responsive evaluation. American Journal of Evalution, 27, 31–43.
3.
AbmaT. A.GreeneJ. C.KarlssonO.RyanK.SchwandtT. A.WiddershovenG. A. M. (2001). Dialogue on dialogue. Evaluation, 7, 164–180.
4.
AlkinM. (2003). Evaluation theory and practice: Insights and new directions. In ChristieC. A. (Ed.), The Practice-Theory Relationship in Evaluation(New Directions for Evaluation, 97, pp. 81–89). San Francisco, CA: Jossey-Bass.
5.
AronsonE.CarlsmithJ. M. (1968). Experimentation in social psychology. In LindzeyG.AronsonE. (Eds.), Handbook of social psychology (2nd ed., Vol. 2, pp. 1–79). Reading, MA: Addison-Wesley.
6.
BaurV. E.VanElterenA. H. G.NierseC. J.AbmaT. A. (2010). Dealing with distrust and power dynamics: Asymmetric relations among stakeholders in responsive evaluation. Evaluation, 16, 233–248.
7.
BeersmaB.DeDreuC. K. W. (2002). Integrative and distributive negotiation in small groups: Effects of task structure, decision rule, and social motive. Organizational Behavior and Human Decision Processes, 87, 227–252.
8.
BrandonP. R. (1998). Stakeholder participation for the purpose of helping ensure evaluation validity: Bridging the gap between collaborative and non-collaborative evaluations. American Journal of Evaluation, 19, 325–337.
9.
BraskampL. A.BrownR. D.NewmanD. L. (1982). Studying evaluation utilization through simulations. Evaluation Review, 6, 114–126.
10.
BrownR. D.NewmanD. L. (1982). An investigation of the effect of different data presentation formats and order of arguments in a simulated adversary evaluation. Educational Evaluation and Policy Analysis, 4, 197–203.
11.
BrownR. D.NewmanD. L.RiversL. S. (1984). A decisionmaking context model for enhancing evaluation utilization. Educational Evaluation and Policy Analysis, 6, 393–400.
12.
BrownR. D.PrenticeD. G. (1987). Assessing decision-making risk and information needs in evaluation. Evaluation Review, 11, 371–381.
13.
CampbellB.MarkM. M. (2006). Toward more effective stakeholder dialogue: Applying theories of negotiation to policy and program evaluation. Journal of Applied Social Psychology, 36, 2834–2863.
14.
CampbellB.McGrathA. L. (2011). Where the rubber hits the road: The development of usable middle-range evaluation theory. In MarkM. M.DonaldsonS. I.CampbellB. (Eds.), Social psychology and evaluation (pp. 346–371). New York, NY: Guilford.
15.
CarnevaleP. J.DeDreuC. K. W. (2006). Motive: The negotiator’s raison d’etre. In ThompsonL. (Ed.), Negotiation theory and research (pp.55–76). New York, NY: Psychology Press.
16.
CarnevaleP. J.PruittD. G. (1992). Negotiation and mediation. Annual Review of Psychology, 43, 531–582.
17.
ChristieC. A. (2011). Advancing empirical scholarship to further develop evaluation theory and practice. Canadian Journal of Program Evaluation, 26, 1–18.
18.
CialdiniR. B. (1980). Full-cycle social psychology. In BickmanL. (Ed.), Applied social psychology annual (Vol. 1, pp. 21–48). Beverly Hills, CA: Sage.
19.
CousinsJ. B. (2003). Utilization effects of participatory evaluation. In KellaghanT.StufflebeamD. L.WingateL. A. (Eds.), International handbook of educational evaluation (pp. 245–265). Boston, MA: Kluwer.
20.
CunninghamG. B.ChelladuraiP. (2004). Affective reactions to cross-functional teams: The impact of size, relative performance, and common in-group identity. Group Dynamics: Theory, Research and Practice, 8, 83–97.
21.
DeDreuC. K. W. (2007). Cooperative outcome interdependence, task reflexivity, and team effectiveness: A motivated information processing perspective. Journal of Applied Psychology, 92, 628–638.
22.
DeDreuC. K. W.BeersmaB.StroebeK.EuwemaM. C. (2006). Motivated information processing, strategic choice, and the quality of negotiated agreement. Journal ofPersonality and Social Psychology, 90, 927–943.
23.
DeDreuC. K. W.EversA.BeersmaB.KluwerE. S.NautaA. (2001). A theory-based measure of conflict management strategies in the workplace. Journal of Organizational Behavior, 22, 645–668.
24.
DeDreuC. K. W.WeingartL. R.KwonS. (2000). Influence of social motives on integrative negotiation: A meta-analytic review and test of two theories. Journal of Personality and Social Psychology, 78, 889–905.
25.
DonaldsonS. I.CranoW. D. (2011). Theory-driven evaluation science and applied social psychology: Exploring the intersection. In MarkM. M.DonaldsonS. I.CampbellB. (Eds.), Social psychology and evaluation (pp. 141–160). New York, NY: Guilford.
26.
DonaldsonS. I.GoolerL. E.ScrivenM. (2002). Strategies for managing evaluation anxiety: Toward a psychology of program evaluation. American Journal of Evaluation, 23, 261–273.
27.
DonaldsonS. I.LipseyM. W. (2006). Roles for theory in contemporary evaluation practice: Developing practical knowledge. In ShawI.GreeneJ. C.MarkM. M. (Eds.), The Sage handbook of evaluation (pp. 56–75). London, England: Sage.
28.
DovidioJ. F.GaertnerS. L.NiemannY. F.SniderK. (2001). Racial, ethnic, cultural differences in responding to distinctiveness and discrimination on campus: Stigma and common group identity. Journal of Social Issues, 5, 167–188.
29.
DovidioJ. F.GaertnerS. L.ValidzicA. (1998). Intergroup bias: Status, differentiation, and a common in-group identity. Journal of Personality and Social Psychology, 75, 109–120.
30.
FlemingM. A. (2011). Attitudes, persuasion, and social influence: Applying social psychology to increase evaluation use. In MarkM. M.DonaldsonS. I.CampbellB. (Eds.), Social psychology and evaluation (pp. 212–241). New York, NY: Guilford.
31.
Geva-MayI.ThorngateW. (2003). Reducing anxiety and resistance in policy and programme evaluations: A socio-psychological analysis. Evaluation, 9, 205–227.
32.
GoldmanL. (1976). A revolution in counseling research. Journal of Counseling Psychology, 23, 543–552.
33.
GreeneJ. C. (2000). Challenges in practicing deliberative democratic evaluation. In RyanK. E.DeStefanoL. (Eds.), Evaluation as a democratic process: Promoting inclusion, dialogue, and deliberation. New Directions for Evaluation, (Vol. 85, pp. 13–26). San Francisco, CA: Jossey-Bass.
34.
GreeneJ. C. (2001). Dialogue in evaluation: A relational perspective. Evaluation, 7, 181–203.
35.
GreeneJ. C. (2007). Mixed methods in social inquiry. San Francisco, CA: John Wiley.
36.
HenryG. T.MarkM. M. (2003). Toward an agenda for research on evaluation. In ChristieC. A. (Ed.), The practice-theory relationship in evaluation (Vol. 97, pp. 69–80). San Francisco, CA: Jossey-Bass.
37.
HeymanR. E.SlepA. M. S. (2004). Analogue behavioral observation. In HaynesS. N.HeibyE. M. (Eds.), Comprehensive handbook of psychological assessment, Vol. 3 Behavioral Assessment (pp. 162–180). Hoboken, NJ: John Wiley.
38.
HeymanR. E.MalikJ.SlepA. M. S. (2009). Analog research methods. In WeinerI. B.CraigheadW. W. (Eds.), The corsini encyclopedia of psychology. John Wiley.
39.
HouseE. R.HoweK. R. (1999). Values in evaluation and social research. Thousand Oaks, CA: Sage.
40.
HouseE. R.HoweK. R. (2000).Deliberative democratic evaluation. New Directions for Evaluation, 85, 3–12.
41.
KramerR. M. (1995). In dubious battle: Heightened accountability, dysphoric cognition, and self-defeating bargaining behavior. In KramerR. M.MessickD. M. (Eds.), Negotiation as a social process (pp. 95–120). Thousand Oaks, CA: Sage.
42.
KramerR. M.PommerenkeP.NewtonE. (1993). The social context of negotiation: Effects of social identity and interpersonal accountability on negotiator decision making. Journal of Conflict Resolution, 37, 633–654.
43.
MarkM. M. (2008). Building a better evidence base for evaluation theory: Beyond general calls to a framework of types of research on evaluation. In SmithN. L.BrandonP. R. (Eds.), Fundamental issues in evaluation (pp. 111–134). New York, NY: Guildford.
44.
MarkM. M.DonaldsonS. I.CampbellB. (2011). The past, present, and possible futures of social psychology and evaluation. In MarkM. M.DonaldsonS. I.CampbellB. (Eds.), Social psychology and evaluation (pp. 4–27). New York, NY: Guilford.
45.
MarkM. M.HenryG. T.JulnesG. (2000). Evaluation: An integrated framework for understanding, guiding, and improving policies and programs. San Francisco, CA: Jossey-Bass.
46.
MillerR. L. (2010). Developing standards for empirical examinations of evaluation theory. American Journal of Evaluation, 31, 390–399.
47.
NierJ. A.GaertnerS. L.DovidioJ. F.BankerB. S.WardC. M.RustM. C. (2001). Changing interracial evaluations and behavior: The effects of a common group identity. Group Processes & Intergroup Relations, 4, 299–316.
48.
PruittD. G.CarnevaleP. J. (1993). Negotiation in social conflict. Buckingham, England: Open University Press.
49.
RapoportA. (1960). Fights, games, and debates. Ann Arbor, MI: University of Michigan Press.
50.
RyanK.DeStefanoL. (2001). Dialogue as a democratizing evaluation method. Evaluation, 7, 188–203.
51.
RyanK. E.JohnsonT. D. (2000). Democratizing evaluation: Meanings and methods from practice. In RyanK. E.DeStefanoL. (Eds.), Evaluation as a democratic process: Promoting inclusion, dialogue, and deliberation (Vol. 85, pp. 39–50). San Francisco, CA: Jossey-Bass.
52.
SannaL. J.PanterA. T.CohenT. R.KennedyL. A. (2011). Planning the future and assessing the past: Temporal biases and debiasing in program evaluation. In MarkM. M.DonaldsonS. I.CampbellB. (Eds.), Social psychology and evaluation (pp. 166–186). New York, NY: Guilford.
53.
ScholtenL.Van KnippenbergD.NijstadB. A.DeDreuC. K. W. (2007). Motivated information processing and group decision-making: Effects of process accountability on information sharing and decision quality. Journal of Experimental Social Psychology, 43, 539–552.
54.
ShadishW. R. (1998). Evaluation theory is who we are. American Journal of Evaluation, 19, 1–19.
55.
ShadishW. R.CookT. D.LevitonL. C. (1991). Foundations of program evaluation: Theories of practice. Newbury Park, CA: Sage.
56.
SmithN. L. (1993). Improving evaluation theory through the empirical study of evaluation practice. Evaluation Practice, 14, 237–242.
57.
SmithN. L. (1979). Requirements for a discipline of evaluation. Studies in Educational Evaluation, 5, 5–12.
58.
SzanyiM.AzzamT.GalenM. (2013). Research on evaluation: A needs assesment. The Canadian Journal of Program Evaluation, 27, 39–64.
59.
TautS.BraunsD. (2003). Resistance to evaluation: A psychological perspective. Evaluation, 9, 247–264.
60.
TenVeldenF.BeersmaB.DeDreuC. K. W. (2010). It takes one to tango: The effects of dyads’ epistemic motivation composition in negotiation. Personality and Social Psychology Bulletin, 36, 1454–1466.
61.
TetlockP. E. (1991). An alternative metaphor in the study of judgment and choice: People as politicians. Theory and Psychology, 1, 451–475.
62.
TetlockP. E. (1992). The impact of accountability on judgment and choice: Toward a social contingency model. Advances in Experimental Social Psychology, 25, 331–376.
63.
TindaleR. S.PosavacE. J. (2011). The social psychology of stakeholder processes: Group processes and interpersonal relations. In MarkM. M.DonaldsonS. I.CampbellB. (Eds.), Social psychology and evaluation (pp. 189–207). New York, NY: Guilford.
64.
VaessenJ.LeeuwF. L. (Eds.). (2010). Mind the gap: Perspectives on policy evaluation and the social sciences. New Brunswick, NJ: Transaction Publishers.
