Restricted accessResearch articleFirst published online 2019-3
“You Get the Great Feeling That You’re Being Heard But in the End You Realize That Things Will Be Done Differently and in Others’ Favor”: An Experimental Investigation of Negative Effects of Participation in Evaluation
Practical-participatory evaluation is often considered an effective approach to enhancing evaluation use. Despite this generally positive connotation, potential negative effects of participatory evaluation have also been discussed in the literature but never empirically investigated before. Drawing upon the social–psychological procedural justice literature, we assumed that stakeholders’ perceptions about the justice of an evaluation might explain possible negative effects of participatory evaluation on evaluation use. Negative effects of participatory evaluation on instrumental and conceptual use were explored in two experiments (Study 1: N = 85; Study 2: N = 78). The results show that negative effects of participation on evaluation use are mediated by stakeholders’ justice perceptions. Implications of the present research include considering stakeholders’ socioemotional needs, negotiation, and transparency when conducting participatory evaluation.
AguinisH.BradleyK. J. (2014). Best practice recommendations for designing and implementing experimental vignette methodology studies. Organizational Research Methods, 17, 351–371. doi:10.1177/1094428114547952
2.
AjzenI. (1991). The theory of planned behavior. Organizational Behavior and Human Decision Processes, 50, 179–211. doi:10.1016/0749-5978(91)90020-T
3.
AlkinM. C.KingJ. A. (2017). Definitions of evaluation use and misuse, evaluation influence, and factors affecting use. American Journal of Evaluation, 76, 1–17. doi:10.1177/1098214017717015
4.
AlkinM. C.TautS. M. (2003). Unbundling evaluation use. Studies in Educational Evaluation, 29, 1–12. doi:10.1016/S0191-491X(03)90001-0
5.
BladerS. L.TylerT. R. (2015). Relational models of procedural justice. In CropanzanoR. S.AmbroseM. L. (Eds.), The Oxford handbook of justice in the workplace (pp. 351–370). New York, NY: Oxford University Press.
6.
BorensteinM. (2009). Effect sizes for continuous data. In CooperH. M.HedgesL. V.ValentineJ. C. (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 221–237). New York, NY: Russell Sage Foundation.
7.
BrandonP. R. (1998). Stakeholder participation for the purpose of helping ensure evaluation validity: Bridging the gap between collaborative and non-collaborative evaluations. American Journal of Evaluation, 19, 325–337. doi:10.1177/109821409801900305
8.
BrandonP. R.FukunagaL. L. (2014). The state of the empirical research literature on stakeholder involvement in program evaluation. American Journal of Evaluation, 35, 26–44. doi:10.1177/1098214013503699
9.
BrocknerJ.FishmanA. Y.RebJ.GoldmanB.SpiegelS.GardenC. (2007). Procedural fairness, outcome favorability, and judgments of an authority’s responsibility. Journal of Applied Psychology, 92, 1657–1671. doi:10.1037/0021-9010.92.6.1657
10.
CohenR. L. (1985). Procedural justice and participation. Human Relations, 38, 643–663. doi:10.1177/001872678503800703
11.
Cohen-CharashY.MuellerJ. S. (2007). Does perceived unfairness exacerbate or mitigate interpersonal counterproductive work behaviors related to envy?Journal of Applied Psychology, 92, 666–680. doi:10.1037/0021-9010.92.3.666
12.
CousinsJ. B. (1996). Consequences of researcher involvement in participatory evaluation. Studies in Educational Evaluation, 22, 3–27. doi:10.1016/0191-491x(96)00001-6
13.
CousinsJ. B.ChouinardJ. A. (2012). Participatory evaluation up close: An integration of research-based knowledge. Greenwich, England: Information Age.
14.
CousinsJ. B.DonohueJ. J.BloomG. A. (1996). Collaborative evaluation in North America: Evaluators’ self-reported opinions, practices and consequences. American Journal of Evaluation, 17, 207–226. doi:10.1177/109821409601700302
15.
CousinsJ. B.WhitmoreE. (1998). Framing participatory evaluation. New Directions for Evaluation, 80, 5–23. doi:10.1002/ev.1114
16.
CropanzanoR. S.RuppD. E.ThorntonM. A.ShaoR. (2016). Organizational justice and organizational citizenship behavior. In PodsakoffP. M.MackenzieS. B.PodsakoffN. P. (Eds.), The Oxford handbook of organizational citizenship behavior. Oxford, England: Oxford University Press.
17.
CullenA. E.CorynC. L. S.RughJ. (2011). The politics and consequences of including stakeholders in international development evaluation. American Journal of Evaluation, 32, 345–361. doi:10.1177/1098214010396076
18.
DaigneaultP. M. (2014). Taking stock of four decades of quantitative research on stakeholder participation and evaluation use: A systematic map. Evaluation and Program Planning, 45, 171–181. doi:10.1016/j.evalprogplan.2014.04.003
19.
de CremerD.TylerT. R. (2005). Am I respected or not? Inclusion and reputation as issues in group membership. Social Justice Research, 18, 121–153. doi:10.1007/s11211-005-7366-3
20.
DeutschM. (1949). A theory of cooperation and competition. Human Relations, 2, 129–152. doi:10.1177/001872674900200204
21.
FishbeinM.TriandisH. C.KanferF. H.BeckerM.MiddlestadtS. E.EichlerA. (2001). Factors influencing behavior and behavior change. In BaumA. S.RevensonT. A.SingerJ. E. (Eds.), Handbook of health psychology (pp. 3–17). Mahwah, NJ: Lawrence Erlbaum.
22.
FleischerD. N.ChristieC. A. (2009). Evaluation use: Results from a survey of U.S. American evaluation association members. American Journal of Evaluation, 30, 158–175. doi:10.1177/1098214008331009
23.
GasticB.IrbyD. J.ZdanisM. (2008). When stakeholders rebel: Lessons from a safe schools program. Studies in Educational Evaluation, 34, 208–211. doi:10.1016/j.stueduc.2008.10.004
24.
GoodwinC.RossI. (1992). Consumer responses to service failures: Influence of procedural and interactional fairness perceptions. Journal of Business Research, 25, 149–163. doi:10.1016/0148-2963(92)90014-3
25.
GreenbergJ.FolgerR. G. (1983). Procedural justice, participation, and the fair process effect in groups and organizations. In PaulusP. B. (Ed.), Springer series in social psychology: Basic group processes (pp. 235–256). New York, NY: Springer.
26.
GreeneJ. C. (1987). Stakeholder participation in evaluation design: Is it worth the effort?Evaluation and Program Planning, 10, 379–394. doi:10.1016/0149-7189(87)90010-3
27.
GreeneJ. C. (1988). Communication of results and utilization on participatory program evaluation. Evaluation and Program Planning, 11, 341–351. doi:10.1016/0149-7189(88)90047-x
28.
GregoryA. (2000). Problematizing participation: A critical review of approaches to participation in evaluation theory. Evaluation, 6, 179–199. doi:10.1177/13563890022209208
HayesA. F. (2013). Introduction to mediation, moderation, and conditional process analysis: A regression-based approach. New York, NY: Guilford Press.
31.
HayesA. F.PreacherK. J. (2014). Statistical mediation analysis with a multicategorical independent variable. The British Journal of Mathematical and Statistical Psychology, 67, 451–470. doi:10.1111/bmsp.12028
32.
HenryG. T.MarkM. M. (2003). Beyond Use: Understanding evaluation’s influence on attitudes and actions. American Journal of Evaluation, 24, 293–314. doi:10.1177/109821400302400302
33.
JacobS.OuvrardL.BélangerJ. F. (2011). Participatory evaluation and process use within a social aid organization for at-risk families and youth. Evaluation and Program Planning, 34, 113–123. doi:10.1016/j.evalprogplan.2010.08.002
34.
JohnsonD. W.GreenseidL. O.ToalS. A.KingJ. A.LawrenzF.VolkovB. (2009). Research on evaluation use. A review of the empirical literature from 1986 to 2005. American Journal of Evaluation, 30, 377–410. doi:10.1177/1098214009341660
35.
JohnsonD. W.JohnsonR. T. (1989). Cooperation and competition: Theory and research. Edina, MN: Interaction Book.
36.
JohnsonD. W.JohnsonR. T.StevahnL. (2011). Social interdependence and program evaluation. In MarkM. M.DonaldsonS. I.CampbellB. (Eds.), Social psychology and evaluation (pp. 288–317). New York, NY: Guilford Press.
37.
KingJ. A.StevahnL. (2013). Interactive evaluation practice: Mastering the interpersonal dynamics of program evaluation. Los Angeles, CA: Sage.
38.
KirkhartK. E. (2000). Reconceptualizing evaluation use: An integrated theory of influence. New Directions for Evaluation, 2000, 5–23. doi:10.1002/ev.1188
39.
LakensD. (2013). Calculating and reporting effect sizes to facilitate cumulative science: A practical primer for t-tests and ANOVAs. Frontiers in Psychology, 4, 1–12. doi:10.3389/fpsyg.2013.00863
40.
LennieJ. (2005). An evaluation capacity building process for sustainable community IT initiatives. Empowering and disempowering impacts. Evaluation, 11, 390–414. doi:10.1177/1356389005059382
41.
LevitonL. C.HughesE. F. (1981). Research on the utilization of evaluations: A review and synthesis. Evaluation Review, 5, 525–548. doi:10.1177/0193841X8100500405
42.
LindE. A.KanferR.EarleyP. C. (1990). Voice, control, and procedural justice: Instrumental and noninstrumental concerns in fairness judgments. Journal of Personality and Social Psychology, 59, 952–959. doi:10.1037/0022-3514.59.5.952
43.
LindE. A.TylerT. R. (1988). The social psychology of procedural justice. New York, NY: Plenum Press.
44.
MathieA.GreeneJ. C. (1997). Stakeholder participation in evaluation: How important is diversity?Evaluation and Program Planning, 20, 279–285. doi:10.1016/S0149-7189(97)00006-2
45.
MarkM. M.HenryG. T. (2004). The mechanisms and outcomes of evaluation influence. Evaluation, 10, 35–57. doi:10.1177/1356389004042326
46.
MarkM. M.ShotlandR. L. (1985). Stakeholder-based evaluation and value judgments. Evaluation Review, 9, 605–626. doi:10.1177/0193841X8500900504
47.
McFarlinD. B.SweeneyP. D. (1992). Distributive and procedural justice as predictors of satisfaction with personal and organizational outcomes. Academy of Management Journal, 35, 626–637. doi:10.2307/256489
48.
MertensD. M. (2009). Transformative research and evaluation. New York, NY: Guilford Press.
49.
MoormanR. H.ByrneZ. S. (2005). How does justice affect organizational citizenship behavior? In GreenbergJ.ColquittJ. A. (Eds.), Handbook of organizational justice (pp. 355–380). Mahwah, NJ: Lawrence Erlbaum Associates.
PothC. A.ShulhaL. (2008). Encouraging stakeholder engagement: A case study of evaluator behavior. Studies in Educational Evaluation, 34, 218–223. doi:10.1016/j.stueduc.2008.10.006
53.
PotterP. W. (2006a). Procedural justice and voice effects. Journal of Organizational Culture, Communications and Conflict, 10, 33–62.
54.
PotterP. W. (2006b). Procedural justice, voice effects and sham: Examining the decision maker from a research context perspective. Journal of Organizational Culture, Communications and Conflict, 10, 61–76.
55.
PreskillH.CaracelliV. (1997). Current and developing conceptions of use: Evaluation use TIG survey results. American Journal of Evaluation, 18, 209–225. doi:10.1016/s0886-1633(97)90028-3
56.
RebienC. C. (1996). Participatory evaluation of development assistance: Dealing with power and facilitative learning. Evaluation, 2, 151–171. doi:10.1177/135638909600200203
57.
RebollosoE.Fernández-RamírezB.CantónP. (2005). The influence of evaluation on changing management systems in educational institutions. Evaluation, 11, 463–479. doi:10.1177/1356389005060263
58.
Rodríguez-CamposL. (2005). Collaborative evaluations: A step-by-step model for the evaluator. Tamarac, FL: Llumina Press.
59.
RoselandD.LawrenzF.ThaoM. (2015). The relationship between involvement in and use of evaluation in multi-site evaluations. Evaluation and Program Planning, 48, 75–82. doi:10.1016/j.evalprogplan.2014.10.003
60.
RuckerD. D.PreacherK. J.TormalaZ. L.PettyR. E. (2011). Mediation analysis in social psychology: Current practices and new recommendations. Social and Personality Psychology Compass, 5, 359–371. doi:10.1111/j.1751-9004.2011.00355.x
61.
ShadishW. R.CookT. D.CampbellD. T. (2002). Experimental and quasi experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin.
62.
ShadishW. R.CookT. D.LevitonL. C. (1991). Foundations of program evaluation: Theories of practice. Newbury Park, CA: Sage.
63.
ShulhaL.WhitmoreE.CousinsJ. B.GilbertN.Al HudibH. (2015). Introducing evidence-based principles to guide collaborative approaches to evaluation: Results of an empirical process. American Journal of Evaluation, 37, 193–217. doi:10.1177/1098214015615230
64.
SpeckK. (2006). Qualität und Evaluation in der Schulsozialarbeit: Konzepte, Rahmenbedingungen und Wirkungen [Quality and evaluation in school social work: Concepts, general conditions and impacts]. Wiesbaden, Germany: VS Verlag für Sozialwissenschaften.
65.
SullinsC. D. (2003). Adapting the empowerment evaluation model: A mental health drop-in centre case example. American Journal of Evaluation, 24, 387–398. doi:10.1177/109821400302400307
66.
TautS. (2007). Methodological and conceptional challenges in studying evaluation process use. Canadian Journal of Program Evaluation, 22, 1–19.
67.
TautS. (2008). What have we learned about stakeholder involvement in program evaluation?Studies in Educational Evaluation, 34, 224–230. doi:10.1016/j.stueduc.2008.10.007
68.
TautS.BraunsD. (2003). Resistance to evaluation: A psychological perspective. Evaluation, 9, 247–264. doi:10.1177/13563890030093002
69.
TurnbullB. (1999). The mediating effect of participation efficacy on evaluation use. Evaluation and Program Planning, 22, 131–140. doi:10.1016/s0149-7189(99)00012-9
70.
TylerT. R.BladerS. L. (2003). The group engagement model: Procedural justice, social identity, and cooperative behavior. Personality and Social Psychology Review, 7, 349–361. doi:10.1207/S15327957PSPR0704_07
71.
van den BosK.LindE. A.VermuntR.WilkeH. A. M. (1997). How do I judge my outcome when I do not know the outcome of others? The psychology of the fair process effect. Journal of Personality and Social Psychology, 72, 1034–1046. doi:10.1037/0022-3514.72.5.1034
72.
van den BosK.WilkeH. A. M.VermuntR.LindE. A. (1998). Evaluating outcomes by means of the fair process effect: Evidence for different processes in fairness and satisfaction judgements. Journal of Personality and Social Psychology, 74, 1493–1503. doi:10.1037//0022-3514.74.6.1493
73.
WeissC. H. (1986). The stakeholder approach to evaluation: Origins and promise. In HouseE. R. (Ed.), New directions for educational evaluation (pp. 145–157). Abingdon, England: Routledge Falmer.
74.
WhitmoreE. (1998). “We need to rebuild this house”: The role of empowerment in evaluation of a Mexican Farmers’ cooperative. In JacksonE. T.KassamY. (Eds.), Knowledge shared: Participatory evaluation in development cooperation (pp. 217–230). West Hartford, CT: Kumarian Press.
75.
ZhaoX.LynchJ. G.JrChenQ. (2010). Reconsidering Baron and Kenny: Myths and truths about mediation analysis. Journal of Consumer Research, 37, 197–206. doi:10.1086/651257
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
