In order for novice evaluators to gauge the success of their evaluations, the ability to identify and recognize indicators of buy-in is an essential skill. While much is written in the evaluation literature about strategies to build buy-in, the indicators of buy-in are largely overlooked. This article shares strategies used for building buy-in from a multi-state evaluation of cardiac care. Indicators of the success of these strategies are then shared with an emphasis on simple, unobtrusive, robust measures that novice evaluators can use to assess stakeholder engagement and responsiveness. Four indicators of buy-in include: timeliness to evaluation team requests; quality and quantity of feedback received; interaction with decision-makers; and investment of in-kind contributions. Awareness of these indicators can boost confidence and authority for students who are new to the field of evaluation. A tracking tool is also showcased which serves as a systematic, responsive prompt to monitor buy-in. The value of the tracking tool is realized in its ability to alert evaluators to potential problems for which corrective actions can be made, assist evaluators in creating realistic expectations, and enhance confidence in evaluation proceedings.
Get full access to this article
View all access options for this article.
References
1.
AlkinM.C.1985, A guide for evaluation decision makers, Sage, Beverly Hills, California.
2.
AzzamT.2010, ‘Evaluator responsiveness to stakeholders’, American Journal of Evaluation, vol. 31, no. 1, pp. 45–65.
3.
BecharS., & Mero-JaffeI.2013, ‘Who is afraid of evaluation? Ethics in evaluation research as way to cope with excessive evaluation anxiety’, American Journal of Evaluation, vol. 1, no. 13, DOI: 10.1177/1098214013512555
4.
Bohanon, H & Wu, M (in press), Developing buy-in for positive behavior support in secondary settings. Preventing school failure, Loyola University Chicago: Loyala eCommons.
5.
BrandonP.R., SmithN.L., TrenholmC., & DevaneyB.2010, ‘Evaluation exemplar: the critical importance of stakeholder relations in a national, experimental abstinence education evaluation’, American Journal of Evaluation, vol. 31, no. 4, pp. 517–531.
6.
BrownJ.2010, ‘Compliance effectiveness: how do we get there?’, Journal of Health Care Compliance, vol. 12, no. 4, pp. 37–63.
7.
BrykA. (ed) 1983, ‘Stakeholder-based evaluation’, New Directions for Evaluation, no. 17, San Francisco, Jossey-Bass, San Francisco.
8.
BoulmetisJ., & DutwinP.2005, The abcs of evaluation: timeless techniques for program and project managers, Jossey-Bass, San Francisco.
9.
BrandonP.R.2014, ‘Participatory evaluation up close: An integration of research-based knowledge’, by JB Cousins & JA Chouinard, reviewed in: American Journal of Evaluation, vol. 35, no. 2, pp. 291–297.
10.
BuchanD.2012, ‘Buy-in and social capital: by-products of social impact assessment’, Impact Assessment and Project Appraisal, vol. 21, no. 3, pp. 168–172.
11.
CartlandJ., Ruch-RossH.S., MasonM., & DonahueW.2008, ‘Role sharing between evaluators and stakeholders in practice’, American Journal of Evaluation, vol. 29, no. 4, pp. 460–477.
12.
CookC., HeathF., & ThompsonR.L.2000, ‘A meta-analysis of responses rates in web-or internet-based surveys’, Educational and Psychological Measurement, vol. 60, no. 6, pp. 821–836.
13.
CousinsJ.B., & ChouinardJ.A.2012, Participatory evaluation up close: an integration of research-based knowledge, Information Age Publishing, Charlotte, North Carolina.
14.
DattaL.2011, ‘Politics and evaluation: more than methodology’, American Journal of Evaluation, vol. 32, no. 2, pp. 273–294, DOI: 10.1177/1098214011400060.
15.
FineA.H., ThayerC.E., & CoghlanA.2000, ‘Program evaluation practice in the nonprofit sector’, Nonprofit Management and Leadership, vol. 10, pp. 331–339.
16.
FinkA.2003, How to sample in surveys, 2nd edition, Sage, Thousand Oaks, California.
17.
GreeneJ.1988, ‘Stakeholder participation and utilization in program evaluation’, Evaluation Review, vol. 12, pp. 91–116.
18.
HarrellA., BurtM., HatryH., RossmanS., RothJ., & SabolW. (n.d.), Evaluation strategies for human services programs: A guide for policymakers and providers, The Urban Institute, Washington.
19.
HoeferR.2000, ‘Accountability in action? Program evaluation in nonprofit human service agencies’, Nonprofit Management and Leadership, vol. 11, no. 2, pp. 167–177.
20.
HowellE.M., & YemaneA.2006, ‘An assessment of evaluation designs: case studies of 12 large federal evaluations’, American Journal of Evaluation, vol. 27, no. 2, pp. 219–236.
21.
HuebnerT.A.2000, ‘Theory-based evaluation: gaining a shared understanding between school staff and evaluators’, New Directions for Evaluation, vol. 2000, issue 87, pp. 79–89. DOI: 10.1002/ev.1184
22.
InnesJ.E., & BooherD.E.1999, ‘Consensus building and complex adaptive systems’, Journal of the American Planning Association, vol. 65, no. 4, pp. 412–423.
23.
KazdinA.E.1979, ‘Unobtrusive measures in behavioral assessment’.Journal of Applied Behavior Analysis, vol. 12, no. 4, pp. 713–724.
24.
KotterJ.P., & WhiteheadL.A.2010, Buy-in: saving your good idea from getting shot down, Harvard Business Review Press, Boston, Massachusetts.
25.
LiketK.C., Rey-GarciaM., & MassK.E.H.2014, ‘Why aren't evaluations working and what to do about it: a framework for negotiating meaningful evaluations in nonprofits’, American Journal of Evaluation, vol. 35, no. 2, pp. 171–188.
26.
LincolnY.S., & GubaE.G.1985, Naturalistic inquiry, Sage, Newbury Park, California.
27.
LorenziN.M., KourobualiA., DetmerD.E., & BloomrosenM.2009, ‘How to successfully select and implement electronic health records (EHR) in small ambulatory practice settings’, BMC Medical Informatics and Decision Making, vol. 9, no. 15, pp. 1–13. DOI: 10.1186/1472-6947-9-15
28.
MadisonD.2005, Process mapping, process improvement, and process management: a practical guide to enhancing work and information flow, Paton Press LLC Chico, California.
29.
MaritzJ., & JoosteK.2011, ‘Debriefing interviews and coaching conversations: strategies to promote student reflexivity and action’, South African Journal of Higher Education, vol. 25, no. 5, pp. 972–986.
30.
MedererH., & SilverB.2010, Buy-in: targeting three levels of barriers to change, paper presented at the 16th National Conference of the College & University Work/Family Association, Cambridge, Massachusetts.
31.
MillerD., & MerrileesB.2013), ‘Rebuilding community corporate brands: a total stakeholder involvement approach’, Journal of Business Research, vol. 66, no. 2, pp. 172–179.
32.
NimonK., ZigarmiD., & AllenJ.2011, ‘Measures of program effectiveness based on retrospective pretest data: are all created equal?’, American Journal of Evaluation, vol. 32, no. 1, pp. 8–28.
33.
PattonM.Q.2010, Developmental evaluation applying complexity concepts to enhance innovation and use, Guilford Press, New York.
PattonM.Q.2001, ‘Evaluation, knowledge management, best practices, and high quality lessons learned’, American Journal of Evaluation, vol. 22, no. 3, pp. 329–336.
36.
PreskillH., & JonesN.2009, ‘A practical guide for engaging stakeholders in developing evaluation questions’ [webinar], Robert Wood Johnson Foundation, http://www.rwjf.org/pr/product.jsp?id=49951.
37.
RengerR.2011, ‘Constructing and verifying program theory using source documentation’, The Canadian Journal of Program Evaluation, vol. 25, no. 1, pp. 51–67.
38.
RengerR., WoodS., WilliamsonS., & KrappS.2011, ‘Systemic evaluation, impact evaluation and logic models’, Evaluation Journal of Australasia, vol. 11, no. 2, pp. 24–30.
39.
RengerR., & BourdeauB.2004, ‘Strategies for values inquiry: an exploratory case study’, American Journal of Evaluation, vol. 25, no. 1, pp. 39–49.
40.
RidziF.2004, ‘Making TANF work: organizational restricting, staff buy-in, and performance monitoring in local implementation’, Journal of Sociology & Social Welfare, vol. 31, no. 2, pp. 27–48.
41.
RogersS.J., AhmedM., HamdallahM., & LittleS.2010, ‘Garnering grantee buy-in on a national cross-site evaluation: the case of ConnectHIV’, American Journal of Evaluation, vol. 31, no. 4, pp. 447–462.
42.
RosasS.R.2006, ‘Nonparticipant to participant: a methodological perspective on evaluator ethics’, American Journal of Evaluation, vol. 27, no. 1, pp. 98–103.
43.
SandersJ.R.1994, The program evaluation standards: how to assess evaluations of educational programs, 2nd edition, Sage, Thousand Oaks, California.
44.
ScrivenM.1993, ‘Hard-won lessons in program evaluation’, New Directions for Program Evaluation, no. 58, Jossey-Bass, San Francisco.
45.
ScrivenM.1991, Evaluation thesaurus, 4th edition, Sage, Newbury Park, California.
46.
ShtivelbandA., & RosecranceJ.2010, ‘Gaining organizational buy in: lessons learned from fifty ergonomists’, Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 54, no. 17, pp. 1277–1281. DOI: 10.1177/154193121005401703
47.
SenGuptaS., HopsonR., & Thompson-RobisonM.2004, ‘Cultural competence in evaluation: an overview’, New Directions for Evaluation, vol. 2004, no. 102, pp. 5–19.
48.
TorresR.T., & PreskillH.2001, ‘Evaluation and organizational learning: past, present and future, American Journal of Evaluation, vol. 22, no. 3, pp. 387–395.
49.
WebbE.J., CampbellD.T., SchwartzR.D., & SechrestL.1966, Unobtrusive measures: nonreactive research in the social sciences, Rand McNally & Company, Chicago, Illinois.
50.
WeissC.H.1978, ‘Improving the linkage between social research and public policy’, in LynnLE (ed), Knowledge and policy: the uncertain connection, National Academy of Sciences, Washington, pp. 23–81.
51.
WilliamsB., & HummelbrunnerR.2009, Systems concepts in action: a practitioner's toolkit, Stanford University Press, Stanford, California.
52.
YackaniczL., KerrR., & LevickD.2010, ‘Physician buy-in for EMRs: what does it really mean?’, Journal of Healthcare Information Management, vol. 24, no. 2, pp. 41–44.
53.
YarbroughD.B., ShulaL.M., HopsonR.K., & CaruthersF.A.2011, The program evaluation standards: a guide for evaluators and evaluation users, 3rd edition, Sage, Thousand Oaks, California.
