A budding area of research is devoted to studying evaluator curriculum, yet to date, it has focused exclusively on describing the content and emphasis of topics or competencies in university-based programs. This study aims to expand the foci of research efforts and investigates the extent to which evaluators agree on what competencies should guide the development and implementation of evaluator education. This study used the Delphi method with evaluators (n = 11) and included three rounds of online surveys and follow-up interviews between rounds. This article discusses on which competencies evaluators were able to reach consensus. Where consensus was not found, possible reasons are offered. Where consensus was found, the necessity of each competency at both the master’s and doctoral levels is described. Findings are situated in ongoing debates about what is unique about what novice evaluators need to know and be able to do and the purpose of evaluator education.
AltschuldJ. W. (1999). The certification of evaluators: Highlights from a report submitted to the Board of Directors of the American Evaluation Association. American Journal of Evaluation, 20, 481–493. http://doi.org/10.1177/109821409902000307
AltschuldJ. W.EngleM. (Eds.). (1994). The preparation of professional evaluators: Issues, perspectives, and programs. New Directions for Program Evaluation, 62. https://doi.org/10.1002/ev1671
4.
AltschuldJ. W.EngleM. (2015). Accreditation, certification, and credentialing: Relevant concerns of U.S. evaluators. New Directions for Evaluation, 145, 1–113. https://doi.org/10.1002/ev20107
AxelrodP. (2002). Values in conflict: The university, the marketplace, and the trials of liberal education. McGill-Queen’s University Press.
7.
BarringtonG. V.FrankC.GauthierB.HicksK. (2015). View from the credentialing board: Where we’ve been and where we’re going. Canadian Journal of Program Evaluation, 29(3), 86–97. http://doi.org/10.3138/cjpe.29.3.86
8.
BoyceA. S.McGowanB. L. (2019). An exploration of two novice evaluation educators’ experiences developing and implementing introduction to evaluation courses. American Journal of Evaluation, 40(1), 119–136. https://doi.org/10.1177/1098214018778812
9.
BrownE. D. (1980). Effective training of program evaluators: A mixture of art and science. New Directions in Program Evaluation, 8, 79–87. https://doi.org/10.1002/ev1263
10.
BurkeM. J.DunlapW. P. (2002). Estimating interrater agreement with the average deviation index: A user’s guide. Organizational Research Methods, 5, 159–172. https://doi.org/10.1177%2F1094428102005002002
11.
BurkeM. J.FinkelsteinL. M.DusigM. S. (1999). On average deviation indices for estimating interrater agreement. Organizational Research Methods, 2, 49–68. https://doi.org/10.1177%2F109442819921004
ChristieC. A.QuiñonesP.FierroL. (2014). Informing the discussion on evaluator training: A look at evaluators’ course taking and professional practice. American Journal of Evaluation, 35(2), 274–290. https://doi.org/10.1177/1098214013503697
16.
ConradC.HaworthJ. G.MillarS. B. (1993). Silent success: Master’s education in the United States. Johns Hopkins University Press.
17.
CreswellJ. W. (2008). Educational research: Planning, conducting, and evaluating quantitative and qualitative research (3rd ed.). Pearson Education.
18.
CreswellJ. W. (2013). Qualitative inquiry and research design: Choosing among five approaches (3rd ed.). SAGE.
DaviesR.MacKayK. (2014). Evaluator training: Content and topic valuation in university evaluation courses. American Journal of Evaluation, 35(3), 419–429. https://doi.org/10.1177/1098214013520066
21.
DeweyJ. (1902). The child and the curriculum. University of Chicago Press.
22.
DeweyJ. D.MontrosseB. E.SchröterD. C.SullinsC. D.JohnR. (2008). Evaluator competencies: What’s taught versus what’s sought. American Journal of Evaluation, 29(3), 268–287. https://doi.org/10.1177/1098214008321152
23.
DonohoeH.StellefsonM.TennantB. (2012). Advantages and limitations of the e-Delphi technique: Implications for health education researchers. American Journal of Health Education, 43(1), 38–46. https://doi.org/10.1080/19325037.2012.10599216
24.
EllisA. K. (2004). Exemplars of curriculum theory. Eye on Education.
FierroL. A.ChristieC. A. (2011). Understanding evaluation training in schools and programs of public health. American Journal of Evaluation, 32(3), 448–468. https://doi.org/10.1177/1098214010393721
27.
FrankJ. R.SnellL. S.CateO. T.HolmboeE. S.CarraccioC.SwingS. R.HarrisP.GlasgowN. J.CampbellC.DathD.HardenR. M.IobstW.LongD. M.MungrooR.RichardsonD. L.SherbinoJ.SilverI.TaberS.TalbotM.HarrisK. A. (2010). Competency-based medical education: Theory to practice. Medical Teacher, 32(8), 638–645. https://doi.org/10.3109/0142159X.2010.501190
28.
GalportN.AzzamT. (2016). Evaluator training needs and competencies: A gap analysis. American Journal of Evaluation, 38(1), 80–100. https://doi.org/10.1177/1098214016643183
29.
GaravaliaL.GredlerM. (2004). Teaching evaluation through modelling: Using the Delphi technique to assess problems in academic programs. Evaluation, 25(3), 375–380. https://doi.org/10.1177/109821400402500307.
30.
Gross DavisB. G. (Ed.). (1986). Teaching of evaluation across the disciplines. New Directions for Program Evaluation, 29, 1–98. https://doi.org/10.1002/ev.1414
GullicksonA. M.KingJ.LaVelleJ. M.ClintonJ. (2019). The current state of educating evaluators: A situation analysis. Evaluation and Program Planning, 75, 20–30. https://doi.org/10.1016/j.evalprogplan.2019.02.012
33.
HarlandT. (2009). The university, neoliberal reform and the liberal educational ideal. In TightM.HuismanJ.HoM. K.MorphewC. C. (Eds.), The Routledge international handbook of higher education (pp. 511–521). Routledge.
34.
HelmerO. (1967). Analysis of the future: The Delphi method. The Rand Corporation.
35.
HouseE. R. (2015). Evaluating: Values, biases, and practical wisdom. Information Age Publishing.
36.
International Development Evaluation Association. (2012). IDEAS competencies for development evaluation evaluators, managers, and commissioners. https://ideas-global.org/policies/
37.
JamesL. R.DemareeR. G.WolfG. (1984). Estimating within-group interrater reliability with and without response bias. Journal of Applied Psychology, 69, 85–98. http://doi.org/10.1037/0021-9010.69.1.85
38.
KingJ. A.AyooS. (2020). What do we know about evaluator education? A review of peer-reviewed publications (1978–2018). Evaluation and Program Planning, 79, 101785. https://doi.org/10.1016/j.evalprogplan.2020.101785
LaVelleJ. M. (2020). Educating evaluators 1976–2017: An expanded analysis of university-based evaluation education programs. American Journal of Evaluation, 41(4), 494–509. http://doi.org/10.1177/1098214019860914
41.
LaVelleJ. M.DonaldsonS. I. (2015). The state of preparing evaluators. New Directions for Evaluation, 145, 39–52. https://doi.org/10.1002/ev
42.
LaVelleJ. M.SabarreN.UmansH. (2020). An empirical examination of evaluation’s presence in the undergraduate curriculum in the United States. American Journal of Evaluation, 41(2), 297–310. https://doi.org/10.1177/1098214019860912
43.
LeBretonJ. M.SenterJ. L. (2008). Answers to 20 questions about interrater reliability and interrater agreement. Organizational Research Methods, 11, 815–852. https://doi.org/10.1177%2F1094428106296642
44.
LeeJ.WallaceT. L.AlkinM. C. (2007). Using problem-based learning to train evaluators. American Journal of Evaluation, 28(4), 536–545. https://doi.org/10.1177/1098214007305613
LevitonL. C. (2014). Some underexamined aspects of evaluation capacity building. American Journal of Evaluation, 35(1), 90–94. https://doi.org/10.1177/1098214013502844
47.
MathisonS. (2008). What is the difference between evaluation and research and why do we care. In SmithN. L.BrandonP. R. (Eds.), Fundamental issues in evaluation (pp. 183–196). Guilford Publications.
48.
MenefeeD. T.ThompsonJ. J. (1994). Identifying and comparing competencies for social work management: A practice driven approach. Administration in Social Work, 18(3), 1–25. https://doi.org/10.1300/J147v18n03_01
49.
MertensD. M. (2014). Research and evaluation in education and psychology: Integrating diversity with quantitative, qualitative, and mixed methods (4th ed.). SAGE.
50.
McDavidJ. C.HuseI. (2015). How does accreditation fit into the picture?New Directions for Evaluation, 145, 53–69. https://doi.org/10.1002/ev20111
MilesM. B.HubermanA. M.SaldañaJ. (2020). Qualitative data analysis: A methods sourcebook (4th ed.). SAGE.
53.
MorellJ. A.FlahertyE. W. (1978). The development of evaluation as a profession: Current status and some predictions. Evaluation and Program Planning, 1(1), 11–17. http://doi.org/10.1016/0149-7189(78)90003-4
54.
NaidooI. A. (2013). Growth and integration in the evaluation profession: Some perspectives for consideration. American Journal of Evaluation, 34(4), 572–576. https://doi.org/10.1177/1098214013493495
55.
National Academy of Sciences. (1999). Designing mathematics or science curriculum programs: A guide for using mathematics and science education standards. National Academy Press.
56.
OkoliC.PawlowskiS. D. (2004). The Delphi method as a research tool: An example, design considerations and application. Information & Management, 42, 15–29. http://doi.org/10.1016/j.im.2003.11.002
PreskillH. (1992). Students, client, and teacher: Observations from a practicum in evaluation. American Journal of Evaluation, 13(1), 39–46. https://doi.org/10.1177/109821409201300109
61.
RallisS. F. (2014). When and how qualitative methods provide credible and actionable evidence: Reasoning with rigor, probity, and transparency. In ChristieC. A.MarkM. M.DonaldsonS. I. (Eds.), Credible and actionable evidence: The foundation for rigorous and influential evaluations (2nd ed., pp. 137–156). SAGE.
62.
SchwandtT. A. (2015). Evaluation foundations revisited: Cultivating a life of the mind for practice. Stanford University Press.
63.
SchwandtT. A. (2017). Professionalization, ethics, and fidelity to an evaluation ethos. American Journal of Evaluation, 38(4), 546–553. https://doi.org/10.1177/1098214017728578
SmithM. F. (1999). Should AEA begin a process for restricting membership in the profession of evaluation?American Journal of Evaluation, 20, 521–531. http://doi.org/10.1177/109821409902000311
67.
SpadyW. G. (1994). Outcome-based education: Critical issues and answers. American Association of School Administrators.
68.
TylerR. W. (1957). The curriculum then and now[Conference session]. Proceedings of the 1956 Invitational Conference on Testing Problems. Princeton, NJ: Educational Testing Service.
69.
UlschakF. L. (1983). Human resource development: The theory and practice of needs assessment. Reston Publishing.
70.
VedungE. (2004). Evaluation research and fundamental research. In StockmannR. (Ed.), Evaluationsforschung: Grundlagen und ausgewählte Forschungsfelder [Evaluation research: Basics and selected fields of research] (pp. 111–134). Leske + Budrich.
71.
von der GrachtH. A. (2012). Consensus measurement in Delphi studies: Review and implications for future quality assurance. Technological Forecasting & Social Change, 79, 1525–1536. https://doi.org/10.1016/j.techfore.2012.04.013
72.
WanzerD. L. (2021). What is evaluation? Perspectives of how evaluation differs (or not) from research. American Journal of Evaluation, 42(1), 28–46. https://doi.org/10.1177/1098214020920710
WorthenB. R. (1999). Critical challenges confronting certification of evaluators. American Journal of Evaluation, 20, 533–555. http://doi.org/10.1177/109821409902000312
