Purpose: The past decade has seen considerable debate about how to best evaluate the efficacy of educational improvement initiatives, and members of the educational leadership research community have entered the debate with great energy. Throughout this debate, the use of randomized experiments has been a particularly contentious subject. This study examines the potential benefits, limitations, and challenges involved in using experiments to evaluate professional development for principals. Approach: We present a case study of an experimental evaluation of a professional development program for principals. The case study is grounded in key themes in recent debates about the use of experiments in educational research, scholarship on challenges in conducting experiments, and experimental studies involving principals. Setting and Sample: The case study was conducted in an urban school district with 48 principals. Implications for Research: The experimental component of the study allowed us to form a trustworthy summary inference about whether or not a professional development program had an overall effect on principals. However, the experiment did not illuminate why or how the program failed to influence principal practice. Using descriptions of the intended curriculum for principals, professional development attendance records, and interview data, we developed an understanding of why the program failed to achieve its intended goals. Based on our experiences, we support continued advocacy of research designs that bring rich evidence to bear about causal mechanisms, implementation conditions, potential measures of delivery of and adherence to treatment protocols, and measures of participants’ exposure to treatment.
The American Association of Colleges for Teacher Education. (2001, March). PK-12 educational leadership and administration (White paper). Washington, DC: Author.
2.
AngristJ. D.ImbensG. W.RubinD. B. (1996). Identification of causal effects using instrumental variables. Journal of the American Statistical Association, 91, 444-455.
3.
ArmitageP. (1983). Statistical methods in medical research. Oxford, England: Blackwell Scientific.
4.
AugustineC. H.GonzalezG.Schuyler IkemotoG.RussellJ.ZellmanG. L.ConstantL.. . . DemboskyJ. W. (2009). Improving school leadership: The promise of cohesive leadership systems. Santa Monica, CA: RAND Corporation.
5.
BarnesC. A.CamburnE. M.SandersB.SebastianJ. (2010). Developing instructional leaders: Using mixed methods to explore the black box of planned change in principals’ professional practice. Educational Administration Quarterly, 46, 241-279.
6.
BermanP. (1981). Educational change: An implementation paradigm. In LehmingR.KaneM. (Eds.), Improving schools: Using what we know (pp. 253-286). London, England: Sage.
7.
BloomH. S. (2005). Learning more from social experiments: Evolving analytical approaches. New York, NY: Russell Sage Foundation.
8.
BloomH. S.BosJ. M.LeeS.-W. (1999). Using cluster random assignment to measure program impacts: Statistical implications for the evaluation of education programs. Evaluation Review, 23, 445-469.
9.
BorkoH. (2004). Professional development and teacher learning: Mapping the terrain. Educational Researcher, 33(8), 3-15.
10.
BormanG. D.SlavinR. E.CheungA. (2005). Success for all: First-year results from the national randomized field trial. Educational Evaluation and Policy Analysis, 27(1), 1-11.
11.
BoruchR. (1997). Randomized experiments for planning and evaluation: A practical guide. Thousand Oaks, CA: Sage.
12.
BoruchR. (2002). The virtues of randomness. Education Next, 2(3), 36-42.
13.
CamburnE. M.SpillaneJ.SebastianJ. (2010). Assessing the utility of a daily log for evaluations involving school principals. Educational Administration Quarterly, 46, 707-737.
14.
CockburnJ.GibberdR. W.ReidA. L.Sanson-FisherR. W. (1987). Determinants of non-compliance with short term antibiotic regimens. British Medical Journal, 295, 814-818.
15.
CohenD. K. (1990). A revolution in one classroom: The case of Mrs. Oublier. Educational Evaluation and Policy Analysis, 12, 327-345.
16.
CohenD. K. (2011). Teaching and its predicaments. Cambridge, MA: Harvard University Press.
17.
Condliffe LangemanE. (2000). An elusive science: The troubling history of education research. Chicago, IL: University of Chicago Press.
18.
CookT. (2002). Randomized experiments in educational policy research: A critical examination of the reasons the educational evaluation community has offered for not doing them. Educational Evaluation and Policy Analysis, 24, 175-199.
19.
CrowG. M. (2007). Foreword. In EnglishF. W.FurmanG. C. (Eds.), Research and educational leadership: Navigating the new National Research Council Guidelines (pp. vii-x). Blue Ridge Summit, PA: Rowman & Littlefield Education.
20.
DavisS.Darling-HammondL.LaPointeM. A.MeyersonD. (2005). School leadership study: Preparing successful principals (Review of Research). Stanford, CA: Stanford University, Stanford Educational Leadership Institute.
21.
DenzinN. K. (1989). The research act: A theoretical introduction to sociological methods (3rd ed.). Englewood Cliffs, NJ: Prentice Hall.
22.
DrakeT. L.RoeW. H. (2003). The principalship. New York, NY: Macmillan College.
23.
EisenhartM.TowneL. (2003). Contestation and change in national policy on “scientifically based” education research. Educational Researcher, 32(7), 31-38.
24.
ElmoreR. F. (2000, Winter). Building a new structure for school leadership. Washington, DC: Albert Shanker Institute.
25.
EnglishF. W. (2007). Introduction. In EnglishF. W.FurmanG. C. (Eds.), Research and educational leadership: Navigating the new National Research Council Guidelines (pp. xi-xv). Blue Ridge Summit, PA: Rowman & Littlefield Education.
26.
GerstenR. (2013). From the editor in chief: The two cultures of educational research?Elementary School Journal, 114, 139-141.
27.
GoldringE.PrestonC.HuffJ. (2013). Conceptualizing and evaluating professional development for school leaders. Planning and Change, 43(3), 1-13.
28.
GrimmettP. P.CrehanE. P. (1987, May). Changes in elementary school principals as a result of laboratory training. Paper presented at the annual meeting of the Canadian Association for Teacher Education, Hamilton, Ontario, Canada.
29.
GutierrezM.TasseT. (2007). Leading with theory: Using a theory of change approach for leadership development evaluations. In HannumK. M.MartineauJ. W.ReineltC. (Eds.), The handbook of leadership development evaluation (pp. 48-70). San Francisco, CA: Jossey-Bass.
30.
HessF. M. (1998). Spinning wheels: The politics of urban school reform. Washington, DC: Brookings Institution Press.
31.
HoweK. R. (2005). The question of education science: Experimentism versus experimentalism. Educational Theory, 55, 307-321.
32.
HullemanC. S.CordrayD. S. (2009). Moving from the lab to the field: The role of fidelity and achieved relative intervention strength. Journal of Research on Educational Effectiveness, 2, 88-110.
33.
JacobR.GoddardR.KimM.MillerR.GoddardY. (2014). Exploring the causal impact of the McREL Balanced Leadership Program on leadership, principal efficacy, instructional climate, educator turnover, and student achievement. Educational Evaluation and Policy Analysis. Advance online publication. doi:10.3102/0162373714549620
34.
LaPointeM.MeyersonD.Darling-HammondL. (2006, April). Preparing and supporting principals for effective leadership: Early findings from Stanford’s School Leadership Study. Paper presented at the 2006 annual meeting of the American Educational Research, San Francisco, CA.
35.
LevineA. (2005). Educating school leaders. New York, NY: Education School Project.
36.
MartinW. J.WillowerD. J. (1981). The managerial behavior of high school principals. Education Administration Quarterly, 17, 69-90.
37.
MaxwellJ. A. (2004). Causal explanation, qualitative research, and scientific inquiry in education. Educational Researcher, 33(2), 3-11.
38.
NelsonM. C.CordrayD. S.HullemanC. S.DarrowC. L.SommerE. C. (2012). A procedure for assessing intervention fidelity in experiments testing educational and behavioral interventions. Journal of Behavioral Health Services & Research, 39, 374-396.
39.
NewmannF. M.SmithB. A.AllensworthE.BrykA. S. (2001). Instructional program coherence: What it is and why it should guide school improvement policy. Educational Evaluation and Policy Analysis, 23, 297-321.
40.
National Research Council. (2002). Scientific research in education (ShavelsonR.TowneL. Eds.). Committee on Scientific Principles for Educational Research. Washington, DC: National Academy Press.
41.
PetersonK. D. (1977). The principal’s tasks. Administrator’s Notebook, 26(8), 1-4.
42.
PetersonK. D. (2002). The professional development of principals: Innovations and opportunities. Educational Administration Quarterly, 38, 213-232.
43.
PetrosinoA.BoruchR. F.RoundingC.McDonaldS.ChalmersI. (2000). The Campbell Collaboration Social, Psychological, Educational and Criminological Trials Register (C2-SPECTR) to facilitate the preparation and maintenance of systematic reviews of social and educational interventions. Evaluation and Research in Education, 14, 206-219.
44.
RaudenbushS. W. (1997). Statistical analysis and optimal design for cluster randomized trials. Psychological Methods, 2, 173-185.
45.
RiehlC. (2007). Research on educational leadership: Knowledge we need for the world we live in. In EnglishF. W.FurmanG. C. (Eds.), Research and educational leadership: Navigating the new National Research Council Guidelines (pp. 133-168). Blue Ridge Summit, PA: Rowman & Littlefield Education.
46.
RossiP. H.LipseyM. W.FreemanH. E. (2004). Evaluation: A systematic approach. Thousand Oaks, CA: Sage.
47.
ShadishW. R.CookT. D.CampbellD. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin.
48.
SlavinR. E. (2002). Evidence-based education policies: Transforming educational practice and research. Educational Researcher, 31(7), 15-21.
49.
ThomasT. A. (1970). Changes in elementary school principals as a result of laboratory training. Eugene: University of Oregon, Center for Advanced Study of Educational Administration.
50.
WallaceF.BlaseK.FixsenD.NaoomS. (2008). Implementing the findings of research: Bridging the gap between knowledge and practice. Alexandria, VA: Educational Research Service.
WeissC. H. (1995). Nothing as practical as good theory: Exploring theory-based evaluation for comprehensive community initiatives for children and families. In ConnellJ. I.KubischA. C.SchorrL. B.WeissC. H. (Eds.), New approaches to evaluating community initiatives: Concepts, methods, and contexts (pp. 65-92). Washington, DC: Aspen Institute.
53.
YoungI. P. (1997). Holmes versus traditional teacher candidates: Labor market receptivity. Journal of School Leadership, 7, 330-344.
