In recent years, states, districts, schools, and external partners have recognized the need to proactively foster the use of data to guide educational decision-making and practice. Understanding that data alone will not guarantee use, individuals at all levels have invested in interventions to support better access to, interpretation of, and responses to data of all kinds. Despite the emergence of these efforts, there has been little systematic examination of research on such efforts.
Purpose/Objective/Research Question/Focus of Study
This article synthesizes what we currently know about interventions to support educators’ use of data—ranging from comprehensive, system-level initiatives, such as reforms sponsored by districts or intermediary organizations, to more narrowly focused interventions, such as a workshop. The article summarizes what is what is known across studies about the design and implementation of these interventions, their effects at the individual and organizational levels, and the conditions shown to affect implementation and outcomes.
Research Design
Literature review.
Data Collection and Analysis
This review entailed systematic searches of electronic databases and careful sorting to yield a total of 41 books, peer-reviewed journal articles, and reports. Summaries of each publication were coded to identify the study methods (design, framework, sample, time frame, data collection), intervention design (level of schooling, focal data and data user, leverage points, components), and findings on implementation, effects, and conditions.
Findings/Results
The review uncovers a host of common themes regarding implementation, including promising practices (e.g., making data “usable” and “safe,” targeting multiple leverage points) and persistent challenges (e.g., developing support that is generic but also customized, sustaining sufficient support). The review also finds mixed findings and levels of research evidence on effects of interventions, with relatively more evidence on effects on educators’ knowledge, skills, and practice than on effects on organizations and student achievement. The article also identifies a set of common conditions found to influence intervention implementation and effects, including intervention characteristics (capacity, data properties), broader context (leadership, organizational structure), and individual relationships and characteristics (trust, beliefs and knowledge).
Conclusions/Recommendations
The review finds that the current research base is limited in quantity and quality. It suggests the need for more methodologically rigorous research and greater attention to the organizational and student-level outcomes of interventions, comparative analyses, interventions that help educators move from knowledge to action, and specific ways in which the quality of data and leadership practices shape the effectiveness of interventions.
Get full access to this article
View all access options for this article.
References
1.
AckoffR. L. (1989). From data to wisdom. Journal of Applied Systems Analysis, 16, 3–9.
2.
AncessJ., BarnettE., & AllenD. (2007) Using research to inform the practice of teachers, schools, and school reform organizations. Theory Into Practice, 46(4), 325–333.
3.
AndersonS. E., LeithwoodK., & StraussT. (2010). Leading data use in schools: Organizational conditions and practices at the school and district levels. Leadership and Policy in Schools, 9(3), 292–327.
4.
BodillyS. J. (1998). Lessons from the New American Schools’ scale-up phase: Prospects for bringing design to multiple schools.Santa Monica, CA: RAND.
5.
BondL., GloverS., GodfreyC., ButlerH., & PattonG. C. (2001). Building capacity for system-level change in schools: Lessons from the Gatehouse Project. Health Education Behavior, 28(3), 368–383.
6.
ChoppinJ. (2002, April). Data use in practice: Examples from the school level. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA.
7.
CoburnC. E. (2010). Partnership for district reform: The challenges of evidence use in a major urban district. In CoburnC. E., & SteinM. K. (Eds.), Research and practice in education: Building alliances, bridging the divide. (pp. 167–182). New York, NY: Rowman & Littlefield.
8.
CoburnC. E., HonigM. I., & SteinM. K. (2009). What's the evidence on district's use of evidence? In BransfordJ., StipekD. J., VyeN. J., GomezL., & LamD. (Eds.), Educational improvement: What makes it happen and why? (pp. 67–86). Cambridge, MA: Harvard Educational Press.
9.
CoburnC. E., & TalbertJ. E. (2006). Conceptions of evidence use in school districts: Mapping the terrain. American Journal of Education.112(4), 469–495.
10.
CoplandM. (2003). Leadership of inquiry: Building and sustaining capacity for school improvement. Educational Evaluation and Policy Analysis, 25, 375–395.
11.
CorcoranT., & LawrenceN. (2003). Changing district culture and capacity: The impact of the Merck Institute for Science Education Partnership. (CPRE Research Report Series RR-054). Philadelphia: Consortium for Policy Research in Education, University of Pennsylvania.
DatnowA., & CastellanoM. (2001). Managing and guiding school reform: Leadership in Success for All schools. Educational Administration Quarterly, 37(2), 219–249.
14.
DemieF. (2003). Using value-added data for school self-evaluation: A case study of practice in inner-city schools. School Leadership & Management, 23(4), 445–467.
15.
DentonC. A., SwansonE. A., & MathesP. G. (2007). Assessment-based instructional coaching provided to reading intervention teachers. Reading and Writing: An Interdisciplinary Journal, 20(6), 569–590.
16.
FeldmanJ., & TungR. (2001, April). Whole school reform: How schools use the data-based inquiry and decision making process. Paper presented at the annual meeting of the American Educational Research Association, Seattle, WA.
17.
FeldmanM., & MarchJ. (1981). Information in organizations as signal and symbol. Administrative Science Quarterly, 26, 171–186.
18.
GallimoreR., ErmelingB.A., SaundersB., & GoldenbergC. (2009). Moving the learning of teaching closer to practice: Teacher education implications of school-based inquiry teams. Elementary School Journal, 109(5), 537–553.
19.
GearhartM., & OsmundsonE. (2009). Assessment portfolios as opportunities for teacher learning. Educational Assessment, 14(1), 1–24.
20.
GillB., HamiltonL., LockwoodJ. R., MarshJ., ZimmerR., HillD., & PribeshS. (2005). Inspiration, perspiration, and time: Operations and achievement in Edison Schools. (No. MG-351-EDU). Santa Monica, CA: RAND. Retrieved from http://www.rand.org/pubs/monographs/2005/RAND_MG351.pdf
21.
GoertzM., OlahL., & RigganM. (2009). From testing to teaching: The use of interim assessments in classroom instruction. (CPRE Research Report No. RR-65). Philadelphia: Consortium for Policy Research in Education, University of Pennsylvania.
22.
HalversonR., & ShapiroB. (in press). Technologies for learning and learners: How data are (and are not) changing schools. (WCER Working Paper). Madison: Wisconsin Center for Education Research.
23.
HamiltonL., HalversonR., JacksonS., MandinachE., SupovitzJ., & WaymanJ. (2009). Using student achievement data to support instructional decision making. (NCEE 2009-4067). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.
24.
HenigJ. R. (2012). The politics of data use. Teachers College Record, 114(11).
25.
HonigM. I. (2004). The new middle management: Intermediary organizations in education policy implementation, Educational Evaluation and Policy Analysis, 26(1), 65–87.
26.
HonigM. I., & IkemotoG. (2008). Adaptive assistance for learning improvement efforts: The case of the Institute for Learning. Peabody Journal of Education, 83(3), 328–363.
27.
HookC. M., & RosenshineB. V. (1979). Accuracy of teacher reports of their classroom behavior. Review of Educational Research, 49, 1–11.
28.
HuffmanD., & KalninJ. (2003). Collaborative inquiry to make data-based decisions in schools. Teaching and Teacher Education, 19(6), 569–580.
29.
IngramD., LouisK. S., & SchroederR. G. (2004). Accountability policies and teacher decision making: Barriers to the use of data to improve practice. Teachers College Record, 106, 1258–1287.
30.
JaquithA., & McLaughlinM. (2009). A temporary, intermediary organization at the helm of regional education reform: Lessons from the Bay Area School Reform Collaborative. Second International Handbook of Educational Change, Springer International Handbooks of Education, 23(1), 85–103.
31.
KerrK. A., MarshJ. A., IkemotoG. S., DarilekH., & BarneyH. (2006). Districtwide strategies to promote data use for instructional improvement. American Journal of Education, 112, 496–520.
32.
KoretzD., MitchellK., BarronS., & KeithS. (1996). Perceived effects of the Maryland School Performance Assessment Program.Los Angeles, CA: National Center for Research on Evaluation, Standards, and Student Testing and RAND.
33.
KronleyR. A., & HandleyC. (2003). Reforming relationships: School districts, external organizations, and systemic change. Report prepared for School Communities That Work: A National Task Force on the Future of Urban Districts. Providence, RI: Annenberg Institute for School Reform at Brown University.
34.
LachatM. A., & SmithS. (2005). Practices that support data use in urban high schools. Journal of Education for Students Placed at Risk, 10(3), 333–349.
35.
LincolnY., & GubaE. (1985). Naturalistic inquiry.Beverly Hills, CA: Sage.
36.
LoveN. (2004). Taking data to new depths. Journal of Staff Development, 25(4), 22–26.
37.
MandinachE., HoneyM., LightD., & BrunnerC. (2008). A conceptual framework for data-driven decision-making. In MandinachE. B., & HoneyM. (Eds.), Data-driven school improvement: Linking data and learning. (pp. 13–31). New York, NY: Teachers College Press.
MarshJ., HamiltonL., & GillB. (2008). Assistance and accountability in externally managed schools: The case of Edison Schools, Inc. Peabody Journal of Education, 83(3), 423–458.
40.
MarshJ., McCombsJ. S., & MartorellF. (2010). How instructional coaches support data-driven decision making: Policy implementation and effects in Florida middle schools. Educational Policy, 24(6), 872–907.
41.
MarshJ. A., PaneJ. F., & HamiltonL. S. (2006). Making sense of data-driven decision making in education: Evidence from recent RAND research. (No. OP-170-EDU). Santa Monica, CA: RAND Corporation.
42.
MasonS. (2002). Turning data into knowledge: Lessons from six Milwaukee Public Schools.Madison: Wisconsin Center for Education Research.
43.
MassellD. (2001). The theory and practice of using data to build capacity: State and local strategies and their effects. In FuhrmanS. H. (Ed.), From the capitol to the classroom: Standards-based reform in the states. (pp. 148–169). Chicago, IL: University of Chicago Press.
44.
MassellD., & GoertzM. E. (2002). District strategies for building instructional capacity. In HightowerA. M., KnappM. S., MarshJ. A., & McLaughlinM. W. (Eds.), School districts and instructional renewal. (pp. 43–60). New York, NY: Teachers College Press.
45.
McDougallD., SaundersW., & GoldenbergC. (2007). Inside the black box of school reform: Explaining the how and why of change at Getting Results schools. International Journal of Disability, Development and Education, 54(1), 51–89.
46.
McLaughlinM. W., & MitraD. (2001). Theory-based change and change-based theory: Going deeper, going broader. Journal of Educational Change, 2, 301–323.
47.
MeansB., PadillaC., & GallagherL. (2010). Use of education data at the local level: From accountability to instructional improvement.Washington, DC: U.S. Department of Education, Office of Planning, Evaluation, and Policy Development
48.
MoodyL., & DedeC. (2008). Models of data-based decision making: A case study of the Milwaukee Public Schools. In MandinachE., & HoneyM. (Eds.), Linking data and learning. (pp. 233–254). New York, NY: Teachers College Press.
49.
MurnaneR., SharkeyN., & BoudettK. (2005) Using student-assessment results to improve instruction: Lessons from a workshop. Journal of Education for Students Placed at Risk, 10(3), 269–280.
50.
NelsonT. H., & SlavitD. (2007). Collaborative inquiry among science and mathematics teachers in the USA: Professional learning experiences through cross-grade, cross-discipline dialogue. Professional Development in Education, 33(1), 23–39.
51.
NewfieldJ. (1980). Accuracy of teacher reports: Reports and observations of specific classroom behaviors. Journal of Educational Research, 74(2), 78–82.
52.
O'DayJ. (2002). Complexity, accountability, and school improvement. Harvard Educational Review, 72, 293–329.
53.
ParkV., & DatnowA. (2008). Collaborative assistance in a highly prescribed whole school reform model: The case of Success for All. Peabody Journal of Education, 83(3), 400–422.
54.
ParkV., & DatnowA. (2009). Co-constructing distributed leadership: District and school connections in data-driven decision making. School Leadership and Management.29(5), 477–494.
55.
PattonM.Q. (2002). Qualitative evaluation and research methods.Thousand Oaks, CA: Sage.
56.
PorterK. E., & SnipesJ. C. (2006). The challenge of supporting change: Elementary student achievement and the Bay Area School Reform Collaborative's focal strategy.New York, NY: MDRC.
57.
QuintJ., SepanikS., & SmithJ. (2008). Using student data to improve teaching and learning: Findings from an evaluation of the Formative Assessments of Student Thinking in Reading (FAST-R) program in Boston elementary schools.New York, NY: MDRC.
58.
ReynoldsD., StringfieldS., & SchafferE. (2006). The High Reliability Schools project: Some preliminary results and analyses. In ChrispeelsJ., & HarrisA. (Eds.), School improvement: International perspectives. (pp. 56–76). London, England: Routledge.
59.
RoehrigA. D., DuggarS. W., MoatsL., GloverM., & MinceyB. (2008). When teachers work to use progress monitoring data to inform literacy instruction: Identifying potential supports and challenges. Remedial and Special Education, 29(6), 364–382.
60.
SmircichL. (1985). Is the concept of culture a paradigm for understanding organizations and ourselves? In FrostP. J., MooreL. F., LouisM. R., LundbergC. C., & MartinJ. (Eds.), Organizational culture. (pp. 55–72). Beverly Hills, CA: Sage.
61.
SlavitD., & NelsonT. H. (2010). Collaborative teacher inquiry as a tool for building theory on the development and use of rich mathematical tasks. Journal of Mathematics Teacher Education, 13, 201–221.
62.
SpillaneJ. P. (2000). Cognition and policy implementation: District policymakers and the reform of mathematics education. Cognition and Instruction, 18(2), 141–179.
63.
SpillaneJ. P., & MieleD. B. (2007). Evidence in practice: A framing of the terrain. In MossP. A. (Ed.), Evidence and decision making. (pp. 46–73). Malden, MA: Blackwell.
64.
StringfieldS., ReynoldsD., & SchafferE. C. (2008). Improving secondary students’ academic achievement through a focus on reform reliability: 4- and 9-year findings from the High Reliability Schools project. School Effectiveness and School Improvement, 19(4) 409–428.
65.
SupovitzJ. A. (2006). The case for district-based reform: Leading, building, and sustaining school improvement.Cambridge, MA: Harvard Education Press.
66.
SupovitzJ. A. (2012). Getting at student understanding—The key to teachers’ use of test data. Teachers College Record, 114(11).
67.
SupovitzJ., & KleinV. (2003). Mapping a course for improved student learning: How innovative schools use student performance data to guide improvement.Philadelphia, PA: Consortium for Policy Research in Education.
68.
SupovitzJ., & WeathersJ. (2004). Dashboard lights: Monitoring implementation of district instructional reform strategies.Philadelphia, PA: Consortium for Policy Research in Education.
69.
SutherlandS. (2004). Creating a culture of data use for continuous improvement: A case study of an Edison Project school. American Journal of Evaluation, 25(3), 277–293.
WeissC. H. (1980). Knowledge creep and decision accretion. Knowledge: Creation, Diffusion, Utilization, 1(3), 381–404.
72.
WohlstetterP., DatnowA., & ParkV. (2008). Creating a system for data-driven decision-making: Applying the principal-agent framework. School Effectiveness and School Improvement Journal, 19(3), 239–259.
73.
YangM., GoldsteinH., RathT., & HillN. (1999) The use of assessment data for school improvement purposes. Oxford Review of Education, 25(4), 469–483.
74.
YoungV. M., & KimD. H. (2010). Using assessments for instructional improvement: A literature review. Education Policy Analysis Archives, 18(19), 1–36.
