Major advances in the number, capabilities, and quality of state, national, and transnational databases have opened up new opportunities for evaluators. Both large-scale data sets collected for administrative purposes and those collected by other researchers can provide data for a variety of evaluation-related activities. These include (a) identifying or highlighting issues that invite greater client attention; (b) establishing the plausibility of policy theories of action; and (c) program evaluation. The authors illustrate through examples from the fields of education, social services, and public health both the opportunities provided by higher quality, interoperable data systems and the challenges encountered when using databases to identify issues of concern, test the plausibility of a proposed theory of action, or evaluate an existing program. The authors then explore implications for roles evaluators may need to play to support these uses of databases and for funding and infrastructure to support greater evaluator involvement in program planning and improvement.
Baker, E.L., Barton, P.E., Darling-Hammond, L., Haertel, E.H., Ladd, H.F., Linn, R.L., ... Shepard, Lorrie, A. (2010). Problems with the use of student test scores to evaluate teachers. EPI Briefing Paper #278. Washington, DC: Economic Policy Institute.
2.
Bearman, P., & Brückner, H. (2001). Promising the future: Virginity pledges and first intercourse. American Journal of Sociology, 106, 859-912.
3.
Bennett, S.E., & Assefi, N.P. ( 2005). School-based teenage pregnancy prevention programs: A systematic review of randomized controlled trials. Journal of Adolescent Health, 36, 72-81.
4.
Brückner, H., & Bearman, P. ( 2005). After the promise: The STD consequences of adolescent virginity pledges. Journal of Adolescent Health, 36, 271-278.
5.
Bryk, A.S., Sebring, P.B., Allensworth, E., Luppescu, S., & Easton, J.Q. ( 2010). Organizing schools for improvement: Lessons from Chicago . Chicago, IL: University of Chicago Press.
6.
Castrechini, S. ( 2009). Educational outcomes for court-dependent youth in San Mateo County. Stanford, CA: John W. Gardner Center for Youth and Their Communities.
7.
Chen, H.T. ( 1996). A comprehensive typology for program evaluation. American Journal of Evaluation, 17, 121-130.
8.
Childress, S.M., Doyle, D.P., & Thomas, D.A. ( 2009). Leading for equity: The pursuit of excellence in Montgomery County Public Schools. Cambridge, MA: Harvard Education Press.
9.
Coburn, C.E., Honig, M.I., & Stein, M.K. ( 2009). What’s the evidence on districts’ use of evidence? In J. D. Bransford, D. J. Stipek, N. J. Vye, L. M. Gomez, & D. Lam (Eds.), The role of research in educational improvement (pp. 67-87). Cambridge, MA: Harvard Education Press.
10.
Constantine, N.A. ( 2008). Converging evidence leaves policy behind: Sex education in the United States. Journal of Adolescent Health, 42, 324-326.
11.
Cousins, J.B., & Earl, L.M. ( 1992). The case for participatory evaluation. Educational Evaluation and Policy Analysis, 14, 397-414.
12.
Donaldson, S.I. ( 2007). Program theory-driven evaluation science: Strategies and applications. Mahwah, NJ: Erlbaum .
13.
Donovan, S., Wigdor, A.K., & Snow, C.E. ( 2003). Strategic education research partnership. Washington, DC: National Research Council.
14.
DuBois, D.L., & Silverthorn, N. (2005). Natural mentoring relationships and adolescent health: Evidence from a national study. American Journal of Public Health, 95, 518-524.
15.
Duncan, A. ( 2009). Robust data gives us the roadmap to reform. Paper presented at the Fourth Annual Institute of Education Sciences Research Conference.
16.
Dynarski, M. ( 2008). Bringing answers to educators: Guiding principles for research syntheses. Educational Researcher, 37, 27-29.
Fine, M., & McClelland, S.I. (2006). Sexuality education and desire: Still missing after all these years. Harvard Educational Review, 76, 297-338.
19.
Gallagher, L.P., Means, B., & Padilla, C. ( 2008). Teachers’ use of student data systems to improve instruction: 2005 to 2007. Washington, DC: Office of Planning, Evaluation and Policy Development, Policy and Program Studies Service, U.S. Department of Education.
20.
Ginsburg, A.L., & Rhett, N. ( 2003). Building a better body of evidence: New opportunities to strengthen evaluation utilization. American Journal of Evaluation, 24, 489-498.
21.
Gray, L., Thomas, N., & Lewis, L. ( 2010). Teachers’ use of educational technology in U.S. public schools: 2009 (NCES 2010-040). Washington, DC: National Center for Education Statistics, Institute of Education Sciences, U.S. Department of Education .
22.
Guiton, G., & Oakes, J. ( 1995). Opportunity to learn and conceptions of educational equality . Educational Evaluation and Policy Analysis, 17, 323-336.
23.
Hanberger, A. ( 2001). Policy and program evaluation, civil society, and democracy . American Journal of Evaluation, 22, 211-228.
24.
Hanushek, E.A. ( 2009). Teacher deselection. In D. Goldhaber & J. Hannaway (Eds.), Creating a new teaching profession (pp. 165-180). Washington, DC: Urban Institute.
25.
Hanushek, E.A., Kain, J.F., O’Brien, D.M., & Rivkin, S.G. ( 2005). The market for teacher quality. NBER Working Paper11154. Cambridge, MA: National Bureau of Economic Research.
26.
Hess, F.M., & Petrilli, M.J. (2006). No child left behind. New York, NY: Peter Lang.
27.
Johnson, K., Greenseid, L.O., Toal, S.A., King, J.A., Lawrenz, F., & Volkov, B. ( 2009). Research on evaluation use: A review of the empirical literature from 1986 to 2005. American Journal of Evaluation , 30, 377-410.
28.
Kapadia, K., Coca, V., & Easton, J.Q. ( 2007). Keeping new teachers: A first look at the influences of induction in the Chicago Public Schools. Chicago, IL : Consortium on Chicago School Research.
29.
Kennedy, M.M. ( 2005). From teacher qualifications to teaching quality. Educational Leadership, 63, 14-19.
30.
Kohler, P.K., Manhart, L.E., & Lafferty, W.E. (2008). Abstinence-only and comprehensive sex education and the initiation of sexuality of teen pregnancy. Journal of Adolescent Health, 42, 344-351.
31.
Lehtonen, M. ( 2006). Deliberative democracy, participation, and OECD peer reviews of environmental policies. American Journal of Evaluation , 27, 185-200.
32.
Leviton, L.C., Khan, L.K., Rog, D., Dawkins, N., & Cotton, D. ( 2010). Evaluability assessment to improve public health policies, programs, and practices. Annual Review of Public Health, 31, 213-233.
33.
London, R.A., & Gurantz, O. ( 2010). State and local data infrastructure for tracking secondary to postsecondary educational outcomes. Stanford, CA: John W. Gardner Center for Youth and Their Communities.
34.
McLaughlin, M.W., & O’Brien-Strain, M. (2008). The youth data archive: Integrating data to assess social settings in a societal sector framework . In M. Shinn & H. Yoshikawa (Eds.), Toward positive youth development: Transforming schools and community programs (pp. 313-332). New York, NY: Oxford University Press.
35.
Means, B., Padilla, C., & Gallagher, L.P. (2010). Use of education data at the local level: From accountability to instructional improvement. Washington, DC: Office of Planning, Evaluation, and Policy Development, U.S. Department of Education.
36.
National Research Council and Institute of Medicine. ( 2002). Community programs to promote youth development. Washington, DC: National Academy Press.
Robelen, E.W.(2009, April). Obama echoes Bush on education ideas. Education Week, 28, 118-19..
39.
Roper, W.L., & Tolleson-Rinehart, S. (2001). Health care data and health: From numbers to outcomes. Pharmacoepidemiology and Drug Safety, 10, 363-366.
40.
Rosenbaum, P.R., & Rubin, D.B. ( 1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70, 41-55.
41.
Sanders, W.L., & Horn, S.P. ( 1994). The Tennessee Value-Added Assessment System (TVAAS): Mixed model methodology in educational assessment. Journal of Personnel Evaluation in Education, 8, 299-311.
42.
Sanders, W.L., & Horn, S.P. ( 1998). Research findings from the Tennessee Value-Added Assessment System (TVAAS) Database: Implications for educational evaluation and research . Journal of Personnel Evaluation in Education, 12, 247-256.
43.
Santelli, J., Ott, M.A., Lyon, M., Rogers, J., Summers, D., & Schiefler, R. ( 2006). Abstinence and abstinence-only education: A review of policies and programs. Journal of Adolescent Health, 38, 72-81.
44.
Smith, T.M., & Ingersoll, R.M. (2004). What are the effects of induction and mentoring on beginning teacher turnover? American Educational Research Journal, 41(3), 681-714.
45.
Thompson, S., & Winking, D. ( 2007). Raising the bar, closing the gap. Strategies , 13, 3-16.
46.
Wayman, J.C. ( 2005). Involving teachers in data-driven decision making: Using computer data systems to support teacher inquiry and reflection. Journal of Education for Students Placed at Risk, 10, 295-308.
47.
Wayman, J.C. ( 2007). Student data systems for school improvement: The state of the field TCEA Educational Technology Research Symposium: Volume 1 (pp. 156-162). Lancaster, PA: ProActive.
48.
Wholey, J.S. ( 2004). Evaluability assessment. In J. S. Wholey, H. P. Hatry, & K. E. Newcomer (Eds.), Handbook of practical program evaluation (pp. 33-62). San Francisco, CA: Jossey-Bass.
