In light of the current debate over the meaning of scientifically based research, we argue that qualitative methods should be an essential part of large-scale program evaluations if program effectiveness is to be determined and understood. This article chronicles the challenges involved in incorporating qualitative methods into the large-scale evaluation of the Waterford Early Reading Program and the decisions made in light of those challenges. We demonstrate that, in spite of the challenges, there are significant benefits associated with using qualitative methods on a large scale. More specifically, by using qualitative methods, we were able to improve the evaluation and generate findings that were meaningful and useful to stakeholders. It is our hope that our experiences can be used by others to inform their work as they seek to incorporate qualitative methods into their large-scale evaluations.
Get full access to this article
View all access options for this article.
References
1.
AdamsM. J. (1990). Beginning to read: Thinking and learning about print. Cambridge, MA: MIT Press.
2.
AschbacherP. R. (1999). Developing indicators of classroom practice to monitor and support school reform (CSE Technical Report 513). National Center for Research on Evaluation, Standards, and Student Testing (CRESST)/UCLA.
3.
BurkhardtH., and SchoenfeldA. H. (2003). Improving educational research: Toward a more useful, more influential, and better-funded enterprise. Educational Researcher, 32(9), 3—14.
4.
ChatterjiM. (2004). Evidence on “what works”: An argument for extended-term mixed-method (ETMM) evaluation designs. Educational Researcher, 33(9), 3—13.
5.
DatnowA., and YonezawaS. (2004). Observing school restructuring in multilingual, multicultural classrooms: Balancing ethnographic and evaluative approaches. In WaxmanH.C., TharpR.G., and HilbergR.S. (Eds.), Observational research in US classrooms: New approaches for understanding cultural and linguistic diversity (pp. 174—204). Cambridge, England: Cambridge University Press.
6.
DuffyG., RoehlerL., and HerrmannB. (1988). Modeling mental processes helps poor readers become strategic readers. Reading Teacher, 40, 514—521.
7.
EhriL. C. (1992). Reconceptualizing the development of sight word reading and its relationship to recoding. In GoughP.B., EhriL.C., and TreimanR. (Eds.), Reading acquisition (pp. 107—143). Hillsdale, NJ: Erlbaum.
8.
EricksonF. (1988). Ethnographic description. In Von Ulrich AmnionJ., DittmarN., and MattheierK. (Eds.), Sociolinguistics (pp. 1081—1095). Berlin, Germany: Walter de Gruyter.
9.
EricksonF., and GutierrezK. (2002). Culture, rigor, and science in educational research. Educational Researcher, 31(8), 21—24.
10.
EisenhartM., and TowneL. (2003). Contestation and change in national policy on “scientifically based” education research. Educational Researcher, 32(7), 31—39.
11.
FeuerM. J., TowneL., and ShavelsonR. (2002). Scientific culture and educational research. Educational Researcher, 31(8), 4—14.
12.
GreeneJ. F. (1998). Qualitative, interpretive evaluation. In ReynoldsA.J., and WalbergH.J. (Eds.), Evaluation research for educational productivity (pp. 135—154). Greenwich, CT: JAI Press.
13.
HansenE. E., LlosaL., and SlaytonJ. (2004). Evaluation of the Waterford Early Reading Program as a supplementary program in the Los Angeles Unified School District 2002–2003 (Planning, Assessment and Research Division Publication No. 170). Program Evaluation and Research Branch, Los Angeles Unified School District, Los Angeles, CA.
14.
MaxwellJ. A. (2004). Causal explanation, qualitative research, and scientific inquiry in Journal item: education. Educational Researcher, 33(2), 3—11.
15.
NogaJ. E. (2002, April). Learning in small classes: Using qualitative methods for evaluation to understand how the process of learning differs in smaller classes (Paper presented at the annual meeting of the American Education Research Association (AERA), LA: New Orleans.
PearsonP. D. (1984). Direct explicit teaching of reading comprehension. In DuffyG., RoehlerL., and MasonJ. (Eds.), Comprehension instruction: Perspectives and suggestions (pp. 222—233). New York: Longman.
18.
PiburnM., and SawadaD. (2001). Reformed teaching observation protocol (RTOP): Reference Manual (ACEPT Technical Report IN00–3). Arizona Collaborative for Excellence in the Preparation of Teachers, Tempe, AZ.
19.
PressleyM., El-DinaryP., GaskinsI., SchuderT., BergmanJ., and AlmasiL.. (1992). Beyond direct explanation: Transactional instruction of reading comprehension strategies. Elementary School Journal, 92, 511—554.
20.
SaxeG. B., GearhartM., and SeltzerM. (1999). Relations between classroom practices and student learning in the domain of fractions. Cognition and Instruction, 17, 1—24.
21.
SlavinR. E. (2002). Evidence-based education policies: Transforming educational practice and research. Educational Researcher, 31(7), 15—21.
22.
SlaytonJ., and LlosaL. (2002). Evaluation of the Waterford Early Reading Program 2001–2002: Implementation and student achievement (Planning, Assessment and Research Division Publication No. 144)Program Evaluation and Research Branch, Los Angeles Unified School District, Los Angeles, CA.
23.
WoodcockR. W. (1998). Woodcock reading mastery tests: Revised normative update, examiner's manual. Circle Pines, MN: American Guidance Service.
24.
YoppH. (1992). Developing phonemic awareness in young children. The Reading Teacher, 45, 696—703.
