Educational technology applications and lessons, such as learning modules, activities, and user-authored systems, face substantial implementation challenges in public schools and post-secondary institutions. The design of technology impacts the willingness of teachers to implement new technology; evaluators examining implementation obstacles can help improve design. In this article, both summative and formative implementation evaluation are discussed in non-technology contexts and compared with current practices with educational technology. Formative evaluation has traditionally focused on making applications and lessons “work”; however, many implementation problems are caused by non-technical/interface concerns and related to the compatibility of the technology with its surroundings. An overview of research on implementation of applications and lessons is provided and linked to a model of formative evaluation for implementation in this area. Technical, curricular/instructional, and practical areas of implementation are examined. Common implementation problems and typical design recommendations are presented. The use of the model is discussed in other areas, such as medicine and assistive technology.
Get full access to this article
View all access options for this article.
References
1.
Baker, E. (2001). Technology: How do we know it works? In W. Heineke & J. Willis (Eds), Methods of evaluating educational technology. Greenwich, CT: Information Age.
2.
Baker, E., Herman, J., & Gearhart, M. (1994). Evaluating the Apple classrooms of tomorrow. In E. L. Baker & H. F. O’Neil Jr. (Ed.), Technology assessment in education and training. Hillsdale, NJ: Earlbaum.
3.
Bates, A.W. (2000). Managing Technological Change: Strategies for College and University Leaders. San Francisco, CA: Jossey-Bass.
4.
Becker, H. J. (1994). How exemplary computer using teachers differ from other teachers: Implications for realizing the potential of computers in schools. Journal of Research on Computing in Education, 26, 291-321.
5.
Becker, H. J. (1998). Running to catch a moving train: Schools and information technologies. Theory into Practice, 37, 20-30.
6.
Becker, H. J. (2000). Access to classroom computers. Communications of the ACM, 43(6), 24-25.
7.
Brown, A., Bransford, J., & Cocking, R. (2000). How people learn: Brain, mind, experience and school. Washington, DC: National Academy Press.
8.
Bullock, C., & Ory, J. (2000). Evaluating instructional technology implementation in a higher education environment. American Journal of Evaluation, 21, 315-328.
9.
Chen, H. (1996). A comprehensive typology of program evaluation. Evaluation Practice, 17, 121-130.
10.
Cuban, L. (1986). Teachers and machines: The classroom use of technology since 1920. New York: Teachers College Press.
11.
Cuban, L. (2002). High access and low use of technologies in high school classrooms: Explaining an apparent paradox. American Educational Research Journal, 38, 813-834.
12.
Dehar, M., Casswel, S., & Duignan, P. (1993). Formative and process evaluation of health promotion and disease prevention programs. Evaluation Review, 17, 204-220.
13.
Dick, W. (1996). The Dick and Carey model: Will it survive the decade?Educational Technology: Research and Development, 44(3), 55-63.
14.
Edwards, P. K. (1987). Conceptual and methodological issues in evaluating emergent programs. Evaluation and Program Planning, 10, 27-33.
15.
Ely, D. (1999). Conditions that facilitate the implementation of educational technology innovations. Educational Technology, 39(26), 23-27.
16.
Ertmer, P. A. (1999). Addressing first- and second-order barriers to change: Strategies for technology integration. Educational Technology Research and Development, 47(4), 47-61.
17.
Flagg, B. N. (1990). Formative evaluation for educational technologies. Hillsdale, NJ: Erlbaum.
18.
Gustafson, K. L., & Branch, R. M. (1997). Revisioning models of instructional development. Educational Technology: Research and Development, 45(3), 73-89.
19.
Hativa, N., & Lesgold, A. (1996). Situational effects in classroom technology implementations: Unfulfilled expectations and unexpected outcomes. In S. T. Kerr (Ed.), Technology and the future of schooling: Ninety-fifth yearbook of the National Society for the Study of Education. Chicago: University of Chicago Press.
20.
International Standards Organization. (2003). International standards in usability and user-centered design. Retrieved September 1, 2003 from: http://www.usabilitypartners.se/usability/standards.html
21.
Knapp, L. R., & Glenn, A. D. (1996). Restructuring schools with technology. Needham Heights, MA: Allyn & Bacon.
22.
Lewis, K. (2003). Eight reasons why software implementation efforts fail. Power and Gas. Retrieved September 1, 2003 from: http://www.qbsol.com/pdfs/PGMQuorum.pdf
23.
Locatis, C., & Al-Nuaim, H. (1999). Interactive technology and authoring tools: A historical review and analysis. Educational Technology: Research and Development, 47(3), 63-75.
24.
Lowenhaupt, M. (2003). Top ten pitfalls. Health Care Technology. Available: http://www.hctproject.com/documents.asp?grID=327&dID=1851
25.
MacArthur, D. J. (1998). Untangling the web: Applications of the Internet and other information technologies to higher learning. Santa Monica: Rand.
26.
Maslowski, R., & Visscher, A. J. (1999). Formative evaluation in educational computing research and development. Journal on Research on Computing in Education, 32, 239-255.
27.
McClintock, C. (1984). Toward a theory of formative program evaluation. In D. Deshler (Ed.), Evaluation for program improvement. New Directions for Continuing Education (No. 24.) (p. 77). New York, NY: Jossey-Bass.
28.
McLaughlin, M. W. (1987). Learning from experience: Lessons from policy implementation. Educational Evaluation and Policy Analysis, 9, 171-178.
29.
Mean, B., Coleman, E., Lewis, E., Quellmalz, E., Marder, C., & Valdes, K. (1997). GLOBE Year 2 evaluation. Retrieved September 1, 2003 from: http://www.sri.com/policy/ctl/assets/images/y2full.pdf
30.
Mills, S., & Ragan, T. (2000). A tool for analyzing implementation fidelity of an integrated learning system. Educational Technology: Research and Development, 48(4), 21-41.
31.
NATRI. (2003). How assistive technology services can go awry. Retrieved September 1, 2003 from: http://natri.uky.edu/resources/reports/awry.html
32.
National Center for Educational Statistics. (2000). Quality of elementary and secondary school environments. Retrieved September 1, 2003 from: NCES.ed.gov/programs/coe/2001/section4/indicator39.shtml
33.
Paterno, F. (1998). Mefisto: Designing user interfaces for air traffic control. ERCIM News. Retrieved September 1, 2003 from: http://www.ercim.org/publication/ErcimNews/enw32/paterno.html
34.
Patton, M. Q. (1978). Utilization focused evaluation. Beverly Hills, CA: Sage Publications.
35.
Pressman, J. L., & Wildavsky, A. (1974). Implementation: How great expectations in Washington are dashed in Oakland. Berkeley, CA: University of California Press.
36.
Rochelle, J.M., Pea, R.D., Hoadley, C.M., Gordin, D.N., & Means, B.M. (2000). Changing how and what children learn in school with computer-based technologies. Children and Computer Technology, 10, 76-101.
37.
Russell, T. L. (2003). The no significant difference phenomenon. Retrieved September 1, 2003 from: http://teleeducation.nb.ca/nosignificantdifference/index.cfm
38.
Sanders, J. S., & Cunningham, D. (1974). A structure for formative evaluation in product development. Review of Educational Research, 43, 205-220.
39.
Scriven, M. (1967). The methodology of evaluation. In R. W. Tyler, R. M. Gagne, & M. Scriven (Eds.), Perspectives on curriculum evaluation: AERA Monograph Series on Curriculum Evaluation (pp. 39-83). Chicago, IL: Rand McNalley.
40.
Scriven, M. (1991). Beyond formative and summative evaluation. In M. McLaughlin & D. C. Phillips (Eds.), Evaluation and education: At quarter century. Chicago, IL: University of Chicago Press.
41.
Seidel, R. J., & Perez, R. S. (1994). In H. F. O’Neil & E. L. Baker (Eds.), Technology assessment in software applications. Hillsdale, NJ: Erlbaum.
42.
Stumpf, S., Zalunardo, R., & Chen, R. (2001). Barriers to telemedicine implementation. ABBC briefing paper. Retrieved September 1, 2003 from: http://abbc.usc.edu/projectsfiles/tmbarrierspaper/hitelemedbarrierspaper.pdf
43.
Weston, T. J., & Barker, L. J. (2001). Designing, implementing, and evaluating web-based learning modules for university students. Educational Technology, 41(4), 15-22.
