For decades colleges and universities have experimented with various technological innovations to deliver instruction. Although not new in higher education, use of learning technology has greatly increased in recent years (Burnaska, 1998). This increased use brings an increased need for understanding the methodologies and approaches best suited to the evaluation of learning technologies in higher education. Here we first review the literature and describe the methods used in a myriad of evaluation studies in this area. Our primary purpose, however, is to describe our experiences evaluating a campus-wide learning technology effort (the SCALE Project) at the University of Illinois at Urbana-Champaign. This evaluation spanned three years and used multiple methods. More importantly, it provided some challenges from which others may learn and benefit.
Get full access to this article
View all access options for this article.
References
1.
Burnaska, K. (1998). Using Mallard to teach an undergraduate Economics Course. Doctoral dissertation, University of Illinois at Urbana-Champaign.
2.
Clark, D. (1992). Getting results with distance education. The Journal of Distance Education, 12(1), 38-51.
3.
Clarke, R. E. (1992). Media use in education. In M. C. Alkin (Ed.), Encyclopedia of educational research in education (pp. 805-814). New York: Macmillan.
4.
Cooley, W. W., & Lohnes, P. R. (1976). Evaluation research in education. New York: Irvington.
5.
Cousins, J. B., & Earl, L. (1992). The case for participatory evaluation. Educational Evaluation and Policy Analysis, 4, 397-418.
6.
Crump, R. E. (1928). Correspondence and class extension work in Oklahoma. Doctoral dissertation, Teachers College, Columbia University.
7.
Draper, S., Brown, M., Henderson, F., & McAteer, E. (1996). Integrative evaluation: an emerging role for classroom studies of CAL. Computers and Education, 26, 1-3.
8.
Ester, D. P. (1995). CAI, lecture, and learning style: The differential effects of instructional method. Journal of Research on Computing in Education, 27(4), 129-139.
9.
Gold, L., & Maitland, C. (1999). What’s the difference? Paper prepared by the Institute for Higher Education Policy.
10.
Goldberg, M. W. (1997). CALOS: First results from an experiment in computer-aided learning.!Proceedings, ALM’s 28th SIGCSE Technical Symposium on Computer Science Education.
11.
Greene, J. (1997). Participatory evaluation. In R. Stake (Ed.), Evaluation and the post modern dilemma. Advances in Program Evaluation, Vol. 3 (pp. 171-203). Greenwich, CT: JAI Press.
12.
Hiltz, R. S. (1997). Impacts of college-level courses via asynchronous learning networks: Some preliminary results. Journal of Asynchronous Learning Networks, 1, 2.
13.
Jones, A., Scanlon, E., Tosunoglu, C., Ross, S., Butcher, P., Murphy, P., & Greenberg, J. (1996). Evaluating CAL at the Open University: 15 years on. Computers in Education, 26(19), 5-15.
14.
Kearsley, G. (1990). Designing education software for international use. Journal of Research on Computing in Education, 23(2), 242-250.
15.
Khalili, A., & Shashaani, L. (1994). The effectiveness of computer applications: A meta-analysis. Journal of Research on Computing in Higher Education, 27(1), 48-61.
16.
Kulik, C. C., & Kulik, J. A. (1991). Effectiveness of computer-based instruction: An updated analysis. Computers and Human Behavior, 7(1), 74-75.
17.
Martin, E. D., & Rainey, L. (1993). Student achievement and attitude in a satellite-delivered high school science course. The American Journal of Distance Learning, 17(1), 22-28.
18.
McAteer, E., Neil, D., Barr, N., Brown, M., Drapes, S., & Henderson, F. (1996). Simulation Software in a Life Sciences practical course. Computers and Education, 26(1-3), 101-112.
19.
McKenna, S. (1995). Evaluating IMM: Issues for researchers, Open Learning Institute. Occasional Paper #17, Charles Sturt University. New South Wales, Australia.
20.
Morgan, D (1998). Planning focus groups. Focus Group Kit 2. Thousand Oaks, CA: Sage.
21.
Oliver, M. (1997). A framework for evaluating the use of learning technology. BP ELT Report no. 1. University of North London. London, England.
22.
Parlett, M., & Hamilton, O. (1976). Evaluation as illumination. In G.V. Glass (Ed.), Evaluation studies review annual (pp. 141-157). Beverly Hills, CA: Sage.
23.
Patton, M. Q. (1997). Utilization-focused evaluation. Thousand Oaks, CA: Sage.
24.
Popham, W. J. (1961). Tape recorded lectures in the college classroom. AV Communications Review (9).
25.
Popham, W. J. (1972). An evaluation guidebook. Los Angeles, CA: Instructional Objectives Exchange.
26.
Reeves, T. C. (1994). Evaluating what really matters in computer-based education. In M. Wild & D. Kirkpatrick (Eds.), Computer education: New perspectives (pp. 219-246). Perth, Australia: MASTEC.
27.
Russell, T. (1999). The no significant difference phenomenon. Raleigh, NC: North Carolina State University.
28.
Schramm, W. (1962). What we know about learning from instructional television: The next ten years. Stanford, CA: Stanford Institute for Communication Research.
29.
Scriven, M. (1972). Pros and cons about goal-free evaluation. Evaluation Comment, 3, 1-8.
30.
Stake, R. E. (1978). Program evaluation, particularly responsive. Kalamazoo, MI: Western Michigan University.
31.
Suchy, R. R., & Baumann, P. C. (1960). The Milwaukee experiment in instructional television: Evaluation report. Milwaukee, WI: Milwaukee Public Schools.
32.
Tyler, R. W. (1942). General statement on evaluation. Journal of Educational Research, 35, 492-501.
33.
Worthen, B., Sanders, J., & Fitzpatrick, J. (1997). Program evaluation: Alternative approaches and practical guidelines. New York: Longman
