An Historical Perspective and a Model for Evaluation of Intelligent Tutoring Systems

Abstract

The purposes of this article are to 1) examine the changes which have occurred in the development and evaluation of ITS systems for the last twenty years, 2) speculate on future directions, and 3) propose a conceptual model to evaluate and institutionalize this technological innovation into training and educational settings. First, we review theoretical and technical dimensions that could be considered relevant to internal evaluation of ITS technology. Secondly, we discuss evaluation dimensions which, while external to ITS per se, are relevant for situational evaluation of any technological innovation. After theoretical and methodological discussions of intelligent features and effectiveness of ITS, its research and development activities are examined from a historical perspective. The development of ITS is divided into three stages and the development focus for each stage is discussed to show the shifts. It is observed that theoretical and methodological criticisms of ITS, needs unique to practical applications, and development of computer technology have influenced the shifts of ITS development focuses: purpose, staffing, hardware and software technology used, and contributions. In stage III, most recent development focuses and future prospects are discussed, including investigation of specific instructional strategies, creation of flexible environments for instructional strategy research, modeling of the human tutor, simulation of human learning and cognition, and creation of multimedia environments. After discussing problems associated with moving ITS from the laboratory to an institutionalized part of the practical world, a conceptual model for evaluating technological innovation in training and educational environments is proposed. This model considers the technology purpose, processes, outcome measures, and the maturity of the technological innovation (the latter especially relevant for ITS). It has three major stages: adoption, implementation, and institutionalization and two major processes: assimilation and accommodation of the innovation. These stages and processes occur within a context of multiple levels of users/stakeholders or decision-makers. Taken together they make up the dimensions of the evaluation model for any technological innovation, including ITS.