Assessing Knowledge Structures: Don't Always Call an Expert

Abstract

Measures of knowledge structures can be used to access and evaluate conceptual understanding for assessment and training purposes. Typically, the quality of an individual's knowledge structure is determined by comparing it to a standard knowledge structure that is an aggregate of the structures of several experts. Recent research suggests that this approach may not be appropriate for all domains. This study investigated different approaches for forming a standard knowledge structure for two knowledge structure measures: relatedness ratings and a diagramming task. Three approaches to developing knowledge standards were compared: a standard derived from expert data, a standard based on high-performing students, and a rational standard developed through an analysis of instructional materials. The knowledge standards were compared in their ability to predict performance on a multiple-choice test. The results showed that comparison of students' structures with a standard constructed by aggregating high-performing student structures produced scores that were independently predictive of performance for both measures, whereas the expert standard resulted in independently predictive knowledge scores for only the diagramming task. For both measures, the high-performer standard and the aggregate expert standard were superior to the rational standard. These results offer support for using standards other than the expert-consensus standard typically used when assessing the quality of knowledge structures.