Modified scoring,traditional item analysis and Sato's caution index used to investigate the reading recall protocol

Abstract

The written recall protocol is increasingly being used in second language reading research as a measure of comprehension. Although the recall protocol is an essay-like instrument, the total score derived is based entirely on summing the discrete propositions correctly recalled. Consequently, item and reliability analyses comparable to those run on multiple-choice tests can and should be performed on recall protocols and on other integrative measures. The purpose of this study is to demonstrate how modified scoring, item and internal consistency analyses, along with Sato's caution indices, can be used to evaluate the quality of the recall protocol as a reading comprehen sion measure. Issues concerning the assumptions underlying classical local independence are discussed with regard to the reading recall protocol and other integrative measures. Results indicate that the procedures can be applied and can yield interpretable results. These results need to be replicated using other texts and other weighting systems. Only when recall protocols are routinely subjected to item and reliability analyses comparable to those performed on other measures, can the instrument be considered a viable alternative.