Predicting item difficulty in a reading comprehension test with an artificial neural network

Abstract

This article reports the results of using a three-layer backpropagation artificial neural network to predict item difficulty in a reading comprehension test. Two network structures were developed: one with the sigmoid function in the output processing unit and the other without the sigmoid function in the output processing unit. The dataset which consisted of a table of coded test items and corresponding item difficulties was partitioned into a training set and a test set in order to train and test the neural networks. To demonstrate the consistency of the neural networks in predicting item difficulty, the training and testing runs were repeated four times starting with a new set of initial weights. Additionally, the training and testing runs were repeated by switching the training set and the test set. The mean squared error values between the actual and predicted item difficulty demonstrated the consistency of the neural networks in predicting item difficulty for the multiple training and testing runs. Significant correlations were obtained between the actual and predicted item difficulties and the Kruskal-Wallis test indicated no significant difference in the ranks of actual and predicted values.