Choosing Research Topics Wisely: The Irony of Item Response Theory as an Example

Abstract

Too often young researchers choose topics to work on for the topic’s convenience rather than its importance. In earlier work we have laid out in broad strokes some areas that would reward progress, but in this essay, we provide guidance toward topic choice that is much more specific, focusing on a fundamental irony of item response theory (IRT). IRT is the name given to a family of models that provide a stochastic description of what happens when a person meets a test item. In general, models are most useful when we can lean on them in areas where data are sparse. Strong models are required when data are very limited, weaker models are possible when there are more data. The recommended size of data samples required to fit IRT models has historically seemed excessively large, yielding the apparent irony that IRT can only be used when data samples are so large that no model is needed. Bayesian work over the past decade or so has revealed that with modern estimation methods, IRT models can be successfully fit with surprisingly modest samples. One practical consequence of expanding the use of IRT to such samples is that it makes possible its use on classroom tests. This can have profound positive implications. In this essay, we describe the apparent irony of IRT and then outline a two-part research project that may resolve matters, thus making the potential provided by IRT models available to a much wider range of applications.