Reader Variability in Mammography and Its Implications for Expected Utility over the Population of Readers and Cases

Abstract

The multiple-reader, multiple-case (MRMC) approach to receiver operating characteristic (ROC) analysis is becoming the dominant assessment paradigm in medical imaging. Its most common version involves having many readers read every patient case in the study, a critical feature since differences among competing imaging modalities are often dominated by differences in reader performance. The present authors have carried out MRMC ROC analysis on a uniquely large data set for mammography. The analysis quantifies the great range of observed reader skill in that data set. It also demonstrates that the sample sizes are sufficiently large that the conclusions generalize to the populations sampled here with little uncertainty from the finite sample size. A schematic approach to bracketing the utility matrix is then used to study trends in the resulting expected utility functions that correspond to the range of observed ROC curves. This is done for both the screening and the diagnostic context. The results raise 2 hypotheses for further investigation. First, it is possible that the present ambiguity surrounding the effectiveness of mammography is due in part to the observed range of reader skills and corresponding expected utility functions. Second, it is possible that computer-assisted modalities for mammography may lead to improvements in the expected utility function not only for screening but also in the diagnostic context, especially for the lower performing readers.