Weighting Components of a Composite Score Using Naïve Expert Judgments About Their Relative Importance

Abstract

A common problem that arises in testing—as well as other contexts such as candidate selection—is how to combine various scores into a weighted composite that reflects expert judgments about each component’s relative importance. For experts to provide nominal weights explicitly, they must fully account for the variances of the components, the covariances among components, and the reliability of each component. This task can be challenging, and in many cases, experts may have greater success making simple judgments about component importance without regard for the variances, covariances, and reliabilities. In this article, it is shown how to estimate the requisite nominal weights when only these kinds of naïve judgments are available, and the analytical solution is demonstrated with a small simulation study. Results from the simulation suggest that the proposed estimators could yield more valid composite scores in practice.