Abstract
Several studies have found differences between subjective preference ratings and objective performance measures. Bailey [Bailey 93] summarizes several, and argues for separate treatment of these concepts. Our results in a multifactor multivariate experiment support Bailey's contention, but adds a new dimension of concern: the use of concurrent versus retrospective subjective ratings. The presentation here will focus on the relationship of performance and concurrent versus retrospective preference ratings. Retrospective ratings may represent users' lasting impressions of a system after a trial use, but may not be good predictors of performance. Concurrent ratings of confidence of accuracy were found, in this study, to be better predictors performance. We offer recommendations about how to make the best use of these different evaluation measures, particularly when they differ.
Get full access to this article
View all access options for this article.