A Rank-Based Method for the Usability Comparison of Competing Products

Abstract

This paper describes a method for a rank-based analysis of data for competitive usability evaluation. The technique works with data collected during scenario-based usability studies. In a scenario-based study, participants are asked to perform realistic tasks (scenarios) with products. Dependent measures commonly include such variables as time-on-task, successful task completion rates, and subjective ratings, reported at the scenario level. Scenario-based studies are sometimes used to set benchmarks or testable behavioral objectives for products. Multivariate statistics such as discriminant analysis can be used with raw data to determine if one product differs from another on the basis of patterns of dependent variables; however, multivariate statistics cannot be used to demonstrate that one product is more usable than another if the designs are based on different usability tradeoffs. Converting raw data to ranks allows the establishment of rank-weighting schemes that combine different dependent measures and allows the assessment of relative product usability. The data that are generated can be analyzed with rank statistical methods. The elimination of various types of biases associated with missing data is also presented. This method of analyzing competitive usability data is a mixture of the subjective and the objective. To use the method, several subjective decisions concerning rank weighting and control of biases must be made before applying the objective part of the method. Application of the method allows a single composite number representing relative usability to be assigned to a product, simplifying product usability comparison.