A Completion Rate Conundrum: Reducing bias in the Single Usability Metric

Abstract

In the Single Usability Metric benchmarking method, the calculation of completion rates creates a bias for completion rates by ignoring the z-score transform that is conducted for satisfaction and time-on-task measures. This artificially inflates all ‘good’ scores and marks some ‘poor’ scores as ‘good’. This paper discusses two methods to augment the SUM so that it will accurately calculate completion rates into the final SUM score.

Keywords

usability user experience measurement

Get full access to this article

View all access options for this article.

References

Barnum

C. M.

(2020). Usability testing essentials: Ready, set…. test!. Morgan Kaufmann.

Broekhuis

van Velsen

Hermens

(2019). Assessing usability of eHealth technology: a comparison of usability benchmarking instruments. International journal of medical informatics, 128, 24-31.

Chacón

Ponsa

Angulo

(2021). Usability study through a human-robot collaborative workspace experience. Designs, 5(2), 35.

de Leon

G. J. L.

Gratuito

J. P. M.

Polancos

R. V

. (2020). The impact of command buttons, entry fields, and graphics on single usability metric (SUM). In Proceedings of the International Conference on Industrial Engineering and Operations Management Dubai.

Nielsen

Levy

(1994) Measuring Usability: Preference vs. Performance. Communications of the ACM, 37, p. 66-76

R Core Team (2023). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. URL https://www.R-project.org/.

Sauro

(2011). What Is A Good Task-Completion Rate?. In MeasuringU website, retrieved February 2023. URL: https://measuringu.com/task-completion/

Sauro

Kindlund

(2005a, April). A method to standardize usability metrics into a single score. In Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 401-409).

Sauro

Kindlund

(2005b) “How long Should a Task Take? Identifying Specification Limits for Task Times in Usability Tests” in Proceeding of the Human Computer Interaction International Conference (HCII 2005), Las Vegas, USA.

10.

Sauro

Kindlund

(2005c, July). Using a single usability metric (SUM) to compare the usability of competing products. In Human Computer Interaction International Conference.

11.

Van Waardhuizen

McLean-Oliver

Perry

Munko

. (2019, May). Explorations on Single Usability Metrics. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (pp. 1-8).