In the Single Usability Metric benchmarking method, the calculation of completion rates creates a bias for completion rates by ignoring the z-score transform that is conducted for satisfaction and time-on-task measures. This artificially inflates all ‘good’ scores and marks some ‘poor’ scores as ‘good’. This paper discusses two methods to augment the SUM so that it will accurately calculate completion rates into the final SUM score.
BarnumC. M. (2020). Usability testing essentials: Ready, set…. test!. Morgan Kaufmann.
2.
BroekhuisM.van VelsenL.HermensH. (2019). Assessing usability of eHealth technology: a comparison of usability benchmarking instruments. International journal of medical informatics, 128, 24-31.
3.
ChacónA.PonsaP.AnguloC. (2021). Usability study through a human-robot collaborative workspace experience. Designs, 5(2), 35.
4.
de LeonG. J. L.GratuitoJ. P. M.PolancosR. V. (2020). The impact of command buttons, entry fields, and graphics on single usability metric (SUM). In Proceedings of the International Conference on Industrial Engineering and Operations Management Dubai.
5.
NielsenJ.LevyJ. (1994) Measuring Usability: Preference vs. Performance. Communications of the ACM, 37, p. 66-76
6.
R Core Team (2023). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. URL https://www.R-project.org/.
SauroJ.KindlundE. (2005a, April). A method to standardize usability metrics into a single score. In Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 401-409).
9.
SauroJ.KindlundE. (2005b) “How long Should a Task Take? Identifying Specification Limits for Task Times in Usability Tests” in Proceeding of the Human Computer Interaction International Conference (HCII 2005), Las Vegas, USA.
10.
SauroJ.KindlundE. (2005c, July). Using a single usability metric (SUM) to compare the usability of competing products. In Human Computer Interaction International Conference.
11.
Van WaardhuizenM.McLean-OliverJ.PerryN.MunkoJ. (2019, May). Explorations on Single Usability Metrics. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (pp. 1-8).
