Extraneous Factors in Usability Testing: Evidence of Decision Fatigue During Sequential Usability Judgments

Abstract

Previous work has examined if cognitive fatigue caused subjects to negatively evaluate products on subjective usability measures but failed to find an effect. The authors suggested that methodological issues may have caused variance, which masked any experimental effect. This study tested that claim by eliminating the extra source of variance. Twenty-six participants voted using six prototype paper voting ballots. The ballots were randomly assigned, with three used before a fatigue manipulation and three after. All the ballots were of a similar usability profile as determined in previous studies. Each ballot was used and evaluated by the subjects using the System Usability Scale (SUS) and the pre-and post-fatigue means were compared, with no significant differences found. Although the reduction in variance did not affect the results, it was discovered that sequential usability judgments may result in decision fatigue. Subjective usability judgments may become increasingly negative with each additional judgment. Evidence from other experiments support this trend and further research may be warranted. Implications for usability researchers and practitioners are discussed.