Reducing Sample Sizes When User Testing with People Who Have,and Who are Simulating Disabilities - Experiences with Blindness and Public Information Kiosks

Abstract

Researchers have suggested running fewer subjects to reduce costs of usability testing of mainstream products (Virzi 1990, 1992, Nielsen and Landauer, 1993, Lewis, 1994). It has been demonstrated that, depending on the average probability of subjects revealing problems (p), most of the problems inherent in an interface will be found in the first few subjects, and adding more subjects gives little return on investment. As part of an investigation aimed at reducing costs in user testing of people with disabilities, a user test was conducted to compare the differences between a group of 15 blind, and 15 blindfolded (sighted) subjects using a touchscreen public information kiosk that was intended for use by people who cannot see. The number and type of problems found by each group were compared, and it was found that the results between each group were mostly similar: the value of p was within 3% (blind p=0.31; blindfolded p=0.34); each group found 46 of a total 53 problems (and therefore did not find 7 problems that the other group found, 6 of which were low frequency problems); 80% of the problems were found by the first 6, and the first 5 subjects of the blind and blindfolded groups respectively; the number of problems found in four evaluation categories were also similar. The need for further study in this area, in terms of tests using different disability types and other types of subjective and objective measures, and possibilities for using mixed subject pools of people who have and people who are simulating disabilities, are discussed.