How Many Participants: A Simple Means for Concurrent Monitoring

Abstract

User trialling, particularly in a design context with tight time and budget constraints, begs the question what makes up a sufficient number of participants for observing proportionally enough of the phenomena at issue; e.g. usability problems. Statistical approaches such as estimating the number of species do not seem to be applicable since the required, mathematically ‘neat’ sampling conditions do not match the gathering of observations with consecutively involved participants in a user trial. Therefore, we resorted to the well-known binomial model, precipitating (without sampling restrictions) an anticipated increase in overlap, i.e., a rising proportion of shared observations between participants in an ongoing trial, or, in other words, diminishing returns in terms of unique observations. In Ergonomics/Human Factors (E/HF) literature, the application of the binomial model has given rise to retrospective assessments involving the number of participants that would have been enough, by hindsight, to discover e.g. 80% of all usability problems, which, by reference to case studies, eventually gave rise to the rule of thumb that about five participants are sufficient. The present paper summarises and extends two earlier papers in providing a simple statistic in order to monitor concurrently the proportion of information gained so far in a trial. Careful scrutiny is given to the origin of estimates being biased downward, that is: the underestimation of the asymptotic number of usability problems to be discovered given the observations after a number of participants. On the basis of both hypothetical examples and empirical studies it is shown that the ‘five-is-enough’ rule of thumb may hold, but may equally well be much too optimistic. With the proposed statistic for concurrent monitoring, it can arguably be decided on whether or not to continue a trial.