Ecological Validity in Laboratory Based Usability Evaluations

Abstract

An interview based survey, looking at the suitability of a laboratory facility for usability testing, raised the issue of ‘ecological validity’. Ecological validity refers to the extent to which the test environment mirrors the environment in which a product would be used in ‘real life’.

Ten ergonomists, nine of whom used the laboratory, were interviewed. Opinions of the laboratory were generally positive; indeed there was a consensus as to the high value of the facility in the ergonomists' work. However, only one of those asked felt that the laboratory provided an ecologically valid testing environment. Initially, this result seems surprising; if the conditions don't provide ecological validity, this would appear to be a limitation on the laboratory's value.

This paper considers the concept of ecological validity; discussing the contexts in which it is important and those in which it is not a priority. The extent to which it can be achieved in the laboratory and how it could be achieved are also considered. Generally, the ergonomists were fairly pessimistic about the prospects for this.

All, bar one, of the ergonomists interviewed also conducted studies outside of the laboratory. The part played by ecological validity in deciding to evaluate in the field is discussed; at what point does the issue become important enough to force evaluations outside of the laboratory?

Five non-ergonomists from the same organisation were also interviewed. Interestingly, they seemed to give ecological validity a higher priority than ergonomists, in terms of influencing the overall value of an evaluation. They were also more inclined to expect ecological validity to be achievable in the laboratory — by using furniture to create an appropriate range of scenarios. This suggests that, even if scenario creation has no real effect on ecological validity (this was the opinion of many of the ergonomists), it may bring ‘propaganda’ benefits in terms of influencing commissioners' attitudes towards ergonomists' work.