How to Make T&E Results Usable and Truly Useful

Abstract

The rationale for a discipline of test and evaluation (T&E) is that systems are developed to achieve certain purposes. This is true whether the systems in question are purely hardware, software, training, socio-political, or some combination of these or still other system types. The major objectives of test and evaluation are to determine whether the systems achieve their purposes and, if they do not, to determine how they can be improved. The human factors results of this effort must be both usable and useful if the T&E practitioner is to have credibility.

The usability of T&E results is a function of the adequacy of the methods used for collecting and analyzing data in the context of real-world constraints, a world in which there may be frequent departures from an ideal test and evaluation environment. The measurement problems inevitably encountered include those associated with the disparity between the engineering and behavioral science disciplines, the complexities of experimental control in operational situations and operational fidelity in non-operational settings, and the absence of direct translations between human performance and the performance of the larger system of which the human is a part.

The usefulness of T&E results is a function of how relevant they are to the decision makers who are managing the system development program, as well as those other individuals who are designing, developing, operating, maintaining, or using the outputs of the system. Unlike the human factors specialist, the T&E “customer” is generally not concerned with a search for the truth. Rather, the customer frequently needs only that information necessary to answer very specific questions. We will fail in our responsibility as human factors professionals if we do not provide the type of information, in the proper format, that will influence the decisions and behavior of the user of that information.

Both the usability and the usefulness of human factors test and evaluation efforts must be properly addressed by T&E researchers and practitioners. This is true if for no other reason than for the sheer pragmatic need to get continuing support and funding for our discipline.

The first two papers in this symposium will emphasize the usability of results in the real-world context of the T&E environment and the last two will emphasize the usefulness of the results to the sponsor or user. The participants have had multiple opportunities to reconcile and more clearly address differences in the positions they espouse through exchanges and critiques of their respective papers.