A Theory of Human Performance Assessment

Abstract

Selection of a measure of effectiveness (MOE) (a mathematical function) and using that measure to evaluate performance demonstrations (or exercises, or experimental trials) without first testing the measure, typically results in a disagreement between two ways of assigning effectiveness scores to each performance demonstration. The two ways of assigning effectiveness scores to each performance demonstration are: effectiveness scores assigned directly by the investigator and effectiveness scores assigned by the MOE selected by the investigator. The disagreement often exists even when comparing the rank ordering of the two sets of scored performance demonstrations.

A disagreement between the two methods means that one method, possibly both, are not correct. The direct assignment of effectiveness scores to each performance demonstration constitutes a test of the MOE. In this paper, we argue that test is typically not conducted and if it were, the MOE (existing untested MOE's) would likely fail the test. We also argue that the investigator should not select an MOE but rather should have an authority (SME) score performance demonstrations and then synthesize an MOE that will pass the test. A method for synthesizing the MOE is presented.