The Metrics Problem in the Study of Cognitive Work and a Proposal for a Family of Solutions

Abstract

We present a measure called i-bar, which is the inverse of the mid-range derived from data on trials-to-criterion. We interpret this measure as a conjoint measurement scale, permitting evaluation of: (1) the sensitivity of the principal performance measure (which is used to set the metric for trials to criterion), (2) the learnability of the work method (i.e., the goodness of the software tool), and (3) the resilience of the work method. It is possible to mathematically model such order statistics, and derive methods for estimating likelihoods. This proposal involves novel ways of thinking about statistical analysis for discrete non-Gaussian distributions. The method we present should be applicable to the study of the effects of any form of training, and any form of intervention, whether to improve legacy work methods or create new cognitive work systems.

Get full access to this article

View all access options for this article.

References

Ballas

J. A.

(2007). Human centered computing for tactical weather forecasting: An example of the “Moving Target Rule.” In Hoffman

R. R.

(Ed.), Expertise out of Context: Proceedings of the Sixth International Conference on Naturalistic Decision Making (pp. 317–326). Boca Raton, FL: CRC Press.

Book

W. F.

(1908). The psychology of skill with special reference to its acquisition in typewriting. Studies in Psychology, Vol. 1. Missoula, MT: University of Montana Press.

Dekker

S. W. A.

Nyce

J. M.

Hoffman

R. R.

(March-April 2003). From contextual inquiry to designable futures: What do we need to get there? IEEE Intelligent Systems, pp. 74–77.

Hoffman

R. R.

Elm

W. C.

(2006, January/February). HCC implications for the procurement process. IEEE: Intelligent Systems, pp. 74–81.

Larsen

R. J.

Marx

M. L.

(2001). An introduction to mathematical statistics and its applications (3rd. ed.). Englewood Cliffs, NJ: Prentice Hall.

Miller

G. A.

, (1986). Dismembering cognition. In Hulse

S. H.

Green

B. F.

Jr. (Eds.) One hundred years of psychological research in America (pp. 277–298). Baltimore, MD: Johns Hopkins University Press.

Neville

Hoffman

R. R.

Linde

Elm

W. C.

Fowlkes

(2008, January/February). The procurement woes revisited. IEEE Intelligent Systems, pp. 72–75.

O'neill

M. J.

(2007). Measuring workplace performance (2nd. ed.). New York: Taylor and Francis.

Schaffer

(2004). Institutionalization of usability. Boston: Addison-Wesley.

10.

Shepherd

(2001). Hierarchical task analysis. London: Taylor and Francis.

11.

Still

D. L.

Temme

L. A.

(2006). Configuring desktop helicopter simulation for research. Aviation Space and Environmental Medicine, 77, 323.