In this paper we propose a new framework for evaluating designs based on work domain analysis, the first phase of cognitive work analysis. We develop a rationale for a new approach to evaluation by describing the unique characteristics of complex systems and by showing that systems engineering techniques only partially accommodate these characteristics. We then present work domain analysis as a complementary framework for evaluation. We explain this technique by example by showing how the Australian Defence Force used work domain analysis to evaluate design proposals for a new system called Airborne Early Warning and Control. This case study also demonstrates that work domain analysis is a useful and feasible approach that complements standard techniques for evaluation and that promotes a central role for human factors professionals early in the system design and development process. Actual or potential applications of this research include the evaluation of designs for complex systems.
Get full access to this article
View all access options for this article.
References
1.
Bisantz, A. M., & Vicente, K. J. (1994). Making the abstraction hierarchy concrete. International Journal of Human-Computer Studies, 40, 83–117.
2.
Burns, C. (2000). Putting it all together: Improving display integration in ecological displays. Human Factors, 42, 226–241.
3.
Charlton, S. G., & O'Brien, T. G. (1996). The role of human factors testing and evaluation in systems development. In T. G. O'Brien, & S. G. Charlton (Eds.), Handbook of human factors testing and evaluation (pp. 13-26). Mahwah, NJ: Erlbaum.
4.
Czaja, S. J. (1997). Systems design and evaluation. In G. Salvendy (Ed.), Handbook of human factors and ergonomics (pp. 594–636). New York: Wiley.
5.
Department of Defence. (1995). The capital equipment procurement manual (CEPMAN).Canberra, Australia: Defence Publishing Service.
6.
Department of Defence. (1999). The defence procurement policy manual (DPPM).Canberra, Australia: Defence Publishing Service.
7.
Dinadis, N., & Vicente, K. J. (1999). Designing functional visualizations for aircraft systems status displays. International Journal of Aviation Psychology, 9, 241–270.
8.
Gabb, A., & Henderson, D. (1995). A review of navy's technical and operational evaluation practices (Defence Science and Technology Organisation Report No. DSTO-TR-0194). Salisbury, Australia: Electronics and Surveillance Research Laboratory.
9.
Gabb, A., & Henderson, D. (1996). Technical and operational tender evaluations for complex military systems (Defence Science and Technology Organisation Report No. DSTO-TR-0303). Salisbury, Australia: Electronics and Surveillance Research Laboratory.
10.
Leveson, N. G. (2000). Intent specifications: An approach to building human-centered specifications. IEEE Transactions on Software Engineering, 26, 15–35.
11.
Malone, T. B. (1996). Human factors test support documentation. In T. G. O'Brien, & S. G. Charlton (Eds.), Handbook of human factors testing and evaluation (pp. 101-116). Mahwah, NJ: Erlbaum.
12.
Meister, D. (1986). Human factors testing and evaluation. In G. Salvendy (Ed.), Advances in human factors/ergonomics (pp. 313-322). Amsterdam: Elsevier Science.
13.
Meister, D. (1996). Human factors test and evaluation in the 21st century. In T. G. O'Brien, & S. G. Charlton (Eds.), Handbook of human factors testing and evaluation (pp. 313-322). Mahwah, NJ: Erlbaum.
14.
Naikar, N., Drumm, D., Pearce, B., & Sanderson, P. M. (2000, November). Designing new teams with cognitive work analysis. Paper presented at the 5th Australian Aviation Psychology Symposium, Manly, Australia.
15.
Naikar, N., Lintern, G., & Sanderson, P. M. (in press). Cognitive work analysis for air defense applications in Australia. In M. D. McNeese, & M. A. Vidulich (Eds.), Cognitive systems engineering state of the art report.Wright-Patterson Air Force Base, OH: Human Systems Information Analysis Center.
16.
Naikar, N., & Sanderson, P. M. (1999). Work domain analysis for training-system definition and acquisition. International Journal of Aviation Psychology, 9, 271–290.
17.
O'Brien, T. G. (1996). Preparing human factors test plans and reports. In T. G. O'Brien, & S. G. Charlton (Eds.), Handbook of human factors testing and evaluation (pp. 117-134). Mahwah, NJ: Erlbaum.
18.
Pejtersen, A. M., & Rasmussen, J. (1997). Systems design and evaluation. In G. Salvendy (Ed.), Handbook of human factors and ergonomics (pp. 1514-1542). New York: Wiley.
19.
Perrow, C. (1984). Normal accidents: Living with high-risk technologies.New York: Basic Books.
20.
Pool, R. (1997). Beyond engineering: How society shapes technology.New York: Oxford University Press.
21.
Rasmussen, J. (1998). Ecological interface design for complex systems: An example: SEAD-UAV systems (AFRL-HE-TR-WP- 1999-0011). Wright-Patterson Air Force Base, OH: Human Effectiveness Directorate, Crew Systems Interface Division.
22.
Rasmussen, J., Pejtersen, A., & Goodstein, L. P. (1994). Cognitive systems engineering.New York: Wiley.
23.
Reason, J. (1990). Human error.Cambridge, England: Cambridge University Press.
24.
Reising, D. V. C., & Sanderson, P. M. (1996). Work domain analysis of a pasteurization plant: Using abstraction hierarchies to analyze sensor needs. In Proceedings of the Human Factors and Ergonomics Society 40th Annual Meeting (pp. 293-297). Santa Monica, CA: Human Factors and Ergonomics Society.
25.
Sanderson, P. M. (2000, November). Cognitive work analysis across the system life-cycle: Achievements, challenges, and prospects. Paper presented at the 5th Australian Aviation Psychology Symposium, Manly, Australia.
26.
Sanderson, P. M., Naikar, N., Lintern, G., & Goss, S. (1999). Use of cognitive work analysis across the system life cycle: Requirements to decommissioning. In Proceedings of the Human Factors and Ergonomics Society 43rd Annual Meeting (pp. 318-322). Santa Monica, CA: Human Factors and Ergonomics Society.
27.
Vicente, K. J. (1995). Supporting operator problem solving through ecological interface design. IEEE Transactions on Systems, Man, & Cybernetics, 25, 529–545.
28.
Vicente, K. J. (1999). Cognitive work analysis: Towards safe, productive, and healthy computer-based work.Mahwah, NJ: Erlbaum.
29.
Walsh, P., Lim, K. Y., & Long, J. B. (1989). Jackson system development and the design of the user interface software. Ergonomics, 32, 1483–1498.
30.
Whitefield, A., Wilson, F., & Dowell, J. (1991). A framework for human factors evaluation. Behaviour and Information Technology, 10, 65–79.
