Proceedings of the Human Factors and Ergonomics Society Annual Meeting

Everyone involved in quantifying the human part of the defense equation has long been frustrated by lack of consistent, “reproducible” and comprehensive data. This void is especially evident in the training and performance data area. The Defense Training and Performance Data Center (TPDC) was established to help redress this problem.

The Occupation Data Base under development at TPDC consists of three files. The first file describes military occupations, providing information about the attributes of the occupation. It addresses the question: what does the occupation look like? The second file describes occupations from a “demographic” perspective, or more simply put, answers the question: who are the people performing the job? It provides a statistical profile (current and historical) of the characteristics, experience, and behavior of the people serving in each military occupation. The third file provides a data file oh training programs, resources, methods and media as these elements relate to military occupations. All occupations within the enlisted, warrant officer, and officer communities are covered in each service, both active and reserve components. These three files are being built from current and historical data and will be updated regularly, thereby providing a comparative perspective with which senior service planners can make informed decisions on defense training issues.

The purpose of this research was to formulate a unitary measure of performance for simulated one-versus-one, within visual range, air-to-air combat. The measure will serve as a criterion for the development and validation of specific measures of ACM skill that can be used to provide diagnostic performance feedback to pilots. Two experiments were conducted in which fighter pilots served as judges and rank-ordered, from most to least desirable, hypothetical ACM engagement outcomes. Outcome variables included (1) whether or not the hypothetical pilot achieved a “kill,” (2) whether or not he survived the mission, (3) the percent of time the pilot was in an offensive, defensive, or neutral posture, (4) length of engagement, and (5) posture at the beginning and end of the engagement (offensive, defensive, or neutral). In order to determine inter-rater agreement among judges in Experiment I, their rankings were correlated. Correlations ranged from .93 to .99. Pilots' rankings of engagement outcomes were subjected to linear regression analyses to derive equations that could be used as a unitary measure of ACM success. The regression equation in Experiment I accounted for 95% of the variance in rankings, and the composite regression model calculated in Experiment II accounted for more than 70% of the variance.

Intelligent tutoring systems offer an exciting new way to train people in areas of complex domains. A simulation-based training system provides the student with the opportunity to manipulate a system without the consequences of real life mistakes. The intelligence required in the tutoring system is focused on the tutor's ability to teach the student efficient, strategic responses. This tutoring demands that the tutor is aware of the student's current ability, specific fault areas, and preferred method of tutoring. Instructional decisions are made by assessing the student's performance. The utility of an intelligent tutoring system depends on its capacity to evaluate the student's performance. Performance assessment then has significant impact on the employment of such a system. The parameters used for performance assessment of a complex task depend on the objective of the tutoring system. We present a description of a generic intelligent tutoring system which will remove the human instructor from the training loop.

Task performance can directly impact on unit/system effectiveness and also indirectly by limiting the performance of other tasks. When assessing the impact of candidate systems designs, training programs, changes in doctrine, MOS staffing etc., the total impact of task performance must be known. A method for calculating the total impact i.e., both direct and indirect effects, is described here in terms of the unit/system/task relationship.

A Moment-to-Moment (MTM) performance measure provides an assessment of the performance of each system task during a mission. The assessment is the total (direct and indirect) impact of task performance on unit/system effectiveness, based on the unit/system measure of effectiveness (MOE). The task performance assessment is calculated by taking into account the performance coupling of each task with those of other tasks. With a MTM measure, task performance is assessed according to its total impact on unit/system effectiveness and not by considering the task to be performed in isolation. The MTM measure can be used to accomplish the following: select tasks for training, establish task performance measures and levels of acceptable performance; evaluate proposed training programs, equipment designs, and procedures; and design system test and evaluation strategies. The paper describes the method of developing MTM measures as applied to measures for maneuvering units.

This paper describes methods for developing automated performance measurement systems used with training ranges and simulators. A prototype automated measurement system designed to assess aircrew performance during strike warfare training is presented as an application of this methodology. Methods are also presented for displaying information which is useful in assessing student progress and for diagnosing training results.

In this paper we briefly highlight relevant laboratory research that provided the theoretical and empirical underpinnings for the development of a task-analytic training methodology. The actual task-analytic methodology, developed to decompose tasks performed to support tactical command and control (C²), air-weapons controller missions, is briefly discussed. The present paper provides the necessary background for the actual application of the methodology. The details of the direct application are presented in a companion paper by Eggemeier, Fisk, Robbins, and Lawless (1988).

Automatic/controlled processing theory maintains that the consistent components of complex skills improve most substantially with training, and that part-task training (PTT) programs should therefore focus on the consistent elements of skilled performance. As an initial step in applying automatic processing principles to PTT, Fisk and Eggemeier (1988) developed a task analytic methodology to identify the consistent components of complex skills. This paper describes the application of the methodology to operator skills required in a complex tactical command and control (C2) system. The results indicate that the methodology can be used to identify a variety of consistent components of complex performance. Possible applications to PTT and issues for future training research are discussed.

This paper describes current developments in automating the processes to author technical information (TI) and deliver it using microcomputers. It describes desirable characteristics which support the presentation of TI for technicians varying in skill levels. Addressed are human factors issues relating to information access, user acceptance, and display formats. Programming is being done in Smalltalk/V, an object oriented language, on a Zenith 248 computer, which is compatible with the IBM PC/AT. The project emphasizes low cost authoring and delivery of information which traditionally has been contained in paper technical manuals. Our intent is to support the Department of Defense initiative to shift from paper to paperless technical manuals.

This paper focuses on the interactions among four constructs during skill acquisition: (1) the dynamic changes in attentional demands of the task to be acquired, (2) individual differences in cognitive and intellectual abilities, (3) conative (motivational), metacognitive processes involved in changes of attentional focus, and (4) knowledge structures acquired through part-task training. An attentional model is reviewed that describes how these variables interact during three phases of skill acquisition (i.e., during declarative knowledge, knowledge compilation, and at the level of proceduralized knowledge). Empirical demonstration of the framework is provided in the context of complex skill acquisition. Supportive results from a series of empirical studies are reviewed.

Formative evaluation in the broadest sense refers to the measurement of some system in order to make direct and immediate differences in the procedures, mechanisms, and goals of that system during development. The objective of this formative evaluation is to address three areas: 1) increase our understanding of how the targeted users make decisions, 2) train the user about how the system makes decisions and presents information, and 3) develop information about interface and modeling changes needed in the system. What is needed in the design of decision aids is an understanding of both how the user perceives the decision process, and how the decisions actually could be effectively made. Both of these issues must be addressed before the user will accept and use the decision aid. The system also needs to be able to accept and use the information that the user considers necessary, as well as to present both recommendations and supporting information in acceptable formats. We have applied a structured interview within our formative evaluation process as a basis for integrating the user in the develop, revise, and deliver cycle. The structured interview was conducted on-line, demonstrating what the system does while explanations of how it works are provided. The responses have provided information about whether the user thinks the system addresses the correct issues, the users agreement with the system analyses, and a report of the users decision processes. By including the user in the review of the developing system, the design of the prototype more accurately reflects the user's decision processes, as well as providing more usable output. This study will provide some insight about one method for evaluating decision aids early in the development process.

Manpower, personnel, training and safety (MPTS) analysis is currently inconsistent and incomplete in its application to Air Force acquisitions. While many problems are managerial, MPTS analysis also suffers from a lack of adequate tools and data bases to analyze weapon system design, project MPTS requirements and suggest trade-offs. These problems are particularly acute in the early phases of the weapon system acquisition process (WSAP), making it difficult to influence design during this critical period. To improve MPTS analysis, a study is being conducted by the Air Force to define a comprehensive and integrated MPTS analytic system for use in the WSAP. The study is identifying what MPTS decisions need to be made in the WSAP, when they need to be made, and how they are interrelated. Current capabilities are then being determined by analyzing the tools and data bases available to support each MPTS decision. The analysis will identify needed improvements to existing tools and data bases and whether new ones need to be developed. This paper discusses the requirement for an improved MPTS system and then describes a variety of managerial and technical initiatives being undertaken to satisfy the requirement. The last section identifies the desired characteristics of an effective MPTS system.

This paper describes the lessons learned from applying embedded training guidelines currently under development to a testbed system. The testbed system was the All Source Analysis System/Enemy Situation Correlation Element (ASAS/ENSCE). ASAS/ENSCE is an emerging computer-based information system. The major result of this effort was the modification of the ET guidelines to better support embedded training development.

The effects of four instructional issues and one simulator design feature for training helicopter shipboard landing on small ships were tested in the Vertical Take-off and Landing Simulator (VTOL) at the Visual Technology Research Simulator (VTRS), Naval Training Systems Center. They were: (1) field of view (VTRS versus a test field of view), (2) task chaining (segmented backward chaining versus whole task training), (3) augmented cueing (augmented cueing versus no augmented cueing), (4) length of training (18, 27, and 36 trials), and (5) the timing of seastate introduction (early versus late). The experiment utilized an in-simulator transfer-of-training paradigm in which pilots who were not proficient in the helicopter shipboard landing task were trained under one of several experimental conditions, then tested on the transfer condition (that represented maximum realism) in the simulator. Thirty-two pilots each completed a total of 54 trials (36 training, 18 transfer). Pilots were tested in the transfer condition (six trials) after their 18th, 27th, and 36th training trial. Of the experimental instructional issues, task chaining had the largest effect, with better performance in all segments of the task for pilots who were trained with the backward-chaining sequence, than for pilots who received whole task training. Augmented cueing did not yield the transfer performance anticipated. Seastate introduction had no effect on performance. Field of view had some marginal effects on vertical performance in the hover, with better performance for pilots who were trained with the combination VTRS field-of-view and backward-chaining. Results suggest a diminished rate of learning after 33 simulator trials (includes 27 training trials and six transfer trials of the first probe).

Fidelity of task structure is proposed to replace physical fidelity, as a cost effective guiding principle in the development of trainers for complex tasks. The paper describes a research project in which a complex computer game was designed to mimic the requirements of flight skill. It was then employed as part trainer within an actual flight training program. The game was mainly directed to teach subjects strategies of attention control and efficient allocation of processing resources. It was found to improve trainees performance in flight. The paper discusses the theoretical roots of the proposed approach and some results of its application in an actual training environment.

Transfer between functionally isomorphic devices can be viewed as a kind of analogical mapping. In this research subjects learned to operate a computer-simulated device and then transferred to a functionally-equivalent device, either immediately or after a delay of one week. Two factors were varied: the systematicity, or causal coherence, of the original device model; and the transparency, or degree of surface similarity between corresponding components in the two devices. The results showed effects of delay, systematicity and transparency. Transfer performance was better in the immediate than in the delayed condition. Both systematicity and transparency improved performance in both immediate and delayed conditions.

The purpose of this paper is to demonstrate the measurement of learning and transfer using a curve-fitting technique discussed in a 1985 Human Factors article by Spears. The data were collected during an experiment that determined if rotation skills could become automated with practice and if the skills could transfer between stimuli. The dependent variables of interest were the slope and intercept of the regression equation relating correct reaction time and degrees of rotation. Curve fitting was accomplished using a common statistical package, BMDP, and an IBM-XT. The curve-fitting technique showed large initial transfer of training on several variables that did not affect the asymptotic level of performance. In contrast standard transfer of training calculations indicated small positive transfer.

Two experiments are presented that examine automaticity and transfer of mental rotation skills. The data from these experiments were analyzed using a curve-fitting technique that represents a departure from traditional methods used to analyze transfer. The first experiment demonstrated significant positive transfer from one letter stimulus to another. The second experiment examined the transfer of rotation skills from a letter to an abstract shape. This experiment demonstrated a reduced amount of transfer from letters to an abstract shape. Both experiments indicated that mental rotation skills may become automatic with practice.

The paper describes an investigation of the effects of computer-based training (CBT) delivery methods on learning and retention in target recognition and identification tasks. Two training methods (drill and practice, and branching computer aided instruction) were administered to subjects at two locations. Measures of cognitive complexity also were obtained on each subject and compared against test scores. Results confirmed that learning did occur with both CBT methods. Results also suggest that training design is not straightforward, and must consider a number of student and training site factors as well as the topic to be trained. Trends, while not highly significant, do suggest directions for further research.

Conceptual graphs are a method of diagraming knowledge structures. They are valuable in a variety of applications including design of instructional materials, student evaluation and diagnosis, and theoretical research. One problem has been the identification of methods for eliciting or measuring the “internal” knowledge structures to develop conceptual graphs.

Several methods of mapping conceptual graphs have been used in various research domains (e.g., free recall, sorting, ordering, etc.). Most of these traditional methods suffer from serious drawbacks. To overcome these problems, a question probe method has been adapted from work done in the area of prose comprehension. The question probe technique is a method for systematically developing a set of “test questions”. The method allows for the direct development of conceptual graphs and quantitative comparison of the graphs.

The question probe technique has been successfully used to systematically develop instructional materials, evaluate/diagnose student understanding of course material, evaluate alternative instructional methods, and assess the relationship between instruction and problem solving performance.

Reported is an experiment which was designed to (1) test the strength-precision relationship of grip and shoulder strength with competitive pistol shooting scores using a nationally ranked collegiate pistol team and (2) determine the effects of a very brief, specifically focused, intense grip and shoulder strength training program (10 min, 3 times per week for 8 weeks) on strength and shooting performance. All members of the Pennsylvania State University Navy ROTC Pistol Team (n = 12) were divided into two matched groups according to pre-test shooting scores. Both groups received identical shooting instruction and practice, but the training group participated in an eight-week strength development program while the control group did not. Maximum isometric strength (peak and 4-sec average force) measures were obtained from electronic output of a hand dynamometer and strain gauge (deltoid contraction from a lateral horizontal shooting position against an arm cuff). Shooting scores (slow fire, timed fire, rapid fire, and total) were used as performance measures.

The 24 intense but brief exercise bouts did not produce group training effects, despite painstaking efforts to use (1) exercises documented to produce training effects, (2) subjects who were highly motivated, (3) accurate and reliable strength testing equipment, and (4) a program which exercised to fatigue each day the target muscles. This finding is particularly alarming because of the strong body of literature suggesting the effect should have occurred. It also suggest that the time spent in strength training might better be spent in shooting. Nonetheless, robust strength-performance correlations were obtained, particularly in two areas: (1) slow fire shooting scores with grip strength and (2) timed and rapid fire with deltoid strength. Furthermore, individual subject analysis revealed that those that improved in strength also improved in shooting skill. This training was effective for improving strength and scores in some subjects, especially in producing increases in deltoid strength concomitant with improvements in timed and rapid fire scores. This work has implications for all types of free-limb shooting activities, for military and civilian populations, and for novice and elite performers.

Three-dimensional objects placed on simulated terrain surfaces are particularly effective as cues for altitude in simulated low-level flight. To conserve the limited edge processing capacity of computer image generators (CIGs), objects have typically been simple in shape and therefore fairly abstract in appearance. The present investigation sought to determine whether the apparent size of more detailed and familiar appearing objects (e.g., trees and bushes) serves as an additional cue for altitude in simulated low-level flight. Results showed no differences in performance between abstract objects and familiar objects. However, performance did improve with increases in object density, at least for some performance measures. These results suggest that CIG processing capacity may be most effectively utilized by increasing object density rather than individual object detail.

This paper describes several lessons learned in the process of getting Computer Conferencing to work for Army Training. One methodological lesson was to better assess the maturity of the phenomenon to be studied and utilize formative evaluation rather than experimentation for immature areas of study. Another lesson was to not over-borrow from prevailing conceptual frameworks as they may be inappropriate for a new context. Some specifics, in case study form, are given to illustrate these lessons learned.

This paper discusses considerations for designing instructional materials for the computer mediated classroom. Specifically, the topics of group versus individual instruction, information processing, media/method tradeoffs, and associated costs were addressed.

This paper explores the issues and dilemmas faced in designing and delivering instructor training for Army engineers to teach in a computer mediated classroom. It presents an account of the design and development and implementation processes, including the evaluation methodology and lessons. Differences between the roles of instructional staff in the computer conference and traditional school environment are also discussed.

As part of a computer mediated conferencing research project, a user interface was developed for the software database management programs. A process of identifying an appropriate interface is discussed as well as the factors which the researchers used to develop the interface. User needs and expectations are identified as the major factors for interface design, and methods of evaluating needs and expectations are discussed. Limitations of designing the interface to suit the particular computer conferencing course are also presented.