AB-47D - Symposium

Behavioral and Cognitive Task Analysis Integration For Assessing Individual and Team Work Activities

Teresa Brenner
The Institute for Job and Occupational Analysis
San Antonio, TX

Kathleen Sheehan
Winfred Arthur, Jr.
Texas A&M University
College Station TX

Winston Bennett, Jr.
Warfighter Training Research Division
U.S. Air Force Research Laboratory
Mesa AZ

Introduction

Traditionally, the same type of task analysis procedures has been used for all types of jobs. Recent changes in the workplace include the increased focus on the cognitive demands of tasks and the increased use of work teams to accomplish these tasks. Researchers have discussed the implications of these trends for task analysis procedures in organizations today.

Cascio (1995) noted an organizational change away from task-based work to more of a process. Jobs are no longer defined by a limited number of tasks, requiring the employee to accomplish a range of tasks that may change over time. As a higher proportion of jobs are focusing on troubleshooting activities, cognitive task analysis may be more appropriate for identifying strategies involved in effective performance.

Another trend involves workers participating as a member of a team which requires sharing information, collaboration, and communication to produce a group outcome. Individuals may play a large or small role in the team process as well as individual responsibilities. Additionally, individuals may also serve on more than one team. Researchers must consider these issues when trying to describe the tasks of team members.

In addition to these trends, some researchers have made a move towards examination of competencies, which may differ for individuals and teams (Cannon-Bowers, Tannenbaum, Salas, & Volpe, 1995). Although competencies share some similarities with traditional knowledge, skills, and abilities (KSAs), some distinguishing characteristics exist between the two frameworks. For example, competencies include knowledge, skills, and attitudes, rather than traditional knowledge, skills, and abilities. Though the knowledge component is similar, competencies include a broader concept of skills than KSAs, by incorporating both a psychomotor and a cognitive component. The concept of abilities is replaced by attitudes in the competencies framework, which are predicted to impact performance (Cannon-Bowers et al., 1995).

Team competencies can be divided into team-specific and task-specific competencies. Team-specific competencies may only apply to a particular team, yet encompass all tasks the team performs. Task-specific competencies may only apply to certain tasks. Cannon-Bowers et al. (1995) revealed 11 knowledge requirements, 8 specific skill dimensions, and 9 attitude requirements for a team. Stevens and Campion's (1994) research appears to corroborate these findings.

Traditional Task Analysis

Traditional task analysis methods serve as a basis for performing work effectively. A task analysis defines a job in terms of KSAs necessary to perform daily tasks. The product includes a list of tasks used to describe the job in terms of relative time spent, difficulty of learning, criticality, importance, and frequency (Dipboye, Smith, & Howell, 1994). This is accomplished by using critical incident techniques, observations, interviews, questionnaires, expert judgments, and self-report (Dipboye et al., 1994). Advantages to this method include the ability to compare different positions, to determine necessary traits and ability, and to serve as the basis in all aspects of Human Resources’ decisions (Dipboye et al., 1994).

Cognitive Task Analysis

Developments in the workplace have allowed the increasing automation of tasks, moving the central focus of many jobs to more strategic and troubleshooting activities. Cognitive task analysis seeks to delineate the mental processes and skills needed to perform a task at high levels of proficiency (Ryder, Redding & Beckshi, 1987). Further, cognitive task analysis depends on uncovering the cognitive demands and how novice and experts respond to these demands (Roth, Woods, & Pople, 1992).

Klein (1993) stated, "Cognitive Task Analysis is directed at the psychological processes underlying the performance… and the subtle cues that may depend on context and experience" (p. 88). In general, the function of cognitive task analysis is to define the actual decision requirements of the task (Klein, 1993). Regardless of the method used, cognitive task analysis should include the following steps: (1) mapping out the task using task analysis; (2) identifying the critical decision points; (3) clustering and linking the decision points; (4) prioritizing the decision points; and (5) diagnosing and characterizing the decisions as to the strategies used, cues signaling the decision points, and the inferences made regarding cues and decision points.

There are some key differences between task analysis and cognitive task analysis (Klein, 1993). In general, task analysis focuses only on observable behavior and does not offer information on overall organization of knowledge. Cognitive task analysis, on the other hand, is directed at the psychological processes underlying the behavior. Cognitive task analysis concentrates on the critical decisions and cognitive processes that separate the expert from the novice. Further, cognitive task analysis provides a description of the overall knowledge organization by looking at the interrelationship between concepts associated with the job. Another benefit of cognitive task analysis is the identification of changes in knowledge structures and mental processes when progressing from novice to expert. Redding (1989) indicated that the following components are essential to cognitive task analysis: assessing individual abilities, assessing changes in knowledge base, identifying task components, identifying differences between novices and experts, identifying the conceptual and procedural knowledge of similar components, and specifying the conditions which best facilitate progression from one knowledge state to another.

Klein (1993) identified four broad classes of cognitive task analysis, including questionnaires and interviews, controlled observation, critical incidents, and analytical methods. Although commonalities can be found among cognitive task analysis methodologies, they all vary with respect to how they elicit expert knowledge, represent expert knowledge, and use the tasks in question to bring about expert performance. The first method, questionnaires and interviews, are standard task analysis techniques that can also be used to probe the cognitive processes underlying observable performance. These techniques can be used to identify cues, goals, options, reasons of choice, knowledge, and help needed to perform the task. These techniques are very popular, easy to implement, and can be used with other cognitive task analysis methods to produce a variety of information. However, these techniques may generate general or idealized information regarding performance rather than cutting through details and contextual nuances (Klein, 1993).

Controlled observation, the second method of cognitive task analysis, uses verbal protocol analysis when experts are instructed to think out loud while performing the task or instructed to provide a retrospective account after completing the task. Another approach to controlled observation is withholding information from the experts while they perform the task to see how they respond to the lack of information and what kinds of questions they might ask. An advantage of controlled observation is that key features of the task can be controlled and manipulated. Additionally, this technique allows for data collection by a computer in addition to an observer. However, this technique may not uncover new factors related to effective performance, since the researchers have prepared the scenarios ahead of time (Klein, 1993; Nisbett & Wilson, 1977). Thus, this method may be more useful in confirming hypotheses rather than generating new ideas.

Critical incidents is the third method of cognitive task analysis, which focuses on the nonroutine aspects of a job or task (Klein, 1993). The first critical incidents approach is the conflict resolution method. This approach entails the expert recounting critical incidents, and the analyst asking the expert to hypothesize what would happen at different points in the incident if things would have occurred differently. In the second approach, critical decision, the expert and analyst recount events in four cycles. It is important to note that during the fourth cycle the experts compare their performance to that of novices. The advantage to comparing knowledge structures includes challenging the expert to come up with exceptions and special cases of the specified incident. However, this method may not be appropriate for procedural tasks because the experts have trouble remembering critical incidents for these types of tasks.

The last method of cognitive task analysis, analytical methods (otherwise know as psychological scaling), typically involves obtaining judgments about concepts, converting the data into pair-wise comparisons, and then deriving a concept structure through multivariate statistical procedures (Redding, 1989). The analytical methods are beneficial in eliminating the ambiguities inherent in asking people to describe their own thought processes, as they do not rely on introspection (Klein, 1993). The limitation of the analytical methods is their tedious and expensive nature. Because of these limitations, analytical methods are met with the greatest resistance compared to other approaches to cognitive task analysis.

Task Analysis for Teams

Despite numerous publications related to team performance and effectiveness, Baker, Salas, and Cannon-Bowers (1998) noted that issues related to team task analysis have received little attention. A team task analysis differs from the traditional task analysis due its focus on teamwork requirements (Baker et al., 1998). Similar to job analysis, team task analysis provides the foundation for many human resource functions in the context of teamwork including: team task design, team composition, team training, and compensation. It also uncovers critical team aspects that drive the selection of measurement methodologies (Tesluk, Mathieu, Zaccaro, & Marks, 1997).

Much of the research to date has applied individual task analysis to teams. Examples have included using critical incidents (Morgan, Glickman, Woodard, Blaiwes, & Salas, 1986; Prince & Salas, 1993) and task analysis rating scales (Stout, Salas, & Carson, 1992). Specifically, Morgan et al. (1986) used critical incident technique to identify the critical behaviors appearing more frequently in effective teams such as communication, cooperation, team spirit, giving suggestions, accepting suggestions, coordination, and adaptability. Stout et al. (1992) had raters view videotapes and rate several behavioral dimensions on a scale ranging from high to low. These ratings were then compared to objective criteria such as the number of targets destroyed during the task. The main criticism of applying task analysis methods to a team is that critical teamwork behaviors such as interdependence, coordination, and cooperation are not measured (Bowers, Baker, & Salas, 1994). However, task analysis techniques are acceptable within the contexts of teams if the task analysis for each individual establishes a link across the job tasks (Dieterly, 1988). Dieterly (1988) suggested that task analyses can be integrated with respect to team goals and not simply to a set of individual tasks that may influence team performance.

There has been a paucity of research on applying task analysis to teams. Researchers have based the measurement of team characteristics on the assessment of individual members’ perceptions of task-important indices. Campion, Medsker, & Higgs (1993) derived a questionnaire by identifying clusters of common group characteristics, which was completed by team members and their managers. Simulators have also been used to examine team processes and performance (e.g., Weaver, Bowers, Salas, & Cannon-Bowers, 1995). These simulations allow for observable outcomes as well as subjective observations of team performance.

The concepts of teamwork and taskwork have been developed in an attempt to differentiate individual and team tasks (Morgan et al., 1986). Taskwork consists of individuals performing individual tasks, whereas teamwork consists of individuals interacting or coordinating tasks that are important to the team’s goals (Baker & Salas, 1996). Bowers et al. (1994) compiled a team task inventory by identifying teamwork behaviors, which were reviewed and modified by Subject-Matter Experts (SMEs). Respondents were then asked to rate each task (behavior) on importance to train, task criticality, task frequency, task difficulty, difficulty to train, and overall task importance. Therefore, the teamwork distinction is important so that future research may include team behaviors such as interaction, coordination, relationships, cooperation, and communication (McIntyre & Salas, 1995). Dieterly (1988) identified eight dimensions along which tasks can be decomposed. These dimensions were grouped under tasks that are not dependent upon a team concept and task characteristics that specifically apply to a team context.

Researchers have several issues to consider such as how to identify team tasks, measure team-level concepts, and integrate teamwork behaviors into the traditional task analysis methods. Until these issues have been addressed, researchers may not have a valid system to generate team tasks because task analysis methods may not capture teamwork behaviors. This has been demonstrated by Bowers et al. (1994), who found a large proportion of unexplained variance when applying task analysis to a team.

Automation of Task Analysis Procedures

Advancements in technology have allowed task analysis procedures to be automated which creates new opportunities for researchers. Automation allows the task analysis procedures to become more accurate and efficient. In addition, these methods can capture observable and subjective views of task performance. A combination of both objective and subjective criteria presents a more complete representation of the individual and team processes and performance.

Conclusions

There is no one best way of conducting a task analysis, whether it focuses on individual or team performance, nor should an analyst rely solely on one technique. Previous research has highlighted the need for using a combination of measures for a more complete representation of the individual and team processes and performance.

For example, using cognitive task analysis in conjunction with task analysis reduces costs and labor. These methods complement each other by providing distinct information (Koubek, Salvendy, & Noland, 1994). Finally, the performance of teams is generally being measured by aggregating individual responses rather than capturing team-level constructs. Researchers need to develop team task analysis by identifying taskwork and teamwork behaviors.

References

Baker, D., & Salas, E. (1996). Analyzing team performance: In the eye of the beholder? Military Psychology, 8 (3), 235-245.

Baker, D. P., Salas, E., & Cannon-Bowers, J. A. (1998). Team task analysis: Lost but hopefully not forgotten. The Industrial and Organizational Psychologist, 35 (3), 79-83.

Bowers, C., Baker, D. & Salas, E. (1994). Measuring the importance of teamwork: The reliability and validity of job/task analysis indices for team training design. Military Psychology, 6 (4), 205-214.

Campion, M. A., Medsker, G., & Higgs, C. (1993). Relations between work group characteristics and effectiveness: Implications for designing effective work groups. Personnel Psychology, 46, 823-847.

Cannon-Bowers, J. A., Tannenbaum, S. I., Salas, E., & Volpe, C. E. (1995). Defining competencies and establishing team training requirements. In R. A. Guzzo & E. Salas (Eds.), Team effectiveness and decision making in organizations (pp. 333-381). San Francisco: Jossey-Bass.

Cascio, W. F. (1995). Applied psychology in human resource management (5th ed.). Upper Saddle River: Prentice Hall.

Dieterly, D. L. (1988). Team performance requirements. In S. Gael (Ed.), The job analysis handbook for business, industry, and government (pp. 766-777). New York: Wiley.

Dipboye, R. L., Smith, C. S., & Howell, W. C. (1994). Understanding industrial and organizational psychology: An integrated approach. Orlando, FL: Harcourt Brace & Company.

Klein, G. A. (1993). Naturalistic decision making: Implications for design. Crew System Ergonomics Information Analysis Center.

Koubek, R. J., Salvendy, G., & Noland, S. (1994). The use of protocol analysis for determining ability requirements for personnel selection on a computer-based task. Ergonomics, 37, 1787-1800.

McIntyre, R. M., & Salas, E. (1995). Measuring and managing for team performance: Emerging principles from complex environments. In R. A. Guzzo & E. Salas (Eds.), Team effectiveness and decision making in organizations (pp. 9-45). San Francisco: Jossey-Bass.

Morgan, B. B., Glickman, A. S., Woodard, E. A., Blaiwes, A. S., & Salas, E. (1986). Measurement of team behavior in a Navy training environment (Tech. Report TR-86-014). Orlando, Fl: Naval Training Systems Center, Human Factors Division.

Nisbett, R. E., & Wilson, T. D. (1977). Telling more than we know: Verbal reports on mental processes. Psychological Review, 84, 231-259.

Prince, C., & Salas, E. (1993). Training for teamwork in the military aircrew: Military CRM programs and research. In E. Weiner, B. Kanki, & R. Helmreich (Eds.), Cockpit resource management. New York: Academic.

Redding, R. E. (1989). Perspectives on cognitive task-analysis: The state of the state of the art. In 33rd Annual Proceedings of the Human Factors Society, (pp. 1348-1352). Santa Monica, CA: Human Factors Society.

Roth, E. M., Woods, D. D., & Pople, H., E. (1992). Cognitive simulation as a tool for cognitive task analysis. Ergonomics, 35 (10), 1163-1198.

Ryder, J. M., & Redding, R. E. (1993). Integrating cognitive task analysis into instructional systems development. Educational Technology Research and Development, 41, 75-96.

Ryder, J. M., Redding, R. E., & Beckshi, P. F. (1987). Training developments for complex cognitive tasks. Proceedings of the Human Factors Society 31st Annual Meeting (pp.1261-1265). Santa Monica, CA: Human Factors Society.

Stevens, M. J., & Campion, M. A. (1994). The knowledge, skills, and ability requirements for teamwork: Implications for human resource management. Journal of Management, 20 (2), 503-530.

Stout, R., Salas, E., & Carson, R. (1994). Individual task proficiency and team process behavior: What’s important for team functioning? Military Psychology, 6 (3), 177-192.

Tesluk, P., Mathieu, J. E., Zaccaro, S. J., & Marks, M. (1997). Task and aggregation issues in the analysis and assessment of team performance. In M. T. Brannick, C. Prince, & E. Salas (Eds.), Team performance assessment and measurement (pp. 197-224). Mahweh, NJ: Lawrence Erlbaum.

Weaver, J., Bowers, C., Salas, E., & Cannon-Bowers, J. (1995). Networked simulations: New paradigms for team performance research. Behavior Research Methods, Inst., & Computers, 21 (1), 12-24.