Military planners and decision-makers often have difficulty understanding the full impact of advanced mission rehearsal, training, and operational technologies on warfighting effectiveness and mission readiness. This is partly due to the fact that potential benefits from these technologies have typically not been quantified in terms of their warfighting or readiness impact, but rather in terms such as "smiling and sweating" or "hell of a ride." These terms, while providing useful information, are not sufficient as evaluation criteria because they do not permit an accurate, quantitative assessment of mission-oriented impact. The practical utility of the technologies should be assessed by establishing a relationship to combat capability and readiness as defined by accepted Measures of Effectiveness (MOEs) and Measures of Performance (MOPs). For example, if combat capability measured at the organizational level could be linked with innovative approaches to rehearsal and training and to aircrew effectiveness measures, decision-makers could more readily evaluate the impact of the new approaches or advanced technology solutions. Until recently, no comprehensive listing or database of aeronautical system MOEs and MOPs existed in the literature and there were no previous attempts to evaluate the measurement characteristics of MOPs and MOEs as evaluation criteria. This paper provides a summary of work to construct a human-system MOE/MOP taxonomy for aeronautical systems. Future research and development (R&D) will use the MOE/MOP taxonomy in structured interviews with decision-makers with responsibility for wartime mission planning and execution to capture their prioritized uses of the MOPs and MOEs for assessing unit readiness and effectiveness. The prioritized MOPs and MOEs will then be examined and applied as criteria to evaluate the impact of advanced Manpower, Personnel, and Training (MPT) technologies on warfighting capabilities.

MPT R&D traditionally validates its products and procedures using outcome criteria such as course grades, "time-to-proficiency reductions," cost-benefits analyses, or more recently, job performance indices. With major R&D program budgets declining, MPT researchers must demonstrate their contribution to overall mission effectiveness to gain support from MAJCOM officials and higher priority in the budgeting cycle. Research funding prioritization may be greatly influenced by demonstrating war-fighting capabilities and readiness impacts. Therefore, incorporating relevant readiness and war-fighting criteria, as measures of benefit and Return on Investment (ROI), into technology demonstrations and impact assessments is critical to future research. Historically, MPT scientists have not focused on empirical links to readiness and war-fighting capability, but these links are now the key to survival in tight fiscal times. The overarching measurement problem is threefold. First, researchers lack an understanding of the type and quality of metrics used by military decision-makers to determine warfighting capability. Second, the potential for using the metrics as evaluation criteria has only recently been examined and documented. Finally, our current paradigms are limited in terms of their capacity to assess the impact, or ROI, of human systems R&D investments in mission related terms (Bennett, 1998; Gould, 1995). Today, effectiveness measures must be linked with war-fighting capability measures. To extend the connection from typical MPT measures to criteria that MAJCOM decision-makers value, a taxonomy of these linkages would be a key advance to assist them in identifying relevant and warfighting organizationally valued criteria.

As a result of the 1992 Joint Training Review, the Chairman of the Joint Chiefs of Staff (CJCS) directed that the Joint Staff develop and institutionalize a requirements-based training system to better focus DoD’s training resources. The first step was to define the required capabilities in terms of mission tasks, conditions, and standards using the Universal Joint Task List (CJCS Manual 3500.04A, 1996). Responding to the CJCS and publication of this manual, the AF embarked on a process of developing and documenting universal mission tasks and metrics, and documented them in the Air Force Task List (AFTL). The original 1996 Aeronautical-System-Human Performance MOE/MOP Taxonomy was given to DoD, Navy, and AF universal task list developers and MAJCOM points of contract (POCs) for inclusion in their task list documents as they added metrics. After an initial draft AFTL that had been formulated similar to the UJTL, the final version of the AFTL was formatted by AF core competencies, and has recently been published in AF Doctrine Document (AFDD) 1-1 (August, 1998). The AFTL includes mission tasks, conditions, and metrics, written at a very general level to be applicable to the AF as a whole. The Air Staff office for operational training (AF/XOOT) recently required that all AF MAJCOMs and units down through the wing-level develop their own unique METL, based on the AFTL, by 15 Jan 99. Since the AFTL is broadly written, subordinate units could use help in developing clear and meaningful METLs and metrics. As these METLs and metrics are developed, they are to be furnished to the Inspector General offices for inclusion in their evaluation process. (AF/XO Message171257Z AUG 98, MAJCOM METL Development.)

From a survey of the literature, military regulations, and networking with experts of aircraft system-related MOEs/MOPs, the UDRI HFG developed a listing of priority metrics used by AF operational flying command officials responsible for wartime missions and identified MOEs/MOPs important to MAJCOM commanders. The taxonomy was organized to contain these metrics, together with a definition, computational formula, and source. In addition, we developed an aircraft maintenance MOEs/MOPs hierarchy depicting the relationships found by decomposing human performance from the more aggregate MOEs down to the most microscopic MOPs that are tracked.

The aircrew portion of the taxonomy was recently updated to reflect the organization of the new AFTL, published in AFDD 1-1 (1998). This portion of the taxonomy also incorporated the AFTL hierarchy. In addition, new aircrew MOEs/MOPs were identified from other sources, thus augmenting the ones found in the AFTL.

The Aeronautical Human-System Performance Taxonomies (aircrew and maintenance) were developed from the following sources.

Based on the extensive literature and military document review of the sources listed above, and networking with Subject Matter Experts (SMEs), we developed the Aeronautical System - Human Performance MOE Taxonomy, which documents major MOEs/MOPs and sources. Any MOE taxonomy rarely can be fully comprehensive, since evaluators need the flexibility to use new and innovative parameters to evaluate weapon systems and war-fighting capability. Such taxonomies will always be a "work in progress." However, we believe that this taxonomy contains an extensive sample of MOEs/MOPs likely to be used. The taxonomy should already assist scientists in identifying MOEs/MOPs that will benefit their research, and if refined, it could be made more comprehensive and easier to use. In addition, it should assist plans and requirements personnel in choosing the most appropriate MOEs/MOPs to communicate clearly to acquisition personnel. The CJCS and Service use of the taxonomy in their universal task list development efforts will ensure this effort has long-lasting effects. Finally, based on the recent update of this taxonomy, we have divided it into two sections, one for maintenance- and aircraft system-related MOEs/MOPs, and one for aircrew metrics. In addition, we developed an aircraft maintenance MOE/MOP hierarchy.

10. Other Human Performance Measures

10.1 Situation Awareness

10.2 Cognitive Workload

10.3 Other

Repair Times and Rates

Turn Time

Break & Abort Rates, & Delivery Reliability

Maintenance Manpower Requirements

Operational Readiness, Availability, & Mission Capable Rates

Supply & Inventory Analysis

Munitions

Maintenance Safety

Personnel Skill and Training Effectiveness

Overall System Performance & Utilization Rates

The extensive literature search and networking with SMEs could not locate a comprehensive hierarchy of aeronautical MOEs/MOPs. Only a few displays of partial MOE/MOP hierarchies are found in the literature, showing only the relationship of a few terms. Therefore, we created a hierarchy of the aircraft maintenance measures based on the relationship of MOE/MOP definitions and aircraft maintenance experience. The maintenance hierarchy has many facets and interrelationships. This complex taxonomy could possibly be arranged in any number of ways; therefore, subjective judgments had to be made to place the MOEs/MOPs in the most relevant places when the definitions would permit several locations. The hierarchy was verified by comparing it to the MAJCOM "Health of the Force" briefings and the formulas contained in AFI 10-602 and other sources. Details of the hierarchies were published in a review and analysis report (CSERIAC-UDRI, 1997).

While there is a limited set of AF missions, mission tasks and subtasks, AF MOEs and MOPs are numerous. Initially, AF organizational tasks and MOEs were not well documented in the literature. When this study began, considerable information was available, but not from widely published sources. Two exceptions to this general rule were (1) MIL-STD-1776A (1994), which contained an extensive, though incomplete, listing of operator/system-related MOEs/MOPs, and (2) AFI 10-602 (USAF/LGMM, 1994), which contained a comprehensive listing of maintenance MOEs/MOPs. There were no comprehensive aircraft-related MOEs/MOPs listings in the unclassified scientific literature. This dearth of listings forced the labor-intensive use of networking to identify AF aeronautical MOEs/MOPs. Through extensive networking, a number of other productive sources of AF MOEs/MOPs were identified. Many of these sources were not formally published, are in the form of informal guides to analysts, or were published as MAJCOM supplements (not widely available).

In the Fall of 1996, the UJTL and Joint Naval Task List became available. The original MOE/MOP taxonomy was provided to the authors of these documents to assist them with MOE/MOP development. The taxonomies developed by this project were given to personnel developing the UJTL and AFTL metrics, and subsequently were used in developing metrics for the AFTL. Now, the UJTL contains the most comprehensive list of warfighter MOEs, except in the aircrew–system area. In this latter area, the AFTL now contains the most comprehensive listing outside of our aircrew taxonomy. Since the taxonomy was able to identify significant measures not in either of the UJTL or AFTL, additional MOEs/MOPs should be added to both lists.

These measures are often developed for evaluation of exercises, wargames, simulations, competitions, operational tests, and ORIs. Like MOPs, surrogate measures often do not measure the full MOE. Despite the fact that surrogate measures cover only some aspects of the MOE and only are able to approximate the MOE, commanders and high-level decision-makers often consider these measures acceptable indicators of readiness conditions. Since MAJCOM decision-makers are often willing to accept these surrogate measures as strong indicators of wartime outcomes, and since the surrogate measures may be more easily measured than MOEs, these measures may be more readily linked with MPT research and effectiveness than MOEs. Since these surrogate measures have already been established as credible, they may convey credibility to the MPT technology being tested. One focus of future research might be to organize surrogate measures by mission area in a hierarchy that illustrates substitute measures that can be used when it is impossible to measure the MOE.

The Aeronautical System - Human Performance MOE Taxonomy, while being the most comprehensive MOE/MOP taxonomy for aeronautical systems to date, is still a work in progress. New metrics are developed each year. In addition, time and funding did not permit a full review of all sources of aircrew MOEs, and a number of aircrew–system measures exist which could be identified. More work needs to be done to complete the aircrew taxonomy. As a result of the most recent effort, the aircrew portion of the taxonomy is now organized by AFTL task, and contains the same AFTL MOEs/MOPs, plus new ones and some criterion measures. After reviewing two aircraft evaluation AFIs, it is clear that much could be gained from a thorough review of all similar evaluation AFIs. This review could help to identify evaluation metrics and criteria where none currently are available in the AFTL. Such an update could assist the field with developing meaningful criteria for METLs. The maintenance portion of the taxonomy and maintenance hierarchy is ready for use. As the taxonomy is incorporated into the CJCS and Service efforts to develop Joint or Service mission task listings, the results of this effort will be provided with a long-lasting home.

Since the MAJCOMs are beginning to define MOEs/MOPs to more fully describe MAP deficiencies and for criteria-for-success in acquisition documents, they also could use the taxonomy. In addition, the recent move to require METLs to be developed at the MAJCOM through wing level, a much wider source of individual MOEs/MOPs may become available that need to be captured and organized. Selected AF and MAJCOM regulations can also be used to improve MOEs/MOPs and evaluation criteria. Therefore, it is incumbent upon researchers to examine deficiencies and METLs for possible targets of opportunity prior to solidifying plans to improve human performance. If MPT researchers want to target their research toward new weapon systems or major system modifications, they should consult the RCM of major systems to determine how system performance could be improved through improved human performance. In cases where the human components of the weapon systems are not listed, the MOE Taxonomy can assist users in identifying MOEs/MOPs that might be helpful in evaluating the human aspects of the system. By studying the METLs by organizational category, MPT researchers can target MOEs that the desired type of organization metric.

The MOE/MOP hierarchy can assist in depicting the relationships between macro versus micro or higher versus lower priority measures. The maintenance hierarchy is mostly complete and has already been useful in selecting MOEs for maintenance training evaluation. On the aircrew side, however, much work remains to be accomplished. Based on the findings of this phase, high-level decision-makers can be polled about their MOE/MOP preferences. The results can then be compared with the written documentation on MOEs/MOPs, and adjustments made to the MOE/MOP taxonomy and hierarchy, as appropriate. As these relationships are verified by decision-makers, it will be possible to see which MOEs/MOPs are most valued. For example, based on our analysis of the MAJCOM "Health of the Force" briefings, we believe that the most important MOE for aircraft maintenance is related to sortie production. It will be interesting to determine which of several sortie production measures decision-makers believe is most descriptive for their purpose.

In this post-Cold War period, when military leaders are pressed to accomplish a wider variety of missions at increasing OPTEMPOs while conducting them in a flawless manner, on-target training is even more critical. R&D must be clearly focused on providing commanders with the best possible training at the least cost. This era has seen unprecedented focus on mission tasks and metrics. However, the Service task lists that contain these tasks and metrics are still in infancy stages, and rarely have more than "cookie cutter" type metrics. Research can provide considerable measurement methodology and result to enhance the reliability and validity of metrics for METL development. In turn, with the metrics organized so that researchers and trainers can easily use them, an improved taxonomy and hierarchy can assist in focusing mission-oriented rehearsal and training systems R&D. Future work needs to complete the aircrew taxonomy and determine the MOE/MOP hierarchy in greater detail than is available from the AFTL. Further efforts to identify decision-maker preferences and priorities for MOEs/MOPs may help focus R&D efforts on mission needs and outcomes. Outcomes from this research will facilitate the integration of mission rehearsal and training research in distributed mission training. Furthermore, the taxonomy can support mission-oriented research targeted at the most critical readiness and performance tasks.