AB-21 - Symposium

Level-Three Training Evaluation: The Coast Guard's Method and Design

Gene Adgate, Mark Rutherford, & Christopher Hall
United States Coast Guard 

Introduction

It is axiomatic that evaluation plays a major role in all phases of the systematic approach to improving human performance.  Implementing performance interventions without following up through evaluation means that positive results, though hoped for, cannot be directly verified.  In such a case, an organization may train, create job aids, buy new equipment or institute any number of other interventions without the means to determine whether these "investments" actually improved performance and provided a reasonable "return." 

Because training is unquestionably among its more resource-intensive and pervasive

interventions, the Coast Guard recently analyzed the evaluation of training to establish policies and procedures to ensure that its training efforts are effective.  This paper describes the Coast Guard's newly implemented training evaluation policy and associated responsibilities and infrastructure. 

The Case of Action

The Coast Guard sponsors over 700 resident and non-resident training courses offered by various training providers.  The term "training provider" in this paper refers to any source that delivers training to Coast Guard members (e.g., Coast Guard Training Centers, contracted training vendors, and other government agencies and military services).  Historically, evaluation methods have varied among training providers with many providers conducting only the most rudimentary student critiques.  Moreover, data received from evaluations were not sufficiently communicated to Program Managers or Training Consultants, the organization-level reviewers who make key decisions regarding training resource allocation.  Without a steady stream of evaluation data, these reviewers have little or no information to identify trends and ensure training is "hitting the mark' in terms of improving or enabling performance. 

Seeking to remedy this situation, the Coast Guard adopted the widely accepted four-level evaluation model first introduced by Dr. Donald Kirkpatrick some four decades ago.  A brief explanation of these levels is included in Figure 1. The levels are ordered in the sequence in which they are assessed; each level provides valuable feedback on distinct aspects of the linkage between training and performance.  The first two levels, referred to as "Internal evaluations," are internal to the training providers and directly affect the design, development, delivery and administration of training.  The second two levels, known as "external evaluations," are external to the training providers and focus on the impact of training on job and organizational performance. 

The Four Levels of Training Evaluation (The Kirkpatrick Model)
Level	Definition	Advantages	Disadvantages
Level 1- "Reaction"	Measures trainees' opinions about the course.  This Is the most common way to evaluate student reaction to the course and provides a measure of immediate customer satisfaction with content, delivery, and environmental factors.  Often referred to as "Smile Sheets."	Low cost and easy to administer.  Provides insights to participant's personal feelings about the course.  Provides quick feedback on successes and failures to the training provider.	Only reflects a quick reading of the participant while they are still in the class.  Results should not be used as a solid basis for changing the educational content or strategy
Level 2- "Learning"	Measures how well participants have mastered the course objectives.  Can include tests of performance immediately before and after the course.	Compared to level 1, this provides more compelling evidence of whether the training program works.	Requires more time and money than level 1. Also requires greater insight to the evaluation process to develop valid measures of learning.
Level 3- "Behavior"	Assesses practical value of training.  Measures how well the knowledge, skills, and/or values from the course are used in the job.  Typically measured 3-6 months after the course.	Provides stronger evidence that the investment in training yields the desired return.  If designed properly, can also identify barriers and obstacles to improved performance.	Requires significantly more investment of time and money. Requires in-depth insight into performance interventions and root causes of performance deficiencies.
Level 4- "Results"	Measures performance improvement, quality improvements, and cost savings to the organization.  Measures the return on investment of the training course.	Provides strong evidence that training program has impact on organization.  Addresses whether the performance is important to the organization's bottom line (e.g., production, safety, sales).	Substantial levels of investment and  expertise are required to develop level 4. Often hard to decide whether or not this level is required.  Linkage from training to org'1 results is hard to establish.

Figure 1: The Four Levels of Training Evaluation

 Policy Analysis

Substantial research and analysis were conducted prior to developing any polices or making any sweeping changes to the existing informal training evaluation performance consultants and a full-time graduate student spent over a year examining related literature and best practices.  Below are observations and conclusions based on the research, benchmarking, and analysis that helped shape a new Coast Guard-specific training evaluation policy and infrastructure. 

1.       Level- 1 and -2 evaluations are clearly the responsibility of the training providers, who are most knowledgeable of the training environment and the trainees' reactions.  The training providers have sole control over how they administer and deliver the course objectives and whether or not the participants acquire the knowledge, skills, or values during training.  Coast Guard Training Centers are staffed with civilian and military professionals well versed in the Instructional Systems Design (ISD) process and are very capable of handling levels 1and 2.  General guidelines are all that is needed to ensure uniformity of practice. 

2.       Level-3 evaluations require a standardized process to ensure the results are communicated consistently and regularly to the appropriate decision-makers.  Under the existing system, many training providers were not even conducting level-3 evaluations.  Where these evaluations were conducted, there were problems including:

Ø      There were no criteria for using and analyzing evaluation data.

Ø      Questions and evaluation forms were non-standard and often did not accurately reflect the course content.

Ø      The evaluations were not built into the training process and therefore were inconsistent.  Identifying negative trends and making decisions on remedial performance interventions were virtually impossible.

Ø      Program Managers and Training Consultants-those who make decisions affecting whether courses are maintained, deleted, or modified-did not receive evaluation data. 

3.       Some level-4 evaluations are in place.  There is a significant body of literature that indicates level 4 is not required for all types of training.  Moreover, it is difficult to isolate the contribution of any one determinant such as training in achieving organizational results. Finally, measurement of some training courses and their competency objectives would either be impossible or would yield invalid and inconclusive results.  For example, even if Coast Guard EMTs never used CPR, the course would still be deemed necessary.  Level 4 requires more research before making changes.  As with levels I and 2, general guidelines will assist Program Managers in continuing to assess the ultimate worth of training.