AB-46 - Paper

Planning and Evaluating a Navy Interactive Electronic Training Manual Prototype

Nora Matos, Agnes Crum, Kelsey Henderson, and ABECS (AW) J. Eggleston

 INTRODUCTION

                 The Product Development division of the Navy Advancement Center (NAC) designed a pilot product for future Electronic Training Manual (ETM) development.  A print-based training manual (TRAMAN) and nonresident training course (NRTC) were converted into an Interactive ETM.  The TRAMAN was converted into a Standard Generalized Markup Language (SGML) format, and the NRTC was converted into interactive tests with remediation.

 PURPOSE OF THE EVALUATION

 The purpose of the evaluation was to (1) measure the technical accuracy of the product, (2) measure the effectiveness of the electronic training environment, and (3) gather information from the target audience about the product.  During the evaluation, the following quality indicators were assessed:  (1) completeness, (2) technical accuracy, (3) instructional soundness, (4) functionality, (5) user friendliness, and (6) aesthetic appearance.

COURSE DESCRIPTION

The ABE 3 & 2 Interactive ETM is an electronic version of the ABE 3 & 2 TRAMAN and NRTC.  The ETM is displayed with the DynaText browser.  The browser has keyword searching, note-taking, and printing capabilities.  The student can search the ETM content for key words, take notes while studying, and print topics of interest or the entire ETM.  Interactive testing with remediation, color-enhanced graphics, animations, audio, and hyperlinks complement the ETM.  Authorware Professional interactive multimedia software package was used to develop the interactive testing.

                 The Interactive ETM is a supplement to the print-based TRAMAN and NRTC.  It contains the same basic technical information. 

SCOPE OF THE EVALUATION

                 The evaluation was an alpha test that included the interactive and printed ETM.

ALPHA TEST       

The alpha test included students participating in one-to-one and small group evaluations, subject matter experts (SMEs), instructional systems specialists (ISSs), and editor‑writer evaluations.  This test was designed to obtain feedback about the course. 

PARTICIPANTS 

The alpha test involved 13 students, 5 observers/interviewers, 4 subject matter experts (SMEs), 2 instructional systems specialists (ISSs), and 2 writer‑editors.

 Students 

Students participating in the evaluation were from the student population at the Naval Air Technical Training Center (NATTC) at Naval Air Station (NAS) Pensacola, Florida.  There were two groups of students:  (1) one-to-one evaluation participants, and (2) small group evaluation participants. 

For the one-to-one evaluations selected students were familiar with the subject and had a strong background in computers.  Students from the upper percentage ranges in aptitude and subject content are often more likely to point out and analyze weaknesses in the student instructions and course materials. 

For the small group evaluation selected students had a broad range of knowledge about the subject and had diverse backgrounds in computers.  The following criteria were used to select the students: (1) aptitude, (2) prior subject knowledge, (3) computer skills, (4) attitude, and (5) general background experience.  Participants represented the students in the ABE 3 & 2 training environment as much as possible. 

Observers/Interviewers 

Observers/interviewers were Saufley Field NETPDTC, Pensacola, Florida ISSs.  Observers/interviewers described events encountered while the students evaluated the course, recorded the students' comments, and interviewed the students after they completed the course.  During the one‑to‑one evaluations, one observer/interviewer was assigned to each student.  During the small group evaluation, one observer/interviewer was assigned to every two or three students.

 SMEs 

SMEs were senior ABE Sailors from the NAC and NATTC.  They evaluated the course for completeness and technical accuracy.  The SMEs completed a “subject matter expert survey” and prepared a “technical deficiency report.”  The SMEs were in a unique position because they have taken the paper-based course before and furnished direct comparisons between the paper-based and ETM products.

 ISSs

 ISSs evaluated the course for instructional soundness and furnished their comments in an “instructional soundness evaluation report.”

 Writer-Editors 

                The writer‑editors evaluated the printed ETM by comparing it to the original print-based TRAMAN.  They conducted a word-by-word comparison of all the text.  They ensured all the original content appeared in the electronic ETM. They ensured all the formatting styles were correct.  They examined the figures and tables for completeness and accuracy.  They reviewed the progress test questions and answers to ensure they complied with NAC examination standards.

 LOCATION AND LOGISTICS 

The one-to-one and small group evaluations were conducted for 1 week each at NETPDTC in the Learning Resource Center (LRC).