Charles N. Holt & Gray Hardaway , IJOA, San Antonio, TX

David Woehr, Texas A&M University, TX

Winston Bennett, Jr., Air Force Research Laboratory, Mesa, AZ

The Air Force Research Laboratory’s Human Effectiveness Directorate (AFRL/HEA) has been conducting research on the best methods to use in automating occupational-type surveys. One area of particular interest has been in developing valid and reliable estimates of the amount of time that individuals spend performing tasks associated with their jobs (Albert, Phalen, Selander, Dittmar, Tucker, & Weissmuller, 1994; Mitchell, Weissmuller, Bennett, Agee & Albert, 1995). Automation has also facilitated exploration of new methods of gathering estimates of actual time in hours and minutes spent performing each task by each respondent. A number of field applications of the automated survey technology have been accomplished so far. A key finding across all studies is that supervisors and subordinates have very positive reactions to the software as a data collection tool. Exit surveys show that instructions are relatively clear, automated surveys are preferred to paper-and-pencil surveys, time to complete is reasonable and typically shorter than paper and pencil, and others in the career field can be expected to successfully complete the survey (Mitchell, Weissmuller, Bennett, Agee & Albert, 1995; Albert, Bennett, Pemberton, Holt & Waldroop, 1997). 

One of the scales that has exhibited promising results in gathering this information is the Actual Time Spent (ATS) Scale. In using the ATS Scale, each respondent is asked how frequently he or she performs each task comprising his or her job during some time period such as a week, month, or year and how long it takes on average for him/her to perform the task once, excluding delays, interruptions, or breaks. The product of these two components of time is the estimate of the total time spent by the respondent on that task. Then, the tasks are fed back to the rater in high-to-low order on total time spent for evaluation and revision of estimates and automatic recalculation and reordering of tasks (until the respondent is satisfied with the results). Therefore, the actual time spent procedure decomposes total time into its two basic components: "time to perform a task once" and "frequency of performance." "Frequency" has been shown to be a highly reliable counter mechanism, a perceptive ability that is consistently more accurate than that which governs our perception of time (Hasher & Zacks, 1984). In addition, "time to perform a task once" has the advantage of representing an average or median value rather than a global estimate (Phalen, 1995). For example, if you were to look at a long column of numbers, you could more quickly approximate a reasonably accurate mean or median value than a sum for that set of numbers. The ATS scale has considerable potential for assessing the impact of new technologies and other operational studies.

Based on their extensive experience in automating both occupational-type surveys and general purpose surveys, AFRL personnel were requested to evaluate savings or costs associated with the introduction of portable computer terminals into the 737th Training Group (TRG) at Lackland AFB, and to determine instructors’ attitudes and opinions concerning the software, hardware, and changes in their work processes (Albert, Bennett, Pemberton, Holt & Waldroop, 1997). The 737th TRG is implementing the terminals into Basic Military Training (BMT) to automate many of the paper and pencil processes performed by their instructors. For example, each terminal provides access through a network server to the instructor’s lesson plans, relevant Air Force regulations, standard forms that the instructor needs to complete on a regular basis, e-mail, and word processing, spreadsheet, presentation graphics, and data-base management software. To make the transition to automation smoother, AFRL is also providing expertise in optimizing the human-terminal interface and efficiency of the associated software, and in designing training for new users of the terminals. In addition to expanding AFRL’s experience with computer-based surveys, this effort will provide a good test-bed for the automated ATS scale.

Basic Military Training Instructors (TIs) have employed time-honored and well-established training procedures in bringing a flight through Basic Military Training. The majority of TI tasks, however, involved paper and pencil record-keeping. The 737^th BMT Squadron at Lackland AFB, TX, had an opportunity to purchase hand-held computers, called CruisePad, in an attempt to automatic many of these processes. The CruisePad system uses FM radio transmitters to connect the CruisePads with a host computer. Without much guidance, squadron personnel wrote their own software in MS Access and brought a few of the CruisePads on line. AFRL agreed to assist with identifying and solving problems associated with this effort. In addition to providing critical programming assistance, AFRL completed an evaluation of the impact of implementation of the CruisePad technology on the tasks normally performed by TIs. The primary goal of this Operational Study was to determine if implementation of the CruisePad technology has an impact on the amount of time TIs spend on their normal work tasks. It was hypothesized that there would be an overall reduction in time spent on tasks; it was also anticipated that there would be a large amount of variance due to the turmoil created with the implementation of the CruisePad technology.

With several experienced TIs serving as Subject Matter Experts (SMEs) and using the Air Force Occupational Squadron-developed TI task list as a starting point, the Institute for Job and Occupational Analysis, under contract to the AFRL, developed a 126-item task list which was incorporated into a computer-based survey using the Air Force Survey Authoring System (AFSAS; see Mitchell, Weissmuller, Tucker, Waldroop, & Bennett, 1996). An "actual time" scale (Albert, Phalen, Selander, Dittmar, Tucker, & Weissmuller, 1994; Phalen, 1995) was used to collect time estimates by incumbents. IJOA programmers inserted the actual time software into an experimental version of the AFSAS software. In addition, several open-ended questions were included with the expectation that comments from CruisePad users would assist in identifying problem areas and improving the CruisePad system.

Identical surveys were administer on two occasions, once just prior to the full implementation of the CruisePad in March 1998 and three and a half months later, in early July, 1998. During the intervening period, there were several significant changes to the software (notably the inclusion of signature recognition software) and hardware (notably the FM antenna system). Also during this period, a number of TIs utilized both the existing paper and pencil methodology and the new CruisePad system for a portion of that four-month period. Additionally, several technical problems identified during the first survey were in the process of being addressed at the time of the second survey; only a portion of those problems had been resolved. Both survey administrations were proctored by AFRL, IJOA and BMT staff and were completed in several group sessions held in a computer laboratory in the 737^th BMT Squadron Headquarters. The first sample consisted of 125 TIs while the second sample consisted of 112 TIs. Two cases were eliminated from the first administration and one case was removed from the second study due to response patterns that suggested that the respondents misunderstood the requirements of the survey. In addition, 12 data points were removed from the first survey and 9 data points were removed from the second survey after being identified by BMT SME personnel as providing obviously out-of-range responses.

Data were summarized using the Statistical Package for Social Sciences (SPSS), version 8.0 PC. A special DOS utility was written to calculate individual time estimate responses into a common metric – hours per task per year. SPSS was utilized instead of CODAP for technical reasons. The background and exit survey data were uploaded using the AFSAS software while the actual time estimates were uploaded using a special, IJOA-developed version of the actual time software described earlier. Table 1 displays the results; it reflects a decrease in time spent on tasks from the first survey to the second.

A similar decrease in the number of hours incumbents reported working per week (in response to a background item asking for an estimate of "hours worked per week") was observed (see Table 2).

The ATS data document average timesavings of just over three hours per week per TI from the first survey to the second, or about 157 hours per year per TI. Table 2 reveals that when estimating the hours worked per week (a more global estimate), the average TI said he/she was working about 4 hours less per week. Both these estimates are in the same direction and in about the same magnitude, and show a trend towards less time being spent on tasks with the use of the CruisePad. Of the 108 tasks performed by the majority of TIs, 56 showed a decrease in actual time spent, while 52 reflected an increase. The estimated savings may be an underestimate since a portion of the respondents was actually using both paper and pencil and CruisePad methodologies. Table 3 shows a direct comparison of the top 12 tasks from each administration.

Considerable information was also provided from the comment section of the survey. Table 4 summarizes these results. The detailed Comments file has been forwarded to BMT for their analysis and use.

The primary objective of the experimental phase of this study was to determine if the respondents’ actual time estimates were consistent with a more concrete estimate of time worked. This was assessed by contrasting their actual time estimates with their responses to the survey item capturing their global estimate of "hours worked per week."

A comparison of actual time data and the more global estimate (survey item asking for "hours per week worked") showed both estimates to be similar (see Table 5).

Although the second administration of the survey was accomplished before many important "bugs" had been completely worked out of the CruisePad system, the operational study identified a trend towards a decrease in time spent on tasks with the CruisePad technology, as hypothesized. As noted, this is probably an underestimate of the total timesavings since many of the respondents were still using both systems at the time of the second survey. It would be beneficial to re-administer this survey when the CruisePad system has stablized and TI's have enough experience to provide realistic estimates.

The experimental study confirms that ATS estimations were consistent with a more global estimate of time spent working on tasks (convergent validity), indicating that further study of ATS is warranted. As evidenced by the high standard deviations seen in both surveys, there may be some problem with how the ATS data collection was operationalized. Future development of ATS and its related software should address e these possible overestimation issues.

In addition to the current operational study, BMT, in collaboration with the Air Force Research Laboratory (AFRL) has developed a prototype electronic classroom where the impact of automation on student feedback assessments and on end-of-course testing is being assessed. Customer surveys are provided to students at the 2^nd and 5^th weeks of training to track satisfaction and to identify potential conflict areas. Typically done as paper-and-pencil surveys, BMT is now using automated data collection and reporting for these surveys. For the automated testing, AFRL and BMT have developed a preliminary test item bank and an automated version of the current paper and pencil end-of-course test. Researchers are presently applying Item Response Theory (IRT) methodology to develop an adaptive testing item bank for future implementation. AFRL has designed and implemented a comprehensive evaluation study of these innovations and their impact on BMT work processes, products and overall efficiency.