We hypothesized that information-processing training improves retention of skills in an ITS. Specifically, we hypothesized that examinees who have the information-processing features in their ITS compile the troubleshooting declarative and procedural knowledge more completely so that they have less decrement in troubleshooting tasks over time.

The sample was comprised of 767 paid participants, between 17-35 years old. The sample was assembled to resemble current or future field test administrators (TAs) with respect to education level, age, primary language, citizenship, and absence of diagnosed reading difficulties.

The research version of the ITS contains two lessons and two final exams. The first lesson, Introduction to Computers and the CAT-ASVAB System, presents conceptual and factual information about computers and the CAT-ASVAB testing system using a tutorial strategy. The ITS includes topic, test question, remediation, and performance feedback screens.

Lesson One, Introduction to Computers and the CAT-ASVAB System

Lesson content. Lesson 1 contains instruction in three topics: (1) CAT-ASVAB Hardware, (2) CAT-ASVAB Test Stations, and (3) Test Modes. CAT-ASVAB hardware includes: a monitor, a DOS-based computer, a keyboard and the associated cables. The CAT-ASVAB also includes three different types of test stations: (1) the Test Administrator (TA) Station, (2) the Examinee Testing (ET) Station, and (3) the File Server. The Test Administrator uses the TA Station to help with administering CAT-ASVAB and to manage the test sessions. Examinees use the ET Stations to take the test, and the File Server allows for communication between the TA and ET Stations.

Instructional design. Lesson One uses simple, algorithmic branching techniques to present conceptual and factual information in a tutorial strategy. For each lesson topic, the program presents a series of instruction screens, a series of review screens, a topic quiz, remediation screens if the student answers quiz items incorrectly, and performance feedback. The screens are presented in a simple linear design.

Performance assessment. A final exam covering all of the content in Lesson One appears at the end of Lesson One. Immediate feedback is presented after each test item, and an overall feedback screen appears at the end of the final exam.

Lesson content. The second lesson, CAT-ASVAB Troubleshooting, is divided into two sections: (1) declarative knowledge and (2) troubleshooting scenarios. The declarative knowledge section teaches fundamental CAT-ASVAB troubleshooting principles and methods and obtains an initial estimate of students’ troubleshooting abilities on four topic: (1) the definition of troubleshooting, (2) preventive maintenance, (3) PC troubleshooting rules, and (4) introduction to scenarios.

Instructional design. The design of this first section of Lesson Two is similar to that of Lesson One. Instruction, review, feedback, and remediation screens are included.

Performance assessment. The performance assessment for this part of Lesson 2 contains multiple-choice trouble-shooting problems with immediate feedback and remediation, as necessary. The problems build on the knowledge that the students are acquiring as they progress through the lesson. Student performance on the first section of Lesson Two provides input for use by the program in the next section of Lesson Two. Every question appearing in the declarative knowledge section is associated with one or more of the 30 problem-solving rules that make up the domain expert for the trouble-shooting scenarios.

Lesson Two, Troubleshooting Scenarios

The troubleshooting principles and methods learned in the declarative knowledge section of the lesson are applied. This section uses artificial intelligence components to develop a model of student performance and compare that model with the program’s expert model.

User interface. Every scenario has three sections: Introduction; Isolate Problem; and Review. The Introduction section describes what took place before the CAT-ASVAB system failure occurred and provides the student with the needed background information to effectively troubleshoot the failure. The Introduction part of the scenarios consists of four to five screens. The Isolate Problem section provides the student with nine different menus from which to choose a troubleshooting procedure. The Isolate Problem section provides students with two additional menu options: DIAGRAM and STEPS. When students select the DIAGRAM option, it displays a graphic depiction of the test station(s) in which the failure is occurring. When a student selects the STEPS menu option, it presents the troubleshooting steps the student already has performed. All troubleshooting scenarios end with a review, consisting of one to nine screens, that summarizes the troubleshooting steps taken to correct the failure.

Lesson content. We performed a cognitive task analysis to derive 74 scenarios that the program uses for instructional and practice purposes. These scenarios require students to correctly select and apply one or more of the 30 problem-solving rules in the domain expert. In addition, we developed an additional 20 scenarios that the program uses only for performance assessment.

Instructional design and performance assessment. The highly modular software contains a domain expert, student model, a diagnostic module, and a tutorial module. The domain expert, or expert model, contains the problem-solving rules that the student needs to learn. The student model maintains a current profile of what the student does and does not know based upon the available information and treatment condition. The diagnostic module uses a Bayesian inferencing approach to assess the probability that the student would correctly apply a given problem-solving rule, and the tutorial module selects the next troubleshooting scenario based on the diagnostic module’s assessment of the current state of the student model.

The second part of Lesson Two consists entirely of troubleshooting scenarios. It requires students to apply troubleshooting principles and methods. A troubleshooting scenario begins with a series of screens that describe a CAT-ASVAB system failure and then provides response options for students to correct the failure. As students complete the scenarios, the program updates its model of the student’s current knowledge state, and the program’s diagnostic module uses artificial intelligence features to compare the updated student model with the expert model. Based on that information, the program’s tutorial module selects the best teaching strategy and scenario to present. The course is self-paced.

For research purposes, we developed for different versions of the troubleshooting scenarios. The four versions are (1) baseline, (2) coaching, (3) information processing, and (4) coaching plus information processing.

Baseline. The control condition includes the same Bayesian inferencing approach and performance assessment components as the other conditions. Corrective feedback explains why the student’s choice is an error and indicates that there is a better approach to a given step without directly suggesting a repair. After three incorrect answers to the same step the baseline condition provides the correct answer. The Baseline condition does not include any of the additional experimental features.

Coaching. The coaching condition adds coaching components to the baseline version (see Graesser, Person, & Huber, 1995; Shaw, Dunkle, & Bendixen, 1995). These coaching components include a student-controlled HINT menu option as well as a program-controlled coach. When selected by the student, the HINT menu option displays advice about which problem-solving step the student should take next. To avoid students overusing hints, program instructions inform students that their progress through training depends on eventually solving problems without the aid of the HINT key. Therefore, the program updates the student model based on whether or not the student accesses the HINT key. In addition, the tutor fills in when the student cannot perform all the steps in a skill, but it facilitates the student’s providing more and more of the work until the tutor operates completely in the background.

Information Processing. Littman and Soloway (1988) stress the importance of including process measures of performance when assessing the effectiveness on an ITS. They note that the purpose of the student or diagnostic model is to capture how students solve problems and not merely that they can solve particular problems. We developed measures of information-processing efficiency that reflect what the student knows and how the student evaluates and solves cognitively complex tasks (see Dillon, 1986, 1997). The information-processing condition adds information-processing treatment and measurement components to the baseline version. To implement this condition, we added features to the interface, developed and tracked additional information-processing measures, and added information-processing efficiency performance criteria to the baseline ITS.

Information-processing interface. We added user-controlled, selective disclosure windows to track students’ information-processing strategies. For the information-processing condition, the declarative section of Lesson Two provides instruction on the most efficient way to troubleshoot failures using the selective-disclosure windows. That lesson instructs students to use their time efficiently by viewing the information available, thinking about the six troubleshooting rules, and thinking about how to apply them to the hardware failure. So, the information in the lesson was taught using instruction in learning material by processing material efficiently

Information-processing measures. The program collects different kinds of data regarding how students process information within a particular window specifically (1) the number of times a student views the menu window (Menu Views), and (2) the amount of time spent with the menu covered (Diagnostic Time), and (3) with the menu uncovered (Procedure Selection Time). Using these data, the program computes Information- Processing Efficiency (IPE) scores and Information-Processing Learning (IPL) scores. The program also computes another IPE score based on the time data. The program computes the Proportion of Diagnostic Time (PDT) as a ratio of Diagnostic Time to total problem time: PDT = (Diagnostic Time) / (Procedure Selection Time + Diagnostic Time). As explained in the declarative part of Lesson Two, students can perform most efficiently in they spend more of their time in Window 1 (i.e., with the window covered) getting all the information they need (their Diagnostic Time) and then selecting and confirming their answer (Procedure Selection Time). Finally, the program calculates two variables that assess students’ improvement over time on the IPE variables. These variables are the Information Processing Learning (IPL) scores. Learning indices measure changes; i.e., improvements in the above information-processing measures.

Information-processing efficiency criteria and feedback. The information-processing condition attempts to improve students’ performance both by improving their information-processing efficiency and by insuring that they process information efficiently before allowing them to progress in the tutor. Information-processing efficiency results in information-processing feedback in addition to the regular feedback that the student receives, such as the following statements: "You are working through the scenarios very efficiently! Thinking the problem through BEFORE accessing Window #2 is a good strategy." If students fail to meet any of the efficiency criteria, they receive corrective information-processing feedback in addition to the regular feedback.

Coaching Plus Information Processing. Both coaching components and information-processing components are added to the baseline condition. When students do not pass any of the information-processing efficiency criteria for a particular problem-solving rule, the program displays the coach for that problem-solving rule just prior to the student’s next attempt at a troubleshooting task involving the rule.

All participants completed the same final exams. Test 1 was taken upon completion of Lesson Two. Test 2 was completed one week after Test 1. The exams were designed to be as similar as possible to the baseline troubleshooting scenarios of Lesson Two. Each of the two final exams contains 92 items. We designed the exams to have at least one item that presented an opportunity for students to correctly apply each of the 30 different problem-solving rules in the domain expert of the ITS. In order to develop an exam that covered all the content areas, it was necessary to duplicate some early aspects of the scenarios that lead up to the new part of the problem. This duplication meant that, for some scenarios, students already had seen the early part of the scenario in a previous scenario, and they already had received feedback about the correct next step. Therefore, we divided the finals into two conceptual parts, one having "first-exposure" items and the other having "second-exposure" items. Of the 92 final exam items, 53 were first-exposure items, and 39 were second-exposure items. Differences found between the four groups on the 53 first-exposure items are due to the impact of the training alone and not to feedback during the final exam process.

We computed alpha reliability coefficients for each of the three sets for the two tests: all 92 items, first-exposure items only, and second-exposure items only. Only second exposure items were below .80. The alpha reliability is .65 for Test 1 second-exposure items and .62 for Test 2 second-exposure items. Test-retest reliability for second-exposure items is .56. Table 1 contains means and standard deviations for Test 1 and Test 2 by group.

No differences were found in Test 1 performance as a function of condition. Data reveal greater training time for students who receive either coaching (F=22.41, p < .05) or IP (F=247.82, p < .05).

Data also were analyzed to test the premise that information-processing principles improve retention of training performance relative to baseline or coaching conditions. We are interested here in the extent to which these procedures reduce any decrement in troubleshooting skills over time. Conditions with information-processing training were expected to yield less decrement in troubleshooting from Test 1 to Test 2. Students demonstrate a decrement in troubleshooting performance over time (F=58.89, p < .05) and data reveal an interaction for Time x IP (F=6.10, p < .05). Thus, the four groups taken as a whole scored, on average, lower (about 1.79 points) when comparing their test scores from Test 1 to Test 2. In addition, the interaction between Time and IP indicates that the difference between participants’ test scores between Test 1 and Test 2 depends on whether or not they were received information-processing training. While students in the information-processing conditions do not demonstrate better performance during the final exam than students in the other conditions, they demonstrate less decrement in training performance at Week 2.

The coaching conditions did not produce an effect on test performance, at Test 1 or Test 2. Participants under coaching showed a decrement in test performance over time. The decrement in test performance, however, was not different from the decrement experienced by students who did not receive coaching. And, students in the coaching conditions required more time (about 11 minutes; .34 more standard deviation units) to complete training than students in the baseline condition.

Data show that participants who receive information-processing training demonstrate a smaller performance decrement over time than students who do not receive information-processing procedures. Students in the IP condition lost, on average, only 0.85 points (or 0.075 standard deviation units) from Test 1 to Test 2, while participants in the Baseline condition lost 2.34 points (or 0.211 standard deviation units). The data offer interesting support for the premise that information-processing procedures help students compile declarative knowledge. Also interestingly, the effect of information-processing training is seen one week after training rather than immediately after training. Because of the fundamental nature of information-processing abilities, such componential knowledge is well learned or consolidated over time and is resistant to decay.

Future research should focus on assessing students’ information-processing abilities prior to assigning them to training conditions. Students high on information-processing efficiency should be able more easily to compile declarative knowledge into procedural knowledge or more completely to strengthen their knowledge and skill in memory. Research efforts should be directed toward determining what instruction or which training conditions will help students improve their information-processing efficiency, thereby facilitating the retention of cognitively complex knowledge and skills.

Dillon, R. F. (1986). Information processing and testing. Educational Psychologist, 20(3), 163-174,

Dillon, R. F. (1997). A new era in testing. In R. F. Dillon (Ed.), Handbook on testing. Westport, CN: Greenwood.

Graesser, A. C., Person, N. K., & Huber, J. D. (1993). Question asking during tutoring and in the design of educational software. In M. Rabinowitz (Ed.), Cognition, instruction, and educational assessment. Hillsdale: Erlbaum.

Littman, D., & Soloway, E. (1988). Evaluating ITSs: The cognitive science perspective. In M. C. Polson & J. J. Richardson (Eds.),l Foundations in intelligent tutoring systems (pp. 209-242). Hillsdale, NJ: Erlbaum.

Shaw, G., Dunkle, M. E., & Bendixen, L. D. (1995). Cognitive processes in well-defined and ill-defined problem solving. In A. C. Graesser, N.K. Person, & J. D. Huber (Eds.), Collaborative dialogue patterns in naturalistic one-to-one tutoring (pp. 295-323). Hillsdale, NJ: Erlbaum.