AB-4 - Paper
ITER: A computerized test to assess the decision making skills. Preliminary applications.
Antonio Peri
Italian Antarctic Project, Rome, Italy
Roberto Mosticoni & Stefano Mosticoni
M-TWO Systems, Rome, Italy
Marta Barbarito
NDA-Argentinian Antarctic Institute, Buenos Aires, Argentina
Introduction
Decision-making and planning skills are important individual qualities in the operational and organizational environment of the Antarctic stations (7,9,14,16,20,21,22,30). In order to assess these abilities in the candidates of the Italian Antarctic expeditions, especially in those having logistic and scientific responsibilities, a computerized test (17), called ITER, was developed.
A preliminary study (13) performed in a population of 110 normal subjects, applicants for the Antarctic expeditions, found correlations between norm referred personality measures (MMPI) and some quantitative indicators of ITER. The results show the cognitive attributes of depressive traits are related to lowest efficacy of less complex solutions; furthermore, self-criticism and uncertainty are related to mazy solving paths and to redundant behaviours with respect to goal orientation.
The general description of the computerized instrument and the results of an experimentation carried out using the Intelligence Structure Test (IST) of R. Amthauer (1) are reported here
Methodology
Sample: 100 men, 26 to 31 years old, mean age: 26.7, all of them law graduated, performed the tests ITER and IST in the same day during the psychological selection of a competitive exam for a job as harbour-masters officer.
Psychological Instruments
ITER
ITER operational environment
The ITER test, inspired on the one used by Barbara and Frederick Hayes-Roth(8), requires the management of a definite time span to accomplish a list of possible errands in a fictitious small town, schematically represented as a map with streets, squares and buildings on the computer screen. Movements in the town are performed using the suitable keys (arrows,etc.) of the keyboard.
Task features
The allowed fictitious time is not sufficient to complete all the tasks mentioned in the test. Therefore the subject has to decide what to do (which errands to perform, which route to take,etc.) in the allotted time. Every move has a time cost: it uses up a predetermined percentage of the overall available time.
Theoretical and practical issues
The solution process includes several decisions (6,26,27), some of them concerning the amount of cognitive resources to allocate to the different cognitive processes (i.e. attention to and recall of the instructions, perception of the map, calculation and reasoning, etc.) during the solution search. Some other decisions, strictly relevant to the task achievement, refer to which direction to take and how to move (going on foot or by underground) in the town, which errands to perform and which to leave out, which criteria to distinguish them (i.e priority, needed time, interconnections amongs the errands). ITER records the result of the abovementioned complex cognitive activity, namely the candidate's behavioral responses which materialize in the path distance covered and in a set of more or less numerous performed errands.
Task complexity (7,29)
The problem situation contains much irrelevant information, which overloads the solution search. ITER proposes a naturalistic, realistic but standardized situation in order to ensure the same operational environment to all candidates. The information provided with the instructions implies inferential and not immediately identifiable priorities. The variables to take into account are many, interconnected, and presumably exceed the working memory of problem solvers. Every move has an immediate and delayed consequence that should be considered for their possible effects on the successive itinerary.
Limitations
The first limitation to keep in mind is the available time span.The candidate has to conclude the actions in a predetermined amount of the allotted time in order to accomplish the superordinate task.
Solution efficiency and complexity
The solution efficiency can be evaluated considering the proportion between the effective work (the time spent in the errands) and the work needed to produce it (the time spent in the movements). The outcome in ITER is measured as a percentage figure computed by the relation between the time, devoted to the errands, and the total time spent in the town.
The solution, represented by the route covered and the errands performed, can be described as a sequence of decisions and actions. The criteria regulating such behaviors indicate the articulation level of the constructs used by the candidate.
A uniform, invariable behavior, failing in strategy,presumably will denote the application of poor, undiversified constructs (5). In order to allow a more generalized evaluation, the above mentioned outcome is corrected according to a parameter, taking into account the tasks concentration in the assigned space and time. In practice, the value of this index is equal to 1 when the performance is comparable in quantity with that which might be obtained with a casual "route" without particular strategies. The value becomes higher than 1 as larger when the planned intervention by the candidate increases.
Rules
The task includes several rules,some concerning the movements' direction on the streets, some others regarding the ways of performing the chosen errands, to read again the list of errands, to end the test etc.
Errors (22)
The problem situation contains several limitations and their infringement, recorded as an error, is not permitted by the system. This makes the subject learn more or less quickly from the errors he/she made.
The error rate can therefore appropriately represent the individual resistance to change, the execution rigidity, responsible sometimes for dogmatic perseveration, obstinacy and, probably, a negative predictor of flexibility in learning new behaviors.
Quantitative indicators
ITER provides the following parameters describing the performance:
1. Duration of the test: the real time spent to complete the test
2. Route time: the fictitious"time" used for movements.
3. Action time: thefictitious"time" employed for the actions in the list of errands.
4. Total time: the sum of the two abovementioned times.
5. Accomplished actions : the number of performed errands.
6. Omitted actions: the errands omitted even when the candidate was topographically in a position to perform them.
7. Serious omissions: the number omitted errands, notwithstanding their being indispensable in relation to the task.
8. Outcome of the test: the percentage relation between the fictitious" time" devoted to the accomplishment of actions and the total fictitious "time".
9. Weighted outcome (or complexity of the solution): the value which compares a casual performance (equal to 1) to a strategic, planned performance (higher than 1).
10. Errors: (expressed in absolute frequencies)
- Wrong route directions (Err.1)
- Errand already carried out (Err.2)
- Place not in the list (Err.3)
- Priority (Err.4): attempts to carry out a task without following the logical sequence
INTELLIGENCE STRUCTURE TEST (IST)
This test is the Italian translation and adaptation ( 3,10 ) of the German Intelligenz Struktur Test (1). Eight of the nine subtest were administered, namely the General Information (GI) which assesses the general word knowledge, the Conceptual Discrimination (CD) that measures the ability to discriminate the different or similar verbal meaning in a group of given words, the Analogical Reasoning (AR) or the ability to understand the relationships amongs the words, the Conceptual Classification (CC) or the ability to identify a synonym, the Arithmetic Problems (AP) or the ability to perform simples calculations, Numerical Reasoning (NR) or the ability to continue a numerical series, Perceptual Fluidity(PF) or the ability to perform a mental reconstruction of geometrical figures, Spatial Ability (SA) or the ability to deal with the tridimensional spatial representations.
Results
Table 1 shows the descriptive statistics for the main quantitative indicators provided by ITER in the sample above mentioned.
Tab.1
| ITER Variables | Mean |
Std Deviation |
Minimum |
Maximum |
| Real Time(Duration of the test) | 14' 05" |
5' 18" |
3' 16" |
33' 09" |
| "Route" Time | 10 h 28' |
8h 31' |
2h 45' |
72h 25' |
| "Action" Time | 2 h 16' |
38' |
50' |
2h 45' |
| "Total" Time | 12 h 44' |
6h 49' |
4h |
75h 10' |
| Accomplished actions | 13.9 |
4.27 |
4 |
17 |
| Omitted actions | 1.86 |
3.04 |
0 |
19 |
| Serious omissions | .83 |
.75 |
0 |
3 |
| Outcome % | 20.99 |
6.63 |
3.65 |
36.36 |
| H (Weighted outcome) | .61 |
.19 |
.10 |
1.06 |
| Err.1 | 15.92 |
13.73 |
0 |
67 |
| Err.2 | 1.04 |
2.16 |
0 |
16 |
| Err.3 | 12.45 |
28.67 |
0 |
256 |
| Err.4 | 2.43 |
2.16 |
0 |
12 |
The mean time spent to perform the test in this group is almost 15 minutes, including the reading of instructions. The task resulted to be very difficult in this group as only 3 subjects out of 100 were able to conclude the actions within the fictitious "time" allowed.
Table 2 shows the significant correlations of the subtests and total score of IST with the ITER quantitative indicators
Tab.2
| GI | CD | AR | CC | AP | NR | PF | SA | Total score IST | |
| Real time | r= -.23 P=.03 |
r= -.22 P=.04 |
r= -.23 P= .03 |
||||||
| "Total" Time | r= -.37 P=.006 |
||||||||
| H (Weighted outcome) | r= .26 P=.01 |
r= .26 P=.01 |
r= .27 P=.01 |
||||||
| Err 1 | r= -.47 P=.000 |
||||||||
| Err 2 | r= -.23 P=.03 |
r= -.26 P=.01 |
r= -.23 P=.03 |
||||||
| Err 4 | r= -.23 P=.03 |
r= -.21 P=.05 |
r= -.22 P=.04 |
The total scores of IST show a positive correlation with the weighted outcome ( H ), a negative correlation with the real and fictitious time used up during the test and with some errors. The subjects who obtained higher scores in the general information (GI) and arithmetic problems (AP) scales completed ITER in a shorter time. Likewise the "total" time used up in the fictitious operations is less in the subjects who scored higher in the conceptual discrimination (CD) scale. Subjects who produced more complex solutions (H) scored higher in the numerical scales (AP and NR). Wrong route directions (Err 1) are higly correlated with lower scores in the verbal meaning discrimination (CD), unsuitable actions like attempts to perform an errand already carried out (Err 2) correlate with lower scores in the same scale (CD) and in the arithmetic ability (AP). Subjects who scored lower in the "logical" scales (AR and CC) made more priority errors (Err.4) that is the attempts to carry out a task without following the logical sequence.
Discussion
The results seem to give evidence of a relationship between intelligence (total scores of IST) and some of its components (several subtests or scales of IST) with several performance indicators provided by ITER. A preliminary exam shows a significant role played by the verbal (GI, CD, AR, and CC) and numerical factors (AP and NR) of IST in the complex cognitive processes required by the test. Surprisingly the spatial factor (FP and SA) seem to play an unimportant role in the task solution process. The negative relationship between real and "total" time, used up by the subjects, and some intelligence measures support the results of previous studies which have shown an increased response or processing time with increasing level of complexity of items (6, 25) in a test and the high correlation between the speed of performance and problem soving ability (8). It is interesting to notice that ITER shows positive as well as negative relations with the intelligence measures. The test results to be able to measure several positive and negative parameters involved in the planning and decision making process.
The low percentage of subjects able to conclude the actions in the fictitious "time" allowed appear to point out that most of the subjects don't take a systematic approach to planning and underestimate the time required for accomplishing the task. These findings confirm the results of the early studies performed by Hayes-Roth (9) where a tendency to plan at high level of abstraction and a motivational factor to accomplish all or most of the tasks under consideration were identified as the main reasons which led to underestimate all time-consuming components of the task.
Conclusions and perspectives
In conclusion ITER is an attempt to introduce task-oriented instruments in the selection process for Antarctic expedition personnel. The test doesn't require too much time to be completed, on an average only 15 minutes. However it is a complex test; it does indeed require the simultaneous and structured processing of information from different domains and an action planning according to priorities, operating rules, time constraints. In front of this complexity the test presents a standard and stable scenario which permits to assess the individual differences. In fact it provides several indexes of individual performance. The preliminary applications of the computerized test ITER appear to be promising. The positive and negative correlation found between some parameters measured in the test and various components of intelligence seem to support furthermore the complexity of the task and suggest a complex, dynamic relationship between intelligence and decision making. Future applications should give useful data to test the ecological validity of the instrument. The test was applied recently in a group of French expeditioners during the winter over in Antarctica and the data are going to be processed in the next future.
References