AB-4 - Paper

ITER: A computerized test to assess the decision making skills.   Preliminary applications.

Antonio Peri
Italian Antarctic Project, Rome, Italy

Roberto Mosticoni & Stefano Mosticoni
M-TWO Systems, Rome, Italy

Marta Barbarito
NDA-Argentinian Antarctic Institute, Buenos Aires, Argentina

Introduction

Decision-making and planning skills are important individual qualities in the operational and organizational environment of the Antarctic stations (7,9,14,16,20,21,22,30). In order to assess these abilities in the candidates of the Italian Antarctic expeditions, especially in those having logistic and scientific responsibilities, a computerized test (17), called ITER, was developed.

A preliminary study (13) performed in a population of 110 normal subjects, applicants for the Antarctic expeditions, found correlations between norm referred personality measures (MMPI) and some quantitative indicators of ITER. The results show the cognitive attributes of depressive traits are related to lowest efficacy of less complex solutions; furthermore, self-criticism and uncertainty are related to mazy solving paths and to redundant behaviours with respect to goal orientation.

The general description of the computerized instrument and the results of an experimentation carried out using the Intelligence Structure Test (IST) of R. Amthauer (1) are reported here

Methodology

Sample:  100 men, 26 to 31 years old, mean age: 26.7, all of them law graduated, performed the tests ITER and IST in the same day during the psychological selection of a competitive exam for a job as harbour-masters officer.

Psychological Instruments

ITER

ITER operational environment

The ITER test, inspired on the one used by Barbara and Frederick Hayes-Roth(8), requires the management of a definite time span to accomplish a list of possible errands in a fictitious small town, schematically represented as a map with streets, squares and buildings on the computer screen. Movements in the town are performed using the suitable keys (arrows,etc.) of the keyboard.

Task features

The allowed fictitious time is not sufficient to complete all the tasks mentioned in the test. Therefore the subject has to decide what to do (which errands to perform, which route to take,etc.) in the allotted time. Every move has a time cost: it uses up a predetermined percentage of the overall available time.

Theoretical and practical issues

The solution process includes several decisions (6,26,27), some of them concerning the amount of cognitive resources to allocate to the different cognitive processes (i.e. attention to and recall of the instructions, perception of the map, calculation and reasoning, etc.) during the solution search. Some other decisions, strictly relevant to the task achievement, refer to which direction to take and how to move (going on foot or by underground) in the town, which errands to perform and which to leave out, which criteria to distinguish them (i.e priority, needed time, interconnections amongs the errands). ITER records the result of the abovementioned complex cognitive activity, namely the candidate's behavioral responses which materialize in the path distance covered and in a set of more or less numerous performed errands.

Task complexity (7,29)

The problem situation contains much irrelevant information, which overloads the solution search. ITER proposes a naturalistic, realistic but standardized situation in order to ensure the same operational environment to all candidates. The information provided with the instructions implies inferential and not immediately identifiable priorities. The variables to take into account are many, interconnected, and presumably exceed the working memory of problem solvers. Every move has an immediate and delayed consequence that should be considered for their possible effects on the successive itinerary.

Limitations

The first limitation to keep in mind is the available time span.The candidate has to conclude the actions in a predetermined amount of the allotted time in order to accomplish the superordinate task.

Solution efficiency and complexity

The solution efficiency can be evaluated considering the proportion between the effective work (the time spent in the errands) and the work needed to produce it (the time spent in the movements). The outcome in ITER is measured as a percentage figure computed by the relation between the time, devoted to the errands, and the total time spent in the town.

The solution, represented by the route covered and the errands performed, can be described as a sequence of decisions and actions. The criteria regulating such behaviors indicate the articulation level of the constructs used by the candidate.

A uniform, invariable behavior, failing in strategy,presumably will denote the application of poor, undiversified constructs (5). In order to allow a more generalized evaluation, the above mentioned outcome is corrected according to a parameter, taking into account the tasks concentration in the assigned space and time. In practice, the value of this index is equal to 1 when the performance is comparable in quantity with that which might be obtained with a casual "route" without particular strategies. The value becomes higher than 1 as larger when the planned intervention by the candidate increases.

Rules

The task includes several rules,some concerning the movements' direction on the streets, some others regarding the ways of performing the chosen errands, to read again the list of errands, to end the test etc.

Errors (22)

The problem situation contains several limitations and their infringement, recorded as an error, is not permitted by the system. This makes the subject learn more or less quickly from the errors he/she made.

The error rate can therefore appropriately represent the individual resistance to change, the execution rigidity, responsible sometimes for dogmatic perseveration, obstinacy and, probably, a negative predictor of flexibility in learning new behaviors.

Quantitative indicators

ITER provides the following parameters describing the performance:

1. Duration of the test: the real time spent to complete the test

2. Route time: the fictitious"time" used for movements.

3. Action time: thefictitious"time" employed for the actions in the list of errands.

4. Total time: the sum of the two abovementioned times.

5. Accomplished actions : the number of performed errands.

6. Omitted actions: the errands omitted even when the candidate was topographically in a position to perform them.

7. Serious omissions: the number omitted errands, notwithstanding their being indispensable in relation to the task.

8. Outcome of the test: the percentage relation between the fictitious" time" devoted to the accomplishment of actions and the total fictitious "time".

9. Weighted outcome (or complexity of the solution): the value which compares a casual performance (equal to 1) to a strategic, planned performance (higher than 1).

10. Errors: (expressed in absolute frequencies)

- Wrong route directions (Err.1)

- Errand already carried out (Err.2)

- Place not in the list (Err.3)

- Priority (Err.4): attempts to carry out a task without following the logical sequence

INTELLIGENCE STRUCTURE TEST (IST)

This test is the Italian translation and adaptation ( 3,10 ) of the German Intelligenz Struktur Test (1). Eight of the nine subtest were administered, namely the General Information (GI) which assesses the general word knowledge, the Conceptual Discrimination (CD) that measures the ability to discriminate the different or similar verbal meaning in a group of given words, the Analogical Reasoning (AR) or the ability to understand the relationships amongs the words, the Conceptual Classification (CC) or the ability to identify a synonym, the Arithmetic Problems (AP) or the ability to perform simples calculations, Numerical Reasoning (NR) or the ability to continue a numerical series, Perceptual Fluidity(PF) or the ability to perform a mental reconstruction of geometrical figures, Spatial Ability (SA) or the ability to deal with the tridimensional spatial representations.

Results

Table 1 shows the descriptive statistics for the main quantitative indicators provided by ITER in the sample above mentioned.

Tab.1

ITER Variables

Mean

Std Deviation

Minimum

Maximum

Real Time(Duration of the test)

14' 05"

5' 18"

3' 16"

33' 09"

"Route" Time

10 h 28'

8h 31'

2h 45'

72h 25'

"Action" Time

2 h 16'

38'

50'

2h 45'

"Total" Time

12 h 44'

6h 49'

4h

75h 10'

Accomplished actions

13.9

4.27

4

17

Omitted actions

1.86

3.04

0

19

Serious omissions

.83

.75

0

3

Outcome %

20.99

6.63

3.65

36.36

H (Weighted outcome)

.61

.19

.10

1.06

Err.1

15.92

13.73

0

67

Err.2

1.04

2.16

0

16

Err.3

12.45

28.67

0

256

Err.4

2.43

2.16

0

12

The mean time spent to perform the test in this group is almost 15 minutes, including the reading of instructions. The task resulted to be very difficult in this group as only 3 subjects out of 100 were able to conclude the actions within the fictitious "time" allowed.

Table 2 shows the significant correlations of the subtests and total score of IST with the ITER quantitative indicators

Tab.2

  GI CD AR CC AP NR PF SA Total score IST
Real time r= -.23
P=.03
      r= -.22
P=.04
      r= -.23
P= .03
"Total" Time   r= -.37
P=.006
             
H (Weighted outcome)         r= .26
P=.01
r= .26
P=.01
    r= .27
P=.01
Err 1   r= -.47
P=.000
             
Err 2   r= -.23
P=.03
    r= -.26
P=.01
      r= -.23
P=.03
Err 4     r= -.23
P=.03
r= -.21
P=.05
        r= -.22
P=.04

The total scores of IST show a positive correlation with the weighted outcome ( H ), a negative correlation with the real and fictitious time used up during the test and with some errors. The subjects who obtained higher scores in the general information (GI) and arithmetic problems (AP) scales completed ITER in a shorter time. Likewise the "total" time used up in the fictitious operations is less in the subjects who scored higher in the conceptual discrimination (CD) scale. Subjects who produced more complex solutions (H) scored higher in the numerical scales (AP and NR). Wrong route directions (Err 1) are higly correlated with lower scores in the verbal meaning discrimination (CD), unsuitable actions like attempts to perform an errand already carried out (Err 2) correlate with lower scores in the same scale (CD) and in the arithmetic ability (AP). Subjects who scored lower in the "logical" scales (AR and CC) made more priority errors (Err.4) that is the attempts to carry out a task without following the logical sequence.

Discussion

The results seem to give evidence of a relationship between intelligence (total scores of IST) and some of its components (several subtests or scales of IST) with several performance indicators provided by ITER. A preliminary exam shows a significant role played by the verbal (GI, CD, AR, and CC) and numerical factors (AP and NR) of IST in the complex cognitive processes required by the test. Surprisingly the spatial factor (FP and SA) seem to play an unimportant role in the task solution process. The negative relationship between real and "total" time, used up by the subjects, and some intelligence measures support the results of previous studies which have shown an increased response or processing time with increasing level of complexity of items (6, 25) in a test and the high correlation between the speed of performance and problem soving ability (8). It is interesting to notice that ITER shows positive as well as negative relations with the intelligence measures. The test results to be able to measure several positive and negative parameters involved in the planning and decision making process.

The low percentage of subjects able to conclude the actions in the fictitious "time" allowed appear to point out that most of the subjects don't take a systematic approach to planning and underestimate the time required for accomplishing the task. These findings confirm the results of the early studies performed by Hayes-Roth (9) where a tendency to plan at high level of abstraction and a motivational factor to accomplish all or most of the tasks under consideration were identified as the main reasons which led to underestimate all time-consuming components of the task.

Conclusions and perspectives

In conclusion ITER is an attempt to introduce task-oriented instruments in the selection process for Antarctic expedition personnel. The test doesn't require too much time to be completed, on an average only 15 minutes. However it is a complex test; it does indeed require the simultaneous and structured processing of information from different domains and an action planning according to priorities, operating rules, time constraints. In front of this complexity the test presents a standard and stable scenario which permits to assess the individual differences. In fact it provides several indexes of individual performance. The preliminary applications of the computerized test ITER appear to be promising. The positive and negative correlation found between some parameters measured in the test and various components of intelligence seem to support furthermore the complexity of the task and suggest a complex, dynamic relationship between intelligence and decision making. Future applications should give useful data to test the ecological validity of the instrument. The test was applied recently in a group of French expeditioners during the winter over in Antarctica and the data are going to be processed in the next future.

References

  1. 1.Amthauer, R., (1970) Intelligenz-Struktur-Test (IST), Gottingen Verlag fur Psychologie
  2. Brehmer, B., European approaches to the study of decision making: focus on process, presentation at the International Applied Military Psychology Symposium, Stockholm, June 1991
  3. Calonghi, L., Polacek, K., Ronco, A., (1974), Test di Struttura dell'Intelligenza TSI di Rudolf Amthauer, Manuale, Organizzazioni Speciali, Firenze
  4. Greeno, J.G. & Simon, H.A., (1984), Problem Solving and Reasoning. Report Number: UPITT/LRDC/ONR/APS-14
  5. Ehrlich, S., (1975), Apprentissage et mémoire chez l'homme, Paris, Presses Universitaires de France, It.tr. (1978) Apprendimento e memoria nell'uomo, Edizioni Paoline, p.77-80
  6. Eysenck, H.J., (1967), Intelligence assessment: A theoretical and experimental approach, British Journal of Educational Psychology,37, 81-98
  7. Frensch, P.A., & Funke, J., (1995), Complex Problem Solving: The European Perspective, Hillsdale, N.J., Lawrence Erlbaum Associates
  1. Halpern, D.F., (1989), Thought and Knowledge. Hillsdale, NJ: LEA, pp.306-403
  2. Hanson, J.R.W., Selection and management of British Antarctic Personnel for prolonged service in Antarctica, 5th SCALOP Symposium, S.Carlos de Bariloche (Argentina), June 1992.
  3. Hayes-Roth, B. & Hayes-Roth, F., (1978), Cognitive processes in planning, Rand/WN-10268-ONR
  1. Hayes-Roth, B., (1980), Estimation of time requirement during planning. Interaction between motivation and cognition. Rand Note N-1581-ONR-Rand Corporation, Santa Monica, CA, 90406
  1. Kaufman, A.S., (1979) Intelligent testing with the WISC-R, Wiley, N.Y.,
  2. Lugg, D.J., (1977), Physiological adaptation and health of an expedition in Antarctica, with comment on behavioural adaptation, ANARE Scientific report N°126
  3. Mion, R., (1972), Analisi fattoriale e presentazione del "Test di struttura dell'intelligenza"-TSI, Bollettino di Psicologia Applicata,N.112-113-114, 134-144
  4. Marsh, J.G., & Shapira, Z., (1982), Behavioral decision theory and organizational decision theory. In G.R. Ungson & D.N.Braunstein (Eds.), Decision making: an interdisciplinary inquiry. Boston, MA, Kent Publishing Co.
  5. Mosticoni, R., Mosticoni, S., Arduino, G., (1992), Setting CASSIOPEA per l'assessment di abilità di base. Validazione concorrente. Rivista di Psichiatria 27, 5°, 209-212
  6. Mosticoni, R., Mosticoni, S., Peri, A., Ruffini, M.C., (1997), MMPI e ITER: uno studio di correlazione fra stile di decisione e aspetti di personalità, Psicoterapia Cognitiva e Comportamentale, 3, 1, 3-11.
  7. Mulligan, R., Selection of station leaders, 6th SCALOP Symposium. Rome, August 1994
  8. Pascual-Leone, J.,(1978), Computational models for metasubjective processes, Behavioral and Brain Sciences, 1, 112-113
  9. Palinkas, L.A., (1985), Sociocultural influences on psychosocial adjustment in Antarctica, Naval Health Research Center, Report N°85-49
  10. Peri, A. (1994), La gestione del tempo:analisi dei problemi in tema di valutazione, Technical Report
  11. Peri, A., Ruffini, M.C., Mosticoni, R.& Mosticoni, S., (1996), Elaborazione di uno strumento psicometrico informatizzato (ITER) per la valutazione della capacità di risolvere problemi di gestione del tempo, Giornale di Medicina Militare, 146, 3, 351-359.
  12. Rasmussen, J., Duncan, K., Leplat, J., (1987), New technology and human error, Chichester, Wiley.
  13. Rivolier, J., (1975), Selection et adaptation psychologiques des sujets vivant en groupes isolés dans l'Antarctique, Paris, CNFRA
  14. Rivolier, J., (1992), Facteurs humains et situations extremes, Paris, Masson
  15. Rothblum, E.D., (1990),Psychological factors the Antarctic, The Journal of Psychology, 124, 3, 253-273
  16. Rowe, H.A.H., (1985), Problem solving and intelligence, Hillsdale, N.J., Lawrence Erlbaum Associates
  17. Simon, H.A., (1957), Models of Man, New York, Wiley, p.198.
  18. Spiegel, M.R., & Byant, N.D., (1978), Is speed processing information related to intelligence and achievement? Journal of Educational Psychology, 70, 904-910
  19. Sternberg, R.J., (1986), Critical Thinking:Its Nature, Measurement, and Improvement National Institute of Education,Washington, D.C.
  20. Sternberg, R.J., (1988), The Psychology of Human Thought. Cambridge: Cambridge University Press,
  21. Sternberg, R.J., (1987), Teorie dell'intelligenza. Milano:Bompiani
  22. Sternberg, R.J. & Frensch, P.A., (1991), Complex problem solving:principles and mechanisms, Hillsdale, N.J., Lawrence Erlbaum Associates
  23. Taylor, A.J.W., (1987), Antarctic Psychology, DSIR SIPC, Wellington