AB-39 - Paper

ARE EXAM SCORES AND OTHER ADVANCEMENT
SYSTEM COMPONENTS RELATED TO NAVY
DUTY STATUS?

Darlene Barrow, Robert Doucette, and Dr. J. Michael Lentz

This presentation will compare the major Navy Duty Status categories on the basis of their exam scores and other advancement system components.  The major Duty Status groups include, Inactive Duty Reservists, active duty Canvasser/Recruiters, active duty Training and Administration of Reserves, and regular Active Duty.  Exam performance comparisons will be based on the same rating exams.  Other advancement system components (award points and performance marks) will also be compared.  A discussion of these results and their implications for different paygrades (E-4/5/6/7) will be presented. 

The Naval Enlisted Advancement System (NEAS) scores exams and advances E-4 candidates in three different duty groups and E‑5 through E-7 candidates in four different duty groups, to the next higher paygrade.  These four groups have different type assignments: 

  1. Inactive duty Reservists (IR) personnel - sailors who work in their rate that have training time and facility limitations, and are assigned to a training site relative to the sailor’s location

  2. Active duty Canvasser/Recruiters (CVR) personnel - sailors on temporary assignment that recruit people for enlistment instead of working in their rate

  3. Active duty Training and Administration of Reserves (TAR) personnel - sailors on continuous assignment that provide training support to naval reservists

  4. Regular Active duty (USN/R) personnel - sailors on continuous assignment who work in their rate

 Active duty Naval Enlisted Advancement candidates for E-4 groups (c) and (d) and E-5 through E-7 groups (b), (c), and (d) participate in the same rating exams on scheduled paygrade dates.  IR candidates take the same rating exams as active duty candidates, but at a later date. 

Figure 1. shows the raw score (RS) to standard score (SS) conversion model used by NEAS for a 150 item test.  The IR candidates utilize the same exam rate raw score to standard score conversion file established by the Active Duty candidates.

Raw score pass cut points for a four-choice 150 item test are derived using the binomial distribution, where n/4 = (150)/4 = 38 is the mean and (npq)1/2 = [(150)(0.25)(0.75)]1/2 = 5 is the standard deviation.  The raw score cut point by paygrade is

E-4 µ +  σ = 38 +  5 = 43
E-5 µ + 2σ = 38 + 10 = 48
E-6/7 µ + 3σ = 38 + 15 = 53

For our purpose, past rating exams for paygrades E-4/5/6 (Series 156) and for paygrade E‑7 (Series 154) were used for

analysis.  Table 1. shows the relationship between the Active Duty and IR administration dates.

 

 

TABLE 1.  ADMINISTRATION DATES 

Exam
Paygrade		 Series Active Duty
Exam Date		 IR Exam
Date
E-4/5/6 156 9-97 2-98
E-7 154 1-97 2-98

For the duty groups by paygrade, we hypothesize the following:

(a)  H0: There are no differences in exam Standard Score (SS) means.

(b)  H0: There are no differences in exam SS means for advanced candidates.

(c)  H0: There are no differences in Performance Mark (PMK) means.

(d)  H0: There are no differences in Award point (AWD) means.

                              METHOD

 One-way Analysis of Variance tests were conducted to determine if there were any significant differences between group means for each null hypothesis.  SPSS (Statistical Package for the Social Sciences) was used to analyze the data.  For each null hypothesis that was rejected by the analysis of variance procedure, SPSS Bonferroni multiple comparison post hoc tests were applied between each set of means.

 RESULTS

All four null hypotheses were rejected for all paygrades.  Significant differences between duty group means are shown in Tables 2 through 5.

TABLE 2.  STANDARD SCORE STATS FOR TOTAL CANDIDATES
E-4 Count Mean Standard
Deviation 		IR CVR TAR USN/R
IR 586 43.6 9.8        
CVR -- -- --        
TAR 522 53.1 9.6 **      
USN/R 23883 49.9 10.0 **   **  
Total 24991 49.8 10.0        
               
E-5 Count Mean Standard
Deviation 		IR CVR TAR USN/R
IR 5153 40.6 9.2        
CVR 12 41.5 8.0        
TAR 1144 48.3 9.3 **      
USN/R 45538 50.0 10.0 ** * **  
Total 51847 49.0 10.3        
               
E-6 Count Mean Standard
Deviation 		IR CVR TAR USN/R
IR 8151 38.2 9.6        
CVR 104 38.7 9.0        
TAR 2380 45.7 9.6 ** **    
USN/R 38022 50.0 10.0 ** ** **  
Total 48657 47.8 10.9        
               
E-7 Count Mean Standard
Deviation 		IR CVR TAR USN/R
IR 6959 38.4 9.9        
CVR 105 33.0 8.7 **      
TAR 2684 45.6 10.1 ** **    
USN/R 38580 50.0 9.9 ** ** **  
Total 48328 48.0 10.7        

NOTE:  * (p < 0.05) and ** (p < 0.01) indicates significant differences which are  shown in the lower triangle.

TABLE 3.  STANDARD SCORE STATS FOR ADVANCED CANDIDATES

E-4

 Count Mean Standard
Deviation 		IR CVR TAR USN/R
IR 528 44.5 9.5        
CVR -- -- --        
TAR 406 55.2 8.8 **      
USN/R 10803 55.4 9.2 **      
Total 11737 54.9 9.5        
               
E-5 Count Mean Standard
Deviation 		IR CVR TAR USN/R
IR 737 47.6 9.1        
CVR 9 43.8 7.9        
TAR 332 54.5 8.4 ** **    
USN/R 6594 60.7 7.9 ** ** **  
Total 7672 59.2 9.0        
               
E-6 Count Mean Standard
Deviation 		IR CVR TAR USN/R
IR 458 48.4 9.1        
CVR 4 52.5 9.7        
TAR 223 57.8 8.4 **      
USN/R 2894 63.5 7.0 ** * **  
Total 3579 61.2 9.0        
               
E-7 Count Mean Standard
Deviation 		IR CVR TAR USN/R
IR 603 44.6 8.8        
CVR 2 48.0 8.5        
TAR 184 55.0 8.3 **      
USN/R 2805 57.7 7.3 **   **  
Total 3594 55.4 9.0        

NOTE: * (p < 0.05) and ** (p < 0.01) indicates significant differences which are  shown in the lower triangle

TABLE 4.  PERFORMANCE MARK STATS FOR TOTAL CANDIDATES
E-4 Count Mean Standard
Deviation 		IR CVR TAR USN/R
IR 584 3.645 0.139        
CVR -- -- --        
TAR 522 3.710 0.158 **      
USN/R 23881 3.712 0.156 **      
Total 24987 3.710 0.156        
               
E-5 Count Mean Standard
Deviation 		IR CVR TAR USN/R
IR 5144 3.688 0.139        
CVR 12 3.642 0.100        
TAR 1144 3.733 0.157 **      
USN/R 45536 3.731 0.146 **      
Total 51836 3.727 0.146        
               
E-6 Count Mean Standard
Deviation 		IR CVR TAR USN/R
IR 8140 3.779 0.132        
CVR 104 3.745 0.139 **      
TAR 2378 3.839 0.114 ** **    
USN/R 38017 3.844   ** **    
Total   0.118          
               
E-7 Count Mean Standard
Deviation 		IR CVR TAR USN/R
IR 6948 3.833 0.120        
CVR 105 3.871 0.067        
TAR 2682 3.930 0.064        
USN/R 38576 3.929 0.070        
Total 48311 3.915 0.086        

NOTE: (p < 0.05) and ** (p < 0.01)  indicates significant differences which are shown in the lower triangle.

TABLE 5.  AWARD STATS FOR TOTAL CANDIDATES
E-4 Count Mean Standard
Deviation 		IR CVR TAR USN/R
IR 586 0.9 1.3        
CVR -- -- --        
TAR 522 0.1 0.5 **      
USN/R 23883 0.4 0.9 **   **  
Total 24991 0.4 0.9        
               
E-5 Count Mean Standard
Deviation 		IR CVR TAR USN/R
IR 5153 2.3 1.9        
CVR 12 2.3 1.9        
TAR 1144 1.9 1.7 **      
USN/R 45538 2.4 2.0 **   **  
Total 51847 2.4 2.0        
               
E-6 Count Mean Standard
Deviation 		IR CVR TAR USN/R
IR 8151 5.3 2.6        
CVR 104 7.3 3.2 **      
TAR 2380 6.7 2.7 **      
USN/R 38022 7.5 2.7 **   **  
Total 48657 7.1 2.8        
               
               

NOTE: ** (p < 0.01) indicates significant differences which are shown in the lower triangle.

                                                                                

                            DISCUSSION

Of considerable interest and concern is percent of candidates failing their enlisted advancement exam.  (Refer to Table 6.)  The USN/R and TAR groups have relatively low fail ratios, 5.14% or less.  The CVR fail percents of 25 to 39 are alarming.  The very significant loss of professional rating knowledge associated with a brief 2 to 3 year recruiting tour needs further investigation.  Of the IR duty group, the E-4 exam candidates have a relatively low fail ratio, 4.61%.  However, the IR fail percent dramatically increases as paygrade increases.  By comparison, the USN/R and TAR groups have only a modest increase in fail ratios with increasing paygrade.  The mean exam standard scores are in general congruence with the ‘fail’ data.

TABLE 6.  PERCENT FAILS BY DUTY GROUP  

IR CVR TAR USN/R
E-4 4.61 -- 0.19 1.97
E-5 16.77 25.00 2.27 3.01
E-6 28.99 25.96 5.13 3.43
E-7 22.00 39.05 5.14 3.36

                           CONCLUSIONS

It is possible that with the progressive passage of time plus very limited exposure to their navy rating, IR and CVR sailors lose much of their functional knowledge.  The data seem to suggest that reservists and canvasser/recruiters need a more active continuing education program.

It should be pointed out that sample size by paygrade for the CVR group is small.  The average exam performance of the CVR and IR duty groups are low and are of concern.  From the perspective of test design, the standard score results are encouraging.  The exams are not intended to be ‘book tests.’  Instead, our goal has always been to test actual on-the-job working knowledge and experience.  The results suggest that sailors working everyday in their profession (rate) are on average performing much better than those that do not.

 Although there are some statistically significant differences in performance marks (evaluations) between groups, there is little functional significance to these differences.

 In a similar vein, the award point comparisons show significant differences in some means between groups at paygrades E-4 and E-5.  However, these differences coupled with the low variability in each group are not considered to be substantial for the award component.  But, at paygrade E-6, the IR award point mean is significantly less than each of the other duty groups.  The magnitude of these differences together with each group’s variability are considered to be substantial for the award component.

 The results of this study suggest that all three measures, exam standard score, performance mark, and award points, are related to navy duty group status.  On the average, sailors (USN/R and TAR) with continuous training and facility access perform better than sailors (IR and CVR) with limited facility access and training time (full-time civilian job or recruiting full-time).