The purpose of this study was to evaluate perceptions of the Defense Finance and Accounting Service (DFAS) workforce regarding a comparison of the former five-element and new pass/fail rating systems. The first phase of this project conducted in November 1996, primarily assessed workforce perceptions under the prior system. A survey was used to explore employee attitudes toward possible changes in their personnel evaluation program (Vandivier, 1996). Five questions were presented for their responses to help gauge their potential acceptance of the proposed changes. The questions dealt with three types of rating plans and whether employees might prefer certain changes in performance awards and critical elements relating to team performance. Employee comments were also requested in the survey. Responses indicated that the prior rating system of five elements (Exceptional to Unacceptable) as least desirable in terms of first choice, the three-level plan (Exceptional, Fully Successful and Unacceptable) seemed marginally desirable, and the basic "pass/fail" rating plan was given the most initial first choices. (Vandivier, Bridges, & Waldkoetter, 1996).

The second phase, focus of this study, assessed perceptions under the new pass/fail system (Vandivier, 1998). Similar survey forms as administered to the workforce during November 1996, were prepared and distributed from late 1997 to January 1998. These surveys were sent to and administered within the selected sample of directorates. Both surveys (prior and current study) included questionnaire items to be completed by employees, and supervisors as employees, and items completed only by supervisors. The second phase survey included two additional questionnaire items (23 and 41) asking about their preference for the new versus prior systems with item 23 for employees and 41 for supervisors. Also, the present study’s survey was adapted so respondents could indicate their last performance rating under the new pass/fail system,rather than the prior five-element (Exceptional to Unacceptable) system. The parallel structure of numerous questionnaire items allowed comparisons of employee and supervisor perceptions.

Structure of the performance management post survey (second phase) consisted of six common background items, 17 scaled items covering performance planning, performance feedback/counseling, and performance appraisal and awards for employees and supervisor-employees, and four supervisory only background items, with 14 scaled items covering content areas indicated above. Both sets of scaled items asked respondents to indicate the extent to which they "agreed or disagreed" to given statements on a five-point scale from "strongly disagree" to "strongly agree" with a parallel column for "don’t know". A total of 1500 survey forms were distributed through the DFAS Center directorates with a total of 746 (49.7%) collected suggesting a high confidence level (95%) in the overall results for this post survey.

A greater number of respondents were in grades 11 and 12 (41.5%), followed by those in grades 6 through 10 (22.5%), with grades 1 through 5 (19.1%), grades 13 or above (16.3%), and others (0.7%). Respondents indicated a range of years in government service with 16 to 25 years (31.1%), followed by 11 to 15 (20.6%), 6 to 10 (18.6%), with 26 or more (16.8%), and less than a year to 5 (12.9%). A total of 88.1% of the collected survey forms were from employees and 11.9% from supervisors. Nearly all respondents (95%) indicated a performance rating of "pass" on the new "pass/fail" rating system as shown by background item 4. Any "don’t know" responses on the five-point scaled items were dismissed for possible analytical use. Analysis consisted of calculating averages (means) and making comparisons across various groupings providing the following most salient results.

The workforce rating yielded a mean of 2.91 (N=720), regarding their preference for the "pass/fail" system, which rather clearly suggests an "undecided" preference, with 3 being the five-point scale midpoint for "undecided." Supervisors responding too, as employees, tended to bias the workforce rating, since the supervisors had a stronger preference for the "pass/fail" system, yielding a mean rating of 4.07 (N=65), which barely exceeded a 4 rating for "agree." Workforce preference for the "pass/fail" system recorded 430 ratings in the "undecided" to "strongly disagree" range, which surely suggests a negative posture even with the inclusion of the more positive supervisory ratings.

While 75% of the supervisors indicated their probable preference for the "pass/fail" system, employees only reflected their lesser preference with just 40% rating in the "agree" to "strongly agree" (4 and 5 point) range. The above differences in ratings and percentages offer, to say the least, very practical significance for further analysis and review, and likely reflect pointed significant differences that may raise further issues in working to implement the "pass/fail" system most effectively in the overall performance management process.

Analysis of the results indicated no significant differences in the overall means for all rated items from pre- to post-testing. Employees yielded mean rating values of 2.96 to 3.25 from the prior survey to the current study with supervisors showing 3.61 to 3.87, respectively (Vandivier, 1998). These rating values would seem to illustrate a cautious perception of the utility of the new "pass/fail" system in its early implementation. Findings further suggest that relatively minor changes have occurred in the performance management processes under the new "pass/fail" system. It has been observed previously that the personnel evaluation and management program must clearly delineate the effectiveness of any rating process and the implementing management guidelines or prepare to address any contradictions with timely and conclusive mediation (Ross & Wieland, 1996).

Estimated times for performances planning comparing employees and supervisors, and for their performance feedback/counseling provided some interesting observations. It was noted that employees (47%) received "less that 30 minutes" performance planning time with supervisors (12%) responding also in that choice. This does not necessarily imply inadequate planning time, since employees (53%) otherwise received "30 minutes or more in performance planning with supervisors (88%) obtaining a similar balance of planning time. There is need for further data analysis to determine what the optimal criteria may be for defining what performance planning times are needed and feasible. Although unlikely, it seems the above planning time estimates may suggest an overall normative condition, but which will vary somewhat among participating directorates and workforce operations.

Further the estimated times comparing employees and supervisors for their respective performance feedback/counseling, revealed employees (43%) received "less that 30 minutes" with 7% of the supervisors falling in this estimated range. Again, there is no evidence to question this operational estimate with employees (57%) and supervisors (93%) receiving "30 minutes or more" for their estimated performance feedback/counseling times. Here it must be observed also that this operational estimate may be relatively accurate, but data analysis must go beyond this stage to develop what required and preferred feedback/counseling times are needed and feasible as well. Perhaps in performance feedback/counseling reviews, this issue can be reconciled by analyzing the differing roles of supervisors and employees to propose operationally feasible times for this critical function. It should be practical to have supervisors and performance management operators express necessary standards for the personnel evaluation program based on their higher levels of task experience. Then, employees may be able to express criterion and more accurate estimates of how performance feedback/counseling and personnel evaluation tasks can be operationally performed (Richman & Quinones, 1996).

In closing, a number of steps are recommended to help the management leadership and the Human Resource Directorate adopt constructive procedures to review the viability of the "pass/fail" rating system (Vandivier, 1998). First the findings should be effectively distributed to provide applicable information to the workforce, and effort should be made to ensure more extensive participation for any future survey efforts. Additionally, there may be sufficient reason to examine the psychological climate in regard to the workforce’s work situations to present the most positive factors in options for changes in the personnel evaluation program (Brown & Leigh, 1996).

Next, a summary review of "high and low" rated items should be made to create a total quality management plan for strategic action. Then, set objectives which can measurably improve the performance management system regarding all aspects of "performance planning, performance feedback/counseling, and performance appraisal and awards." Lastly, the principal study analyst should be retained in a consulting and advisory capacity to clarify existing information and participate in any changes to modify or redesign the newly implemented "pass/fail" system.

Brown, S.P. & Leigh, T.W. (1996). A new look at psychological climate and its relationship to job involvement, effort and performance. Journal of Applied Psychology, 81, 358-368.

Richman, W.L. & Quiriones, M.A. (1996). Task frequency rating accuracy: The effect of task engagement and experience. Journal of Applied Psychology, 81, 512-524.

Ross, W.H. & Wieland, C. (1996). Effects of interpersonal trust and time pressure on managerial mediation strategy in a simulated organizational dispute. Journal of Applied Psychology, 81, 228-248.

Vandivier, P.L. (1996). A "DFAS" performance management survey Unpublished manuscript. Indianapolis, IN.

Vandivier, P.L., Bridges, L.T., & Waldkoetter, R.0. (1996). A survey of proposed changes in a personnel evaluation program. Proceedings of the 38th Annual Conference of the International Military Testing Association. San Antonio, TX: U.S. Air Force.

Vandivier, P.L. (1998). A post pilot performance management survey study. Unpublished manuscript. Indianapolis, IN.