Reference reports : a meta-analytic review of predictive validity and an experimental study of rating accuracy : a thesis presented in partial fulfilment of the requirements for the degree of Doctor of Philosophy in Psychology at Massey University

Abstract

Reference reports are a commonly used selection method in New Zealand and overseas. Although popular with practitioners, they have attracted little attention from researchers. To ascertain the predictive validity of reference reports a meta-analytic review was conducted. Results of the preliminary analysis provided a bare-bones estimated mean validity of 0.15, and a fully corrected estimated mean validity of 0.32. Substantial variance remained unaccounted for following corrections for sampling error. Subsequent moderated meta-analyses, based on degree of structure and psychometric soundness of the reference reports, was found to account for much of the variation in observed validity coefficients. Highly structured reports were found to be consistently superior to unstructured reports. Improvements in the validity of highly structured reports can be attributed to the control of leniency in ratings. However, no studies to date have evaluated the accuracy of referees' ratings. Drawing on the performance rating literature, an experimental study examining the influence of scale format, ratee characteristics, rating purpose, and rater affect on the accuracy of ratings was implemented. Asymmetrical, positively toned scales were found to reduce leniency in ratings compared to unstructured and Likert-type rating forms. Raters who expressed liking for the ratee were more lenient in their ratings compared to raters who expressed neutral or antagonistic feelings toward the ratee. No significant effects for rating purpose and ratee characteristics were apparent. Overall, the investigation points to deficiencies in the way reference reports are presently employed, and highlights the need for a more rigorous approach in their development and application.