In an effort to standardize academic application procedures, the authors developed the Standardized Letters of Recommendation (SLR) to capture important cognitive and noncognitive qualities of graduate school candidates. The SLR, which consists of seven scales, is applied to an intern-selection scenario. Both professor ratings (n = 414) during the application process and mentor ratings of the selected students (n = 51) are collected using the SLR. A multidimensional Rasch investigation suggests that the SLR displays satisfactory internal consistency, model fit, and item fit. The two cognitive scales, knowledge and analytical skills, are found to be the best predictors for intern selection. The professor ratings are systematically higher than the mentor ratings. Possible reasons for the rating discrepancies are discussed. Also, implications for how the SLR can be used and improved in other selection situations are suggested.
Aamont, M.G., Bryan, D.A., & Whitcomb, A.J. (1993). Predicting performance with letters of recommendation. Public Personnel Management , 22, 81-90.
2.
Aamont, M.G., Nagy, M.S., & Thompson, N. (1998, June). Employment references: Who are we talking about?Paper presented at the annual meeting of the International Personnel Management Association Assessment Council, Chicago.
3.
Adams, R.J., Wilson, M., & Wang, W. (1997). The multidimensional random coefficients multinomial logit model. Applied Psychological Measurement, 21, 1-23.
4.
Agresti, A. (1996). An introduction to categorical data analysis. New York: Wiley.
5.
Akaike, H. (1974). A new look at the statistical model identification . IEEE Transactions on Automatic Control, 19, 716-723.
6.
Akaike, H. (1987). Factor analysis and AIC. Psychometrika , 52, 317-332.
7.
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing . Washington, DC: American Psychological Association.
8.
Andrich, D. (1978). A rating formulation for ordered response categories . Psychometrika, 43, 561-573.
9.
Bock, R.D., & Aitken, M. (1981). Marginal maximum likelihood estimation of item parameters: Applications of an EM algorithm. Psychometrika , 46, 443-459.
10.
Bozdogan, H. (1987). Model selection and Akaike's information criterion (AIC): The general theory and its analytical extensions. Psychometrika, 52, 345-370.
11.
Briggs, D.C., & Wilson, M. (2003). An introduction to multidimensional measurement using Rasch models. Journal of Applied Measurement, 4, 87-100.
12.
Ceci, S.J., & Peters, D. (1984). A naturalistic study of the effects of confidentiality . American Psychologist, 39, 29-31.
13.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
14.
Enright, M.K., & Gitomer, D.H. (1989). Toward a description of successful graduate students (GRE Board Professional Rep. No. 89-09). Princeton, NJ: Educational Testing Service.
15.
International Personality Item Pool. (2001 ). A scientific collaboratory for the development of advanced measures of personality traits and other individual differences. Retrieved November 28, 2006, from http://ipip.ori.org/
16.
Kim, S., & Kyllonen, P.C. (2006). Rasch rating scale modeling of data from the standardized letters of recommendation (RR-06-33). Princeton, NJ: Educational Testing Service..
17.
Knouse, S.B. (1983). The letter of recommendation: Specificity and favorability of information. Personnel Psychology, 36, 331-341.
18.
Kuncel, N.R., Hezlett, S.A., & Ones, D.S. (2001). A comprehensive meta-analysis of the predictive validity of the Graduate Record Examination. Implications for graduate student selection and performance. Psychological Bulletin, 27, 162-181.
19.
Kyllonen, P.C., & Kim, S. (2005, April). Personal qualities in higher education: Dimensionality of faculty ratings of students applying to graduate school . Paper presented at the annual meeting of the American Educational Research Association, Montreal, Canada.
20.
Kyllonen, P.C., Walters, A.M., & Kaufman, J. (2005). Noncognitive constructs and their assessment in graduate education: A review. Educational Assessment, 10, 153-184.
21.
Lopez, S.J., Oehlert, M.E., & Moberly, R.L. (1996). Selection criteria for American Psychological Association-accredited internship programs: A survey of training directors . Professional Psychology: Research and Practice, 27, 518-520.
22.
McCarthy, J.M., & Goffin, R.D. (2001). Improving the validity of letters of recommendation: An investigation of three standardized reference forms. Military Psychology, 13, 199-222.
23.
Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13-103). Washington, DC: American Council on Education & National Council on Measurement in Education .
24.
Messick, S. (1995). Standards of validity and the validity of standards in performance assessment. Educational Measurement: Issues and Practice, 14, 5-8.
25.
Miller, R.K., & Van Rybroek, G.J. (1988). Internship letters of recommendation: Where are the other 90%? Professional Psychology: Research and Practice, 19, 115-117.
26.
Range, L.M., Menyhert, A., Walsh, M.L., Hardin, K.N., Ellis, J.B., & Craddick, R. (1991). Letters of recommendation: Perspectives, recommendations, and ethics. Professional Psychology: Research and Practice , 22, 389-392.
27.
Robiner, W.N., Saltzman, S.R., Hoberman, H.M., Semrud-Clikeman, M., & Schirvar, J.A. (1998). Psychology supervisors' bias in evaluations and letters of recommendation. Clinical Supervisor, 16, 49-72.
28.
Schwarz, G. (1978). Estimating the dimension of a model. Annals of Statistics, 6, 461-464.
29.
Walpole, M.B., Burton, N.W., Kanyi, K., & Jackenthal, A. (2002). Selecting successful graduate students: In-depth interviews with GRE users (Graduate GRE Board Research Rep. No. 99-11). Princeton, NJ: Educational Testing Service.
30.
Walters, A.M., Kyllonen, P.C., & Plante, J.W. (2003, April). Preliminary research to develop a standardized letter of recommendation. Paper presented at the annual meeting of the American Educational Research Association, San Diego, CA.
31.
Wang, W. (1999). Direct estimation of correlations among latent traits within IRT framework. Methods of Psychological Research Online, 4, 47-70.
32.
Wang, W., Wilson, M., & Adams, R.J. (1997). Rasch models for multidimensionality between and within items. In M. Wilson & G. Engelhard (Eds.), Objective measurement: Theory into practice (Vol. IV; pp. 139-154). Norwood, NJ: Ablex.
33.
Wilson, M. (2005). Constructing measures: An item response modeling approach. Mahwah, NJ: Erlbaum.
34.
Wolfe, E.W., & Smith, E.V., Jr. (2007a). Instrument development tools and activities for measure validation using Rasch models: Part I. Instrument development tools. Journal of Applied Measurement, 8, 97-123.
35.
Wolfe, E.W., & Smith, E.V., Jr. (2007b). Instrument development tools and activities for measure validation using Rasch models: Part II. Validation Activities. Journal of Applied Measurement, 8, 204-233.
