This research used logistic regression to model item responses from a popular 360-degree assessment-for-development survey. The model used method of survey delivery and rater group to identify items exhibiting differential item functioning (DIF). The methods of survey delivery were pencil-and-paper and online by Web page. The rater groups were self, boss, peer, and direct report. The sample consisted of 374 survey families where a survey family was a set of four surveys: self, boss, peer, and direct report. Half the survey families were pencil-and-paper administration; half were online administration. The flagging procedure used effect size from Wald chi-square statistics. Results indicated little evidence that DIF existed due to method of survey delivery, lending additional support for the use of the Internet to deliver 360-degree surveys. Approximately 10% of items exhibited DIF attributable to rater group, though in every instance the magnitude of the DIF was small, suggesting that the impact of DIF on the 360-degree feedback could be slight. There was no evidence of an interaction between method of delivery and rater group. The maximum likelihood parameter estimates indicated DIF resulted from either hierarchical complexity or contingency theory. This research suggested that such forms of DIF could be naturally occurring phenomena in 360- degree assessment because both contingency and complexity can influence the perceptions by raters of a manager’s performance.
Get full access to this article
View all access options for this article.
References
1.
Anastasi, A. (1982). Psychological testing (5th ed.). New York: Macmillan.
2.
Arnau, R. C., Thompson, R. L., & Cook, C. (2000). Do different response formats change the latent structure of responses? An empirical investigation using taxometric analysis. Educational and Psychological Measurement, 1, 23-44.
3.
Barnette, J. J. (2000). Effects of stem and Likert response option reversals on survey internal consistency: If you feel the need, there is a better alternative to using those negatively worded items. Educational and Psychological Measurement, 60, 361-370.
4.
Benson, J. (1987). Detecting item bias in affective scales. Educational and Psychological Measurement, 47, 55-67.
5.
Benson, J., & Hocevar, D. (1985). The impact of item phrasing on the validity of attitude scales for elementary school children. Journal of Educational Measurement, 22, 231-240.
6.
Clauser, B. E., & Mazor, K. M. (1998). Using statistical procedures to identify differentially functioning test items. Educational Measurement: Issues and Practices, 2, 31-44.
7.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Hillsdale, NJ: Lawrence Erlbaum.
8.
Collins, W. C., Raju, N. S., & Edwards, J. (2000). Assessing Differential Functioning in a satisfaction scale. Journal of Applied Psychology, 85, 451-461.
9.
Craig, B., Raju, N. S., Zieleskiewicz, J. R., & Formen, A. (1999). Measurement equivalence of the benchmarks ratings across four rating sources. Paper presented at SIOP Conference, Atlanta, GA.
10.
Cronbach, L. J. (1950). Further evidence on response sets and test design. Educational and Psychological Measurement, 10, 3-31.
11.
Donovan, M. A., Drasgow, F., & Probst, T. M. (2000). Does computerizing paper-and-pencil job attitude scales make a difference? New IRT analyses offer insight. Journal of Applied Psychology, 85, 305-313.
12.
Drasgow, F., & Kanfer, R. (1985). Equivalence of psychological measurement in heterogeneous populations. Journal of Applied Psychology, 70, 662-680.
13.
Evan, W. M., & Miller, J. R. (1969). Differential effects on response bias of computer vs. conventional administration of a social science questionnaire: An exploratory methodological experiment. Behavioral Science, 14, 216-227.
14.
Facteau, J. D., & Craig, S. B. (2001). Are performance appraisal ratings obtained from different rating sources comparable? Journal of Applied Psychology, 86(2), 215-227.
15.
Fiedler, F. E. (1978). The contingency model and the dynamics of the leadership process, In L. Berowitz (Ed.)., Advances in experimental social psychology. New York: Academic Press.
16.
Fiedler, F. E., & Chemers, M. M. (1982). Improving leadership effectiveness: The leader match concept (2nd ed.). New York: John Wiley.
17.
Finegan, J. E., & Allen, N. J. (1994). Computerized and written questionnaires: Are they equivalent? Computers in Human Behavior, 10, 483-496.
18.
Hambleton, R. H., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of item response theory. Newbury Park, CA: Sage.
19.
Hershey, P., & Blanchard, K. H. (1969). Life cycle theory of leadership. Training and Development Journal, 23, 26-34.
20.
Hershey, P., & Blanchard, K. H. (1982). Management of organizational behavior (4th ed.). Englewood Cliffs, NJ: Prentice Hall.
21.
Holland, P. W., & Thayer, D. T. (1988). Differential item performance and the Mantel-Haenszel procedure. In H. Wainer & H. I. Braun (Eds.), Test validity (pp. 129-145). Hillsdale, NJ: Erlbaum.
22.
Hsieh, F. Y. (1989). Sample size tables for logistic regression. Statistics in Medicine, 8, 795-802.
23.
Jacques, E. (1996). Requisite organization: A total system for effective managerial organization and managerial leadership for the 21st century. Cambridge, MA: Cason Hall.
24.
Jacques, E., & Clement, S. D. (1994). Executive leadership: A practical guide to managing complexity. Cambridge, MA: Basil Blackwell.
25.
Kantor, J. (1991). The effects of computer administration and identification on the Job Descriptive Index (JDI). Journal of Business and Psychology, 5, 309-323.
26.
Kiesler, S., & Sproull, L. S. (1986). Response effects in the electronic survey. Public Opinion Quarterly, 50, 402-413.
27.
King, W. C., & Miles, E. W. (1995). A quasi-experimental assessment of the effect of computerizing noncognitive paper-and-pencil measurements: A test of measurement equivalence. Journal of Applied Psychology, 80, 643-651.
28.
Knight, R. G., Chisholm, B. J., Marsh, N. V., & Godfrey, H. P. (1988). Some normative, reliability, and factor analytic data for the revised UCLA Loneliness Scale. Journal of Clinical Psychology, 44, 203-206.
29.
Koson, D., Kitchen, C., Kochen, M., & Stodolosky, D. (1970). Psychological testing by computer: Effect on response bias. Educational and Psychological Measurement, 30, 803-810.
30.
Laffitte, L. J., Raju, N. S., Scott, J. C., & Fasolo, P. M. (1998, April). Examination of the measurement equivalence of a 360° feedback assessment with confirmatory factor analysis and item response theory. Paper presented at the annual conference of the Society for Industrial and Organizational Psychology, Dallas, TX.
31.
Lautenschlager, G. J., & Flaherty, V. L. (1990). Computer administration of questions: More desirable or more social desirability? Journal of Applied Psychology, 75, 310-314.
32.
Lee, J., Moreno, K. E., & Sympson, J. B. (1986). The effects of mode of test administration on test performance. Educational and Psychological Measurement, 46, 467-474.
33.
Likert, R. A. (1932). A technique for the measurement of attitudes. Archives of Psychology, 140, 44-53.
34.
Lindsey, E., Homes, V., & McCall, M. W., Jr. (1987). Key events in executives’lives (Tech. Rep. No. 32). Greensboro, NC: Center for Creative Leadership.
35.
Loehlin, J. C. (1992). Latent variable models: An introduction to factor, path and structural analysis (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
36.
Lord, F. (1980). Application of item response theory to practical testing problems. Hillsdale, NJ: Erlbaum.
37.
Lukin, M. E., Dowd, T., Plake, B. S., & Kraft, R. G. (1985). Comparing computerized MMPI. Journal of Personality Assessment, 13, 407-412.
38.
Lushene, R. E., O’Neil, H. F., & Dunn, T. (1974). Equivalent validity of a completely computerized MMPI. Journal of Personality Assessment, 38, 353-361.
39.
Mantel, N., & Haenszel, W. (1959). Statistical aspects of the analysis of data from retrospective studies of disease. Journal of the National Cancer Institute, 22, 719-748.
40.
Maurer, T. J., Raju, N. S., & Collins, W. C. (1998). Peer and subordinate performance appraisal measurement equivalence. Journal of Applied Psychology, 83, 693-702.
41.
McCall, M. W., Jr., & Lombardo, M. M. (1983, February). What makes a top executive? Psychology Today, pp. 26-31.
42.
McCall, M. W., Jr., Lombardo, M. M., & Morrison, A. M. (1988). The lessons of experience: How successful executives develop on the job. Lexington, MA: Lexington Books.
43.
Miller, T. R., & Spray, J. A. (1993). Logistic discriminant function analysis for DIF identification of polytomously scored items. Journal of Educational Measurement, 30, 107-122.
44.
Oshima, T. C., Raju, N. S., & Flowers, C. (1997). Development and demonstration of multidimensional IRT-based internal measures of differential functioning of items and tests. Journal of Educational Measurement, 34, 253-272.
45.
Penny, J., & Johnson, R. L. (1999). How group differences in matching criterion distribution and IRT item difficulty can influence the magnitude of the Mantel-Haenszel chi-square DIF index. Journal of Experimental Education, 67, 343-366.
46.
Pilotte, W. J., & Gable, R. K. (1990). The impact of positive and negative item stems on the validity of a computer anxiety scale. Educational and Psychological Measurement, 50, 603-610.
47.
Potosky, D., & Bobko, P. (1997). Computer versus paper-and-pencil administration mode and response distortion in noncognitive selection tests. Journal of Applied Psychology, 82, 293-299.
48.
Raju, N. S. (1999). Psychometric analysis of the Benchmarks survey. Chicago: Center for Research and Service, Illinois Institute of Technology.
49.
Raju, N. S., van der Linden, W., & Fleer, P. (1995). An IRT-based internal measure of test bias with applications for differential item functioning. Applied Psychological Measurement, 19, 353-368.
50.
Reckase, M. D. (1979). Unifactor latent trait models applied to multi-factor tests: Results and Implications. Journal of Educational Statistics, 4, 207-230.
51.
Reilly, R. R., & Warech, M. A. (1993). The validity and fairness of alternatives to cognitive tests. In L. C. Wing & B. R. Gifford (Eds.), Policy issues in employment testing (pp. 131-224). Norwell, MA: Kluwer Academic.
52.
Riordan, C., & Vandenberg, R. (1994). A central question in cross-cultural research: Do employees of different cultures interpret work-related measures in an equivalent manner? Journal of Management, 20, 643-671.
53.
Rogers, H. J. (1989). A logistic regression procedure for detecting item bias. Dissertation Abstracts International, 50, 3928A. (University Microfilms No. 90-11,788)
54.
Rogers, H. J., & Swaminathan, H. (1993). A comparison of logistic regression and Mantel-Haenszel procedures for detecting differential item functioning. Applied Psychological Measurement, 17, 105-116.
55.
Rosenfeld, P., Doherty, L. M., Vicino, S. M., Kantor, J., & Greaves, J. (1989). Attitude assessment in organizations: Testing three microcomputer-based survey systems. Journal of General Psychology, 116, 145-154.
56.
Rossi, P. H., Wright, J. D., & Anderson, A. B. (1983). Handbook of survey research. New York: Academic Press.
57.
Rozensky, R. H., Honor, L. F., Rasinski, K., Tovian, S. M., & Herz, G. I. (1986). Paper-and-pencil versus computer-administered MMPIs: A comparison of patient’s attitudes. Computers in Human Behavior, 2, 111-116.
58.
Samejima, F. (1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika Monograph, 17.
59.
Samejima, F. (1979). A new family of models for the multiple choice item (Office of Naval Research Report 79-4). Knoxville: University of Tennessee.
60.
Skinner, H. A., & Allen, B. A. (1983). Does the computer make a difference? Computerized versus face-to-face versus self-report assessment of alcohol, drug, and tobacco use. Journal of Consulting and Clinical Psychology, 51, 267-275.
61.
Smith, P. C., Kendall, L., & Hulin, C. L. (1969). The measurement of satisfaction in work and retirement. Chicago: Rand McNally.
62.
Stanton, J. M. (1998). An empirical assessment of data collection using the Internet. Personnel Psychology, 51(3), 709-725.
63.
Swaminathan, H., & Rogers, H. J. (1990). Detecting differential item functioning using logistic regression procedure. Journal of Educational Measurement, 27, 361-370.
64.
Van de Vijver, F. J. R., & Harsveld, M. (1994). The incomplete equivalence of the paper-and-pencil and computerized versions of the General Aptitude Test Battery. Journal of Applied Psychology, 79, 852-859.
65.
Van Velsor, E., & Leslie, J. B. (1991). Feedback to managers: Vol. 1. A guide to evaluating multirater feedback instruments. Greensboro, NC: Center for Creative Leadership.
66.
White, D. M., Clements, C. B., & Fowler, R. D. (1985). A comparison of computer administration with standard administration of the MMPI. Computers in Human Behavior, 1, 153-162.
67.
Whittemore, A. (1981). Sample size for logistic regression with small response probability. Journal of the American Statistical Association, 42, 415-427.
68.
Wilson, F. R., Genco, K. T., & Yager, G. G. (1985). Assessing the equivalence of paper-and-pencil vs. computerized tests: Demonstration of a promising methodology. Computers in Human Behavior, 1, 265-275.
Yukl, G. A., & van Fleet, D. (1982). Cross-situational, multi-method research on military leader effectiveness. Organizational Behavior and Human Performance, 30, 87-108.
