Organizational researchers routinely use attitudinal surveys to track organizational development and identify areas for intervention. However, seemingly trivial changes to the survey instrument, such as question wording or question order, can introduce measurement artifacts leading to differences in observed responses that are not due to actual employee attitudinal change. Traditional methods for assessing the presence of artifacts because of survey changes require additional survey administration using multiple survey forms and random assignment. However, the item response theory method illustrated in this study eliminates the need for additional data collection, offers a more rigorous design, and requires fewer organizational resources.
Baker, F.B. (1995). EQUATE 2.1. A Fortran-based computer program used for equate item parameters between two groups. Madison : University of Wisconsin.
2.
Bless, H., Schwarz, N., & Wänke, M. (2003). The size of context effects in social judgment . In J. P. Forgas, K. D. Williams, & W. von Hippel (Eds.), Social judgments: Implicit and explicit processes (pp. 180-197). New York: Cambridge University Press.
3.
Byrne, B.M., Shavelson, R.J., & Muthen, B. (1989). Testing for the equivalence of factor covariance and mean structures: The issue of partial measurement invariance. Psychological Bulletin, 105(3), 456-466.
4.
Candell, G.L., & Drasgow, F. (1988). An iterative procedure for linking metrics and assessing item bias in item response theory. Applied Psychological Measurement, 12, 253-260.
5.
Church, A.H. (2001). Is there a method to our madness? The impact of data collection methodology on organizational survey results. Personnel Psychology, 54, 937-969.
6.
Church, A.H., & Waclawski, J. (2001). Designing and using organizational surveys: A seven step process. San Francisco: Jossey-Bass.
7.
Cohen, J. (1983). The cost of dichotomization. Applied Psychological Measurement, 7, 249-254.
8.
Colasanto, D., Singer, E., & Rogers, T. (1992). Context effects on responses to questions about AIDS. Public Opinion Quarterly, 56, 515-518.
9.
Donoghue, J.R., & Isham, S.P. (1998). A comparison of procedures to detect item parameter drift. Applied Psychological Measurement, 22, 33-51.
10.
Drasgow, F., Levine, M.V., Tsien, S., Williams, B.A., & Mead, A.D. (1995). Fitting polytomous item response theory models to multiple-choice tests. Applied Psychological Measurement , 19, 143-165.
11.
Ellis, B.B., Minsel, B., & Becker, P. (1989). Evaluation of attitude survey translations: An investigation using item response theory. International Journal of Psychology, 24, 665-684.
12.
Embretson, S.E., & Reise, S.P. (2000). Item response theory for psychologists. Mahwah, NJ: Lawrence Erlbaum.
13.
Facteau, J.D., & Craig, S.B. (2001). Are performance appraisal ratings from different rating sources comparable? Journal of Applied Psychology, 86, 215-227.
14.
Flowers, C.P., Oshima, T.C., & Raju, N.S. (1999). A description and demonstration of the polytomous-DFIT framework. Applied Psychological Measurement, 23, 309-326.
15.
Gaskell, G.D., O'Muircheartaigh, C.A., & Wright, D.B. (1994). Survey questions about the frequency of vaguely defined events: The effects of response alternatives. Public Opinion Quarterly, 58, 241-254.
16.
Golembiewski, R.T., Billingsley, K., & Yeager, S. (1975). Measuring change and persistence in human affairs: Types of change generated by OD designs. Journal of Applied Behavioral Science, 12, 133-157.
17.
Groves, R.M. (1991). Measurement error across the disciplines. In P. B. Biemer, R. M. Groves, L. E. Lyberg, N. A. Mathiowetz, & S. Sudman (Eds.), Measurement error in surveys (pp. 1-25). New York: John Wiley & Sons.
18.
Hambleton, R.K., Swaminathan, H., & Rogers, H.J. (1991). Fundamentals of item response theory. Newbury Park, CA: Sage.
19.
Higgs, A.C., & Ashworth, S.D. (1996). Organizational surveys: Tools for assessment and research. In A. I. Kraut (Ed.), Organizational surveys: Tools for assessment and change (pp. 19-40). San Francisco: Jossey-Bass.
20.
Johnson, R.H. (1996). Life in the consortium: The mayflower group. In A. I. Kraut (Ed.), Organizational surveys: Tools for assessment and change (pp. 285-309). San Francisco : Jossey-Bass.
21.
Kraut, A.I. (1996). Introduction: An overview of organizational surveys . In A. I. Kraut (Ed.), Organizational surveys: Tools for assessment and research (pp. 1-14). San Francisco: Jossey-Bass.
22.
Kraut, A.I., & Saari, L.M. (1999). Organizational surveys: Coming of age for a new era. In A. I. Kraut & A. K. Korman (Eds.), Evolving practices in human resource management: Responses to a changing world of work (pp. 302-327). San Francisco: Jossey-Bass.
23.
Lautenschlager, G.J., Flaherty, V.L., & Park, D.G. (1994). IRT differential item functioning: An examination of ability scale purifications. Educational and Psychological Measurement, 54, 21-31.
McDonald, R.P. (1999). Test theory: A unified treatment. Mahwah, NJ: Lawrence Erlbaum.
26.
Meade, A.W., & Lautenschlager, G.J. (2004). A comparison of item response theory and confirmatory factor analytic methodologies for establishing measurement equivalence/invariance. Organizational Research Methods, 7, 361-388.
27.
Meade, A.W., Lautenschlager, G.J., & Hecht, J.E. (2005). Establishing measurement equivalence and invariance in longitudinal data with item response theory. International Journal of Testing, 5, 279-300.
28.
Meade, A.W., Lautenschlager, G.J., & Johnson, E.C. (2007). A Monte Carlo examination of the sensitivity of the differential functioning of items and tests framework for tests of measurement invariance with Likert data. Applied Psychological Measurement , 31, 430-455.
29.
Park, D.G., & Lautenschlager, G.J. (1990). Improving IRT item bias detection with iterative linking and ability scale purification. Applied Psychological Measurement, 14, 163-173.
30.
Raju, N. (1999). DFITPS6. A Fortran-based computer program for calculating DIF/DTF [computer software]. Chicago: Illinois Institute of Technology.
31.
Raju, N.S., Laffitte, L.J., & Byrne, B.M. (2002). Measurement equivalence: A comparison of methods based on confirmatory factor analysis and item response theory. Journal of Applied Psychology, 87, 517-529.
32.
Raju, N.S., van der Linden, W. J., & Fleer , P.F. (1995). IRT-based internal measure of differential functioning of items and tests . Applied Psychological Measurement, 19, 353-368.
33.
Rogelberg, S.G., Church, A.H., Waclawski, J., & Stanton, J.M. (2002). Organizational survey research. In S. G. Rogelberg (Ed.), Handbook of research methods in industrial and organizational psychology (pp. 141-160). Malden, MA: Blackwell.
34.
Ryan, A.M., Horvath, M., Ployhart, R.E., Schmitt, N., & Slade, L.A. (2000). Hypothesizing differential item functioning in global employee opinion surveys. Personnel Psychology, 53(3), 531-562.
35.
Sackett, P.R., & Larson, J.R., Jr. (1990). Research strategies and tactics in industrial and organizational psychology. In M. D. Dunnette & L. M. Hough (Eds.), Handbook of industrial and organizational psychology: Vol. 1 (2nd ed., pp. 419-489). Palo Alto, CA: Consulting Psychologists Press.
36.
Samejima, F. (1969). Estimation of latent ability using a response pattern of graded scores (Psychometric Monographs No. 18). Iowa City, IA: Psychometric Society.
37.
Schuman, H., Kalton, G., & Ludwig, J. (1983). Context and contiguity in survey questionnaires . Public Opinion Quarterly, 47, 112-115.
38.
Schuman, H., & Presser, S. (1981). Questions and answers in attitude surveys: Experiments on question form, wording, and context. New York: Academic Press.
39.
Schwarz, N. (1995). Social cognition: Information accessibility and use in social judgment. In E. E. Smith & D. N. Oshenson (Eds.), Thinking: An invitation to cognitive science (2nd ed., pp. 345-376). Cambridge, MA: MIT Press.
40.
Schwarz, N. (1999). Self-reports-How the questions shape the answers . American Psychologist, 54(2), 93-105.
41.
Schwarz, N. (in press). Attitude construction: Evaluation in context. Social Cognition . Retrieved April 25, 2007, from http://sitemaker.umich.edu/norbert.schwarz/social_cognition
42.
Schwarz, N., & Bless, H. (1992). Assimilation and contrast effects in attitude measurement: An inclusion/ exclusion model. Advances in Consumer Research, 19, 72-77.
43.
Schwarz, N., Hippler, H.J., Deutsch, B., & Strack, F. (1985). Response scales-Effects of category range on reported behavior and comparative judgments. Public Opinion Quarterly , 49, 388-395.
44.
Schwarz, N., Knauper, B.R., Hippler, H.J., Noelle-Neumann, E., & Clark, L. (1991). Rating scales: Numeric values may change the meaning of scale labels. Public Opinion Quarterly, 55, 570.
45.
Schwarz, N., Strack, F., Muller, G., & Chassein, B. (1988). The range of response alternatives may determine the meaning of the question-Further evidence on informative functions of response alternatives. Social Cognition, 6, 107-117.
46.
Sirken, M.G., & Schechter, S. (1999). Interdisciplinary survey methods research. In M. G. Sirken, D. J. Herrmann, S. Schechter, N. Schwarz, J. M. Tanur , & R. Tourangeau (Eds.), Cognition and survey research (pp. 1-10). New York: John Wiley & Sons.
47.
Smith, T.W. (1991). Context effects in the general social survey. In P. B. Biemer, R. M. Groves, L. E. Lyberg, N. A. Mathiowetz, & S. Sudman (Eds.), Measurement error in surveys (pp. 57-71). New York: John Wiley & Sons.
48.
Stanton, J.M., Sinar, E.F., Balzer, W.K., & Smith, P.C. (2002). Issues and strategies for reducing the length of self-report scales. Personnel Psychology, 55, 167-194.
49.
Stark, S. (2001). MODFIT: A computer program for model-data fit . Urbana-Champaign: University of Illinois.
50.
Stark, S., Chernyshenko, O.S., & Drasgow, F. (2006). Detecting differential item functioning with confirmatory factor analysis and item response theory: Toward a unified strategy. Journal of Applied Psychology, 91(6), 1292-1306.
51.
Steinberg, L. (1994). Context and serial-order effects in personality measurement-Limits on the generality of measuring changes the measure. Journal of Personality and Social Psychology, 66(2), 341-349.
52.
Steinberg, L. (2001). The consequences of pairing questions: Context effects in personality measurement. Journal of Personality and Social Psychology, 81(2), 332-342.
53.
Stocking, M.L., & Lord, F.M. (1983). Developing a common metric in item response theory . Applied Psychological Measurement, 7, 201-210.
54.
Thissen, D. (1991). MULTILOG users guide: Multiple categorical item analysis and test scoring using item response theory [computer software and manual]. Chicago: Scientific Software International.
55.
Thissen, D. (2001). Software for the computation of the statistics involved in item response theory likelihood-ratio tests for differential item functioning. Chapel Hill: University of North Carolina at Chapel Hill.
56.
Tourangeau, R. (1999). Context effects on answers to attitude questions . In M. G. Sirken, D. J. Herrmann, S. Schechter , N. Schwarz, J. M Tanur, & R. Tourangeau (Eds.), Cognition and survey research (pp. 111-122). New York: John Wiley and Sons.
57.
Tourangeau, R., & Rasinski, K.A. (1988). Cognitive processes underlying context effects in attitude measurement. Psychological Bulletin , 103, 299-314.
58.
Tourangeau, R., Rasinski, K., Bradburn, N., & D'Andrade, R. (1989). Carryover effects in attitude surveys. Public Opinion Quarterly, 53, 495-524.
59.
Tourangeau, R., Rips, L.J., & Rasinski, K. (2000). The psychology of survey response. New York: Cambridge University Press.
60.
Tourangeau, R., Singer, E., & Presser, S. (2003). Context effects in attitude surveys: Effects on remote items and impact on predictive validity. Sociological Methods & Research, 31, 486-513.
61.
Vandenberg, R.J., & Lance, C.E. (2000). A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods, 3, 4-69.
62.
Wells, C.S., Subkoviak, M.J., & Serlin, R.C. (2002). The effect of item parameter drift on examinee ability estimates. Applied Psychological Measurement, 26, 77-87.
63.
Wilson, T.D., & Hodges, S.D. (1992). Attitudes as temporary constructions. In L. L. Martin & A. Tesser (Eds.), The construction of social judgments (pp. 37-65). Hillsdale, NJ: Lawrence Erlbaum.
