The criterion problem is a serious conceptual and methodological challenge that has confronted investigators and managers in the organizational sciences for over 100 years. This paper is an introduction to the criterion problem and some of the complex, multifaceted issues it encompasses. It defines and discusses important concepts (e.g. the problem itself, conceptual criteria, operational criteria). It explores broad dilemmas that afflict all types of criteria and narrower dilemmas that are more individually relevant to criteria traditionally considered objective or subjective. It concludes by examining prospects for the criterion problem both in general and within cross-cultural management.
AbugreJBDebrahYA (2019) Assessing the impact of cross-cultural communication competence on expatriate business operations in multinational corporations of a Sub-Saharan African context. International Journal of Cross Cultural Management19(1): 85–104.
2.
AckermanPL (2017) Adult intelligence: The construct and the criterion problem. Perspectives on Psychological Science12(6): 987–998.
3.
AlaybekBDalalRS (2021) Within-person variability in job performance. In: RauthmannJF (ed), The handbook of personality dynamics and processes. Amsterdam, Netherlands: Elsevier, pp. 1087–1118.
4.
AllportGW (1958) What units shall we employ? In: LindzeyG (ed), Assessment of human motives. Rinehart, pp. 239–260.
5.
American Association for the Advancement of Science (2005) So much more to know. Science (New York, N.Y.)309(5731): 78–102.
6.
American Psychological Association (1954) Technical recommendations for psychological tests and diagnostic techniques. Psychological Bulletin51(2): 1–38.
7.
AnastasiA (1986) Evolving concepts of test validation. Annual Review of Psychology37(1): 1–16.
8.
AnsbacherHL (1944) German industrial psychology in the fifth year of war. Psychological Bulletin41(9): 605–614.
9.
AstinAW (1964) Criterion-centered research. Educational and Psychological Measurement24(4): 807–822.
10.
AustinJTCrespinTR (2006) Problems of criteria in industrial and organizational psychology: Progress problems and prospects. In: BennettWLanceCEWoehrDJ (eds), Performance measurement: Current perspectives and future challenges. Lawrence Erlbaum, pp. 9–48.
11.
AustinJTVillanovaP (1992) The criterion problem: 1917-1992. Journal of Applied Psychology77(6): 836–874.
12.
AustinJTVillanovaPKaneJS, et al. (1991) Construct validation of performance measures: Issues, development, and evaluation of indicators. In: FerrisGRRowlandKM (eds), Research in personnel and human resources management. JAI Press, vol. 9, pp. 159–233.
13.
BassBM (1952) Ultimate criteria of organizational worth. Personnel Psychology5: 157–173.
14.
BayraktarS (2019) A diary study of expatriate adjustment: Collaborative mechanisms of social support. International Journal of Cross Cultural Management19(1): 47–70.
15.
BechtoldtH (1947) Problems in establishing criterion measures. In: StuitD. B. (ed), Personnel research and test development in the Bureau of Naval Personne. NJ, USA: Princeton University Press, pp. 357–379.
16.
BechtoldtHP (1951). Selection. In StevensSS (Ed.), Handbook of experimental psychology (pp. 1237–1267). NJ, USA: Wiley.
17.
BechtoldtHP (1959) Construct validity: A critique. American Psychologist14(10): 619–629.
18.
BellowsRM (1941) Procedures for evaluating vocational criteria. Journal of Applied Psychology25(5): 499–513.
19.
BensonPG (1978) Measuring cross-cultural adjustment: The problem of criteria. International Journal of Intercultural Relations2(1): 21–37.
20.
BinghamWV (1926) Measures of occupational success. Harvard Business Review5: 1–10.
21.
BinningJFBarrettGV (1989) Validity of personnel decisions: A conceptual analysis of the inferential and evidential bases. Journal of Applied Psychology74(3): 478–494.
22.
BlumMLNaylorJC (1968) Industrial Psychology: Its Theoretical and Social Foundations. NY, USA: Harper & Row.
23.
BohmD (2002) Wholeness and the Implicate Order. England, UK: Routledge.
24.
BommerWHJohnsonJLRichGA, et al. (1995) On the Interchangeability of Objective and Subjective Measures of Employee Performance: a Meta-analysis. Personnel Psychology48(3): 587–605.
25.
BormanWC (1991) Job behavior, performance, and effectiveness. In: DunnetteMDHoughLM (eds), Handbook of industrial & organizational psychology. Consulting Psychologists Press, 2, pp. 271–326.
26.
BormanWCBryantRHDorioJ (2010) The measurement of task performance as criteria in selection research. In: FarrJLTippinsNT (eds), Handbook of employee selection. Taylor & Francis, pp. 439–461.
27.
BormanWCBuckDEHansonMA, et al. (2001) An examination of the comparative reliability, validity, and accuracy of performance ratings made using computerized adaptive rating scales. Journal of Applied Psychology86(5): 965–973.
28.
BormanWCMotowidloSJ (1997) Task performance and contextual performance: The meaning for personnel selection research. Human Performance10(2): 99–109.
29.
BormanWCSmithTN (2012) The use of objective measures as criteria in I/O psychology. In: SchmittN (ed), The Oxford handbook of personnel assessment and selection. Oxford University Press, pp. 532–542.
30.
BradburnNMCartwrightNFullerJ (2017) A theory of measurement. In: ClimansMc (ed), Measurement in Medicine: Philosophical Essays on Assessment and Evaluation. Rowman & Littlefield International, pp. 73–88.
BrogdenHETaylorEK (1950a) The theory and classification of criterion bias. Educational and Psychological Measurement10(2): 159–183.
33.
BrogdenHETaylorEK (1950b) The dollar criterion?applying the cost accounting concept to criterion construction. Personnel Psychology3: 133–154.
34.
CampbellJP (1990) Modeling the performance prediction problem in industrial and organizational psychology. In: DunnetteMHoughLM (eds), Handbook of industrial and organizational psychology. 2nd ed.Consulting Psychologists Press, 1, pp. 687–731.
35.
CampbellJP (1994) Alternative models of job performance and the implications for selection and classification. In: RumseyMGWalkerCBHarrisJH (eds), Personnel selection and classification. Erlbaum, pp. 33–51.
36.
CampbellJP (2012) Behavior, performance, and effectiveness in the 21st century Kozlowski. In: JSW (ed), The Oxford handbook of organizational psychology. Oxford, UK: Oxford University Press, pp. 159–195.
37.
CampbellJPDunnetteMDLawlerEE, et al. (1970) Managerial Behavior, Performance, and Effectiveness. NY, USA: McGraw-Hill.
38.
CampbellJPKnappDK (eds), (2001) Exploring the Limits in Personnel Selection and Classification. Mahwah, NJ, USA: Lawrence Erlbaum Associates.
39.
CampbellJPMcCloyRAOpplerSH, et al. (1993) A theory of performance. In: SchmittNBormanWC (eds), Personnel selection in organizations. San Francisco, CA: Jossey-Bass, pp. 35–70.
40.
CampbellJPWiernikBM (2015) The modeling and assessment of work performance. Annual Review of Organizational Psychology and Organizational Behavior2(1): 47–74.
41.
CascioWFAguinisH (2004) Applied Psychology in Human Resource Management. 6th ed.Pearson College Division.
CozzensP (2017) The Earth Is Weeping: The Epic Story of the Indian Wars for the American West. Vintage.
44.
CritesJO (1969) Vocational Psychology: The Study of Vocational Behavior and Development. McGraw-Hill Company.
45.
CronbachLJ (1971) Test validation. In: ThorndikeRL (ed), Educational measurement. 2nd ed.American Council on Education, pp. 443–507.
46.
CronbachLJ (1989) Construct validity after thirty years. In: LinnRL (ed), Intelligence: Measurement, theory, and public policy. University of Illinois Press, pp. 147–171.
47.
CronbachLJ (1990) Essentials of Psychological Testing. 5th ed.HarperCollins.
48.
CronbachLJMeehlPE (1955) Construct validity in psychological tests. Psychological Bulletin52(4): 281–302.
49.
CuretonEE (1951) Validity. In: LindquistEF (ed), Educational measurement. American Council on Education, p. 621.
50.
DawesTMuru-LanningMLapsleyH, et al. (2021) Hongi, Harirū and Hau: Kaumātua in the time of COVID-19. Journal of the Royal Society of New Zealand51(sup1): S23–S36.
51.
DeNisiAS (2008) Rain, snow, and sleet are just different types of precipitation. Academy of Management Perspectives22(1): 15–17.
52.
DorseyDWCortinaJMLuchmanJ (2010) Adaptive and citizenship-related behaviors at work. In: FarrJLTippinsNT (eds), Handbook of employee selection. Taylor & Francis, pp. 463–487.
53.
DunnetteMD (1963) A note on the criterion. Journal of Applied Psychology47(4): 251–254.
54.
DunnetteMD (1992) It was nice to be there: Construct validity then and now. Human Performance5(1–2): 157–169.
55.
ElyRJThomasDA (2020) Getting serious about diversity. Harvard Business Review98(6): 114–122.
FarhJ-LZhongC-BOrganDW (2004) Organizational citizenship behavior in the People's Republic of China. Organization Science15(2): 241–253.
58.
FeiglH (1945) Operationism and scientific method. Psychological Review52(5): 250–259.
59.
FishEH (1917) Human engineering. Journal of Applied Psychology1(2): 161–174.
60.
FiskeAP (2020) The lexical fallacy in emotion research: Mistaking vernacular words for psychological entities. Psychological Review127(1): 95–113.
61.
FiskeDW (1951) Values, theory, and the criterion problem. Personnel Psychology4: 93–98.
62.
FiskeDW (ed), (1981) Problems with language imprecision. Jossey-Bass.
63.
FlanaganJC (ed) (1948) The Aviation Psychology Program of the Army Air Forces Report. U.S. Government Printing Office.
64.
FlanaganJC (1956) The evaluation of methods in applied psychology and the problem ofcriteria. Occupational Psychology30: 1–9.
65.
FreydM (1923–1924) Measurement in vocational selection: An outline of research procedure. Journal of Personnel Research2: 215377–249385. 268-284.
66.
FriedmanL (1990) Degree of redundancy between time, importance, and frequency task ratings. Journal of Applied Psychology75(6): 748–752.
67.
GelfandMJAycanZErezM, et al. (2017) Cross-cultural industrial organizational psychology and organizational behavior: A hundred-year journey. Journal of Applied Psychology102(3): 514–529.
68.
GelfandMJLeslieLMFehrR (2008) To prosper, organizational psychology should… adopt a global perspective. Journal of Organizational Behavior29(4): 493–517.
69.
GhiselliEE (1956) Dimensional problems of criteria. Journal of Applied Psychology40(1): 1–4.
70.
GhiselliEE (1959) 2. The Generalization of Validity. Personnel Psychology12(3): 397–402.
71.
GhiselliEEBrownCW (1955) Personnel and Industrial Psychology. 2nd ed.McGraw-Hill.
72.
GomezCTaylorKA (2018) Cultural differences in conflict resolution strategies. International Journal of Cross Cultural Management18(1): 33–51.
73.
GongYChangS (2007) The relationships of cross-cultural adjustment with dispositional learning orientation and goal setting: A longitudinal analysis. Journal of Cross-Cultural Psychology38(1): 19–25.
74.
GottfredsonLS (1991) The evaluation of alternative measures of job performance. In: Commission on Behavioral and Social Sciences and Education (ed), Performance assessment for the workplace. National Academy Press, 2, pp. 75–126.
75.
GuionRM (1961) Criterion Measurement and Personnel Judgments. Personnel Psychology14: 141–149.
76.
GuionRM (1965a) Industrial psychology as an academic discipline. American Psychologist20(10): 815–821.
77.
GuionRM (1965b) Personnel Testing. McGraw-Hill.
78.
GuionRM (1976) Recruiting, selection, and job placement. In: DunnetteMD (ed), Handbook of industrial and organizational psychology. Rand McNally, pp. 777–828.
79.
GuionRM (1991). Personnel assessment, selection, and placement. In: DunnetteMDHoughLM (eds), Handbook of industrial and organizational psychology. 2nd ed.Consulting Psychologists' Press, 2, pp. 328–397.
80.
GuionRM (1998) Assessment, Measurement, and Prediction for Personnel Decisions. Erlbaum.
HackingI (1999) The Social Construction of what?. Harvard University Press.
83.
HajroAPudelkoM (2010) An analysis of core-competences of successful multinational team leaders. International Journal of Cross Cultural Management10(2): 175–194.
84.
HampsonSEJohnOPGoldbergLR (1986) Category breadth and hierarchical structure in personality: studies of asymmetries in judgments of trait implications. Journal of Personality and Social Psychology51(1): 37–54.
85.
HarrisDRutledgeS (2010) Models and predictors of teacher effectiveness: A comparison of research about teaching and other occupations. Teachers College Record112(3): 914–960.
86.
HelmreichRLSawinLLCarsrudAL (1986) The honeymoon effect in job performance: Temporal increases in the predictive power of achievement motivation. Journal of Applied Psychology71(2): 185–188.
87.
HenemanRL (1986) The relationship between supervisory ratings and results-oriented measures of performance: A Meta-analysis. Personnel Psychology39(4): 811–826.
88.
HullCL (1928) Aptitude Testing. World Book Company.
89.
JacobsonKJHoodJNVan BurenHJIII (2014) Workplace bullying across cultures. International Journal of Cross Cultural Management14(1): 47–65.
90.
JamesLR (1973) Criterion models and construct validity for criteria. Psychological Bulletin80(1): 75–83.
91.
JenkinsJG (1946) Validity for whatJournal of Consulting Psychology10(2): 93–98.
92.
JungstMJanssensB (2020) Task conflict and extra-role performance: A cross-national perspective between east and west. International Journal of Cross Cultural Management20(1): 71–87.
93.
KaneJS (1996) The conceptualization and representation of total performance effectiveness. Human Resource Management Review6(2): 123–145.
94.
KellHJLubinskiD (2015) Intellectual abilities for counseling interventions, practice, and theory: Dismissing their significance for learning and work constitutes. In: HartungPJSavickasMLWalshWB (eds), APA handbook of career intervention, Volume 1: Foundations. Foundations: American Psychological Association, Vol. 1, pp. 303–326.
95.
KellHJMartin‐RaughMPCarneyLM, et al. (2017) Exploring methods for developing behaviorally anchored rating scales for evaluating structured interview performance. ETS Research Report Series1: 1–26.
96.
KentJDMcCarthyMT (2016) Holistic Review in Graduate Admissions: A Report from the Council of Graduate Schools. Council of Graduate Schools.
97.
KleinmuntzB (1967) Personality Measurement: An Introduction. Dorsey Press.
98.
KomakiJBloodMRHolderD (1980) Fostering friendliness in a fast food franchise. Journal of Organizational Behavior Management2(3): 151–164.
99.
KuhnT (2000) The Road since Structure. University of Chicago Press.
100.
LangJWBKellHJ (2020) General mental ability and specific abilities: Their relative importance for extrinsic career success. Journal of Applied Psychology105(9): 1047–1061.
LofquistEAMatthiesenSB (2018) Viking leadership: How Norwegian transformational leadership style effects creativity and change through organizational citizenship behavior (OCB). International Journal of Cross Cultural Management18(3): 309–325.
103.
McCormickEJIlgenD (1985) Industrial and Organizational Psychology. 8th ed.Prentice-Hall.
104.
MessickS (1980) Test validity and the ethics of assessment. American Psychologist35(11): 1012–1027.
105.
MinbashianAEarlJBrightJEH (2013) Openness to experience as a predictor of job performance trajectories. Applied Psychology62(1): 1–12.
106.
MontagTMaertzCPJrBaerM (2012) A critical analysis of the workplace creativity criterion space. Journal of Management38(4): 1362–1386.
107.
MotowidloSJKellHJ (2013). In: SchmittNWHighhouseS (eds), Job performanceHandbook of industrial and organizational psychology. 2nd ed.Wiley, pp. 82–103.
108.
MotowidloSJSchmitMJ (1999). Performance assessment in unique jobs. In IlgenDPulakosE (Eds.), The changing nature of performance: Implications for staffing, motivation, and development (pp. 56–86). Jossey-Bass.
109.
MurphyKR (1989a) Dimensions of job performance. In: DillonR.PellingrinoJ. (eds), Testing: Applied and theoretical perspectives. Praeger, pp. 218–247.
110.
MurphyKR (1989b) Is the relationship between cognitive ability and job performance stable over time?Human Performance2(3): 183–200.
111.
MurphyKR (2009) 9 Validity, Validation and Values. Academy of Management Annals3(1): 421–461.
112.
NathanBRAlexanderRA (1988) A Comparison of Criteria for Test Validation: a Meta-analytic Investigation. Personnel Psychology41(3): 517–535.
113.
NewtonPEShawSE (2014) Validity in Educational and Psychological Assessment. Sage Publications Ltd.
114.
OswaldFLSchmittNKimBH, et al. (2004) Developing a biodata measure and situational judgment inventory as predictors of college student performance. Journal of Applied Psychology89(2): 187–207.
115.
PreacherKJZhangGKimC, et al. (2013) Choosing the optimal number of factors in exploratory factor analysis: A model selection perspective. Multivariate Behavioral Research48(1): 28–56.
116.
PulakosEDO'LearyRS (2010). Defining and measuring results of workplace behavior. In: FarrJLTippinsNT (eds), Handbook of employee selection. Routledge, pp. 513–528.
117.
RavetzJR (1977) Science as a never-ending quest. Nature268(18): 671.
118.
RomaniLBarmeyerCPrimeczH, et al. (2018) Cross-Cultural Management Studies: State of the Field in the Four Research Paradigms*. International Studies of Management & Organization48(3): 247–263.
119.
RonanWWPrienE (1966) Toward a Criterion Theory: A Review and Analysis of Research and Opinion. Smith Richardson Foundation.
120.
RotundoM (2002) Defining and Measuring Individual Level Job Performance: A Review and Integration. DOI: 10.1017/CBO9781107415324.004.
121.
RotundoMSackettPR (2002) The relative importance of task, citizenship, and counterproductive performance to global ratings of job performance: A policy-capturing approach. Journal of Applied Psychology87(1): 66–80.
122.
RuncoMA (2009) Simplifying theories of creativity and revisiting the criterion problem: A comment on Simonton's (2009) hierarchical model of domain-specific disposition, development, and achievement. Perspectives on Psychological Science4(5): 462–465.
123.
RyansDG (1949) The criteria of teaching effectiveness. Journal of Environmental Radioactivity42(9): 690–699.
124.
SampsonEE (1978) Scientific paradigms and social values: Wanted-a scientific revolution. Journal of Personality and Social Psychology36(11): 1332–1343.
125.
SchmidtFLKaplanLB (1971) Composite vs. multiple criteria: A review and resolution of the controversy. Personnel Psychology24(3): 419–434.
126.
SchneiderWH (1992) After Binet: French intelligence testing, 1900-1950. Journal of the History of the Behavioral Sciences28(2): 111–132.
127.
SeashoreRH (1939) Work methods: an often neglected factor underlying individual differences. Psychological Review46(2): 123–141.
128.
SeashoreRH (1951) Work and motor performance. In: StevensSS (ed), Handbook of experimental psychology. Wiley, pp. 1341–1362.
129.
ShawJBEricksonANasirzadehF (2015) Destructive leader behavior. International Journal of Cross Cultural Management15(3): 329–345.
130.
ShayJPBaackS (2006) An empirical investigation of the relationships between modes and degree of expatriate adjustment and multiple measures of performance. International Journal of Cross Cultural Management6(3): 275–294.
131.
SlifeBDWrightCDYancharSC (2016) Using operational definitions in research: A best-practices approach. The Journal of Mind and Behavior37(2): 119–139.
132.
SmithPC (1976). Behaviors, results, and organizational effectiveness: The problem of criteria. In: DunnetteMD (ed), Handbook of industrial and organizational psychology. Rand McNally, pp. 745–775.
133.
SpeerAB (2018) Quantifying with words: An investigation of the validity of narrative-derived performance scores. Personnel Psychology71(3): 299–333.
134.
StagnerR (1961) Psychology of Personality. 3rd ed.McGraw-Hill Book Company, Inc.
135.
TarasVKirkmanBLSteelP (2010) Examining the impact of Culture's consequences: A three-decade, multilevel, meta-analytic review of Hofstede's cultural value dimensions. Journal of Applied Psychology95(3): 405–439.
136.
ThorndikeRL (1949) Personnel Selection: Test and Measurement Techniques. John Wiley.
137.
ThurstoneLL (1955) The criterion problem in personality research. Educational and Psychological Measurement15(4): 353–361.
ToopsHA (1944) The criterion. Educational and Psychological Measurement4(1): 271–295.
140.
VallacherRRWegnerDM (1989) Levels of personal agency: Individual variation in action identification. Journal of Personality and Social Psychology57(4): 660–671.
141.
VallacherRRWegnerDM (1987) What do people think they're doing? Action identification and human behavior. Psychological Review94(1): 3–15.
142.
VitelesMS (1932) Industrial Psychology. Norton.
143.
ViswesvaranCOnesDS (2000) Perspectives on models of job performance. International Journal of Selection and Assessment8(4): 216–226.
144.
WallaceSR (1965) Criteria for whatAmerican Psychologist20(6): 411–417.
145.
WallaceSRWeitzJ (1955) Industrial psychology. Annual Review of Psychology6(1): 217–250.
146.
WernimontPFCampbellJP (1968) Signs, samples, and criteria. Journal of Applied Psychology52(5): 372–376.
WigginsJ (1973) Personality and Prediction: Principles of Personality Assessment. Addison-Wesley.
149.
WoehrDJRochS (2012) Supervisory performance ratings. In: SchmittN. (Ed.), The Oxford handbook of personnel assessment and selection. Oxford University Press, pp. 517–531.
