A Test-Retest Reliability Generalization Meta-Analysis of Judgments Via the Policy-Capturing Technique

Abstract

Policy capturing is a widely used technique, but the temporal stability of policy-capturing judgments has long been a cause for concern. This article emphasizes the importance of reporting reliability, and in particular test-retest reliability, estimates in policy-capturing studies. We found that only 164 of 955 policy-capturing studies (i.e., 17.17%) reported a test-retest reliability estimate. We then conducted a reliability generalization meta-analysis on policy-capturing studies that did report test-retest reliability estimates—and we obtained an average reliability estimate of .78. We additionally examined 16 potential methodological and substantive antecedents to test-retest reliability (equivalent to moderators in validity generalization studies). We found that test-retest reliability was robust to variation in 14 of the 16 factors examined but that reliability was higher in paper-and-pencil studies than in web-based studies and was higher for behavioral intention judgments than for other (e.g., attitudinal and perceptual) judgments. We provide an agenda for future research. Finally, we provide several best-practice recommendations for researchers (and journal reviewers) with regard to (a) reporting test-retest reliability, (b) designing policy-capturing studies for appropriate reportage, and (c) properly interpreting test-retest reliability in policy-capturing studies.

Keywords

policy capturing test-retest reliability reliability generalization judgment analysis meta-analysis

Get full access to this article

View all access options for this article.

References

Aguinis

Bradley

K. J.

(2014). Best practice recommendations for designing and implementing experimental vignette methodology studies. Organizational Research Methods, 17, 351–371.

*Aiman-Smith

Bauer

T. N.

Cable

D. M.

(2001). Are you attracted? Do you intend to pursue? A recruiting policy-capturing study. Journal of Business and Psychology, 16, 219–237.

Aiman-Smith

Scullen

Barr

(2002). Conducting studies of decision making in organizational contexts: A tutorial for policy-capturing and other regression-based technologies. Organizational Research Methods, 5, 388–414.

*Albers-Miller

N. D.

(1999). Consumer misbehavior: Why people buy illicit goods. Journal of Consumer Marketing, 16, 273–287.

*Andersen

Larsen

J. K.

Schultz

Nielsen

B. M.

Kørner

Behnke

Munk-Andersen

Butler

Allerup

Bech

(1989). The brief psychiatric rating scale. Psychopathology, 22, 168–176.

Appelbaum

Cooper

Kline

R. B.

Mayo-Wilson

Nezu

A. M.

Rao

S. M.

(2018). Journal article reporting standards for quantitative research in psychology: The APA Publications and Communications Board task force report. American Psychologist, 73, 3–25.

*Athanasou

Cooksey

(2001). Judgment of factors influencing interest: An Australian study. Journal of Vocational Education Research, 26, 77–96.

Ashton

R. H.

(2000). A review and analysis of research on the test–retest reliability of professional judgment. Journal of Behavioral Decision Making, 13, 277–294.

Baer

R. A.

Ballenger

Berry

D. T. R.

Wetter

M. W.

(1997). Detection of random responding on the MMPI-A. Journal of Personality Assessment, 68, 139–151.

10.

*Baker

Thompson

(2012). Initiating artificial nutrition support: A clinical judgement analysis. Journal of Human Nutrition and Dietetics, 25, 427–434.

11.

*Barley

E. A.

Jones

P. W.

(2005). Analysis of the cues used by patients when making assessments of their asthma severity. European Respiratory Journal, 25, 671–676.

12.

Barr

Hitt

(1986). A comparison of selection decision models in manager versus student samples. Personnel Psychology, 39, 599–617.

13.

Baumeister

R. F.

Heatherton

T. F.

(1996). Self-regulation failure: An overview. Psychological Inquiry, 7, 1–15.

14.

*Bech

Haaber

Joyce

C. R. B.

(1986). Experiments on clinical observation and judgement in the assessment of depression: Profiled videotapes and judgement analysis. Psychological Medicine, 16, 873–883.

15.

*Beckstead

J. W.

Pezzo

M. V.

Beckie

T. M.

Shahraki

Kentner

A. C.

Grace

S. L.

(2014). Physicians’ tacit and stated policies for determining patient benefit and referral to cardiac rehabilitation. Medical Decision Making, 34, 63–74.

16.

*Beckstead

J. W.

Stamp

K. D.

(2007). Understanding how nurse practitioners estimate patients’ risk for coronary heart disease: A judgment analysis. Journal of Advanced Nursing, 60, 436–446.

17.

Berchtold

(2016). Test-retest: Agreement or reliability? Methodological Innovations, 9, 1–7.

18.

Bevan

Steger

J. A.

(1971). Free recall and abstractness of stimuli. Science, 172, 597–599.

19.

*Blume

B. D.

Baldwin

T. T.

Rubin

R. S.

(2009). Reactions to different types of forced distribution performance evaluation systems. Journal of Business and Psychology, 24, 77–91.

20.

*Bolton

K. R.

(2005). Organizational characteristics, goal orientation, and organizational attractiveness: A person-organization fit perspective [Publication No. 1427916] [Master’s thesis, University of Houston]. ProQuest Dissertations & Theses Global.

21.

*Bonaccio

(2006). Determining the relative importance of antecedents to advice utilization during decision-making with and without missing information [Publication No. 3251589] [Doctoral dissertation]. ProQuest Dissertations & Theses Global.

22.

*Bonaccio

Dalal

R. S.

(2010). Evaluating advisors: A policy-capturing study under conditions of complete and missing information. Journal of Behavioral Decision Making, 23, 227–249.

23.

*Bradshaw

G. L.

Shaw

(1992). Forecasting solar flares: Experts and artificial systems. Organizational Behavior and Human Decision Processes, 53, 135–157.

24.

*Braspenning

Sergeant

(1994). General practitioners’ decision making for mental health problems: Outcomes and ecological validity. Journal of Clinical Epidemiology, 47, 1365–1372.

25.

Brown

T. R.

(1972). A comparison of judgmental policy equations obtained from human judges under natural and contrived conditions. Mathematical Biosciences, 15, 205–230.

26.

*Brown

T. J.

Allgeier

E. R.

(1996). The impact of participant characteristics, perceived motives, and job behaviors on co-workers’ evaluations of workplace romances. Journal of Applied Social Psychology, 26, 577–595.

27.

*Browne

J. P.

O’Boyle

C. A.

McGee

H. M.

McDonald

N. J.

Joyce

C. B.

(1997). Development of a direct weighting procedure for quality of life domains. Quality of Life Research, 6, 301–309.

28.

*Brundin

Gustafsson

(2013). Entrepreneurs’ decision making under different levels of uncertainty: The role of emotions. International Journal of Entrepreneurial Behavior & Research, 19, 568–591.

29.

*Cable

D. M.

Graham

M. E.

(2000). The determinants of job seekers’ reputation perceptions. Journal of Organizational Behavior, 21, 929–947.

30.

*Cable

D. M.

Judge

T. A.

(1994). Pay preferences and job search decisions: A person-organization fit perspective. Personnel Psychology, 47, 317–348.

31.

*Carkenord

D. M.

Stephens

M. G.

(1994). Understanding student judgments of teaching effectiveness: A “policy capturing” approach. The Journal of Psychology, 128, 675–682.

32.

*Carson

K. P.

Cardy

R. L.

Dobbins

G. H.

(1991). Performance appraisal as effective management or deadly management disease two initial empirical investigations. Group & Organization Management, 16, 143–159.

33.

Carter

E. C.

Kofler

L. M.

Forster

D. E.

McCullough

M. E.

(2015). A series of metaanalytic tests of the depletion effect: Self-control does not seem to rely on a limited resource. Journal of Experimental Psychology: General, 144, 796–815.

34.

*Carvalho

M. A.

(2005). Accessing legislative voting behavior: A multimethod research approach [Publication No. 3196120] [Doctoral dissertation]. ProQuest Dissertations & Theses Global.

35.

Catano

V. M.

Brochu

Lamerson

C. D.

(2012). Assessing the reliability of situational judgment tests used in high-stakes situations. International Journal of Selection and Assessment, 20, 333–346.

36.

*Chen

W. Y.

Wang

M. L.

Hsu

B. F.

(2013). When PJ fit and PO fit meet guanxi in a Chinese selection context. Journal of Technology Management in China, 8, 174–189.

37.

*Cheyne

Dalgleish

Tucker

Kane

Shetty

Ashalatha.

McLeod

Niven

(2012). Risk assessment and decision making about in-labour transfer from rural maternity care: A social judgment and signal detection analysis. BMC Medical Informatics and Decision Making, 12, Article 122. https://doi.org/10.1186/1472-6947-12-122

38.

*Chhinzer

N. N.

(2007). Evaluating layoff techniques: A policy-capturing study of voluntary versus involuntary layoffs [Publication No. 978-0-494-28125-3] [Doctoral dissertation]. ProQuest Dissertations & Theses Global.

39.

*Christie

L. A.

(1996). Expertise in nurses’ clinical judgments: The role of cognitive variables and experience [Doctoral dissertation]. ProQuest Dissertations & Theses Global.

40.

Cooksey

R. W.

(1996). Judgment analysis: Theory, methods, and applications. Academic Press.

41.

Cortina

J. M.

(1993). What is coefficient alpha? An examination of theory and applications. Journal of Applied Psychology, 78, 98–104.

42.

*Crocker

R. K.

Banfield

(1986). Factors influencing teacher decisions on school, classroom, and curriculum. Journal of Research in Science Teaching, 23, 805–816.

43.

Cronbach

L. J.

(1988). Five perspectives on the validity argument. In Wainer

Braun

H. I.

(Eds.), Test validity (pp. 3–17). Erlbaum.

44.

*Cugueró-Escofet

Fortin

Canela

M. A.

(2014). Righting the wrong for third parties: How monetary compensation, procedure changes and apologies can restore justice for observers of injustice. Journal of Business Ethics, 122, 253–268.

45.

*Dahling

J. J.

Thompson

M. N.

(2010). Contextual supports and barriers to academic choices: A policy-capturing analysis. Journal of Vocational Behavior, 77, 374–382.

46.

Dalal

R.S.

Bhave

D.P.

Fiset

(2014). Within-person variability in job performance: A theoretical review and research agenda. Journal of Management, 40, 1396–1436.

47.

*Dalal

R. S.

Bonaccio

(2010). What types of advice do decision-makers prefer? Organizational Behavior and Human Decision Processes, 112, 11–23.

48.

*Davies

M. L.

(2011). Detecting and preventing financial abuse of older adults: Examining decision making by health, social care and banking professionals. [Publication No. U578889] [Doctoral dissertation]. Dissertations & Thesis Global.

49.

Davies

D. R.

Parasuraman

(1982). The psychology of vigilance. Academic Press.

50.

*de Cueto

J. F.

(2012). Relative importance of false positives in the selection process [Publication No. 3517008] [Doctoral dissertation]. ProQuest Dissertations & Theses Global.

51.

*De Goede

M. E.

Van Vianen

A. E.

Klehe

U. C.

(2013). A tailored policy-capturing study on po fit perceptions: The ascendancy of attractive over aversive fit. International Journal of Selection and Assessment, 21, 85–98.

52.

*Deshpande

S. P.

Joseph

(1994). Variation in compensation decisions by managers: An empirical investigation. The Journal of Psychology, 128, 41–50.

53.

*Deshpande

S. P.

Schoderbek

P. P.

(1992). Pay allocations by hospital administrators: An empirical analysis. Journal of Healthcare Management, 37, 321–332.

54.

*Deshpande

S. P.

Schoderbek

P. P.

(1993). Pay-allocations by managers: A policy-capturing approach. Human Relations, 46, 465–479.

55.

*DeTienne

D. R.

Shepherd

D. A.

De Castro

J. O.

(2008). The fallacy of “only the strong survive”: The effects of extrinsic motivation on the persistence decisions for under-performing firms. Journal of Business Venturing, 23, 528–546.

56.

*Di Stasio

(2014). Why education matters to employers: A vignette study in Italy, England and the Netherlands. [Unpublished doctoral dissertation]. Universiteit van Amsterdam.

57.

*Dineen

B. R.

Noe

R. A.

Wang

(2004). Perceived fairness of web-based applicant screening procedures: Weighing the rules of justice and the role of individual differences. Human Resource Management, 43, 127–145.

58.

*Dorsey

D. W.

(1997). Managerial merit pay allocation: An analysis of alternative judgment models using regression and fuzzy expert system techniques [Publication No. 9724003] [Doctoral dissertation]. ProQuest Dissertations & Theses Global.

59.

*Drenth

D. J.

(2002). Differences in work preferences as a function of gender and ethnicity: A policy capturing approach [Publication No. 3052641] [Doctoral dissertation]. ProQuest Dissertations & Theses Global.

60.

*Drover

A. W.

(2014). The influence of angel investor characteristics on venture capitalist decision making [Publication No. 3640752] [Doctoral dissertation]. ProQuest Dissertations & Thesis Global.

61.

*Dulebohn

Martocchio

J. J.

(1998). Employees’ perceptions of the distributive justice of pay raise decisions: A policy capturing approach. Journal of Business and Psychology, 13, 41–64.

62.

Ebert

R. J.

Kruse

T. E.

(1978). Bootstrapping the security analyst. Journal of Applied Psychology, 63, 110–119.

63.

Egly

Driver

Rafal

R. D.

(1994). Shifting visual attention between objects and locations: Evidence from normal and parietal lesion subjects. Journal of Experimental Psychology: General, 123, 161–177.

64.

Evans

S. C.

Amaro

C. M.

Herbert

Blossom

J. B.

Roberts

M. C.

(2018) “Are you gonna publish that?” Peer-reviewed publication outcomes of doctoral dissertations in psychology. PLOS ONE, 13(2), Article e0192219. https://doi.org/10.1371/journal.pone.0192219

65.

*Fehr

(2007). When apologies work: The benefits of matching apology content to victims and context [Publication No. 1450214] [Doctoral dissertation]. ProQuest Dissertations & Theses Global.

66.

Fishbein

Ajzen

(2009). Predicting and changing behavior: The reasoned action approach. Taylor & Francis.

67.

Fredrickson

B. L.

Kahneman

(1993). Duration neglect in retrospective evaluations of affective episodes. Journal of Personality and Social Psychology, 65, 45–55.

68.

Friedman

(1979). Framing pictures: The role of knowledge in automatized encoding and memory for gist. Journal of Experimental Psychology: General, 108, 316–355.

69.

Garb

H. N.

(1999). Call for a moratorium on the use of the Rorschach Inkblot Test in clinical and forensic settings. Assessment, 6, 313–317.

70.

*Gebing

T. A.

(2000). What is she thinking about? A policy capturing investigation of battered women’s decision to stay in violent relationships [Publication No. 9995031] [Doctoral dissertation]. ProQuest Dissertations & Theses Global.

71.

*Geller

S. E.

(1994). The influence of psychosocial factors on heart transplantation [Publication No. 9424974] [Doctoral dissertation]. ProQuest Dissertations & Theses Global.

72.

*German

(2011). Capturing the justice judgment: An application of the theory of representative design in two policy capturing studies in organizational justice [Unpublished doctoral dissertation]. Durham University.

73.

*German

Fortin

Read

(2016). Justice judgments: Individual self-insight and between-and within-person consistency. Academy of Management Discoveries, 2, 33–50.

74.

*Golay

L. M.

(2016). A judgment analysis of psychological contracts: Priorities of part-time and full-time employees in relation to fulfillment and obligation to stay [Unpublished doctoral dissertation]. University of Connecticut.

75.

*Gorman

C. D.

Clover

W. H.

Doherty

M. E.

(1978). Can we learn anything about interviewing real people from “interviews” of paper people? Two studies of the external validity of a paradigm. Organizational Behavior and Human Performance, 22, 165–192.

76.

Graham

M. E.

Cable

D. M.

(2001). Consideration of the incomplete block design for policy-capturing research. Organizational Research Methods, 4, 26–45.

77.

Greco

L. M.

O’Boyle

E. H.

Cockburn

B. S.

Yuan

(2018). Meta-analysis of coefficient alpha: A reliability generalization study. Journal of Management Studies, 55, 583–618.

78.

Hagger

M. S.

Chatzisarantis

N. L.

Alberts

Anggono

C. O.

Batailler

Birt

Zwienenberg

(2016). A multi-lab pre-registered replication of the ego depletion effect. Perspectives on Psychological Science, 11, 546–573.

79.

Hagger

M. S.

Wood

Stiff

Chatzisarantis

N. L. D.

(2010). Ego depletion and the strength model of self-control: A meta-analysis. Psychological Bulletin, 136, 495–525.

80.

*Hall

J. A.

(1979). On the adequacy of current authoritative guidelines for the review and evaluation of systems development controls [Publication No. 8013017] [Doctoral dissertation]. ProQuest Dissertations & Theses Global.

81.

*Hanisch

D. N.

Rau

S. B.

(2014). Application of metric conjoint analysis in family business research. Journal of Family Business Strategy, 5, 72–84.

82.

Hardiman

Dufresne

Mestre

(1989). The relation between problem categorization and problem solving among experts and novices. Memory & Cognition, 17, 627–638.

83.

*Harries

(1995). Judgement analysis of patient management: General practitioners’ policies and self-insight [Publication No. U072152] [Doctoral dissertation]. ProQuest Dissertations & Theses Global.

84.

*Harries

Evans

J. St. B. T.

Dennis

Dean

(1996). A clinical judgement analysis of prescribing decisions in general practice. Le Travail Humain, 59, 87–109.

85.

*Harries

P. A.

(2004). Occupational therapists’ judgement of referral priorities: Expertise and training. [Publication No. 10052525] [Doctoral dissertation]. ProQuest Dissertations & Thesis Global.

86.

*Harries

P. A.

Gilhooly

(2003). Identifying occupational therapists’ referral priorities in community health. Occupational Therapy International, 10, 150–164.

87.

*Harries

Gilhooly

(2011). Training novices to make expert, occupationally focused, community mental health referral decisions. British Journal of Occupational Therapy, 74, 58–65.

88.

Heerwegh

(2009). Mode differences between face-to-face and web surveys: An experimental investigation of data quality and social desirability effects. International Journal of Public Opinion Research, 21, 111–121.

89.

Hedges

Olkin

(1985). Statistical methods for meta-analysis. Academic Press.

90.

*Hergert

(2016). Personality, situation, and infidelity in romantic relationships. [Unpublished doctoral dissertation]. University of Hagen.

91.

*Heron

R. L.

(1997). Intuition and deliberation: A study of women’s sexual choices [Publication No. 9840305] [Doctoral dissertation]. ProQuest Dissertations & Theses Global.

92.

Hoffman

B. J.

Kennedy

C. L.

LoPilato

A. C.

Monahan

E. L.

Lance

C. E.

(2015). A review of the content, criterion-related, and construct-related validity of assessment center exercises. Journal of Applied Psychology, 100, 1143–1168.

93.

*Hoffman

P. J.

Slovic

Rorer

L. G.

(1968). An analysis-of-variance model for the assessment of configural cue utilization in clinical judgment. Psychological Bulletin, 69, 338–349.

94.

*Holcomb

R. C.

(2011). Improving judgment performance and cognitive systems engineering by examining the relationship between task properties and cognitive mode [Publication No. 3455046] [Doctoral dissertation]. ProQuest Dissertations & Theses Global.

95.

*Hollenbeck

J. R.

(1984). Control theory, self-focus and behavior in organizations (self-consciousness, self-awareness, retail sales) [Publication No. 8241599] [Doctoral dissertation]. ProQuest Dissertations & Thesis Global.

96.

*Holtz

B. C.

Han

(2017). Resource foci, valence, and distributive justice effects: A meta-analysis and policy capturing study. In Academy of Management Proceedings (Vol. 2017, No. 1, p. 11586). Academy of Management.

97.

*Hoye

G. V.

Weijters

Lievens

Stockman

(2016). Social influences in recruitment: When is word-of-mouth most effective? International Journal of Selection and Assessment, 24, 42–53.

98.

IBM Corp. 2010. IBM SPSS Statistics for Macintosh, Version 19.0.

99.

*Ikomi

P. A.

Guion

R. M.

(2000). The prediction of judgment in realistic tasks: An investigation of self-insight. The International Journal of Aviation Psychology, 10, 135–153.

100.

*Irish

P. A.

III . (1987). Capturing the policies of time-constrained decision makers: The effects of deadline control, cue structure and individual difference variables [Accession Order No. 8728022] [Doctoral dissertation]. ProQuest Dissertations & Theses Global.

101.

*Jacklin

Svedalis

Harries

Darzo

Vincent

(2008). Judgment analysis: A method for quantitative evaluation of trainee surgeons’ judgments of surgical risk. The American Journal of Surgery, 195, 183–188.

102.

*Kaplan

Engelsted

Lei

Lockwood

(2018). Unpackaging manager mistrust in allowing telework: Comparing and integrating theoretical perspective. Journal of Business and Psychology, 33, 365–382.

103.

Karelaia

Hogarth

R. M.

(2008). Determinants of linear judgment: A meta-analysis of lens model studies. Psychological Bulletin, 134, 404–426.

104.

Karren

R. J.

Barringer

M. W.

(2002). A review and analysis of the policy-capturing methodology in organizational research: Guidelines for research and practice. Organizational Research Methods, 5, 337–361.

105.

*Karver

M. S.

(2000). The dangerousness to self judgment: Help line counselors’ accuracy, agreement, and insight into judgment [Publication No. 9996254] [Doctoral dissertation]. ProQuest Dissertations & Theses Global.

106.

Kaufmann

Reips

U. D.

Wittmann

W. W.

(2013). A critical meta-analysis of lens model studies in human judgment and decision-making. PLOS ONE, 8(12), Article e83528. https://doi.org/10.1371/journal.pone.0083528

107.

*Kennedy

D. J.

(1999). On the road again: An investigation of the situational and intentional antecedents of job relocation decisions in the service sector [Publication No. 9914661] [Doctoral dissertation]. ProQuest Dissertations & Theses Global.

108.

*Kight

W. D.

(2010). An analysis of reasonableness models for research assessments [Publication No. 3396317] [Doctoral dissertation]. ProQuest Dissertations & Theses Global.

109.

Knight

A. P.

Eisenkraft

(2015). Positive is usually good, negative is not always bad: The effects of group affect on social integration and task performance. Journal of Applied Psychology, 100, 1214–1227.

110.

Koslowsky

Sagie

(1993). On the efficacy of credibility intervals as indicators of moderator effects in meta-analytic research. Journal of Organizational Behavior, 14, 695–699.

111.

*Kroecker

T. S.

(1993). Situational and individual influences upon ethical decision-making in organizations: A policy capturing approach [Publication No. 9324225] [Doctoral dissertation]. ProQuest Dissertations & Theses Global.

112.

*Kung

Y. H.

(1997). The enhancement of self-insight through frequency count on cue usage [Publication No. 9815959] [Doctoral dissertation]. ProQuest Dissertations & Theses Global.

113.

*Kutcher

E. J.

III . (2007). Assessing fit in the interview: How candidates consider content and context cues to person organization fit [Publication No. 3310188] [Doctoral dissertation]. ProQuest Dissertations & Theses Global.

114.

LeBreton

J. M.

Senter

J. L.

(2008). Answers to 20 questions about interrater reliability and interrater agreement. Organizational Research Methods, 11, 815–852.

115.

*Lee

Dalal

R. S.

(2011). The effects of extremities on ratings of dynamic performance appraisal. Human Performance, 24, 99–118.

116.

Leigh

D. E.

(1986). Union preferences, job satisfaction, and the union-voice hypothesis. Industrial Relations, 25, 65–71.

117.

*Lelchook

(2010). The use of humor by leaders in response to situational stressors [Publication No. 1474720] [Master’s thesis]. ProQuest Dissertations & Theses Global.

118.

*Levi

(1989). Expert systems should be more accurate than human experts: Evaluation procedures from human judgment and decision making. IEEE Transactions on Systems, Man, and Cybernetics, 19, 647–657.

119.

F. F.

VanRullen

Koch

Perona

(2002). Rapid natural scene categorization in the near absence of attention. Proceedings of the National Academy of Sciences, 99, 9596–9601.

120.

*Lievens

Highhouse

Corte

(2005). The importance of traits and abilities in supervisors’ hirability decisions as a function of method of assessment. Journal of Occupational and Organizational Psychology, 78, 453–470.

121.

Lipsey

M. W.

Wilson

D. B.

(2001). Practical meta-analysis (Vol. 49). SAGE.

122.

*Liu

S. H.

(1993). Priority of criteria used for occupational ratings: A study of Gottfredson’s theory of circumscription and compromise [Publication No. 9334387] [Doctoral dissertation]. ProQuest Dissertations & Theses Global.

123.

*Lopina

E. C.

(2015). Understanding older workers’ decisions to participate in voluntary training opportunities [Publication No. 3721050] [Doctoral dissertation]. ProQuest Dissertations & Thesis Global.

124.

*MacDonald

H. A.

(2008). Motivational differences in feedback-seeking intentions: A cultural analysis [Publication No. 978-0-494-43310-2] [Doctoral dissertation]. ProQuest Dissertations & Theses Global.

125.

*MacDonald

H. A.

Sulsky

L. M.

Spence

J. R.

Brown

D. J.

(2013). Cultural differences in the motivation to seek performance feedback: A comparative policy-capturing study. Human Performance, 26, 211–235.

126.

Mackay

Barr

Kletke

(1992). An empirical investigation of the effects of decision aids on problem-solving processes. Decision Sciences, 23, 648–672.

127.

Mackworth

N. H.

(1948). The breakdown of vigilance during prolonged visual search. Quarterly Journal of Experimental Psychology, 1, 6–21.

128.

*Marcus

MacDonald

H. A.

Sulsky

L. M.

(2015). Do personal values influence the propensity for sustainability actions? A policy-capturing study. Journal of Business Ethics, 127, 459–478.

129.

*Marron

G. F.

(1994). Work and family issues: The impact of eldercare on work force policies and job choice decisions [Publication No. 9501263] [Doctoral dissertation]. ProQuest Dissertations & Theses Global.

130.

Marta-Pedroso

Freitas

Domingos

(2007). Testing for the survey mode effect on contingent valuation data quality: A case study of web based versus in-person interviews. Ecological Economics, 62, 388–398.

131.

*McDonough

T. A.

(2010). A policy capturing investigation of battered women’s decisions to stay in violent relationships. Violence and Victims, 25, 165–184.

132.

*McGee

H. M.

O’Boyle

C. A.

Hickey

O’Malley

Joyce

C. R. B.

(1991). Assessing the quality of life of the individual: The SEIQoL with a healthy and a gastroenterology unit population. Psychological Medicine, 21, 749–759.

133.

McGraw

K. O.

Wong

S. P.

(1992). A common language effect size statistic. Psychological Bulletin, 111, 361–365.

134.

*McIlroy

J. M. H.

(2005). The effect of decision condition in a judgmental policy capturing exercise [Publication No. 0-494-07626-7] [Doctoral dissertation]. ProQuest Dissertations & Theses Global.

135.

Meade

A. W.

Craig

S. B.

(2012). Identifying careless responses in survey data. Psychological Methods, 17, 437–455.

136.

*Meixner

W. F.

(1985). The effect of the organizational environment on judgment consensus: The case for professional government auditors in the state of Texas [Publication No. 8514267] [Doctoral dissertation]. ProQuest Dissertations & Theses Global.

137.

*Mellewigt

Thomas

Weller

Zajac

E. J.

(2017). Alliance or acquisition? A mechanisms-based, policy-capturing analysis. Strategic Management Journal, 38, 2353–2369.

138.

*Miller

T. L.

Wesley

C. L.

(2010). Assessing mission and resources for social change: An organizational identity perspective on social venture capitalists ‘decision criteria. Entrepreneurship Theory and Practice, 34, 705–733.

139.

*Mishra

Roch

S. G.

(2017). Do all raters value task, citizenship, and counterproductive behaviors equally: An investigation of cultural values and performance evaluations. Human Performance, 30, 193–211.

140.

Moher

Liberati

Tetzalaff

Altman

D. G.

, & the PRISMA Group. (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLOS Medicine, 6(7), Article e1000097. https://doi.org/10.1371/journal.pmed.1000097

141.

Molenaar

P. C.

Campbell

C. G.

(2009). The new person-specific paradigm in psychology. Current Directions in Psychological Science, 18, 112–117.

142.

*Monahan

C. J.

Muchinsky

P. M.

(1985). Intrasubject predictions of vocational preference: Convergent validation via the decision theoretic paradigm. Journal of Vocational Behavior, 27, 1–18.

143.

*Murnieks

C. Y.

Cardon

M. S.

Sudek

White

T. D.

Brooks

W. T.

(2016). Drawn to the fire: The role of passion, tenacity and inspirational leadership in angel investing. Journal of Business Venturing, 31, 468–484.

144.

*Murphy

K. R.

(1979). Convergent and discriminant validity of subjectively weighted models and regression models of decision making processes [Publication No. 8006031] [Doctoral dissertation]. ProQuest Dissertations & Theses Global.

145.

*O’Boyle

C. A.

(1994). The schedule for the evaluation of individual quality of life (SEIQoL). International Journal of Mental Health, 23, 3–23.

146.

*Oberoi

D. V.

Jiwa

Mcmanus

Parsons

(2016). Do men know which lower bowel symptoms warrant medical attention? A web-based video vignette survey of men in western Australia. American Journal of Men’s Health, 10, 474–486.

147.

*Ohme

Zacher

(2015). Job performance ratings: The relative importance of mental ability, conscientiousness, and career adaptability. Journal of Vocational Behavior, 87, 161–170.

148.

*Okopny

D. R.

(1985). Planning an international audit: An empirical investigation of internal auditor judgment [Publication No. 8526357] [Doctoral dissertation]. ProQuest Dissertations & Theses Global.

149.

*Olsen

J. E.

Martins

L. L.

(2016). Racioethnicity, community makeup, and potential employees’ reactions to organizational diversity management approaches. Journal of Applied Psychology, 101, 657–672.

150.

*Ostgaard

(1998). Team characteristics: Preferences and linkages to work attitudes [Publication No. 9826847] [Doctoral dissertation]. ProQuest Dissertations & Theses Global.

151.

Paivio

(1969). Mental imagery in associative learning and memory. Psychological Review, 76, 241–263.

152.

Paivio

(1971). Imagery and verbal processes. Holt, Rinehart, and Winston.

153.

*Parker

D. L.

(1989). Organizational culture and support for innovation: A policy-capturing approach [Publication No. 9023203] [Doctoral dissertation]. ProQuest Dissertations & Theses Global.

154.

*Paterson

Dowding

Harries

Cassells

Morrison

Niven

(2008). Managing the risk of suicide in acute psychiatric inpatients: A clinical judgement analysis of staff predictions of imminent suicide risk. Journal of Mental Health, 17, 410–423.

155.

*Pearson

A. M.

(2009). An exploratory examination of the impact of web functionality across the customer service life cycle: A multi-criteria approach [Publication No. 3358703] [Doctoral dissertation]. ProQuest Dissertations & Theses Global.

156.

*Phelps

R. H.

(1978). Expert livestock judgment: A descriptive analysis of the development of expertise [Publication No. 78-2413] [Doctoral dissertation]. ProQuest Dissertations & Theses Global.

157.

*Polk

K. M.

(2017). Can interview structure be manipulated to enhance applicant reactions [Publication No. 13837215] [Doctoral dissertation]. ProQuest Dissertations & Thesis Global.

158.

*Porter

C. M.

Parrigon

S. E.

Woo

S. E.

Saef

R. M.

Tay

(2016). Cultural and intellectual openness differentially relate to social judgments of potential work partners. Journal of Personality, 85, 632–642.

159.

*Qureshi

A. A.

(1993). An investigation of auditor judgment in the evaluation of contingent legal liabilities [Publication No. 9416553] [Doctoral dissertation]. ProQuest Dissertations & Theses Global.

160.

*Raaphorst

Groeneveld

Van de Walle

(2018). Do tax officials use double standards in evaluating citizen-clients? A policy-capturing study among Dutch frontline tax officials. Public Administration, 96, 134–153.

161.

Rensink

R. A.

(2002). Change detection. Annual Review of Psychology, 53, 245–277.

162.

*Reuer

J. J.

Tyler

B. B.

Tong

T. W.

C. W.

(2012). Executives’ assessments of international joint ventures in China: A multi-theoretical investigation. Management and Organization Review, 8, 311–340.

163.

*Reynolds

(2019). Predictors of interest in voluntary participation for mindfulness training [Unpublished master’s thesis]. San Diego State University.

164.

Roethlisberger

F. J.

Dickson

W. J.

(1939). Management and the worker. Harvard University Press.

165.

Rossi

P. H.

Anderson

A. B.

(1982). The factorial survey approach. In Rossi

P. H.

Nock

S. L.

(Eds.), Measuring social judgments (pp. 15–67). SAGE.

166.

*Rotundo

Sackett

P. R.

(2002). The relative importance of task, citizenship, and counterproductive performance to global ratings of job performance: A policy-capturing approach. Journal of Applied Psychology, 87, 66–80.

167.

*Russell

C. J.

Van Sell

(2012). A closer look at decisions to quit. Organizational Behavior and Human Decision Processes, 117, 125–137.

168.

Rynes

S. L.

Schwab

D. P.

Heneman

H. G.

(1983). The role of pay and market pay variability in job application decisions. Organizational Behavior and Human Performance, 31, 353–364.

169.

*Scheuch

(1983). A social judgment theory analysis of pregnancy resolution policies among college students and counselors (abortion) [Publication No. 8417999] [Doctoral dissertation]. ProQuest Dissertations & Thesis Global.

170.

Schmidt

F. L.

Hunter

J. E.

(2015). Methods of meta-analysis: Correcting error and bias in research findings (3rd ed.). SAGE.

171.

Schönbrodt

F. D.

Perugini

(2013). At what sample size do correlations stabilize? Journal of Research in Personality, 47, 609–612.

172.

*Sekiguchi

(2003). The role of person–organization fit and person–job fit in managers’ hiring decisions: The effects of work status and occupational characteristics of job openings [Publication No. 3102714] [Doctoral dissertation]. ProQuest Dissertations & Theses Global.

173.

*Sekiguchi

Huber

V. L.

(2011). The use of person–organization fit and person–job fit information in making selection decisions. Organizational Behavior and Human Decision Processes, 116, 203–216.

174.

Shanteau

Stewart

T. R.

(1992). Why study expert decision making? Some historical perspectives and comments. Organizational Behavior and Human Decision Processes, 53, 95–106.

175.

*Shepherd

D. A.

(1999). Venture capitalists’ assessment of new venture survival. Management Science, 45, 621–632.

176.

*Shepherd

D. A.

Zacharakis

(2003). A new venture’s cognitive legitimacy: An assessment by customers. Journal of Small Business Management, 41, 148–167.

177.

*Sherer

P. D.

Schwab

D. P.

Heneman

H. G.

(1987). Managerial salary-raise decisions: A policy-capturing approach. Personnel Psychology, 40, 27–38.

178.

*Silva-Castan

J. R.

(1996). The selection of institutions of higher education as consultants by small business owner/managers: A policy-capturing approach [Publication No. 9704935] [Doctoral dissertation]. ProQuest Dissertations & Theses Global.

179.

*Sinclair

A. L.

(2003). Disentangling contributions of process elements to the fair process effect: A policy-capturing approach [Publication No. 3087505] [Doctoral dissertation]. ProQuest Dissertations & Theses Global.

180.

Sireci

S. G.

Sukin

(2013). Test validity. In Geisinger

K. F.

(Editor-in-Chief) APA handbook of testing and assessment in psychology (Vol. 1, pp. 61–84). American Psychological Association.

181.

*Soltani

(2013). Organizational citizenship behavior, gender, and performance ratings: Does the rater matter? [Unpublished doctoral dissertation]. San Diego State University.

182.

*Stewart

T. R.

Middleton

Downton

Ely

(1984). Judgments of photographs vs. field observations in studies of perception and judgment of the visual environment. Journal of Environmental Psychology, 4, 283–302.

183.

*Stewart

T. R.

Moninger

W. R.

Brady

R. H.

Merrem

F. H.

Stewart

T. R.

Grassia

(1989). Analysis of expert judgment in a hail forecasting experiment. Weather and Forecasting, 4, 24–34.

184.

*Strawser

J. R.

(1985). An empirical investigation of auditor judgment: Factors affecting perceived audit risk [Publication No. 8605222] [Doctoral dissertation]. ProQuest Dissertations & Theses Global.

185.

*Teas

R. K.

(1985). An analysis of the temporal stability and structural reliability of metric conjoint analysis procedures. Journal of the Academy of Marketing Science, 13, 122–142.

186.

*Tews

M. J.

Stafford

Zhu

(2009). Beauty revisited: The impact of attractiveness, ability, and personality in the assessment of employment suitability. International Journal of Selection and Assessment, 17, 92–100.

187.

*Thomas

F. D.

III . (2009). Using judgment analysis to identify at-risk drivers and to evaluate the effectiveness of training for changing drivers’ perceptions of crash risk [Publication No. 3383930] [Doctoral dissertation]. ProQuest Dissertations & Theses Global.

188.

*Thompson

N. J.

(2013). Leader effectiveness in the eye of the beholder: Self-affirming implicit policies in leader perception [Publication No. 10598026] [Doctoral dissertation]. ProQuest Dissertations & Thesis Global.

189.

*Tomassetti

A. J.

(2017). On the reliability of judgments made in policy capturing measures: A meta-analytic review and experimental analysis [Unpublished doctoral dissertation]. George Mason University.

190.

*Tomassetti

A. J.

Dalal

R. S.

Kaplan

S. A.

(2016). Is policy capturing really more resistant than traditional self-report techniques to socially desirable responding? Organizational Research Methods, 19, 255–285.

191.

*Tomkovick

C. L.

(1993). An empirical investigation of product specific and product non-specific factors in making product continuance/discontinuance decisions [Publication No. 9328830] [Doctoral dissertation]. ProQuest Dissertations & Theses Global.

192.

Toth

A. A.

Banks

G. C.

Mellor

O’Boyle

E. H.

Dickson

Davis

D. J.

DeHaven

Bochantin

Borns

(2020). Study preregistration: An evaluation of a method for transparent reporting. Journal of Business and Psychology. Advance online publication. https://doi.org/10.1007/s10869-020-09695-3

193.

*Trouba

E. J.

(2007). A person-organization fit study of the Big Five personality model and attraction to organizations with varying compensation system characteristics [Publication No. 3321996] [Doctoral dissertation]. ProQuest Dissertations & Theses Global.

194.

*Turner

S. R.

(2008). The impact of role conceptualization on the process and outcomes of decision making in an educational context [Unpublished doctoral dissertation]. The University of Tennessee.

195.

*Tye

M. G.

(2005). Selection of expatriates: A study of decision-making models [Publication No. 3200703] [Doctoral dissertation]. ProQuest Dissertations & Theses Global.

196.

*Vega

R. P.

(2015). Why use flexible work arrangements? A policy capturing study examining the factors related to flexible work arrangement utilization [Publication No. 3706954] [Doctoral dissertation]. ProQuest Dissertations & Theses Global.

197.

Viechtbauer

(2010). Conducting meta-analyses in R with the metafor package. Journal of Statistical Software, 36(3), 1–48.

198.

*Viswesvaran

Barrick

M. R.

(1992). Decision-making effects on compensation surveys: Implications for market wages. Journal of Applied Psychology, 77, 588–597.

199.

Viswesvaran

Schmidt

F. L.

Deshpande

S. P.

(1994). A meta-analytic method for testing hypotheses about clusters of decision makers. Organizational Behavior and Human Decision Processes, 58, 304–321.

200.

*Voegel

J. A.

(2016). An exploratory examination of the ethical decision making process of entrepreneurs through the theory of planned behavior lens: A policy-capturing approach [Publication No. 10163453] [Doctoral dissertation]. ProQuest Dissertations & Thesis Global.

201.

*Wang

(2005). Factors determining Taiwanese nurses’ clinical judgments about hospitalized elderly patients with acute confusion [Publication No. 3208335] [Doctoral dissertation]. ProQuest Dissertations & Theses Global.

202.

*Warnick

B. J.

Murnieks

C. Y.

Mcmullen

J. S.

Brooks

W. T.

(2018). Passion for entrepreneurship or passion for the product? A conjoint analysis of angel and VC decision-making. Journal of Business Venturing, 33, 315–332.

203.

*Webber

S. A.

(1994). A comparison of judgmental decision modeling techniques: Models of internal control procedure evaluation using the analytic hierarchy process and policy-capturing ANOVA [Publication No. 9509744] [Doctoral dissertation]. ProQuest Dissertations & Theses Global.

204.

Wegman

L. A.

Hoffman

B. J.

Carter

N. T.

Twenge

J. M.

Guenole

(2018). Placing job characteristics in context: Cross-temporal meta-analysis of changes in job characteristics since 1975. Journal of Management, 44, 352–386.

205.

*Werner

Ones

D. S.

(2000). Determinants of perceived pay inequities: The effects of comparison other characteristics and pay-system communication. Journal of Applied Social Psychology, 30, 1281–1309.

206.

*Wexler

B. R.

(2017). Beauty is beneficial: An examination of candidate facial attractiveness, gender, qualification, and customer visibility on online recruitment intentions [Publication No. 10270836] [Doctoral dissertation]. ProQuest Dissertations & Thesis Global.

207.

*Whorton

(2015). Marked: A policy capturing investigation of job applicant tattoos as stigmatizing marks in blue and white collar employment [Publication No. 3710439] [Doctoral dissertation]. ProQuest Dissertations & Thesis Global.

208.

Williams

L. J.

O’Boyle

E. H.

J. J.

(2020). Condition 9 and 10 tests of model confirmation: A review of James, Mulaik, and Brett (1982) and contemporary alternatives. Organizational Research Methods, 23, 6–29.

209.

Wilson

D. B.

(2006). Meta-analysis macros for SAS, SPSS, and Stata. http://mason.gmu.edu/∼dwilsonb/ma.html

210.

*Wood

M. S.

Bylund

Bradley

(2016). The influence of tax and regulatory policies on entrepreneurs’ opportunity evaluation decisions. Management Decision, 54, 1160–1182.

211.

*Wood

M. S.

Williams

D. W.

(2014). Opportunity evaluation as rule-based decision making. Journal of Management Studies, 51, 573–602.

212.

*Yang

Thompson

Hamm

R. M.

Bland

Foster

(2013). The effect of improving task representativeness on capturing nurses’ risk assessment judgements: A comparison of written case simulations and physical simulations. BMC Medical Informatics and Decision Making, 13, 62–73.

213.

*Yarab

P. E.

(1999). Not all rivals are created equal: A policy capturing approach to examining jealousy and threat based on the characteristics of a rival [Publication No. 9950994] [Doctoral dissertation]. ProQuest Dissertations & Theses Global.

214.

*Zacharakis

A. L.

McMullen

J. S.

Shepherd

D. A.

(2007). Venture capitalists’ decision policies across three countries: an institutional theory perspective. Journal of International Business Studies, 38, 691–708.

215.

*Zacher

Dirkers

T. B.

Korek

Hughes

(2017). Age-differential effects of job characteristics on job attraction: A policy-capturing study. Frontiers in Psychology, 8, Article 1124. https://doi.org/10.3389/fpsyg.2017.01124

216.

Zedeck

(1977). An information processing model and approach to the study of motivation. Organizational Behavior and Human Performance, 18, 47–77.

217.

*Zhang

(2008). Corporate social responsibility, applicants’ ethical predispositions, and organizational attraction: A person–organization fit perspective [Publication No. 3288923] [Doctoral dissertation]. ProQuest Dissertations & Theses Global.

218.

*Zhang

Gowan

M. A.

(2012). Corporate social responsibility, applicants’ individual traits, and organizational attraction: A person–organization fit perspective. Journal of Business and Psychology, 27, 345–362.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.19 MB