The Item-Number Distortion Effect in Rank Order Testing: An Example Using the Clifton Strengthsfinder Inventory

Abstract

A possible distortion effect in tests that use (a) forced-choice ipsative scaling and (b) different numbers of items to determine categorical representation was investigated. It was proposed that the greater the number of items used to determine representation in a category, the lower the likelihood that category will be selected. The Clifton StrengthsFinder questionnaire, in which responses to 177 items are used to determine which of 34 talent themes are respondents' top five “Signature Themes,” was examined to ascertain whether or not the aforementioned distortion effect affected test results. Analysis of 1,198,645 completions of the questionnaire provided confirmatory evidence for the distortion effect. These findings call into question the trustworthiness of the Clifton StrengthsFinder questionnaire and any test that uses a similar design format.

Get full access to this article

View all access options for this article.

References

Asplund

Lopez

S. J.

Hodges

Harter

(2009) The Clifton StrengthsFinder 2.0 technical report: development and validation. Princeton, NJ: The Gallup Organization. Retrieved May 21, 2012, from http://strengths.gallup.com/private/Resources/CSFTechnicalReport031005.pdf.

Cattell

R.B.

(1944) Psychological measurement: ipsative, normative, and interactive. Psychological Review, 51, 292–303. DOI: 10.1037/h0057299.

Cornwell

J. M.

Dunlap

W. P.

(1994) On the questionable soundness of factoring ipsative data: a response to Saville & Willson (1991). Journal of Occupational and Organizational Psychology, 67, 89–100. DOI.org/10.1111/j.2044-8325.1994.tb00553.x.

Edwards

A. L.

(1959) Edwards Personal Preference Schedule. New York: Psychological Corp.

Greer

Dunlap

W. P.

(1997) Analysis of variance with ipsative measures. Psychological Methods, 2(2), 200–207. DOI.org/10.1037/1082-989X.2.2.200.

Guilford

J. P.

(1954) Psychometric methods. (2nd ed.) New York: McGraw-Hill.

Hambleton

R. K.

Swaminathan

Rogers

H. J.

(1991) Fundamentals of item response theory. Newbury Park, CA: Sage.

Huber

Payne

J. W.

Puto

(1982) Adding asymmetrically dominated alternatives: violations of regularity and the similarity hypothesis. Journal of Consumer Research, 9, 90–98. DOI.org/10.1086/208899.

Inscape Publishing. (2008) Everything DiSC® Classic research report. Minneapolis, MN: Inscape. Retrieved May 21, 2012, from http://www.manager-tools.com/docs/DiSC_Validation_Research_Report.pdf.

10.

Johnson

C. E.

Wood

Blinkhorn

S. F.

(1988) Spuriouser and spuriouser: the use of ipsative personality tests. Journal of Occupational Psychology, 61, 153–162. DOI: 10.1111/j.2044-8325.1988.tb00279.x.

11.

Kim

H. S.

Drolet

(2003) Choice and self-expression: a cultural analysis of variety-seeking. Journal of Personality and Social Psychology, 85(2), 373–382. DOI: 10.1037/0022-3514.85.2.273.

12.

Kraemer

H. C.

Theimann

(1987) How many subjects? Statistical power analysis in research. Newbury Park, CA: Sage.

13.

McClelland

D. C.

Atkinson

J. W.

Clark

Lowell

(1953) The achievement motive. New York: Appleton-Century-Crofts.

14.

Meade

A. W.

(2004) Psychometric problems and issues involved with creating and using ipsative measures for selection. Journal of Occupational and Organizational Psychology, 77, 531–552. DOI: 10.1348/0963179042596504.

15.

Myers

I. B.

(1962) The Myers-Briggs type indicator manual. Palo Alto, CA: Consulting Psychologists Press.

16.

Novick

M. R.

(1966) The axioms and principal results of classical test theory. Journal of Mathematical Psychology, 3, 1–18. DOI: 10.1016/0022-2496(66)90002-2.

17.

Samuelson

Zeckhauser

(1988) Status quo bias in decision making. Journal of Risk and Uncertainty, 1, 7–59. DOI.org/10.1007/BF00055564.

18.

Simonson

(1989) Choice based reasons: the case of attraction and compromise effects. Journal of Consumer Research, 16, 158–174. DOI.org/10.1086/209205.

19.

Tsetsos

Usher

Chater

(2010) Preference reversal in multiattribute choice. Psychological Review, 117(4), 1275–1293. DOI: 10.1037/a0020580.

20.

Tversky

(1972) Elimination by aspects: a theory of choice. Psychological Review, 79, 281–299. DOI.org/10.1037/h0032955.