Sage Journals: Discover world-class research

Abstract

If numerical reasoning items are administered under time limits, will two dimensions be required to account for the responses, a numerical ability dimension and a speed dimension? A total of 182 college students answered 74 numerical reasoning items. Every item was taken with and without time limits by half the students. Three psychometric models were fit to the data—one including no time-limit effect, one including a fixed time-limit effect, and one including a random effect of time limits. The latter model best fit the data, suggesting that a speed dimension, the random effect of time limits, is needed to account for time-limited responses. The estimated reliability of the Speed scores was .39. Despite this low reliability, Speed scores were correlated with American College Testing (ACT) math scores and response times. Speed scores added significantly to the ACT math score variance accounted for by the numerical reasoning dimension in the model. A within-person log-odds ratio interpretation of the Speed score is proposed. Possible methods of improving Speed score reliability and methods for studying the speed dimension are discussed.

Keywords

hierarchical linear model linear logistic model multidimensional item-response theory numerical reasoning quantitative reasoning Rasch model random effects response time limits test time limits speededness

Get full access to this article

View all access options for this article.

References

Akaike

(1974). A new look at statistical model identification. IEEE Transactions on Automatic Control, 19, 716-723.

Attali

(2005). Reliability of speeded number-right multiple-choice tests. Applied Psychological Measurement, 29, 357-368.

Cho

S.-J.

Rabe-Hesketh

(2011). Alternating imputation posterior estimation of models with crossed random effects. Computational Statistics & Data Analysis, 55, 12-25.

Davidson

W. M.

Carroll

J. B.

(1945). Speed and level components in time-limit scores: A factor analysis. Educational and Psychological Measurement, 5, 411-427.

Davison

M. L.

Kim

S.-K.

(2009). Factor analytic modeling of within person variation in score profiles. Multivariate Behavioral Research, 44, 668-687.

De Boeck

(2008). Random item IRT models. Psychometrika, 73, 533-559.

De Boeck

Wilson

(2004). Explanatory item response models. New York, NY: Springer.

Hambleton

R. K.

Swaminathan

(1985). Item response theory: Principles and applications. Boston, MA: Kluwer-Nihoff.

Jöreskog

K. G.

(1971). Statistical analysis of sets of congeneric tests. Psychometrika, 36, 109-133.

10.

Kamata

(2001). Item analysis by the hierarchical generalized linear model. Journal of Educational Measurement, 38, 79-93.

11.

Kang

Cohen

A. S.

(2007). IRT model selection methods for dichotomous items. Applied Psychological Measurement, 31, 331-358.

12.

Kang

Cohen

A. S.

Sung

H.-J.

(2009). Model selection indices for polytomous items. Applied Psychological Measurement, 33, 499-518.

13.

Kim

S.-K.

Davison

M. L.

Frisby

C. L.

(2007). Confirmatory factor analysis and profile analysis via multidimensional scaling. Multivariate Behavioral Research, 42, 1-32.

14.

Lohman

D. F.

(1979). Spatial ability: A review and reanalysis of the correlational literature (Technical Report No. 8). Stanford, CA: Stanford University, Aptitude Research Project, School of Education. (NTIS No. AD-A108–003).

15.

Lord

F. M.

(1956). A study of speed factors in tests and academic grades. Psychometrika, 21, 31-50.

16.

Nunnally

J. C.

(1978). Psychometric theory (2nd ed.). New York, NY: McGraw-Hill.

17.

Peterson

N. G.

(1993). Review of issues associated with speededness of GATB tests. Washington, DC: American Institutes for Research.

18.

Raudenbush

S. W.

Bryk

A. S.

Cheong

Y. F.

Congdon

(2004). HLM 6: Hierarchical linear and nonlinear modeling. Lincolnwood, IL: Scientific Software International.

19.

Thissen

D. M.

(1976/1977). Incorporating response latencies in latent trait estimation (Doctoral dissertation, University of Chicago, IL). Dissertation Abstracts International, 37, 4658B. (University Microfilms No. T-26116).

20.

Thissen

D. M.

(1983). Timed testing: An approach using item response theory. In Weiss

D. J.

(Ed.), New horizons in testing: Latent trait test theory and computerized adaptive testing (pp. 179-203). New York, NY: Academic Press.

21.

Van Breukelen

G. J. P.

(2005). Psychometric modeling of response speed and accuracy with mixed and conditional regression. Psychometrika, 70, 359-376.

22.

van der Linden

W. J.

(2006). A lognormal model for response times on test items. Journal of Educational and Behavioral Statistics, 31, 181-204.

23.

van der Linden

W. J.

(2009). Conceptual issues in response-time modeling. Journal of Educational Measurement, 46, 247-272.

24.

Wise

S. L.

DeMars

C. E.

(2009). A clarification of the effects of rapid guessing on coefficient α: A note on Attali’s “Reliability of Speeded Number-Right Multiple-Choice Tests.” Applied Psychological Measurement, 33, 488-490.

Modeling Individual Differences in Numerical Reasoning Speed as a Random Effect of Response Time Limits

Abstract

Keywords

Get full access to this article

References