Extending a Practical Method for Developing Alternate Test Forms Using Independent Sets of Items

Abstract

This study describes alternate test form development for a Situational Judgment Inventory (SJI) predicting college performance. Principal axis factor analysis of responses to the SJI lent support for a general factor, yet each SJI form sampled items across 12 distinct rationally derived content areas. The first step of developing alternate forms involved random and representative sampling of SJI items across each content area, creating a large number of preliminary 36-item SJI test forms. Gibson and Weiner (1998) provided criteria for selecting alternate forms; however, the authors of the present study extended this approach in the next step of selecting alternate forms based on their estimated criterion-related validity with grade point average. Results provide initial support for the 144 alternate forms generated. This general approach reflects a practical and methodologically sound means of developing alternate forms of types of measures that are rationally heterogeneous yet empirically homogeneous.

Keywords

parallel forms test banks item banks test development reliability analysis

Get full access to this article

View all access options for this article.

References

American Educational Research Association , American Psychological Association , & National Council on Measurement in Education . (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.

Armstrong, R. D. , Jones, D. H. , & Wang, Z. (1994). Automated parallel test construction using classical test theory. Journal of Educational Statistics, 19, 73-90.

Armstrong, R. D. , Jones, D. H. , & Wu, I.-L. (1992). An automated test development of parallel tests from a seed test. Psychometrika, 57, 271-288.

Atkinson, R. C. (2001, April). Standardized tests and access to American universities. The 2001 Robert H. Atwell Distinguished Lecture, delivered at the 83rd Annual Meeting of the American Council on Education, Washington, DC.

Clause, C. S. , Mullins, M. E. , Nee, M. T. , Pulakos, E. , & Schmitt, N. (1998). Parallel test form development: A procedure for alternate predictors and an example. Personnel Psychology, 51, 193-298.

Cohen, J. (1988). Statistical power analysis for the behavioral sciences(2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.

Cortina, J. M. (1993). What is coefficient alpha? An examination of theory and application. Journal of Applied Psychology, 78, 98-104.

Cronbach, L. J. ,& Meehl, P. C. (1955). Construct validity in psychological tests. Psychological Bulletin, 52, 281-302.

Ghiselli, E. E. , Campbell, J. P. ,& Zedeck, S. (1981). Measurement theory for the behavioral sciences.San Francisco: Freeman.

10.

Gibson, W. M. , & Weiner, J. A. (1998). Generating random parallel test forms using CTT in a computer-based environment. Journal of Educational Measurement, 35, 297-310.

11.

Hakstian, A. R. ,& Barchard, K. A. (2000). Toward more robust inferential procedures for coefficient alpha under sampling of both subjects and conditions. Multivariate Behavioral Research, 35, 427-456.

12.

Hakstian, A. R. ,& Whalen, T. E. (1976). Ak-sample significance test for independent alpha coefficients. Psychometrika, 41, 219-231.

13.

Humphreys, L. G. (1956). The normal curve and the attenuation paradox in test theory. Psychological Bulletin, 53, 472-476.

14.

Lord, F. M. , & Novick, M. R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.

15.

McDonald, R. P. (1999). Test theory: A unified treatment. Mahwah, NJ: Lawrence Erlbaum.

16.

Motowidlo, S. J. , Russell, T. L. , Carter, G. W. , & Dunnette, M. D. (1989). Revision of the management selection interview: Final report. Minneapolis, MN: Personnel Decisions Research Institute.

17.

Nunnally, J. C. , & Bernstein, I. H. (1994). Psychometric theory(3rd ed.). New York: McGraw-Hill.

18.

Oswald, F. L. , Schmitt, N. , Kim, B. H. , Ramsay, L. J. , & Gillespie, M. A. (2004). Developing a biodata measure and situational judgment inventory as predictors of college student performance. Journal of Applied Psychology, 89, 187-207.

19.

Raju, N. S. , van der Linden, W. J. ,& Fleer, P. F. (1995). IRT-based internal measures of differential functioning of items and tests. Applied Psychological Measurement, 19, 353-368.

20.

Rosenthal, R. , & Rubin, D. B. (1986). Meta-analytic procedures for combining studies with multiple effect sizes. Psychological Bulletin, 99, 400-406.

21.

Sackett, P. R. , & Ellingson, J. E. (1997). The effects of forming multi-predictor composites on group differences and adverse impact. Personnel Psychology, 50, 707-721.

22.

SAS Institute, Inc. (1999). SAS/IML user’s guide. Version 8. Cary, NC: Author.

23.

Stanton, J. M. , Sinar, E. F. , Balzer, W. K. ,& Smith, P. C. (2002). Issues and strategies for reducing the length of self-report scales. Personnel Psychology, 55, 167-194.

24.

van der Linden, W. J. (1998). Optimal assembly of psychological and educational tests. Applied Psychological Measurement, 22, 195-211.

25.

Wightman, L. F. (1998). Practical issues in computerized test assembly. Applied Psychological Measurement, 22, 292-302.