Effects of Test Preparation on the Validity of a of a Graduate Admissions Test

Abstract

Test score improvement has been the major concern in nearly all the extant studies of special preparation, or "coaching," for tests. Recently, however, logical analyses of the possible outcomes and implications of special test preparation (Anastasi, 1981; Cole, 1982; Messick, 1981) have suggested that the issue of test score effects is but one aspect of the controversy sur rounding coaching; the impact of special preparation on test validity is an equally germane consideration. Although the assumption is sometimes made that coaching can serve only to dilute the construct validity and impair the predictive power of a test, some kinds of special preparation may, by reducing irrelevant sources of test difficulty, actually improve both con struct validity and predictive validity. This study ex amined the relationships of both internal and external criteria to Graduate Record Examination (GRE) candi dates' performance on several analytical ability item types, obtained under several test preparation condi tions. The purpose was to assess the effects of these various preparations on test reliability and validity. The preparation conditions were those previously shown to be effective, in varying degrees, in improv ing examinee performance on two of three analytical item types (Powers & Swinton, 1982, 1984). The data for this study were those collected by Powers & Swin ton (1982, 1984). The results suggest that GRE ana lytical ability scores may relate more strongly to aca demic performance after special test preparation than under more standard conditions and that they may re late less to measures of other cognitive abilities (ver bal and quantitative scores). No consistent effects were detected on either the internal consistency or the convergent validity of the analytical measure.

Get full access to this article

View all access options for this article.

References

Anastasi, A. (1981). Coaching, test sophistication, and developed abilities . American Psychologist, 36(10), 1086-1093.

Cole, N. (1982). The implications of coaching for ability testing . In A. Wigdor & W. R. Garner (Eds.), Ability Testing: Uses, consequences, and controversies, Part II: Documentation sections (pp. 389-414). Washington DC: National Academy Press.

Kendall, L.M. (1964). The effects of varying time limits on test validity . Educational and Psychological Measurement, 24, 789-800.

Kingston, N.M. , & Dorans, N.J. (1982). The effect of the position of an item within a test on item responding behaviors: An analysis based on item response theory (GRE Board Professional Report GREB 79-126P and ETS Research Report 82-22) . Princeton NJ: Educational Testing Service.

Levy, S. (1979, March). ETS and the "coaching" cover-up. New Jersey Monthly, pp. 51-54, 82-89.

Lieberman, D. (1979, June 20). You can study for aptitude tests. The Hartford Advocate, pp. 6-7, 16.

Linn, R.L. (1977). On the treatment of multiple scores for Law School Admission Test repeaters (Report #LSAC-77-4). In Law School Admission Council, Reports of LSAC Sponsored Research: Volume III, 1975-1977 . Princeton NJ: Law School Admission Council.

Marron, J.E. (1965). Preparatory school test preparation: Special test preparation, its effect on College Board scores and the relationship of affected scores to subsequent college performance. West Point NY : United States Military Academy, Research Division, Office of the Director of Admissions and Registrar.

Messick, S. (1981). The controversy over coaching: Issues of effectiveness and equity. In B. F. Green (Ed.), Issues in testing: Coaching, disclosure, and ethnic bias (pp. 21-53). San Francisco: Jossey-Bass.

10.

Olsen, M. , & Schrader, W.B. (1959). The use of preliminary and final Scholastic Aptitude Test scores in predicting college grades (College Entrance Examination Board Research and Development Reports, and Statistical Report #59-19). Princeton NJ: Educational Testing Service.

11.

Ortar, G.R. (1960). Improving test validity by coaching. Educational Research, 2, 137-142.

12.

Powers, D.E. , & Swinton, S.S. (1981). Extending the measurement of graduate admission abilities beyond the verbal and quantitative domains. Applied Psychological Measurement, 5, 141-158.

13.

Powers, D.E. , & Swinton, S.S. (1982). The effects of self-study of test familiarization materials for the analytical section of the GRE Aptitude Test (GREB Research Report GREB No. 79-9). Princeton NJ: Educational Testing Service.

14.

Powers, D.E. , & Swinton, S.S. (1984). Effects of self-study for coachable test item types. Journal of Educational Psychology, 76, 266-278.

15.

Stricker, L.J. (1982). Test disclosure and retest performance on the Scholastic Aptitude Test (College Board Report No. 82-7 and ETS RR No. 82-48) . Princeton NJ: Educational Testing Service.

16.

Swinton, S.S. , & Powers, D.E. (1983). A study of the effects of special preparation on GRE analytical scores and item types. Journal of Educational Psychology, 75, 104-115.

17.

Swinton, S.S. , Wild, C.L. , & Wallmark, M. (1983). Investigation of practice effects on item types in the Graduate Record Examinations Aptitude Test (GREB Professional Report No. 80-1 CP and ETS RR 82-56). Princeton NJ: Educational Testing Service.

18.

Wilson K.M. (1981). A study of the validity of the restructured GRE Aptitude Test for predicting first-year performance in graduate study (GREB Research Report GREB No. 78-6). Princeton NJ: Educational Testing Service.

19.

Wilson, K.M. (1985). The relationship of GRE General Test item-type part scores to undergraduate grades (GREB Professional Report GREB No. 81-22P) Princeton NJ: Educational Testing Service.