There is much debate on the merits and pitfalls of standardized tests for college admission, with questions regarding the format (multiple-choice vs. constructed response), cognitive complexity, and content of these assessments (achievement vs. aptitude) at the forefront of the discussion. This study addressed these questions by investigating the relationship between SAT Mathematics (SAT-M) item characteristics and the item’s ability to predict college outcomes. Using multiple regression, SAT-M item characteristics (content area, format, cognitive complexity, and abstract/concrete classification) were used to predict three outcome measures: the correlation of item score with first-year college grade point average, the correlation of item score with mathematics course grades, and the percentage of students who answered the item correctly and chose to major in a mathematics or science field. Separate models were run including and excluding item difficulty and discrimination as covariates. The results revealed that many of the item characteristics were related to the outcome measures and that item difficulty and discrimination had a mediating effect on several of the predictor variables, particularly on the effects of nonroutine/insightful items and multiple-choice items.
AtkinsonR. C.GeiserS. (2009). Reflections on a century of college admissions tests. Educational Researcher, 38, 665-676.
2.
BennettR. E. (1993). On the meanings of constructed response. In BennettR. E.WardW. C. (Eds.), Construction versus choice in cognitive measurement. (pp. 1-27). Hillsdale, NJ: Lawrence Erlbaum.
3.
BrelandH. M.CampR.JonesR. J.MorrisM. M.RockD. A. (1987). Assessing writing skill. New York, NY: College Board.
4.
BridgemanB. (1991). Essays and multiple choice tests as predictors of college freshman GPA. Research in Higher Education, 32, 319-332.
5.
BridgemanB. (1992). A comparison of quantitative questions in open-ended and multiple-choice formats. Journal of Educational Measurement, 29, 253-271.
6.
BridgemanB.LewisC. (1991). Sex differences in the relationship of Advanced Placement essay and multiple-choice scores to grades in college courses (RR-91-48). Princeton, NJ: Educational Testing Service.
7.
DoransN. J.LawrenceI. M. (1987). The internal construct validity of the SAT (ETS Research Rep. RR 87-35). Princeton, NJ: Educational Testing Service.
8.
DoubT. W.LawrenceI. M. (1996, April). Item-level dimensionality of SAT I verbal and mathematical reasoning tests. Paper presented at the meeting of the American Educational Research Association, New York, NY.
9.
EnrightM. K.MorleyM.SheehanK. M. (2002). Items by design: The impact of systematic feature variation on item statistical characteristics. Applied Measurement in Education, 15, 49-74.
10.
EwingM.HuffK.AndrewsM.KingK. (2005). Assessing the reliability of skills measured by the SAT (College Board Research Note RN-24). New York, NY: College Board.
11.
GeiserS.StudleyR. (2002). UC and the SAT: Predictive validity and differential impact of the SAT I and SAT II at the University of California. Educational Assessment, 8, 1-26.
12.
GierlM. J.TanX.WangC. (2005). Identifying content and cognitive dimensions on the SAT (College Board Research Rep. No. 2005-11). New York, NY: College Board.
13.
HaladynaT. M. (2004). Developing and validating multiple-choice test items. London, England: Psychology Press.
14.
KaminskiJ. A.SloutskyV. M.HecklerA. F. (2008). The advantage of abstract examples in learning math. Science, 320, 454-455.
15.
KobrinJ. L.CamaraW. J.MilewskiG. B. (2002). The utility of the SAT I and SAT II for admissions decisions in California and the nation (College Board Research Rep. No. 2002-6). New York, NY: College Board.
16.
KobrinJ. L.PattersonB. F.ShawE. J.MatternK. D.BarbutiS. M. (2008). Validity of the SAT for predicting first-year college grade point average (College Board Research Rep. No. 2008-5). New York, NY: College Board.
17.
KoedingerK. R.AlibaliM. W.NathanM. J. (2008). Trade-offs between grounded and abstract representations: Evidence from algebra problem solving. Cognitive Science, 32, 366-397.
18.
KoedingerK. R.NathanM. J. (2004). The real story behind story problems: Effects of representation on quantitative reasoning. Journal of the Learning Sciences, 13, 129-164.
19.
LinnR.BakerE. L.DunbarS. B. (1991). Complex, performance-based assessment: Expectations and validation criteria. Educational Researcher, 20(8), 15-21.
20.
McDonaldM. (2002). Systematic assessment of learning outcomes: Developing multiple-choice exams. Sudbury, MA: Jones & Bartlett.
21.
National Association for College Admissions Counseling. (2008). Report of the Commission on the Use of Standardized Tests in Undergraduate Admissions. Arlington, VA: Author.
22.
National Center for Education Statistics. (2009). Students who study science, technology, engineering, and mathematics (STEM) in postsecondary education (Stats in Brief, NCES 2009-161). Washington, DC: U.S. Department of Education.
23.
National Science Board. (2010). Preparing the next generation of STEM innovators: Identifying and developing our nation’s human capital (NSB-10-33). Arlington, VA: Author. Retrieved from http://www.nsf.gov/nsb/publications/2010/nsb1033.pdf
24.
NunesT.SchliemannA. D.CarraherD. W. (1993). Street mathematics and school mathematics. New York, NY: Cambridge University Press.
25.
OhH.SathyV. (2006). Construct comparability and continuity in the SAT (Unpublished research report). New York, NY: College Board.
26.
O’NeillK. A.McPeekW. M. (1993). Item and test characteristics that are associated with differential item functioning. In HollandP.WainerH. (Eds.), Differential item functioning (pp. 255-336). Hillsdale, NJ: Lawrence Erlbaum.
27.
RamistL.LewisC.McCamley-JenkinsL. (2001). Using achievement tests/SAT II Subject Tests to demonstrate achievement and predict college grades: Sex, language, ethnic, and parental education groups. (College Board Research Report No. 2001-5). New York: College Board.
28.
RoussosL. A.NortonL. L. (1998, March). LSAT item-type validity study (Tech. Rep. 98-01). Newtown, PA: Law School Admission Council.
29.
RyanA. M.GregurasG. J. (1998). Life is not multiple choice: Reaction to the alternatives. In HakelM. D. (Ed.), Beyond multiple choice: Evaluating alternatives to traditional testing for selection (pp. 183-202). Mahwah, NJ: Lawrence A. Erlbaum.
30.
SebrechtsM. M.EnrightM.BennettR. E.MartinK. (1996). Using algebra word problems to assess quantitative ability: Attributes, strategies, and errors. Cognition and Instruction, 14, 285-343.
31.
VasavadaN.CarmanE.HartB.LuisierD. (2010). Common core state standards alignment: ReadiStep, PSAT/NMSQT and SAT (College Board Research Rep. No. 2010-5A). New York, NY: College Board.
