Sage Journals: Discover world-class research

Abstract

This study defines subpopulation item parameter drift (SIPD) as a change in item parameters over time that is dependent on subpopulations of examinees, and hypothesizes that the presence of SIPD in anchor items is associated with bias and/or lack of invariance in three psychometric outcomes. Results show that SIPD in anchor items is associated with a lack of invariance in dimensionality structure of an anchor test, a lack of invariance in scaling coefficients across subpopulations, and a lack of invariance in ability estimates. It is demonstrated that these effects go beyond what can be understood from item parameter drift or differential item functioning.

Keywords

item parameter drift invariance differential item functioning fairness

Get full access to this article

View all access options for this article.

References

Alexander

K. L.

Entwisle

D. R.

Dauber

S. L.

(1996). Children in motion: School transfers and elementary school achievement. Journal of Educational Research, 90, 3-12.

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. Washington, DC: American Psychological Association.

Andrich

Hagquist

(2015). Real and artificial differential item functioning in polytomous items. Educational and Psychological Measurement, 75, 185-207.

Battauz

(2014). equateIRT: Direct, chain and average equating coefficients with standard errors using IRT methods. R package version 1.2. Retrieved from http://CRAN.R-project.org/package=equateIRT.

Baxter

S. D.

Royer

J. A.

Hardin

J. W.

Guinn

C. H.

Devlin

C. M.

(2011). The relationship of school absenteeism with body mass index, academic achievement, and socioeconomic status among fourth-grade children. Journal of School Health, 81, 417-423.

Birnbaum

(1968). Some latent trait models and their use in inferring an examinee’s ability. In Lord

F. M.

Novick

M. R.

(Eds.), Statistical theories of mental test scores (pp. 397-460). Reading, MA: Addison-Wesley.

Bock

R. D.

Muraki

Pfeiffenberger

(1988). Item pool maintenance in the presence of item parameter drift. Journal of Educational Measurement, 25, 275-285.

Chen

(2013). Remove or keep: Linking items showing item parameter drift (Doctoral dissertation). Retrieved from ProQuest Dissertations and Theses database. (UMI No. 3560290)

DeMars

(2012). Detection of item parameter drift over multiple test administrations. Applied Measurement in Education, 17, 265-300.

10.

Deng

Melican

(2009, April). An investigation of scale drift in computer adaptive test. Paper presented at the annual meeting of the National Council on Measurement in Education, San Diego, CA.

11.

Dorans

N. J.

(2004). Using subpopulation invariance to assess test score equity. Journal of Educational Measurement, 41, 43-68.

12.

Dorans

N. J.

Holland

P. W.

(1993). DIF detection and description: Mantel-Haenszel and standardization. In Holland

P. W.

Wainer

(Eds.), Differential item functioning (pp. 35-66). Hillsdale, NJ: Lawrence Erlbaum.

13.

Dorans

N. J.

Liu

(2009). Score equity assessment: Development of a prototype analysis using SAT© mathematics test data across several administrations (ETS Research Report No. RR-09-08). Princeton, NJ: Educational Testing Service.

14.

Education Week. (2015). The Every Student Succeeds Act: Explained. Education Week, 35(14), 17.

15.

Giordano

Subhiyah

Hess

(2005, April). An analysis of item exposure and item parameter drift on a take-home recertification exam. Paper presented at the annual meeting of the American Educational Research Association, Montreal, Quebec, Canada. Retrieved from http://eric.ed.gov/?id=ED497708.

16.

Goldstein

(1983). Measuring changes in educational attainment over time: Problems and possibilities. Journal of Educational Measurement, 20, 369-377.

17.

Han

K. T.

Wells

C. S.

Sireci

S. G.

(2012). The impact of multidirectional item parameter drift on IRT scaling coefficients and proficiency estimates. Applied Measurement in Education, 25, 97-117.

18.

Holland

P. W.

Thayer

D. T.

(1988). Differential item performance and the Mantel-Haenszel procedure. In Wainer

Braun

H. I.

(Eds.), Test validity (pp. 129-145). Hillsdale, NJ: Lawrence Erlbaum.

19.

Huggins

A. C.

(2014). The effect of differential item functioning in anchor items on population invariance of equating. Educational and Psychological Measurement, 74, 627-658.

20.

Human Resources Research Organization, & Harcourt Assessment. (2006). Reading and mathematics: Technical report for 2005 FCAT test administrations. San Antonio, TX: Author. Retrieved online from http://fcat.fldoe.org/fcatpub5.asp.

21.

Joint Committee on Testing Practices. (2004). Code for fair testing practices in education. Washington, DC: Author. Retrieved from http://www.apa.org/science/programs/testing/fair-code.aspx.

22.

Kane

M. T.

(2006). Validation. In Brennan

R. L.

(Ed.), Educational measurement (4th ed., pp. 17-64). Westport, CT: American Council on Education/Praeger.

23.

Karger

Hitchcock

(2003). Access to the general curriculum for students with disabilities: A brief for parents and teachers. Wakefield, MA: National Center on Accessible Instructional Materials. Retrieved from http://aim.cast.org/learn/historyarchive/backgroundpapers/brief_parents_teachers#.VRLwQOGnRod.

24.

Klein

(2015, December 10). President signs ESEA Rewrite, giving states, districts bigger say on policy. Education Week. Retrieved from http://blogs.edweek.org/edweek/campaign-k-12/2015/12/president_barack_obama_signs_e.html?cmp=eml-enl-eu-news1.

25.

Krause

(2012). Assessment of item parameter drift of known items in a university placement exam (Master’s thesis, Arizona State University). Retrieved from ProQuest Dissertations and Theses database. (UMI No. 963541958).

26.

Wang

(2015). Assessment of differential item functioning under cognitive diagnosis models: The DINA model example. Journal of Educational Measurement, 52, 28-54.

27.

Mislevy

R. J.

(1982, March). Five steps toward controlling item parameter drift. Paper presented at the annual meeting of the American Educational Research Association, New York, NY.

28.

Muthén

L. K.

Muthén

B. O.

(2012). Mplus user’s guide. 7th ed.Los Angeles, CA: Muthén & Muthén.

29.

No Child Left Behind: State Plans, 20 U.S.C. § 1111 (2001).

30.

Penfield

R. D.

Camilli

(2007). Differential item functioning and item bias. In Rao

C. R.

Sinharay

(Eds.) Handbook of statistics: Vol. 26. Psychometrics (pp. 125-167). New York, NY: Elsevier.

31.

R Development Core Team. (2011). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Retrieved from http://www.R-project.org/.

32.

Rennie Center of Education Research and Policy. (2013). Research meets policy: Opportunity to learn science?Boston, MA: Author.

33.

Rossell

C. H.

(2003). Policy matters in teaching English language learners: New York and California. New York, NY: ERIC Clearinghouse on Urban Education.

34.

Rupp

A. A.

Zumbo

(2006). Understanding parameter invariance in unidimensional IRT models. Educational and Psychological Measurement, 66, 63-84.

35.

Snyder

C. W.

Mereku

K. D.

Amedahe

F. K.

Etsey

Adu

(2013). National assessment and opportunity to learn in education reform. International Journal of Educational Reform, 22, 76-102.

36.

Stocking

M. L.

Lord

F. M.

(1983). Developing a common metric in item response theory. Applied Psychological Measurement, 7, 201-210.

37.

Sukin

T. M.

(2010). Item parameter drift as an indication of differential opportunity to learn: An exploration of item flagging methods & accurate classification of examinees (Doctoral dissertation, University of Massachusetts Amherst). Retrieved from ProQuest Dissertations and Theses database. (UMI No. 814814255)

38.

Sykes

R. C.

Fitzpatrick

A. R.

(1992). The stability of IRT b values. Journal of Educational Measurement, 29, 201-211.

39.

U.S. Department of Education (2009). Standards and assessments peer review guidance: Information and examples for meeting requirements of the No Child Left Behind Act of 2001. Washington, DC: Author. Retrieved from http://www2.ed.gov/admins/lead/account/peerreview/index.html

40.

U.S. Senate (2015). The Every Child Achieves Act of 2015 (draft bill). Washington, DC: Author. Retrieved from http://blogs.edweek.org/edweek/campaign-k-12/Senate%20Bill.pdf.

41.

Ujifusa

(2014, January 14). All 50 states now using testing or student performance for school accountability. Education Week. Retrieved from http://blogs.edweek.org/edweek/state_edwatch/2014/01/all_50_states_now_using_testing_or_student_performance_for_school_accountability.html

42.

Wells

C. S.

Hambleton

R. K.

Kirkpatrick

Meng

(2014). An examination of two procedures for identifying consequential item parameter drift. Applied Measurement in Education, 27, 214-231.

43.

Wells

C. S.

Subkoviak

M. J.

Serlin

R. C.

(2002). The effect of item parameter drift on examinee ability estimates. Applied Psychological Measurement, 26, 77-87.

44.

Yang

W. L.

Gao

(2008). Invariance of score linkings across gender groups for forms of a testlet-based college-level examination program examination. Applied Psychological Measurement, 32, 45-61.

45.

Xin

(2014). Effects of item parameter drift on vertical scaling with the nonequivalent groups with anchor test (NEAT) design. Educational and Psychological Measurement, 74, 227-235.

46.

Harris

D. J.

Gao

(2008). Invariance of equating functions across different subgroups of examinees taking a science achievement test. Applied Psychological Measurement, 32, 62-80.

47.

Zimowski

M. F.

Muraki

Mislevy

R. J.

Bock

R. D.

(2003). BILOG-MG 3 for Windows: Multiple-group IRT analysis and test maintenance for binary items [Computer software]. Lincolnwood, IL: Scientific Software International.

Psychometric Consequences of Subpopulation Item Parameter Drift

Abstract

Keywords

Get full access to this article

References