A multiscale criterion-referenced test that featured two presumably equivalent forms (A and B), was administered to 1,667 Head Start children at each of four points over an academic year. Using a randomly equivalent groups design, three equating methods were applied: common-item IRT equating using concurrent calibration, linear transformation, and equipercentile transformation. The methods were compared by examining mean score differences, weighted mean squared difference, and Kolmogorov's D statistics for each subscale. The results indicated that over time the IRT equating method and conventional equating methods exhibited different patterns of discrepancy between the two test forms. IRT equating yielded marginally smaller form-to-form mean score differences and generated slightly f ewer distributional discrepancies between Forms A and B than both linear and equipercentile equating. However, the results were mixed indicating that more studies are needed to provide additional information on the relative merits and weaknesses of each approach.
Get full access to this article
View all access options for this article.
References
1.
AllenM. J.YenW. M. (1979) Introduction to measurement theory. Monterey, CA: Brooks/Cole.
2.
Carrow-WoolfolkE. (1995) Oral and Written Language Scales — Listening Comprehension Scale. Circle Pines, MN: American Guidance Service, Inc.
3.
ChenW.RevickiD. A.LaiJ.CookK. F.AmtmannD. (2009) Linking pain items from two studies onto a common scale using item response theory. Journal of Pain Symptom Management, 38(4), 615–628.
4.
ConoverW. J. (1999) Practical nonparametric statistics. (3rd ed.) New York: John Wiley.
5.
CookL. L.DouglasJ. B. (1982). Analysis of fit and vertical equating with the three-parameter model. Paper presented at the American Educational Research Association Annual Meeting, New York, March.
6.
CookL. L.EignorD. R. (1991) An NCME instructional module on IRT equating methods. Educational Measurement: Issues and Practice, 10(3), 37–45.
7.
CrockerL.AlginaJ. (1986) Introduction to classical and modern test theory. Fort Worth, TX: Harcourt.
8.
du ToitM. (Ed.) (2003) IRT from SSI, BILOG-MG, MULTILOG, PARSCALE, TESTFACT. Lincolnwood, IL: Scientific Software International.
9.
DunnL. M.DunnL. M. (1997) Peabody Picture Vocabulary Test-Third Edition Form A. Circle Pines, MN: American Guidance Service, Inc.
10.
FrancisD. J.SantiK. L.BarrC.FletcherJ. M.VariscoA.FoormanB. R. (2008) Form effects on the estimation of students' oral reading fluency using DIBELS. Journal of School Psychology, 46, 315–342.
11.
GinsburgH. P.BaroodyA. J. (2003) Test of Early Mathematics Ability-Third Edition Form A. Austin, TX: PRO-ED.
12.
GreenB. F. (1995) Comparability of scores from performance assessments. Educational Measurement: Issues and Practice, 14(4), 13–15.
13.
HambletonR. K.SwaminathanH.RogersH. J. (1991) Fundamentals of item response theory. Newbury Park, CA: Sage.
14.
HanT.KolenM.PohlmannJ. (1997) A comparison among IRT true- and observed-score equating and traditional equipercentile equating. Applied Measurement in Education, 10, 105–121.
15.
HansonB. A.BéguinA. A. (1999) Obtaining a common scale for IRT item parameters using separate versus concurrent estimation in the common item nonequivalent groups equating design. Paper presented at the Annual Meeting of the National Council on Measurement in Education, Montreal, April.
16.
HarrisD. J.CrouseJ. D. (1993) A study of criteria used in equating. Applied Measurement in Education, 6(3), 195–240.
17.
HarrisD. J.KolenM. J. (1990) A comparison of two equipercentile equating methods for common item equating. Educational and Psychological Measurement, 50, 61–71.
JodoinM. G.KellerL. A.SwaminathanH. (2003) A comparison of linear, fixed common item, and concurrent parameter equating procedures in capturing academic growth. The Journal of Experimental Education, 71(3), 229–250.
KolenM. J. (1981) Comparison of traditional and item response theory methods for equating tests. Journal of Educational Measurement, 18, 1–18.
23.
KolenM. J.BrennanR. L. (2004) Test equating, scaling, and linking: methods and practices. (2nd ed.) New York: Springer-Verlag.
24.
KolenM. J.HarrisD.J. (1990) Comparison of item preequating and random groups equating using IRT and equipercentile methods. Journal of Educational Measurement, 27, 27–39.
25.
KolenM. J.WhitneyD. R. (1982) Comparisons of four procedures for equating the General Tests of Educational Development. Journal of Educational Measurement, 19, 279–293.
26.
LeeW.BanJ. (2010) A comparison of IRT linking procedures. Applied Measurement in Education, 23, 23–48.
LordF. M. (1977) Practical applications of item characteristic curve theory. Journal of Educational Measurement, 14, 117–138.
29.
McDermottP. A.FantuzzoJ. W.WatermanC.AngeloL. E.WarleyH. P.GadsdenV. L.ZhangX. (2009) Measuring preschool cognitive growth while it's still happening: the Learning Express. Journal of School Psychology, 47, 337–366.
30.
MontagueM.PenfieldR. D.EndersC.HuangJ. (2010) Curriculum-based measurement of math problem solving: a methodology and rationale for establishing equivalence of scores. Journal of School Psychology, 48(1), 39–52. DOI: 10.1016/j.jsp.2009.08.002.
31.
PetersenN. S.CookL. L.StockingM. L. (1983) IRT versus conventional equating methods: a comparative study of scale stability. Journal of Educational Statistics, 8, 137–156.
32.
PetersenN. S.MarcoG. L.StewartE. E. (1982) A test of the adequacy of linear score equating models. In HollandP. W.RubinD. R. (Eds.), Test equating. New York, NY: Academic Press, Inc. Pp. 71–135.
33.
Public Law 107–110 (No Child Left Behind Act of 2001). (2002) Most recent set of amendments to the Elementary and Secondary Education Act of 1965. Retrieved from http://www.ed.gov.
34.
ReidD. K.HreskoW. P.HammillD. D. (2001) Test of Early Reading Ability-Third Edition Form A. Austin, TX: PRO-ED.
35.
SkaggsG.LissitzR. W. (1986) IRT test equating: relevant issues and a review of recent research. Review of Educational Research, 56, 495–529.
36.
ThissenD.WainerH. (2001) Test scoring. Mahwah, NJ: Erlbaum.
37.
TongY.KolenM. J. (2005) Assessing equating results on different equating criteria. Applied Psychological Measurement, 29, 418–432.
38.
U.S. Department of Health and Human Services. (2001) Head Start FACES: longitudinal findings on program performance. Third progress report. Washington, DC: Administration for Children and Families.
39.
U.S. Department of Health and uman Services. (2003) Head Start National Reporting System – Direct Child Assessment Fall and Spring. Washington, DC: Administration for Children and Families, Administration on Children, Youth, and Families, & Head Start Bureau.
40.
U.S. Department Health and Human Services. (2010) Head Start Impact Study Technical Report. Washington, DC: Administration for Children and Families.
41.
van der LindenW. J. (2000) A test-theoretic approach to observed score equating. Psychometrika, 65, 437–456.
42.
van der LindenW. J.LuechtR. M. (1998) Observed-score equating as a test assembly problem. Psychometrika, 63, 401–418.
