Sage Journals: Discover world-class research

Abstract

Empirical growth benchmarks, as introduced by Hill, Bloom, Black, and Lipsey (2008), are a well-known way to contextualize effect sizes in education research. Past work on these benchmarks, both positive and negative, has largely avoided confronting the role of vertical scales, yet technical issues with vertical scales trouble the use of such benchmarks. This article introduces vertical scales and outlines their role in the creation of empirical benchmarks for growth. I then outline three strands of recent vertical scale research that call into question the grounds for relying on these benchmarks. I conclude with recommendations for researchers looking to contextualize observed effects of educational interventions without confounding their effects with vertical scaling artifacts.

Keywords

assessment educational policy effect size evaluation experimental research item response theory program evaluation psychometrics research methodology testing

Get full access to this article

View all access options for this article.

References

Baird

M. D.

Pane

J. F.

(2019). Translating standardized effects of education programs into more interpretable metrics. Educational Researcher, 48(4), 217–228. https://doi.org/10.3102/0013189X19848729

Betebenner

(2009). Norm- and criterion-referenced student growth. Educational Measurement: Issues and Practice, 28(4), 42–51. https://doi.org/10.1111/j.1745-3992.2009.00161.x

Birnbaum

(1968). Some latent trait models and their use in inferring an examinee’s ability. In Lord

F. M.

Novick

M. R.

(Eds.), Statistical theories of mental test scores (pp. 395–479). Addison-Wesley.

Bloom

H. S.

Hill

C. J.

Black

A. R.

Lipsey

M. W.

(2008). Performance trajectories and performance gaps as achievement effect-size benchmarks for educational interventions. Journal of Research on Educational Effectiveness, 1(4), 289–328. https://doi.org/10.1080/19345740802400072

Bolt

D. M.

Deng

Lee

(2014). IRT model misspecification and measurement of growth in vertical scaling. Journal of Educational Measurement, 51(2), 141–162. https://doi.org/10.1111/jedm.12039

Briggs

D. C.

(2013). Measuring growth with vertical scales. Journal of Educational Measurement, 50(2), 204–226. https://doi.org/10.1111/jedm.12011

Briggs

D. C.

(2019). Interpreting and visualizing the unit of measurement in the Rasch Model. Measurement, 146, 961–971. https://doi.org/10.1016/j.measurement.2019.07.035

Briggs

D. C.

Dadey

(2015). Making sense of common test items that do not get easier over time: Implications for vertical scale designs. Educational Assessment, 20(1), 1–22. https://doi.org/10.1080/10627197.2014.995165

Briggs

D. C.

Domingue

(2013). The gains from vertical scaling. Journal of Educational and Behavioral Statistics, 38(6), 551–576. https://doi.org/10.3102/1076998613508317

10.

Briggs

D. C.

Weeks

J. P.

(2009). The impact of vertical scaling decisions on growth interpretations. Educational Measurement: Issues and Practice, 28(4), 3–14. https://doi.org/10.1111/j.1745-3992.2009.00158.x

11.

Castellano

K. E.

A. D.

(2015). Practical differences among aggregate-level conditional status metrics: From median student growth percentiles to value-added models. Journal of Educational and Behavioral Statistics, 40(1), 35–68. https://doi.org/10.3102/1076998614548485

12.

Cohen

(1969). Statistical power analysis for the behavioral sciences. Academic Press.

13.

Dadey

Briggs

D. C.

(2012). A meta-analysis of growth trends from vertically scaled assessments. Practical Assessment, Research and Evaluation, 17. https://doi.org/10.7275/F2BM-6R59

14.

Domingue

(2014). Evaluating the equal-interval hypothesis with test score scales. Psychometrika, 79(1), 1–19. https://doi.org/10.1007/s11336-013-9342-4

15.

Embretson

S. E.

Reise

S. P.

(2000). Item Response Theory for Psychologists. Lawrence Erlbaum Associates.

16.

Haebara

(1980). Equating logistic ability scales by a weighted least squares method. Japanese Psychological Research, 22(3), 144–149. https://doi.org/10.4992/psycholres1954.22.144

17.

Hill

C. J.

Bloom

H. S.

Black

A. R.

Lipsey

M. W.

(2008). Empirical benchmarks for interpreting effect sizes in research. Child Development Perspectives, 2(3), 172–177. https://doi.org/10.1111/j.1750-8606.2008.00061.x

18.

Kolen

M. J.

Brennan

R. L.

(2014). Test Equating, Scaling, and Linking (3rd ed.). Springer New York. https://doi.org/10.1007/978-1-4939-0317-7

19.

Kraft

M. A.

(2020). Interpreting effect sizes of education interventions. Educational Researcher, 49(4), 241–253. https://doi.org/10.3102/0013189X20912798

20.

Lissitz

R. W.

(2012). Exploring the full-information bifactor model in vertical scaling with construct shift. Applied Psychological Measurement, 36(1), 3–20. https://doi.org/10.1177/0146621611432864

21.

Lord

F. M.

Novick

M. R.

(1968). Statistical theories of mental test scores. Addison-Wesley.

22.

Lortie-Forgues

Sio

U. N.

Inglis

(2021). How should educational effects be communicated to teachers? Educational Researcher, 50(6), 345–354. https://doi.org/10.3102/0013189X20987856

23.

Loyd

B. H.

Hoover

H. D.

(1980). Vertical equating using the Rasch model. Journal of Educational Measurement, 17(3), 179–193. https://doi.org/10.1111/j.1745-3984.1980.tb00825.x

24.

Luce

R. D.

Tukey

J. W.

(1964). Simultaneous conjoint measurement: A new type of fundamental measurement. Journal of Mathematical Psychology, 1(1), 1–27. https://doi.org/10.1016/0022-2496(64)90015-X

25.

Marco

G. L.

(1977). Item characteristic curve solutions to three intractable testing problems (ETS Research Bulletin Series). https://onlinelibrary.wiley.com/doi/10.1002/j.2333-8504.1977.tb01136.x

26.

Martineau

J. A.

(2006). Distorting value added: The use of longitudinal, vertically scaled student achievement data for growth-based, value-added accountability. Journal of Educational and Behavioral Statistics, 31(1), 35–62. https://doi.org/10.3102/10769986031001035

27.

Maul

Torres Irribarra

Wilson

(2016). On the philosophical foundations of psychological measurement. Measurement, 79, 311–320. https://doi.org/10.1016/j.measurement.2015.11.001

28.

Rasch

(1960). Probabilistic models for some intelligence and attainment tests. Danish Institute for Educational Research.

29.

Samejima

(2000). Logistic positive exponent family of models: Virtue of asymmetric item characteristic curves. Psychometrika, 65(3), 319–335. https://doi.org/10.1007/BF02296149

30.

Simpson

(2021). Benchmarking a misnomer: A note on “Interpreting Effect Sizes in Education Interventions.” Educational Researcher, 0013189X2098544. https://doi.org/10.3102/0013189X20985448

31.

Stocking

M. L.

Lord

F. M.

(1983). Developing a common metric in Item Response Theory. Applied Psychological Measurement, 7(2), 201–210. https://doi.org/10.1177/014662168300700208

32.

Tong

Kolen

M. J.

(2007). Comparisons of methodologies and results in vertical scaling for educational achievement tests. Applied Measurement in Education, 20(2), 227–253. https://doi.org/10.1080/08957340701301207

33.

Tong

Kolen

M. J.

(2010). Scaling: An ITEMS Module: Winter 2010. Educational Measurement: Issues and Practice, 29(4), 39–48. https://doi.org/10.1111/j.1745-3992.2010.00192.x

Vertical Scales,Deceleration,and Empirical Benchmarks for Growth

Abstract

Keywords

Get full access to this article

References