In instructional sensitivity research, it is important to evaluate the validity argument about the extent to which student performance on the assessment can be used to infer differences in instructional experiences. This study examines whether three different measures of mathematics instruction consistently identify mathematics assessments as being sensitive to instruction. Mixed findings across fourth-grade (n = 8,298) and fifth-grade (n = 9,336) students and their teachers across three school districts raise questions as to whether different ways of measuring instruction provide similar inferences about the instructional sensitivity of assessments. This raises validity concerns about the quality of inferences based on different measures of instruction.
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
2.
BrennanR. L. (2013). Commentary on “Validating the Interpretations and Uses of Test Scores.”Journal of Educational Measurement, 50, 74-83.
3.
BursteinL. (1983). A word about this issue [Editor’s note]. Journal of Educational Measurement, 20, 99-102.
4.
BursteinL. (1989). Conceptual considerations in instructionally sensitive assessment (CSE Technical Report 333). Los Angeles, CA: Center for Research on Evaluation, Standards, and Student Testing.
5.
CantrellS. M. (2012). The Measures of Effective Teaching Project: An experiment to build evidence and trust. Education Finance and Policy, 7, 203-218. doi:10.1162/EDFP_a_00062
6.
CasabiancaJ. M.LockwoodJ. R.McCaffreyD. F. (2015). Trends in classroom observation scores. Educational and Psychological Measurement, 75, 311-337.
7.
CorrentiR.MartinezJ. F. (2012). Conceptual, methodological, and policy issues in the study of teaching: Implications for improving instructional practice at scale. Educational Assessment, 17, 51-61.
8.
CoxR. C.VargasJ. S. (1966). A comparison of item selection techniques for norm-referenced and criterion-referenced tests. Paper presented at the annual meeting of the National Council on Measurement in Education, Chicago, IL.
9.
D’AgostinoJ. V.WelshM. E.CorsonN. M. (2007). Instructional sensitivity of a state’s standards-based assessment. Educational Assessment, 12, 1-22.
10.
DanielsonC. (2011). The framework for teaching evaluation instrument. Princeton, NJ: Danielson Group.
11.
DerryS. J.PeaR. J.BarronB.EngleR. A.EricksonF.GoldmanR.… SherinB. L. (2010). Conducting video research in the learning sciences: Guidance on selection, analysis, technology, and ethics. Journal of the Learning Sciences, 19, 3-53.
12.
FlodenR. E. (2002). The measurement of opportunity to learn. In PorterA. C.GamoranA. (Eds.), Methodological advances in cross-national surveys of educational achievement (pp. 231-267). Washington, DC: National Research Council.
13.
GrossmanP.CohenJ.RonfeldtM.BrownL. (2014). The test matters: The relationship between classroom observation scores and teacher value added on multiple types of assessment. Educational Researcher, 43, 293-303.
14.
HaertelE. (2013). Getting the help we need. Journal of Educational Measurement, 50, 84-90.
15.
HaladynaT. M.RoidG. (1981). The role of instructional sensitivity in the empirical review of criterion-referenced test items. Journal of Educational Measurement, 18, 39-53.
16.
HannaG. S.BennettJ. A. (1984). Instructional sensitivity expanded. Educational and Psychological Measurement, 44, 583-596.
17.
HansonR. A.McMorrisR. F.BaileyJ. D. (1986). Differences in instructional sensitivity between item formats and between achievement test items. Journal of Educational Measurement, 23, 1-12.
18.
HarnischD. L. (1983). Item response patterns: Applications for educational practice. Journal of Educational Measurement, 20, 191-206.
19.
HillH. C.BlunkM. L.CharalambousC. Y.LewisJ. M.PhelpsG. C.SleepL.BallD. L. (2008). Mathematical knowledge for teaching and the mathematical quality of instruction: An exploratory study. Cognition and Instruction, 26, 430-511.
20.
HillH. C.GrossmanP. (2013). Learning from teacher observations: Challenges and opportunities posed by new teacher evaluation systems. Harvard Educational Review, 83, 371-394.
21.
HoA. D.KaneT. J. (2013). The reliability of classroom observations by school personnel. Seattle, WA: Bill and Melinda Gates Foundation.
22.
IngM. (2016). Initial considerations when applying an instructional sensitivity framework: Partitioning the variation between and within classrooms for two mathematics assessments. Applied Measurement in Education, 29, 122-131.
23.
IngM.WebbN. M. (2012). Characterizing mathematics classroom practice: Impact of observation and coding choices. Educational Measurement: Issues and Practice, 31, 14-26.
24.
Inter-University Consortium for Political and Social Research. (2013). Year 1 section-level analytical file 4th-8th grade codebook (ICPSR 34309-0001). Ann Arbor, MI: Author.
25.
KaneM. T. (2013a). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50, 1-73.
26.
KaneM. T. (2013b). Validation as a pragmatic, scientific activity. Journal of Educational Measurement, 50, 115-122.
27.
KaneT. J.KerrK. A.PiantaR. C. (Eds.). (2014). Designing teacher evaluation systems: New guidance from the Measures of Effective Teaching Project. San Francisco, CA: Jossey-Bass.
28.
KennedyM. M. (1999). Approximations to indicators of student outcomes. Educational Evaluation and Policy Analysis, 21, 345-363.
29.
Learning Mathematics for Teaching. (2011). Measuring the mathematical quality of instruction. Journal of Mathematics Teacher Education, 14, 25-47.
30.
LinnR. L. (1983). Testing and instruction: Links and distinctions. Journal of Educational Measurement, 20, 179-189.
31.
MehrensW. A.PhillipsS. E. (1986). Detecting impacts of curricular differences in achievement test data. Journal of Educational Measurement, 23, 185-196.
32.
MehrensW. A.PhillipsS. E. (1987). Sensitivity of item difficulties to curricular validity. Journal of Educational Measurement, 24, 357-370.
33.
MessickS. (1989). Validity. In LinnR. (Ed.), Educational measurement (3rd ed., pp. 13-103). Washington, DC: American Council on Education/Macmillan.
34.
MillerM. D.LinnR. L. (1988). Invariance of item characteristic functions with variations in instructional coverage. Journal of Educational Measurement, 25, 205-219.
35.
MuthénB. O. (1989). Using item-specific instructional information in achievement modeling. Psychometrika, 54, 385-396.
36.
MuthénB. O. (1994). Instructionally sensitive psychometrics: Applications to the Second International Mathematics Study. In WestburyI.EthingtonC. A.SosniakL. A.BakerD. P. (Eds.), In search of more effective mathematics instruction (pp. 293-324). Norwood, NJ: Ablex.
37.
MuthénB. O.HuangL. C.KhooS. K.GoffG. H.NovakJ. R.ShinJ. C. (1995). Opportunity-to-learn effects on achievement: Analytical aspects. Educational Evaluation and Policy Analysis, 17, 371-403.
38.
MuthénB. O.KaoC. F.BursteinL. (1991). Instructionally sensitive psychometrics: Application of a new IRT-based detection technique to mathematics achievement test items. Journal of Educational Measurement, 28, 1-22.
39.
NaumannA.HochweberJ.KliemeE. (2016). A psychometric framework for the evaluation of instructional sensitivity. Educational Assessment, 21, 89-101.
40.
PhillipsS. E.MehrensW. A. (1987). Curricular differences and unidimensionality of achievement test data: An exploratory analysis. Journal of Educational Measurement, 24, 1-16.
41.
PiantaR. C.HamreB. K. (2009). Conceptualization, measurement, and improvement of classroom processes: Standardized observation can leverage capacity. Educational Researcher, 38, 109-119.
42.
PiantaR. C.HamreB.HayesN.MintzS.LaParoK. M. (2008). Classroom Assessment Scoring System–Secondary (CLASS-S). Charlottesville: University of Virginia.
43.
PolikoffM. S. (2010). Instructional sensitivity as a psychometric property of assessments. Educational Measurement: Issue and Practice, 29(4), 3-14.
44.
PolikoffM. S. (2016). Evaluating the instructional sensitivity of four states’ student achievement tests. Educational Assessment, 21, 102-119.
45.
PophamW. J. (1971). Indices of adequacy for criterion-referenced test items. In PophamW. J. (Ed.), Criterion-referenced measurement: An introduction (pp. 79-98). Englewood Cliffs, NJ: Educational Technology.
46.
PophamW. J. (2007). Instructional sensitivity of tests: Accountability’s dire drawback. Phi Delta Kappan, 89, 146-150.
47.
RaudenbushS. W.BrykA. S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Thousand Oaks, CA: Sage.
48.
Ruiz-PrimoM. A.LiM.WillsK.GiamellaroM.LanM. C.MasonH.SandsD. (2012). Developing and evaluating instructionally sensitive assessments in science. Journal of Research in Science Teaching, 49, 691-712.
49.
Ruiz-PrimoM. A.ShavelsonR. J.HamiltonL. S.KleinS. (2002). On the evaluation of systematic science education reform: Searching for instructional sensitivity. Journal of Research in Science Teaching, 39, 369-393.
50.
SatoT. (1975). The construction and interpretation of S-P tables. Tokyo, Japan: Meiji Tosho.
51.
SchlesingerL.JentschA. (2016). Theoretical and methodological challenges in measuring instructional quality in mathematics education using classroom observations. ZDM, 48, 29-40.
52.
SchoenfeldA. H. (2013). Classroom observations in theory and practice. ZDM Mathematics Education, 45, 607-621.
53.
ShavelsonR. J.WebbN. M.BursteinL, (1986). The measurement of teaching. In WittrockM. (Ed.), Handbook of research on teaching (3rd ed., pp. 50-91). New York, NY: Macmillan.
54.
SteinM. K.KaufmanJ. (2010). Selecting and supporting the use of mathematics curricula at scale. American Educational Research Journal, 47, 663-693.
55.
WeissI. R.PasleyJ. D.SmithP. S.BanilowerE. R.HeckD. J. (2003). Looking inside the classroom: A study of K-12 mathematics and science education in the United States. Chapel Hill, NC: Horizon Research.
56.
WhiteM.RowanB. (2013). User guide to measure of effective teaching longitudinal database. Ann Arbor: Inter-University Consortium for Political and Social Research, University of Michigan.
