Developing Situated Measures of Science Instruction Through an Innovative Electronic Portfolio App for Mobile Devices: Reliability,Validity,and Feasibility

Abstract

Adoption of new instructional standards in science demands high-quality information about classroom practice. Teacher portfolios can be used to assess instructional practice and support teacher self-reflection anchored in authentic evidence from classrooms. This study investigated a new type of electronic portfolio tool that allows efficient capture of classroom artifacts in multimedia formats using mobile devices. We assess the psychometric properties of measures of quality instruction in middle school science classrooms derived from the contents of portfolios collected using this novel tool—with instruction operationalized through dimensions aligned to the Next Generation Science Standards. Results reflect low rater error and adequate reliability for several dimensions, a dominant underlying factor, and significant relations to some relevant concurrent indicators. Although no relation was found to student standardized test scores or course grades, portfolio ratings did relate to student self-efficacy perceptions and enjoyment of science. We examine factors influencing measurement error, and consider the broader implications of the results for assessing the validity of portfolio score interpretations, and the feasibility and potential value of this type of tool for summative and formative uses, in the context of large-scale instructional improvement efforts.

Keywords

generalizability theory measures of instruction performance assessment portfolio assessment validity

Get full access to this article

View all access options for this article.

References

Allen

C. D.

Penuel

W. R.

(2015). Studying teachers’ sensemaking to investigate teachers’ responses to professional development focused on new standards. Journal of Teacher Education, 66(2), 136–149.

Ball

D. L.

Forzani

F. M.

(2009). The work of teaching and the challenge for teacher education. Journal of Teacher Education, 60(5), 497–511.

Borko

Stecher

B. M.

Martinez

J. F.

Kuffner

K. L.

Barnes

Arnold

S. C.

(2006). Using classroom artifacts to measure instructional practices in middle school science: A two-state field test (No. 690). Center for Research on Evaluation, Standards and Student Testing, University of California.

Cohen

Goldhaber

(2016). Building a more complete understanding of teacher evaluation using classroom observations. Educational Researcher, 45(6), 378–387.

Correnti

Martinez

J. F.

(2012). Conceptual and methodological issues in the study of teaching: Mission statement for improving instruction at scale. Educational Assessment, 17, 2–3.

Darling-Hammond

Hyler

M. E.

Gardner

(2017). Effective teacher professional development. Learning Policy Institute.

Desimone

LeFloch

(2004). Are we asking the right questions? Using cognitive interviews to improve surveys in education research. Educational Evaluation and Policy Analysis, 26(1), 1–22.

Gitomer

D. H.

(2009). Measurement issues and assessment for teaching quality. SAGE.

Grossman

McDonald

(2008). Back to the future: Directions for research in teaching and teacher education. American Educational Research Journal, 45(1), 184–205.

10.

Hushman

Marley

(2015). Guided instruction improves elementary student learning and self-efficacy in science. The Journal of Educational Research, 108(5), 371–381.

11.

Ing

(2018). What about the “instruction” in instructional sensitivity? Validity issues in research on instructional sensitivity. Educational and Psychological Measurement, 78(4), 635–652.

12.

Ing

Chinen

Jackson

Smith

(2020). When should i use a measure to support instructional improvement at scale? The importance of considering both intended and actual use in validity arguments. Educational Measurement: Issues and Practice, 40(1), 92–100. https://doi.org/10.1111/emip.12393

13.

Kane

(2006). Validation. In Brennan

R. L.

(Ed.), Educational measurement (4th ed., pp. 17–64). American Council on Education and Praeger.

14.

Kane

T. J.

Staiger

D. O.

(2012). Gathering feedback for teaching: Combining high-quality observations with student surveys and achievement gains. Bill & Melinda Gates Foundation. http://www.metproject.org/reports.php

15.

Kennedy

M. M.

(1999). Approximations to indicators of student outcomes. Educational Evaluation and Policy Analysis, 21, 345–363.

16.

Kersting

N. B.

Sherin

Stigler

J. W.

(2014). Automated Scoring of Teachers’ OpenEnded Responses to Video Prompts: Bringing the Classroom Video Analysis (CVA) Assessment to Scale. Educational & Psychological Measurement, 74(6), 950–974.

17.

Kloser

Edelman

Floyd

Martinez

J.F.

Stecher

Srinivasan

(2021) Interrogating practice or show and tell?: Using a digital portfolio to anchor a professional learning community of science teachers. Journal of Science Teacher Education, 31(2) 210–241.

18.

Knapp

M. S.

(1997). Between systemic reforms and the mathematics and science classroom: The dynamics of innovation, implementation, and professional learning. Review of Educational Research, 67(2), 227–266.

19.

Krajcik

Codere

Dahsah

Bayer

Mun

(2014). Planning instruction to meet the intent of the next generation science standards. Journal of Science Teacher Education, 25(2), 157–175.

20.

Kuhn

(2010). Teaching and learning science as argument. Science Education, 94(5), 810–824.

21.

Latour

(1986). Laboratory life: The construction of scientific facts. Princeton University Press.

22.

Lemke

J. L.

(1990). Talking science: Language, learning, and values (Vol. 1). Ablex Publishing Corporation.

23.

Martinez

J. F

Borko

Stecher

Luskin

Kloser

(2012). Measuring classroom assessment practice using instructional artifacts: A validation study of the QAS notebook. Educational Assessment, 17(2-3), 107–131.

24.

Matsumura

L. C.

Garnier

H. E.

Slater

S. C.

Boston

M. D.

(2008). Toward measuring instructional interactions “at-scale.” Educational Assessment, 13(4), 267–300.

25.

Mayer

(1999). Measuring instructional practice: Can policy makers trust survey data? Educational Evaluation and Policy Analysis, 21(1), 29–45.

26.

Medley

D. M.

Mitzel

H. E.

(1963). Measuring classroom behavior by systematic observation. In Gage

N. L.

(Ed.), Handbook of research on teaching (pp. 247–328). Rand McNally.

27.

Meyer

J. P.

Cash

A. H.

Mashburn

(2011). Occasions and the reliability of classroom observations: Alternative conceptualizations and methods of analysis. Educational Assessment, 16(4), 227–243. https://doi.org/10.1080/10627197.2011.638884

28.

Muthén

B. O.

Muthén

L. K.

(2012). Mplus (Version 7.0) [Computer software].

29.

National Board for Professional Teaching Standards. (2019). Scoring guide: Understanding your scores. https://www.nbpts.org/wp-content/uploads/2021/05/NBPTS_Scoring_Guide.pdf

30.

National Research Council. (2012). A framework for K-12 science education: Practices, crosscutting concepts, and core ideas. The National Academies Press.

31.

Podolsky

Kini

Darling-Hammond

(2019). Does teaching experience increase teacher effectiveness? A review of US research. Journal of Professional Capital and Community, 4(4), 286–308.

32.

Polikoff

M. S.

Gasparian

Korn

Gamboa

Porter

A. C.

Smith

Garet

M. S.

(2020). Flexibly using the surveys of enacted curriculum to study alignment. Educational Measurement: Issues and Practice, 39, 38–47. https://doi.org/10.1111/emip.12292

33.

Pollio

Hochbein

C. D.

(2015). The association between standards-based grading and standardized test scores as an element of a high school reform model. Teachers College Record: The Voice of Scholarship in Education, 117, 1–28.

34.

Raudenbush

S. W.

Bryk

A. S.

(2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). SAGE.

35.

Raudenbush

S. W.

Sadoff

(2008). Statistical inference when classroom quality is measured with error. Journal of Research on Educational Effectiveness, 1, 138–154.

36.

Rockoff

J. E.

(2004). The impact of individual teachers on student achievement: Evidence from panel data. American Economic Review, 94(2), 247–252.

37.

Rowan

Correnti

(2009). Studying reading instruction with teacher logs: Lessons from a study of instructional improvement. Educational Researcher, 38, 120–131.

38.

Stanford Center for Assessment, Learning, and Equity. (2018). Educative assessment & meaningful support: 2017 edTPA administrative report. http://edtpa.aacte.org/news-area/2017-edtpa-annual-administrative-report.html

39.

Schunk

D. H.

(1989). Self-efficacy and achievement behaviors. Educational Psychology Review, 1, 173–208.

40.

Schweig

J. D.

(2014). Quantifying error in survey measures of school and classroom environments. Applied Measurement in Education, 27(2), 133–157.

41.

Scruggs

T. E.

Brigham

F. J.

Mastropieri

M. A.

(2013). Common core science standards: Implications for students with learning disabilities. Learning Disabilities Research & Practice, 28(1), 49–57.

42.

Seidel

Shavelson

R. J.

(2007). Teaching effectiveness research in the past decade: The role of theory and research design in disentangling meta-analysis results. Review of Educational Research, 77(4), 454–499.

43.

Shavelson

R. J.

Webb

N. M.

(1991). Generalizability theory: A primer. SAGE.

44.

Shulman

(1998). Teacher portfolios: A theoretical activity. In Lyons

(Ed.), With Portfolio in Hand (pp. 23–37). Teachers College Press.

45.

Smith

P. S.

(2010, March 21–24). New tools for investigating the relationship between teacher content knowledge and student learning [Paper presentation]. Presented at the 2010 NARST Annual Conference, Philadelphia, PA, United States.

46.

Stecher

V.N.

Hamilton

Ryan

Robyn

Lockwood

J. R.

(2006). Using structured classroom vignettes to measure instructional practices in mathematics. Educational Evaluation and Policy Analysis, 28(2), 101–130.

47.

Stefani

Mason

Pegler

(2007). The educational potential of e-portfolios. Routledge.

48.

Strudler

Wetzel

(2011). Electronic portfolios in teacher education: Forging a middle ground. Journal of Research on Technology in Education, 44(2), 161–173.

49.

Tangmunarunkit

Hsieh

C. K.

Longstaff

Nolen

Jenkins

Ketcham

. . . Estrin

(2015). Ohmage: A general and extensible end-to-end participatory sensing platform. ACM Transactions on Intelligent Systems and Technology (TIST), 6(3), Article 38.

50.

University of California, Los Angeles. (2018). e-QIS (3.10.27) [Mobile application software]. https://play.google.com/store/apps/details?id=org.ohmage.x.eqis

51.

William

Grant Foundation, Spencer Foundation, & Bill & Melinda Gates Foundation. (2015). Measuring instruction in higher education: Summary of a convening. https://wtgrantfoundation.org/library/uploads/2015/11/Measuring-Instruction-in-Higher-Education.pdf

52.

Windschitl

Thompson

Braaten

Stroupe

(2012). Proposing a core set of instructional practices and tools for teachers of science. Science Education, 96(5), 878–903.

53.

Zee

Koomen

(2016). Teacher self-efficacy and its effects on classroom processes, student academic adjustment, and teacher well-being: A synthesis of 40 years of research. Review of Educational Research, 86(4), 981–1015.

54.

Zeichner

(2012). The turn once again toward practice-based teacher education. Journal of Teacher Education, 63(5), 376–382.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.61 MB