Restricted accessResearch articleFirst published online 2022-12
Developing Situated Measures of Science Instruction Through an Innovative Electronic Portfolio App for Mobile Devices: Reliability,Validity,and Feasibility
Adoption of new instructional standards in science demands high-quality information about classroom practice. Teacher portfolios can be used to assess instructional practice and support teacher self-reflection anchored in authentic evidence from classrooms. This study investigated a new type of electronic portfolio tool that allows efficient capture of classroom artifacts in multimedia formats using mobile devices. We assess the psychometric properties of measures of quality instruction in middle school science classrooms derived from the contents of portfolios collected using this novel tool—with instruction operationalized through dimensions aligned to the Next Generation Science Standards. Results reflect low rater error and adequate reliability for several dimensions, a dominant underlying factor, and significant relations to some relevant concurrent indicators. Although no relation was found to student standardized test scores or course grades, portfolio ratings did relate to student self-efficacy perceptions and enjoyment of science. We examine factors influencing measurement error, and consider the broader implications of the results for assessing the validity of portfolio score interpretations, and the feasibility and potential value of this type of tool for summative and formative uses, in the context of large-scale instructional improvement efforts.
AllenC. D.PenuelW. R. (2015). Studying teachers’ sensemaking to investigate teachers’ responses to professional development focused on new standards. Journal of Teacher Education, 66(2), 136–149.
2.
BallD. L.ForzaniF. M. (2009). The work of teaching and the challenge for teacher education. Journal of Teacher Education, 60(5), 497–511.
3.
BorkoH.StecherB. M.MartinezJ. F.KuffnerK. L.BarnesD.ArnoldS. C. (2006). Using classroom artifacts to measure instructional practices in middle school science: A two-state field test (No. 690). Center for Research on Evaluation, Standards and Student Testing, University of California.
4.
CohenJ.GoldhaberD. (2016). Building a more complete understanding of teacher evaluation using classroom observations. Educational Researcher, 45(6), 378–387.
5.
CorrentiR.MartinezJ. F. (2012). Conceptual and methodological issues in the study of teaching: Mission statement for improving instruction at scale. Educational Assessment, 17, 2–3.
6.
Darling-HammondL.HylerM. E.GardnerM. (2017). Effective teacher professional development. Learning Policy Institute.
7.
DesimoneL.LeFlochK. (2004). Are we asking the right questions? Using cognitive interviews to improve surveys in education research. Educational Evaluation and Policy Analysis, 26(1), 1–22.
8.
GitomerD. H. (2009). Measurement issues and assessment for teaching quality. SAGE.
9.
GrossmanP.McDonaldM. (2008). Back to the future: Directions for research in teaching and teacher education. American Educational Research Journal, 45(1), 184–205.
10.
HushmanC.MarleyC. (2015). Guided instruction improves elementary student learning and self-efficacy in science. The Journal of Educational Research, 108(5), 371–381.
11.
IngM. (2018). What about the “instruction” in instructional sensitivity? Validity issues in research on instructional sensitivity. Educational and Psychological Measurement, 78(4), 635–652.
12.
IngM.ChinenS.JacksonK.SmithT. (2020). When should i use a measure to support instructional improvement at scale? The importance of considering both intended and actual use in validity arguments. Educational Measurement: Issues and Practice, 40(1), 92–100. https://doi.org/10.1111/emip.12393
13.
KaneM. (2006). Validation. In BrennanR. L. (Ed.), Educational measurement (4th ed., pp. 17–64). American Council on Education and Praeger.
14.
KaneT. J.StaigerD. O. (2012). Gathering feedback for teaching: Combining high-quality observations with student surveys and achievement gains. Bill & Melinda Gates Foundation. http://www.metproject.org/reports.php
15.
KennedyM. M. (1999). Approximations to indicators of student outcomes. Educational Evaluation and Policy Analysis, 21, 345–363.
16.
KerstingN. B.SherinB.StiglerJ. W. (2014). Automated Scoring of Teachers’ OpenEnded Responses to Video Prompts: Bringing the Classroom Video Analysis (CVA) Assessment to Scale. Educational & Psychological Measurement, 74(6), 950–974.
17.
KloserM.EdelmanA.FloydC.MartinezJ.F.StecherB.SrinivasanJ. (2021) Interrogating practice or show and tell?: Using a digital portfolio to anchor a professional learning community of science teachers. Journal of Science Teacher Education, 31(2) 210–241.
18.
KnappM. S. (1997). Between systemic reforms and the mathematics and science classroom: The dynamics of innovation, implementation, and professional learning. Review of Educational Research, 67(2), 227–266.
19.
KrajcikJ.CodereS.DahsahC.BayerR.MunK. (2014). Planning instruction to meet the intent of the next generation science standards. Journal of Science Teacher Education, 25(2), 157–175.
20.
KuhnD. (2010). Teaching and learning science as argument. Science Education, 94(5), 810–824.
21.
LatourB. (1986). Laboratory life: The construction of scientific facts. Princeton University Press.
22.
LemkeJ. L. (1990). Talking science: Language, learning, and values (Vol. 1). Ablex Publishing Corporation.
23.
MartinezJ. FBorkoH.StecherB.LuskinR.KloserM. (2012). Measuring classroom assessment practice using instructional artifacts: A validation study of the QAS notebook. Educational Assessment, 17(2-3), 107–131.
MayerD. (1999). Measuring instructional practice: Can policy makers trust survey data?Educational Evaluation and Policy Analysis, 21(1), 29–45.
26.
MedleyD. M.MitzelH. E. (1963). Measuring classroom behavior by systematic observation. In GageN. L. (Ed.), Handbook of research on teaching (pp. 247–328). Rand McNally.
27.
MeyerJ. P.CashA. H.MashburnA. (2011). Occasions and the reliability of classroom observations: Alternative conceptualizations and methods of analysis. Educational Assessment, 16(4), 227–243. https://doi.org/10.1080/10627197.2011.638884
28.
MuthénB. O.MuthénL. K. (2012). Mplus (Version 7.0) [Computer software].
National Research Council. (2012). A framework for K-12 science education: Practices, crosscutting concepts, and core ideas. The National Academies Press.
31.
PodolskyA.KiniT.Darling-HammondL. (2019). Does teaching experience increase teacher effectiveness? A review of US research. Journal of Professional Capital and Community, 4(4), 286–308.
32.
PolikoffM. S.GasparianH.KornS.GamboaM.PorterA. C.SmithT.GaretM. S. (2020). Flexibly using the surveys of enacted curriculum to study alignment. Educational Measurement: Issues and Practice, 39, 38–47. https://doi.org/10.1111/emip.12292
33.
PollioM.HochbeinC. D. (2015). The association between standards-based grading and standardized test scores as an element of a high school reform model. Teachers College Record: The Voice of Scholarship in Education, 117, 1–28.
34.
RaudenbushS. W.BrykA. S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). SAGE.
35.
RaudenbushS. W.SadoffS. (2008). Statistical inference when classroom quality is measured with error. Journal of Research on Educational Effectiveness, 1, 138–154.
36.
RockoffJ. E. (2004). The impact of individual teachers on student achievement: Evidence from panel data. American Economic Review, 94(2), 247–252.
37.
RowanB.CorrentiR. (2009). Studying reading instruction with teacher logs: Lessons from a study of instructional improvement. Educational Researcher, 38, 120–131.
SchunkD. H. (1989). Self-efficacy and achievement behaviors. Educational Psychology Review, 1, 173–208.
40.
SchweigJ. D. (2014). Quantifying error in survey measures of school and classroom environments. Applied Measurement in Education, 27(2), 133–157.
41.
ScruggsT. E.BrighamF. J.MastropieriM. A. (2013). Common core science standards: Implications for students with learning disabilities. Learning Disabilities Research & Practice, 28(1), 49–57.
42.
SeidelT.ShavelsonR. J. (2007). Teaching effectiveness research in the past decade: The role of theory and research design in disentangling meta-analysis results. Review of Educational Research, 77(4), 454–499.
43.
ShavelsonR. J.WebbN. M. (1991). Generalizability theory: A primer. SAGE.
44.
ShulmanL. (1998). Teacher portfolios: A theoretical activity. In LyonsN. (Ed.), With Portfolio in Hand (pp. 23–37). Teachers College Press.
45.
SmithP. S. (2010, March21–24). New tools for investigating the relationship between teacher content knowledge and student learning [Paper presentation]. Presented at the 2010 NARST Annual Conference, Philadelphia, PA, United States.
46.
StecherB.LeV.N.HamiltonL.RyanG.RobynA.LockwoodJ. R. (2006). Using structured classroom vignettes to measure instructional practices in mathematics. Educational Evaluation and Policy Analysis, 28(2), 101–130.
47.
StefaniL.MasonR.PeglerC. (2007). The educational potential of e-portfolios. Routledge.
48.
StrudlerN.WetzelK. (2011). Electronic portfolios in teacher education: Forging a middle ground. Journal of Research on Technology in Education, 44(2), 161–173.
49.
TangmunarunkitH.HsiehC. K.LongstaffB.NolenS.JenkinsJ.KetchamC.. . . EstrinD. (2015). Ohmage: A general and extensible end-to-end participatory sensing platform. ACM Transactions on Intelligent Systems and Technology (TIST), 6(3), Article 38.
WindschitlM.ThompsonJ.BraatenM.StroupeD. (2012). Proposing a core set of instructional practices and tools for teachers of science. Science Education, 96(5), 878–903.
53.
ZeeM.KoomenH. (2016). Teacher self-efficacy and its effects on classroom processes, student academic adjustment, and teacher well-being: A synthesis of 40 years of research. Review of Educational Research, 86(4), 981–1015.
54.
ZeichnerK. (2012). The turn once again toward practice-based teacher education. Journal of Teacher Education, 63(5), 376–382.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
