Evaluating the Psychometric Qualities of a Rating Scale to Assess Pre-Service Teachers’ Lesson Plan Development in the Context of a Secondary-Level Music Performance Classroom

Abstract

The purpose of this study was to evaluate the psychometric quality (i.e., validity and reliability) of a rating scale to assess pre-service teachers’ lesson plan development in the context of secondary-level music performance classrooms. The research questions that guided this study include: (1) What items demonstrate acceptable model fit for the construct of lesson plan development in the context of a secondary-level music performance classroom? (2) How does the structure of the rating scale vary across items? and (3) Does differential severity emerge for academic administrators or music education content specialists across items? Using multiple teacher effectiveness frameworks, the lesson plans in this study were evaluated using a 4-point Likert-type rating scale (e.g., strongly agree, agree, disagree, strongly disagree) consisting of five domains: (a) instructional planning, (b) instructional delivery, (c) differentiated instruction, (d) assessment uses, and (e) assessment strategies. Secondary-level school administrators (n = 8) and music education content specialists (n = 8) rated 32 lesson plans using a balanced incomplete assessment network. The multifaceted Rasch measurement partial credit model was used in this study. Results suggest higher rater severity among administrators than music specialists. Of the 68 potential pairwise interactions examined in the study, 5 (7.4 %) of those were found to be statistically significant, which indicates that 5 raters demonstrated differential severity across at least one lesson plan. Implications for student teacher preparation, teacher effectiveness, and the validity of measures are discussed.

Keywords

lesson plan rating scale Rasch model reliability validity

Get full access to this article

View all access options for this article.

References

Akyuz

Dixon

J. K.

Stephan

(2013). Improving the quality of mathematics teaching with effective planning practices. Teacher Development, 17(1), 92–106. doi:10.1080/13664530.2012.753939

Baghaei

(2008). The Rasch model as a construct validation tool. Rasch Measurement Transactions, 22(1), 1145–1146.

Berg

M. H.

Miksza

(2010). An investigation of preservice music teacher development and concerns. Journal of Music Teacher Education, 20(1), 39–55. doi:10.1177/1057083710363237.

Bond

T. G.

Fox

C. M.

(2015). Applying the Rasch model: Fundamental measurement in the human sciences (3rd ed.). New York, NY: Taylor & Francis Group, Routledge.

Brittin

R. V.

(2005). Preservice and experienced teachers’ lesson plans for beginning instrumentalists. Journal of Research in Music Education, 53, 26–39. doi:10.1177/002242940505300103

Butler

(2001). Preservice music teachers’ conceptions of teaching effectiveness, microteaching experiences, and teaching performance. Journal of Research in Music Education, 49, 258–272. doi:10.2307/3345711

Butt

(2006). Lesson plan (2nd ed.). London: Continuum International Publishing Group.

The Center for Educator Effectiveness, McREL International. (2013). McREL’s research-based teacher evaluation system: The CUES framework. Denver, CO: McREL International.

Chaffin

C. R.

(2009). Perceptions of instrumental music teachers regarding the development of effective rehearsals. Bulletin of the Council for Research in Music Education, 181, 21–36.

10.

Conway

(2002a). Curriculum writing in music. Music Educators Journal, 88(6), 54–59. doi:10.2307/3399806

11.

Conway

(2002b). Perceptions of beginning teachers, their mentors, and administrators regarding preservice music teacher preparation. Journal of Research in Music Education, 50, 20–36. doi:10.2307/3345690

12.

Coppola

A. J.

Scricca

D. B.

Connors

G. E.

(2004). Supportive supervision: Becoming a teacher of teachers. Thousand Oaks, CA: Corwin Press.

13.

Council for the Accreditation of Educator Preparation. (2016). 2013 CAEP standards. Retrieved from http://caepnet.org/standards/introduction

14.

Danielson

(2007). Enhancing professional practice: A framework for teaching (2nd ed.). Alexandria, VA: Association for Supervision & Curriculum Development.

15.

Danielson

(2013). The framework for teaching: Evaluation instrument. Princeton, NJ: The Danielson Group.

16.

Dumas

(1999). Usability testing methods: Subjective measures, Part II—Measuring attitudes and opinions. Washington, DC: American Institutes for Research.

17.

Duncan

(2011). Our future, our teachers: The Obama administration’s plan for teacher education reform and improvement. Retrieved from https://www.ed.gov/sites/default/files/our-future-our-teachers.pdf

18.

edTPA, Annual Administrative Report. (2015). Educative assessment and meaningful support. Stanford, CA: Stanford Center for Assessment, Learning, and Equity.

19.

Engelhard

(1997). Constructing rater and task banks for performance assessments. Journal of Outcome Measurement, 1(1), 19–33.

20.

Engelhard

(2008). Differential rater functioning. Rasch Measurement Transactions, 21(3), 281–385.

21.

Engelhard

Jr. (2013). Invariant measurement: Using Rasch models in the social, behavioral, and health sciences. New York, NY: Psychology Press.

22.

Engelhard

Jr. Perkins

A. F.

(2011). Person response functions and the definition of units in the social sciences. Measurement: Interdisciplinary Research & Perspective, 9, 40–45. doi:10.1080/15366367.2011.558787

23.

Frey

Fisher

Moore

(2005). Designing responsive curriculum: Planning lessons that work. Lanham, MD: Rowman & Littlefield Education.

24.

Furr

R. M.

Bacharach

V. R.

(2007). Psychometrics: An introduction. Thousand Oaks, CA: SAGE Publications.

25.

Houston

Beech

(2002). Designing lessons for the diverse classroom: A handbook for teachers. Retrieved from http://www.fldoe.org/core/fileparse.php/7690/urlt/0070084-4dclessn.pdf

26.

Kilpatrick

Swafford

Findell

(2001). Adding it up: Helping children learn mathematics. Washington, DC: National Academy Press.

27.

Kirk

R. E.

(1995). Experimental design: Procedures for the behavioral sciences (3rd ed.). Pacific Grove, CA: Brooks/Cole.

28.

Lane

J. S.

(2006). Undergraduate instrumental music education majors’ approaches to score study in varying musical contexts. Journal of Research in Music Education, 54, 215–230. doi:10.1177/002242940605400305

29.

Lane

J. S.

Talbert

M. D.

(2015). Examining lesson plan use among instrumental music education majors during practice teaching. Journal of Music Teacher Education, 24(3), 83–96. doi:10.1177/1057083713514979

30.

Lehman

(2014). How are we doing? In Brophy

T. S.

Lai

M.-L.

Chen

H.-F.

(Eds.), Music assessment and global diversity: Practice, measurement and policy (pp. 3–17). Chicago, IL: GIA Publications, Incorporated.

31.

Linacre

J. M.

(1989). Many-facet Rasch measurement. Chicago, IL: Mesa Press.

32.

Linacre

J. M.

(1994). Sample size and item calibration stability. Rasch Measurement Transactions, 7(4), 328.

33.

Linacre

J. M.

(2002). Optimizing rating scale category effectiveness. Journal of Applied Measurement, 3, 86–106.

34.

Linacre

J. M.

(2014). Facets (Version 3.71.4) [Computer software]. Chicago, IL: MESA Press.

35.

Linacre

J. M.

(2017). Reliability and separation of measures. Retrieved from http://www.winsteps.com/winman/reliability.htm

36.

Linacre

J. M.

Wright

B. D.

(2004). Construction of measures from many-facet data. In Smith

E. V.

Smith

R. M.

(Eds.), Introduction to Rasch measurement: Theories, models, and applications (pp. 296–321). Maple Grove, MN: JAM Press.

37.

“Linking terminology: Raw score and Rasch.” (2007). Rasch Measurement Transactions, 20(4), 1076.

38.

Marzano Research Laboratory. (2013). The Marzano teacher evaluation model. Englewood, CO: Marzano Research Laboratory. Retrieved from http://tpep-wa.org/wp-content/uploads/Marzano_Teacher_Evaluation_Model.pdf

39.

Marzano

R. J.

(2003a). Classroom management that works. Alexandria, VA: Association for Supervision and Curriculum Development.

40.

Marzano

R. J.

(2003b). What works in schools: Translating research into action. Alexandria, VA: Association for Supervision and Curriculum Development.

41.

Marzano

R. J.

(2006). Classroom assessment and grading that work. Alexandria, VA: Association for Supervision and Curriculum Development.

42.

Marzano

R. J.

(2007). The art of science and teaching. Alexandria, VA: Association for Supervision and Curriculum Development.

43.

Marzano

R. J.

Frontier

Livingston

(2011). Effective supervision: Supporting the art and science of teaching. Alexandria, VA: Association for Supervision and Curriculum Development.

44.

Marzano

R. J.

Pickering

D. J.

Pollock

J. E.

(2001). Classroom instruction that works. Alexandria, VA: Association for Supervision and Curriculum Development.

45.

Masters

G. N.

(1982). A Rasch model for partial credit scoring. Psychometrika, 47, 149–174. doi:10.1007/BF02296272

46.

Miksza

Berg

M. H.

(2013). Transition from student to teacher: Frameworks for understanding pre-service music teacher development. Journal of Music Teacher Education, 23, 10–26. doi:10.1177/1057083713480888

47.

National Association of Schools of Music. (2016). National Association of Schools of Music handbook. Retrieved from https://nasm.arts-accredit.org/wp-content/uploads/sites/2/2015/11/NASM_HANDBOOK_2016-17.pdf

48.

National Education Association. (2011). New policy statement on teacher evaluation and accountability. Retrieved from http://www.nea.org/grants/46326.htm

49.

Nelson

J. A.

(2012). Effects of teacher evaluations on teacher effectiveness and student achievement (Unpublished master’s thesis). Northern Michigan University, Marquette, Michigan.

50.

Paul

S. J.

(1998). The effects of peer teaching experiences on the professional teacher role development of undergraduate instrumental music education majors. Bulletin of the Council for Research in Music Education, 137, 73–92.

51.

Schleuter

(1991). Student teachers’ preactive and postactive curricular thinking. Journal of Research in Music Education, 39, 48–65. doi:10.2307/3344608

52.

Schmidt

(2005). Preservice string teachers’ lesson-planning processes: An exploratory study. Journal of Research in Music Education, 53, 6–25. doi:10.1177/002242940505300102

53.

Shuler

S. C.

Brophy

T. S.

Sabol

F. R.

McGreevy-Nichols

Schuttler

M. J.

(2015). Arts assessment in an age of accountability: Challenges and opportunities in implementation, design, and measurement. In Braun

H. I.

(Ed.), Meeting the challenges to measurement in an era of accountability (pp. 183–216). New York, NY: Routledge, Taylor & Francis Group.

54.

Snyder

D. W.

(1998). Classroom management for student teachers. Music Educators Journal, 84(4), 37–40. doi:10.2307/3399115

55.

Stronge & Associates. (2013). Stronge teacher evaluation system: A validation report. Retrieved from http://www.cesa6.org/effectiveness_project/Validation-Report-of-Stronge-Evaluation-System.pdf

56.

Stronge

J. H.

Ward

(2013). Virginia teacher evaluation and Virginia Performance-Pay Incentives (VPPI) pilot: An evaluation report. Richmond, VA: Virginia Department of Education.

57.

Taylor

(2014). Integrating arts learning with the common core state standards. Retrieved from http://ccsesa.org/wp-content/uploads/2014/12/FINAL-Common-Core-Publication.compressed.pdf

58.

Teachout

D. J.

(1997). Preservice and experienced teachers’ opinions of skills and behaviors important to successful music teaching. Journal of Research in Music Education, 45, 41–50. doi:10.2307/3345464

59.

U.S. Department of Education. (2009). Race to the top program executive summary. Retrieved from https://www2.ed.gov/programs/racetothetop/executive-summary.pdf

60.

Virginia Department of Education. (2012). Guidelines for uniform performance standards and evaluation criteria for teacher. Retrieved from http://www.doe.virginia.gov/teaching/performance_evaluation/teacher/index.shtml

61.

Wesolowski

B. C.

(2014). Documenting student learning in music performance: A framework. Music Educators Journal, 101(1), 77–85.

62.

Wesolowski

B. C.

(2015). Tracking student achievement in music performance: Developing student learning objectives for growth model assessments. Music Educators Journal, 102(1), 39–47.

63.

Wesolowski

B. C.

Wind

S. A.

Engelhard

(2016). Examining rater precision in music performance assessment: An analysis of rating scale structure using the multifaceted Rasch partial credit model. Music Perception, 33, 662–678.

64.

Wind

S. A.

Engelhard

Jr. Wesolowski

B. C.

(2016). Exploring the effects of rating designs and rater fit on achievement estimates within the context of music performance assessment. Educational Assessment, 21, 278–299.

65.

Woods

(2015). Teacher keys effectiveness system: Implementation handbook. Retrieved from http://www.gadoe.org/School-Improvement/Teacher-and-Leader-Effectiveness/Documents/TKES&20Handbook%20-713.pdf.

66.

Wright

B. D.

(1977). Solving measurement problems with the Rasch model. Journal of Educational Measurement, 14, 97–116. doi:10.1111/j.1745-3984.1977.tb00031.x

67.

Wright

B. D.

Linacre

J. M.

(1994). Reasonable mean-square fit values. Rasch Measurement Transactions, 8(3), 370.

68.

Wright

B. D.

Stone

M. H.

(1979). Best test design. Chicago, IL: MESA Press.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.47 MB

0.00 MB