The Development of a Secondary-Level Solo Wind Instrument Performance Rubric Using the Multifaceted Rasch Partial Credit Measurement Model

Abstract

The purpose of this study was to describe the development of a valid and reliable rubric to assess secondary-level solo instrumental music performance based on principles of invariant measurement. The research questions that guided this study included (1) What is the psychometric quality (i.e., validity, reliability, and precision) of a scale developed to assess secondary-level solo music performance? (2) Do the proposed items fit the measurement model, and if so, how do the items vary in difficulty? and (3) How does the structure of the rating scale vary across individual items? The psychometric considerations in this study included calibrations of items, persons, raters, school level, musical instrument, and rating scale structure using the Multifaceted Rasch Partial Credit Measurement Model. A 13-member cohort of music content experts participated as raters in this study. A total of 75 video performances of secondary-level solo and ensemble performances were evaluated. The result was the development of the Music Performance Rubric for Secondary-Level Instrumental Solos (MPR-2L-INSTSOLO), a 30-item rubric consisting of rating scale categories ranging from two to four performance criteria. Implications for consequential validity, rater training, standard setting, and benchmarking are discussed.

Keywords

assessment invariant measurement Many Facet Partial Credit Model Rasch rubric

Get full access to this article

View all access options for this article.

References

Abeles

H. F.

(1973). Development and validation of a clarinet performance adjudication scale. Journal of Research in Music Education, 21, 246–255.

Andrich

D. A.

(1996). Measurement criteria for choosing among models for graded responses. In von Eye

Clogg

C. C.

(Eds.), Analysis of categorical variables in developmental research (pp. 3–35). Orlando. FL: Academic Press.

Bergee

M. J.

(1987). An application of the facet-factorial approach to scale construction in the development of a rating scale for euphonium and tuba music performance. Dissertation Asbtracts International, 49(05), 1086A.

Bond

T. G.

Fox

C. M.

(2015). Applying the Rasch model: Fundamental measurement in the human sciences, 3rd edition. New York, NY: Routledge.

Buckly

Marion

(2011). A survey of approaches used to evaluate educators in non-tested grades and subjects. Retrieved from http://www.nciea.org/publication_PDFs/BuckleyMarion_SummaryofApproachesfornon-testedgrades.pdf

Christensen

K. B.

Engelhard

Jr. Salzberger

(2012). Ask the experts: Rasch vs. factor analysis. Rasch Measurement Transactions, 26, 1373–1378.

Cooksey

J. M.

(1977). A facet-factorial approach to rating high school choral music performance. Journal of Research in Music Education, 25, 100–114.

DCamp

C. B.

(1980). An application of the facet-factorial approach to scale construction in the development of a rating scale for high school band performance. Iowa City: University of Iowa.

Duke

R. A.

Byo

J. L.

(2007). The habits of musicianship: A radical approach to beginning band. Austin: University of Texas Center for Music Learning. Retrieved from papers3://publication/uuid/912917D1-6EF0-449F-A1B6-939466C40505

10.

Dumas

(1999). Usability testing methods: Subjective measures, part II - measuring attitudes and opinions. Washington, DC: American Institutes for Research.

11.

Engelhard

(1997). Constructing rater and task banks for performance assessments. Journal of Outcome Measurement, 1, 19–33.

12.

Engelhard

(2009). Using item response theory and model-data fit to conceptualize differential item and person functioning for students with disabilities. Educational and Psychological Measurement, 69, 585–602. doi:10.1177/0013164408323240

13.

Engelhard

(2013). Invariant measurement: Using Rasch models in the social, behavioral, and health sciences. New York, NY: Routledge.

14.

Georgia Department of Education . (2012). Overview to the 2012 TKES/LKES pilot evaluation report. Retrieved from https://www.gadoe.org/School-Improvement/TeacherandLeaderEffectiveness/Documents/Pilot%20Report_Overview%20and%20Report%20Combined%201-10-13.pdf

15.

Horowitz

R. A.

(1994). The development of a rating scale for jazz guitar improvisation performance (Doctoral thesis) Columbia University, Teachers College, New York, NY.

16.

Jones

(1986). An application of the facet-factorial appraoch to scale construction in the development of a rating scale for high school vocal solo performance. Dissertation Asbtracts International, 47(4), 1230.

17.

Kreiner

Christensen

K. B.

(2004). Analysis of local dependence and multidimensionality in graphical loglinear Rasch models. Communications in Statistics—Theory and Methods, 33, 1239–1276.

18.

Linacre

J. M.

(1989). Many facet Rasch measurement. Chicago, IL: MESA Press.

19.

Linacre

J. M.

(2002). Optimizing rating scale category effectiveness. Journal of Applied Measurement, 3, 85–106.

20.

Linacre

J. M.

(2014). Facets. Chicago, IL: MESA Press.

21.

Marton

(1986). Phenomenography: A research approach to investigating different understandings of reality. Journal of Thought, 21, 29–49.

22.

Masters

G. N.

(1982). A Rasch model for partial credit scoring. Psychometrika, 47, 149–174.

23.

Meredith

(1993). Measurement invariance, factor analysis and factorial invariance. Psychometrika, 58, 525–543.

24.

Messick

(1989). Validity. In Linn

R. L.

(Ed.), Educational measurement (3rd ed., pp. 13–103). New York, NY: Macmillian.

25.

Nichols

J. P.

(1991). A factor analysis approach to the development of a rating scale for snare drum performance. Dialogue in Instrumental Music Education, 15, 11–31.

26.

Pazitka-Munroe

W. L.

(2003). The development and validation of an audition instrument to measure vocal performance of college singers auditioning for choral ensembles. Dissertation Asbtracts International, 63(8), 2821.

27.

Pfenninger

R. C.

(1990). The development and validation of three rating scales for the objective measurement of jazz improvisation achievement. Dissertation Asbtracts International, 51(8), 2674A.

28.

Russell

B. E.

(2010). The development of a guitar performance rating scale using facet-factorial approach. Bulletin of the Council for Research in Music Education, 184, 21–34.

29.

Shuler

S. C.

Norgaard

Blakeslee

M. J.

(2014). The new national standards for music educators. Music Educators Journal, 101, 41–49. doi:10.1177/0027432114540120

30.

Smith

B. P.

Barnes

G. V.

(2007). Development and validation of an orchestra performance rating scale. Journal of Research in Music Education, 55, 268–280. doi:10.1177/002242940705500307

31.

Smith

D. T.

(2009). Development and validation of a rating scale for wind jazz improvisation performance. Journal of Research in Music Education, 57, 217–235. doi:10.1177/0022429409343549

32.

Steele

J. L.

Hamilton

L. S.

Stecher

B. M.

(2010). Incorporating student performance measures into teacher evaluation systems. Retrieved from https://cdn.americanprogress.org/wp-content/uploads/issues/2010/12/pdf/student_teacher_eval.pdf

33.

Union

I. T.

(2004). Objective perceptual assessment of video quality: Full reference television. ITU-T Telecommunication Standardization Bureau. Retrieved from http://scholar.google.com/scholar?hl=en&btnG=Search&q=intitle:Objective+perceptual+assessment+of+video+quality+:+Full+reference+television#6

34.

U.S. Department of Education. (2009). Race to the Top program executive summary. Retrieved from https://www2.ed.gov/programs/racetothetop/executive-summary.pdf

35.

Vagias

W. M.

(2006). Likert-type scale response anchors. Clemson, SC: Clemson University.

36.

van der Linden

W. J

. (1992). IRT in the 1990s—Which models work best. Retrieved from http://www.rasch.org/audio/IRT-van-der-Linden-2.mp3

37.

Wesolowski

B. C.

(2014). Documenting student learning in music performance: A framework. Music Educators Journal, 101, 77–85. doi:10.1177/0027432114540475

38.

Wesolowski

B. C.

(2015). Tracking student achievement in music performance: Developing student learning objectives for growth model assessments. Music Educators Journal, 102, 39–47. doi:10.1177/0027432115589352

39.

Wesolowski

B. C.

(2016). Assessing jazz big band performance: The development, validation, and application of a facet-factorial rating scale. Psychology of Music, 44, 324–339. doi:10.1177/0305735614567700

40.

Wesolowski

B. C.

Wind

S. A.

Engelhard

(2015). Rater fairness in music performance assessment: Evaluating model-data fit and differential rater functioning. Musicae Scientiae, 19, 147–170. doi:10.1177/1029864915589014

41.

Wesolowski

B. C.

Wind

S. A.

Engelhard

(2016a). Examining rater precision in music performance assessment: An analysis of rating scale structure using the multifaceted Rasch partial credit model. Music Perception, 5, 662–678.

42.

Wesolowski

B. C.

Wind

S. A.

Engelhard

(2016b). Rater analyses in music performance assessment: Application of the Many Facet Rasch Model. In Connecting practice, measurement, and evaluation: Selected papers from the 5th International Symposium on Assessment in Music Education (pp. 335–356). Chicago, IL: Gia Publications.

43.

Wilson

(2005). Constructing measures: An item response modeling approach. New York, NY: Taylor & Francis.

44.

Wraugh

R. F.

Chapman

E. S.

(2005). An analysis of dimensionality using factor analysis (true-score theory) and Rasch measurement: What is the difference? Which method is better? Journal of Applied Measurement, 6, 80–99.

45.

Wright

B. D.

(1977). Solving measurement problems with the Rasch model. Journal of Educational Measurement, 14(2), 97–116.

46.

Wright

B. D.

(1996). Comparing Rasch measurement and factor analysis. Structural Equation Modeling, 3(1), 3–24.

47.

Wright

B. D.

Douglas

G. A.

(1986). The rating scale model for objective measurement. Retrieved from http://www.rasch.org/memo35.pdf

48.

Wright

B. D.

Linacre

J. M.

(1994). Reasonable mean-square fit values. Rasch Measurement Transactions, 8, 370.

49.

Zdzinski

S. F.

Barnes

G. V.

(2002). Development and validation of a string performance rating scale. Journal of Research in Music Education, 50, 245–255.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.39 MB