Evaluation of music performance in competitive contexts often produces discrepancies between the expert judges. These discrepancies can be reduced by using appropriate rubrics that minimise the differences between judges. The objective of this study was the design and validation of an analytical evaluation rubric, which would allow the most objective evaluation possible of a musical solo performance in a regulated official competition. A panel of three experts created an analytical rubric made up of five review criteria and three scoring levels, together with their respective indicators. To validate the rubric, two independent panels of judges used it to score a sample of recordings. An examination was made of the dimensionality, sources of error, inter-rater reliability and internal consistency of the scores coming from the experts. The essential unidimensionality of the rubric was confirmed. No differential effects between raters were found, nor were significant differences seen in each rater’s internal consistency. The use of a rubric as tool for evaluating music performance in a competitive context has positive effects, improving reliability and objectivity of the results, both in terms of intra-rater consistency and agreement between raters.
American Psychological Association, American Educational Research Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. American Psychological Association.
4.
AzzaraC. D. (1993). The effect of audiation-based improvisation techniques on musical achievement of elementary music students. Journal of Research in Music Education, 41, 328–342.
5.
BairdJ. A.MeadowsM.LeckieG.CaroD. (2017). Rater accuracy and training group effects in expert- and supervisor-based monitoring systems. Assessment in Education: Principles, Policy and Practice, 24(1), 44–59. https://doi.org/10.1080/0969594X.2015.1108283
6.
BarnicleS. P. (1993). Evaluating the choral performer. CMEA News, 44(2), 24–26.
BergeeM. J. (1987). An investigation of the facet-factorial approach to scale construction in the development of a rating scale for euphonium and tuba music performance. Dissertation Abstracts International, 49, 1086.
9.
BergeeM. J. (1988). The use of an objectively constructed rating scale for the evaluation of brass juries: A criterion-related study. Missouri Journal of Research in Music Education, 5(5), 6–25.
10.
BergeeM. J. (1997). Relationships among faculty, peer, and self-evaluation of applied performances. Journal of Research in Music Education, 45, 601–612.
11.
BurnsedV.HinkleD.KingS. (1985). Performance evaluation reliability at selected concert festivals. Journal of Band Research, 21(1), 22–29.
12.
Calderón-GarridoC.Navarro-GonzálezD.Lorenzo-SevaU.Ferrando-PieraP. J. (2019). Multidimensional or essentially unidimensional? A multi-faceted factor-analytic approach for assessing the dimensionality of tests and items. Psicothema, 31, 450–457. https://doi.org/10.7334/psicothema2019.153
CongdonP. J.McQueenJ. (2000). The stability of rater severity in large-acale assessment programs. Journal of Educational Measurement, 37(2), 163–178. https://doi.org/10.1111/j.1745-3984.2000.tb01081
16.
CookseyJ. M. (1977). A facet-factorial approach to rating high school choral music performance. Journal of Research in Music Education, 25, 100–114.
17.
EckesT. (2009). Many-facet Rasch measurement. In TakalaS. (Ed.), Reference supplement to the manual for relating language examinations to the common European framework of reference for languages: Learning, teaching, assessment (section H) (pp. 2–54). Council of Europe/Language Policy.
Fernandez-AlonsoR.WoitschachP.MunizJ. (2019). Rubrics do not neutralize rater effects: A many-faceted Rasch model estimation. Revista de Educación, 386, 85–107. https://doi.org/10.4438/1988-592X-RE-2019-386-428
21.
FiskeH. E.Jr. (1975). Judge-group differences in the rating of secondary school trumpet performances. Journal of Research in Music Education, 23(3), 186–196.
22.
FiskeH. E.Jr. (1977). Relationship of selected factors in trumpet performance adjudication reliability. Journal of Research in Music Education, 25, 256–263.
23.
FlôresR. G.GinsburghV. A. (1996). The Queen Elisabeth musical competition: How fair is the final ranking?The Statistician, 45(1), 97–104.
24.
GordonE. (2002). Rating scales and their uses for evaluating achievement in music performance. Gregorian Institute of America.
25.
GwetK. L. (2014). Handbook of inter-rater reliability. The definitive guide to measuring the extent of agreement among raters. Advanced Analytics, LLC.
26.
GyagendaI.EngelhardG. (2009). Using classical and modern measurement theories to explore rater, domain, and gender influences on student writing ability. Journal of Applied Measurement, 10(3), 225–246.
27.
HernándezA.PonsodaV.MuñizJ.PrietoG.ElosuaP. (2016). Revisión del modelo para evaluar la calidad de los tests utilizados en España. [Assessing the quality of tests in Spain: Revision of the Spanish test review model]. Papeles del Psicólogo, 37, 192–197.
28.
HorowitzR. A. (1994). The development of a rating scale for jazz guitar improvisation performance. Dissertation Abstracts International, 11, 3443.
29.
JonesH.Jr. (1986). An application of a facet-factorial approach to scale construction in the development of a rating scale for high school solo vocal performance. Dissertation Abstracts International, 47, 1230.
30.
JonssonA.SvingbyG. (2007). The use of scoring rubrics: Reliability, validity and educational consequences. Educational Research Review, 2(2), 130–144.
31.
JuslinP. (2003). Five facets of musical expression: A psychologist’s perspective on music performance. Psychology of Music, 31(3), 273–302. https://doi.org/10.1177/03057356030313003
32.
KooT. K.LiM. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155–163. http://doi.org/10.1016/j.jcm.2016.02.012
33.
KuoS. A. (2007). Which rubric is more suitable for NSS liberal studies? Analytic or holistic?Educational Research Journal, 22(2), 179–199.
34.
LatimerM. E.Jr. (2007). Adjudicator reliability: A comparison of the use of state festival choir and global score audition forms. Contributions to Music Education, 34(2), 67–82.
35.
LatimerM. E.BergeeM. J.CohenM. L. (2010). Reliability and perceived pedagogical utility of a weighted music performance assessment rubric. Journal of Research in Music Education, 58(2), 168–183. https://doi.org/10.1177/0022429410369836
36.
LeckieG.BairdJ. (2011). Rater effects on essay scoring: A multilevel analysis of severity drift, central tendency, and rater experience. Journal of Educational Measurement, 48(4), 399–418. https://doi.org/10.1111/j.1745-3984.2011.00152.x
37.
LevinowitzL. M. (1989). An investigation of preschool children’s comparative capability to sing songs with and without words. Bulletin of the Council for Research in Music Education, 100, 14–19.
38.
LinacreJ. M.EngelhardG.TatumD. S.MyfordC. M. (1994). Measurement with judges: Many-faceted conjoint measurement. International Journal of Educational Research, 21(6), 569–577. https://doi.org/10.1016/0883-0355(94)90011-6
39.
LordF. M. (1980). Application of Item Response Theory to practical testing problems. Lawrence Erlbaum.
40.
LunzM. E.WrightB. D.LinacreJ. M. (1990). Measuring the impact of judge severity on examination scores. Applied Measurement in Education, 3(4), 331–345. https://doi.org/10.1207/s15324818ame0304_3
41.
McCormickL. (2009). Higher, faster, louder: Representations of the international music competition. Cultural Sociology, 3(1), 5–30.
42.
McGrawK. O.WongS. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1(1), 30–46.
43.
McNamaraT. F. (1996). Measuring second language performance. Longman.
MorenoR.MartínezR. J.MuñizJ. (2018). Test item taxonomy based on functional criteria. Frontiers in Psychology, 9, Article 1175. https://doi.org/10.3389/fpsyg.2018.01175
46.
MuñizJ.Fonseca-PedreroE. (2019). Diez pasos para la construcción de un test [Ten steps for test development]. Psicothema, 31, 7–16. https://doi.org/10.7334/psicothema2018.291
47.
NicholsJ. P. (1991). A factor analysis approach to the development of a rating scale for snare drum performance. Dissertation Abstracts International, 46, 3282.
48.
NitkoA. J. (2001). Educational assessment of students (3rd ed.). Pearson Merrill; Prentice Hall.
49.
ParkT. (2010). An investigation of an ESL placement test of writing using multi- faceted Rasch measurement (Vol. 4, Working Papers in TESOL and Applied Linguistics). Teachers College, Columbia University.
50.
ParkeC. S. (2001). An approach that examines sources of misfit to improve performance assessment items and rubrics. Educational Assessment, 7(3), 201–225.
PlatzF.KopiezR. (2012). When the eye listens: A meta-analysis of how audio-visual presentation enhances the appreciation of music performance. Music Perception: An Interdisciplinary Journal, 30(1), 71–83.
53.
PophamW. J. (1997). What’s wrong – And what’s right – With rubrics. Educational Leadership, 55(2), 72–75.
54.
PrietoG. (2011). Evaluación de la ejecución mediante el modelo many-facet Rasch measurement [Performance assessment using the Many-Facet Rasch Measurement]. Psicothema, 23(2), 233–238.
55.
ReddyM. Y.AndradeH. (2010). A review of rubric use in higher education. Assessment & Evaluation in Higher Education, 35(4), 435–448.
56.
RutkowskiJ. (1990). The measurement and evaluation of children’s singing voice development. Quarterly Journal of Music Teaching and Learning, 1(1–2), 81–95.
57.
SaalF. E.DowneyR. G.LaheyM. A. (1980). Rating the ratings: Assessing the psychometric quality of rating data. Psychological Bulletin, 88(2), 413–428.
58.
SaundersT. C.HolahanJ. M. (1997). Criteria specific rating scales in the evaluation of high school instrumental performance. Journal of Research in Music Education, 45, 259–272.
59.
ShroutP. E.FleissJ. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2), 420–428.
60.
SmithB. P.BarnesG. V. (2007). Development and validation of an orchestra performance rating scale. Journal of Research in Music Education, 55, 268–280.
61.
StemlerS. E. (2004). A comparison of consensus, consistency, and measurement approaches to estimating interrater reliability. Practical Assessment, Research & Evaluation, 9, Article 4. https://scholarworks.umass.edu/pare/vol9/iss1/4/
62.
Suárez-ÁlvarezJ.González-PrietoC.Fernández-AlonsoR.GilG.MuñizJ. (2014). Psychometric assessment of oral expression in English language in the university entrance examination. Revista de Educación, 364, 93–118. https://doi.org/10.4438/1988-592X-RE-2014-364-256
63.
SudweeksR. R.ReeveS.BradshawW. S. (2005). A comparison of generalizability theory and many-facet Rasch measurement in an analysis of college sophomore writing. Assessing Writing, 9, 239–261.
64.
ThompsonS.WilliamonA. (2003). Evaluating evaluation: Musical performance assessment as a research tool. Music Perception: An Interdisciplinary Journal, 21(1), 21–41.
65.
WaddellG.WilliamonA. (2017). Eye of the beholder: Stage entrance behavior and facial expression affect continuous quality ratings in music performance. Frontiers in Psychology, 8, Article 324. https://doi.org/10.3389/fpsyg.2017.00513
66.
WangZ.YaoL. (2013). The effects of rater severity and rater distribution on examinees’ ability estimation for constructed-response items. ETS Research Report Series, 2013(2), 1–22. https://doi.org/10.1002/j.2333-8504.2013.tb02330.x
67.
WesolowskiB. (2012). Understanding and developing rubrics for music performance assessment. Music Educators Journal, 98, 36–42. https://doi.org/10.1177/0027432111432524
68.
WesolowskiB. (2016). Exploring rater cognition: A typology of raters in the context of music performance assessment. Psychology of Music, 45(3), 375–399. https://doi.org/10.1177/0305735616665004
69.
WesolowskiB.WindS. A. (2017). Investigating rater accuracy in the context of secondary-level solo instrumental music. Musicae Scientae, 23(2), 157–176. https://doi.org/10.1177/1029864917713805
70.
WesolowskiB.WindS. A.EngelhardG. (2016). Examining rater precision in music performance assessment: An analysis of rating scale structure using the multifaceted Rasch partial credit model. Music Perception: An Interdisciplinary Journal, 33(5), 662–678. https://doi.org/10.1525/mp.2016.33.5.662
71.
WoitschachP.Díaz-PérezC.Fernández-ArgüellesD.Fernández-CastañónJ.Fernández-CastilloA.Fernández-RodríguezL.González-CanalM. C.López-MarquésI.Martín-EspinosaD.Navarro-CabreroR.Osendi-CadenasL.Riesgo-FernándezD.Suárez-GarcíaZ.Fernández-AlonsoR. (2018). Efectos del corrector en las evaluaciones educativas de alto impacto [Rater effects in high-impact educational assessments]. Revista Electrónica de Metodología Aplicada, 23(1), 12–27. https://doi.org/10.17811/rema.23.1.2018.12-27
72.
WolfeE. W.McVayA. (2012). Application of latent trait models to identifying substantively interesting raters. Educational Measurement, 31(3), 31–37. https://doi.org/10.1111/j.1745-3992.2012.00241.x
73.
ZdzinskiS. F.BarnesG. V. (2002). Development and validation of a string performance rating scale. Journal of Research in Music Education, 50, 245–255.
74.
ZoanettiN.ChampionH. (2016). Mitigating the halo effect: Managing the wow factor in music performance assessment. Journal of Research in Music Performance, (Spring) 36–52. http://dx.doi.org/10.21061/jrmp.v0i0.738
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
