The purpose of this study was to examine differential item functioning (DIF) in a rubric used to assess middle-school solo and ensemble performances. This study was guided by the following research questions: (a) does measurement equivalence for all items exist when used to measure subgroups of students based on their musical instrument? (b) what patterns of differential item functioning effects exist for items when used to measure subgroups of students based on their musical instrument? and (c) what size of differential item functioning effects exists for items when used to measure subgroups of students based on their musical instrument? In total, 17 adjudicators evaluated 138 middle-school instrumental students (ages 11–13) in the context of a live, formal solo and ensemble performance assessment. Using the Many Facets Rasch Partial Credit measurement model, measurement equivalence for all items did not exist when used to measure subgroups of students based on their musical instrument (2(252) = 634.00, p < .01). Of the 252 total pairwise interactions examined between items and instruments, 57 (22.62%) significant interaction terms were identified. Overall, 26 (10.3%) significant interaction terms demonstrated a moderate to large effect (|DIF| >/= .63 logits), nine (3.6%) significant interaction terms demonstrated a slight to moderate effect (|DIF| = (.43, .63 logits)), and 22 (8.7%) significant interaction terms demonstrated a negligible effect (|DIF| < .43 logits). Implications for the fairness of music performance assessments and improved assessment protocols are discussed.
AbelesH. F.HofferC. R.KlottmanR. H. (1994). Foundations of music education (2nd edition). New York, NY: Schirmer Books.
2.
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for Educational and Psychological Testing. Washington, DC: American Educational Research Association
3.
AndoY. (1988). Architectural acoustics: Blending sound sources, sound fields, and listeners. New York, NY: Springer.
4.
AustinJ. R. (1988). The effect of music contest format on self-concept, motivation, achievement, and attitude of elementary band students. Journal of Research in Music Education, 36, 95–107.
5.
BanisterS. (1992). Attitudes of high school band directors toward the value of marching band and concert band contests and selected aspects of the overall band program. Missouri Journal of Research in Music Education, 29, 49–57.
6.
BergeeM. J.McWhirterJ. L. (2005). Selected influences on solo and small-ensemble festival ratings: Replication and extension. Journal of Research in Music Education, 53, 177–190.
7.
BergeeM. J.PlattM. C. (2003). Influence of selected variables on solo and small-ensemble festival ratings. Journal of Research in Music Education, 51, 342–353.
8.
BergeeM. J.WestfallC. R. (2005). Stability of a model explaining selected extramusical influences on solo and small-ensemble festival ratings. Journal of Research in Music Education, 53, 253–271.
9.
BergeeM. J. (2007). Performer, rater, occasion, and sequence as sources of variability in music performance assessment. Journal of Research in Music Education, 55, 344–358.
10.
BerminghamG. A. (2000). Effects of performers’ external characteristics on performance evaluations. Update: Applications of Research in Music Education, 18, 3–7.
11.
BondT. G.FoxC. M. (2015). Applying the Rasch model: Fundamental measurement in the human sciences (3rd edition). New York, NY: Routledge.
12.
BoyleD. J. (1992). Program evaluation for secondary school music programs. NASSAP Bulletin, 76, 63–68.
13.
BradshawL. (2016). Diagnostic classification models. In RuppA. A.LeightonJ. P. (Ed.), The Wiley handbook of cognition and assessment: Frameworks, methodologies, and applications (pp. 297–326). West Sussex, UK: John Wiley & Sons, Inc.
14.
BurnsedV.HinkleD.KingS. (1985). Performance evaluation reliability at selected concert festivals. Journal of Band Research, 21, 22–29.
15.
CiorbaC. R.SmithN. Y. (2009). Measurement of instrumental and vocal undergraduate performance juries using a multidimensional assessment rubric. Journal of Research in Music Education, 57, 5–15.
16.
ColwellR. J.HewittM. P. (2011). The teaching of instrumental music (4th Edition). New York, NY: Taylor and Francis.
17.
ConradD. (2003). Judging the judges: Improving rater reliability at music contests. NFHS Music Association Journal, 20, 27–31.
18.
CrochetL. S. (2006). Repertoire selection practices of band directors as a function of teaching experience, training, instructional level, and degree of success. (Unpublished doctoral dissertation). University of Miami, Coral Gables, FL.
19.
DavidsonJ. W.CoimbraD. D. C. (2001). Investigating performance evaluation by assessors of singers in a music college setting. Musicae Scientiae, 5, 33–53.
20.
DavidsonJ. W. (1993). Visual perception of performance manner in the movements of solo musicians. Psychology of Music, 21, 103–113.
21.
DavidsonJ. W. (1994). Which areas of a pianist’s body convey information about expressive intention to an audience?Journal of Human Movement Studies, 26, 279–301.
22.
DavidsonJ. W. (1995). What does the visual information contained in musical performances offer the observer? Some preliminary thoughts. In SteinbergR. (Ed.), The music machine: Psychophysiology and psychopathology of the sense of music (pp. 105–113). Berlin, Germany: Springer-Verlag.
23.
DukeR. A.ByoJ. L. (2011). The habits of musicianship: A Radical approach to beginning band. Austin, TX: University of Texas Center for Music Learning.
Educational Testing Service (2016). ETS international principles for the fairness of assessments: A manual for developing locally appropriate fairness guidelines for various countries. Retrieved from https://www.ets.org/s/about/pdf/fairness_review_international.pdf
26.
EngelhardG. E. (1997). Constructing rater and task banks for performance assessments. Journal of Outcome Measurement, 1, 19–33.
27.
EngelhardG. E. (2013). Invariant measurement: Using Rasch models in the social, behavioral, and health sciences. New York, NY: Routledge.
28.
EngelhardG. E.MyfordC. M. (2003). Monitoring faculty consultant performance in the advanced placement English literature and composition program with a Many-faceted Rasch Model (ETS Research Report).
29.
EngelhardG. E.WindS. A. (2018). Invariant measurement with raters and rating scales: Rasch models for rater-mediated assessments. New York, NY: Routledge.
30.
FiskeH. E. (1975). Judge-group differences in the rating of secondary school trumpet performers. Journal of Research in Music Education, 23, 186–196.
31.
FiskeH. E. (1983). The effect of a training procedure in music performance evaluation on judge reliability. Ontario, Canada.
32.
FranklinJ. O. (1979). Attitudes of school administrators, band directors, and band students towards selected activities of the public school band program. (Unpublished doctoral dissertation). Northwestern State University of Louisiana, Natchitoches, LA.
33.
GriffithsN. K.ReayJ. L. (2018). The relative importance of aural and visual information in the evaluation of Western canon music performance by musicians and nonmusicians. Music Perception: An Interdisciplinary Journal, 35, 364–375.
34.
HashP. M. (2012). An analysis of the ratings and interrater reliability of high school band contests. Journal of Research in Music Education, 60, 81–100.
35.
HollandP. W.ThayerD. T. (1988). Differential item performance and the Mantel-Haenszel procedure. In WainerH.BraunH. I. (Eds.), Test validity (pp.129–146). Engelwood Cliffs, NJ: Erlbaum.
36.
HowardK. K. (1994). A survey of Iowa high school band students’ self-perceptions and attitudes toward types of music contests. (Unpublished doctoral dissertation). University of Iowa, Iowa City, IA.
37.
HowardR. L. (2002). Repertoire selection practices and the development of a core repertoire for the middle school concert band. (Unpublished doctoral dissertation). University of Florida, Gainsville, FL.
38.
HurstC. W. (1994). A nationwide investigation of high school band directors’ reasons for participating in music competitions. (Unpublished doctoral dissertation). The University of North Texas, Denton, TX.
39.
KingS. E.BurnsedV. (2007). A study of the reliability of adjudicator ratings at the 2005 Virginia Band and Orchestra Directors Association state marching band festivals. Journal of Band Research, 27–33.
40.
KirchhoffC. (1988). The school and college band: wind band pedagogy in the United States. In GatesJ. T. (Ed.), Music education in the United States: Contemporary issues (pp. 259–276). Tuscaloosa, AL: The University of Alabama Press.
41.
KopiezR.WolfA.PlatzF. (2017). Small influence of performing from memory on audience evaluation. Empirical Musicology Review, 12(1–2), 2–14.
42.
KraheC.HahnU.WhitneyK. (2015). Is seeing (musical) believing? The eye versus the ear in emotional responses to music. Psychology of Music, 43(1), 140–148.
43.
LatimerM. E.BergeeM. J.CohenM. L. (2010). Reliability and perceived pedagogical utility of a weighted music performance assessment rubric. Journal of Research in Music Education, 58, 168–183.
44.
LinacreJ. M.WrightB. D. (2004). Construction of measures from many-facet data. In SmithE. V.SmithR. M. (Eds.), Introduction to Rasch Measurement: Theories, Models, and Applications (pp.296–321). Maple Grove, MN: JAM Press.
45.
LinacreJ. M. (1989). Many facet Rasch measurement. Chicago, IL: MESA Press.
46.
LinacreJ. M. (1994). Sample size and item calibration [or person measure] stability. Rasch Measurement Transactions, 7(4), 328.
NorrisC. E.BorstJ. D. (2007). An examination of the reliabilities of two choral festival adjudication forms. Journal of Research in Music Education, 55, 237–251.
54.
PenfieldR. D.CamilliG. (2007). Differential item functioning and item bias. In RaoC. R.SinharayS. (Eds.), Handbook of statistics 26: Psychometrics (pp. 125–167). Amsterdam, The Netherlands: Elsevier.
55.
PlatzF.KopiezR. (2012). When the eye listens: A meta-analysis of how audio-visual presentation enhances the appreciation of music performance. Music Perception: An Interdisciplinary Journal, 30, 71–83.
56.
RuppA.TemplinJ.HensonR. (2010). Diagnostic measurement: Theory, methods, and applications. New York, NY: Guilford.
57.
SaundersT. C.HolahanJ. M. (1997). Criteria-specific rating scales in the evaluation of high school instrumental performance. Journal of Research in Music Education, 45, 259–272.
58.
SchubertE. (2002). Continuous response methodology applied to expressive performance. In StevensC.BurnhamD.McPhersonG.SchubertE.RenswickJ. (Eds.), Proceedings of the Seventh International Conference on Music Perception and Cognition (pp. 83–86). Adelaide, Australia: Causal Productions.
59.
SilveyB. A. (2009). The Effects of band labels on evaluators’ judgments of musical performance. Update: Applications of Research in Music Education, 28, 47–52.
60.
TsayC. J. (2014). The vision heuristic: Judging music ensembles by sight alone. Organizational Behavior and Human Decision Processes, 124, 24–33.
61.
WaddellG.WilliamonA. (2017). Eye of the beholder: Stage entrance behavior and facial expression affect continuous quality ratings in music performance. Frontiers in Psychology- Cognitive Science, 8.
62.
WapnickJ.DarrowA. A.KovacsJ.DalrympleL. (1997). Effects of physical attractiveness on evaluation of vocal performance. Journal of Research in Music Education, 45, 470–479.
63.
WapnickJ.MazzaJ. K.DarrowA. A. (1998). Effects of performer attractiveness, stage behavior, and dress on evaluation of violin performance evaluation. Journal of Research in Music Education, 46, 510–521.
64.
WapnickJ.MazzaJ. K.DarrowA. A. (2000). Effects of performer attractiveness, stage behavior, and dress on evaluation of children’s piano performances. Journal of Research in Music Education, 48, 323–335.
65.
WesolowskiB. C. (2019). Item response theory and music testing. In BrophyT. S. (Ed.), The Oxford handbook of assessment policy and practice in music education (pp. 437–460). New York, NY: Oxford University Press.
66.
WesolowskiB. C.AmendR. M.BarnsteadT. S.EdwardsA. S.EverhartM.GoinsQ.GroganR. J.IIIHercegA. M.JenkinsS. I.JohnsP. M.McCarverC. J.SchapsR E.SorrellG. W.WilliamsJ. D. (2017). The development of a secondary-level solo wind instrument performance rubric using the Multifaceted Rasch Partial Credit Measurement Model. Journal of Research in Music Education, 65, 95–119.
67.
WesolowskiB. C.AthanasM.BurtonJ.EdwardsA. S.EdwardsK. E.GoinsQ.IrbyA.JohnsP.MusselwhiteD. J.ParidoB.SorrellG.ThompsonJ. (2018). Judgmental standard setting: The development of objective content and performance standards for secondary –level solo instrumental music assessment. Journal of Research in Music Education, 66, 224–245.
68.
WesolowskiB. C.WindS. A.EngelhardG. EJr.. (2015). Rater fairness in music performance assessment: Evaluating model-data fit and differential rater functioning. Musicae Scientiae, 19, 39–47.
69.
WindS. A.EngelhardG. E.Jr.WesolowskiB. C. (2016). Exploring the effects of rating designs and rater fit on achievement estimates within the context of music performance assessment. Educational Assessment, 21, 278–299.
70.
WindS. A.OoiP. S.EngelhardG. EJr.. (in press). Exploring decision consistency and decision accuracy across rating designs in rater-mediated music performance assessments. Musicae Scientiae. doi: 10.1177/1029864918761184
71.
WrightB. D.LinacreJ. M. (1994). Reasonable mean-square fit values. Rasch Measurement Transactions, 8, 370.
WrightB. D. (1999). Common sense for measurement. Rasch Measurement Transactions, 13, 704.
74.
ZwickR.ThayerD. T.LewisC. (1999). An empirical Bayes approach to Mantel-Haenszel DIF analysis. Journal of Educational Measurement, 36, 1–28.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
