Rating scale analysis techniques provide researchers with practical tools for examining the degree to which ordinal rating scales (e.g., Likert-type scales or performance assessment rating scales) function in psychometrically useful ways. When rating scales function as expected, researchers can interpret ratings in the intended direction (i.e., lower ratings mean “less” of a construct than higher ratings), distinguish between categories in the scale (i.e., each category reflects a unique level of the construct), and compare ratings across elements of the measurement instrument, such as individual items. Although researchers have used these techniques in a variety of contexts, studies are limited that systematically explore their sensitivity to problematic rating scale characteristics (i.e., “rating scale malfunctioning”). I used a real data analysis and a simulation study to systematically explore the sensitivity of rating scale analysis techniques based on two popular polytomous item response theory (IRT) models: the partial credit model (PCM) and the generalized partial credit model (GPCM). Overall, results indicated that both models provide valuable information about rating scale threshold ordering and precision that can help researchers understand how their rating scales are functioning and identify areas for further investigation or revision. However, there were some differences between models in their sensitivity to rating scale malfunctioning in certain conditions. Implications for research and practice are discussed.
AdamsR. J.WuM. L.WilsonM. (2012). The Rasch rating model and the disordered threshold controversy. Educational and Psychological Measurement, 72(4), 547–573. https://doi.org/10.1177/0013164411432166
2.
AndrichD. A. (1978). A rating formulation for ordered response categories. Psychometrika, 43(4), 561–573. https://doi.org/10.1007/BF02293814
3.
AndrichD. A. (2010, June). The detection of a structural halo when multiple criteria have the same generic categories for rating [Conference session]. International Conference on Rasch Measurement, Copenhagen, Denmark.
4.
AndrichD. A. (2013). An expanded derivation of the threshold structure of the polytomous Rasch model that dispels any “threshold disorder controversy.”Educational and Psychological Measurement, 73(1), 78–124. https://doi.org/10.1177/0013164412450877
5.
AndrichD. A. (2015). The problem with the step metaphor for polytomous models for ordinal assessments. Educational Measurement: Issues and Practice, 34(2), 8–14. https://doi.org/10.1111/emip.12074
BondT. G.YanZ.HeeneM. (2020). Applying the Rasch model: Fundamental measurement in the human sciences (4th ed.). Routledge.
8.
BonifayW. (2019). Multidimensional item response theory (1st ed.). SAGE.
9.
BozdağF.BilgeF. (2022). Scale Adaptation for refugee children: Sense of school belonging and social contact. Journal of Psychoeducational Assessment. Advance online publication. https://doi.org/10.1177/07342829221094402
10.
BriggsD. C.WilsonM. (2003). An introduction to multidimensional measurement using Rasch Models. Journal of Applied Measurement, 4(1), 87–100.
11.
BuchholzJ.HartigJ. (2019). Comparing attitudes across groups: An IRT-Based Item-Fit Statistic for the analysis of measurement invariance. Applied Psychological Measurement, 43(3), 241–250. https://doi.org/10.1177/0146621617748323
12.
ChenW.-H.ThissenD. (1997). Local dependence indexes for item pairs using item response theory. Journal of Educational and Behavioral Statistics, 22(3), 265–289.
13.
CrockerL.AlginaJ. (1986). Introduction to classical and modern test theory. Holt, Rinehart and Winston.
14.
DeAyalaR. J. (2009). The theory and practice of item response theory. The Guilford Press.
15.
EmbretsonS. E.ReiseS. P. (2000). Item response theory for psychologists. Lawrence Erlbaum.
16.
EngelhardG.WindS. A. (2018). Invariant measurement with raters and rating scales: Rasch models for rater-mediated assessments. Taylor & Francis.
17.
EngelhardG.Jr.WangJ.WindS. A. (2018). A tale of two models: Psychometric and cognitive perspectives on rater-mediated assessments using accuracy ratings. Psychological Test and Assessment Modeling, 60(1), 33–52.
18.
FinchH. (2011). The impact of missing data on the detection of nonuniform differential item functioning. Educational and Psychological Measurement, 71(4), 663–683.
19.
ForrestC. B.TuckerC. A.Ravens-SiebererU.PratiwadiR.MoonJ.TeneralliR. E.BeckerB.BevansK. B. (2016). Concurrent validity of the PROMIS® pediatric global health measure. Quality of Life Research, 25(3), 739–751. https://doi.org/10.1007/s11136-015-1111-7
20.
GreenS. B.YangY. (2015). Evaluation of dimensionality in the assessment of internal consistency reliability: Coefficient alpha and omega coefficients. Educational Measurement: Issues and Practice, 34(4), 14–20. https://doi.org/10.1111/emip.12100
21.
HaddadC.KhouryC.SalamehP.SacreH.HallitR.KheirN.ObeidS.HallitS. (2021). Validation of the Arabic version of the eating attitude test in Lebanon: A population study. Public Health Nutrition, 24(13), 4132–4143. https://doi.org/10.1017/S1368980020002955
22.
HagedoornE. I.PaansW.JaarsmaT.KeersJ. C.van der SchansC. P.LuttikM. L.KrijnenW. P. (2018). Translation and psychometric evaluation of the Dutch Families Importance in Nursing Care: Nurses’ attitudes scale based on the Generalized Partial Credit Model. Journal of Family Nursing, 24(4), 538–562. https://doi.org/10.1177/1074840718810551
23.
HambletonR. K.van der LindenW. J.WellsC. S. (2010). IRT models for the analysis of polytomously scored data: Brief and selected history of model building advances. In NeringM. L.OstiniR. (Eds.), Handbook of polytomous item response theory models (pp. 21–42). Routledge.
24.
KornettiD. L.FritzS. L.ChiuY.-P.LightK. E.VelozoC. A. (2004). Rating scale analysis of the Berg balance scale. Archives of Physical Medicine and Rehabilitation, 85(7), 1128–1135. https://doi.org/10.1016/j.apmr.2003.11.019
25.
LinacreJ. M. (1989). Many-facet Rasch measurement. MESA Press.
26.
LinacreJ. M. (2002). Optimizing rating scale category effectiveness. Journal of Applied Measurement, 3(1), 85–106.
27.
LinacreJ. M. (2004). Optimizing rating scale category effectiveness. In SmithE. V.Jr.SmithR. M. (Eds.), Introduction to Rasch measurement theory: Models and applications (pp. 258–278). JAM Press.
28.
LuoG. (2005). The relationship between the Rating Scale and Partial Credit models and the implication of disordered thresholds of the Rasch models for polytomous responses. Journal of Applied Measurement, 6(4), 443–455.
MellenberghG. J. (1995). Conceptual notes on models for discrete polytomous item responses. Applied Psychological Measurement, 19(1), 91–100. https://doi.org/10.1177/014662169501900110
31.
MolenaarI. W. (1997). Nonparametric models for polytomous responses. In van der LindenW. J.HambletonR. K. (Eds.), Handbook of modern item response theory (pp. 369–380). Springer.
32.
MoorsG. (2008). Exploring the effect of a middle response category on response style in attitude measurement. Quality & Quantity, 42(6), 779–794. https://doi.org/10.1007/s11135-006-9067-x
33.
MurakiE. (1997). A generalized partial credit model. In van der LindenW. J.HambletonR. K. (Eds.), Handbook of modern item response theory (pp. 153–164). Springer. https://doi.org/10.1007/978-1-4757-2691-6_9
34.
MurakiE.MurakiM. (2018). Generalized partial credit model. In van der LindenW. J. (Ed.), Handbook of item response theory (Vol. 1, pp. 127–138). CRC Press.
35.
NeringM. L.OstiniR. (Eds.). (2010). Handbook of polytomous item response theory models. Routledge.
36.
R Core Team. (2021). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.R-project.org/
37.
ReckaseM. D. (2009). Multidimensional item response theory (1st ed.). Springer.
SamejimaF. (1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika Monograph Supplement, 34(Part 2, No. 17), 1–97.
40.
SeolH. (2016). Using the bootstrap method to evaluate the critical range of misfit for polytomous Rasch fit statistics. Psychological Reports, 118(3), 937–956.
41.
SmithR. M. (2004). Fit analysis in latent trait models. In SmithE. V.SmithR. M. (Eds.), Introduction to Rasch measurement (pp. 73–92). JAM Press.
42.
Van Zile-TamsenC. (2017). Using Rasch analysis to inform rating scale development. Research in Higher Education, 58(8), 922–933.
43.
WalkerA. A.JenningsJ. K.EngelhardG. (2018). Using person response functions to investigate areas of person misfit related to item characteristics. Educational Assessment, 23(1), 47–68. https://doi.org/10.1080/10627197.2017.1415143
44.
WaughR. F. (2002). Creating a scale to measure motivation to achieve academically: Linking attitudes and behaviours using Rasch measurement. British Journal of Educational Psychology, 72(1), 65–86. https://doi.org/10.1348/000709902158775
45.
WesolowskiB. C.WindS. A.EngelhardG. (2016). Examining rater precision in music performance assessment: An analysis of rating scale structure using the Multifaceted Rasch Partial Credit Model. Music Perception, 33(5), 662–678. https://doi.org/10.1525/mp.2016.33.5.662
46.
WindS. A.GuoW. (2019). Exploring the combined effects of rater misfit and differential rater functioning in performance assessments. Educational and Psychological Measurement, 79(5), 962–987. https://doi.org/10.1177/0013164419834613
47.
WindS. A.JonesE. (2019). The effects of incomplete rating designs in combination with rater effects. Journal of Educational Measurement, 56(1), 76–100. https://doi.org/10.1111/jedm.12201
48.
WindS. A.TsaiC.-L.GrajedaS. B.BerginC. (2018). Principals’ use of rating scale categories in classroom observations for teacher evaluation. School Effectiveness and School Improvement, 29(3), 485–510. https://doi.org/10.1080/09243453.2018.1470989
49.
WindS. A.WalkerA. A. (2019). Exploring the correspondence between traditional score resolution methods and person fit indices in rater-mediated writing assessments. Assessing Writing, 39, 25–38. https://doi.org/10.1016/j.asw.2018.12.002
50.
WolfeE. W. (2013). A bootstrap approach to evaluating person and item fit to the Rasch model. Journal of Applied Measurement, 14(1), 1–9.
51.
WolfeE. W.JiaoH.SongT. (2014). A family of rater accuracy models. Journal of Applied Measurement, 16(2), 153–160.
WrightB. D.MokM. M. C. (2004). An overview of the family of Rasch measurement models. In SmithE. V.SmithR. M. (Eds.), Introduction to Rasch measurement (pp. 1–24). JAM Press.
54.
WuM.AdamsR. J. (2013). Properties of Rasch residual fit statistics. Journal of Applied Measurement, 14(4), 339–355.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
