Student evaluation of teaching (SET) assesses students’ experiences in a class to evaluate teachers’ performance in class. SET essentially comprises three facets: teaching proficiency, student rating harshness, and item properties. The computerized adaptive testing form of SET with an established item pool has been used in educational environments. However, conventional scoring methods ignore the harshness of students toward teachers and, therefore, are unable to provide a valid assessment. In addition, simultaneously estimating teachers’ teaching proficiency and students’ harshness remains an unaddressed issue in the context of online SET. In the current study, we develop and compare three novel methods—marginal, iterative once, and hybrid approaches—to improve the precision of parameter estimations. A simulation study is conducted to demonstrate that the hybrid method is a promising technique that can substantially outperform traditional methods.
AcuñaE. A. V. (2017). Response styles in student evaluation of teaching. Doctoral dissertation. University of Toronto.
2.
AndersonH. M.CainJ.BirdE. (2005). Online student course evaluations: Review of literature and a pilot study. American Journal of Pharmaceutical Education, 69(1), 5. https://doi.org/10.5688/aj690105
3.
BirnbaumA. (1968). Some latent trait models and their use in inferring an examinee’s ability. In LordF. M.NovickM. R.BirnbaumA. (Eds.), Statistical theories of mental test scores (pp. 397–479). Addison-Wesley.
4.
BooneW. J.TownsendJ. SStaverJ. R. (2016). Utilizing multifaceted Rasch measurement through FACETS to evaluate science education data sets composed of judges, respondents, and rating scale items: An exemplar utilizing the elementary science teaching analysis matrix instrument. Science Education, 100(7), 221–238. https://doi-org.ezproxy.uio.no/10.1002/sce.21210
5.
ChangH.-H.YingZ. (1996). A global information approach to computerized adaptive testing. Applied Psychological Measurement, 20(3), 213–229. https://doi.org/10.1177/014662169602000303
6.
ChenS.-Y.AnkenmannR. D.SprayJ. A. (1999). Exploring the relationship between item exposure rate and test overlap rate in computerized adaptive testing(ACT Research Report Series, 99, Issue 5). ACT.
7.
ChengY.ChangH. H. (2009). The maximum priority index method for severely constrained item selection in computerized adaptive testing. British Journal of Mathematical and Statistical Psychology, 62(Pt 2), 369–383. https://doi.org/10.1348/000711008X304376
8.
DempsterA. P.LairdN. M.RubinD. B. (1977). Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society: Series B (Methodological), 39(1), 1–22. https://doi.org/10.1111/j.2517-6161.1977.tb01600.x
9.
DommeyerC. J.BaumP.HannaR. W.ChapmanK. S. (2004). Gathering faculty teaching evaluations by in-class and online surveys: Their effects on response rates and evaluations. Assessment & Evaluation in Higher Education, 29(5), 611–623. https://doi.org/10.1080/02602930410001689171
10.
FisherR. A. (1922). On the mathematical foundations of theoretical statistics. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character, 222(594–604), 309–368. https://doi.org/10.1098/rsta.1922.0009
11.
GamlielE.DavidovitzL. (2005). Online versus traditional teaching evaluation: Mode can matter. Assessment & Evaluation in Higher Education, 30(6), 581–592. https://doi.org/10.1080/02602930500260647
12.
HarwellM.StoneC. A.HsuT.-C.KirisciL. (1996). Monte Carlo studies in item response theory. Applied Psychological Measurement, 20(2), 101–125. https://doi.org/10.1177/014662169602000201
13.
HolzbachR. L. (1978). Rater bias in performance ratings: Superior, self-and peer ratings. Journal of Applied Psychology, 63(5), 579–588. https://doi.org/10.1037/0021-9010.63.5.579
14.
HoytW. T. (2000). Rater bias in psychological research: When is it a problem and what can we do about it?Psychological Methods, 5(1), 64–86. https://doi.org/10.1037/1082-989x.5.1.64
15.
HsuehI.-P.ChenJ.-H.WangC.-H.ChenC.-T.SheuC.-F.WangW.-C.HouW.-H.HsiehC.-L. (2010). Development of a computerized adaptive test for assessing balance function in patients with stroke. Physical Therapy, 90(9), 1336–1344. https://doi.org/10.2522/ptj.20090395
16.
KemberD.WongA. (2000). Implications for evaluation from a study of students' perceptions of good and poor teaching. Higher Education, 40(1), 69–97. https://doi.org/10.1023/A:1004068500314
17.
KingsburyG. G.ZaraA. R. (1989). Procedures for selecting items for computerized adaptive tests. Applied Measurement in Education, 2(4), 359–375. https://doi.org/10.1207/s15324818ame0204_6
18.
LeungC.-K.ChangH.-H.HauK.-T. (2000). Content balancing in stratified computerized adaptive testing designs. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA.
19.
LinacreJ. M. (1993). Generalizability theory and many-facet Rasch measurement [Paper presentation]. In The Annual Meeting of the American Educational Research Association, Atlanta, GA, USA, April 12-16 1993. https://eric.ed.gov/?id=ED364573
MurakiE. (1992). A generalized partial credit model: Application of an EM algorithm. Applied Psychological Measurement, 16(2), 159–176. https://doi.org/10.1177/014662169201600206
23.
PaoloA. M.BonaminioG. A.GibsonC.PartridgeT.KallailK. (2000). Response rate comparisons of e-mail-and mail-distributed student evaluations. Teaching and Learning in Medicine, 12(2), 81–84. https://doi.org/10.1207/S15328015TLM1202_4
24.
ReckaseM. D. (2009). Multidimensional item response theory models. In ReckaseM. D. (Ed.), Multidimensional item response theory (pp. 79–112). Springer. https://doi.org/10.1007/978-0-387-89976-3_4
25.
RudnerL. M. (2009). Implementing the graduate management admission test computerized adaptive test. In van der LindenW. J.GlasA. W. C. (Eds.), Elements of adaptive testing (pp. 151–165). Springer. https://doi.org/10.1007/978-0-387-85461-8_8
ThissenD.MislevyR. J. (2000). Testing algorithms. In WainerH.DoransN. J.EignorD.FlaugherR.GreenB. F.MislevyR. J.SteinbergL.ThissenD. (Eds.), Computerized adaptive testing: A prime (2nd ed.,pp. 125–158). Lawrence Erlbaum Associates. https://doi.org/10.4324/9781410605931-13
29.
van der LindenW. J. (2005). A comparison of item-selection methods for adaptive tests with content constraints. Journal of Educational Measurement, 42(3), 283–302. https://doi.org/10.1111/j.1745-3984.2005.00015.x
30.
WainerH.DoransN. J.FlaugherR.GreenB. F.MislevyR. J. (2000). Computerized adaptive testing: A primer. Lawrence Erlbaum Associates. https://doi.org/10.4324/9781410605931-13
31.
WangT.KolenM. J. (2001). Evaluating comparability in computerized adaptive testing: Issues, criteria and an example. Journal of Educational Measurement, 38(1), 19–49. https://doi.org/10.1111/j.1745-3984.2001.tb01115.x
32.
WeissmanA. (2007). Mutual information item selection in adaptive classification testing. Educational and Psychological Measurement, 67(1), 41–58. https://doi.org/10.1177/0013164406288164
33.
WesolowskiB. C.WindS. A.Engelhard JrG. (2016). Examining rater precision in music performance assessment: An analysis of rating scale structure using the multifaceted Rasch partial credit model. Music Perception: An Interdisciplinary Journal, 33(5), 662–678. https://doi.org/10.1525/mp.2016.33.5.662
34.
WindS. A.JonesE. (2019). The effects of incomplete rating designs in combination with rater effects. Journal of Educational Measurement, 56(1), 76–100. https://doi-org.ezproxy.uio.no/10.1111/jedm.12201
35.
WolfeE. W. (2004). Identifying rater effects using latent trait models. Psychology Science, 46(1), 35–51.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
