We present a new model-based approach to the analysis of agreement between raters in a situation where all raters have supplied dichotomous ratings of the same cases in a sample. The model is a logistic regression model with random effects – a Rasch model. In the rater setting, the Rasch model includes parameters that allow raters to have different propensities to score a given set of individuals positively or negatively – the rater bias. An exact score test of the hypothesis of no rater bias is proposed and is shown to be an exact generalised McNemar's test. Based on the model, we suggest quantifying the rater variation as a suitable measure of the variation of the rater odds ratios. An important example that will serve to motivate and illustrate the proposed model, is the study of Umbilical artery Doppler velocimetry used by obstetricians to assess the status of a foetus. The purpose of the assessment is to improve the foetus' chance of survival by choosing the optimal time of elective delivery. In the study, data related to 139 perinatal deaths were sent to 32 experts who were asked whether the use of Doppler velocimetry might have prevented each death.
CohenJ. A coefficient for agreement for nominal scales. Educ Psychol Bull1960; 70: 213–220.
8.
KraemerHCPeriyakoilVSNodaA. Kappa coefficients in medical research. Stat Med2002; 21(14): 2109–2129.
9.
ShoukriMMMartinSWMianIUH. Maximum likelihood estimation of the kappa coefficient from models of matched bivariate responses. Stat Med1995; 4: 83–99.
10.
FleissJL. Statistical methods for rates and proportions, 2nd edn. New York: Wiley, 1981.
11.
FeinsteinARCicchettiDV. High agreement but low kappa: I. the problems of two paradoxes. J Clin Epidemiol1990; 43: 543–549.
12.
LandisJRKochGG. A one-way components of variance model for categorical data. Biometrics1977; 33: 671–679.
13.
CrowderMJ. Inference about the intraclass correlation coefficient in the Beta-binomial ANOVA for proportions. J R Stat Soc, Ser B, Methodol1979; 41: 230–234.
KangS-HChingS-JAhnCW. Exact tests for one sample correlated binary data. Biometric J2005; 47: 188–193.
16.
TannerMAYoungMA. Modeling agreement among raters. J Am Stat Assoc1985; 80: 175–180.
17.
DarrochJNMcCloudPI. Category distinguishability and observer agreement. Aus J Stat1986; 28: 371–388.
18.
EspelandMAHandelmanSL. Using latent class models to characterise and assess relative error in discrete measurements. Biometrics1989; 45: 587–599.
19.
LindsayBCloggCGregoJ. Semiparametric estimation in the Rasch model and related exponential response models, including a simple latent class model for item analysis. J Am Stat Assoc1991; 86: 96–107.
20.
UebersaxJS. Statistical modeling of expert ratings on medical treatment appropriateness. J Am Stat Assoc1993; 88: 421–427.
21.
CookRJFarewellVT. Conditional inference for subject-specific and marginal agreement: Two families of agreement measures (corr: 1998v26 p381). Can J Stat1995; 23: 333–344.
22.
RaschG. On general laws and the meaning of measurement in psychologyWiley, Berkley University Press, 1961.
23.
FischerGHMolenaarIW. Rasch models: foundations, recent developments, and applications, New York: Springer-Verlag Inc, 1995.
24.
SmithEVSmithRM. Introduction to rasch measurement, Maple Grove, MN: JAM Press, 2006.
25.
AndersenEB. A goodness of fit test for the Rasch model. Psychometrika1973; 38: 123–140.
26.
TjurT. A connection between Rasch's item analysis model and a multiplicative Poisson model. Scand J Stat1982; 9: 23–30.
MolenaarIW. Some improved diagnostics for failure in the rasch model. Psychometrika1983; 48(1): 49–72.
29.
WestergaardHBLanghoff-RossJLingmanGMarsalKKreinerS. A critical appraisal of the use of umbilical artery doppler ultrasound in high-risk pregnancies: use of meta-analysis in evidence-based obstetrics. Ultrasound Obstet Gynecol2001; 17: 466–476.
30.
BlandJMAltmanDG. Statistical methods for assessing agreement between two methods of clinical measurement. The Lancet1986; 327: 307–310.
31.
RaschG. Probabilistic models for some intelligence and attainment tests, Copenhagen: Nielsen and Lydiche, 1960.
32.
ChristensenKBKreinerS. Testing the fit of latent regression models. Commun Stat2004; 33(6): 1341–1356.
33.
LarsenKPetersenJHBudtz-JørgensenEEndahlL. Interpreting parameters in the logistic regression model with random effects. Biometrics2000; 56(3): 909–914.
34.
AndersenEB. Discrete statistical models with social science applications, New York: North-Holland/Elsevier, 1980.
35.
Barndorff-NielsenO. Information and exponential families in statistical theory, New York: Wiley, 1978.
36.
AndersenEB. Asymptotic properties of conditional maximum likelihood estimators. J R Stat Soc B1970; 32: 283–301.
37.
KeldermanH. Computing maximum likelihood estimates of loglinear models from marginal sums with special attention to loglinear item response theory. Psychometrika1992; 57: 223–245.
38.
Breslow NE and Day NE. Statistical methods in cancer research. Vol. I – The analysis of case-control studies., volume 32 of IARC scientific publications. Lyon: International Agency for Research on Cancer, 1980.
