The comparison of two measurement systems is important in medical and other contexts. A common goal is to decide if a new measurement system agrees suitably with an existing one, and hence whether the two can be used interchangeably. Various methods for assessing interchangeability are available, the most popular being the limits of agreement approach due to Bland and Altman. In this article, we review the challenges of this technique and propose a model-based framework for comparing measurement systems that overcomes those challenges. The proposal is based on a simple metric, the probability of agreement, and a corresponding plot which can be used to summarize the agreement between two measurement systems. We also make recommendations for a study design that facilitates accurate and precise estimation of the probability of agreement.
BarnettVD. Simultaneous pairwise linear structural relationships. Biometrics1969; 25: 129–142.
11.
LudbrookJ. Confidence in Altman-Bland plots: A critical review of the method of differences. Clin Exp Pharmacol Physiol2010; 37: 143–149.
12.
ChoudharyPKNagarajaHNMeasuring agreement in method comparison studies - A review. In: BalakrishnanNet al. (ed). Advances in ranking and selection, multiple comparisons, and reliability: Methodology and applications, Boston: Birkhauser, 2005, pp. 215–244.
13.
BarnhartHXHaberMJLiL. An overview on assessing agreement with continuous measurements. J Biopharm Stat2007; 17: 529–569.
14.
LinL. Overview of agreement statistics for medical devices. J Biopharm Stat2008; 18: 126–144.
15.
RyanTPWoodallWH. The most-cited statistical papers. J Appl Stat2005; 32: 461–474.
16.
BlandJMAltmanDG. Measuring agreement in method comparison studies. Stat Meth Med Res1999; 8: 135–160.
17.
BlandJMAltmanDG. Agreement between methods of measurement with multiple observations per individual. J Biopharmaceut Stat2007; 17: 571–582.
18.
Voelkel JG and Siskowski BE. A study of the Bland-Altman plot and its associated methodology. Technical Report, Center for Quality and Applied Statistics, Rochester Institute of Technology, 2005.
19.
BlandJMAltmanDG. Comparing methods of measurement: Why plotting difference against standard method is misleading. Lancet1995; 346: 1085–1087.
20.
BlandJMAltmanDG. Applying the right statistics: Analyses of measurement studies. Ultrasound Obstet Gynecol2003; 22: 85–93.
21.
ManthaSRoizenMFFleisherLet al.Comparing methods of clinical measurement: Reporting standards for Bland and Altman analysis. Anesth Analg2000; 90: 593–602.
22.
DewitteKFierensCStöcklDet al.Application of the Bland-Altman plot for interpretation of method-comparison studies: A critical investigation of its practice. Clin Chem2002; 48: 799–801.
23.
LinLHedayatASSinhaBet al.Statistical methods in assessing agreement: Models, issues, and tools. J Am Stat Assoc2002; 97: 257–270.
24.
Oldford RW. On modified QQ plots. Personal communication, 2014.
25.
O’BrienPetrieJLittlerWet al.The British Hypertension Society protocol for the evaluation of measuring devices. J Hypertension1993; 11: S43–S62.
26.
Stevens NT. Assessment and comparison of continuous measurement systems. PhD Thesis, University of Waterloo, Canada, 2014.
27.
LinL. Total deviation index for measuring individual agreement with applications in laboratory performance and bioequivalence. Stat Med2000; 19: 255–270.
28.
Maple 18. Maplesoft. Waterloo, ON: Waterloo Maple Inc. www.maplesoft.com.
