A meta-analysis of studies examining the interrater reliability of the standard practice of peer assessments of quality of care was conducted. Using the Medline, Health Planning and Administration, and SCISEARCH databases, the English-language literature from 1966 through 1991 was searchedfor studies of chance corrected agreement among peer reviewers. The weighted mean kappa of 21 independent findings from 13 studies was .31. Comparison of this result with widely used standards suggests that the interrater reliability of peer assessment is quite limited and needs improvement. Research needs to be directed at modifying the peer review process to improve its reliability or at identifying indexes of quality with sufficient validity and reliability that they can be employed without subsequent peer review.
Get full access to this article
View all access options for this article.
References
1.
American Medical Association
. (1992). American medical association's policy positions. Quality Assurance and Utilization Review,7, 34-36.
2.
Bigby, J.
, Dunn, J., Goldman, L., Adams, J. B., Jen, P., Landefeld, C. S., & Komaroff, A. L. (1987). Assessing the preventability of emergency hospital admissions. American Journal of Medicine, 83, 1031-1036.
3.
Brennan, T. A.
, Leape, L. L., Laird, N. M., Herbert, L., Localio, A. R., Lawthers, A. G., Newhouse, J. P., Weiler, P. C., & Hiatt, H. H. (1991). Incidence of adverse events and negligence in hospitalized patients (Results of the Harvard Medical Practice Study I). New England Journal of Medicine, 324, 370-376.
4.
Brennan, T. A.
, Localio, R. J., & Laird, N. L. (1989). Reliability and validity of judgments concerning adverse events suffered by hospitalized patients. Medical Care, 27, 1148-1157.
5.
Brennan, T. A.
, Localio, R. J., Leape, L. L., Laird, N. M., Peterson, L., Hiatt, H. H., & Barnes, B. A. (1990). Identification of adverse events occurring during hospitalization. Annals of Internal Medicine,112, 221-226.
6.
Brewster, A. C.
, Karlin, B. G., Hyde, L. A., Jacobs, C. M., Bradbury, R. C., & Chae, Y. M. (1985). MEDISGRPS: A clinically based approach to classifying hospital patients at admission. Inquiry,22,377-387.
7.
Brook, R. H.
(1973). Quality of care assessment: A comparison offive methods of peer review (DHEW Publication HRA-74-3100). Washington, DC: U.S. Department of Health, Education, and Welfare, Public Health Service, Health Resources Administration, Bureau of Health Services Research and Evaluation.
8.
Caplan, R. A.
, Posner, K., Ward, R. J., & Cheney, F. W. (1988). Peer reviewer agreement for major anesthetic mishaps. Quality Review Bulletin, 14, 363-368.
9.
Caplan, R. A.
, Posner, K. L., & Cheney, F. W. (1991). Effect of outcome on physicianjudgments of appropriateness of care. Journal of the American Medical Association, 265, 1957-1960.
10.
Cicchetti, D. V.
, & Feinstein, A. R. (1990). High agreement but low kappa: II. Resolving the paradoxes. Journal of Clinical Epidemiology, 43, 551-558.
11.
Clarke, A.
(1990). Are readmissions avoidable?British Medical Journal, 301, 1136-1138.
12.
Craddick, J. W.
(1987). Improving quality and resource management through medical management analysis.Rockville, MD: Medical Management Analysis International.
13.
Daily, E. F.
, & Morehead, M. A. (1956). A method of evaluating and improving the quality of medical care. American Journal of Public Health, 46, 848-854.
14.
Department of Veterans Affairs
. (1989). Review of mortality in VA medical centers.Washington, DC: Author.
15.
Dixon, W. J.
, Brown, M. B., Engelman, L., & Jennrich, R. I. (1990). BMDP statistical software manual: Volume I (pp. 269-270). Berkeley: University of California Press.
16.
DuBois, R. W.
, & Brook, R. H. (1988). Preventable deaths: Who, how often and why? Annals of Internal Medicine, 109, 582-589.
17.
Feinstein, A. R.
, & Cicchetti, D. V. (1990). High agreement but low kappa: I. The problem of two paradoxes. Journal of Clinical Epidemiology, 43, 543-550.
18.
Fleiss, J.
(1981). Statistical methods forrates and proportions (2nd ed., pp. 212-236). New York: Wiley.
19.
Fleiss, J. L.
, & Cohen, J. (1973). The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and Psychological Measurement, 33, 613-619.
20.
Franki, S. E.
, Breeling, J. L., & Goldman, L. (1991). Preventability of emergent hospital readmission. American Journal of Medicine, 90, 667-674.
21.
Goldman, R. L.
(1992). The reliability of peer assessments of quality of care. Joumal of the American Medical Association,267, 958-961.
22.
Goldman, R. L.
, & Walder, D. J. (1992). An initial assessment of the veterans affairs occurrence screening program. Quality Review Bulletin, 18, 327-332.
23.
Government Accounting Office
. (1988). Intemal controls needed to strengthen controls over payments by Medicare intermediaries (HRD-89-8). Washington, DC: Author.
24.
Grove, W. M.
, Andreasen, N. C., McDonald-Scott, P., Keller, M. B., & Shapiro, W. B. (1981). Reliability studies of psychiatric practice: Theory and practice. Archives of General Psychiatry,38,408-413.
25.
Hannan, E. L.
, Bernard, H. R., O'Donnell, J. F., & Kilburn, H., Jr. (1989). A methodology for targeting hospital cases for quality of care record reviews. American Journal of Public Health, 79,430-436.
26.
Hartz, A.
, Gottlieb, M., Kuhn, E., & Rimm, A. (1989). National validation study of medicare mortality studies (Project No. 1247-M. Technical report to the Health Standards and Quality Review Bureau). Baltimore, MD: HCFA.
27.
Hastings, G. E.
, Sonneborn, R., Lee, G. H., Vick, L., & Sasmor, L. (1980). Peerreview checklist: Reproducibility and validity of a method for evaluating the quality of ambulatory care. American Journal of Public Health, 70, 222-228.
28.
Health Data Institute
. (1987). National DRG validation study (Report prepared for the Office of Inspector General, U.S. Department of Health and Human Sciences pursuant to Contract HHS-100-87-0015). Lexington, MA: Author.
29.
Hedges, L. V.
, & Olkin, I. (1985). Statistical methodsfor meta-analysis.Orlando, FL: Academic Press.
30.
Horn, S. D.
, & Pozen, M. W. (1977). An interpretation of implicit judgments in chart review. Journal of Community Health,2, 251-258.
31.
Horn, S. D.
, & Williamson, J. W. (1977). Statistical methods for reliability and validity testing: An application to nominal group judgment in health care. Medical Care, 25, 922-928.
32.
Hulka, B. S.
, Romm, F. J., Parkerson, G. R., Jr., Russell, I. T., Clapp, N. E., & Johnson, F. S. (1979). Peer review in ambulatory care: Use of explicit criteria and implicit judgments. Medical Care,17(3 Suppl.), 1-73.
33.
Hunter, J. E.
, & Schmidt, F. L. (1990). Methods of meta-analysis: Correcting error and bias in research findings.Newbury Park, CA: Sage.
34.
Institute of Medicine
. (1990). A strategy for quality assurance: Volume LWashington, DC: National Academy Press.
35.
Izzeoni, L. I.
(1989). Using severity information for quality assessment: Areview of three cases by five severity measures. Quality Review Bulletin, 15, 376-382.
36.
Jencks, S. F.
(1990). Quality assurance. Journal of the American Medical Association, 263, 2679-2680.
37.
Jencks, S. F.
, Daley, J., Draper, D., Thomas, N., Lenhart, G., & Walker, J. (1988). Interpreting hospital mortality data: The role of clinical risk adjustment. Journal of the American Medical Association, 260, 3611-3616.
38.
Kaiser-Permanente Medical Care Program
. (1978). Quality of medical care research in methods of assessment and assurance.Los Angeles: Author.
39.
Klaucke, D. N.
, Revicki, D. A., & Brown, R. E. (1988). Investigation of mortality and severe morbidity associated with anesthesia: Pilot study.Washington, DC: Centers for Disease Control and Battelle Human Affairs Research Centers.
40.
Koran, L. M.
(1975). The reliability of clinical methods, data and judgments. New England Journal of Medicine,293,695-701.
41.
Krippendorff, K.
(1970). Bivariate agreement coefficients for reliability of data. In E. F. Borgatta (Ed.), Sociological methodology (pp. 139-150). San Francisco, CA: Jossey-Bass.
42.
Kroeger, H. H.
, Altman, I., Clark, D. A., Johnson, A. C., & Sheps, C. G. (1965). The office practice of internists: I. The feasibility of evaluating quality of care. Journal of the American Medical Association, 193, 371-376.
43.
Landis, J. R.
, & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159-174.
44.
Lembcke, P. A.
(1967). Evaluation of medical audit. Journal of the American Medical Association, 199,543-550.
45.
Mayer, W. M.
, Clinton, J. J., & Newhall, D. (1988). A first report of the department of defense external civilian peer review of medical care. Journal of the American Medical Association,260,2690-2693.
46.
Medical Review Project
, Empire State Medical, Scientific and Educational Foundation, Inc. (1967). Rochester region perinatal study. New York State Journal of Medicine, 67, 1205-1210.
47.
Morehead, M. A.
, Donaldson, R. S., & Burt, F. E. (1964). A study of quality of hospital care secured by a sample of teamster family members in New York City.New York: Columbia University School of Public Health and Administrative Medicine.
48.
Office of Health Finance and Data Analysis
. (1989). Medicare hospital mortality data 1989.Chicago: American Hospital Association.
49.
O'Leary, D. S.
(1988, July/August). President's column. Joint Commission Perspectives, pp. 2-3.
50.
Palmer, H. R.
, Strain, R., Maurer, J.V.W., Rothrock, J. K., & Thompson, M. S. (1984). Quality assurance in eight adult medicine group practices. Medical Care, 22, 632-642.
51.
Peterson, 0. L.
, Andrews, L. P., Spain, R. S., & Greenberg, B. G. (1956). An analytical study of North Carolina general practice 1953-1954. Journal of Medical Education,31(12), 1-162.
52.
Posner, K. L.
, Caplan, R. A., & Cheney, F. W. (1991). Physician agreement in judging clinical performance. Anesthesiology,75(Suppl. 3A), A1068-A1068.
53.
Posner, K. L.
, Sampson, P. D., Caplan, R. A., Ward, R. J., & Cheney, F. W. (1990). Measuring interrater reliability among multiple raters: An example of methods for nominal data. Statistics in Medicine, 9, 1103-1115.
54.
Revicki, D. A.
, Klaucke, D. N., Brown, R. E., & Caplan, R. A. (1990). Reliability of ratings of anesthesia contribution to adverse surgical outcomes. Quality Review Bulletin,16,404-408.
55.
Richardson, F. M.
(1972). Peer review of medical care. Medical Care, 10, 29-39.
56.
Rosenfeld, L. S.
(1957). Quality of medical care in hospitals. American Journal ofPublic Health,47, 856-865.
57.
Rosenthal, R.
(1984). Meta-analytic procedures for social research.Beverly Hills, CA: Sage.
58.
Rubenstein, L. V.
, Kahn, K. L., & Reinisch, E. J. (1990). Changes in quality of care for five diseases measured by implicit review, 1981 to 1986. Journal of the American Medical Association, 264, 1974-1979.
59.
Rubenstein, L. V.
, Kahn, K., Reinisch, E. J., Sherwood, M., Rogers, W. H., & Brook, R. H. (1989). Structured implicit review of the medical record: A new method of quality assessment. Clinical Research, 37, 324A-324A.
60.
Rubin, H. R.
, Rogers, W. H., Kahn, K. L., Rubenstein, L. V., & Brook, R. H. (1992). Watching the doctor watchers: How well do peer review organization methods detect hospital care quality problems?Journal of the American Medical Association,267, 2349-2354.
61.
Rutstein, D. D.
, Berenberg, W., Chalmers, T. C., Child, C. G., Fishman, A. P., & Perrin, E. B. (1976). Measuring the quality of medical care: A clinical method. New England Journal of Medicine, 294, 582-584.
62.
Sanazaro, P. J.
(1980). Quality assessment and quality assurance in medical care. Annual Review of Public Health, 1, 37-68.
63.
Sanazaro, P. J.
, & Worth, R. M. (1985). Measuring clinical performance of individual internists in office and hospital practice. Medical Care, 23, 1097-1114.
64.
Shrout, P. E.
, Spitzer, R. L., & Fleiss, J. L. (1987). Quantification of agreement in psychiatric diagnosis revisited. Archives of General Psychiatry, 44, 172-177.
65.
Spitznagel, E. L.
, & Helzer, J. E. (1985). A proposed solution to the base rate problem in the kappa statistic. Archives of General Psychiatry, 42, 725-728.
66.
Thomas, J. W.
, & Longo, D. R. (1990). Application of severity measurement systems for hospital quality management. Hospital and Health Services Administration,35, 221-243.
67.
Williamson, J. W.
, Fehlauer, C. S., Gaiennie, J., & Smith, C. S. (1991). Assessing quality of ambulatory care: A comparative analysis in rural versus tertiary general medical clinics. Quality Assurance and Utilization Review, 6, 8-15.
