Background. In this article, the authors determine the reproducibility of appropriateness ratings for cataract surgery between a multidisciplinary physician panel that convened and a multidisciplinary physician panel that completed ratings by mail.Methods. Eighteen panelists, who constituted 2 distinct multidisciplinary panels, rated 2894 clinical scenarios as an appropriate, inappropriate, or uncertain indication to perform cataract surgery. Each panel’s summary score for each scenario was calculated. Weighted kappa values were determined to assess the level of agreement between the ratings of the 2 panels.Results. The panels had a substantial level of agreement overall, with a weighted kappa statistic of 0.64. There was agreement on about 68% of the scenarios, and serious disagreement, in which one panel rated an indication appropriate and the other rated it inappropriate, occurred in only 1% of the ratings.Conclusion. There was substantial agreement about the ratings by the 2 panels. The panel that convened rated fewer scenarios uncertain and more appropriate, suggesting the impact of group dynamics and face-to-face discussion on resolution of uncertainty.
Javitt JC, Kendix M, Tielsch JM, et al.Geographic variation in utilization of cataract surgery. Med Care. 1995;33:90-105.
2.
Goldzweig CL, Mittman BS, Carter GM, et al.Variations in cataract extraction rates in Medicare prepaid and fee-for-service settings. JAMA. 1997;277:1765-1768.
3.
Steinberg EP, Tielsch JM, Schein OD, et al.National study of cataract surgery outcomes: variation in 4-month postoperative outcomes as reflected in multiple outcome measures. Ophthalmology. 1994;101:1131-1141.
4.
US Department of Health and Human Services, Public Health Service, Agency for Health Care Policy and Research. Cataract in adults: management of functional impairment. Rockville MD: US Department of Health and Human Services, Public Health Service, Agency for Health Care Policy and Research; 1993.
5.
American Academy of Ophthalmology. Cataract in the otherwise healthy adult eye. San Francisco, CA: American Academy of Ophthalmology; 1989.
6.
Tobacman JK, Zimmerman B, Lee P, Hilborne L, Brook RH. Assessment of appropriateness of cataract surgery in ten academic medical centers in 1990. Ophthalmology. 1996;103:207-215.
7.
Desai P.The National Cataract Surgery Survey, II: clinical outcomes. Eye. 1993;7:489-494.
8.
Desai P, Minassian DC, Reidy A.National cataract surgery survey 1997-8: a report of the results of the clinical outcomes. Br J Ophthalmol. 1999;83:1336-1340.
9.
Leape LL, Park RE, Kahan JP, Brook RH. Group judgments of appropriateness: the effect of panel composition. Quality Assurance Health Care. 1992;4:151-159.
10.
Fraser GM, Pilpel D, Kosecoff J, Brook RH. Effect of panel composition on appropriateness ratings. Int J Quality Health Care. 1994;6:251-255.
11.
Kahan JP, Park RD, Leape LL, et al.Variations by specialty in physician ratings of the appropriateness and necessity of indications for procedures. Med Care. 1996;34:512-523.
12.
Scott EA, Black N.Appropriateness of cholecystectomy in the United Kingdom—a consensus panel approach. Gut. 1991;32:1066-1070.
13.
Naylor CD, McGlynn EA, Leape LL, et al.Coronary angiography and revascularization: defining procedural indications through formal group processes. Can J Cardiol. 1994;10(1):41-48.
14.
Leape LL, Hilborne LH, Park RE, et al.The appropriateness of use of coronary artery bypass graft surgery in New York state. JAMA. 1993;269:753-760.
15.
Park RE, Fink A, Brook RH, et al.Physician ratings of appropriate indications for six medical and surgical procedures. Am J Public Health. 1986;76:766-772.
16.
Lee PP, Kamberg CJ, Hilborne LH, et al.Cataract surgery: a literature review and ratings of appropriateness and cruciality. Santa Monica, CA: RAND Corporation; 1993.
17.
SAS Institute. SAS/STAT software: changes and enhancements through release 6.11. Cary, NC: SAS Institute Inc; 1996.
18.
O’Connell AL, Dobson AJ. General observer-agreement measures on individual subjects and groups of subjects. Biometrics. 1984;40:973-983.
19.
Seigel DG, Podgor MJ, Remaley NA. Acceptable values of kappa for comparison of two groups. Am J Epidemiol. 1992;135:571-578.
20.
Scott EA, Block N.When does consensus exist in expert panels?J Public Health Med. 1991;13:35.
21.
Shekelle PG, Kahan JP, Bernstein SJ, Leape LL, Kamberg CF, Park RE. The reproducibility of a method to identify the overuse and underuse of medical procedures. N Engl J Med. 1998;338:1888-1895.
22.
Tobacman JK, Scott IU, Cyphert S, Zimmerman B.Reproducibility of measures of overuse of cataract surgery by three physician panels. Med Care. 1999;37:937-945.
