Much research has been devoted to addressing challenges in achieving reliable assessments of physicians’ clinical performance but less work has focused on whether valid and accurate classification decisions are feasible. This study used 957 physicians certified in internal medicine (IM) or a subspecialty, who completed the American Board of Internal Medicine (ABIM) Diabetes Practice Improvement Module (PIM). Ten clinical and two patient-experience measures were aggregated into a composite measure. The composite measure score was highly reliable (r = .91) and classification accuracy was high across the entire score scale (>0.90), which indicated that it is possible to differentiate high-performing and low-performing physicians. Physicians certified in endocrinology and those who scored higher on their IM certification examination had higher composite scores, providing some validity evidence. In summary, it is feasible to create a psychometrically robust composite measure of physicians’ clinical performance, specifically for the quality of care they provide to patients with diabetes.
American Board of Internal Medicine. (n.d.). ABIM HIPAA business associate agreement. Retrieved March 3, 2010, from http://www.abim.org/pdf/hipaa/hipaa_compliance.pdf
2.
American Diabetes Association. (2009). Standards of medical care in diabetes- 2009. Diabetes Care , 32, S13-S61.
3.
Bridges to Excellence. (2009). Diabetes care link. Retrieved March 3, 2010, from http://bridgestoexcellence.org/Documents/DCL%20Measures.pdf
4.
Brennan, T.A., Horwitz, R.I., Duffy, F.D., Cassel, C.K., Goode, L.D., & Lipner, R.S. ( 2004). The role of physician specialty board certification status in the quality movement. Journal of the American Medical Association , 292, 1038-1043.
5.
Doran, T., Fullwood, C., Reeves, D., Gravelle, H., & Roland, M. ( 2008). Exclusion of patients from pay-for-performance targets by English physicians. New England Journal of Medicine, 359, 274-284.
6.
Duffy, F.D., Lynn, L.A., Didura, H., Hess, B., Caverzagie, K., Grosso, L., & . . . Holmboe, E.S. (2008). Self-assessment of practice performance: Development of the ABIM Practice Improvement Module (PIMSM). Journal of Continuing Education in the Health Professions , 28, 38-46.
7.
Dunn-Rankin, P. ( 1983). Scaling methods. Hillsdale, NY: Lawrence Erlbaum.
8.
Epstein, A.M. ( 2007). Pay for performance at the tipping point. New England Journal of Medicine, 356, 515-517.
9.
Greenfield, S., Kaplan, S.H., Kahn, R., Ninomiya, J., & Griffith, J.L. (2002). Profiling care provided by different groups of physicians: Effects of patient case-mix (bias) and physician-level clustering on quality assessment results. Annals of Internal Medicine, 136, 111.
10.
Hess, B.J., Weng, W., Lynn, L.A., Holmboe, E.S., & Lipner, R.S. (in press). Setting a fair performance standard for physicians’ quality of patient care. Manuscript submitted for publication.
11.
Holmboe, E.S., Wang, Y., Meehan, T.P., Tate, J.P., Ho, S.-Y., Starkey, K.S., & Lipner, R.S. ( 2008). Association between maintenance of certification examination scores and quality of care for Medicare beneficiaries. Archives of Internal Medicine, 168, 1396-1403.
12.
Holmboe, E.S., Wang, Y., Tate, J.P., & Meehan, T.P. ( 2006). The effects of patient volume on the quality of diabetic care for Medicare beneficiaries. Medical care, 44, 1073-1077.
13.
Kaplan, S., Griffith, J., Price, L., Pawlson, L., & Greenfield, S. (2009). Improving the reliability of physician performance assessment: Identifying the ‘‘physician effect’’ on quality and creating composite measures. Medical Care, 47, 378-387.
14.
Kerr, E.A., Hofer, T.P., Hayward, R.A., Adams, J.L., Hogan, M.M., McGlynn, E.A., & Asch, S.M. ( 2007). Quality by any other name?: A comparison of three profiling systems for assessing health care quality. Health services research, 42, 2070-2087.
15.
Landon, B.E., & Normand, S.-L.T. (2008). Performance measurement in the small office practice: Challenges and potential solutions. Annals of Internal Medicine, 148, 353-377.
16.
Landon, B.E., Normand, S.-L.T., Blumenthal, D., & Daley, J. ( 2003). Physician clinical performance assessment: Prospects and barriers. Journal of the American Medical Association , 290, 1183-1189.
17.
Lee, W.-C., Brennan, R.L., & Wan, L. ( 2009). Classification consistency and accuracy for complex assessments under the compound multinomial model. Applied Psychological Measurement, 33, 374-390.
18.
Lipner, R.S., Weng, W., Arnold, G.K., Duffy, F.D., Lynn, L.A., & Holmboe, E.S. ( 2007). A three-part model for measuring diabetes care in physician practice. Academic Medicine, 82, S48-S52.
19.
McGlynn, E.A., Asch, S.M., Adams, J., Keesey, J., Hicks, J., DeCristofaro, A., & Kerr, E.A. ( 2003). The quality of health care delivered to adults in the United States. New England Journal of Medicine, 348, 2635-2645.
20.
Miller, T.P., Brennan, T.A., & Milstein, A. ( 2009). How can we make more progress in measuring physicians’ performance to improve the value of care?Health Affairs, 28, 1429-1437.
21.
Mosier, C. ( 1943). On the reliability of a weighted composite. Psychometrika, 8, 161-168.
22.
Norcini, J., Lipner, R., & Kimball, H. ( 2001). The certification status of generalist physicians and the mortality of their patients after acute myocardial infarction. Academic Medicine, 76, S21-S23.
23.
Norcini, J.J., Holmboe, E.S., & Hawkins, R. ( 2008). Evaluation challenges in the era of the outcomes-based education. In E. S. Holmboe & R. Hawkins (Eds.), A practical approach to the evaluation of clinical competence (pp. 1-9). Philadelphia, PA : Mosby-Elsevier.
24.
O’Kane, M.E. ( 2007). Performance-based measures: The early results are in . Journal of Managed Care Pharmacy, 13, S3-S6.
25.
Pham, H.H., Schrag, D., Hargraves, J.L., & Bach, P.B. ( 2005). Delivery of preventive services to older adults by primary care physicians. Journal of the American Medical Association , 294, 473-481.
26.
Reeves, D., Campbell, S.M., Adams, J., Shekelle, P.G., Kontopantelis, E., & Roland, M.O. ( 2007). Combining multiple indicators of clinical quality: An evaluation of different analytic approaches. Medical Care , 45, 489-496.
27.
Rittenhouse, D.R., & Shortell, S.M. (2009). The patient-centered medical home. Journal of the American Medical Association, 301, 2038-2040.
28.
Rosenthal, M.B. ( 2007). Pay-for-performance: Will the latest payment trend improve care? Journal of the American Medical Association, 297, 740-744.
29.
Scholle, S.H., Roski, J., Adams, J.L., Dunn, D.L., Kerr, E.A., Dugan, D.P., & Jensen, R.E. ( 2008). Benchmarking physician performance: Reliability of individual and composite measures. American Journal of Managed Care, 14, 829-838.
30.
Tamblyn, R., Abrahamowicz, M., Dauphinee, W.D., Hanley, J.A., Norcini, J., Girard, N., & . . . Brailovsky, C. (2002). Association between licensure examination scores and practice in primary care. The Journal of the American Medical Association, 288, 3019-3026.
