Governments have traditionally looked to the medical profession for leadership in health planning and have charged the profession with the responsibility of establishing and monitoring standards of medical practice. Training program accreditation and licensure/certification exams have been used as the primary methods of preventing unqualified individuals from entering medical practice. Despite the critical nature of the decision made at the time of licensure/certification, there is no information about the validity of these examinations for predicting subsequent practice and health outcome. In this article, the assumptions implicit in the current use of licensing/certifying examinations are identified, the relevant evidence is reviewed, and the implications of this evidence for current methods of measurement are discussed.
Get full access to this article
View all access options for this article.
References
1.
Anderson, G. M.
, & Lomas, J. (1989). Regionalization of coronary artery bypass surgery. Medical Care, 27(3), 290-297.
2.
Avorn, J.
, & Soumerai, S. B. (1983). Improving drug-therapy decisions through educational outreach: A randomized controlled trial of academically based "detailing."New England Journal of Medicine, 308, 1457-1463.
3.
Bass, M. J.
, Buck, C., Turner, L., Dickie, G., Platt, G., & Robinson, H. C. (1986). The physician's actions and the outcome of illness in family practice. Journal of Family Practice, 23(1), 43-47.
4.
Bellamy, N.
, Brooks, P. M., Emmerson, B. T., Gilbert, J. R., Campell, J. & McCredie, M. (1989). A survey of current prescribing practices of anti-inflammatory and urate-lowering drugs in gouty arthritis in New South Wales and Queensland. Medical Journal of Australia, 151, 531-537.
5.
Bertakis, K. D.
, Roter, D., & Putnam, S. M. (1991). The relationship of physician medical interview style to patient satisfaction. Journal of Family Practice, 32(2), 175-181.
6.
Bliss, M. R.
(1981). Prescribing for the elderly. British Medical Journal, 283, 203-206.
7.
Brockway, B. S.
(1978). Evaluating physician competency: What difference does it make?Evaluation and Program Planning, 1, 211-220.
8.
Brody, D. S.
, Miller, S. M., Lerman, C. E., Smith, D. G., Lazaro, C. G., & Blum, M. J. (1989). The relationship between patients' satisfaction with their physicians and perceptions about interventions they desired and received. Medical Care, 27, 1027-1035.
9.
Brooks, R. H.
, Kamberg, C. J., Mayer-Oakes, A., Beers, M. H., Raube, K., & Steiner, A. (1989). Appropriateness of acute medical care for the elderly: An analysis of the literature. Santa Monica, CA: RAND.
10.
Buller, M. K.
, & Buller, D. B. (1987). Physicians' communication style and patient satisfaction. Journal of Health and Social Behavior, 28, 375-388.
11.
Chassin, M. R.
, Brook, R. H., Park, R. E., Keesey, J., Fink, A., Kosecoff, J., & Kahn, K. (1986). Variations in the use of medical and surgical services by the medicare population. New England Journal of Medicine, 314(5), 285-290.
12.
Cohen, M. M.
, Naylor, C. D., Basinski, A.S.H., Ferris, L. E., Llewellyn-Thomas, H. A., & Williams, J. I. (1992). Small-area variations: What are they and what do they mean?Canadian Medical Association Journal, 146(4), 467-470.
13.
Comstock, L. M.
, Hooper, E. M., Goodwin, J. M., & Goodwin, J. S. (1982). Physician behaviors that correlate with patient satisfaction. Journal of Medical Education, 57, 105-112.
14.
Cummins, R. O.
, Jarman, B., & White, R. M. (1981). Do general practitioners have different "referral thresholds"?British Medical Journal, 282, 1037-1040.
15.
Dans, E.
, Keruly, J. C., Brinker, J. A., Tabatznik, B., Pristoop, A. S., & Buchanan, S. G. (1988). Analysis of physician and hospital differences of "negative" coronary angiogram rates. Archives of Internal Medicine, 148, 2633-2637.
16.
Davis, P. B.
, & Lay Yee, R. (1990). Patterns of care and professional decision making in a New Zealand general practice sample. New Zealand Medical Journal, 103, 309-312.
17.
Durand, R. P.
, Levine, J. H., Lichtenstein, L. S., Fleming, G. A., & Ross, G. R. (1988). Teachers' perceptions concerning the relative values of personal and clinical characteristics and their influence on the assignment of students' clinical grades. Medical Education, 22, 335-341.
18.
Edwards, R. (1986, March). Keynote address. Presented to the University of Manitoba Faculty of Medicine meeting, Winnipeg.
19.
Eisenberg, J. M.
(1986). Doctor s decisions and the cost of medical care. Ann Arbor, MI: Health Administration Press Perspectives.
20.
Falvo, D.
, & Tippy, P. (1988). Communicating information to patients—Patient satisfaction and adherence as associated with resident skill. Journal of Family Practice, 26(6), 643-647.
21.
Ferguson, J. A.
(1990). Patient age as a factor in drug prescribing practices. Canadian Journal on Aging, 9, 278-295.
22.
Gerace, T. M.
, & Sangster, J. F. (1987). Factors determining patients' satisfaction in a family practice residency teaching center. Journal of Medical Education, 62, 485-490.
23.
Goran, M. J.
, Williamson, J. W., & Gonnella, J. S. (1973). The validity of patient management problems. Journal of Medical Education, 48, 171-177.
24.
Grand'Maison, P.
, Lescop, J., Rainsberry, P., & Brailovsky, C. A. (1992). Large-scale use of an objective, structured clinical examination for licensing family physicians. Canadian Medical Association Journal, 146(10), 1735-1740.
25.
Grasela, T. H., Jr.
, & Green, J. A. (1990). A nationwide survey of prescribing patterns for thrombolytic drugs in acute myocardial infraction. Pharmacotherapy, 10, 35-41.
26.
Greenfield, S.
, Cretin, S., Worthman, L. G., Dorey, F. J., Solomon, N. E., & Goldberg, G. A. (1981). Comparison of a criteria map to a criteria list in quality of care assessment for patients with chest pain: The relation of each to outcome. Medical Care, 19, 255-272.
27.
Grymonpre, R. E.
, Mitenko, P. A., Sitar, D. S., Aoki, F. Y., & Montgomery, P. R. (1988). Drug-associated hospital admissions in older medical patients. Journal ofAmerican Geriatric Society, 36, 1092-1098.
28.
Hartzema, A. G.
, & Christensen, D. B. (1983). Nonmedical factors associated with the prescribing volume among family practitioners in an HMO. Medical Care, 21, 990-1000.
29.
Headache Study Group of the University of Western Ontario
. (1986). Predictors of outcome in headache patients presenting to family physicians-A one year prospective study. Headache, 26, 235-294.
30.
Hutchinson, T. A.
, Flegal, K. M., Kramer, M. S., Leduc, D. G., & Ho Ping Kong, H. (1986). Frequency, severity and risk factors for adverse drug reactions in adult outpatients: A prospective study. Journal of Chronic Diseases, 39, 533-533.
31.
Kane, M. T.
(1985). Definitions and strategies for validating licensure examinations. In J. C. Fortune (Ed.), Understanding testing in occupational licensing (pp. 45-64). San Francisco, CA: Jossey-Bass.
32.
Kane, M. T.
(1994). Validating interpretive arguments for licensure and certification examinations. Evaluation & the Health Professions, 17(2), 133-159.
33.
Kaplan, S. H.
, Greenfield, S., & Ware, J. E. (1989). Assessing the effects of physician-patient interactions on the outcomes of chronic disease. Medical Care, 27(suppl.), 110-127.
34.
Karch, F. E.
, & Lasagna, L. (1975). Adverse drug reactions: A critical review. Journal of the American Medical Association, 234, 1236-1241.
35.
Kassebaum, D. G.
(1990). The measurement of outcomes in the assessment of educational program effectiveness. Academic Medicine, 65, 293-296.
36.
Klessig, J.
, Robbins, A. S., Wieland, D., & Rubenstein, L. (1989). Evaluating humanistic attributes of internal medicine residents. Journal of General Internal Medicine, 4, 514-521.
37.
Knickman, J. R.
, & Foltz, A. M. (1984). Regional differences in hospital utilization. Medical Care, 22, 971-986.
38.
Kopelow, M. L.
, Schnabl, G. K., Hassard, T. H., Tamblyn, R. M., Klass, D. J., Beazley, G., Hechter, F, & Grott, M. (1992). Assessment of performance in the office setting with standardized patients: Assessing practicing physicians in two settings using standardized patients. Academic Medicine, 67(10), S 19-S21. 218 Evaluation & the Health Professions / June 1994
39.
Krasny, J.
, & Ferrier, I. R. (1991). A closer look at health care in Canada. Health Affairs, 10, 152-164.
40.
LaDuca, A.
(1994). Validation of professional licensure examinations: Professions theory, test design, and construct validity. Evaluation & the Health Professions, 17(2), 178-197.
41.
Leape, L. L.
, Brennan, T. A., Laird, N., Lawthers, A. G., Localio, A. R., Barnes, B. A., Hubert, L., Newhouse, J. P., Weiler, P. C., & Hiatt% H. (1991). The nature of adverse events in hospitalized patients: Results of the Harvard Medical Practice Study II. New England Journal of Medicine, 324, 377-384.
42.
Leape, L. L.
, Park, R. E., Solomon, D. H., Chassin, M. R., Kosecoff, J., & Brook, R. H. (1989). Relation between surgeons' practice volumes and geographic variation in the rate of carotid endarterectomy. New England Journal of Medicine, 321(10), 653-657.
43.
Learnan, L. A.
, Avorn, J., Everitt, D. E., & Rosenthal, R. (1990). Pygmalion in the nursing home: The effects of caregiver expectations on patient outcomes. Journal of American Geriatric Society, 38,797-803.
44.
Lesar, T. S.
, Bricestand, L. L., Delcoure, K., Parmalee, J. C., Masta-Garnic, V., & Pohl, H. (1990).Medication prescribing errors in a teaching hospital. Journal of the American Medical Association, 263(17), 2328-2334.
45.
Lindsay, M. I., Jr.
, Hermans, P. E., Nobrega, F. T., & lstrup, D. M. (1976). Quality of care assessment I. Outpatient management of acute bacterial cystitis as the model. Mayo Clinic Proceedings, 51, 307-312.
46.
Lindsay, M. I., Jr.
, Nobrega, F. T., Offord, K. P., Carter, E. T., Rutherford, B. D., Kernel, A. I., & Mankin, H. T. (1977). Quality of care assessment. II. Outpatient medical care following hospital dismissal after myocardial infarction. Mayo Clinic Proceedings, 52, 220-227.
47.
Madeley, R. J.
, Evans, J. R., & Miur, B. (1991). The use of routine referral data in the development of clinical audit and management in north Lincolnshire. Journal of Public Health Medicine, 12(12), 22-27.
48.
Maronde, R. F.
, Lee, P. V., McCarron, M. M., & Seibert, S. (1971). Astudy of prescribing patterns. Medical Care, 9, 383-395.
49.
Mates, S.
, & Sidel, V. W. (1981). Quality assessment by process and outcome methods: Evaluation of emergency room care of asthmatic adults. American Journal of Public Health, 71,687-693.
50.
McAuley, R.
, Paul, W. M., Morrison, G. H., Beckett, R. F., & Goldsmith, C. H. (1990). Five year results of the peer assessment program of the college of physicians and surgeons of Ontario. Canadian Medical Association Journal, 143(11), 1193-1199.
51.
McAuliffe, W. E.
(1978). Studies of process outcome correlations in medical care evaluations: A critique. Medical Care, 16,907-930.
52.
McLaughlin, C. G.
, Normolle, D. P., Wolfe, R. A., McMahon, L. F., & Griffith, J. R. (1989). Small-area variation in hospital discharge rates. Medical Care, 27(5), 507-521.
53.
McPherson, K.
, Wennberg, J. E., Hovind, O. B., & Clifford, P. (1982). Small-area variations in the use of common surgical procedures: An international comparison of New England, England, and Norway. New England Journal of Medicine, 307(21), 1310-1314.
54.
Molstad, S.
, Hovelius, B., Kroon, L., & Melander, A. (1990). Prescribing of antibiotics to out-patients in hospital clinics, community health centres and private practice. European Journal of Clinical Pharmacology, 39, 9-12.
55.
Morehead, M. A.
, Daily, E. F., & Sharpiro, S. (1958). Quality of medical care provided by family physicians as related to their education, training and methods of practice. New York: Health Insurance Plan of Greater New York.
56.
Muijen, M.
, & Silverstone, T. (1987). A comparative hospital survey of psychotropic drug prescribing. British Journal of Psychiatry, 150, 501-504.
57.
Nobrega, F. T.
, Morrow, G. W., Smoldt, R. K., & Offord, K. P. (1977). Quality assessment in hypertension: Analysis of process and outcome methods. New England Journal qf Medicine, 296, 145-148.
58.
Nolan, L.
, & O'Malley, K. (1988). Prescribing for the elderly. Part 1: Sensitivity of the elderly to adverse drug reactions. Journal of American Geriatric Society, 36, 142-149.
59.
Norcini, J. J.
(1994). Research on standards for professional licensure and certification examinations. Evaluation & the Health Professions, 17(2), 160-177.
60.
Norcini, J. J.
, Lipner, R. S., Benson, J. A., Jr., & Webster, G. D. (1985). An analysis of the knowledge base of practicing internists as measured by the 1980 recertification examination. Annuals of Internal Medicine, 102, 385-389.
61.
Norcini, J. J.
, Meskauskas, J. A., Langdon, L. O., & Webster, G. D. (1986). An evaluation of a computer simulation in the assessment of clinical competence. Evaluation & the Health Professions, 9(3), 286-290.
62.
Palmer, R. H.
, & Reilly, M. C. (1979). Individual and institutional variables which may serve as indicators of quality of medical care. Medical Care, 17(7), 693-736.
63.
Pasley, B.
, Vernon, P., Gibson, G., McCauley, M., & Andoh, J. (1987). Geographic variations in elderly hospital and surgical discharge rates, New York State. American Journal of Public Health, 77(6), 679-684.
64.
Paul-Shaheen, P.
, Clark, J. D., & Williams, D. (1987). Small area analysis: Areview and analysis of the North American literature. Journal of Health Politics, Policy and Law, 12(4), 741-809.
65.
Pineault, R.
(1977). The effect of medical training factors on physician utilization behaviour. Medical Care, 15(1), 52-67.
66.
Popham, W. J.
, & Shapiro, D. (1978). Guest editorial: Measurement requisites for competency assurance in the health professions. Evaluation & the Health Professions, 1, 9-15.
67.
Ramsey, P. G.
, Carline, J. D., Inui, T. S., Larson, E. B., LoGerfo, J. P., & Weinrich, M. D. (1989). Predictive validity of certification by the American Board of Internal Medicine. Annals of Internal Medicine, 110, 719-726.
68.
Rethans, J. J.
, Sturmans, F., Drop, R., & van der Vleuten, C. (1990). Performance and competence of general practitioners. In I. Hart & R. Harden (Eds.), Further developments in the assessment of clinical competence. Montreal: Heal Publications.
69.
Rethans, J.J.E.
, & Van Boyen, C.P.A. (1987). Simulated patients in general practice: A different look at the consultation. British Medical Journal, 294, 809-812.
70.
Rhee, S.
(1976). Factors determining the quality of physician performance in patient care. Medical Care, 14, 733-738.
71.
Roos, L. L.
(1983). Supply, workload and utilization: A population-based analysis of surgery in rural Manitoba. American Journal of Public Health, 73(4), 414-421.
72.
Roos, L. L.
, Fisher, E. S., Sharp, S. M., Newhouse, J. P., Anderson, G., & Bubolz, T. A. (1990). Postsurgical mortality in Manitoba and New England. Journal of the American Medical Association, 263(18), 2453-2458.
73.
Roos, L. L.
, & Sharp, S. M. (1989). Innovation, centralization, and growth coronary arterybypass graft surgery in Manitoba. Medical Care, 27(5), 441-452.
74.
Roos, N. P.
(1980). Impact of the organization of practice on quality of care and physician productivity. Medical Care, 18(4), 347-359.
75.
Roos, N. P.
(1984). Hysterectomy: Variations in rates across small areas and across physicians' practices. American Journal of Public Health, 74(4), 327-335.
76.
Roos, N. P.
(1989). Predicting hospital utilization by the elderly: The importance of patient, physician, and hospital characteristics. Medical Care, 27(10), 905-919.
77.
Roos, N. P.
, Flowerdew, G., Wajda, A., & Tate, R. B. (1986). Variations in physicians' hospitalization practices: A population-based study in Manitoba, Canada. American Journal of Public Health, 76(1), 45-51.
78.
Roos, N. P.
, & Roos, L. L. (1981). High and low surgical rates: Factors for area residents. American Journal of Public Health, 71(6), 591-600.
79.
Roos, N. P.
, & Roos, L. L. (1982). Surgical rate variations: Do they reflect the health or socioeconomic characteristics of the population?Medical Care, 20(9), 945-959.
80.
Sackett, D. L.
, Haynes, R. B., & Tugwell, P. (1985). Clinical epidemiology: A basic science for clinical medicine. Boston/Toronto: Little, Brown.
81.
Sampalis, J. S.
(in press). The surgeon-effect: Potential impact in surgical trials. American Journal of Surgery.
82.
Sanz, E. J.
, Bergman, U., & Dahlstrom, M. (1989). Paediatric drug prescribing. A comparison of Tenerife (Canary Islands, Spain) and Sweden. European Journal of Clinical Pharmacology, 37,65-68.
83.
Schneeweiss, R.
, Ellsbury, K., Hart, G., & Geyman, J. P. (1989). The economic impact and multiplier effect of a family practice clinic on an academic medical center. Journal of the American Medical Association, 262(3), 370-375.
84.
Schwenk, T. L.
, Marquez, J. T., Lefever, R. D., & Cohen, M. (1989). Physician and patient determinants of difficult physician-patient relationships. Journal of Family Practice, 28, 59-63.
85.
Soumerai, S. B.
, McLaughlin, T. J., & Avorn, J. (1989). Improving drug prescribing in primary care: A critical analysis of the experimental literature. Milbank Quarterly, 67, 268-317.
86.
Starfield, B.
, & Scheff, D. (1972). Effectiveness of pediatric care: The relationship between processes and outcome. Pediatrics, 49, 547-552.
87.
Subcommittee on Evaluation of Humanistic Qualities in the Internist
. (1985). A guide to awareness and evaluation of humanistic qualities in the internist. Philadelphia, PA: American Board of Internal Medicine.
88.
Tamblyn, R. M., Abrahamowicz, M., Schnarch, B., Colliver, J., Benaroya, S., & Snell, L. (in press). Can standardized patients predict real patient satisfaction with the doctor-patient relationship? Teaching and Learning in Medicine.
89.
Tamblyn, R. M.
, & Battista, R. N. (1993). Changing clinical practice. Journal of Continuing Education in the Health Professions, 13, 273-288.
90.
Tamblyn, R. M.
, Benaroya, S., Snell, L., McLeod, P., Schnarch, B., & Abrahamowicz, M. (in press). The feasibility and value of using patient satisfaction ratings to evaluate medical residents and clerks. Journal of General Internal Medicine.
91.
Tamblyn, R. M., McLeod, P., Sampalis, J., Abrahamowicz, M., & Dauphinee, W. D. (1993). Training and practice research team [Final report for Project #911403-102]. Available from Fonds de la Recherche de Santé du Québec, 550 Sherbrooke West, Montreal, Quebec H3A 1B9, Canada.
92.
Thomas, K. B.
(1987). General practice consultations: Is there any point in being positive? BritishMedical Journal, 294, 1200-1202.
93.
Tugwell, P.
(1979). A methodological perspective on process measures of the quality of medical care. Clinical & Investigative Medicine, 2, 113-121.
94.
Turnbull, J. M.
(1989). What is... normative versus criterion-referenced assessment. Medical Teacher, 11, 145-150.
95.
Uhlmann, R. F.
, Inui, T. S., Pecoraro, R. E., & Carter, W. B. (1988). Relationship of patient request fulfillment to compliance, glycemic control, and other health care outcomes in insulin-dependent diabetes. Journal of General Internal Medicine, 3, 458-463.
96.
Wartman, S. A.
, Morlock, L. L., Malitz, F. E., & Palm, E. A. (1983). Patient understanding and satisfaction as predictors of compliance. Medical Care, 21, 886-891.
97.
Wennberg, J.
, & Gittelsohn, A. (1991). Small area variations in health care delivery. Science, 182, 1102-1123.
98.
Wennberg, J. E.
(1983). On patient need, equity, supplier-induced demand, and the need to assess the outcome of common medical practices. Medical Care, 23(5), 512-520.
99.
Wennberg, J. E.
, Barnes, B. A., & Zubkoff, M. (1982a). Professional uncertainty and the problem of supplier-induced demand. Social Science and Medicine, 16, 811-824.
100.
Wennberg, J. E.
, Barnes, B. A., & Zubkoff, M. (1982b). Professional uncertainty and the problem of supplier-induced demand. Social Science and Medicine, 16, 811-824.
101.
Wessling, A.
, Boethius, G., & Sjoqvist, F. (1990). Prescribing monitoring of drug dosages in the county of Jamtland and Sweden as a whole in 1976,1982,1985. European Journal of Clinical Pharmacology, 38,329-334.
102.
Wilkin, D.
, & Smith, A. G. (1987). Variation in general practitioners' referral rates to consultants. Journal of the Royal College of General Practitioners, 37, 350-353.
103.
Winefield, H. R.
, & Murrell, T.G.C. (1991). Speech patterns and satisfaction in diagnostic and prescriptive stages of general practice consultations. British Journal of Medical Psychology, 64, 103-115.
104.
Wooliscroft, J. O.
, Calhoun, J. G., Billiu, G. A., Stross, J. K., MacDonald, M., & Templeton, B. (1989). House officer interviewing techniques: Impact on data elicitation and patient perceptions.Journal of General Internal Medicine, 4, 108-114.
105.
Young, M. J.
, Fried, L. S., Eisenberg, J., Hershey, J., & Williams, S. (1987). Do cardiologists have higher thresholds for recommending coronary arteriography than family physicians?Health Services Research, 22(5), 623-635.
