Assessments of clinical skills of medical students rely increasingly on standardized patients demonstrating medical cases with faculty rating performance. The common finding of inconsistency of scores across cases is often referred to as case specificity. A multivariate generalizability model reveals that overall case specificity cannot explain inconsistency in this example. Results from the multivariate model suggest directions for enhancing the table of specifications, optimizing scoring, and improving case selection to reduce inconsistency in scores across cases.
Get full access to this article
View all access options for this article.
References
1.
Barrows, H. S. (1987). Simulated (standardized) patients and other human simulations. Chapel Hill, NC: Health Sciences Consortium.
2.
Boodoo, G. M., & O’Sullivan, P. (1982). Obtaining generalizability coefficients for clinical evaluations. Evaluation & the Health Professions, 5, 345-358.
3.
Brailovsky, C. A., & Grand’Maison, P. (2000). Using evidence to improve evaluation: A comprehensive psychometric assessment of a SP-based OSCE licensing examination. Advances in Health Sciences Education, 5, 207-219.
4.
Brennan, R. L. (1998). Raw-score conditional standard errors of measurement in generalizability theory. Applied Psychological Measurement, 22, 307-331.
5.
Brennan, R. L. (2000). Performance assessments from the perspective of generalizability theory. Applied Psychological Measurement, 24, 339-353.
6.
Brennan, R. L., & Johnson, E. G. (1995). Generalizability of performance assessments. Educational Measurement: Issues & Practice, 14, 9-12.
7.
Colliver, J. A., Markwell, S. J., Vu, N. V., & Barrows, H. S. (1990). Case specificity of standardized-patient examinations: Consistency of performance on components of clinical competence within and between cases. Evaluation & the Health Professions, 13, 252-261.
8.
Colliver, J. A., & Swartz, M. H. (1997). Assessing clinical performance with standardized patients. JAMA, 278, 790-791.
9.
Colliver, J. A., Verhulst, S. J., Williams, R. G., & Norcini, J. J. (1989). Reliability of performance on standardized patient cases: A comparison of consistency measures based on generalizability theory. Teaching and Learning in Medicine, 1, 31-37.
10.
Colliver, J. A., & Williams, R. G. (1993). Technical issues: Test application. Academic Medicine, 68, 454-463.
11.
De Champlain, A. F., Margolis, M. J., King, A., & Klass, D. J. (1997). Standardized patients’ accuracy in recording examinees’ behaviors using checklists. Academic Medicine, 72, S85-S87.
12.
Fitzgerald, J. T., Wolf, F. M., Davis, W. K., Barclay, M. L., Bozynski, M. E., Chamberlain, K. R., et al. (1994). A preliminary study of the impact of case specificity on computer-based assessment of medical student clinical performance. Evaluation & the Health Professions, 17, 307-321.
13.
Graybill, F. A. (1969). Introduction to matrices with applications in statistics. Belmont, CA: Wadsworth.
14.
Gross, L. J. (1994). Interrater reliability reconsidered: Performance assessment using one examiner per candidate. Evaluation & the Health Professions, 17, 465-484.
15.
Jarjoura, D. (1981). Variance components models for content and passage effects in the English usage and natural sciences subtests of the ACT Assessment Program. ACT Technical Bulletin, 39. Iowa City: The American College Testing Program.
16.
Jarjoura, D. (1983). Best linear prediction of composite universe scores. Psychometrika, 48, 525-539.
17.
Jarjoura, D. (1985). Tolerance intervals for true scores. Journal of Educational Statistics, 10, 1-17.
18.
Jarjoura, D. (1986). An estimator of examinee-level measurement error variance that considers test form difficulty adjustments. Applied Psychological Measurement, 10, 175-186.
19.
Jarjoura, D., & Brennan, R. L. (1982). A variance components model for measurement procedures associated with a table of specifications. Applied Psychological Measurement, 6, 161-171.
20.
Jarjoura, D., & Brennan, R. L. (1983). Multivariate generalizability models for tests developed from tables of specification. In L. J. Fyans, Jr. (Ed.), Generalizability theory: Inferences and practical applications. San Francisco: Jossey-Bass.
21.
Kendall, M., & Stuart, A. (1977). The advanced theory of statistics: Distribution theory (Vol. 1). New York: Macmillan.
22.
Lee, G., & Frisbie, D. (1999). Estimating reliability under a generalizability theory model for test scores composed of testlets. Applied Measurement in Education, 12, 237-255.
23.
Lee, G., Dunbar, S., & Frisbie, D. (2001). The relative appropriateness of eight measurement models for analyzing scores from tests composed of testlets. Educational and Psychological Measurement, 61, 958-975.
24.
Linn, R., & Burton, E. (1994). Performance-based assessments: Implications of task specificity. Educational Measurement: Issues and Practice, 13, 5-8.
25.
Marcoulides, G. (1996). Estimating variance components in generalizability theory: The covariance structure analysis approach. Structural Equation Modeling, 3, 290-299.
26.
Moore, A. D., & Young, S. (1997, October). Clarifying the blurred image: Estimating the interrater reliability of performance assessments. Presented at the annual meeting of the Northern Rocky Mountain Educational Research Association, Jackson, WY.
27.
Prislin, M. D., Giglio, M., Lewis, E. M., Ahearn, S., & Radecki, S. (2000). Assessing the acquisition of core clinical skills through the use of serial standardized patient assessments. Academic Medicine, 75, 480-483.
28.
Rajaratnam, N., Cronbach, L. J., & Gleser, G. C. (1965). Generalizability of stratified parallel tests. Psychometrika, 30, 39-56.
29.
Regehr, G., Freeman, R., Hodges, B., & Russell, L. (1999). Assessing the generalizability of OSCE measures across content domains. Academic Medicine, 74, 1320-1322.
30.
SAS. (2001). Proc mixed. Cary, NC: Author.
31.
Sireci, S. G., Thissen, D., & Wainer, H. (1991). On the reliability of testlet-based tests. Journal of Educational Measurement, 28, 237-247.
32.
Swanson, D. B., & Norcini, J. J. (1989). Factors influencing reproducibility of tests using standardized patients. Teaching and Learning in Medicine, 1, 158-166.
33.
Swartz, M. H., Colliver, J. A., Bardes, C. L., Charon, R., Fried, E. D., & Moroff, S. (1997). Validating the standardized-patient assessment administered to medical students in the New York City Consortium. Academic Medicine, 72, 619-626.
34.
Tamblyn, R. M., Klass, D. K., Schanbl, G. K., Kopelow, M. L. (1990). Factors associated with the accuracy of standardized patient presentation. Academic Medicine, 65, S55-S56.
35.
van der Vleuten, C. P., & Swanson, D. B. (1990). Assessment of clinical skills with standardized patients: State of the art. Teaching & Learning in Medicine, 2, 58-76.
36.
Wainer, H., & Lukhele R. (1997). How reliable are TOEFL scores? Educational and Psychological Measurement, 57, 741-758.
37.
Wang, M. W., & Stanley, J. C. (1970). Differential weighting: A review of methods and empirical studies. Review of Educational Research, 40, 663-705.
