Reliability and Validity of A Procedure to Measure Diagnostic Reasoning and Problem-Solving Skills Taught in Predoctoral Orthodontic Education

Abstract

The purpose of this study was to assess the reliability and validity of a procedure to measure diagnostic-reasoning and problemsolving skills taught in predoctoral orthodontic education. The procedure consisted of stimulus material and a constant set of 33 multiple-choice items. Intercase correlations obtained from 71 second-year dental students rangedfrom .41 to .85 and average .60. These correlations changed by at most .05 when different measures of general achievement (DA T, undergraduate GPA and freshman Dental GPA) were partialed out, suggesting that only a small portion of the relationship between cases can be attributed to general achievement. The correlations of attendance with test scores ranged from .06 to .43. These data were interpreted as supporting the reliability and construct validity of the measurement procedure. The experimental testing method has a number of features that are appealing, including a relatively realistic testing situation, ease of making case changes, and ability lo be administered to large groups. It was concluded that the experimental procedure offers a viable method of assessing problem-solving skills in orthodontics and may have applications in other areas.

Get full access to this article

View all access options for this article.

References

Case, S. M. , Swanson, D. B. , & Stillman, P. L. (1988). Evaluating diagnostic pattern recognition: The psychometric characteristics of a new item format. In Proceedings of the 27nd Annual Conference on Research in Medical Education (pp. 3-8). Chicago, IL: Association of American Medical Colleges.

Elstein, A. S. , Shulman, L. S. , & Sprafka, S. A. (1978). Medical problem solving: An analysis of clinical reasoning. Cambridge, MA: Harvard University Press.

Hively, W. , Maxwell, G. , Rabehl, G. , Sension, D. , & Lundin, S. (1973). Domain-referenced curriculum evaluation: A technical handbook and a case study from the minnemast project. In M. C. Alkin (Ed.), CSE Monograph Series in Evaluation, 1. Los Angeles: Center for the Study of Evaluation, University of California.

Jacobs, R. M. (1988). Ten-year study of strategies for teaching clinical inference in predoctoral orthodontic education. Journal of Dental Education, 52, 235-244.

McGuire, C. , & Babbott D. (1967). Simulation technique in the measurement of problem-solving skills. Journal of Educational Measurement, 4, 1-10.

Mehrens, W. A. , & Lehman, I. J. (1973). Measurement and evaluation in education and psychology. New York: Holt, Rinehart & Winston.

Norman, G. R. (1989). Educational research and the changing definition of competence. National Board Examiner, 36(4), 2-5.

Norman, G. R. , Feightner, J. W. , Tugwell, P. , Muzzin, I. J. , & Guyatt, G. (1983). The generalizability of measures of clinical problem-solving. In Proceedings of the 22ndAnnual Conference on Research in Medical Education (pp. 110-114). Washington, DC: Association of American Medical Colleges.

Swanson, D. B. (1987). A measurement framework for performance-based tests. In I. R. Hart , R. M. Harden , & R. J. Walton (Eds.), Further developments in assessing clinical competence (pp. 13-45). Montreal: Can-Heal Publications.

10.

Vu, N. , Dawson-Saunders, B. , & Barrows, H. (1987). Use of a medical reasoning aptitude test to help predict performance in medical school. Journal of Medical Education, 62, 325-335.

11.

Webster, G. D. , Shea, J. A. , Norcini, J. J. , Grosso, L. J. , & Swanson, D. B. (1988). Strategies in comparison of methods for scoring patient management problems: Use of external criteria to validate scores. Evaluation & the Health Professions, 11, 231-248.