Abstract
The American College of Veterinary Pathologists commissioned a role delineation survey to define the specialized tasks, knowledge, and tools that define the current practice of veterinary clinical pathology and veterinary anatomic pathology. The survey also identified when competence was acquired for each task (i.e., before certification or after certification). The response rate by diplomates was high, with approximately 50% of practicing pathologists within each specialty responding to each survey. Using the survey results, all tasks for each specialty were classified as either appropriate or unsuitable for testing in the certifying examinations. The role delineation survey data will facilitate the creation of test plans that objectively define the content in each certifying examination, the evaluation and enhancement of training curricula, and the optimization of continuing education opportunities for practicing veterinary pathologists.
Keywords
In 2006, the Council of the American College of Veterinary Pathologists (ACVP) determined that a study should be performed to define the roles of practicing veterinary anatomic pathologists and veterinary clinical pathologists. This activity was intended to provide objective data to be used 1) to demonstrate that the certification examinations are fair, relevant, and legally defensible; 2) to improve the curricula of training programs in veterinary pathology; and 3) to guide the creation of continuing education programs for established veterinary pathologists. A Role Delineation Task Force of 9 ACVP diplomates comprising 6 anatomic pathologists and 3 clinical pathologists was selected by the Council to design and oversee the study.
In response to requests from the ACVP, 5 consulting firms with recognized credentials in professional testing services provided proposals. In 2007, the ACVP engaged CASTLE Worldwide to help define the current roles, tasks, and requisite knowledge of practicing veterinary anatomic pathologists and veterinary clinical pathologists.
Methods
To define the questions to be used in the role delineation surveys, separate, face-to-face focus groups of ACVP diplomates from each veterinary pathology specialty and with representation from diverse areas of practice were held in November 2007. Focus group members are listed in Table 1. CASTLE staff led by Dr. James Henderson, the supervising consultant, facilitated these focus groups. Working separately and in combination, the anatomic pathology and clinical pathology focus groups defined the relevant Elements of Practice (broad activities of the profession), Tasks (more specific activities within each Element of Practice), Foundational Sciences (broad disciplines of importance to proficient professional practice), Specific Sciences (more specific areas of knowledge within a Foundational Science), and Tools (methods and instruments employed in undertaking their regular Tasks) used by veterinary pathologists in the current professional practice of each specialty. These groups also provided recommendations on demographic questions to be used to better understand and use survey responses.
ACVP role delineation focus groups.
∗Also a member of the Role Delineation Task Force.
The information from the focus group sessions was used by Dr. Henderson and the Role Delineation Task Force to create the ACVP role delineation surveys for each specialty. All active ACVP diplomates were asked to complete the survey for their specialty during February and March 2008. Diplomates certified in both veterinary anatomic pathology and veterinary clinical pathology were asked to complete both surveys. Diplomates were instructed to answer each question in the context of the respondent's current role and practice. For this survey, current roles, tasks, and skills are those used in today's practice, regardless of their age, novelty, or complexity. Objective information regarding the current roles and required knowledge of individual practicing pathologists offers the best guidance for design of relevant educational programs and selection of appropriate examination content. Diplomates were specifically instructed not to offer their opinions on what tasks, knowledge, and tools might be important to the practice of veterinary pathology in the future. Emeritus (retired) ACVP diplomates were excluded from the survey because responses based on their past roles or opinions might not accurately reflect the actual current practice of the veterinary pathology specialties.
Usable responses were received from 713 of 1,237 anatomic pathology diplomates and 133 of 266 clinical pathology diplomates, resulting in response rates equaling or exceeding 50%. The testing consultant expressed confidence that this high response rate would produce reliable data.
The rating scales for each category of question used in the surveys (Elements of Practice, Tasks, Foundational Sciences, Specific Sciences, and Tools) are provided in Table 2. Mean scores and standard deviations were calculated for each survey item. Mean ratings from all diplomates and from a subset consisting of diplomates certified in 2003–2007 were calculated separately. Data provided by diplomates certified recently (since 2003) were analyzed based on the premise that information acquired from recently qualified practitioners may more accurately reflect the demographics and real-world roles of entry-level veterinary pathologists than would responses obtained from all diplomates. In general, responses from all diplomates and recent diplomates were similar. A posthoc comparison of results obtained from diplomates employed by academic institutions and by industrial firms was undertaken to determine the similarities and differences in roles of veterinary pathologists in these two broad practice settings. This latter evaluation was performed to examine potential differences in training and certification needs of pathologists practicing in these two environments.
Rating choices available to survey respondents.
Survey Results
The questions asked and the results obtained from the role delineation surveys for each veterinary pathology specialty are available in the Role Delineation Study Report on the ACVP Role Delineation and Training Program Development web page (http://www.acvp.org/roledelin/index.php). Detailed survey results from all diplomates and from recent (2003–2007) diplomates of each specialty are listed separately. In addition, comparisons of results from diplomates of each specialty employed by academic institutions and industry are available.
Utilization of the Role Delineation Data
The ACVP will use the results of the role delineation surveys to perform several distinct but overlapping functions of great importance to the organization.
Generation of objective specifications for crafting certifying examinations
One obvious use for the role delineation data is the creation of written Test Plans; that is, examination specifications that guide the selection and distribution of examination content for each specialty. Such Test Plans define the tasks that should be evaluated during the certifying examinations and the relative weight or emphasis of examination content to be assigned to each task. Written Test Plans are a critical means of ensuring that certifying examinations are objective measures of proficiency in important job-related tasks, and therefore are legally defensible.
Anatomic Pathology and Clinical Pathology Test Plan Working Groups were appointed by the ACVP Council in the fall of 2008 to create separate Test Plans for each specialty (Table 3). Most members had ACVP Examination Committee experience. Two members from each Working Group also had served on the ACVP Role Delineation Task Force to provide continuity. The 3 charges given to these working groups were to:
Set threshold criteria based on role delineation survey results to define tasks and knowledge that should be evaluated within the certifying examinations for veterinary clinical pathology and veterinary anatomic pathology. The criteria were established to emphasize specific tasks because legal precedent and scientific literature indicate that the most suitable data to use in defining appropriate content for professional certifying examinations are current tasks. Identify tasks and knowledge that should not be evaluated within the formal certifying examinations for veterinary clinical pathology and veterinary anatomic pathology. Tasks may be excluded based on role delineation survey data or because some important tasks cannot be adequately evaluated in a traditional examination setting. For the tasks that should be assessed in the certifying examinations, develop a systematic process to determine the relative distribution (emphasis) of each task within the examination content.
Test plan working groups.
∗Served on the ACVP Role Delineation Task Force.
†Experience on the ACVP Examination Committee.
The Test Plan Working Groups identified those tasks that should and should not be assessed in each examination based on role delineation survey results. The tasks that should be included in the clinical pathology and anatomic pathology certifying examinations are listed in Table 4 and Table 5, respectively. Tasks that should not be included in the certifying examinations are listed in Table 6 (clinical pathology) and Table 7 (anatomic pathology). The criteria used to define tasks and knowledge to be included in certifying examinations are explained in the Test Plans that are available to ACVP members, training faculty, candidates, students, and the general public (http://www.acvp.org/roledelin/index.php).
Clinical pathology tasks to be tested in the certifying examination.
Anatomic pathology tasks to be tested in the certifying examination.
Clinical pathology tasks identified as unsuitable for evaluation by examination.
Anatomic pathology tasks identified as unsuitable for evaluation by examination.
Selecting the content of certifying examinations
Following approval by the ACVP Council, the Test Plans will guide future ACVP Examination Committees during the selection of test items for certifying examinations. Test Plans based on objective role delineation data will allow the Examination Committee to select appropriate examination content, demonstrate that content is relevant to the current practice of veterinary pathology, effectively defend the content of the certifying examinations in response to potential legal challenges, and design and implement improvements to examination content, processes, and structure. The Test Plans and Role Delineation Survey Report will be used by the Examination Committee to:
Assign each examination question to a specific task included in the Test Plans. Create and select new questions that meet the specifications of the Test Plans. Exclude questions from examinations that do not meet the specifications cited in the Test Plans. Balance the relative distribution of tasks covered in examination content as detailed in the Test Plans. Retrieve archived questions by task and other relevant information for possible use on future examinations. In fact, this capability will be an essential means of standardizing the examinations from year to year, and it should also lessen the work load on the Examination Committee by reducing the number of new questions that must be written for each examination.
As the Examination Committee uses the Test Plans, opportunities to improve the format and structure of future examinations should be considered.
Improvement of veterinary pathology education
Using role delineation data and the Test Plans, training coordinators and mentors in veterinary pathology training programs will be able to improve the educational experience for their trainees. Access to objective role delineation data will allow training program faculty to:
Optimize theoretical (knowledge-based) and applied (task-oriented) activities to better prepare trainees for the certifying examination and successful practice of veterinary pathology. Identify and fill gaps in training curricula to provide a more balanced and complete educational program. Establish standardized means for assessing proficiency for pathology tasks that are essential for entry-level competence but that cannot be reliably or readily tested in a written examination format.
Objective and accessible Test Plans should assist educators in maximizing the students' learning experiences while simultaneously easing the burden on training faculty.
Improvement of continuing education for ACVP diplomates
Role delineation survey information will assist the ACVP in identifying and providing the most relevant continuing education opportunities that support the most important roles for practicing veterinary pathologists in each specialty. Tasks essential to the effective practice of veterinary pathology that are mastered after certification are important topics for continuing education programs.
Defense of the ACVP
A major concern of the ACVP Council in this litigious age is to protect the organization and its members from the stress and expense of lawsuits, especially those that might be brought by individuals who have been unsuccessful in their attempt(s) to pass a veterinary pathology certifying examination. The definition of the key roles and tasks of veterinary pathologists using objective role delineation data, the existence of written test specifications (the Test Plans) based on this information, and the regular use of the Test Plans in creation of the certifying examinations will facilitate defense of the ACVP by demonstrating that the examination content is relevant and appropriate to the current practice of each veterinary pathology specialty. The role delineation survey data and Test Plans also will assist the ACVP in creating and executing a documented process to objectively set a fair and reasonable minimal passing score (“cut score”) for each certifying examination. Finally, the experience gained in the current endeavor will allow the ACVP Council to oversee future efforts to design and execute new role delineation surveys on an intermittent basis to evaluate shifts in the current roles of veterinary pathologists and to predict changes in the professional roles of veterinary pathologists in the future.
Looking Forward
The role delineation documents and Test Plans should be reviewed once every few years to determine whether new surveys are needed to capture shifts in the professional roles and responsibilities of practicing veterinary pathologists. Based on these reviews, role delineation surveys likely should be repeated every 3 to 7 years; rapid changes in the profession would justify more frequent surveys. Each time new role delineation surveys are available, the Test Plans should be updated or re-created to reflect the evolution of the tasks, knowledge, and tools required in the current practice of veterinary clinical pathology and veterinary anatomic pathology.
The role delineation surveys conducted in 2008 were specifically designed to evaluate the current roles of veterinary pathologists. The Role Delineation Task Force determined that asking questions about both the present roles and the future of the profession in the same survey instrument would further complicate a survey that already was quite complex. Accordingly, the ACVP may want to consider creating another survey to explore future trends arising within the profession. Such a “forward-looking” analysis likely would provide information to enhance educational programs intended to prepare trainees and current diplomates for future roles in veterinary pathology.
