Sage Journals: Discover world-class research

Abstract

The American Board of Toxicology (ABT), in consultation with ACT Credentialing & Career Services (ACT), performed a practice analysis study of general toxicology in 2020-21. This work follows up on an initial practice analysis commissioned by the ABT and conducted in 2014-2015, results of which were published in 2016. The purpose of the current, second-generation study was to update and validate the existing process-based delineation of practice of general toxicologists, including major domains of responsibility and tasks performed in practice. In addition, the study included the review, update, and validation of the knowledge areas required by toxicologists developed by subject-matter experts (SMEs) that have been used for ABT examination development initiatives. Consistent with best practices in the field of credentialing, ABT also contracted with ACT to conduct 2 follow-on activities: a study to evaluate the reliability of a reduced-length ABT examination and a standard setting study to establish a valid passing score for the updated examination. In addition to informing ongoing ABT certification examination and question writing activities, it is anticipated that the results of this practice analysis will be of value to those responsible for developing graduate and undergraduate toxicology curricula, creating continuing education content, and authoring textbooks covering the contemporary practice of toxicology.

Keywords

practice analysis toxicology credentialing professional competence board certification

Introduction

The American Board of Toxicology (ABT) was incorporated in 1979 with the purpose of establishing standards of competence in the professional practice of toxicology and identifying those who meet these standards. Along with meeting eligibility criteria, individuals must pass the ABT certification examination. Successful candidates are awarded the Diplomate of the American Board of Toxicology® (DABT) credential. The first examination was administered in 1980. In the intervening years, a total of 4068 toxicologists have earned certification; as of January 1, 2023, there were 2631 currently certified ABT Diplomates worldwide.

The mission of ABT is to identify, maintain, and evolve a standard for professional competence in the field of toxicology; the ABT vision is to establish a globally recognized credential in toxicology that represents competency and commitment to human health and the environment. As the scientific discipline of toxicology advances, the ABT Board of Directors (BoD) is committed to keeping the certification program current and relevant to the present-day practice of toxicology. Ensuring that the content of the ABT examination accurately reflects the knowledge required in the professional practice of toxicology is fundamental to meeting the standards and recommendations outlined by third party accreditation standards.^1-3

In 2014, the ABT BoD, in collaboration with the Society of Toxicology (SOT), contracted Professional Examination Service (now ACT Credentialing & Career Services or ACT) to conduct a practice analysis of toxicology to determine the knowledge and competencies needed to practice as a general toxicologist, with results used to develop and validate the blueprint for the ABT certification examination. In the context of credentialing, validity is defined as “the degree to which accumulated evidence supports outcome decisions made with respect to all requirements for obtaining a credential (e.g., education, experience, and assessment instruments).” The 2014 practice analysis study identified and validated 6 practice domains (the major practice areas performed by toxicologists) with the corresponding tasks (the activities performed within a practice domain), and knowledge required to perform the tasks.⁴

The first certification examination based on the practice analysis-based test specifications was administered in 2017. In conjunction with the new examination outline, the examination was also reduced from a 300-question exam administered in three 3-hour segments over one and a half days to a 200-question examination administered in two 3-hour time segments on a single day.

This paper describes ABT activities and developments in 3 major areas since 2016: 1) the second professional practice analysis study and the resulting updated test specifications; 2) the test length and reliability study; and 3) the standard setting study.

The Practice Analysis Study

In keeping with certification industry best practices which require conducting practice analyses at regular intervals, the ABT BoD contracted with ACT in 2020 to update and validate the 2014 practice analysis. A properly conducted and documented practice analysis study is central to the construction of a valid and defensible examination. Both the study and the development of test specifications were performed by ACT in accordance with credentialing industry standards, guidelines, and best practices and met all internationally recognized criteria.^1-3

The practice analysis was conducted in 2 general phases—a qualitative update of the domains, tasks, and knowledge by subject matter experts (SMEs), and a quantitative phase using a large-scale survey of ABT Diplomates to collect data, assess/refine, and validate the elements of the updated delineation. Following finalization of the updated delineation, empirically derived test specifications (i.e., the percentage of the exam focused on content in each domain or subdomain) were established using data from the validation survey.

Qualitative Update: Methodology

The qualitative assessment began with a focus group of thought leaders familiar with the exam. The focus group concentrated on perceived strengths and weaknesses of the existing content outline, including identifying gaps and areas of redundancy/overlap across existing domains and tasks. The focus group also provided information about major trends in the profession, recent and anticipated changes in the roles and work functions of toxicologists, and the impact of these changes on the competencies and knowledge required by general toxicologists. The focus group output was used as a resource by the Practice Analysis Task Force (PATF).

The major work of the practice analysis was performed by the ABT BoD-appointed 10-member PATF. Members were SMEs representative of a range of toxicology practice settings, roles, educational backgrounds, areas of expertise, and experience levels. ACT facilitated ten 2-hour virtual meetings of the PATF, with preparatory work conducted by committee members between meetings. The PATF updated the delineation through an iterative process, constantly refining and augmenting drafted practice-specific domains, tasks, and knowledge. Interim versions of the work product were circulated to PATF members after each meeting for review and comment.

The first set of 5 virtual PATF meetings resulted in an initial draft update to the domain/task/knowledge delineation (a.k.a. exam content outline). Following these, ACT circulated this draft delineation to additional external SMEs to perform an independent peer review to critically evaluate the work in progress. The reviewers' feedback informed the PATF’s subsequent discussions during their next series of 3 virtual meetings to further refine and finalize the delineation. A subgroup of the task force participated in 2 additional meetings to update the lengthy knowledge list. A graphic depiction of the development process and names of PATF members, thought leaders, and SMEs is found in Figure 1.

Figure 1.

Task force overview and members.

Qualitative Update: Results

As a result of the iterative development process, the delineation was re-organized into a structure consisting of 5 domains, 2 of which had component sub-domains, 57 tasks and 140 knowledge topics. While the current study largely confirmed the results of the previous practice analysis, several changes occurred. Domain 1. Design, Execute and Interpret Toxicological Studies was divided into 3 subdomains. Based on focus panel feedback from ABT examination committee members related to overlapping content, and following discussion over multiple PATF meetings, Domain 2. Descriptive Toxicology: Environmental, Clinical, Non-clinical and Forensic Investigations was eliminated as a stand-alone content area. Overlapping tasks in the domain were shifted to domains where content was logically subsumed or were combined with existing tasks. Four of the domains were deemed testable on the exam. A fifth domain, Contribution to the Profession, was identified as contributing to a rich description of general toxicology practice. However, because it outlined activities beyond the point of credentialing, the PATF did not delineate knowledge in this domain. An overview of the updated delineation is provided in Table 1.

Table 1.

Updated Delineation of Practice of Toxicology.

Domains/Subdomains	Task Statements	Knowledge Topics
1. Conduct of Toxicological Studies	13	31
A. Design	4	15
B. Execute	5	8
C. Interpret	4	8
2. Mechanistic Toxicology	8	27
3. Risk Assessment	15	55
A. Hazard Identification	4	16
B. Exposure Assessment	4	11
C. Dose Response Assessment	3	16
D. Risk Characterization and Management	4	12
4. Applied Toxicology	12	25
5. Contribution to the Profession ^a	9	NA
Total	57	138

^aDomain not testable.

Validation Survey: Methodology

The updated delineation of practice was validated through an online survey of practicing toxicologists. The survey was designed to collect feedback regarding all elements of the delineation of practice. To reduce the time burden, one half of the respondents was randomly routed to rate tasks and the other half was randomly routed to rate knowledge. All respondents rated the domains and completed a background questionnaire.

Tasks

Two rating scales were used to evaluate the tasks, one designed to assess the respondents own work patterns (Frequency) and the other designed to assess the task in the context of the practice of toxicologists in general (Importance). The frequency of task performance in the respondent’s own job was rated on a 5-point scale with the following choices: 1 = never/not applicable to my job, 2 = rarely (less than once a month), 3 = occasionally (at least once a month), 4 = frequently (at least once a week), or 5 = very frequently (at least daily). The importance to toxicology practice in general was rated on a 4-point scale which included the following response options: 1 = not, 2 = minimally, 3 = moderately, or 4 = highly important.

Knowledge

One rating scale was used by respondents to evaluate the knowledge—the importance of the knowledge to the respondent’s own job. The same 4 response options were offered as for the tasks: 1 = not, 2 = minimally, 3 = moderately, or 4 = highly important.

Domains

Two rating scales were used by respondents to evaluate the domains and subdomains. Respondents allocated the percentage of their toxicology work time they spent performing activities related to each domain during the past 12 months. For the 2 domains with subdomains, respondents allocated their percentages of time at the subdomain level and these percentages were summed to produce a domain percentage. Respondents also rated the importance of each domain or subdomain to the work of toxicologists in general. Because domain ratings would subsequently be used to derive test specifications, greater precision in domain importance ratings was achieved by offering 5 response options: 1 = not, 2 = minimally, 3 = moderately, 4 = highly, or 5 = critically.

Feedback on the Delineation

As a check on the completeness of the delineation, respondents answered questions related to potentially missing content and on how well they felt the delineation of domains, tasks, and knowledge represented toxicology practice.

Pilot Test of the Survey

The survey was pilot tested in May 2021. Fifteen pilot testers participated, including 4 external SMEs, 8 ABT BoDs, and 3 members of the PATF. Based on their comments, minor technical enhancements to the survey presentation were implemented.

Survey Administration

The large-scale validation survey was launched in July 2021 and was open for 4 weeks. All certified ABT Diplomates were included in the survey invitation list. ABT sent a pre-survey email to all invitees, and ACT followed up with an invitation containing a unique password-protected link to the survey. Additional reminders were sent over the survey window to non-respondents. In addition, the American College of Toxicology circulated an anonymous link to the survey that its members could access.

Validation Survey: Results

Demographic Characteristics of Respondents

A total of 733 respondents completed the survey, for an overall response rate of close to 30%. Most respondents accessed the survey from the link distributed to ABT Diplomates, with only 7 respondents accessing the survey link distributed by the American College of Toxicology to its membership. Approximately half the respondents completed each version of the survey, with 51% rating the tasks and 49% rating the knowledge.

Respondents had a mean of 23 years of experience as a toxicologist, with 10% having less than 10 years, 37% having 11-20 years, 26% having 21-30 years, and 27% having more than 30 years of experience. Eighty-five percent of respondents held research or applied doctoral degrees, 11% a master’s level degree, and 2% a bachelor’s level degree. Overall, the largest percentage of respondents (48%) worked in industry, followed by 22% in consulting (either with a firm or independently), 10% for the federal government, and 7% in contract laboratories. The primary focus area for the largest percentage of respondents was pharmaceutical (47%) followed by regulatory (13%) and food or cosmetic safety (7%). Respondents could specify multiple areas of specialization, with 83% indicating they currently specialize in general toxicology, 59% in regulatory toxicology, and 47% in risk assessment among the most common areas of expertise.

Seventy-four percent of respondents were from the United States, representing 46 states or territories, and 26% were from outside the US, representing 23 countries. The greatest number of non-US respondents was from India (35), followed by China (32), Canada (21), and Switzerland (20). Fifty-eight percent of respondents were male, 36% were female, and about 6% either indicated they preferred not to respond or did not answer. Respondents from the United States were asked to identify their racial/ethnic background; the greatest number identified as Caucasian (62%), 17% identified as Asian, 3% identified as Black, and 2% identified as Hispanic. Two percent indicated more than one race or ethnicity and almost 12% either indicated they preferred not to respond or did not answer.

Domains

Respondents overwhelmingly validated the domain structure by indicating that they spent a significant percentage of their toxicology work time in each domain or subdomain and rating all domains and subdomains as Moderately to Highly important to the work of toxicologists. The domain in which respondents spent the most time was Risk Assessment (37%), followed by Conduct of Toxicology Studies (35%). Respondents spent lesser amounts of time performing Applied Toxicology (9%), Mechanisms of Toxicology (7%) and Contribution to the Profession (8%). All domains and subdomains were highly rated on the Importance scale, with the subdomain Interpret Toxicological Studies rated as most important, with a mean of 4.7 on the 5-point scale.

Tasks

Most of the 57 tasks were performed at a high level of frequency. Distributions of the count of mean task frequency ratings, grouped into ranges, are shown in Figure 2. The most frequently performed task was Interpret and integrate study results with other available data (e.g., literature, existing data) into a scientifically cogent narrative to develop conclusions and/or inform next steps (mean = 4.4), and the next most frequently performed task was Identify systemic and local effects, target organs, dose response, thresholds of effect, and reversibility (mean = 4.3). Six of the tasks achieved mean frequency ratings below 2.0, indicating they are performed less than monthly. Five of these were in the Applied Toxicology domain, and the other was in the Contribution to the Profession domain.

Figure 2.

Distribution of counts of mean task frequency ratings.

All tasks were rated highly on the Importance scale. Distributions of the count of mean task importance ratings, grouped into ranges, are shown in Figure 3. Nineteen tasks achieved mean ratings of 3.5 or higher, meaning they were Moderately to Highly important. The most highly rated task on this scale was also highest in frequency: Interpret and integrate study results into a scientifically cogent narrative to develop conclusions and/or inform next steps (mean = 3.8). Five other tasks had mean importance ratings at or just below 3.8. No tasks were rated below 2.5 on the Importance scale. A full report of the mean task ratings and importance are shown in Table 1 of the Supplemental Information.

Figure 3.

Distribution of counts of mean importance ratings.

Knowledge

Most knowledge was rated highly on the scale for Importance. The distribution of counts of mean knowledge ratings, as related to an individual’s own work, is shown in Figure 4. Nineteen knowledge topics had mean ratings of 3.5 or higher (above the midpoint between Moderately and Highly important). The highest rated knowledge topics were Major effects and endpoints measured in acute lethality/toxicity, repeat dose toxicity, reproductive toxicity, developmental toxicity, genotoxicity, carcinogenicity, sensitization, local effects/tolerance, immunotoxicity, and phototoxicity (mean = 3.79), and the next most highly-rated knowledge was Considerations for identifying toxic responses including dose/concentration, duration of treatment, life-stage, endpoints, route of administration, alternative in vitro, and ecotoxicological studies (mean = 3.75).

Figure 4.

Distribution of counts of mean knowledge importance ratings.

Eight knowledge areas had mean importance ratings just below 2.0, indicating that they were rated lower than Minimally important. Two of these were in subdomain 3D Risk Characterization and Management and 6 were in the Applied Toxicology domain. A full report of mean knowledge ratings is shown in Table 2 of the Supplemental Information.

Completeness of the Delineation

After completion of task or knowledge and domain ratings, respondents were asked how completely they thought the delineation represented the work of toxicologists. Their responses further supported that the domain structure, tasks, and knowledge updated and validated during the practice analysis reflected the work of general toxicologists well. As shown in Figure 5, 89% of respondents thought the delineation completely or mostly represented current toxicology practice. All write in responses were provided to the PATF for review, which determined that none of the feedback warranted a change in the delineation. Many responses mentioned the difficulty of conceptualizing a profession that is practiced in such a diversity of setting and roles.

Figure 5.

Assessment of completeness of the delineation to general toxicology practice.

After careful review and discussion during its final meeting, the PATF endorsed the validation of all 57 tasks, and endorsed the validation of 138 of the 140 knowledge topics. The 2 lowest rated knowledge topics were not retained in the delineation.

Development of Test Specifications

In a practice analysis that has the development of test specifications for a certification program as one of its primary goals (i.e., the percentage of the exam focused on content in each domain or subdomain), data from the validation survey are used to generate empirically derived test specifications. Two commonly used approaches were implemented to provide hypothetical exam weights.⁵ The top-down method uses domain percentage of time, weighted by importance ratings, to derive potential test weights. The bottom-up method starts at the level of the task ratings to derive potential test weights. The results of 2 approaches were averaged to provide proposed test specifications for the examination. These were presented to the ABT BoD in November 2021 and were approved for use in constructing the 2022 certification exam (Figure 6).

Figure 6.

Test specification for the American Board of Toxicology certification exam.

The full delineation resulting from the updated practice analysis study is presented in Table 2, where k indicates Knowledge Topics.

Table 2.

Delineation of General Toxicology Practice.

Domain I. Conduct of Toxicological Studies

A. Design

1.1. Design scientifically valid studies to answer questions or address defined hypotheses.

k1 Elements of a scientific hypothesis

k2 Elements of a scientifically valid hypothesis-driven research or regulatory guideline study, including selection of statistical analyses

k3 Experimental (e.g., test article or agent) and control (e.g., positive and negative control agents, untreated controls) groups used to evaluate scientific hypotheses and/or for regulatory decision-making

k4 Considerations for identifying toxic responses including dose/concentration, duration of treatment, life-stage, endpoints, route of administration, alternative in vitro and ecotoxicological studies

k5 Principles of the 3Rs (Reduce/Refine/Replace) for animal use

k6 Selection of appropriate species for study

1.2. Comply with applicable regulations and guidelines specific to the agent when designing a study.

k7 Standard US and global non-clinical testing guidelines for pharmaceuticals and chemicals (e.g., GLP, OECD, USEPA, ICH, USFDA, EMA)

1.3. Identify the characteristics of the test agent necessary for the study (e.g., chemical identity and physical properties).

k8 Analytical methods commonly used in toxicology for test agent characterization (e.g., HPLC, MS, ELISA)

k9 Characterization of formulated test agent for administration (e.g., stability, homogeneity)

k10 Solubility, stability, formulation, vehicles, and excipients

k11 Content and qualification of impurities in new drug substances per regulatory guidance (e.g., ICH Q3A[R2]) including organic (process or drug-related) impurities, inorganic impurities, and residual solvents

1.4. Identify and apply new testing methods and techniques that are fit for purpose.

k12 Accepted standards of validation applied to testing methods

k13 Validated alternative testing methods in toxicology

k14 Study design for new assays (e.g., dermal sensitization, irritation, cell-based assays, “omics” technologies)

k15 Limitations and advantages of assays, including specificity and sensitivity

B. Execute

1.5. Ensure compliance with regulatory guidances, Good Laboratory Practice (GLPs), and standard operating procedures (SOPs) when performing studies.

k16 Standard US and global non-clinical testing guidelines for pharmaceuticals and chemicals (e.g., GLP, OECD, USEPA, ICH, USFDA, EMA)

1.6. Characterize toxicological effects in vivo.

k17 Typical in vivo models used in toxicology

k18 Relevant endpoints measured in standard animal toxicological studies

1.7. Characterize toxicological effects in vitro.

k19 Typical in vitro models used in toxicology

k20 Relevant endpoints measured in standard in vitro toxicological studies

1.8. Characterize toxicological effects in silico, incorporating computational models and methods.

k21 Standard in silico methods and their application in toxicology (e.g., quantitative structure activity relationships [QSARs], read-across, PBPK modeling)

k22 Artificial intelligence, deep and machine learning used in toxicology

1.9. Characterize toxicological effects in a field or clinical setting.

k23 Endpoints obtained from standard controlled field (ecotoxicological) studies, human clinical trials, and studies with human subjects, and epidemiological studies

C. Interpret

1.10. Analyze test results using tools such as informatics, statistics, and modeling.

k24 Standard data analysis methods, approaches, and techniques used in toxicology, including the role of bioinformatics, statistics, and computer modeling

k25 Distinctions between toxicological and biological significance

1.11. Identify systemic and local effects, target organs, dose response, thresholds of effect, and reversibility.

k26 Criteria, methodologies, and guidelines for analyzing, interpreting, and identifying target organs and adverse effects

1.12. Prepare research reports or summaries that are fit for purpose.

k27 Elements of a toxicology study report, including all supporting subparts

k28 Written communication styles, protocols, and approaches used to tailor reports to specific target audiences (e.g., other scientists, managers, sponsor, regulatory bodies, the general public)

k29 Narrative, tabular, and graphical presentation of findings and interpretations

1.13. Interpret and integrate study results with other available data (e.g., literature, existing data) into a scientifically cogent narrative to develop conclusions and/or inform next steps.

k30 Basic principles of and approaches to systematic literature review (e.g., risk of bias, search strategies)

k31 Principles, strategies, heuristics, methods and approaches for interpreting the results or relevance for human or animal safety

Domain II. Mechanistic Toxicology

2.1. Develop hypotheses to investigate and/or identify mechanisms and understand toxicological outcomes.

k32 Steps involved in developing a mechanistic hypothesis

k33 Methods to test mechanistic hypotheses using in vitro and in vivo studies

k34 Established molecular/signaling pathways associated with chemical-specific toxicities in animal tissues/in vitro systems

k35 Physiology and organ function as related to toxicology

k36 Role of genetic variant phenotypes e.g., (KOs, SNPs) in mechanisms of toxicity

k37 Application of “omics” technologies appropriate to test hypotheses under investigation

2.2. Assess role of species differences in mechanism of toxicity.

k38 Standard parameters used to predict or describe differences in localized and systemic toxicity (e.g., ADME, PK/TK, PK/PD, parent versus metabolite)

k39 Role of species-specific toxicokinetics and toxicodynamics in mechanism of toxicity

k40 Effects of species differences in anatomy and physiology on uptake and absorption (e.g., respiratory tract, dermal, oral)

2.3. Identify susceptibility factors that may impact toxicological outcomes.

k41 Attributes (e.g., genetic polymorphisms, life-stage, gender, background disease-states) affecting individual susceptibility to test agents, toxins, or toxicants

2.4. Assess mode of action (MOA)/adverse outcome pathway (AOP) for relevance to humans or other target species, including sensitive subpopulations and individuals.

k42 Adverse outcome pathway concept

k43 Fundamental molecular toxicological mechanisms and their relationship to adverse responses

k44 Molecular level (e.g., receptor/protein level) toxicant interactions

2.5. Distinguish direct and indirect action, primary/secondary, and on-target versus off-target effect.

k45 Toxicity resulting directly from a targeted interaction/direct insult (e.g., at a cellular/receptor/pathway level)

k46 Toxicity resulting from a secondary/indirect effect (e.g., delayed ossification due to maternal toxicity, reduced fetal body weight, food consumption, stress, off-target interaction)

k47 How exposure to toxicants leads to downstream effects in individuals and populations

2.6. Translate toxicological results across biological levels (sub-cellular, cellular, tissue, individual, species, populations) using principles of systems toxicology.

k48 Methods to translate experimental findings across levels of biological organization

k49 Methods to apply findings/results from simple and/or lower-level experimental systems to more complex, higher-level systems

k50 Anatomical, physiological, biochemical, pharmacological, and toxicological differences across species, and magnitude of specific effects

k51 Utility and limitations of analytical/computational tools to study mechanisms of toxicity and interpret disruption of biological networks

k52 Relationship of “omics” data to organism-level effects

2.7. Apply results from mechanistic studies to toxicological outcomes, prevention, and clarification of risk (human, environmental, animal).

k53 The relationships between mechanism or MOA of chemical(s), human/environmental exposure, and the regulation of certain products (e.g., endocrine disruptors, microbeads, tobacco/nicotine, persistent bioaccumulative toxins (PBTs), nanoparticles)

k54 Mechanistic or MOA data for a chemical from in vitro/in vivo animal studies that inform AOPs

k55 Toxic effects of a chemical mixture, including concepts of synergism, potentiation, additive, and antagonistic

k56 Cellular, biochemical, and molecular mechanisms of chemical toxicity used for the interpretation and evaluation of risk to humans, animals, and the environment

2.8. Apply disease and genetically engineered models to mechanistic investigations.

k57 Application of mechanistic data to the development of alternative animal disease models

k58 Difference between animal disease models and human conditions, including strengths and limitations of disease models

Domain III. Risk Assessment

A. Hazard Identification

3.1. Characterize, describe, and interpret effects, target organs, and test system endpoints of toxicological concern.

k59 Major effects and endpoints measured in acute lethality/toxicity, repeat dose toxicity, reproductive toxicity, developmental toxicity, genotoxicity, carcinogenicity, sensitization, local effects/tolerance, immunotoxicity, and phototoxicity

k60 Ecotoxicological studies

k61 Toxicity by different routes of exposure

k62 Relevant toxic agents such as pesticides, metals, solvents and vapors, radiation, nanomaterials, air pollution, and naturally occurring toxins (e.g., plant- and animal-based, mycotoxins)

k63 Toxic responses of organ systems (e.g., blood, bone, immune system, gastrointestinal tract, liver, kidney, respiratory tract, nervous system, eye, heart, skin, reproductive organs, endocrine system)

k64 Adverse versus non-adverse or adaptive effects

k65 Concepts of false negative and false positive results

k66 Chemical-specific biomarkers of effect

3.2. Apply appropriate in silico, in vitro, in vivo systems, models, and study types for safety evaluation.

k67 Application of data from various methods (e.g., structure-activity, in vitro or short-term studies, animal bioassays, human epidemiologic studies) for hazard identification

k68 Relevant target organism(s) for selection of appropriate test species used in safety evaluation

3.3. Identify toxicological approaches using exposure routes that are appropriate surrogates or alternatives to the primary exposure route of concern.

k69 Basic principles of SAR, QSAR, and other in silico approaches and how those outputs can be used in a risk assessment context

k70 Types of oral toxicity studies applied to hazard identification (e.g., gavage versus dietary or drinking water)

k71 Types of inhalation toxicity studies as they apply to hazard identification (e.g., aerosol versus vapor versus intratracheal instillation)

k72 Metabolic differences between various routes of administration (e.g., sublingual versus oral)

k73 Anatomical differences across species for various routes of exposure (e.g., inhalation, oral and dermal exposures)

3.4. Assess impact of anticipated versus unanticipated effects on organism, species, population, and environmental functions.

k74 Target and off-target effects at organism, species, population, and environmental levels

B. Exposure Assessment

3.5. Select appropriate exposure endpoints to document exposure to toxicants in test systems and populations.

k75 Measures and units of exposure (e.g., dose, airborne concentration, plasma level) reported in toxicological investigations and assessments

k76 Relevance of biomarkers of exposure in individuals, populations, and the environment

k77 Disproportional, unique, reactive metabolites, and toxicophores arising in specific species

3.6. Assess internal exposure in tissues, organs, systems, or whole organism resulting from an administered or exposure dose.

k78 Routes of exposure and toxic agent effects on cells, tissues, organisms, and the environment

k79 Absorption (A), distribution (D), metabolism (M), excretion (E), kinetics (PK or TK), and storage following toxicant exposure

k80 Considerations for measuring the appropriate constituent (e.g., parent versus metabolite, unbound/free or total)

3.7. Assess toxicant exposure in the general public, occupationally exposed individuals, populations, and the environment using appropriate technologies (e.g., analytical, bioassay, biomonitoring).

k81 Emerging and existing technologies used to assess exposure in target populations (e.g., biomonitoring endpoints/dosimeters, sentinel species, spatially explicit models)

k82 Generally accepted default values used in human exposure and dose calculations (e.g., body weight, surface area, inhalation rates)

3.8. Assess and document behavior, environmental fate, and transport of chemicals entering the environment.

k83 Major concepts and processes in environmental fate and transport (efate) assessments (e.g., persistence, biotransformation, bioavailability, bioaccumulation, biomagnification)

k84 Environmental and/or occupational exposure patterns relevant to human and other environmental receptors (e.g., conceptual site model)

k85 Environmental fate and transport processes that influence exposure

C. Dose Response Assessment

3.9. Quantitatively and/or qualitatively characterize relationships between dose (or concentration) and incidence and severity of health effects or toxicological endpoints.

k86 Interpretation of different types of dose-response curves including those for: (a) lethality, (b) threshold responses, (c) linear, nonlinear, non-monotonic, responses, and (d) hormesis

k87 Interpretation of frequency and cumulative dose-response curves

k88 Parameters of dose-response curve (e.g., slope, plateau, midpoint)

k89 ED50, EC50, LD50, and LC50 from dose response curve

k90 General principles and approaches of benchmark dose (BMD) methodologies, including statistical methods and dose-response modeling

k91 Outputs of physiologically-based pharmacokinetic (PBPK) modeling and their utility in characterizing dose-response relationships

k92 Methods for dosimetric adjustments across species and exposure duration (e.g., HED/HEC calculations)

k93 Dose-response extrapolations and how to characterize their associated uncertainties in susceptible populations, including traditional uncertainty factors (UFs), chemical-specific adjustment factors (CSAFs), data-derived extrapolation factors (DDEFs)

3.10. Analyze toxicity data to determine safety margin and first in human dose.

k94 First-in-Human dose (FIH) criteria/standards per regulatory guidance(s)

3.11. Analyze toxicity data to determine safe dose or concentration, toxicologically relevant dose, reference dose, protective threshold, or benchmark dose.

k95 Threshold and non-threshold (linear) dose-response curves

k96 Integration of endpoints (e.g., organ weights, clinical chemistry, histopathology) in comprehensive dose-response assessment

k97 Dose-response dynamics of essential nutrients and hormetic responses

k98 Calculation of Therapeutic Index (TI)

k99 Point of departure (POD) (e.g., LOAEL, BMD, NOAEL, HNSTD, MABEL, STD10) and uncertainty factors and their use in risk assessment

k100 Concepts and uses of safety thresholds (e.g., TTC, Cramer Class, REACH DNELs and DMELs, reference values, ADI, health-based OELs)

k101 Approaches for allometric scaling including default values and when they should be applied

Risk Characterization and Management

3.12. Integrate various lines of toxicity evidence (e.g., animal, in vitro, epidemiological, read-across) with exposure information to characterize potential health risks.

k102 Metrics used in risk characterization (e.g., margin of exposure (MOE), margin of safety (MOS), hazard quotient (HQ), Hazard index (HI), probability estimates) and their application and limitations

k103 Aggregate exposure and risk and cumulative risk in the context of risk characterization

k104 Principles of mixtures toxicity and potential application in risk characterization (e.g., dose-response additivity, toxicity equivalency factors)

3.13. Perform weight of evidence analyses to reach conclusions and inform decision making.

k105 Weight of evidence criteria (WoE) criteria for integrating potentially disparate lines of evidence (e.g., Bradford Hill criteria)

k106 Systematic review concepts, processes, and procedures (e.g., Cochrane review)

3.14. Assess health risks and management options to protect occupational, environmental, or public health by deriving protective safety limits and mitigation strategies.

k107 Factors that contribute to and modify risk characterization (e.g., engineering, economic, social, political)

k108 Principles of risk perception and communication

k109 Precautionary principle and its implications

k110 Health protective exposure limits for various populations (e.g., immunocompromised, child-bearing potential, pregnant, lactating, elderly, asthmatic), exposure durations, and exposure scenarios

3.15. Evaluate and implement alternatives to reduce risk through actions such as emergency risk management and reduction of chemical exposure or risks.

k111 Strategies, technologies, practices, and procedures to reduce exposure and/or mitigate toxicity (e.g., antidotes, remediation, engineering controls, personal protective equipment)

k112 Cleanup goals, emission limits, safe concentrations

k113 Health-based guidance values (e.g., Provisional Advisory Levels (PALs), Acute Exposure Guideline Levels (AEGLs))

Domain IV. Applied Toxicology

4.1. Characterize, describe, and interpret effects of ecotoxicological concern in individuals, populations, communities, and ecosystems.

k114 Differences in responses between individuals and populations

k115 indirect effects from exposure in an ecological community on susceptible populations

4.2. Respond to public health issues, including new or emerging public health concerns.

k116 Elements of response and risk communication plan, risk perception, and mitigation strategies

k117 Lessons learned from events of major toxicological significance (e.g., Bhopal, radium dial painters, cytokine storm release syndrome, Minamata Bay, thalidomide, DDT)

k118 Toxicological impact of emerging risks (e.g., climate change, infrastructure issues, food scarcity, infectious diseases)

4.3. Assess impact on public health from environmental toxicants resulting from ecological disruptions.

k119 Impact of threats to ecosystem services (e.g., clean/safe water, clean air) on human health

k120 Effects of toxicants on disease prevalence (e.g., zoonoses) in animal-human interactions

4.4. Investigate health outcomes in exposed groups using measured or modeled exposures.

k121 Exposure reconstructions for dose-response evaluation in the context of epidemiological findings

4.5. Use population-based biomonitoring studies to ascertain temporal trends in environmental exposures.

k122 Population-level national studies and databases (e.g., NHANES)

k123 Cross-sectional versus longitudinal epidemiology studies

k124 Toxicological implications of biomonitoring data reported as part of a community public health investigation

4.6. Identify sensitive or susceptible subpopulations and identify factors or circumstances that would increase the risks of adverse health effects in incidents or situations of toxicological concern in a community or occupational setting.

k125 Factors related to susceptibility and increased risk in groups or sub-groups of individuals (e.g., comorbidities, age, ethnicity)

k126 Biomarkers of susceptibility (e.g., polymorphisms, genetic variants)

4.7. Collaborate in the development and manufacture of safe, sustainable, and environmentally responsible pharmaceutical and consumer products.

k127 Principles of green chemistry, sustainable substitution, regrettable substitution, benign by design

k128 Extractables and leachables (e.g., packaging and food contact, medical devices, primary container closure systems)

k129 Toxicological or hazard criteria for designations (e.g., EPA SaferChoice, USDA Biobased, NSF International)

4.8. Evaluate the safety/toxicity or risk/benefit of agents for the treatment of diseases.

k130 Potential side effects from use of common therapeutics/drugs and the mitigation strategies for each

k131 Elements of drug risk-benefit analysis and margins of safety (MOS) calculations

4.9. Contribute to product stewardship, including evaluating products and communicating potential environmental, health, or safety risks and mitigation strategies.

k132 Tiered approaches to gathering toxicological information

k133 Potential unintended effects from product use or misuse, potential environmental impacts resulting from use, and mitigation strategies through labeling and manufacturing controls

4.10. Evaluate the clinical signs and symptoms of toxicity in the context of relevant standards, metrics, and reference values.

k134 Translatability of pre-clinical data to clinical results or findings

k135 Standard sources of information for toxicity-related metrics and reference values

4.11. Develop or provide clinical treatment recommendations for poisoning incidents, including antidotes.

k136 Mechanism of toxic action and how it relates to treatment of poisoning, including physical measures (e.g., decontamination procedures, decorporation) and clinical treatment options (e.g., antidotes)

4.12. Adhere to regulation pertaining to toxic chemicals.

k137 Regulations and standards relevant to occupational and consumer health (e.g., OSHA, CPSC, ASTM Standards)

k138 Regulations relevant to public exposure to chemicals (e.g., TSCA, REACH, FIFRA)

Domain V. Contribution to the Profession

5.1. Improve or provide technical opinion to help improve regulatory guidelines to ensure guidelines reflect current scientific advances.

5.2. Provide testimony about toxic agents and products and their effects.

5.3. Communicate the results of toxicological or risk evaluations to technical, governmental, or public groups via a variety of methods (e.g., publications [peer reviewed and otherwise], technical reports, summary documentation reviewed], abstracts, presentations, and posters).

5.4. Provide toxicological expertise on project teams across the range of work settings and industries (e.g., advise on study design, regulatory-required studies, risk mitigation; support product development, aid in interpretation of results).

5.5. Respond to requests from organizations (e.g., companies, governmental/regulatory authorities, scientific societies) for technical expertise and comments about toxins/toxicants and their effects.

5.6. Provide education, training, and mentorship for students, colleagues, and others in the field (e.g., instructional course work, laboratory practical, continuing education courses, chairing or presenting in scientific sessions).

5.7. Disseminate information to co-workers, management, and appropriate organizations.

5.8. Establish, facilitate, or support programs to educate the public on environmental or public health issues.

5.9. Participate in professional scientific or not-for profit organizations (e.g., serve as elected or appointed officer, volunteer on task force or committee).

The Test Length and Reliability Study

Like all other aspects of life, the SARS-CoV-2 virus and resulting global pandemic created challenges and opportunities for the ABT. For the first time in 40 years, the ABT certification exam (which historically was an in-person event administered annually on a single day in limited cities) was not administered in 2020. Instead, the ABT BoD focused on implementation of a computer-based exam and subsequently administered an electronic 2021 exam on a single day worldwide using partnering testing administration sites. Feedback indicated some candidates experienced difficulties scheduling the exam due to the 6+ hours of seat time required for each candidate (i.e., two 3-hour blocks with a break).

The computer-based administration captured the amount of time candidates used to answer individual questions to complete the exam (i.e., latency). These data provided an opportunity to further investigate the feasibility of a shorter examination.

The ABT BoD recognized the potential value of a shortened examination. In addition to easing scheduling issues due to long seat times, question development efforts would be less burdensome and question exposure reduced. The ABT BoD also recognized any action related to test length needed to occur in a manner consistent with credentialing best practices and only after careful study of the impact. This led to contracting with ACT’s Psychometrics Unit to conduct a reliability study.

Prior to the reliability study, the certification exam consisted of 200 questions distributed across 4 domains and 7 subdomains and a resulting reliability score (coefficient alpha) of .92. Any reduction in the number of test questions would need to ensure sufficient coverage of the content outline to demonstrate candidates have sufficient knowledge and skill to attain the credential while still maintaining the psychometric content validity of the examination.

Using data from the 2021 exam, ACT’s Psychometrics Unit conducted a study to determine the number of operational items (i.e., questions) required to yield reliability estimates comparable to those produced by the then-current operational form, while still conforming to the test specifications to ensure adequate content coverage. The reliability study included 2 steps: a theoretical prediction of the reliability of shortened forms of the examination and the empirical validation of the theoretical prediction.

First, the Spearman-Brown prophecy formula^6,7 was used to theoretically predict test score reliability of tests with various shorter lengths. The reliability of test scores is, in part, a function of the number of test items; it tends to increase with more test items and decrease with fewer test items.⁸ The Spearman-Brown prophecy formula is one way to estimate the reliability of a shorter or longer test based on the reliability of the existing test.

Then, a simulation study was conducted to empirically generate reliability based on the 200-item certification exam for test length reductions of 5%, 10%, 15%, 20%, 25%, and 30%. A proportional sample of questions was removed from each of the domains and subdomains so that each simulated form met the test specifications. The reliability and the standard error of measurement (SEM) were calculated for each simulation. This proportional item sampling and reduction was repeated 100 times for each of 6 reduced test lengths.

Table 3 summarizes the results of the predicted reliability from the Spearman-Brown formula and the empirical reliability from the simulation. Because the reduction in test length was proportional across domains and subdomains, the number of questions per reduced test length slightly differs from the proportional reduction with the Spearman-Brown prophecy formula. The average reliabilities from the simulation study perfectly matched the estimated reliabilities from the Spearman-Brown prophecy formula.

Table 3.

Predicted and Empirical Reliability.

% Reduction	Spearman-Brown		Empirical Reliability				SEM
% Reduction	Length	Reliability	Length	Mean	Min	Max	Mean	Min	Max
5	190	.92	190	.92	.91	.92	6.00	5.97	6.03
10	180	.91	178	.91	.91	.92	5.81	5.75	5.86
15	170	.91	170	.91	.90	.91	5.67	5.61	5.72
20	160	.90	159	.90	.89	.91	5.49	5.43	5.60
25	150	.90	151	.90	.89	.91	5.34	5.28	5.45
30	140	.89	141	.89	.88	.90	5.16	5.09	5.25

The results of the test length reduction study indicated that the reliability of a reduced-length examination with 140 scored questions would still be sufficiently high for the purposes of certification. When constructed with precise measurement (i.e., low conditional standard error of measurement) near the passing score, a shortened ABT certification examination will not reduce the accuracy of pass/fail decisions. Based on the latency data from the 2021 computer-based examination, an exam consisting of 160 questions (140 scored and 20 pretest) would result in an exam lasting approximately 3 to 4 hours. The results from this study support the development of an exam consisting of 140 scored questions across the domains and subdomains of content coverage specified by the updated test blueprint. Based on these data, the ABT BoD voted to adopt this change in length beginning with the 2022 examination administration.

The Standard Setting Study

In keeping with certification industry best practices, the ABT BoD contracted with ACT to perform a standard setting study to assess changes resulting from the updated practice analysis and the reduction in exam length. Standard Setting exercises are conducted to set the passing score for the initial form of an exam developed following a job analysis study or other major change in an exam, such as a reduction in length.⁹ The standard setting study was performed for the shortened ABT certification examination, which was administered to 259 candidates on October 11, 2022.

The standard setting study was conducted in conformance with best practices in the testing and credentialing industry. ACT employed 2 approaches to complete the standard setting, the modified-Angoff method, an industry-standard criterion-referenced approach for setting the standard,¹⁰ as well as the modified-Hofstee method (Range Estimation).⁵ The use of 2 complementary methods for standard setting provides an organization with robust data on which to base its final decision about the passing standard for the exam.

The standard setting process used a committee of 10 toxicology SMEs from diverse backgrounds. The SMEs attended an online training webinar presented by ACT, followed by three 3-hour virtual meetings. The SMEs were tasked with identifying the point on the theoretical continuum that separates the test taker who is minimally qualified—that is, just able to pass the exam—from one who is not. Using the modified Angoff method, the participants evaluated each individual item on the exam in the context of the performance of the hypothetical candidate. An iterative rating process involving calibration, discussion and re-rating questions was employed, taking into account the objective difficulty of each question based on actual candidates' performance. The Angoff activity was followed by a shorter method—a holistic review of the exam using a modified Hofstee method. This approach involves estimating the highest and lowest acceptable passing score. When both processes were complete, the SME ratings were translated into a passing score and impact analyses were conducted using actual examination data. A recommended passing standard was presented to the ABT BoD at its November 2022 meeting, and this passing score was approved by the ABT BoD.

One hundred eighty-seven candidates achieved the passing score of the 2022 certification examination and became Diplomates of the American Board of Toxicology®. While the ABT does not release the passing score, which is not uncommon in the credentialing industry, the percentage of candidates passing the examination each year is both tracked and publicized by the ABT. The percentage of candidates earning the DABT credential in 2022 was 72%. This is consistent with the passing percentages since the first practice analysis resulted in format changes in 2017, lending further support for the structural changes to the exam made in 2022. Further evidence that there was no negative impact on overall exam performance may be found by comparing statistical analysis data over the years. The average difficulty (the percentage of candidates that correctly answered the question) and discrimination (the point-biserial statistic that indicates how well a question discriminates between stronger and weaker candidates) have remained consistent over time (Table 4). Equally important, the reliability of the exam has remained stable and consistent across administrations, regardless of the number of items on the examination form. As shown in Table 4, the exam reliability (coefficient alpha) was .913 in 2022; in 2021, it was .920, both strong measures of reliability.

Table 4.

Comparison of Exam Statistics, Including Reliability, Across Time.

	2022		2021		2019		2018		2017
	Diff.	Disc.	Diff.	Disc.	Diff.	Disc.	Diff.	Disc.	Diff.	Disc.
Mean	.64	.26	.63	.23	.64	.23	.63	.23	.60	.23
SD	.17	.10	.18	.11	.20	.11	.19	.11	.19	.13
Reliability	.91		.92		.92		.92		.92

Discussion

The ABT BoD is committed to regularly conducting an updated practice analysis to ensure the examination adequately represents the knowledge and skills required of a toxicologist in a rapidly changing discipline. The second practice analysis of toxicology was executed in 2020-2021 to meet this commitment. The outcome of this practice analysis represents a validated description of the domains and tasks associated with the contemporary active practice of toxicology and the knowledge bases needed by practicing toxicologists. The study described herein explored the importance that practicing toxicologists place on each task, the frequency with which the tasks are performed, and the requisite knowledge needed. For instructors of undergraduate and graduate curricula, these data provide valuable information about the knowledge and skills needed by newly graduating students to be successful in toxicology. This information is also valuable for editors and authors of textbooks covering the science of toxicology, particularly in the emerging areas of toxicogenomics, bioinformatics, systems biology, epigenetics, and computational toxicology.

A 30% survey response rate is considered quite strong for a lengthy and complex practice analysis survey, and the demographics of survey respondents generally align with demographics previously reported in other surveys of practicing toxicologists.^4,11 Some caveats: the representation from academic toxicologists was quite low, as was representation from non-certified toxicologists, and those with doctoral degrees were overrepresented in the pool of survey respondents. As this practice analysis is, to the authors' knowledge, the only rigorous delineation of the practice of toxicology, identifying methodology to improve non-DABT, academic, and nondoctoral participation could be one goal in designing the survey methodology for the next validation.

For the individual toxicologist, the benefits of board certification extend beyond an objective demonstration of the breadth of their knowledge and their competence in the field. According to the Tenth Triennial Toxicology Salary Survey,¹¹ a certified toxicologist with 3-5 years of experience (the earliest a candidate with a PhD is eligible to sit for ABT certification) can expect to earn 42% more than a toxicologist without certification. Interestingly, this impact is much more pronounced for women than for men. A woman with certification earns 41% more than her non-certified counterpart over the course of a 29-year career while a man could expect to earn 21% more than his non-certified counterpart over the same time frame. Notably, salary differences between male and female toxicologists appear to have achieved relative parity (on average) across the range of 3-29 years of experience when certification has been obtained.

The ABT is committed to encouraging the science of toxicology and advancing the discipline. With the publication of these analyses, which include a study of test length and reliability, as well as a standard setting study, the ABT BoD provides transparency regarding the methods and data used to develop the standards on which the ABT certification program is based. This paper shows the foundations of ABT certification and the competencies of those who earn the credential as defined by those standards. In addition, it offers valuable information to employers who expect excellence from Diplomates of the American Board of Toxicology®. In conducting and publicizing these studies, the ABT demonstrates its service and dedication to the field of toxicology and to those who devote their professional lives to that science.

Supplemental Material

Supplemental Material - Updating the Standards of Professional Competence in the Field of Toxicology: The Second Generation of Best Practice

Supplemental Material for Updating the Standards of Professional Competence in the Field of Toxicology: The Second Generation of Best Practice by Nicole V. Soucy, Susie Masten, Carla Caro, Ann M. Arthur, Nadia H. Moore, Michelle J. Hooth, and Robert Mitkus in International Journal of Toxicology

Footnotes

Acknowledgments

The ABT BoD greatly appreciates the time, effort, and priority that all task force members, reviewers, thought leaders, and Board members invested in this effort.

Author Contributions

Soucy, N.V. contributed to analysis and interpretation and drafted manuscript; Masten, S. contributed to conception, contributed to acquisition, analysis, and interpretation, and drafted manuscript; Caro, C. contributed to conception and design, contributed to acquisition, analysis, and interpretation, and drafted manuscript; Arthur, A. contributed to conception and design, contributed to acquisition, analysis, and interpretation, and drafted manuscript; Moore, N. contributed to analysis and interpretation and critically revised manuscript; Hooth, M. contributed to analysis and interpretation and critically revised manuscript; Mitkus, R. contributed to acquisition, analysis, and interpretation and critically revised manuscript. All authors gave final approval and agree to be accountable for all aspects of work ensuring integrity and accuracy.

Declaration of Conflicting Interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: S. Masten is the Executive Director for the American Board of Toxicology and is retained under contract by the BoD to provide this service. C. Caro and A. Arthur are employees of ACT and were retained as consultants by the American Board of Toxicology to provide their scientific expertise in the areas of credentialing and psychometrics, respectively. N. Moore is a current ABT board member, serving in an individual capacity; she is employed by J.S. Held LLC, and has provided consultation for a broad spectrum of toxicological services and expert testimony in litigation. M. Hooth is a former ABT board member, serving in an individual capacity; she is employed by the National Institute of Environmental Health Sciences. N.V. Soucy is a former ABT board member, serving in an individual capacity; she is employed by Boston Scientific Corporation. R. Mitkus is a former ABT board member, serving in an individual capacity. He is employed by NAMSA.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Nicole V. Soucy

Supplemental Material

Supplemental material for this article is available online.

References

American Educational Research Association, American Psychological Association, and National Council on Measurement in Education . Standards For Educational and Psychological Testing. Washington, DC: American Education and Research Association; 2014.

International Organization for Standardization . ISO/IEC 27024: Conformity Assessment — General Requirements for Bodies Operating Certification of Persons. Geneva, Switzerland: ISO; 2012.

National Commission for Certifying Agencies . Standards for the Accreditation of Certification Programs. Washington, DC: Institute for Credentialing Excellence; 2021.

Wood

Weis

Caro

Roe

. A practice analysis of toxicology. Regul Toxicol Pharmacol. 2016;82:140-146.

Raymond

Nuestel

. Determinizing the content of credentialing exams. In: Downing S

Haladyna

, eds. Handbook of Test Development. Mahwah, NJ: Lawrence Erlbaum Associates; 2006.

Spearman

. Correlation calculated with faulty data. Br J Psychol. 1910;3:271-295.

Brown

. Some experimental results in the correlation of mental abilities. Br J Psychol. 1910;3:296-322.

Thorndike

. Applied Psychometrics. Boston, MA: Houghton Mifflin Company; 1982.

Cizek

. Standard setting. In: Downing

Haladyna

, eds. Handbook of Test Development. Mahwah, NJ: Lawrence Erlbaum Associates; 2006.

10.

Angoff

. Scales, norms, and equivalent scores. In: Thorndike

, ed. Educational Measurement. Washington, DC: American Council on Education; 1971.

11.

Sullivan

Gad

. Tenth triennial toxicology salary survey. Int J Toxicol. 2020;39(3):189-197.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.21 MB