Sage Journals: Discover world-class research

Abstract

Background

One of the challenges to using some flavor chemicals in aerosol products is the lack of route of administration specific toxicology data.

Methods

Flavor chemicals (88) were divided into four different flavor mixtures based upon chemical compatibility and evaluated in 2-week dose-range-finding and subsequent 90-day nose-only rodent inhalation studies (OECD 413 and GLP compliant). Sprague-Dawley rats were exposed to vehicle control or one of three increasing concentrations of each flavor mixture.

Results

In the dose-range-range-finding studies, exposure to flavor mixture four resulted in adverse nasal histopathology in female rats at the high dose, resulting in this flavor mixture not being evaluated in a 90-day study. In the 90-day studies daily exposures to the three flavor mixtures did not induce biologically meaningful adverse effects (food consumption, body weights, respiratory physiology, serum chemistry, hematology, coagulation, urinalysis, bronchoalveolar lavage fluid analysis and terminal organ weights). All histopathology findings were observed in both vehicle control and flavor mixture exposed animals, with similar incidences and/or severities, and therefore were not considered flavor mixture related.

Conclusion

Based on the absence of adverse effects, the no-observed-adverse-effect concentration for each 90-day inhalation study was the highest dose tested, 2.5 mg/L of the aerosolized high dose of the three flavor mixtures.

Keywords

Inhalation toxicology flavor chemicals electronic nicotine delivery system aerosols

Introduction

From an inhaled product development toxicology persepctive, one of the challenges to use of some flavor chemicals in aerosol products is the lack of route of administration specific data (e.g., inhalation), which has been cited by several authors for Electronic Nicotine Delivery System (ENDS) products.^1–3 Typically, because flavor chemicals used in ENDS are Generally Recognized as Safe for use in food (U.S. Food Drug and Cosmetic Act, sections 201(s) and 409), there is some oral toxicology, mutagenicity and/or genotoxicity and sensitization data, but rarely inhalation data. A couple of recent articles have summarized in vitro and in vivo data on flavors used in ENDS products.^2,4 There has also been some research that attempts to correlate toxicology data between oral and inhalation routes of exposure.^5–9 Usually, these efforts result in application of a safety factor (also called assessment factor, correction factor, etc.) in the risk assessment process for route-to-route extrapolation (oral to inhalation). For example, Schroder et al.,⁹ recommended a safety factor of three compared to a safety factor of two that Schroder et al.,⁹ indicated was proposed by European Chemicals Agency.¹⁰ The exception to the safety factor approach was Ford et al.,⁸ that found when comparing chemicals that had both oral and inhalation data, if local respiratory effects were ignored, the threshold of concern was similar between the two routes of exposure.

The limited published inhalation data using flavor chemicals, that have been used in ENDS products, can be grouped into three different testing strategies: product testing using either prototype or marketed ENDS products,^11–13 single flavor ingredient studies,^14–20 and mixtures of flavor chemicals studies.^21,22 An example of an early ENDS prototype product being tested reported results of 90-day nose only inhalation study using prototype ENDS products with targeted dosing at 3.2, 9.6, and 32.0 mg/kg/day, with one prototype ENDS formulation containing 17.6% proprietary flavors.¹¹ The 90-day nose-only study was conducted in accordance with Organization for Economic Cooperation and Development (OECD) guideline 413²³ and consistent with U.S. Good Laboratory Practice requirements (21 CFR Part 58). The No Observed Effect Level for all formulations was 9.6 mg/kg/day due to body weight decreases (i.e. exceeding 10%).¹¹ An example of a single flavor ingredient study targeting a flavor ingredient used in ENDS products was a 28-day nose-only inhalation study using two-isopropyl-N,2,3-trimethylbutyramide (WS-23) in a mixture of propylene glycol and glycerin with the aerosol being generated from a commercial ENDS product.¹⁹ The study was designed in accordance OECD guideline 403 and 412^24,25 and found the No Observed Adverse Effect Level (NOAEL) to be at least 342.85 mg/m³, the highest dose tested. Finally, a mixture of flavor ingredients was tested^21,22 utilizing a read across approach^26–30 for 245 ENDS flavor chemicals by grouping them into 26 distinct chemical groups based on structural similarities and reported potential metabolic and biological effects. Flavors chemicals (chemically compatible) from each group that were predicted to show the highest toxicity were combined into a single formulation.³¹ This mixture of 26 flavor chemicals with and without nicotine in a mixture of propylene glycol and glycerin at three exposure doses that were generated using a nebulizer was evaluated in a 90-day nose-only inhalation study.²¹ The 90-day nose-only inhalation study used an OECD 413 compliant study design and was conducted consistent with OECD Good Laboratory Practices.^23,32 Results of these exposures (using all standard OECD and novel systems toxicology endpoints) to the mixture of 26 flavor chemicals demonstrated no significant additive biological changes to those seen with a mixture of nicotine, propylene glycol, and glycerin.²¹ The same flavor chemical mixture was used in a 5-week nose-only inhalation study using AJ mice.²² A capillary aerosol generator was used to heat the formulations containing the flavor chemical mixture to 245°C–255°C for aerosol generation. Exposure to the heated mixture of flavors caused no notable lung inflammation and only minimal adaptive changes in the nose and larynx.²² In the majority of these published ingredient studies, the flavor chemicals were tested as a mixture with the primary ENDS formulation chemicals (propylene glycol and glycerin), which addressed the concerns about reaction products between some flavor chemicals used in ENDS products and propylene glycol and glycerin.^33,34

Each of the flavor chemical testing strategies has advantages/disadvantages and may reflect information needs during the different stages of product development. Testing a single or mixture of flavor chemicals in inhalation studies can provide toxicology information suitable for ENDS product formulation development, while product testing is usually performed on finished ENDS products. The advantage of finished ENDS product testing is that it uses the product intended for marketing with the potential disadvantage of elongating the time prior to marketing, which in the best case is the time required to plan, schedule, execute, and evaluate the data obtained from the inhalation study. The advantage of single ingredient testing is greater flexibility in dose range testing that represents a realistic ENDS product flavor load (percent of formulation i.e. flavor chemicals) and depending on the vehicle used, could provide a more direct evaluation of the effect of an individual ingredient. The disadvantage of single flavor ingredient testing is that it does not provide information on any complex ingredient interactions and requires more cautious interpretation when considering the acceptability for use in a multiple ENDS flavor formulation. The advantage of testing mixtures of flavor chemicals is that numerous flavor chemicals and their potential toxicological interactions (antagonistic or synergistic) can be evaluated simultaneously,^21,22 however if there is an indication of toxicity, all tested flavors chemicals are implicated as there is no way to isolate the toxic response to a single or subset of tested flavor chemicals without additional studies. Additionally, because a mixture of flavor chemicals is tested, chemical compatibility and a narrower dose range for each flavor chemical can occur before the total product flavor load exceeds that normally found in ENDS products. The purpose of this inhalation work was to evaluate 88 flavor chemicals for potential use in ENDS products, which necessitated evaluation as four mixtures of flavor chemicals due to their chemical compatibility.

Materials and methods

Due to space limitations at the testing facility, this work was divided up into two separate inhalation study projects with each project evaluating two mixtures of flavor chemicals. A 2-week nose-only inhalation dose range finding study was performed for each of the four flavor chemical mixtures prior to conducting the 90-day nose only inhalation study for each of the four flavor chemical mixtures. The 2-week nose-only inhalation studies were performed to confirm that the target exposure concentration of 2.5 mg/L was appropriate for their subsequent 90-day nose-only inhalation study. The 90-day nose-only inhalation studies were conducted using the OECD guideline 413³⁵ study design (90-days [13-week] of exposure followed by 42-day [6-week] of recovery) and was conducted in compliance with U.S. Good Laboratory Practices (21 CFR Part 58) by a contract research organization (Charles River Laboratories, Ashland, Ashland, OH). Publication of this work is being done in accordance with the Animal Research: Reporting of In Vivo Experiments guidelines.³⁶

Animals

All studies used Sprague-Dawley rats (Crl:CD[SD]) and were performed at Charles River, Ashland, in facilities that were accredited by the Association for Assessment and Accreditation of Laboratory Animal Care International (AAALAC International). All animal experiments were approved by the Charles River Ashland Institutional Animal Care and Use Committee (review/approval numbers 1,324,173, 1,324,174, 1,324,192, and 1,324,193). The rats (male and nulliparous female) were bred under specific pathogen-free conditions and were obtained from a commercial supplier (Charles River Laboratories, Raleigh, NC). Animals were individually identified (subcutaneous transponders) and housed two to four animals (same sex and dosing group) per polycarbonate cage with Bed-o’ Cobs® bedding material (ScottPharma Solutions, Marlborough, MA) and enrichment items (e.g., gnawing device, treats and/or nesting material) during non-exposure periods. Certified rodent diet 5CR4 (PMI Nutrition International, LLC,) and fresh drinking water via automated watering system (reverse osmosis treated municipal tap water that was also ultraviolet irradiated) was supplied ad libitum to each cage. Animal rooms utilized a 12-hr. light/12-hr. Dark cycle and were supplied with fresh filtered conditioned laboratory air (20 °C–26°C; 30%–70% relative humidity). The animal husbandry was conducted in accordance with the Guide for the Care and Use of Laboratory Animals.³⁷ Animals were acclimated to laboratory conditions for a minimum 6 days for the 2-week dose-range-finding study and 7 days for the 90-day inhalation study. After acclimation to laboratory conditions, animals were subsequently acclimated over 4 days to the nose-only exposure tubes. Animals were placed in the nose-only exposure tubes for increasing amounts of time breathing laboratory air culminating in a six-hour period on day 4 of exposure tube acclimation. Animals did not have access to food or water during inhalation exposures. Animals were assigned to groups using a stratified randomization scheme designed to achieve similar group mean body weights (±20% of mean/sex) with males and females randomized separately. Animals were approximately 7 weeks of age at the beginning of 2-week dose-range-finding studies and 90-day inhalation studies.

Test article

A total of 88 neat flavor chemicals were divided into four different flavor mixtures based upon chemical compatibility/solubility (e.g., precipitation or phase separation) and the criteria that at the high dose the total amount of flavors did not exceed 6% (wt/wt; Table 1). The flavor load criteria of 6% was based upon reported total flavor levels in a majority of ENDS products.^38–40 All neat flavor chemicals were food grade and compliant with Food Chemical Codex, if available. USP grade propylene glycol and USP grade, Food Chemical Codex compliant glycerin were used in each flavor mixture (Table 1) and vehicle control. The vehicle control consisted of the base formulation; 40%/60% (w/w) propylene glycol/glycerin. When not being used for aerosol generation, the vehicle control and all formulations were stored at room temperature (18 °C–24°C), protected from light, and capped with nitrogen. The vehicle control and all flavor mixtures were prepared specifically for this work (they are not commercially available, however individual flavors are available from various commercial sources) in an ISO 90,001:2015 certified facility using commercial ENDS liquid manufacturing techniques that met the SQF Code for food manufacturing by Tobacco Technologies, Inc. (Sykesville, MD.). Tobacco Technologies, Inc. performed stability studies on the high dose of each of the four different flavor mixtures using a single flavor chemical as a marker to demonstrate at least a 6-week shelf-life. A single lot of each of the four different flavor mixtures was used in the 2-week dose range finding studies, while three lots of the four different flavor mixtures were used in the 90-day inhalation studies.

Table 1.

Four flavor mixture formulations listing each neat flavor ingredient.

Flavor ingredient name	CAS#	FEMA #	Inclusion level %
Flavor ingredient name	CAS#	FEMA #	Low	Mid	High
Flavor mixture 1
Glycerin	56-81-5	2525	60	60	60
Propylene glycol	57-55-6	2940	38.8175	37.635	35.27
2,3,5,6-Tetramethyl pyrazine	1124-11-4	3237	0.0075	0.015	0.03
2,4-Decadienal, (E,E)-	25152-84-5	3135	0.025	0.05	0.1
2-Acetyl pyridine	1122-62-9	3251	0.01	0.02	0.04
3-Methyl-1,2-Cylopentanedione	80-71-7	2700	0.25	0.5	1.0
4-(Para-Hydroxyphenyl)-2-Butanone	5471-51-2	2588	0.05	0.1	0.2
Alpha ionone	127-41-3	2594	0.1	0.2	0.4
Beta damascenone	23696-85-7	3420	0.03	0.06	0.12
Beta ionone	79-77-6	2595	0.1	0.2	0.4
Butyl acetate	123-86-4	2174	0.05	0.1	0.2
Ethyl acetate^a	141-78-6	2414	0.05	0.1	0.2
Ethyl lactate	97-64-3	2440	0.025	0.05	0.1
Ethyl phenylacetate	101-97-3	2452	0.02	0.04	0.08
Hexanal	66-25-1	2557	0.1	0.2	0.4
Hexyl alcohol	111-27-3	2567	0.025	0.05	0.1
Isoamyl phenylacetate	102-19-2	2081	0.02	0.04	0.08
Isobutyl acetate	110-19-9	2175	0.05	0.1	0.2
L-menthyl acetate	2623-23-6	2668	0.1	0.2	0.4
Methyl caproate	106-70-7	2708	0.025	0.05	0.1
Methyl phenylacetate	101-41-7	2733	0.02	0.04	0.08
Phenethyl phenylacetate	102-20-5	2866	0.02	0.04	0.08
Styrallyl acetate (alpha-methylbenzyl acetate)	93-92-5	2684	0.05	0.1	0.2
Trans-2-nonenal	18829-56-6	3213	0.025	0.05	0.1
Veratryl aldehyde	120-14-9	3109	0.03	0.06	0.12
Total percentage of neat flavorchemicals in flavor mixture 1			1.1825%	2.365%	4.73%
Flavor mixture 2
Glycerin	56-81-5	2525	60	60	60
Propylene glycol	57-55-6	2940	38.59	37.18	34.36
2-Methyl-2-Pentenoic acid	3142-72-1	3195	0.025	0.05	0.1
2-Methyl butyl acetate	624-41-9	3644	0.075	0.15	0.3
2-Methyl-1-Butanol	137-32-6	3998	0.125	0.25	0.5
Anisyl acetone	104-20-1	2672	0.025	0.05	0.1
Benzyl alcohol	100-51-6	2137	0.55	1.1	2.2
Cis-6-nonenol	35854-86-5	3465	0.005	0.01	0.02
cis-jasmone	488-10-8	3196	0.04	0.08	0.16
Citral	5392-40-5	2303	0.025	0.05	0.1
Decanal	112-31-2	2362	0.025	0.05	0.1
Diethyl sebacate	110-40-7	2376	0.025	0.05	0.1
Dimethyl benzyl carbinyl butyrate^a	10094-34-5	2394	0.025	0.05	0.1
d-limonene	5989-27-5	2633	0.025	0.05	0.1
Ethyl methyl phenyl glycidate	77-83-8	2444	0.025	0.05	0.1
Heptanal	111-71-7	2540	0.025	0.05	0.1
Melonal	106-72-9	2389	0.025	0.05	0.1
Methyl 3-nonenoate	13481-87-3	3710	0.1	0.2	0.4
Methyl thiobutyrate	2432-51-1	3310	0.005	0.01	0.02
Neofolione	111-79-5	2725	0.025	0.05	0.1
Nona-2-trans, 6-ci-Dienal	557-48-2	3377	0.0025	0.005	0.01
Nootkatone	4674-50-4	3166	0.0025	0.005	0.01
Propyl acetate	109-60-4	2925	0.1	0.2	0.4
Propyl caproate	626-77-7	2949	0.05	0.1	0.2
T,T,2,4, undecadienal	30361-29-6	3422	0.05	0.1	0.2
Thiomenthone	38462-22-5	3177	0.005	0.01	0.02
Trithioacetone	828-26-2	3475	0.025	0.05	0.1
Total percentage of neat flavor chemicals in flavor mixture 2			1.411%	2.82%	5.64%
Flavor mixture 3
Glycerin	56-81-5	2525	60	60	60
Propylene glycol	57-55-6	2940	39.4935	38.987	37.974
2-Isobutyl thiazole	18640-74-9	3134	0.01	0.02	0.04
3-Acetyl pyridine	350-03-8	3424	0.025	0.05	0.1
4-Vinyl guaiacol	7786-61-0	2675	0.0025	0.005	0.01
Acetoin acetate	4906-24-5	3526	0.015	0.03	0.06
acetophenone	98-86-2	2009	0.025	0.05	0.1
Amyl formate^a	638-49-3	2068	0.025	0.05	0.1
Butyl anthranilate	7756-96-9	2181	0.05	0.1	0.2
Cinnamyl isobutyrate	103-59-3	2297	0.005	0.01	0.02
Cinnamyl isovalerate	140-27-2	2302	0.0025	0.005	0.01
Cis-3-hexenyl butyrate ((Z)-3-hexen-1-yl butyrate)	16491-64-4	3402	0.05	0.1	0.2
Cis-3-hexenyl caproate	31501-11-8	3403	0.0025	0.005	0.01
diethyl Malonate	105-53-3	2375	0.025	0.05	0.1
diethyl Succinate	123-25-1	2377	0.025	0.05	0.1
Ethyl benzoate	93-89-0	2422	0.025	0.05	0.1
Ethyl-3-methylthiopropionate	13327-56-5	3343	0.0035	0.007	0.014
Hexyl 2-methylbutanoate	10032-15-2	3499	0.025	0.05	0.1
Hexyl caproate	6378-65-0	2572	0.025	0.05	0.1
Hexyl isobutyrate	2349-07-7	3172	0.025	0.05	0.1
Methyl benzoate	93-58-3	2683	0.025	0.05	0.1
methyl Heptenone	110-93-0	2707	0.025	0.05	0.1
Methyl nonyl ketone	112-12-9	3093	0.00025	0.0005	0.001
Milk lactone (5,6-decenoic acid)	72881-27-7	3742	0.025	0.05	0.1
Octanone-2 (methyl hexyl ketone)	111-13-7	2802	0.00025	0.0005	0.001
sulfuryl Acetate	656-53-1	3205	0.01	0.02	0.04
Valencene	4630-07-3	3443	0.025	0.05	0.1
Vanillyl ethyl ether	13184-86-6	3815	0.025	0.05	0.1
Whiskey lactone	39212-23-2	3803	0.005	0.01	0.02
Total percentage of neat flavor chemicals in flavor mixture 3			0.5065%	1.013%	2.026%
Flavor mixture 4
Glycerin	56-81-5	2525	60	60	60
Propylene glycol	57-55-6	2940	39.41	38.82	37.64
6-Amyl-alpha pyrone	27593-23-3	3696	0.025	0.05	0.1
6-Methyl coumarin	92-48-8	2699	0.0025	0.005	0.01
bis(2-Methyl-3-furyl)disulfide	28588-75-2	3259	0.05	0.1	0.2
Delta-3-carene	13466-78-9	3821	0.1	0.2	0.4
Ethyl cinnamate	103-36-6	2430	0.025	0.05	0.1
Furfuryl thio acetate	13678-68-7	3162	0.05	0.1	0.2
Linalool oxide	1365-19-1	3746	0.025	0.05	0.1
Methyl furfuryl disulfide	57500-00-2	3362	0.025	0.05	0.1
Methyl-2-furoate^a	611-13-2	2703	0.025	0.05	0.1
methyl-2methyl-3-furyl disulfide	65505-17-1	3573	0.05	0.1	0.2
Neryl acetate	141-12-8	2773	0.0025	0.005	0.01
sulfurol (4-Methyl-5-thiazole ethanol)	137-00-8	3204	0.075	0.15	0.3
Trans-2-decenal	3913-81-3	2366	0.05	0.1	0.2
Total percentage of neat flavor chemicals in flavor mixture 4			0.505%	1.01%	2.02%

^aSelected as the chemical marker for 6-week shelf-life study.

Two-week dose-range-finding studies

Flavor mixtures one and two were tested in the first 2-week dose-range-finding study and flavor mixtures three and four were tested in the second 2-week dose-range-finding study. The same study design was used for both 2-week dose-range-finding studies (Table 2). Animals in both 2-week dose-range-finding studies were exposed 6hr/d, 5 days/week for 2 weeks (10 days of exposure). A Capillary Aerosol Generator (CAG) was used to provide a continuous aerosol stream. The CAG has been used in several inhalation studies^20,41–43 and its design has been well documented.^44–50 Briefly, the CAG heats the vehicle control or flavor formulation that is pumped (one-eighth inch piston pump head and V-300 stroke rate controller; Fluid Metering Inc.; Syosset, NY) through it to a defined temperature (270 ± 10°C) creating a vapor that is subsequently cooled with humidified dilution air. The same target CAG temperature was used for all exposure groups (Table 2) and was controlled with Digi Sense Temperature Controller (TC9600, Cole-Palmer, Vernon Hills, IL) using resistive heating elements and thermocouples. This temperature range was chosen to mimic coil temperatures reported for Electronic Nicotine Delivery Systems.^49,51,52 To achieve the desired particle size distribution (Mass median aerodynamic diameter approximately 1 µm) and target aerosol concentration (2.5 mg/L), humidified dilution air was introduced into two places prior to the nose-only exposure towers. To cool the heated vapor from the CAG, approximately 90 L/min of room temperature humidified HEPA filtered dilution air was used. To adjust exposure concentration, approximately 37 L/min of room temperature HEPA filtered air was introduced at the top of the stainless-steel nose-only exposure towers (14.1 L internal volume) (Figure 1). Separate stainless-steel nose-only exposure towers were used for each exposure group (Table 2). To minimize non-inhalation routes of exposure and maximize respiratory tract exposures to the flavor mixtures nose-only exposures were used.^53,54 Animal position on the nose-only exposure towers was changed daily. Exposure atmosphere targeted mean temperature and relative humidity were 22°C ± 3°C and 50% ± 20%, respectively. Temperature and relative humidity monitoring for the vehicle control group occurred at least 3 times daily and for all flavor mixture groups at 60-minute intervals throughout the exposure period in each nose-only exposure tower using a Vaisala temperature and humidity probe (Model No. HMT110, Vaisala, Helsinki, Finland). To assess exposure to vehicle control and flavor mixtures, filter samples from the nose-ports of the nose-only inhalation towers used for each exposure group were collected and assessed weekly using validated analytical chemistry assays for propylene glycol, glycerin and water. Exposure concentrations were measured gravimetrically at least five times daily by collecting aerosol on pre-weighed 44 mm glass fiber filters (Grade F31904; GE Healthcare, Buckinghamshire, UK) and dividing the accumulated mass by the sampled air volume. Based upon these measurements, adjustments were made to dilution airflow to maintain the target exposure concentration. Since the loading of males and females animals were offset due to the number of animals, separate samples were collected for males and females. The mass median aerodynamic diameter (MMAD) and geometric standard deviation (GSD) were measured weekly to define the particle size distribution for each exposure group using a cascade impactor (InTox, Moriarty, NM; model 02-005 in brass).

Table 2.

Inhalation study design for 2-week dose range finding and 90-day studies.

Study	Group -test material	Target exposure concentration (mg/L)	Number of animals
			Two-week		90-days		Recovery
			Males	Females	Males	Females	Males	Females
1	Vehicle control¹	2.5	5	5	10	10	5	5
1	Flavor mixture 1 - low	2.5	5	5	10	10	5	5
1	Flavor mixture 1 - mid	2.5	5	5	10	10	5	5
1	Flavor mixture 1 - high	2.5	5	5	10	10	5	5
1	Flavor mixture 2 – low	2.5	5	5	10	10	5	5
1	Flavor mixture 2 – mid	2.5	5	5	10	10	5	5
1	Flavor mixture 2 - high	2.5	5	5	10	10	5	5
2	Vehicle control^a	2.5	5	5	10	10	5	5
2	Flavor mixture 3 - low	2.5	5	5	10	10	5	5
2	Flavor mixture 3 - mid	2.5	5	5	10	10	5	5
2	Flavor mixture 3 - high	2.5	5	5	10	10	5	5
2	Flavor mixture 4 – low	2.5	5	5	10	10	5	5
2	Flavor mixture 4 – mid	2.5	5	5	10	10	5	5
2	Flavor mixture 4 - high	2.5	5	5	10	10	5	5

^aVehicle was 40% USP grade propylene glycol/60% USP grade/Food Chemical Codex complaint glycerin (w/w).

Figure 1.

Schematic of the aerosol generation and exposure system.

90-Day inhalation studies

The same study design was used for both 90-day inhalation studies (Table 2). Animals in both 90-day inhalation studies were exposed 6hr/d, 5 days/week for 13 weeks. To assess reversibility of any potential changes (e.g., hematology, clinical chemistry, histopathology, etc.), a 43-day post-inhalation period (recovery) was used for sub-groups of rats (Table 2). To accommodate necropsy capacity limitations for flavor mixtures 1 and 2, exposure initiation was staggered so that the necropsy after 90 days of exposure occurred on days 92, and 93 while the recovery necropsy occurred on day 134 of the study. The same nose-only exposure towers, aerosol generation and monitoring of the exposure atmospheres that was conducted for the 2-week dose range finding studies was also performed in the 90-day inhalation studies. However, due to the number of animals for flavor mixtures one and two in the first 90-day inhalation study separate exposure towers were used for males and females. In addition to the endpoints evaluated in the 2-week studies, respiratory physiology was measured near the end of the exposure period, urine was collected for analysis immediately prior to necropsy and histopathology evaluation was performed on additional organs/tissues (Supplemental Table 1).

Biological endpoints

Twice daily in-life observations (before and after exposure with animals in their cages) were performed on exposure days and once per day on non-exposure days in the dose range finding and 90-day inhalation studies. Detailed clinical observations were conducted weekly for all exposed animals. Body weights (non-fasted) and food consumption were measured every other day in the 2-week dose-range-finding studies through termination and weekly for each animal upon arrival through termination in the 90-day studies. In the 90-day inhalation studies, ophthalmic examinations were performed by a board-certified veterinary ophthalmologist prior to exposure, during week 4 and 13 of exposure, and near the end of the recovery period using an indirect ophthalmoscope and slit lamp biomicroscope. Prior to ophthalmic examination animals were treated with tropicamide.

Confirmation of exposure was performed by measuring the concentration of propylene glycol in rat plasma using a validated analytical assay for all exposure groups, on day 5 in the 2-week dose-range-finding studies and on days 3 and last day of exposure in the 90-day studies.^20,43 Blood samples were obtained within 10 min post exposure in both the 2-week range finding studies and 90-day inhalation studies via jugular vein. In the 90-day inhalation studies animals from the recovery group animals (5 animals/sex/group) were used. An anticoagulant tube (K₂EDTA) was used to collect the plasma, which was gently mixed and centrifuged within 2-hr of collection. The resultant plasma was transferred to a polypropylene tube and stored at −70°C until analysis.

Respiratory physiology measurements (tidal volume and frequency) for vehicle control and high dose flavor mixture groups were not measured in the 2-week dose range finding studies, but were measured once during the 90-day inhalation studies (week 12 for flavor mixtures one and two and week 11 for flavor mixture 3) using a nose-only single-chamber restraint plethysmograph (Buxco Electronics, Inc., Wilmington, NC). Five animals/sex/group were assessed. Baseline measurements started 10 min prior to exposure and continued for the first 30 min of exposure with only measurements collected during the last 15 min of exposure used for statistical analysis. Data acquisition logging rate was set to 5 s and respiratory data were sampled at 500 Hertz. Any respiratory frequency data exceeding 350 breaths/minute or tidal volume data greater than 5.0 mL was considered physiologically unlikely and excluded from analysis. Minute volume was calculated as the product of the measured tidal volume and respiratory frequency.

Terminal endpoints

In the 90-day studies, animals were fasted for at least 8 h prior to blood collection while housed in metabolism cages for urine collection. Blood was collected via the jugular vein and placed into tubes with potassium K₂EDTA for hematology, and tubes with no anticoagulant for clinical chemistry. Blood was collected from anesthetized animals (using isoflurane inhalation) into tubes with sodium citrate for coagulation parameters at the time of euthanasia via the vena cava. The euthanasia was completed by exsanguinating the anesthetized animals.

Urinary volume, pH, and specific gravity as well as glucose, ketone and protein concentration were measured. The hematology endpoints included red blood cell indices (mean corpuscular hemoglobin concentration, mean corpuscular volume, red blood cell and reticulocyte count and hematocrit), platelet count, mean platelet volume, white blood cell count, differential count, as well as coagulation parameters (function included prothrombin time, activated partial thromboplastin and fibrinogen). Clinical chemistry endpoints included: Total Protein, Albumin, Globulin, Albumin/Globulin ratio, Alkaline phosphatase, Alanine aminotransferase, Aspartate aminotransferase, Gamma glutamyltransferase, Cholesterol, Creatine kinase, Creatine, Glucose, Triglycerides, Total bilirubin Direct and Indirect bilirubin, Sodium, Calcium, Potassium, Chloride, Phosphorus and Urea Nitrogen.

In both the 2-week dose-range-finding and 90-day inhalation studies, three bronchoalveolar lavages (BALF) were performed on the right lung of all animals after lung weight was recorded and before the lungs were inflated with fixative. The cell pellets from the three lavages per animal were combined and resuspended for counting. Total cell counts were obtained using a hemocytometer. Total protein concentration, lactate dehydrogenase, and alkaline phosphatase enzyme activities from the supernatant of the first lavage were determined using automated methods. For differential counts (cytology), at least 200 cells per slide (cytospin preparation) were identified (if possible) as macrophages, neutrophils, lymphocytes, eosinophils, basophils, or others.

Macroscopic evaluation for gross lesions was conducted on all animals at all necropsies (Supplemental Table 1). Organ weights (absolute) were obtained for all organs at each necropsy and paired organs were weighed together (Supplemental Table 1). In the dose-ranging-finding study organ weight to terminal (fasted) body weight ratio was calculated and in the 90-day studies organ weight to terminal (fasted) body weight and organ weight to brain weight ratios were calculated. In the dose-ranging-finding studies (after 10 days of exposure), histopathological evaluation was performed on respiratory tract tissues (including six different sections of the nose from oropharynx to nasopharynx, larynx, trachea, main carina-intersection of trachea and two major bronchi, extrapulmonary bronchus, and lung) and selected other organs from vehicle control and all flavor mixture groups (Supplemental Table 1). After 90 days of exposure, the same respiratory tract tissues, and a more complete list of organs (Supplemental Table 1) were evaluated. After recovery, histopathology was only performed on tissues of interest in the high dose flavor mixture groups after 90 days of exposure. All histopathology was performed by a board-certified veterinary pathologist using a 5-level severity grade scale of minimal, mild, moderate, marked and severe who also identified and examined the tissues of interest at recovery. Animals were fasted the day prior to necropsy.

Statistical analysis

Statistical comparisons between flavor mixture groups and the respective vehicle control group were performed to indicate where potentially toxicologically significant differences might be present. In addition to being statistically significantly different, expert interpretation which included analysis of factors such as dose-response and coherence among findings were required for a result to be considered toxicologically significant. The 5% level (p < .05) for statistical significance was used throughout the evaluations. SAS®, ver. 9.4 (Cary N.C.) was used for statistical analysis.

In general, means, standard deviations (or % coefficient of variation or standard error, when deemed appropriate), ratio, percentages, numbers, and/or incidences were reported depending on the dataset. The pairwise comparisons for statistical analysis in the 2-week dose range finding studies compared each flavor mixture group to the respective vehicle control (e.g., Low dose flavor mixture 1 vs Study 1 Vehicle Control; Mid dose flavor mixture 1 vs Study 1 Vehicle Control; etc.). The same comparisons were performed in the 90-day studies for the flavor mixtures studied. Any group with less than three observations were excluded from analysis. Levene’s test was used to assess the homogeneity of group variances. The groups were compared using an overall one-way ANOVA F-test if Levene’s test was not significant or the Kruskal-Wallis test if it was significant. If the overall F-test or Kruskal-Wallis test was significant, then pairwise comparisons using two sided tests were conducted using Dunnett’s or Dunn’s test, respectively.

Descriptive statistics for each respiratory physiology data time period were tabulated and included: arithmetic mean (Mean), number of subjects (N), least squares mean (LS Mean), and standard error of the LS Mean (LSM s.e.). Analysis of covariance was used to evaluate the effect of exposure to flavor ingredient mixture during each analysis segment using exposure group and baseline data (covariate) (average of 10 min prior to exposure) as factors in the model. If a significant effect to flavor mixture exposure was found, step-down adjusted t-tests were used to compare exposure to each flavor mixture to the respective vehicle control treatment using the overall segment means. Step-down simulated adjustment was used to adjust t-tests for pair-wise comparisons, which were evaluated at the 0.05 level of significance, using two-tailed tests.

Results

Two-week dose-range-finding studies

In the first 2-week dose-range-finding study all exposure concentration and particle size distribution parameters were within acceptable ranges (Supplemental Table 2). Exposure to the flavor mixtures one and two was confirmed with analytical measurements of propylene glycol in rat plasma (Supplemental Table 3). All animals except for one male in the flavor mixture 1, low dose group survived to the scheduled necropsy. The cause of death for this animal could not be determined, however due to the lack of effect in other animals, including the mid and high dose groups of flavor mixture 1, a test article-related effect was considered unlikely. Notable microscopic changes in this animal consisted of mild perivascular mixed infiltrates in the lungs and minimal alveolar proteinosis. There were no other microscopic findings in this animal. No statistically significant differences in mean body weights were found (Figure 2; Supplemental Tables 4 and 5). Although a few differences (statistically significant and non-statistically significant) were noted in body weight gain, food consumption, hematology, clinical chemistry, BALF, absolute organ weights or organ weights normalized to body mass, these were sporadic in nature, consistent with biological variation with no consistency across doses, sexes or time points (Supplemental Tables 6-19). There was also no flavor mixture one or two macroscopic (Supplemental Tables 20) or histopathology findings (Figure 3; Supplemental Tables 38). Based upon the lack of findings in the 2-week dose range finding study, the flavor mixture exposure level of 2.5 mg/L was used in the subsequent (first) 90-day inhalation study.

Figure 2.

Two-week dose range finding studies mean bodyweight as function of exposure day for males (top panels) and females (bottom panels) with Study one in the left column and Study two in the right column.

Figure 3.

Representative images of the nasal turbinates (section IV) from female mice exposed to vehicle (a) and high dose groups of Flavor Mixtures 1 (b), 2 (c), 3 (d), and 4 (e) from both 2-week dose range finding studies. Rats exposed to the high dose of Flavor Mixture four showed notable, diffuse thinning of the olfactory epithelium throughout the nasal turbinates (arrows) at low magnification. Vehicle exposed rat olfactory epithelium (f) shows the orderly arrangement of nuclei of the olfactory sensory neurons with thin cilia projecting into the nasal lumen. Flavor Mixture four high dose group exposed rat olfactory epithelium (g) shows notable degeneration is present, characterized by loss of sensory neurons and sustentacular cells, cellular apoptosis (filled arrowheads), presence of inflammatory cells (open arrowhead), and loss of cilia along with evidence of regeneration (disorganized and poorly differentiated cells with large nuclei). Hematoxylin and Eosin (H&E) staining. Magnification is 30x and scale bar is 700 µm for A, B, C, D, E. Magnification is 400x and scale bar is 60 µm for F and G.

In the second 2-week dose-range-finding study (flavor mixtures 3 and 4) all animals survived to the scheduled necropsy and all exposure concentration and particle size distribution parameters were within acceptable ranges (Supplemental Table 2). Exposure to flavor mixtures three and four was confirmed with analytical measurements of propylene glycol in rat plasma (Supplemental Table 3). There were also a few differences (statistically significant and non-statistically significant) noted in clinical observations, hematology, clinical chemistry, BALF endpoints, absolute organ weights, organ weights relative to body weight or macroscopic findings, these were sporadic in nature, consistent with biological variation with no consistency across doses, sexes, or time points (Supplemental Tables 21-37). A statistically significantly lower body weight was measured on day 8 for female rats in flavor mixture 4 (Supplemental Table 22). Lower (not statistically significant) mean body weight gains and food consumption (Supplemental Tables 23-26) resulting in a non-statistically significant subsequent lower mean body weight on day 12 was observed for female rats in the flavor mixture four high dose group (Figure 2; Supplemental Table 22). No flavor mixture three related effects on body weight gains, food consumption (Supplemental Tables 23 and 25) or body weight on day 12 was observed (Figure 2: Supplemental Table 23). No adverse histopathology results were noted for flavor mixture three exposed animals (Figure 3; Supplemental Table 39), however olfactory epithelial degeneration/regeneration was noted in nasal levels III, IV, V, and VI for female rats in the flavor mixture four high dose group (Figure 3; Supplemental Table 39). Due to the adverse nasal histopathology findings flavor mixture four was not evaluated in a subsequent 90-day inhalation study. Based upon the lack of findings in the 2-week dose-range-finding study for flavor mixture 3, the same flavor mixture three target exposure level of 2.5 mg/L was used in the subsequent (second) 90-day inhalation study.

90-Day inhalation studies – exposure characterization

All mean temperature and relative humidity measurements from each nose-only exposure tower were within the target ranges (19°C–25°C and 30%–70%; Table 3). All mean exposure concentrations were within 4% (0.1 mg/L) of the 2.5 mg/L target exposure concentration with no statistically significant differences between groups (Table 3). Mean MMADs met the OECD guideline criteria of <2 microns³⁵ ranging from 1.1 to 1.6 microns, which was slightly larger than the targeted 1 micron (Table 3).

Table 3.

Characterization of 90-day inhalation exposures (mean ± SD).

Study	Test material – Sex	Mean ± standard deviation (SD)
		Temperature °C^a	Relative humidity %^a	Exposure concentration mg/L^a (N = 396)	Particle size distribution (N = 13)		Exposure chemical characterization mg/L (N = 78)
					MMAD	GSD	Propylene glycol		Glycerin		Water
					MMAD	GSD	Mean ± SD	% Of Exp. Con	Mean ± SD	% Of Exp. Con	Mean ± SD
1	Vehicle control
	Males	22.6 ± 0.7	46.4 ± 2.2	2.6 ± 0.2	1.6 ± 0.2	1.8 ± 0.1	0.76 ± 0.06	29.2	1.50 ± 0 × 10^f	57.7	0.57 ± 0.05^f
	Females	22.6 ± 0.7	46.3 ± 2.2	2.6 ± 0.2	1.6 ± 0.2	1.8 ± 0.1	0.77 ± 0.06	29.6	1.50 ± 0.10	57.7	0.57 ± 0.05
1	Flavor mixture 1 – low
	Males	22.4 ± 0.4	44.6 ± 1.5	2.4 ± 0.2	1.3 ± 0.2	1.8 ± 0.1	0.67 ± 0.04	27.9	1.40 ± 0.09^f	58.3	0.56 ± 0.06^f
	Females	22.3 ± 0.6	44.5 ± 1.5	2.4 ± 0.2	1.3 ± 0.2	1.8 ± 0.1	0.67 ± 0.04	27.9	1.40 ± 0.09	58.3	0.56 ± 0.05
1	Flavor mixture 1 – mid
	Males	22.1 ± 0.4	45.3 ± 1.4	2.5 ± 0.2	1.4 ± 0.2	1.8 ± 0.1	0.70 ± 0.06	28.0	1.46 ± 0.11^f	58.4	0.55 ± 0.05^f
	Females	22.1 ± 0.5	45.2 ± 1.4	2.5 ± 0.2	1.4 ± 0.2	1.8 ± 0.1	0.69 ± 0.06	27.6	1.44 ± 0.09	57.6	0.56 ± 0.05
1	Flavor mixture 1 – high
	Males	21.7 ± 0.5	46.6 ± 1.5	2.6 ± 0.3	1.3 ± 0.3	1.9 ± 0.1	0.86 ± 0.04	33.1	1.23 ± 0.12^f	47.3	0.54 ± 0.04^f
	Females	21.7 ± 0.6	46.5 ± 1.5	2.6 ± 0.2	1.3 ± 0.3	1.8 ± 0.1	0.85 ± 0.04	32.7	1.22 ± 0.11	46.9	0.54 ± 0.04
1	Flavor mixture 2 – low
	Males	21.5 ± 0.8	47.9 ± 1.8	2.5 ± 0.2	1.3 ± 0.2	1.9 ± 0.1	0.70 ± 0.04	28.0	1.42 ± 0.04^f	56.8	0.55 ± 0.06^f
	Females	21.5 ± 0.7	47.7 ± 1.6	2.5 ± 0.2	1.3 ± 0.2	1.9 ± 0.1	0.70 ± 0.04	28.0	1.42 ± 0.04	56.8	0.55 ± 0.06
1	Flavor mixture 2 – mid
	Males	21.6 ± 0.8	51.5 ± 2.1	2.6 ± 0.1	1.3 ± 0.2	1.8 ± 0.1	0.69 ± 0.03	26.5	1.51 ± 0.07^f	58.1	0.58 ± 0.06^f
	Females	21.7 ± 0.7	51.2 ± 2.0	2.6 ± 0.2	1.3 ± 0.2	1.8 ± 0.1	0.69 ± 0.03	26.5	1.50 ± 0.08	57.7	0.58 ± 0.06
1	Flavor mixture 2 – high
	Males	22.2 ± 0.6	46.1 ± 1.7	2.5 ± 0.2	1.6 ± 0.3	1.8 ± 0.1	0.61 ± 0.09	24.4	1.42 ± 0.02^g	56.8	0.55 ± 0.05^f
	Females	22.3 ± 0.5	45.8 ± 1.4	2.5 ± 0.2	1.6 ± 0.3	1.8 ± 0.1	0.60 ± 0.09	24.0	1.40 ± 0.21	56.0	0.55 ± 0.05
2	Vehicle control	22.6 ± 0.4	43.7 ± 2.2	2.5 ± 0.3^d	1.1 ± 0.1	1.6 ± 0.2	0.65 ± 0.12^b	26.0	1.37 ± 0.18^b	54.8	0.45 ± 0.09^c
2	Flavor mixture 3 - low	22.9 ± 0.4	42.2 ± 2.0	2.5 ± 0.2	1.3 ± 0.2	1.7 ± 0.2	0.64 ± 0.05	25.6	1.41 ± 0.09	56.4	0.45 ± 0.10
2	Flavor mixture 3 - mid	22.8 ± 0.5	44.0 ± 2.1	2.5 ± 0.3^e	1.4 ± 0.2	1.8 ± 0.2	0.66 ± 0.10	26.4	1.37 ± 0.19	54.8	0.44 ± 0.08
2	Flavor mixture 3 - high	23.3 ± 0.6	41.8 ± 2.1	2.5 ± 0.1	1.6 ± 0.1	1.8 ± 0.1	0.66 ± 0.05	26.4	1.45 ± 0.13	58.0	0.45 ± 20.08

Exp. = Exposure; Con. = concentration; SD = standard deviation.

^aMeasurements made over 66 days.

^bN = 81.

^cN = 79.

^dN = 412.

^eN = 403.

^fN = 77.

^gN = 76.

As expected with use of humectants, propylene glycol and glycerin, as the primary formulation ingredients, chemical analysis of nose-port aerosol samples confirmed that approximately 18%–21% of those samples were water (Table 3). Using the actual mean exposure concentration and measured amounts of propylene glycol and glycerin measured for each group (Table 3), the percentage of their measured aerosol mass ranged between approximately 24%–33% and 47%–58%, respectively. Serum measurements of propylene glycol concentration (Table 4) confirmed exposure for all groups, with values for males in all groups being lower at day 91 than day 3.

Table 4.

Rat Plasma Propylene Glycol Concentrations (μg/ml) on Days 3and 91 of Exposures. All values represent the mean of five replicates unless otherwise noted.

Study	Test material – Sex	Units	Day 3	Day 91
Study	Test material – Sex	Units	Mean ± SD	Mean ± SD
1	Vehicle control
	Males	µg/ml	42.6 ± 6.8	28.1 ± 3.8
	Females	µg/ml	48.6 ± 14.9	41.4 ± 17.5
1	Flavor mixture 1 – low
	Males	µg/ml	46.9 ± 7.3	33.7 ± 6.9
	Females	µg/ml	38.0 ± 7.4	40.5 ± 3.9
1	Flavor mixture 1 – mid
	Males	µg/ml	52.9 ± 4.6	30.7 ± 3.0
	Females	µg/ml	40.0 ± 6.6	40.8 ± 6.7
1	Flavor mixture 1 – high
	Males	µg/ml	51.5 ± 9.6	36.0 ± 7.4
	Females	µg/ml	62.7 ± 8.4	46.1 ± 4.5
1	Flavor mixture 2 – low
	Males	µg/ml	48.9 ± 13.4	29.3 ± 4.5
	Females	µg/ml	42.1 ± 8.7	42.6 ± 12.4
1	Flavor mixture 2 – mid
	Males	µg/ml	51.3 ± 14.5	25.8 ± 5.5
	Females	µg/ml	43.2 ± 9.4	40.7 ± 12.7
1	Flavor mixture 2 – high
	Males	µg/ml	35.4 ± 4.7	22.5 ± 2.7
	Females	µg/ml	32.0 ± 4.9	30.6 ± 9.0
2	Vehicle control
	Male	µg/ml	63.9 ± 11.5	28.7 ± 5.3
	Females	µg/ml	58.3 ± 15.6	30.4 ± 8.2
2	Flavor mixture 3 – low
	Males	µg/ml	42.1 ± 8.5	30.1 ± 6.0
	Females	µg/ml	33.6 ± 3.5	26.2 ± 3.7
2	Flavor mixture 3 – mid
	Males	µg/ml	45.0 ± 14.6	26.3 ± 4.9
	Females	µg/ml	33.8 ± 14.4	28.4 ± 1.2
2	Flavor mixture 3 – high
	Males	µg/ml	53.0 ± 7.7	29.5 ± 4.9
	Females	µg/ml	42.7 ± 7.0	24.5 ± 2.3

90-Day inhalation studies - mortality/morbidity/in-life observations

In the first 90-day inhalation study using flavor mixtures 1 and 2, there were two animals that did not survive to the scheduled necropsy. A male in the flavor mixture 2, high dose was found dead prior to exposure on day 87. There were no clinical observations, effects on body weight, or macroscopic or microscopic findings and the cause of death was undetermined. Given the lack of mortality in other treatment groups, this finding was considered unrelated to flavor mixture exposure. The second unscheduled death occurred in a female rat in the flavor mixture 1, high dose group that was euthanized in extremis during the recovery period on Day 104 following clinical observations of decreased activity, partially closed eyes, hunched posture, labored breathing, and erected fur. In addition, this animal had lost 11.9% of body weight between days 99–104. The cause of death was attributed to malignant lymphoma that was observed in multiple organs and tissues throughout the body. Spontaneous lymphohematopoietic neoplasia, including malignant lymphoma, has been reported in young Sprague Dawley rats including female Sprague-Dawley rats 11-15 weeks old.^55,56 No other indications of lymphoma were found in any of the other female rats in flavor mixture one regardless of dose group, therefore in this case, malignant lymphoma was considered an incidental finding and not related to flavor mixture exposure. In the second 90-day inhalation study using flavor mixture 3, all animals survived until the scheduled necropsies. There were no flavor mixture related clinical observations or ophthalmic lesions indicative of toxicity in either 90-day inhalation study.

Mean group body weights (Figure 4; Supplemental Tables 40-42) were unaffected by exposure to any of the flavor mixtures at any dose level. Although there were sporadic statistically significantly differences in body weight gains, they did not correlate with food consumption data as there were no statistically significant differences in food consumption between flavor mixture exposure groups and respective vehicle control groups (Supplemental Tables 43-48). Because there was no consistent pattern between successive body weight gain measurements nor between sexes within the same measurement and in all but one case (Mix 3-M and Mix3-H group male rats) no dose response (Supplemental Tables 43-45), the sporadic statistically significant differences in body weight gain were not considered flavor mixture exposure related.

Figure 4.

90-day studies mean bodyweights for males (top panels) and females (bottom panels) with Study in the left column and Study two in the right column. Recovery animals were not fasted after 90 days of exposure as were the rest of the animals that were fasted prior to necropsy.

The only statistically significant difference in respiratory physiology was a higher respiratory frequency noted in the Mix1-H group females at 16-30 min postdosing (Supplemental Table 49). This change was not considered to be flavor mixture related because it was of minimal difference compared to the baseline respiratory frequency.

90-Day inhalation studies – terminal endpoints

Although there were several differences (statistically significant and non-statistically significant) in hematology, coagulation, clinical chemistry, and urinalysis endpoints these were sporadic in nature, consistent with biological variation with no consistency across doses, sexes, or time points (Hematology – Supplemental Tables 50-57; Coagulation – Supplemental Tables 58-65; Clinical chemistry – Supplemental Tables 66-73; Urinalysis – Supplemental Tables 74-81). For example, there were no statistically significant changes in hematology endpoints during the 90-day inhalation study for any flavor mixture exposure group compared to their respective vehicle control group (Supplemental Tables 50-53), but during the recovery phase of the study there were statistically significant differences between flavor mixture exposure groups and the respective vehicle control groups (Supplemental Tables 54-57). For flavor mixture 1 at recovery in female rats at the low and mid dose groups, monocytes levels were statistically significantly lower than the level in vehicle control, however the level of monocytes were not different than vehicle control for the high dose group and no differences were seen at recovery for male rats. For coagulation endpoints the only three statistically significant differences occurred in flavor mixture three exposure groups compared to vehicle control values. After 13 weeks of exposure to flavor mixture 3, male rats in the mid and high dose groups had statistically significantly less fibrinogen (mg/dL), however this decrease was not dose responsive and was not seen in female rats (Supplemental Table 59). At the end of the recovery, female rats exposed to the low dose of flavor mixture 3, had a statistically significantly higher prothrombin time compared to vehicle control, however there was no dose response and was not present in male rats (Supplemental Table 65). For clinical chemistry endpoints, the only two statistically significant differences occurred in female rats exposed to flavor mixture 3. After 90 days of exposure, female rats in the low dose group had statistically significantly less aspartate aminotransferase and female rats in the high dose group had statistically significantly less chloride, however there was not a dose response, and a similar pattern was not present in male rats (Supplemental Tables 66-69). For urinalysis there were three statistically significant differences from vehicle control in male rats exposed to flavor mixture one and one statistically significant difference from vehicle control for female rats exposed to flavor mixture 2. The male rats exposed to flavor mixture one in the mid and high dose groups had statistically significantly higher urinary pH and the mid dose group also had a higher urinary volume compared to vehicle control. The female rats exposed to flavor mixture 2 at the mid dose group had a statistically significant lower urinary pH than the vehicle control. None of the statistically significant differences in urinary endpoints exhibited a dose response or were evident in the opposite sex.

There were no statistically significant changes in BALF cytology, total protein, or lactate dehydrogenase, for any flavor mixture exposure group after 90 days of exposure when compared to vehicle control values (Supplemental Tables 82-87). The only statistically significant difference after 90 days of exposure in BALF endpoints was a statistically significant increase in bronchoalveolar alkaline phosphatase level in female rats exposed to the high dose of flavor mixture three compared to vehicle control (Supplemental Table 84). However, at recovery there were two statistically significant differences that were found in flavor mixture 3 exposed groups: more lymphocytes/µL were found in male rats exposed at the low dose and a lower percent neutrophiles in female rats exposed at the high dose (Supplemental Table 87). None of the statistically significant differences in BALF endpoints exhibited a dose response or were evident in both sexes.

The only statistically significant differences in absolute organ weights after 90 days of exposure occurred in male rats exposure to flavor mixture three at the low dose (Supplemental Tables 88-91). Statistically significantly lower organ weights compared to vehicle control organ weights were found for the epididymis, pituitary gland, kidney, and testis (Supplemental Table 90). When compared on a percent terminal body weight and brain weight basis only the pituitary gland remained statistically significantly lower compared to the vehicle control (Supplemental Tables 92-99). There were no microscopic correlates and there was no dose response so these differences were considered incidental and not related to flavor mixture three exposure. After 90 days of exposure when compared on a terminal body weight or brain weight basis, there were other isolated organ weight differences in flavor mixture groups compared to their respective vehicle control groups, however there was no dose response or other patterns, or correlating data to suggest these values were toxicologically relevant. Interesting, although not observed after 90 days of exposure, after recovery, there were statistically significantly higher mean thyroid/parathyroid weights observed in male rats for the flavor mixture one low, mid and high dose groups and flavor mixture two high dose group (absolute organ weights and organ weight relative to terminal body weight and brain weight) and in the flavor mixture 2 mid dose group (organ weight relative to brain weight) compared to the mean thyroid/parathyroid weight from the vehicle control group (Supplemental Table 100-111). However, most individual rat values fell within the contract research organization’s historical control range, with the exception of one flavor mixture two high dose male rat. As noted above, significant differences in thyroid/parathyroid weights (absolute, percent body weight, or per brain weight) were not observed in animals after 90 days of exposure and there was no significant difference in thyroid/parathyroid weights in the recovery group females relative to the vehicle controls. Further, there were no microscopic correlates. Therefore, higher mean thyroid/parathyroid weights were attributed to individual animal variation and not flavor mixture exposure. At recovery, there were other isolated organ weight values that were statistically different from their respective vehicle controls. There were, however, no patterns, trends, or correlating data to suggest these values were toxicologically relevant. Thus, these other organ weight differences observed were considered incidental and unrelated to flavor mixture exposure. There were no notable findings in the macroscopic examination from any of the flavor mixture exposed groups when compared to their respective vehicle control groups after 90 days of exposure or at recovery (Supplemental Tables 112-116).

Histopathology results after 90 days of exposure showed goblet cell hyperplasia and metaplasia, transitional epithelial cell hyperplasia, and degeneration/regeneration of the olfactory epithelium in the nasal sections of both vehicle control and all flavor mixture exposed animals, with a similar incidence and/or severity in all groups and therefore, were not considered flavor mixture exposure related (Figures 5 and 6; Supplemental Tables 117-118). Other histopathology observations after 90 days of flavor mixture exposure were considered incidental, of the nature commonly observed in this strain and age of rats, and/or were of similar incidence and severity in vehicle control and flavor mixture exposed groups and, therefore, were considered unrelated to flavor mixture exposure (Supplemental Tables 119-120). At the recovery necropsy, the microscopic findings observed were considered incidental, of the nature commonly observed in this strain and age of rats, and/or were of similar incidence in vehicle control and flavor mixture exposed groups and, therefore, were considered unrelated to flavor mixture exposure (Supplemental Tables 121-122).

Figure 5.

Representative images of the nasal turbinates (section IV) and lungs from a single control female rat (a and b), high dose group Mixture one exposed female rat (c and d), and high dose group Mixture two exposed female rat (e and f). The olfactory epithelium of the nasal turbinates were microscopically similar between the control female (a), the high dose group Mixture 1 (c) and 2 (e) exposed female rats with no evidence of degeneration or regeneration. Similarly, the lungs from the control female rats (b) and the high dose group Flavor Mixtures 1 (d) and 2 (f) exposed female rats were similar and unremarkable. Hematoxylin and Eosin (H&E) staining. Magnification is 30x and scale bar 700 µm is for A, C, E. Magnification is 100x and scale bar is 300 µm for B, D, F.

Figure 6.

Representative images of the rat nasal turbinates (section IV) and lungs from a single control female (a and Bb) and high dose group of flavor mixture three exposed female rat (c and d). The olfactory epithelium of the nasal turbinates were microscopically similar between the control female (a) and high dose group of flavor mixture three exposed female rat (c) with no evidence of degeneration or regeneration. Similarly, the lungs from the control female (b) and high dose group of flavor mixture three exposed female rat (d) were similar and unremarkable. Hematoxylin and Eosin (H&E) staining. Magnification is 30x and scale bar is 700 µm for A and C. Magnification is 100x and scale bar is 300 µm for B and D.

Discussion

In the 2-week dose range finding studies, despite having one of the lower total flavor loads, exposure to flavor mixture four at 2.5 mg/L demonstrated significant toxicity that resulted in flavor mixture 4 not being studied in the subsequent 90-day inhalation studies. Consistent with a primary disadvantage of testing mixtures of flavor chemicals, we don’t have any insight into which flavor chemical, combination of flavor chemicals or their potential reaction/degradation product(s) might be responsible for the observed results for flavor mixture 4. In vitro cytotoxicity studies in rat or human nasal epithelium with exposure to each flavor chemical in mixture four might help identify an individual chemical or chemicals of concern.^57,58 No individual flavor ingredient inhalation studies could be found for any of the flavor chemicals in flavor mixture 4. None of the flavor chemicals in flavor mixture 4 were evaluated or tested in the previously described read across approach for 245 flavor chemicals for use in ENDS products.^{21,22,26–31} A few flavor chemicals in flavor mixtures 1, 2, and 3 were part of the previously described read across approach for 245 flavor chemicals (beta-damascenone, beta ionone, ethyl acetate, hexanal, l-menthyl acetate, Nootkatone, and Whiskey lactone) for use in ENDS products.^{21,22,26–31}

In the 90-day inhalation studies, the mean measured exposure temperature and humidity being within respective target ranges combined with mean measured exposure concentrations being within 0.1 mg/L (4%) of the target exposure concentration and the similarity in the particle size distribution among the flavor mixture exposure groups, enables comparison of biological responses between these flavor mixture exposure groups and their respective vehicle control group. The consistency in the amount of water measured in the aerosol was not surprising due to use of the humectants propylene glycol and glycerin in the vehicle control and flavor formulation groups. The humidified dilution air used was the likely water source. The chemical measurements of propylene glycol and glycerin at the nose ports are also generally consistent with their proportion in the flavor mixture formulations considering the rapid water absorption and rapid propylene glycol evaporation from aerosols recently reported.⁵⁹ There was no statistically significant difference in measured rat plasma propylene glycol concentrations between groups within time points. Although sporadic statistically significant differences from vehicle control were noted for some endpoints in the flavor mixture exposed groups (e.g., less fibrinogen in mid and high Flavor mixture three dose group female rats, tidal volume; etc.), no dose-response or consistency in response between sexes was noted. No Flavor Mixture exposure effects were observed on body weight, food consumption, respiratory physiology, hematology, coagulation, clinical chemistry, urinalysis, bronchoalveolar lavage fluid parameters, organ weights, ophthalmic endpoints, macroscopic observations, or histopathology findings. These two 90-day subchronic nose only inhalation studies demonstrated that daily exposures to the three flavor mixtures up to 2.5 mg/L TPM for 6 h/day did not induce biologically meaningful adverse effects in rats. Results of our 90-day inhalation studies using three different mixtures of flavor chemicals are consistent with those using the read across approach for 245 Flavor Ingredients used in ENDS products in that the mixture of flavor chemicals tested did not result in additional toxicity compared to the vehicle control.^21,22

In summary, administration of the three flavor mixtures by nose-only inhalation to Crl:CD(SD) rats at a target concentration of 2.5 mg/L of aerosolized formulations for 6 h per day for 5 days per week for 13 weeks was well tolerated. Based on the absence of adverse effects, the NOAEC was considered to be the highest dose tested, 2.5 mg/L of aerosolized high dose for each flavor mixture.

Supplemental Material

Supplemental Material - Evaluation of mixtures of flavor chemicals in a 90-day nose-only exposures in sprague-dawley rats

Supplemental Material for Evaluation of mixtures of flavor chemicals in a 90-day nose-only exposures in sprague-dawley rats by Michael J Oldham, Rahat Wadhwa Desai, James Randazzo, Brent E Walling, Guy Lalonde, and Roxana Weil in Human & Experimental Toxicology

Footnotes

Acknowledgments

The authors would like to thank the staff at Charles River Laboratories for a well-conducted study and Maleeha Agha for assistance with data presentation.

Author contributions

M.J.O. and R.W. conceived the study, R.W. obtained funding for the study, M.J.O., R.W.D., J.R., and G.L. designed the study, R.W.D. monitored the study, J.R. was the study director, B.E.W. was the veterinary pathologist for the study and who provided the histopathology images, M.J.O. wrote the first draft and all authors reviewed and edited the manuscript and approved the final version of the manuscript.

Declaration of conflicting interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: All authors are employees of Juul Labs, Inc. or the contract research organization that was compensated for performing the work.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The work was funded by Juul Labs, Inc.

ORCID iD

Michael J Oldham

Data Availability Statement

Data sets are provided in the supplemental material.*

Supplemental Material

Supplemental material for this article is available online.

References

Alhadyan

Sivaraman

Onyenwoke

. E-Cigarette flavors, sensory perception, and evoked responses. Chem Res Toxicol 2022; 35: 2194–2209. DOI: 10.1021/acs.chemrestox.2c00268.

COT, The Committee on Toxicity of Chemicals in Food, Consumer Products and the Environment (COT) . Framework for risk assessment of flavouring compounds in electronic nicotine (and non-nicotine) delivery systems (E(N)NDS – e-cigarettes), Report, COT Statement Number 2020/01, 2019.

Effah

Taiwo

Baines

, et al. Pulmonary effects of e-liquid flavors: a systematic review. J Toxicol Environ Health B Crit Rev 2022; 25: 343–371. DOI: 10.1080/10937404.2022.2124563.

Stefaniak

LeBouf

Ranpara

, et al. Toxicology of flavoring- and cannabis-containing e-liquids used in electronic delivery systems. Pharmacol Ther 2021; 224: 107838. DOI: 10.1016/j.pharmthera.2021.107838.

Pepelko

Withey

. Methods for route-to-route extrapolation of dose. Toxicol Ind Health 1985; 1: 153–170.

Rennen

MAJ

Nordheim

Houben

, et al. Prediction of local irritant effects after repeated dermal and respiratory exposure to chemicals. Reg Tox Pharm 2002; 36: 253–261.

Rennen

MAJ

Bouwman

Wilschut

, et al. Oral-to-inhalation route extrapolation in occupational health risk assessment: a critical assessment. Reg Tox Pharm 2004; 39: 5–11.

Ford

Davis

Gardner

. Development of inhalation thresholds of concern for the evaluation of tobacco products. Toxicologist. 2006; 2317: 474.

Schröder

Escher

Hoffmann-Dörr

, et al. Evaluation of route-to-route extrapolation factors based on assessment of repeated dose toxicity studies compiled in the database RepDose®. Toxicol Lett 2016; 261: 32–40.

10.

ECHA, European Chemicals Agency . Guidance on information requirements and chemical safety assessment. Chapter R.8: characterisation of dose [Concentration]-Response for human health. Helsinki, Finland: European Chemical Agency, 2012, https://echa.europa.eu/documents/10162/13632/information_requirements_r8_en.pdf (accessed 3 November 2023).

11.

Werley

Kirkpatrick

Oldham

, et al. Toxicological assessment of a prototype e-cigaret device and three flavor formulations: a 90-day inhalation study in rats. Inhal Toxicol 2016; 28(1): 22–38.

12.

Lee

Hoeng

Harbo

, et al. Biological changes in C57BL/6 mice following 3 weeks of inhalation exposure to cigarette smoke or e-vapor aerosols. Inhal Toxicol 2018; 30: 553–567. DOI: 10.1080/08958378.2019.1576807.

13.

Desai

Demir

Tsolakos

, et al. Comparison of the toxicological potential of two JUUL ENDS products to reference cigarette 3R4F and filtered air in a 90-day nose-only inhalation toxicity study. Food Chem Toxicol 2023; 179: 113917. DOI: 10.1016/j.fct.2023.113917.

14.

Christoph GR Subchronic Inhalation Neurotoxicity Study of Ethyl Acetate in Rats . Report, Haskell Laboratory Report No. 454-96; OTS0558887. Prepared by Newark (DE): E.I. du Pont de Nemours and Company, Haskell Laboratory for Toxicology and Industrial Medicine and submitted to Washington (DC). Washington: U.S. Environmental Protection Agency (U.S. EPA), Office of Pollution Prevention and Toxics, 1997a.

15.

Christoph GR Subchronic Operant Behavior Study of Ethyl Aetate by Inhalation in Rats . Report, Haskell Laboratory Report HLR-453-96 ; OTS0558886. Prepared by Newark (DE): E.I. du Pont de Nemours and Company, Haskell Laboratory for Toxicology and Industrial Medicine and submitted to Washington (DC). Washington: U.S. Environmental Protection Agency (U.S. EPA), Office of Pollution Prevention and Toxics, 1997b.

16.

Thake

Kaempfe

, et al. Neurotoxicity evaluation of rats after subchronic inhalation exposure to isobutanol. Neurotoxicology 1999; 20(6): 889–900.

17.

Cho

Lim

J-H

Song

M-K

, et al. Toxicogenomic analysis of the pulmonary toxic effects of hexanal in F344 rat. Environ Toxicol 2017; 32(2): 382–396. DOI: 10.1002/tox.22242.

18.

Matsushita

Toyoda

Morikawa

, et al. A 13-week subchronic toxicity study of vanillin propylene glycol acetal in F344 rats. Food Chem Toxicol 2019; 132: 110643. DOI: 10.1016/j.fct.2019.110643.

19.

Z-H

Liu

Y-S

X-D

, et al. Acute and subacute inhalation toxicity assessment of WS-23 in Sprague-Dawley rats. J Appl Toxicol 2021; 41(11): 1826–1838. DOI: 10.1002/jat4166.

20.

Oldham

Desai

Randazzo

, et al. Evaluation of the toxicity of WS-5 in a 90-days nose-only exposure in Sprague-Dawley rats. Toxicol Res Appl 2023; 7: 23978473231169078. DOI: 10.1177/23978473231169078.

21.

Sciuscio

Kogel

, et al. Evaluation of toxicity of aerosols from flavored e-liquids in Sprague-Dawley rats in a 90-day OECD inhalation study, complemented by transcriptomics analysis. Arch Toxicol 2020; 94: 2179–2206. DOI: 10.1007/s00204-020-02759-6.

22.

Wong

Luettich

Cammack

, et al. Assessment of inhalation toxicity of cigarette smoke and aerosols from flavor mixtures: 5-week study in A/J mice. J Appl Toxicol 2022; 42: 1701–1722. DOI: 10.1002/jat.4338.

23.

OECD 413 . Guidelines for the testing of chemicals, subchronic inhalation toxicity: 90-day study. Paris: OECD, 2009.

24.

OECD 403 . Guidelines on the testing of chemicals. Acute inhalation toxicity. Paris: OECD, 2009.

25.

OECD 412 . Guidelines on the testing of chemicals, 28-day (subacute) inhalation toxicity study. Paris: OECD, 2018.

26.

Ehman

Langston

Kumar

, et al. Preclinical testing of flavors in e-vapor products - Part 1: selection of representative flavor mixtures for toxicological evaluations using a structural grouping approach. In: 73rd Tobacco science research conference. Leesburg, Virginia USA: Lansdowne Resort, 2019.

27.

Smith

Miller

Shah

, et al. Preclinical testing of flavors in e-vapor products, part 2: preparation and stability characterization of representative flavor mixtures. In: 73rd Tobacco science research conference. Leesburg, Virginia USA: Lansdowne Resort, 2019.

28.

Doshi

Zhang

Kumar

, et al. Preclinical testing of flavors in e-vapor products - Part 3: in vitro cytotoxicity and genotoxicity of representative flavor mixtures. In: 73rd Tobacco science research conference. Leesburg, Virginia USA: Lansdowne Resort, 2019.

29.

Zhang

Smith

Anderson

, et al. Preclinical testing of flavors in e-vapor products - Part 4: flavor transfer from the liquid to the aerosol for inhalation exposure. In: 73rd Tobacco science research conference. Leesburg, Virginia USA: Lansdowne Resort, 2019.

30.

Sciuscio

Langston

Kumar

, et al. Flavor ingredients in E-vapor products: a structure-based grouping approach to predict their biological activity. In: Toxicologist 2047. San Diego, CA: 61st Society of Toxicology Meeting, 2022.

31.

Sciuscio

Calvino-Martin

Kumar

, et al. Toxicological assessment of flavor ingredients in E-vapor products. Front Toxicol 2022; 4: 878976. DOI: 10.3389/ftox.2022.878976.

32.

OECD test guideline 1 . OECD series on principles of good laboratory practice and compliance monitoring. Paris: OECD, 1998.

33.

Erythropel

Jabba

DeWinter

, et al. Formation of flavorant-propylene Glycol Adducts with novel toxicological properties in chemically unstable E-cigarette liquids. Nicotine Tob Res 2019; 21: 1248–1258. DOI: 10.1093/ntr/ntyl92.

34.

Gschwend

Jenkins

Jones

, et al. A wide range of Flavoring−Carrier fluid adducts form in E-cigarette liquids. Chem Res Toxicol 2023; 36(1): 14–22. DOI: 10.1021/acs.chemrestox.2c00200.

35.

OECD 413 . Guidelines for the testing of chemicals, 90-day (subchronic) inhalation toxicity study. Paris: OECD, 2018.

36.

Percie du Sert

Hurst

Ahluwalia

, et al. The ARRIVE guidelines 2.0: updated guidelines for reporting animal research. PLoS Biol 2020; 18: e3000410. DOI: 10.1371/journal.pbio.3000410.

37.

National Research Council . Guide for the Care and use of laboratory animals. 8th ed. Washington, DC: The National Academies Press, 2011.

38.

Oldham

Zhang

Rusyniak

, et al. Particle size distribution of selected electronic nicotine delivery system products. Food Chem Toxicol 2018; 113: 236–240. DOI: 10.1016/j.fct.2018.01.045.

39.

Omaiye

McWhirter

Luo

, et al. High concentrations of flavor chemicals are present in electronic cigarette refill fluids. Sci Rep 2019; 9: 2468. DOI: 10.1038/s41598-019-39550-2.

40.

Omaiye

Luo

McWhirter

, et al. Flavour chemicals, synthetic coolants and pulegone in popular mint-flavoured and menthol-flavoured e-cigarettes. Tob Control 2022; 31(e1): e3–e9. DOI: 10.1136/tobaccocontrol-2021-056582.

41.

Werley

McDonald

Lilly

, et al. Non-clinical safety and pharmacokinetic evaluations of propylene glycol aerosol in Sprague-Dawley rats and Beagle dogs. Toxicology 2011; 287: 76–90. DOI: 10.1016/j.tox.2011.05.015.

42.

Szostak

Wong

Titz

, et al. A 6-month systems toxicology inhalation study in ApoE-/- mice demonstrates reduced cardiovascular effects of E-vapor aerosols compared with cigarette smoke. Am J Physiol Heart Circ Physiol 2020; 318: H604–H631. DOI: 10.1152/ajpheart.00613.2019.

43.

Langston

Randazzo

Kogel

, et al. Thirteen-week nose-only inhalation exposures of propylene glycol aerosols in Sprague Dawley rats with a lung systems toxicology analysis. Toxicol Res Appl 2021; 5: 239784732110210. DOI: 10.1177/23978473211021072.

44.

Gupta

Hindle

Byron

, et al. Investigation of a novel condensation aerosol generator: solute and solvent effects. Aerosol Sci Technol 2003; 37: 672–681.

45.

Howell

Sweeney

. Aerosol and a method and apparatus for generating an aerosol. USA: Patent 5,743,251, USA, 1998.

46.

Nichols

Cox

Nguyen

. Dual capillary fluid vaporizing device. USA: Patent 6,715,487, 2004.

47.

Shen

Hindle

Byron

. Effect of energy on propylene glycol aerosols using the capillary aerosol generator. Int J Pharm 2004; 275: 249–258.

48.

Pham

Nguyen

. The aerosolization of cromolyn sodium using a capillary aerosol generator. Patent PCT/US2004/013897, WIPO, March 2005; 7.

49.

Werley

Miller

Kane

, et al. Prototype e-cigarette and the capillary aerosol generator (CAG) comparison and qualification for use in subchronic inhalation exposure testing. Aerosol Sci Technol 2016; 50: 1284–1293.

50.

Goedertier

Weber

Lucci

, et al. Use of capillary aerosol generator in continuous production of controlled aerosol for non-clinical studies. J Vis Exp 2022; 182: e61021. DOI: 10.3791/61021.

51.

Black

Alston

Holloway

, et al. Computational simulation of aerosol generation and temperature regulation performance of nicotine salt pod system. CORESTA Conference (Smoke – Technology Joint Study Groups). Hamburg, Germany: CORESTA, 2019, pp. 6–10.

52.

Talih

Salman

El-Hage

, et al. Characteristics and toxicant emissions of JUUL electronic cigarettes. Tob Control 2019; 28: 678–680, DOI: 10.1136/tobaccocontrol-2018-054616.

53.

Langard

Nordhagen

A-L

. Small animal inhalation chambers and the significance of dust ingestion from the contaminated coat when exposing rats to zinc chromate. Acta Pharmacol Toxicol 1980; 46: 43–46.

54.

Phalen

. Inhalation studies: foundations and techniques. 1st ed. Boca Raton, Florida: CRC Press, 1984, pp. 123–150.

55.

Yoshizawa

Kinoshita

Emoto

, et al. Spontaneously occurring lymphohematopoietic tumors in three young Sprague Dawley rats. Exp Toxicol Pathol 2016; 68(5): 301–305.

56.

Son

W-C

Gopinath

. Early occurrence of spontaneous tumors in CD-1 mice and Sprague-Dawley rats. Toxicol Pathol 2004; 32: 371–374.

57.

Kilgour

Simpson

Alexander

, et al. A rat nasal epithelial model for predicting upper respiratory tract toxicity: in vivo–in vitro correlations. Toxicology 2000; 145(1): 39–49. DOI: 10.1016/s0300-483x(99)00180-8.

58.

Tratnjec

Sibinovska

Kristan

, et al. In vitro ciliotoxicity and cytotoxicity testing of repeated chronic exposure to topical nasal formulations for safety studies. Pharmaceutics 2021; 13(11): 1750. DOI: 10.3390/pharmaceutics13111750.

59.

David

Parmentier

Taurino

, et al. Tracing the composition of single e-cigarette aerosol droplets in situ by laser-trapping and Raman scattering. Sci Rep 2020; 10: 7929. DOI: 10.1038/s41598-020-64886-5.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

3.97 MB