Semi-automated generation of pictures for the Mouse Grimace Scale: A multi-laboratory analysis (Part 2)

Abstract

German

Spanish

French

The Mouse Grimace Scale (MGS) is an established method for estimating pain in mice during animal studies. Recently, an improved and standardized MGS set-up and an algorithm for automated and blinded output of images for MGS evaluation were introduced. The present study evaluated the application of this standardized set-up and the robustness of the associated algorithm at four facilities in different locations and as part of varied experimental projects. Experiments using the MGS performed at four facilities (F1–F4) were included in the study; 200 pictures per facility (100 pictures each rated as positive and negative by the algorithm) were evaluated by three raters for image quality and reliability of the algorithm. In three of the four facilities, sufficient image quality and consistency were demonstrated. Intraclass correlation coefficient, calculated to demonstrate the correlation among raters at the three facilities (F1–F3), showed excellent correlation. The specificity and sensitivity of the results obtained by different raters and the algorithm were analysed using Fisher's exact test (p < 0.05). The analysis indicated a sensitivity of 77% and a specificity of 64%. The results of our study showed that the algorithm demonstrated robust performance at facilities in different locations in accordance with the strict application of our MGS setup.

Keywords

MGS Mouse Grimace Scale pain assessment severity assessment

Based on the developments in experimental animal research and implementation of the European Union Directive (2010/63 EU)¹ on the protection of animals used for scientific purposes, ensuring the highest possible level of animal well-being has become a major priority in animal studies. Article 15 of this directive mandates a severity assessment of each procedure in an animal study.¹ Based on this requirement, methods of evaluating any changes in animal well-being and estimating potential suffering are necessary.

Because most rodent species used in animal research are flight and prey animals,^2,3 these animals avoid overtly exhibiting signs or vocalizing their pain.⁴ Due to this lack of self-indication of pain severity by the animals,⁴ objective criteria must be implemented to assess pain severity.

Researchers have employed various methods for this purpose, including clinical examinations and scoring sheets,⁵ specific stress parameter evaluations and behavioural tests.^6–8

The Mouse Grimace Scale (MGS), a noninvasive method of visually recognizing pain on the basis of facial expressions of mice,⁹ has become an established method for identifying acute pain in mice and has been repeatedly used in animal experiments.^3,10,11 In the original publication of Langford et al.,⁹ the MGS pictures for analysis were cropped from pre-recorded videos, selected and then scored manually.

In a recent study, our group could improve the MGS set-up by video recording up to four animals simultaneously. Additionally, a tool for automated image selection for blinded MGS analysis was introduced (Ernst et al.,).¹² Automation and standardization of technical processes is necessary to minimize subjective influences and avoid selection and performance biases. The aim of the present study was to investigate the application of the modified and improved MGS set-up and the robustness of the automated process in a multi-laboratory analysis. The application and conformity of the set-up and image selection tool were also assessed.

Materials and methods

Study design

To confirm the applicability of the improved MGS set-up and automated image selection tool (Ernst et al., under review), animal research studies at four experimental facilities (F1–F4) at different locations employed this approach. Most of the studies were part of assessments that evaluated the severity of procedures using animals for scientific purposes. The recently introduced modified MGS set-up was analogously implemented in all the studies.

At F1, the MGS set-up was implemented in a refinement study on the possible benefits of infiltration with local anaesthetics (lidocaine–bupivacaine) in combination with systemic paracetamol administration via drinking water compared with systemic analgesia only after surgery. Male and female C57Bl/6J mice were subjected to a minor laparotomy and treated with a combination of local and systemic analgesia, local or systemic analgesia alone, or anaesthesia and systemic analgesia only. Among other behavioural tests, the MGS was used to assess changes in animal well-being and potential pain severity. Baseline measurements were taken at the same time points during the day as postoperative measurements: 1, 6 and 24 h after surgery.

At F2, a project on pain severity assessment after inducing liver fibrosis was conducted. Fibrosis was induced using CCl₄ dissolved in germ oil. Male C57Bl/6N mice were intraperitoneally injected (50 µl) either with 0.6 ml/kg CCl₄ in mixed germ oil or germ oil only (control) three times per week for 4 weeks. MGS scoring was performed 1 h before and after injections. Baseline measurements were taken before starting the induction of liver fibrosis.

At F3, a project on the effects of intraperitoneal transmitter implantation or a corresponding SHAM operation on different clinical and behavioural parameters in female C57Bl/6J mice was conducted. To differentiate between the effects of the surgical procedure itself and the transmitter, SHAM-operated mice were monitored as a control group. MGS scoring was performed at 30 and 180 mins after surgery on the same day and on days 1, 2, 3, 5 and 7.

At F4, male and female mice, wild-types for A1783V mutation and with or without the presence of Cre were used for animal experiments.¹³ All mice underwent surgery during which a telemetry device (HD-X02, DSI, St Paul, USA) was subcutaneously implanted and an electrode was implanted into the hippocampus. The electrode leads were fixed with three screws in the skull and covered with paladur (Heraeus®, Hanau, Germany). Baseline measurements were taken 1 day prior to surgery after a habituation phase of 10 mins. Video recordings were taken 1 h prior to surgery and at 1, 3, 6, 25 and 49 h following gain of consciousness after surgery.

All the studies were conducted in accordance with the legal requirements, and anaesthesia and analgesia interventions were obtained if appropriate. Human endpoint protocols were applied for all the studies. Additional details concerning procedures or surgeries can be found in the supplementary material.

Ethical statement

MGS evaluations were a preliminary part of the experiments. According to the 3R principles,¹⁴ no additional animals were used to perform this study. All the studies were conducted in accordance with EU Directive (2010/63 EU) and the legal provisions of German Animal Welfare Act (TierSchG).¹⁵

Cantonal Veterinary Office, Zurich, Switzerland, approved the animal housing and experimental procedures for the project at F1 under licence number 097/2017. For the project at F2, the permission licence was granted by Governmental Animal Care and Use Committee (Landesamt für Natur, Umwelt und Verbraucherschutz, LANUV AZ: 84-02.04.2014.A417, North Rhine Westphalia, Germany). All experiments performed at F3 were approved by Niedersächsisches Landesamt für Landwirtschaft und Lebensmittelsicherheit (LAVES) under licence number 15/1905. For the project at F4, all investigations were approved by the Government of Upper Bavaria (license number 55.2-1-54-2532-168-2016).

Animals

Two facilities (F1 and F3) used C57Bl/6J mice, one facility (F1) used C57Bl6/N mice, and one facility (F4) used transgenic mice with a C57Bl6 background selected according to the researchers' interest in their main study. The choice of sex, age and strain was independently made and was a nonexclusive part of the present study. Only a black coat colour and the presence of adult animals were indicated as relevant to the present study. Additional information concerning housing and husbandry conditions according to the ARRIVE guidelines¹⁶ can be found in the supplementary material.

MGS set-up

With the modified and improved MGS video recording set-up, four animals could be simultaneously filmed under standardized conditions. To maximize the quality of MGS pictures, four equally sized MGS boxes (9 cm × 5 cm × 5 cm), which were placed in an observation rack located within a light tent, were illuminated from the side, bottom and front. Additional air holes were drilled into the front as well as into the lid of the boxes to reduce fogging. At F1, the recording time was set at approximately 5 mins, and at F2, F3 and F4, the recording time was set at 10 mins.

For automated analysis, box positions in the videos were manually defined and 300 images from each box were automatically extracted. Subsequently, the algorithm analysed the extracted box images using a fully convolutional architecture¹⁷ to detect the position and size of the animals' eyes. Eye areas in the images were automatically measured, and all the images in which the largest visible eye had an area of at least 100 pixels were considered suitable for MGS scoring (image size: approximately 500 × 500 pixels). Among these images, 10 images per animal were randomly selected by the algorithm for further manual scoring. For this purpose, we developed a tool that displays the images of all trials and animals in a randomized and blinded manner to minimize bias.

Evaluation process

At each facility, 100 images each rated as positive or negative by the algorithm were selected for image quality evaluation; therefore, a total of 200 images per facility were evaluated. Images rated as positive were suitable for MGS scoring, whereas those rated as negative could not be used. Evaluation of image quality was performed by three raters at three facilities. All the raters had comparable experience in the performance and assessment of MGS images. The criteria for fulfilling individual evaluation points were discussed with all the participants in advance.

For manual selection, a maximum score of 54 points could be achieved by fulfilling all positive criteria. Six evaluation criteria (mouse in profile, eyes recognizable, ears recognizable, nose recognizable, mouse in steady position and general image quality) were assessed and assigned a maximum of nine points per criterion. The quality gradations for fulfilment within the evaluation criteria were as follows: 1–3 = poor, 4–6 = moderate, and 7–9 = excellent. Images rejected due to non-fulfilment of even any one criterion were given a score of −1. As a cut-off value, images with a score of ≤30, that is 55% of the maximum score, or a score of −1 for any evaluation criterion were rated as negative. In this study, the focus of the algorithm was mainly on the detection of the eye.

Statistical analysis

GraphPad Prism (GraphPad Prism, Version 7, La Jolla California USA, www.graphpad.com) and R software (version 3.4.1)¹⁸ were used for data analysis. Intraclass correlation coefficient (ICC), which is an estimate of inter-rater reliability, was calculated using the ICC function from the interrater reliability (irr) library¹⁸ using a two-way ANOVA to assess agreement. Fisher's exact test was used to analyse specificity and sensitivity. To identify reasons for false positive assignments by the algorithm, data were analysed using one-way ANOVA and Tukey's multiple comparison test. The data were considered statistically significant at p < 0.05.

Results

The number of scorable images at the facilities and their distribution are presented in Figure 1. For each facility, 100 positive and negative images each were selected by the algorithm and evaluated by raters for image quality. Examples of such images from each facility are shown in Figure 1. Regarding facility and location: at F1, an average of 97 images (standard deviation (SD) = 17.08, n = 3, n corresponds to the number of raters (one rater per facility)) were suitable for MGS evaluation; at F2, an average of 86.25 images (SD = 9.52, n = 3) were suitable; and at F3, an average of 96 images (SD = 27.67, n = 3) were suitable. Because of deviations in performance from the initial set-up protocol in terms of colour and illumination, videos from F4 could not be included for image-quality evaluation.

Figure 1.

Examples of images of all facilities: F1, top left; F2, top right; F3, bottom left; and F4, bottom right. The number of scorable images between the facilities is presented. The data distribution within the different facilities shows a bimodal distribution. The cut-off value is indicated by a dashed line. The data distribution for F4 cannot be displayed because of differences in colour, brightness and the presence of a head implant, given that the algorithm could not generate adequate images without further adjustment.

ICC, which was calculated to demonstrate the correlation between raters at the three facilities (F1–F3) showed an excellent correlation (Table 1);¹⁹ however, no significant differences were detected in their correlation coefficients. The results obtained by different raters and the algorithm, as analysed using Fisher's exact test (p < 0.05), indicated a sensitivity of 77% and a specificity of 64% (Table 2).

Table 1.

Results of the analysis of intraclass correlation coefficient (ICC) values and their 95% confidence intervals as determined using a two-way ANOVA. r0 is a specification of the null hypothesis (H1: r = r0). H1: r > r0 denotes that a one-sided F-test was performed.

Models: 2-Way	Fl v. F2	Fl v. F3	F2 v. F3	Fl v. F3
Typ:Agreement
Subjects	600	600	600	600
Raters	2	2	2	2
ICC (2)	0.95 [0.94, 0.96]	0.91 [0.89,0.92]	0.95 [0.94,0.96]	0.966 [0.95,0.96]
F-test, HO: r0 = 0; H1: r0 > 0	F (599, 600) = 19.2	F (599, 600) = 10.7	F (599, 600) = 19.1	F (599, 1200) = 22.5
p value	<0.05	<0.05	<0.05	<0.05
95% confidence interval for ICC Population Values	0.939 < ICC < 0.956	0.89 < ICC < 0.92	0.939 < ICC < 0.955	0.949 < ICC < 0.961

Table 2.

Specificity and sensitivity of the algorithm.

Effect size	Value	95% confidence interval
Sensitivity	0.7774	0.7245–0.8226
Specificity	0.6425	0.5944–0.6879
Positive predictive value	0.5983	0.5466–0.6479
Negative predictive value	0.8082	0.7613–0.8477

Among the 600 rated images, an average of 88.67 (SD = 30.16) images assessed by the algorithm were evaluated as false positives compared with those assessed by the raters. A total of 44 similar images were rated as false positives by the raters compared with the algorithm. For images evaluated as false positives by the algorithm depending on the different evaluation criteria, the number of rejected images is presented in Figure 2. The results showed significant differences according to the ‘nose recognizable’ and ‘mouse in steady position’ criteria.

Figure 2.

The number of rejected images per criterion with median and 95% confidence interval calculated using a one-way ANOVA: F (6,14) = 18,67, *Tukey's multiple comparisons test adjusted p value < 0.05.

Discussion

The present study evaluated the applicability and robustness of the modified MGS set-up and the algorithm for image selection. Figure 1 shows that the standardized set-up is applicable and reproducible and that a sufficient number of images were deemed suitable for analysis at three facilities in different locations. Reproducibility is an important attribute in the performance of animal experiments.²⁰ Vasilevsky et al. have demonstrated that the applicability of scientific methods is dependent on the ability of reproducing other studies and building on previous work, and they noted that a lack in the provision of methodological details considerably reduces this reproducibility.²¹ This conclusion is supported by the findings of our study such that deviations from the standardized protocol (e.g. colour and illumination level) at one facility rendered it impossible for the algorithm to evaluate images without adapting the algorithm to this particular set-up.

Miller and Leach describe in their study deviations in MGS values between different mouse strains and sexes.²² In order to investigate the selection criteria for MGS images, we have deliberately included animals of different breeds that have a C57Bl/6 background and represent both sexes. In the selection of the pictures, we could not detect any signs for gender-specific selection criteria for MGS. The application of the algorithm shows, in principle, a positive result rate, with a sensitivity of 77% and specificity of 64%. The reduction in false positives supports the use of this algorithm. With regard to specificity, the ‘nose recognizable’ and ‘mouse in steady position’ criteria are decisive for the selection of false positive results (Figure 2). Recognition of the ‘nose recognizable’ criterion appears to result in a reduced reliability. This has also been reported by other studies and may not be algorithm-dependent alone.²³

Conclusion

The present study demonstrated the applicability of the improved MGS set-up and the functionality of the associated algorithm at three facilities in different locations. Limitations in the specificity of the algorithm, especially because of the lack of detection of moving animals, are currently being adjusted by improving the algorithm, and this should result in a reduction in the number of images rated as false positives in future studies (Kopaczka et al.,²⁴ submitted to the 2019 Annual International Conference of EMBC)

Supplemental Material

LAN881664 Supplemetal Material - Supplemental material for Semi-automated generation of pictures for the Mouse Grimace Scale: A multi-laboratory analysis (Part 2)

Supplemental material, LAN881664 Supplemetal Material for Semi-automated generation of pictures for the Mouse Grimace Scale: A multi-laboratory analysis (Part 2) by Lisa Ernst, Marcin Kopaczka, Mareike Schulz, Steven R Talbot, Birgitta Struve, Christine Häger, André Bleich, Mattea Durst, Paulin Jirkof, Margarete Arras, Roelof Maarten van Dijk, Nina Miljanovic, Heidrun Potschka, Dorit Merhof and Rene H Tolba in Laboratory Animals

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The study was funded in part by German Research Foundation (Deutsche Forschungsgemeinschaft - DFG) FOR 2591 Consortium and ME3737/18-1. Reference numbers are listed in the supplementary material.

References

EU EP. Directive 2010/63/EU of the European Parliament and of the Council of 22 September 2010 on the protection of animals used for scientific purposes. Document 32010L0063. European Parliament, 22 September 2010.

Faller

McAndrew

Schneider

, et al. Refinement of analgesia following thoracotomy and experimental myocardial infarction using the Mouse Grimace Scale. Exp Physiol 2015; 100: 164–172.

Matsumiya

Sorge

Sotocinal

, et al. Using the Mouse Grimace Scale to re-evaluate the efficacy of postoperative analgesics in laboratory mice. JAALAS 2012; 51: 42–49.

Kanzler

Rix

Czigany

, et al. Recommendation for severity assessment following liver resection and liver transplantation in rats: Part I. Lab Anim 2016; 50: 459–467.

Morton

Griffiths

. Guidelines on the recognition of pain, distress and discomfort in experimental animals and a hypothesis for assessment. Vet Rec 1985; 116: 431–436.

Seibenhener

Wooten

. Use of the Open Field Maze to measure locomotor and anxiety-like behavior in mice. JoVE 2015, pp. e52434.

Peng

Zhang

Dong

, et al. Battery of behavioral tests in mice to study postoperative delirium. Sci Rep 2016; 6: 29874.

Pratt

Fuchs

Sluka

. Assessment of avoidance behaviors in mouse models of muscle pain. Neuroscience 2013; 248: 54–60.

Langford

Bailey

Chanda

, et al. Coding of facial expressions of pain in the laboratory mouse. Nat Methods 2010; 7: 447–449.

10.

Miller

Kitson

Skalkoyannis

, et al. Using the mouse grimace scale and behaviour to assess pain in CBA mice following vasectomy. Appl Anim Behav Sci 2016; 181: 160–165.

11.

Miller

Leach

. Using the mouse grimace scale to assess pain associated with routine ear notching and the effect of analgesia in laboratory mice. Lab Anim 2015; 49: 117–120.

12.

Ernst L, Kopaczka M, Schulz M, et al. Improvement of the Mouse Grimace Scale set-up for implementing a semi-automated Mouse Grimace Scale scoring (Part 1). Lab Anim 2020; 54: 83–91.

13.

Tang

Silva

Tsark

, et al. A Cre/loxP-deleter transgenic line in mouse strain 129S1/SvImJ. Genesis 2002; 32: 199–202.

14.

Russell

WMS

Burch

. The Principles of Humane Experimental Technique, London: Methuen, 1959.

15.

Tierschutzgesetz in der Fassung der Bekanntmachung vom 18. Mai 2006 (BGBl. I S. 1206, 1313), das zuletzt durch Artikel 1 des Gesetzes vom 17. Dezember 2018 (BGBl. I S. 2586) geändert worden ist.

16.

Kilkenny

Browne

Cuthill

, et al. Animal research: Reporting in vivo experiments: the ARRIVE guidelines. Br J Pharmacol 2010; 160: 1577–1579.

17.

Navab N, Hornegger JM, Wells W, et al. Medical Image Computing and Computer-Assisted Intervention. In: Proceedings of the 18th International Conference MICCAI, Munich, Germany, October 5–9, 2015, Part III. 2015, pp. 234–241. Germany: Springer International publishing AG Switzerland.

18.

Team RC. R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing, 2018.

19.

Cicchetti

. Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instrument in psychology. Psychol Assess 1994; 6: 284–290.

20.

Tilson

Schroeder

. Reporting of results from animal studies. Environ Health Perspect 2013; 121: A320–321.

21.

Vasilevsky

Brush

Paddock

, et al. On the reproducibility of science: Unique identification of research resources in the biomedical literature. Peer J 2013; 1: e148.

22.

Miller

Leach

. The Mouse Grimace Scale: A clinically useful tool?. PloS One 2015; 10: e0136000.

23.

Dalla Costa

Pascuzzo

Leach

, et al. Can grimace scales estimate the pain status in horses and mice? A statistical approach to identify a classifier. PloS One 2018; 13: e0200339.

24.

Kopaczka M, Tillmann D, Ernst L, et al. Assessment of Laboratory Mouse Activity in Video Recordings Using Deep Learning Methods. In: International Engineering in Medicine and Biology Conference (EMBC), Berlin, Germany, 2019, Paper FrA15.2.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.02 MB