With the development of deep learning, many researches have been devoted to the explanatory nature of Neural Networks (NNs), among which the Information Bottleneck (IB) theory based on information theory has received the most attention. Previous IB work has focused on small MLP or several-layer CNN networks, and a small number of datasets, and the applicability of its conclusions in popular image classification models cannot be verified, especially lacking in transfer learning. To address the above issues, we experiment on an industrial surface defects classification task to investigate IB and performance variations in ResNet transfer learning. The local densities of feature parameters in ResNet transfer learning are calculated by adaptive bins and K-nearest neighbour method. The mutual information of input image information-model parameters and model parameters-output information is calculated, and it is found that there are two stages of fitting and compression of CNN features in transfer learning. The IB representation method is proposed, and the value is found to be positively correlated with the CNN model performance, which provides an innovative idea to explain the transfer learning performance of the model from the IB perspective. Evaluating 81 transfer learning tasks with the derived IP conclusions captures 22.2% poor models for re-optimization. To our knowledge, we are the first study to analyze IB in transfer learning and generalize to the understanding of common CNN image classification models.
LeCunYBengioYHintonG. Deep learning. Nature2015; 521: 436–444.
2.
WangHRajB. On the origin of deep learning. arXiv preprint arXiv:1702.07800, 2017.
3.
BengioY, et al.Learning deep architectures for ai. Found Trend Mach Learn2009; 2: 1–127.
4.
AlainGBengioY. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644, 2016.
5.
Shwartz-ZivRTishbyN. Opening the black box of deep neural networks via information. arXiv preprint arXiv:1703.00810, 2017.
6.
TishbyNZaslavskyN. Deep learning and the information bottleneck principle. In: 2015 IEEE information theory workshop (itw), 2015, pp.1–5. IEEE.
7.
SaxeAMBansalYDapelloJ, et al.On the information bottleneck theory of deep learning. J Stat Mech: Theory Exp2019; 2019: 124020.
8.
AbrolVTannerJ. Information-bottleneck under mean field initialization, 2020.
9.
NoshadMZengYHeroAO. Scalable mutual information estimation using dependence graphs. In: ICASSP 2019-2019 IEEE international conference on acoustics, speech and signal processing (ICASSP), 2019, pp.2962–2966. IEEE.
10.
TorreyLShavlikJ. Transfer learning. In: Handbook of research on machine learning applications and trends: algorithms, methods, and techniques, 2010, pp.242–264. IGI global.
11.
ChenXChenJHanX, et al.A light-weighted cnn model for wafer structural defect detection. IEEE Access2020; 8: 24006–24018.
12.
FuGSunPZhuW, et al.A deep-learning-based approach for fast and robust steel surface defects classification. Opt Lasers Eng2019; 121: 397–405.
13.
ZhuxiMLiYHuangM, et al.A lightweight detector based on attention mechanism for aluminum strip surface defect detection. Comput Indust2022; 136: 103585.
GaninYLempitskyV. Unsupervised domain adaptation by backpropagation. In: International conference on machine learning, 2015, pp.1180–1189. PMLR.
16.
YosinskiJCluneJBengioY, et al.How transferable are features in deep neural networks? Adv Neural Inf Process Syst2014; 27: 1–9.
17.
ZamirARSaxAShenW, et al.Taskonomy: disentangling task transfer learning. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp.3712–3722.
18.
YouKLiuYWangJ, et al.Logme: Practical assessment of pre-trained models for transfer learning. In: International conference on machine learning, 2021, pp.12133–12143. PMLR.
19.
AlzubaidiLDuanYAl-DujailiA, et al.Deepening into the suitability of using pre-trained models of imagenet against a lightweight convolutional neural network in medical imaging: an experimental study. PeerJ Comput Sci2021; 7: e715.
20.
HeKZhangXRenS, et al.Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp.770–778.
21.
SimonyanKZissermanA. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
22.
SandlerMHowardAZhuM, et al.Mobilenetv2: Inverted residuals and linear bottlenecks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp.4510–4520.
23.
CheplyginaV. Cats or cat scans: transfer learning from natural or medical image source data sets? Curr Opin Biomed Eng2019; 9: 21–27.
24.
ImotoKNakaiTIkeT, et al.A cnn-based transfer learning method for defect classification in semiconductor manufacturing. IEEE Trans Semicond Manufact2019; 32: 455–459.
25.
DengJDongWSocherR, et al.Imagenet: a large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition, 2009, pp.248–255. Ieee.
26.
CaoZYouKLongM, et al.Learning to transfer examples for partial domain adaptation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp.2985–2994.
27.
WangZDaiZPóczosB, et al.Characterizing and avoiding negative transfer. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp.11293–11302.
28.
TishbyNPereiraFCBialekW. The information bottleneck method. arXiv preprint physics/0004057, 2000.
29.
GeigerBC. On information plane analyses of neural network classifiers–a review. IEEE Trans Neural Netw Learn Syst2021; 33: 7039–7051.
30.
KwakN. Principal component analysis based on l1-norm maximization. IEEE Trans Pattern Anal Mach Intell2008; 30: 1672–1680.
31.
AyindeBOInancTZuradaJM. Redundant feature pruning for accelerated inference in deep neural networks. Neural Netw2019; 118: 148–158.
32.
KirschALyleCGalY. Scalable training with information bottleneck objectives. In: Proceedings of the international conference on machine learning (icml): workshop on uncertainty and robustness in deep learning, virtual, 2020, pp.17–18.
33.
HjelmRDFedorovALavoie-MarchildonS, et al.Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv:1808.06670, 2018.
34.
AlemiAAFischerIDillonJV, et al.Deep variational information bottleneck. arXiv preprint arXiv:1612.00410, 2016.
35.
SongKYanY. A noise robust method based on completed local binary patterns for hot-rolled steel strip surface defects. Appl Surf Sci2013; 285: 858–864.
36.
YangJFuGZhuW, et al.A deep learning-based surface defect inspection system using multiscale and channel-compressed features. IEEE Trans Instrum Meas2020; 69: 8032–8042.
37.
JiangLYuanBWangY, et al.Ma-yolo: A method for detecting surface defects of aluminum profiles with attention guidance. IEEE Access2023; 11: 71269–71286.
38.
HuangYQiuCYuanK. Surface defect saliency of magnetic tile. Vis Comput2020; 36: 85–96.
39.
ZhangDSongKWangQ, et al.Two deep learning networks for rail surface defect inspection of limited samples with line-level label. IEEE Trans Indust Inform2020; 17: 6731–6741.
KylbergG. Kylberg texture dataset v. 1.0. Centre for Image Analysis, Swedish University of Agricultural Sciences and Uppsala University, 2011.
42.
FritzMHaymanECaputoB, et al.The kth-tips database, 2004.
43.
DorafshanSThomasRJMaguireM. Sdnet2018: An annotated image dataset for non-contact concrete crack detection using deep convolutional neural networks. Data Brief2018; 21: 1664–1668.
44.
CuiYSongYSunC, et al.Large scale fine-grained categorization and domain-specific transfer learning. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp.4109–4118.
