TextureMask: A merged architecture for low-resolution instance segmentation

Abstract

Instance segmentation has a wide range of applications, including video surveillance, autonomous driving, and behavior analysis. Nevertheless, as a type of pixel-level segmentation, its prediction performance in practice is substantially affected by low-resolution (LR) images resulting from the limitations of image acquisition equipment and poor acquisition conditions. Moreover, because their immense computational costs prevent the implementation of existing segmentation models on embedded devices, the development of a lightweight segmentation model has become an urgent necessity. However, it is challenging to achieve sound results with high efficiency and portability. From another perspective, to improve understanding of detailed objects, an architecture is needed that promotes an advanced interpretation of the segmentation, that is, a refined mask with texture. Our main contribution, called TextureMask, consists of the MobileNet-FPN for Mask R-CNN methods, segmentation with cropping, and a gradient sensitivity map, which are then merged into a unified map to refine and enrich the mask with texture information. Furthermore, preprocessing and post-processing algorithms are incorporated. Experiments demonstrated that our technique exhibits good pixel-level segmentation performance in terms of both accuracy and computational efficiency for a given LR input, and it can be easily implemented in embedded platforms.

Keywords

Mask R-CNN MobileNet light weight sensitivity map instance segmentation

Get full access to this article

View all access options for this article.

References

Krizhevsky

Sutskever

and Hinton

G.E.

, ImageNet classification with deep convolutional neural networks, Communications of the ACM 60 (2012), 84–90.

et al., Mask R-CNN, IEEE Transactions on Pattern Analysis and Machine Intelligence 42 (2020), 386–397.

Ronneberger

Fischer

and Brox

, U-Net: Convolutional networks for biomedical image segmentation, 2015, ArXiv, abs/1505.04597.

Zhao

et al., Pyramid scene parsing network, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 6230–6239.

Akilan

et al., A 3D CNN-LSTM-based Image-to-Image foreground segmentation, IEEE Transactions on Intelligent Transportation Systems 21 (2020), 959–971.

Gao

et al., Salient object detection in the distributed cloud-edge intelligent network, IEEE Network 34 (2020), 216–224.

Banerjee

and Maji

, A spatially constrained probabilistic model for robust image segmentation, IEEE Transactions on Image Processing 29 (2020), 4898–4910.

Perazzi

et al., Learning video object segmentation from static images, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 3491–3500.

Caelles

et al., One-shot video object segmentation, in: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 5320–5329.

10.

Cheng

et al., SegFlow: Joint learning for video object segmentation and optical flow, in: IEEE International Conference on Computer Vision (ICCV), 2017, pp. 686–695.

11.

Cheng

et al., Fast and accurate online video object segmentation via tracking parts, in: IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, pp. 7415–7424.

12.

Huang

and Schwing

, VideoMatch: matching based video object segmentation, 2018, ArXiv, abs/1809.01123.

13.

and Loy

C.C.

, Video object segmentation with joint re-identification and attention-aware mask propagation, ECCV, 2018.

14.

Bhatti

A.H.

Rahman

A.U.

and Butt

, Unsupervised video object segmentation using conditional random fields, Signal, Image and Video Processing 13 (2019), 9–16.

15.

Lin

et al., Towards optimal structured CNN pruning via generative adversarial learning, in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 2785–2794.

16.

Xin

and Lin

S.H.

, Pruning and retraining method for a convolution neural network, U.S. Patent Application 15/699, 438, filed March 14, 2019.

17.

Moon

et al., Memory-reduced network stacking for edge-level CNN architecture with structured weight pruning, IEEE Journal on Emerging and Selected Topics in Circuits and Systems 9 (2019), 735–746.

18.

Nogami

et al., Optimizing weight value quantization for CNN inference, in: International Joint Conference on Neural Networks (IJCNN), 2019, pp. 1–8.

19.

Banner

Nahshan

and Soudry

, Post training 4-bit quantization of convolutional networks for rapid-deployment, NeurIPS, 2019.

20.

Zhang

and Li

, Unleashing the power of soft logic for convolutional neural network acceleration via product quantization, in: Proceedings of the 2019 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays, 2019.

21.

Chandra

, Data bandwidth reduction in deep neural network SoCs using history buffer and Huffman coding, in: International Conference on Computing, Power and Communication Technologies (GUCON), 2018, pp. 1–3.

22.

Howard

A.G.

et al., MobileNets: Efficient convolutional neural networks for mobile vision applications, 2017, ArXiv, abs/1704.04861.

23.

Sandler

et al., MobileNetV2: Inverted residuals and linear bottlenecks, in: IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, pp. 4510–4520.

24.

Zhang

et al., ShuffleNet: An extremely efficient convolutional neural network for mobile devices, in: IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, pp. 6848–6856.

25.

et al., ShuffleNet V2: Practical guidelines for efficient CNN architecture design, ECCV, 2018.

26.

Tan

and Le

Q.V.

, EfficientNet: Rethinking model scaling for convolutional neural networks, 2019, ArXiv, abs/1905.11946.

27.

Girshick

R.B.

et al., Rich feature hierarchies for accurate object detection and semantic segmentation, in: IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 580–587.

28.

Girshick

R.B.

, Fast R-CNN, in: IEEE International Conference on Computer Vision (ICCV), 2015, pp. 1440–1448.

29.

Ren

et al., Faster R-CNN: towards real-time object detection with region proposal networks, IEEE Transactions on Pattern Analysis and Machine Intelligence 39 (2015), 1137–1149.

30.

et al., Spatial pyramid pooling in deep convolutional networks for visual recognition, IEEE Transactions on Pattern Analysis and Machine Intelligence 37 (2015), 1904–1916.

31.

Ventura

et al., RVOS: End-to-end recurrent network for video object segmentation, in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 5272–5281.

32.

Zhu

et al., Improving semantic segmentation via video propagation and label relaxation, in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 8848–8857.

33.

Vuola

Akram

and Kannala

, Mask-RCNN and U-net ensembled for nuclei segmentation, in: IEEE 16th International Symposium on Biomedical Imaging (ISBI), 2019, pp. 208–212.

34.

Johnander

et al., A generative appearance model for end-to-end video object segmentation, IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 8945–8954.

35.

Zhang

et al., Context encoding for semantic segmentation, in: IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, pp. 7151–7160.

36.

Zeng

et al., Dmm-net: Differentiable mask-matching network for video object segmentation, in: IEEE/CVF International Conference on Computer Vision (ICCV), 2019, pp. 3928–3937.

37.

Wang

et al., RDSNet: A new deep architecture for reciprocal object detection and instance segmentation, AAAI, 2020.

38.

Chen

et al., Hybrid task cascade for instance segmentation, in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 4969–4978.

39.

Wang

et al., Understanding convolution for semantic segmentation, in: IEEE Winter Conference on Applications of Computer Vision (WACV), 2018, pp. 1451–1460.

40.

Xie

et al., Polarmask: Single shot instance segmentation with polar representation, in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 12190–12199.

41.

Maninis

et al., Video object segmentation without temporal information, IEEE Transactions on Pattern Analysis and Machine Intelligence 41 (2019), 1515–1530.

42.

et al., See more, know more: Unsupervised video object segmentation with co-attention siamese networks, in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 3618–3627.

43.

Ying

et al., Embedmask: Embedding coupling for one-stage instance segmentation, 2019, ArXiv, abs/1912.01954.

44.

Wang

et al., Deep networks for image super-resolution with sparse prior, in: IEEE International Conference on Computer Vision (ICCV), 2015, pp. 370–378.

45.

Gregor

and LeCun

, Learning fast approximations of sparse coding, in: International Conference on International Conference on Machine Learning (ICML), 2010, pp. 399–406.

46.

Dong

et al., Image super-resolution using deep convolutional networks, IEEE Transactions on Pattern Analysis and Machine Intelligence 38 (2016), 295–307.

47.

BinoSebastian

Unnikrishnan

and Balakrishnan

, Gray level co-occurrence matrices: generalization and some new features, 2012, ArXiv, abs/1205.4831.

48.

Dalal

and Triggs

, Histograms of oriented gradients for human detection, IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR) 1 (2005), 886–893.

49.

Zhang

, The optimality of naive Bayes, in: FLAIRS Conference, 2004.

50.

Breiman

, Random forests, Machine Learning 45 (2004), 5–32.

51.

Freund

and Schapire

, A decision-theoretic generalization of on-line learning and an application to boosting, J. Comput. Syst. Sci. 55 (1995), 119–139.

52.

Suykens

and Vandewalle

, Least squares support vector machine classifiers, Neural Processing Letters 9 (2004), 293–300.

53.

Smilkov

et al., SmoothGrad: removing noise by adding noise, 2017, ArXiv, abs/1706.03825.

54.

Springenberg

J.T.

et al., Striving for Simplicity: The all convolutional net, 2015, arXiv, abs/1412.6806.

55.

Taylor

G.W.

et al., Convolutional learning of spatio-temporal features, ECCV, 2010.

56.

Hara

Kataoka

and Satoh

, Learning spatio-temporal features with 3D residual networks for action recognition, in: IEEE International Conference on Computer Vision Workshops (ICCVW), 2017, pp. 3154–3160.

57.

Sun

et al., Human action recognition using factorized spatio-temporal convolutional networks, in: IEEE International Conference on Computer Vision (ICCV), 2015, pp. 4597–4605.

58.

Moschetti

and Debes

, Data adapting motion estimation and subsampling, Visual Communications and Image Processing, 2000.

59.

Luo

D.E.

et al., Patent No. 7,978,770, Washington, DC: U.S. Patent and Trademark Office, 2011.

60.

Chen

and Chiueh

, One-dimensional full search motion estimation algorithm for video coding, IEEE Trans, Circuits Syst. Video Technol 4 (1994), 504–509.

61.

Huynh-Thu

and Ghanbari

, Scope of validity of PSNR in image/video quality assessment, Electronics Letters 44 (2008), 800–801.

62.

Wang

et al., Image quality assessment: from error visibility to structural similarity, IEEE Transactions on Image Processing 13 (2004), 600–612.

63.

Zhang

et al., FSIM: a feature similarity index for image quality assessment, IEEE Transactions on Image Processing 20 (2011), 2378–2386.

64.

Powers

, Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation, ArXiv, abs/2010.16061.