Sage Journals: Discover world-class research

Abstract

Recent improvements in deep learning techniques show that deep models can extract more meaningful data directly from raw signals than conventional parametrization techniques, making it possible to avoid specific feature extraction in the area of pattern recognition, especially for Computer Vision or Speech tasks. In this work, we directly use raw text line images by feeding them to Convolutional Neural Networks and deep Multilayer Perceptrons for feature extraction in a Handwriting Recognition system. The proposed recognition system, based on Hidden Markov Models that are hybridized with Neural Networks, has been tested with the IAM Database, achieving a considerable improvement.

Keywords

Handwriting recognition deep learning convolutional neural networks

Get full access to this article

View all access options for this article.

References

Baldi,

Brunak,

Frasconi,

Soda and

Pollastri, Exploiting the past and the future in protein secondary structure prediction, Bioinformatics (Oxford, England) 15(11) (1999), 937–946. doi:10.1093/bioinformatics/15.11.937.

Bauer, Manual of information to accompany The Wellington Corpus of Written New Zealand English, Technical report, Department of Linguistics, Victoria University, Wellington, New Zealand, 1993.

Bengio,

A.C.

Courville and

Vincent, Unsupervised feature learning and deep learning: A review and new perspectives. CoRR, abs/1206.5538, 2014.

Bengio,

LeCun and

Hinton, Deep learning, Nature 521 (2015), 436–444. doi:10.1038/nature14539.

Bertolami and

Bunke, Hidden Markov model-based ensemble methods for offline handwritten text line recognition, Pattern Recognition 41(11) (2008), 3452–3460. doi:10.1016/j.patcog.2008.04.003.

A.L.

Bianne-Bernard,

Menasri,

Al-Hajj Mohamad,

Mokbel,

Kermorvant and

Likforman-Sulem, Dynamic and contextual information in HMM modeling for handwritten word recognition, IEEE Transactions on Pattern Analysis and Machine Intelligence 33(10) (2011), 2066–2080. doi:10.1109/TPAMI.2011.22.

C.M.

Bishop, Neural networks for pattern recognition, Oxford University Press, 1995.

Bluche, Deep neural networks for large vocabulary handwritten text recognition, PhD thesis, Université Paris Sud – Paris XI, 2015.

Bluche,

Ney and

Kermorvant, Feature extraction with convolutional neural networks for handwritten word recognition, in: 12th International Conference on Document Analysis and Recognition (ICDAR), 2013, pp. 285–289.

10.

Bluche,

Ney and

Kermorvant, Tandem HMM with convolutional neural network for handwritten word recognition, in: 38th International Conference on Acoustics Speech and Signal Processing (ICASSP), 2013, pp. 2390–2394.

11.

Bluche,

Ney and

Kermorvant, A comparison of sequence-trained deep neural networks and recurrent neural networks optical modeling for handwriting recognition, in: Slsp-2014, 2014, pp. 1–12.

12.

Bourlard and

Morgan, Connectionist Speech Recognition – A Hybrid Approach, Series in Engineering and Computer Science, Vol. 247, Kluwer Academic, 1994.

13.

R.M.

Bozinovic and

S.N.

Srihari, Off-line cursive script word recognition, IEEE Trans. Pattern Anal. Mach. Intell. 11(1) (1989), 68–83. doi:10.1109/34.23114.

14.

Bunke, Recognition of cursive roman handwriting – past, present and future, in: International Conference on Document Analysis and Recognition, Vol. 1, 2003, pp. 448–459.

15.

Caillault,

Viard-Gaudin and

Rahim Ahmad, MS-TDNN with global discriminant trainings, in: International Conference on Document Analysis and Recognition (ICDAR), 2005, pp. 856–860.

16.

Doetsch,

Kozielski and

Ney, Fast and robust training of recurrent neural networks for offline handwriting recognition, in: 14th International Conference on Frontiers in Handwriting Recognition (ICFHR), 2014, pp. 279–284.

17.

Dreuw,

Doetsch,

Plahl and

Ney, Hierarchical hybrid MLP/HMM or rather MLP features for a discriminatively trained Gaussian HMM: A comparison for offline handwriting recognition, in: International Conference on Image Processing (ICIP), 2011, pp. 3541–3544.

18.

Dreuw,

Heigold and

Ney, Confidence and margin-based MMI/MPE discriminative training for online handwriting recognition, International Journal of Document Analysis and Recognition 14(3) (2011), 273–288. doi:10.1007/s10032-011-0160-x.

19.

España-Boquera,

M.J.

Castro-Bleda,

Gorbe-Moya and

Zamora-Martinez, Improving offline handwritten text recognition with hybrid HMM/ANN models, IEEE Trans. on Pattern Analysis and Machine Intelligence 33(4) (2011), 767–779. doi:10.1109/TPAMI.2010.141.

20.

W.N.

Francis and

Kucera, Brown corpus manual, manual of information to accompany a standard corpus of present-day edited American English, Technical report, Department of Linguistics, Brown University, Providence, Rhode Island, US, 1979.

21.

Graves,

Fernández,

Gomez and

Schmidhuber, Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks, in: 23rd International Conference on Machine Learning (ICML), ACM, 2006, pp. 369–376.

22.

Graves and

Jaitly, Towards end-to-end speech recognition with recurrent neural networks, in: 31st International Conference on Machine Learning (ICML), 2014, pp. 1764–1772.

23.

Graves,

Liwicki,

Fernández,

Bertolami,

Bunke and

Schmidhuber, A novel connectionist system for unconstrained handwriting recognition, IEEE Transactions on Pattern Analysis and Machine Intelligence 31(5) (2009), 855–868. doi:10.1109/TPAMI.2008.137.

24.

Graves and

Schmidhuber, Framewise phoneme classification with bidirectional LSTM networks, in: International Joint Conference on Neural Networks (IJCNN), Vol. 4, 2005, pp. 2047–2052.

25.

Graves and

Schmidhuber, Offline handwriting recognition with multidimensional recurrent neural networks, in: Advances in Neural Information Processing Systems (NIPS), 2009, pp. 545–552.

26.

Grézl,

Karafiát,

Kontár and

Černocký, Probabilistic and bottle-neck features for LVCSR of meetings, in: International Conference on Acoustics, Speech and Signal Processing (ICASSP), Vol. 4, 2007.

27.

Hochreiter and

Schmidhuber, Long short-term memory, Neural computation 9(8) (1997), 1735–1780. doi:10.1162/neco.1997.9.8.1735.

28.

Impedovo, More than twenty years of advancements on frontiers in handwriting recognition, Pattern Recognition 47(3) (2014), 916–928. doi:10.1016/j.patcog.2013.05.027.

29.

Jaeger,

Manke,

Reichert and

Waibel, Online handwriting recognition: The NPen++ recognizer, International Journal on Document Analysis and Recognition 3(3) (2001), 169–180. doi:10.1007/PL00013559.

30.

Johansson,

Atwell,

Garside and

Leech, The tagged LOB corpus: User’s manual, Technical report, Norwegian Computing Centre for the Humanities, Bergen, Norway, 1986.

31.

Kozielski,

Doetsch and

Ney, Improvements in RWTH’s system for off-line handwriting recognition, in: 12th International Conference on Document Analysis and Recognition (ICDAR), IEEE, 2013, pp. 935–939.

32.

Krizhevsky,

Sutskever and

G.E.

Hinton, ImageNet classification with deep convolutional neural networks, in: Advances in Neural Information Processing Systems (NIPS),

Pereira,

C.J.C.

Burges,

Bottou and

K.Q.

Weinberger, eds, Vol. 25, Curran Associates, Inc., 2012, pp. 1097–1105.

33.

Lecun,

Bottou,

Bengio and

Haffner, Gradient-based learning applied to document recognition, Proceedings of the IEEE 86(11) (1998), 2278–2324. doi:10.1109/5.726791.

34.

LeCun,

Cortes and

Burges, MNIST handwritten digit database, http://yann.lecun.com/exdb/mnist/.

35.

Liwicki,

Graves,

Bunke and

Schmidhuber, A novel approach to on-line handwriting recognition based on bidirectional long short-term memory networks, in: 9th International Conference on Document Analysis and Recognition (ICDAR), 2007, pp. 367–371.

36.

U.V.

Marti and

Bunke, The IAM-database: An English sentence database for offline handwriting recognition, International Journal on Document Analysis and Recognition 5 (2002), 39–46. doi:10.1007/s100320200071.

37.

Marukatat,

Artieres,

Gallinari and

Dorizzi, Sentence recognition through hybrid neuro-Markovian modeling, in: 6th International Conference on Document Analysis and Recognition (ICDAR), 2001, pp. 731–735.

38.

F.J.

Och, Minimum error rate training in statistical machine translation, in: 41st Annual Meeting on Association for Computational Linguistics, ACL’03, Vol. 1, 2003, pp. 160–167.

39.

Pastor Pellicer, Neural networks for document image and text processing, PhD thesis, Universitat Politècnica de València. Departamento de Sistemas Informáticos y Computación, Valencia, Spain, 2017.

40.

Pastor-Pellicer,

España-Boquera,

M.J.

Castro-Bleda and

Zamora-Martínez, A combined convolutional neural network and dynamic programming approach for text line normalization, in: 13th International Conference on Document Analysis and Recognition (ICDAR), 2015.

41.

Pastor-Pellicer,

España-Boquera,

Zamora-Martínez,

Zeshan Afzal and

M.J.

Castro-Bleda, Insights on the use of convolutional neural networks for document image binarization, in: The International Work-Conference on Artificial Neural Networks, Vol. 9095, 2015, pp. 115–126.

42.

Pham,

Bluche,

Kermorvant and

Louradour, Dropout improves recurrent neural networks for handwriting recognition, in: International Conference on Frontiers in Handwriting Recognition (ICFHR), 2014, pp. 285–290.

43.

Plamondon and

S.N.

Srihari, On-line and off-line handwriting recognition: A comprehensive survey, IEEE Trans. Pattern Anal. Mach. Intell. 22(1) (2000), 63–84. doi:10.1109/34.824821.

44.

Plötz and

G.A.

Fink, Markov models for offline handwriting recognition: A survey, International Journal of Document Analysis and Recognition 12 (2009), 269–298. doi:10.1007/s10032-009-0098-4.

45.

Poznanski and

Wolf, CNN-N-gram for HandwritingWord recognition, in: Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 2305–2314.

46.

Puigcerver, Are multidimensional recurrent layers really necessary for handwritten text recognition? in: Document Analysis and Recognition (ICDAR), 2017 14th IAPR International Conference on, Vol. 1, IEEE, 2017, pp. 67–72. doi:10.1109/ICDAR.2017.20.

47.

L.R.

Rabiner, A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition, 1989.

48.

Russakovsky,

Deng,

Su,

Krause,

Satheesh,

Ma,

Huang,

Karpathy,

Khosla,

Bernstein,

A.C.

Berg and

Fei-Fei, ImageNet large scale visual recognition challenge, International Journal of Computer Vision 115(3) (2015), 211–252. doi:10.1007/s11263-015-0816-y.

49.

T.N.

Sainath,

Kingsbury and

Ramabhadran, Auto-encoder bottleneck features using deep belief networks, in: International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2012, pp. 4153–4156.

50.

K.M.

Sayre, Machine recognition of handwritten words: A project report, Pattern Recognition 5(3) (1973), 213–228. doi:10.1016/0031-3203(73)90044-7.

51.

Schenkel,

Guyon and

Henderson, On-line cursive script recognition using time-delay neural networks and hidden Markov models, Machine Vision and Applications 8(4) (1995), 215–223. doi:10.1007/BF01219589.

52.

Schmidhuber, Deep learning in neural networks: An overview, CoRR abs/1404.7828 (2014).

53.

Schuster and

K.K.

Paliwal, Bidirectional recurrent neural networks, IEEE Transactions on Signal Processing 45(11) (1997), 2673–2681. doi:10.1109/78.650093.

54.

A.W.

Senior and

A.J.

Robinson, An off-line cursive handwriting recognition system, in: IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 20, 1998, pp. 309–321.

55.

Simonyan and

Zisserman, Very deep convolutional networks for large-scale image recognition, ArXiv, 2014, pp. 1–14.

56.

Singer and

R.P.

Lippman, A speech recognizer using radial basis function neural networks in an HMM framework, in: International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Vol. 1, IEEE, 1992, pp. 629–632.

57.

Stadermann, A hybrid SVM/HMM acoustic modeling approach to automatic speech recognition, in: International Conference on Spoken Language Processing (ICSLP), 2004.

58.

Stolcke, SRILM: An extensible language modeling toolkit, in: International Conference on Spoken Language Processing (ICSLP), 2002, pp. 901–904.

59.

Szegedy,

Liu,

Jia,

Sermanet,

Reed,

Anguelov,

Erhan,

Vanhoucke and

Rabinovich, Going deeper with convolutions, in: Conference on Computer Vision and Pattern Recognition (CVPR), 2015, pp. 1–12.

60.

A.H.

Toselli,

Juan,

González,

Salvador,

Vidal,

Casacuberta,

Keysers and

Ney, Integrated handwriting recognition and interpretation using finite-state models, International Journal of Pattern Recognition and Artificial Intelligence 18(4) (2004), 519–539. doi:10.1142/S0218001404003344.

61.

A.H.

Toselli,

Romero,

Pastor and

Vidal, Multimodal interactive transcription of text images, Pattern Recognition 43(5) (2010), 1814–1825. doi:10.1016/j.patcog.2009.11.019.

62.

J.M.

Vilar, Efficient computation of confidence intervals for word error rates, in: International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2008, pp. 5101–5104.

63.

Vincent,

Larochelle,

Lajoie,

Bengio and

P.-A.

Manzagol, Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion, Journal of Machine Learning Research 11(3) (2010), 3371–3408.

64.

Vinciarelli, A survey on off-line cursive word recognition, Pattern Recognition 35(7) (2002), 1433–1446. doi:10.1016/S0031-3203(01)00129-7.

65.

Voigtlaender,

Doetsch and

Ney, Handwriting recognition with large multidimensional long short-term memory recurrent neural networks, in: 15th International Conference on Frontiers in Handwriting Recognition (ICFHR), 2016, pp. 228–233. doi:10.1109/ICFHR.2016.0052.

66.

Wang,

Zhang,

Shen,

Zhang,

Lu,

Wu and

Wang, Intel math kernel library, in: High-Performance Computing on the Intel ^® Xeon Phi™ , Springer, 2014, pp. 167–188.

67.

Zamora-Martínez

et al., April-ANN Toolkit, a Pattern Recognizer in Lua, Artificial Neural Networks Module, 2013, https://github.com/pakozm/ [github.com]april-ann.

68.

Zamora-Martínez,

Frinken,

España-Boquera,

M.J.

Castro-Bleda,

Fischer and

Bunke, Neural network language models for off-line handwriting recognition, Pattern Recognition 47(4) (2014), 1642–1652. doi:10.1016/j.patcog.2013.10.020.

69.

Zeyer,

Beck,

Schlüter and

Ney, Ctc in the context of generalized full-sum hmm training, in: Interspeech, 2017, pp. 944–948. doi:10.21437/Interspeech.2017-1073.

Handwriting recognition by using deep learning to extract meaningful features

Abstract

Keywords

Get full access to this article

References