The location attention mechanism has been widely applied in deep neural networks. However, as the mechanism entails heavy computing workload, significant memories consumed for weights storage, and shows poor parallelism in some calculations, it is hard to achieve high efficiency deployment. In this paper, the field-programmable gate array (FPGA) is employed to implement the location attention mechanism in hardware, and a novel fusion approach is proposed to connect the convolutional layer with the fully connected layer, which not only improves the parallelism of both the algorithm and the hardware pipeline, but also reduces the computation cost for such operations as multiplication and addition. Meanwhile, the shared computing architecture is used to reduce the demand for hardware resources. Parallel computing arrays are utilized to time-multiplex a single computing array, which can speed up the pipeline parallel computing of the attention mechanism. Experimental results show that for the location attention mechanism, the FPGA’s inference speed is 0.010 ms, which is around a quarter of the speed achieved by running it with GPU, and its power consumption is 1.73 W, which is about 2.89% of the power consumed by running it with CPU. Compared with other FPGA implementation methods of attention mechanism, it has less hardware resource consumption and less inference time. When applied to speech recognition tasks, the trained attention model is symmetrically quantized and deployed on the FPGA. The result shows that the word error rate is only 0.79% higher than that before quantization, which proves the effectiveness and correctness of the hardware circuit.
RenS., HeK., GirshickR., et al., Faster r-cnn: Towards real-time object detection with region proposal networks[J], Advances in Neural Information Processing Systems28 (2015), 91–99.
2.
DevlinJ., ChangM.W., LeeK., et al., Bert: Pre-training of deep bidirectional transformers for language understanding[J]. arXiv preprint arXiv:1810.04805, 2018.
3.
ChanW., JaitlyN., LeQ., et al., Listen, attend and spell: A neural network for large vocabulary conversational speech recognition[C]//IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE (2016), 4960–4964.
4.
ChoK., Van MerriënboerB.,
GulcehreC., et al., Learning phrase representations using RNN encoder-decoder for statistical machine translation[J]. arXiv preprint arXiv:1406.1078, 2014.
5.
ManaswiN.K., Deep learning with applications using python: chatbots and face, object, and speech recognition with tensorflow and keras[M], Apress, 2018.
6.
Sharif RazavianA.,
AzizpourH.,
SullivanJ., et al., CNN features off-the-shelf: an astounding baseline for recognition[C]//, Proceedings of the IEEE conference on computer vision and pattern recognition workshops (2014), 806–813.
7.
GreffK., SrivastavaR.K., KoutníkJ., et al., LSTM: A search space odyssey[J], IEEE Transactions on Neural Networks and Learning Systems28(10) (2016), 2222–2232.
8.
ChungJ., GulcehreC., ChoK.H., et al., Empirical evaluation of gated recurrent neural networks on sequence modeling[J]. arXiv preprint arXiv:1412.3555, 2014.
9.
VaswaniA., ShazeerN., ParmarN., et al., Attention is all you need[C]//, Advances in Neural Information Processing Systems (2017), 5998–6008.
10.
LuongM.T., PhamH. and ManningC.D., Effective approaches to attention-based neural machine translation[J], arXiv preprint arXiv:1508.04025, 2015.
11.
GolubD. and HeX., Character-level question answering with attention[J]. arXiv preprint arXiv:1604.00727, 2016.
12.
BasiriM.E., NematiS., AbdarM., et al., ABCDM: An attention-based bidirectional CNN-RNN deep model for sentiment analysis[J], Future Generation Computer Systems115 (2021), 279–294.
13.
TianY., SongY. and XiaF., Joint Chinese Word Segmentation and Part-of-speech Tagging via Multi-channel Attention of Character N-grams[C]//, Proceedings of the 28th International Conference on Computational Linguistics (2020), 2073–2084.
14.
WangK., ZhongH., YuN., et al., Nonintrusive load monitoring based on sequence-to-sequence model with attention mechanism[C]//, Zhongguo Dianji Gongcheng Xuebao/Proceedings of the Chinese Society of Electrical Engineering39(1) (2019), 75–83.
15.
ShenY.H., HeK.X. and ZhangW.Q., SAM-GCNN: A gated convolutional neural network with segment-level attention mechanism for home activity monitoring[C]//, 2018 IEEE International Symposium on Signal Processing and Information Technology (ISSPIT). IEEE (2018), 679–684.
16.
HuD., An introductory survey on attention mechanisms in NLP problems[C]//, Proceedings of SAI Intelligent Systems Conference. Springer, Cham, (2019), 432–448.
17.
ChaudhariS., MithalV., PolatkanG., et al., An attentive survey of attention models[J], arXiv preprint arXiv:1904.02874, 2019.
18.
IrieK., TüskeZ., AlkhouliT., et al., LSTM, GRU, highway and a bit of attention: an empirical overview for language modeling in speech recognition[C]//, Interspeech (2016), 3519–3523.
19.
MnihV., HeessN. and GravesA., Recurrent models of visual attention[C]//, Advances in Neural Information Processing Systems (2014), 2204–2212.
20.
XiaoT., XuY., YangK., et al., The application of two-level attention models in deep convolutional neural network for fine-grained image classification[C]//, Proceedings of the IEEE Conference on Computer Vision and Pattern recognition (2015), 842–850.
21.
JaderbergM., SimonyanK. and ZissermanA., Spatial transformer networks[J], Advances in Neural Information Processing Systems28 (2015), 2017–2025.
22.
HuJ., ShenL. and SunG., Squeeze-and-excitation networks[C]//, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2018), 7132–7141.
23.
LiH., XiongP., AnJ., et al., Pyramid attention network for semantic segmentation[J], arXiv preprint arXiv:1805.10180, 2018.
24.
WooS., ParkJ., LeeJ.Y., et al., Cbam: Convolutional block attention module[C]//, Proceedings of the European conference on computer vision (ECCV) (2018), 3–19.
25.
DuW., ZhangL., SunL., et al., Research and application of semantic understanding based on Attention-RNN[J], Procedia Computer Science183 (2021), 337–340.
26.
ChorowskiJ., BahdanauD., SerdyukD., et al., Attention-based models for speech recognition[J], arXiv preprint arXiv:1506.07503, 2015.
27.
KimS., HoriT. and WatanabeS., Joint CTC-attention based end-to-end speech recognition using multi-task learning[C]//, 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, (2017), 4835–4839.
28.
InagumaH. and KawaharaT., Alignment Knowledge Distillation for Online Streaming Attention-based Speech Recognition[J], arXiv preprint arXiv:2103.00422, 2021.
29.
YanS., YuW. and Shui-qingW.U., Machine Translation System Based on Self-Attention Model[J], Computer and Modernization2019(07), 9.
30.
VestiasM. and NetoH., Trends of CPU, GPU and FPGA for high-performance computing[C]//, 2014 24th International Conference on Field Programmable Logic and Applications (FPL). IEEE, (2014), 1–6.
31.
NurvitadhiE., SheffieldD., SimJ., et al., Accelerating binarized neural networks: Comparison of FPGA, CPU, GPU, and ASIC[C]//, 2016 International Conference on Field-Programmable Technology (FPT). IEEE, (2016), 77–84.
32.
KesturS., DavisJ.D. and WilliamsO., Blas comparison on fpga, cpu and gpu[C]//, 2010 IEEE computer society annual symposium on VLSI. IEEE, (2010), 288–293.
33.
NurvitadhiE., SimJ., SheffieldD., et al., Accelerating recurrent neural networks in analytics servers: Comparison of FPGA, CPU, GPU, and ASIC[C]//, 2016 26th International Conference on Field Programmable Logic and Applications (FPL). IEEE, (2016), 1–4.
34.
ZhaoR., NiuX., WuY., et al., Optimizing CNN-based object detection algorithms on embedded FPGA platforms[C]//, International Symposium on Applied Reconfigurable Computing. Springer, Cham, (2017), 255–267.
35.
LiH., LinZ., ShenX., et al., A convolutional neural network cascade for face detection[C]//, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2015), 5325–5334.
36.
LyuY., BaiL. and HuangX., Real-time road segmentation using lidar data processing on an fpga[C]//, 2018 IEEE International Symposium on Circuits and Systems (ISCAS). IEEE, (2018), 1–5.
37.
LeeM., HwangK., ParkJ., et al., FPGA-based low-power speech recognition with recurrent neural networks[C]//, 2016 IEEE International Workshop on Signal Processing Systems (SiPS). IEEE, (2016), 230–235.
38.
WhittingtonJ., DeoK., KleinschmidtT., et al., FPGA implementation of spectral subtraction for in-car speech enhancement and recognition[C]//, 2008 2nd International Conference on Signal Processing and Communication Systems. IEEE, (2008), 1–8.
39.
BahouraM. and EzzaidiH., FPGA-implementation of discrete wavelet transform with application to signal denoising[J], Circuits, Systems, and Signal Processing31(3) (2012), 987–1015.
40.
ShengxueZ., English corpus translation system based on FPGA and machine learning[J], Microprocessors and Microsystems (2020), 103464.
41.
OnoT., ShojiT., WaidyasooriyaH.M., et al., Fpga-based acceleration of word2vec using opencl[C]//, 2019 IEEE International Symposium on Circuits and Systems (ISCAS). IEEE, (2019), 1–5.
42.
QiuJ., WangJ., YaoS., et al., Going deeper with embedded fpga platform for convolutional neural network[C]//, Proceedings of the 2016 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays (2016), 26–35.
43.
ChangA.X.M., MartiniB. and CulurcielloE., Recurrent neural networks hardware implementation on FPGA[J], arXiv preprint arXiv:1511.05552, 2015.
44.
JamelT.M. and KhammasB.M., Implementation of a sigmoid activation function for neural network using FPGA[C]//, 13th Scientific Conference of Al-Ma’moon University College (2012), 13.
45.
SledevicT., Adaptation of convolution and batch normalization layer for CNN implementation on FPGA[C]//, 2019 Open Conference of Electrical, Electronic and Information Sciences (eStream). IEEE, (2019), 1–4.
46.
YanW., TongW. and ZhiX., FPGAN: an FPGA accelerator for graph attention networks with software and hardware co-optimization[J], IEEE Access8 (2020), 171608–171620.
47.
LuS., WangM., LiangS., et al., Hardware Accelerator for Multi-Head Attention and Position-Wise Feed-Forward in the Transformer[J], arXiv preprint arXiv:2009.08605, 2020.
48.
LiB., PandeyS., FangH., et al., Ftrans: energy-efficient acceleration of transformers using fpga[C]//, Proceedings of the ACM/IEEE International Symposium on Low Power Electronics and Design (2020), 175–180.
49.
MossD.J.M., KrishnanS., NurvitadhiE., et al., A customizable matrix multiplication framework for the intel harpv2 xeon+ fpga platform: A deep learning case study[C]//, Proceedings of the 2018 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays (2018), 107–116.
50.
JiangJ., MirianV., TangK.P., et al., Matrix multiplication based on scalable macro-pipelined FPGA accelerator architecture[C]//, 2009 International Conference on Reconfigurable Computing and FPGAs. IEEE, (2009), 48–53.
51.
HamT.J., JungS.J., KimS., et al., A∧ 3: Accelerating Attention Mechanisms in Neural Networks with Approximation[C]//, 2020 IEEE International Symposium on High Performance Computer Architecture (HPCA). IEEE, (2020), 328–341.
