Person identification using biometric features is an effective method for recognizing and authenticating the identity of a person. Multimodal biometric systems combine different biometric modalities in order to make better predictions as well as for achieving increased robustness. This paper proposes a touchless multimodal person identification model using deep learning techniques by combining the gait and speech modalities. Separate pipelines for both the modalities were developed using Convolutional Neural Networks. The paper also explores various fusion strategies for combining the two pipelines and shows how various metrics get affected with different fusion strategies. Results show that weighted average and product fusion rules work best for the data used in the experiments.
VenkatramanS. and DelpachitraI., Biometrics in banking security: A case study,415–430. 10, Inf Manag Comput Security16 (2008).
2.
OloyedeM.O. and HanckeG.P., Unimodal and multimodal biometric sensing systems: A review, IEEE Access4 (2016), 7532–7555.
3.
LecunY., BengioY. and HintonG., Deep learning, Nature Cell Biology521(7553) (2015), 436–444.
4.
CastroF.M., Marin-JimenezM.J., GuilN. and de la BlancaN.P., Automatic learning of gait signatures for people identification, (2016).
5.
SokolovaA. and KonushinA., Pose-based deep gait recognition, (2017).
6.
SinghS., GyaourovaA., BebisG. and PavlidisI., Infrared and visible image fusion for face recognition, Proceedings of SPIE - The International Society for Optical Engineering5404 04 (2004).
7.
Gopal and SrivastavaS., Accurate human recognition by score-level and feature-level fusion using palm–phalanges print, Arabian Journal for Science and Engineering43(2) (2018), 543–554. [Online]. Available: https://doi.org/10.1007/s13369-017-2644-6.
8.
KowtkoM.A., Biometric authentication for older adults, in IEEE Long Island Systems, Applications and Technology (LISAT) Conference 2014, (2014), 1–6.
9.
ZhouX. and BhanuB., Feature fusion of face and gait for human recognition at a distance in video, in, 18th International Conference on Pattern Recognition (ICPR’06)4 (2006), 529–532.
10.
GhateM.D.J. and PatilS.B., Robust combination method for privacy protection using fingerprint and face biometrics, in 2015 4th International Conference on Reliability, Infocom Technologies and Optimization (ICRITO) (Trends and Future Directions), (2015), 1–6.
FrischholzR.W. and DieckmannU., Biold: a multimodal biometric identification system, Computer33(2) (2000), 64–68.
13.
ChaudharyG., SrivastavaS., BhardwajS. and BhargavaS., Fusion of palm-phalanges print with palmprint and dorsal hand vein, Applied Soft Computing47 05 (2016).
14.
ChaudharyG., SrivastavaS. and BhardwajS., A robust biometrics multilevel fusion scheme, International Journal of Image Mining2 (2016), 100. 01.
15.
NarulaP., BhargavaS., SrivastavaS. and HanmandluM., Multimodal biometric system based on information set theory and refined scores, Soft Computing, 03 (2016).
16.
AroraP., SrivastavaS., HanmandluM. and BhargavaS., Robust authentication using dorsal hand vein images, IEEE Intelligent Systems34(2) (2019), 25–35.
17.
AlexandreL., CampilhoA. and KamelM.S., On combining classifiers using sum and product rules, Pattern Recognition Letters22 (2001), 1283–1289. 10
18.
HanmandluM., GroverJ., MadasuV.K. and VasirkalaS., Score level fusion of hand based biometrics using t-norms, in 2010 IEEE International Conference on Technologies for Homeland Security (HST) (2010), 70–76.
19.
CasazzaP.G. and PetersonJ., Weighted fusion frame construction via spectral tetris, Advances in Computational Mathematics40(2) (2014), 335–351. [Online]. Available: https://doi.org/10.1007/s10444-013-9310-7.
20.
ChingH.C. and EswaranC., A nature based fusion scheme for multimodal biometric person identity verification at a distance, in 2009 International Conference on Signal Acquisition and Processing (2009), 94–97.
21.
HanJ. and BhanuB., Individual recognition using gait energy image, IEEE Transactions on Pattern Analysis and Machine Intelligence28(2) (2006), 316–322.
LamT.H., CheungK. and LiuJ.N., Gait flow image: A silhouette-based gait representation for human identification, Pattern Recognition44(4) (2011), 973–987.
24.
LeeC.P., TanA.W. and TanS.C., Gait probability image: An information-theoretic model of gait representation, Journal of Visual Communication and Image Representation25(6) (2014), 1489–1492.
25.
AroraP. and SrivastavaS., Gait recognition using gait gaussian image, in 2015 2nd International Conference on Signal Processing and Integrated Networks (SPIN) (2015), 791–794.
26.
NarulaP., SrivastavaS. and SinghalS., Analysis of gait flow image and gait gaussian image using extension neural network for gait recognition, International Journal of Rough Sets and Data Analysis3 (2016), 45–64. 04
27.
AroraP., SrivastavaS., AroraK. and BarejaS., Improved gait recognition using gradient histogram gaussian image, Procedia Computer Science58 (2015), 408–413. second International Symposium on Computer Vision and the Internet (VisionNet’15).
28.
NarulaP., HanmandluM. and SrivastavaS., Gait based authentication using gait information image features, Pattern Recognition Letters 06 (2015).
SarinS., ChughA., MittalA. and SrivastavaS., Comparing cnn architectures for gait recognition using optical flows, in International Conference on Innovative Computing and Communications. Singapore: Springer Singapore, (2021), 887–893.
31.
WolfT., BabaeeM. and RigollG., Multi-view gait recognition using 3d convolutional neural networks, 2016 IEEE International Conference on Image Processing (ICIP), (2016), 4165–4169.
32.
HansenJ.H.L. and HasanT., Speaker recognition by machines and humans: A tutorial review, IEEE Signal Processing Magazine32(6) (2015), 74–99.
33.
ReynoldsD.A. and RoseR.C., Robust text-independent speaker identification using gaussian mixture speaker models, IEEE Transactions on Speech and Audio Processing3(1) (1995), 72–83.
34.
GroverM.S., KumarY., SarinS., VafaeeP., HamaM. and ShahR.R., Multi-modal automated speech scoring using attention fusion, (2020).
35.
McLarenM., LeiY., SchefferN. and FerrerL., Application of convolutional neural networks to speaker recognition in noisy conditions, in INTERSPEECH, (2014).
FarnebäckG., Two-framemotion estimation based on polynomial expansion, in Proceedings of the 13th Scandinavian Conference on Image Analysis, ser. SCIA’03. Berlin, Heidelberg: Springer-Verlag, (2003), 363–370.
38.
ChoiK., FazekasG. and SandlerM., Automatic tagging using deep convolutional neural networks, (2016).
39.
OordA.v.d., DielemanS. and SchrauwenB., Deep contentbased music recommendation, in Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 2, ser. NIPS’13. Red Hook, NY, USA: Curran Associates Inc., (2013), 2643–2651.
StevensS.S., VolkmannJ. and NewmanE.B., A scale for the measurement of the psychological magnitude pitch, The Journal of the Acoustical Society of America8(3) (1937), 185–190, [Online]. Available: https://doi.org/10.1121/1.1915893.
42.
MooreB., An Introduction to the Psychology of Hearing. Leiden, The Netherlands: Brill, (2013). [Online]. Available: https://brill.com/view/title/24210.
43.
ChoiK., FazekasG., ChoK. and SandlerM., A comparison on audio signal preprocessing methods for deep neural networks on music tagging, 09 (2017).
44.
SimonyanK. and ZissermanA., Very deep convolutional networks for large-scale image recognition, (2014).
45.
SrivastavaN., HintonG., KrizhevskyA., SutskeverI. and SalakhutdinovR., Dropout: A simple way to prevent neural networks from overfitting, Journal of Machine Learning Research15(56) (2014), 1929–1958. [Online]. Available: http://jmlr.org/papers/v15/srivastava14a.html.
46.
ClevertD.-A., UnterthinerT. and HochreiterS., Fast and accurate deep network learning by exponential linear units (elus), (2015).
47.
TrottierL., GiguèreP. and Chaib-draaB., Parametric exponential linear unit for deep convolutional neural networks, (2016).
48.
KingmaD.P. and BaJ., Adam: A method for stochastic optimization, (2014).
