Sign languages, which are utilized by millions of deaf or hearing-impaired individuals globally, offer unique visual communication that varies across cultures. Yet, most current communication technologies are primarily designed for spoken languages, creating barriers for deaf people.
Objective
This study presents a novel sign language recognition method that learns from a few samples and requires no retraining for new signs, leveraging a unique approach inspired by the face recognition domain.
Methods
Using contrastive learning, a deep learning technique for representation learning, the model embeds each sign’s holistic representation, by clustering similar signs and distancing the dissimilar ones in the feature space.
Results
The method was evaluated on custom Multi-Examples Sign Language and Few-shot Sign Language datasets and outperformed state-of-the-art methods with an accuracy of 97.3% (an improvement of 6%).
Conclusions
This paper develops a novel method that integrates a contrastive learning approach tailored to address the unique challenges of this area. It also identified two key parameters (on the model performance) in the sign recognition domain: the impact of sample size and the influence of different regions of the signer’s body. This research offers an efficient solution to improve deaf people’s interaction with modern technologies.
KyleJGKyleJWollB. Sign language: the study of deaf people and their language. Cambridge University Press, 1988.
3.
BraggDKollerOBellardM, et al.Sign language recognition, generation, and translation: an interdisciplinary perspective. Proceedings of the 21st International ACM SIGACCESS Conference on Computers and Accessibility, 2019, pp. 16–31.
4.
KafleSHuenerfauthM. Predicting the understandability of imperfect English captions for people who are deaf or hard of hearing. ACM Transactions on Accessible Computing (TACCESS)2019; 12: 1–32. https://doi.org/10.1145/3325862
5.
MüllerMEblingSAvramidisE, et al.First WMT shared task on sign language translation (WMT-SLT22). 24th Annual Conference of the European Association for Machine Translation (EAMT 2023), 2023, pp. 507–508.
6.
LiDRodriguezCYuX, et al.Word-level deep sign language recognition from video: a new large-scale dataset and methods comparison. Proceedings of the IEEE/CVF winter conference on applications of computer vision, 2020, pp. 1459–1469.
7.
AlgahtaniSZuhairyRAlmarghalaniO, et al.Wearable Saudi sign language recognition device based on neural network. 2024 21st International Multi-Conference on Systems, Signals & Devices (SSD). IEEE, 2024, pp. 199–208.
8.
ShahinNWatfaM. Deaf and hard of hearing in the United Arab Emirates interacting with Alexa, an intelligent personal assistant. Technol Disabil2020; 32: 255–269. https://doi.org/10.3233/tad-200286
9.
WilliamsRJZipserD. A learning algorithm for continually running fully recurrent neural networks. Neural Comput1989; 1: 270–280. https://doi.org/10.1162/neco.1989.1.2.270
10.
LinC-HHsuK-CLiangC-K, et al.A disease-specific language representation model for cerebrovascular disease research. Comput Methods Progr Biomed2021; 211: 106446. https://doi.org/10.1016/j.cmpb.2021.106446
BorgMCamilleriKP. Sign language detection “in the wild” with recurrent neural networks. ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019, pp. 1637–1641.
13.
ChaikaewASomkuanKYuyenT. Thai sign language recognition: an application of deep neural network. 2021 joint international conference on digital arts, media and technology with ECTI northern section conference on electrical, electronics, computer and telecommunication engineering. IEEE, 2021, pp. 128–131.
14.
AlyamiSLuqmanHHammoudehM. Isolated Arabic sign language recognition using a transformer-based model and landmark keypoints. ACM Transactions on Asian and Low-Resource Language Information Processing2024; 23: 1–19. https://doi.org/10.1145/3584984
15.
DuarteAPalaskarSVenturaL, et al.How2sign: a large-scale multimodal dataset for continuous American sign language. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 2735–2744.
AthitsosVNeidleCSclaroffS, et al.The American sign language lexicon video dataset. 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops. IEEE, 2008, pp. 1–8.
JozeHRVMs-aslKO. A large-scale data set and benchmark for understanding American sign language. British Machine Vision Conference Cardiff, Wales, UK, 2019.
20.
WilburRKakAC. Purdue RVL-SLLL American sign language database, 2006.
21.
ChenTKornblithSNorouziM, et al.A simple framework for contrastive learning of visual representations. International conference on machine learning. PMLR, 2020, pp. 1597–1607.
22.
SchroffFKalenichenkoDPhilbinJ. Facenet: a unified embedding for face recognition and clustering. Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 815–823.
23.
FerreiraPMPernesDRebeloA, et al.Desire: deep signer-invariant representations for sign language recognition. IEEE Trans Syst Man Cybern: Systems2021; 51: 5830–5845. https://doi.org/10.1109/tsmc.2019.2957347
24.
SharmaSKumarK. ASL-3DCNN: american sign language recognition technique using 3-D convolutional neural networks. Multimed Tool Appl2021; 80: 26319–26331. https://doi.org/10.1007/s11042-021-10768-5
25.
AdeyanjuIABelloOOAdegboyeMA. Machine learning methods for sign language recognition: a critical review and analysis. Intelligent Systems with Applications2021; 12: 200056. https://doi.org/10.1016/j.iswa.2021.200056
26.
ZhouZTamVWLamEY. SignBERT: a BERT-based deep learning framework for continuous sign language recognition. IEEE Access2021; 9: 161669–161682. https://doi.org/10.1109/access.2021.3132668
27.
KumarEKKishorePKumarDA, et al.Early estimation model for 3D-discrete indian sign language recognition using graph matching. Journal of King Saud University-Computer and Information Sciences2021; 33: 852–864. https://doi.org/10.1016/j.jksuci.2018.06.008
28.
WenFZhangZHeT, et al.AI enabled sign language recognition and VR space bidirectional communication using triboelectric smart glove. Nat Commun2021; 12: 5378. https://doi.org/10.1038/s41467-021-25637-w
29.
Engberg-PedersenEBrentariDKDavisJ. Formal and functional factors in classifier choice: evidence from American Sign Language and Danish Sign Language. Open Linguist2025; 11: 20250043. https://doi.org/10.1515/opli-2025-0043
30.
OrtegaGSchiefnerALazarusN, et al.A lexical database of British Sign Language (BSL) and German Sign Language (DGS): iconicity ratings, iconic strategies, and concreteness norms. Behav Res Methods2025; 57: 139. https://doi.org/10.3758/s13428-025-02660-z
YinKAtwellKHochgesangJA, et al.The importance of including signed languages in natural language processingSign language machine translation. Springer, 2024, pp. 73–87.
33.
CamgozNCKollerOHadfieldS, et al.Sign language transformers: joint end-to-end sign language recognition and translation. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 10023–10033.
34.
TaoTZhaoYLiuT, et al.Sign language recognition: a comprehensive review of traditional and deep learning approaches, datasets, and challenges. IEEE Access2024.
35.
KasapbaşiAElbushraAEAOmarA-H, et al.DeepASLR: a CNN based human computer interface for American Sign language recognition for hearing-impaired individuals. Computer methods and programs in biomedicine update2022; 2: 100048. https://doi.org/10.1016/j.cmpbup.2021.100048
36.
MorrisA. How a visual language evolves as our world does. The New York Times, 2022.
37.
AmirATabaBBergD, et al.A low power, fully event-based gesture recognition system. Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 7243–7252.
38.
SunCShrivastavaASinghS, et al.Revisiting unreasonable effectiveness of data in deep learning era. Proceedings of the IEEE international conference on computer vision, 2017, pp. 843–852.
39.
GirshickR. Fast R-CNN. In: Fast r-cnn in proceedings of the ieee international conference on computer vision. IEEE[Google Scholar], 2015, vol 2, pp. 1440–1448. https://doi.org/10.1109/iccv.2015.169
40.
LugaresiCTangJNashH, et al.Mediapipe: a framework for building perception pipelines. Third Workshop on Computer Vision for AR/VR Long Beach, CA, USA, 2019.
41.
VoulodimosADoulamisNDoulamisA, et al.Deep learning for computer vision: a brief review. Comput Intell Neurosci2018; 2018: 7068349. https://doi.org/10.1155/2018/7068349
42.
TarwaniKMEdemS. Survey on recurrent neural network in natural language processing. Int J Eng Trends Technol2017; 48: 301–304.
43.
HochreiterS. Long short-term memory. Neural Computation MIT-Press1997.
44.
ChoK. Learning phrase representations using RNN encoder-decoder for statistical machine translation. Conference on Empirical Methods in Natural Language Processing. Doha, 2014.
45.
BromleyJGuyonILeCunY, et al.Signature verification using a “siamese” time delay neural network. Adv Neural Inf Process Syst1993; 6: 737–744.
AranOAkarunL. A multi-class classification strategy for Fisher scores: application to signer independent sign language recognition. Pattern Recogn2010; 43: 1776–1788. https://doi.org/10.1016/j.patcog.2009.12.002
CaoZSimonTWeiS-E, et al.Realtime multi-person 2d pose estimation using part affinity fields. Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 7291–7299.
50.
DengXYangSZhangY, et al.Hand3d: hand pose estimation using 3d neural network. IEEE Trans Image Process2017; 27: 1888–1900.
51.
CardenasEJEChavezGC. Multimodal hand gesture recognition combining temporal and pose information based on CNN descriptors and histogram of cumulative magnitudes. J Vis Commun Image Represent2020; 71: 102772. https://doi.org/10.1016/j.jvcir.2020.102772
52.
AdhikarySTalukdarAKSarmaKK. A vision-based system for recognition of words used in Indian sign language using mediapipe. 2021 sixth international conference on image information processing (ICIIP). IEEE, 2021, pp. 390–394.
53.
WangYLiRLiG. Sign language recognition using Mediapipe. International Conference on Computer Graphics, Artificial Intelligence, and Data Processing (ICCAID 2022). SPIE, 2023, pp. 807–813.
54.
ChaikaewA. An applied holistic landmark with deep learning for Thai sign language recognition. 2022 37th International Technical Conference on Circuits/Systems, Computers and Communications (ITC-CSCC). IEEE, 2022, pp. 1046–1049.
55.
FarhanYMadiAA. Real-time dynamic sign recognition using mediapipe. 2022 IEEE 3rd International Conference on Electronics, Control, Optimization and Computer Science (ICECOCS) 2022. IEEE, pp. 1–7.
BohacekMHrúzM. Learning from what is already out there: few-shot sign language recognition with online dictionaries. 2023 IEEE 17th International conference on automatic face and gesture recognition (FG) 2023. IEEE, pp. 1–6.
58.
FoodyGMMathurASanchez-HernandezC, et al.Training set size requirements for the classification of a specific class. Rem Sens Environ2006; 104: 1–14. https://doi.org/10.1016/j.rse.2006.03.004
59.
ShangYJiangKWangL, et al.The 30-days hospital readmission risk in diabetic patients: predictive modeling with machine learning classifiers. BMC Med Inf Decis Making2021; 21: 1–11. https://doi.org/10.1186/s12911-021-01423-y
60.
JoshiGSinghSVigR. Taguchi-TOPSIS based HOG parameter selection for complex background sign language recognition. J Vis Commun Image Represent2020; 71: 102834. https://doi.org/10.1016/j.jvcir.2020.102834
61.
DreuwPSteinDDeselaersT, et al.Spoken language processing techniques for sign language recognition and translation. Technol Disabil2008; 20: 121–133. https://doi.org/10.3233/tad-2008-20207
62.
TchuenteDSenningerDPietschH, et al.Providing more regular road signs infrastructure updates for connected driving: a crowdsourced approach with clustering and confidence level. Decis Support Syst2021; 141: 113443. https://doi.org/10.1016/j.dss.2020.113443
63.
ZhangXChanFTYanC, et al.Towards risk-aware artificial intelligence and machine learning systems: an overview. Decis Support Syst2022; 159: 113800. https://doi.org/10.1016/j.dss.2022.113800
64.
ViraniNIyerNYangZ. Justification-based reliability in machine learning. Proceedings of the aaai conference on artificial intelligence, 2020, pp. 6078–6085.
65.
NapoliDJSutton-SpenceR. Order of the major constituents in sign languages: implications for all language. Front Psychol2014; 5: 376. https://doi.org/10.3389/fpsyg.2014.00376
66.
FengJSunJShaoD, et al.Improving the robustness of machine reading comprehension via contrastive learning. Appl Intell2023; 53: 9103–9114. https://doi.org/10.1007/s10489-022-03947-w
67.
WadhawanAKumarP. Sign language recognition systems: a decade systematic literature review. Arch Comput Methods Eng2021; 28: 785–813. https://doi.org/10.1007/s11831-019-09384-2
68.
ConkerN. Sign language interpreting in the Turkish public service domains in the 21st century. Çeviribilim ve Uygulamaları Dergisi2020: 42–60. https://doi.org/10.37599/ceviri.811094
69.
RidhaAMShehiebW. Assistive technology for hearing-impaired and deaf students utilizing augmented reality. 2021 IEEE Canadian Conference on Electrical and Computer Engineering (CCECE). IEEE, 2021, pp. 1–5.
LohHWOoiCPSeoniS, et al.Application of explainable artificial intelligence for healthcare: a systematic review of the last decade (2011–2022). Comput Methods Progr Biomed2022; 226: 107161. https://doi.org/10.1016/j.cmpb.2022.107161
72.
KatılmışZKarakuzuC. ELM based two-handed dynamic turkish sign language (TSL) word recognition. Expert Syst Appl2021; 182: 115213. https://doi.org/10.1016/j.eswa.2021.115213
73.
XuLXiXPattonA, et al.A cross-language comparison of sentence recognition using American English and Mandarin Chinese HINT and AzBio sentences. Ear Hear2021; 42: 405–413. https://doi.org/10.1097/AUD.0000000000000938
74.
Desmeules-TrudelFZamunerTS. Spoken word recognition in a second language: the importance of phonetic details. Second Lang Res2023; 39: 333–362. https://doi.org/10.1177/02676583211030604
75.
WeiCZhaoJZhouW, et al.Semantic boundary detection with reinforcement learning for continuous sign language recognition. IEEE Trans Circ Syst Video Technol2020; 31: 1138–1149. https://doi.org/10.1109/tcsvt.2020.2999384
76.
SharmaSGuptaRKumarA. Continuous sign language recognition using isolated signs data and deep transfer learning. J Ambient Intell Hum Comput2023; 14: 1531–1542. https://doi.org/10.1007/s12652-021-03418-z
77.
ZahediMKeysersDDeselaersT, et al.Combination of tangent distance and an image distortion model for appearance-based sign language recognition. Pattern Recognition: 27th DAGM Symposium, Vienna, Austria, August 31-September 2, 2005 Proceedings 27 2005. Springer, pp. 401–408.
