The use of computers as a communication tool by people with disabilities can serve as an alternative effective to promote social interactions and the more inclusive and active participation of people in society.
OBJECTIVE:
This paper presents a systematic mapping of the literature that provides a survey of scientific contributions where Computer Vision is applied to enable users with motor and speech impairments to access computers easily, allowing them to exert their communicative abilities.
METHODS:
The mapping was conducted employing searches that identified 221 potentially eligible scientific articles published between 2009 and 2019, indexed by ACM, IEEE, Science Direct, and Springer databases.
RESULTS:
From the retrieved papers, 33 were selected and categorized into themes of this research interest: Human-Computer Interaction, Human-Machine Interaction, Human-Robot Interaction, Recreation, and surveys. Most of the chosen studies use sets of predefined gestures, low-cost cameras, and tracking a specific body region for gestural interaction.
CONCLUSION:
The results offer an overview of the Computer Vision techniques used in applied research on Assistive Technology for people with motor and speech disabilities, pointing out opportunities and challenges in this research domain.
AscariREdOSSilvaLPereiraR. Personalized gestural interaction applied in a gesture interactive gamebased approach for people with disabilities. In: Proceedings of the 25th International Conference on Intelligent User Interfaces; 2020. pp. 100–110.
2.
WangRHViswanathanPCzarnuchSBogerJNejatGMihailidisA. Developing advanced assistive technologies for older adults with dementia: Lessons learned. Proceedings of Rehabilitation Engineering Society of North America (RESNA). 2013; 22–24.
3.
AscariREdOSPereiraRSilvaL. Computer vision-based methodology to improve interaction for people with motor and speech impairmen. ACM Transactions on Accessible Computing (ACM TACCESS). 2020; 13(4): 1–33.
4.
BetkeMGipsJFlemingP. The Camera Mouse: visual tracking of body features to provide computer access for people with severe disabilities. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 2002; 10(1): 1–10.
5.
MarnikJ. BlinkMouse-On-Screen Mouse Controlled by Eye Blinks. In: Information Technologies in Biomedicine. Springer; Vol. 4, 2014. pp. 237–248.
6.
AscariREdOSPereiraRSilvaL. Towards a Methodology to Support Augmentative and Alternative Communication by Means of Personalized Gestural Interaction. In: Proceedings of the 17th Brazilian Symposium on Human Factors in Computing Systems. ACM; 2018. p. 38.
7.
MontaniniLCippitelliEGambiESpinsanteS. Low complexity head tracking on portable android devices for real time message composition. Journal on Multimodal User Interfaces. 2015; 9(2): 141–151.
8.
AscariREdOSPereiraRSilvaL. Mobile interaction for augmentative and alternative communication: a systematic mapping. SBC Journal on 3D Interactive Systems. 2018; 9(2): 105–118.
9.
MauriCGranollers i SaltiveriTLorés VidalJGarcíaM. Computer vision interaction for people with severe movement restrictions. Human Technology: An Interdisciplinary Journal on Humans in ICT Environments. 2006; 2(num 1): 38–54.
10.
MorikawaCLyonsMJ. Design and evaluation of vision-based head and face tracking interfaces for assistive input. In: Smart Technologies: Breakthroughs in Research and Practice. IGI Global; 2018. pp. 67–96.
11.
RiveraLADeSouzaGN. Haptic and Gesture-Based Assistive Technologies for People with Motor Disabilities. In: Assistive Technologies and Computer Access for Motor Disabilities. IGI Global; 2014. pp. 1–27.
12.
GevarterCO’ReillyMFRojeskiLSammarcoNLangRLancioniGE, et al. Comparisons of intervention components within augmentative and alternative communication systems for individuals with developmental disabilities: a review of the literature. Research in Developmental Disabilities. 2013; 34(12): 4404–4414.
13.
BiswasP. Inclusive Human Machine Interaction for India: A Case Study of Developing Inclusive Applications for the Indian Ulation – User Interaction. Springer; 2014.
14.
KruegerMWGionfriddoTHinrichsenK. VIDEOPLACE – an artificial reality. In: ACM SIGCHI Bulletin. ACM; Vol. 16, 1985. pp. 35–40.
15.
JacobRJ. The use of eye movements in humancomputer interaction techniques: what you look at is what you get. ACM Transactions on Information Systems (TOIS). 1991; 9(2): 152–169.
16.
PetersenKVakkalankaSKuzniarzL. Guidelines for conducting systematic mapping studies in software engineering: an update. Information and Software Technology. 2015; 64: 1–18.
17.
KitchenhamB. Procedure for undertaking systematic reviews. Computer Science Depart-ment, Keele University (TRISE-0401) and National ICT Australia Ltd (0400011T 1), Joint Technical Report. 2004.
18.
MunzlingerENarcizoFBde QueirozJER. Sistematização de revisões bibliográficas em pesquisas da área de IHC. In: Companion Proceedings of the 11th Brazilian Symposium on Human Factors in Computing Systems. Brazilian Computer Society; 2012. pp. 51–54.
19.
EscaleraSAthitsosVGuyonI. Challenges in multimodal gesture recognition. In: Gesture Recognition. Springer; 2017. pp. 1–60.
20.
Graham-KnightKTzanetakisG. Adaptive music technology using the Kinect. In: Proceedings of the 8th ACM International Conference on PErvasive Technologies Related to Assistive Environments; 2015. pp. 1–4.
21.
KurauchiAFengWMorimotoCBetkeM. HMAGIC: head movement and gaze input cascaded pointing. In: Proceedings of the 8th ACM International Conference on PErvasive Technologies Related to Assistive Environments; 2015. pp. 1–4.
22.
SalehSBernsK. Nonverbal communication with a humanoid robot via head gestures. In: Proceedings of the 8th ACM International Conference on PErvasive Technologies Related to Assistive Environments; 2015. pp. 1–8.
23.
MissimerEBetkeM. Blink and wink detection for mouse pointer control. In: Proceedings of the 3rd International Conference on PErvasive Technologies Related to Assistive Environments; 2010. pp. 1–8.
24.
PaquetteIKwanCBetkeM. Menu controller: making existing software more accessible for people with motor impairments. In: Proceedings of the 4th International Conference on PErvasive Technologies Related to Assistive Environments; 2011. pp. 1–8.
25.
McMurroughCRichJMetsisVNguyenAMakedonF. Low-cost head position tracking for gaze point estimation. In: Proceedings of the 5th International Conference on PErvasive Technologies Related to Assistive Environments; 2012. pp. 1–4.
26.
FengWChenMBetkeM. Target reverse crossing: a selection method for camera-based mousereplacement systems. In: Proceedings of the 7th International Conference on PErvasive Technologies Related to Assistive Environments; 2014. pp. 1–4.
27.
GhanemSConlyCAthitsosV. A survey on sign language recognition using smartphones. In: Proceedings of the 10th International Conference on PErvasive Technologies Related to Assistive Environments; 2017. pp. 171–176.
28.
SchrammelJLeitnerMTscheligiM. Semantically structured tag clouds: an empirical evaluation of clustered presentation approaches. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems; 2009. pp. 2037–2040.
29.
PalSMangalNKKhoslaA. Development of assistive application for patients with communication disability. In: 2017 International Conference on Innovations in Green Energy and Healthcare Technologies (IGEHT). IEEE; 2017. pp. 1–4.
30.
FineGTsotsosJK. Examining the feasibility of face gesture detection using a wheelchair mounted camera. In: Proceedings of the 1st ACM SIGMM International Workshop on Media Studies and Implementations That Help Improving Access to Disabled Users; 2009. pp. 19–28.
31.
GaoGWDuanXY. An overview of human-computer interaction based on the camera for disabled people. In: Advanced Materials Research. Trans Tech Publ; Vol. 219, 2011. pp. 1317–1320.
32.
KaramchandaniHChauTHobbsDMumfordL. Development of a low-cost, portable, tablet-based eye tracking system for children with impairments. In: Proceedings of the International Convention on Rehabilitation Engineering & Assistive Technology; 2015. pp. 1–4.
33.
CristinaSCamilleriKP. Model-based head pose-free gaze estimation for assistive communication. Computer Vision and Image Understanding. 2016; 149: 157–170.
34.
RosalesCJácomeLCarriónJJaramilloCPalmaM. Computer vision for detection of body expressions of children with cerebral palsy. In: 2017 IEEE Second Ecuador Technical Chapters Meeting (ETCM). IEEE; 2017. pp. 1–6.
35.
KakkothSSGhargeS. Real Time Hand Gesture Recognition & its Applications in Assistive Technologies for Disabled. In: 2018 Fourth International Conference on Computing Communication Control and Automation (ICCUBEA). IEEE; 2018. pp. 1–6.
36.
AscariREdOSSilvaLPereiraR. Personalized Interactive Gesture Recognition Assistive Technology. Proceedings of the 18th Brazilian Symposium on Human Factors in Computing Systems. 2019; 1–12.
37.
KumarSRaiAAgarwalABachaniN. Vision based human interaction system for disabled. In: 2010 2nd International Conference on Image Processing Theory, Tools and Applications. IEEE; 2010. pp. 441–446.
38.
DimentLHobbsD. A gesture-based virtual art program for children with severe motor impairments – development and pilot study. Journal of Assistive, Rehabilitative & Therapeutic Technologies. 2014; 2(1): 23206.
39.
MohammadUAnasM. Design of a low cost DIY moving wheel chair using ATmega1284P based on retina movement for the persons disabled with quadriplegia. In: 2015 Annual IEEE India Conference (INDICON). IEEE; 2015. pp. 1–4.
40.
SambrekarURamdasiD. Human computer interaction for disabled using eye motion tracking. In: 2015 International Conference on Information Processing (ICIP). IEEE; 2015. pp. 745–750.
41.
LambKMadheS. Automatic bed position control based on hand gesture recognition for disabled patients. In: 2016 International Conference on Automatic Control and Dynamic Optimization Techniques (ICACDOT). IEEE; 2016. pp. 148–153.
42.
ChattorajSVishwakarmaKPaulT. Assistive system for physically disabled people using gesture recognition. In: 2017 IEEE 2nd International Conference on Signal and Image Processing (ICSIP). IEEE; 2017. pp. 60–65.
43.
MartinsPRodriguesHRochaTFranciscoMMorgadoL. Accessible options for deaf people in e-learning platforms: technology solutions for sign language translation. Procedia Computer Science. 2015; 67: 263–272.
44.
TavaresCSMAntheroFScozM. Analysis of Methods for Evaluation of Assistive Technologies Focused on Computational Access of People with Cerebral Palsy. In: Congress of the International Ergonomics Association. Springer; 2018. pp. 1411–1419.
45.
Manresa-YeeCVaronaJPeralesFJSalinasI. Design recommendations for camera-based head-controlled interfaces that replace the mouse for motion-impaired users. Universal Access in the Information Society. 2014; 13(4): 471–482.
46.
PaulyLSankarD. Non intrusive eye blink detection from low resolution images using HOG-SVM classifier. International Journal of Image, Graphics & Signal Processing. 2016; 8(10).
47.
GraumanKBetkeMLombardiJGipsJBradskiGR. Communication via eye blinks and eyebrow raises: video-based human-computer interfaces. Universal Access in the Information Society. 2003; 2(4): 359–373.
48.
ChauMBetkeM. Real time eye tracking and blink detection with usb cameras. Boston University Computer Science Department; 2005.
49.
XuGWangYZhangX. Human computer interaction for the disabled with upper limbs amputation. In: 2010 2nd International Conference on Advanced Computer Control. IEEE; Vol. 3, 2010. pp. 120–123.
50.
ParmarKMehtaBSawantR. Facial-feature based Human-Computer Interface for disabled people. In: 2012 International Conference on Communication, Information & Computing Technology (ICCICT). IEEE; 2012. pp. 1–5.
51.
UtaminingrumFFauziMASariYAPrimaswaraRAdinugrohoS. Eye Movement as Navigator for Disabled Person. In: Proceedings of the 2016 International Conference on Communication and Information Systems; 2016. pp. 1–5.
52.
NakazawaNAikawaSMatsuiT. Development of Communication Aid Device for Disabled Persons Using Corneal Surface Reflection Image. In: Proceedings of the 2nd International Conference on Graphics and Signal Processing; 2018. pp. 16–20.
53.
SharmaSVarkeyBAcharyKHakulinenJTurunenMHeimonenT, et al. Designing gesture-based applications for individuals with developmental disabilities: guidelines from user studies in India. ACM Transactions on Accessible Computing (TACCESS). 2018; 11(1): 1–27.
54.
KrishnamurthiRPatanRGandomiAH. Assistive pointer device for limb impaired people: a novel Frontier Point Method for hand movement recognition. Future Generation Computer Systems. 2019; 98: 650–659.
55.
GaoXShiLWangQ. The design of robotic wheelchair control system based on hand gesture control for the disabled. In: 2017 International Conference on Robotics and Automation Sciences (ICRAS). IEEE; 2017. pp. 30–34.
56.
ZhangJZhuangLWangYZhouYMengYHuaG. An egocentric vision based assistive co-robot. In: 2013 IEEE 13th International Conference on Rehabilitation Robotics (ICORR). IEEE; 2013. pp. 1–7.
57.
ZhaoQTuDXuSShaoHMengQ. Natural humanrobot interaction for elderly and disabled healthcare application. In: 2014 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). IEEE; 2014. pp. 39–44.
ChattorajSVishwakarmaKPaulT. Assistive system for physically disabled people using gesture recognition. In: 2017 IEEE 2nd International Conference on Signal and Image Processing (ICSIP). IEEE; 2017. pp. 60–65.
60.
MartinsPRodriguesHRochaTFranciscoMMorgadoL. Accessible options for deaf people in e-learning platforms: technology solutions for sign language translation. Procedia Computer Science. 2015; 67: 263–272.
61.
TavaresCSMAntheroFScozM. Analysis of Methods for Evaluation of Assistive Technologies Focused on Computational Access of People with Cerebral Palsy. In: Congress of the International Ergonomics Association. Springer; 2018. pp. 1411–1419.
62.
Manresa-YeeCVaronaJPeralesFJSalinasI. Design recommendations for camera-based head-controlled interfaces that replace the mouse for motion-impaired users. Universal Access in the Information Society. 2014; 13(4): 471–482.
63.
PaulyLSankarD. Non intrusive eye blink detection from low resolution images using HOG-SVM classifier. International Journal of Image, Graphics & Signal Processing. 2016; 8(10).
64.
RosenbergDScottK. Use case driven object modeling with UML. Springer; 1999.
65.
KintschADePaulaR. A framework for the adoption of assistive technology. SWAAAC 2002: Supporting learning through assistive technology. 2002; 1–10.
66.
BhattacharyaSSamantaDBasuA. Performance models for automatic evaluation of virtual scanning keyboards. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 2008; 16(5): 510–519.
67.
HankeHKneesD. A phase-field damage model based on evolving microstructure. Asymptotic Analysis. 2017; 101: 149–180.
68.
LefeverE. A hybrid approach to domainindependent taxonomy learning. Applied Ontology. 2016; 11(3): 255–278.
69.
MeltzerPSKallioniemiATrentJM. Chromosome alterations in human solid tumors. In: VogelsteinBKinzlerKW, eds. The Genetic Basis of Human Cancer. New York: McGraw-Hill; 2002. pp. 93–113.
70.
MurrayPRRosenthalKSKobayashiGSPfallerMA. Medical Microbiology. 4th ed. St. Louis: Mosby; 2002.
71.
WilsonE. Active vibration analysis of thin-walled beams. University of Virginia; 1991.
72.
LalanneDNigayLRobinsonPVanderdoncktJLadryJF, et al. Fusion engines for multimodal input: a survey. In: Proceedings of the 2009 International Conference on Multimodal Interfaces. ACM; 2009. pp. 153–160.
73.
NerisanuRNerisanuRAManiuINeamtuB. Cerebral palsy and eye-gaze technology. Interaction, perspective and usability. A review. Acta Medica Transilvanica. 2017; 22(4).
74.
AnastasiouDJianCZhekovaD. Speech and gesture interaction in an ambient assisted living lab. In: Proceedings of the 1st Workshop on Speech and Multimodal Interaction in Assistive Environments. Association for Computational Linguistics; 2012. pp. 18–27.
75.
AnastasiouDJianCStahlC. A German-Chinese speech-gesture behavioural corpus of device control in a smart home. In: Proceedings of the 6th International Conference on PErvasive Technologies Related to Assistive Environments; 2013. pp. 1–6.
76.
WangKJTungHWHuangZThakurPMaoZHYouMX. EXGbuds: universal wearable assistive device for disabled people to interact with the environment seamlessly. In: Companion of the 2018 ACM/IEEE International Conference on Human-Robot Interaction; 2018. pp. 369–370.
77.
NerisanuRNerisanuRAManiuINeamtuB. Cerebral palsy and eye-gaze technology. Interaction, perspective and usability. A review. Acta Medica Transilvanica. 2017; 22(4).
78.
HemmingssonHAhlstenGWandinHRytterströmPBorgestigM. Eye-gaze control technology as early intervention for a non-verbal young child with high spinal cord injury: a case report. Technologies. 2018; 6(1): 12.
