An exploratory case study on letter-based,head-movement-driven communication

Abstract

BACKGROUND:

With alternative and augmentative communication (AAC) people with complex communication needs (CCN) become more independent and express themselves to the fullest extent possible. In finding the best AAC solution, mobile technology and ICT (information and communications technology) provide new opportunities every day. Although a wide range of assistive technologies (AT) are available, matching person and technology (MPT) and setting the optimal parameters individually are essential. For an AAC solution to be optimal for letter-based communication it has to be easy-to-use, comfortable, and fast.

OBJECTIVES:

For people with severe speech and physical impairments (SSPI), one method to interact with a computer is using head-movement-driven mouse. There are different on-screen devices available for typing via head movements, and much work has been done to compare them in terms of the time required for typing. Dasher is one of the fastest software tools with a setting option for zooming speed. An optimistic initial model (OIM) based on Markov decision process (MDP) has already been shown to optimize this zooming speed for increasing the typing efficiency of persons without SSPI. Since this reinforcement learning component has so far been tested on neurotypical users only (e.g., research assistants), in the present case study we involved a user with SSPI. Our question was whether the algorithm can optimize its own parameters in these circumstances.

METHODS:

To document all relevant aspects of the human-computer interaction log files, screen and webcam videos were collected. These input data were later analyzed with mathematical methods based on the OIM reward systems feedbacks. In addition, manual interpretation using semi-supervised machine video annotation was carried out for analyzing screen events and user behaviors.

RESULTS:

The human annotations of the recorded video data indicated that the participant had at least two different typing strategies. In contrast with the data from a previous study, in our study the artificial intelligence (AI) component was unable to find optimal settings similar to those attained when only one typing strategy was used by subjects without SSPI.

CONCLUSIONS:

To maximize communication efficiency, a more complex assistive tool may be more appropriate. Closer cooperation between different areas of expertise is suggested in order to achieve solutions employing various methods.

Keywords

Augmentative and alternative communication (AAC)assistive technology (AT)head movement driven typing self-improvement matching person and technology (MPT)

Get full access to this article

View all access options for this article.

References

American Speech-Language-Hearing Association: Roles and responsibilities of speech-language pathologists with respect to augmentative and alternative communication: Technical report, 2004.

Light,

Gulens,

Rebuilding communicative competence and self-determination, Augmentative and Alternative Communication for Adults with Acquired Neurologic Disorders 2000. 137-179.

Light,

McNaughton,

Communicative competence for individuals who require augmentative and alternative communication: A new definition for a new era of communication?

Augmentative and Alternative Communication. 2014; 30(1): 1-18.

Oklahoma funding for assistive technology. Available from: https://www.ok.gov/abletech/documents/Tech%20Act-Individuals%20with%20Disabilities.pdf.

Fried-Oken,

Beukelman,

DR.

Hux,

Current and future AAC research considerations for adults with acquired cognitive and communication impairments.

Assistive Technology. 2012; 24(1): 56-66.

Scherer,

M.J.

Outcomes of assistive technology use on quality of life. Disability & Rehabilitation. 1996; 18(9): 439-448.

McNaughton,

Light,

The iPad and mobile technology revolution: Benefits and challenges for individuals who require augmentative and alternative communication.

Augmentative and Alternative Communication. 2013; 29(2): 107-116.

Chantry,

Dunford,

How do computer assistive technologies enhance participation in childhood occupations for children with multiple and complex disabilities? A review of the current literature.

The British Journal of Occupational Therapy. 2010; 73(8): 351-365.

Von Tetzchner,

Martinsen,

Introduction to sign teaching and the use of communication aids. Whurr Publisher Ltd., London. 1992, pp. 52-55.

10.

Scherer,

M.J.

, et al. Predictors of assistive technology use: The importance of personal and psychosocial factors. Disability and Rehabilitation. 2005; 27(21): 1321-1331.

11.

Scherer,

M.J.

Craddock,

Matching person & technology (MPT) assessment process.

Technology and Disability. 2002; 14(3): 125-131.

12.

Copestake,

Augmented and alternative NLP techniques for augmentative and alternative communication. In: Proceedings of the ACL workshop on Natural Language Processing for Communication Aids. 1997.

13.

Hansen,

J.P.

et al. Gaze typing compared with input by head and hand. In: Proceedings of the 2004; symposium on Eye tracking research & applications. ACM. 2004.

14.

Foulds,

. Communication rates of nonspeech expression as a function in manual tasks and linguistic constraints. 1980. 83-87.

15.

Beukelman,

Pat,

Augmentative and alternative communication: Supporting children and adults with complex communication needs, 2005.

16.

Hoag,

L.A.

et al. Trade-offs between informativeness and speed of message delivery in augmentative and alternative communication. Journal of Speech, Language, and Hearing. 2004; 47(6): 1270-1285.

17.

Lőrincz,

Takács,

AGI architecture measures human parameters and optimizes human performance.

Artificial General Intelligence. 2011; 321-326.

18.

Wills,

S.A.

MacKay,

D.J.C.

DASHER – an efficient writing system for brain-computer interfaces.

Neural Systems and Rehabilitation Engineering. 2006; 14(2): 244-246.

19.

Hévízi,

Gy.

et al. Hidden Markov model finds behavioral patterns of users working with a headmouse driven writing tool. Neural Networks, 2004.

20.

Ward,

D.J.

MacKay,

D.J.C.

Fast hands-free writing by gaze direction. arXiv preprint cs/0204030, 2002.

21.

Dasher Project. Available from: http://wwwinference.phy.cam.ac.uk/dasher/DasherSummary2.html.

22.

Szita,

Takács,

Lőrincz,

Epsilon-MDPs: Learning in varying environments.

J. Mach. Learn. 2003; 3: 145-174.

23.

Szita,

Lőrincz,

The many faces of optimism: a unifying approach. In: 25th Int. Conf. on Mach. Learn. Omnipress, Helsinki, 2008; 1048-1055.

24.

Molnár,

Gy.

Lőrincz,

Pásztor,

Csapó,

Internet-based development of thinking skills in young schoolchildren. Paper to be presented at the 16th European Conference for Research on Learning and Instruction. Limassol, Cyprus, August 25-29, 2015; 833.

25.

Jeni,

Cohn,

J.F.

De La Torre,

Facing Imbalanced Data-Recommendations for the Use of Performance Metrics.

Affective Computing and Intelligent Interaction (ACII), 2013 Humaine Association Conference on. IEEE, 2013.

26.

Sunghyun,

et al. Crowdsourcing micro-level multimedia annotations: The challenges of evaluation and interface. In: Proceedings of the ACM multimedia 2012 workshop on Crowdsourcing for multimedia. ACM, 2012.

27.

Fabian,

H.C.

Niebles,

J.C.

Collecting and annotating human activities in web videos. In: Proceedings of International Conference on Multimedia Retrieval. ACM, 2014.

28.

Gireeja,

Varshney,

L.R.

To crowdsource or not to crowdsource. Workshops at the Twenty-Sixth AAAI Conference on Artificial Intelligence, 2012.

29.

Eickhoff,

. Crowd-powered experts: Helping surgeons interpret breast cancer images. In: Proceedings of the First International Workshop on Gamification for Information Retrieval. ACM, 2014.

30.

Vondrick,

Patterson,

Ramanan,

Efficiently scaling up crowdsourced video annotation.

International Journal of Computer Vision. 2013; 101(1): 184-204.

31.

et al. LabelMovie: Semi-supervised machine annotation tool with quality assurance and crowd-sourcing options for videos. Content-Based Multimedia Indexing (CBMI), 2014.

32.

Cohen,

. Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological Bulletin. 1968; 70(4): 213.

33.

Free Virtual Keyboard. Available at: http://freevirtualkeyboard.com/.

34.

Sandberg,

A.D.

Hjelmquist,

A comparative, descriptive study of reading and writing skills among non-speaking children: a preliminary study.

International Journal of Language & Communication Disorders. 1996; 31(3): 289-308.

35.

Jeni,

, et al. Spatio-temporal event classification using time-series kernel based structured sparsity. Computer Vision-ECCV 2014. Springer International Publishing. 2014, pp. 135-150.