This paper presents our proposed multimodal system that is composed of fusion and fission engines that take into account the contextual information of the user. The fusion engine combines various input modalities to determine the overall context of a given situation. It yields the situation that needs action. This action is subdivided into smaller tasks that are sent to actuators, gadgets and other output modalities for implementation. The process is handled by the fission engine. Our goal is to build a system that helps people with disabilities interact with their ambient intelligent environment using their natural communication skills, such as speech and gesture. Both the fusion and fission engines rely on ontology which is the knowledge base of our system while taking into consideration contextual semantic information. The system design is validated through case simulation using formal specifications using colored Petri nets. This work is our contribution to the ongoing research on robotic application to render services to the handicapped and the elderly.
O.Adjali, M.D.Hina, S.Dourlens and A.Ramdane-Cherif, Multimodal fusion, fission and virtual reality simulation for an ambient robotic intelligence, Procedia Computer Science52 (2015), 218–225. doi:10.1016/j.procs.2015.05.060.
2.
É.André, M.M.Benmoussa and C.Choppy, Formalising concurrent UML state machines using coloured Petri nets, Formal Aspects of Computing28(5) (2016), 805–845.
3.
R.A.Bolt, “Put-that-there”. Voice and gesture at the graphics interface, ACM SIGGRAPH Computer Graphics14 (1980), 262–270. doi:10.1145/965105.807503.
4.
R.A.Brooks, A robust layered control system for a mobile robot, IEEE Journal of Robotics and Automation2(1) (1986), 14–23.
5.
J.Choi, S.K.Kim and A.Tsourdos, Verification of heterogeneous multi-agent system using MCMAS, International Journal of Systems Science46 (2013), 634–651.
6.
R.S.Cost, Y.Chen, T.Finin, Y.Labrou and Y.Peng, Using colored Petri nets for conversation modeling, in: Issues in Agent Communication, Lecture Notes in AI, Vol. 1916, Springer, 2000, pp. 178–192.
7.
D.Costa and C.Duarte, Adapting multimodal fission to user’s abilities, in: Universal Access in Human–Computer Interaction: Design for All and eInclusion, Lecture Notes in Computer Science, Vol. 6765, 2011, pp. 347–356.
8.
S.Dourlens and A.Ramdane-Cherif, Modeling & understanding environment using semantic agents, Journal of Artificial Intelligence and Soft Computing Research1(4) (2011), 301–314.
9.
S.Dourlens, A.Ramdane-Cherif and E.Monacelli, Multi levels semantic architecture for multimodal interaction, Journal Applied Intelligence38(4) (2013), 586–599. doi:10.1007/s10489-012-0387-3.
10.
C.Galindo, A.Cruz-Martin, J.L.Blanco, J.A.Fernández-Madrigal and J.Gonzalez, A multi-agent control architecture for a robotic wheelchair, Applied Bionics and Biomechanics3(3) (2006), 179–189. doi:10.1155/2006/612940.
11.
A.Iglesias, Pure Petri nets for software verification and validation of semantic web services in graphical worlds, International Journal of Future Generation Communication and Networking3(1) (2010), 33–46.
12.
K.Jensen, L.M.Kristensen and L.Wells, Coloured Petri nets and CPN tools for modelling and validation of concurrent systems, International Journal on Software Tools for Technology Transfer9(3) (2007), 213–254. doi:10.1007/s10009-007-0038-x.
13.
R.R.Murphy and R.C.Arkin, SFX: An architecture for action oriented sensor fusion, in: RSJ International Conference on Intelligent Robot and Systems, 1992, pp. 1079–1086.
14.
M.N.Nicolescu and M.J.Matarić, A hierarchical architecture for behaviour-based robots, in: The First International Joint Conference on Autonomous Agents and Multi-Agent Systems, Bologna, 2002, pp. 227–233.
15.
S.L.Oviatt, Advances in robust multimodal interface design, IEEE Computer Graphics and Applications23(5) (2003), 62–68.
16.
S.L.Oviatt, Multimodal interfaces, in: The Human–Computer Interaction Handbook: Fundamentals, Evolving Technologies and Emerging Applications, 2nd edn, CRC Press, 2008, pp. 286–304, Chapter 14.
17.
S.L.Oviatt and P.R.Cohen, Multimodal interfaces that process what comes naturally, Communications of the ACM43(3) (2000), 45–53. doi:10.1145/330534.330538.
18.
T.Perperis, T.Giannakopoulos, A.Makris, D.I.Kosmopoulos, S.Tsekeridou, S.J.Perantonis and S.Theodoridis, Multimodal and ontology-based fusion approaches of audio and visual processing for violence detection in movies, Expert Systems with Applications38 (2011), 14102–14116.
19.
P.Prodanov and A.Drygajlo, Bayesian networks based multi-modality fusion for error handling in human–robot dialogues under noisy conditions, Speech Communication45 (2005), 231–248. doi:10.1016/j.specom.2004.10.015.
20.
G.Pruvost, T.Heinroth, Y.Bellik and W.Minker, User interaction adaptation within ambient environments, in: Next Generation Intelligent Environments, 2011, pp. 153–194. doi:10.1007/978-1-4614-1299-1_5.
21.
A.S.Rao and M.P.Georgeff, BDI-agents: From theory to practice, in: Proceedings of the First International Conference on Multiagent Systems (ICMAS’95), San Francisco, 1995, pp. 312–319.
22.
C.Rousseau, Y.Bellik and F.Vernier, WWHT: Un modèle conceptuel pour la presentation multimodale d’information, in: Proceedings of the 17th International Conference on Francophone sur l’Interaction Homme-Machine, 2005. doi:10.1145/1148550.1148558.
23.
D.Schnelle-Walk, S.Radeck-Arneth and J.Striebinger, Multimodal dialog management in a smart home context with SCXML, in: Proceedings of the 2nd Workshop on Engineering Interactive Systems with SCXML, 2015, pp. 23–25. doi:10.13140/RG.2.1.3790.5120.
24.
B.Srinath Reddy and O.A.Basir, Concept-based evidential reasoning for multimodal fusion in human–computer interaction, Applied Soft Computing10 (2010), 567–577. doi:10.1016/j.asoc.2009.08.026.
25.
The Arlyn ChairBot, http://www.osbornej.com/ChairBotHome.html, retrieved February 7, 2016.
26.
M.Wooldridge, Reasoning About Rational Agents, MIT Press2000. ISBN 0-262-23213-8.
27.
A.Zaguia, A.Wahbi, M.Miraoui, C.Tadj and A.Ramdane-Cherif, Modeling rules fission and modality selection using ontology, Journal of Software Engineering and Applications6(7) (2013), 354–371. doi:10.4236/jsea.2013.67045.
