Abstract
Abstract
Objective:
The aim of this article is to describe a game engine that has all the characteristics needed to support rehabilitation at home. The low-cost tracking devices recently introduced in the entertainment market allow measuring reliably at home, in real time, players' motion with a hands-free approach. Such systems have also become a source of inspiration for researchers working in rehabilitation. Computer games appear suited to guide rehabilitation because of their ability to engage the users. However, commercial videogames and game engines lack the peculiar functionalities required in rehabilitation: Games should be adapted to each patient's functional status, and monitoring the patient's motion is mandatory to avoid maladaptation. Feedback on performance and progression of the exercises should be provided. Lastly, several tracking devices should be considered, according to the patient's pathology and rehabilitation aims.
Subjects and Methods:
We have analyzed the needs of the clinicians and of the patients associated in performing rehabilitation at home, identifying the characteristics that the game engine should have.
Results:
The result of this analysis has led us to develop the Intelligent Game Engine for Rehabilitation (IGER) system, which combines the principles upon which commercial games are designed with the needs of rehabilitation. IGER is heavily based on computational intelligence: Adaptation of the difficulty level of the exercise is carried out through a Bayesian framework from the observation of the patient's success rate. Monitoring is implemented in fuzzy systems and based on rules defined for the exercises by clinicians. Several devices can be attached to IGER through an input abstraction layer, like the Nintendo® (Kyoto, Japan) Wii™ Balance Board™, the Microsoft® (Redmond, WA) Kinect, the Falcon from Novint Technologies (Albuquerque, NM), or the Tyromotion (Graz, Austria) Timo® plate balance board. IGER is complemented with videogames embedded in a specific taxonomy developed to support rehabilitation progression through time.
Conclusions:
A few games aimed at postural rehabilitation have been designed and developed to test the functionalities of the IGER system. The preliminary results of tests on normal elderly people and patients with the supervision of clinicians have shown that the IGER system indeed does feature the characteristics required to support rehabilitation at home and that it is ready for clinical pilot testing at patients' homes.
Introduction
However, it was soon recognized that the videogames developed for the entertainment market are not adequate for the pace and the goals of rehabilitation8–10 : Their fast interaction, which can be barely matched to the patient's residual functional abilities, and the wealth of targets and distractors make usability low and may produce strain and anxiety. Thus, one hurdle facing the successful use of exercise videogames (or exergames) in older adults and functionally impaired population is that many off-the-shelf videogames are too complex for use by these groups. 8 Videogames must, therefore, be developed to take into consideration the cognitive and physical limitations of the population(s) for which they are intended. This is the reason why games explicitly dedicated to rehabilitation have been recently developed.
In addition, in the REWIRE 4 project, the need for developing game engines of a new generation, specifically targeted for rehabilitation purposes, has been clearly identified. Such engines should have a twofold goal: The first, common to all game engines, is to provide all functionalities for the game play, and the second, associated with their therapeutic role, is to provide real-time monitoring and advice to the patient. Rehabilitation game engines should (1) adapt the difficulty and the game play to the patients' abilities to avoid frustration and, most importantly, (2) should monitor maladaptation and wrong postures, as these would make rehabilitation more harmful than effective. The game engine should also be able (3) to give an adequate real-time feedback to the patient while exercising.
Another open problem when applying games in rehabilitation is the definition of a rehabilitation schedule based on important training principles (e.g., objective progression in the therapy, with an increasing level of difficulty mapped onto the rehabilitation plan and goals).
In the following, we discuss how these issues have been addressed inside the Rewire platform. In particular, we illustrate the Intelligent Game Engine for Rehabilitation (IGER) system, developed specifically to support rehabilitation through games, and how this can be integrated with the definition of an adequate taxonomy for rehabilitation progression through time inspired by the work of Gentile. 11
Materials and Methods
The development of an effective game engine for rehabilitation has to be based on inputs from clinicians and patients. Several meetings have been organized to elicit such specifications. The key issue is that rehabilitation games cannot work as stand-alone applications but must be included into a broader structure involving patients, therapists, clinicians, hospitals, and institutions at the regional/national level. This is specifically the approach pursued inside the REWIRE project, recently funded by the European Commission. The REWIRE platform implements such a broad perspective by integrating three main hierarchical components: A hospital station (HS), a networking station, and a patient station (PS), under the assumption that such a structure enables effective support of at-home rehabilitation.
The HS is used by clinicians to define and schedule the rehabilitation at home. It also monitors the patient's progression remotely and supports a virtual community of patients and clinicians that helps, educates, and motivates the patients. The network station is installed at the health provider site, at a regional level. It provides advanced data mining functionalities to discover patterns in rehabilitation treatments among hospitals and regions. The PS is installed at patients' homes and has at its core the IGER (Fig. 1), which guides patients through their actual rehabilitation schedule using engaging and targeted videogames and monitors patients' movements and their correct execution of the exercises.
The Intelligent Game Engine for Rehabilitation (IGER) and its game and control modules. The connections with the patient and the therapist at the hospital and the virtual therapist, which provides real-time feedback to the patient, are highlighted. Color images available online at www.liebertonline.com/g4h
An additional hospital communication module of the station allows patients to interact with the clinicians at the hospital and to download the rehabilitation program and the configuration of the associated games. REWIRE contains also a lifestyle module that collects data on the patient's daily activity through a body sensor network that is used, along with environmental and physiological data, to tune the games' difficulty level, assess potential risks, and advise clinicians on the therapy.
Finally, a virtual community module, managed by the hospital, acts as a client of the patients' community and allows patients to interact with a peer group.
Rehabilitation scheduling
Rehabilitation exercises are designed by therapists with specific goals, for example, (1) to increase or maintain mobility of the joints and surrounding soft tissues, (2) to develop coordination through control of individual muscles, (3) to increase muscular strength and endurance, or (4) to promote relaxation and relief of tension. 12
For each exercise, therapists define the correct way to perform it, and, in one-on-one sessions, they check that patients perform only correct movements and maintain correct postures so as to avoid maladaptation and, therefore, inadequate rehabilitation.
The variety of exercises is quite large and depends on the therapists' idiosyncrasies. Thus, it is difficult to map the therapists' rehabilitation exercises, their goals, and the mandatory posture/movement constraints to their implementation as videogames.
In our work, we guide the mapping between therapy and engaging videogames by using Gentile's taxonomy of motor skills, 11 which identifies high-level features of rehabilitation routines and organizes them in a hierarchical structure (Fig. 2).
Gentile's taxonomy, 11 with a possible mapping of exercises and a possible progression for balance and posture rehabilitation. The arrow indicates a possible progression of the patient and the possible mappings of the “Fruit Catcher” game described hereafter. Color images available online at www.liebertonline.com/g4h
Gentile's taxonomy 11 is a two-dimensional matrix that contains exercises with increasing complexity of the addressed functionality (from body stability to body movement) in its columns. In its rows are the conditions under which the exercises are executed, from simple static environments to time-varying situations with inter-trial variability.
With this taxonomy, it becomes easier to design and configure games for rehabilitation. In fact, any exercise for motor skill rehabilitation (proposed by any therapist) can be classified in one element of the taxonomy according to a progression in recovering functionality. At the same time, rehabilitative videogames can be developed by targeting one element of the taxonomy rather than a specific exercise proposed by a specific therapist, thus broadening the number of therapists and patients who might use it.
The IGER game engine
The IGER is the core component of the PS. It comprehends a game engine and a game control unit. The former provides all the basic gaming functionalities (input data, animation, collision detection, rendering, and game logic); the latter controls the game, and it has been developed to match the needs of games for rehabilitation:
1. It schedules the games, chosen and configured according to the framework shown in Figure 2. 2. It adapts the game difficulty level to the actual patient's performance capacity, so that an adequate challenge level is maintained. 3. It supervises the gaming sessions and monitors whether or if the patient's movements comply with the specifications set by the therapist. 4. It displays a virtual therapist (VT) to advise and provide feedback to the patient on the therapy.
Games defined for rehabilitation have to be parametric. Specifically, games should be defined by parameters that can be regulated and adapted, depending on the level of game difficulty. Such parameters are initialized inside the hospital, where the clinician prescribes the therapy and are continuously adapted to each patient's progression. Such parameters can be, for instance, either the frequency of the obstacles and of the targets inside the game or the range of movements required to complete a game successfully. The control unit implements a real-time adaptation of the parameters through a Bayesian model 10 : The patient's performance is monitored, and the success rate is computed online. The parameters are then increased or decreased such that a certain success rate is maintained, where the amount of change is provided by the model.
The patient's movement is also used for monitoring. To this aim, a set of rules of correct execution is defined inside the HS. Such rules represent a knowledge base from the therapist's experience and are coded into logical propositions like, for instance, “do not bend the trunk” or “keep your feet slightly apart.” Such rules are then fuzzyfied, defining a maximum range allowed for correct execution (e.g., maximum trunk bending allowed in 20°) and associating automatically intermediate ranges with the values in between (Fig. 3). A visual interface, showing an avatar inside the game environment, helps the therapists in defining the constraints on the patient's motion.
The graphical interface used to define the motion constraint inside the hospital station. The automatic fuzzyfication of a parameter on which a rule has been defined is shown. Color images available online at www.liebertonline.com/g4h
At run-time the knowledge-based monitor inside the control unit (Fig. 1) compares the movement of all the body segments with the rules defined inside the HS by the clinicians and raises an alarm level that can be graded: Risky situation, bad movement, or wrong movement. If more than one constraint is violated, the most severe alarm level is raised, implicitly combining the fuzzy rules with the or logical operator.
The monitor supervises also the online adaptation process. Depending on the severity of the alarm raised, the monitor can stop increasing the game difficulty until the patient achieves a correct way of playing.
Once the alarm has been raised, the control unit has to give an adequate feedback to the patient. Inside the IGER we have implemented several methods to provide such feedback. The first method consists of displaying a visual feedback in the form of a text or an icon (e.g., a smiley face) and an audio feedback in the form of a warning sound (Fig. 4d). The second method consists in showing a VT that can warn the patient that he or she is not doing the exercise correctly (Fig. 4a). This is an avatar therapist; her face is animated through morphing expressions that are combined randomly to give her a variety of expressions, occasionally tilting the head or blinking the eyes. The three-dimensional model features also simplified lip syncing for a more realistic voice feedback, allowing playing back standard recorded audio messages while the lips of the three-dimensional model move reasonably. The second form of an avatar is a three-dimensional mascot (Fig. 4b), similar to what Nintendo does with its Wii Balance Board avatar in the “WiiFit” games. The third form of feedback is a video of a real therapist (Fig. 4c), recorded and played back when needed.
The feedback implemented inside the Intelligent Game Engine for Rehabilitation:
In all cases, when the alarm rises to the wrong level, the game is paused, and the avatar inside the game shows the correct posture and movement superimposed on the patient's last movement on the screen to the patient.
Principles of game design
We have built through our framework a set of minigames, each designed according to the goals and the requirements of the underlying exercises and on the good game design practice13–15 : Meaningful play, flow theory, and sense of presence have been all incorporated into the design.
Meaningful play states that each game action must have a direct and clearly distinguishable feedback as well as a reasonably lasting effect. This helps the patient in understanding what he or she can and cannot do. It was achieved here through a clear video–audio feedback of the success or failure of a patient's action, represented respectively with a “√” green symbol displayed on the screen and a nice beep and a “X” red symbol and an annoying beep.
The second basic principle is the theory of flow, which states that when the skills of the user are matched by the level of challenge posed by the game, the user enters a state of complete focus and immersion in which he or she loses track of time. 14 The patient's appropriate level of exercise difficulty is achieved inside IGER through an adaptation mechanism based on a Bayesian framework. 10 Scenarios and position of targets and distractors of each game are randomized, making the patient feel challenged by always different situations. Music matched to the game scenarios is played during the rehabilitation sessions. The scoring system reflects the rehabilitation nature of the games8–10,16: No negative points is given, or no “death” occurs, and an actual score value reflects the accuracy and/or speed of execution of the movement. Former scores are shown along with the actual score to demonstrate improvement over time. All these elements contribute to create a flow experience that, in turn, contributes in focusing on the game. This hides the burden of therapeutic repetitive tasks and the difficulties arising from impairments, under the entertaining experience of a game. This is even more important when we consider that post-stroke patients often fall into depression. 17
The sense of presence is another strong point associated with IGER. As the avatar follows the patient's movements with no appreciable delays, the patient can feel that the avatar represents him- or herself inside the virtual environment, and therefore the patient has a strong perception of being the actor in the game. 18 This is associated with the use of “hands-free” tracking of motion, which does not require attaching any device to the patient, allowing the most natural interfacing with the games.
Games for rehabilitation
In our work, we aim to create a set of minigames built on clinically valid exercises and comprehending both monitoring and adaptability. We designed and realized several game concepts that address posture and balance rehabilitation. These games can be mapped on a set of exercises defined through Gentile's taxonomy 11 and share the same theme (in our case, the farm theme) to provide a continuity in gaming during the patient's rehabilitation process. To exemplify, we present here two of these minigames: “Animal Feeder” and “Fruit Catcher” (Fig. 5), which highlight the flexibility of our approach and the monitoring and adaptation capabilities.
Two minigames of the patient station:
The “Animal Feeder” minigame aims to train patients' balance by implementing a dual task. The exercise requires the patient to kneel in front of the display and to move the impaired arm to touch different targets, which are represented by three hungry cows that have to be fed. The cows keep requesting food by mooing while opening their mouths (Fig. 5a). The player controls a virtual hand in first-person view. He or she has first to collect some hay and then feed it to one of the hungry cows; when the cow has been fed, the player's score increases. If a cow remains hungry for too long, it groans, and the player's score decreases. In addition, a pitchfork positioned to the left of the player must be kept upright by the player with his or her other hand. If the player fails to do so, the pitchfork breaks, and the player's score decreases. “Animal Feeder” can be currently played with different devices. In particular, here the player's hand was tracked through a Kinect camera positioned above the display, while the pitchfork was controlled by Novint Technologies (Albuquerque, NM) Falcon® haptic device, which produces an elastic force to the patient's hand proportional to the hand displacement. Note that devices not used for game play can still be used for additional monitoring of the patient. For instance, Figure 5b shows how Nintendo's Balance Board is used to monitor the position of the foot center of mass while performing a game. In this case, the oscillations do not affect the game play, but provide useful information in terms of a patient's balance capability.
The “Fruit Catcher” game (Fig. 5b) is built on two different exercises. For the first exercise, the patient is required to shift his or her upper body to the left and to the right side, while keeping the feet still on the ground. For the second exercise, the patient must move laterally inside the minigame. In the basic concept of “Fruit Catcher,” the player must catch fruits falling from the top of a tree. The player avatar stands below the tree with a virtual basket on its head and can move the body laterally to catch the fruits with the basket. The player's score increases when a fruit falls into the basket. The fruits fall from different heights and from different positions on the horizontal axis and at a different frequency. Like the “Animal Feeder” game, it can be played with different devices (e.g., the Nintendo Wii Balance Board or the Microsoft Kinect). The choice of the input device, in this case, depends also on the exercise rationale, showing an example of separation between game and exercise. When using the Nintendo Wii Balance Board, the player is constrained to keep the feet on the board. Thus, only the first exercise can be done this way. When using the Microsoft Kinect, instead, the player can move freely around the play area. In this latter case, the exercise can be either of the two described above. In Figure 2a, we can see where the two exercises are mapped over Gentile's taxonomy. 11 A possible progression inside the taxonomy is shown in Figure 2: The patient starts with an exercise classified as 1A, which belongs to the lowest classification, defining an easy exercise. In this case, using the “Fruit Catcher” game as an example, the patient may only have to stand still and not to move in order to not disturb the tree; otherwise, its fruits would fall down. As the patient progresses and improves strength and balance capability, the game may move toward the 1D box, adding both body transport (requiring the patient to move around) and object manipulation (adding the basket on the head and the requirement to catch fruits). Then, the patient may be presented with inter-trial variability, moving the classification to 2D. At last, the patient may be required to meet birds flying around the virtual area and disturbing his or her focus, thus moving the classification toward 4D.
To accommodate such a variety of tracking devices, we have inserted an Input Abstraction Layer inside the IGER. Our prototype currently supports the Sony (Tokyo, Japan) PlayStation® PS3™ Eye camera, the Microsoft Kinect camera and its microphone array, the Wii Balance Board, or the Tyromotion (Graz, Austria) Timo® plate and two haptic devices (the Omni® Phantom® [Sensable Technologies, Wilmington, MA] and the Novint Falcon). The input devices can be combined for additional control over game elements or for implementing dual tasks.
Discussion
Many authors have acknowledged the power of immersion associated with virtual reality and games in particular.19,20 This is referred to as presence; it would facilitate the rehabilitation sessions by providing a fun, interactive environment that challenges the skills of the player and immerses the player in a virtual world. Such an approach would limit the typical boredom and fatigue in traditional rehabilitation: The patient, while exercising, should feel like a player, focused on having fun while playing the game. To fully achieve this, games should be designed according to good games design principles to avoid coming up short in engaging the patients.
However, engaging games are not enough: Adaptation, monitoring, and real-time evaluation of the movements should be provided to the patient in real time. Such functionalities are not provided by commercial games, but are indeed incorporated inside the IGER system. In fact, (1) IGER allows configuring the game inside the HS, setting the parameters associated with the level of difficulty most adequate to the patient's functional status, and then it continuously adapts these parameters to the patient's performance to continuously challenge the patient at the proper level at any point during the rehabilitation process. (2) IGER allows the continuous monitoring of patients' actions and postures within the game engine, and it aims to enforce a correct execution of the rehabilitation movements. (3) IGER integrates a real-time feedback to the patient of his or her performance and of possible wrong movements, according to the rules set by the therapists.
Monitoring and real-time feedback have been designed to substitute for, although to a limited degree, the therapist. In one-on-one sessions, these functions are provided by the human therapist, who cannot be present at home during rehabilitation. The monitoring role is achieved by the fuzzy monitor that implements the rule defined by the therapist him- or herself, mainly to avoid maladaptation. The rules operate in real time on the motion data and are downloaded from the HS at game configuration time.
The social aspects of rehabilitation are very important for patients and must be considered when designing a platform for home rehabilitation. For instance, in one-on-one sessions, the therapist is a major source of motivation for the patient, 21 but this component is missing during home rehabilitation. To avoid isolation, which diminishes the appeal of a rehabilitation session, IGER provides an informative evaluation to the patient, besides displaying a balanced score that reflects performance on the actual exercise, by means of a VT that, through speech, evaluates the patient's performance of the exercise.
Given the importance of the issue, we have evaluated several feedback modalities. Icons (e.g., a smiley face) and sounds are especially useful because they provide immediate and easily understood feedback to the patient. Moreover, visual icons and sounds have a more immediate effect than the VT; they are less intrusive and less annoying when repeated often. For this reason, we use them for the simple and more frequent warnings.
The VT, because of its resemblance to a real therapist, allows the patient to feel the presence of a more serious character supervising his or her activity. The mascot, on the other hand, can be beneficial as a motivator because of its funny aspect and movements, enhancing the playful aspect of the application. The video is useful because of its realism and the presence of a real person, who could be preferred by the patient for a greater degree of empathy with the therapist. This implies a somehow stereotyped video style, as the amount of variability in the video appearance is much less than with mascots or three-dimensional models. We believe that there is not a single best feedback, but each patient will have his or her own preference, depending on personal idiosyncrasies.
The VT and the mascot can also be displayed between two games (exercises) of the same session to introduce the exercises and accompany the patient along his or her training period. In addition, they appear during the introduction of the game to explain its rules and during the course of the game, encouraging and motivating the players according to their performance. Lastly, they appear when the game is stopped because of wrong movements, explaining to the patient what went wrong and what the patient should do.
A proper game design and games specifically designed for therapeutic purposes are not sufficient. A complete rehabilitation program is required, and this can be defined only by a clinician. Besides the game engine, a clear definition of a rehabilitation path is required to support intensive prolonged rehabilitation processes. The therapist has still the fundamental role of the supervisor, although remotely from the hospital. The succession of games inside the rehabilitation program is customized by the therapist for each patient according to his or her impairments; the program is defined at the hospital, according to the rehabilitation goals set with the patient and the current position of the patient inside the taxonomy (Fig. 2). The therapist can also fully configure each game, thus tailoring the underlying exercise to the patient's health status. For instance, the therapist can define, for each exercise, what input device should be used, how long the exercise should last, how far the targets can travel, and how fast they can move. Finally, the therapist has an important role concerning the definition of the knowledge base on which the rules monitor a patient's movements and his or her constraints are built upon. A personalized rehabilitation program can therefore be obtained.
Conclusions
This work is the result of a tight collaboration among game developers, human movement scientists, and physical therapists and aims at building engaging games that implement therapeutic exercises targeted to at-home rehabilitation. We designed the IGER platform to take up some of the therapist's functions during home rehabilitation. Although completely replacing the therapist's skill is beyond reach, the IGER system can make rehabilitation at home a viable option, especially if intertwined with periodic rehabilitation sessions at the reference hospital with the reference clinicians maintaining, therefore, a personal relationship that is presumably fundamental for prolonged rehabilitation.
