BCI Sensor Based Environment Changing System for Immersion of 3D Game

Abstract

We present a methodology about game environment system based on BCI (brain computer interface) for immersion of FPS game play. FPS game is a video game genre centered on gun shooting and projectile weapon-based combat through a first-person perspective; immersion is important factor in this game. FPS game designer makes efforts to raise player's immersion using interface, graphic effect, sound, and so forth. We implemented real-time game environment system using game player's emotional state information from the BCI sensor to raise the degree of immersion in FPS game. We measured user's EEG signal using MindSet to take user's emotional state on playing game. We can get user's emotional state in 4 categories (attention/inattention, meditation/uneasiness) and this emotional state interacts with game environment system real time. In game environment space, each player experiences different environmental situation (lighting, fog) because each player has emotional state on each scene. We implemented game environment system to be changed (lighting, fog) on playing FPS game. Player can feel the change of game environment by our system. We found out that game environment system can be applied to various game genres to raise player's immersion on playing game.

1. Introduction

When computer games interact with game player, we can measure the neurophysiological information such as blood pressure, skin conductance, and brain wave signal. Monitoring such affective biofeedback signal can be useful to improve the gaming experience [1]. FPS (first-person shooters are a type of three-dimensional shooter game [2], featuring a first-person point of view with which the player sees the action through the eyes of the player character) game is a video game genre centered on gun shooting and projectile weapon-based combat through a first-person perspective. Immersion is important factor in this game genre. Figure 1 is a capture image of Korean popular FPS game, “Sudden Attack” (http://sa.nexon.com/main/index.aspx). FPS game developers try to raise the immersion on playing game using game graphics, sound effect, and so forth. In this paper, we present a methodology about game environment system based on BCI for immersion of FPS game using EEG measurement sensor device (MindSet) (http://www.neurosky.com/products/mindset.aspx). We used a MindSet device to measure player's emotional state. Game player wearing this device is enjoying the game that is changing sky to interact with player's emotional state. Game player is immersed in game through the change of sky environment. EEG is the recording of electrical activity along the scalp and measures voltage fluctuations resulting from ionic current flows within the neurons of the brain [3]. It provides a noninvasive means of reliably monitoring brain activity spatially and temporally. The EEG signal may be one of the most predictable and reliable physiological indicators to measure the level of alertness [4].

Figure 1

Game playing screen shot of FPS game “Sudden Attack.”

EEG was used for medical research at first. It has developed every year and broadened the scope of application. Currently, education, culture, life, and so forth are focused on EEG's application in more than medical purposes such as EEG-controlled game, attention training program using EEG, and wheelchair control. EEG's practical approach has been attempted at many fields [5]. In our system, game player can control the game sky environment consciously or unconsciously. Figure 2 shows process of implemented game environment system using EEG.

Figure 2

Process of game environment system to measure EEG.

The following section will describe the related work. Section 3 will implement the game environment system using EEG. In Section 4, we will discuss the results of our research.

2. Related Works

Our study uses commercially available dry EEG devices, MindSet. Usually, EEG related research uses commercially available dry EEG devices. The NeuroSky offers reliable low level EEG headsets and stable software tools for researchers and developers. Many researchers have succeeded in testing these single-channel devices in recent projects. In this section, we present EEG, MindSet, and game environment system. Human body condition is affected by environment. Also day or night can affect human body condition because human has biological clock. Game player's emotional state controls and changes the environment of game in the implemented system.

2.1. Biological Clock

Our research is related to human biological clock to feel day, night, and other environments. We express this concept using lighting and fog in this system. Biological clock affects the daily rhythm of many physiological processes. Although circadian rhythms tend to be synchronized with cycles of light and dark, other factors such as ambient temperature, meal times, stress, and exercise can influence the timing as well [6]. In humans, the average internal temperature is 37.0°C (98.6°F), though it varies among individuals. However, no person always has exactly the same temperature at every moment of the day. The lowest temperature occurs about two hours before the person normally wakes up. Additionally, temperatures change according to activities and external factors [7]. We applied these concepts into game environment system. Depending on player's concentration, the virtual world's day, night, and fog in game are determined with player's emotional state. When player concentrates into game, game environment lighting is sunshine sky or dark night; at feeling meditation and uneasiness game environment fog is thin or dense sky.

2.2. EEG

BCI is a computer interface related to technologies by brain wave. Recently, BCI research has been getting active. EEG means the flow of electricity that is formed when signal is transmitted between cranial nerves. In other words, EEG is called Electroencephalography. EEG-based technology has become more popular in serious games design and developments since new wireless headsets that meet consumer demand for wearability, price, portability, and ease-of-use. Originally, EEG-based technologies were used in neurofeedback games and brain-computer interfaces. Now, such technologies could be used in entertainment, e-learning, and new medical applications [8]. EEG is widely researched; for example, control of electronic devices using brain wave is used to manage movement. That is, this method is similar to using characteristics of human mental activities [9]. Human brain wave consists of 5 kinds of factor, δ (delta), θ (theta), α (alpha), β (beta), and γ (gamma). Brain wave signals are extracted from the EEG device. It is difficult to visualize and identify the result data. Thus, we measured the result data through the FFT analysis in real time. As a result, we can measure the player's emotional state. EEG is the electronic flow when delivering the signal between nervous system and a cranial nerve. When measuring EEG, the complicated form structure appears. This kind of data is “raw data”. It can't be extracted from brain wave as each frequency [10]. These raw data contain more static signal than extraction signal part. It has to be filtered at preconditioning process. After this process, raw data is normalized. It is also divided by the frequency and is different according to the emotional state of mind. The features of each frequency are shown in Table 1.

Table 1

Types and features of EEG.

Type	Frequency (Hz)	Normally
Delta	0.5–4 Hz	Hypnoidal
Theta	4–7 Hz	Slow wave sleep
Alpha	8–12 Hz	Stable wave
Beta	12–30 Hz	Action wave

In Table 1, there are delta waves (frequency 0.5–4 Hz), theta waves (frequency 4–7 Hz), alpha waves (frequency 8–13 Hz), beta waves (frequency 14–30 Hz), and gamma waves (frequency 30 Hz and over) [11]. Alpha waves are associated with meditation and relaxation; all types can also be attributed to attention or concentration at slower frequencies [12]. Beta waves are related to concentration. Alpha waves appear when human mind and body are stable. Alpha waves tend to be extracted a lot in stress-free mental state [13].

2.3. MindSet

Three main EEG-related application fields have been researched for several years with the development of wireless EEG devices; expanding the EEG applications out of the lab became possible as follows:

• (a)

BCI applications that help disabled people to communicate with machines [14, 15];

(b)

BCI applications for video games as game controllers [16];

(c)

neurofeedback games [17].

Standard medical EEG devices use a conductive gel to facilitate the reading of the signals. Dry active sensor technology does not need such gel. For this reason, headsets based on NeuroSky technology are low-cost and easy to handle [18]. The MindSet (Figure 3) wireless Bluetooth headset features brainwave-reading and mental-state-translational technology from NeuroSky, Inc., a Silicon Valley company. With earlier NeuroSky partner announcements in the toy (Uncle Milton Force Trainer, under a Lucas Licensing deal) and video gaming industries (Square Enix Judecca), the Toshiba-NeuroSky product launch represents the first BCI peripheral directed to mainstream PC users. The measurement hardware is Brain Wave Interface developed by NeuroSky in America. Attention and meditation are distributable by low price device. It is suitable for unprofessional or professional using of dry active sensor. Mindset analyzes and checks brain wave in a real time using Neurofeedback system and there is a technology which helps to change according to the brain wave needs [19]. We can extract player's emotion (attention/inattention, meditation/uneasiness) from this device.

Figure 3

MindSet.

Emotion recognition process by BCI sensors is Figure 4.

• (1)

Read brain waves and perform the FFT.

(2)

Based on the results of FFT, calculate each asymmetric value and SEF value.

(3)

Calculated value mapping and normalizing depending on attention/inattention, meditation/uneasiness.

(4)

Determine the emotion.

Figure 4

Emotion recognition by BCI sensors.

2.4. Emotion Recognition

Through EEG signal analysis we can get theta, alpha, and beta values. Based on developed algorithm, we can get player's emotion (attention/inattention, meditation/uneasiness); (1) is the expression for attention measurement using EEG. mBata is middle Bata waves (frequency 16–20 Hz) and SMR is low Bata waves (frequency 12–15 Hz) [20]. Consider

$$\begin{array}{cc}& \frac{\text{SMR}+m\text{Bata}}{\text{Theta}}.\end{array}$$

(1)

Figure 5 is a sensor process and Figure 6 is test result of experiment. Our experiment is focused on attention and meditation because these two emotions are most related with game play. Equation (2) is how EEG measurement can make virtual world environment:

$$\begin{array}{cc}& \frac{1}{100}\left[\frac{\text{attention}\mathrm{\hspace{0.17em}\hspace{0.17em}}\text{value}}{\text{meditation}\mathrm{\hspace{0.17em}\hspace{0.17em}}\text{value}}\right]\left[\begin{array}{c}1\\ 8\end{array}\right]=\left[\begin{array}{c}\text{lighting}\\ \text{fog}\end{array}\right].\end{array}$$

(2)

Figure 5

Sensor measurement and normalization.

Figure 6

Emotion classification with MindSet.

3. Unity Environment System

To implement game environment system, we used Unity3D engine. It is a cross-platform game engine with a built-in IDE developed by Unity Technologies. It is used to develop video games for web plug-ins, desktop platforms, consoles, and mobile devices. The graphics engine uses Direct3D, OpenGL, OpenGL ES, and proprietary APIs. There is support for bump mapping, reflection mapping, parallax mapping, screen space ambient occlusion (SSAO), dynamic shadows using shadow maps, render-to-texture, and full-screen postprocessing effects. We can express day and night, rain, and so forth using Unity like other 3D game engines. And there are many ways to express various environments in Unity. Fog and lighting have an effect on FPS game play because FPS view is almost the same with real world. Fog and lighting obstruct the view of game play. View is important in FPS game play. In this paper, we implement environment simulation system using EEG. Implemented system is in Table 2. Implemented system process is as Figure 7. Unity3D's redersetting is in Figure 8. Fog is set from 0 to 1. The maximum value of fog is 1. Thus, meditation is mapped in a reverse way.

• (1)

Player's emotion (attention/inattention, meditation/uneasiness) is determined in real time via MindSet.

(2)

Measured emotion changes the game's environment (attention-lighting, meditation-fog).

(3)

Game character is affected by lighting and fog for play time.

Table 2

Emotional state game module.

Emotion	Expression of environment
Attention	Sunshine (lighting)
Inattention	Dark (lighting)
Meditation	Thin (fog)
Uneasiness	Dense (fog)

Figure 7

System process.

Figure 8

Rendering setting of Unity3D.

We inserted ThinkGear Native library into Unity project as plug-in to use MindSet; flow chart is in Figure 9. Plug-in in Unity is in Figure 10.

Figure 9

ThinkGear Native Library flow chart.

Figure 10

ThinkGear as plug-in in Unity project.

4. Results and Test

Player wears EEG measurement device for game environment system in FPS for playing time. The graph according to each player's emotional state is in Figure 11. Female player's meditation average value is higher than the male player's. And male player's attention average value is higher than the female player's. Figures 12, 13, 14, 15, and 16 are results of each situation. Virtual world's environment (lighting, fog) is changed by player's emotional state real time. We tested our system on 6 players. Experiment results are as follows. Images are captured for playing game.

Figure 11

EEG measurement results.

Figure 12

Result of image 1 (attention 67/meditation 81).

Figure 13

Result of image 2 (attention 22/meditation 87).

Figure 14

Result of image 3 (attention 79/meditation 46).

Figure 15

Result of image 4 (attention 23/meditation 18).

Figure 16

Result of image 5 (attention 49/meditation 20).

5. Summary and Conclusion

Wireless sensor networks are rapidly evolving into ubiquitous sensor networks [21]. Recently, wireless sensors have been proposed for the assisted living and residential monitoring and also physiological sensors are used to monitor vital signs, heartbeats, and brain waves. Sensor data is sent periodically via wireless links to a personal computer that analyzes the data [22]. Personal area network for intrabody communication using human body as the transmission medium enables wireless communication without transmitting radio waves through the air [23].

We proposed a game environment system using wireless EEG sensor device for immersion of FPS game play. We measured player's EEG using MindSet device for game playing and evaluated the signal to determine player's emotional state. This normalized value operates our game environment system in 3D virtual space. In 3D virtual space, each player's experience change the game environment situation according to his emotional state. This system can afford game player to recognize his concentration situation and increase player's immersion for playing game. The implemented system can be applied to various 3D game genres and raise players' immersion for playing game. Our system takes place in a virtual world, but the actual emotional state of player is shown in the content. This system can make player feel closer to game virtual world. In the future research, we will implement more precise game environment system according to physiological signal related to players' age classification, ethnic group, and so forth.