Sleep Monitoring System Using Kinect Sensor

Abstract

Sleep activity is one of crucial factors for determining the quality of human life. However, a traditional sleep monitoring system onerously requires many devices to be attached to human body for achieving sleep related information. In this paper, we proposed and implemented the sleep monitoring system which can detect the sleep movement and posture during sleep using a Microsoft Kinect v2 sensor without any body attached devices. The proposed sleep monitoring system can readily gather the sleep related information that can reveal the sleep patterns of individuals. We expect that the analyzed sleep related data can significantly improve the sleep quality.

1. Introduction

Recently, IoT (Internet of Things) technology has received great attention as one of the key issues for future technology and it provides the wireless connection to achieve the necessary information between any devices and systems. This IoT technology is expected to provide more convenient living environments to people in the near future. IoT technology is generally related with human behaviors as a commend input from various information that can be utilized to control the devices such as human face detection [1] and human gestures [2–4]. IoT technology has been also applied for the energy saving to efficiently control the limited natural resources [5].

This information can be a key element to design and implement the smart home [6]. Smart home platform has complicated connections to be communicated each other between all devices and natural behaviors of individuals should be detected by sensors to trigger the devices or systems [7]. Smart home platform can be included or combined with other technologies and the human health related wellness technology is one of them. The proposed sleep monitoring can be one of the elements for smart home system and it can take care or improve the human life through sleep state monitoring and it is one part of improving human wellness.

Recent human life is intensively busy and has been exposed to stressful work tasks and these cause the increasing number of insomnia patients as shown in Figure 1 [8]. Sleep disturbing has been known for causing the increased learning disabilities, inefficient work process, and unnecessary accidents that can cause serious illness [9]. Sleep disturbing can be caused by various factors and the diagnosis of sleep disturbing is also a highly difficult task. For sleep disturbing examination, patients should visit the hospital and should wear various devices; thus, they cannot sleep well due to unfamiliar sleep environments in hospital.

Figure 1

Sleep difficulties at least once a week (%).

In this paper, we introduce the Kinect sensor based simple sleep monitoring system to gather the sleep movement, posture, and environment not only to collect sleep movement information but also to extract the overall sleep information. Since it does not require us to wear any troublesome sensing devices, the proposed system can provide more natural and comfort sleep environments to collect the sleep related information.

2. Related Works

Lee et al. [10] used acceleration sensors which are attached on epigastrium, mesogastrium, and lower abdomen using sleepwear. This study used 3 acceleration sensors that can detect the sleep posture and breathing during sleep. This detection is using voltage information from acceleration sensors. However, this approach cannot detect breathing when a patient is lying on side and front, because 3 axis acceleration sensors cannot detect the movement of chest and abdomen.

U-healthcare based sleep monitoring system for sleep control and remote monitoring is applied with sensors and switches, and it shows an example of healthcare system using modified sleep mat [11]. This example is comprised of related sensing and receiving data part and controlling and all sleep related situations monitoring part. It used TMO (time-triggered message-triggered object) scheme model, and it collects the real time sleep data. Sleep detection module is separated in 3 parts which are position detect sensor, sound detect sensor, and vibration sensor to warning.

In the monitoring system, on/off sensor collects the sleep motion data and it determines that the sleep position is good or bad. When the decision case is bad, the vibration sensor is working for user to notify the situation.

In sleep monitoring via depth video image compression and analysis [12], the sleep detection is monitored by a Microsoft Kinect v1 using depth image and video. In this study, they did not use skeleton tracking approach which is included in the Kinect v1 SDK. The main purpose of this study was the detection of hypopnoea though the depth video image. The program can detect the change in depth image via the movement of the chest. However, it has limitation which is it can detect only hypopnoea that is just one of the symptoms of sleep disturbing.

Smart bedroom for elderly using Kinect is also purposed and implemented to help them by providing smart home environments using Kinect v1 [13]. This system mainly detects the arms motion and gives a warning for the risk of bed falling. To notify the risk of bed falling, it sets up the four corner points in the bed. This system detects and tracks the human body using skeleton tracking method in Kinect v1 SDK, and it notifies the warning when those skeleton points are out of the corners in bed. Also, it can recognize the arms motion that conveniently requests the help of other people for the elderly who find it difficult to get up from the bed. It tracks the angle of arm which is calculated by the elbow-to-wrist vector. However, it has the same limitation in general video recognition approach which makes it difficult to detect the human body when people use a blanket in bed.

3. Sleep State Monitoring

Generally, there are two ways to detect the sleep state. We can directly attach some sensors to human body or we can record body movements during sleep to analyze the sleep quality. The direct sensor attachment based method can provide some important sleep related information from human body. In addition, it can provide scientifically accurate sleep state information. However, the wearing of complicated sensing devices leads to uncomfortable sleep environments and it may negatively affect the measurement of sleep state. Figure 2 shows the necessary devices for nocturnal polysomnography to diagnose the sleep state.

Figure 2

Polysomnography devices.

Video based approach is also able to conveniently provide the sleep state, but it is more difficult to analyze the sleep state than attached sensors. Recently, color, infrared, and depth information can be obtained by camera and some other unwearable sensors are also able to achieve the sleep related information. In order to monitor the sleep state with video, it is necessary to recognize the shape of human body and then the image processing based approach is essential to extract the human body from the input images. Infrared and IIR (Imaging Infrared), differ integrator, skin segmentation [14], and skin color and edge detection [15] have been widely applied to detect and track the movement of human body from video images.

3.1. Microsoft Kinect v2

In this paper, we utilized the Microsoft Kinect v2 to detect and track the movement of human body. We selected the Kinect v2 which is the latest version of Kinect and this version of Kinect provides the position of 25 joints, and detection range has been more improved than previous Kinect v1. In addition, Kinect v2 changed IR (infrared) irradiation to TOF (Time of Flight) which is configured with one pair of receptors.

The principle of TOF is the light reflection that measures the distance by the receptor. This new depth detection method provides much more accurate depth information than Kinect v1. Kinect v2 is only performed under Windows 8 OS (Operating System) and over Visual Studio 2012. Therefore, the proposed system is implemented with C++ using Kinect v2 SDK. We also utilized OpenCV library to detect and track the human body movement during sleep. The proposed system can recognize the human body through an infrared sensor and a depth sensor which are shown in Figure 3.

Figure 3

Depth and infrared image from Kinect v2.

The infrared sensor of Kinect v2 detects the reflected infrared rays and provides the movement of users with pixel by pixel information. The depth sensor can achieve the depth information which is made up with CMOS sensor that converts light to electrical image signal with infrared ray projector. The Kinect v2 includes the specific human body information and a matching algorithm which can detect human body movement with sensing information. Therefore, we can readily gather the sleep movement related information with Kinect.

3.2. Architecture of Sleep Monitoring System

Figure 4 shows the device architecture of proposed sleep monitoring system that automatically communicates with each device. This service stores the daily sleep state of users into the sleep monitoring data server.

Figure 4

Device relationship diagram.

This sleep monitoring system stores user related information such as name, age, height, weight, and medical history. Also, sleep related information is stored into the database which is shown in Table 1.

Table 1

Data field of system.

Input data	Name
	Age
	Gender
	Height
	Weight
	Brief medical history
Monitoring data	Movement information during sleep
	Change of sleep posture
	Room temperature and humidity

4. Measurement of Sleep Movement

In order to monitor user's movements during sleep, the proposed system should not only detect the human shape from image but also continuously track the movement of human body. Unlike wearing many sensors on human body, the proposed system uses Kinect v2 camera for an input image to detect and track the movement of human body.

Since the Kinect v2 provides detecting of human body joints, it is possible to estimate the degree of human body movement. Because the Kinect v2 requires the minimum distance for object detection, we have to locate the Kinect v2 camera on the appropriate position which is between 0.5 m and 4.5 m from object. Therefore, we installed the aluminum angles to locate the Kinect camera at 2.0 m high on the bed. Figure 5 shows the configuration of proposed sleep monitoring system and the located Kinect camera to detect whole human body joints without any distortions.

Figure 5

Configuration of the proposed sleep monitoring system.

The Kinect v2 provides x, y, and z positions of 25 joints and distance data of objects from depth sensor. Since it is necessary to detect the overall movements of the human body, we store x, y, and z positions of all joints in every half second and Figure 6 shows the detected 25 joint positions. Because sleep movements mainly occur by limbs, we selected critical 19 joints relating with the sleep movements through the comparison and they are head, left shoulder, left elbow, left wrist, left hand, left hip, left knee, left ankle, left foot, left thumb, right shoulder, right elbow, right wrist, right hand, right hip, right knee, right ankle, right foot, and right thumb. Figure 7 depicts the total sleep movement value of 19 major joints and movement value of other joints. Since minor joints do not critically affect the body movement during sleep, we excluded the value of these joints from our calculation.

Figure 6

Detected 25 joints for body tracking.

Figure 7

Comparing the movement value of major joints and minor joints.

The sleep movement value is calculated by using the Euclidean distance between previous position of joints and current position of joints from image sequences. All movement values of major joints are calculated every 0.5 seconds. The formula to obtain the sleep movement value is shown as (1). SM (sleep movement) is a distance of moved coordinate from previous coordinate of 19 key joints to current coordinate. ${\mathcal{X}}_{\mathcal{i}}$ c is current $\mathcal{X}$ position, and ${\mathcal{X}}_{\mathcal{i}}$ p is previous $\mathcal{X}$ position in image sequences:

$$\begin{array}{l}\text{SM}\\ ={\displaystyle \sum }_{\mathcal{i}=\mathrm{1}}^{19}\sqrt{{{\left({\mathcal{X}}_{\mathcal{i}}\text{c}-{\mathcal{X}}_{\mathcal{i}}\text{p}\right)}^{\mathrm{2}}+\left({\mathcal{Y}}_{\mathcal{i}}\text{c}-{\mathcal{Y}}_{\mathcal{i}}\text{p}\right)}^{\mathrm{2}}+{\left({\mathcal{Z}}_{\mathcal{i}}\text{c}-{\mathcal{Z}}_{\mathcal{i}}\text{p}\right)}^{\mathrm{2}}}.\end{array}$$

(1)

After calculating the first sleep movement value, the sleep monitoring system keeps accumulating the sleep movement value until the user wakes up. At the same time, the proposed sleep monitoring system compares image with 5 reference sleep postures to determine user's current sleep posture and Figure 8 shows the flowchart of proposed sleep monitoring system.

Figure 8

Flowchart of proposed sleep monitoring system.

Sleep monitoring system follows the above flow of process and the sleep monitoring result is provided by a smart phone application. The main page of smart phone application is shown in Figure 9.

Figure 9

Main page of sleep monitoring application on smart phone.

In addition, TI (Texas Instruments) sensor tag which is shown in Figure 10 is used to obtain the temperature and humidity of sleep environments. This sensor is connected with sleep monitoring application by Bluetooth, and this sensor provides 6 main pieces of data such as temperature, humidity, pressure, accelerometer, gyroscope, and magnetometer. For this study, we measured the sleep related environments with temperature and humidity sensors.

Figure 10

TI sensor tag for temperature and humidity measurement.

5. Sleep Movement Data Analysis

For experimental test, 20 students participated for sleep state monitoring and the average age of participants was 23 years. We performed the sleep monitoring around 7 hours and the participants' bodies were not covered with any blankets to accurately detect the sleep motions. We tried to provide similar sleep environments such as time of falling asleep, temperature, lighting and noise level of room, and humidity conditions for participants. There are general 6 sleep postures which are shown in Figure 11 such as foetus, log, yearner, soldier, freefaller, and starfish.

Figure 11

Typical 6 sleep postures.

In our system, we monitored the sleep posture of participants with 5 categories. Figure 12 shows the result of sleep movement and the change of sleep posture during sleep experiment. In Figure 12, x axis is a sleep time and y axis is a sleep movement value for left figure and y axis is type number of sleep postures for right figure, respectively.

Figure 12

Result of experiment (movement and posture).

The sleep movement value is sharply altered when the body posture is changed. After the body posture is stabilized, the detected motion data is also almost stable. The proposed sleep monitoring system can precisely count the body movements and it returns the correct counts number for body posture change. In other words, the number of high peeks from the steady constant values signifies the change of body posture.

In this study, we added up all SM values at each hour to make a decision for sleep state. Since we measure the sleep movement value every 0.5 seconds, we summed up 7,200 times. For HSM (Hourly Sleep Movement), collected range of HSM values was 0.0~5.5 and we standardized this HSM value in 0 to 1.0 by (2).

In our system, we decide the sleep quality by the following: under 0.25 as a good with deep sleep, 0.25 to under 0.59 as a normal with regular sleep, and 0.59 to 1 as a bad with light sleep which is shown in (3). This criterion is decided from the range of entire SM value with video monitoring and user evaluation for sleep. In our experimental test, deep sleep was 43 hours (30.7%), regular sleep was 61 hours (43.6%), and light sleep was 36 hours (25.7%) in 140 testing hours. Table 2 shows the judgment of sleep movement per hour for 20 participants and the grade of sleep movement is the average of HSM value:

$$\begin{array}{cc}& \text{HSM}=\frac{{\sum }_{\mathcal{t}=\mathrm{0}}^{\mathrm{7200}}\left(\text{SM}/1000\right)-\min }{\max -\min }\end{array}$$

(2)

$\begin{array}{cc}& \text{Good\hspace{0.17em}\hspace{0.17em}(Deep\hspace{0.17em}\hspace{0.17em}Sleep):\hspace{0.17em}\hspace{0.17em}}\mathrm{0}<\text{HSM}<\mathrm{0.25}\\ \text{Normal\hspace{0.17em}\hspace{0.17em}(Normal\hspace{0.17em}\hspace{0.17em}Sleep):\hspace{0.17em}\hspace{0.17em}}\mathrm{0.25}\le \text{HSM}<0.59\\ \text{Bad\hspace{0.17em}\hspace{0.17em}(Light\hspace{0.17em}\hspace{0.17em}sleep):\hspace{0.17em}\hspace{0.17em}}\mathrm{0.59}\le \text{HSM}\le \mathrm{1}.0.\end{array}$

(3)

Table 2

The amount of movement per hour.

P	T
	1 h	2 h	3 h	4 h	5 h	6 h	7 h	Grade
Case 1	0.34	0.37	0.11	0.51	0.00	0.63	0.69	N
Case 2	0.37	0.40	0.69	0.63	0.40	0.60	0.77	N
Case 3	0.49	0.74	0.09	0.34	0.06	0.89	0.91	N
Case 4	0.31	0.11	0.06	0.09	0.06	0.66	0.57	G
Case 5	0.11	0.03	0.46	0.00	0.54	0.03	0.03	G
Case 6	0.63	0.51	0.57	0.49	0.20	0.09	0.23	N
Case 7	0.40	0.14	0.40	0.43	0.34	0.37	0.37	N
Case 8	0.29	0.43	0.11	0.40	0.03	0.63	0.14	G
Case 9	0.43	0.46	0.43	0.51	0.74	0.69	0.40	N
Case 10	0.20	0.74	0.80	0.77	0.71	0.49	0.40	B
Case 11	0.31	0.31	0.40	0.34	0.49	0.51	0.49	N
Case 12	0.14	0.63	0.60	0.17	0.14	0.31	0.34	N
Case 13	0.34	0.57	0.11	0.29	0.37	0.09	0.11	G
Case 14	0.49	0.06	0.71	0.66	0.11	0.77	0.34	N
Case 15	0.17	0.11	0.09	0.34	0.43	0.14	0.34	G
Case 16	0.06	0.57	0.51	0.11	0.57	0.60	0.66	N
Case 17	0.03	0.57	0.11	0.06	0.63	0.54	0.66	N
Case 18	0.31	0.34	0.60	0.17	0.63	0.34	0.31	N
Case 19	0.49	0.69	0.71	0.66	0.60	0.46	0.43	B
Case 20	0.17	0.63	0.09	0.03	0.11	0.34	0.60	G

P: participant, T: time, and h: hour; Grade (G: good, N: normal, and B: bad).

6. Sleep Posture Data Analysis

Sleep posture is also an important factor for sleep quality. In general, people have 6 typical sleep postures that are founded through the research targeted for one thousand people by SAAS (Sleep Assessment and Advisory Service). According to their research, the sleep posture has some relationship with human characters. Thus, we analyze the sleep posture with 5 categories: foetus type, log type, yearner type, soldier type, and freefaller and starfish type which are shown in Figure 11.

To determine the sleep posture, we mainly applied the boundary information for location of 4 key joints (left hand, left knee, right hand, and right knee). The posture change is robustly detected by the proposed posture verification algorithm which is based on the location of 4 joints in predefined boundary. The proposed posture verification algorithm finds three base lines (middle, upper, and lower line) to determine the sleep posture with boundary information. The middle line is defined with extended line between neck joint and spine base joint. The upper line is determined with the spine midjoint and it is perpendicular to middle line. The lower line is determined with spine base joint and it is also perpendicular to middle line. With these three base lines which are shown in Figure 13, we can readily determine 5 sleep postures with locations of 4 key joints in boundary.

Figure 13

Sleep postures based on boundary information.

In addition to the posture verification algorithm with boundary information, we used SURF (Speeded-Up Robust Features) method in openCV library to figure out the sleep posture with extraction of feature points. The reference postures are predefined by image and the proposed algorithm extracts the feature points between reference and current image and then determines the posture with feature point matching which is shown in Figure 14.

Figure 14

Feature points matching with SURF method.

Figure 15 shows the decisions of sleep posture type with the proposed sleep monitoring system. The sleep posture is determined with the combination of boundary and SURF method and we gave the priority to boundary information based method. When the position result is not well determined by boundary information based method due to failure of joint detection, we utilized the result of SURF based method to alleviate this problem.

Figure 15

Decision of sleep posture type.

7. Conclusions

In this paper, we designed and implemented the sleep monitoring system using Microsoft Kinect v2 that does not require users to attach any inconvenient devices. The proposed system provides comfortable and natural sleep environments for experiment; thus, we believe that it is well suitable to monitor the ordinary sleep conditions. Also, it can monitor the exact number of sleep position changes during sleep. According to the experimental result data, we can detect sleep disorders such as restless legs syndrome in further research. The proposed sleep monitoring system can measure the sleep quality and can notify the daily sleep condition with sleep time, sleep posture, and sleep quality using application. It can also figure out the pleasant sleep condition for users with room humidity and temperature.

However, the proposed system has a drawback which is that users could not use blankets, because depth sensor cannot work with blanket which covers human body. In addition, it is difficult to recognize front and back of sleep posture. For further research, we will add multi-Kinect cameras to achieve more detailed information from the different range of viewpoints. Also, we will apply the Kinect v2 sound array sensor that can determine snoring and apnea of users [16]. We believe that we can comprehensively improve the proposed sleep monitoring system using the Kinect and unattached sensors for analyzing critical sleep factors such as snoring and external sleep disturbance sources.