Intelligent human-centric lighting for mental wellbeing improvement

Abstract

In recent years, the main area of interest in the issue of influencing mental states of people is the impact of lighting on human beings, their wellbeing but also workplace productivity. This work discusses in detail the problem of positively influencing people using intelligent technologies, especially the role of the colors. We describe techniques and technologies needed to implement the case study of an intelligent lighting system. The system proposed can detect humans from an IP camera, find faces, and detect emotion. The main aim is to adjust the lights accordingly to the emotional result to improve the mood of people while taking into consideration the principles of color psychology and daytime. We have evaluated our case study solution in a real-world environment and collected the feedback from participants in the form of a questionnaire. Evaluation of participants’ wellbeing was based on their subjective statements. There were several ideas on further functionality extension which needs to be explored. Among them is including wearable devices to the proposed system, validate the emotional results according to them, but also determine the impact of an increasing number of users interacting with the system at the same time.

Introduction

Technologies are evolving and improving, and with that, peoples’ concerns are becoming increasingly important. One of the areas, lighting, endorses this tendency, introducing the approach of human-centric lighting (HCL). The main aim of HCL is adjusting the light for the optimal needs of humans.

Multiple factors play a role in influencing people’s inner state and wellbeing. Taking into consideration for instance color, which can be defined as the light represented by wavelengths absorbed by the eyes. Light can be decomposed into a spectrum of six colors: red, orange, yellow, green, blue, and violet. The red has the longest wavelength, while violet the shortest. ¹ It is proven that some of the wavelengths of light may have specific impacts on human body. ² For example, the human circadian rhythms may be sensitive to short wavelength of light, and blue light may enhance the brain’s cognitive performance. ³

Color psychology includes the meanings of colors that the human brain determines. Light waves passing through retina are converted into electrical impulses that are processed in the hypothalamus, which is the control center of the body’s biological clock. ⁴ The biological clock can form your emotions and mood. The most common colors examined by psychologists are red, blue, yellow, green, violet, orange, black, and pink. There could be positive and negative psychological impacts of colors. ⁵ For example, red can stimulate the body and raise the pulse, and it can give the person the notion that time is passing faster than it is. It also represents physical courage, but it can be seen as an aggressive color. Blue is the color of trust and intelligence. Yellow could have positive effects and encourage confidence and optimism. ⁴

Using the artificial lights set to the right brightness and color temperature can supplement the daylight and make use of the biological effects of the natural course of daylight. This kind of conditions can create a pleasant atmosphere with a positive impact on the productivity, health, and wellbeing of humans. ⁶

The challenge is how to configure the artificial lighting according to the changing environment. Setting the brightness and color temperature manually can be tedious and distracting, which can result in outperformance of the gained benefits. This article addresses the challenge by proposing a system for lighting adjustment without human intervention. Our approach is based on the recognition of human emotions that represent the input parameters for the correct lighting configuration. The aim is to improve mental wellbeing while making decisions entirely autonomously.

The main contribution is twofold. First, we summarize the impact of lighting on human emotions and describe the possibilities of how intelligent technologies can improve mental wellbeing. Second, we present a case study based on the system that configures the artificial lights without human intervention. In contrast to other approaches, our solution is fully autonomous and can adapt to the changing conditions in the environment.

The rest of the article is structured as follows: first, the HCL is discussed and its applicability is highlighted in several application domains. Second, the intelligent lighting system is proposed and then evaluated in the presented case study. Finally, the summary and future work is provided in conclusion.

HCL

In recent years, there is an emerging need to develop applications that are trying to meet people’s emotional needs through the control of color temperature, illuminance levels, and light direction. HCL is the process by which the correct control of the lighting can improve the emotional wellbeing, comfort, health, and productivity of people in the environment. ⁷ According to Walerczyk, ⁸ there are several factors taken into consideration (Figure 1):

• Circadian rhythms—blue light at night can intrude our regular hormone flow;

• Mood—certain colors or light can affect emotions, mood, and cognition;

• Visual acuity—adding more blue to the light can increase the visual acuity in other words, clarity of vision.

• Perception—factors that may help people perceive the course of the day, the amount of energy they should have at a particular time of day or the year in addition to the amount of light and spectral distributions for circadian rhythms,

• Sustainability—dimming and Kelvin changing LED systems can save notable amounts of energy, which decreases carbon footprint;

• Productivity—high correlated color temperature fluorescent lights could present a beneficial intervention to enhance wellbeing and productivity.

Figure 1.

Human-centric lighting.

The term lighting control is used to define stand-alone control of the lighting within an environment. An automated lighting control system is an intelligent network-based solution that includes exchanging information between a variety of inputs and outputs with the use of one or more central computing agents. These systems can be used both indoor and outdoor and can provide the appropriate amount of light when and where it is necessary. They are used to minimize energy costs. Nowadays, the color temperature is taken into consideration as a new, yet important element of lighting control solutions. ⁷

Applications of HCL

HCL can be used in several application domains—healthcare, workplaces, education, retail, industrial, and home environment.

Healthcare

There is evidence of improved recovery rates and enhanced medicated efficiency and also stimulation by light and colors despite outdoor conditions. ⁹ Philips HealWell ¹⁰ is a commercial solution of patient room lighting that is creating a more effective healing environment using the natural power of light by adjusting lighting automatically with a human circadian rhythm which helps hospital patients to sleep better thus heal faster. There is, for example, LED ceiling module and LED cove lighting, ¹⁰ which is simulating daylight and helps patients to recover faster by supporting their circadian rhythms.

Workplaces

Some of the health disorders are linked to abnormal amounts of blue-rich light during working hours of the day, taking into consideration 5 days working week, steps need to be taken to get natural light. ¹¹ By simulating natural light in workspaces, it is possible to improve concentration and energy, but it can also increase employee’s motivation and commitment and improve work performance, awareness, and satisfaction. ⁷

Education

Research done by Govn et al. ¹² shows that ambient lighting can have a positive impact on children’ chronobiological system, moods, and academic performance. By adjusting lights, it is possible to increase concentration as well as decrease fatigue in classrooms. ⁷ In the light study conducted by Transfer Center for Neuroscience and Learning in collaboration with Osram, ¹³ students with the biologically optimized lighting had up to third fewer errors in the concentration test. Contributing to the positive results was the fact that biologically optimized light provides stimulation to the body as if the person was outdoors. Therefore, this light can counteract social jetlag—tiredness in the mornings frequently observed in young people, notably.

Industry

HCL can help with biorhythm balancing in nightshift workers. Output and error rate of monotonous work can be improved. ⁸ It can influence the performance of the workers by way of several mechanisms—visual performance, visual comfort, visual ambiance, interpersonal relationships, biological clock, stimulation, job satisfaction and problem-solving. ¹⁴ Philips GreenWarehouse ¹⁵ is stating significant energy and cost reductions than in traditional lighting systems, reducing the carbon footprints but also a better quality of light which will increase the comfort of the workers.

Smarthome

In the area of residential solutions, a potential cause of health disorders is mostly excessive exposure to the blue light, especially during the night hours. Smart lights are trying to help people eliminate this kind of lights by simulating natural light. ⁸ Philips Wake-up light ¹⁶ simulates a natural sunrise for a refreshing wake up, and sunset, which is preparing people to sleep. It comes with an adjustable brightness level according to preferences of the user, and it has been clinically proven to improve the general wellbeing of people after waking up:

Ninety-two percent of users say this wake-up light wakes them up pleasantly, while 88% of users say that the wake-up light is a better way to wake up than how they did before. Besides, 92% of users find it easier to get out of bed.

Sleepace Nox Music Smart Sleep Light ¹⁷ is a smart lamp that could help people living with insomnia. By light and sound programs that adapt to user’s body clock, it can improve the quality of sleep by red-light wavelengths, as the red light helps raise levels of melatonin to promote sleep, while delicate sounds can help the user to calm down. The user can adjust the color of light.

Eye 2 H ¹⁸ is an intelligent system that uses an aggregate image and facial recognition methods to identify and analyze the human emotion of the people living in a smart home to set the brightness of the light to the level which may influence people’s wellbeing.

Motivation

According to research conducted on current solutions, to authors’ best knowledge, there is no system proposed that adjusts itself to the users without their intervention and also offers color adjustment at the same time.

Solutions mentioned in the preceding section adjust to people’s need but in a general way, which is an essential step in developing smart lighting systems, but it lacks the higher level of personalization and intelligence. If the user demands these systems to provide for their specific needs, it needs to be set manually by providing appropriate input, or the system can set the brightness, but not the color of the light. In this article, the main goal is to propose a system that will be able to improve the mental wellbeing of people by colored lighting options depending on their current emotional state without the need of specifying it manually.

Case study

To achieve the goal, an intelligent lighting system was developed. The idea behind the system is to extract real-time visual information about the user from a camera. As the user’s face is detected in these video frames, emotion recognition is performed. According to these results, the system of the LED strips in the environment is set to specific color according to the theoretical concepts of the psychology of colors and daytime with a goal to improve the mental wellbeing of a user.

The system consists of several hardware components (Figure 2). The IP camera is connected to the Raspberry Pi 3 Model B through local network. The program logic and MQTT broker run on the Raspberry Pi, which acts as a central processing unit of proposed solution. The mobile devices and ESP8266 modules are communicating with Raspberry Pi through MQTT protocol via TCP/IP. The communication between the ESP8266 module and the LED strips is based on the radio frequency (RF). One or multiple ESP8266 modules can control one or multiple LED strips. It depends on the limitations of RF communication, such as distance and spatial arrangement of the components in the working environment.

Figure 2.

System scheme.

The software compounds of two parts—one of them is training of the neural network for emotion detection and the other one is main program logic where emotion classification, with the model produced from the first step, is executed, resulting in color choice in the RGB LED strip.

Emotion detection

First of all, the face needs to be detected. A face detection algorithm proposed by Viola and Jones ¹⁹ based on the concept of cascade classifiers was implemented to detect faces in the video frames. Cascade function is trained from a lot of positive and negative examples. Then it is used to detect faces or objects in general in other visual inputs. From these examples, feature extraction is performed using Haar features, which suits as a convolutional kernel. Each feature stands for a single value retrieved by subtracting the sum of pixels under the white part from the sum of pixels under the black part. An enormous number of features obtained is reduced by the integral image. Despite these eliminations, there are a lot of irrelevant features. Acquiring only the relevant features is achieved by AdaBoost. To speed up this process, even more, the concept of the cascade classifier is introduced. The basic idea in the cascade classifiers lies in the division of the features into several stages. In the first stages, there are fewer features and if in the selected region of interest detection fails in some of these stages, this region is discarded and thus not recomputed again. The region which passes through all stages successfully is the face. This classifier is trained to detect the frontal faces and provides robust and rapid visual detection. The face detected is framed by the bounding box in the window, where the input from the IP camera is shown.

To classify the emotions, the classification model was trained. The model is a fully convolutional neural network, which combines the two concepts, residual modules and depth-wise separable convolutions.

Residual modules ²⁰ adjust the desired mapping between two consecutive layers in a way that the learned features become the difference between the initial feature map and the desired features. The desired features $H(x)$ are adjusted in favor of solving an easier learning problem $F(x)$ such that

$$H\left(x\right)=F\left(x\right)+x$$

One of the benefits of this method is that it allows backpropagating the gradient more easily in deep neural networks. ²¹

Depth-wise separable convolutions ²² are able to reduce the number of parameters by dividing the processes of feature extraction and combination within a convolutional layer. Depth-wise separable convolutions consist of two types of layers: depth-wise convolutions and point-wise convolutions. They separate the spatial cross-correlations from the channel cross-correlations by first applying a $D\times D$ filter on every $M$ input channels and then applying $N\times 1\times 1\times M$ convolution filters to combine the $M$ input channels into $N$ output channels. Applying $1\times 1\times M$ convolutions incorporate each value in the feature map without taking into consideration their spatial relationships within the channel.

The architecture is shown in Figure 3. It consists of four residual depth-wise separable convolutions. Each convolution is followed by a batch normalization operation and ReLU activation function. In the last layer, the global average pooling as well as a soft-max activation function are applied to produce an output. This architecture was proposed in Arriaga et al., ²¹ and we find it sufficient for creating the prototype of our system.

Figure 3.

The architecture of classification model—convolutional neural network.

For the training of this neural network, the FER2013 dataset ²³ was used. This dataset consists of 28,709 $48\times 48$ training, 3589 validation and 3589 testing grayscale images of people’s facial expressions under 7 different categories. Each image belongs to one of the following categories: angry, disgust, fear, happy, sad, surprise, and neutral. An example sample is shown in Figure 4. The training was done on the personal computer with the following characteristics: 2-core Intel i7 3537U CPU with base frequency at 2.00 GHz, 8GB of RAM memory, 2GB GeForce GT720M GPU, and 5400RPM HDD drive. The training stopped after 129 epochs when there was no improvement in validation loss function, which means that the best possible result was achieved. The accuracy of the validation set is 66% with a loss of 96%.

Figure 4.

Samples from the FER2013 dataset. ²³

Decision-making

There are several steps in the main program logic running on the Raspberry Pi, as shown in Figure 5. First of all, the input from the IP camera is taken and processed. On these video frames, the face is detected, and if the face is present in the video frame, the emotion can be detected. Based on the daytime and emotion result, the desired color value is set. To reduce the amount of information processed, the time between the two following emotion classifications was set to 1 min.

Figure 5.

The logic of one cycle from the input frame to the final color setting.

Even though the emotional classification model can identify seven categories of the emotions, for the system, four categories were chosen—angry, sad, happy, and neutral. Surprise, fear and disgust were excluded from the model. The surprise is considered to be immediate to reaction to a specific event; thus it is not necessary for us to try to maintain it or improves it. Similarly, disgust is an immediate response. Fear is a response that arises from the perception of threat leading to a confrontation with the source of fear. Fear should be eliminated by removing the source of fear.

According to the research done, there are different responses to the colors in the human body in the daytime and the night (Table 1). This was taken into consideration and implemented in a way, that in the daytime, light with a higher blue content is used to enhance productivity—blue and green. In the night, lower blue content lighting is used—red, orange, yellow, and violet.

Table 1.

Color response according to emotion and daytime.

Time	Emotion
	Angry	Sad	Happy	Neutral
Day
Night

There are six colors used in the system, assorted to the emotions chosen according to certain principles of the psychology of colors:

• Blue—lowers blood pressure, inspires mental control, clarity, and creativity;

• Green—relaxes the muscles, helps to breathe deeper and slower, creates feelings of comfort, relaxation and calmness;

• Violet—associated with bringing peace and calmness;

• Red—increases blood pressure;

• Yellow—proven to stimulate the brain and can make a person more alert and decisive, happy and uplifting, aiding organization, and encourages optimism;

• Orange—relieves feelings of self-pity and, antidepressant.

Communication

For the communication between the Raspberry Pi and the ESP8266 module(s), the MQTT protocol was used. The Raspberry Pi acts as a MQTT broker but also as the publisher. The subscribers are subscribed to the MQTT topic, and these can be either ESP8266 modules or mobile device connected to the network. Even though the protocol itself is a lightweight solution, there are several restrictions implemented to reduce the number of messages exchanged. If the emotion processed is the same as the one sent previously, the current emotion status won’t be sent. If the emotion processed is different than the one sent previously, the current information is compared to the emotion one step and two steps before. The three following emotional states are being compared. By defining this constraint, the system is assured that the emotion has changed and will be partially prone to the errors which can occur because of misclassification in the model. If the two consecutive states confirm the same emotion, the message is sent, and color is set accordingly. If the user is not satisfied with the choice of the color made by the system, it is possible to change the light settings to the desired ones. For that, a simple iOS application was developed. This application is designed so user can switch between intelligent and manual light control.

Discussion

The demonstration model was implemented and evaluated in the real-world environment—the office as shown in Figure 6.

Figure 6.

Demonstration model installed in the real environment.

Ten participants tested the system and their feedback was collected using the questionnaire to review the results. We were interested in knowing their opinion on the system functionality, the impact of the lighting on their wellbeing, and on suggestions for further improvements.

The main results obtained from the questionnaire are summarized in the Table 2. Our main concerns were the correctness of the colors chosen, the pace of the changes, the overall impact on the mood, possible disturbance of the system on the user, and overall wellbeing after the test. The answers were given numerical representation from 1 to 5, where 1 represents the lowest score and 5 the highest, while in section considering pace of color changes, users choosing 1 thought that colors could be changed slower and vice versa.

Table 2.

Results of questionnaires.

Participant	Color correctness	Pace of changes	Impact on mood	Disturbance	Overall wellbeing
1	4	1	3	1	4
2	5	1	4	2	4
3	4	2	3	1	3
4	5	1	4	3	5
5	2	3	2	2	4
6	3	2	5	1	3
7	4	3	2	3	4
8	5	2	3	3	4
9	1	2	1	1	2
10	3	1	4	2	4
Mean	3.6	1.8	3.1	1.9	3.7

The overall impressions from the participants were positive; the most appreciated feature was the fully autonomous functionality without the need for user intervention. The system behaved as expected, and the lighting was adjusting according to the detected emotions. The evaluation of participants’ wellbeing was based on their subjective statements, but all of them reported positive mood after the test. It could be partly caused by their enthusiasm for seeing the system in action, but they also described the lighting colors as a positive stimulus.

Nevertheless, we also received several suggestions for further improvement. For instance, the transition between the colors was reported to be too sharp and a little distracting. It would be more pleasing if the colors were gradually fading from one to another, thus making the change more subtle. Also, the time frame between the consecutive detections could be more than one minute. It is not necessary to adjust the lights so often as the mood is not likely to change that fast. The outcome of the discussion with participants was that the solution could fade from the previous color to the new one in the time frame of 5 min. All these comments should be considered in the next system versions.

In real conditions, it was observed that sad or neutral emotions are being detected most of the time. One of the difficulties is also that people are often not showing emotions for a prolonged time period. Sometimes it could be just seconds.

During the system testing and the evaluation process, there were several ideas on further functionality extension. As for now, we developed the iOS mobile application in which the user can set the lighting according to their preferences. Nonetheless, it would be interesting to store and analyze these user inputs and use them as the training data so the system could enhance its performance by taking into consideration these preferences. The system would learn with every interaction and could provide a higher level of personalization for the users.

The emotion detection reached the accuracy of 66%, which was sufficient for the prototype, but we would like to obtain more accurate results. Thus, it is necessary to develop a more robust emotional model. It can be achieved by developing a different architecture of a deep neural network or enhancing the existing one proposed by Arriaga et al. ²¹

Another way of attaining more accurate emotional results could be data fusion from multiple sensors and its combinations and modifications. One of the ideas is to incorporate the wearable devices to the system and use them to acquire additional information about the user, like heart rate data or physical movements. Attributes such as the heart rate and current activity could be good indicators for understanding the wellbeing of the user. Further research on this topic needs to be done to understand the extraction of individual states and emotions from vital signals and then to map them onto the existing architecture of the system. The results could either confirm or deny the results from facial detection and vice versa. For instance, if the user would be exercising, higher heart rate does not have to mean that he is angry, so with more types of user data, we could provide more accurate light condition knowing what is happening with the user in the environment. The wearables can be highly diverse, so utilizing paradigms like the edge computing ²⁴ and cloud computing ²⁵ would facilitate the management process.

Besides, we would like to take into consideration emotion and mood in the group of people interacting with our system, because it is possible that more than one person could be present within sight of the camera which means they would also be detected. For now, we did not consider this, because our scenarios were purposefully set just for one person at the time. So taking into factors the concepts of collective sensing, we could analyze the emotions of multiple users and set the resulting color as the weighted output of all users. For example, the question would be, if multiple people are watching a TV or playing a table game, how we can adjust the lights to suit all of them?

More scenarios and user test cases need to be executed to understand the impact of the system on the wellbeing of users better. We want to test the system in the three different environments—in-home health care, smart home and industrial. From gaining preferences from different environments and industries, we could make specific changes for each of the application domains.

The hardware parts could be exchanged for the existing solutions provided by companies mentioned earlier, so it would not have to be necessary for us to take care of the hardware implementation and maintenance. Already existing hardware solutions could provide more pleasant and reliable visual and performance results.

Conclusion

In conclusion, this article addressed the topic of improving mental wellbeing using intelligent technologies. In particular, we described the impact of lighting on human emotions and proposed a solution capable of adjusting light color and brightness according to the user’s needs. In contrast to other HCL lighting systems, our approach is fully autonomous and does not require any manual intervention.

The benefits of such a system include the elimination of distractions caused by manual configuration, the dynamic adjustments of lighting to create a pleasant atmosphere, and the positive impact on productivity, health, and wellbeing of humans, among others. The contribution of this article is twofold: (1) we provided the overview of how intelligent technologies can influence the mental wellbeing with the focus on HCL and (2) we presented the case study based on the intelligent lighting system that utilizes artificial intelligence to recognize user’s emotions and set the appropriate lights color and brightness.

The proposed solution can be applied to several application domains, including, but not limited to, healthcare, workplaces, education, industry, and smart home. Our research group is particularly active in the smart healthcare domain as we are working on the cognitive healthcare platform called CHECkuP. ²⁶ The platform utilizes the state-of-the-art in the Internet of Things and Artificial Intelligence to provide the edge-enabled solution for improving the quality of home care and to facilitate the transition from hospital-centered to home-centered healthcare. Our aim is to incorporate the presented HCL system to CHECkuP as one of the building blocks that influence patients’ health and wellbeing with intelligent technologies. The patients are especially sensitive to all subconscious perceptions so adjusting the environment toward their wellbeing would be a significant benefit, which is our motivation for future work.

The presented case study proved the feasibility of our approach and showed the possibilities for improvement. For instance, even though the system is fully autonomous, it could also take into consideration the human input while making decisions about the lighting settings. In addition, the emotion detection accuracy could be increased with other sensors, such as the wearables devices. Recognizing the right emotion can sometimes be a difficult task even for humans, so extending the number of monitored attributes would help to provide better results. As a result, the intelligent HCL systems require further research, but their utilization is highly desirable in many areas, such as in smart healthcare as described by the platform CHECkuP.