On the design of a human–robot interaction strategy for commercial vehicle driving based on human cognitive parameters

Abstract

A proper design of human–robot interaction strategies based on human cognitive factors can help to compensate human limitations for safety purposes. This work is focused on the development of a human–robot interaction system for commercial vehicle (Renault Twizy) driving, that uses driver cognitive parameters to improve driver’s safety during day and night tasks. To achieve this, eye blink behavior measurements are detected using a convolutional neural network, which is capable of operating under variable illumination conditions using an infrared camera. Percentage of eye closure measure values along with blink frequency are used to infer diver’s sleepiness level. The use of such algorithm is validated with experimental tests for subjects under different sleep-quality conditions. Additional cognitive parameters are also analyzed for the human–robot interaction system such as driver sleep quality, distraction level, stress level, and the effects related to not wearing glasses. Based on such driver cognitive state parameters, a human–robot interaction strategy is proposed to limit the speed of a Renault Twizy vehicle by intervening its acceleration and braking system. The proposed human–robot interaction strategy can increase safety during driving tasks for both users and pedestrians.

Introduction

Human–robot interaction (HRI) is a multidisciplinary field which studies interactions between people and robots, which may help humans in complex tasks, reducing the workload and increasing safety conditions. ¹ Approaches based on HRI are focused on designing and evaluating the interaction among humans and robots to accomplish an activity in a collaborative manner. ² The design of HRI strategies is still a difficult problem, in part for the human factors complexity. ³ Several HRI approaches have been studied for different problems such as: vehicle navigation,^4,5 humanoids, ⁶ rescue robots, ⁷ assistant robots, ⁸ collaborative robots, ⁹ and among others. However, HRI is a relatively new research field which is still under development; therefore, many of its guidelines and design criteria may depend on the context and application taxonomy. ¹⁰ Collaboration is one of the most important aspects of HRI. Since humans and robots have their own capabilities and limitations, they can complement their skills to achieve certain task, which may impact positively on its efficiency and flexibility. Moreover, can HRI help to tackle difficult problems which cannot be accomplished if humans and robots work separately. ¹¹ There are several approaches that have used HRI methods for vehicle-driving applications, for example, using a robotic architecture as an accident-prevention safety system intended to give information and prevent if a risky situation arises. ¹² The research carried out by Lugano ⁵ is focused on virtual assistants and studies their role and functions for automated vehicle applications. Other works have studied collision avoidance among robots and humans using extrapolation of human intentions and optimization methods. ¹³ In a previous work by Schaefer et al., ¹⁴ a shared situation awareness architecture is studied, which can aid users to reach less dependence on manual control. Although those are useful methodologies, the study of HRI systems for driving tasks based on human cognitive parameters is still a challenging topic. Moreover, the autonomy level based on a natural interaction between a driver and the vehicle still needs to be analyzed properly.^3,15,16 In this context, this work uses several human cognitive parameters during vehicle driving to determine when the human is capable of driving properly. For this, cognitive parameters such as sleepiness, distraction, and stress should be considered.^17,18 Those are important parameters to consider, and the lack of cognitive skills is one of the main factors related with traffic accidents in many countries such as Chile, United States, and France.^19–22 To design an online algorithm capable of obtaining cognitive parameters such as sleepiness and distraction levels during driving tasks is challenging due to problems such as illumination changes and high demanding computing times. In addition, how to use cognitive parameters with an HRI strategy during driving is a key factor to be analyzed for safety purposes. The aim of this work is to develop an online algorithm capable of detecting cognitive parameters such as sleepiness and distraction levels, along with the effects related to not wearing glasses. Additional cognitive measures such as stress level and quality of sleep are also considered as prior information. Then, an HRI strategy is proposed to manage cooperatively driving tasks based on the human cognitive state, allowing the system to limiting the driver control over the vehicle if the human cognition is poor.

Previous work

In this section, a brief review is made about previous work related to the detection of cognitive parameters under different illumination changes. In addition, an implementation of a sleepiness-detection system for a vehicle using a cell phone camera is reviewed. Such analysis provides background and several guidelines to consider for the proposed work.

Selecting a cognitive parameter-extraction algorithm

A cognitive parameter-extraction algorithm was developed, which can work under illumination variations. The algorithm was tested and compared with other approaches in previous work of Vasconez and Auat Cheein. ²³ Processing times were considered during the eye status–detection procedure, which needed to detect short, medium, and long eye blink (71–100 ms, 101–170 ms, and 171–300 ms, respectively). ²⁴ In addition, it was realized that a comparison of the computing times and recognition performance of several illumination normalization methods can be found in the research of Struc. ²⁵ The Yale B images were used to validate the classification efficiency of such algorithms. ²⁶ Results showed that using independent component analysis (ICA), dual-tree complex wavelet transform (DT-CWT) with hard threshold criteria, and Euclidean distance to extract eye features was a viable solution for classifying cognitive parameters reaching classification rates on the Yale B images from 90% to 96% for different lighting levels.^23,25 Then, Haar cascade–based architecture proposed by Paul Viola and Michael Jones ²⁷ was implemented as a face-and-eyes localization method, which allowed online classification to obtain eye blinking information. Such architecture was able to achieve fast enough processing times, and efficient localization and classification of the human face and eye characteristics in real environments. The algorithm was capable of obtaining classification rates for eye blink features from 84% to 90% on 0.0552 s. A normalization of such results from 0 to 1 was carried out for the classification results and the processing times for several lighting normalization approaches, which is illustrated in in Figure 1.

Figure 1.

Normalized face classification results tested on Yale B database ²³ (see Appendix 1).

Sleepiness-detection system

In previous work of Vasconez and Auat Cheein, ²⁸ an HRI system was implemented to control a vehicle’s maximum allowed speed based on the driver’s sleepiness level. Such algorithm was tested using the online illumination invariant eye blink–detection system used by Vasconez and Auat Cheein, ²³ along with the percentage of eye closure (PERCLOS) measure and the eye blink frequency. Such measures provides valuable information to infer the driver’s sleepiness level.^19,29 The interface developed to detecte drowsiness is illustrated in Figure 2, and an image of the experiment is shown in Figure 3, which was tested for very sleepy, moderate sleepy, and awake drivers. For this, video frames were obtained from a cell phone camera that was connected to a computer using IP-webcam android application. The experiments were tested under different lighting conditions inside a Renault Twizy. One of the main ideas for developing such algorithm was that this approach could be used in the near future for every driver who has a cell phone camera that can be attached in the car each time the driver needs it. The experiments showed that even changing slightly the lighting conditions over the face from 60 to 3600 luxes in a controlled environment, the eye blinking can be detected, obtaining performances from 84% to 90% and fast processing times, up to 0.0552 s. Finally, several HRI strategies based on the sleepiness level were proposed, which could improve safety during driving tasks. Further details are presented in another study by Vasconez and Auat Cheein. ²⁸

Figure 2.

Online sleepiness-detection interface: (a) video image, (b) driver status, (c) eye status, (d) eye-detection results, (e) eye feature result. ²⁸

Figure 3.

Online sleepiness detection inside a vehicle under different illumination levels. ²⁸

Proposed system architecture

Although the systems reviewed in the previous sections^23,28 can be a cheap solution to detect sleepiness during day under limited illumination changes, they cannot be used at night, and it has problems detecting closing eyes, and eye states if drivers are using glasses or sunglasses. Additional problems arise due to the variations in face pose and distance and background objects. Finally, the algorithm proposed by Viola and Jones ²⁷ works properly for face feature localization problems. However, several classes from the scene are required, for example, the objects which the driver uses that may distract he or she, the use of glasses, sunglasses, and among others.

The motivations of this work are summarized as follows: (1) to obtain an online sleepiness-detection system using convolutional neural networks (CNNs) that is capable of detecting face cognitive parameters during day and night, even if drivers are using glasses or sunglasses; (2) to implement a distraction-detection system using CNN based on semantic human–object interaction (HOI); (3) to analyze other cognitive parameters such as sleep quality, stress level, and the adequate use of glasses; (4) to propose HRI collaborative strategies to command the driving system of a Renault Twizy vehicle based on human cognitive parameters which can be detected online (sleepiness, distraction, and use of glasses) or using prior information (sleep quality and stress level).

Sleepiness-detection system using CNN

In last years, deep-learning approaches have become a useful method for image-recognition and object-detection applications. Some deep learning approaches have managed to approach or even surpass the performance of humans in certain tasks. ³⁰ In particular, CNNs are capable of obtaining better results at image classification and detection problems than various traditional computer vision methods. ³¹

Several efficient and fast detection methods, which can localize and classify different classes on a single image, have been developed in recent years.^32,33 Among the object-detection methods that achieved promising results, the following can be mentioned: Fast R-CNN, ³⁴ Faster R-CNN, ³⁵ single-shot multibox detector (SSD), ³⁶ and YOLO. ³⁷ However, most of these methods are optimized for accuracy, and they often rely on models which are too slow for practical usage. Thus, a speed/accuracy trade-off decisions are necessary depending on the applications to obtain desired results, which depend on factors such as which feature extractor to be used, the input image sizes, and among others. ³³

In this work, TensorFlow object-detection aplication programming interface (API)^33,38 is used to detect objects and features related to cognitive factors inside a Renault Twizy vehicle. In this work, the SSD model is trained as the meta-architecture and MobileNet as the feature extractor. Detailed information about architecture, hyper-parameter tuning, and model details can be found in Huang et al. ³³ and Howard et al. ³⁸ MobileNet feature-extractor model was chosen because it is faster than others models, such as Inception V2, which aims to obtain fast processing times for online applications. Initial weights are imported from a previous training on COCO data set, which allows training with fewer images to obtain high performance in detection results.^33,38,39 The training set used contains 840 images from seven subjects which were taken inside a Renault Twizy vehicle and 500 to test the algorithm performance. The objects in the images were labeled using LabelImg software. ⁴⁰ An infrared (IR) camera was positioned in the vehicle to detect face features during day and night driving tasks. The image size used was 480 × 270 pixels. The algorithm is capable of detecting eight object classes, which are: face, open eye, closed eye, head, hand, glasses, sunglasses, and cell phone. Each algorithm iteration takes 0.0604 s using a Nvidia Quadro M4000 video card processor, which allows detecting short, medium, and long eye blinks. Based on the detected eye features, the PERCLOS measure and eye blink frequency were computed to detect driver sleepiness.^19,29 The illumination levels were between 2 to 4500 luxes of artificial light on an indoors controlled environment.

Distraction-detection system using CNN

Several studies reveal that cell phone use while driving can cause traffic signals to not be detected, slower reaction times, and less attention to the relevant context associated with driving, which is directly related to traffic accidents.^41,42

To tackle the problem of inferring distraction state for a driver, an action–recognition approach based on semantic representations is used to obtain inherent driver activity information.^43,44 Ziaeefar and Bergevin ⁴³ used semantic space to classify semantic approaches, which studies human characteristics related to their activities such as the body parts, objects, scenes, and attribute features. Based on those features, the semantic methods are categorized as follows: approaches based on body parts such as pose or poselet, HOI, scenes, and attributes. In particular, HOI analyzes the existence of particular objects in a scene and can provide valuable information to recognize a specific human activity. The configuration between these objects and humans can indicate how they are interacting. ⁴⁵ Recognizing and characterizing the objects is useful to determine the human activity.^43,46 In this work, we use HOI semantic representations. The relevant recognized objects are head, hand, and cell phone. Based on those objects detected inside the vehicle, it is possible to infer when a human is holding a cell phone, and if such action lasts, a certain amount of time (in this work, 10 s), then the driver is considered to be distracted. The trained CNN algorithm uses the same SSD as the meta-architecture and the MobileNet feature-extraction model explained in the previous section and by Huang et al. ³³ and Howard et al. ³⁸ Three possible categories of human distraction state related to cell phone are considered as follows: not distracted, holding the cell phone, and distracted. The IR camera is specially positioned in the vehicle to detect such objects, which allow for detecting during day and night driving. ⁴⁷

Driver cognitive parameters information

Different cognitive and behavioral parameters of drivers have been analyzed in the literature, such as sleepiness, ⁴⁸ stress level, ⁴⁹ distraction level, ⁵⁰ and among others. The combination of several cognitive parameters can provide additional information to increase the performance of accident prevention systems. In particular, the use of combined cognitive parameters with HRI strategies for cooperative vehicle driving tasks can help to reduce traffic accidents. The analysis of the cognitive parameters that are taking into account for the proposed HRI architecture is presented as follows.

Sleepiness-detection criteria

Sleepiness is one of the main parameters to consider for vehicle accident–prevention systems.^19,22 Sleepiness is a complex cognitive parameter to detect because it needs fast processing times and robustness against illumination and pose changes. Different studies agree that PERCLOS measures along with eye blink frequency data can contribute with reliable data to infer the driver’s sleepiness level.^19,29 PERCLOS measures are described as the interval of time in which the eyelids are 80% or more closed over the pupil. ¹⁹ Such measures can be extracted each 60 seconds and usually are accurate enough after five or more minutes, since the driver begins to be monitored. The PERCLOS parameter can be obtained using the following equation

$$f_{Perclos}=\frac{n_{close}}{N_{total}}\times 100\%$$

(1)

where the amount of frames with the eye closed is $n_{\mathrm{close}}$ , and $N_{\mathrm{total}}$ is the total amount of frames for the total monitoring period of time. Moreover, blink frequency measures can be used alongside PERCLOS to enhance sleepiness inference accuracy. According to Zhang et al. ¹⁹ and Wierwille et al., ²⁹ when the driver is not sleepy, blink frequency reachs between 15 and 30 (times/min), and when the driver’s sleepiness level is high, this variable is less than 7 (times/min). Blink frequency equation is presented below

$$f_{Blinkfreq}=\frac{n}{N}\times 100\%$$

(2)

where the amount of eye blinks is n, and the total number of frames during a time interval (in this case is 1 min) in which the driver is monitored is N.

As stated by Wierwille et al., ²⁹ the threshold for PERCLOS values can be obtained from a multiple-regression method which uses observable measures, such as steering, seat movement, lane, heading, brain activity (electroencephalogram (EEG) measures), activity trackers data, and heart rate parameters. The use of PERCLOS combined with blink frequency measures can be a key tool develop a sleepiness-detection system for driving as is mentioned in Zhang et al. ¹⁹ Based on such guidelines, the PERCLOS thresholds values that we use are $t_1=0.25$ and $t_2=0.15$ . In the case when PERCLOS value is higher than 0.25, the driver cognitive state is labeled as sleepy; whereas, when PERCLOS value is less than 0.25 but higher than 0.15 and the blinking frequency is no more than 10, the person cognitive state is labeled as moderate sleepy. In any other case, the person cognitive state is labeled as awake.^19,29

Sleep quality

Sleep quality information is a key concept required to reduce human accidents while driving. This parameter can be used as prior information for the HRI system to evaluate the cognitive state of the driver. A brief analysis of the main concepts of sleep quality parameter and the considerations used in this work for the HRI system is presented as follows.

Sleep quality is an important cognitive parameter widely used in clinical treatment because poor sleep problems are related to several sleep and medical disorders. ⁵¹ However, there are several applications which could be benefited from sleep quality information such as safety during machinery operation and driving. ⁵² Sleep information can be obtained based on objective measures using devices such as a digital EEG, electromyogram (EMG), electrooculogram (EOG), and electrocardiogram (ECG), ⁵³ and among others. Subjective measures were also used such as the Pittsburgh Sleep Quality Index (PSQI), which is still a reliable, quick, and easy method to perform sleep-quality assessments. ⁵⁴ However, these methods require to be performed in a clinic or laboratory by sleep specialists. For that reason, it is difficult to automatically obtain daily information of sleep quality, which is what our HRI system requires to evaluate the cognitive state of the driver. In this context, several wearable activity-tracking devices developed in recent years have made it possible to track the sleep quality with reasonable performances.^51,55 These devices usually work analyzing muscular movements (actigraphy) and heart rate information. Some of the latest activity-tracking devices can estimate sleep quality information based on quantitative aspects of sleep, such as sleep duration, deep sleep, rapid eye movement (REM) sleep, and among others.^51,55 These parameters are related to the sleep stages that a person experiments when sleep, which are: stage 1 and 2 (light sleep), stage 3 (deep sleep), and REM sleep.^56,57 Stage 1 is the transition between waking and sleep states. The brain waves that predominate in this stage are alpha and theta. Stage 2 presents a rapid brain activity known as sleep spindles combined with sleep structures known as K complexes. Stage 3 (deep sleep) is the most restorative stage of sleep. In this stage, the body starts a bone and muscle regeneration process, and the immune system is reinforced. The predominant brain waves In this phase are the delta. During stage 4 (REM sleep), the brain is almost as active as when the person is awake. This stage is usually characterized by REMs, low muscle tone, and rapid low-voltage EEG. ⁵³ This is the stage of sleep that is associated not only with dreaming but also with memory consolidation, learning, and creativity. In other words, this is the stage that energizes the cognition-related capabilities.^55,56 A sleep cycle is the progression through the stages of sleep, this is from stage 1 to stage 4 (REM) before beginning the progression again with stage 1. A complete sleep cycle takes an average of 90–120 min resulting in four to five cycles per sleep time during night. The first sleep cycles of the night have relatively short REM sleeps and long periods of deep sleep but later in the night, REM periods lengthen and deep sleep time decreases.^56,57

The amount of sleep varies depending on the person because every human has a biological clock which is aligned to day/night cycles. ⁵⁵ However, the National Sleep Foundation (NSF) recommends sleeping at least 7–9 h during the night for an adult aged 18–64 years. ⁵⁸ In this context, adults spend about 15%–20% of their total sleep time in deep sleep, and this percentage is reduced when people are older. ⁵⁵ It is similar with the amount of REM sleep, which takes about 20%–25% of the total sleep time. ⁵⁵ Therefore, measuring deep sleep and REM sleep for a normal night (7–9 h of sleep), considering those percentages would be a reasonable indicator of sleep quality.^55,57 However, if the total sleep time is reduced, the amount of deep sleep and REM sleep also decreases, which means that the overall performance of the person both physically and cognitively will be affected. For example, Wiesner et al. ⁵³ studied the effects-selective deprivation of REM sleep in persons. The results show the cognitive and emotional state the participants were affected when persons were deprived of REM sleep, presenting anxiety and irritability. ⁵³ Therefore, the emotional stability of the persons seemed to depend on REM sleep. The study also indicates that REM sleep intervenes in the consolidation of neutral and emotional memories. However, the results showed that only late sleep (second half of total sleep time) with enough REM sleep can improve the consolidation of emotional memories.

The above is one of the reasons why parameters such as REM sleep can be used as a metric for the evaluation of sleep quality, since they affect the cognitive and emotional state of people. For a fine adjustment of our HRI system, it could be considered sleep quality parameters only for the case where the driver sleeps between 7 and 9 hours of total sleep, since in these ranges, the recommended deep sleep and REM sleep parameters are known, which are 15%–20% of the total sleep time for deep sleep, and 20%–25 % of the total sleep time for REM sleep. However, the relation of these parameters and the driving performance of a person is still an open problem. In particular, when the sleep hours are reduced, for example, to 5 and 2 h of total sleep. For this reasons, it is necessary to use other strategies that allow us to use sleep-quality parameters that have been proved to affect human driving performances. In this context, it has been demonstrated that sleep deprivation (which affect the total sleep time and all the parameters therein) increases the risk of driving accidents related to human errors.^52,59,60 In particular, the partial sleep deprivation of young adult drivers (especially males) has been related to a much higher risk of a vehicular accidents. ⁵² Therefore, in this work, we use the sleep deprivation concept to obtain acquire sleep-quality information required to improve driving using the proposed HRI system.

Williamson and Feyer ⁶¹ have demonstrated that after 17–19 h of sleep deprivation, driver’s performance can be equal or worse than a driver with blood alcohol concentration (BAC) of 0.05%, and such value can increase up to 0.1% if the sleep deprivation is prolonged more. That is a key factor to consider since a BAC of 0.08% is considered a legally drunk person. ⁶² Such information directly relates to sleep deprivation with the BAC parameter, which is widely known to affect driving ability.^52,62 Considering such statements, the information about sleep quality in this work is related to the sleep deprivation parameters, which will be used as prior information to complement the online sleepiness-detection system. For this, an activity tracker watch with motion detector can automatically and continuously extract information about drivers sleep quality. Thus, a Garmin Vivofit watch is used to obtain the driver sleep-quality information, as illustrated in Figure 4. In this work, several considerations about sleep quality are proposed based on results shown by the NSF ⁵⁸ and Williamson and Feyer. ⁶¹ Thus, if the driver sleeps more than 7 h, he has high sleep quality that day, 5–7 h means medium sleep quality, and less than 5 h means poor sleep quality.

Figure 4.

Watch activity tracker—sleep-quality information. ⁶³

Distraction level

According to the National Highway Transportation Safety Administration (NHTSA), ⁶⁴ there are three types of distraction that can impair driver’s skills: visual, manual, and cognitive distractions. It is considered as a visual distraction when the driver performs a task that keeps his eyes away from the road. A manual distraction causes a driver to release the steering wheel using one or both hands to perform a task such as grasping a coffee mug or using a cell phone. Finally, cognitive distractions are related to any mental process that diverts the driver’s attention from driving.

Among the main sources of distraction that affect the performance of the driver are the use of cell phones, eating, or drinking, changing the radio station, and among others. Although a driver can be affected by multiple distractions, using a cell phone is one of the main causes of road accidents. Usually, the driver holds a cell phone near to the ear to make or receive a call or manipulates the cell phone manually for texting or using social networks. In particular, texting is one of the most dangerous activities while driving, since it constitutes a visual, manual, and cognitive distraction at the same time. Most countries have implemented laws to prohibit texting while driving. ⁶⁴

In this context, we propose the use of the convolutional object-detection method based on SSD described in previous sections to detect when a driver is using a cell phone, which is directly related to visual, manual, and cognitive distraction. For this, we use a semantic action-recognition strategy based on HOI information to infer human activity. Semantic action recognition can provide meaningful information that can help to reduce vehicle accidents in highways by detecting driver’s distraction.^41–43 In this work, we propose that detecting hands and a cell phone classes is related to the action “holding a cell phone,” which may indicate that the driver is texting or making or receiving a call. Since the use of a cell phone for any activity implies a visual, manual, and cognitive distraction for the driver, we consider the driver to have a low level of distraction if he holds a cell phone for less than 10 s (watch the time on the cell phone, an incoming message notification, and among others.). In addition, if the driver holds a cell phone for more than 10 s, then the person is considered very distracted (texting, making or receiving a call, and among others). The reference time of 10 s has been selected empirically for this experiment.

Stress level

Nowadays, driver stress is a problem that is gaining importance. It can cause a decrease of cognitive skills, which is related to poor driving performance and a high likelihood of traffic accidents. ⁴⁹ Effects of stress on driving performance may depend on previous cognitive state of the driver, the complexity of current traffic environment, and among others. Several approaches that study stress effects use sensors (e.g, ECG and EMG) to study natural episodes of driver stress in public road highways. ⁶⁵ In this work, drivers stress information is obtained from heart rate variability (HRV) measures which can be obtained using a cell phone built-in heart rate sensor to measure levels of stress on the body. ⁶⁶ Thus, the stress level is considered as high, moderate, and low, which can be obtained from the Samsumg S6 health application or similar. For a real scenario, the HRI system could receive heart rate measures by synchronizing with the cell phone when the driver gets inside the vehicle, which may allow for obtaining the latest stress measure. The HRI system could also use an alarm or a visual indicator to request the driver to perform the stress measurement the moment he enters the vehicle. Another option could be the use of any available commercial steering wheel that monitors the heart rate using haptic sensors, which could be the best choice to obtain online measurements.

Use of glasses

Studies suggest that driving without glasses when the driver require them is illegal and dangerous.^67,68 If the driver has a visual impairment and do not use glasses, it is an indicator that he or she is not in optimal conditions to detect traffic signals, people, or damage on the road.^69,70 Drivers need to have periodic eye tests to prevent accidents from visual impairment. ⁷¹ The non-use of glasses is a punishable offense that receives monetary fines and reduction of driver license points. ²² In this work, the HRI collaborative system is able to detect whether a driver is wearing glasses or sunglasses. Based on this information, the system can take preventive measures to try to reduce accidents, such as reducing the current speed to a limit allowed by the HRI system. For this, the system should have prior information about whether the driver depends on the use of glasses or not, which can be set manually.

Sleepiness- and distraction-detection results

The visual results of the sleepiness-detection architecture can be visualized in Figure 5 for awake and sleepy states, respectively (very sleepy state can also be detected). The object-detection results for each of the eight classes can be observed in Table 1. For each of the 500 images, several algorithm performance indicators were obtained, which are: true positives (TP), false negatives (FNs), false positives (FPs), F1 score (F1), and average precision (AP). It can be observed that open and closed eyes can be detected with high-precision rates, between 76% and 81%, respectively. This is an important result, since a considerable number of tests were performed using glasses and sunglasses (see Table 1). In addition, it can be observed that the object-detection processing time required to make an inference (0.0552 s) is fast enough to detect eye blinking. In the case of cell phone, head and hand detection, it can be observed that the number of FNs and FPs are high. It was observed that these problems are related to occlusion problems. Nevertheless, the proposed approach still obtain decent detection performances for these classes. In general, the object detection results can be improved with a large data set, which also help to generalize the algorithm to make it able to detect features of people that are not in the data set.

Figure 5.

(a) Online sleepiness detection—open eyes detected using sunglasses, (b) online sleepiness detection—closed eyes detected using glasses.

Table 1.

Object-detection results.

Classes	TP	FN	FP	F1 (%)	AP (%)
Face	240	11	0	98	99
Open eye	127	63	0	80	76
Closed eye	43	17	0	83	81
Sunglasses	48	7	4	90	92
Glasses	34	6	1	91	93
Hand	69	32	24	71	77
Head	180	13	43	87	94
Cell phone	181	53	17	84	85

TP: true positives; FN: false negatives; FP: false positives; F1: F1 score; AP: average precision.

As explained in previous sections, when the system detects hands and a cell phone, a holding cell phone message is shown, and if this last for less than 10 s, then the driver is considered to have a low distraction level. If such a state takes more than 10 s, then the driver is considered as very distracted. The driver-distraction state results are illustrated in Figure 6 for not distracted and distracted states, respectively.

Figure 6.

(a) Online distraction detection—not distracted driver and (b) online distraction detection—distracted driver.

Validation of the sleepiness-detection algorithm

Following the criteria mentioned in previous sections and by Teran-Santos et al. ⁵⁹ and Bloomfield et al., ⁶⁰ there is a strong association between sleep less than7–5 h and have cognitive skills similar to a person with BAC of 0.05%. Measurements were obtained from three test subjects (S1, S2, and S3), two men and one woman, while they use city car driving simulator. ⁷² The mean age of subjects were 29 years, and all were doctorate students. None of the subjects have previous exposure to the sleepiness-detection system, and they do not have knowledge about how the PERCLOS or blink frequency parameters works because that might cause a bias in the results, for example, if they are aware that more blinking could mean more sleepiness levels. Each one of the subjects have a different sleep quality, which is 2, 5, and 8 h of sleep, respectively. The number of hours each user slept was verified using an activity tracker watch to verify that the sleep time of each of them is correct. Eye behavior measurements were taken for 30 min for each test subject while using the driving simulator, and five random samples of 1 min were taken after 15 min of driving to be analyzed using the sleepiness-detection algorithm. This consideration is important because the results during the first 15 min showed that most of the samples of the sleepy subject (S3) gave the state of the subjects as awake (FNs). These results were considered as not reliable which is in accordance with Zhang et al. ¹⁹ and Wierwille et al., ²⁹ which indicates that PERCLOS values usually are reliable enough after at least six or more minutes of continuous eye features measures. Nevertheless, the results during the next 15 min are illustrated in Table 2. It can be observed that the sleepiness-detection algorithm can detect awake, sleepy, and very sleepy cognitive states from a 60% to an 80% confidence for drivers that did not have good enough sleep quality. It also can be observed some important FN results such as the case of the subject S3, which detects 20% of the time an awake state when it was very sleepy, or S2 that detects 40% of the time an awake state when the driver was sleepy. This is in part because PERCLOS requires some time to obtain reliable results.^19,29 Moreover, the detection error of the proposed algorithm might also influence the detection of eye blinks, which may be an important factor to affect the result of the sleepiness-detection system. Finally, there are many factors that have yet to be studied to improve the sleepiness level, such as the influence of the driving environment on the driver because a very quiet and relaxing environment could make the driver fall asleep more easily; the influence of the age of the driver because older people may have different sleep requirements and cognition capabilities and among others.

Table 2.

Sleepiness-detection results after 15 min of monitoring.

Subject	Slep time	Awake	Sleepy	Very Sleepy
S1	8 h	4	1	0
S2	5 h	2	3	0
S3	2 h	1	0	4

Vehicle instrumentation and traction architecture

To implement the HRI system based on human cognitive factors, it is necessary to monitor and control vehicle movement variables. In this work, the vehicle control system is formed by the acceleration module, the sensing module, and the breaking module. ²⁸

Currently, vehicles have a large number of sensors that provide multiple data for both the user who drives the vehicle, and for the internal systems that owns the car. Thus, data such as speed, position of the pedals, and all those necessary to perform a speed control, are supplied to the control units through a communication interface called CAN-BUS. The commercial electric vehicle used in this work to test the proposed architecture is a Renault Twizy, which has an autonomy of 100 km and can reach up to 80 km/h in urban areas. This electric car posses an accelerator system that depends on an electronic resistive device. In this stage, the signals from the connection between the acceleration pedal and the car electronic controller is intervened using CAN-BUS protocol, which allows to monitor and control acceleration and velocity parameters. To read the data of the CAN-BUS communication bus, it is first necessary to physically connect to this bus. This has been performed through an OBD-II to DB9 link, and this, in turn, is linked to the Sparkfun OBD-II UART board. ²⁸ Such device allows the link with the vehicle using OBD-II protocol. To compute the Sparkfun OBD-II UART device information, an Arduino Mega 2560 was programmed. Then it was necesarry to use an FTDI component to link the Sparkfun OBD-II UART board with a laptop. The OBD-II version used was ISO 15765-4, also known as CAN 11 bits 500 [Kbps]. The velocity and acceleration information of this system can be obtained each 100 ms. The architecture presented in this section is used to control the speed limits that will depend on the cognitive state of the driver. In the case of the electronic car braking module, it is not possible to perform an electronic control directly, since this system works on the basis of a hydraulic mechanism. For this, an ultramotion actuator is used to control the brake pedal externally. This stage of the architecture can be used in emergency cases where the driver is not with sufficient cognitive ability to react in the required time by itself, such as the case of a high sleepiness level. The Renault Twizy vehicle is shown in Figure 7, and the braking system of the electric vehicle is based on the ultramotion actuator in Figure 8.

Figure 7.

Renault Twizy vehicle.

Figure 8.

Vehicle brake system using ultramotion actuator.

Criteria of maximum allowed speed

The Chilean National Traffic Safety Commission (CONASET) ²² states that fast driver reaction times are equal or more relevant as driving at low speeds to reduce traffic incidents. Moreover, several works demonstrate that sleepy drivers have reaction times as poorly as legally drunk persons, which are usually involved in driving accidents. ⁶¹ For this reason, it is recommended to drive at moderate speeds in urban roads. In this context, CONASET proposes that the speed level should be no more than 60 (km/h) in urban roads. The probability for a pedestrian to survive an accident depending on the speed of the vehicle can be observed in Table 3.

Table 3.

Probability of a pedestrian to survive a car accident based on vehicle speed.

Survival probability	speed limit (km/h)
High	<30
Moderate	45
Low	>60

HRI collaborative driving strategies

In this work, the cooperation between the driver and the vehicle uses the following HRI scheme

$$I={\lambda }_{\mathrm{Tot}}H+(1-{\lambda }_{\mathrm{Tot}})R$$

(3)

$${\lambda }_{\mathrm{Tot}}={\lambda }_1+{\lambda }_2+\cdots +{\lambda }_n$$

(4)

in these equations, ${\lambda }_{\mathrm{Tot}}$ is a coefficient that indicates the cognitive capacity of a human driver $(H)$ . On the contrary, $(1-{\lambda }_{\mathrm{Tot}})$ indicates how much the robotic vehicle $(R)$ should support the human driver. The ${\lambda }_{\mathrm{Tot}}$ parameter changes over time and can be obtained from the driver cognitive parameters presented in previous sections, which are also illustrated in Table 4.

Table 4.

Coefficient $\lambda$ values for each cognitive factor.

Cognitive parameter	$\lambda$	$\lambda$ value
Awake	${\lambda }_1$	0.2
Sleepy		0.1
Very sleepy		0
High sleep quality	${\lambda }_2$	0.2
Moderate sleep quality		0.1
Poor sleep quality		0
Not distracted	${\lambda }_3$	0.2
Low distraction level		0.1
High distraction level		0
Low stress level	${\lambda }_4$	0.2
Moderate stress level		0.1
High stress level		0
Wearing glasses	${\lambda }_5$	0.2
Not wearing glasses		0
Total	${\lambda }_{\mathrm{Tot}}$	1

The sleepiness level is obtained online using the sleepiness-detection system proposed, which detects eye blink behavior to provide the PERCLOS and eye blink frequency information by applying equations (1) and (2). Thus, as illustrated in Table 4, the parameter value is ${\lambda }_1=0.2$ if the driver is awake, ${\lambda }_1=0.1$ if the driver is sleepy, and ${\lambda }_1=0$ if the driver is very sleepy.

The sleep quality parameter is obtained offline using the activity tracker watch. Thus, as illustrated in Table 4, the parameter value is ${\lambda }_2=0.2$ if the driver has high sleep quality (at least 7 h of sleep), ${\lambda }_2=0.1$ if the driver has moderate sleep quality (5–7 h of sleep), and ${\lambda }_2=0$ if the driver has poor sleep quality (less than 5 h of sleep).

The distraction level is obtained online using the proposed object-detection algorithm and the HOI information related to hands and cell phone detection. Thus, as can be observed in Table 4, the parameter value is ${\lambda }_3=0.2$ , if the driver is not distracted (not holding the cell phone), ${\lambda }_3=0.1$ if the driver is holding the cell phone for a small amount of time (low distraction level), and ${\lambda }_3=0$ if the driver distracted (holding the cell phone for more than 10 s).

The stress level is obtained offline using the cell phone’s built-in heart rate sensor to measure levels of stress. However, it also can be obtained online using a commercial steering wheel that monitors the heart rate using haptic sensors. As is shown in Table 4, the parameter value is ${\lambda }_4=0.2$ if the driver has low stress level, ${\lambda }_4=0.1$ if the driver has moderate stress level, and ${\lambda }_4=0$ if the driver has high stress level.

Finally, the use of glasses parameter is obtained online using the proposed object-detection algorithm. As is presented in Table 4, the parameter value is ${\lambda }_5=0.2$ if the driver is using glasses (if he require its use) and ${\lambda }_5=0$ if he does not use them.

As it can be observed in Table 4, ${\lambda }_n$ parameters vary over time depending on the cognitive parameters of the driver. In addition, considering the speed that is related to pedestrians fatalities illustrated in Table 3, a maximum allowed speed criterion that depend on ${\lambda }_{\mathrm{Tot}}$ is shown in Table 5. If ${\lambda }_{\mathrm{Tot}}$ value, which is related to driver cognitive capacity, is high, it means that the driver is capable of reaching the desirable reaction times and is able to reach the speed of 60 (km/h). If the driver cognitive abilities are poor, then the vehicle maximum permissible speed decrease as illustrated in Table 5. For example, if the driver is sleepy, has a moderate sleep quality, and a high stress level, then ${\lambda }_{\mathrm{Tot}}=0.6$ , and the maximum speed level allowed for the robot is 50 (km/h) as indicated in Table 5. In general, depending on the parameter ${\lambda }_{\mathrm{Tot}}$ , the vehicle can increasingly gain control over the driving decisions and should try to keep the driver safe, such as use the brake or reduce the speed. Thus, the collaborative autonomy level of the vehicle change online depending on the driver cognitive factors. Such considerations help to reduce the accident likelihood, which can save drivers and pedestrians lives.

Table 5.

Maximum allowed safety speed depending on the cognitive factor of the driver.

${\lambda }_{\mathrm{Tot}}$ value	Maximum allowed speed (km/h)
$1\le {\lambda }_{\mathrm{Tot}}\le 0.8$	60
$0.8<{\lambda }_{\mathrm{Tot}}\le 0.6$	50
$0.6<{\lambda }_{\mathrm{Tot}}\le 0.4$	40
$0.4<{\lambda }_{\mathrm{Tot}}$	30

In future work, the vehicle might not only support the person in the driving tasks but also could interact with the driver to notify their cognitive status and proposes sharing driving tasks if a risky situation arises. If the person do not want to share control over the vehicle, the robotic system can make suggestions to maintain if possible the coefficient ${\lambda }_{\mathrm{Tot}}$ in levels where the person can still drive properly. In the case of sleepiness or stress, the robotic vehicle should interact or communicate with the person by an appropriate method or interface so that it can stay focused on the task, or stay awake with a suitable type of music and temperature, which might have a positive impact on the reaction times. The ribituc vehicle should also change the position of the seat or recommend the driver a place nearby to rest or have a coffee.^73,74 The same strategy apply to distraction using cell phone, where the robot can suggest the driver to stop the vehicle until the call is finished or decrease velocity.

Conclusion

In this work, an HRI system based on human cognitive parameters for cooperative driving tasks was implemented to increase driver safety. The system was capable of recognizing cognitive features online such as sleepiness and distraction in an artificial illumination environment (2–4500 luxes) using an IR camera, which can permit day and night driving applications. The system was also capable of detecting eye features even if the driver wears glasses or sunglasses. The benefits of the proposed approach compared with other architectures used in previous works were analyzed. The proposed detection algorithm for sleepiness and distraction inference were trained and tested using seven participants. For this, the use of SSD as meta-architecture and MobileNet as feature extractor allows reaching a balance between speed (0.0604 s) and performance (AP from 76% to 99% depending on the class) for this application. The sleepiness-detection system was tested for three subjects with different sleep-quality levels, showing that the system was capable of recognizing sleepiness states such as awake, sleepy, and very sleepy after 15 min of driving. However, more test participants should be considered to obtain statistics for future research, and a large data set is also required to generalize the detection algorithm to be used for any person. Finally, the development of a systems which control brake and acceleration were proposed to command a Renault Twizy vehicle, which alongside a proper HRI strategy that is based on cognitive parameters, can help to mitigate the occurrence of accidents and increase safety conditions.

Human–robot collaborative driving methods based on driver cognitive parameters were studied for a Renault Twizy vehicle, which can provide helpful guidelines to increase safety in urban areas. Thus, depending on the human cognitive skills that can be detected online (sleepiness, distraction, and use of glasses) and others as prior information (sleep quality and stress level), the HRI system can increasingly take control over the vehicle to compensate human cognitive deficiencies.