Abstract
We developed a robotic service system by applying a user model-based application for supporting daily life. Our robotic service system is designed to provide appropriate services to users depending on their needs; thus, we applied a user model-based application, which can help to select and filter user information for our system in order to provide appropriate services to users.
For achieving the design goal, we first collected motion logs. We collected the motion data from acceleration sensors that were carried by users for creating motion logs. Life logs could then be created from these motion logs. Finally, we constructed user models by mining life logs so as to enable our robotic service system to support users. We discuss different service modes of our robotic service system. The research shows how to determine different user models from the motion data and how to apply these models and log data to support not only individuals but also families and communities.
1. Introduction
Recently, daily life has become more convenient due to developments in robotics. Robotic systems support our daily lives in many ways, but the services that such systems provide are usually for the masses and seldom for people's individual needs. Everyone has his/her own unique personality and different needs, thus services should be provided to people according to their needs. In other words, the future development trend among applications is personalization. We developed a robotic service system based on the concept of personalization. We obtain different user models by mining people's life logs and we provide different services to users depending on the user models based on their life logs. Therefore, we focused on applying user models constructed by mining life logs so as to provide different services to users in order to achieve this development goal. Moreover, we want to directly obtain a person's data; thus, we used sensors carried by users in order to obtain their life logs.
A system is able to use several applications and these applications include functions that help to achieve the system development goal. An application can be a program or a group of programs. A program can be classified as low-level, which includes an input device (for example, sensors) or as high-level, which includes data processing (for example, databases and data analysis programs). Furthermore, a low-level program's functionality and the data precision from low-level equipment are more advanced. Therefore, this improvement allows us to apply low-level programs to a wider area (for example, to collect motion data). On the other hand, high-level programs have also improved; thus, they are able to provide more powerful data processing capability in offering various services to users. Therefore, a robotic service system that collects original data from low-level programs and uses high-level programs to process the data will result in a more powerful system. Low-level programs can help to accurately collect the data that we need and high-level programs can aid in the handling data and providing services to users in order to achieve our goal.
We applied a user model-based application that mining life logs in developing our robotic service system. We collected life logs from motion data, which was directly obtained from sensors carried by users. Because motion data cannot be directly applied, we use fuzzy inference and fuzzy membership functions to analyse the original motion data so as to obtain motion logs [1]. Motion logs can help obtain life logs, which can be mining to construct user models. After obtaining user models, the system will be able to provide services to users depending on the user models and provide services to not only individuals but also to families and communities.
2. Background
User models can be used to help select or filter user information (for example, basic data, purchase history and use history) and they are frequently used in web intelligence. User models are used widely. A user model may be a specific module with user knowledge, which is part of an interface, or it may refer to the style of interaction that an interface engenders without reference to specific user knowledge [2]. The usages are different depending upon the design goals.
There are many related studies that have used user information as a basis for providing services. Nakamura et al. [3] designed a recommendation system that applies personal attributes and their preferences as basic recommendation data. Mizutome et al. [4] developed a system that depends on users' life logs to suggest ways to manage electric power. These studies described the concept of constructing user models. Furthermore, there have been many studies using user models for services. Joung et al. [5] described a survey that focuses on defining a general user model to represent user information and proposed a user model for capturing user information for personalization applications. Jennings and Higuchi [6] proposed a user model designed for browsing. It provides personal news services to those who are uncertain what information they want. These investigations use personal attributes, preferences, life logs, user's basic data, user information and environmental information to construct user models. However, we find that these models can obtain usable user models, but they usually use indirect methods to collect data from people.
Because of the reason described above, we want to collect the needed data directly from people, which will involve obtaining motion data from sensors carried by users, in order to construct user models for our robotic service system.
3. System structure
Our robotic service system is composed of three subsystems: motion detection, data mining and service provision. Motion detection detects and obtains the original motion data to obtain motion logs. Data mining analyses motion logs to obtain life logs and mining life logs to construct user models. Service provision provides services to people depending on their needs, with a service range from individuals to families and communities. To explain the design concept of our robotic service system, we use a health maintenance system for the elderly as an example. The system applies the user models to provide exercise suggestions, show daily life conditions and provide socializing information that could help elderly people to communicate with each other.
3.1 Motion detection
We automatically collected original data from the users. Because user models are constructed by mining life logs and because life logs can be obtained from motion logs, we used 3D acceleration sensors with ZigBee communication (designed by System Craft Inc.) and Actimarker EW4800 (designed by Panasonic Electric Works) to detect motion and calculate the amount of exercise (see Figure 1.). The 3D acceleration sensors with ZigBee communication can record 3D acceleration data 20 times per second within the valid range of the ZigBee network. A sensor on a user's body collects coordination data and sends the data to a receiving device. Then, using the fuzzy inference and fuzzy membership functions for the motion recognition [1], we can obtain motion logs from the motion data. Actimarker calculates the amount of exercise as metabolic equivalents (METs) per minute, which represents the intensity of physical activity and depends on body weight and exercise time to calculate the amount of calories burned. This measurement is defined by the Ministry of Health, Labour and Welfare, Japan [7].
The sensors used in our research. On the left are 3D acceleration sensors with ZigBee communication designed by System Craft Inc. On the right is the Actimarker EW4800 designed by Panasonic Electric Works
3.2 Data mining
For mining and constructing user models, we have to obtain life logs. A series of motion logs constitute the data of a person's actions. Through the action data and the location of where the motion takes place, we can obtain life logs. Then, for preparing the mining step, the life logs are processed by classification and normalization and by calculating the life log's specific gravity. Finally, the pre-processed data of life logs can be mining to construct a user model. We used the Waikato Environment for Knowledge Analysis (Weka) [8] developed by Waikato University as the data mining tool and chose clustering as the data mining method.
3.3 Service providing
Our robotic service system provides services to a user according to the user model. Because the service provision range is not only for one person but also for their family and community, the service concept will differ according to the user's role in the system. Moreover, we want to use the most intuitional method for providing services to users; thus, we designed two types of services to provide and apply them according to the user needs in our robotic service system. One type of service involves using a robot to communicate with a user and provide physical assistance while the other involves using a smartphone to show information in augmented reality. We used the robot ApriPoco by Toshiba and the iPhone 3Gs by Apple Inc.
4. Constructing the user model
The method for constructing user models by data mining includes two subsystems: motion detection and data mining. In this section, we discuss an experiment that involved applying a user model-based application and explain its process.
As described in Section 3.1, we used 3D acceleration sensors with Zigbee communication to detect motion data. We applied the motion detection method proposed by Ishiguro et al. [1]. As with their method, we applied a fuzzy inference method as the motion recognition algorithm and a fuzzy membership function into the system for recognizing motion. The recognized motion and the time taken to perform the complete the motion was then saved into a database. This data is the sub-data of the motion logs.
After collecting motion logs, we were able to pre-process data for the next mining step. Because we need life logs to use in the mining step, we analysed the data of a series of motion logs and the location of the motion occurrence to obtain life logs. Moreover, the calorie data also became sub-data of the life logs to help calculate the amount of exercise. After obtaining life logs, we began to pre-process them by classification and normalization and by calculating the life log's specific gravity so as to prepare for the next mining step. We divided the motion data into three grades of exercise intensity: heavy motion, moderate motion and light motion. Heavy motion consists of running, in which the METs*hour is more than 4. Moderate motion consists of walking, in which the METs*hour is between 3 and 4. Light motion consists of other motions, in which the METs*hour is less than 3. According to the recommendation by the Ministry of Health, Labour and Welfare of Japan [7], a person has to do 26 METs*hour exercise per week to maintain health and the effective exercise that may be included within the 26 METs*hour exercise per week described above should be more than 3 METs*hour. Thus, we depended on this definition to calculate the motion score. Because every effective exercise should be more than 3 METs*hour, we calculated the occurrence rate of heavy and moderate motion for determining the score of effective motion (Table 1). In this research, the occurrence duration was 1 hour. Because exercise intensity is different when a user performs a heavy motion or a light motion, we also focused on a person performing a heavy motion for one hour and determined the score by the result (Table 2). The scores of the occurrence rate of moderate and heavy motions and the score of the time it took a person to complete heavy motions are added to determine the total score. The total score and METs*hour (the amount of exercise) are the elements of the next mining step.
Score table for heavy and moderate motion occurrence ratios
Score table for time of motion in one hour
We used Weka [8] and the expectation maximization (EM) algorithm of clustering analysis as the mining method. The method shows the possibility of mining life logs to construct user models. Because our target was to construct a user model and provide appropriate services, we chose clustering analysis for constructing a user model from the user data. The EM algorithm is useful in finding a probabilistic model that depends upon unobserved latent variables [9]. Thus, we used the EM algorithm of the clustering analysis as the mining method and applied the clustering results to construct a user model.
We invited ten students to participate in our study. We collected 20 hours of motion and exercise data from the participants. One item of data includes one hour of motion and exercise data. The 20 items of data could be used to calculate the score and the amount of METs*hour for mining. We obtained three clusters (see Figure 2.). Cluster 1 includes that data where the METs*hour is about 126.9, and the score was over 3; Cluster 3 includes that data where the METs*hour is about 100, and the score was over 3; Cluster 2 includes that data not included in Cluster 1 or 3. Cluster 1 contains 3 items of data, accounting for 15%. Clustering 2 contains 11 items of data, accounting for 55%. Cluster 3 contains 6 items of data, accounting 30%. Based on the results, we designed three exercise levels: Inactive (Cluster 2), Moderate exercise (Cluster 3) and Active (Cluster 1). The user model could then be constructed depending on these results. This proves the possibility of using motion data and the amount of exercise to construct user models.
User model constructed by mining. After mining using clustering, we can obtain user models from users' life logs.
5. Service concept of the system
Our robotic service system provides services to users according to user models. Because the service targets include individuals, families and communities, the system will provide different services not only according to the user model but also according to the user's role in this system.
We use a health maintenance system for the elderly as an example to explain the design concept of our robotic service system. The system's target was to collect data from elderly individuals, whether they were all housed in the same care facility or in individual homes, and provide exercise support for individuals. Moreover, this data can be shared with families and nurses to help prevent illness. With the select and filter function of the system, the elderly could also use the information provided by the system to meet other people. The application encouraged the elderly to extend their social circles.
5.1 Services for the elderly
The provision of a service for the elderly was the target of collecting direct motion data and a user model-based application was used to provide exercise recommendations. To provide such a service to the elderly, the system must detect the user's health situation and provide physical assistance; thus, we used sensors to obtain motion data and a robot to provide an intuitional service. We used “ApriPoco” by Toshiba as the assistant robot. The robot is equipped with a 3D camera, microphone and speaker and can do simple actions. Because the robot has a camera, we also collected image data to augment the user data.
As described in Section 4, we collected motion data from the elderly using 3D acceleration sensors and Actimarker to obtain motion logs, and used a series of motion logs to obtain life logs. A user model could then be constructed by mining the life logs. Apripoco can match user models to offer exercise suggestions, such as reminding users to exercise and depending upon a user's situation to suggest doing the appropriate amount of exercise for maintaining health. For example, if a user simply needs light exercise, ApriPoco will suggest that they take a walk. If a user is not exercising enough, ApriPoco will suggest that they do gymnastics (see Figure 3.). Apripoco can also simulate the exercise so as to encourage users to do the exercise.
The system uses a robot called “ApriPoco” to give exercise suggestions to users
Because the robot has a 3D camera that can collect image data, we also used the camera to obtain eating and home logs to augment user data. Eating logs include the time spent eating and the amount of food eaten. Home logs include the name of electronic devices, the start time, time spent and the rate of use of the devices in their homes. We used OpenCV as the image processing software. The camera captures images and uses the colour of the dish or bowl to detect where the food is being placed and uses the area of detectable space to detect how much food was consumed (see Figure 4.). The three types of data, which are the eating start time, the amount of food eaten and the time spent eating, are saved into the database as eating logs. The method of obtaining home logs is almost the same. We used the background distribution technique to recognize the place in the home where electric devices were used and the change in the images to detect the use rate of the devices. This data are saved into the database as home logs.
Image processing for detecting the amount of food consumed
5.2 Service for family (nursing)
A service for the families or nurses of the elderly was the target in providing data of elderly individuals' daily lives. This service enables families or nurses to review the details of their family members'/patients' life, eating logs and home logs by using a web browser on a PC or a mobile device (see Figure 5.). We also developed an augmented reality application that allows the elderly and their families to directly view the usage history of their electronic devices in the home logs. The function uses the pointing movement system developed by Sato et al. [10]. The original system uses “pointing” which uses a hand to point to a target to indicate the robot's direction of movement. We use the same basic concept, but we focus on only a part of the “pointing direction” function that can detect the place where the user's hand pointing to. The function can help determine the direction where the smartphone toward. The function helps us to develop an augmented reality application that allows the elderly or their families to use a smartphone to point to the target whereby the smartphone can display the information that the elderly or their families want to know. After determining this, the system can compare the location of the electronic device and the pointing direction to determine whether or not the user is using a mobile phone to look at the electric equipment, and then show the usage history of that device on the mobile screen. The user can obtain more details by touching the device image displayed on the mobile screen.
Data can be reviewed using a web browser on a smartphone
5.3 Service for community
The service for the community encourages the elderly to extend their social circles. As described in Section 4, life logs can show daily activities, including information on where and how long they spend doing something during their normal daily routine. With this information, we can predict the possible interests that an elderly individual may have. In this research, we used common interests as an incentive to encourage the elderly to communicate with other people. We designed an information-providing application using augmented reality on a smartphone. The application is similar to the augmented reality application of home logs, but it is based on the user's interests. The application can be used in public spaces (for example, a garden, a kitchen or a library. When a person is in a public space, the system will detect this action and display the data to those who have the same interest and inform them when it may be possible to meet. The service encourages the elderly to go outside and socialize with each other for extending their social circles (see Figure 6.).
The system detects the user's pointing direction and displays the data of anyone who has same interests and when it may be possible to meet them. This application will help the elderly to find other people who have similar interests and encourage them to improve their social skills.
6. Conclusion
We developed a robotic service system by applying a user model-based application for supporting daily life. We used sensors to collect motion data to obtain life logs and mining life logs in order to construct user models for providing appropriate services. The capability of the system also extends from individuals to families (and nurses) and communities. The experiment with mining life logs demonstrates the possibility of collecting motion data to obtain life logs and constructing user models by mining these life logs. We use a health maintenance system for the elderly to explain the design concept of our system. User models can be constructed by mining life logs; user models can help provide appropriate services to users; and log data can help provide services not only to individuals but also to families and communities.