Sage Journals: Discover world-class research

Abstract

Technology has recently been developed which offers an excellent opportunity to design systems with the ability to help people in their own houses. In particular, assisting elderly people in their environments is something that can significantly improve their quality of life. However, helping elderly people outside their usual environment is also necessary, to help them to carry out daily tasks like shopping. In this paper we present a person-following shopping cart assistance robot, capable of helping elderly people to carry products in a supermarket. First of all, the paper presents a survey of related systems that perform this task, using different approaches, such as attachable modules and computer vision. After that, the paper describes in detail the proposed system and its main features. The cart uses ultrasonic sensors and radio signals to provide a simple and effective person localization and following method. Moreover, the cart can be connected to a portable device like a smartphone or tablet, thus providing ease of use to the end user. The prototype has been tested in a grocery store, while simulations have been done to analyse its scalability in larger spaces where multiple robots could coexist.

1. Introduction

Recently, the proliferation of big supermarkets and shopping centers, added to the rapid development of robot technology, has produced robotic systems for helping people in these particular environments. Some specific types of applications, such as enhancing the physical capabilities of the user, helping in the transport of products by providing a mobile shopping basket, and improving the information given to the user in a more intelligent manner, have been described in scientific literature.

Assisting elder people in their environments is a way of significantly improving their quality of life. As an example, networks of sensors and actuators embedded in buildings (i.e., Ambient Intelligence) can provide useful functions. In fact, several European Union projects have focused on this particular subject. However, helping elderly people outside their usual environment is also necessary, to help them to carry out daily tasks like shopping.

Population ageing is a concern for most developed countries and it is forecast that it will accelerate in the years to come. The main goal in the field of assistance robotics is to help users in their everyday lives, in tasks such as moving around, handling objects, or interacting, possibly remotely, with other people, either friends or relatives, or professionals. In this context, robot devices are aimed not only at assisting people, instead of replacing them, but also at adopting the role of companions. Therefore, they must be designed to interact with untrained people, and a great effort must be made to enable users to accept them easily.

Moreover, these assistance robots may have to operate in unstructured environments that cannot be customized, such as homes, shops, or parks. This implies a need for the robot that is conscious of its surroundings, which requires the use of a variety of sensors. In addition, many of the things that need to be handled, such as food, do not come in standard sizes and shapes, as they often do in an industrial context.

In this paper we present a person following shopping cart and assistance robot named CompaRob, the companion robot (Figure 1), capable of easily helping elderly people to carry products in a supermarket. First of all, the paper presents a survey of related systems designed to help people in supermarkets and commercial malls [1–3] and to help people shopping and carrying their articles in a supermarket [4, 5] using different approaches, such as attachable modules and computer vision. After that, the paper describes in detail the proposed system: a mobile platform that implements person following behaviour that uses ultrasound and radio transmissions technology to provide robustness and ease of use from the point of view of the user. Finally, the paper analyzes the scalability of this system for increasing the number of robots in a simulated environment and its usability in a small grocery store.

Figure 1

CompaRob: a person following shopping cart assistance robot based on the Erratic robotic mobile platform equipped with a shopping basket.

2. Review of Developed Systems

2.1. Information and Help Systems

In the context of information and help systems, the services that those systems provide to the customers in malls and big supermarkets are mainly centered on indicating the location of certain shops and offering nonspecific advertisement information based on customer identification.

In [1] a system is described that has the ability to accompany the user in the search of a list of articles previously specified. The system does not carry the food. Moreover, the robot performs navigation and obstacle detection by using a camera and a laser range finder.

Using a Robovie IIF platform with humanoid aspect, in [2] a system is described that interacts with the persons in a mall and is able to give them directions about the localization of the products or even recognizing them through RFID tags. The user input is provided by means of a speech recognition system.

An intelligent multirobot system for big stores is proposed in [3], where the robots may develop cooperative tasks in order to detect available space in the shelves, checking the real location of the products by looking at their barcodes, knowing the number of articles in a shelf by using RFID tags, making an inventory of the available products, and helping customers to localize specific articles.

2.2. Shopping Cart Robots

In recent years, some studies have addressed the idea of developing an autonomous shopping cart. As an example, the possibility of developing a porting system for large-sized facilities like a shopping mall or an airport is studied in [4, 5]. To achieve this purpose, they have attached a drive system to a shopping cart, in order to transmit power to the cart wheels. To perform the desired person following capability, they use a stereo camera installed in front of the cart. One advantage of this system is that the person does not have to wear any attached element. In contrast, as they are using recognition by colour segmentation, the person has to wear clothes of a specific colour. Additionally, they have to face all the problems related to computer vision: high computational cost, light variability problems, background colour interference problems, and so forth.

A more complete prototype is proposed in [6], composed of two robots, a companion and a carrier. In order to localize the position of the companion, a set of cameras is used, which allows the detection of robots and humans present in the scenario. The carrier robot is able to follow the companion one, which is able to guide the customer to the desired position, as well as follow him while waiting for the next action to be performed. A system for tracking shopping carts in indoor environments is developed in [7]. This system consists of the installation of a computer and a video camera on the shopping carts, for them to be able to self-localize and send their position to a centralized system. We find this system interesting, as it could also be a first step to having an autonomous cart with person following capabilities, but as a disadvantage, it requires a complex system to be installed on the cart, and a building with preexistent infrastructure.

The study carried out in [8] concludes that elder people interact in a better way with robots carrying the shopping basket when they have a humanoid aspect and provide conversational capabilities, although this fact implies a high cost and complex system. In [9, 10] a shopping help system is developed, able to obtain the shopping list from a mobile device through a QR code, carry the shopping basket and show at each moment which articles are on it, and communicate with the supermarket computer system to inform about the location of articles. It uses a laser range finder, sonar, and contact sensors (bumpers) to navigate.

A special case is presented in the system proposed in [11], where the user controls the robot remotely by specifying the products to be grasped. Then, the robot navigates through the supermarket to perform the task autonomously. A lot of work is provided in the design of the artificial hand used to grasp a wide variety of food products. No details are provided in the paper about the localization and navigation techniques used by the robot. It seems a very useful system for handicapped people. On the other hand, some elderly people prefer being present in the supermarket to touch the products and make a more detailed selection.

2.3. Robot Person following Behaviour

Person following performed by a robot (see Figure 2) has traditionally been an important research topic in the machine vision area. It is natural to imagine that mobile robots of the future, especially those operating in public places, will be expected to have this skill. As an example, in [12, 13], two different efficient person-tracking algorithms for a vision-based mobile robot using two independently moving cameras are presented. With this approach it is possible to carry out real-time person following in indoor environments. In [14], the authors describe a robot with following function and returning function using a monocular camera.

Figure 2

Robot following an elder person using ultrasound and radio sensors. An ultrasound ring transmitter is attached to the elder person's leg. The robot can follow the elder person using the received signals.

A different human tracking method using template matching in range data obtained from a laser range finder (LRF) is described in [15]. The block matching is performed using stored templates, which correspond to appearances of human legs. In [16], the authors use laser scans and filtering techniques for tracking moving objects and persons. In [17], the authors use a laser-based person-tracking method and two different approaches to person following, direction-following and path-following, and a combination of both methods is proposed, a hybrid approach, with the robot automatically selecting which method to use. In [18] the laser-based leg detector is combined with a Kinect system to detect the user position, predict its trajectory, and make the vehicle follow it. In [19], the laser range finder is combined with two cameras to detect the user position and the presence of obstacles, while the main goal of the paper is to provide the robot with different behaviour strategies (e.g., Go To User, Follow User, Local Search, and Global Search) and motion ways (Free Space and Cluttered Space) selecting the appropriate one in every situation.

A robot tracking system based on a camera and a light-emitting device is used in [20]. Their purpose is to develop an autonomous mobile robot which can follow a human being. The robot with a camera looks at a human having a light-emitting device. The device consists of two LEDs. The robot can estimate the position of the human from the distance between two LEDs, and the direction, from the position of the LED on the captured image.

In [21], a double sensor system is used to perform person following by a robot. The robot can accompany a person using vision-based target detection and avoid obstacles with ultrasonic sensors while following the person. In [22] a camera is used to detect both the person to be followed and the environmental obstacles, while RFID tags are used to obtain the robot's localization.

In [23], a shopping help robot is proposed for visually impaired people, using RFID tags for product identification and a laser range finder for localization and navigation, while, in [24], RFID is mixed with visio techniques to identify the user in a crowd and track his or her position in order to make it possible to follow him.

As can be seen, several approaches have been developed using different kinds of sensor information: computer vision, laser range finder devices, light-emitting diodes used as a light pattern, ultrasonic sensors to detect obstacles, and so forth. In Table 1, a comparison between the reviewed systems is shown.

Table 1

Comparison between main features of several reviewed systems.

	Sensors				Capabilities			Interaction
	Camera	LRF $^{*}$	RFID	Other	Following	Carry items	Find items	User	Environment
Chen and Birchfield 2007 [13]	Binocular	NO	NO	NO	$✓$	NO	NO	NO	A
Chung et al. 2012 [15]	NO	$✓$	NO	NO	$✓$	NO	NO	NO	A
Cosgun et al. 2013 [18]	$✓$	$✓$	NO	Kinect	$✓$	n/a	NO	Remote Skype chat	A, F, H
Garcia-Arroyo et al. 2012 [9]	$✓$	n/a	NO	Bumpers/QR reader/sonar	NO	$✓$	NO	Display, keyboard	F, G
Germa et al. 2009 [24]	$✓$	$✓$	User detection	n/a	$✓$	NO	NO	Touch panel	F
Granata and Bidaud 2012 [19]	Two cameras	$✓$	NO	n/a	$✓$	n/a	n/a	n/a	A
Gross et al. 2009 [1]	Omnidirectional plus frontal	$✓$	NO	Sonar sensors, bumpers	NO	NO	Guide user to the items	Touch panel, speech, head movements	F
Kanda et al. 2010 [2]	Binocular	NO	Person identification	Speech recognition, floor sensors	NO	NO	Give directions	Speech/arm and head movements	H
Katic and Rodic 2010 [3]	$✓$	$✓$	Inventory keeping	Infrared	$✓$	$✓$	$✓$	Human assistance	A, C, E, F, I
Kulyukin et al. 2005 [23]	NO	$✓$	Localization	n/a	NO	$✓$	$✓$	Keypad, speech	D
Matsuhira et al. 2010 [6]	Stereo plus omnidirectional	$✓$	NO	Touch and force sensors/sonar	$✓$	$✓$	$✓$	Touch panel	A, F, I, J
Nagumo and Ohya 2001 [20]	$✓$	NO	NO	User LED	$✓$	NO	NO	NO	A
Nishimura et al. 2006 [4]	Stereo	$✓$	Future	Encoder	$✓$	$✓$	NO	LEDs	A, H
Shieh et al. 2006 [22]	$✓$	$✓$	Localization	n/a	$✓$	n/a	$✓$	LCD screen	A, D
Tomizawa et al. 2007 [11]	$✓$	$✓$	NO	n/a	NO	$✓$	NO	Remote chat	F, H, K
Tsuda et al. 2009 [14]	$✓$	NO	NO	NO	$✓$	$✓$	Return to origin	NO	A
Yoshimi et al. 2006 [21]	Binocular	NO	NO	Ultrasonic	$✓$	$✓$	n/a	Touch panel, speech	A, F
Zender et al. 2007 [16]	NO	$✓$	NO	Odometry	$✓$	n/a	NO	Speech	A
Zimmerman 2006 [7]	Monochromatic	NO	NO	Barcode for localiz./odom.	No autonom. movement	$✓$	Not implemented	Display for items list	B
CompaRob [35]	NO	Obstacle avoidance $^{* *}$	Identify items	User RF/sonar	$✓$	$✓$	NO	Smartphone/tablet PC	F

$^{*}$ Laser range finder.

$^{* *}$ Optional feature.

n/a: information not available.

A: obstacle (and user) detection.

B: barcode for localization.

C: barcode for items identification.

D: RFID for localization.

E: RFID for items identification.

F: data connection with warehouse.

G: data connection with user.

H: teleoperation.

I: mutual interaction with other robots.

J: environment cameras for localization.

K: object manipulation.

The system proposed in this paper is a shopping cart robot that implements a person following behaviour. The objective is to have a simple and robust system, easy to use, specially for older people, and that represents a relatively low cost solution compared to other approaches. The proposed person following system is based on a combination of radio and ultrasound signals to perform measurements between the person and the robot. Once the system takes a pair of measurements, a simple operation estimates the position of the human to be followed. The implementation requires a couple of measurement devices installed on the robot and an ultrasound ring attached to the person's leg. This detection technique, as different to other implementations, avoids the use of cameras and the large amount of computation necessary to image processing.

This system has already proven its efficiency in the context of the EU GUARDIANS project [25, 26], where the risky and low visibility environments required a robust and reliable system. Its strong points are the simplicity, robustness, and its independence to changes in visibility conditions. Compared to other proposed methods, computational requirements are low, so it is possible to implement the whole system in a cost-effective programmable board.

3. System Description: The Shopping Cart Assistance Robot

In this section, the proposed shopping cart assistance robot named CompaRob is presented. The temporal development of the system and the achieved milestones are represented in Figure 4. To the best of the authors' knowledge, it is the first shopping assistance robot that combines modularity, simplicity, and ease of use at a reasonable cost that is especially targeted to help the elderly people in such a common activity like doing shopping. As it will be described below, it is based on a commercial robotic platform and several accessories like an RFID tag reader that provides additional control of the products that are being added to the cart. Moreover, when the system is connected to a mobile device like a tablet computer or smartphone, it provides additional information to the user and allows the adjustment of different person following parameters.

3.1. The Mobile Platform

The CompaRob is based on the Erratic-Videre mobile robot platform [27], a platform which has been equipped with a shopping basket. It is a compact and powerful platform and is capable of carrying a full load of robotics equipment, including an integrated PC, laser range finder, and other sensors. The selection was motivated because it is a simple platform, reasonably priced, highly expandable, and robust. Additionally, this platform is very flexible, as it has been designed to work either with an on-board PC, using the power supply from the robot batteries, or with a laptop, with its own battery. Additionally, the hardware is compatible with Player/Stage [28] and ROS (Robot Operating System) [29], both open-source and widely employed in the robotics community. The platform is equipped with three 12V 7AH lead-acid batteries, providing a reasonable autonomy for the described purpose (around 2 hours for a normal shopping use). The platform weighs 13.5 Kg including the base, batteries, and accessories. The supported payload is 20 Kg.

3.2. Ultrasound Localization System

The localization system used in the shopping cart robot has been adapted from the previous results obtained in the field of person following under emergency situations and low visibility conditions [30, 31]. This system uses ultrasound-based signals to make measurements between two points (see Figure 3). It also uses radio signals for synchronization purposes. The developed ultrasonic localization system involves the following devices: (i)

Hagisonic sonars (transmitters/receivers): two different models of ultrasonic sonar [32] have been studied and used for implementing time difference of arrival (TDoA) in order to derive position information. Those sensors (HG-M40DAI and HG-M40DAII) are ultrasonic object detectors and range finders that offer very-short-to-medium-range detection. The AII model offers a special narrow directional response, 25~30 degrees in vertical direction to minimize the reflection of unwanted sound waves.

(ii)

Radio modules, standard radio transmitter/receiver working at 433 MHz: those modules allow the modulation of digital signals up to 1 KHz at maximum distances of around 20–30 m. Those wireless modules are intended to provide a synchronization mechanism, by sending real-time signals between the different robots that could be present.

(iii)

Handy Board, general purpose board based on the 68HC11 microcontroller [33]: it is widely used in the field of mobile robots for educational, hobbyist, and industrial purposes.

Figure 3

Position determination by trilateration. The robot has an estimation of the elderly position by doing two independent measurements.

Figure 4

CompaRob shopping cart temporal development evolution and summary of the achieved milestones.

3.3. TDoA with RF and Ultrasound

With this approximation, we try to determine the 2D position of a physical target (i.e., a person) with respect to a mobile robot by using the time difference of arrival of two different signals each one with a known propagation speed (i.e., radio and ultrasound; see Figure 5).

Figure 5

TDoA principle for measuring distances between robots by using the time difference of arrival of two different signals (electromagnetic and ultrasonic) each one with a known propagation speed.

The first step was to determine the feasibility of the use of ultrasound signals to obtain distances. In order to do that, several laboratory experiments have been done to evaluate some available ultrasound transmitters/receivers using wires to connect both transmitter and receiver. Those sensors (HG-M40DAI and HG-M40DAII) are ultrasonic object detectors and range finders that offer very-short-to-medium-range detection. The AII model offers a special narrow directional response, 25–30 degrees in vertical direction to minimize the reflection of unwanted sound waves. We obtain larger transmission distances by removing the frontal cover of the sensors that acts as an anisotropic filter.

After performing those experimentations, we concluded that the measurement of the time of flight of ultrasound waves could be a feasible way of estimating distances. The estimation of distances by measuring the time of propagation of ultrasonic waves can be useful for several localization methods based on the knowledge of some distances. Further experimentation demonstrated that this technique offered a good performance of the sensors for distances up to 7-8 meters.

The next step was to introduce synchronization radio signals in order to estimate distances between two independent nodes (i.e., a robot and the person or two robots). To achieve this purpose, the theoretical idea of the time difference of arrival (TDoA) was implemented (see Figure 5). The idea is to measure the time of flight of the ultrasound signals using the radio signal as synchronization method. The time of flight of the radio signal is considered constant for the range of distances that we are going to measure (from 0 to a maximum of 8 meters).

The most challenging task for implementing the TDoA was to find a hardware platform that could take those measurements, being able to control both, radio and ultrasound sensors. After exhaustive research about its capabilities, the Handy Board [33], widely used for teaching purposes in Jaume I University, was used. Some additional electronics were necessary for coupling those components to the board (see Figure 6). The programming of the board was done with a mixture of C programming language for the global routine and assembly language for the real-time reading and writing of signals.

Figure 6

Radio/ultrasound measurement system module implemented with a Handy Board, a general purpose board based on the 68HC11 microcontroller.

The measurement capabilities of the boards were tested, and the result is that the obtained measurements were quite accurate, being able to measure up to 8 meters with a maximum error of around 3 cm.

3.4. Trilateration for Position Determination

The prototype presented in this paper uses a couple of measurement boards to determine the 2D position of the person. The two radio/ultrasound measurement systems (see Figure 7) are separated by a known distance $d_{r}$ .

Figure 7

Erratic robot prototype, to be used as a shopping cart, equipped with a couple of radio/ultrasound measurement systems.

With this configuration, and using an algebraic trilateration method (see [34] for a nonalgebraic approach based on constructive geometric arguments), we can measure the distances from an emitter located at P point (see Figure 8). The P point is the position of the person that we want to locate (i.e., the elderly; see Figure 3).

Figure 8

Trilateration principle. (a) The P point in the intersection of $C_{1}$ and $C_{2}$ . (b) Uncertainty area due to measurements errors of $d_{1}$ and $d_{2}$ .

In order to obtain the $(x, y)$ coordinates of the P point we proceed as follows: the P point is in the intersection of the two circumferences $C_{1}$ and $C_{2}$ . Let the equation of the circumference be $(x - a)^{2} + (y - b)^{2} = r^{2}$ , where $(a, b)$ is the center, r is the radius, and $(x, y)$ are the points of the circumference.

Knowing both center points $(- d_{r} / 2,0)$ and $(d_{r} / 2,0)$ , and the distances $d_{1}$ , $d_{2}$ from the object located at $P (x, y)$ , we can write the following system of equations:

\begin{matrix} {(x + \frac{d_{r}}{2})}^{2} + y^{2} = d_{1}^{2}, \\ {(x - \frac{d_{r}}{2})}^{2} + y^{2} = d_{2}^{2} . \end{matrix}

(1)

Solving this system of equations, we obtain the point $P (x, y)$ where the emitter is located:

\begin{matrix} x = \frac{d_{1}^{2} - d_{2}^{2}}{2 d_{r}}, \\ y = \frac{\sqrt{- d_{1}^{4} + 2 d_{1}^{2} (d_{2}^{2} + d_{r}^{2}) - d_{2}^{4} + d_{r}^{2} (2 d_{2}^{2} - d_{r}^{2})}}{2 d_{r}} . \end{matrix}

(2)

The equation for the y coordinate provides two solutions (positive and negative), but we only use the positive solution because ultrasound sensors are only receiving signals from both y positive quadrants, as they are mounted in positive y direction and not in negative y direction. Once we know the relative position of the P point with respect to the robot, a control algorithm computes the resultant linear and angular speeds for controlling the robot.

3.5. Trilateration Geometrical Error Study

In order to assess the error produced by individual measurements when using the above trilateration method, a geometrical study was made for the most common configurations.

When using the implemented trilateration technique, two independent measurements $d_{1}$ and $d_{2}$ are taken by two measuring devices (see Figure 8(a)), from the receiving points $(- d_{r} / 2,0)$ and $(d_{r} / 2,0)$ to the object located at $P (x, y)$ , equipped with a transmission device. Individual measurement errors in $d_{1}$ and $d_{2}$ produce an uncertainty area (see Figure 8(b)) when calculating $P (x, y)$ by trilateration.

As an example, in Figure 9, we present a geometrical representation of the uncertainty areas in which the point $P (x, y)$ could be if we consider an individual error for each measurement of ±5 cm (an upper bound, typical error is around ±1 cm). The uncertainty area is represented for increasing distances from the emitter to the receivers (from 1 to 8 meters). In this figure, the emitter is situated at 40° from the perpendicular to the segment that connects the two receivers. When increasing the distance, the uncertainty area becomes bigger for the horizontal coordinate (becomes wider) but remains small for the vertical coordinate. The horizontal uncertainty is asymmetric due to the angular displacement of the emitter. The error region increases with the distance from the transmitter to the receivers but it can be reduced by increasing the separation distance ( $d_{r}$ ) between receivers that, in practice, is limited by the robot dimensions.

Figure 9

Trilateration geometrical error study when the emitter is moving away in diagonal line. Distance between receivers: 50 cm. Emitter-Robot distance: 1–8 m. Emitter Angle: 40°. Angular uncertainty: 14 + 17°.

Even this theoretical error study reveals that the position determined by the method can be affected by an important horizontal error; in practice, the individual measures obtained from the devices are much smaller than the theoretical maximum, so the position of the person can be accurately determined and tracked by using a simple filtering technique.

3.6. Attachable Ultrasound Ring

The shopping cart assistance robot is able to follow an elderly person that has an ultrasound ring attached to his or her leg (see Figure 10). This prototype device is composed of a 9 V battery, a radio transmitter, and five ultrasound transmitters. The device is continuously emitting radio and ultrasound signals in order to indicate to the assistance robot the position of the elderly person.

Figure 10

An ultrasound ring transmitter is attached to the elderly person's leg. The robot can follow the person using the received signals.

The particular design of the prototype was chosen so that it can be easily attached to the leg and simple in usage from the point of view of the user (only requires to use an on/off switch). Its design is simple and discrete, and its simplicity implies also reliability. It is also important to take user acceptability into account, especially when designing products that are designed for the elderly.

After testing the system with a small pilot group and gathering the opinion of several potential users, we can conclude that the device is well accepted among elderly people. It would even be desirable not having to use any attached device, the ultrasound ring is a necessary element for the robot to work, and it does not affect in any way the mobility of the person.

3.7. RFID Tag Reader

The CompaRob has been equipped with an RFID tag reader, allowing it to keep control of the products that are being added to the basket (see Figure 11). The list of products is sent to a user interface (described in Section 3.8) so the user can see the list of purchased products and their total cost. The tag reader is a Skyetek SkyeModule M2, which connects to the on-board computer using a USB interface.

Figure 11

RFID tag reader, allowing keeping control of the products that are being added to the basket (a). The list of products is sent to a tablet computer (b).

The software architecture includes a C++ server running in the robot's on-board computer, which provides a TCP/IP interface to an Android device. At the moment of writing the server is running on a Windows Virtual Machine, due to some drivers issues, which made it impossible to run the server directly on the Linux operating system and ROS platform.

3.8. User Interface

The CompaRob is connected wirelessly to a tablet computer or smartphone, allowing the user to check the shopping list. Moreover, the user can control some parameters of the robot related to the person following functionality (described in Section 4) by means of a setup configuration screen.

In this context, the user is able to, for example, adjust the following distance, as well as modifying the following speed and the robot turn delay (see Figure 12). This will allow the user to tune the way he/she wants the robot to perform. Moreover, by using this interface, the user can know other robot parameters, like the battery status and so forth.

Figure 12

User interface to get the actual products' list (a) and adjusting some parameters related to the robot person following navigation (b).

4. Person following Functionality

The person following functionality is implemented in a computer program allocated on the on-board PC of the robot. The PC runs an Ubuntu 12.04 LTS distribution, and the robot is controlled using ROS, thanks to the erratic_player node that wraps the Erratic driver found in Player project. The system functionality is partially implemented in Java (the interface to the measurement units and the positioner) and Python (using rospy, a pure Python client library for ROS), to implement the person following functionality and to interface with ROS and the hardware platform controller.

The person following functionality is implemented using a 4-state machine (see Figure 13). Each state represents one different position of the followed person with respect to the robot (see Figure 14). The 4 states are described as follows: (i)

Follow: the current position $P (x, y)$ of the followed person is into the working area tagged as Follow (see Figure 14). The control program computes the robot speed in a way that the position of the person $P (x, y)$ converges to the desired position $P (x^{'}, y^{'})$ . To achieve this goal, the rotation speeds of both robot wheels are computed using a proportional controller.

(ii)

Wait_1: the current position $P (x, y)$ of the followed person is into the nearest area, tagged as Wait_1, at a distance from the robot that is less than that desired. This situation happens, for example, when the person comes back to take something else from the shelves, or when the person puts something into the cart. In this state, the robot can behave in two different ways depending on the user configuration: (1) to stop moving, so that the person can approach the robot to put something on the basket or (2) to slightly move backwards, providing more freedom to the user when moving forwards and backwards. In this latter case, the user can leave things in the basket by approaching to the robot from the side (Wait_2 state).

(iii)

Wait_2: in this state, the person has moved to the side of the robot and leaves the coverage area of ultrasound receivers. This situation happens, for example, when the person wants to leave something on the basket, or when the person changes direction. In this case, the robot waits for a configurable period of time (turn delay), just in case the person comes back to the coverage area. If the person comes back to the coverage area, the new state will be Wait_1 or Follow. Otherwise, after waiting an amount of time, the program will transition to the Search state.

(iv)

Search: the robot starts rotating slowly in order to search for the person. This situation happens, for example, after the person has changed direction. Once the person has been located, the possible next states will be Follow or Wait_1.

Figure 13

Person following behaviour 4-state machine. Each state represents a different position of the followed person with respect to the robot.

Figure 14

Person following behaviour: the person can be in one of the 3 shown regions, corresponding to Wait_1, Wait_2, and Follow states.

When the program starts its execution, both measurement units installed on the robot are queried. Depending on the read value, the initial state will be one of the following: Follow, Wait_1, or Wait_2. After the initialization, the program will evolve as described above.

5. Simulation Results

Several 2D simulations of the robots using Player/Stage and ROS have been carried out to test the feasibility and scalability of the presented system and are described below. The videos of the simulations are available online [35].

5.1. Small Grocery Store Simulation

The first simulation recreates a small grocery store (see Figure 15), in which there are five people doing shopping and five robots each one following one of the five persons. The environment is based on a small grocery, consisting of three corridors with shelves on each side. The customers move along the corridors and stop in front of the shelves to pick a product.

Figure 15

Simulation of five persons and robots on a small grocery, consisting of three corridors with shelves on each side.

We were particularly interested in the scalability of our system; that is, how would the system perform as the number of robots increases? Due to the interferences between the sonar sensors of different robots, the frequency of the pulses is decreased. In this simulation the positioning system provides new measurements at a rate of merely 2 Hz. Such low frequency allows the group of robots to make measurements of each person in turn, without overlapping.

Robots are programmed to follow the persons at a target distance of 1 m. Each robot follows the person depicted in the same colour. Figure 16 presents the distances between each person and the corresponding follower robot, during an interval of 70 seconds. In such an interval, a common person performs between three and five stops and motions.

Figure 16

Person-robot distances between each person and the corresponding follower robot, during an interval of 70 seconds.

When a person starts to move, the distance to the robot increases momentarily, with peaks at about 2.2–2.4 m, since the velocities of robots are limited, and the feedback control loop is purely proportional. But the positioning system guides the robot towards the person, following the appropriate path until the person stops again in front of a shelf.

Upon the person approaching the robot below a given threshold, it will move slowly backwards to keep the preselected distance (e.g., see the distance of the robot r_5 in the plot between seconds 60 and 70), according to the person following functionality described in Section 4.

5.2. Large Grocery Store Simulation

The previous follower algorithm works only for small distances. We have tested a more complex follower behaviour in a larger shopping environment. Figure 17 depicts the new simulation environment, consisting of a larger shop with a higher number of corridors and shelves. The robots are programmed to calculate a trajectory towards the position of the person, given the map of the environment, and a proper self-positioning system. We have run a simulation of the five people doing shopping over a period of six minutes. Distances between each person and the corresponding follower robot are shown in Figure 18. Each robot updates the position of the followed person at 1 Hz. The preprogrammed distance is 1 m and most of the time the robots remain within a distance to the person shorter than 2.5 m. But the distance may increase due to the delay in the update, and the moving obstacles (other persons and/or robots). With our available sensors, the positioning of the person can be reliably determined up to a distance of 8 m (threshold shown in the figure). Two robots (r_1 and r_4) approach such distance in two instants of the simulation. In such cases, the robot should stop and wait for the person to come within the detection distance once more.

Figure 17

Simulation of five people shopping in a larger environment.

Figure 18

Person-robot distances in the large shop environment. A detection threshold for the ultrasound sensor is also shown.

6. Field Experiments (I): Person following Functionality in a Small Grocery Store

To evaluate the usability, feasibility, and most convenient robot following parameters in a real environment, several experiments were carried out in a small grocery store (Figure 19). The system was tested with a small pilot group of elderly people, with the aim to get some feedback related to usability and adequate robot parameters. All of them were required to do their usual shopping, following approximately the trajectory shown in Figure 20.

Figure 19

The shopper with the CompaRob in a small grocery store used as a real validation scenario.

Figure 20

Approximate floor plan of the grocery store, the real validation scenario. The trajectory followed by the shopper and the robot is marked in a dotted line.

Two different linear following speeds were selected, first at 0.2 m/s and then at 0.3 m/s, following the preferences of the users. For each selected speed, the distance from the robot to the person was recorded during an interval of 70 seconds, as in the first simulation experiment. The aim now is to test the robustness of the proposed following method at low measuring rates (i.e., 2 Hz) coming from the positioning system. Also, different following distances were tested $d = {0.2, 0.5, 1.0}$ , to observe the behaviour at different linear speeds. As can be seen in Figure 21 (linear speed 0.2 m/s), when the person starts to move, the distance to the robot increases and the robot tries to minimize it (using a simple proportional controller) to the preselected following distances depicted in horizontal lines. The small ripple pattern that can be observed in all the graphs corresponds to the walking pattern, as the ultrasonic ring was attached to the person's leg. This pattern does not affect the follower functionality in any way.

Figure 21

Person-robot distances in the real grocery environment for different following distances $d = {0.2, 0.5, 1.0}$ while a person does shopping. The linear following speed was set to 0.2 m/s.

As can also be observed, when the person does a stop to pick something up, the distance decreases as the robot approaches the person. This happens faster in Figure 22 as the linear speed is 0.3 m/s, slightly greater than in the experiments depicted in Figure 21 (0.2 m/s). When the person approaches the robot, for example, to put something into the basket, the robot remains static if the user approaches from the side, according to the state machine explained in Section 4. Contrarily, if the user approaches from the front, the robot begins to move slowly backwards to keep the predefined distance, providing more freedom to the user when moving along the corridor.

Figure 22

Person-robot distances in the real grocery environment for different following distances $d = {0.2, 0.5, 1.0}$ while a person does shopping. The linear following speed was set to 0.3 m/s.

For the tested following distances, the robot behaves adequately, but if greater distances were required (i.e., a user that prefers a greater robot distance), an obstacle avoidance system should be used to avoid collisions. To tackle this issue, the platform is also equipped with a Hokuyo URG-04LX Scanning laser range finder (LRF) [36] and uses a local planner for obstacle avoidance, based on the Dynamic Window Approach [37] to local navigation on a plane. Given the reference position of the person to follow and a costmap obtained by a laser range finder, the local planner produces velocity commands to send to the mobile base. The information for the sonar needs to be combined with the laser measurements in order for the detected person not to be treated as an obstacle. Nevertheless, with the use of an LRF this device increases significantly the cost of the platform, so as an alternative, a much cheaper optical triangulation-based distance sensor (like the Sharp GP2D12 distance measuring sensor) could be used instead.

Further details, like a video with the person following system in action and some preliminary usability results can also be seen online [35].

7. Field Experiments (II): Usability Tests with Two Robots in a Large Grocery Store

Based on the first user experiments and suggestions, an improved version of the robot has been designed, and a first prototype (named CompaRob-2; see Figure 23) was tested in a larger grocery store (see Figure 24). In the evolved version, a standard shopping cart bag was fitted to the robot by means of a PVC structure, improving the ergonomics and the aesthetics. The previous prototype had a 22 × 25 × 40 cm ( $H \times L \times W$ ) bag, while the improved one is 65 × 29 × 44 cm (see the details in Figure 25). The upper part of the PVC structure can be used as a handle both to slightly lean on it and to move the cart. Also, the height of the bag has been increased, making it easier to put articles inside and increasing as well the capacity of the cart. Below the bag, a specific protected space has been enabled to hold the computer and sensors.

Figure 23

CompaRob-2 prototype: an improved version of CompaRob shopping cart based on first user usability tests and suggestions.

Figure 24

Usability tests with two robots in a larger grocery store. Both CompaRob and CompaRob-2 robots were tested with different people doing normal shopping.

Figure 25

Working drawings and dimensions of the original CompaRob (a) and the new CompaRob-2 robot (b).

In this second experiment, the new platform was teleoperated, using a remote laptop connected wirelessly to the computer on board the robot, and using the ROS teleop package. The objective at this prototyping stage was to get as much feedback as possible from users before the design and development of an improved person following functionality. Five different people were asked to do normal shopping in a grocery store, acting as natural as possible and forgetting about having to push the shopping cart.

The user experiences were collected in a series of interviews. All of them indicated that this new version is more comfortable than the previous one, mainly with the new design of the bag. A summary of the main ideas brought by the users in order to improve the system is the following: (i)

Make the robot even more light, in order to allow the user to move it manually in case of necessity.

(ii)

Design a tool or elevator that helps elder people to extract the goods in a simple and easy manner.

(iii)

The use of an intelligent arm watch instead of an attachable belt to the leg.

Moreover, a set of four questions was asked to the four people that tested the robots. Their responses to a Likert 1–5 scale are summarized in a chart (Figure 26). As can be seen, all of them are very satisfied with the new design, and they mainly suggest the use of an intelligent arm watch as an interaction device with the robot.

Figure 26

Summary of the responses given by the four interviewed users, related to usability and overall satisfaction with the two versions of the robot.

8. Conclusion and Future Work

In this paper we have presented a person following system for a mobile assistance robot that is based on a combination of radio and ultrasound signals to perform measurements between the person and the robot. One of the objectives of the work was to develop a simple, reliable, and easy to use system that could provide freedom of movements for the elderly people. To do so, the mobile robot follows the person by reading the signals from an ultrasound ring attached to the person's leg.

A second important objective was the user acceptability. After testing the system with some potential users, we were able to conclude that this shopping cart robot is well accepted among elderly people, because they understand that it is an improvement in their lives, a technological advance, instead of something that replaces their physical capabilities.

As future work, we are planning to integrate the recently proposed indoor radio-based localization system [38] into the shopping cart. This system is based on fingerprinting techniques and can be applied to robot and person localization in order to know their position in a particular environment [39], just using simply the WiFi/ZigBee transceiver information, and providing a 1.5 m localization error. This could improve the robot navigation inside a shopping center, and the robot could also show the person where to go in order to find a particular product. Combining the radio/sonar localization for navigation and the fingerprinting global approach, more sophisticated applications will be possible. In fact, further work can, for example, provide a more intuitive user interface by providing the possibility of searching for the available products in the store, creating a wish list, and let the robot conduct the user to the proper place in an efficient and autonomous manner. This would require a better integration of the robot interface with the store database. On the other hand, if the user wants to lead the navigation, the smartphone can provide orientations to find the products related to the actual position of the robot and the user.

Moreover, some experiments are being performed to enhance the global localization system of the person by using the eZ430-Chronos watch [40], a device that enables the integration of a sophisticated radio-based localization system and provides a bidirectional communication link between user and the robot.

Footnotes

Conflict of Interests

The authors declare that there is no conflict of interests regarding the publication of this paper.

Acknowledgments

This work was partly supported by Spanish Ministry under Grant DPI2014-57746-C3 (MERBOTS Project) and by Universitat Jaume I Grants P1-1B2015-68 and PID2010-12.

References

Gross

H.-M.

Boehme

Schroeter

Mueller

Koenig

Einhorn

Martin

Merten

Bley

TOOMAS: interactive shopping guide robots in everyday use—final implementation and experiences from long-term field trials

Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS ′09)

October 2009

St. Louis, Mo, USA

IEEE

2005 2012

10.1109/iros.2009.5354497

2-s2.0-76249094575

Kanda

Shiomi

Miyashita

Ishiguro

Hagita

A communication robot in a shopping mall

IEEE Transactions on Robotics 2010 26 5 897 913

10.1109/TRO.2010.2062550

2-s2.0-77957700115

Katic

Rodic

Intelligent multi robot systems for contemporary shopping malls

Proceedings of the 8th International Symposium on Intelligent Systems and Informatics (SISY ′10)

September 2010

Subotica, Serbia

109 113

Nishimura

Itou

Kikuchi

Takemura

Mizoguchi

A study of robotizing daily items for an autonomous carrying system—development of person following shopping cart robot

Proceedings of the 9th International Conference on Control, Automation, Robotics and Vision (ICARCV ′06)

December 2006

Singapore

1 6

10.1109/icarcv.2006.345272

2-s2.0-34547162819

Nishimura

Takemura

Mizoguchi

Development of attachable modules for robotizing daily items -person following shopping cart robot-

Proceedings of the IEEE International Conference on Robotics and Biomimetics (ROBIO ′07)

December 2007

Sanya, China

IEEE

1506 1511

10.1109/robio.2007.4522387

2-s2.0-49249137462

Matsuhira

Ozaki

Tokura

Sonoura

Tasaki

Ogawa

Sano

Numata

Hashimoto

Komoriya

Development of robotic transportation system—shopping support system collaborating with environmental cameras and mobile robots

Proceedings of the 41st International Symposium on Robotics (ISR ′10) and 6th German Conference on Robotics (ROBOTIK ′10)

2010

Munich, Germany

1 6

Zimmerman

T. G.

Tracking shopping carts using mobile cameras viewing ceiling-mounted retro-reflective bar codes

Proceedings of the IEEE International Conference on Computer Vision Systems (ICVS ′06)

January 2006

IEEE

10.1109/icvs.2006.60

2-s2.0-33749406325

Iwamura

Shiomi

Kanda

Ishiguro

Hagita

Do elderly people prefer a conversational humanoid as a shopping assistant partner in supermarkets?

Proceedings of the 6th ACM/IEEE International Conference on Human-Robot Interaction (HRI ′11)

March 2011

Lausanne, Switzerland

449 457

10.1145/1957656.1957816

2-s2.0-79953152344

Garcia-Arroyo

Marin-Urias

L. F.

Marin-Hernandez

Hoyos-Rivera

G. D. J.

Design, integration, and test of a shopping assistance robot system

Proceedings of the 7th Annual ACM/IEEE International Conference on Human-Robot Interaction (HRI ′12)

March 2012

Boston, Mass, USA

ACM

135 136

10.1145/2157689.2157722

2-s2.0-84859938695

10.

Marin-Hernandez

de Jesus Hoyos-Rivera

García-Arroyo

Marin-Urias

L. F.

Conception and implementation of a supermarket shopping assistant system

Proceedings of the 11th Mexican International Conference on Artificial Intelligence (MICAI ′12)

November 2012

San Luis Potosi, Mexico

IEEE

26 31

10.1109/micai.2012.21

2-s2.0-84872584545

11.

Tomizawa

Ohba

Ohya

Yuta

Remote food shopping robot system in a supermarket—realization of the shopping task from remote places

Proceedings of the IEEE International Conference on Mechatronics and Automation (ICMA ′07)

August 2007

Harbin, China

IEEE

1771 1776

10.1109/icma.2007.4303818

2-s2.0-37049015565

12.

Kwon

Yoon

Park

J. B.

Kak

A. C.

Person tracking with a mobile robot using two uncalibrated independently moving cameras

Proceedings of the IEEE International Conference on Robotics and Automation (ICRA ′05)

April 2005

Barcelona, Spain

2877 2883

13.

Chen

Birchfield

S. T.

Person following with a mobile robot using binocular feature-based tracking

Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS ′07)

November 2007

San Diego, Calif, USA

815 820

10.1109/iros.2007.4399459

2-s2.0-51349132679

14.

Tsuda

Harimoto

Saitoh

Konishi

Mobile robot with following function and autonomous return function

Proceedings of the ICROS-SICE International Joint Conference (ICCAS-SICE ′09)

August 2009

Fukuoka, Japan

635 640

15.

Chung

Kim

Yoo

Moon

C.-B.

Park

The detection and following of human legs through inductive approaches for a mobile robot with a single laser range finder

IEEE Transactions on Industrial Electronics 2012 59 8 3156 3166

10.1109/tie.2011.2170389

2-s2.0-84859718483

16.

Zender

Jensfelt

Kruijff

G.-J. M.

Human- and situation-aware people following

Proceedings of the 16th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN ′07)

August 2007

JeJu Island, South Korean

1131 1136

10.1109/roman.2007.4415250

2-s2.0-48749132558

17.

Gockley

Forlizzi

Simmons

Natural person-following behavior for social robots

Proceedings of the ACM/IEEE Conference on Human-Robot Interaction (HRI ′07)

March 2007

Washington, DC, USA

ACM

17 24

10.1145/1228716.1228720

2-s2.0-34548247198

18.

Cosgun

Florencio

D. A.

Christensen

H. I.

Autonomous person following for telepresence robots

Proceedings of the IEEE International Conference on Robotics and Automation (ICRA ′13)

May 2013

Karlsruhe, Germany

IEEE

4335 4342

10.1109/icra.2013.6631191

2-s2.0-84887284435

19.

Granata

Bidaud

A framework for the design of person following behaviors for social mobile robots

Proceedings of the 25th IEEE/RSJ International Conference on Robotics and Intelligent Systems (IROS ′12)

October 2012

Vilamoura, Portugal

IEEE

4652 4659

10.1109/iros.2012.6385976

2-s2.0-84872310260

20.

Nagumo

Ohya

Human following behavior of an autonomous mobile robot using light-emitting device

Proceedings of the 10th IEEE International Workshop on Robot and Human Interactive Communication (ROMAN ′01)

September 2001

Bordeaux, France

225 230

10.1109/roman.2001.981906

2-s2.0-84881097623

21.

Yoshimi

Nishiyama

Sonoura

Nakamoto

Tokura

Sato

Ozaki

Matsuhira

Mizoguchi

Development of a person following robot with vision based target detection

Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS ′06)

October 2006

Beijing, China

IEEE

5286 5291

10.1109/iros.2006.282029

2-s2.0-34250661914

22.

Shieh

M.-Y.

Chan

Y.-H.

Lin

Z.-X.

J.-H.

Design and implementation of a vision-based shopping assistant robot

Proceedings of the IEEE International Conference on Systems, Man and Cybernetics (SMC ′06)

October 2006

Taipei, Taiwan

IEEE

4493 4498

10.1109/icsmc.2006.384852

2-s2.0-34548131302

23.

Kulyukin

Gharpure

Nicholson

Robocart: toward robot-assisted navigation of grocery stores by the visually impaired

Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS ′05)

August 2005

Edmonton, Canada

2845 2850

10.1109/IROS.2005.1545107

24.

Germa

Lerasle

Ouadah

Cadenat

Devy

Vision and RFID-based person tracking in crowds from a mobile robot

Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS ′09)

October 2009

St. Louis, Mo, USA

IEEE

5591 5596

10.1109/iros.2009.5354475

2-s2.0-76249123444

25.

GUARDIANS

Group of unmanned assistant robots deployed in aggregative navigation supported by scent detection (GUARDIANS)

EU Project 2007 IST-045269 https://www.shu.ac.uk/research/meri/research/guardians-project

26.

Sales

Maŕn

Cervera

Rodríguez

Pérez

Multi-sensor person following in low-visibility scenarios

Sensors 2010 10 12 10953 10966

10.3390/s101210953

2-s2.0-78650262911

27.

Erratic-Videre Erratic-Videre Mobile Robot Platform 2010

Erratic-Videre

http://wiki.ros.org/Robots/Erratic

28.

Player

The Player Project. Free Software tools for robot and sensor applications

2010, http://playerstage.sourceforge.net

29.

Quigley

Conley

Gerkey

B. P.

Faust

Foote

Leibs

Wheeler

A. Y.

ROS: an open-source Robot Operating System

Proceedings of the ICRA Workshop on Open Source Software

May 2009

Kobe, Japan

30.

Sales

El-Habbal

Marín

Witkowski

Cervera

Nomdedeu

Rückert

Localization of networked mobile sensors and actuators in low-visibility conditions

RISE (IARP/EURON Workshop on Robotics for Risky Interventions and Environmental Surveillance) 2010

Sheffield, UK

Sheffield Hallam University

31.

Naghsh

Saez-Pons

Penders

Gancet

Motard

Nomdedeu

Sales

Cervera

Marín

Sanz

Sebastià

Remote and in-situ multirobot interaction for firefighters interventions under smoke conditions

Proceedings of the 2nd IFAC Symposium on Telematics Applications (TA ′10)

October 2010

Timisoara, Rumania

109 115

10.3182/20101005-4-ro-2018.00037

2-s2.0-80051942801

32.

Hagisonic Ultrasonic Sensors 2010

Hagisonic

http://www.hagisonic.com/

33.

Handy-Board The Handy Board, 2010, http://www.handyboard.com/

34.

Thomas

Ros

Revisiting trilateration for robot localization

IEEE Transactions on Robotics 2005 21 1 93 101

10.1109/tro.2004.833793

2-s2.0-14044260564

35.

CompaRob

CompaRob: the shopping cart assistance robot

2015, http://sites.google.com/a/uji.es/elderlyassistance/comparob

36.

Hokuyo Hokuyo. URG-04LX Scanning Laser Range Finder, 2010, http://www.hokuyo-aut.jp/02sensor/07scanner/urg_04lx.html

37.

Fox

Burgard

Thrun

The dynamic window approach to collision avoidance

IEEE Robotics and Automation Magazine 1997 4 1 23 33

10.1109/100.580977

2-s2.0-0031098777

38.

Marti

J. V.

Sales

Marin

Jimenez-Ruiz

Localization of mobile sensors and actuators for intervention in low-visibility conditions: the ZigBee fingerprinting approach

International Journal of Distributed Sensor Networks 2012 2012 10

951213

10.1155/2012/951213

2-s2.0-84866149945

39.

Martí

J. V.

Sales

Marín

Sanz

P. J.

Multi-sensor localization and navigation for remote manipulation in smoky areas

International Journal of Advanced Robotic Systems 2013 10, article 211

10.5772/55808

40.

Fink

Lange

Beikirch

Radio-based indoor localization using the eZ430-Chronos platform

Proceedings of the IEEE 1st International Symposium on Wireless Systems (IDAACS-SWS ′12)

September 2012

Offenburg, Germany

IEEE

19 22

10.1109/idaacs-sws.2012.6377623

2-s2.0-84872179417

CompaRob: The Shopping Cart Assistance Robot

Abstract

1. Introduction

2. Review of Developed Systems

2.1. Information and Help Systems

2.2. Shopping Cart Robots

2.3. Robot Person following Behaviour

3. System Description: The Shopping Cart Assistance Robot

3.1. The Mobile Platform

3.2. Ultrasound Localization System

3.3. TDoA with RF and Ultrasound

3.4. Trilateration for Position Determination

3.5. Trilateration Geometrical Error Study

3.6. Attachable Ultrasound Ring

3.7. RFID Tag Reader

3.8. User Interface

4. Person following Functionality

5. Simulation Results

5.1. Small Grocery Store Simulation

5.2. Large Grocery Store Simulation

6. Field Experiments (I): Person following Functionality in a Small Grocery Store

7. Field Experiments (II): Usability Tests with Two Robots in a Large Grocery Store

8. Conclusion and Future Work

Footnotes

Conflict of Interests

Acknowledgments

References