Sage Journals: Discover world-class research

Abstract

Location-based services for mobile users provide a convenient means of filtering information based on current geographical position. However, the information in traditional location service systems is provided through an abstract metaphor that the user has to understand and translate into action. In this paper, a mobile augmented reality-based system for individual location recommendation is proposed. The system consists of the modules of user location, marker detection, three-dimensional (3D) display and location guide by combing augmented reality with navigation technology. Through such a system, the position and orientation information from any sensory source were pervasively extracted and used to enhance the association to the real world by combining video techniques and 3D-graphics in an augmented reality view. Experimental results demonstrate that the proposed system is both efficient and effective in helping people locate their position. Compared with the traditional Global Positioning Systems-based location services system, the proposed system can also enhance users’ senses experience by providing users with a more intuitive, 3D, dynamic information display and sharing capabilities.

Keywords

Augmented reality mobile location individual recommendation collaborative filtering

Introduction

There is a massive amount of information available on the internet about travel destinations and their associated resources (such as accommodation, restaurants, museums or events), and this wealth of information may be particularly useful for those users who plan to visit an unknown destination. However, the extensive list of possibilities offered by search engines may be overwhelming. Evaluating this long list of options is a very complex and time consuming task for tourists in order to select the internet search result that best fits their needs.¹ However, personalization systems can be of great benefit to tourists by customizing these resources by offering recommendations based on their preferences and their current context.² The use of a Web-based interface is the most common option, particularly as this kind of interface allows users to look for information in a user-friendly manner. Nevertheless, Web-based applications are usually not designed to be used during the stay since most of the users will not have easy access to computers with an internet connection.

Recently, there has been a considerable increase in systems that offer mobile interfaces due in large part to the popularity of mobile devices with internet capabilities and, even more recently, the popularity of smart phones. The primary advantage of these mobile devices is that the system can be used in any place with an internet connection, so that users may access information, discover places or modify their trips during their stay. They are particularly relevant in recommender systems,³ whose objective is to filter out irrelevant options and to provide personal and relevant information to each particular user. When opening this service item, a mobile user can easily use their mobile phone to learn their present location as well as to search for and inquire into a variety of places located nearby. By location services, a mobile user could also be located, monitored and tracked in real-time.⁴

Currently, almost all social networking platforms, such as Twitter, Google and MySpace, provide such basic location service functions, like location sharing, location sign, location identification and so on.⁵ The latest research report by Pew² revealed that 58% of smart phone online users have used location based service (LBS), 55% used the navigation and location-related recommendation and 12% shared their location information.

However, in traditional mobile location service systems (e.g. the Global Positioning Systems (GPS)-based location service systems), the information is provided through an abstract metaphor that the user has to interpret and then project to the real world. Meanwhile, most commercially available GPS systems are too inaccurate to provide practical meaningful location-based information.⁶

Motivated by the aforementioned problems, in order to provide users with a more intuitive, three-dimensional (3D), dynamic information display and with sharing capabilities and help people locate their position, this paper developed an augmented reality (AR)-based system for mobile location recommendation. Experimental results show that with such a system, a user could obtain a real experience of the information natural acquisition effect according to his (or her) visual transformation and its dynamical recombination, and this is very helpful for people to locate their positions. The proposed system has several distinct advantages over all other location positioning systems. First, it is an economical solution because the marker that identifies each location is simply printed on paper, and mobile phones with cameras are general devices that can be implemented in software. Second, our system is not limited by signal propagation and multiple reflections.

The remainder of this paper is organized as follows. “Related works” introduces related works in detail. “The system architecture” provides detailed insight into the architecture of this system. “System implementation” describes the system implementation, and “Experiments” details the experimental results, examples of this scenario are also given. Finally, the paper is concluded in “Conclusions”.

Related works

Over the last few years, the application of AR in location services has attracted industry attention as a means of reducing the level of abstraction for the user and of improving position accuracy.^7,8 AR uses a calculated field position and camera angle to impose a layer of virtual objects over the “real-world” background.⁷ Through AR, users can not only immerse themselves in the combined virtual and real-world scenes but also interact with the virtual objects and access relevant and useful information.⁹

According to the identification and information acquisition, mobile AR technology is classified into location-based AR and vision-based AR. Location-based AR uses the mobile device’s GPS or uses current location information achieved by the network to first position an area, and then the virtual information corresponding to the current position is obtained.¹⁰ The first successful location-based AR application is called iButterfly.¹¹ By using GPS, iButterfly obtains the users’ positions, and then users can capture the butterflies that are set beforehand to achieve business objectives. The Wikitude AR Travel Guide, created around 2007 with the advent of the iOS and Android platforms, was the first popular AR application for the G1 Android phone, and it located and displayed tourist attraction information.^10,11 However, due to the absence of targeted technical support, AR application functions on smart terminals remained very simple for the following two years. Over time, Apple Inc. became aware of the prospect of AR applications and, therefore, provided support for AR in their iOS 3.1 (iPhone 3Gs) released in September 2009, and thus began to promote the development of the AR operating system level. After the 3Gs release, the first AR-based LBS applications, “Nearest Tube”, appeared on the iPhone, which was used to check the position of the subway in London.^12,13 With the development of the iPhone4, Apple Inc. provided gyroscope technology to LBS, which brought a more stable and smooth screen experience to AR. The first client to use AR+LBS technology was the famous consumer reviews site “TryLive”, that allowed users to use their mobile phones to search for information about businesses that were located around them. Another location services-based social networking site from the USA, Brightkite, cooperated with McDonald’s in developing an AR-based system in which users could open their mobile phone on a street, and any McDonald’s stores around the user would be displayed.^14,15 In addition, combined with the Brightkite user system, the user can see how many friends are nearby to upload their photos, add comments and so on. Subsequently, the famous AR R&D (Research and Development) company called Layer was established, and an augmentation content store was released in April 2010, which has since become a platform for companies in this field.¹⁶ In Park et al.,¹⁷ an AR-based system was developed for showing cultural heritage; meanwhile, in Barile et al.¹⁸ and Lee et al.,¹⁹ the lane-level guidance and the drive service had paid attention to AR technology which would help drivers change lanes and navigate.

Different from location-based AR, vision-based AR must first obtain images by a camera, and then match the captured images with a given template. The template registers the corresponding virtual information on the images.²⁰ Compared with location-based AR, vision-based AR provides more information about the specified target in real scenes by virtually adding text, images and videos. Recently, AR experiential campaigns focus on the entire experience developed principally for the consumer.²¹ And some people have applied AR for teaching.²² However, it cannot accurately designate the target position. Conversely, location-based AR can acquire the target position in the real scene. However, its navigation is not able to generate accurate position data when the destinations are within a small area, for example, the buildings in a university campus.

The system architecture

This recommendation system is developed under the Android system of intelligent mobile phones. It combines AR with navigation technology. By obtaining detailed information, it not only customizes to the user, but also shows the user a correct route to the target location. To achieve these goals, this system realizes the modules of user location, marker detection, 3D display and location guide. The system architecture is shown in Figure 1. It begins by reading the real scene images taken by a mobile phone. Then landmark detection is accomplished through a comprehensive analysis of image objects and landmark information based on basic image shapes and the topology relationship. The extraction, detection and identification of the real scene image marker of the AR system are next. Then the marker ID is determined. The marks are all different from one another and any given building is represented by a different marker. Markers carry much information about the building that the marker represents. Once the marker is identified, the position and orientation of the camera with respect to the marker is computed. Then its corresponding 3D virtual object is generated based on the recognized ID. Finally, a virtual registration of virtual objects is completed according to the camera position and orientation, and the effect of Fusion Reality is output to the display device. In addition, the route guide can be generated according to the current position of the marker in a situation in which the user cannot ensure the current position. Details of each model are described in the following.

Figure 1.

System architecture.

3.1 Landmark detection

This process detects a marker from an input image. It is the most important step for AR-based mobile location recommendation. The image illustrates the environment in front of the user, for example, doors, walls or hallways. This process primarily distinguishes the candidate marker and non-marker regions (e.g. background) in the image and detects real landmarks in the candidate marker regions.

Usually, the analysis of image target information is mainly based on image features of objects in the scene such as the shape, the color and the target connection. In this study, in order to obtain the target image information, a method²³ that retrieves the target image from an image database according to the shape of the input object is employed. This image retrieving method accounts for various factors related to mobile location services. For the details of this retrieving method, the literature²⁴ can be referred to. The landmark detection process is shown in Figure 2.

Figure 2.

Landmark detection.

In general, images taken from mobile devices contain a lot of noise. Furthermore, simply smoothing out the images also degrades edges. To address this problem, a bilateral filter is employed in this paper. The bilateral filter²⁵ is a nonlinear, edge-preserving smoothing filter. It is usually defined as

B_{x} = \frac{1}{k (x)} \sum_{ξ \in Ω} c (ξ, x) s (I_{ξ}, I_{x}) I_{ξ}

(1)

where $B_{x}$ is the output bilateral filter result and $Ω$ denotes the neighboring pixels locations, including the current location x. $c (ξ, x)$ and $s (I_{ξ}, I_{x})$ are Gaussian functions that measure the geometric closeness and photometric similarity between the neighborhood center x and a nearby point $ξ$ . $I_{ξ}$ represents the intensity value of $ξ$ . $k (x)$ is a normalized factor that is defined by

k_{x} = \sum_{ξ \in Ω} c (ξ, x) s (I_{ξ}, I_{x})

(2)

By means of a nonlinear combination of nearby image values, the bilateral filter smoothes images while preserving edges.

After removing noise, and to detect landmarks, vertical edges were detected by applying dual vertical edge filters: canny edge detection in grayscale and saturation images. Saturation images were used in order to detect gray buildings in the blue sky background. Some edges were detected well in the grayscale, whereas others were better detected in the saturation images. After any edges were detected, a mask was applied to remove short, adjacent and horizontal edges, and then the Hough transform was applied to obtain line equations. For an efficient process, a method must be introduced to reduce the extent of each image that must be searched for a marker. Therefore, we adapted vision techniques to a process that aids in marker detection. This process consists of image threshold, labeling, contour detecting, square checking and pattern matching. First, the input images are turned into binary images using the threshold method with a specified threshold value. Then, square regions (candidate marker regions) are distinguished from each binary image by the labeling, contour detecting and square checking processes. The marker detection process finds all the squares in each binary image. For each square region, a pattern inside the square is scanned and compared to pre-trained pattern templates. As a result, location information is detected by matching the detected pattern in a square region to a pre-registered pattern template that designates a specific location in a user’s environment.

3.2 Location calculation

In the location-based services, the system should dynamically obtain users’ location information and this information must be able to meet a certain accuracy. Currently, the most used technologies related to location knowledge discovery includes the location knowledge discovering techniques for discovering network information (such as text, video, audio and images) and pre-recognition techniques for unknown location. However, the social attributes related technologies for location include the relationship of positions and people (map matching, transportation identifying, users’ frequent pattern mining and user environment judging), the relationship of positions and positions, the location-based social network link prediction and so on.

In this study, based upon traditional GIS (Geographic Information System), the concept of Sensor is introduced to achieve an all-dimensional information registration of the computer-generated virtual objects and users’ real environment first. Then, a user’s motion trail in the 3D space is obtained by a user’s sensor, as well as from Web galleries,^26,27 maps provided by Google or Baidu and image landmark information databases that come with the system. Then, the landmark objects in a mobile captured image are retrieved from these databases. Next, the location of the image landmark is computed, and finally, a located user’s coordinates and azimuth information is determined. During these processes, a transformation termed as landmark coordinates to a camera coordinate system, should be done, as is shown in Figure 3. In which, the origin of the landmark coordinate is the center of the landmark, and its z-axis is perpendicular to the Web map surface, which is given as

P_{m} = [X_{m} Y_{m} Z_{m} 1]^{T}

(3)

Figure 3.

Coordinate system.

Additionally, the camera coordinates z-axis is perpendicular to the lens surface as

P_{c} = [X_{c} Y_{c} Z_{c} 1]^{T}

(4)

where T is a matrix that represents landmark coordinates with reference to camera coordinates. $P_{m}$ is the 3D position of the landmark in the world coordinate, and $P_{c}$ is the 3D position of the landmark in the camera coordinate. $X_{m}$ , $Y_{m}$ and $Z_{m}$ are the world coordinates of the landmark. $X_{c}$ , $Y_{c}$ and $Z_{c}$ are the camera coordinates of the landmark. The relative transformation between the landmark and camera is

P_{c} = T * P_{m}

(5)

In order to improve the location computation performance, a user-based collaborative method was also employed in this study, in which the system makes recommendations based on groups of users with similar preferences through matching a user’s opinions with the ones of the other users in the system. The similarity between users is normally computed by comparing the ratings that they give to some of the items. When the system identifies which people share similar interests with the current user, the items that those people liked are recommended to this user. In this approach, some feedback about the provided recommendations is necessary in order to know which items the user has liked or disliked (e.g. which places the user enjoyed visiting).

One of the main objectives in our mobile recommender system proposal is to add the capacity of adapting recommendations to the current user’s location. To do so, we propose the use of a location-aware pre-filtering to provide recommendations. The location-aware pre-filtering defines the recommendation areas considered by the system through the means of a Router.^28–31 The “Router” is a user defined parameter that establishes the limits where the items might be recommended.

Given a set of recommendable items that are geographically located as

A = {a_{1}, a_{2}, \dots a_{n}}

(6)

where $a_{i}$ is the ith recommendation item, then an a subset, $A' \in A$ , that includes those items that are suitable to be recommended to the user according to his or her location, is computed by the parameter “Router”.

Next, the subset $A'$ is re-ranked according to the interest rate $ν_{i}$ of the item $a_{i}$ for the target user. The interest rate $ν_{i}$ of the item $a_{i}$ for the target user is computed according to the distance from each item $a_{i}$ to the target user using the following formula³²

υ_{i} = e^{- λ \times dist (u, a_{i})}

(7)

where $λ$ is a parameter that express the user’s distance sensitivity, and $dist (u, a_{i})$ is the Euclidean distance from the user u to the item $a_{i}$ . The value of $λ$ is determined by experiments.

Finally, a recommender list $RL = {(a_{i}, ν_{i}), i = 1 \dots N}$ , sorted by $ν_{i}$ in descending order, will be obtained.

3.3 Enhancing mobile recommendation with augmented reality

Currently, the AR technology primarily uses mobile phone positioning systems and inertial sensors (GPS, compass, acceleration sensors, etc.) to determine the location of the terminal. Then this is combined with the back end information servers and the related virtual content provided by the search, advertising and image recognition engine. However, such a solution based on a mobile compass or GPS cannot be closely combined with camera-photographed images because of large position error.

In this study, to obtain the relative orientation of the camera so as to realize the superposition of the virtual information and the real scenes information seamlessly, the user pointed object is identified by processing the camera captured data, and the target image is tracked and analyzed frame by frame. Figure 4 details the processing procedure step by step.

Figure 4.

Calculation of the AR view.

Firstly, a user photographs the scene around him or her with a mobile phone, which then reads the real scene images and uploads them to a sever. Then the image objects and the landmark information in the image are analyzed based on the basic shape and its topology. Next, the mark of the real scene image of the AR system is extracted and identified. Finally, the ID of the mark is determined according to the business datum and information. The marker is designed as a black square within a pattern inside. The pattern on the marker refers to the building. Each marker has its own feature for each building. These markers are stored as the template images and are used for consequential image recognition. The images with data are given an identifier that is linked to data of the buildings. If the whole marker is not included in the image, the system will not provide the information for users. When the mark is identified, it is combined with the GIS information and then used to compute the position and orientation of the camera with respect to the mark: that is, the external parameter matrix of the camera. Then, its corresponding 3D virtual objects are generated based on the recognized ID. Finally, a virtual registration on virtual objects is completed according to the camera position and orientation, and the effect of Fusion Reality is output to the display device, recommended items are also portrayed on this 3D map according to the user location on the real world. As depicted in Figure 4, the map is rendered under a perspective projection, allowing users to visually search over entire cities or areas even on small mobile displays. Consequently, tourists can become familiar with the zone they are visiting and form an accurate cognitive map of the area around them. As well, 3D virtual representations are, in general, easier and faster to understand than two-dimensional (2D) abstract maps.

Location annotation

The immersive user experience is further enhanced by matching the virtual view with the user’s current view in the physical world according to his/her location and view direction. To seamlessly annotate the location information on the user’s view, we used the OpenGL graphics library. The location information is provided using virtual 3D direction sign graphics and text such as “go straight”, “turn left” and “turn-right”. The location information is obtained from a real scene taken with a cell phone camera, which is used as a reference point for target tracking. With this reference point, the image location mark can be determined. After identification of the image location mark, the position and orientation of the camera with respect to the image mark is computed. Here, the image coordinates are a 2D system and are given as

P_{img} = [x y 1]^{T} * P_{c}

(8)

where $P_{img}$ is transformed points from the camera coordinates in to the OpenGL perspective viewing volume, and x and y are the 2D camera coordinates of a reference point. $P_{img}$ will be used as the location for rendering virtual models. Then, the corresponding 3D virtual object based on the recognized mark is generated. Next, the virtual objects are registered according to the camera position and orientation. Finally, the effect of fusion reality is output to the display device. The location information is annotated on the top-left position of the user’s view. The user can see the 3D sign on the input image.

System implementation

For the implementation of the proposed system, a mixture mode of client-server and browser-server (B/S) was employed in this study, as shown in Figure 5. The client application is used to support the user’s daily activities and to guide him/her through the environment. A server is used to enforce synchronization between client applications and web services. The AR components and the GIS components were designed independently, the AR components were coded with OpenGL and the GIS components were coded with Java. Meanwhile, the Sensor concept was introduced into the traditional GIS application. The Sensor involves user input and external data resources. It combines a traditional paper map image of a locality with a satellite image. Through the system’s user interface, a user provides a custom map of a particular location. A corresponding satellite map is retrieved from an internet source, such as Google Maps. As a result, users can not only locate the coordinate information, but they can also obtain other users’ location information according to other users’ sensors, making the users’ datum acquisition range more accurate.

Figure 5.

The mode used for system implementation.

The system is developed by AutoNaviMap API (Application Programming Interface), OpenGL API and AR API, and is tested on an Android mobile phone. Auto Navi MAP API provides the system’s GPS support. OpenGL API provides the system’s display support and AR API provides the system’s identification support.

To end-users, the system operates in the model of camera-view. To easily operate various functions that are provided by the system, the operation interface was designed to include button and drop-down list controls. Figure 6 shows the system interface implemented into an Android-based device. It tracks user’s geo-locations using a GPS and a compass, as well as an accelerometer, and then displays an overlay of a custom map image with live scenes through the device’s rear camera. This map indicates and tracks a user’s current location and heading direction. It can be zoomed in and out or rotated in any direction. Every known point of interest is clearly marked and labeled. While walking, the information is updated in real-time, following the user’s movement.

Figure 6.

Screen shots exemplifying the system’s user interface.

Experiments

We have built a fully operational prototype that implements the ideas described in previous sections, and it has been subjected to empirical experimentation. All experiments were performed in downtown Xi’an by capturing images with a smart phone and with GPS and digital compass measurements. The goal of these experiments was to answer the two following questions, which reflect the main contributions of the paper.

Is the AR a good way to provide location-aware recommendations on mobile devices?

Does location-awareness increase the quality and usefulness of the system?

The first experiment attempts to formally quantify the efficiency of the algorithm for landmark information detection. In this experiment, a total of 100 locations on images captured during day and night with users’ mobile phones were used. We followed equation (7) to test the performance of the location detection system.

R = \frac{c}{n} \times 100

(9)

where n is the number of locations included in the captured images, c is the number of locations recognized by the system and R indexes the landmark detection rate; a high R demonstrates a better detection performance of landmark. For this experiment, the term “ground-truth marker” is used to denote the ground-truth bounding square marked around a region of each marker. The ground truth markers consist of the code marker data and the position of the upper left corner of the marker(s) in the given testing images. These testing images are manually selected and marked by an experienced expert. The marked images are then sent to the Google Maps API to compute the position of upper left corner of the marker. By this way, the “ground-truth marker” is automatically generated. The experimental results are shown in Table 1. As can be seen from Table 1, our test yielded a landmark detection rate of 98% in daylight and a landmark detection rate of 90% at night. This is acceptable for this type of application.

Table 1.

Evaluation of landmark detection.

	Ground-truth marker	The number of location recognized by the system
Images during day	100	98
Images during night	100	90

We also evaluated the proposed system’s location performance through comparing it with a traditional GPS-based location system. This evaluation was performed by a total of 10 real end users (none of them were inexperienced with smart phones). The evaluation consisted of two test sets: one set used a commercial GPS navigator and the other one used our proposed system. In order to ensure that each test person has the same initial information, we presented information about how the system works and provided information about the city of Xi’an to each of the test persons. Afterwards, we gave each test person the task of planning a city tour in Xi’an. There were certain points that the test persons had to implement into their tours. For example, they should stay nearby the “Bell Tower”, a famous Chinese attraction in Xi’an. They had the opportunity to use their smart phone to acquire images and information about attractions, as well as a list of attractions in the area. In this way, we ensured that all of the test persons had the same chances to implement an attractive city tour. The experimental results are shown in Table 2 and Figure 7, respectively. It can be seen from Figure 7 that with the proposed location system, 67% of false landmarks can be removed with only one true landmark lost. However, with the traditional GPS-based system, only 46% of false landmarks can be removed with one true landmark lost.

Table 2.

Evaluation of the usability of the system with respect to the user interfaces.

Methods	Number of people with location accuracy
With GPS	8
With our system	9

Figure 7.

Evaluation of the usability of the system with respect to the user interfaces.

Table 2 confirmed the above results. As can be seen in Table 2, 8 of 10 users with the commercial GPS can locate their position accurately; however, 9 of 10 users with our application can locate their position accurately.

Users’ evaluation and impressions in the above mentioned experiment were also collected using the following questionnaire, and the results are shown in Figure 8.

Q1 I’m satisfied with the comments suggested by the traditional GPS-based location system.

Q2 I’m satisfied with the comments suggested by the proposed location system.

Q3 Taking into account my location increases the usefulness of the recommendations.

Q4 I found it useful to adapt recommendations to the distance I’m willing to travel.

Q5 I prefer an augmented reality-based system over a traditional GPS-based location system.

Figure 8.

User’s evaluation and impressions of the system.

It can be found from Figure 8 that evaluators showed a clear preference for the proposed location recommendation system over the traditional GPS-based location system. Comparing the user satisfaction expressed in Q1 and Q2, we observe that the evaluations were very positive for both versions; however the proposed location recommendation system was substantially more satisfying. Questions Q3 and Q4 answer the second question formulated at the beginning of this section: our experimentation proved that location-aware recommendations were perceived more useful than non-location-aware ones by most users.

The evaluators were also asked to compare the proposed system with the baseline GPS-based variant; most of them preferred the proposed system (Q5: 60%). This result seems to indicate that, in general, users felt very attracted to the AR-based recommendation system, although there is still room for improvement.

We also evaluated the usability of the system with respect to the user interfaces and methods of visualizing information through the above mentioned experiment. Upon completing them, evaluators filled out a usability questionnaire once per tested variant. Following, we reproduce the questionnaire, which included the below questions.

Q6 It was easy to use this system.

Q7 The information provided by the system is clear and easy to understand.

Q8 I think that this system is of good practical use.

The results are shown in Figure 9. As can be seen from Figure 9 the testers have expressed that both systems were equally easy to use (Q6). However, the proposed system is more clear than GPS-based systems to understand (Q7) and has good practical use (Q8). These results confirm our initial hypothesis that the AR-based system is a valid tool for providing location-aware recommendations on mobile devices.

Figure 9.

Evaluation of the usability of the proposed system.

A further evaluation on the proposed system’s location performance during different periods in one day was also given. In this experiment the proposed system is compared with a GPS-based location system. The experiment results are shown in Figure 10. As can be seen from Figure 10, a significant improvement in location performance can be obtained with the proposed system during the period of day (i.e. from 10 a.m. to 12 p.m., and from 6 p.m. to 8 p.m.). However, during the period of night (i.e. from 9 p.m. to 10 p.m.), the location performance with different location systems is almost equal; this means that using solely GPS is almost already enough at night. The reason for the difference in the results is that extracting distinguishing local features is very difficult for night queries, which is quite different from the day time.

Figure 10.

A comparison of location performance during different periods of one day with different location systems.

The effect of image de-noise processing on landmark detection was also evaluated. In this experiment, a comparison is made of the image without de-noising as well as comparisons with various classical de-noising algorithms, as is shown in Table 3, in which, a total of 100 locations on images captured during the day is provided.

Table 3.

Evaluation of image de-noising on landmark detection.

De-noise method	Ground-truth markers	The number of locations recognized by the system
Without de-noising	100	80
Gaussian filter	100	85
Median filter	100	90
Filter used in work	100	98

It can be seen from Table 3 that image noises seriously affect landmark detection. For example, without de-noising, only 80 of the 100 landmarks can be correctly recognized by the system from images captured with a smart phone camera by the system. However, after the image is de-noised, landmark detection accuracy is obviously improved. This is clearly demonstrated by both the Gaussian filter and the Median filter as these methods improve the accuracy to 85% and 90%, respectively. However, the bilateral filter that was used in this work improved the accuracy to 98%. These results demonstrate that the approach employed in this study is superior to other approaches.

The location calculation method used in this work was also evaluated by comparing it to other existing similar methods, such as those reported in Keitler⁶ and Kim and Jun.¹¹ The methods’ performances were evaluated using the free response receiver operating characteristic (FROC) curves. The FROC curve expresses an overall sensitivity as a function of the number of false positives per image at a specific operating point on the curve. The results are shown in Figure 11.

Figure 11.

Average location results with different location methods.

It can be seen from Figure 11 that the method used in this work results in a 63% reduction in the number of error locations per image (from 6 to 2.2 per image) while maintaining an original sensitivity between 75% and 86%. The method in Keitler⁶ results in a reduction of 55% (from 6 to 2.7 per image) while maintaining a sensitivity between 70% and 86%, and the method in Kim and Jun¹¹ results in a reduction of 46% (from 6 to 3.2 per image) while maintaining a sensitivity between 68% and 86%. These results suggest that the performance of the method used in this work is superior to that of others.

All of the above experimental results clearly demonstrate that the proposed system is both efficient and effective in helping people locate their position. As an example, an application case of the system is shown in Figures 12–13. Figure 13 shows the input image overlaid with the user’s current location and a direction sign, as well as text boxes that display the recognized location. This interface is displayed on the mobile phone.

Figure 12.

An application case of the system.

Figure 13.

A global view of the application case.

Figure 13 shows a global view of the application case in Figure 12. When the system calculates a route to the destination, the user can activate the AR mode. The information entities are then shown, including their geographic coordinates. The data that determine the position of each entity on the screen is obtained from the mobile device via GPS and a digital compass, and the entities are displayed as graphical icons. A set of identifying colors and other visual characteristics has been established to distinguish each type of entity (see Figure 13).

The user can interact with the AR view by touching an entity. A description of the area is then displayed, and it is possible to open the multimedia resources related to the task or location.

Conclusions

In this paper, a novel system based on AR for mobile location publication and sharing is developed. In this system, the B/S development mode is employed, and the AR and mobile location information sharing technology is combined. In order to evaluate the performance of the proposed system, experiments in downtown Xi’an were performed by capturing images with a smart phone and with GPS and digital compass measurements. This evaluation was performed by a total of 10 real-end users (none of them were inexperienced with smart phones). The location calculation method used in this work was also evaluated by comparing it to other existing similar methods using the FROC analysis. Compared to the traditional GPS location services system, our method can be easily applicable utilizing existing maps, and it is simple and computationally low. Meanwhile, by combining computer graphics with real world objects, vision-based AR can provide users with a more intuitive, 3D, dynamic information display and sharing capabilities, thus enhancing the user’s senses experience.

The proposed system can be applied to various consumer applications, such as the door plate system, notice board system, shopping assistance system, bus service route guide system, and so on. It is especially suitable for urban environments with tall buildings where GPS measurements are inaccurate.

Footnotes

Acknowledgements

The authors gratefully acknowledge the helpful comments and suggestions of the reviewers.

Academic Editor: Zhipeng Cai

Declaration of conflicting interests

The authors declare that there is no conflict of interests regarding the publication of this paper.

Funding

This work is partially supported by a grant from the National Natural Science Foundation of China (grant number 61202198), Nature Science Foundation of Shaanxi province (grant number 2014JQ8299), Nature Science Foundation of Shaanxi Education Department (grant number 2013JK1136), the Nature Science Foundation of Science Department of PeiLin count at Xi’an (grant number GX1213), and Specialized Research Fund for the Doctoral Program of Higher Education of China (grant number 20136118120010).

References

Skinner

Long

Vice

. Augmented interaction: Applying the principles of augmented cognition to human-technology and human-human interactions (Lecture Notes in Computer Science, vol. 8027). Berlin, Heidelberg: Springer, 2013, pp.764-773.

Gao

Liu

Personalization in web computing and informatics: Theories, techniques, applications, and future research. Inform Syst Front 2010; 12: 607–629.

Adomavicius

Tuzhilin

Toward the next generation of recommender systems: A survey of the state-of-the-art and possible extensions. IEEE Trans Knowl Data Eng 2005; 17: 734–749.

Schobesberger

Integrating user and usability research in web-mapping application design, modern trends in cartography. Berlin: Springer International Publishing, 2015, pp.147–158.

Azuma

Baillot

Behringer

. Recent advances in augmented reality. IEEE Comput Graph Appl 2001; 21(6): 34–47.

Keitler

. Mobile augmented reality based 3D snapshots. In: Proceedings of IEEE and ACM international symposium on mixed and augmented reality, Orlando, FL, USA, 19–22 October 2009, pp.199–200.

Ryberg

Christiansson

Eriksson

. Accuracy investigation of a vision based system for pose measurements. In: The 9th international conference on control, automation, robotics and vision (ICARCV’06), 5–8 December 2006, pp.1–6. Singapore: IEEE.

Borras

Moreno

Valls

Intelligent tourism recommender systems: A survey. Expert Syst Appl 2014; 41: 7370–7389.

Qin

Liu

Qiu

. Campus navigation system based on mobile augmented reality. In: Proceedings of 2013 6th international conference on intelligent networks and intelligent systems (ICINIS), 1–3 November 2013, pp.139–142. Shenyang, China: IEEE.

10.

Marner

Smith

Thomas

BH.

Mapping 2D input to 3D immersive spatial augmented reality. In: IEEE symposium on 3D user interfaces, Arles, France, 23–24 March 2015, pp.171–172. IEEE.

11.

Kim

Jun

HS.

Vision-based location positioning using augmented reality for indoor navigation. IEEE Trans Consum Electron 2008; 54: 954–962.

12.

Paucher

Turk

. Location-based augmented reality on mobile phones. In: Proceedings of IEEE international conference on computer vision and pattern recognition workshops, San Francisco, CA, 13–18 June 2010, pp.9–16. IEEE.

13.

Langlotz

Wagner

Mulloni

. Online creation of panoramic augmented reality annotations on mobile phones. IEEE Pervasive Comput 2012; 11: 56–63.

14.

Huey

Sebastian

Drieberg

Augmented reality based indoor positioning navigation tool. In: Proceedings of the 2011 IEEE conference on open systems, Langkawi, Malaysia, 25–28 September 2011, pp.256–260. IEEE.

15.

Cheng

Lin

. AR-based positioning for mobile devices. In: Proceedings of the 40th International Conference on Parallel Processing Workshops, Taipei City, 13–16 September 2011, pp. 63–70. IEEE.

16.

Jaewon

Kyusung

Cho

Rojas

. Real-time scalable recognition and tracking based on the server-client model for mobile augmented reality. In: Proceedings of IEEE international symposium on VR innovation, Singapore, 19–20 March 2011, pp.267–272. IEEE.

17.

Park

Lee

Park

. Digital map based pose improvement for outdoor augmented reality. In: Proceedings of IEEE international symposium on mixed and augmented reality, Atlanta, GA, 5–8 November 2012, pp.309–310. IEEE.

18.

Barile

Calandra

Caso

. ICT solutions for the OR. C. HE. STRA project: from personalized selection to enhanced fruition of cultural heritage data. In: Tenth international conference on signal-image technology and internet-based systems (SITIS), Marrakech, 23–27 November 2014, pp.501–507. IEEE.

19.

Lee

J-W

Yoon

C-R

Kang

. Development of lane-level guidance service in vehicle augmented reality system. In: Advanced communication technology (ICACT), Seoul, Korea, 1–3 July 2015, pp.263–266. IEEE.

20.

Eyraud

Zibetti

Baccino

Allocation of visual attention while driving with simulated augmented reality. Transp Res Part F Traffic Psychol Behav 2015; 32: 46–55.

21.

Stoyanova

Georgieva

. Comparison of consumer purchase intention between interactive and Augmented Reality shopping platforms through statistical analyses. In: Innovations in intelligent systems and applications (INISTA), Madrid, Spain, 2–4 September 2015, pp.1–8. IEEE.

22.

Kysela

Pavla

Using augmented reality as a medium for teaching history and tourism. Soc Behav Sci 2015; 174: 926–931.

23.

Gherghina

Olteanu

Tapus

. A marker-based augmented reality system for mobile devices. In: Proceedings of 11th Roedunet international conference, Sinaia, Romania, 17–19 January 2013, pp.1–6. IEEE.

24.

Wanneng

Shu

Jiangqing

Wang

. An optimized multi-hop routing algorithm based on clonal selection strategy for energy-efficient management in wireless sensor networks. Sens Transduc 2013; 22: 8–14.

25.

El Choubassi

Kozintsev

Augmenting 3D urban environment using mobile devices. In: Proceedings of 10th IEEE international symposium on mixed and augmented reality, Basel, Switzerland, 26–29 October 2011, pp.241–242. IEEE.

26.

Wanneng

Shu

Lixin

Ding

Xingdong

. Chaotic clone optimization algorithm based on equivalent division strategy. Chongqing Daxue Xuebao (Ziran Kexue Ban) 2012; 35(8): 34–41.

27.

Chang

Qing

Tan

. Augmented reality system design and scenario study for location-based adaptive mobile learning. In: Proceedings of IEEE 13th international conference on computational science and engineering, Hong Kong, 11–13 December 2010, pp.20–27. IEEE.

28.

Srisuk

. Bilateral filtering as a tool for image smoothing with edge preserving properties. In: Electrical engineering congress (iEECON), 2014 international, Chonburi, Thailand, 19–21 March 2014, pp.1–4. IEEE.

29.

Yang

Hwang

SY.

ITravel: A recommender system in mobile peer to peer environment. J Syst Softw 2013; 86: 12–20.

30.

Gavalas

Konstantopoulos

Mastakas

. Mobile recommender systems in tourism. J Netw Comput Appl 2014; 39: 319–333.

31.

Ricci

Mobile recommender systems. Int J Inf Technol Tour 2011; 12: 205–31.

32.

Xinyu

Zhongchun

Zhenmei

Zhang

. A location-aware recommender system for tourism mobile commerce. In: The 2nd international conference on information science and engineering, Hangzhou, China 4–6 December 2010, pp.1709–1711. IEEE.