Development of Mobile Social Network Systems Using Real-Time Facial Authentication and Collaborative Recommendations

Abstract

We propose a location-based mobile social networking system (SNS) that integrates biometric facial authentication using a modified local directional pattern (MLDP), collaborative content recommendations, and communication among users. The proposed system is an effective fusion of a traditional social networking system, a location-based system, and a provider of intelligent recommendations. The prototype and service scenario in this study show possibilities for technical advancement and the future extension of mobile social networking services. To verify the proposed system, we performed experiments to determine the recognition rate for a user's facial image on a real-world smartphone and the preference prediction accuracy of a collaborative filtering-based recommendation system. Through our implementation and preference prediction experiments, we confirmed that the proposed system is highly effective and applicable to convergence using a location-based service and a content recommender.

1. Introduction

A social networking service (SNS) focuses on building and reflecting social networks and social relations among people who share interests or activities. It provides personal connections to individual users via Internet applications by offering communities, instant messenger programs, blogs, and user-created content services. In recent years, SNSs have become popular through the ubiquitous computing concept. SNS is now evolving from a PC-based service to a mobile-based service via an IT convergence platform, and a location-based system (LBS) is optimal for the new mobile environment. A mobile SNS is the SNS and LBS convergence product [1–4].

Studies of LBS trends show that the system is becoming a more general purpose for users with global positioning system (GPS) enabled mobile devices, such as smartphones, tablet PCs, and portable multimedia players [5, 6]. Existing studies, products, and services related to GPS-enabled mobile devices show that the devices provide a way to share various location-tagged content generated by the user, such as photographs, movie clips, text, and sound. A mobile device user with such a system can access location-tagged content and then share the content via that device [7–9]. SNSs that were formerly PC based have now converted to a mobile platform. This technological and industrial trend has developed throughout the years [10]. Recently, a problem regarding the information security of SNSs and LBSs has resurfaced, for they contain the user's personal data, including their location data [11]. Furthermore, the flood of information shared by users from their mobile device at any time and from anywhere is becoming an issue. This low-quality content introduced to the system eventually extends the time required to search for more useful information. Therefore, a program within the framework of a personal LBS on a mobile device is required to resolve this problem.

In this paper, we present a personal location-based mobile SNS that integrates facial recognition-based user authentication, collaborative recommendations, and communication among users of an SNS. The proposed system is a convergence of the SNS, the LBS, and a recommendation system (RS) and consists of a single security module, the user authentication module (UAM), two key filtering modules, a distance filtering module (DFM), and a preference filtering module (PFM). First, the DFM searches for location-tagged content within a specific user-centered radius and improves the recommendation efficiency by determining the user's current location. The PFM, as the RS, then selects and recommends content of interest by considering the user's preferences based on collaborative filtering (CF) [12, 13]. The PFM determines and personalizes the real-time recommendations using a memory-based CF framework. The UAM improves security for hand-held PC SNSs by using the user's facial image. We applied facial recognition software to a smartphone in unconstrained illuminated conditions using modified local directional pattern (MLDP) and image covariance-based face feature extraction algorithms, such as two-dimensional principal component analysis (2DPCA) [14] and the alternative two-dimensional principal component analysis (A2DPCA) [15], which are accurate high-speed pattern recognition algorithms suitable for the mobile environment [16–18]. Then, the MLDP image is directly inserted into the image covariance-based face feature extraction algorithm. In comparison with a previous work using binary pattern descriptors such as local directional pattern (LDP) to extract histogram features, the proposed method has enriched a different approach.

Generally, the binary patterns used in a previous work have produced long histograms, leading to high computational complexity for a large face database. Therefore, this work directly utilizes a face image transformed by the MLDP descriptor as the input data for the image covariance-based feature extraction algorithms. The proposed approach has the advantage of retaining illumination effects which can be degraded by the MLDP descriptor. Because the image covariance-based feature extraction algorithms, such as 2DPCA and A2DPCA, are line-based local features, they are also more robust to illumination variation than global features, such as principal component analysis (PCA) [19]. The convergence of DFM and PFM is a novel approach toward solving the scalability problem of the CF.

In our proposed SNS, we intend to expand the application of CF to the LBS. We suggest a prototype of an LBS-based personalized mobile SNS that uses RS and mobile-based real-time facial authentication, which is a state-of-the-art authentication method. The facial image used for authentication can also be used to represent the user on the mobile SNS. We proved that the CF-based RS is effective for an LBS-based SNS with an in-depth CF experiment. The experimental results showed that the CF was excellent at recommending location-tagged content, as well as making traditional recommendations such as movies, books, and other e-commerce content. We also showed the successful convergence of LBS, RS, and SNS in the service scenario, which comprises an authentication process for a user's facial image and a syndication method for location-tagged content among people who share interests or activities.

This paper is organized as follows. In Section 2, we describe the SNS, LBS, RS, and mobile biometric facial authentication systems and explain existing studies and their products in detail. In Section 3, we propose a mobile social networking system using facial recognition-based user authentication and collaborative recommendation for location-based content as a convergence of SNS, LBS, and RS. In Section 4, we show the experimental results for each module, including the facial recognition rate and the user preference prediction accuracy of the RS. We explain our prototype in Section 5 and conclude our study in Section 6.

2. Related Works

2.1. Location-Based System and Service

An LBS is an information and entertainment service that utilizes the geographical position of a mobile device and is accessible via mobile devices through the mobile network. In the United Kingdom, networks do not use triangulation. The LBS services use a single base station with a radius of inaccuracy to determine a phone's location. A mobile carrier company in Singapore implemented an LBS system in 2007 that involved many local marketers and was reported to be a huge success in terms of subscriber acceptance. Mobile systems do not restrict users to a fixed desktop location but allow ubiquitous mobile activities. Tourist guides are the most common application scenario for location-based services. A mobile city guide for the city of Lancaster was one of the first systems to integrate “personalized” information for the user. More recent projects have integrated multimedia into mobile city guides. However, these systems do not address the dynamic generation of personalized content. This is the objective of the current study [5].

In Web development, a mashup is a Web application that combines data from more than one source into a single integrated tool. The term mashup implies easy, fast integration that is frequently performed by accessing open APIs and data sources to produce results that were not the original goal of the data owners. A study by Floyd et al. showed how mashup techniques can be used for rapid prototyping in the user-centered software development processes [6]. A study at the Human-Computer Interaction Institute of Carnegie Mellon University showed that mashups can even be utilized in user programming [7]. Another study emphasized the great benefits of “enterprise” mashups, which are information-heavy applications that integrate distributed business information within an enterprise in a quick and dynamic way [8]. Most existing mashups have been programmed manually. However, there are a number of mash-up platforms that facilitate development. For example, “Mash-o-matic” [9] can be used to generate geomashups, which integrate information based on a certain location. With well-known online tools, mashups can be constructed out of predefined components and combined using interactive drag-and-drop methods. These were an early form of modern SNSs like Facebook, YouTube, and Myspace. The SNS acts to provide relationships among the users in an online environment. Looking into recent studies concerning SNSs using LBS, state-of-the-art intelligent methodologies for the mobile environment have been applied to SNSs. Yamamoto et al. proposed a social network service based on the locations of students [20]. To acquire the locations of students, data from both the login records of educational computers and the attendance records of classes in the university are required. The analysis of login records and attendance records is used to create a friendship relation map between SNS users. This study is a kind of intelligent location-based SNS in a conditional mobile environment using user behavior. Kim et al. focused on personalization, which is the process of tailoring information to individual users' characteristics or preferences [21]. He has also studied personalized SNSs using machine intelligence [22]. The study provides a social media platform composed of GPS-equipped smart phones, a social media server system, and integrated service ontology to achieve a personalized social user-created-content service. It enables a more intelligent and active information sharing system. The ontology server in the study includes predefined categories of family, friends, coworkers, romantic partners, and location-based relations, and service modules create a personalized recommendation list of user-created content in the SNS database.

SNS technology has been changing and evolving quickly to answer the needs of individuals. A recent issue at the application layer has been the convergence between mobile SNS and intelligent ability represented by personalization. This paper proposes a more sophisticated mobile SNS based on state-of-the-art content recommendation methods and an intuitive user authentication module.

2.2. Mobile-Based Biometric Facial Authentication

Biometrics technologies refer to the identification of individuals based on their distinguishing biological or behavioral traits. These traits include facial features, speech, fingerprints, gait, hand veins, retinas, irises, palm prints, ears, and written signatures. Biometric security systems provide convenience and a high degree of stability since they do not require passwords or physical tokens. The convenience of biometric security systems and their reliability have led to the integration of biometric systems into devices such as desktop computers, laptop computers, PDAs, smartphones, and mobile phones [14]. Few studies have investigated biometric security systems in mobile devices. For example, studies regarding facial recognition have been conducted by Venkataramani et al., Tao and Veldhuis, Hadid et al., Zuo and de With, and Rahman et al. [17, 18, 23–25]. Facial recognition algorithms are generally divided into six categories, as shown in Table 1.

Table 1

Algorithmic categories of facial recognition.

Category	Characteristic
Geometric	Extraction of geometric features of a facial image besides suffering from various problems similar to those in most traditional methods
Eigenface	Feature extraction based on principal component analysis (PCA). Its response is sensitive to changes in environment
Fisherface	Feature extraction based on Fisher linear discriminate (FLD). It is robust to changes in illumination
Support vector machine (SVM)	Uses PCA and SVM and shows a high recognition rate, although requires high calculation cost and time
Neural network (NN)	Recognizes the user's face using a sliding window based on a black-and-white image. The training process is difficult
Fuzzy NN	Shows a higher recognition rate than the NN-based method, but a higher calculation cost is required

2.3. Personalized Recommendation

Content-based filtering (CBF) chooses a new item based on items that have been explicitly rated by the user in the past. Though CBF is based on the user's explicit preferences, the method varies according to the information type. In the case of movie retrieval, for example, CBF searches for movies similar to those that have been highly rated by the user. Various attributes can be used, such as actor, genre, release date, or producer. If we can extract common features from the movies that have been rated highly by the user, then this approach should be effective. For image and music retrieval, CBF extracts an object from the image or a rhythm from the music and then searches for other images containing the object or for other music with the same rhythm.

Collaborative filtering (CF) is a state-of-the-art algorithm used to create personalized recommendations. It analyzes the preferences of each user and searches for users similar to the target user. It then predicts a rating for a new item based on the previous rating scores of similar users. Existing studies have proposed methods to improve the data scarcity problem, which decreases prediction accuracy. Recent study trends can be classified into the three approaches described below [12, 13].

The first method reduces the empty space of nonrated items by reducing the dimensionality of the user-item rating matrix using a model-based CF. The second method expands the memory-based CF framework and the use of additional information with an existing similarity method. The third method strengthens existing approaches by developing a new similarity method. Data scarcity problems result from incorrect calculations of the similarity between items or users, which is the main cause of less accurate predictions. For this reason, improving the similarity method is the most radical approach; however, it has the advantage of being applicable to the two other methods.

3. Proposed System

Figure 1 shows the architecture of the authors' proposed system. Every user in the system can register and request location-tagged multimedia content via an Internet-enabled mobile device. A user performing the registration uses the camera or microphone of his own mobile device to record content. At this point, the user's location, consisting of latitude and longitude, is tagged along with the content. The content, including the user's text comments, is sent to the content-sharing server as location-tagged content. These steps are denoted by ① to ④ in Figure 1. A user receives personalized content through a request, and then the user rates his preference after experiencing the content. This process is denoted by steps ⑤ to ⑨. The key modules of the proposed system, the DFM and PFM, are described in the next subsections.

Figure 1

The system architecture.

3.1. The UAM: User Authentication Module

In this subsection, we describe the facial recognition system, which is composed of image acquisition and facial region detection, feature extraction of the facial image, and a recognition phase. For this study, we employed the AdaBoost algorithm based on Haar-like features for facial region detection. The nearest neighbor classifier using 2D-PCA as a feature vector was also used in the process of facial recognition. Specifically, 2D-PCA computes the corresponding eigenvectors more efficiently than PCA because the size of the image covariance matrix is equal to the width of the images, which is quite small compared to the size of a covariance matrix in PCA. Thus, we employed 2D-PCA in the facial recognition system of a mobile device.

The first phase of facial recognition is facial region detection in an image acquired by a digital camera on a mobile device. For this study, we employed the AdaBoost algorithm based on the Haar-like features introduced by Viola and Jones for object detection [17]. The Haar-like features are used to search the face region, and the prototypes are trained until the face region is accurately represented through the AdaBoost learning algorithm. The AdaBoost algorithm is a method used to construct a strong classifier by combining weak multiple classifiers. A weak classifier $h_{j}$ is a single-layer perceptron and is defined by,

\begin{matrix} h_{j} (x) = {\begin{cases} 1 & if f_{j} (x) < θ_{j} \\ 0 & otherwise . \end{cases} \end{matrix}

(1)

Each weak classifier is associated with a feature $f_{j}$ and a threshold $θ_{j}$ , and x indicates the size of the subwindow. The Haar-like features obtained by the AdaBoost learning algorithm are classified in stages, as shown in Figure 2. An image is scanned at all possible locations and scales using a subwindow. A series of classifiers is then applied to all of the subwindows. The initial stage eliminates a large number of negative examples with very little processing. All of the subwindows are classified as “face” or “nonface” regions by analyzing the entire cascade. The number of processed Haar-like features is larger in the higher classification stages than in the lower stages, and the Haar-like features processed at a higher classification stage can render more detailed face regions.

Figure 2

Cascade structure for facial region detection.

To improve recognition performance in uncontrolled illumination environments, such as those encountered on smartphones, we perform a preprocessing procedure in which the MLDP operator is applied to the detected face image. Moreover, we devised a new methodology that directly combines the MLDP image with image covariance-based face feature extraction algorithms such as 2DPCA and A2DPCA. Note that this is a different approach than that used in a previous work, which employed binary pattern descriptors to acquire histogram features. Recently, local binary pattern (LBP) has received increasing interest for use in face representation to overcome the problem of performance degradation caused by illumination variation [26, 27]. The LBP operator labels the pixels of an image by thresholding a $3 \times 3$ neighborhood of each pixel with the center value and considering the results as a binary number. More recently, the LDP operator was introduced to allow for more robust facial expression recognition [28]. While the LBP operator uses information on intensity changes around the pixels, the LDP operator uses the edge response values of the neighborhood pixels and encodes the image texture. Given a central pixel in the image, the eight-directional edge response values $m_{i} (i = 0, 1, \dots, 7)$ are computed by Kirsch masks and are converted to absolute values. Then, the most prominent directions of the k number with high response values are selected to generate the LDP code. In other words, only the responses of the top-k bits are set to 1, and the remaining $(8 - k)$ bits are set to 0. In Figure 3, we depict the calculation process of the LDP code with $k = 3$ . Generally, it is known that the LDP code with $k = 3$ shows better performance than those with other k values [28]. Note that bits having edge response values of −399 and −503 are set to 1 in Figure 3, since the edge response values are converted to absolute values.

Figure 3

An example of LDP code.

To provide a more stable pattern from the impulse noise and to make an independent pattern against the k value of the LDP, we modify the LDP operator by considering all directions of the 3*3 neighborhood pixels. Because the LDP only selects k components with high response values, unselected components that had relatively high response values compared to those of the selected components are ignored during pattern creation. Thus, we devise the MLDP operator by considering the response values of all neighborhood pixels. This operator is derived as

\begin{matrix} MLDP = \sum_{i = 0}^{7} s (m_{i} - g_{c}) \times 2^{i}, \\ s (x) = {\begin{cases} 1, & x \geq 0, \\ 0, & x < 0, \end{cases} \end{matrix}

(2)

where

m_{i}

and

g_{c}

denote the ith most significant directional response of the neighborhood pixels and the center pixel value, respectively. Here, we use the center pixel value to encode image textures by comparing all response values, and we directly use the response values without performing the absolute operation at

m_{i}

, unlike in the LDP.

In Figure 4, we depict a sample image transformed using the MLDP and the LDP operators for face images. With fewer noise components than the LDP images, the proposed MLDP image has stronger facial texture components than the LDP image. In addition, the proposed MLDP has merit in that it is more robust than previous LBP and LDP operators against impulse noise. In Figure 5, we show examples of several binary patterns for a small image patch, before and after adding impulse noise. After adding the noise, we can observe that the 2nd bit of the LDP has changed from 1 to 0. However, the MLDP shows the same pattern, that is, 01111100; thus, it provides a more stable pattern than those of the LBP or LDP. Note that the MLDP is computed using the edge response values without performing the absolute operation as shown in Figure 5, while the LDP carries out the absolute operation. For example, bit values having edge response values of −399 and −503 were set to 1, as shown in Figure 5(a), since the edge response values are converted to absolute values.

Figure 4

Comparison of LDP and MLDP images: (a) raw image, (b) LDP image, and (c) MLDP image.

Figure 5

Stability of MLDP: (a) binary patterns for image patch and (b) binary patterns for image patch with impulse noise.

Principal component analysis (PCA) is a well-known feature extraction and data representation technique that is widely used in the areas of pattern recognition, computer vision, and signal processing [23, 24]. The central underlying concept is to reduce the dimensionality of a dataset while retaining variations as much as possible [25]. In PCA-based facial representation and recognition methods, the 2D facial image matrices must have been previously transformed into 1D image vectors column by column or row by row. However, concatenating 2D matrices into 1D vectors often results in a high-dimensional vector space, where it is difficult to accurately evaluate the covariance matrix due to its large size and the relatively small number of training samples [14]. Furthermore, computing the eigenvectors of a large covariance matrix is very time-consuming.

To overcome these problems, a new technique called two-dimensional principal component analysis (2D-PCA) was proposed to directly compute the eigenvectors of the so-called image covariance matrix without a matrix-to-vector conversion [14]. Because the size of the image covariance matrix is equal to the width of the images, which is quite small compared to the size of a covariance matrix in PCA, 2D-PCA evaluates the image covariance matrix more accurately and computes the corresponding eigenvectors more efficiently than PCA. It was reported in [15] that the recognition accuracy of several face databases is higher using 2D-PCA than when using PCA, and the extraction of image features is computationally more efficient using 2D-PCA than when using PCA [15]:

\begin{matrix} \bar{A} = \frac{1}{M} \sum_{k} A_{k}, \end{matrix}

(3)

\begin{matrix} G = \frac{1}{M} \sum_{k = 1}^{M} {(A_{k} - \bar{A})}^{T} (A_{k} - \bar{A}) . \end{matrix}

(4)

Unlike PCA, which treats 2D images as 1D image vectors, 2D-PCA views an image as a matrix. Consider an m by n image matrix A. Let $X \in R^{n \times d}$ be a matrix with orthonormal columns, $n \geq d$ . Projecting A onto X yields an m by d matrix $Y = A X$ . In 2D-PCA, the total scatter of the projected samples is used to determine a good projection matrix, X. Suppose that there are M training face images, denoted by m by n matrices $A_{k}$ $(k = 1, 2, \dots, M)$ , and the average image is denoted as in (3). Then, the image covariance matrix, G, can be evaluated as shown in (4).

It has been shown that the optimal value for the projection matrix $X_{opt}$ is composed of the orthonormal eigenvectors $X_{1}, X_{2}, \dots, X_{d}$ of G corresponding to the d largest eigenvalues, that is, $X_{opt} = [X_{1}, X_{2}, \dots, X_{d}]$ . Since the size of G is only n by n, computing its eigenvectors is very efficient. The optimal projection vectors of 2D-PCA, $X_{1}, X_{2}, \dots, X_{d}$ , are used for feature extraction. For a given facial image A, the feature vectors $Y_{opt} = [Y_{1}, Y_{2}, \dots, Y_{d}]$ , in which Y has a dimension of m by d matrices, are obtained by projecting the images into the eigenvectors as follows:

\begin{matrix} Y_{k} = (A - \bar{A}) X_{k}, k = 1, 2, \dots, d . \end{matrix}

(5)

Similar to 2DPCA, A2DPCA is also employed for face recognition in this work. While 2DPCA essentially works in the row direction of images, A2DPCA works in the column direction [15]. When compared to 2DPCA, A2DPCA is different in that the covariance matrix is computed by

\begin{matrix} G = \frac{1}{M} \sum_{k = 1}^{M} (A_{k} - \bar{A}) {(A_{k} - \bar{A})}^{T} . \end{matrix}

(6)

After feature extraction through 2DPCA or A2DPCA, the nearest neighbor classifier is used to measure the similarities between the training feature and the test feature. The nearest neighbor classifier is a simple classifier that requires no specific training phase. Here the distance between the two arbitrary feature vectors, Y and $Y^{k}$ , is defined by:

\begin{matrix} d (Y, Y^{k}) = \sqrt{\sum_{i = 1}^{m} \sum_{j = 1}^{d} {(y_{i, j}^{k} - y_{i, j})}^{2}} . \end{matrix}

(7)

In this formula, k is $1, 2, \dots, M$ , and M is the total number of training images. Assume that each of these features is assigned a given identity (class), $w_{k}$ . If a feature vector of the test image is given as Y, the recognition result is decided by finding the class $w_{k}$ that has the minimum distance, as follows:

\begin{matrix} d (Y, Y_{r}) = \min_{j} d (Y, Y_{j}), Y_{r} \in w_{k}, then Y \in w_{k} . \end{matrix}

(8)

3.2. The DFM: Distance Filtering Module

The DFM searches for the location-tagged content of a place in a specific user-centered radius. The radius is established by user selection. The error value varies according to the approach and the real distance. The simplest method considers the two-dimensional coordinates according to the latitude and longitude, as shown in (9), which is given as a directional distance on the map:

\begin{matrix} d (A, B) = \sqrt{α (| A_{lat} - B_{lat} |) + β (| A_{lon} - B_{lon} |)} . \end{matrix}

(9)

In this formula, α is the number of kilometers per 1 degree of latitude in the target area, and β is the number of kilometers per 1 degree of longitude in the target area. If it is possible to measure the distance on the road, another artificial intelligence method should be considered, such as the $A^{*}$ or Dijikstra algorithm. The Dijikstra algorithm fuses the breadth-based search with the depth-based search. The $A^{*}$ algorithm adds a heuristic function to the Dijikstra algorithm. Both are methods used to search for the shortest distance between two points. In this study, we determined the distance between two points using the distance accrued between the nodes.

3.3. The PFM: Preference Filtering Module

A key module in the system considers content-sharing personalization. This module refers to a preference database and then generates a content recommendation list based on the memory-based CF [12]. However, the memory-based CF has a scalability problem that increases the calculation cost as the content and number of users increase. The PFM, which is based on the results from the DFM, helps to ameliorate this problem. This module recommends personalized content for a user by predicting a preference rating score for new content using results from the DFM. CF is mainly applied to electronic commerce; in this study, we attempted to apply CF to LBS and confirm its effectiveness. Our scheme and generalized steps for the PFM are shown below.

Step 1.

Collect ratings for the experimental location-based content from a large number of users and then construct the preference database. The target user must have evaluated location-based content in the preference dataset because this is a necessary condition to use CF.

Step 2.

Search the top- k similar users who rated the target item. Considering linear similarity algorithms such as PCC, COS, and RMS is important when searching for similar users. The vector space model-based method includes cosine similarity (COS), which is frequently used in information retrieval. The COS model assumes that the rating of each user is a point in vector space and then evaluates the cosine angle between the two points by considering the common rating vectors $X = {x_{1}, x_{2}, x_{3}, \dots, x_{n}}$ and $Y = {y_{1}, y_{2}, y_{3}, \dots, y_{n}}$ of users X and Y and representing these vectors as dot products and magnitudes. COS has frequently been used for performance comparisons in the CF area. The cosine angle between vectors X and Y is given by

\begin{array}{l} COS (X, Y) = \frac{X \cdot Y}{∥ X ∥ ∥ Y ∥} \\ = \frac{x_{1} y_{1} + x_{2} y_{2} \dots x_{n} y_{n}}{\sqrt{x_{1}^{2} + y_{1}^{2}} \sqrt{x_{2}^{2} + y_{2}^{2}} \dots \sqrt{x_{n}^{2} + y_{n}^{2}}} . \end{array}

(10)

One correlation-based method utilizes the Pearson dot-product correlation coefficient (PCC), which is normally used to evaluate the association intensity between two variables. PCC is given by

\begin{matrix} PCC (X, Y) = \frac{\sum_{i = 1}^{n} (x_{i} - \bar{X}) (y_{i} - \bar{Y})}{\sqrt{\sum_{i = 1}^{n} {(x_{i} - \bar{X})}^{2}} \sqrt{\sum_{i = 1}^{n} {(y_{i} - \bar{Y})}^{2}}} . \end{matrix}

(11)

Our research team proposed row moment-based similarity (RMS) in August 2011. This method is more effective than traditional methods under data sparsity conditions. The RMS is given by

\begin{array}{l} RMS (u_{i}, u_{j}) = 1 - \frac{1}{r^{k}} \frac{1}{n} \sum_{v = 1}^{n} {(| u_{i, v} - u_{j, v} |)}^{k} \\ = 1 - \frac{1}{r^{k}} \sum_{z = 1}^{m} Pr (D = d_{z}) \cdot d_{z}^{k} . \end{array}

(12)

Step 3.

Predict ratings based on a preference database to determine the expected rating of a target user for the target content. Accordingly, the PFM predicts the rating of the target user for the target content with weight similarities. Equation (13) shows the prediction method, where u indicates the target user, i indicates the target item, j indicates other users, and w is the similarity between u and j [12]:

\begin{matrix} p (u, i) = \bar{r_{u}} + \frac{\sum_{j = 1}^{n} w_{j, v} (r_{j, i} - \bar{r_{j}})}{\sum_{j = 1}^{n} w_{u, j}} . \end{matrix}

(13)

3.4. Characteristics of the System

The proposed system contains various academic and practical advantages. First, concerning the UAM, the system is largely protected against security problems and is more tractable than existing security methods. We will show the processing results of the UAM on a smartphone. Second, concerning the DFM and PFM, the system employs a novel approach for solving the scalability problem of the CF. The convergence of DFM and PFM in the proposed system shows that the use of characteristics in the application area can improve the scalability problem. Third, concerning the PFM, application of the CF is extended to the LBS. The CF is commonly applied to increase sales in e-commerce, but here it is used to create recommendations for users. Last, the generalized form of a novel SNS system is shown by a convergence of RS, LBS, and the existing SNS.

4. Experiment and Result

4.1. Experimental Results for Facial Recognition

To verify the UAM, we used a smartphone with a 1 GHz processor, 512 MB of memory, a 5 MP front-facing camera, and the Android-based Gingerbread version 2.3 operating system. For the purposes of this performance evaluation, we constructed a database consisting of frontal facial images with lighting maintained at about 200 lux in an indoor environment. The experimental database contains a total of 1,000 images of 50 subjects, that is, 20 images per individual. This study used 250 images for training, and the remaining images were used to evaluate performance.

To evaluate the recognition performance of the proposed face recognition system, we employed images from the Yale B database [24] and the CMU-PIE database [25]. The Yale B database consists of 640 face images of 10 subjects in a frontal pose and in 64 illumination conditions. The CMU-PIE database contains more than 40,000 facial images of 68 individuals, 21 illumination conditions, 13 poses, and four different expressions. From these, we selected illumination images of 68 individuals in frontal poses. So, the CMU-PIE set consists of 21 images of 68 individuals (21 × 68 images in total). First, we investigated the detection rates and detection times for each database. The detection rates were 91.71% and 94.57% for the Yale B database and the CMU-PIE database, respectively. Also, the average detection times for one image were 182 ms and 187 ms for the Yale B database and the CMU-PIE database, respectively. Undetected images, including false-detected images, were added to the experimental datasets by manually cropping the associated regions. Based on the datasets described, the performance evaluation of the proposed approach was carried out using various recognition approaches, such as PCA, 2DPCA, A2DPCA, Gabor-wavelets based on LBP [26], and support vector machine (SVM) based on LDP [27].

Next, we investigated the recognition performance of the proposed system with two databases. During the performance evaluation, we partitioned each database into training and testing sets. For the Yale B database, we used 45 face images for each subject, and the database was further subdivided into four subsets depending on the direction of light [28]. For the Yale B database, we employed the first subset for training, and the remaining subsets were used for testing. For the CMU-PIE databases, five images from each person were used for training and the remaining images were used for testing. Here, we selected illumination-invariant images for training, and the remaining images were employed for testing.

For the Yale B database, the recognition results for different recognition algorithms such as PCA, 2DPCA, and A2DPCA are shown in Figures 6, 7, and 8, respectively. To further show the relationship between the recognition rate and the dimensions of the feature vectors, we provide the recognition results along with the different dimensions. Furthermore, we summarized the maximum recognition rates in Table 2. The maximum recognition rates were found to be 93.98%, 98.98%, and 98.98% for PCA, 2DPCA, and A2DPCA, when using the MLDP image. Here, we can observe that the maximum rates were achieved with the proposed approach, which uses image covariance-based feature extraction algorithms, that is, 2D-PCA and A2D-PCA, and the MLDP image. For the CMU-PIE database, the recognition results for PCA, 2DPCA, and A2DPCA are shown in Figures 9, 10, and 11, respectively. Also, we summarized the maximum recognition rates in Table 2. The maximum recognition accuracies had values of 94.55%, 96.25%, and 96.17% for PCA, 2DPCA, and A2DPCA, respectively, when using the MLDP image. As with the recognition results from the Yale B database, we can observe that the maximum rates were achieved with the proposed approach. Consequently, the proposed approach using 2DPCA and A2DPCA based on the MLDP images achieved the best recognition rates for all databases. Also, the proposed approach showed better performance compared to previous methods, such as Gabor-wavelets based on the LBP approach [26] and SVM based on the LDP approach [27].

Table 2

Summary of recognition accuracies.

Recognition approaches		Maximum recognition rate
		Yale B	CMU-PIE
PCA	Raw	43.55%	24.55%
	Histogram	71.30%	51.32%
	MLDP	93.98%	94.55%

2DPCA	Raw	43.72%	25.73%
	Histogram	71.94%	53.08%
	MLDP	98.98%	96.25%

A2DPCA	Raw	43.55%	25.73%
	Histogram	72.62%	53.01%
	MLDP	98.98%	96.17%

Gabor + LBP	Raw	66.77%	62.79%
Gabor + LBP	Histogram	85.76%	82.94%

SVM + LDP	Raw	71.46%	64.57%
SVM + LDP	Histogram	92.17%	87.46%

Figure 6

Recognition results using PCA for the Yale B database.

Figure 7

Recognition results using 2DPCA for the Yale B database.

Figure 8

Recognition results using A2DPCA for the Yale B database.

Figure 9

Recognition results using PCA for the CMU-PIE database.

Figure 10

Recognition results using 2DPCA for the CMU-PIE database.

Figure 11

Recognition results using A2DPCA for the CMU-PIE database.

4.2. The PFM: Preference Prediction Accuracy and Recommendation Quality

To verify the PFM, we collected content and user preference scores over a period of one year. We obtained 687 images consisting of location-tagged content from 52 users. The users rated shared content on a scale from 1 to 5. The total number of rating data points was 9,473. The data format was {user id, content id, rating score}. We randomly divided the rating dataset using an 8 : 2 ratio, then predicted a rating of 20%, and measured the absolute average error (MAE).

Figure 12 shows the general prediction accuracy of the proposed system using our dataset. COS is the cosine coefficient, PCC is the Pearson correlation coefficient, and RMS is the raw-moment similarity method [13]. The existing CF area shows that the MAE values ranged from 0.73 to 0.75 in the MovieLens 100 k dataset, which has the same rating scale as our dataset. Our experimental results showed 0.5406 to be the best MAE, confirming that applying the CF to the LBS was more effective than the more common application area of e-commerce.

Figure 12

Full-rating experimental result.

Figure 13 shows the results of the artificial data scarcity experiment. This experiment creates an artificial data scarcity condition by restricting the number of coitem ratings between two variables. The coitem ratings used to measure similarity were randomly selected. This experiment showed that our method was robust with regard to the data scarcity problem. We applied an optimal neighborhood size around each similarity method, similar to those seen in Figures 12 and 13, to show that the user's preference was reflected more strongly in the personal LBS. In other words, we confirmed strong performance of the proposed method in spite of a comparatively small amount of ratings data, which showed a good convergence of the CF and LBS. However, an intensive study using many more rating data points is required.

Figure 13

Artificial data scarcity experimental results.

The second experiment for the PFM addressed the recommendation quality of the full-rating experimental results. The measures were precision, recall, and F-score. The precision and recall are defined by (14), where $N (N_{u Re c} \cap L_{u Re f})$ is the number of contents in the recommended list of SNS contents, and $N (N_{u Re c})$ is the number of recommended SNS contents $N (N_{u Re f})$ is the number of watched SNS contents. The F-score is 2 * (Precision * Recall)/(Precision + Recall). The results are shown in Table 3. Our dataset showed that the Top-40 collaborative filtering recommendation method provided the best results:

\begin{matrix} precision (u) = \frac{N (L_{u Re c} \cap L_{u Pre f})}{N (L_{u Re c})}, \\ recall (u) = \frac{N (L_{u Re c} \cap L_{u Pre f})}{N (L_{u Pre f})} . \end{matrix}

(14)

Table 3

Experimental results for recommendation quality.

Top-n	Precision	Recall	F-score
35	0.761	0.734	0.747
40	0.730	0.773	0.751
45	0.690	0.795	0.739
50	0.657	0.817	0.728

5. Application

5.1. Prototype Implementation Environment

We implemented a prototype of the proposed system. Since mobile applications based on Java are slower than applications written in native C/C++ languages, we used the Native Development Kit (NDK) for mobile operating systems and Java Native Interfaces (JNI) [29, 30] to improve the performance of the mobile application. Further, the system frequently requires memory access, which involves complex calculations due to the real-time image processing. Consequently, the code related to facial detection and recognition algorithms used in this study was written in native C, and other codes related to the Graphical User Interface (GUI) were written in Java with the Software Development Kit (SDK) for mobile operating systems.

5.2. Service Scenario

Figure 14 shows the results from the implementation of the proposed system. In the UAM, the system detects a user's face and then saves the facial image file. The UAM recognizes a user's face based on the facial image file, which could be used to recognize the user during the next authentication or to represent the current user to other users on the map. Existing SNS services use a text input method for authentication, while the proposed system in this study uses a state-of-the-art, biometric authentication method. Our method also includes a remarkable level of security for protecting personal information in the mobile SNS, which could contain the personal information of many users.

Figure 14

Actual images and a scenario based on a free Internet map.

After authentication by the UAM, the system shows the recommended content list, which is the processing result of preference filtering by PFM and distance filtering by DFM, as a user-oriented personalized content list recommended according to a default radius that can be changed by the user. The system shows the location of the user and the recommended content based on the mobile maps. The map is implemented using mobile mash-up technology. The PFM and DFM in the proposed system select personalized content from a large collection of location-tagged content based on the ratings of users who showed similar tastes.

When the user wants to view detailed information about the location-tagged content, he or she will click on the content from a printed list. Clicked content will be highlighted for a moment and then will show detailed information such as comments and ratings from other users. The [+] symbol next to a username in the comments means that the target user can be added to the “following” list. In the proposed system, the “following” list is a group of users with similar ratings values. Users clicking the follow button [+] will cause the system to count the selected users' ratings and their rating averages similar to the user-closest results based on PFM. For instance, the message and trace function between followers can be considered. Namely, a user can communicate with other users and followers based on these facilities; this ability is intrinsic to the SNS. We are able to improve this ability with intuitive user authentication by user facial recognition, personalization by sharing content by CF, and user position recognition by LBS.

In summary, the proposed system improves mobile SNSs using state-of-the-art technologies, including the ability to post from a blog. Images from the user may also be used as markers on a map to show followers the size, location, or conditions of the area. This prototype can be used in two-way TVs, PCs, tablet PCs, and car navigation systems by changing their application platforms to IT convergence solutions.

6. Conclusion

This study proposed a new mobile SNS system that combines LBS, RS, and user facial authentication technology. The user authentication method in the proposed system is a secure way to protect personal information on a mobile SNS. A PFM and DFM in the proposed system select personalized content from a collection of location-tagged content based on the ratings of users with similar preferences. The CF has constraints that apply when an individual's preference has not been clearly defined. A user can communicate with other users based on his comments and preference ratings for shared content. We improved mobile social networks by adding user facial recognition, personalization for sharing content, and user position information. We created a prototype and a scenario based on a smartphone for testing. In this study, we applied a state-of-the-art personalization method, CF to LBS, and validated it. We expanded its formerly restricted e-commerce application to include quantitative performance to verify its effectiveness. The proposed system can be extended by integration into other platforms, that is, two-way TVs, Web-map services, and car navigation systems.

The proposed system was created for location-based services and personalized recommendations. Accordingly, it was compared to existing SNSs that have been developed based on personal blogs. The proposed system's convergence with existing SNSs will be a novel study topic for next-generation mobile SNSs. We are pursuing further study with an in-depth analysis of the system and its suitability for various applications and completing additional verification.

Footnotes

Acknowledgments

This research was supported by the Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education, Science and Technology (2013059769) and the Basic Science Research Program through NRF of Korea, funded by MOE (NRF-2010-0020210).

References

Miluzzo

Lane

N. D.

Fodor

Peterson

Musolesi

Eisenman

S. B.

Zheng

Campbell

A. T.

Sensing Meets Mobile Social Networks: The Design, Implementation and Evaluation of the CenceMe Application

Proceedings of the 6th ACM Conference on Embedded Network Sensor Systems

2008

337 350

Chang

Y.-J.

Liu

H.-H.

Chou

L.-D.

Chen

Y.-W.

Shin

H.-Y.

A general architecture of mobile social network services

Proceedings of the 2nd International Conference on Convergent Information Technology (ICCIT '07)

November 2007

kor

151 156

2-s2.0-49049102940

10.1109/ICCIT.2007.4420252

Boyd

D. M.

Ellison

N. B.

Social network sites: definition, history, and scholarship

Journal of Computer-Mediated Communication 2007 13 1 210 230

2-s2.0-37249053884

10.1111/j.1083-6101.2007.00393.x

Beach

Gartrell

Akkala

Elston

Kelley

Nishimoto

Ray

Razgulin

Sundaresan

Surendar

Terada

Han

WhozThat? Evolving an ecosystem for context-aware mobile social networks

IEEE Network 2008 22 4 50 55

2-s2.0-48749092900

10.1109/MNET.2008.4579771

Lee

H.-H.

K.-R.

Hong

K.-S.

Location-based mixed-map application development for mobile devices

Human Interface and the Management of Information 2009 5618 2 403 412

2-s2.0-76249122211

10.1007/978-3-642-02559-4_44

Floyd

I. R.

Cameron Jones

Rathi

Twidale

M. B.

Web mash-ups and patchwork prototyping: user-driven technological innovation with web 2.0 and open source software

Proceedings of the 40th Annual Hawaii International Conference on System Sciences (HICSS '07)

January 2007

2-s2.0-39749191218

10.1109/HICSS.2007.612

Wong

Hong

J. I.

Making mashups with marmite: towards end-user programming for the web

25th SIGCHI Conference on Human Factors in Computing Systems 2007, CHI 2007

May 2007

usa

1435 1444

2-s2.0-35348863809

10.1145/1240624.1240842

Jhingran

Enterprise information mashups: integrating information

Proceedings of the 32nd international Conference on Very Large Data Bases (ACM VLDB '06)

2006

3 4

Murthy

Maier

Delcambre

Mash-o-matic

Proceedings of the 2006 ACM Symposium on Document Engineering (DocEng '06)

October 2006

205 214

2-s2.0-34247326564

10.1145/1166160.1166214

10.

Breslin

Decker

The future of social networks on the internet: the need for semantics

IEEE Internet Computing 2007 11 6 86 90

2-s2.0-36349019854

10.1109/MIC.2007.138

11.

Larsen

Urry

Axhausen

K. W.

Networks and tourism. Mobile Social Life

Annals of Tourism Research 2007 34 1 244 262

2-s2.0-33845926219

10.1016/j.annals.2006.08.002

12.

Adomavicius

Tuzhilin

Toward the next generation of recommender systems: a survey of the state-of-the-art and possible extensions

IEEE Transactions on Knowledge and Data Engineering 2005 17 6 734 749

2-s2.0-20844435854

10.1109/TKDE.2005.99

13.

Kwon

H.-J.

Hong

K.-S.

Personalized smart TV program recommender based on collaborative filtering and a novel similarity method

IEEE Transactions on Consumer Electronics 2011 57 3 1416 1423

2-s2.0-80052945814

10.1109/TCE.2011.6018902

14.

Yang

Zhang

Frangi

A. F.

Yang

J.-Y.

Two-dimensional PCA: a new approach to appearance-based face representation and recognition

IEEE Transactions on Pattern Analysis and Machine Intelligence 2004 26 1 131 137

2-s2.0-0742268833

10.1109/TPAMI.2004.1261097

15.

Zhang

Zhou

Z.-H.

(2D)2 PCA: two-directional two-dimensional PCA for efficient face representation and recognition

Neurocomputing 2005 69 1–3 224 231

2-s2.0-27844449908

10.1016/j.neucom.2005.06.004

16.

Jain

A. K.

Ross

Prabhakar

An introduction to biometric recognition

IEEE Transactions on Circuits and Systems for Video Technology 2004 14 1 4 20

2-s2.0-0742290133

10.1109/TCSVT.2003.818349

17.

Venkataramani

Qidwai

Vijayakumar

B. V. K.

Face authentication from cell phone camera images with illumination and temporal variations

IEEE Transactions on Systems, Man and Cybernetics C 2005 35 3 411 418

2-s2.0-23944507759

10.1109/TSMCC.2005.848183

18.

Tao

Veldhuis

R. N. J.

Biometric authentication for a mobile personal device

Proceedings of the 3rd Annual International Conference on Mobile and Ubiquitous Systems

July 2006

2-s2.0-50249137111

10.1109/MOBIQW.2006.361741

19.

Turk

Pentland

Eigenfaces for recognition

Journal of Cognitive Neuroscience 1991 3 1 71 86

2-s2.0-0026065565

20.

Yamamoto

Takumi

Matsuo

Location-based social network services employing student cards for university

Proceedings of the International Workshop on Location Based Social Networks (LBSN '09)

November 2009

21 24

21.

Kim

J.-T.

Lee

J.-H.

Lee

H.-K.

Paik

E.-H.

Provision of the personalized social network service based on the locality/sociality relations

Proceedings of the 4th International Conference on Internet and Web Applications and Services (ICIW '09)

May 2009

235 238

2-s2.0-70349755378

10.1109/ICIW.2009.110

22.

Kim

J.-T.

Lee

J.-H.

Lee

H.-K.

Paik

E.-H.

Design and implementation of the location-based personalized social media service

Proceedings of the 5th International Conference on Internet and Web Applications and Services (ICIW '10)

May 2010

116 121

2-s2.0-77954515696

10.1109/ICIW.2010.25

23.

Hadid

Heikkilä

J. Y.

Silven

Pietikäinen

Face and eye detection for person authentication in mobile phones

Proceedings of the Conference on Distributed Smart Cameras

September 2007

101 108

2-s2.0-47349131254

10.1109/ICDSC.2007.4357512

24.

Zuo

de With

P. H. N.

Real-time embedded face recognition for smart home

IEEE Transactions on Consumer Electronics 2005 51 1 183 190

2-s2.0-18844441808

10.1109/TCE.2005.1405718

25.

Rahman

Ren

Kehtarnavaz

Real-time implementation of robust face detection on mobile platforms

Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP '09)

April 2009

1353 1356

2-s2.0-70349214864

10.1109/ICASSP.2009.4959843

26.

Ahonen

Hadid

Pietikäinen

Face description with local binary patterns: application to face recognition

IEEE Transactions on Pattern Analysis and Machine Intelligence 2006 28 12 2037 2041

2-s2.0-33947167478

10.1109/TPAMI.2006.244

27.

Zhang

Shan

Chen

Gao

Local gabor binary patterns based on mutual information for face recognition

International Journal of Image and Graphics 2007 7 4 777 793

28.

Jabid

Kabir

M. H.

Chae

Robust facial expression recognition based on local directional pattern

ETRI Journal 2010 32 5 784 794

2-s2.0-78049351030

10.4218/etrij.10.1510.0132

29.

Chang

Tan

Zhu

Developing mobile applications on the android platform

Mobile Multimedia Processing 2010 5960 264 286

2-s2.0-77952004483

10.1007/978-3-642-12349-8_15

30.

Liang

Java Native Interface: Programmer's Guide and Specification 1999

Addison-Wesley