A Dynamic Users’ Interest Discovery Model with Distributed Inference Algorithm

Abstract

One of the key issues for providing users user-customized or context-aware services is to automatically detect latent topics, users’ interests, and their changing patterns from large-scale social network information. Most of the current methods are devoted either to discovering static latent topics and users’ interests or to analyzing topic evolution only from intrafeatures of documents, namely, text content, without considering directly extrafeatures of documents such as authors. Moreover, they are applicable only to the case of single processor. To resolve these problems, we propose a dynamic users’ interest discovery model with distributed inference algorithm, named as Distributed Author-Topic over Time (D-AToT) model. The collapsed Gibbs sampling method following the main idea of MapReduce is also utilized for inferring model parameters. The proposed model can discover latent topics and users’ interests, and mine their changing patterns over time. Extensive experimental results on NIPS (Neural Information Processing Systems) dataset show that our D-AToT model is feasible and efficient.

1. Introduction

With a dynamic users’ interest discovery model, one can answer a range of important questions about the content of information uploaded or shared to social network service (SNS), such as which topics each user prefers, which users are similar to each other in terms of their interests, which users are likely to have written documents similar to an observed document, and who are influential users at different stages of topic evolution, and it also helps characterize users as pioneers, mainstream, or laggards in different subject areas.

Users’ interests have shown their increasing importance for the development of personalized web services and user-centric applications [1, 2]. Hence, users’ interest modeling has been attracting extensive attentions during the past few years, such as (a) Author-Topic (AT) model [3–5], (b) Author-Recipient-Topic (ART) model [6–8], Role-Author-Recipient-Topic (RART) model [6–8], and Author-Persona-Topic (APT) model [9], (c) Author-Interest-Topic (AIT) model [10] and Latent-Interest-Topic (LIT) model [11], and (d) Author-Conference-Topic (ACT) model [12].

In fact, when people enjoy SNS with their smart devices including phones and tablets, each user's interest is usually not static. However, the above models are devoted to discovering static latent topics and user's interests. Moreover, they are applicable only to the case of single processor. Of course, one can perform some post hoc or pre hoc analysis [4, 13] to discover changing patterns over time, but this misses the opportunity for time to improve topic discovery [14], and it is very difficult to align corresponding topics [15]. Currently, attention for dynamic models is mainly focused on analyzing topic evolution only from text content, such as Dynamic Topic Model (DTM) [16], continuous time DTM (cDTM) [17], and Topic over Time (ToT) [14].

This paper mainly focuses on the dynamic users’ interest discovery model, especially collapsed Gibbs sampling following the main idea of MapReduce [18]. Figure 1 gives a detailed illustration for discovering dynamic users’ interests. Our previous work [19, 20] is limited to inference algorithm on single-processor.

Figure 1

The illustration for discovering dynamic users’ interests.

The organization of the rest of this work is as follows. In Section 2, we firstly discuss two related generative models, Author-Topic (AT) model and Topic over Time (ToT) model, and then introduce in detail our proposed Author-Topic over Time (AToT) model. Sections 3 and 4 describe the collapse Gibbs sampling methods used for inferring the model parameters and distributed inference algorithm version, respectively. In Section 5, extensive experimental evaluations are conducted, and Section 6 concludes this work.

2. Generative Models for Documents

Before presenting our Author-Topic over Time (AToT) model, we first describe two related generative models: AT model and ToT model. The notation is summarized in Table 1.

Table 1

Notation used in the generative models.

Symbol	Description
K	Number of topics
M	Number of documents
V	Number of unique words
A	Number of unique authors
$N_m$	Number of word tokens in document m
$A_m$	Number of authors in document m
a	Single author index, $a\in [\mathrm{1},A]$
k	Single topic index, $k\in [\mathrm{1},K]$
m	Single document index, $m\in [\mathrm{1},M]$
n	Single word token index, $n\in [\mathrm{1},N_m]$
v	Single word index, $v\in [\mathrm{1},V]$
${\mathit{a}}_m$	Authors in document m, ${\mathit{a}}_m\subseteq [\mathrm{1},A]$
${\mathbf{\vartheta }}_a$	Multinomial distribution of topics specific to the author a.
${\mathbf{\vartheta }}_m$	Multinomial distribution of topics specific to the document m.
${\mathbf{\phi }}_k$	Multinomial distribution of words specific to the topic k.
${\mathbf{\psi }}_k$	Beta distribution of timestamp specific to the topic k.
$z_{m,n}$	Topic associated with the nth token in the document m
$w_{m,n}$	nth token in document m
$x_{m,n}$	Chosen author associated with the word token $w_{m,n}$
$t_{m,n}$	Timestamp associated with the nth token in the document m
α	Dirichlet priors (hyperparameter) to the multinomial distribution ϑ
β	Dirichlet priors (hyperparameter) to the multinomial distribution φ

2.1. Author-Topic (AT) Model

Rosen-Zvi et al. [3–5] propose an Author-Topic (AT) model for extracting information about authors and topics from large text collections. Rosen-Zvi et al. model documents as if they were generated by a two-stage stochastic process. An author is represented by a probability distribution over topics, and each topic is represented as a probability distribution over words. The probability distribution over topics in a multiauthor paper is a mixture of the distributions associated with the authors.

The graphical model representations for AT model are shown in Figure 2. The AT model can be viewed as a generative process, which can be described as follows.

• (1)

For each topic $k\in [\mathrm{1},K]$ ,

• (i)

draw a multinomial ${\mathit{\phi }}_k$ from $\text{Dirichlet}(\mathit{\beta })$ ;

(2)

for each author $a\in [\mathrm{1},A]$ ,

• (i)

draw a multinomial ${\mathit{\vartheta }}_a$ from $\text{Dirichlet}(\mathit{\alpha })$ ;

(3)

for each word $n\in [\mathrm{1},N_m]$ in document $m\in [\mathrm{1},M]$ ,

• (i)

draw an author assignment $x_{m,n}$ uniformly from the group of authors ${\mathit{a}}_m$ ;

(ii)

draw a topic assignment $z_{m,n}$ from $\text{Multinomial}({\mathit{\vartheta }}_{x_{m,n}})$ ;

(iii)

draw a word $w_{m,n}$ from $\text{Multinomial}({\mathit{\phi }}_{z_{m,n}})$ .

Figure 2

The graphical model representation of the AT model.

2.2. Topic over Time (ToT) Model

Unlike other dynamic topic models that rely on Markov assumptions or discretization of time, each topic in Topic over Time (ToT) model [14] is associated with a continuous distribution over timestamps, and, for each generated document, the mixture distribution over topics is influenced by both word cooccurrences and the document's timestamp. Thus, the meaning of a particular topic can be relied upon as constant, but the topics’ occurrence and correlations change significantly over time.

The graphical model representations for ToT model are shown in Figure 3. The ToT is a generative model of timestamps and the words in the timestamped documents. The generative process can be described as follows.

• (1)

For each topic $k\in [\mathrm{1},K]$ ,

• (i)

draw a multinomial from $\text{Dirichlet}(\mathit{\beta })$ ;

(2)

for each document $m\in [\mathrm{1},M]$ ,

• (i)

draw a multinomial ${\mathit{\vartheta }}_m$ from $\text{Dirichlet}(\mathit{\alpha })$ ;

(ii)

for each word $n\in [\mathrm{1},N_m]$ in document m,

• (a)

draw a topic assignment $z_{m,n}$ from $\text{Multinomial}({\mathit{\vartheta }}_m)$ ;

(b)

draw a word $w_{m,n}$ from $\text{Multinomial}({\mathit{\phi }}_{z_{m,n}})$ ;

(c)

draw a timestamp $t_{m,n}$ from $\text{Beta}({\mathit{\psi }}_{z_{m,n}})$ .

Figure 3

The graphical model representation of the ToT model.

2.3. Author-Topic over Time (AToT) Model

The graphical model representations for AToT model are shown in Figure 4. The AToT model can be viewed as a generative process, which can be described as follows.

• (1)

For each topic $k\in [\mathrm{1},K]$ ,

• (i)

draw a multinomial ${\mathit{\phi }}_k$ from $\text{Dirichlet}(\mathit{\beta })$ ;

(2)

for each author $a\in [\mathrm{1},A]$ ,

• (i)

draw a multinomial ${\mathit{\vartheta }}_a$ from $\text{Dirichlet}(\mathit{\alpha })$ ;

(3)

for each word $n\in [\mathrm{1},N_m]$ in document $m\in [\mathrm{1},M]$ ,

• (i)

draw an author assignment $x_{m,n}$ uniformly from the group of authors ${\mathit{a}}_m$ ;

(ii)

draw a topic assignment $z_{m,n}$ from $\text{Multinomial}({\mathit{\vartheta }}_{x_{m,n}})$ ;

(iii)

draw a word $w_{m,n}$ from $\text{Multinomial}({\mathit{\phi }}_{z_{m,n}})$ ;

(vi)

draw a timestamp $t_{m,n}$ from $\text{Beta}({\mathit{\psi }}_{z_{m,n}})$ .

Figure 4

The graphical model representation of the AToT model.

From the above generative process, one can see that AToT model is parameterized as follows:

•  

${\mathit{\vartheta }}_a\hspace{0.17em}|\hspace{0.17em}\mathit{\alpha }~\text{Dirichlet}(\mathit{\alpha })$

 

${\mathit{\phi }}_k\hspace{0.17em}|\hspace{0.17em}\mathit{\beta }~\text{Dirichlet}(\mathit{\beta })$

 

$z_{m,n}\hspace{0.17em}|\hspace{0.17em}{\mathit{\vartheta }}_{x_{m,n}}~\text{Multinomial}({\mathit{\vartheta }}_{x_{m,n}})$

 

$w_{m,n}\hspace{0.17em}|\hspace{0.17em}{\mathit{\phi }}_{z_{m,n}}~\text{Multinomial}({\mathit{\phi }}_{z_{m,n}})$

 

$x_{m,n}\hspace{0.17em}|\hspace{0.17em}{A}_m~\text{Multinomial}(\mathrm{1}/A_m)$

 

$t_{m,n}\hspace{0.17em}|\hspace{0.17em}{\mathit{\psi }}_{z_{m,n}}~\text{Beta}({\mathit{\psi }}_{z_{m,n}})$ .

As a matter of fact, a paper is usually written by the first author and reprint author. If one wants to differentiate the contributions of the first author and reprint author from those of other coauthors, it is very easy for AToT model to set different weights for different authors. But since there are no criteria to guide the corresponding weights, we just set the equal weights for all coauthors in this work; that is to say, $x_{m,n}\hspace{0.17em}|\hspace{0.17em}{A}_m$ follows the uniform distribution.

3. Inference Algorithm

For inference, the task is to estimate the sets of the following unknown parameters in the AToT model: $(1)\hspace{0.17em}\hspace{0.17em}\mathbf{\Phi }=\{{\mathit{\phi }}_k{\}}_{k=\mathrm{1}}^K$ , $\mathbf{\Theta }=\{{\mathit{\vartheta }}_a{\}}_{a=\mathrm{1}}^A$ , and $\mathbf{\Psi }=\{{\mathit{\psi }}_k{\}}_{k=\mathrm{1}}^K$ and (2) the corresponding topic and author assignments $z_{m,n},x_{m,n}$ for each word token $w_{m,n}$ . In fact, inference cannot be done exactly in this model. A variety of algorithms have been used to estimate the parameters of topics models, such as variational EM (expectation maximization) [21, 22], expectation propagation [23, 24], belief propagation [25], and Gibbs sampling [19, 20, 26, 27]. In this work, collapsed Gibbs sampling algorithm [26] is used, since it provides a simple method for obtaining parameter estimates under Dirichlet priors and allows combination of estimates from several local maxima of the posterior distribution.

In the Gibbs sampling procedure, we need to calculate the conditional distribution $P(z_{m,n},x_{m,n}\hspace{0.17em}|\hspace{0.17em}\mathit{w},{\mathit{z}}_{\neg (m,n)},{\mathit{x}}_{\neg (m,n)},\mathit{t},\mathit{a},\mathit{\alpha },\mathit{\beta },\mathbf{\Psi })$ , where ${\mathit{z}}_{\neg (m,n)},{\mathit{x}}_{\neg (m,n)}$ represents the topic and author assignments for all tokens except $w_{m,n}$ , respectively. We begin with the joint distribution $P(\mathit{w},\mathit{z},\mathit{x},\mathit{t}\hspace{0.17em}|\hspace{0.17em}\mathit{a},\mathit{\alpha },\mathit{\beta },\mathbf{\Psi })$ of a dataset, and, using the chain rule, we can get the conditional probability conveniently as

$$\begin{array}{l}P\left(z_{m,n},x_{m,n}\hspace{0.17em}|\hspace{0.17em}\mathit{w},{\mathit{z}}_{\neg \left(m,n\right)},{\mathit{x}}_{\neg \left(m,n\right)},\mathit{t},\mathit{a},\mathit{\alpha },\mathit{\beta },\mathbf{\Psi }\right)\\ \hspace{0.5em}\propto \frac{n_{z_{m,n}}^{\left(w_{m,n}\right)}+{\beta }_{w_{m,n}}-\mathrm{1}}{{\sum }_{v=\mathrm{1}}^V\left(n_k^{\left(v\right)}+{\beta }_v\right)-\mathrm{1}}\times \frac{n_a^{\left(z_{m,n}\right)}+{\alpha }_{z_{m,n}}-\mathrm{1}}{{\sum }_{k=\mathrm{1}}^K\left(n_{x_{m,n}}^{\left(k\right)}+{\alpha }_k\right)-\mathrm{1}}\\ \times \text{Beta}\left({\mathit{\psi }}_{z_{m,n}}\right),\end{array}$$

(1) where $n_k^{(v)}$ is the number of times tokens of word v are assigned to topic k and $n_a^{(k)}$ represents the number of times author a is assigned to topic k. Detailed derivation of Gibbs sampling for AToT is provided in the appendix.

If one further manipulates the above (1), one can turn it into separated update equations for the topic and author of each token, suitable for random or systematic scan updates:

$$\begin{array}{l}P\left(x_{m,n}\hspace{0.17em}|\hspace{0.17em}{\mathit{x}}_{\neg \left(m,n\right)},\mathit{z},\mathit{a},\mathit{\alpha }\right)\propto \frac{n_{x_{m,n}}^{\left(z_{m,n}\right)}+{\alpha }_{z_{m,n}}-\mathrm{1}}{{\sum }_{k=\mathrm{1}}^K\left(n_{x_{m,n}}^{\left(k\right)}+{\alpha }_k\right)-\mathrm{1}},\end{array}$$

(2)

$$\begin{array}{l}P\left(z_{m,n}\hspace{0.17em}|\hspace{0.17em}\mathit{w},{\mathit{z}}_{\neg \left(m,n\right)},\mathit{x},\mathit{t},\mathit{\alpha },\mathit{\beta },\mathbf{\Psi }\right)\\ \propto \frac{n_{z_{m,n}}^{\left(w_{m,n}\right)}+{\beta }_{w_{m,n}}-\mathrm{1}}{{\sum }_{v=\mathrm{1}}^V\left(n_k^{\left(v\right)}+{\beta }_v\right)-\mathrm{1}}\times \frac{n_a^{\left(z_{m,n}\right)}+{\alpha }_{z_{m,n}}-\mathrm{1}}{{\sum }_{k=\mathrm{1}}^K\left(n_{x_{m,n}}^{\left(k\right)}+{\alpha }_k\right)-\mathrm{1}}\\ \hspace{0.5em}\hspace{0.5em}\hspace{0.5em}\times \text{Beta}\left({\mathit{\psi }}_{z_{m,n}}\right).\end{array}$$

(3)

During parameter estimation, the algorithm keeps track of two large data structures: an $A\times K$ count matrix $n_a^{(k)}$ and a $K\times V$ count matrix $n_k^{(v)}$ . From these data structures, one can easily estimate the Φ and Θ as follows:

$$\begin{array}{cc}& {\phi }_{k,v}=\frac{n_k^{\left(v\right)}+{\beta }_v}{{\sum }_{v=\mathrm{1}}^V\left(n_k^{\left(v\right)}+{\beta }_v\right)},\end{array}$$

(4)

$$\begin{array}{cc}& {\vartheta }_{a,k}=\frac{n_a^{\left(k\right)}+{\alpha }_k}{{\sum }_{k=\mathrm{1}}^K\left(n_a^{\left(k\right)}+{\alpha }_k\right)}.\end{array}$$

(5)

As for Ψ, similar to [14], for simplicity and speed, we update it after each Gibbs sample by the method of moments [28]:

$$\begin{array}{cc}& {\psi }_{k,\mathrm{1}}={\stackrel{-}{t}}_k\left(\frac{{\stackrel{-}{t}}_k\left(\mathrm{1}-{\stackrel{-}{t}}_k\right)}{s_k^{\mathrm{2}}}-\mathrm{1}\right),\\ {\psi }_{k,\mathrm{2}}=\left(\mathrm{1}-{\stackrel{-}{t}}_k\right)\left(\frac{{\stackrel{-}{t}}_k\left(\mathrm{1}-{\stackrel{-}{t}}_k\right)}{s_k^{\mathrm{2}}}-\mathrm{1}\right),\end{array}$$

(6) where ${\stackrel{-}{t}}_k$ and $s_k^{\mathrm{2}}$ indicate the sample mean and biased sample variance of the timestamps belonging to topic k, respectively. The readers are invited to consult [28] for details. In fact, similar to [14], since the Beta distribution with the support $[0,1]$ can behave many more shapes including the bell curve than Gaussian distribution, it is utilized to model the timestamps. But Wang and McCallum [14] did not provide much detail on how to handle documents with 0 and 1 timestamps so that they have some probability, so the time range of the data is normalized to $[0.01,0.99]$ in the paper.

With (2)–(6), Gibbs sampling algorithm for AToT model is summarized in Algorithm 1. The procedure itself uses only seven larger data structures, the count variables $n_a^{(k)}$ and $n_k^{(v)}$ , which have dimension $A\times K$ and $K\times V$ , respectively, their row sums $n_a$ and $n_k$ with dimensions A and K, Beta parameters Ψ with dimension $K\times \mathrm{2}$ , and the state variable $z_{m,n}$ , $x_{m,n}$ with dimension $W={\sum }_{m=\mathrm{1}}^M{N}_m$ .

Algorithm 1: Gibbs sampling algorithm for AToT model.

Algorithm AToTGibbs( $\{\mathit{w}\},\{\mathit{a}\},\{\mathit{t}\},\mathbf{\alpha },\mathbf{\beta },\mathbf{\psi },K$ )

Input:    word vectors $\{\mathit{w}\}$ , author vector $\{\mathit{a}\}$ , time vector $\{\mathit{t}\}$ , hyperparameters

       $\mathbf{\alpha },\mathbf{\beta }$ , Beta parameters ψ, topic number K

Global data: count statistics $\{n_a^{(k)}\}$ , $\{n_k^{(v)}\}$ and their sums $\{n_a\}$ , $\{n_k\}$

Output:   topic associations $\{\mathit{z}\}$ , author associations $\{\mathit{x}\}$ , multinomial parameters

      Φ and θ, Beta parameter estimates ψ, hyperparameter estimates α, β

// initialization

zero all count variables, $n_a^{(k)}$ , $n_a$ , $n_k^{(v)}$ , $n_k$

for all documents $m\in [\mathrm{1},M]$   do

  for all words $n\in [\mathrm{1},N_m]$ in document m  do

     sample topic index $z_{m,n}~\text{Multinomial}(\mathrm{1}/K)$

     sample author index $x_{m,n}~\text{Multinomial}(\mathit{p})$ with $p_a=\{\begin{array}{cc}\frac{\mathrm{1}}{A_m},\hfill & a\in {\mathit{a}}_m\hfill \\ \mathrm{0},\hfill & \text{otherwise}\hfill \end{array}$

     // increment counts and sums

      $n_{x_{m,n}}^{(z_{m,n})}+=1$ ; $n_{x_{m,n}}+=1$ ; $n_{z_{m,n}}^{(w_{m,n})}+=1$ ; $n_{z_{m,n}}+=1$

// Gibbs sampling over burn-in period and sampling period

while not finished do

  for all documents $m\in [\mathrm{1},M]$   do

     for all words $n\in [\mathrm{1},N_m]$ in documents m  do

        // decrement counts and sums

         $n_{x_{m,n}}^{(z_{m,n})}-=1$ ; $n_{x_{m,n}}-=1$ ; $n_{z_{m,n}}^{(w_{m,n})}-=1$ ; $n_{z_{m,n}}-=1$

        sample author index $\tilde{a}$ according to (2)

        sample topic index $\tilde{z}$ according to (3)

        // increment counts and sums

         $n_{\tilde{a}}^{(\tilde{k})}+=1$ ; $n_{\tilde{a}}+=1$ ; $n_{\tilde{k}}^{(w_{m,n})}+=1$ ; $n_{\tilde{k}}+=1$

        update ψ according to (6)

  if converged and L sampling iterations since last read out then

     // different parameters read outs are averaged

     read out parameter set Φ according to (4)

     read out parameter set θ according to (5)

4. Distributed Inference Algorithm

Our distributed inference algorithm, named as D-AToT, is inspired by AD-LDA algorithm [29, 30], following the main idea of the well-known distributed programming model, MapReduce [18]. The overall distributed architecture for AToT model is shown in Figure 5.

Figure 5

The overall distributed architecture for AToT model.

As stated in Figure 5, the master firstly distributes M training documents over P mappers, with nearly equal number $M/P$ of documents on each mapper. Specifically, D-AToT partitions document $\{\mathit{w}\}$ , $\{\mathit{a}\}$ , and $\{\mathit{t}\}$ into $\{\{{\mathit{w}}_{|p}\}{\}}_{p=\mathrm{1}}^P$ , $\{\{{\mathit{a}}_{|p}\}{\}}_{p=\mathrm{1}}^P$ , and $\{\{{\mathit{t}}_{|p}\}{\}}_{p=\mathrm{1}}^P$ and corresponding topic and author assignments $\{\mathit{z}\}$ and $\{\mathit{x}\}$ into $\{\{{\mathit{z}}_{|p}\}{\}}_{p=\mathrm{1}}^P$ and $\{\{{\mathit{x}}_{|p}\}{\}}_{p=\mathrm{1}}^P$ , where $\{{\mathit{w}}_{|p}\}$ , $\{{\mathit{a}}_{|p}\}$ , $\{{\mathit{t}}_{|p}\}$ , $\{{\mathit{z}}_{|p}\}$ , and $\{{\mathit{x}}_{|p}\}$ exist only on mapper p. The Author-Topic count $\{n_a^{(k)}\}$ and topic-word count $\{n_k^{(v)}\}$ are likewise distributed, denoted as $\{n_{a|p}^{(k)}\}$ and $\{n_{k|p}^{(v)}\}$ on mapper p, which are used to temporarily store local Author-Topic and topic-word counts.

In each Gibbs sampling iteration, each mapper p updates $\{{\mathit{z}}_{|p}\}$ and $\{{\mathit{x}}_{|p}\}$ by sampling $z_{m,n|p}$ and $x_{m,n|p}$ from the following posterior distributions:

$$\begin{array}{l}P\left(x_{m,n|p}\hspace{0.17em}|\hspace{0.17em}{\mathit{x}}_{\neg \left(m,n\right)|p},{\mathit{z}}_{|p},{\mathit{a}}_{|p},\mathit{\alpha }\right)\\ \propto \frac{n_{x_{m,n}}^{\left(z_{m,n}\right)}+{\alpha }_{z_{m,n}}-\mathrm{1}}{{\sum }_{k=\mathrm{1}}^K\left(n_{x_{m,n}}^{\left(k\right)}+{\alpha }_k\right)-\mathrm{1}},\\ P\left(z_{m,n|p}\hspace{0.17em}|\hspace{0.17em}{\mathit{w}}_{|p},{\mathit{z}}_{\neg \left(m,n\right)|p},{\mathit{x}}_{|p},{\mathit{t}}_{|p},\mathit{\alpha },\mathit{\beta },\mathbf{\Psi }\right)\\ \propto \frac{n_{z_{m,n}}^{\left(w_{m,n}\right)}+{\beta }_{w_{m,n}}-\mathrm{1}}{{\sum }_{v=\mathrm{1}}^V\left(n_k^{\left(v\right)}+{\beta }_v\right)-\mathrm{1}}\times \frac{n_a^{\left(z_{m,n}\right)}+{\alpha }_{z_{m,n}}-\mathrm{1}}{{\sum }_{k=\mathrm{1}}^K\left(n_{x_{m,n}}^{\left(k\right)}+{\alpha }_k\right)-\mathrm{1}}\\ \hspace{0.5em}\hspace{0.5em}\hspace{0.5em}\times \text{Beta}\left({\mathit{\psi }}_{z_{m,n}}\right)\end{array}$$

(7) and updates local $n_{a|p}^{(k)}$ and $n_{k|p}^{(v)}$ according to the new topic and author assignments. After each iteration, each mapper sends the local counts to the reducer and then the reducer updates Ψ and broadcasts the global $n_a^{(k)}$ , $n_k^{(v)}$ , and Ψ to all mappers. After all sampling iterations, the reducer calculates the Φ and Θ according to (4)-(5).

5. Experimental Results and Discussions

NIPS proceeding dataset is utilized to evaluate the performance of our model, which consists of the full text of the 13 years of proceedings from 1987 to 1999 Neural Information Processing Systems (NIPS) Conferences. The dataset contains 1,740 research papers and 2,037 unique authors. The distribution of the number of papers over year is shown in Table 2.

Table 2

Distribution of number of papers over year in NIPS dataset.

Year	Number of papers
1987	90 (5.2%)
1988	95 (5.5%)
1989	101 (5.8%)
1990	143 (8.2%)
1991	144 (8.3%)
1992	127 (7.3%)
1993	144 (8.3%)
1994	140 (8.0%)
1995	152 (8.7%)
1996	152 (8.7%)
1997	151 (8.7%)
1998	151 (8.7%)
1999	150 (8.6%)

In addition to downcasing and removing stop words and numbers, we also remove the words appearing less than five times in the corpus. After the preprocessing, the dataset contains 13,649 unique words and 2,301,375 word tokens in total. Each document's timestamp is determined by the year of the proceedings. In our experiments, K is fixed at 100 and the symmetric Dirichlet priors α and β are set at 0.5 and 0.1, respectively. Gibbs sampling is run for 2000 iterations.

5.1. Examples of Topic, Author Distributions, and Topic Evolution

Table 3 illustrates examples of 8 topics learned by AToT model. The topics are extracted from a single sample at the 2000th iteration of the Gibbs sampler. Each topic is illustrated with (1) the top 10 words most likely to be generated conditioned on the topic, (b) the top 10 authors which have the highest probability conditioned on the topic, and (c) histograms and fitted beta PDFs which show topics evolution patterns over time.

Table 3

An illustration of 8 topics from a 100-topic solution for the NIPS collection. The titles are our own interpretation of the topics. Each topic is shown with the 10 words and authors that have the highest probability conditioned on that topic. Histograms show how the topics are distributed over time; the fitted beta PDFs is shown also.

Topic 87		Topic 37		Topic 11		Topic 88
SVM and Kernel methods		Neural networks		Reinforcement learning		EM and mixture models
Word	Prop.	Word	Prop.	Word	Prop.	Word	Prop.
set	0.0188195	learning	0.01106740	state	0.0468466	density	0.0279477
support	0.0187117	network	0.00948016	learning	0.0252876	log	0.0217790
vector	0.0186039	neural	0.00780503	belief	0.0213999	distribution	0.0186946
kernel	0.0160163	input	0.00682192	policy	0.0182191	mixture	0.0178379
function	0.0146146	model	0.00681643	function	0.0175122	method	0.0144108
svm	0.0138060	training	0.00604202	action	0.0150383	gaussion	0.0142394
training	0.0129974	data	0.00597611	states	0.0148615	likelihood	0.0140681
problem	0.0124583	figure	0.00594316	reinforcement	0.0118574	entropy	0.0132113
space	0.0119731	networks	0.00560813	actions	0.0118574	gaussians	0.0123546
solution	0.0115957	function	0.00554222	mdp	0.0102670	form	0.0113264
Author	Prop.	Author	Prop.	Author	Prop.	Author	Prop.
Scholkopf_B	0.949692	Reggia_J	0.979832	Zhang_N	0.629412	Barron_A	0.608507
Crisp_D	0.888975	Todorov_E	0.976750	Rodriguez_A	0.578235	Wainwright_M	0.372871
Laskov_P	0.706170	Horne_B	0.974146	Dietterich_T	0.342954	Mukherjee_S	0.340927
Steinhage_V	0.634973	Thmn_S	0.973083	Sallans_B	0.228042	Li_J	0.337108
Chapelle_O	0.610385	Weigend_A	0.972806	Walker_M	0.189143	Jebara_T	0.253203
Li_Y	0.513418	McCallum_R	0.969777	Koller_D	0.1885150	Millman_K	0.171569
Herbrich_R	0.454384	Camana_R	0.969388	Yeung_D	0.1213730	Fisher_J	0.148230
Gordon_M	0.425090	Slaney_M	0.969382	Thrun_S	0.0842081	Ihler_A	0.128369
Vapnik_V	0.330421	Miikkulainen_R	0.968541	Konda_V	0.0680365	Beal_M	0.126578
Dom_B	0.286036	Bergen_J	0.968358	Parr_R	0.0468006	Hansen_L	0.0849109
Topic 47		Topic 78		Topic 51		Topic 58
Speech recognition		Bayesian learning		Eye recognition and factor analysis		Data model and learning algorithm
Word	Prop.	Word	Prop.	Word	Prop.	Word	Prop.
hmm	0.0415364	bayesian	0.0243032	sejnowski	0.0265409	learning	0.00904655
speech	0.0392921	sampling	0.0184560	eye	0.0265409	model	0.00752741
hmms	0.0216579	prior	0.0178563	ica	0.0183324	neural	0.00705102
mixture	0.0179708	distribution	0.0148578	vor	0.0159531	data	0.00700339
suffix	0.0104362	monte	0.0127588	disparity	0.0153583	function	0.00683930
probabilistic	0.00995527	carlo	0.0118592	head	0.0135738	network	0.00624646
probabilities	0.00947434	model	0.0109597	position	0.0125031	input	0.00593946
singer	0.00883310	posterior	0.0105099	eeg	0.0119083	set	0.00561128
acoustic	0.00883310	priors	0.00946041	parietal	0.0109566	networks	0.00556365
saul	0.00867279	sample	0.00901063	salk	0.0105997	figure	0.00545249
Author	Prop.	Author	Prop.	Author	Prop.	Author	Prop.
Rigoll_G	0.460882	Schuurmans_D	0.651505	Sejnowski_T	0.410459	Gray_M	0.974482
Singer_Y	0.437547	Sykacek_P	0.495506	Pouget_A	0.269781	Dimitrov_A	0.973538
Nix_D	0.192342	Andrieu_C	0.413324	Anastasio_T	0.112957	Galperin_G	0.97094
Saul_L	0.170699	Rasmussen_C	0.344185	Horiuchi_T	0.0328485	Malik_J	0.968536
Hermansky_H	0.0795602	Zlochin_M	0.244745	Albright_T	0.0099278	Davies_S	0.966534
Roweis_S	0.0391364	Beal_M	0.157807	Jousmaki_V	0.00791139	Cook_G	0.96519
Attias_H	0.0357538	Hansen_L	0.122773	Fredholm_H	0.00681818	Ghosn_J	0.964184
Movellan_J	0.033414	Herbrich_R	0.0882701	Bohr_J	0.00643777	Orponen_P	0.964184
Schuster_M	0.0293324	Downs_O	0.0694726	Ramanujam_N	0.00621891	Yen_S	0.963001
Muller_K	0.028258	Williams_C	0.0652069	Dixon_L	0.00585938	Chatterjee_C	0.962627

5.2. Author Interest Evolution Analysis

In order to analyze further author interest evolution, it is interesting to calculate

$$\begin{array}{cc}& P\left(z,t\hspace{0.17em}|\hspace{0.17em}a\right)=P\left(z\hspace{0.17em}|\hspace{0.17em}a\right)p\left(z\hspace{0.17em}|\hspace{0.17em}t\right)={\vartheta }_{a,z}\times \text{Beta}\left({\mathit{\psi }}_z\right).\end{array}$$

(8) In this subsection, we take Sejnowski_T as an example, who published 43 papers in total from 1987 to 1999 in the NIPS conferences, as shown in Figure 6(a). The research interest evolution for Sejnowski_T is reported in Figure 6(b), in which the area occupied by a square is proportional to the strength of his research interest.

Figure 6

The distribution of number of publications and research interest evolution for Sejnowski_T.

From Figure 6(b), one can see that Sejnowski_T's research interest focused mainly on Topic 51 (Eye Recognition and Factor Analysis), Topic 37 (Neural Networks), and Topic 58 (Data Model and Learning Algorithm) but with different emphasis from 1987 to 1999. In the early phase (1989–1993), Sejnowski_T's research interest is only limited to Topic 51 and then extended to Topic 37 in 1994 and Topic 58 in 1996 with great research interest strength and finally back to Topic 51 after 1997. Anyway, Sejnowski_T did not change his main research direction, Topic 51, which is verified from his homepage again.

5.3. Predictive Power Analysis

Similar to [5], we further divide the NIPS papers into a training set ${𝒟}^{\text{train}}$ of 1,557 papers and a test set ${𝒟}^{\text{test}}$ of 183 papers of which 102 are single-authored papers. Each author in ${𝒟}^{\text{test}}$ must have authored at least one of the training papers. The perplexity, originally used in language modeling [31], is a standard measure for estimating the performance of a probabilistic model. The perplexity of a test document $\tilde{m}\in {𝒟}^{\text{test}}$ is defined as the exponential of the negative normalized predictive likelihood under the model:

$$\begin{array}{l}\text{perplexity}\left({\mathit{w}}_{\tilde{m},·},{\mathit{t}}_{\tilde{m},·}\hspace{0.17em}|\hspace{0.17em}{\mathit{a}}_{\tilde{m}},\mathit{\alpha },\mathit{\beta },\mathbf{\Psi }\right)\\ =\exp \left[-\frac{\ln P\left({\mathit{w}}_{\tilde{m},·},{\mathit{t}}_{\tilde{m},·}\hspace{0.17em}|\hspace{0.17em}{\mathit{a}}_{\tilde{m}},\mathit{\alpha },\mathit{\beta },\mathbf{\Psi }\right)}{N_{\tilde{m}}}\right]\end{array}$$

(9) with

$$\begin{array}{l}P\left({\mathit{w}}_{\tilde{m},·},{\mathit{t}}_{\tilde{m},·}\hspace{0.17em}|\hspace{0.17em}{\mathit{a}}_{\tilde{m}},\mathit{\alpha },\mathit{\beta },\mathbf{\Psi }\right)\\ =\frac{\mathrm{1}}{{\left[A_{\tilde{m}}\right]}^{N_m}}\times {\displaystyle \sum }_{{\mathit{z}}_{\tilde{m},·}}\text{Beta}\left({\psi }_{z_{\tilde{m},n},\mathrm{1}},{\psi }_{z_{\tilde{m},n},\mathrm{2}}\hspace{0.17em}|\hspace{0.17em}{𝒟}^{\text{train}}\right)\\ \hspace{0.5em}\hspace{0.5em}\hspace{0.5em}\times {\displaystyle \int }p\left(\mathbf{\Phi }\hspace{0.17em}|\hspace{0.17em}\mathit{\beta },{𝒟}^{\text{train}}\right){\displaystyle \sum }_{{\mathit{z}}_{\tilde{m},·}}{\phi }_{z_{\tilde{m},n},w_{\tilde{m},n}}d\mathbf{\Phi }\\ \hspace{0.5em}\hspace{0.5em}\hspace{0.5em}\times {\displaystyle \int }p\left(\mathbf{\Theta }\hspace{0.17em}|\hspace{0.17em}\mathit{\alpha },{𝒟}^{\text{train}}\right){\displaystyle \sum }_{{\mathit{x}}_{\tilde{m},·}}{\vartheta }_{x_{\tilde{m},n}},z_{\tilde{m},n}d\mathbf{\Theta }\mathbf{.}\end{array}$$

(10)

We approximate the integrals over Φ and Θ using the point estimates obtained in (4)-(5) for each sample $s\in \{\mathrm{1,2},\dots ,\mathrm{10}\}$ of assignments x , z and then average over samples. Figure 7 shows the results for the AToT model and AT model in a post hoc fashion on 102 single-authored papers. It is not difficult to see that the perplexity of AToT model is smaller than that of AT model when the number of topics > 10, which indicates that AToT model outperforms AT model.

Figure 7

Perplexity of the 102 single-authored test documents.

6. Conclusions

With a dynamic users’ interest discovery model, one can answer many important questions about the content of information uploaded or shared to SNS. Based on our previous work, Author-Topic over Time (AToT) model [19], for documents using authors and topics with timestamps, this paper proposes a dynamic users’ interest discovery model with distributed inference algorithm following the main idea of MapReduce, named as Distributed AToT (D-AToT) model. The D-AToT model combines the merits of AT and ToT models. Specifically, it can automatically detect latent topics, users’ interests, and their changing patterns from large-scale social network information. The results on NIPS dataset show the increase of salient topics and more reasonable users’ interest changing patterns.

One can generalize the approach in the work to construct alternative dynamic models from other static users’ interest discovery models and ToT model with distributed inference algorithm. As a matter of fact, our work currently is limited to deal with the users and latent topics with timestamps in SNS. Though NIPS proceeding dataset is a benchmark data for academic social network, the D-AToT model ignores the links in SNS. In ongoing work, novel topic model, considering the links in SNS, will be constructed to identify the users with similar interests from social networks.