A two-stage deep semantic model for movie recommendation with long-term user profiling on Spark

Abstract

With the growth of the film market, users are often overwhelmed by irrelevant information, making it difficult to make accurate movie choices. To address this, we propose an improved deep structured semantic model (DSSM) movie recommendation algorithm based on Spark technology. The algorithm employs two DSSMs: one to extract users’ long-term preferences and another for final movie recommendations. The preference model integrates explicit and implicit user interaction features, while the recommendation model utilizes recent viewing history and search behavior. Experiments were conducted on the MovieLens 100K and MovieLens 1M datasets, using metrics such as accuracy, recall, mean reciprocal ranking (MRR), diversity, and normalized discounted cumulative return (NDCG). The results show that the improved algorithm outperforms others in recall, with rates of 36.0% and 42.1% for the top 40 recommendations on the MovieLens 100K and 1M datasets, respectively. Additionally, the algorithm’s NDCG scores for the top 10 recommendations were 0.525 and 0.554, higher than those of competing algorithms. The system, built on Spark technology, passes all performance tests, with an average response time of under 600 ms. These results demonstrate that the proposed DSSM-based movie recommendation algorithm can provide accurate recommendations while overcoming the challenge of data sparsity. The research not only provides an efficient and accurate solution for the field of movie recommendation, but also shows the significant advantages of Spark technology in processing large-scale data sets, which has a wide range of application value and promotion prospects.