Quality assessment and sexual health information analysis of mobile dating applications for men who have sex with men: A comprehensive evaluation using the User Version of the Mobile App Rating Scale

Abstract

Objective

Numerous mobile apps designed for men who have sex with men (MSM) are launched in recent years, potentially serving as crucial platforms for disseminating sexual health information and promoting healthy sexual behaviors. This study aimed to evaluate the quality of dating apps for MSM on the apps market and analyze the sexual health information they provide.

Methods

A systematic search was conducted in major apps platforms in China (including Apple, OPPO, Huawei, and Tencent) to identify current MSM dating apps. Apps fitting the inclusion criteria were independently assessed by two MSM users via using the User Version of Mobile App Rating Scale (uMARS), a user-assessed quality scale for mobile apps.

Results

Fifty mobile apps were identified and assessed, with an overall uMARS score of 3.64 ± 0.53 out of 5. More than half (29 out of 50) of apps included sexual health information.

Conclusions

The overall quality of the apps was satisfactory, whereas significant variations existed in sexual health information content and focus. It is recommended that app developers can further collaborate with public health experts to enhance the accuracy and utility of sexual health information on these platforms.

Keywords

Mobile apps quality assessment app store men who have sex with men HIV sexual health

Introduction

Background

As information technology advances and mobile internet services improve, the number of smartphone users has increased significantly. Statistics have shown that the number of smartphone users in China has reached approximately 1.078 billion by 2023, with a smartphone usage rate of 74.4%.¹ Meanwhile, the number of mobile application downloads has reached a remarkable 1113 billion downloads.² Among all categories of applications (apps), social networking apps, particularly dating apps, have become increasingly prevalent in the digital generation, which fundamentally changed how people connect and interact.³ Previous studies estimate that there are approximately 8 million men who have sex with men (MSM) in mainland China, representing 1.73% of adult males.⁴ The widespread and active utilization of dating apps among this population is notable, with a rate of 77.9%.⁵

Men who have sex with men dating apps have dramatically altered the patterns of social interaction within this community through innovative digital affordances and geolocation features.^6,7 China MSM communities use dating platforms not only for casual encounters but also for fostering friendships and building community connections, providing a private space for social interaction and identity expression.⁸

However, research has identified that MSM dating apps have created a virtual environment associated with unprotected sexual behaviors and other high-risk sexual activities, raising several health concerns in China.⁹ Empirical studies have demonstrated that online partner-seeking behaviors through these platforms are associated with increased sexual health risks. Specifically, researchers found that such behaviors were correlated with higher rates of condomless sexual intercourse and increased sexually transmitted infection (STI) diagnosis.¹⁰ This risk is particularly pronounced among younger users, as mobile dating apps could facilitate HIV transmission risk due to limited sexual health knowledge and unsafe sex practices.¹¹ Furthermore, the dating apps’ usage-related sexual health issues can be complicated by substance use behaviors among all types of users. Recent studies in China have revealed that alcohol and drug use can significantly increase sexual risk-taking behaviors among MSM dating app users, potentially exacerbating the spread of HIV and other STIs.^12,13

These technological advancements not only offer unprecedented opportunities for social networking for MSM but also present both opportunities and challenges for public health. For the MSM community, dating apps can be the more effective platforms for disseminating health information than common disease prevention apps, particularly in areas or societies where the visibility of LGBTQ+ community is limited.¹⁴ These platforms potentially serve as crucial channels for health information dissemination, especially given that the inaccessibility to sexual health services due to cultural barriers or social stigma may exist in some societies.¹⁵ Several factors could enhance the trustworthiness of sexual health information on these platforms, including the partnerships with recognized healthcare institutions, content verification by medical professionals, and transparent sourcing of health recommendations.^16,17 Additionally, app features may encourage users to engage with and follow sexual health guidance, such as interactive educational content, personalized health reminders, and tailored behavioral feedback based on individual risk assessments.¹⁸

In response to these challenges, various initiatives have emerged. Various nongovernmental organizations have collaborated with dating platform to promote sexual health and HIV prevention. Some apps have begun incorporating features related to substance use awareness and sexual health education, ranging from user profile indicators to educational resources.^19,20 Despite the incorporation of health-related functions in some apps, these features were not systematically assessed across different platforms. A previous study evaluated 43 MSM apps in China using the original Mobile App Rating Scale (MARS).²¹ However, their assessment relied on expert evaluations using the original MARS scale rather than considering user perspectives. Additionally, the mobile app landscape has evolved substantially since their data collection in 2018. In this study, we assessed MSM dating apps by employing the User Version of the MARS (uMARS) to capture end-user experiences, providing new perspectives for evaluation. Furthermore, we incorporated more apps into this study and conducted a more granular analysis of sexual health features. This approach provides app quality and health promotion potential from users’ perspectives, which identify current gaps and provide evidence-based recommendations for the future development and strategies of sexual health promotion.

Objective

The objective of this study was to evaluate the quality of mainstream MSM dating apps for the MSM community in Chinese app stores using the uMARS. Meanwhile, we aim to survey whether these apps provide sexual health information and assess their room for improvement, and we anticipate these outcomes can draw app developers’ and public health experts’ attention to MSM dating apps.

Methods

Selection of app markets

Android and iOS are two of the most prevalent smartphone operating systems in China, collectively accounting for over 84% of the market. Android has a 64% market share, while iOS accounts for 20%.²² Furthermore, the third-ranked Harmony OS accounting for 16% market share is currently compatible with Android applications. Due to the unavailability of Google Play services for app downloads in China, the Huawei App Market, Tencent App Store, and OPPO Software Store are the three most prominent Android app markets, according to the Talking’s App Distribution Active Ranking for 2023.²³ Based on the aforementioned market data, three mainstream Android app platforms in China (Huawei, Tencent, and OPPO APP Store) and the Apple App Store in the iOS market were selected for this study to search for mobile apps designed for MSM dating.

Search strategy

The study employed Qimai, a professional mobile application data analysis platform, for systematic data collection.²⁴ Qimai Data are a renowned platform that provides search ranking and app multidimensional data from iOS and Android app markets, including Huawei, Tencent, OPPO, Apple, and other App stores. A cross-search approach was employed, conducting searches across multiple app markets and using a series of relevant search terms. Four Chinese search terms were selected for analysis, including “homosexual,” “comrade,” “gay friend,” “gay” (these terms in this manuscript were translated from Chinese), and one English search term, “gay,” were used to searched in four major app markets.

The search was scheduled from 28 May 2024, and the app market country/region was designated as China. Considering the homepage effect in app market search and download,²⁵ the top 50 apps in the search results were analyzed.

The methods above enable the comprehensive capture of the performance of pertinent apps across diverse platforms and search terms, thereby ensuring the apps we acquired were as exhaustive and representative as possible.

App selection criteria and evaluator characteristics

Initially, the author (XZ) conducted an application search using predefined search terms and identified potential applications via screening of application titles and descriptions. The author has 4 years of experience operating mobile Internet products and a comprehensive understanding of the application market, including app uploading, version control, and app store optimization strategies.

Subsequently, the apps were further screened by two evaluators who were recruited online. The evaluators downloaded the apps according to inclusion and exclusion criteria based on the initial list screened by the authors. They then proceeded to assess the apps to confirm their eligibility independently. The apps of iOS version were preferred for apps evaluation; under the circumstances that the iOS version was unavailable, the apps of Android version were applied. In the event of disagreement between the two evaluators during the eligibility process, the issue was resolved through mutual negotiation to reach a consensus.

Evaluators were recruited through the following process: an announcement was posted on the online platform (Xiaohongshu). Requirements for evaluators were as follows: (a) at least 18 years of age, (b) MSM population, and (c) having used MSM dating apps within the past six months.

Apps that meet the following criteria will be included: (a) app designed for MSM dating (containing functions to view profiles and chat); (b) app interface supports Chinese or English; and (c) app is free to download. Apps that meet the following criteria will be excluded: (a) cannot be downloaded, (b) unable to access app content, (c) main function requires payment, and (d) apps for games, or news, or audio-visual applications rather than chatting.

Two evaluators with prior MSM dating app experience were recruited. Evaluator A (23 years old, living in Guangzhou) holds an associate degree with basic English proficiency (Gaokao English score: 71/150). Evaluator B (21 years old, living in Wuhan) is an undergraduate student with intermediate English proficiency (CET-4 score: 465/710, passed). Both of them are native Chinese speakers, evaluator A required translation assistance for English interfaces, while Evaluator B could assess English apps independently. The familiarization of apps and discussion about the apps interface was conducted prior to assessment in order to minimize the divergence due to users’ perspectives.

Data extraction

To analyze all the applications in depth, we used a semistructured database approach for documentation recording. The following characteristics of the applications were acquired:

General characteristics

Application name

App category

Smartphone platform

Star rating

Number of comments

Developer country/region

Duration of release

Age rating

Language

Location services

Sexual health-related functions

Personal information of HIV status: HIV status, last time tested, test reminders

Personal information of sexual safety: condoms, prophylaxis, treatment prevention

Personal information of vaccination: Covid-19, HPV, monkeypox/immunization, meningitis, hepatitis

Access for health resources: frequently asked questions (FAQs) for sexual health, testing, medicines (preexposure prophylaxis (PrEP)/PEP), drugs, alcohol

This approach allowed us to systematically assess and compare each app’s characteristics and features of sexual health promotion features, which provided detailed data support for further analysis.

Evaluation of app quality

This study assessed apps from the user’s perspective, so the evaluators recruited for this study were not industry experts but ordinary users. We used uMARS to assess the quality of the apps, which was derived from MARS, that was developed initially to assess mHealth apps as a reliable tool with high internal consistency and inter-rater reliability. The uMARS is a tool for assessing the quality of mHealth apps, which is adapted from the MARS scoring tool and is intended to serve as a simpler, more user-friendly alternative method for assessing apps.²⁶ MARS requires training and expertise in mHealth and related health fields to administer, whereas uMARS eliminates the need for trained experts and provides a reliable tool for application developers and users to assess the quality of mHealth applications.

The uMARS scale utilizes a 5-point Likert scale (1 = inadequate to 5 = excellent) for rating each item.

Scores are interpreted as follows:

1 to <2: poor quality

2 to <3: fair quality

3 to <4: good quality

4 to 5: excellent quality

The overall app quality score is calculated as the mean of four subscale scores: engagement, functionality, aesthetics, and information, along with two additional subscales as references: subjective quality and perceived impact.

Engagement assesses how engaging the app is for users. Sample questions include:

“Is the app fun/entertaining to use? Does it have components that make it more fun than other similar apps?”

“Is the app interesting to use? Does it present its information in an interesting way compared to other similar apps?”

“Does it allow you to customize the settings and preferences that you would like (e.g., sound, content, and notifications)?”

Functionality evaluates usability and technical performance. Sample questions include:

“How accurately/fast do the app features (functions) and components (buttons/menus) work?”

“How easy is it to learn how to use the app; how clear are the menu labels, icons, and instructions?”

“Does moving between screens make sense? Does the app have all necessary links between screens?”

Aesthetics focuses on the app's visual appeal and design quality. Sample questions include:

“Is the arrangement and size of buttons, icons, menus, and content on the screen appropriate?”

“How high is the quality/resolution of graphics used for buttons, icons, menus, and content?”

“How good does the app look? Is it pleasant, professionally designed, and visually consistent?”

Information examines the relevance and credibility of the content provided in the app. Sample questions include:

“Is app content correct, well-written, and relevant to the goal/topic of the app?”

“Is the information within the app comprehensive but concise?”

“Does the information within the app seem to come from a credible source?”

Additionally, subjective quality reflects the user’s overall satisfaction, asking:

“Would you recommend this app to people who might benefit from it?”

“What is your overall (star) rating of the app?”

Perceived impact measures the app’s influence on user behavior and knowledge. Sample questions include:

“This app has increased my knowledge/understanding of the health behavior.”

“The app has changed my attitudes toward improving this health behavior.”

“Use of this app will increase/decrease the health behavior.”

For both subjective quality and perceived impact sections, higher scores indicate more positive evaluations. This standardized scoring system enables consistent assessment across applications and facilitates comparative analysis.

Before scoring the apps, both evaluators read and familiarized themselves with the original uMARS protocol and its Chinese-translated version. Using the uMARS, both evaluators preassessed the apps excluded from the study, Grindr, based on our inclusion and exclusion criteria. The results were also discussed until an agreement was reached to better understand the purpose and meaning of each item on the scale.

We used the version of the apps downloaded on 28 May 2024, and two evaluators were asked to conduct independent tests from 29 May 2024 to 5 June 2024. All apps included in this study were rated with uMARS, and evaluators spent at least 10 min on scoring for each app and finding sexual health-related information as detailed as possible. If there was a large difference (≥2) in quality subscale rating for a particular app, the app was discussed between reviewers and reassessed to reach a consensus. The above methods were used to ensure objectivity and consistency in the assessment process, ensuring the reliability and validity of the results.

Statistical analysis

Descriptive statistics were applied to analyze the application’s general characteristics and sexual health-related information. In accordance with the recommendations of uMARS developers,²⁶ the mean of the assessor scores was calculated for uMARS scores, as well as for each domain and subscale.

To ascertain the interassessor reliability of the uMARS scales, we calculated the intragroup correlation coefficients (ICCs) between assessors, utilizing a two-way mixed, absolute consistency, average measurement model to estimate the reliability of the measures between them.²⁷ We employed linear regression analysis to determine if there is a relationship between the app’s star rank in the App Store or the number of reviews and the uMARS score. All data analyses were performed using SPSS Statistics (IBM, Version 26.0).

Results

Systematic search result

We searched the Chinese app store using specified search terms and identified potentially qualified 668 Android apps and 747 iOS apps. App screening process is illustrated in the accompanying figure (Figure 1). The authors included and excluded apps based on the titles and descriptions displayed in the app market. After removing duplicates, a total of 64 apps met the inclusion criteria.

Figure 1.

Application screening flowchart.

After further review by two evaluators, six apps were inaccessible due to network problems, six apps forced payment after logging in, one app was no longer available in the app store when downloaded, and another app did not meet the targeted MSM population, so 12 apps were excluded. Finally, 50 apps were included in this study.

General characteristics

The app's name, type of application, developer’s country/region, launch time, star rating, number of reviews, age rating, language, and location services are acquired. Supplemental Appendix 1 contains a comprehensive array of information.

Among the 50 applications included in this study (Table 1), 41 are compatible with both the Android and iOS platforms, 8 are exclusive to the iOS platform, and 1 is exclusive to the Android platform. The categorization of the apps was based on their classification within the app market. The result of categorization was as follows: social (46 apps), entertainment (2 apps), and lifestyle (2 apps). Regarding the launch time of the apps, as of 10 June 2024, 10 apps have been listed on the market for more than 10 years. The earliest app, Recon, was released on 18 December 2009. In terms of the number of app developers, China is in the lead with 56% (28/50), followed by the United States with 16% (8/50) and the United Kingdom with 6% (3/50). Of these apps, 40% (20/50) have an app market star rating up to 4. Chinese language support is available in 76% of apps (38/50).

Table 1.

Basic characteristics of the application (N = 50).

Characteristics of apps		N	%
Smartphone platform	Both	41	82.00
	Only iOS	8	16.00
	Only Android	1	2.00
App category^a	Social networking	46	92.00
	Entertainment	2	4.00
	Lifestyle	2	4.00
Released time^b	<3 years	8	16.00
	3–5 years	22	44.00
	5–10 years	10	20.00
	>10 years	10	20.00
Developer country/region^c	China	28	56.00
	United States	8	16.00
	Unknown	3	6.00
	United Kingdom	3	6.00
	Germany	2	4.00
	Other	8	16.00
Star rating^a	1–2.9	4	8.00
	3–3.9	23	46.00
	4–5	20	40.00
	Inapplicability	3	6.00
Number of comments^a	1–99	24	48.00
	100–999	14	28.00
	1000–10,000	6	12.00
	>10,000	3	6.00
	Inapplicability	3	6.00
Age rating^a	17+	47	94.00
	9+	1	2.00
	4+	1	2.00
	Inapplicability	1	2.00
Language	Chinese	38	76.00
	English	12	24.00
Location services	Yes	48	96.00
	No	2	4.00

One hundred forty-nine apps get data from the Apple App Store; one app gets data from the Android market.

The years the app will be available on the app market as of 10 June 2024.

Obtained from the information displayed on the application’s official website and the user agreement.

Most dating apps (96%, 48/50) employ geolocation services, which users can utilize to showcase nearby individuals, check distances and locations of other users, or tag their locations when they update post. Furthermore, three apps were identified that set age ratings to permit access to users below the legal adult age. The mean App Market star rating of the 50 apps included in this study was 3.76 (SD = 0.98), with minimum rating of 1 and maximum rating of 5.

Sexual health features

Over half of MSM dating apps include sexual health-related features (58%, 29/50). The three most prevalent features, in terms of frequency, were the provision of FAQ on sexual health (42%, 21/50), the availability of profiles indicating whether users were on PrEP (20%, 10/50), and information on personal HIV test status and HIV test service access (16%, 8/50) (Table 2). The distribution of complete sexual health features can be viewed in Supplemental Appendix 2.

Table 2.

Sexual health-related features of the app (N = 50).

Health information of apps		N	%
Personal information of HIV status	Status	8	16.00
	Last tested date	6	12.00
	Testing reminders	4	8.00
Personal information of sexual safety	PrEP^a	10	20.00
	Condoms	6	12.00
	Undetectable^b/TasP^c	7	14.00
Personal information of vaccination	COVID-19	4	8.00
	HPV	4	8.00
	Monkeypox/immunization	3	8.00
	Meningitis	3	6.00
	Hepatitis	3	6.00
Access for health resources	FAQ	21	42.00
	Testing	8	16.00
	Medications	7	14.00
	Drugs	5	10.00
	Alcohol	4	8.00

Preexposure prophylaxis (PrEP) is a medication taken by people at risk of HIV to prevent infection from sex or injection drug use.

When people with HIV have an undetectable viral load, they cannot transmit HIV to others through sexual transmission.

Treatment as prevention (TasP) refers to a strategy of reducing the viral load in HIV-infected people to the undetectable levels through effective antiretroviral therapy, thereby preventing the spread of the virus.

HPV: human papillomavirus; FAQ: frequently asked question.

User Version of MARS scores

The mean uMARS score for all applications was 3.64 (SD = 0.53) out of score 5. The aesthetics dimension exhibited the highest mean score (3.98, SD = 0.56) among the uMARS quality subscale dimensions. Conversely, the engagement dimension demonstrated the lowest mean score (3.18, SD = 0.77). Additionally, the information dimension exhibited scattered ratings (3.34, SD = 0.83), with an inter-rater uMARS ICC (ICC = 0.833, 95% CI 0.582–0.821) (in Table 3). This indicates a high level of agreement. A notable discrepancy is observed in the uMARS scores of the apps in different languages (overall uMARS), with the apps offering the Chinese language exhibiting a higher average uMARS score than those offering the English language. The mean uMARS scores of Chinese language applications were 3.69 (2.83–4.77), while those of English language applications were 3.48 (2.8–4.09) (Figure 2). The raw data for uMARS scores can be viewed in Supplemental Appendix 3.

Figure 2.

User Version of Mobile App Rating Scale (uMARS) dimension scores. Each point represents the score for an individual app. The box plot shows the median, first and third quartiles, and the minimum and maximum scores.

Table 3.

Descriptive results of Mobile App Rating Scale scores.

App Quality Ratings	Rater1, mean (SD)	Rater2, mean (SD)	Average, mean (SD)	ICC (95% CI)
Engagement	2.98 (0.77)	3.39 (0.83)	3.18 (0.77)	0.764 (0.205–0.908)
Functionality	3.85 (0.42)	4.01 (0.57)	3.95 (0.45)	0.602 (0.344–0.766)
Aesthetics	3.86 (0.49)	4.09 (0.72)	3.98 (0.56)	0.635 (0.396–0.785)
Information	3.37 (0.84)	3.53 (0.90)	3.45 (0.83)	0.785 (0.587–0.891)
App quality^a	3.52 (0.49)	3.76 (0.63)	3.64 (0.53)	0.833 (0.582–0.821)
Subjective quality	1.84 (0.54)	2.12 (0.87)	1.98 (0.63)	0.509 (0.260–0.692)
Perceived impact	1.84 (1.00)	2.11 (1.17)	1.97 (1.04)	0.781 (0.617–0.876)

ICC: intraclass correlation coefficient.

App quality: the average of engagement, functionality, aesthetics, and information.

We also applied simple linear regression to predict the relationship between star ratings in the app store and mean uMARS scores, which showed no statistical significance (F1, 45 = 0.002, p > 0.05), R² = 0.000 (Figure 3). The relationship between the number of user reviews and the uMARS score showed a significant positive correlation between two variables (F1, 45 = 11. 297, p < 0.05), R² = 0. 201 (Figure 4).

Figure 3.

Regression analyses of the association between the star rating in the app store and the mean User Version of Mobile App Rating Scale (uMARS).

Figure 4.

Regression analyses of the association between the number of reviews and the mean User Version of Mobile App Rating Scale (uMARS).

Discussion

Principal finding

This study comprehensively evaluated the 50 MSM dating apps and analyzed the quality using the uMARS scale. This assessment encompassed four key domains: engagement, functionality, aesthetics, and information. We examined the subjective quality and perceived impact of the apps. Furthermore, we quantified the number of in-app sexual health messages. Overall, the quality of 43 apps was considered as acceptable, with uMARS score higher than score 3. Additionally, three apps (Finka, Blued, and GYOU) were found to have exceptional quality in both overall uMARS quality and subjective quality scores, with scores exceeding 4.0 and 3.0, respectively. Furthermore, 30% (12/50) of the apps reached acceptable level, with score higher than 3, on the perceived impact score. Moreover, 58% (29/50) of the apps provided sexual health information.

Geographic location and health services access

The popularity and application of smartphone location services may give rise to a notable shift in MSM dating patterns, which gradually shift from face-to-face approach to both online and offline approaches. Recently, more and more people look for sexual partners online through the Internet or mobile applications.^5,27,28 The provision of real-time and convenient location services enable dating apps to offer location- and distance-based matching and personalized services, thereby assisting users in identifying potential dates in their vicinity area, enhancing interaction efficiency, and increasing user experience.²⁹

Furthermore, some MSM dating apps have formed partnerships with health organizations to promote sexual health information and resources through app profile fields and educational content.^29–31 For example, nine apps in this study (including five apps in China and four international apps) allow users to locate MSM-friendly health resources such as HIV testing institution and sexual health clinics. However, it is important to note that these partnerships typically focus on educational content and resource sharing rather than data collection or surveillance. Therefore, while dating apps can play an important role in connecting users to health resources and information via geographic location function, this should be done through transparent partnerships that respect user privacy and comply with app policies to avoid controversial issues.³² Future development of health-related features should prioritize user consent, data protection, and clear communication about how personal information will be used.

Functional differences due to culture and region

Cultural and regional differences impact on the functionality of MSM dating apps, which vary in design and functionality across cultures and regions to align with the specific needs and social environments of local users. In regions where the social culture and attitudes toward the homosexual community are more conservative or discriminatory, the design of MSM dating apps may be more cautious. In order to meet the specific requirements of different regions, the application development team will customize and adapt the features in accordance with the legal requirements and social milieu. Some apps will default in some areas, where homosexuality is a sensitive issue, to enable higher privacy protection settings,³³ including:

1. Anonymous use: Users can use it anonymously, avoiding using real names or photos and reducing the risk of being identified.

2. Hidden location: Hiding distance may be a wise choice for users living in regions where homosexuality or male–male interactions are criminalized, such as Russia and the Middle East, and the app will enable the “hide distance” feature by default for people in these regions.

3. Security alerts: When a user travels to a risk country and launches the app, it will see an alert reminding user the presence of local laws criminalizing homosexual acts.

In this study, the top three dating apps, judging by the number of sexual health information features, are all from America, including Jack'd, SCRUFF, and Hornet. All three apps allow users to input data about three main categories, including HIV testing status, protection preferences, and vaccinations, which contribute to higher sexual health feature points. Moreover, this functionality was also found in apps from other regions, whereas this feature was not identified in any apps developed by mainland Chinese developers. It is worth mentioning that when we checked the Blued's user agreement, even though the application’s personal information protection policy mentions that it collects personal HIV testing information, including status and last tested date, it is not displayed on the user interface.³⁴ By parsing the installation package and personal information protection policy, we found that the Blued's source code contained information about HIV and PrEP (Figure 5). We indicate that the application’s developer had considered this feature before, but it may have not yet been launched for some reason.

Figure 5.

Blued APK resource code: personal preexposure prophylaxis (PrEP) and HIV testing information.

Furthermore, the profile of vaccination status revealed that three America apps (Jack'd, SCRUFF, and Hornet) have listed options for meningitis and hepatitis, whereas no other apps have these options. This may be related to the prevalence of such sexually transmitted diseases among the MSM community in the United States. For example, in 2012, an outbreak of meningitis cases among MSM in New York, America, prompted public health agencies to recommend meningitis vaccination for MSM group. This recommendation was disseminated through paid banners and pop-up advertisements on MSM apps, including Grindr.³⁵ Hepatitis A and B vaccines have been developed since the late 1980s, and the MSM community has been identified as a high-risk group of hepatitis, so the recommendation for such vaccination was proposed.³⁶

Consequently, the consideration of user habits and cultural factors when designing functionality is essential to make dating apps be tailored to the situation of each region in order to meet users’ needs and make efforts to promote healthy sexual behaviors.

Alcohol and drug

The influence of alcohol and drug use on the transmission of HIV and other STIs is prominent and intricate. The consumption of these substances may alter individuals’ behaviors, including a diminution of their capacity for judgment and self-control, hence increase the risk of STI transmission.^34,35 However, the apps included in this study provided less detailed information on the specific effects of such substances.

Only seven apps referred to the use of drugs, and six apps (including Blued, Blued Lite, BRISH, Chance, Closet Match, and TRANSLOVE) have in-app warnings about the dangers of drug abuse. At the same time, one app (Gsland) permitted users to add a “no drugs” label to their profiles; thereby, user can explicitly express their attitudes toward drugs. Furthermore, six apps provided descriptions of the risk of alcohol consumption, and two apps (BRISH and TRANSLOVE) include a page that outlines the potential risks associated with alcohol consumption. The other four apps (Catscha, gayfriednly, myBOY, and WE) allow users to indicate their alcohol consumption habits in their profiles. Although this feature enables users to convey their stance on alcohol consumption, there is no clear description of the dangers of alcohol consumption and STI, neither the clear education nor warning information, which may fail to increase users’ awareness of the risk of STI transmission due to alcohol overconsumption. In addition, as recommended in the NICE AF report,³⁷ app developers should consider including more features related to drug and alcohol use. This will not only increase users’ health awareness but may also be an aspect of competitive differentiation for apps. However, local laws and regulations and cultural sensitivities also need to be considered when implementing these recommendations in the Chinese context.

Protection of minors

Generally speaking, MSM dating apps require user must reach the age of majority. Marketplaces for such apps typically adhere to local laws and regulations that mandate developers to assign the corresponding age rating when making their apps available.^35,36 Secondly, users are typically required to provide their date of birth upon registration or while utilization of certain features. It is a common practice among developers to display a pop-up window indicating whether the user is an adult or a date option to select age or date of birth, even though the minors can actually circumvent the age restriction by providing a false age. However, the analysis of the apps included in this study revealed that two of these apps had lower age grading requirements in the app market, including Youth booklet (9+) and Form Marriage Circle (4+).

Even though the above methods can serve as a primary screening tool, minors can still easily circumvent these restrictions. Therefore, it is recommended that app marketplaces should enhance their vetting procedures for apps information. App developers should implement more robust measures to protect minors, such as manual review, age prediction models,³⁸ natural language processing,³⁹ and other techniques to identify underage users’ information, and thus further preventing minors from accessing their apps.

Limitations

Our study has the following limitations. First, the app market is changing rapidly. Since mobile apps are frequently updated, new apps are released every day, and app availability may change very often, so this study can only provide an overview of the current situation.

Second, our research objectives were app stores in China. Due to regional differences in developers’ decisions on app availability, this may not accurately represent the overall functionality of apps presented in other regions. For example, the app Blued from a Chinese developer included in this study is only released in China; Grindr, the LGBTQ social app with the largest number of users in the world, has been delisted from the China and Saudi Arabia app markets due to policy reasons.⁴⁰ It is worth mentioning that some apps, such as Sniffies, did not choose to be listed on the mobile app market in order to avoid being supervised by the app market, and users can only access them through website applications.⁴¹

Third, only free apps and nonpaid feature experiences were included in the research, and no assessment was conducted on apps with paid features, which may limit the scope of the study.

Fourth, the quality of the apps was evaluated using a user assessment tool, the uMARS scale, which could be different from results obtained MARS scale. Finally, each app is required to be used by the evaluator for at least 10 min, whereas for the apps with more functions, if the evaluator evaluates the application in the shortest time period, it may not be enough to fully assess.

Future research should consider (a) incorporating a broader range of applications, including those distributed through nonmarketplace channels such as web apps and mini-programs, to provide a more comprehensive evaluation of the MSM app ecosystem; (b) expanding the inclusion criteria to cover both free and premium features for a fuller aspect of available services and their impact on users; (c) employing multiple assessment tools for comprehensive quality measurements; and (d) recruiting evaluators with diverse linguistic backgrounds to ensure accurate assessment of multilingual applications. Such approaches would offer a more complete understanding of the evolving MSM dating app ecosystem and its potential for sexual health promotion.

Conclusions

This research revealed a diversity of dating apps developed for the MSM community. The quality of these apps was evaluated using the uMARS scale, which demonstrated that the overall quality was satisfactory. There was no correlation between the quality of the apps and their star rating, although there was a positive and weak correlation between the quality of apps and the number of reviews. Furthermore, in the analysis of sexual health-related information within these apps, it was found that over half of the apps contained such information. However, there were notable discrepancies in the quantity and content of the sexual health information provided among these apps.

Mobile apps have the potential to disseminate positive and accurate public health information, as well as disease prevention and health promotion, which represent a prominent opportunity for future development of mobile health. It is recommended that developers of MSM dating mobile apps should engage in more communication and collaboration with public health experts to optimize the utilization of these platforms for delivering sexual health-related information to target populations and paying more attention to services tailored to each region. Furthermore, developers can ensure the accuracy and usefulness of the sexual health-related information within the apps, thus providing more reliable sexual health guidance to users and improving overall user experience and satisfaction.

Supplemental Material

sj-docx-1-dhj-10.1177_20552076241308939 - Supplemental material for Quality assessment and sexual health information analysis of mobile dating applications for men who have sex with men: A comprehensive evaluation using the User Version of the Mobile App Rating Scale

Supplemental material, sj-docx-1-dhj-10.1177_20552076241308939 for Quality assessment and sexual health information analysis of mobile dating applications for men who have sex with men: A comprehensive evaluation using the User Version of the Mobile App Rating Scale by Xu Zhou and Wei-Yi Wu in DIGITAL HEALTH

Supplemental Material

sj-docx-2-dhj-10.1177_20552076241308939 - Supplemental material for Quality assessment and sexual health information analysis of mobile dating applications for men who have sex with men: A comprehensive evaluation using the User Version of the Mobile App Rating Scale

Supplemental material, sj-docx-2-dhj-10.1177_20552076241308939 for Quality assessment and sexual health information analysis of mobile dating applications for men who have sex with men: A comprehensive evaluation using the User Version of the Mobile App Rating Scale by Xu Zhou and Wei-Yi Wu in DIGITAL HEALTH

Supplemental Material

sj-docx-3-dhj-10.1177_20552076241308939 - Supplemental material for Quality assessment and sexual health information analysis of mobile dating applications for men who have sex with men: A comprehensive evaluation using the User Version of the Mobile App Rating Scale

Supplemental material, sj-docx-3-dhj-10.1177_20552076241308939 for Quality assessment and sexual health information analysis of mobile dating applications for men who have sex with men: A comprehensive evaluation using the User Version of the Mobile App Rating Scale by Xu Zhou and Wei-Yi Wu in DIGITAL HEALTH

Footnotes

Acknowledgments

We appreciated the evaluators who participated in this quality assessment. We would like to thank the reviewers for their valuable review comments.

Contributorship

Conceptualization: XZ and WY-W; methodology: XZ; formal analysis: XZ; resources: XZ; data curation: XZ; writing—original draft: XZ; writing—review and editing: XZ and WY-W. All authors have read and agreed to the published version of the manuscript.

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Ethical approval

This study only focuses on mobile applications and does not involve the collection of any personal information or data. Therefore, there are no ethical issues or controversies in this study.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

Guarantor

XZ.

ORCID iDs

Xu Zhou

Wei-Yi Wu

Supplemental material

for this article is available online.

References

Dominant players in the Chinese smartphone market - unveiling the powerhouses 2024. Czech Republic: GO-Globe, 2024. https://www.go-globe.com/dominant-players-in-the-chinese-smartphone-market/ (accessed 1 July 2024).

State of Mobile 2024 . data.ai, https://www.data.ai/en/go/state-of-mobile-2024/ (accessed 15 June 2024).

Bandinelli

Gandini

. Dating apps: the uncertainty of marketised love. Cult Sociol 2022; 16: 423–441.

Wang

. Spatiotemporal analysis of men who have sex with men in mainland China: social app capture-recapture method. JMIR Mhealth Uhealth 2020; 8: e14800.

Wei

Chen

Zhang

, et al. Relationship between gay app use and HIV testing among men who have sex with men in Shenzhen, China: a serial cross-sectional study. BMJ Open 2019; 9: e028933.

Trottier

. Constructing sexual fields: Chinese gay men’s dating practices among pluralized dating apps. Social Media + Society 2021; 7: 20563051211009014.

Cummings

. ‘Now you can see who’s around you’: negotiating and regulating gay intimacies on mobile media in the People’s Republic of China. In: Cabañes

JVA

Uy-Tioco

(eds) Mobile media and social intimacies in Asia: reconfiguring local ties and enacting global relationships. Dordrecht: Springer Netherlands, 2020, pp.15–30.

Ward

. Looking for “interesting people”: Chinese gay men’s exploration of relationship development on dating apps. Mob Media Commun 2020; 8: 342–359.

Tang

Best

Zhang

, et al. Gay mobile apps and the evolving virtual risk environment: a cross-sectional online survey among men who have sex with men in China. Sex Transm Infect 2016; 92: 508–514.

10.

Cao

Liu

Stein

, et al. Faster and riskier? Online context of sex seeking among men who have sex with men in China. Sex Transm Dis 2017; 44: 239–244.

11.

Clark

. Mobile dating apps could be driving HIV epidemic among adolescents in Asia Pacific, report says. Br Med J 2015; 351:h6493.

12.

Wang

Yang

PKH

, et al. Influence of social media on sexualized drug use and chemsex among Chinese men who have sex with men: observational prospective cohort study. J Med Internet Res 2020; 22: e17894.

13.

Liu

Qian

H-Z

Ruan

, et al. Alcohol use among Chinese men who have sex with men: an epidemiological survey and meta-analysis. Biomed Res Int 2014; 414381: 11.

14.

Miao

Chan

. Social constructivist account of the world’s largest gay social app: case study of Blued in China. Inf Society 2020; 36: 214–225.

15.

Wang

Safe on blued? A qualitative exploration of sex, risk, and stigma on a gay social application in China. Int J Sex Health 2023; 35: 596–607.

16.

Muessig

Bien

Wei

, et al. A mixed-methods study on the acceptability of using eHealth for HIV prevention and sexual health care among men who have sex with men in China. J Med Internet Res 2015; 17: e100.

17.

Cao

Gupta

Wang

, et al. Social media interventions to promote HIV testing, linkage, adherence, and retention: Systematic review and meta-analysis. J Med Internet Res 2017; 19: e394.

18.

Zhang

Reynolds

Farley

, et al. Preferences for an HIV prevention mobile phone app: a qualitative study among men who have sex with men in China. BMC Public Health 2019; 19: 297.

19.

Wohlfeiler

Hecht

Volk

, et al. How can we improve online HIV and STD prevention for men who have sex with men? Perspectives of hook-up website owners, website users, and HIV/STD directors. AIDS Behav 2013; 17: 3024–3033.

20.

Wohlfeiler

Hecht

Raymond

, et al. How can we improve HIV and STD Prevention Online for MSM?, https://www.ncsddc.org/wp-content/uploads/2017/08/how_can_we_improve_hiv_and_std_prevention_online_for_msm_full_report_0.pdf (accessed 12 June 2024).

21.

Yang

Long

Luo

, et al.

The characteristics and quality of mobile phone apps targeted at men who have sex with men in China: a window of opportunity for health information dissemination?

JMIR Mhealth Uhealth 2019; 7: e12573.

22.

Global smartphone sales share by operating system. Counterpoint. https://www.counterpointresearch.com/insights/global-smartphone-os-market-share/ (accessed 10 June 2024).

23.

Talkingdata Intelligence App Ranking . http://mi.talkingdata.com/app-rank.html?type=102000 (accessed 10 June 2024).

24.

Qimai Data . https://www.qimai.cn/ (accessed 12 June 2024).

25.

Ghose

Han

. Estimating demand for mobile applications in the new economy. Manage Sci 2014; 60: 1470–1488.

26.

Stoyanov

Hides

Kavanagh

, et al. Development and validation of the user version of the Mobile Application Rating Scale (uMARS). JMIR Mhealth Uhealth 2016; 4: e72.

27.

Hallgren

. Computing inter-rater reliability for observational data: an overview and tutorial. Tutor Quant Methods Psychol 2012; 8: 23–34.

28.

Bien

Best

Muessig

, et al. Gay apps for seeking sex partners in China: implications for MSM sexual health. AIDS Behav 2015; 19: 941–946.

29.

Miller

. “They’re the modern-day gay bar”: exploring the uses and gratifications of social networks for men who have sex with men. Comput Hum Behav 2015; 51: 476–482.

30.

Sun

Stowers

Miller

, et al. Acceptability and feasibility of using established geosocial and sexual networking mobile applications to promote HIV and STD testing among men who have sex with men. AIDS Behav 2015; 19: 543–552.

31.

Danlan Charity Foundation . https://www.danlan.org/project/l8nlq (accessed 5 July 2024).

32.

Hayes

Snow

. Privacy and security issues associated with mobile dating applications. In Proceedings of the Conference on Information Systems Applied Research 2018; 11: 4823.

33.

Silverberg

Eric

. Location security & privacy: an inside look. Jackd Support. https://support.jackd.com/portal/en/kb/articles/location-security-privacy-an-inside-look-23-12-2023 (accessed 23 June 2024).

34.

BLued Personal Information Protection Policy . https://activity.blued.cn/activityblued/pcview/UUIo7dhT?blued_mode=disable_bounce (accessed 29 June 2024).

35.

Kratz

Weiss

Ridpath

, et al. Community-based outbreak of Neisseria meningitidis serogroup C infection in men who have sex with men, New York City, New York, USA, 2010−2013. Emerg Infect Dis 2015; 21: 1379–1386.

36.

Chan

DPC

Sun

H-Y

Wong

HTH

, et al. Sexually acquired hepatitis C virus infection: a review. Int J Infect Dis 2016; 49: 47–58.

37.

NICE AF . https://niceaf.org/ (accessed 10 June 2024).

38.

Benkhelifa

Laallam

. Exploring demographic information in online social networks for improving content classification. J King Saud Univ 2020; 32: 1034–1044.

39.

Qin

Wang

Cheng

, et al. Demographic information prediction: a portrait of smartphone application users. IEEE Trans Emerg Top Comput 2018; 6: 432–444.

40.

Censored countries & regions . Help Center, https://help.grindr.com/hc/en-us/articles/1500010811581-Censored-countries-regions (accessed 3 July 2024).

41.

Sniffies Notifications are Here . Sniffies HUSH, https://sniffieshush.com/blogs/articles/sniffies-notifications-are-here (accessed 22 June 2024).

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.02 MB

0.03 MB

0.02 MB