Quality assessment of online osteosarcoma–related videos in Mainland China: A cross-sectional study

Abstract

Background

Information about orthopedics diseases on the Internet has not been extensively assessed. Our purpose was to evaluate the quality of online information of osteosarcoma on current video-sharing platforms in mainland China.

Method

TikTok and Bilibili were independently queried from June to July 2023 by four independent researchers using the Microsoft Edge web browser. Information about the videos and creators was recorded, and descriptive analyses were conducted.

Results

After data extraction, a total of 95 videos were included, in which 43 videos were uploaded by certified doctors (45.3%), with 35 videos (36.8%) uploaded by certified orthopedic surgeons. Of the content of these videos, 78.9% were introduction (n = 75), 64.2% were on professional knowledge (n = 61), 28.4% were on treatment (n = 27), while 5.3% were on surgical techniques (n = 5). The mean DISCERN total score was 43.8 ± 13.4, and the mean JAMA score was 3.8 ± 0.3.

Conclusions

Videos about osteosarcoma on current video-sharing platforms were extensive, but were not comprehensive and professional. Although current online videos have the potential to improve public awareness on osteosarcoma, due to their quality and content, were not assessed to be good sources for medical education.

Keywords

internet orthopedics osteosarcoma TikTok video platform level II cross-sectional study

Introduction

Osteosarcoma is the most common primary bone sarcomas, with an incidence of 0.3/100,000/year.¹ Children and Adolescents had the highest incidence rate of osteosarcoma (0.8–1.1/100,000/year at age 15–19 years), in which most patients should be considered candidates for limb salvage.² Early detection and diagnosis of osteosarcoma can avoid tumor metastasis and improve survival rate.³ Despite of the rapid development of diagnostic and therapeutic techniques, osteosarcoma thus remains a challenging diagnosis for patients and their families. There is a potential for informational and psychological support materials online.⁴ However, public awareness of osteosarcoma is still poor, which leads to inaccuracy in identification and delays in early diagnosis and treatment. While medical information on video-sharing platforms has great potential on guiding patients and educating medical students, inaccurate and even misleading content may have secondary effects on harming the physical and mental health of the public and the healthy development of the online healthcare industry.⁵ As patients continue to turn to the Internet for health information and use this information in decision-making related to their health, it is important that the information presented online should be accurate and thorough.

Up to June 2022, the user size of online videos in mainland China had reached 995 million.⁶ In this rising digital age, abundant content of video-sharing platforms has brought great convenience to people’s lives. With the rapid development of the Internet, the way to acquire medical information and knowledge is no longer limited to traditional approaches such as newspapers, television, and offline teaching. Instead, people are increasingly accustomed to acquiring medical information and advise on video-sharing platforms.⁷ In Hong Kong, China, about 78.19% adult urban patients had ever searched medical information online, 36.81% of online medical information seekers searched information before the medical visit, 8.65% searched information after the visit.⁸ As one of the most popular video-sharing platforms, TikTok (named as “DouYin” in China mainland: ByteDance, Beijing, China) has more than 500 million daily users in mainland China.⁹ In 2022, more than 35,000 certified doctors uploaded about 4.43 million popular science videos on TikTok.¹⁰ Meanwhile, Bilibili (https://www.bilibili.com) is also popular and essential channels for health information dissemination in China, with user-friendly experiences on computers, tablets, and smart mobile phones. Although YouTube is very popular globally, but in China, some commercial disputes and copyright issues related to YouTube have led the Chinese government to prohibit Chinese netizens from accessing the Web site, which is the reason why YouTube was not included in this study.

However, since video-sharing platforms always focus more on entertainment rather than education and professional knowledge, current research suggested that medical content on video-sharing platforms is variable and not comprehensive, with questionable authenticity and credibility.¹¹ It is necessary to evaluate the quality of medical videos available on video-sharing platforms. Previous research on the evaluation of popular science videos highlighted the need for more targeted and widespread professional knowledge on the Internet to direct prospective patients to a surgeon.¹² Since the proportion of doctors and professional knowledge can reflect the professionalism of online videos, the popularity is one of the most important criteria for evaluating the quality of popular science videos as well. Most previous researches have used indicators such as the number of views, likes, and comments in the quality assessment. The online time of videos can reflect whether those recommended videos are the latest updated and to some extent, the updating activity of the platform’s creators.

As many scholars have conducted multidimensional discussions on the current dissemination of medical videos, there is still limited research focusing on osteosarcoma. Therefore, the purpose of this study was to assess the current audience engagement and quality of videos related to osteosarcoma on the most popular video-sharing platforms, TikTok and Bilibili, in mainland China.

Materials and methods

We conducted a cross-sectional study on Tiktok Chinese version and Bilibili. The search terms was “骨肉瘤” (osteosarcoma in Chinese). All searches were independently queried for the keyword by using the Microsoft Edge web browser. Researchers performed independent searches from June to July 2023. “Incognito” or “Privacy” browser modes were employed to prevent any influence of cache or cookies on the next search. After each search process, the Internet cookies in web browser and computer were completely cleared.

The total number of videos on each video-sharing platform were recorded. Four different authors (two on TikTok and two on Bilibili) separately searched and recorded the data of their first 40 search results.

Exclusion criteria: (1) duplicated videos, (2) not in Chinese language, (3) only with picture and without any presentation or explanation.

After excluding those videos which did not meet our criteria for analysis, a total of 95 videos (44 on TikTok and 51 on Bilibili) were included and analyzed. Uploaders were divided into four categories: certified doctor, certified orthopedic surgeon, industry/company, and other individuals. The number of videos by certified doctors, of certified orthopedic surgeons, of industry/company and of other individuals were recorded and assessed. The following video features were assessed: total video duration, number of months online, number of “likes”, number of comments, and number of views. Each single video was reviewed to determine if broad-based disease-related content was present. We classified the video content into six categories: (1) Instruction: personal stories of patients, disease introduction, disease risk factors, and simple introduction of treatment; and videos uploaded by non-certified orthopedic surgeons were all classified into this category, (2) Diagnosis: complete and comprehensive diagnostic points, including etiology, pathological morphology, pathophysiology and detailed classification; (3) Imaging: reviewing and diagnosis of imaging examinations, (4) Treatment: medication, conservative management, surgical selection, etc., (5) Surgical techniques, (6) Prognosis: rehabilitation, clinical efficacy, postoperative precautions. Descriptive statistics were used to evaluate the results of the research. All analyses were performed using Microsoft Excel.

As for quality assessment, DISCERN, JAMA Benchmark Criteria were conducted. DISCERN score is a practical tool to evaluate the reliability of online videos.²⁰ Meanwhile, the mean of JAMA scores can be used to evaluate the general quality of online videos.²⁵ DISCERN consisted of 16 questions in total, with each question scored from 1 to 5 points. Questions were divided into three sections: reliability (questions 1–8), quality information about treatment options (questions 9–15), and overall score (question 16). The total DISCERN score was calculated by summing up scores over questions 1–15. All videos were divided into five categories based on their total DISCERN score: very poor (<27), poor (27–36), fair (37–50), good (51–62), and excellent (63–75). Journal of the American Medical Association (JAMA) benchmark criteria were used to evaluate online health information reliability. This included four criteria: authorship, attribution, disclosure, and currency. Each satisfied criterion counted 1 point, with a maximal score of 4 points. To avoid bias, all 4 researchers in our study conducted the scoring independently, blinded to each other’s score, and the average score was taken.

Since this study focused on the quality assessment of videos contributed and viewed by the public on social media platforms, no ethics committee approval was required.

Statistical analysis

Descriptive statistics were calculated for the demographic data. A paired t-test was used to analyse within-group scores changes. RM-ANOVAs were used to assess the relationship between the variables, with a significance level of p < .05. Analyses were performed using SPSS software version 23.0.

Result

A total of 653 search results (304 on TikTok and 353 on Bilibili) were included by searching the keyword “骨肉瘤”. After data extraction, a total of 95 videos (44 on TikTok and 51 on Bilibili) were included for analysis (See Figure 1). The general features of 95 videos included were recorded in Table 1. Among them, 43 videos were uploaded by certified doctors (45.3%), with 35 videos (36.8%) uploaded by certified orthopedic surgeons. Of these videos, 78.9% contain content of introduction (n = 75), while professional knowledge accounted for 64.2% (n = 61). (See Figure 2). Contents of treatment had 28.4% (n = 27), while surgical techniques had the least proportion at 5.3% (n = 5).

Figure 1.

Selection process implemented in this study.

Table 1.

General features of search results on TikTok (Mean ± SD).

	TikTok (n = 44)		Bilibili (n = 51)		p value
	Mean	SD	Mean	SD	p value
Number of likes	13,857.0	41,037.7	6718.6	32,862.1	>.05
Number of comments	978.2	2952.7	478.4	2802.9	>.05
Number of views	—	—	35,957.5	140,934.5	—
Duration (s)	114.3*	130.8	504.4	708.0	<.001
Number of months online	8.4*	9.0	14.6	15.0	<.001
DISCERN score	44.6*	13.4	50.4	6.3	<.001

*suggest a significant difference with Bilibil (p < .001).

Figure 2.

Detailed results of broad-based disease-related content of two platforms (%).

Audience engagement and broad-based disease-related content

The proportion of certified doctors and orthopedic surgeons on Tiktok was significantly higher than that on Bilibili (p < .001) (See Figure 3). The number of likes and comments of the videos on Bilibili was slightly higher than those of Tiktok, though there was no statistical difference between them (p > .05). The mean duration and online time of Tiktok videos were significantly shorter than Bilibili (p < .001). (See Table 2).

Figure 3.

Proportion of certified doctors, orthopedic surgeons, industry/company, and other individuals of two platforms (%).

Table 2.

Quality assessments according to video source and identities.

	Reliability (questions 1–8)		Quality information about treatment options (questions 9–15)		Overall score (question 16)		JAMA benchmark criteria
	Mean	SD	Mean	SD	Mean	SD	Mean	SD
Certified doctor (n = 43)	28.6	6.1	16.7	4.2	3.1	0.9	2.5	1.2
Orthopedic surgeon (n = 35)	28.8	6.4	16.7	4.3	3.0	0.9	2.5	1.2
Industry/company (n = 13)	29.1	9.4	15.2	6.8	2.9	1.1	2.5	1.1
Other individuals (n = 26)	28.9	8.0	14.6	4.7	3.2	0.9	2.8	1.2

Evaluation outcomes of DISCERN classification and JAMA benchmark

The mean DISCERN total score was 43.8 ± 13.4, and the mean JAMA score was 3.8 ± 0.3. The DISCERN classification data showed that among all videos involved, 5.4% were “very poor”, 6.3% were “poor”, 46.3% were “fair”, 38.9% were “good”, and only 3.2% were “excellent”. DISCERN scores and JAMA scores had no significant correlation with the video source and uploader (see Table 2). In other words, videos uploaded by different video-sharing platforms, certified orthopedic surgeons or non-orthopedic individual/groups were not superior in quality assessment.

Discussion

This is the first study to evaluate and to compare the reliability and quality of osteosarcoma related videos on video-sharing platforms in mainland China. We retrieved a total of 653 results from two video-sharing platforms (304 from TikTok and 349 from Bilibili). All videos included were open and free to the public. Current research suggested that websites on the first search results page receive almost 95% of web traffic, leaving only 5% for remaining search results pages.¹³ Our results confirmed the importance of the first-page results after retrieval. For the general public, the high proportion of introduction (78.9%, n = 75) on Tiktok and Bilibili about osteosarcoma can play a role at guiding them to discover abnormal symptoms and seek medical attention promptly. When evaluating the professional knowledge (64.2%, n = 61), the content on diagnosis (TikTok: 29.4%; Bilibili: 27.3%) and treatment (TikTok: 25.5%; Bilibili: 25%) accounted the largest proportion, while the content on imaging (TikTok: 21.2%; Bilibili: 9.1%) and surgical technique (TikTok: 2%; Bilibili: 9.1%) was relatively limited. Some studies suggested that medical students could learn professional knowledge on video-sharing platforms.¹⁴ In our opinion, the professional orthopedic knowledge and information on current video-sharing platforms is still far from comparable to professional books and literature.

Zhang et al suggested that the most medical information on TikTok were low-quality content and could be easily accessible.¹⁵ In our study, the “good” (38.9%) and “excellent” (3.2%) DISCERN classification account for a small portion of all searched results. In our research, the mean DISCERN values of included results was 43.8, with an average JAMA scores of 2.7, indicating a relatively intermediate quality and reliability. Many current studies were consistent this situation as well. Abey et al examined YouTube videos using same instruments and found modified DISCERN, and JAMA scores were 2 (IQR: 2-2), and 2 (IQR: 2-2), respectively, suggesting an intermediate quality.¹⁶ Kunze et al examined YouTube videos and found mean JAMA score of 1.55.¹⁷ The proportion of certified doctors (45.3%) and orthopedic surgeons (36.8%) were unsatisfactory. Though our results showed taht the DISCERN scores and JAMA scores had no significant correlation with the type of video uploaders (p > .05), sufficient certified doctors and orthopedic surgeons may provide in-time online feedback on professional questions from patients and general public. Meanwhile, the identity of certified doctors, especially orthopedic surgeons, can guide patients to correctly choose hospitals and departments. We suggested that orthopedic surgeons should not only actively participate in creating popular science and profession-related videos, but also make effort to improve the quality of online videos. Meanwhile, some studies showed that videos released by medical specialists and institutions did not receive more public attention and higher quality evaluation rates than those from other sources,¹⁸ which dampened the doctor’s enthusiasm of creating and uploading. The low proportion of of videos produced by certified doctors could result in insufficient professional quality, even increasing the risk of erroneous or misleading information.¹⁹ Perhaps those platforms should prioritize the recommendation of videos uploaded by certified doctors when the searched term is found to be medical related.

The online time of Tiktok videos are significantly shorter than Bilibili (p < .001), which may indicate that Tiktok is more likely to be limited in pushing the latest updated videos to users. Meanwhile, the mean duration of Tiktok videos are significantly shorter than Bilibili (p < .001). Current study suggested that shorter video content is more likely to arouse user interest.²⁰ However, a short video may not fully contain knowledge of all aspects of a disease. To achieve the effect of medical education, we suggest that orthopedic doctors should further improve the comprehensiveness of their videos. The ranking algorithm of video-sharing platforms not only considers the quality of the videos, but also more considers the likes, comments, reposts, and completion rates.²¹ Video content of current platforms is mostly entertainment oriented. However, for medical videos, accuracy and professionalism are more important than entertainment and popularity. Compared with other types of videos, medical videos have low views, likes, and comments, which is a problem that would need to be considered and solved when creating videos in the future.

Telemedicine and online consultation have currently been widely accepted and have received good social feedback due to the rapid development and advancement of the Internet.²² The Internet has become an important way for patients and the general public to obtain disease-related and treatment-related information because it can be accessed conveniently and is free of charge.²³ If the advantages of the Internet can be utilized to improve public awareness of osteosarcoma, the efficiency of early detection and diagnosis could be effectively improved, which would be beneficial for those afflicted with osteosarcoma. The user size of live streaming in China had reached 716 million in 2022, accounting for 68.1% of all Internet users.²⁴ As one of the most popular and populous live stream platforms, TikTok allows orthopedic surgeons and medical specialties to directly interact with patients and general public. This function has also greatly promoted the popularity of medical science popularization on this platform.

Limitation

There are several limitations to this investigation. First, we only analyzed the disease-related content of those videos, and a more detailed analysis of the video content is needed; secondly, different users could have different search results due to their different daily preference. Moreover, this study was a short-term effect study and data from a long-term follow-up survey were lacking. Therefore, more researches with larger sample sizes will be needed in the future.

Conclusion

This study showed that videos about osteosarcoma on TikTok were extensive but not comprehensive and professional. Current online videos could effectively improve public awareness on osteosarcoma, but were not of sufficient quality to be useful for medical education. Meanwhile, the proportion of videos produced by certified orthopedic surgeons and those covering professional knowledge were low. Regulations on the management and screening of these videos need to be improved their quality and attract more orthopedic surgeons to participate in online video creation.

Footnotes

Acknowledgements

We thank the authors of the included studies for their help.

Author contributions

C.C completed the manuscript writing, W.L, G.C X.Z and Y.Y were responsible for videos searching, X.Z and D.Z collected and analyzed the data. All authors reviewed the final article.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Ethical statement

ORCID iD

Chang Chen

Data availability statement

The datasets used and/or analysed during the current study available on supplementary material from the corresponding author on reasonable request.

References

Cole

Gianferante

Zhu

, et al. Osteosarcoma: a surveillance, epidemiology, and end results program-based analysis from 1975 to 2017. Cancer-Am Cancer Soc. 2022; 5: 128.

Jakob

Andreou

Bedke

, et al. Ten recommendations for sarcoma surgery: consensus of the surgical societies based on the German S3 guideline “adult soft tissue sarcomas”. LANGENBECK ARCH SURG 2023; 408: 272.

Wang

, et al. Guideline for limb-salvage treatment of osteosarcoma. Orthop Surg 2020; 12: 1021–1029.

Lam

Roter

Cohen

. Survey of quality, readability, and social reach of websites on osteosarcoma in adolescents. Patient Educ Couns 2013; 90: 82–87.

Bahar-Ozdemir

Ozsoy-Unubol

Akyuz

. Is YouTube a high-quality source of information on cancer rehabilitation? J Cancer Surviv 2022; 16: 1016–1022.

China Internet Network Information Center (CNNIC) . (2022). The 50th statistical report on China’s Internet development. Beijing: China Internet Network Information Center (CNNIC). Available at: https://P020221209344717199824.pdf_(cnnic.com.cn) (Accessed September 1, 2022).

Bethell

Anastasio

Taylor

, et al. Evaluating the distribution, quality, and educational value of videos related to shoulder instability exercises on the social media platform TikTok. J Am Acad Orthop Surg Glob Res Rev 2023; 7: e23.00034.

Wong

Cheung

. Online health information seeking and eHealth literacy among patients attending a primary care clinic in Hong Kong: a cross-sectional survey. J Med Internet Res 2019; 21: e10831.

Song

Xue

Zhao

, et al. Short-video apps as a health information source for chronic obstructive pulmonary disease: information quality assessment of TikTok videos. J Med Internet Res 2021; 23: e28318.

10.

Iqbal

. TikTok revenue and usage statistics. 2023. Available at: https://www.businessofapps.com/data/tik-tok-statistics/

11.

Phair

Dalmia

Sanon

, et al. The current state of vascular surgery presence and educational content in Google and YouTube internet search results. J Vasc Surg 2021; 74: 616.e6–624.e6.

12.

Ferhatoglu

Kartal

Ekici

, et al. Evaluation of the reliability, utility, and quality of the information in sleeve gastrectomy videos shared on open access video sharing platform YouTube. Obes Surg 2019; 29: 1477–1484.

13.

Kong

Song

Zhao

, et al. TikTok as a health information source: assessment of the quality of information in diabetes-related videos. J Med Internet Res 2021; 23: e30409.

14.

Fraticelli

Smentek

Tardivo

, et al. Characterizing the content related to oral health education on TikTok. Int J Environ Res Publ Health 2021; 18: 13260.

15.

Zhang

Feng

Peng

, et al. Using structural equation modeling to examine pathways between physical activity and sleep quality among Chinese TikTok users. Int J Environ Res Publ Health 2022; 19: 5142.

16.

Abed

Ray

Smathers

, et al. Assessment of video quality and reliability of YouTube videos regarding meniscus tear rehabilitation. Cureus 2023; 15: e36299.

17.

Kunze

Krivicich

Verma

, et al. Quality of online video resources concerning patient education for the meniscus: a YouTube-based quality-control study. Arthroscopy 2020; 36: 233–238.

18.

Yilmaz Ferhatoglu

Kudsioglu

. Evaluation of the reliability, utility, and quality of the information in cardiopulmonary resuscitation videos shared on Open access video sharing platform YouTube. Australas Emerg Care 2020; 23: 211–216.

19.

Xiong

Luo

Zhou

, et al. A profile of patients’ and doctors’ perceptions, acceptance, and utilization of e-health in a deprived region in southwestern China. PLOS Digit Health 2023; 2: e0000238.

20.

Sun

Zheng

. Quality of information in gallstone disease videos on TikTok: cross-sectional study. J Med Internet Res 2023; 25: e39162.

21.

Zhang

Han

Tang

, et al. Popular science and education of cosmetic surgery in China: quality and reliability evaluation of Douyin short videos. Health Expect. 2023; 26: 1221–1226.

22.

Kruse

Molina-Nava

Kapoor

, et al. Analyzing the effect of telemedicine on domains of quality through facilitators and barriers to adoption: systematic review. J Med Internet Res 2023; 25: e43601.

23.

Ping

, et al. TikTok browsing for anxiety relief in the preoperative period: a randomized clinical trial. Complement Ther Med 2021; 60: 102749.

24.

Jia

Pang

Huang

, et al. Understanding consumers’ continuance intention to watch streams: a value-based continuance intention model. Front Psychol 2023; 14: 1073301.

25.

Akpolat

Kurdal

. Is quality of YouTube content on Bankart lesion and its surgical treatment adequate? J Orthop Surg Res 2020; 15: 78.