Analysis of P Values in the Abstract Compared With the Main Text of Randomized Controlled Trials and Clinical Trials of Mesenchymal Stromal Cells for the Treatment of Knee Osteoarthritis

Abstract

Background:

Randomized controlled trials (RCTs) and clinical trials (CTs) are critical for evaluating treatment effectiveness, with abstracts serving as the primary source of the study that is read.

Purpose:

This study explored selective reporting of significant P values in the abstracts of RCTs and CTs on mesenchymal stromal cells (MSCs) for the treatment of knee osteoarthritis (OA). The purpose was to compare the rates of significant P values reported in the abstract compared with the main text of RCTs and CTs on MSCs for the treatment of knee OA.

Study Design:

Systematic review.

Methods:

PubMed (MEDLINE) and Embase searches were conducted using related terms and were filtered for RCTs and CTs. There were 2 reviewers who analyzed the abstract and main text independently and extracted study characteristics and every P value reported in the abstract and main text of each article. The distribution of significant P values between the abstract and main text was compared using a paired-samples t test and odds ratios. Significance was set at P < .05.

Results:

The initial search yielded 156 articles, with 61 meeting the inclusion criteria. Our analysis showed a significant discrepancy in the distribution of P values between the abstract and main text. In abstracts, 79.3% of P values were significant compared with 53.8% in main texts. The odds of significant P values being reported in the abstract over the main text were more than 3 times higher (odds ratio, 3.3). The paired-samples t test confirmed that abstracts were significantly more likely to report significant findings (P = 7.5 × 10⁻⁸). Primary outcomes were reported in 90.2% of abstracts, with 18.0% being significant, 47.5% being not significant, and 34.4% being not applicable because of qualitative results. Functional and quality of life scores were the most common primary outcomes in 34 studies, followed by safety and adverse events in 25 studies, pain measurements in 20 studies, and imaging outcomes in 17 studies.

Conclusion:

Our study highlights a significant risk of reporting bias in RCTs and CTs of MSCs for the treatment of knee OA. With growing interest in the use of MSCs, it remains crucial for busy clinicians to recognize the possibility of selective reporting bias in abstracts, as this could potentially influence clinical decision making.

Keywords

MSCs knee osteoarthritis abstract bias

Knee osteoarthritis (OA) is a common condition involving degeneration of the joint with cartilage loss and bony changes.³³ Mesenchymal stromal cells (MSCs; also known as mesenchymal stem cells) can differentiate into mesodermal lineages such as osteocytes and chondrocytes. The use of MSCs for the treatment of musculoskeletal conditions has garnered significant attention for their pluripotent potential, including increased exploration of their clinical utility in treating knee OA.^21,30,32 The burgeoning interest is evidenced by accelerating rates of research on biologics for cartilage repair and OA treatment, with rates increasing by 16.4% annually from 2006 to 2021.³⁵ This has not only sparked interest in the medical community but has also gained the attention of patients who seek clinical consultations and conduct online searches for stem cell injections to treat hip and knee OA.^3,31 Despite increased public and research interest on the topic, systematic reviews of biologics for the treatment of OA have found that studies are at risk of bias, with small numbers of patients and the lack of a control and randomization.^1,13

Randomized controlled trials (RCTs) and clinical trials (CTs) are the gold standard for evaluating the effectiveness of treatment methods or interventions and are paramount for making evidence-based medical treatment decisions.^4,16,18 Abstracts provide a summary of study findings and are essential for efficiently conveying critical results.¹⁹ Beyond article titles, abstracts are the main and frequently sole section of an article that is read, highlighting the critical findings of studies; therefore, accurate and complete reporting is vital as clinicians and researchers often make important judgments based only on information from abstracts.^5,18,24,25 Previous studies have investigated the bias and quality of data reported in abstracts. Spin bias,³⁴ or reporting that could alter the interpretation of results and misguide readers, has been examined in several fields, including orthopaedics.⁷ Studies investigating spin in systematic reviews and meta-analyses of treatment for lumbar disc herniation found spin in 29.4% of abstracts, another found spin in 35.4% of abstracts for the surgical treatment of knee OA, and finally, spin was found in 65.1% of abstracts on treatment for Achilles tendon ruptures.^2,8,29 Selective reporting was the most common form of spin found by the latter 2 studies.^8,29 Selective reporting, a form of bias in which only the most favorable results are highlighted, is often observed in abstracts, potentially misrepresenting the full findings of the study. This issue is particularly concerning in knee OA research. To address this, Assem et al⁴ explored a way to identify selective reporting in abstracts by comparing the proportion of significant P values in the abstract with those in the main text. To our knowledge, no study has investigated selective reporting in the abstracts of RCTs and CTs on MSCs for the treatment of knee OA. This study aimed to compare the rates of significant P values reported in the abstract compared with the main text of these studies and to identify trends in reported outcomes. We hypothesized that the abstracts of RCTs and CTs involving MSCs for the treatment of knee OA would show a higher proportion of significant P values compared with the corresponding main text. This may suggest the presence of selective reporting bias in which more significant results are highlighted in the abstracts than in main texts. This analysis addresses a gap in the literature by focusing on MSC-based interventions for knee OA, a rapidly expanding area of research in which abstract-level reporting may influence how findings are interpreted and communicated.

Methods

Article Selection

Our study utilized a systematic review approach, involving a comprehensive literature search, study selection, data extraction, and qualitative synthesis following PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses)²³ guidelines (Figure 1). This study examined RCTs and CTs of knee OA treated with MSCs for selective reporting of significant P values. PubMed (MEDLINE) and Embase searches were conducted on July 23, 2023, using the terms “mesenchymal stem cells” or “mesenchymal stromal cells” or “MSCs” and “knee arthritis” or “osteoarthritis” and “therapy” or “treatment” or “regeneration.” Studies were included if they were a prospective RCT or CT involving human participants, MSCs, and primary knee OA and if the full article was available in English. Studies were uploaded to a datasheet, and duplicates were removed. All articles were screened independently for inclusion by 2 reviewers (N.M. and K.W.). Both reviewers agreed on the inclusion or exclusion of studies.

Figure 1.

Flow diagram.

Data Extraction

The abstract and main text of studies meeting the inclusion criteria were analyzed independently (N.M. and K.W.), and data collected included the following: number of significant P values in the abstract and text, total number of reported P values in the abstract and text, journal/book, publication year, intervention, control, primary outcomes, secondary outcomes, impact factor, and financial disclosures. Statistical significance was determined by the article and reported P value, with significance at P < .05. Any outcomes without data for statistical significance were not included in the analysis. The measured outcomes were further categorized as follows.

Pain measurements: This included the Numeric Pain Rating Scale, visual analog scale, Western Ontario and McMaster Universities Osteoarthritis Index (WOMAC) pain subscale, Knee Injury and Osteoarthritis Outcome Score (KOOS) pain subscale, and other specific pain measures such as pain on walking and number of stairs until pain appears.

Functional and quality of life scores: This encompassed the KOOS (excluding the pain subscale), WOMAC (excluding the pain subscale), Lequesne algofunctional index, 12-item Short Form Health Survey, International Knee Documentation Committee form, Lysholm score, Oxford Knee Score, and Numeric Rating Scale–11.

Imaging outcomes: This included magnetic resonance imaging (MRI), radiography, and specific MRI scores such as the Whole-Organ MRI Score, T2 values for cartilage morphology and collagen content, serial changes in the cartilage defect, International Cartilage Repair Society grade, Magnetic Resonance Observation of Cartilage Repair Tissue, hip-knee-ankle angle, and Kellgren-Lawrence grade.

Safety and adverse events: This category included adverse events, severe adverse events, tolerability, and specific safety assessments such as laboratory tests, vital signs, physical examinations, and specific criteria such as the Common Terminology Criteria for Adverse Events (Version 4.0).

Specific clinical scores and indices: This encompassed the Knee Society Score, need for total hip replacement, and clinical failure rate.

Miscellaneous outcomes: This included other specific outcomes mentioned such as range of motion, patellar crepitus, presence of synovial fluid, and bone marrow aspirate concentrate characterization.

Coding was performed independently by the 2 reviewers (N.M. and K.W.), who alternated codes on half of the studies, and flagged audits were performed throughout the review process by the same 2 coders to ensure that assessments were reproducible. A third reviewer (B.T.F.) performed random audits of 10 studies (each coded by N.M. and K.W.), and finally, one of the reviewers (K.W.) then appraised the entire database.

Statistical Analysis

We conducted a statistical analysis to compare the distribution of statistically significant P values (P < .05) between abstracts and main texts. A paired-samples t test was used for this comparison. Additionally, odds ratios (ORs) were calculated to assess the likelihood of encountering significant P values in the abstract versus main text. Assem et al⁴ provided a methodological reference for identifying selective reporting in abstracts. Their approach involved calculating the OR to analyze the association between significant P values and what was reported in the abstract. This comparison allows us to assess the presence of reporting bias in surgical RCTs. Descriptive statistics were utilized to summarize the frequency of primary outcome categories reported in the reviewed studies. All analyses were conducted using Python (Version 3.7), with statistical significance set at P < .05.

Results

The initial search returned 156 articles, with 61 meeting inclusion criteria. Our analysis of the distribution of statistically significant P values between the abstract and main text of articles revealed a significant discrepancy. A total of 213 P values were reported in the abstract, with 169 (79.3%) being significant and 44 (20.7%) being insignificant. In comparison, a total of 1935 P values were reported in the main text, with 1041 (53.8%) being significant and 894 (46.2%) being insignificant (Figure 2). OR analysis showed that the abstract was over 3 times more likely to report significant P values compared with the main text (OR, 3.299). The paired-samples t test comparing the distribution of significant and nonsignificant P values in abstracts and main texts highlighted the differences in reporting, with abstracts significantly more likely to report significant findings (P = 7.47 × 10⁻⁸).

Figure 2.

The frequency of significant (P < .05) and insignificant (P > .05) P values reported in (1) abstracts versus (2) main texts.

Primary outcomes were reported in 55 (90.2%) abstracts and not reported in 6 (9.8%) abstracts. Among the studies with reported primary outcomes, 40 were registered, and 15 were not. Among the articles analyzed, 11 (18.0%) reported significant primary outcomes, 29 (47.5%) found their primary outcomes to be not significant, and 21 (34.4%) were categorized as “not applicable” because of the primary outcomes being qualitative and without statistical analysis such as safety and adverse events. Functional and quality of life scores were the most reported primary outcomes in 34 studies, followed by safety and adverse events in 25, pain measurements in 20, and imaging outcomes in 17 (Figure 3 and Table 1). The mean abstract word count was 285 words, and there was no association between the abstract word count and the proportion of significant P values (P > .05).

Figure 3.

Frequency of primary outcomes in abstracts.

Table 1

Primary Outcomes in Abstracts

	n (%)
Pain measurements	20 (19.61)
Functional and quality of life scores	34 (33.33)
Imaging outcomes	17 (16.67)
Safety and adverse events	25 (24.51)
Specific clinical scores and indices	2 (1.96)
Miscellaneous outcomes	4 (3.92)

Discussion

Abstracts were over 3 times more likely to report significant P values compared with main texts. Additionally, our investigation into primary outcomes revealed that while 90.2% of abstracts reported primary outcomes, they were absent in 9.8% of them. Our research underscores a notable inconsistency in how P values are reported between the abstract and main text of RCTs and CTs focused on MSC therapy for knee OA. This discrepancy could potentially misguide readers who primarily consult abstracts for rapid evaluations of the literature, thereby fostering a biased understanding of the treatment's efficacy.

Because of the limited space in abstracts, authors might prioritize primary outcomes, which are more likely to show significance, or focus on presenting the most clinically relevant findings that could influence practice. Thus, secondary outcomes and nonsignificant results may be underreported or omitted, potentially leading to an incomplete understanding of the study's overall findings and their broader implications. A review revealed that CTs with positive findings were 3.9 times more likely to be published than those with negative or nonsignificant results.¹⁷

Lehmen et al²⁰ identified substantial discrepancies between abstracts and main texts, noting inconsistencies in 75% of the lumbar spine surgery studies that they reviewed. Notably, only 60% of the abstracts reported relevant statistically significant results. Their analysis revealed that important details, such as methods of randomization and primary outcomes, were often missing or poorly described in abstracts.^9-11,20 Pocock et al²⁷ pointed out that CTs often include vast amounts of data, making it difficult to interpret results accurately, especially with excessive significance testing. They suggested a more transparent approach in which primary treatment comparisons are predefined and greater emphasis is placed on the actual size of treatment effects and the use of confidence intervals rather than arbitrary significance thresholds.²⁷ Confidence intervals are better because they provide a range of values within which the true effect size is likely to lie, offering more information about the precision and reliability of the estimated effect. This approach avoids the binary interpretation of results as simply “significant” or “not significant,” which can be misleading. P values in observational research can be especially deceptive and should not be interpreted as probabilities. This core problem likely explains why P values in cohort studies tend to be the most extreme, as data from large cohorts are frequently published repeatedly.^15,26

Selective reporting, as evidenced in this study, can significantly affect clinical decision making. Specifically, abstracts were found to contain a higher proportion of significant P values compared with their corresponding main texts. This is similar to what Rongen and Hannink²⁸ discovered when they examined the consistency between registered and published primary outcomes in orthopaedic surgical RCTs. They found discrepancies in 53.8% of the trials, usually favoring significant results. Such selective reporting undermines the validity of evidence-based medicine by increasing false-positive results and potentially misleading health care professionals and policymakers.^10,28 Such reporting may lead clinicians and researchers to develop a biased view of the effectiveness of MSC therapy for knee OA, affecting treatment recommendations and future research directions. This study underscores the need for careful and critical appraisal of published data, advocating for a thorough evaluation of the entire text rather than relying solely on the abstract. Research reporting general health measurements should carefully consider power calculations, as underpowered secondary outcomes may fail to reach statistical significance, despite potential clinical relevance. This can contribute to selective reporting in abstracts in which well-powered primary outcomes are emphasized while secondary outcomes remain underreported. Recognizing this issue helps to provide a more balanced interpretation of study findings. Numerous studies have warned against relying solely on abstracts, as it could mislead readers, particularly those who depend only on abstracts for quick assessments of the scientific literature, thereby perpetuating a biased interpretation of the treatment's efficacy.^6,14 Our findings emphasize the need for a careful and critical appraisal of published data. Utilizing the full medical literature for clinical decisions has been shown to be more effective.^12,22

Our study had several limitations. We did not evaluate the quality of the individual studies or compare journals based on their quality metrics. Comparing journals based on established quality indicators could provide more insight into the impact of journal quality on research findings. Future directions could involve the development of stricter guidelines for abstract reporting and the implementation of checks that ensure consistency across abstracts and main texts.

Conclusion

While abstracts are a convenient way to access scientific information, the current findings emphasize the risk of relying solely on them for understanding a study's full context. Alternatively, authors might aim to attract readers or journal editors by presenting an increased number of significant findings. With a substantial discrepancy observed in the prevalence of reported significant P values between the abstract and main text, our study accentuates the potential for abstracts to misrepresent research findings. This bias not only affects clinical judgment but also distorts the scientific narrative surrounding effective interventions for knee OA. Our findings advocate for enhanced editorial oversight and the encouragement of transparent reporting practices.

Footnotes

Final revision submitted April 16, 2024; accepted June 24, 2025.

One or more of the authors has declared the following potential conflict of interest or source of funding: B.T.F. has received support for research from the National Institutes of Health, the Department of Veterans Affairs, and Stryker; has received support for education from Evolution Surgical; and is an editorial board member of the Journal of Shoulder and Elbow Surgery. AOSSM checks author disclosures against the Open Payments Database (OPD). AOSSM has not conducted an independent investigation on the OPD and disclaims any liability or responsibility relating thereto. Ethical approval was not required for this systematic review as it analyzed data from previously published studies and did not involve human participants or animals.

ORCID iDs

Nesa Milan

Zubin Master

References

Abdelrazik

Mesenchymal stem cells: a hope or a hype?

Int J Mol Sci. 2023;24(17):13218.

Abu-Zahra

Mayfield

Thompson

, et al. Evaluation of spin in systematic reviews and meta-analyses of minimally invasive surgical techniques and standard microdiscectomies for treating lumbar disc herniation. Global Spine J. 2024;14(2):731-739.

Arthurs

Nordan

Hultgren

, et al. Patients seeking stem cell therapies: a prospective qualitative analysis from a regenerative medicine consult service. NPJ Regen Med. 2022;7:20.

Assem

Adie

Tang

Harris

IA.

The over-representation of significant P values in abstracts compared to corresponding full texts: a systematic review of surgical randomized trials. Contemp Clin Trials Commun. 2017;7:194-199.

Bavdekar

Gogtay

Writing an abstract for a research manuscript: providing an honest, succinct and complete summary. J Assoc Physicians India. 2015;63(12):64-67.

Boutron

Altman

Hopewell

Vera-Badillo

Tannock

Ravaud

Impact of spin in the abstracts of articles reporting results of randomized controlled trials in the field of cancer: the SPIIN randomized controlled trial. J Clin Oncol. 2014;32(36):4120-4126.

Boutron

Dutton

Ravaud

Altman

DG.

Reporting and interpretation of randomized controlled trials with statistically nonsignificant results for primary outcomes. JAMA. 2010;303(20):2058-2064.

Carr

Dye

Arthur

, et al. Evaluation of spin in the abstracts of systematic reviews and meta-analyses covering treatments for Achilles tendon ruptures. Foot Ankle Orthop. 2021;6(1):24730114211000637.

Chan

Altman

DG.

Identifying outcome reporting bias in randomised trials on PubMed: review of publications and survey of authors. BMJ. 2005;330(7494):753.

10.

Chan

Hróbjartsson

Haahr

Gøtzsche

Altman

DG.

Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. JAMA. 2004;291(20):2457-2465.

11.

Chan

Krleža-Jerić

Schmid

Altman

DG.

Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research. CMAJ. 2004;171(7):735-740.

12.

Crowley

Owens

Schardt

, et al. A web-based compendium of clinical questions and medical evidence to educate internal medicine residents. Acad Med. 2003;78(3):270-274.

13.

Delanois

Etcheson

Sodhi

, et al. Biologic therapies for the treatment of knee osteoarthritis. J Arthroplasty. 2019;34(4):801-813.

14.

Fontelo

Gavino

Sarmiento

RF.

Comparing data accuracy between structured abstracts and full-text journal articles: implications in their use for informing clinical decisions. BMJ Evid Based Med. 2013;18(6):207-211.

15.

Greenland

Randomization, statistics, and causal inference. Epidemiology. 1990;1(6):421-429.

16.

Hariton

Locascio

JJ.

Randomised controlled trials: the gold standard for effectiveness research. BJOG. 2018;125(13):1716.

17.

Hopewell

Loudon

Clarke

Oxman

Dickersin

Publication bias in clinical trials due to statistical significance or direction of trial results. Cochrane Database Syst Rev. 2009;2009(1):MR000006.

18.

Hua

Walsh

Glenny

Worthington

Structure formats of randomised controlled trial abstracts: a cross-sectional analysis of their current usage and association with methodology reporting. BMC Med Res Methodol. 2018;18:6.

19.

Kamel

El-Sobky

TA.

Reporting quality of abstracts and inconsistencies with full text articles in pediatric orthopedic publications. Res Integr Peer Rev. 2023;8:11.

20.

Lehmen

Deering

Simpson

Carrier

Bono

CM.

Inconsistencies between abstracts and manuscripts in published studies about lumbar spine surgery. Spine. 2014;39(10):841-845.

21.

Lopa

Colombini

Moretti

De Girolamo

Injective mesenchymal stem cell-based treatments for knee osteoarthritis: from mechanisms of action to current clinical evidences. Knee Surg Sports Traumatol Arthrosc. 2019;27(6):2003-2020.

22.

McGowan

Hogg

Campbell

Rowan

Just-in-time information improved decision-making in primary care: a randomized controlled trial. PLoS One. 2008;3(11):e3785.

23.

Page

McKenzie

Bossuyt

, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. Int J Surg. 2021;88:105906.

24.

Pitkin

RM.

Accuracy of data in abstracts of published research articles. JAMA. 1999;281(12):1110.

25.

Pitkin

Branagan

MA.

Can the accuracy of abstracts be improved by providing specific instructions? A randomized controlled trial. JAMA. 1998;280(3):267.

26.

Pocock

Collier

Dandreo

, et al. Issues in the reporting of epidemiological studies: a survey of recent practice. BMJ. 2004;329(7471):883.

27.

Pocock

Hughes

Lee

RJ.

Statistical problems in the reporting of clinical trials. N Engl J Med. 1987;317(7):426-432.

28.

Rongen

Hannink

Comparison of registered and published primary outcomes in randomized controlled trials of orthopaedic surgical interventions. J Bone Joint Surg Am. 2016;28(5):403-409.

29.

Siex

Nowlin

Ottwell

, et al. Evaluation of spin in the abstracts of systematic reviews and meta-analyses covering surgical management, or quality of life after surgical management, of osteoarthritis of the knee. Osteoarthr Cartil Open. 2020;2(4):100121.

30.

Song

Zhang

, et al. Mesenchymal stem cells in knee osteoarthritis treatment: a systematic review and meta-analysis. J Orthop Translat. 2020;24:121-130.

31.

Strotman

Novicoff

Nelson

Browne

JA.

Increasing public interest in stem cell injections for osteoarthritis of the hip and knee: a Google Trends analysis. J Arthroplasty. 2019;34(6):1053-1057.

32.

Thej

Kumar Gupta

The role of mesenchymal stromal cells in the management of osteoarthritis of the knee. In: Ahmed Al-Anazi

, ed. Update on Mesenchymal and Induced Pluripotent Stem Cells. IntechOpen; 2020: 65–82. Accessed September, 2024. Available at: https://www.intechopen.com/chapters/67252.

33.

Wei

Bao

Intra-articular mesenchymal stem cell injection for knee osteoarthritis: mechanisms and clinical evidence. Int J Mol Sci. 2022;24(1):59.

34.

Woolley

Milan

Master

Feeley

BT.

Evaluation of spin in clinical trials of mesenchymal stromal cells for the treatment of knee osteoarthritis: a systematic review. Am J Sports Med. 2025;53(9):2264-2272.

35.

Zhang

Schon

. The current status of clinical trials on biologics for cartilage repair and osteoarthritis treatment: an analysis of ClinicalTrials.gov data. Cartilage. 2022;13(2):19476035221093065.