Abstract
Background:
The aim of the study was to analyze the type and study design of publications in the Journal of Indian Orthodontic Society (JIOS) and the changes in their trends from 2001 to 2020.
Materials and Methods:
All the online publications in JIOS were classified as background resource, original articles, and case reports. The original articles were further divided into 7 types based on study design. The annual and overall data were compiled. To evaluate the change in trends, the study period was divided into 2 block years: 2001 to 2010 and 2011 to 2020. The differences in the total number of articles and distribution of articles by type and study design were analyzed using Mann–Whitney and chi-square/Fisher’s exact tests, respectively.
Results:
Background resource, original articles, and case reports comprised 38.7%, 43.9%, and 17.4% of the total articles (n = 845), respectively. Non-clinical (34.5%) and cross-sectional (32.35%) studies formed the bulk of original articles. There were 17 randomized controlled trials (RCTs) (4.58%) and 5 systematic reviews (1.35%) among the original articles. The total number of publications were higher for years in the second block compared to those in the first (Mdn of 63.5 vs 14.5, P < .001). The differences in the distribution according to type were not statistically significant [X2(2) = 2.052, P = .35]. A reduction in the share of cohort studies and increase in RCTs were found in the second block (F = 19.174, P = .002).
Conclusions:
The higher proportion of background resource publications and those with study designs lower in the hierarchy of evidence is a matter of concern. Though slow paced, the increase in the number of RCTs and systematic reviews over the past few years is encouraging.
Keywords
Introduction
The hallmark of any scientific journal is the quality of its publications. The research question(s), the method(s) adopted to answer them, and appropriate reporting are three major factors that influence the scientific quality of an article. In the era of evidence-based medicine, the level of evidence provided by publications assumes special importance, and this is largely dependent on the research design used in a study. Hence, assessing these characteristics of articles published in a journal is a valuable introspection and an indispensable quality control measure.
The Journal of Indian Orthodontic Society was started in 19681 as the official journal of The Indian Orthodontic Society, with the primary aim of advancement of orthodontic research in the country. The online version of the journal has been available from 2001, and it currently publishes 4 issues per year. A recent editorial in the journal reported that it receives nearly 250 submissions per year, 1 and this number is on a constant rise. A bibliometric analysis of articles from 2008 to 2017 2 found the proportion of original articles, case reports, and systemic reviews to be 55.3%, 39.9%, and 4.8%, respectively. For the articles published between 2011 and 2016, the data from Gyawali et al 3 indicate that the proportion of original articles and case reports were 55.3% and 19.8%, respectively. Cross-sectional studies were the most common study design. Randomized controlled trial (RCT) and systematic review comprised just 1% and 0.5% of the total publications, respectively.
However, the publications in JIOS have not been comprehensively investigated for their study designs over a sufficiently longer period. Also, the changes in publication trends over the years have not been previously explored. As this information would be immensely useful to all the stake holders of the journal, the current study was undertaken to
analyze the type and study design of articles published in the JIOS from 2001 to 2020 and
to analyze the changes in trends, if any, in the total number and types of articles during this period.
Materials and Methods
All the publications from 2001 till May 20, 2020, available online at
The publications were broadly divided into three types as background resource publications, original articles, and case reports/case series. The background resource publications included editorial, letter to editor, review (narrative, interview, discussion, invited, special, orthodontics and beyond, and review and abstracts), short communication, clinical Innovation, technique tips and transfer, and PG forum. After careful appraisal, the original articles were divided based on study design as non-clinical, cross-sectional, case control, cohort, quasi-experimental, RCT, and systematic review/ meta-analysis. The non-clinical category included all animal, in-vitro, finite element method (FEM), and record-based studies that did not involve direct human participation.
Data Acquisition and Analysis
A consensus discussion on the process and classification of study designs was held among the assessors, prior to initiation of the study. All the articles were screened and subsequently categorized independently by two assessors, a senior resident and professor of orthodontics, with 2 and 16 years of experience, respectively. The categorization of each publication was then cross-verified for agreement. If found concordant, they were directly included for data entry. If discordant, the category of publication was finalized after assessment and discussion with the third assessor, a professor with 20 years of experience following postgraduation. The process of classification is summarized in Figure 1.
Flow Chart of Evaluation and Classification of Publications.
To evaluate the intra- and inter-rater agreement between the two independent assessors, categorization of 50 randomly selected publications was repeated after 1 month. The extent of agreement for classification of publications by article type and study design were analyzed with Cohen’s Kappa.
The data were tabulated in Microsoft Excel 2016 (Microsoft Corporation, Redmond, Washington, USA). The number and proportion of articles in each category were calculated year-wise and for the entire evaluation period.
To evaluate the changes in trends in the number and distribution of publications by type (background resource, original, or case report), the evaluation period was divided into 2 blocks of 10 years each, as 2001 to 2010 and 2011 to 2020. The differences in the total number of publications between these block years were analyzed with Mann–Whitney U test. The differences in the distribution of publication type were analyzed with chi-square test. The differences in the distribution of original articles by study design were evaluated with Fisher’s Exact test and Bonferroni correction for multiple comparisons. The alpha level for all tests was fixed at 0.05. The statistical analyses were performed in IBM SPSS statistics 23 for Windows (IBM, Armonk, NY)
Results
A total of 73 issues were available online for the study period. The fourth issue of 2009 available online was an erroneous repetition of the fourth issue of 2008, and hence was excluded from the study. A total of 881 publications were screened for content from the remaining 72 issues. After eliminating 36 non-scientific items, 845 publications were included for analysis. The year-wise data related to all the article type and sub-categories are presented in Table 1.
The Number and Distribution of Publications in JIOS From 2001 to 2020 by Categories.
Out of a total of 845 articles, 327 were background resource publications (38.7%), 371 were original articles (43.91%), and 147 (17.4%) were case reports (Table 2, Figure 2), respectively.
Distribution of Publications in JIOS from 2001 to 2020 by Type (n, %).
Distribution of Publications in JIOS by Categories in Block years 2001 to 2010 and 2011 to 2020.
Non-clinical (n = 128, 34.5%), cross-sectional (n = 120, 32.35%), and quasi-experimental studies (n = 63, 16.98%) formed the major portion of the original articles (Table 3). Cohort and case-control study designs comprised 7.28% and 2.96% of the original articles, respectively. Seventeen RCTs (4.58%) and 5 systematic reviews (1.35%) were found in the entire evaluation period.
The distribution of all the included articles (n=845) by the categories in the hierarchy of evidence pyramid is presented in Figure 3’.
Distribution of Publications in JIOS by Categories of Hierarchy of Evidence.
Distribution of Original Articles in JIOS by Study Designs in Block Years 2001 to 2010 and 2011 to 2020.
The descriptive statistics for article types by block years are given in Table 2. Mann–Whitney U test indicated that the total number of articles published in the second block (Mdn = 63.5) was greater than the first (Mdn = 14.5), U = 98, P < .001. However, chi-square test for comparison of distribution of articles by type (background resource, original article, and case report) indicated that the differences were not statistically significant [X2(2) = 2.052, P = .35].
The block-wise distribution of original articles by study design is presented in Table 3. Fisher’s exact test indicated a statistically significant decrease in the share of cohort studies (19.12% vs 4.62%) and an increase in that of RCT (0% vs 5.61%) in the second block as compared to the first (F = 19.174, P = .002).
For intra-rater agreement, Cohen’s Kappa indicated a perfect agreement for classification of publications by type (k < .001, P < .001) and near perfect agreement for classification by study design (k = .97, P < .001). Inter-rater agreements for the same were marginally less but nonetheless near perfect for classification by both type (K = .97, P < .001) and study design (k = .92, P < .001).
Discussion
This study explored the number, type, and study design of articles in JIOS available online from 2001 to 2020. A 4-fold increase in the number of the publications was noted in the current decade as compared to previous one. An increase in the number of issues per year, the number of articles per issue, research output in the country, especially in the form of the post-graduate dissertations, the publication pressure, and requirements from universities and regulatory council for career advancement are a few factors that could have resulted in this drastic increase. For the period from 2011 to 2016, JIOS fared better than 4 other South Asian orthodontic journals with respect to the total number of publications. 3 In general, JIOS is still far behind American Journal of Orthodontics and Dentofacial Orthopedics (AJODO), Angle Orthodontist (AO), and European Journal of Orthodontics (EJO) in the number of publications per annum.2, 4-8 This is understandable given the fact that JIOS is relatively a new journal and publishes less issues and articles per year. Reduced visibility, lower citation metrics, and not being indexed in Medline/PubMed could be other factors that reduce the number of submissions to the journal and indirectly the output. From the data of Aura-Tormos et al, 8 it can be deciphered that JIOS had published lesser number of articles than Korean journal of orthodontics and Journal of orthodontics but more than Australian Orthodontic Journal, Seminars in Orthodontics and Orthodontics and craniofacial Research for the years 2007 to 2017.
Original articles and case reports comprised 43.9% and 17.4%, respectively, of the total publications in JIOS for the 20-year period. Interestingly, a similar picture was noted for publications by Indian orthodontists from 1990 to 2011 in Medline-indexed journals. 9 In general, the share of original articles in contemporary reputed orthodontic journals appears to be around 70% or more.5,6,8 EJO has been consistent in having the maximum share of research articles and least case reports.5, 6, 8, 10 It is notable that orthodontic journals in Science Citation Index had more than 80% research articles and less than 10% case reports. 11 The inclusion of some categories of opinion-based publications like editorials and letter to editor in the total number of articles, unlike previous studies, could have reduced the share of original articles in JIOS noted in the study. However, from the data of Gyawali et al, 3 this effect appears to be modest. A relatively lower share of original articles and higher share of background resource articles and case reports in JIOS are a matter of concern to be addressed. Though there was a decrease in the share of case reports and a corresponding increase in original articles during the second block years, these differences were not statistically significant. In a similar comparison of 2 5-year periods, a statistically significant reduction in case reports and an increase in original articles were noted for AO but not AJODO and EJO. 6
Study design has a major impact on the quality of evidence provided by any publication. It also influences their citation metrics, 12 rejection rates, 13 and the dissemination of the knowledge. 12 Non-clinical and cross-sectional studies formed a major share of original articles in JIOS. RCTs and systematic reviews comprised only 4.5% and 1.3% of total publications, respectively. The less share of study designs providing high-quality evidence is a matter of concern, though not unique to JIOS. 7 The share of systematic reviews/meta-analysis and RCTs ranged from 0.7% to 6.3% and 6.5% to 21.9%, respectively, for orthodontic journals assessed from 2007 to 2017. 8 Earlier bibliometric studies of orthodontic journals studies have shown either no change 10 or a decrease 8 in the share of RCT’s over years. In contrast, JIOS has shown an increase in the share of RCTs in the second decade. The number of systematic reviews also increased, though not statistically significant. In fact, all the RCTs and systematic reviews have appeared only from 2013 onwards. This is an encouraging trend in JIOS.
Some difficulties noted during the evaluation of the original articles need to be mentioned here. Under-reporting of information was a common problem. Frequently, the title and abstract were non-informative about the study design, necessitating a thorough scrutiny of the material and methods section of the article. Sometimes, information was inadequate or inexplicit even in this section, forcing the authors to take an educated guess from the available information. Misclassification of study designs, failure to identify and report the study as an RCT, and case report or narrative review presented as original article were few other issues encountered during the venture.
This study is not without limitations. Only the articles available online were included for analysis in the study. Cross-verification with the print version of the journal to locate and include the articles missing in the online version would have yielded more comprehensive results. Further, only 2 issues in 2020 have been published and included in the study. In many instances, the categorization of article into a group was done based on the face value of author’s statements, though substantiating data were unavailable; for example, a study was categorized as RCT if it was mentioned that the study participants were allocated randomly, even though randomization details were not available to verify the claim. Further, it was not in the purview of this study to analyze the actual quality of the research reported in the article. Hence, it is possible that the actual quality of evidence from the article may not commensurate with the quality expected by virtue of its position in the hierarchy of evidence.
We wish to offer few suggestions for improving the research and reporting quality of publications in JIOS. All the major stakeholders in this process, that is, the authors, the editorial board (including the peer reviewers), and the readers, need to understand and shoulder this responsibility. Frequent and focused workshops for potential authors and reviewers about research methodology and reporting standards would be helpful in the long run. Adherence to internationally accepted reporting standards by the authors and strict enforcement of the same by the peer reviewers is the need of the hour. Publishing the pre-publication history of the articles is another suggested option. This is likely to improve the accountability and stringency of the peer review system and thereby its efficiency in weeding out poor-quality articles. Encouraging audience interaction through online blog integrated with the journal homepage can serve as a feedback and another quality check. Finally, the need to index the journal in Medline and PubMed cannot be overstated.
Conclusions
The following are the salient conclusions of this study that explored the type, study design, and change in trends of publications in JIOS online from 2001 to 2020:
There is a 4-fold increase in the total number of publications in the period 2011 to 2020 as compared to 2001 to 2010.
Background resource, original articles, and case reports comprised 38.7%, 43.9%, and 17.4% of the total articles, respectively. There was no statistically significant change in the distribution of these types during the 2 decades of observation.
Non-clinical (34.5%) and cross-sectional (32.35%) studies were the common study designs among original articles. A reduction in cohort studies and increase in RCTs were noted in the block years 2011 to 2020.
The is an evident need to improve the share of original articles, especially those with study designs higher in the hierarchy of evidence. The recent increase in the number of RCTs and systematic reviews published in the journal is a welcome change in this direction.
Footnotes
Acknowledgment
We sincerely thank Dr A. Balamani, Professor and Head, Department of Orthodontics, Mahatma Gandhi Postgraduate Institute of Dental Sciences, Puducherry, for her technical guidance and assistance during the conduct of the study.
Statement of Informed Consent and Ethical Approval
Necessary ethical clearances and informed consent was received and obtained respectively before initiating the study from all participants.
Declaration of Conflicting Interests
Funding
The authors received no financial support for the research, authorship, and/or publication of this article.
Voluntary Disclosure
Balasubramanian Madhan is currently an editorial board member in the Journal of Indian Orthodontic Society.