Abstract
Introduction
The role of the scholar was introduced to the early CanMEDS framework in 1993 by the Royal College of Physicians and Surgeons of Canada to bring emphasis to the role of the physician scientist. 1 The persistence of this role establishes scholarly work as a pillar of medical education, and sets an expectation for the participation in some form of research experience in the education of medical trainees. Participation in research has been shown to improve residents’ critical thinking, academic productivity, influence career paths, which ultimately impacts patient care.2-4
Although research work is not specifically required for residency in radiology and other specialties, residency programs do consider pre-residency research during applicant review. Information regarding each applicant’s academic profile is available through the Canadian Resident Matching Service (CaRMS). 5 The CaRMS application portal allows applicants to list a variety of research-related activities, which include research experience, publications, and presentations. For publications and presentations, the applicants are able to list not only peer-reviewed publications or presentations but also non-peer reviewed activities and pending publications or presentations which have not yet been presented at the time of application. 6 The residency programs use the applicant information gathered on CaRMS as the primary source in deciding whether to offer applicants an interview and also in how prospective residents are ranked.
Although there has been some decline in competitiveness since 1997, radiology remains a popular specialty having been one of the top 10 discipline choice for Canadian Medical Graduates at least since 2012.7-9 Because of the perceived competitiveness, it is conceivable that a temptation may exist for applicants to misrepresent parts of the application, with parts of the application that are difficult to verify being at most risk.
Previous studies have demonstrated varying rates of dishonesty or inaccuracies associated with the application process to several in-demand specialties.10-19 Amongst these papers are some earlier studies with misrepresentation rates amongst applicants being as high as 34.1% for gastrointestinal fellowship in 1995, 34% for emergency medicine residency in 2002, and 20.4% for emergency medicine residency in 1995–1996.17-19 Specific to medical imaging are misrepresentation rates of 16%, 11%, and 4% of applicants claiming publications when applying to radiology or radiology fellowship programs in 1998, 2000, and 2013 papers, respectively.11,20,21 These misrepresentation rates are a concern and require reassessment in our current Canadian context as no Canadian data has been previously reported.
The objective of our study is two-folds. First, to determine research experience as defined by publications and presentations of radiology applicants to CaRMS. Second, to determine the current rates of research publication and presentation inaccuracies in the applicant profiles to our residency program at the University of Saskatchewan.
Methods
Ethics exception was provided by the Research Ethics Board at the at the University of Saskatchewan. The common application forms of Canadian medical graduates submitted to our radiology residency program via CaRMS were reviewed over 2 consecutive application cycles—2019–2020 and 2020–2021 (hereon called year 1 and year 2). Basic demographic information was collected, which included school name from the “Medical Education” section, and non-physician graduate degrees from the “Non-Medical Education” section.
Information submitted under the “Publications/Presentations” section was assessed. Publications listed as “article” for publication type and “yes” for peer-reviewed status were included in assessment. Publications listed as any other publication type, such as “book chapter,” “thesis,” “web article,” and/or listed as “no” for peer-reviewed status were not assessed. The PubMed-MEDLINE database was principally used to verify publication citations. When citations were not found, the official website of the listed journal was consulted. When the journal of the citation was not found, the journal was validated using Ulrich’s Periodicals Directory. Internet search engine, using a various permutation of keywords from the author name, article title, and/or journal name, was used as a last resort. Inaccuracies of articles were defined as (1) non-existence of article, (2) non-authorship, (3) self-promotion to a higher authorship status, and (4) self-demotion to a lower authorship status.
Peer-reviewed articles listed as “pending” for publication status were assessed independently using the same criteria and verification process. Publication delay from the date of CaRMS application deadline was taken into consideration. For consistency between the 2 years assessed, only articles identified within 6 months of the CARMS application deadline were assessed for each year.
Presentations listed as “conference,” “oral,” or “poster” for presentation type, regardless of peer-reviewed status, and publications listed as “abstract” for publication type were assessed together in an independent category. Presentations listed as anything else were not assessed, which included “thesis defense,” “workshop,” “seminar,” “keynote", “rounds,” and “educational exhibit.” Conference presentations were classified as internal (such as local research days), external (provincial, national, or international conference), or uncertain. Verifiability of presentations/abstracts was determined via the availability of conference programs and/or list of abstracts on the official website of the specified society, association, conference, residency program, and/or journal. Presentations from the annual scientific meeting of the Canadian Association of Radiology, online data were supplemented with books of abstracts dating back to 2011. Internet search engine was used as the last resort. A presentation/abstract was deemed unverifiable if the conference program or list of abstracts was unavailable from the above sources. Additionally, a presentation was deemed unverifiable if the official conference program or list of abstracts only listed the presenting author, and the applicant was a non-presenting author. Inaccuracies of presentations/abstracts were defined as (1) non-existing presentation, (2) non-authorship, (3) self-promotion to a higher authorship status, (4) self-demotion to a lower authorship status, and (5) double entry.
For all categories, the inaccuracies that could result in exaggeration of experience (nonexistent article, non-authorship, self-promotion, and double entry) were labeled as misrepresentations.
Descriptive data are provided. Subset analysis of errors by schools or educational background was not performed to preserve anonymity.
Results
Demographic and Research Experience of Applicants from CaRMS Cycle 2019–2020 and 2020–2021.
CaRMS: Canadian Resident Matching Service.
Inaccuracies Found in Reported Publications and Pending Publications of Applicants from CaRMS Cycle 2019–2020 and 2020–2021.
CaRMS: Canadian Resident Matching Service.
Most applicants (60%) did not report pending publications. Thirty-two percent of the applicants had 1–2 pending publications while 8%, had 3 or more pending applications (Table 1). A total of 77 pending publications were reported from both years (Table 2). Twenty-seven (35%) of the pending publications reached publication status 6 months post-CaRMS application deadline. Out of those that reached publication, a total of 3 inaccuracies were found: 1 self-demotion to lower authorship status (from 2nd author to 3rd author), 1 self-promotion to higher authorship status (from 58th author to 6th author), and 1 non-authorship (Table 2).
In total, there were 257 publications confirmed from both the publications and pending publications sections. These were in a total of 180 different journals with the most popular 3 journals as follows: European Radiology (11 articles), Canadian Association of Radiologists Journal (9 articles), and American Journal of Roentgenology (AJR) (7 articles).
Inaccuracies Found in Reported Presentation/Abstracts of Applicants from CaRMS Cycle 2019–2020 and 2020–2021.
CaRMS: Canadian Resident Matching Service.
aBreakdown of internal vs external presentation with inaccuracies in the results section.
bThe presentation/abstract demonstrating the inaccuracy type of double entry has a simultaneous inaccuracy type of self-promotion.
The 296 external presentations were given at 178 different host events with the top 3 most popular as follows: Canadian Association of Radiologists Conference (23 presentations), Radiological Society of North America (17 presentations), and Society for Neuroscience (10 presentations).
Inaccuracies and Misrepresentations Found in Research Experience of Applicants from CaRMS Cycle 2019–2020 and 2020–2021.
CaRMS: Canadian Resident Matching Service.
*Misrepresentations are inaccuracies that could result in exaggeration of experience (nonexistent article/presentation, non-authorship, self-promotion, and double entry).
Discussion
This study represents the first study to report on the research experience of applicants to a diagnostic radiology residency program in Canada. Our study found that 88% of applicants had research experience as defined by presentations or publications and 28% of applicants held advanced degrees in Master’s and/or PhD. This is comparable to another Canadian study performed by Hames et al in 2018, which reports 86% of diagnostic radiology residents having research experience before residency and 19% holding a nonphysician graduate degree. 22
Prior studies have reported various rates of academic misrepresentation with a low of 1.3% in plastic surgery residency applicants and a high of 34% in gastrointestinal fellowship and emergency medicine residency applicants.10-19 Based on our findings, the rate of misrepresentation found in publications and pending publications that reached publication is low at 8 total inaccuracies out of 307 articles (2.6%). Excluding the 3 inaccuracies of self-demotion which would not lead to a competitive benefit, there were only 5 potential misrepresentations/exaggerations amongst the 307 cited articles (1.6%), found in 2 out of 106 total applicants. There have been 3 prior studies on misrepresentation amongst applicants to radiology residency or fellowship programs.11,20,21 A 1996 study of radiology residency applicants from Indiana found 13 non-existent articles (7 listed as published and 6 listed as pending published) amongst 128 potentially verifiable articles (10.1%) with no assessment of misrepresentations in author order. 21 A 1998 study of applicants to a body breast fellowship program at Memorial Sloan Kettering/Cornell found errors in 39 of 222 (17%) articles cited, or 15 (8%) when excluding the 24 journals that could not be found to verify the article. 20 These errors were amongst 16% of applicants citing a publication. 20 A 2011 study of radiology residency applicants at Harvard reported 5 (1.1%) inaccuracies in 422 articles listed, with misrepresentations committed by 3.6% of applicants. 11 Our findings are similar to the 2011 Harvard study with low rates of misrepresentation in both.
Eisenberg attributes the widespread availability of PubMed and the inclusion of articles from less well-known online journal, combined with a stratified search approach in being the primary reason that more reported articles are being comprehensively identified, in recent as opposed to earlier studies. 11 This is supported by a study from Meeks et al showing decreasing rates of misrepresentation among cited articles on orthopedics residency applications from 18.0% in 1999 and 20.6% in 2007 to only 1.2% in 2017. 15 Meeks et al credit the inclusion of a PubMed identifier number in applications as one of the reasons for decreasing misrepresentation rates.
Only 38% of articles listed as pending peer reviewed publications were identified as published within 6 months. Some of this could be due to publication delays by medical journals. Based on studies of 13 journals from 1996 to 2016 publication delays can be lengthy with median times of 87–147 days from submission to acceptance and 99–567 days from submission to publication. 23 Specific to radiology the mean time from acceptance to publication of AJR articles was 243 days in 2012. 24 However, delays between acceptance and publication may now be shorter as evidenced by a review of 2016 medical journal papers showing median times from acceptance to publication of 68 days. 25 This finding could also relate to uncertainty regarding the term “pending” with applicants potentially interpreting this as meaning anything from actively writing a manuscript, submitted but under review, accepted pending revisions, to accepted pending publication. Although CaRMS does define instruction to “Check the Pending box if your publication has been accepted, but is not yet published,” 26 this is neither clearly seen nor accessible as it is located in an outdated 2011 document not directly linked to the application process. 25 This inaccessible definition, as well as low rates of misrepresentation found in pending articles that reached publication, indicates that the cause of few articles reaching publication is more likely a result of uncertainty regarding the term “pending” as opposed to intentional misrepresentation/exaggeration by applicants. To remedy this issue, we will provide feedback to CaRMS with the results of this study to urge them to clarify the criteria for the category “pending” during the CaRMS application process. We believe this would provide standardized information and increase confidence of the program in the applicants’ profile.
Approximately half (47%) of the reported presentations/abstracts were not found with the described search method, which renders verifying the existence of the many of the entered presentation/abstract impossible. Inability to verify was more common with internal than external presentations. This category also had the highest rate of inaccuracies in 12.6% of the 221 verifiable entries. Excluding 9 cases of self-demotion to lower authorship, there were 17 (9%) verifiable titles demonstrating misrepresentation from 17 (16%) applicants, which included 9 self-promotions, 1 non-authorship, 9 non-existent presentation/abstract, and 1 double entry. Five out of 9 inaccuracies of self-promotion in authorship occurred among middle authors (i.e., not first or last author), making the effects of these to be of uncertain importance and therefore should be considered less strongly. In terms of self-demotion in authorship status, although it does not inflate a candidate’s application, it is accounted as an inaccuracy to better understand the degree of background inaccuracies in the application when ideally there should be a perfect match between the application and the publications/presentations. The category of presentations/abstracts was difficult to assess for multiple reasons including variable completeness and permanence of on-line records, in-person changes not captured in the official program, only listing the presenting author as opposed to the whole research team, and others. For the same reasons, the presentations/abstracts category is conceivably the most vulnerable category to academic falsification due to the applicants potentially understanding the difficulties of confirming activities from incomplete online sources. Due to these inherent challenges and uncertainties in the verification process, inaccuracies and misrepresentations found in the presentation/abstract category account for the bulk of the total number of applicants with inaccuracies and misrepresentations in any category. Although there are multiple prior studies on accuracy of publications on residency applications, to our knowledge this paper is the first assessing accuracy of claimed presentations/abstracts on residency applications.
Despite a slightly decreasing competitiveness trend in Canadian match results from 2010–2020 to diagnostic radiology residency, most recently reported by Li et al, 9 our data over a limited timeline does make some interesting observations. Although we only assessed 2 years of applicants, there was greater research experience in the second year of applicants compared to the first year across all categories including number of applicants with non-physician graduate degrees, publications, and presentations/abstracts. Most notably in the publication category, where there are nearly half as many applicants entering CaRMS with no publication experience in year 2 than that observed for year 1. Therefore, despite the decreased number of applicants listing radiology as first or only choice in CaRMS applications shown by Li et al, 19 there remains competitive candidates with extensive listed research experience in the candidate pool.
There was a wide range of journals and conferences cited in applications. This included only 9/257 (4%) of articles in the CARJ and 23/296 (8%) of external presentations at the CAR ASM. This is in contradistinction to previous studies on Canadian resident research day projects where 14% of generated publications were in the CARJ and 93 residents made 195 research presentations at the CAR ASM.27,28 This difference is likely because although CaRMS applicants are interested in radiology, their background is not exclusive to the discipline so they would be expected to have a wider range publishing journals and presenting conferences.
We recognize there are limitations to our study. Misrepresentations do not necessarily mean deceptive self-promotion as it could simply represent a mistake or misunderstanding of factors such as the importance of author order and publication status. Therefore, the true rate of intentional academic dishonesty is difficult to determine. There is limited verifiability of conference presentations and abstracts, which may result in the underestimation of inaccuracies or misrepresentations in this category. This is a single center study reflecting the experience of one medium sized diagnostic radiology residency program in Canada. Therefore, a small sample size is an important limitation of the study. However, there is good geographic representation from across Canada and applicants tend to apply across the country indicating relevance beyond our single center. We hope dissemination of the results of this study will improve awareness about the importance for an accurate representation of scholarly work in residency applications. This heightened awareness may help decrease inaccuracies in residency applications for future years.
Conclusion
Despite some uncertainty with scientific articles reported as pending and scientific presentations, radiology residency applicants are accurately representing their published articles with a negligible number of misrepresentations. Canadian radiology residency programs should regard the publication profiles of the applicants with a high level of confidence.
Footnotes
Acknowledgments
We gratefully acknowledge the assistance of the Canadian Resident Matching Service (CaRMS) in making this study possible.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
