Appraising the Quality of Reporting of Vascular Surgery Studies That Use the American College of Surgeons National Surgical Quality Improvement Program (NSQIP) Database

Abstract

Objective

The American College of Surgeons National Surgical Quality Improvement Program (NSQIP) is an important data source for observational studies. While there are guides to ensure appropriate study reporting, there has been no evaluation of NSQIP studies in vascular surgery. We sought to evaluate the adherence of vascular-surgery related NSQIP studies to best reporting practices.

Methods

In January 2022, we queried PubMed for all vascular surgery NSQIP studies. We used the REporting of studies Conducted using Observational Routinely-collected Health Data (RECORD) statement and the JAMA Surgery (JAMA-Surgery) checklist to assess reporting methodology. We also extracted the Journal Impact Factor (IF) of each article.

Results

One hundred and fifty-nine studies published between 2002 and 2022 were identified and analyzed. The median score on the RECORD statement was 6 out of 8. The most commonly missed RECORD statement items were describing any validation of codes and providing data cleaning information. The median score on the JAMA-Surgery checklist was 2 out of 7. The most commonly missed JAMA-Surgery checklist items were identifying competing risks, using flow charts to help visualize study populations, having a solid research question and hypothesis, identifying confounders, and discussing the implications of missing data. We found no difference in the reporting methodology of studies published in high vs low IF journals.

Conclusion

Vascular surgery studies using NSQIP data demonstrate poor adherence to research reporting standards. Critical areas for improvement include identifying competing risks, including a solid research question and hypothesis, and describing any validation of codes. Journals should consider requiring authors use reporting guides to ensure their articles have stringent reporting methodology.

Keywords

research reporting quality national surgical quality improvement program RECORD statement observational study

Introduction

The American College of Surgeons National Surgical Quality Improvement Program (NSQIP) is a large-scale registry that is an important data resource for surgical research.^1–4 Nationwide registries like the NSQIP are powerful data sources because of the significant amount of information they contain; The 2021 NSQIP participant use file (PUF) contains demographic, procedural, and outcome related data on 983,851 cases from 685 different sites while 15 other separate NSQIP PUFs contain over 9.6 million procedures.⁵ Although the size and breadth of the procedures and variables are strengths of the NSQIP, the inherent complexity of NSQIP and other analogous large-scale databases predispose them to unwanted variation in research reporting methodology.⁶ Inadequate research reporting can obscure sources of biases, mask the strengths and limitations of data sources, and lead to faulty conclusions. 7 In recent years two research reporting guides have been created specifically for observational studies based on datasets such as NSQIP to help authors follow strong reporting standards. In 2015, The REporting of studies Conducted using Observational Routinely-collected Health Data (RECORD) statement was created as an extension of the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement and focuses on reporting items specific to the needs of studies that use large-scale databases. 7,8 Separately, in 2018, the editors of JAMA Surgery created a checklist (JAMA-Surgery) as a guide for authors using the NSQIP and ten other databases. 9

Although other investigators have examined reporting methodology in different surgical specialties, none have examined vascular surgery studies based on NSQIP data.^10–13 As such, we sought to use the RECORD statement and JAMA-Surgery checklist to appraise the quality of reporting among all published vascular surgery NSQIP studies. Furthermore, we sought to understand the association between Journal Impact Factor (IF) and article reporting quality, given that Journal IF has historically been associated with article quality. 14,15 We hypothesized that vascular surgery NSQIP studies would demonstrate poor to moderate adherence to the RECORD statement and JAMA-Surgery checklist reporting criteria. In particular, we expected the identification of competing risks and clearly defining study inclusion/exclusion criteria to be the most heavily missed reporting items based on our prior work. 14

Methods

Study Selection

In January 2022, we queried PubMed for all NSQIP studies. We included only vascular surgery studies that used the NSQIP. We excluded all non-vascular surgery clinical research studies, studies that used multiple data sources, and studies that qualitatively described the NSQIP. We also excluded all editorials, commentaries, and reviews.

Following study retrieval and duplicate removal, six study team members (AM, WU, KL, AD, JR, CV) reviewed the title and abstracts of all studies for potential inclusion. The title and abstract of each study were reviewed by two reviewers and required the approval of both reviewers to be considered for full-text review. A third reviewer was consulted for any disagreements. All six reviewers assessed articles for review during full-text review. Two reviewers needed to approve each article during full-text review, and a third study team member was consulted to resolve any disagreements about study inclusion/exclusion. Our study sample selection process followed the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) methodology.¹⁶ This study was deemed exempt from review by the University of Florida Institution Review Board (IRB202101723). Informed consent was not obtained or needed for this study since our study sample was made up of scientific articles available to the public.

Evaluation of Article Reporting Quality

After consulting the literature for the most commonly used checklists in appraising research reporting, we chose to utilize two appraisal guides to analyze each study:^11-14

1. The Reporting of studies Conducted using Observational Routinely-collected Health Data (RECORD) statement: The RECORD statement is made up of thirteen unique items Table 1. We excluded five items (Table 1 items 3, 6, 8, 10, and 13) due to their lack of relevance to NSQIP studies. The RECORD statement was created by an international group of 100 health data researchers collectively known as the RECORD Initiative.⁸

2. JAMA Surgery (JAMA-Surgery) Checklist: The JAMA-Surgery checklist is made up of 10 items Table 2. Three items from the JAMA-Surgery checklist were excluded from our analysis (Table 2 items 3, 4, and 7) because we were unable to assess them accurately.

Table 1.

The REporting of Studies Conducted Using Observational Routinely-Collected Health Data (RECORD) Statement.

Item #	RECORD statement item
Title and abstract
1	The type of data used should be specified in the title or abstract. When possible, the name of the databases used should be included
2	If applicable, the geographic region and time frame within which the study took place should be reported in the title or abstract
3	If linkage between databases was conducted for the study, this should be clearly stated in the title or abstract
Methods
4	The methods of study population selection (such as codes or algorithms used to identify subjects) should be listed in detail. If this is not possible, an explanation should be listed in detail
5	Any validation studies of the codes or algorithms used to select the population should be referenced. If validation was conducted for this study and not published elsewhere detailed methods and results should be provided
6	If the study involved linkage of databases, consider use of a flow diagram or other graphical display to demonstrate the data linkage process, including the number of individuals with linked data at each stage
7	A complete list of codes and algorithms used to classify exposures, outcomes, confounders, and effect modifiers should be provided. If these cannot be reported, and explanation should be provided
8	Authors should describe the extent to which the investigators had access to the database population used to create the study population
9	Authors should provide information on the data cleaning methods used in the study
10	State whether the study included person-level, institution-level, or other data linkage across two or more databases. The methods of linkage and methods of linkage quality evaluation should be provided
Results
11	Describe in detail the selection of the persons included in the study (ie, study population selection), including filtering based on data quality, data availability, and linkage. The selection of included persons can be described in the text and/or by means of the study flow diagram
Discussion
12	Discuss the implications of using data that were not created or collected to answer the specific research question(s). Include discussion of misclassification bias, unmeasured confounding, missing data, and changing eligibility over time, as they pertain to the study being reported
Other
13	Authors should provide information on how to access any supplemental information such as the study protocol, raw data, or programming code

Table 2.

The Journal of American Medical Association-Surgical Section (JAMA-Surgery) Checklist.

Item #	JAMA-surgery checklist item
1	Have a solid research question and clear hypothesis. Consider using the FINER (feasible, interesting, novel, ethical, relevant) or PICO (patient, population, or problem; intervention, prognostic factor, or exposure; comparison or intervention; outcome) criteria to develop these
2	Ensure compliance with the institutional review board and data use agreements
3	Conduct a thorough literature review. Use a reference management program for ease in manuscript development
4	Make sure this is the best data set available and that it has the appropriate variables to answer your research question
5	Clearly define the inclusion criteria, exclusion criteria, and outcome variables. Use a flow diagram to describe final patient selection
6	Identify potential confounders and use risk adjustment to minimize bias. Consider using a directed acyclic graph to represent potential associations. Avoid use of causal language in reporting results of these observational studies
7	Ensure that data variables have not changed over time. If so, account for this
8	Ensure that competing risks are identified and addressed
9	Ensure that data issues, such as missing data, are discussed and that any sensitivity analyses or imputations performed are reported in a clear and cohesive way
10	Ensure that your article has a clear take-home message that addresses how you research advances current knowledge and has important policy or clinical implications

Both reporting guides were used to appraise each study. Each item on the RECORD statement and JAMA-Surgery checklist was assigned one point. After excluding items from both checklists, scores out of 7 (JAMA-Surgery) and out of 8 (RECORD) total points were assigned. Six study team members (AM, WU, KL, AD, JR, CV) scored papers. Each paper was independently scored by two team members, and a third team member was consulted for any disagreements. After scoring was complete, pooled Cohen’s kappa scores were calculated.¹⁷ We obtained journal IF percentiles from Journal Citation Reports.¹⁸ We classified journals with an IF greater than or equal to the 90th percentile of one of their core journal categories as having high IF.¹⁸ We classified all other journals as low IF.

Statistical Analysis

Continuous variables are represented by medians and interquartile ranges (IQR), and categorical variables are presented using percentages and frequencies. Fisher’s exact test was used as indicated. Pearson’s correlation coefficient was used to assess for a correlation between scores on the RECORD statement and the JAMA-Surgery Checklist. The significance threshold for all analyses was a 2-tailed P-value ≤.05. All statistical analysis was conducted using R (R Foundation for Statistical Computing, Vienna, Austria) and Microsoft Office Analysis ToolPak (Microsoft Corporation, Redmond, Washington).

Results

Search Results

As of January 2022, we found 3420 NSQIP articles indexed in PubMed. After duplicate removal 3418 articles remained. After conducting a thorough title and abstract review we further excluded 3223 studies, which left 195 articles to be reviewed. After a full-text review of all 195 articles, we further excluded 36 articles, which left us with a final study sample of 159 studies Figure 1. Our sample includes articles published between 2002 and 2022. 30 unique journals are represented in our study sample, and nearly half of all NSQIP vascular surgery studies were published in the Journal of Vascular Surgery (n = 75). Aneurysm research (n = 64) was the most common topic of focus of the articles in our study sample. Table 3 displays the characteristics of our study sample.

Figure 1.

Preferred reporting items for systematic reviews and meta-analyses diagram illustrating study sample selection process.

Table 3.

Study Sample Breakdown.

Study characteristics
Publishing journal	N = 159
Journal of Vascular Surgery	75
Annals of Vascular Surgery	26
Journal of Surgical Research	7
Vascular and Endovascular Surgery	6
The American Journal of Surgery	4
European Journal of Vascular and Endovascular Surgery	4
Journal of the American College of Surgeons	4
Vascular	4
Cureus	3
The American Surgeon	2
Annals of Surgery	2
Journal of the American Society of Nephrology	2
Journal of Endovascular Therapy	2
Surgery	2
Angiology	1
Aorta (stamford)	1
Journal of the American College of Cardiology	1
Journal of the American Heart Association	1
Journal of Cardiothoracic and Vascular Anesthesia	1
Journal of Clinical Medicine	1
Journal of Neurosurgery	1
Journal of Surgical Education	1
The Journal of Vascular Access	1
Journal of Vascular and Interventional Neurology	1
Journal of Vascular Surgery: Venous and Lymphatic Disorders	1
JAMA Surgery	1
Medical Care	1
Neurosurgery	1
Stroke	1
Surgical Innovation	1
Article disease topic of focus
Aneurysm	64
PAD	43
Cerebrovascular	27
Other	13
Diabetes/Dialysis	5
Venous	5
Year Published
2022	3
2021	23
2020	14
2019	7
2018	16
2017	24
2016	7
2009-2015	57
2002-2008	8
Checklist
RECORD score (out of 8): Median (IQR)	6 (5,6)
JAMA-surgery score (out of 7): Median (IQR)	2 (2,3)

Reporting Guideline Criteria Fulfilment

The median criteria fulfilment score for the RECORD statement was 6 (IQR 5,6), and the median fulfilment score for the JAMA-Surgery checklist was 2 (IQR 2,3). Two RECORD statement items were fulfilled in less than half of the study sample—item 9: Provided data cleaning methodology and item 5: Described any validation of codes—were fulfilled in 10% and <1% of studies respectively Table 4. Five items from the JAMA-Surgery checklist were fulfilled in under 50% of articles: item 9: discussed implications of missing data (38%); item 6: clearly identified potential confounders (25%); item 1: solid research question and hypothesis (20%); item 5: clearly defined inclusion criteria, exclusion criteria, and included a flow chart to visualize study population selection (12%); and item 8: identified and addressed competing risks (1%) were fulfilled in less than half of all articles. Five items from the RECORD statement were fulfilled in more than 90% of articles (RECORD items 12, 2, 7, 11, and 1), while only one item from the JAMA-Surgery checklist was similarly fulfilled (JAMA-Surgery item 10) Table 4. The Pearson coefficient between criteria fulfillment of the RECORD statement and JAMA-Surgery checklist was r = .19. Pooled Cohen’s kappa results showed our study scoring had an inter-rater reliability above 80% for both the RECORD statement and JAMA-Surgery checklist.

Table 4.

RECORD Statement and JAMA-Surgery Checklist Item Fulfilment.

Checklist item	n = 159	% Fulfilled
RECORD statement
1, the type of data or name of dataset is specified in title/abstract	159	100
11, method of population selection, with inclusion and exclusion criteria specified	155	97
7, specified outcomes and exposures of interest	154	97
2, defined geographic region and timeframe in title/abstract	152	96
12, discussed the implication of data not available or collected to answer research question	146	92
4, defined codes used to select population	91	57
9, provided data cleaning methodology	16	10
5, described any validation of codes	1	<1
JAMA-surgery checklist
10, clear take home message that addresses advancement of knowledge	158	99
2, statement ensuring IRB compliance	107	67
9, discussed implications of missing data	60	38
6, clearly identified potential confounders	39	25
1, solid research question and hypothesis	31	20
5, clearly defined inclusion criteria, exclusion criteria, and included a flow chart to visualize study population selection	19	12
8, identified and addressed competing risks	2	1

High vs Low Impact Factor (IF) Journals

Out of the 159 articles in our study sample, we were able to find the associated Journal IF for 157 studies. 15 articles met the criteria to be considered published in high IF journals, and 142 articles were considered to be published in low IF journals. The median RECORD statement fulfilment score for high IF journals was 6 (IQR 5,6), while the median score for low IF journals was 5.5 (IQR 5,6). The median JAMA-Surgery checklist score for high IF journals was 3 (IQR 2,3), while the median score for low IF journals was 2 IQR.^2,3 There was no significant difference in the fulfillment of any RECORD statement or JAMA-Surgery item between high vs low IF journals Table 5.

Table 5.

Record Statement and JAMA-Surgery Checklist Item Fulfilment Stratified by Journal Impact Factor (High vs Low).

Checklist Criteria	High IF n = 15	Low IF n = 142	P-value
Record #
1, the type of data or name of dataset is specified in title/abstract	15	142	1.0
2, defined geographic region and timeframe in title/abstract	15	135	1.0
4, defined codes used to select population	9	81	.7371
5, describe any validation of codes	0	1	1.0
7, specified outcomes and exposures of interest	15	138	1.0
9, provided data cleaning methodology	1	15	1.0
11, detailed explanation of selection of persons included in study from database population	15	137	1.0
12, discussed the implication of data not available or collected to answer research question	14	130	1.0
Median score (IQR)	6 (5,6)	5.5 (5,6)
JAMA-surgery #
1, solid research question and hypothesis	2	29	.7371
2, statement ensuring IRB compliance	13	92	.0927
5, clearly defined inclusion criteria, exclusion criteria, and included a flow chart to visualize study population selection	0	18	.221
6, clearly identified potential confounders	6	33	.2043
8, identified and addressed competing risks	0	2	1.0
9, discussed implications of missing data	3	56	.1711
10, clear take home message that addresses advancement of knowledge	15	141	1.0
Median score (IQR)	3 (2,3)	2 (2,3)

^*P-values calculated using Fisher’s Exact test.

Discussion

Our data confirm our hypothesis that vascular surgery studies that use NSQIP data have a poor adherence to research reporting standards. In particular, NSQIP studies frequently failed to identify competing risks, use flow charts to help visualize study populations, have a solid research question and hypothesis, identify confounders, and discuss the implications of missing data. In addition, vascular surgery NSQIP studies neglect to describe any validation of codes or describe data cleaning efforts.

Previously published studies investigating the reporting standards of NSQIP articles have shown varied research reporting adherence. For example, a study by Yolcu et al investigating neurosurgery NSQIP studies and another by Khachfe et al assessing pancreatic surgery NSQIP articles found that NSQIP studies have satisfactory reporting. 11,12 In contrast, a study by El Moheb et al found that NSQIP emergency general surgery articles demonstrate suboptimal reporting. 13 Similar to our findings that two of the most heavily missed reporting items were using flow charts to help visualize study populations (12% fulfilled) and describing code validation (<1% fulfilled), El Moheb et al noted the same criteria were fulfilled in only 11% and 3% of emergency general surgery studies respectively.¹³ These limited data points showing similar areas of deficient reporting quality suggests common areas for improvement exists among different surgical subspecialties. Furthermore, our previous investigation of reporting standards among Vascular Quality Initiative (VQI) articles also found suboptimal reporting.¹⁵ Given our similar findings among vascular surgery articles using two different databases suggests improved research reporting may be broadly needed within the vascular surgery literature.

Although there were several checklist criteria that were missed in most studies, we highlight a few that are especially important below. The most commonly missed item from the JAMA-Surgery checklist, the identification of competing risks, was only addressed in 1% of studies in our study population. Competing risks are events that “compete” with one another, such that the occurrence of one event prevents the occurrence of another “competing” event.^19-21 An example of a competing risk would be mortality in a study of major limb amputation among patients with advanced peripheral arterial disease. A cohort study by Abdel-Qatir et al investigating stroke risks in patients with atrial fibrillation, found that neglecting competing risks led to biased estimates and a stroke incidence estimate that was overestimated by 39%.¹⁹ Similar neglect in vascular surgery NSQIP studies could lead to misleading results.

Another important but frequently missed reporting item from the JAMA-Surgery checklist is item 1, which recommends including a solid research question and hypothesis, but was fulfilled in only 20% of studies. Having both a research question and hypothesis is important to adequately define the population of interest and narrow the scope of the study.⁹ This explicitly helps to avoid scientific “fishing expeditions.” We acknowledge that some studies are intended to be “hypothesis generating” and may not have clear hypotheses. We believe this should be noted explicitly in these studies rather than left ambiguous. Moreover, use of large-scale registry databases like NSQIP generally should not be used for exploratory analyses since multiple statistically significant associations can commonly be identified with little clinical relevance so it is the authors opinion that a minority of reports should be submitted without clear hypotheses.

The most commonly missed item from the RECORD statement, describing any validation of codes, was also only fulfilled in 1% of our study sample. By not describing or citing validation studies, the transparency and reliability of studies are severely diminished.⁷ Put plainly, if there are no validation studies for the codes used to select the study population readers may not be sure if the study sample accurately represents the population of interest. In cases where there is no code validation data, authors should explicitly note this in the methods and discussion. While not every reporting criteria item holds the same importance in strong research reporting, the aforementioned items are especially impactful and should be addressed in future NSQIP studies to ensure appropriate rigor and reliability.

Interestingly our study also found that Journal IF was not associated with differences in reporting standards and that studies published in high IF journals did not have statistically better reporting adherence. None of the RECORD statement or JAMA-Surgery checklist items had a reporting adherence that varied significantly between articles in high and low IF journals. Our findings agree with El Moheb et al who found that emergency general surgery NSQIP articles generally do not vary by journal IF.¹³ In addition, in our previous study that used the RECORD statement and JAMA-Surgery checklist to appraise articles that used another large-scale database, the Vascular Quality Initiative (VQI), we found no association between Journal IF and reporting adherence.¹⁴ While our IF analysis may be skewed by a relatively small number of high IF articles, based on the cumulative findings of this investigation and of similar studies we believe that issues with research reporting affect both high and low IF journals alike.

While not every reporting criterion on the RECORD statement and JAMA-Surgery checklist weigh equally in the evaluation of robust research reporting, our study highlights key takeaways such as addressing competing risks, including a solid research question and hypothesis, and using validated codes to significantly improve the reporting standards of NSQIP articles. To ensure that future studies adhere to strong research reporting standards we encourage journals to require authors submitting articles that use large-scale databases to follow the reporting criteria of both the RECORD statement and the JAMA-Surgery checklist. By incorporating reporting guides into the peer-review process, the reporting rigor and transparency of future NSQIP articles will improve.

Our study has certain limitations. First, we only queried PubMed to create our study population; therefore, it is possible that we missed some NSQIP articles that are not indexed in MEDLINE and the other online sources PubMed queries. In addition, the majority of articles in our study sample were published before 2015; this may limit the generalizability of our findings to more recently published NSQIP articles. Second, while we equally assigned each reporting item one point for fulfillment, we concede that different reporting items are not equally important in their adherence. Finally, while the RECORD statement and JAMA-Surgery checklists are thorough, the two reporting guides we used do not capture every domain of robust research reporting. As a result, our appraisal is not a comprehensive analysis of every research reporting criterion.

Conclusion

Vascular surgery NSQIP articles demonstrate a poor adherence to research reporting standards. Areas to improve the reporting rigor of vascular surgery NSQIP studies include identifying competing risks, using flow charts to help visualize study populations, having a solid research question and hypothesis, identifying confounders, discussing the implications of missing data, and describing any validation of codes and data cleaning efforts. Future efforts should focus on ensuring journals work with authors to incorporate standards of robust reporting methodology in scientific articles.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

This work was supported by the National Institutes of Health [K23HL161428]. The sponsor played no role in the design, methods, data collection, analysis, or preparation of this article.

ORCID iDs

Amin A. Mirzaie

Walker R. Ueland

Samir K. Shah

References

Hall

Hart

Cohen

Hoyt

. The American college of surgeons national surgical quality improvement program: Achieving better and safer surgery. Jt Comm J Qual Patient Saf. 2015;41(5):199-204. doi:10.1016/s1553-7250(15)41026-8.

Khuri

. The NSQIP: A new frontier in surgery. Surgery. 2005;138(5):837-843. doi:10.1016/j.surg.2005.08.016.

Shiloach

Frencher

Steeger

, et al. Toward robust information: Data quality and inter-rater reliability in the American college of surgeons national surgical quality improvement program. J Am Coll Surg. 2010;210(1):6-16. doi:10.1016/j.jamcollsurg.2009.09.031.

Ingraham

Richards

Hall

. Quality improvement in surgery: The American college of surgeons national surgical quality improvement program approach. Adv Surg. 2010;44:251-267. doi:10.1016/j.yasu.2010.05.003.

https://www.facs.org/media/yaol5yoj/nsqip_puf_userguide_2020.pdf. Accessed September 1, 2022.

Hemkens

Benchimol

Langan

, et al. The reporting of studies using routinely collected health data was often insufficient. J Clin Epidemiol. 2016;79:104-111. doi:10.1016/j.jclinepi.2016.06.005.

Benchimol

Smeeth

Guttmann

, et al. The REporting of studies conducted using observational routinely-collected health data (RECORD) statement. PLoS Med. 2015;12(10):e1001885. doi:10.1371/journal.pmed.1001885.

von Elm

Altman

Egger

, et al. The strengthening the reporting of observational studies in epidemiology (STROBE) statement: Guidelines for reporting observational studies. Ann Intern Med. 2007;147(8):573-577. doi:10.7326/0003-4819-147-8-200710160-00010.

Haider

Bilimoria

Kibbe

. A checklist to elevate the science of surgical database research. JAMA Surg. 2018;153(6):505-507. doi:10.1001/jamasurg.2018.0628.

10.

Gebran

Bejjani

Badin

, et al. Critically appraising the quality of reporting of american college of surgeons TQIP studies in the era of large data research. J Am Coll Surg. 2022;234(6):989-998. doi:10.1097/XCS.0000000000000182.

11.

Yolcu

Wahood

Alvi

Kerezoudis

Habermann

Bydon

. Reporting methodology of neurosurgical studies utilizing the American college of surgeons-national surgical quality improvement program database: A systematic review and critical appraisal. Neurosurgery. 2020;86(1):46-60. doi:10.1093/neuros/nyz180.

12.

Khachfe

Habib

Salhab

Fares

Chahrour

Jamali

. American college of surgeons NSQIP pancreatic surgery publications: A critical appraisal of the quality of methodological reporting. Am J Surg. 2022;223(4):705-714. doi:10.1016/j.amjsurg.2021.06.012.

13.

El Moheb

Sabbagh

Badin

, et al. Appraising the quality of reporting of american college of surgeons NSQIP emergency general surgery studies. J Am Coll Surg. 2021;232(5):671-680. doi:10.1016/j.jamcollsurg.2021.01.012.

14.

Mirzaie

Delgado

DuPuis

, et al. Assessing the quality of reporting of studies using Vascular Quality Initiative (VQI) data. J Vasc Surg. 26:2022. doi:10.1016/j.jvs.2022.06.024.

15.

Banks

Dellavalle

. Emerging alternatives to the impact factor. OCLC Syst Serv. 2008;24(3):167-173. doi:10.1108/10650750810898200.

16.

Page

Moher

Bossuyt

, et al. PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews. BMJ. 2021;372:n160. doi:10.1136/bmj.n160.

17.

De Vries

Elliott

Kanouse

Teleki

. Using pooled kappa to summarize interrater agreement across many items. Field Methods. 2008;20(3):272-282. doi:10.1177/1525822X08317166.

18.

Analytics

. 2022 Journal Citation Reports®. 2022. Accessed March 10, 2022. https://jcr.clarivate.com/jcr/home

19.

Abdel-Qadir

Fang

Lee

, et al. Importance of considering competing risks in time-to-event analyses: Application to stroke risk in a retrospective cohort study of elderly patients with atrial fibrillation. Circ Cardiovasc Qual Outcomes. 2018;11(7):e004580. doi:10.1161/CIRCOUTCOMES.118.004580.

20.

Basak

Mistry

Chen

. Understanding Competing Risks. Int J Radiat Oncol Biol Phys. 2021;110(3):636-640. doi:10.1016/j.ijrobp.2021.01.008.

21.

Jepsen

Vilstrup

Andersen

. The clinical course of cirrhosis: The importance of multistate models and competing risks analysis. Hepatology. 2015;62(1):292-302. doi:10.1002/hep.27598.