Sage Journals: Discover world-class research

Abstract

Summary

Objective

To investigate whether the h index (a bibliometric tool which is increasingly used to assess and appraise an individual's research performance) could be improved to better measure the academic performance and citation profile for individual healthcare researchers.

Design

Cohort study.

Setting

Faculty of Medicine, Imperial College London, UK.

Participants

Publication lists from 1 January 2000 until 31 December 2009 for 501 academic healthcare researchers from the Faculty of Medicine.

Main outcome measures

The h index for each researcher was calculated over a nine-year period. The citation count for each researcher was differentiated into high (h² upper), core (h² centre) and low (h² lower) visibility areas. Segmented regression model (sRM) was used to statistically estimate number of high visibility publications (sRM value). Validity of the h index and other proposed adjuncts were analysed against academic rank and conventional bibliometric indicators.

Results

Construct validity was demonstrated for h index, h² upper, h² centre, h² lower and sRM value (all P < 0.05). Convergent validity of the h index and sRM value was shown by significant correlations with total number of publications (r =0.89 and 0.86 respectively, P<0.05) and total number of citations (r = 0.96 and 0.65, respectively, P<0.05). Significant differences in h index and sRM value existed between non-physician and physician researchers (P < 0.05).

Conclusions

This study supports the construct validity of the h index as a measure of healthcare researcher academic rank. It also identifies the assessment value of our developed indices of h² upper, h² centre, h² lower and sRM. These can be applied in combination with the h index to provide additional objective evidence to appraise the performance and impact of an academic healthcare researcher.

Introduction

Academic healthcare institutions employ various systems to quantify research performance. This may enhance research quality and innovation so that the latest discoveries can be translated into medical advances.^1,2 The simplest measure of research performance is the number of publications a researcher produces; however, this bibliometric fails to recognize the quality of a researcher's performance.² The number of citations received by a researcher's publications may determine the scientific impact of his or her research; however, this may be distorted by self-citations and negatively associated citations.^3,4

DECLARATIONS

Competing interests

All authors have completed the Unified Competing Interest form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare that all authors have no relationships with any company that might have an interest in the submitted work in the previous three years; their spouses, partners, or children have no financial relationships that may be relevant to the submitted work; and all authors have no non-financial interests that may be relevant to the submitted work

Funding

None declared

Ethical approval

None declared

Guarantor

None declared

Contributorship

VMP was involved in design, acquisition of data, analysis of data, interpretation of data and drafted article; HA in design, acquisition of data, analysis of data and critical revision; LB in design and analysis of data; RM in design and analysis of data; JM in acquisition of data and drafted article; PS in critical revision; AD in concept and critical revision; and TA was involved in concept, interpretation of data and critical revision

The h index is an alternative bibliometric designed to measure the importance, significance and broad impact of a scientist's cumulative research contributions.⁵ A researcher has index h if h of his or her publications (N_p) have at least h citations each and the other (N_p—h) publications have ≤h citations each.⁵ For example, an academic with an h index of 20 must have 20 publications with at least 20 citations each. Evaluating research performance using the h index is rapidly increasing. The strongest indication of its acceptability is that its calculation has been integrated into the contemporary citation databases of Web of Science and Scopus.^6-8

The h index is frequently criticized for the problems of self-citations, field dependency and multiple authorship, which are common to the majority of current bibliometrics.⁹ If researchers are evaluated only on the basis of the h index, they may be unfairly treated mainly due to three reasons. Firstly, the h index does not fully consider highly cited papers, so that the impact of a researcher with a low h index and a few very highly cited papers will be undervalued. Secondly, the h index does not fully consider productivity of a researcher, so that publications with citations less than the h index are not included. Thirdly, the h index draws a sharp distinction between publications which have higher impact (publications with citations equal to the h index or more) and publications with less importance (for example, the h index assumes that in a researcher with an h index of 20 a publication with 19 citations is not important). In addition to the h index a more reasonable criterion is necessary which separate papers with visible impact from publications with less or no impact.⁹

Recently, a small set of indicators were developed in order to solve these three limitations. The first two limitations are addressed by specifying the percentage of citations which fall into the high, core and low visibility areas of a researcher's publications. The third limitation is addressed by a special form of regression which determines the threshold of publications with visible impact from publications with less or no impact.⁹

At present the assessment and appraisal of academic healthcare researchers relies on a peer review process to judge an individual's research performance, clinical commitment and educational contribution. These domains are further analysed to identify an individual's impact and external visibility.^10,11 The h index has been considered as an objective tool that can assess research performance through citation analysis, although it has not been formally validated in healthcare researchers.²

In this study we aimed to: (1) quantify the high, core and low visibility publications of individual researchers within the Faculty of Medicine at Imperial College London; (2) statistically estimate the number of high visibility publications of each researcher within our group by using a segmented regression model (sRM); (3) demonstrate the construct validity of the h index and our proposed adjuncts by comparing them with a qualitative peer-reviewed process of academic promotion in healthcare research; and (4) determine the convergent validity of the h index and our proposed adjuncts by correlating them with conventional bibliometric indicators.

Methods

Sample population

The Faculty of Medicine Imperial College London was established in 1997, and is one of Europe's largest medical institutions in terms of its staff and student population and its research income.¹² It incorporates eight campuses in North and West London, and is divided into six divisions: Clinical Sciences, Kennedy Institute of Rheumatology, Medicine, National Heart and Lung Institute, School of Public Health, and Surgery and Cancer.¹² We generated a list of all Faculty of Medicine Imperial College academics in employment on the 31 December 2009 from the intranet database. We included academics that were ranked in the hierarchical system in the following grades: lecturer, senior lecturer, reader and professor. Academics can be promoted from lecturer to senior lecturer to reader and then to professor. Each promotion has to be approved by an academic promotions committee which takes into account contributions to the following: education, research, leadership, management, professional and clinical practice.¹² We excluded academics that were research fellows, associates, officers and assistants. We extracted the first name, surname, academic rank, physician status (whether the academic was a physician or not) and department for each of the academics included in the study.

Generating an academic's publication list

We used SciVerse Scopus to evaluate research output because it offers sophisticated tools to search, analyse and visualize research, as well as export to citation management software (Table 1).⁷

Table 1

Content coverage of SciVerse Scopus^6,7

Launched November 2004
More than 18,000 titles 17,000 peer-reviewed journals (including 1200 Open Access journals) 600 trade publications 350 book series 3.7 million conference papers from proceedings and journals
42.5 million records 22 million records with references back to 1996 20.5 million records pre-1996 which go back as far as 1823
359 million scientific webpages indexed via Scirus (science-specific Internet search engine)
24 million patent records from five patent offices ‘Articles-in-Press’ from over 3500 journals

The Scopus Author Identifier uses an algorithm that creates a publication set for an author based on their affiliation, address, subject area and source title, dates of publication, citations and co-authors. We used the surname and initials to search for the publication set authored by each academic. When more than one publication set was identified for the academic's name, we viewed the titles and abstracts of each publication set to determine which sets should be combined. After we selected all the relevant publication sets, we used the ‘Show documents’ button to generate a list of the academic's publications, with their corresponding citation counts. We selected publications from this list that were published between 1 January 2000 until 31 December 2009, and we exported the list to Microsoft Excel. We sorted the publication list in order of descending citations, and calculated the total number of publications, total number of citations and h index.

Calculating h² upper, h² centre and h² lower

We differentiated the geometrical area of the citation count into the high visibility area (h² upper), the core visibility area (h² centre) and the low visibility area (h² lower). Figure 1a shows a schematic curve of number of citations versus publications ranked in order of decreasing citations. The h index refers to the area h x h (h² centre), which does not take into consideration the areas starting at h citations (h² upper) or starting at h publications (h² lower). We calculated h² upper, h² centre and h² lower as a percentage of total citation counts using the following equations:

\begin{matrix} h^{2} upper= \frac{Σ_{j = 1}^{h} ({cit}_{j} - h)}{Σ_{j = 1}^{n} ({cit}_{j})} \times 100 \\ h^{2} center= \frac{h \times h}{Σ_{j = 1}^{n} ({cit}_{j})} \times 100 \\ h^{2} lower= \frac{Σ_{j = h + 1}^{n} {cit}_{j}}{Σ_{j = 1}^{n} {cit}_{j}} \times 100 \end{matrix}

Figure 1

(a) Schematic curve of number of citations versus publication rank showing h² upper, h² centre and h² lower, and (b) schematic curve of cumulative number of citations versus publication rank showing sRM value⁹

The equations demonstrate that h² upper was calculated by subtracting h x h from the sum of citations of publications with citation counts greater than an academic's h index value. The sum of h² upper and h² centre as a percentage of total citation counts was subtracted from 100 to calculate h² lower.

Calculating sRM value

We defined the statistical estimate of the number of high visibility publications as the sRM value. Figure lb shows a schematic curve of cumulative citations versus publications ranked in order of decreasing citations. The quadratic curve in the first part depicts those publications with high citations that contribute most to the total citation count of a researcher (‘core’ publications with high visibility). The linear curve in the second part illustrates those publications which have little contribution to the total citation count (publications with low visibility). The h index underestimates the point of intersection of the two curves (Figure 2), so we used segmented regression to statistically model this point to obtain the sRM value. The following model for the cumulative citation counts y_j was assumed, whereby x_j is the rank of the publication j and z₀ was the point of intersection of the two curves: if x_j < z₀

\begin{matrix} y_{j} = b_{0} + b_{1} x_{j} + b_{2} x_{j}^{2} + e_{j,} & e_{j} \sim N (0, σ_{e}^{2}) \end{matrix}

Figure 2

Calculation of h² upper, h² centre, h² lower and sRM value for one of the professors of clinical medicine (R²_sRM = .99)

otherwise,

\begin{matrix} j_{j} = b_{0} + b_{1} z_{0}^{2} + b_{3} (x_{j} - z_{0}) + e_{j}, & e_{j} \sim N (0, σ_{e}^{2}) \end{matrix}

The x values for j ranged from 1 to k. The z₀ value was an unknown parameter that was defined as the maximum of the quadratic function:

z_{0} = \frac{- b_{1}}{2 b_{2}}

The size of the residual variance (σ²_e), or the proportion of explained variance to total variance (R²), can quantify the model fit. The sRM value was calculated for an academic if their publication set met the following criteria: (i) the citation count could be clearly distinguished into two parts, (ii) the algorithm converged, (iii) R² was greater than 0.90, (iv) the point of intersection of the curves lay within the range of publications and (v) there were at least 15-20 publications in the set.

Statistical analysis

We statistically analysed for construct validity, as well as convergent validity: The bibliometric indicators were construct valid, if they showed significantly mean differences according to the hierarchy of academic rank (lecturer, senior lecturer, reader and professor). The bibliometric indicators were convergent valid, if they showed at least low to moderate correlations among the set of indicators, as well as low to moderate correlations with the h index. Bibliometric data as a rule and the calculated bibliometric indicators are not normally distributed, so we used Box-Cox power transformation to normalize the data.¹³ Bibliometric data often have extreme values or outliers (e.g. highly cited publications) which might distort the variance. To adjust for extreme values we weighted the data: extreme values were assigned a smaller weight than the other values (M-estimator). Weights were estimated using Robust regression.^14,15 We used two-factorial multivariate analysis of variance (MANOVA) on unweighted power transformed data followed by a two-factorial univariate analysis of variance (ANOVA) based on weighted power transformed data to analyse the differences between bibliometric indicators and academic rank or physician status.¹⁶ We also applied post hoc Scheffe's test to explore significant differences between the groups. As the numbers of observations in each group were unequal, we employed Type III sums of squares. We adjusted for the year of first publication of an academic within the time frame of this study by using covariance adjustment. We also used Pearson correlation coefficient to calculate correlations on the power transformed data. We used SAS 9.2 for all analyses. (SAS Institute Inc., SAS 9.2 Enhanced Logging Facilities, Cary, NC, USA: SAS Institute Inc., 2008).

Results

The bibliometric data for 501 academics in the Faculty of Medicine were analysed. The cohort was unbalanced because the academic ranks were heterogeneous in terms of the number of academics and physician status (χ²(3)=48.1, P < 0.05, Cramer's V = 0.31); professors and senior lecturers were more likely to be physicians than lecturers and readers (Table 2). Out of the 501 academics, 398 (79.4%) had at least one publication in the year 2001. The criteria for the calculation of sRM value were fulfilled in 438 academics. The h² upper values were not power transformed because they were approximately normally distributed.

Table 2

Composition of the academics in the Faculty of Medicine

	Physician status
	Non-physician		Physician
Academic rank	Observed, n (%)	Expected, n (%)	Observed, n (%)	Expected, n (%)	Total, n (%)
Lecturer	84(31.8)	57 (21.6)	25(10.6)	52 (21.9)	109(21.8)
Senior lecturer	44(16.7)	68 (25.8)	84 (35.4)	60 (25.3)	128(25.5)
Reader	40 (15.1)	33(12.5)	23 (9.70)	30(12.7)	63(12.6)
Professor	96 (36.4)	106(40.2)	105(44.3)	95(40.1)	201 (40.1)
Total, n (%)	264(100.0)	264(100.0)	237 (100.0)	237 (100.0)	501 (100.0)
Row percent	52.7	52.7	47.3	47.3	100

The expected value demonstrates the relationship between academic rank and physician status in terms of over and under representation

In the MANOVA, construct validity was demonstrated for total number of publications, total number of citations, h index, h² upper, h² centre, h² lower and sRM value, because there were statistically significant differences between the different academic ranks (Wilks’ Lambda = 0.62, F(18, 1377.9) = 14.11, P < 0.05; Table 3). There were statistically significant differences between all the bibliometrics and the groups with different physician status (physician or non-physician) (Wilks’ Lambda = 0.90, F(6487) = 8.55, P < 0.05) and a statistically significant interaction effect between academic rank and physician status (Wilks’ Lambda = 0.94, F(18,1377.9) = 1.68, P < 0.05).

Table 3

This table demonstrates the significant increase or decrease ( p < 0.05) in the mean values of each bibliometric indicator with academic rank

	N_ptOt▴	N_ctOt ▴	h index ▴	h² upper ▾	h² centre ▴	h ² lower ▴	sRM value▴
Non-physician
Lecturer	20.2	491.7	10.0	65.5	23.8	9.51	2.8
Senior lecturer	32.7	766.3	13.1	57.4	28.0	14.6	20.9
Reader	37.6	835.6	14.4	53.4	29.3	17.3	24.4
Professor	69.9	2045.8	21.4	53.4	27.6	19.0	35.7
Physician
Lecturer	16.7	256.6	6.4	65.2	20.4	10.4	11.9
Senior lecturer	31.4	562.3	10.2	56.0	28.0	13.5	18.6
Reader	59.8	964.0	15.6	49.9	28.2	21.9	28.5
Professor	82.5	2388.8	21.9	53.0	26.7	20.2	36.6

N_ptot, total number of publications; N_ctot, total number of citations; ▴, increases when academic rank increases; ▾, decreases when academic rank increases

In the ANOVA, construct validity was demonstrated for total number of publications, total number of citations, h index, h² upper, h² centre, h² lower and sRM value, because there were statistically significant differences between the different academic ranks (P < 0.05; Table 4; Figure 3). Groups with different physician status showed statistically significant differences regarding total number of citations, h index, h² upper and sRM value (P < 0.05). Academic rank and physician status showed a statistically significant interaction effect with respect to total number of publications and h index (P < 0.05). Professors (physician or non-physician) had more publications than the other academic ranks (P < 0.05). Physician professors and physician readers had more publications than their non-physician counterparts (P < 0.05). The more recent the year of first publication of an academic the lower were his or her total number of citations (regression coefficient = -0.12, P < 0.05) and h² upper (regression coefficient = -0.011, P < 0.05). Lecturers had the smallest h index and total number of citations, but they showed the highest h² upper (about 65%) of all groups.

Table 4

Results of seven two-factor analyses of variance (academic rank, physician status, interaction) with covariate-adjustment (year of publication)

Effect	SSQ	df	MSQ	F	R²
Total number of publications^p
Year of first publication	1.22	1	1.22	1.09
Academic rank	245.78	3	81.92	72.69^*
Physician status	0.26	1	0.26	0.23
Academic rank x physician status	10.35	3	3.45	3.06^*
Error	554.55	492	1.13
Total	858.43	500			0.35
Total number of citations^p
Year of first publication	25.30	1	25.30	5.34^*
Academic rank	889.83	3	296.61	62.60^*
Physician status	18.60	1	18.60	3.93^*
Academic rank x physician status	37.18	3	12.39	2.62
Error	2312.13	488	4.74
Total	3373.65	496			0.31
h index^p
Year of first publication	2.69	1	2.69	2.75
Academic rank	196.42	3	65.47	66.81^*
Physician status	5.73	1	5.73	5.85^*
Academic rank x physician status	8.23	3	2.74	2.80^*
Error	478.46	488	0.98
Total	716.46	496			0.33
h² upper^np
Year of first publication	0.25	1	0.25	11.10^*
Academic rank	1.00	3	0.33	14.98^*
Physician status	0.001	1	0.001	0.06
Academic rank x physician status	0.03	3	0.01	0.43
Error	10.96	492	0.02
Total	12.30	500			0.11
h² centre
Year of first publication	0.04	1	0.04	3.03
Academic rank	0.21	3	0.07	5.69^*
Physician status	0.04	1	0.04	3.02
Academic rank x physician status	0.01	3	0.004	0.36
Error	5.90	485	0.01
Total	6.17	493			0.05
h² lower^p
Year of first publication	0.18	1	0.18	2.22
Academic rank	5.92	3	1.97	23.80^*
Physician status	0.05	1	0.05	0.55
Academic rank x physician status	0.28	3	0.09	1.11
Error	40.78	492	0.08
Total	48.44	500			0.16
sRM value^p
Year of first publication	0.02	1	0.02	0.69
Academic rank	3.62	3	1.21	34.87^*
Physician status	0.15	1	0.15	4.18^*
Academic rank x physician status	0.10	3	0.03	0.97
Error	14.38	416	0.04
Total	18.5	342			40.22

P, power transformed; np, not power transformed; SSQ, sums of squares; df, degree of freedom; MSQ, mean square; F, F-test; R², coefficient of determination; Type III sums of squares

P<0.05

Figure 3

Differences between academic rank, physician status and (a) h index, (b) sRM value, (c) h² upper, h² centre and h² lower

Convergent validity of the h index and sRM value was demonstrated by significant correlations with total number of publications (r = 0.89 and 0.86 respectively, P < 0.05) and total number of citations (r = 0.96 and, respectively, P < 0.05; Table 5).

Table 5

Matrix of inter-correlations (Pearson) between different bibliometrics (power transformed variables, N= 438)

	N_ptot	N_ctot	h index	h² upper	h² centre	h² lower	sRM
N_ptot	1
N_ctot	0.84^*	1
h index	0.89^*	0.96^*	1
h² upper	-0.39^*	0.06	-0.17^*	1
h² centre	0.21^*	0.04	0.29^*	-0.64^*	1
h² lower	-0.74^*	-0.41^*	-0.53^*	0.66^*	-0.42^*	1
sRM	0.86^*	0.65^*	0.84^*	-0.76^*	0.61^*	-0.76^*	1

N_ptot, total number of publications; N_ctot, total number of citations

P<0.05

Discussion

We have demonstrated how to calculate h² upper, h² centre, h² lower and sRM value in order to enhance bibliometric assessment and appraisal of research performance of an academic healthcare researcher. The sRM value is the first indicator which is based on statistical and not numerical concepts. Our study supports construct and convergent validity of the h index and sRM value.

Criticisms of the h index have led to the development of numerous variants of the h index.^9,17 However, many of these variants are highly correlated with the h index and offer limited value in research performance assessment.^9,17,18

Our study presents h² upper, h² centre, h² lower and the sRM value, which complements and provides additional information to the h index. For example, in our study the lecturers had few publications and had a low h index, but their small publication set was well cited. A multi-dimensional bibliometric approach to research performance assessment may identify patterns of productivity, such as prolific researchers (high h² centre and upper, low h² lower), mass producers (high h² lower, low h² centre and upper) and perfectionists (high h² upper, low h² centre and lower).^9,19 This can provide practical information that can be conveyed in various forms such as graphical images (Figure 4) to support academics and supervisors in planning and managing careers, as well as assisting in the decisions for academic promotion and funding.²⁰

Figure 4

Representative grid categorizing researchers as prolific, perfectionist or mass producers according to h² upper, h² centre and h² lower characteristics

Previous studies have shown construct validity of the h index in healthcare specialties such as anaesthesia, neurosurgery, radiology and urology.^21-25 Our study is the largest construct validation study of the h index in academic healthcare researchers within one of the largest Faculties of Medicine in Europe. It also takes into account the length of academic career and physician status. Academic physicians (clinician scientists) will spend approximately half of their time in clinical care, and many physicians believe that bibliometric analyses favour basic scientific research over clinical research.^26-28 Our study confirms that differences do exist between physician and non-physician academic healthcare researchers. This should be considered when measuring the research performance of academic physicians with bibliometrics such as the h index.

Our study has limitations associated with the disadvantages of the h index, which does not consider degree of co-authorship, gratuitous authorship, age of the researcher, context of citations, self-citations and scientific discipline.^9,17,22 We did not consider the impact of document type on research performance. We did not account for the bias that may exist of researchers preferring to cite work from their own country. We have extracted bibliometric data for academics from a single centre specializing in medicine, so the findings of this study may not be generalizable to other academic institutions or specialties. The h index does give an indication of the absolute number of citations (h x h), but this information is deficient in the sRM value. We collected data over a defined timeframe, so our results cannot be used as an absolute benchmark for comparing academic healthcare researchers (Figure 4). The sRM value is not easily calculable because it relies on segmented regression modelling using statistical software.

There are differences in the publication sources for Web of Science, Scopus and Google Scholar, which produce quantitatively and qualitatively different citation counts for healthcare reasearchers.²⁹ We only used Scopus to extract the bibliometric data of the academic healthcare researchers, and it is unknown whether physician bias exists for any of the citation databases. Miscalculation of bibliometric data is more likely if an author has a common name, if they have published using a combination of names, if they have worked in a variety of institutions or if they have extensive research interests.²²

Future research may include studies that explore the relationship between our measures of high visibility (h² upper and sRM) and the impact and external visibility of an individual's research. This may objectively measure the benefit of an individual's research to the wider economy and society. It may also offer impartial evidence for selection and promotion committees to guide and support healthcare researchers in achieving enhanced academic goals.

Conclusions

Our study supports construct validity of the h index as a measure of academic rank for healthcare researchers. However, the h index describes only a small proportion of information about the academic output of a researcher. As a result, it may not accurately reflect an individual's research performance. The use of h² upper, h² centre, h² lower and sRM value in combination with the h index may provide additional objective evidence to assess and appraise the impact of an academic healthcare researcher. This may identify patterns of academic activity that can support research quality and productivity of innovative researchers. It can also offer guidance for less creative researchers to optimize their academic profiles. The ultimate aim of applying these h index values is to guide and support the academic output and impact of healthcare researchers to support translational medicine for the improvement of patient care.

Footnotes

Acknowledgements

None declared

References

Ashrafian

, Rao

, Darzi

, Athanasiou

Benchmarking in surgical research.

Lancet 2009; 374: 1045–7

Patel

, Ashrafian

, Ahmed

. How has healthcare research performance been assessed? A systematic review. J R Soc Med 2011; 104: 251–61

Bornmann

, Daniel

H-D.

What do citation counts measure? A review of studies on citing behavior.

J Document 2008; 64: 45–80

Radicchi

, Fortunato

, Castellano

Universality of citation distributions: toward an objective measure of scientific impact.

Proc Natl Acad Sci USA 2008; 105: 17268–72

Hirsch

JE.

An index to quantify an individual's scientific research output.

Proc Natl Acad Sci USA 2005; 102: 16569–72

Patel

, Ashrafian

, Almoudaris

. Measuring academic performance for healthcare researchers with the H index: which search tool should be used? Med Princ Pract 2012; 1: 2–3

Elsevier. Scopus Website. 2004. See www.scopus.com (last accessed October 2012)

Thomson-Reuters. ISI Web of Knowledge Website. 2004. See www.isiknowledge.com/ (last accessed October 2012)

Bornmann

, Mutz

, Daniel

The h index research output measurement: two approaches to enhance its accuracy.

J Informetrics 2010; 4: 407–14

10.

Lucey

, Sedmak

, Notestine

, Souba

Rock stars in academic medicine.

Acad Med 2010; 85: 1269–75

11.

Patel

, Warren

, Humphris

. What does leadership in surgery entail? ANZ J Surg 2010; 80: 876–83

12.

Imperial College London. See http://www1.imperial.ac.uk/medicine/about/

13.

Box

GEP

, Cox

DR.

An analysis of transformations.

J R Stat Soc B 1964; 26: 211–52

14.

Huber

PJ.

Robust Statistics. New York: John Wiley & Sons, Inc, 1981

15.

Hampel

, Ronchetti

, Rousseeuw

, Stahel

WA.

Robust Statistics: The Approach Based on Influence Functions. New York: John Wiley & Sons, Inc, 1986

16.

Stevens

JP.

Applied Multivariate Statistics for the Social sciences. New York: Taylor & Francis, 2009

17.

Bornmann

, Daniel

HD.

The state of h index research.

Is the h index the ideal way to measure research performance? EMBO Rep 2009; 10: 2–6

18.

Bornmann

, Mutz

, Hug

, Daniel

H-D.

A multilevel meta-analysis of studies reporting correlations between the h index and 37 different h index variants.

J Informetrics 2011; 5: 346–59

19.

Cole

, Cole

JR.

Scientific output and recognition: a study in the operation of the reward system in science.

Am Sociol Rev 1967; 32: 377–90

20.

Ioannidis

JP.

More time for research: fund people not projects.

Nature 2011; 477: 529–31

21.

Benway

, Kalidas

, Cabello

, Bhayani

SB.

Does citation analysis reveal association between h-index and academic rank in urology?

Urology 2009; 74: 30–3

22.

Bould

, Boet

, Sharma

, Shin

, Barrowman

, Grantcharov

h-indices in a university department of anaesthesia: an evaluation of their feasibility, reliability, and validity as an assessment of academic performance.

Br J Anaesth 2011; 106: 325–30

23.

Lee

, Kraus

, Couldwell

WT.

Use of the h index in neurosurgery. Clinical article.

J Neurosurg 2009; 111: 387–92

24.

Rad

, Brinjikji

, Cloft

, Kallmes

DF.

The H-index in academic radiology.

Acad Radiol 2010; 17: 817–21

25.

Spearman

, Quigley

, Wilberger

JE.

Survey of the h index for all of academic neurosurgery: another power-law phenomenon?

J Neurosurg 2010; 113: 929–33

26.

Durso

, Christmas

, Kravet

, Parsons

, Wright

SM.

Implications of academic medicine's failure to recognize clinical excellence.

Clin Med Res 2009; 7: 127–33

27.

Sheridan

DJ.

Reversing the decline of academic medicine in Europe.

Lancet 2006; 367: 1698–701

28.

Ashrafian

, Patel

, Skapinakis

, Athanasiou

Nobel Prizes in medicine: are clinicians out of fashion?

J R Soc Med 2011; 104: 387–9

29.

Kulkarni

, Aziz

, Shams

, Busse

JW.

Comparisons of citations in Web of Science, Scopus, and Google Scholar for articles published in general medical journals.

JAMA 2009; 302: 1092–6

Enhancing the h index for the objective assessment of healthcare researcher performance and impact

Abstract

Summary

Objective

Design

Setting

Participants

Main outcome measures

Results

Conclusions

Introduction

Methods

Sample population

Generating an academic's publication list

Calculating h2 upper, h2 centre and h2 lower

Calculating sRM value

Statistical analysis

Results

Discussion

Conclusions

Footnotes

Acknowledgements

References

Calculating h² upper, h² centre and h² lower