Abstract
The Australian public strongly supports health and medical research and believes in its benefits to quality of life. Health and medical research findings are amongst the most heavily covered issues in the Australian media, and charitable community support for health and medical research is widespread. Yet, public investment in health and medical research is low by OECD standards. Other developed nations are rapidly expanding their investment in health and medical research to gain the anticipated health, health care and economic benefits. [1], p.30]
There is intense community, government and professional interest in the current state of Australian mental health research. This is reflected not only in recent government [2], community [3–5] and scientific reports [6], but also in the priority settings of the most recent National Mental Health Plan [7]. For the first time, innovation, research and sustainability are described as critical elements of our National Mental Health Strategy [7]. The most recent report of the Mental Health Council of Australia concludes that if we are to reduce the ongoing costs of mental disorders in Australia, we must develop new preventative, early intervention and sustained treatment strategies [8].
In 2003, a case was put to the Prime Minister's Science, Engineering and Innovation Council that Australia should increase its investment in research in basic and clinical neurosciences [6]. Previously, the case had been put for significant investment in early child development and family support [9] and this has now resulted in a substantial national program [10]. Specifically, the earlier expansion of treatments for depression in primary care in Australia, and the more recent work of beyond blue: the national depression initiative, have already produced population-based benefits [11]. On the basis of impressive trial evidence [12], arguments have now been advanced for rapid expansion of e-health-based mental health information and intervention services [13].
However, when such arguments for increased funding for research and development are advanced, it is critical that we specify how we will measure potential benefits. Traditional academic markers of research success have included both measurements of inputs (e.g. increased dollar funding from both government and non-government funding sources, increased success at competitive grants through both Australian national and international funding sources) and at least some measurements of outputs (e.g. number of scientific publications, publications in high quality international journals). Until recently, the impact factor of the journal served as a proxy for the likely impact of individual articles.
With regard to these markers of output, the advent of more powerful electronic tracking methodologies has led to the development of new capabilities that allow us to measure the direct impact of individual articles. Additionally, one can now trace the impact of individual researchers, groups of researchers, research institutions or whole countries through counting the citations of published articles. Importantly, citations can be followed over time to give a reasonable indication of changes in national, institutional or individual productivity.
Our goals in this report are to describe some measures of the current status of Australian mental health research and to propose potential methods for tracking changes in research output over the next decade. Specifically, we describe the research output of nations, Australian states, Australian and New Zealand institutions and Australian and New Zealand researchers, using citation rates derived from two international databases (Institute for scientific information (ISI) Essential Science Indicators (ESIs; [14]) and ISI Web of Science (WOS) [15]).
Method
Research outputs
Citations for countries and institutions were calculated from the ISI ESI, while performance for particular individuals was examined using: (i) ISI ESI; (ii) ISI Highly Cited Researchers indices; (iii) ISI WOS; and (iv) further examination of highly cited papers from WOS.
Institute for Scientific Information indexes academic literature across many scientific disciplines. This indexation forms the basis of the ESI and WOS databases.
ISI ESI
The ISI ESI database provides performance statistics for countries (top 50%; by field of research), institutions (top 1%; based on published author affiliations) and individual authors (top 1%). The ESI database incorporates all ISI databases and ISI-indexed journals. Letters, article corrections and abstracts are omitted and ISI does not include books or book chapters. The ESI data are based on the previous 10 years and the database is updated every 2 months. Citations are grouped according to the area of study. The four ESI field of research categories included in the current manuscript are: ‘psychiatry/psychology’ (which covers 10 topics from applied, diagnosis and treatment, through to social); ‘clinical medicine’ (which covers 25 topics from anaesthesiology through general and internal medicine to urology); ‘neuroscience and behaviour’ (which covers basic and clinical neurology, psychopharmacology, biobehavioural psychology, molecular psychology, cellular and molecular neuroscience, neuronal development and neuronal function underlying higher cognitive processes); and ‘social sciences general’ (which covers 11 topics from communication to public health and administration to social work and social policy) [16].
ISI Highly Cited Researchers
The ISI also publishes information on highly researchers based on ISI citation databases. In the current manuscript, ‘highly cited researchers’ are among the top 250 most cited researchers within a specific research category for the period 1983–2002 (the most recent available data).
ISI Web of Science
The ISI WOS database enables simultaneous searches of three citation indexes: science (which includes 150 scientific disciplines); social sciences (which includes 50 social science disciplines); and arts and humanities (which includes 6800 major science and social science journals). These data describe only the individual citations of authors and citations to specific papers. They provide total citations across all research areas and hence are not restricted to one of the subject areas.
Highly cited papers
All manuscripts identified from the WOS searches for individual authors were collated and ordered by the number of citations. Results are presented for Australian and New Zealand authored manuscripts with 100 or more citations.
Comparison of research outputs
These four measures reflect general research output but vary in the exact measure of research output or the timeframe over which the output is measured. It is, therefore, likely that these measures will perform differently on an individual basis. The differences in outcome for the citation systems will reflect differences in search strategies, categorization of research categories and lack of specificity in linking individuals to specific citations. Additionally, the timeframe for assessment is different. The ISI uses a 10-year (rolling) timeframe for ESI (e.g. citation rates based on papers published during 1992–2002) and a 20-year (rolling) timeframe for Highly Cited Researchers. The WOS database provides the current number of citations for publications dating back to 1980, although other publication timeframes can be selected.
Procedure
We first considered data accessed on 1 November 2003 from ESI that covered the period from 1 January 1993 to 30 August 2003. The top 1500 most cited authors in the scientist world rankings in the category ‘psychiatry/psychology’ were reviewed to identify Australian and New Zealand researchers. This formed the basis of our sample. Authors on this list were contacted to verify their publishing details (including all initials used in publication and author affiliations) and asked if prominent publishers within the field of psychiatry or psychology in Australia and New Zealand had been inadvertently omitted. With considerable cooperation of the many researchers who were identified in the 2003 ESI database, we were then able to track the citations of individual researchers within the WOS for the previous decade. We initially asked authors to search the WOS database for the previous decade (1993–2003) to collect information on the total number of citations from their publications. As citations vary significantly over time (citation databases are updated weekly), and authors ran searches over a total period of 6 months, we repeated all WOS searches on 3 May 2004. The Highly Cited Researchers database was searched to identify Australian and New Zealand authors in mental health related fields and was accessed at the same time. Data on highly cited papers were extracted from the WOS searches.
Population estimates obtained from the Geo Hive global statistics website [17] during November 2003 were used to calculate citations per capita. Figures for England and Wales were retrieved from the Office for National Statistics in the UK [18]. Population estimates for Australian States and Territories were obtained from the Australian Bureau of Statistics' demographic figures for December 2003 [19].
Results
Indicators for countries and institutions
For the decade 1 January 1994 to 7 June 2004, the top 15 countries by citations in the research category ‘psychiatry/psychology’ were identified. Australia was ranked number five. For those 15 countries, we then compared standings for citations per paper and citations per capita. New Zealand and Canada advanced in ranking in these comparisons (Table 1).
Ranking of top 15 countries by total number of citations, citations per paper and citations per capita for the Essential Science Indicator (ESI) research field ‘psychiatry/psychology’ as of 7 June 2004
These data cover the period from 1 January 1994 to 1 April 2004.
With the 2003 ESI country ranking data, we repeated the same exercise for the category ‘neuroscience and behaviour’. Here there is little difference between Australia and New Zealand, but both countries rank significantly below Canada (Table 2). These rankings increased once citations per paper and citations per capita were considered. Within Australia and New Zealand, we then sought to identify the strengths of various states and academic institutions (Table 3). Here, population distribution and institutional resources vary considerably and may have marked effects. Relative to its population, the strength of Victoria in ‘neuroscience and behaviour’ and ‘clinical medicine’ is notable, while the performance of the Australian Capital Territory in ‘psychiatry/psychology’ and ‘clinical medicine’ is also noteworthy. The strengths of the University of New South Wales and Macquarie University in ‘psychiatry/psychology’ are notable, while in ‘neuroscience and behaviour’ and ‘clinical medicine’ the performance of the University of Sydney stands out. Within Victoria the strength of work within the ‘neuroscience and behaviour’ category at the University of Melbourne draws attention.
Ranking of top 10 countries by total number of citations, citations per paper and citations per capita for the Essential Science Indicator (ESI) research field ‘neuro science and behaviour’ as of 1 November 2003
These data cover the period from 1 January 1993 to 30 August 2003.
Total number of citations in the areas of ‘psychiatry/psychology’, ‘neuroscience and behaviour’, ‘clinical medicine’ and ‘social sciences general’ by Australian and New Zealand institutions as of 1 November 2003
† Could also be University of Newcastle, England.
Indicators of performance for individual researchers
The performance for particular individuals was examined using four indices: ESI, WOS citations, Highly Cited Researchers and highly cited papers (from WOS searches). Tables 4 and 5 describe the performance of individual researchers. Essential science indicator rankings based on citations in ‘psychiatry/psychology’ are reported for 2004 in Table 4 while Table 5 reports rankings in 2004 using WOS citations.
2004 Essential Science Indicator (ESI) ranking of Australian and New Zealand academics in the top 1000 in the World by total number of citations
This table includes academics working in Australia and New Zealand as in December 2003. † Academic institution represents current affiliation; ‡Denotes the academic has been identified as a Highly cited researcher in the category of ‘psychiatry/psychology’; §Denotes the academic is female. ANU, The Australian National University; Monash, Monash University; MQ, Macquarie University; QCF, Queensland Cancer Fund; QIMR, Queensland Institute of Medical Research; UMelb, University of Melbourne; UNSW, University of New South Wales; UOtago, University of Otago; UQ, University of Queensland; USyd, University of Sydney; UWA, University of Western Australia; UWS, University of Western Sydney.
2004 Web of Science (WOS) ranking of Australian and New Zealand academics by total number of citations
This table includes academics working in Australia and New Zealand as in December 2003. †Academic institution represents current affiliation; ‡N/A indicates the academic was not in the top 1500 in the World according to ESI; §Denotes the academic is female; ¶<30 indicates the academic was in the top 1500 in the World according to ESI (but not the top 1000). ANU, The Australian National University; Griffith, Griffith University; Monash, Monash University; MQ, Macquarie University; QCF, Queensland Cancer Fund; QIMR, Queensland Institute of Medical Research; UAdelaide, University of Adelaide; UMelb, University of Melbourne; U New, University of Newcastle; UNSW, University of New South Wales; UOtago, University of Otago; UQ, University of Queensland; USyd, University of Sydney; UWA, University of Western Australia; UWS, University of Western Sydney.
ISI ESI rankings
Importantly, no Australian researcher appears in the top 100 of world rankings, though one New Zealander does (DM Fergusson; Table 4). In 2004, 12 Australian researchers appeared in the top 500 world-ranked researchers and 26 were in the top 1000. Of the top 30 researchers in 2004, 10 were from universities in New South Wales, five were from The Australian National University, five were from universities in Victoria and four from institutions in Queensland. From the University of New South Wales, six researchers were associated with the Mood Disorders Unit; and from the Australian National University all five listed researchers were part of the Centre for Mental Health Research. A small number of researchers were not only listed in the ‘psychiatry/ psychology’ ESI category in 2004, but were also significantly listed in the ‘social sciences general’ or ‘clinical medicine’ categories.
ISIWeb of Science rankings
This led to the generation of 20]), epidemiology and epidemiological methods (e.g. Ormel et al. −283 citations [21]), cognitive sciences (e.g. Coltheart et al. −464 citations [22]), phenomenology (e.g. Yehuda and McFarlane −171 citations [23]) and risk factor research (e.g. Jones et al. −337 citations [24]) but less impact in clinical trials or clinical neuroscience.
Highly cited individual researchers for the last 20 years
The Australians who appear in this list for the category ‘psychology/ psychiatry’ are AF Jorm, HW Marsh, G Parker and for ‘social sciences general’ WD Hall.
Discussion
This presentation of citation rates for the last 10 years provides some important insights into the current national and international standing of Australian mental health research. Before over interpreting between and within country differences, the amount of resource investment in research and many other contributing factors need to be considered. The Wills report [1] noted ‘With 0.3% of the world's population, Australia produces about 2.5% of the world's health and medical research output’ (p.5). However, ‘In 1995, Australia spent 0.115% of GDP or $28 per capita on health and medical research and development. The GDP-weighted OECD average for developed nations was significantly higher at 0.174% or $66 per capita’ (p.7). For that period, public expenditure on health and medical research was highest in Switzerland (0.315%), the US was mid-ranking (0.211%) and New Zealand was lowest (0.051%).
More recent evidence [25] suggests that Australia's total public expenditure in health-related R&D was 0.12% of gross domestic product during 2002 and that the comparisons with other countries had not changed significantly. Of the four major Organization for Economic Cooperation and Development countries, public expenditure was highest in the US (0.25%), France was mid-ranking (0.20%) while the UK and Germany were relatively low (0.13% and 0.11%, respectively) [25]. Research investment in mental health in Australia (both public and private expenditure) is low by comparison with other major health areas (2000–2001: mental health comprised around 3% of total expenditure, compared with cancer at 9% and pharmaceutical treatments at 8%) [26].
Although citation systems are only one measure of research output, there is evidence that they do relate to real-world health impacts. For example, in a recent review of the most cited articles ever published in the Medical Journal of Australia, Dr John Cade's 1949 description of the therapeutic effects of lithium carbonate led the field (888 citations) [27]. Consequently, having taken account of the resources, issues and Australia's generally high ranking in overall medical research performance, several comparisons in mental health are notable.
First, on a number of key measures we do not perform as well as comparable countries such as Canada or New Zealand. Given the relatively low level of national investment in research, the very high impact of a small number of New Zealand researchers should be the subject of active discussion. Their performance may simply reflect the quality of those individuals or it may reflect their capacity to pursue long-term, high value and internationally collaborative projects.
There is a high degree of variability in performance between our major research institutions, and performance in mental health research is not strongly tied to the overall scientific performance of the home institution. Instead the major factors appear to be the strength of some individual researchers, their national or international networks, and/or the strength of certain consolidated research groups over a sustained period. Among the individual researchers, NG Martin, at Queensland Institute of Medical Research working in association with the Australian Twin Registry and his network of international genetic and genetic epidemiology colleagues is notable. Other notable performers include M Coltheart and RM Rapee working in cognitive sciences and psychology at Macquarie University and RA Bryant in psychology at the University of New South Wales. The consolidated research groups of AF Jorm and colleagues at the Centre for Mental Health Research, the Australian National University (H Christensen, AE Korten, B Rodgers), G Parker and his colleagues at the Black Dog Institute (formerly the Mood Disorders Unit), University of New South Wales (P Mitchell, K Wilhelm, D Hadzi-Pavlovic, previously P Boyce and I Hickie) and PD McGorry at the University of Melbourne and ORYGEN Youth Health (HJ Jackson, C Pantelis) stand out.
Although it may be argued that consolidated research groups simply inflate citation rates through double counting (as each author receives credit for each paper published), it is important to note the extent to which the contributions of these groups have also been recognized through other mechanisms (e.g. NHMRC Program Grant awards) or the extent to which they have led important international movements. Moreover, the citations of consolidated research groups reflect the collective work of younger researchers and PhD students, whose performance reflects the capacity-building associated with these research centres. For the Centre for Mental Health at The Australian National University, stand-out publications have highlighted epidemiological aspects of ageing, mental health literacy, e-health services and the lead researchers (AF Jorm, H Christensen) have high citation per paper ratios. Similarly, for the University of Melbourne and ORYGEN Youth Health team stand-out publications (and high citations per paper) have highlighted the effects of clinical interventions in early or prodromal psychosis, as well as the neuroimaging changes that accompany transition to psychosis (PD McGorry).
The extent to which our research efforts might be more strategic and obviously responsive to population need or community expectations needs to be articulated [2], [3]. The most recent overview of Australian mental health research highlighted important differences between current investigator-driven activity and both community expectation and population health burden [28], [29]. Consequently, other potential markers of success now need to be considered. At the input end we might consider more closely responsiveness to community priorities, targeting of population-health burden, consumer and carer participation and broadening of non-government agency support. From an output perspective, factors such as impacts on population health or health service delivery are deserving of greater attention. Development of specific metrics for each of these constructs is challenging. Some key processes such as the NHMRC Program Grant assessment procedures have attempted to operationalize these perspectives, placing weight not only on quantitative measures of productivity but also on peer-assessed notions of international and national impact [30]. Indirect measures of public health penetration or impact in the community might be achieved through Internet searches. These might reveal the extent to which applications have been picked up and transmitted beyond academic research publication.
Can we set realistic goals for Australian mental health research? In our view, we should at least aspire to match New Zealand and Canada in terms of per capita citation rates for our key specialty areas. Both these countries appear to derive great benefit from their international collaborative programs. Second, we need to back those individuals, and those groups of researchers, that add real value to the Australian research environment. Some have access to unique resources (e.g. Australian Twin Registry and international twin collections) while others have pursued internationally leading paradigm shifts in epidemiology, cognitive sciences, mental health literacy, e-health, interventions in first episode psychosis or primary care psychiatry.
We need to point to the limitations of our empirical data. Citation tracking exercises are fraught with difficulties [31]. In the ESI system particularly, there are issues related to a lack of capacity to distinguish between common names, variability in use of first and second initials (e.g. C Nemeroff vs CB Nemeroff) and allocation of researchers and their citations to various categories. For example, one of Australia's most productive researchers WD Hall, is listed as a citation classic in ‘social sciences general’ but not ‘psychiatry/psychology’. Further, the citation rates vary considerably between categories with citations for those who appear in ‘psychiatry/psychology’ generally being much lower than those who appear in ‘clinical medicine’. Individual researchers may not be highly ranked in the ESI if their work is shared between subject areas. Nevertheless, we maintain that these citations are of value but may best be pursued by an independent body working in close association with active researchers. We would suggest that this process be conducted on a regular basis (e.g. biannually) and overseen either by an appropriate collegiate body such as the Australasian Society for Psychiatric Research or an administrative arm of our National Mental Health Strategy.
Footnotes
Acknowledgements
We thank the Australian and New Zealand researchers in mental health for providing information and assistance for this project.