Abstract
Keywords
In a companion paper, we have compared the amount of research on various mental disorders in Australia with the disease burden and health system costs of these disorders [1]. A limitation of this approach is that it is disease focused; although it is possible to use burden and cost data to rank diseases for research priority, such data are not so easily applied to other dimensions of research (e.g. the setting, methodology or research topic).
Another approach to setting priorities involves consulting experts or other stakeholders. A number of methods have been used for consulting stakeholders, including consensus workshops of experts [2, 3], focus and Delphi groups of clinicians [4], surveys of service providers [5], surveys of researchers [6, 7], focus groups of research consumers [8], and surveys of other stakeholders such as politicians and the general community [9]. This research has focused on the priorities of a limited range of stakeholders. Moreover, to our knowledge, there is no published evidence concerning stakeholder priorities across a range of mental health conditions, and with respect to different topics, settings and target groups.
The present paper compares the distribution of mental health research with the priorities rated by a range of stakeholder groups. Ten different stakeholder groups were surveyed including researchers, committees that evaluate research, clinical service providers (general practitioners, psychiatrists, clinical psychologists and mental health nurses), members of the National Mental Health Working Group, consumers of mental health services and carers. Stakeholders were surveyed about types of mental disorders which should be covered, the topic of the research, the setting in which the research should be carried out, and the desirability of including various subpopulation groups as the focus of the research. We also asked stakeholders about the sorts of factors that should be used to guide research priorities and their sources of information about mental health research. We were particularly interested in areas of research that all stakeholder groups rated as a high priority, but also in any major differences in priorities between stakeholder groups.
Method
Content analysis of current research
The project analysed the content of current research by examining a year's worth of published articles in mental health and a year's worth of competitive grant funding. We analysed the content of publications and grants according to four different schemes:
The type of mental disorder being investigated (ICD-10 code). The goals of the research, for example, prevention, service evaluation, genetics, prevalence, risk factors. The participant type and setting where the research was carried out, for example, members of the community, patients in primary care, patients in specialist care, students. The inclusion of special interest groups in the research, for example, indigenous peoples, children and adolescents, older people, people from rural and remote areas.
The classification scheme was applied to a year's worth of publications and a year's worth of competitive research grants. The publications and grants were identified using methods described in a companion paper [1].
Stakeholder questionnaire
We devised a questionnaire on research priorities which was based around the scheme used for classifying current research activity. The questionnaire asked respondents to rate priorities for various mental disorders, settings in which research can be carried out, topics of research and subgroups of the population. Because many of the categories of research involve technical terms, respondents were also provided with a lay definition or some examples, such as, ‘animal models (i.e. researching the equivalents of human mental disorders in laboratory animals)’. Priorities were rated on a scale from ‘very low’ (assigned a value of 1) through to ‘very high’ (assigned a value of 5).
The stakeholder questionnaire next presented a section which began: ‘When rating research priorities for mental disorders, people take various factors into account, such as how much the disorder costs, how much suffering it causes, etc. Please rate the following factors for importance to you in setting priorities for research in Australia’. These instructions were followed by a list of 23 factors to be rated on the fivepoint scale from very low to very high. The 23 factors to be rated are shown in abbreviated form in Table 9. To construct the items for this section of the questionnaire, the authors compiled an initial list of items and then asked researcher colleagues to identify any factors that were omitted. Items were then added to cover these factors.
The stakeholder questionnaire concluded with a section which began: ‘A lot of mental health research is carried out in Australia. How do you find out about what research is being done? Please rate the importance to you of the following sources of information about mental health research in Australia’. These instructions were followed by a list of 20 sources to be rated on the same 5-point scale. The 20 sources were: academic text books; being told by family and friends; being told by health professionals; federal, state or territory departments of health; electronic and other databases (e.g. PubMed & PsycLit); free leaflets and booklets from pharmacies, health food shops, etc.; magazines; newspapers; organizations for consumers or carers (e.g. newsletters); personal contacts with researchers; pharmaceutical advertising; professional or academic journals; professional or academic conferences; professional or academic training courses; radio programmes; self-help books; the internet; television documentaries; television news; videos.
In case the order of the items had an influence on the ratings made, two versions of the questionnaire were used, one of which had items in alphabetical order and the other in reverse alphabetical order. Respondents were randomly sent one or the other.
Stakeholder groups surveyed
We used the following procedures in order to sample the various stakeholder groups:
Researchers were sampled from the list of senior authors of all the mental health research articles published in 1998.
Committees that evaluate research grants were the members of the National Health and Medical Research Council (NHMRC) Mental Health Grant Review Panel and the Research Committee of the Australian Rotary Health Research Fund for 2000.
National Mental Health Working Group consisted of senior Commonwealth, state and territory mental health administrators.
General practitioners and psychiatrists were sampled from the list of medical practitioners held by the Health Insurance Commission.
Clinical psychologists were sampled from the membership of the College of Clinical Psychologists of the Australian Psychological Society.
Mental health nurses were sampled from the membership of the Australian and New Zealand College of Mental Health Nurses.
Consumers and carers were the most difficult to sample because there is no central list. Furthermore, many consumers and carers would have little familiarity with Australian research. We therefore decided to focus on consumer and carer advocates. Even for advocates there was no single organization whose membership could be used as a sampling frame. Instead we contacted consumer and carer members of the board of the Mental Health Council of Australia (MHCA) and members of consumer and carer advisory groups (CAGs) set up by state and territory governments. Because the numbers of consumers and carers in these organizations were small, we used a snowball technique to expand the sample. Each of the seven MHCA consumer and carer representatives was asked to provide details of consumers or carer advocates whom they believed would be suitable participants for the study. In addition, the MHCA consumer and carer representatives were asked to identify three key people from the list who had developed extensive consumer and/or carer networks and whom we could approach to provide contact details of potentially suitable participants. One of the MHCA carers requested that state Associations of Relatives and Friends of the Mentally Ill (ARAFMI) organizations provide contact details to us directly. Two secondary carers/consumers participants contacted us and volunteered details of potential participants from their organization/network. All consumers and carers identified by the above process were included in the sample.
Questionnaires were sent to a random sample of 200 persons from each of the lists of researchers, general practitioners, psychiatrists, clinical psychologists and mental health nurses. Questionnaires were also sent to all 11 members of the NHMRC Mental Health Grant Review Panel, all 10 members of the research committee of the Australian Rotary Health Research Fund (ARHRF) and all nine members of the National Mental Health Working Group (NMHWG). For consumers and carer advocates, questionnaires were sent to 83 people associated with consumer/carer advisory groups and 179 others identified through our snowball sampling procedure. The response rates from each of these groups were calculated as: number responded/ (number sampled – number not contactable). The response rates (and achieved sample sizes) were: researchers 56% (n = 100), NHMRC panel 91% (n = 10), ARHRF committee 80% (n = 8), NMHWG 67% (n = 6), general practitioners 30% (n = 58), psychiatrists 53% (n = 105), clinical psychologists 53% (n = 106), mental health nurses 60% (n = 120), consumers/carers (from CAGs) 72% (n = 59), and consumers/carers (other) 64% (n = 114).
Statistical methods
In interpreting the data, we have examined the three categories in each section of the questionnaire rated highest for each stakeholder group. However, these mean ratings are estimates which are subject to sampling error. To indicate the precision of the estimates, we also report standard errors for stakeholder survey results. Standard errors of the tabulated means are chiefly determined by the size of the group, from six in the National Mental Health Working Group to 120 nurses. However, the underlying standard deviation of the distribution of priority ratings is not constant, but becomes smaller as the mean approaches either end of the 1–5 priority scale. This function was approximated by a quadratic curve fitted by weighted regression to the observed standard deviations in the cells of each table, and the fitted standard deviations were used to derive the standard errors tabulated. This provides more accurate estimates of the standard errors than would result from using a single ANOVA-based error estimate.
The statistical significance of the differences in ratings was tested in each table using a mixed linear model, in which the correlation between scores from the same individual respondent was modelled using an additive random effect. The analysis was done using the ‘xtreg’ program in the package STATA [10].
With the ratings of factors that should be taken into account in setting research priorities, we used principal component analysis to reduce the 23 items to a smaller set of dimensions. A scree plot was used to determine the number of factors retained for varimax rotation. Principal component scores were calculated using the regression method.
Results
Priorities for research on various categories of mental disorder
Table 1 shows the breakdown of current research on categories of mental disorder. Substance use disorders receive the most publications, followed by affective and anxiety disorders. Substance use disorders also receive the most competitive funding, followed by childhood conditions and affective disorders and dementia. Table 2 shows the stakeholders' ratings for the same categories. It can be seen that some stakeholder groups tend to give overall higher ratings than others (e.g. the ARHRF committee compared to psychiatrists). Therefore in interpreting the ratings, we have emphasized the rank order of ratings in each category rather than the absolute values. Considering the top three rating categories for each stakeholder group, affective disorders rank consistently highly, as do suicide and schizophrenia to a lesser extent. However, for other categories there are various degrees of disagreement. Substance use disorders are ranked highly by researchers, general practitioners and the NMHWG and childhood conditions are ranked highly by researchers, psychiatrists, clinical psychologists and consumers. The NHMRC panel stands out for its high ranking for dementia.
Percentage of research carried out on various categories of mental disorder
Mean priority ratings by stakeholder groups for research on various categories of mental disorder
Priorities for research in various settings
Table 3 shows the distribution of research in community, primary care and specialist settings. Most research is carried out in specialist settings, with less done in the community and very little in primary care. Specialist settings also receive a considerably greater allocation of grant funds than community and primary care research. Although we coded research carried out in other settings (e.g. nursing homes, schools), these were not included in the priorities questionnaire because they would simply reflect priorities asked about elsewhere in the questionnaire (e.g. older people, the physically ill, children and adolescents). Table 4 shows the stakeholders' priorities for the three main settings: ‘patients in hospitals or other specialist settings’, ‘patients in primary care (e.g. seen in general practice)’, and ‘people in the general community’. No stakeholder group allocated their highest rating to research in specialist settings. The other stakeholder groups were fairly evenly divided in assigning their top ranking to community and primary care settings. The difference in mean priority between the latter two settings was minor except for the NMHWG, which rated primary care substantially higher than community settings.
Percentage of research carried out in various settings
Mean priority ratings (standard error) by stakeholder groups for research in various settings
Priorities for research on various topics
Table 5 shows the distribution of current research according to various topics. The topics currently receiving the most publications are risk factors and associations, assessment and classification, and description of disorders, whereas the topics receiving the most funding are genetic causes, risk factors and physiological and anatomical differences. Table 6 shows the stakeholders' priorities for the same topics. The topic rated highly with most consistency is prevention and promotion. It is notable that this topic accounted for only 1.9% of the articles. However, the funding of prevention and promotion research was higher at 8.8%, perhaps reflecting a greater emphasis on this area in recent years and the impact of the Mental Health Promotion and Prevention National Action Plan [11]. Psychological and social treatments were ranked highly by researchers, the NHMRC panel, the ARHRF committee, psychiatrists, clinical psychologists and consumers. Such treatments are one of the bigger categories of current research, being covered by 8.7% of articles and 9.5% of funding. Service evaluation was ranked highly by researchers, general practitioners, clinical psychologists, mental health nurses, consumers and the NMHWG. It is perhaps not surprising that most provider groups and consumers of services see this topic as important. Service evaluation accounts for 5.6% of the articles and 4.7% of the grant funding, making it a moderate area of research activity. The final topic rated highly by a number of stakeholder groups was training and education of mental health professionals. This topic was ranked highly by the NMHWG, general practitioners, psychiatrists, nurses, consumers and carers. This ranking may reflect dissatisfaction with current training. Training and education is currently only a minor topic of research, accounting for 2.4% of articles and 1.5% of grant funding. Finally, it is notable that drug treatment research was rated as a very high priority by carers but not by other stakeholder groups. Drug treatments accounted for a moderate percentage of articles (6.4%) but only 1% of competitive funding. The discrepancy between articles and competitive funding is probably because most drug treatment research is carried out with noncompetitive industry funding.
Percentage of published research and competitive grant funding on various topics
Mean priority ratings by stakeholder groups for research on various topics
Priorities for research on various subgroups of the population
Table 7 shows the amount of research involving various subgroups. It can be seen that the most researched subgroups are children and adolescents (13.5% of articles and 21.4% of funding), those with physical illness or injury (7.1% of funding and 8.1% of articles), and older people (6.8% of articles and 14.2% of funding). Table 8 shows the stakeholders' priority ratings for the same subgroups. The groups most consistently rated highly are children and adolescents, Aboriginal and Torres Strait Islander peoples, and the socially and economically disadvantaged. The latter two groups are not currently well researched. Aboriginal and Torres Strait Islander peoples currently account for only 1.1% of articles and 1.6% of funding, while socially and economically disadvantaged people account for 0.7% of articles and 0.5% of funding.
Percentage of research carried out on various subgroups of the Australian population
Mean priority ratings by stakeholder groups for research on various subgroups of the Australian population
Factors that should be used in setting priorities
Table 9 shows the mean importance ratings for each factor which might be taken into account in setting research priorities. The factors considered to be important by most of the stakeholder groups were suffering to the individual affected, the potential for the research to make a difference to people's lives, and whether the disorder affects people for most of their life. There were also some notable differences in ratings between the stakeholder groups. The NHMRC Panel rated as important the quality of the research being done on the topic and the potential for research to advance scientific knowledge. The Rotary Committee showed a similar trend. However, these were not rated as major factors for any of the other stakeholder groups. Another interesting difference is that the consumers rated as important the involvement of consumers in planning the research. The carers showed a trend in the same direction, but this factor was not rated highly by any of the other stakeholder groups.
Mean ratings by stakeholder groups for importance of various factors in setting priorities
In a principal component analysis of the importance ratings, three clear components emerged. The first was labelled Personal Impact and had high loadings (< 0.4) on the following items: financial cost to the individual affected, involvement of mental health consumers in the planning of the research, involvement of mental health carers in the planning of the research, suffering to family and friends of the person affected, suffering to the individual affected, the amount of contact you have with people who have this disorder, and the extent to which you personally or those close to you have been affected by the disorder. The second component was labelled Scientific Quality and had high loadings on: fortuitous opportunities for research, high quality of the research being done on the topic, how little research has already been done on the topic, new opportunities for research from technological advances, productivity of the researchers working on the topic, and the potential of the research to advance scientific knowledge. The third component, Societal Relevance, had high loadings on: how common the disorder is, suffering to family and friends of those affected, suffering to society as a whole, whether the disorder affects people early in life, whether the disorder affects people for most of their life, whether the disorder causes premature death, and whether the disorder is becoming more common. Mean scores on these principal components were compared across the stakeholder groups. For the Personal Impact component, the consumers and carers scored high, while the researchers, committees that evaluate research and the NMHWG scored low. On the Scientific Quality component, the committees that evaluate research scored much higher than any of the other groups, including the researchers. The Societal Relevance component showed the smallest difference between the groups.
Importance of sources of information about research
Table 10 shows the mean importance ratings for the 20 sources of information. It can be seen that some groups tended to give much lower ratings overall (e.g. general practitioners). For this reason, the data have again been interpreted using the top-rated sources for each group, regardless of the actual magnitude of the ratings. The top three information sources for each group are shown in bold in the table. All of the professional groups (clinicians, researchers and administrators) included journals and conferences in their top three sources. As well, all of the clinician groups had training courses in their top three sources, while the researchers and committees that evaluate research included electronic and other databases among their three most important sources. The administrators (NMHWG) were the only group to have departments of health in their top three sources. The consumers and carer groups differed from all the others, ranking organizations for consumers and carers the highest as a source of information, followed by television documentaries and being told by health professionals.
Mean ratings by stakeholder groups for importance of various sources of information about research
Discussion
Topics of research
Different groups of stakeholders tend to have differing perspectives on research priorities. However, there are a number of areas of agreement. Affective disorders and suicide were consistently rated as a high priority. However, these are not currently receiving the major research attention. When various research settings were rated, primary care and community research was consistently seen as a higher priority than research in specialist settings. By contrast, much more research is currently carried out in specialist settings than in primary care or the community. When topics of research were rated, stakeholder groups largely agreed about the importance of research on prevention and promotion, psychological and social treatments, and evaluation of mental health services. Similarly, consumers, carers, clinicians and senior mental health administrators rated research into the training and education of mental health professionals as a high priority. With the exception of psychological and social treatments, these topics are currently minor areas of research. When subgroups of the population were rated, there was agreement that children and adolescents, Aboriginal and Torres Strait Islander peoples, and socially and economically disadvantaged people are priorities. However, there is currently very little research on the latter two groups.
Factors considered in setting priorities
There were a number of factors that most of the stakeholder groups agreed were important in setting research priorities. However, more interesting were the substantial differences. The groups that have the most power to set priorities are the committees that evaluate research and they saw scientific quality as being very important. They accorded a higher level of importance to scientific quality than any other group, including the researchers. By contrast, the people who are expected to be the ultimate beneficiaries of research, the consumers and carers, wanted research priorities to be guided by the impact on the individual. The desire for individual impact to be taken into account is hardly surprising, but stands in contrast to the views of researchers and committees that evaluate research, who rated individual impact much lower. The least disagreement was seen in ratings about the importance of taking societal relevance into account.
Tobin and Hickie have argued for a greater partnership between academic research and clinical services [12]. Our findings imply that this partnership needs to be broadened to include greater dialogue between those who carry out and evaluate research on the one hand and consumers and carers on the other [13]. There is a trend for consumers and carers to desire more involvement as partners in research as well as in services, as encapsulated in the slogan ‘Nothing about us without us’. If there is to be greater partnership, then researchers will need to better communicate the need for scientific quality to consumers and carers. On the other hand, if the desires of consumers and carers are to be met, there may be a need to modify the criteria for evaluating grant applications, so that the practical relevance of the research is given some weighting in addition to scientific quality.
Sources of information about research
When interpreting stakeholders' priority ratings, it is useful to know where they get their information about mental health research. Stakeholders may have varying degrees of knowledge about what is currently happening in Australian mental health research. There has been very little previous research on information sources for mental health professionals [14] and, as far as we can ascertain, none at all on mental health researchers, consumers and carers. We found that the biggest gulf in sources of information is between various professional groups on the one hand and consumers and carers on the other. While traditional sources of research dissemination, such as journals and conferences, are important for professionals, these do not reach consumers and carers. Because consumers and carers are the ultimate beneficiaries of research knowledge, and their political support is important for securing better research funding, it is vital that they be kept informed about Australian research. The consumers and carers allocated the highest importance rating to consumer and carer organizations as sources of information. These organizations may need more attention from researchers as a means of information dissemination additional to the traditional ones.
While electronic databases such as PubMed and PsycLit are important to researchers, the internet is not yet a major source of research information for any group. Nevertheless, it is undoubtedly a growing source of information for both professionals and the public [15]. All stakeholder groups gave only low to medium importance ratings to the internet, but for most of the groups it was rated higher than traditional mass media including newspapers, magazines, TV news and documentaries. The ratings given to the internet must also be seen in context that the Web barely existed a decade ago and would at that time have received ratings of ‘very low’ for all groups.
Limitations
Some limitations of using stakeholder opinions to assess priorities must be acknowledged. The stakeholders we surveyed represent the interests of some disorders better than others. The best example is dementia. Dementia is not a well researched area when compared to its burden and health system costs [1]. However, it was not rated among the highest priorities by most stakeholder groups. This may be because dementia is not commonly dealt with by psychiatrists, clinical psychologists or mental health nurses. Nor are the consumer or carer groups we surveyed generally concerned with dementia. A different picture might have emerged if we had surveyed geriatricians, gerontic nurses and members of the Carers Association.
Another limitation is that stakeholder perceptions may be influenced by their expertise and knowledge about mental health research. For example, Sanson-Fisher et al. [9] found that politicians and community members were influenced by false perceptions that illicit drugs are a bigger health problem than tobacco and alcohol. It is possible that differences in priorities among stakeholder groups reflect, in part, varying degrees of knowledge about the current state of Australian mental health research. Researchers are likely to have a greater knowledge than clinicians or consumer and carer advocates, particularly about opportunities for rapid scientific advances. Some of the stakeholders may also have had trouble distinguishing priorities for research from priorities for service provision. Although the two sets of priorities may correspond, they do not necessarily have to.
A final limitation of the findings is that different stakeholder groups may allocate a high rating to the same research area for different reasons. For example, some health practitioners may advocate additional service research because they believe that it will demonstrate the effectiveness, or relative effectiveness, of particular services, whereas consumers and carers may believe that service research is necessary in order to highlight the inadequacies of current service provision.
Conclusion
Despite these limitations, there are a number of areas that the diverse stakeholder groups agree are high priorities, but where current research activity is low. These clearly merit greater attention from mental health researchers.
Acknowledgements
This research was funded by a consultancy from the Mental Health and Special Programs Branch of the Commonwealth Department of Health and Aged Care. We thank the staff of the Department for their help with the project.