Abstract
Objective
To describe in detail the methods followed in each of the qualitative and quantitative surveys of national health information, research and knowledge systems and research institutions.
Design
Cross-sectional surveys.
Setting
National health information and research systems, and 847 health research institutions in 42 countries in the World Health Organization (WHO) African Region.
Participants
Key informants from health research institutions, ministries of health and statistical offices.
Main outcome measures
Stewardship, financing, ethics, human and material resources and output of health information and research systems.
Results
Key informants were used to collect data to assess national research systems in 44 countries in the Region. The same method was followed in assessing national information systems in 17 countries and knowledge systems for health in 44 countries. These assessments included a detailed review of the state of data sources in the Region and their effect on measuring progress on the health-related Millennium Development Goals. A concurrent survey employed a structured questionnaire (the WHO Health Research Systems Analysis Questionnaire) to assess the capacity of 847 health research institutions in 42 countries. Stewardship, financing, ethics, human and material resources and research output were assessed.
Conclusions
The logistics and resources involved in surveying the knowledge landscape in the 42 countries were substantial. However, the investment was worthwhile as the results of the surveys can be used to inform policy-making and decision-making, as well as to establish a regional database of national health information, research and knowledge systems.
Keywords
Introduction
A health research system has been defined as the people, institutions and activities involved in the generation and dissemination of knowledge. The health research system is an integral part of the health system. It should produce research findings that are synthesised into high-quality information and knowledge, and inform the development and strengthening of the health system. A systematic assessment of existing health research and knowledge systems will inform the development of policies and strategies to strengthen national health research systems. Such analyses were conducted and covered issues of governance/stewardship, ethical oversight, financing, resource inputs, outputs, dissemination and impact of national health research findings.
A health information system (HIS) has been described as an integrated effort to collect, process, report and use health information and knowledge to influence policy-making, programme action and research. There have not been comprehensive reviews of the state of national HISs in the World Health Organization (WHO) African Region, particularly the state of their inputs (resources), processes (selection of indicators and data sources; data collection and management) and outputs (information products; information dissemination and use).
Knowledge directly affects human health, and an effective health system needs effective knowledge management. Knowledge management, in this context, includes how to apply known interventions when resources exist, how to allocate limited resources for these known interventions according to identified priorities, and how to secure additional resources. These issues are of direct relevance to the African Region, where the know-do gap is mostly a consequence of inadequate capacity to apply existing knowledge. This may include, among other things, insufficient trained human resources, networks and communities of practice, and infrastructure. Knowledge management provides a set of methodologies and tools to overcome precisely these challenges. There have been no studies that assess the role of knowledge in the health system.
Major areas of enquiry, type of survey and target population included in the various surveys, 2009.
Methods
Surveying national health research systems
In September 2006, the Regional Office asked each WHO Country Office representative to designate a country focal person for health information, research and knowledge management. In December 2006, these focal persons from 46 countries of the Region were invited to a workshop in Brazzaville, Congo, for orientation on the Regional Office’s activities on health information, research and knowledge management, and for training on the main tools that were to be used for collection of data during the surveys.
The health research surveys employed two tools, focusing on the national and institutional levels. The national-level survey used a tool originally employed for a similar survey in the Region.
2
The institutional survey employed a tool developed by WHO as part of its Health Research System Analysis Initiative.3,4 The survey on knowledge systems for health used two survey processes (all tools used for the collection of data are in online supplementary material):
One in which key data regarding knowledge systems were collected from international data sources, the results of which are not included in this report A second in which a questionnaire-based survey was conducted to collect data about the use of knowledge in health policy and clinical services provision
The Health Research System Analysis Initiative tools included questions grouped into seven modules.
5
Each module covered a specific topic:
The tool was initially pilot tested by WHO in 10 low- and middle-income countries, including three African countries (Cameroon, Senegal and the United Republic of Tanzania). The tools were translated into French and Portuguese before their use in the surveys. Questionnaires in English were sent to institutions in 19 countries, in French in 19 countries and in Portuguese in four countries.
The main criterion for considering an institution as a ‘health research institution’ was that it should be engaged in the conception or creation of new knowledge, products, processes, methods and systems related to any aspect of health, such as factors affecting health and ways of promoting and improving it. Institutions could be departments of medical schools, universities, teaching or non-teaching hospitals, independent research institutions, governmental agencies, pharmaceutical and other for-profit and not-for profit businesses, charities and non-governmental organisations.
The WHO initially created a database of research institutions in the Region from various sources in order to build a reasonably comprehensive inventory. Sources used included:
An Internet search International directories such as Thomson Reuters Web of Knowledge, the Nature Yearbook and the World Higher Education Database The Institute for Scientific Information databases (Science Citation Index, Social Science Citation Index, Art and Humanities Citation Index) An international health research journals subscription database WHO sources such as the Health InterNetwork Access to Research Initiative (HINARI) Phase 1 and 2; WHO collaborating centres; WHO research activity database; WHO Medical Schools Directory; the Global Forum for Health Research, Forum 4 and 6 participants; the WHO master mailing list, expert advisory panel International funders, including the USA National Institutes of Health grants and awards to foreign countries data Existing data and directories of ministries of health, science and technology at the national level Distribution by country of the 1882 institutions identified as a sampling frame for the survey (graph on the left) and percentage of institutions that responded to the survey (
The list of health research institutions corresponding to a particular country was then sent to the country focal team to verify and confirm whether the listed institutions still existed and were operational, and whether they had carried out research on health or health-related topics in the past five years. A regular exchange with the coordinating team at the Regional Office to rectify discrepancies and inconsistencies, and add or remove relevant or irrelevant institutions, eventually led to a final list of all institutions conducting health research in countries, totalling 1882 institutions in the 46 countries of the Region. Of these, a total of 847 institutions conducting health research in 42 countries were included in this report. The number of institutions in each country that responded to the questionnaire ranged from 2% in Rwanda to 100% in Guinea Bissau. Overall, 45% of the institutions listed on the sampling frame responded to the survey (Figure 1).
It was decided to include all of the research institutions in the survey, thus this was more of a census than a survey. This decision was taken because it was envisaged that national and regional databases of health research institutions could be created that could later be used by national groups and the Regional Office to formulate national policies and strategies to strengthen national health research systems. Regular monitoring and evaluation of national health research systems would also be possible by establishing national and regional observatories based on such exhaustive data.
Characteristics of health research institutions in 42 sub-Saharan African countries, 2009.
aNumber of respondent health institutions, out of 847 surveyed.
The modules were designed to be self-completed and involved several individuals working together within an institution to gather and discuss the responses. The modules were sent to heads of institutions by email or fax or given to them personally by the consultant. Each module was designed to be filled out by the appropriate specialist in the institution. The head of the institution had the responsibility of designating the appropriate specialist to fill in the relevant module and noted their names on the last page of the first module. In most questions, the survey focused on explicit policies and actions within the institution, rather than the opinions of the respondent.
Once collected, data were entered at the country level and shared online using DataCol (the WHO data entry program), which was easily accessible to WHO Country Offices via the Internet. Each country focal point had full access to the data that they entered and could make modifications in case of errors. The coordinators checked the data on a daily basis for completeness and consistency and, in consultation with country focal points and national consultants, made the necessary modifications. After the completion of this process, the data were exported and processed in a Microsoft Excel format and were available for further analysis in
Surveying national HISs
The national HIS surveys were managed and carried out by the WHO African Regional Office and the Health Metrics Network. The assessment focused on a comprehensive approach to HIS described in the Health Metrics Network’s and WHO’s At the country level, it aims to focus investment and technical assistance for the standardised development of HIS and serves as a baseline for HIS assessment and diagnosis, a roadmap for the development of HIS plans, and provides for ongoing monitoring and evaluation At the country and global levels, it enables access to and use of better health information
The HIS assessment questionnaire comprised about 200 items organised in components: resources, indicators, data sources, data management, information products dissemination and use (see online supplementary material). The assessment tool was in a Microsoft Excel format and key informants discussed and scored each item. The results of the assessment of each component were generated as tables and graphs. These results were discussed and shared with stakeholders.
The stakeholder group and its steering committee designated an existing agency, such as the HIS section or unit of the ministry of health, to carry out certain administrative tasks (e.g. communications and procurement) required to conduct an assessment. The assessments were carried out in the setting of a workshop or meetings of several groups, or with individual interviews of key informants, or a combination of all these for soliciting inputs from all key stakeholders.
The two important mapping steps in the HIS assessment were the mapping of all stakeholders, partners and donors that were willing to strengthen their HIS, and the mapping of all the components of the HIS in order to develop consensus on priority weaknesses. The list of stakeholders included producers, users and financers at different levels (subnational versus national) of health information and other social statistics.
Countries have included in the assessment:
Representatives of bilateral donors and organisations of the United Nations system active in development and in monitoring progress towards the Millennium Development Goals such as United Nations Children’s Fund, United Nations Development Programme, United Nations Population Fund and WHO The World Bank Representatives of key non-governmental organisations and civil society
These stakeholders were aware that the HIS assessment would very soon be followed by a comprehensive strategic planning process to which they would also be asked to contribute.
Surveying national systems on knowledge in health
A number of data items relevant to systems on knowledge in health are not available from international sources. In some countries, the data are not available at all, while in others, much or all of the data characterising their knowledge systems can be found in their national health information or national health informatics systems, public health associations and other institutes and bodies. Regional and country focal points were aware of such locally published data. In some countries, all or most of the data were available from locally published sources. For items where there were no locally published data, we opted to ask key informants in each country to give us an estimate or an opinion.
The initial survey focused on analysing knowledge systems in national health system layers dealing with:
Health policy formulation, implementation and monitoring The provision of clinical services (our definitions for these layers of the health system are given below)
The questionnaire (in online supplementary material) sought to collect the opinions and estimates of experts on those items for which we found no published sources. We tried to eliminate questions for which published data already exist. Some of the questions in the questionnaire were matters of fact, while others were matters of opinion. At this stage in our efforts to describe the knowledge systems, we collected both qualitative and quantitative data.
The questionnaires were developed by WHO, making use of the advice of in-house survey and questionnaire specialists. As the original intention was to use them for a global survey, the questionnaires were then field-tested through full-day consensus workshops. The first such workshop was held in Prague (Czech Republic), and then in Addis Ababa (Ethiopia), Sana’a (Yemen), Manila (the Philippines), Bangkok (Thailand) and New Delhi (India). After these workshops, the questionnaires were adjusted and finalised for wider distribution. The health information, research and knowledge management focal point in every country in the Region received the questionnaires and collected data, making use of key informants.
We considered the health policy layer of the national health system as including all the elements involved in policy development, implementation and monitoring in the health sector. In a given setting, an event (an identified problem) triggers the policy deliberation. The process, which is not linear and involves interactions among different stakeholders at all times, often ends with a policy decision (budget, regulation, institutional arrangement), which may or may not be implemented and subsequently evaluated for improvement.
A country’s health policy may be centrally originated and managed, or decentralised to subnational entities (regions, provinces and states) or to bodies operating at national or even local levels (professional associations, syndicates or unions). If policy-making is evidence-based, the evidence may include research results; feedback from and studies of experience and implementation, guidelines, norms and standards developed internationally, regionally or locally; or other sources.
The questions aimed to clarify the knowledge translation process relating to health policy in a given country, identifying the principal stakeholders, actors and flows. Specific questions addressed the:
Responsibility and type of work of the person completing the questionnaire Bodies and groupings involved in health policy formulation Role of the ministry of health and other central policy-setting and implementing bodies Sources and types of evidence used Dissemination and use of knowledge about health policy Knowledge about monitoring and evaluation of health policy
The second questionnaire covered knowledge in the provision of clinical services at the local level. Direct patient care accounts for a significant portion of the health enterprise in countries. Clinical practice includes various preventive or care interventions (e.g. education, diagnosis, medical or surgical treatment, follow-up) for individuals in settings such as medical offices, hospitals or regular home care (note that in-and-out, point-of-service interventions or commodity delivery to populations were not covered here).
The questions aimed to clarify the knowledge translation process relating to clinical services provision in a given country, identifying the principal stakeholders, actors and flows. Specific questions addressed the:
Responsibility and area of work of the person completing the questionnaire Structure of delivery of clinical and health services in the country Knowledge related to policy and procedures Education and networking Knowledge sources and targets
Conclusions
The logistics and resources involved in surveying the knowledge landscape in the countries of the African Region were substantial. However, the investment was worthwhile as the results of the surveys can be used for:
Convening national consultations to develop or reinforce strategic frameworks and plans Holding subregional and regional consultations with stakeholders and the development of strategic frameworks Establishing a regional database of national health research systems and the African Health Observatory
In future surveys, more time and effort would be required to ensure that the surveys are completed by as many institutions and key individuals as possible.