Abstract
Introduction
Evaluation has been defined as “attributing value to an intervention by gathering reliable and valid information about it in a systematic way, and by making comparisons, for the purposes of making more informed decisions or understanding causal mechanisms or general principles.” 1 Understanding of evaluation as an ongoing activity rather than a one-time action has grown in recent years, leading to reassessment of evaluation theories and developing tools to assess activities and programs at different stages of a program's life cycle.
e-Health is widely accepted as an option for improving health services and information. e-Health is defined as the cost-effective and secure use of information and communications technologies in support of health and health-related fields, including healthcare services, health surveillance, health literature, and health education, knowledge, and research. 2 As e-health spreads globally, questions about its feasibility and impact need to be addressed. 3 Health systems must evaluate e-health in relation to factors that affect its use at different stages of implementation. 4 Different factors that play important roles in development, implementation, and operation of e-health must be understood and evaluated. This study analyzes different theories of evaluation applicable to e-health to obtain a better understanding of factors related to e-health and those stages of application they are most likely to influence.
Evaluation of e-health programs can serve several purposes, such as (1) to determine success or failure of programs in reaching explicit objectives, (2) to ascertain if there are more effective and less costly alternatives to achieve the same objectives, (3) to determine if the program has undesirable and unintended effects, and (4) to make informed policy decisions on continuation, termination, or changes in the program and whether to increase, maintain, or decrease funding for it. 5
Along with a better understanding of evaluation issues, it is also important to build standardized and validated tools for evaluation of e-health programs. Such tools can provide a systematic and comprehensive approach to evaluating e-health initiatives and make it easier to compare. This study reports development and validation of one such tool.
Materials and Methods
The conceptual framework for this study has been developed through a literature search of different theories and concepts on evaluation of e-health and a process of iterative review of this literature by several users of e-health. The methodology for developing the conceptual framework is described in three stages:
1. Literature review (identify theories and concepts related to evaluation of e-health)
2. Analysis (plot evaluation theories against the e-health life cycle and identify relevant themes)
3. Consolidation (develop a matrix aligning evaluation themes and stages of e-health implementation)
Literature Review
In-depth review of peer-reviewed and gray literature* was conducted to identify theories and concepts of evaluation either used in e-health or found relevant to applications of e-health. To broaden the search, theories of evaluation used for health programs and in related areas (such as health technology assessment) were also searched during the literature review. For peer-reviewed articles, the PubMed search engine was used to find English language articles published in the last 10 years. Key words used were “e-health,” “telemedicine,” “health technology,” “evaluation,” “outcomes,” “impact,” “themes,” and “theories.” The gray literature was searched applying the same strategy using Google.
Analysis
Following the literature review and analysis, the next step was to discuss the theories with a group of users in e-health and experts in evaluation. This was done jointly at the Aga Khan University in Karachi, Pakistan and the University of Calgary in Calgary, AB, Canada. A list of “evaluation themes” was developed through analysis of these theories by plotting them against the following life cycle stages of e-health implementation 6 :
a. Development: establishing active and viable e-health solutions based on clear needs, informed by an environmental scan, needs assessment, readiness assessment, existing capacity (technical, financial and human resources) and implemented within a broad social and equity-based perspective. 7
b. Implementation: preparing for and introducing the initial practice of an e-health solution. Implementation encompasses many process steps, including procurement of software and hardware, operational planning, installation, configuration, running, testing, and preliminary quality assurance, and requires significant change management initiatives. 8
c. Integration: combining of separate components, subsystems, or health issues and resolution of any problems in their interactions to ensure improvement in the quality and effectiveness of health services and care. 9 The word is used in the context of computing (e.g., integrating telehealth and health informatics activities), but is also applied to healthcare “system” activities (e.g., integrating e-health into traditional healthcare systems) and healthcare “business” processes (e.g., integrating activities of different healthcare institutions into a single functional entity).
d. Sustained operation: the ongoing state where an e-health application has performed satisfactorily without interruption for an extended period of time (months to years). This refers to the stage where routine health activities are “seamlessly” carried out using e-health as an integral component.
Consolidation
Evaluation themes for e-health identified through analysis of theories (Stage 2) were entered into a matrix against different stages of the e-health life cycle. A group of e-health researchers from 12 Asian countries who were part of the PAN Asian Collaborative for evidence-based e-Health Adoption and Application (PANACeA) were invited to review the framework and identify factors for evaluating each theme at different stages of the e-health life cycle. 10
Results
Findings of this study led to the development of a new and comprehensive e-health evaluation framework (the Khoja–Durrani–Scott [KDS] framework) for e-health evaluation. The framework was used to design an e-health evaluation tool, which is currently being validated. The findings are described under the steps discussed in Materials and Methods.
Literature Review
From the initial search of over 500 abstracts and conference proceedings, 60 scientific and gray resources were selected as primary references for review. A hard copy of each primary reference was obtained, reviewed, and summarized by the project research associate. From these, 40 articles were selected and forwarded to a three-member panel for final selection of key references and identification of evaluation theories and concepts relevant to e-health:
a. Needs-based evaluation theory. This theory is based on the idea that any new solution or program should focus on addressing social needs of the population. 11 For health-related interventions, it is important to identify gaps in the current system and services, which may require a new solution.
b. Behavioral change theory. Several theories exist for behavioral change, but the most common and relevant was Prochaska's Transtheoretical Model of Behavior Change. This theory deals with the adoption of change by individuals or groups of people, such as organizations, and identifies five stages: (1) Precontemplation is the stage at which there is no intention to change behavior in the foreseeable future. (2) Contemplation is the stage in which individuals are deciding whether or not there is a need to take action to correct the problem. (3) Preparation is a stage entered into once the individual decides there is a need for action. Specific plans of action are developed during this stage as the individual chooses among alternative potential solutions. (4) Action is the stage when someone intends to modify the life style in order to adopt new behavior. (5) Maintenance is the stage where people consolidate the gains attained from their actions. 12
c. Health outcomes and impact assessment. Health outcomes research has developed as a means of evaluating the effectiveness of healthcare interventions and as an approach to informing resource allocation. The use of a health outcomes approach in health promotion has made increasing demands on evaluation methodologies to demonstrate program effectiveness and ultimate impact. 13
d. Technology evaluation. Technology evaluation highlights mechanisms associated with the adoption and diffusion of healthcare technologies, realizing that the capacity to capitalize on the health gains from the adoption of technology varies in society. This variation is caused by the segmentation of individuals into distinct social and cultural groups. The model helps in understanding these factors and improving adoption and benefits of technology by addressing these influences. 14
e. Sociotechnical analysis. Sociotechnical systems analysis provides a powerful framework with which to analyze the reasons behind the poor acceptability, uptake, and performance of many information and communication technology interventions. Perhaps because healthcare systems are so dependent on complex human organizational structures, they seem particularly suitable to sociotechnical analysis. 15
f. Economic assessment. Economic evaluation of various healthcare interventions helps allocate resources in an efficient way and facilitates prioritization. Different methods of economic evaluation have been developed to help guide decisions and inform policymaking. 16
g. Ethical consideration. This is taking into account a person's well-being by using morality in one's behavior. It addresses the moral and ethical issues arising from clinical practice, medical and biological research, resource allocation, and access to biomedical technology. 17
h. Policy and legal framework. e-Health policy can be defined as “a set of statements, directives, regulations, laws, and judicial interpretations that direct and manage the life cycle of e-health.” It is important to design policies to guide proper implementation of e-health initiatives, complemented by the legal framework to ensure that malpractices, such as inappropriate access to patient information, breach of privacy, or inadequate quality of care are controlled. 18
Analysis
Each evaluation theory or concept was examined based on its relevance to e-health and to the part of the e-health life cycle it may influence the most. The alignment of evaluation theories with different stages of the e-health life cycle is shown in Figure 1.
The alignment of evaluation theories with different stages of the e-health life cycle.
On the basis of the analysis, a list of evaluation themes was identified that covers a wide spectrum of evaluation of information and communication technology use in healthcare. These themes are listed below:
a. Health services outcomes. 19 Health services outcomes are based on the principle that every clinical or health intervention produces a change in the health status of a patient or community and that change can be measured on the basis of:
• Change in disease or health status
• Impact on quality of life
• Change in health indicators
b. Technology outcomes. 20 Technology refers to the software, hardware, and connectivity infrastructure used to implement and sustain any e-health solution. e-Health technology outcomes can be measured in terms of appropriateness, relevance, use, safety, and effectiveness of the technology.
c. Affordability and cost-effectiveness. 21 This considers the extent to which a health service is affordable, as measured by its cost relative to the amount that the user (patient, family member, clinician) is able—or willing—to pay for it. Cost-effectiveness will determine the least costly system that is capable of delivering a specified set of outcomes.
d. Social and behavioral impact. Social impact assessment includes the processes of analyzing, monitoring, and managing the intended and unintended social consequences, both positive and negative, of planned interventions (policies, programs, plans, projects) and any social change processes invoked by those interventions. Its primary purpose is to bring about a more sustainable and equitable biophysical and human environment. 22
e. Ethics. A general term for what is often described as the science of morality. In philosophy, ethical behavior is behavior that is good. The goal of a theory of ethics is to determine what is good, both for the individual and for society as a whole. 23 Ethics in e-health address the moral and ethical issues arising from clinical practice, medical and biomedical research, resource allocation, use, and access to technology solutions.
f. Readiness and change. 24 e-Health readiness refers to the preparedness of healthcare institutions and individuals to implement programs that involve use of e-health in the provision and management of health services. The process of change management includes changes in the existing business processes and ensuring appropriate training and support for adopting new processes.
g. e-Health policy. e-Health policy (defined as “a set of statements, directives, regulations, laws, and judicial interpretations that direct and manage the life cycle of e-health” 18 ) is required to facilitate structured and consistent e-health practice.
Consolidation
A matrix containing evaluation themes and the stages of the e-health life cycle was developed. This matrix was named the KDS Framework for e-Health Evaluation. The framework was shared with e-health researchers from PANACeA, who provided their comments on the framework and also identified factors that could be evaluated under each theme at different stages of e-health programs. Examples of these factors are shown in Table 1. These factors have been populated using the literature review and interviews with domain experts. The findings will be used to develop e-health evaluation tools, which will be the next stage of research.
Khoja–Durrani–Scott Evaluation Framework Showing Different Themes of Evaluation and Associated Factors That Influence the Stages of the e-Health Life Cycle
DALY, disability-adjusted life-year; QALY, quality-adjusted life-year.
The Kds Evaluation Tool
Based on the KDS framework, evaluation tools were developed for evaluating e-health programs. Four separate evaluation tools were developed for the four different stages of the e-health life cycle, and each of these tools has a separate set of questions for the three different types of users (i.e., managers, healthcare providers, and clients). The questions for each group of respondents are divided into the seven outcome themes described in the above framework. The KDS evaluation tools can be seen at
Discussion
The KDS Framework offers a comprehensive platform for developing an evaluation tool for e-health programs. The framework builds on existing theories of health and technology evaluation and covers the range of areas affected or influenced by e-health interventions (health, social, technical, economic, ethical, and policy aspects of e-health programs). Evaluation of any e-health program should not be limited to health outcomes or economic analysis, but should cover all themes identified in the KDS Framework.
Using the above framework, researchers at the Aga Khan University and the University of Calgary have developed tools for e-health evaluation that could be used for e-health programs in a variety of settings and stages. The tools cover the factors identified in each box of the KDS Framework described above. Separate tools have been developed for managers, healthcare providers, and clients to understand their perspective on the performance of e-health initiatives. The tools will be validated thoroughly before making them available for general use.
Major strengths of the KDS tool include (1) conducting a thorough literature search to include all established theories of evaluation in health and technology, (2) involving experts in assessing and critiquing the evaluation theories and identifying relevant themes, and (3) discussing the framework with a number of e-health researchers and evaluators involved in a closed network and having them identify factors for building the evaluation tools.
In conclusion, the KDS Framework offers a comprehensive approach to e-health program evaluation and has provided a platform to develop e-health evaluation tools for wider use across the globe.
Footnotes
Acknowledgments
We thank Ms. Syeda Suha Tirmizi (Research Officer at the Aga Khan University) for her help in drafting and finalizing the manuscript. We also appreciate the support of our PANACeA partners for their continuing input into the development of the framework and tools. The authors also thank the International Development Research Centre, Canada, for funding this effort through the PANACeA initiative.
Disclosure Statement
No competing financial interests exist.
*
Gray literature is defined as any literature that is not published in academic peer-reviewed journals and is not available through indexed databases for review.