Beyond monitoring functionality to results evaluation of eHealth interventions: Development and validation of an eHealth evaluation framework

Abstract

Evidencing eHealth interventions, benefits generates data as a basis for assessing whether observed changes in behavior, processes or healthcare outcomes can be attributed to the eHealth interventions. Generating such evidence requires the use of frameworks or some other type of organizing schemes to help in guiding the process and making sense of eHealth systems and the findings. The frameworks available in literature do not clearly guide on how to monitor eHealth implementation and evaluate eHealth implementation results. This study aimed to develop and validate an eHealth evaluation framework to guide the process of monitoring eHealth implementations and evaluation of eHealth results in terms of outcomes and impact on healthcare in developing countries. The Design Science Research Methodology was followed to conduct this study. Recommendations from an eHealth evaluation exploratory study in Uganda and other eHealth evaluation literature formed key inputs into the design and development of the framework. The framework consists of a generic reference model with eHealth monitoring and evaluation dimensions, performance indicators, and guidelines on how to conduct eHealth monitoring and evaluation. The eHealth evaluation framework received high acceptance (>80%) as regards its fitness for purpose during its validation.

Keywords

eHealth monitoring evaluation results-based management

Introduction

Across the world, healthcare systems are facing pressures to guarantee equitable access to quality and affordable healthcare. Healthcare administrators and policy makers are expected to select and implement interventions that increase the quality and efficiency of services, care and support high performance of health systems.^1,2 The application of information and communication technology (ICT) in healthcare management, referred to as eHealth,³ is becoming more popular and has proved to improve the effectiveness, access, quality and efficiency of healthcare systems.^3–5 eHealth has facilitated various medicine and public health operations including patient and public health data management (electronic health records), provision of remote healthcare services (telemedicine/teleHealth), health information and services through mobile telephone technology (mHealth), health knowledge management and distant learning for health workers (eLearning), connection of medical devices (internet of things), and other areas like improved planning, organization, management of health services, and more recently the management of large public health data. The Internet has also facilitated communication between healthcare providers and their clients, sharing of information and knowledge among healthcare providers and researchers, all of which have contributed to better disease management.^6,7 Many developed countries have implemented well-developed eHealth applications that have benefited their healthcare systems.

Learning from the developed world, developing countries are implementing ICT solutions as a means to improve accessibility to quality and equitable healthcare.^8,9 Uganda, like most other developing countries, has employed eHealth to improve healthcare delivery and public health^8,10 and this has yielded positive results, especially in disease control and prevention through disease surveillance.¹¹ Implemented since 2011, Uganda’s national DHIS2-based electronic Health Management Information System (eHMIS), one of the key nationally scaled up eHealth platforms, is used to capture, validate, analyze and visualize aggregate data about health events and services reported by healthcare providers, especially public healthcare facilities, across the country. Apart from the national eHMIS, other eHealth platforms are used in Uganda to support delivery of various functions including management of human resources for health, supply of essential medicines and medical supplies, laboratory information management, and integrated disease surveillance and response.

Such an increasing investment in eHealth has also increased demand for evidence to prove that there are benefits and results realized from eHealth applications. The evaluation of eHealth interventions helps to generate data that can be used as a basis for assessing whether observed changes in behavior, processes or health outcomes can be attributed to the eHealth interventions.^12,13 The concept of “evaluation” is defined as a systematic and objective assessment of an intervention that aims to determine the fulfillment of objectives, efficiency, effectiveness, impact and sustainability.¹⁴ Related to evaluation is the concept of “monitoring” which is the routine collection, review, and analysis of data intended to measure implementation progress for an eHealth initiative, and results into adjustments in intervention activities necessary to maintain or improve the quality and consistency of the eHealth deployment. World Health Organization (WHO) emphasizes that evaluation measures changes in outcome and impact that are attributed to the eHealth initiative.¹³

Due to various eHealth evaluation complexities, various literature recommends the use of frameworks or some other type of organizing schemes to help in guiding the evaluation process but also making sense of eHealth systems and evaluation findings.^12,13 Several ICT evaluation frameworks directly related to the health domain and healthcare including Total Evaluation and Acceptance Methodology (TEAM) framework,¹⁵ HOT-Fit framework,¹⁶ Performance of Routine Information System Management (PRISM) framework,¹⁷ Khoja‐Durrani‐Scott (KDS) Framework¹⁸ and the Evidence in eHealth Evaluation Model¹⁹ exist and present several and varying approaches to eHealth evaluation. However, most of the real-world case studies that have undertaken evaluation of their eHealth interventions, have not used such standard evaluation frameworks and theories;¹⁹ this could be attributed to their lack of guidance on how to clearly state/evidence the contributions of eHealth interventions to service delivery and healthcare status.¹⁹ Further, much as the existing frameworks consider eHealth systems use, user uptake, satisfaction and other outcome indicators as success measures, results for such indicators do not automatically translate to changes in behavior and healthcare outcomes.²⁰ Last but not least, these frameworks do not address needs for comprehensive evaluation of eHealth interventions in countrywide eHealth settings;²¹ and lastly, the evaluation aspects provided by the frameworks are not classified in any way to guide on aspects to be considered when monitoring eHealth implementation or evaluating eHealth results. Such limitations are also confirmed in an exploratory study about eHealth evaluation practices and challenges in developing countries with a focus on Uganda, which reported scantiness of evaluations for the several eHealth interventions being implemented in the country. The study further reported that eHealth implementers focused on monitoring the eHealth interventions' functionality and adoption rather than their outcome and impact, with limited capacity and unavailability of guidelines on eHealth evaluation being the key limitations.²² Table 1 presents a summary analysis of the several frameworks and the categorizations of eHealth evaluation aspects they cover including organizational, technological, human/social, clinical, cost and economic, ethical and legal, and transferability aspects.

Table 1.

Analysis of the existing eHealth evaluation models and frameworks.

Framework	Aspects of eHealth evaluation
Framework	Organizational aspect	Technological aspect	Human and social aspect	Clinical aspect	Cost aspect	Ethical and legal aspect	Transferability aspect	Differentiate monitoring and evaluation?
TEAM¹⁵	Organizational structure and roles	X	X	X	X	X	X	X
HOT-Fit¹⁶	Organizational Structure; Organizational Environment	System, information and service quality	System use; User satisfaction	Net benefits	Net benefits	X	System use	X
PRISM¹⁷	Organizational factors	Technical factors; RHIS processes; RHIS performance	Behavioral factors	Health system performance; Health status	X	X	X	X
Khoja-Durrani-Scott (KDS)¹⁸	Readiness and change outcomes	Technology outcomes	Behavioral and socio technical; Readiness and change outcomes	Healthoutcomes	Economicoutcomes	Ethical outcome; Policy outcomes	X	X
Evidence in eHealth¹⁹	Organizational aspect	Technological aspect	Human and social aspect	Clinical aspect	Cost aspect	Ethical and legal aspect	Transferability aspect	X

X means “framework does not take the aspect into consideration”.

Accordingly, this study aimed to develop and validate an eHealth evaluation framework to guide monitoring of eHealth implementations and evaluation of their results in terms of outcomes and impact on healthcare in developing countries, using Uganda as a case study.

Methodology

Study setting

The study was conducted in a period between 2019 and 2020 within the context of Uganda as a developing country. The study was well documented based on literature on eHealth evaluation practices and challenges. The study was conducted in close consultation with Uganda Ministry of Health (Division of Health Information), which is the custodian of digital health and health information management in Uganda.

Study design

The Design Science Research Methodology (DSRM)^23,24 was followed to conduct this study. DSRM was used because of its ability to study the connection between research and professional practices by designing, implementing and evaluating artifacts that address a specific need, and embodies the creative, artistic, and goal-oriented spirit within which purposeful artifacts are created.²⁵ Particularly, the six DSRM activities were carried out as detailed below;

Activity 1: Problem identification and motivation - the problem and motivation for this study was identified by the researchers from literature. There was a need for a framework to guide monitoring of eHealth implementations and evaluation of results from eHealth interventions. The researchers also consulted officials at Uganda Ministry of Health (Division of Health Information) who acknowledged the identified problem and provided authorization to the researchers to work with eHealth practitioners in the country towards achieving the study objective.

Activity 2: Definition of the objectives of a solution - the objective of this study was defined by the researchers as “to develop and validate an eHealth evaluation framework to guide monitoring of eHealth implementations and evaluation of their results in terms of outcomes and impact on healthcare”.

Activity 3: Design and development – by synthesizing and complementing recommendations from various eHealth evaluation literature, the researchers designed and developed the eHealth evaluation framework constituting three components including; i) a reference model that demonstrates key aspects of eHealth monitoring and evaluation, ii) performance indicators for measuring eHealth performance, and iii) a set of guidelines on how to conduct monitoring and evaluation of eHealth interventions.

Activity 4: Demonstration – the researchers demonstrated the utility of the eHealth evaluation framework using the walk-through approach to selected eHealth practitioners in a workshop setting. The framework was demonstrated to twenty-one (21) eHealth experts in Uganda who attended the workshop out of purposively selected thirty (30) experts who were invited by the Ministry of Health (Division of Health Information) to participate in the workshop due to their experience in eHealth implementation. The twenty-one experts comprised of 5 (24%) females and 16 (76%) males with varying experiences in eHealth and from different organization settings including development partners, implementing partners, health facilities, research or academic institutions and Ministry of Health (see Table 2). Development partners are bilateral donors and multilateral agencies that provide essential funding to the health sector, implementing partners are organizations that work with Ministry of Health to implement program activities, health facilities are locations where healthcare is provided, academic institutions are educational institutions that contribute to research on health related topics, and Ministry of Health is the government body responsible for the health system in Uganda. This mix of participants helped to avoid bias in the framework validation results.²⁶

Table 2.

Respondent’s level of eHealth experience and organizations’ role.

Organization’s role	Respondent’s years of eHealth experience					Grand total
Organization’s role	1–3 years	4–6 years	7–10 years	11–15 years	Over 15 years	Grand total
Development partner		2	3			5 (24%)
Health facility	1		1			2 (10%)
Implementing partner	1		1	1		3 (14%)
Ministry/Policy maker	1	2			1	4 (19%)
Research/Academic institution	2	2	3			7 (33%)
Grand Total	5 (24%)	6 (28%)	8 (38%)	1 (5%)	1 (5%)	21

Activity 5: Evaluation - the framework was validated against design science validation criteria suggested by Aier and Fischer²⁷ including utility, internal consistency, external consistency, broad purpose and scope, simplicity, and fruitfulness of further research (see Table 3). Following from the walk-through demonstration, all the twenty-one eHealth practitioners evaluated the framework’s utility and provided their feedback through a questionnaire²⁸ regarding their agreement or disagreement on whether the framework met each validation criterion. The resultant data from filled questionnaires was entered and analyzed in Microsoft Excel.

Table 3.

Validation criteria for the eHealth evaluation framework.

No	Criteria/Parameters	Explanation
1	Utility: The ability of the eHealth evaluation framework to fulfill its purpose
1.1	Effectiveness	The framework meets its higher-level purpose of guiding the process of monitoring eHealth implementation and evaluating the eHealth intervention results
1.2	Efficiency	The framework can be applied at a minimal cost/reduces the effort and other resources to perform the monitoring and evaluation tasks in comparison to the eHealth investment
2	Internal consistency: How well are terms and components of the framework and their relationship adequately defined
2.1	Completeness	The framework covers all aspects of monitoring and evaluation of eHealth implementation
2.2	Depth and breadth	The scope/level of detail covered in each aspect of the framework is appropriate
3	External consistency: How well the framework fits and aligns well with other knowledge in the field of eHealth monitoring and evaluation
3.1	Fidelity	The framework appropriately captures and represents aspects of eHealth monitoring and evaluation
4	Broad purpose and scope: How well the framework can be adapted to different purposes and scope without its usefulness decreasing
4.1	Generality	The framework is flexible for adaption to varying settings of eHealth interventions
4.2	Robustness	The framework is able to achieve and sustain its intended objectives irrespective of varying settings of eHealth implementations
5	Simplicity: How simple is the framework
5.1	Ease of use	The framework requires little effort or training to use
5.2	Understandability	Users of the framework are able to develop a shared/common understanding of its components and the eHealth monitoring and evaluation process
5.3	Elegance	Users are willing to apply the framework to guide their eHealth monitoring and evaluation process
6	Fruitfulness of further research: How well the framework contributes to the information systems (IS) and health informatics (HI) body of knowledge
6.1	Contribution to IS and HI knowledge bases	The framework aptly contributes knowledge to the information systems and health informatics research disciplines

Activity 6: Communication - the developed and validated eHealth evaluation framework was communicated through various stakeholder meetings, and also disseminated through this manuscript.

Ethical consideration

Researchers obtained written authorization from Uganda Ministry of Health to conduct the study within Uganda’s eHealth implementation settings. For participants engaged in the demonstration and validation of the framework, the researchers verbally informed participants that participation to give feedback about the framework was voluntary, and submission of feedback would mean consent to participate.

Results

Design and development of an eHealth evaluation framework

An eHealth evaluation framework was designed and developed. It constitutes of three components-i) a reference model that demonstrates key aspects of eHealth monitoring and evaluation; ii) performance indicators for measuring eHealth performance and how these can be operationalized; and iii) a set of guidelines on how to conduct monitoring and evaluation of eHealth interventions.

i). eHealth evaluation reference model derivation and description

Findings of the exploratory field study²² on indicators supposed to be assessed during the eHealth evaluation were corroborated with the suggestions from other eHealth evaluation frameworks in literature^13,16–19 to derive substantial and fundamental all-encompassing eHealth evaluation domains and sub-domains that are supposed to be assessed (see detailed derivation²⁹). The resultant indicator domains and sub-domains were further aligned with the results chain provided in the results-based management strategy (see Figure 1)³⁰ and the evaluation questions provided in the categorization of the eHealth indicators (see Figure 2)¹³ in order to further distinguish between the aspects of “monitoring” and “evaluation” of eHealth implementation results in the developed eHealth evaluation reference model (see Figure 3). The Results-based management strategy provides that all actors, contributing directly or indirectly to achieving a set of results, should ensure that their processes, products and services contribute to the achievement of the desired results in terms of outputs, outcomes and higher-level goals or impact. It is a results chain of inputs-activities-outputs-outcomes-impact on the basis that actions taken at one level will lead to a result at the next level; and in this sense, the results chain stipulates the sequence of actions taken to achieve a particular result. The categorization of eHealth indicators provides guidance on the key monitoring and evaluation questions that can help to assess various areas through which eHealth interventions can achieve results, and the questions include a) Does the technology work; b) How do people interact with the technology; c) How does the technology improve the service delivery process; and d) How do improvements in service delivery affect health?

Figure 1.

Results chain from the results-based management strategy.³⁰

Figure 2.

Categorization of eHealth indicators.¹³

Figure 3.

An eHealth evaluation reference model to evaluate eHealth interventions.

The eHealth evaluation reference model provides a basis for evaluating the performance of eHealth implementations along the results chain, starting with the programme inputs and activities until the impact. The model indicates the key eHealth evaluation questions that should be answered; the key domains of performance indicators, and the data sources where the data needed to measure the indicators can be collected from at each stage of the results chain. The monitoring of programme inputs and activities answers whether the eHealth technology works; and it relates to the assessment of the feasibility of the eHealth implementation and the inputs for developing the initiative. Monitoring of eHealth outputs assesses how the users interact with the initiative; and it relates to the service output measures intended to capture and assess the immediate results of the eHealth initiative as well as the usability measures to help quantify how the users interact with it. Evaluation of eHealth outcomes helps to ascertain how the eHealth improves programme processes and service delivery; and it captures the effect of the eHealth implementation on programme service utilization outputs or the extent to which the clients use the programme service, and intermediate population-level outcomes. The “evaluation of impact” focuses on establishing how improvements in service delivery impact health by capturing long-term outcomes, and impact on health that are attributed to the health intervention being supported by the eHealth initiative. Also, to note is a feedback loop in monitoring of the programme inputs and activities, and eHealth outputs. From a technical perspective, eHealth inputs affect the respective outputs and users’ experience, while the user experience/feedback informs the eHealth development process. Feedback from end-user adoption and satisfaction rates loops around to further inform the eHealth development process and determines the quality of the technological inputs. This, in turn, affects the performance of the revised version of the eHealth system among end-users, making the technology development process iterative.

ii). eHealth evaluation framework indicators and operationalization tools derivation

An indicator is defined as “a quantitative or qualitative factor or variable that provides a simple and reliable means to measure achievement, to reflect the changes connected to an intervention or to help assess the performance of a development actor”.³¹ In order to demonstrate how the eHealth interventions’ performance can be measured at each stage of the results chain in the reference model, suggestions and best practices provided in various eHealth evaluation literature^{13,16–20,22,30,32} were synthesized and used to develop a set of example indicators and their respective operationalization tools (see detailed list³³). To note is that, this set of example indicators and operationalization tools are general and non-exhaustive; henceforth requiring that priority indicators and their operationalization tools for use to monitor and evaluate a specific eHealth intervention should be selected based on the relevance of the intervention and modified to reflect the specific objectives of the intervention.

iii). Guidelines for conducting evaluation of eHealth interventions derivation

The last component of the eHealth evaluation model provides a set of guidelines that are intended to guide/facilitate the users through a step-by-step process of planning and executing activities for monitoring and evaluation of eHealth interventions. Informed by findings of the exploratory study by Ashaba and Nabukenya,²² and the guidance from World Health Organization, International Telecommunication and Uganda eHealth Strategy,^13,20,32 this paper provides a set of key strategies and course of actions that should be followed with adaptation to the eHealth context, while planning for and conducting monitoring and evaluation of eHealth interventions in Uganda and other low- and middle-income countries with similar or related eHealth settings;

1. Establish dedicated resources and effort to ensure an effective process, and such considerations should be provided from the inception of the initiative. eHealth implementers should plan and budget for monitoring and evaluation of the interventions during planning and costing of their eHealth programmes, to ensure that appropriate resources are dedicated;³²

2. Identify all key stakeholders of the eHealth initiative, involve them and consider their interests in the eHealth intervention while designing the monitoring and evaluation plans, and ensure their consensus on the public health need or goal that the eHealth addresses or supports;¹³

3. Make use of/refer to the eHealth reference model (Figure 3), indicator domains and sub-domains (see details²⁹) as well as the examples indicators (see details³³) as guides to think through the necessary monitoring and evaluation aspects of the eHealth intervention. The reference model provides a graphical representation to aid the clarification of monitoring and evaluation scopes and identification of expected causal links between inputs and activities, outputs, outcomes and impacts for eHealth implementations, while the indicator domains and indicators provide a non-exhaustive list of some examples of indicators that can be measured under each stage of the results chain;

4. Define the appropriate eHealth initiative implementation phases and timeframes²⁰ to specify and harmonize the stages of the eHealth maturity, timing for delivery of the defined results and frequency of measuring the results. It should be noted that defining the eHealth initiative implementation phases helps to refine and focus on the appropriate results and indicators to be monitored or evaluated at each stage of maturity of the eHealth intervention. Stakeholders should agree on the stage of maturity of the eHealth intervention and determine the appropriate monitoring or evaluation activities to avoid embarking on premature.¹³ The mapping shown in Table 4 can be used to match the six stages of maturity of an eHealth intervention with the results chain and eHealth evaluation questions;

5. Define stakeholder priorities and their projected impact of the eHealth intervention and outcomes required to cause the defined impact, and the respective impact and outcome measurable indicators while ensuring that interests of all identified stakeholders are addressed. It should be noted that evaluation of an eHealth intervention can be limited to its process and early outcomes without the need to evaluate its long-term outcomes and impact if the eHealth is supporting a health intervention with known efficacy based on prior research. For example, an evaluation of an eHealth intervention for vaccination reminders should focus on measuring the percentage of children who received timely vaccination as a result of the eHealth prompt (e.g. text message) and need not seek to measure the direct impact of vaccination on the rate of child mortality from the disease they are vaccinated against, since the effectiveness of timely vaccination will have been established through prior research;¹³

6. Define eHealth outputs required to lead to each defined eHealth outcome and the respective eHealth output indicators.²⁰ The outputs should include immediate results of the eHealth intervention and should also capture usability measures that help to quantify how users interact with the eHealth intervention;

7. Define eHealth inputs and activities and their respective measurable indicators while taking consideration of all the technical, organizational, policy and behavioral factors that influence delivery of a working eHealth application. Indicators for availability of resources required to support monitoring and evaluation functions should also be considered while defining indicators for the eHealth inputs and activities;

8. Define the baseline and target measures for the defined indicators to clearly label the situation before the eHealth implementation (baseline measures) and expected achievements (target measures) after the eHealth implementation.²⁰ A measure of the baseline situation facilitates the ability to compare situation statuses before and after the eHealth implementation. It necessitates research and analysis to determine baseline measures for each indicator, while internal analysis and input from the subject-matter experts and other health experts are necessary to determine the target measures;³²

9. Define the necessary monitoring and evaluation governance and processes/activities.^20,32 Considering that monitoring and evaluation tasks are conducted at different and various times and by various parties, it is imperative that their governance model is defined to guide on the available functions, roles, responsibilities, and required mechanisms to deliver them;

10. Prepare for and initiate the monitoring and evaluation process that should continue ongoing in parallel with the eHealth implementation,^20,32 with monitoring or evaluation undertaken at the defined timeframes for each of the indicators. The preparation involves defining detailed monitoring and evaluation timelines and milestones and ensuring the availability of all the resources required to support the monitoring and evaluation activities including the monitoring and evaluation teams and data collection tools;

11. Execute the monitoring and evaluation data collection activities and track indicator measures according to the eHealth maturity stage, timing for delivery of the defined results, defined indicators, frequency of measuring the results, and defined monitoring and evaluation timelines;

12. Analyze collected data and report on actual situation versus target eHealth adoption and results as per defined indicators (see example reporting tool in Table 5). The analysis may also include identification of causes of divergences in actual and target performance at each stage in the results chain for each timeframe of measuring results;

13. Engage stakeholders to agree on the corrective actions to address divergence between target and actual results, and provide for management of changes in the eHealth implementation scope, if required, to implement the agreed corrective actions;^20,32

14. For indicators that may be unrealistic or unachievable within the required timeframes, engage stakeholders to review and agree on refined indicators, targets and timeframes for consideration in the next round of results measurement.

Table 4.

Mapping eHealth maturity stages with the results chain and eHealth evaluation questions.

Stages of eHealth maturity	Pre-prototype	Prototype	Pilot	Demonstration	Scale-up	Integration/sustainability
Results chain stage	Programme and eHealth inputs and activities		eHealth outputs	eHealth outcomes	eHealth impact
eHealth evaluation question	Does the eHealth technology work?		How do users interact with the eHealth?	How does the eHealth improve programme processes/service delivery?	How do improvements in service delivery impact health?

Table 5.

Example of eHealth monitoring and evaluation reporting tool.

Result	Indicator	Reporting frequency	Baseline (Period)	Reporting period 1		Reporting period 2
Result	Indicator	Reporting frequency	Baseline (Period)	Target	Actual	Target	Actual
eHealth impact
Impact 1	Indicator 1.1
Impact 1	Indicator 1.2
Impact 2	Indicator 2.1
Impact 2	Indicator 2.2
eHealth outcomes
Outcome 1	Indicator 1.1
Outcome 1	Indicator 1.2
Outcome 2	Indicator 2.1
Outcome 2	Indicator 2.2
eHealth outputs
Output 1	Indicator 1.1
Output 1	Indicator 1.2
Output 2	Indicator 2.1
Output 2	Indicator 2.2
eHealth inputs and activities
Input 1	Indicator 1.1
Input 1	Indicator 1.2
Input 2	Indicator 2.1
Input 2	Indicator 2.2

Validation of the eHealth evaluation framework

Figure 4 provides the criteria score analysis of the feedback regarding the framework validation. We observed that 80% and above of the participants agreed that the model met all the criteria, apart from efficiency (76%) and ease of use (71%). The results indicated that the framework is able to fulfill its purpose (average score of 85.5% on effectiveness and efficiency), its components and their relationship are adequately defined (score of 81% on each of completeness, and depth and breadth), and also fits and aligns well with other knowledge in the field of eHealth (score of 81% on fidelity). The validation results demonstrated acceptance that the framework is suitable for adaptation to eHealth interventions of different purposes and scope (score of 95% on each of generality and robustness), and further indicate the framework is simple to use (average score of 78.5% on ease of use and understandability).

Figure 4.

eHealth evaluation framework validation results.

Discussion

The developed framework consists of a generic reference model that demonstrates key aspects (dimensions) of eHealth monitoring and evaluation, performance indicators used to measure eHealth performance and their operationalization tools, as well as a set of guidelines on how to conduct monitoring and evaluation of eHealth interventions. The framework was developed with consideration of guidance and recommendations from varied literature and it facilitates the understanding of the distinction between the notions of “monitoring” and “evaluation” of eHealth interventions. While the monitoring notion focuses on routine collection, review, and analysis of data intended to measure implementation progress for an eHealth intervention and results into adjustments in intervention activities necessary to maintain or improve the quality and consistency of the eHealth deployment; the evaluation notion deals with measuring the changes in outcome and impact that are attributed to the eHealth intervention. This distinction is worth a significant asset in the research disciplines and practice of eHealth, medical/health informatics and information systems (with attention to healthcare).

From the framework validation results (Figure 4), such a high score of 80% and above on each criterion apart from efficiency (76%) and ease of use (71%) suggested that participants appreciated the framework’s fitness for purpose. The remaining 24% and 29% of the participants, who did not agree on the framework’s efficiency and ease of use respectively, may gain confidence in the framework and appreciate its ease of use and application with minimal resources after having more time of training and application of the framework in eHealth evaluations as suggested by other studies.^23,34,35 The high acceptance of the framework confirmed the same experience as that from an artifact validation study by Al Nadabi where acceptance was as high as 80% and above.³⁶ The high acceptance of the framework by the eHealth experts in Uganda further suggest its applicability to other low- and middle-income countries with similar eHealth settings, which may require none to less customization of the framework to fit/align with their respective eHealth evaluation goals.³⁷

It should be noted, however, that the framework demonstration and validation activities were undertaken amid the COVID-19 pandemic which resulted in conducting the activities while strictly adhering to the Uganda Ministry of Health COVID-19 infection prevention and control guidelines. This limited the extent of stakeholder engagement and the field activities that could be conducted. Demonstration and validation of the framework’s utility was done through walk-throughs in a workshop setting where COVID-19 related guidelines could easily be enforced. This approach could not provide feedback about the framework’s application in real-life eHealth implementation settings. To counteract this limitation, the researchers are currently conducting empirical validation of the framework to further understand its applicability and benefits in the field.

Conclusion

In this research, we present both the relationship and difference between monitoring of eHealth interventions implementation and evaluation of outcomes and impact realized from the interventions. We developed and validated an eHealth evaluation framework to guide the process of monitoring and evaluating eHealth interventions performance in healthcare. The high acceptance (>80%) that the framework met its validation criteria suggested its fitness for purpose during its validation. To get started with using the eHealth evaluation framework, eHealth implementers will need to first read and understand its three components in order to appreciate how to apply it depending on the nature and purpose of the eHealth intervention in consideration. The framework is applicable to the evaluation of individual eHealth interventions and it yields substantial results if its guidelines’ prescription is followed in a timely manner (that is, starting with the initial planning phases of the eHealth interventions). We recommend the application of the framework to evaluations in real-life settings and/or scenarios of eHealth interventions in Uganda and other low- and middle-income countries to further evidence its applicability and benefits. Further research could assess the suitability of the framework for monitoring and evaluation of eHealth in the context of developed countries.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Justus Ashaba

Josephine Nabukenya

References

Kieny

Bekedam

Dovlo

, et al. Strengthening health systems for universal health coverage and sustainable development. Bull World Health Organ 2017; 95(7): 537.

Kruk

Gage

Arsenault

, et al. High-quality health systems in the sustainable development goals era: time for a revolution. Lancet Glob Health 2018; 6(11): e1196–e1252.

Stoffers

. The promise of ehealth for primary care: Opportunities for service delivery, patient–doctor communication, self-management, shared decision making and research. Eur J Gen Pract [Internet]Mar 26 [cited 2022 Mar 24]. 2018; 24. Available from: https://www.tandfonline.com/doi/abs/10.1080/13814788.2018.1449779

Hyppönen

Ronchi

Adler-Milstein

. Health care performance indicators for health information systems. Evid-based Health Inform 2016: 222, 181–194.

Murray

Hekler

Andersson

, et al. Evaluating digital health interventions: key questions and approaches. Am J Prev Med 2016; 51(5): 843–851.

World Health Organization . Innovative technologies that address global health concerns: outcome of the call: global initiative on health technologies, 2010. World Health Organization, 2010.

Olu

Muneene

Bataringaya

, et al. How can digital health technologies contribute to sustainable attainment of universal health coverage in Africa? A perspective. Front Public Health 2019; 7: 341.

Scott

Mars

. Telehealth in the developing world: current status and future prospects. Smart Homecare Technol Telehealth 2015; 3: 25–37.

Lewis

Synowiec

Lagomarsino

, et al. E-health low- and middle-income countries: findings from the center for health market innovations. Bull World Health Organ 2012; 90(5): 332–340.in.

10.

Blaya

Fraser

HSF

Holt

. E-health technologies show promise in developing countries. Health Aff Proj Hope 2010; 29(2): 244–251.

11.

Kiberu

Mars

Scott

. Barriers and opportunities to implementation of sustainable e-health programmes in Uganda: a literature review. Afr J Prim Health Care Fam Med 2017; 9(1): 1–10.

12.

Lau

Kuziemsky

. Handbook of eHealth evaluation: an evidence-based approach. 2016.

13.

World Health Organization . Monitoring and evaluating digital health interventions: a practical guide to conducting research and assessment. World Health Organization, 2016.

14.

World health Organization . WHO evaluation practice handbook. World Health Organization, 2013.

15.

Grant

Plante

Leblanc

. The TEAM methodology for the evaluation of information systems in biomedicine. Comput Biol Med 2002; 32(3): 195–207.

16.

Yusof

Kuljis

Papazafeiropoulou

, et al. An evaluation framework for health information systems: human, organization and technology-fit factors (HOT-fit). Int J Med Inf 2008; 77(6): 386–398.

17.

Aqil

Lippeveld

Hozumi

. PRISM framework: a paradigm shift for designing, strengthening and evaluating routine health information systems. Health Policy Plan 2009; 24(3): 217–228.

18.

Khoja

Durrani

Scott

, et al. Conceptual framework for development of comprehensive e-health evaluation tool. Telemed E-Health 2013; 19(1): 48–53.

19.

Enam

Torres-Bonilla

Eriksson

. Evidence-based evaluation of ehealth interventions: systematic literature review. J Med Internet Res 2018; 20(11): e10971.

20.

Uganda Ministry of Health . Uganda National eHealth Strategy 2017-2021 [Internet]. Uganda Ministry of Health, 2016. Available from: https://health.go.ug/sites/default/files/Nationale_Health%20Strategy_0.pdf

21.

Kruus

. Developing an evaluation framework for the country-wide electronic prescribing system in Estonia. Tallinna Teh 2013.

22.

Ashaba

Nabukenya

. Assessing evaluation of ehealth interventions in Uganda: practices, challenges and insights. J Health Inform Afr 2020; 7(1): 12–23.

23.

Venable

Pries-Heje

Baskerville

. FEDS: a framework for evaluation in design science research. Eur J Inf Syst 2016; 25(1): 77–89.

24.

Cruz

AMRD

. Design science research for IS/IT projects: focus on digital transfo rmation. In: 2020 15th Iberian Conference on Information Systems and Technologies (CISTI), 2020, 1–6.

25.

Hevner

Chatterjee

. Design science research in information systems. In: Hevner

Chatterjee

(eds). Design research in information systems: theory and practice [Internet]. Boston, MA, USA: Springer US, 2010, 9–22. [cited 2022 Mar 24]. Available from: https://doi.org/10.1007/978-1-4419-5653-8_2

26.

Mugisha

Nankabirwa

Tylleskär

, et al. A usability design checklist for mobile electronic data capturing forms: the validation process. BMC Med Inform Decis Mak 2019; 19(1): 1–11.

27.

Aier

Fischer

. Criteria of progress for information systems design theories. Inf Syst E-Bus Manag 2011; 9(1): 133–172.

28.

Ashaba

Nabukenya

. eHealth evaluation framework validation questionnaire [Internet]. figshare; 2022 [cited 2022 Oct 18]. Available from: https://doi.org/10.6084/m9.figshare.21354294

29.

Ashaba

Nabukenya

. Derivation of eHealth monitoring and evaluation reference model domains and subdomains. figshare, 2022. [Internet][cited 2022 Oct 18]. Available from: https://doi.org/10.6084/m9.figshare.21354039

30.

United Nations Development Group . Results-based management handbook: harmonizing rbm concept and approaches for improved development results at country level, 2011.

31.

Regional Committee for Africa 63 . Utilizing ehealth solutions to improve national health systems in the African Region (Document AFR/RC63/9) [Internet]. WHO. Regional Office for Africa, 2013. Report No.: AFR/RC63/R5. Available from: https://apps.who.int/iris/handle/10665/94187

32.

World Health Organization, International Telecommunication Union . National eHealth strategy toolkit. International Telecommunication Union, 2012.

33.

Ashaba

Nabukenya

. Example eHealth monitoring and evaluation indicators and their operationalization tools. [Internet]. figshare, 2022. [cited 2022 Oct 18]. Available from: https://doi.org/10.6084/m9.figshare.21354183

34.

O’Malley

Perdue

Petracca

. A framework for outcome-level evaluation of in-service training of health care workers. Hum Resour Health 2013; 11(1): 1–12.

35.

Ee Dawn Kit

. Towards a professional development framework for mathematical modelling: the case of Singapore teachers. ZDM 2018; 50(1): 287–300.

36.

Al Nadabi

ZSH

. A validation framework for an online English language Exit Test: A case study using Moodle as an assessment management system. The University of Queensland, 2017.

37.

Hongoro

Rutebemberwa

Twalo

, et al. Analysis of selected policies towards universal health coverage in Uganda: the policy implementation barometer protocol. Arch Public Health 2018; 76(1): 1–12.