Strengthening the evidence base for mHealth in clinical practice: Conducting research with standalone or interoperable systems – a viewpoint

Abstract

Objective

The aim of this viewpoint is to inform mobile health (mHealth) evidence development in using standalone or interoperable systems in hospital practice.

Methods

There is a gap between mHealth research and its widespread uptake in clinical practice. Evidence generation is not keeping up with the introduction and implementation of technologies. This is partly a consequence of the technology characteristics and the way research is conducted in a clinical setting. Research and development of mHealth technology can be conducted standalone in a laboratory like setting, standalone in a clinical setting or interoperable with already existing technology in hospital practice.

Results

Standalone systems operate relatively independent from an organizations' existing infrastructure. Using laboratory settings does not reflect the complexity of real-life, but in clinical practice this may be suitable for research assessing usability, feasibility or even clinical and process outcomes at a small scale. Realizing research and development on interoperable mHealth technology solutions, especially with operational EMR systems, is a challenging, time- and resource intensive process and requires large(r) investments, as it is often complicated by a myriad of interfering factors. Interoperable systems are however a more sustainable option in the long run, and generated evidence reflects the real hospital care setting and this option may therefore facilitate dissemination. Choosing either a standalone or interoperable setting affects the research design, the implementation pace and ultimately widespread adoption of the mHealth technology.

Conclusion

We recommend to include these technology characteristics in implementation frameworks and think of evaluation research designs in an early phase.

Introduction

The use of technology to improve health, well-being and healthcare ¹ changes healthcare delivery. Reflecting the societal trend towards digitalization but also as a possible contribution in delivering patient-centred and cost-effective care. ² Digital health is a broad term encompassing, e.g. telehealth, telemedicine, mobile health (mHealth) and electronic medical records (EMRs). ³ The EMR is considered the primary software system in hospital care and is used by healthcare professionals from one healthcare organization. ⁴ The COVID-19 pandemic highlighted the importance of the use of telehealth to provide care from a distance, ⁵ for example, by using video consultation or remote patient monitoring. Market and technology push are very strong in this field, and the use of telehealth solutions is especially promoted as it may lead to reduction in hospital visits and hospitalizations. Telehealth was so far predominantly introduced and evaluated for chronic conditions, especially heart failure and chronic obstructive pulmonary disease, with only small to moderate effects in well-designed evaluations. ⁶ Telehealth services are often delivered with mobile applications and devices. Mobile health (mHealth) is referred to as a medical and public health practice supported by mobile devices for communication, information access, health monitoring ⁷ and behaviour change purposes. ⁸ Mobile devices and wearables are widely used in everyday life, but application in healthcare is often lagging behind. In hospital practice (or in some countries: the domain of medical specialist care), mobile devices and applications are used for remote patient monitoring, education and self-management purposes. Although positive effects are reported for mHealth initiatives on, for example, chronic disease management ⁹ and health outcomes, ⁸ evidence on its effectiveness varies ¹⁰ and is limited, ⁹ and widespread use remains challenging.^9,11 Many initiatives remain in the pilot phase, ¹² and there remains a gap between research and uptake in clinical practice. ¹³ mHealth implementation in hospital care, diffusion on a large scale and its translation in transformation of care are not accomplished yet or at least delayed. Implementation of mHealth technology is complex and affected by multiple factors, of which the technology setup is a crucial factor. Different stakeholders, such as clinicians, hospital executives and IT innovators, have to deal with this issue from their various perspectives. Sufficient and adequate evidence is required to support decision-making on the implementation of innovative mHealth technology in hospital practice, appropriate budget allocation and coverage decision, as well as for professionals to gain confidence in adopting it in practice. ¹⁴ As sufficient evidence on effectiveness and efficiency is often lacking, budget impact analysis can only be based on lacunar feasibility or pilot evaluation data; obtaining a realistic overview of these aspects let alone a proper cost-effectiveness analysis is a challenge. Information on technology characteristics and setup in hospital practice has to be part of this evidence development.

Generating evidence is affected by the technology setup, and the use and effect of mHealth technology solutions in hospital practice can either be evaluated by using (1) standalone systems in a laboratory setting (e.g. academic environment) or in an (isolated) clinical setting or using (2) interoperable systems that operates by definition in the clinical practice setting and linked to the EMR. Presently there is no guidance based on scientific evaluation to decide on these options in research and development (R&D) in hospital practice.

Aim of this viewpoint

The aim of this viewpoint is to inform mHealth evidence development in using standalone and/or interoperable systems in hospital practice. We describe the differences and the consequences of the two options in relation to research and early-stage implementation. At first, the challenges on combining evidence development and (early-stage) mHealth implementation in hospital practice are explained, followed by the differences between standalone and interoperable systems. Finally, practical insights for applicability in hospital care settings, implications for research and support in deciding on the appropriate research setup are provided.

Challenges in combining mHealth evidence development with early-stage implementation

Several issues occur when combining research with mHealth implementation in clinical practice. A first challenge, related to the uptake of mHealth, is clarified by using the innovation S-curve (see Figure 1). The development of an innovative technology, such as a mHealth solution, starts from a new angle with often a lower initial quality or performance level and then accelerates; this is followed by maturity and eventually the next decline phase.^15,16 Especially when the existing technology is at the end of its life cycle curve or incremental innovation is simply not possible, a dynamic phase commences. To be successful in the healthcare setting, an upcoming technology should rather be of equal quality or provide direct added value or be likely to provide that soon.^17,18 As technology push is often strong and digital solutions commonly enter the market without a proper research base, formal implementation in terms of coverage may be even more challenging.

Figure 1.

Innovation S-curve, from Apreda et al., 2016. ¹⁶

A second issue lies in the gap between conducting research in hospital practice and the rapid pace of technology development, with new versions of digital solutions entering the market. Conducting decent medical research takes time, especially since the randomized controlled trial is still seen as the gold standard. ¹⁹ Therefore, published results may be outdated once the study is finished. ¹³ To increase the efficiency of the R&D process in hospital practice, speed up the process and gain more evidence from research in practice, different research designs might be useful such as pragmatic trials.^19,20 A pragmatic approach can actually support decision-making on uptake in daily clinical practice, because in pragmatic trials effects are measured in clinical practice with relevant outcomes for key stakeholders^13,19 in, for example, hospital practice. Although the assessment of treatment and patient-related outcomes is preferable, (other) proximal outcomes can also be chosen, ²¹ especially since these are more directly affected by the intervention,^13,22 for example, mHealth technology use and acceptance.

A third and related issue is that the decision to conduct mHealth research using standalone or interoperable systems, especially with operational EMR systems, is often underexposed in scientific research. Various frameworks are available as a guideline for implementation and evaluation, such as the Consolidated Framework for Implementation Research, ²³ the CeHRes roadmap ²⁴ and the NASSS (non-adoption, abandonment, scale-up, spread, sustainability) framework. ²⁵ As is clarified in these frameworks, implementation is a complex process^23–25 and is facilitated or hindered by a set of interacting factors.^25,26 A key element in implementation is the technology itself^27,28 and the setting in which it is implemented. The human, organization and technology-fit (HOT-fit) framework, to evaluate health information systems, for example, illustrates the interaction of relevant factors such as technology, human and organization. To our knowledge, no comprehensive model is available to evaluate the differences in, and consequences of, using standalone or EMR interoperable systems, in mHealth. Decision-making on using standalone or interoperable systems is certainly relevant because it adds dynamics that affect the research design and the pace of research and of possibilities of adoption, especially in hospital practice.

Evidence development and early-stage implementation: evaluation with standalone and interoperable systems?

Standalone systems

Standalone systems can be evaluated in a laboratory-like setting, for example, in an academic setting, with limited outside influences. However, when a study runs out of funding, ²⁰ it may not be possible to test the intervention in clinical practice using the real-world environment. By exception, it is possible to proceed from a laboratory setting to daily practice without much ado, and more often the complexity of the real-life setting requires additional adaptations or investments. The interaction with or dependency on primary hospital information systems is often not evaluated nor clarified a priori in research projects. Working with completely functional IT mock-up systems could be a solution in the R&D phase. However, this is usually too expensive and cumbersome. IT systems that are operational in daily clinical practice such as the EMR in hospitals often lack innovative features since these are developed for ‘standard’ use on a large scale. Innovations within these systems are commonly only provided in case of high demand from larger numbers of organizations or professionals. The dependency on the R&D planning of large software suppliers, for which competition is often limited, can be a barrier for innovative hospital organizations. Technology start-up companies often fill this gap and are leading in providing innovative and often standalone mHealth technology solutions. Before a mHealth solution possibly results in a sustainable implementation solution, it involves prototyping and testing. ²⁹ This emphasizes the relevance for decision-making on using standalone or interoperable mHealth technology in hospital practice, especially in early-stage implementation.

Clinicians commonly prefer the least possible numbers of clicks in using systems and balance this against perceived added value and speed of implementation of standalone features. Standalone systems are easier to study in a laboratory-like setting (e.g. academic environment) or even in clinical practice, because they run relatively independent of primary hospital IT systems. The EMR is considered the primary system for healthcare professionals in hospital practice. Increasingly, connectivity with patients is added through patient portals ³⁰ and mHealth technology, for example, for remote patient monitoring. New mHealth solutions are often provided by start- and scale-ups, with often a vulnerable position ²⁶ in a competing market but also with higher levels of flexibility. Standalone systems are in limited extent dependent on an organizations’ technical infrastructure and therefore less complex to implement and conduct research upon. Integration with existing care processes, and required time investment from professionals for implementation, is limited. However, healthcare professionals work in different systems, and, therefore, this setup can delay acceptance and affect research outcomes. Research using a standalone mHealth technology provides the possibility to evaluate mHealth technology in clinical practice on a small scale, to clarify users’ satisfaction and technology acceptance, at a relatively rapid pace, without doing large investments.

Overall, standalone systems are beneficial in isolated processes, with limited dependency on existing care processes and systems. These systems are often tailored to specific wishes and needs from organizations or healthcare professionals, leading to increased usefulness and quality of information for a specific domain. Standalone systems provide good solutions to move forward with new initiatives but with risk of failure on the long term.

Interoperable systems

Interoperability is necessary to achieve integration between mHealth technologies from third parties and already existing systems in hospitals, such as the EMR. This can be challenging since this requires collaboration between multiple organizations, their investment agendas and procedures. ³¹ In this case, the interoperability framework is used to illustrate interoperability of different systems within one organization. Agreements on multiple levels are needed: legal and regulatory, policy, care process, information, applications and IT infrastructure.^31,32 Legal and regulatory agreements are always a precondition for implementation in clinical practice. Integrating a mHealth technology in an organization’s information and technical infrastructure should also be in line with a care organizations’ policy, for example, regarding data processing and data protection. ³² Lack of integration in a care process is seen as a barrier in previous studies^33,34 and can hamper the enthusiasm of users (e.g. doctors and nurses). Determining which information should be transferred between these systems and with which level of detail is important to decide how information is being exchanged with the use of the new application. Technical specifications about the new application(s) are also necessary to assess the level of complexity (e.g. regarding technology standards) in order to achieve interoperability on application and IT infrastructure level (see Figure 2).

Figure 2.

Model for interoperability: the Nictiz layer model. ³²

Interoperability may have a positive effect on implementation and uptake^34,35 and may provide a sustainable solution in achieving upscaling for mHealth solutions. Implementation of interoperable mHealth solutions is complex, especially in combination with research on possibly disruptive digital technology that can interfere with the hospitals’ investments and software version update calendar. Dependency on the planning of large (EMR) software suppliers, with often a monopoly position, can delay the implementation process. The EMR, for example, is frequently updated with new releases and planned updates that can be a precondition to achieve interoperability and may also require alignment with the investment calendar of the organization. Support from senior management is essential as various stakeholders are involved ³⁴ and financial resources are required. mHealth solutions interoperable with existing hospital information systems for research purposes have the advantage of exact reflection of use in clinical practice and will enable healthcare professionals to work in one system, therefore not unnecessarily complicating their work. However, implementation is complex due to dependencies of various stakeholders, implementation planning that is often dominated by the hospital's operational priorities and dependency of external software suppliers.

Although the implementation of interoperable systems is more complex, time-, resource- and cost-intensive, it is a more sustainable solution on the long term. Interoperable system implementation is more complex but reflects a real-life setting in conducting research where a standalone system is less complex but cumbersome. Assessment of the need for interoperability with the existing hospital information infrastructure is an important factor to successfully implement mHealth technology in hospital care settings and to support decision-making about the technology setup for research purposes. ³⁶

Implications for research

Standalone systems are useful for conducting research: (1) relatively independently from hospital IT systems; (2) to assess technology usability, feasibility, and users’ acceptance on a small scale; (3) with fixed budget and resource allocation; and (4) as a proof of principle or as a pre-phase for use on a larger scale or to prepare for interoperability with existing infrastructure.

Interoperable systems, especially with the EMR, are useful for conducting research: (1) to approximate technical conditions in complex hospital care settings, (2) to assess a broad range of process and clinical outcomes reflecting use in daily clinical practice, (3) to realistically estimate budget impact or cost-effectiveness for broader implementation in hospital practice, and (4) to enable large-scale use by most providers that are not early adopters.

A disadvantage of standalone approaches can be the fact that budget impact and cost-effectiveness might not be estimated in a realistic way. In contrast, in case of cost-effectiveness data on interoperable systems resulting in unfavourable conclusions, de-implementation might be needed.

Not all mHealth implementations are science-based. Technology companies’ approach, for example, mHealth, is development of their product. Evidence development following the – more well-known – traditional approach of translational research is lacking. In an evidence-based environment, such as healthcare, this impedes adoption of wide-scale use in daily practice, where also evidence development is complex without adoption in practice. This highlights the need for innovative, pragmatic, research approaches. ³⁷

Conclusions

The technology setup, standalone or interoperable, for mHealth technology solution implementation and evaluation may affect research design, time to launch, implementation pace, adoption and even the intervention outcomes. mHealth technology can be evaluated standalone in laboratory settings, commonly not reflecting the complexity of real-life hospital care, or in an isolated clinical setting. Standalone mHealth technology in an isolated clinical setting, such as hospital practice, may be suitable for research conducted relatively independent from hospital IT systems, such as the EMR. Research aimed at assessing, for example, feasibility and users’ acceptance on a small scale, against relatively low cost with a fixed budget and resource allocation. It can be also beneficial as a pre-phase for use on a larger scale with interoperable systems. Interoperable mHealth solution, especially with the EMR, is a more sustainable option to assess a broad range of outcomes, to better estimate budget impact and cost-effectiveness to predict success at a wider scale in daily clinical practice. Realizing EMR interoperable mHealth solutions is a challenging, time- and resource-intensive process and requires large(r) investment, as it is often complicated by a myriad of interfering factors such as technology, organizational and individual factors. We recommend to include the technology design and technology characteristics in implementation frameworks and research designs with assessment a priori. More scientific and pragmatic evaluation with different IT systems in hospital care setting is needed to support implementation and diffusion on a large scale and to translate in transformation of care.