Sage Journals: Discover world-class research

Abstract

This study reviews the quality of evidence reported in mobile health intervention literature in the context of developing countries. A systematic search of renowned databases was conducted to find studies related to mobile health applications published between a period of 2013 and 2018. After a methodological screening, a total of 31 studies were included for data extraction and synthesis. The mobile health Evidence Reporting and Assessment checklist developed by the World Health Organization was then used to evaluate the rigor and completeness in evidence reporting. We report several important and interesting findings. First, there is a very low level of familiarity with the mobile health Evidence Reporting and Assessment checklist among the researchers and mobile health intervention designers from developing countries. Second, most studies do not adequately meet the essential criteria of evidence reporting mentioned in the mobile health Evidence Reporting and Assessment checklist. Third, there is a dearth of application of design science–based methods and theory-based frameworks in developing mobile health interventions. Fourth, most of the mobile health interventions are not ready for interoperability and to be integrated into the existing health information systems. Based on these findings, we recommend for robust and inclusive study plans to deliver highly evidence-based reports by mobile health intervention studies that are conducted in the context of developing countries.

Keywords

mERA developing nations evidence reporting mobile health systematic review

Introduction

It is predicted that the total market size of mobile health, also known as mHealth, will be almost 60 billion dollars by 2020.¹ There are more than 100,000 mHealth applications (apps) available in the Google Play and Apple’s App Stores² that are either available for free or at a minimal cost to the users. Mobile technologies (smartphone, smartwatch, etc.) nowadays are integrated with a plethora of sensors and sophisticated functions. The increasing adoption of these smart devices in developing countries provides opportunities for mHealth applications to deliver versatile and useful healthcare services where clinical resources are in short supply. As the demand for quality healthcare increases, a global shortage of healthcare resources, including nurses and physicians, exists, specifically in underdeveloped countries. To meet such demand, healthcare providers are increasingly relying on technology such as mHealth platforms as a feasible substitute for in-person interactions. As healthcare systems in developing countries are considered immature and often times dysfunctional, the mHealth technology can become a disruptive intervention with positive impact in improving the health of ordinary citizens and to deliver quality care to inaccessible population. In efforts to accelerate progress in achieving sustainable development goals, especially in developing countries, the World Health Organization (WHO)³ has identified the use of mHealth technology by the public as an important step in providing a cost-effective, convenient, and transparent access to healthcare.

mHealth technology was designed to improve the quality of care and to minimize costs to patients. To achieve these goals, mHealth technology provides several important applications, including educating and awareness building, providing clinical and non-clinical decision support systems (DSSs), epidemic outbreak tracking, training of healthcare workers and remote monitoring, and many others.⁴ Despite the availability of thousands of mHealth apps and tremendous prospects and broader scopes of this technology that are beneficial for managing health, the adoption of mHealth in developing countries is relatively slow. Few pieces of research, to date, are focused to understand this low adoption phenomenon. Reviews of existing literature suggest several barriers contribute to the low adoption of mHealth technology in developing countries. These barriers include inadequacy in health literacy, lack of effectiveness, safety, privacy, awareness, and poor integration with the traditional healthcare system.² Given these known barriers of mHealth technology adoption, researchers have called for more research to understand this issue from a different perspective.

In the early days of mobile phone adoption, the domain of mHealth was mostly limited to conceptual and proof-of-concept stages. At the time, the matters of mass adoption, scalability, and effectiveness of the technology were not thoroughly considered by researchers and practitioners. Consequently, the lack of evidence of mHealth effectiveness led to the deferral of many mHealth projects.⁵ According to the World Bank, mHealth interventions about better public health outcomes show very limited evidence.⁶ Given the lack of evidence, researchers call for broader and more adaptive evaluation plans to identify the evidence of positive outcomes, scalability, and sustainability for mHealth intervention projects.⁷ Evaluation of reporting of evidence of mHealth technology remains an unexplored area of research and, thus, very important and utmost interest of Information Systems (IS) researchers in recent years.⁸

The success of any new technology relies on the successful integration, diffusion, and continued use by intended users. Evaluation of success or failure is critical and one of the essential parts of technology implementation⁹ that has to be continuous, comprehensive, and systematic to find evidence of what results have been achieved against the target. Thus, the development and use of standard criteria for the evaluation of mHealth interventions is necessary for evidence reporting from the mHealth studies. The evidence reporting by mHealth evaluation process helps in mHealth policy formation as well as in mHealth intervention selection, promotion, and program expansion.¹⁰ Evidence of mHealth efficiencies and impacts are generally reported by publishing journal articles, white papers, reports, exhibits, and news articles. However, these publications are often biased, incomplete, and incomparable; lack study quality; and have discrepancies in research findings. Many of these problems are due to the absence of standard and rigorous evaluation methodologies for reporting evidence for improving healthcare services by mHealth interventions. Because of these reasons, in 2016, the WHO collaborated with mHealth researchers to formulate an evaluation guideline, known as the mHealth Evidence Reporting and Assessment (mERA) checklist, for evidence reporting on mHealth interventions.⁵

The mERA checklist enables researchers to follow a reliable and quality methodology for scrutinizing, reporting, and evaluating evidence of mHealth interventions. Researchers in the field believe that the importance of such methodology-specific evaluation guidance for evidence reporting is substantial and necessary for the effective integration of mHealth technology. Hence, many mHealth studies have recommended for having rigorous methodologies and proper guidelines for mHealth intervention evaluation and evidence reporting.⁵

Objective

Informed and motivated by the significance of evidence reporting, this study aims to review the quality of evidence reporting for the mHealth intervention studies in developing countries. As an evaluation tool, we used the mERA checklist to evaluate the quality of evidence reporting in selected studies. According to the best of our knowledge, no similar study to date has been found focusing on this topic in the context of developing countries.

There are different types of evaluation guidance for reporting evidence of technology-based healthcare interventions. For example, Grading of recommendations assessment, development, and evaluation (GRADE),¹¹ Consolidated standards of reporting trials (CONSORT),¹² Collaborative Adaptive Interactive Technology (CAIT) framework,¹³ Mobile Application Rating Scale (MARS),¹⁴ Trial of Intervention Principles (TIP) framework,¹⁵ mHealth Assessment and Planning for Scale (MAPS) Toolkit,¹⁶ and WHO’s mHealth Evidence Reporting and Assessment (mERA) checklist.¹⁰ In this study, we used mERA as it is the most recent, concise, and specific to mHealth interventions. In addition, researchers use mERA as it improves the completeness of reporting of mHealth interventions.^10,17,18

Methodology

We conducted this study in four steps: database search, relevance appraisal, data extraction, and evaluation of evidence reporting. As this is a systematic review of existing literature, all steps are qualitative.

Database search

In this step, a systematic search of six prominent online journal databases was carried out to find extant relevant mHealth studies that are in the context of developing countries. The selected studies were published within the timeframe of the last 5 years from 2013 to 2018. We followed the search strategies of systematic reviews and executed on the selected database to fetch related studies.¹⁹ To do so, we first identified three unique search domains that are related to mHealth research. These search domains include “mobile technology,” “health,” and “developing countries” and are specified in Table 1. We then specified multiple combinations of search terms that belong to our search domains also shown in Table 1. These terms were used to perform a thorough search of six online journal databases. The lists of selected databases are specified in Table 2. For the inclusion of an article, at least one search term was required from each of the three search domains. In addition to the database search, we performed a manual search based on references available in various studies. The initial search query returned with a total number of 39,152 articles.

Table 1.

Search terms.

Search domains	Search terms
Mobile technology	“mobile health” OR “mHealth” or apps OR “mobile support” OR “mobile intervention” OR “mobile technology” OR “mobile app” OR “mobile application”
	AND
Health	“healthcare” OR “health care” OR “health” OR “clinical” OR “health promotion” OR “follow up” OR “health support” OR “decision” OR “support”
	AND
Developing countries	“developing country” OR “developing nation” OR “low income”
Not included in the search	“telehealth” OR “telemedicine” OR “ehealth” OR “digital technology” OR “wearable technology” OR “digital media” OR “mobile technology” OR “social media” OR “helpline” OR “hotline” OR “SMS” OR “text messaging” OR “messaging”

Table 2.

Database searching summary.

Database	Search field	The initial number of studies in the search list
PubMed	All	1332
ScienceDirect	All	20,600
ProQuest Central	All	1614
Wiley Online Library	All	238
EBSCOhost	All	2068
Google Scholar	All	13,300
Total		39,152

Relevance appraisal

For relevance appraisal, the authors followed the PRISMA 2009 flow diagram known as “Preferred Reporting Items for Systematic Review and Meta Analyses” as recommended by L’Engle et al.²⁰ Figure 1 shows the results of the following PRISMA flow diagram. PRISMA assisted two authors to select the most potentially relevant studies based on various exclusion and inclusion criteria, details of which are provided in Figure 1 as well. For exclusion, the authors initially screened the title and abstract of every study based on the exclusion criteria. For inclusion, the full texts of the studies were screened carefully using the various inclusion criteria. The inclusion criteria were developed based on a specific framework known as the PICO framework. PICO framework is known to be used in evidence-based medicine to frame and answer a series of healthcare-related questions. PICO stands for Problem, Intervention, Comparison, and Outcome,²¹ which uses questions and answers to facilitate a literature search. Following the PICO framework, the authors had to have mutual consent for the exclusion and inclusion of any studies found in the initial database search. The number of identified, excluded, and included studies through the PRISMA flow diagram and PICO framework is shown in detail in Figure 1.

Figure 1.

Results of PRISMA flow diagram.

Data extraction

Once irrelevant studies are excluded, we focus on a much smaller set of studies that were considered relevant for this research. In this step, our goal is to extract data from these studies. During the data extraction process, one reviewer was tasked with extracting data and coding while the second reviewer rechecked the accuracy of the first reviewer’s data extraction. The data extraction process and coding are significant for data synthesis, which involved keeping notes about the name of mHealth interventions, targeted population, context/country, study objective, methods, and results. Furthermore, the selected articles were categorized according to their characteristics.

Evaluation of evidence reporting

For evaluating the quality of evidence reporting of each study, we used the mERA checklist following the guidelines by Iribarren et al.⁸ In this step, we graded the evidence related to mHealth intervention in each study based on the criteria specified in the mERA checklist.¹⁰ The mERA checklist has two parts, where the first part is known as the core checklist that has 16 essential criteria to evaluate the completeness of reporting. The second part of the mERA checklist is a checklist with 29 methodology criteria to evaluate the evidence reporting and to synthesize study findings (Supplemental File). While the core checklist focuses on the content, context, and technical aspect of the studies, the second part, on the other hand, focuses on the study designs and methodologies.^10,20 Table 3 shows two parts of the mERA checklist as discussed.

Table 3.

mERA checklist (details in the attached Supplemental Information).

Core/essential mHealth criteria	Methodology reporting criteria
1. Infrastructure 2. Technology platform 3. Integration/interoperability 4. Intervention delivery 5. Intervention content 6. Usability 7. User feedback 8. Access 9. Cost assessment10. Adoption inputs11. Scalability12. Contextual adaptability13. Replicability14. Data security15. Compliance16. Fidelity of the intervention	1. Rationale and scientific background 2. Objectives and hypotheses of the studies 3. Logical model and theoretical framework 4. Study design 5. Intervention outcomes 6. Data collection methods 7. Participant eligibility 8. Participant recruitment 9. Biasness10. Sampling11. Setting and locations12. Comparator13. Data sources and measurement14. Enrollment15. Description of the study population16. Reporting on outcomes17. Summary of evidence18. Limitations19. Generalizability and external validity20. Conclusions and interpretation21. Funding22. Ethical considerations23. Competing interests24. Statistical methods25. Missing data26. Attribution27. Analytical methods28. Data validation29. Reflexivity of account provided

Core/essential mHealth criteria

Methodology reporting criteria

1. Infrastructure 2. Technology platform 3. Integration/interoperability 4. Intervention delivery 5. Intervention content 6. Usability 7. User feedback 8. Access 9. Cost assessment10. Adoption inputs11. Scalability12. Contextual adaptability13. Replicability14. Data security15. Compliance16. Fidelity of the intervention

1. Rationale and scientific background 2. Objectives and hypotheses of the studies 3. Logical model and theoretical framework 4. Study design 5. Intervention outcomes 6. Data collection methods 7. Participant eligibility 8. Participant recruitment 9. Biasness10. Sampling11. Setting and locations12. Comparator13. Data sources and measurement14. Enrollment15. Description of the study population16. Reporting on outcomes17. Summary of evidence18. Limitations19. Generalizability and external validity20. Conclusions and interpretation21. Funding22. Ethical considerations23. Competing interests24. Statistical methods25. Missing data26. Attribution27. Analytical methods28. Data validation29. Reflexivity of account provided

mERA: mobile health Evidence Reporting and Assessment.

Before applying the grading technique, close attention was paid to practical applications of the mERA checklist and its reliability. For grading, the inter-rater reliability is significant for reliable qualitative studies where more than one rater or grader is involved in grading the contents of data. Inter-rater reliability measures the degree of agreement between the raters. As our grading procedure involved multiple graders, it was important and necessary to test their inter-rater reliability. To do so, we performed a pre-evaluation where three random studies were selected by the following recommendation put forwarded by Agarwal et al.¹⁰ In the grading process, the graders assigned a value of either 1 or 0 for each of the studies, where 1 means that a specific evaluation criterion of mERA is met, and 0 means that a specific evaluation criterion of mERA is not met. To assess inter-rater reliability, we used Cohen’s kappa coefficient (K).²² It is important to note that in the case of grading healthcare studies, K value as low as 0.41 can be acceptable.²³ The pre-evaluation results show that the inter-rater reliabilities were 0.87, 0.87, and 0.92 for the three randomly selected studies, which are much higher than the recommended acceptable threshold. The pre-evaluation results confirm the reliability of the mERA evaluation technique that we will follow for the actual evaluation. In the next section, we discuss the results for the main evaluation of the selected studies.

Results

Categories of studies

The full-text screening and PICO evaluation during the PRISMA process returned a total of 31 studies for final evaluation consideration. We now classify these studies based on several categories—study types (qualitative study, quantitative study, and mixed study), formative versus summative, service types, health issues, and study location. These classifications were made by carefully reading the 31 selected articles’ abstracts, objectives, and methodologies. Table 4 shows the classifications of the selected studies, their respective frequency (n), and distribution percentages for each of the categories.

Table 4.

The classifications and percentage of the types of studies.

Categories of studies	Percentage
Study type (n = 31)
Qualitative (n = 4)	13
Quantitative (n = 23)	74
Mixed (n = 4)	13
Formative evaluation (n = 9)
Control group (n = 2)	22
Observational (n = 1)	11
Descriptive (n = 6)	67
Summative evaluation (n = 22)
Control group (n = 5)	23
Observational (n = 5)	23
Descriptive (n = 12)	55
Service types (n = 31)
Decision support (n = 3)	10
Diagnosis/screening/assessment (n = 9)	29
Follow-up/adherence/track/support (n = 14)	45
Surveillance (n = 3)	10
Appointment (n = 1)	3
Information/awareness (n = 1)	3
Health issues (n = 31)
HIV (n = 2)	6
Maternal/antenatal (n = 6)	19
Stroke (n = 1)	3
Hypertension (n = 2)	6
Cognitive development (n = 1)	3
Diabetes (n = 2)	6
Kidney (n = 1)	3
Smoking (n = 3)	10
Sexual education (n = 1)	3
Drug addicts (n = 1)	3
Mental health (n = 1)	3
Cancer (n = 1)	3
Pneumonia (n = 1)	3
Weight loss (n = 1)	3
Sexual violence (n = 1)	3
Dengue surveillance (n = 1)	3
Infectious disease surveillance (n = 1)	3
General support (n = 4)	13
Study locations (n = 31)
Asia (n = 17)	55
Africa (n = 8)	26
Europe (n = 2)	6
Latin America (n = 3)	10
Middle America (n = 1)	3

For the first classification category, 13 percent of the studies are classified as qualitative, 74 percent are quantitative, and 13 percent are mixed studies. Based on the methodologies used for evidence reporting, the selected studies are then categorized as formative and summative methodologies. The formative evidence reporting methodology involves the evaluation of evidence during application design and prototype testing phases of mHealth intervention, whereas the summative evidence reporting methodology involves the evaluation of evidence during launches and pilot phases.²⁴ A total of nine studies (29%) are categorized as formative, while 22 studies (71%) fit the criteria of summative studies. In addition, 11 percent of the formative and 23 percent of the summative evaluations are observational study, whereas 67 percent of the formative and 55 percent of the summative evaluations are descriptive. These classifications are important to know in which phases the mHealth intervention was mostly reported about evidence.

While making categories of the studies according to the service types, most of the studies belong to follow-up (45%) and disease diagnosis (29%) categories. Furthermore, the percentage of service types like decision support and surveillance is about 10 percent. In addition, of 31 studies, maternal/antenatal and cessation of smoking support were the most focused area (19% and 10%, respectively). Finally, of the five different locations, Asian and African countries are the most studied context with 55 and 26 percent, respectively.

Study summaries and synthesis

For synthesizing data, we extracted specific data to summarize all 31 studies. This was done by reading their study objectives, methodologies, and results. Supplemental Appendix 1 individually shows the targeted population, study contexts, study objectives, methodologies, and the results of each study. The findings show that there is a big demand for tailored services in the area of a DSS such as a smoking cessation program and hypertension management where mHealth shows its useful application. Moreover, the feasibility test of mHealth programs for people with depressive disorders shows that mHealth is useful in saving time and expense. Ginsburg et al.²⁵ studied the usability and acceptability of the mHealth program for pneumonia and reported a positive outcome of mHealth intervention. Similarly, for chronic kidney disease, mHealth programs are found to be satisfactory to users. mHealth is also widely used in the dissemination of information for sexual education.²⁶

When summarizing the types of services provided through mHealth interventions, it was also found that antenatal service was mostly focused on the behavioral changes in seeking antenatal care services as well as on the satisfactory level of them to find evidence of the usefulness. Another study by Benski et al.²⁷ particularly focused on how to provide antenatal care services according to WHO guidelines using mHealth intervention. On the other hand, Gunawardena et al.²⁸ focused on diabetes and pre-diabetes management using mHealth intervention called Smart Glucose Manager, whereas Islam et al.²⁹ studied the impact of mHealth on the improvement of lifestyle of diabetic patients. Similarly, two other studies^30,31 focused on HIV prevention and management which is a significant area in mHealth study. mHealth programs reported in these studies also focused on the provision of healthcare to the elderly people,³² hazard reporting in a working site,³³ sexual violence reporting, and setting up appointments. In the areas of weight loss and diet control program, evidence of effective uses of mHealth is also reported.

mHealth is also used effectively for ecological momentary assessment (EMA) for tobacco and drug use, although EMA is not found effective as a self-report survey tool in China. Francis-Lyon et al.³⁴ found that mHealth is a useful intervention for conducting child cognitive assessment in a poverty-stricken context like in Kenya. García et al.³⁵ conducted a quasi-experiment on the early detection of stroke symptoms by developing a prototype app that was reported very useful. mHealth is also being used on the surveillance of the frequency of particular diseases related to socio-cultural and economic contexts, and as well as infectious diseases such as dengue transmission.

About the methodologies used for evidence reporting, the findings also show that the technique of randomized controlled trial (RCT) was highly preferred in identifying the impact of mHealth on HIV disease awareness and management, such as the studies by Guo et al.³¹ and Bull et al.³⁰ The RCT was also applied for evidence reporting on managing hypertension,³⁶ drug addiction,³⁷ and diabetes management²⁸ through mHealth intervention. We have noticed the effective application of theories for designing mHealth services. For example, Ilozumba et al.³⁸ used the Theory of Change to determine contextual factors that affect the outcomes of mHealth intervention for pregnant women. Similarly, Bernardes-Souza et al.³⁹ used the theory of planned behavior to examine the feasibility and user’s perception of facial-aging apps that can discourage secondary school students from smoking.

mERA checklist findings and scores

The mERA checklist was developed to systematically identify the inadequacy, biasness, and usefulness of evidence reporting in mHealth studies. This checklist evaluates the rigor and completeness in reporting. It was developed specifically with a focus on low- to middle-income country contexts. The list of criteria incorporated in the checklist can evaluate a wide range of study designs and can aggregate and synthesize a study based on relevant content. The checklist contains a total of 45 mERA evaluation criteria, 16 of which are essential mHealth criteria and 29 are reporting and methodology criteria, as shown in Table 3. As described in the methodology section, we graded every evaluation criterion of mERA with a value of either 1 or 0. The grading along with the relevant findings derived from 31 studies is presented in Tables 5 and 6. In each of the tables, the value of 1 means that the specific evaluation criterion is met in a corresponding study, and 0 means that the specific evaluation criterion is not met in a corresponding study. Findings for essential mHealth criteria and methodology criteria are discussed in the next sections.

Table 5.

Evaluating evidence using mERA Essential mHealth Criteria.

Authors of 31 studies		McNabb et al.⁴⁰	Gunawardena et al.²⁸	Benski et al.²⁷	García et al.³⁵	Francis-Lyon et al.³⁴	O’Connor et al.⁴¹	Alsharif and Philip⁴²	Soong et al.⁴³	Afroze et al.⁴⁴	Han et al.³⁷	Islam et al.²⁹	Lwin et al.⁴⁵	Musyimi et al.⁴⁶	Soto-Perez-De-Celis et al.³²	Rishikesh³³	Bull et al.³⁰	Lestantri et al.⁴⁷	Guo et al.³¹	Borsari et al.⁴⁸	Alnasser et al.⁴⁹	Praveen et al.³⁶	Ilozumba et al.³⁸	Malik et al.⁵⁰	Mishori et al.⁵¹	Li et al.⁵²	Ginsburg et al.²⁵	Lopez et al.²⁶	Miah et al.⁵³	Sobrinho et al.⁵⁴	Wang et al.⁵⁵	Bernardes-Souza et al.³⁹	Percent
Sl.	Essential mHealth criteria
1	Infrastructure	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	100
2	Technology platform	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	100
3	Integrate to existing system	0	1	1	0	0	1	0	1	0	1	0	1	0	0	0	1	0	0	1	0	0	1	0	1	1	1	0	0	0	0	0	39
4	Intervention delivery	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	100
5	Intervention content	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	100
6	Usability	1	1	1	1	1	1	1	1	0	1	1	0	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	94
7	User feedback	1	1	1	0	1	1	1	1	0	1	1	1	1	1	1	1	1	1	1	1	1	1	0	1	0	1	1	1	1	1	1	87
8	Access	1	1	0	0	0	0	0	1	0	1	0	0	1	0	0	0	0	0	1	0	1	1	0	1	0	0	0	0	0	1	0	32
9	Cost assessment	0	0	1	0	1	0	0	0	0	0	1	0	1	0	1	1	1	1	1	1	1	1	0	0	0	1	1	0	1	1	0	52
10	Adoption inputs (training)	1	1	1	0	0	0	0	1	0	0	0	1	1	1	1	1	0	1	1	0	1	1	0	1	1	1	0	0	1	0	0	55
11	Scalability	1	0	0	1	0	0	0	0	0	0	0	0	0	0	0	1	0	0	0	0	1	1	0	1	0	1	0	1	0	0	0	26
12	Contextual adaptability	0	0	1	0	1	0	1	0	0	1	1	1	0	0	1	0	0	1	0	1	1	0	0	1	1	1	1	1	0	1	1	55
13	Replicability	0	1	0	1	0	1	1	0	1	0	0	0	0	0	0	0	0	0	1	1	1	0	1	1	1	0	0	1	0	1	1	45
14	Data security	1	0	0	1	1	0	0	1	0	0	0	0	0	0	0	0	0	0	0	1	1	0	0	1	1	0	0	1	0	1	1	35
15	Compliance	1	1	1	0	1	0	0	1	0	0	0	0	0	1	0	0	0	0	0	1	0	0	0	0	0	0	0	1	1	0	0	29
16	Fidelity of the intervention	1	1	1	1	0	0	1	1	0	1	0	1	1	1	0	1	1	1	1	1	1	1	0	1	1	1	1	1	1	1	1	81

mERA: mobile health Evidence Reporting and Assessment. 1 = criterion is met, 0 = criterion is not met.

Table 6.

Evaluating evidence using mERA reporting and methodology criteria.

Authors of 31 studies		McNabb et al.⁴⁰	Gunawardena et al.²⁸	Benski et al.²⁷	García et al.³⁵	Francis-Lyon et al.³⁴	O’Connor et al.⁴¹	Alsharif and Philip⁴²	Soong et al.⁴³	Afroze et al.⁴⁴	Han et al.³⁷	Islam et al.²⁹	Lwin et al.⁴⁵	Musyimi et al.⁴⁶	Soto-Perez-De-Celis et al.³²	Rishikesh³³	Bull et al.³⁰	Lestantri et al.⁴⁷	Guo et al.³¹	Borsari et al.⁴⁸	Alnasser et al.⁴⁹	Praveen et al.³⁶	Ilozumba et al.³⁸	Malik et al.⁵⁰	Mishori et al.⁵¹	Li et al.⁵²	Ginsburg et al.²⁵	Lopez et al.²⁶	Miah et al.⁵³	Sobrinho et al.⁵⁴	Wang et al.⁵⁵	Bernardes-Souza et al.³⁹	Percent
Sl.	Reporting and methodology criteria
1	Rationale and scientific background	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	100
2	Objectives and hypotheses	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	100
3	Logic model/theoretical framework	0	0	1	1	0	1	1	1	0	0	0	1	0	0	1	0	1	0	1	1	1	1	0	1	0	1	0	1	1	1	1	58
4	Study design	1	1	1	1	1	1	1	1	0	1	1	1	1	1	1	1	1	1	1	1	1	1	0	1	1	1	1	1	1	1	1	94
5	Outcomes	1	1	1	1	1	1	1	1	0	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	97
6	Data collection methods	1	1	1	0	0	1	1	1	0	1	1	1	1	1	1	1	1	1	1	1	1	1	0	1	1	1	1	1	1	1	1	87
7	Participant eligibility	1	1	1	1	0	0	1	1	0	1	0	1	1	1	1	1	1	1	1	1	1	1	0	1	1	1	1	0	1	1	1	81
8	Participant recruitment	1	1	1	1	0	0	1	1	0	1	0	1	1	1	1	1	1	1	1	1	1	1	0	1	1	0	1	0	1	1	1	77
9	Bias	0	1	0	0	0	0	0	0	0	1	0	0	0	0	0	0	0	1	0	0	1	0	0	0	0	0	0	0	1	0	1	19
10	Sampling	1	1	1	0	0	0	0	1	0	1	0	1	1	1	0	1	0	1	0	0	1	1	0	0	0	1	1	0	0	1	0	48
11	Setting and locations	1	1	1	0	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0	0	1	1	1	0	1	1	1	87
12	Comparator	0	1	0	0	0	0	0	0	0	1	0	0	0	0	0	1	0	1	0	0	1	1	0	0	0	0	1	0	0	0	0	23
13	Data sources and measurement	1	1	1	0	0	0	0	1	0	1	0	1	1	1	0	1	1	1	1	1	1	1	0	1	1	1	1	1	1	1	1	74
14	Enrollment	1	1	1	0	0	0	0	1	0	1	0	1	1	1	0	1	1	1	1	0	1	1	0	1	0	1	1	0	0	1	1	61
15	Description of study population	0	1	1	0	0	0	0	1	0	1	0	1	1	1	0	1	1	1	1	0	1	1	0	1	0	1	1	0	1	1	1	61
16	Reporting on outcomes	1	1	1	1	1	1	0	1	0	1	0	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	90
17	Summary of evidence	1	1	1	1	0	0	1	1	0	1	0	1	1	1	0	1	1	1	1	1	0	1	0	1	1	1	1	1	1	1	1	77
18	Limitations	1	1	1	1	0	1	1	1	1	1	1	1	1	1	1	0	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	94
19	Generalizability and external validity	0	0	1	0	0	0	0	1	0	1	0	1	0	1	0	0	0	0	0	0	0	0	0	0	1	1	0	0	0	0	1	26
20	Conclusions and interpretation	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	100
21	Funding	1	1	1	0	0	1	0	1	0	0	0	1	0	1	0	1	0	1	1	0	1	1	0	1	1	1	1	0	1	0	0	55
22	Ethical considerations	1	1	1	0	0	0	0	1	0	0	0	0	1	0	0	1	0	1	1	0	1	1	0	0	0	0	1	0	1	1	0	42
23	Competing interests	1	1	1	0	0	1	0	1	0	1	0	1	1	1	0	1	0	1	0	1	1	1	0	1	1	1	1	0	1	1	1	68
25	Statistical methods	1	1	1	0	0			1	0	1	0	1	1	1	0	1	1	0	1		1	1	0		0	1	1	1	1	1	1	70
26	Missing data	0	1	0	0	0			1	0	1	0	0	1	0	0	0	0	1	0		0	0	0		0	0	0	0	1	0	0	22
27	Attribution	1	1	1	0	0			1	0	1	0	0	0	1	0	1	1	1	1		0	1	0		0	1	1	1	1	1	1	63
28	Analytical methods						1	1						1							1		1		1				1		1		100
29	Data validation						0	0						0							0		1		0				1		0		25
30	Reflexivity of account provided						0	0						0							0		1		0				0		0		13

mERA: mobile health Evidence Reporting and Assessment. 1 = criterion is met, 0 = criterion is not met.

Findings on the essential mHealth criteria

Table 5 shows the findings of the 16 essential mHealth criteria of the mERA checklist. All 31 studies included infrastructure-related discussion, which is the first item in the checklist. This criterion provides information about the minimum infrastructure required for each mHealth intervention and, thus, facilitates understanding of the viability, generalizability, and replicability of the mHealth interventions. In regard to the second item in the checklist, technology platform, all 31 studies once again cover discussion regarding this criterion without which it is not possible to identify the constraints that might exist and inhibit the implementation and replicability of a mHealth program. Item 3 in the checklist, integration/interoperability, however, was only covered by 12 of 31 or 39 percent of studies. This criterion is used to present a discussion on how the proposed or tested mHealth program fits and integrates with the relevant health information systems (HISs). The next two items in the checklist, intervention delivery and intervention content, are once again covered by all studies under consideration. While the fourth item in the checklist, intervention delivery, is used to describe the channels and mode of service delivery, the fifth item, intervention content, on the other hand, is used to discuss the characteristics, functionalities, and customizations of the contents that were being delivered. Intervention content also includes discussion regarding whether the contents were delivered according to relevant guidelines or no guideline was followed in the process. The analysis shows that 29 studies or 94 percent covered usability criteria, 87 percent of the studies considered users’ feedback, and only 32 percent of the studies considered identifying on any barrier or facilitator to access mHealth technology. These observations are significant because although the majority of the studies discussed usability as an important factor and considered users’ feedback regarding mHealth technology, very few authors went a step forward to identify the facilitators or the barriers to technology access.

The results of our analysis show that 52 and 55 percent of the studies, respectively, considered cost aspect and pre-adoption requirements (e.g. training) of mHealth adoptions. While only 26 percent of the studies considered scalability for mHealth interventions, 55 percent of the studies described contextual adaptability criterion. A total of 14 of the 31 studies or 45 percent provided any instruction or flowchart, or shared code(s) or algorithm for replicability. Another interesting finding is that only 35 percent of the studies discussed data security of mHealth application and only 29 percent of the studies discussed any compliance requirements by the mHealth applications. These findings suggest that despite security and compliance being important aspects of technology adoption and use, they did not get adequate attention from mHealth researchers. Finally, among the essential criteria, only 25 studies or 81 percent of the studies discussed the fidelity (item 16) when describing the outcomes of mHealth intervention.

Findings on the methodology criteria

In this section, we discuss our findings for the 29 reporting and methodological criteria as shown in Table 6. The analysis shows that all the studies covered item 1 and item 2, which is expected as both items (objectives and scientific background) are essential parts of scientific studies. However, item 3 is covered by 58 percent of the studies as not all studies are theoretically grounded. We feel that this coverage is still very good as more than half of the studies used a logical model(s) to show how mHealth intervention can influence users’ behavior for seeking healthcare support. The next three items in the checklist are very similar, and the findings are very similar as well. The item for study design is covered by 94 percent of the studies, the item for study outcome measures against the study objectives is covered by 97 percent of the studies, and the item for the data collection method is covered by 87 percent of the studies. The next two items focus on study participants and as much as 80 percent described the eligibility criteria of the study respondents and 77 percent of the studies mentioned how the study participants were recruited. Study biasness (item 9) was covered by only 19 percent of the studies where the authors discussed and addressed the influence of research on the results. Although less than 48 percent of the studies (15 studies) mentioned what sampling techniques were used, most of the studies (27 of 31 studies) mentioned about the study areas and settings. Also, it is found that only seven studies or 23 percent mentioned about testing comparison groups.

It was found that 74 percent of the studies (23 studies) mentioned the data sources (item 13; e.g. survey) and measurement (e.g. Likert-type scale). Moreover, only 19 studies (61%) mentioned how the participants of studies were screened and enrolled based on eligibility (item 14). Also, regarding item 15 (description of the study population), it is observed that only 19 studies (61%) describe the study population. In addition, most of the studies (90%) presented the study findings or outcomes (item 16) with enough details. Only 24 studies or 77 percent of the studies summarized evidence (item 17) found in the respective studies, and 94 percent of the studies addressed the research limitation (item 18). It is interesting to find only 26 percent of the studies discussed generalizability and external validity, which are very important aspects of empirical research. For describing the conclusions and interpretation, we found all the studies (100%) complied with this item and finished their study with adequate research synthesis and conclusion. About 55 percent of the studies mentioned funding sources (if funded) and 13 of the 31 studies (42%) mentioned ethical considerations which are a very important requirement regarding health-related research. In addition, 68 percent of the studies (21 studies) discussed competing interests, which is item 23 on the checklist.

While evaluating evidence reporting, we found that 19 of the 27 studies (70%) mentioned what types of statistical methods were applied and only 6 studies or 22 percent addressed the issue of missing data. Correspondingly, it is found that only 17 studies (63%) interpreted the attributions from the evidence found (item 26). Subsequently, the authors found that there were 4 qualitative and 4 mixed-method studies of 31 studies. In this regard, eight of the eight studies (100%) mentioned any analytical method(s) used in the qualitative part, and two of the eight studies (25%) applied data validation (e.g. triangulation) techniques. And finally, one of the eight studies (13%) addressed the issue of biasness that could arise from different factors, such as relationships between researchers’ roles and respondents, or the words and phrases used in the questionnaires. Table 6 provides a summary of evidence for the mERA checklist.

Discussion: summary of evidence reporting

After presenting the findings above which were extracted and synthesized from the 31 studies, Table 7 presents the mERA score and percentage of all mERA checklist criteria met by each of the studies. Despite the presence of robustness and diversity in the research designs and methods applied in these studies, a lack of comprehensiveness and completeness in evidence reporting are common. Our analyses have revealed that the average of the percentage of mERA criteria met by all the studies is 63 percent. It implies that still, a big percentage of the selected studies do not report on the evidence of mHealth interventions properly as recommended by the WHO. This finding of low-quality evidence reporting also conforms with the findings reported by Marcolino et al.⁵⁶ Similar finding by Wu et al.⁵⁷ also confirms that evidence is still very limitedly reported in the mHealth intervention studies, especially in the context of resource-limited settings.

Table 7.

The sum of points (mERA score) and percentage of each article.

Authors	Research type	Essential criteria met (%)	Methodology criteria met (%)	Sum of points	% per article
Ilozumba et al.³⁸	Mixed	75	90	38	84
Gunawardena et al.²⁸	Quantitative	75	92	36	80
Soong et al.⁴³	Qualitative	75	92	36	80
Praveen et al.³⁶	Quantitative	88	85	36	80
Benski et al.²⁷	Quantitative	75	88	35	78
Wang et al.⁵⁵	Mixed	75	76	34	76
Han et al.³⁷	Quantitative	56	88	33	73
Guo et al.³¹	Quantitative	63	88	33	73
García et al.³⁵	Quantitative	75	81	33	73
McNabb et al.⁴⁰	Quantitative	75	77	32	71
Bull et al.³⁰	Quantitative	69	81	32	71
Borsari et al.⁴⁸	Quantitative	75	77	32	71
Mishori et al.⁵¹	Qualitative	88	69	32	71
Sobrinho et al.⁵⁴	Quantitative	63	85	32	71
Musyimi et al.⁴⁶	Mixed	63	72	31	69
Lopez et al.²⁶	Quantitative	56	85	31	69
Bernardes-Souza et al.³⁹	Quantitative	63	81	31	69
Lwin et al.⁴⁵	Quantitative	56	81	30	67
Soto-Perez-De-Celis et al.³²	Quantitative	56	81	30	67
Alnasser et al.⁴⁹	Qualitative	75	62	28	62
Li et al.⁵²	Quantitative	69	62	27	60
Miah et al.⁵³	Mixed	75	52	27	60
Lestantri et al.⁴⁷	Quantitative	50	69	26	58
Alsharif and Philip⁴²	Qualitative	56	50	22	49
O’Connor et al.⁴¹	Quantitative	50	50	21	47
Rishikesh³³	Quantitative	56	46	21	47
García et al.³⁵	Quantitative	56	42	20	44
Francis-Lyon et al.³⁴	Quantitative	63	27	17	38
Islam et al.²⁹	Quantitative	50	31	16	36
Malik et al.⁵⁰	Quantitative	38	23	12	27
Afroze et al.⁴⁴	Quantitative	38	19	10	22
			Average		63

mERA: mobile health Evidence Reporting and Assessment.

From the above-discussed findings, it was found that the highest and lowest percentages of mERA criteria met by the studies were 84 and 22 percent, respectively. It was also found that the studies that have used RCTs have met the high percentages of methodology criteria. For example, the percentages of methodology criteria met by the studies by Gunawardena et al.,²⁸ Han et al.,³⁷ Bull et al.,³⁰ Guo et al.,³¹ and Praveen et al.³⁶ were 92, 88, 81, 88, and 85 percent, respectively. However, the percentages of essential criteria met by their studies were 75, 56, 69, 63, and 88 percent, respectively. Therefore, it can be inferred that the percentages for corresponding essential criteria and methodology criteria do not commensurate well. The higher percentage of reporting on methodology criteria than the essential criteria indicates high methodological quality in intervention impact assessment but a low level of comprehensiveness in the essential functionalities and requirements of mHealth interventions.

The higher percentage of reporting on methodology criteria can also imply that although the comprehensive methodologies were applied to report on the evidence of mHealth interventions, because of the lower percentages of meeting the evidence reporting from the perspective of essential criteria, the chance of widespread adoption of the mHealth intervention is still very low. For instance, Table 5 shows lower percentages for accessibility (32%), replicability (45%), scalability (26%), data security (privacy; 35%), compliance (29%), and integration and interoperability with the existing systems (39%) that lead to a low mHealth adoption rate. Such issues like low accessibility and scalability were also reported in the study by Bockting et al.⁵⁸ Besides, low evidence reporting on the criteria of cost consideration (52%) and adoption input (promotion and training required; 55%) indicates significant barriers in the low-setting context.

From the perspective of methodology criteria, low criteria meeting for missing data (22%) in quantitative studies and data validation (25%) in qualitative studies can indicate a high presence of biasness in the evidence reporting for mHealth interventions. Also, the low percentages of biasness (19%) and comparator reporting (23%) indicate poor quality of evidence reporting on mHealth programs. Furthermore, more than 50 percent of the studies did not use any logic model or theoretical framework that can lead to higher adoption of mHealth interventions vis-à-vis contextual requirements and demands.

Conclusion

The mERA checklist brings a valuable tool to the researchers for evaluating a wide range and quality of studies on mHealth interventions for low-setting developing world context where the scope of the positive impact of mHealth projects is very high. First released in 2016, until today, the mERA checklist is the most useful tool for comprehensive and transparent evidence reporting on mHealth interventions. The mERA checklist does not evaluate the quality of any mHealth study but evaluates the evidence reported in the mHealth studies. According to our knowledge, this is the first study focusing on the quality of evidence reporting in the studies of mHealth interventions that exist in the context of the developing world.

As mentioned in the discussion section above, the evidence reporting on mHealth research in low-setting developing countries significantly lacks essential criteria meeting, which indicates that mHealth interventions in the context of developing countries have still a long way to go to deliver the optimal benefits. Also, it is found that most of the mHealth studies selected did not follow any design science–based method and theory-based framework for developing mHealth interventions. Thus, robust and more inclusive development plans and studies are required to deliver highly evidence-based recommendations for the improvement and widespread adoption of mHealth interventions in developing countries.

The high presence of studies without peer review from the developing countries is concerning as these studies mostly fail to meet the criteria on evidence reporting severely. Thus, the low percentages of meeting the essential criteria in the developing countries are not surprising as the mERA checklist is still not very familiar to the mHealth researchers in developing countries. Hence, we acknowledge the need for more dissimilation and increased application of mERA checklist to eventually improve the usability and adaptability of mHealth technologies. The extensive application of mERA can also indirectly increase the quality of mHealth evidence reporting. Finally, the findings also reveal that the lack of interoperability and integration with the existing HIS is significantly low in the developing world. To increase and ensure interoperability, the synchronous effort by the industry stakeholders, health policymakers, funding institutes, and mHealth developers is a prerequisite.

A possible limitation of this study is that it is likely possible to include more peer-reviewed studies by altering the search strategy applied for developing country context. It is also important to acknowledge the possibility of unintentional exclusion of some eligible studies. Another limitation of this study is that this study initially searched and used only articles/studies in the context of developing countries; hence, there is no scope of any comparison between studies from developing countries to developed countries in this particular study. We believe that this is a major direction for future studies. Furthermore, the generalization of the findings related to some mERA criteria cannot be possible because of the wide range of socio-economic and contextual differences that exist in the developing world. Thus, studies based on more specific and homogeneous context are needed for obtaining consistent findings on the quality of evidence reporting in the mHealth intervention studies.

Supplemental Material

Appendix – Supplemental material for Mobile health interventions in developing countries: A systematic review

Supplemental material, Appendix for Mobile health interventions in developing countries: A systematic review by Md Rakibul Hoque, Mohammed Sajedur Rahman, Nymatul Jannat Nipa and Md Rashadul Hasan in Health Informatics Journal

Supplemental Material

Supplementary_File – Supplemental material for Mobile health interventions in developing countries: A systematic review

Supplemental material, Supplementary_File for Mobile health interventions in developing countries: A systematic review by Md Rakibul Hoque, Mohammed Sajedur Rahman, Nymatul Jannat Nipa and Md Rashadul Hasan in Health Informatics Journal

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Md Rakibul Hoque

Supplemental material

Supplemental material for this article is available online.

References

Statista. mHealth: statistics & facts. Hamburg: Statista, 2018.

Carroll

Moorhead

Bond

, et al. Who uses mobile phone health apps and does use matter? A secondary data analytics approach. J Med Internet Res 2017; 19: e125.

WHO. Use of appropriate digital technologies for public health, 2018, https://apps.who.int/gb/ebwha/pdf_files/WHA71/A71_20-en.pdf.

Latif

Rana

Qadir

, et al. Mobile health in the developing world: review of literature and lessons from a case study. IEEE Access 2017; 5: 11540–11556.

Labrique

Vasudevan

Weiss

Establishing standards to evaluate the impact of integrating digital health into health systems. Glob Health Sci Pract 2018; 6: 5–17.

Qiang

Yamamichi

Hausman

, et al. Mobile applications for the health sector. Washington, DC: World Bank, 2011.

White

Burns

Giglia

, et al. Designing evaluation plans for health promotion mHealth interventions: a case study of the Milk Man mobile app. Health Promot J Austr 2016; 27(3): 198–203.

Iribarren

Cato

Falzon

, et al. What is the economic evidence for mHealth? A systematic review of economic evaluations of mHealth solutions. PLoS ONE 2017; 12(2): e0170581.

O’Connor-Fleming

Parker

Higgins

, et al. A framework for evaluating health promotion programs. Heal Promot J Aust 2006; 17: 61–66.

10.

Agarwal

LeFevre

Lee

, et al. Guidelines for reporting of health interventions using mobile phones: mobile health (mHealth) evidence reporting and assessment (mERA) checklist. BMJ 2016; 352: i1174.

11.

BMJ. What is GRADE? 2020, https://bestpractice.bmj.com/info/toolkit/learn-ebm/what-is-grade/

12.

Turner

Shamseer

Altman

, et al. Consolidated standards of reporting trials (CONSORT) and the completeness of reporting of randomised controlled trials (RCTs) published in medical journals. Cochrane Database Syst Rev 2012; 11: MR000030.

13.

O’Grady

Witteman

Bender

, et al. Measuring the impact of a moving target: towards a dynamic framework for evaluating collaborative adaptive interactive technologies. J Med Internet Res 2009; 11: e20.

14.

Stoyanov

Hides

Kavanagh

, et al. Mobile application rating scale: a new tool for assessing the quality of health mobile applications. JMIR Mhealth Uhealth 2014; 3: e27.

15.

Mohr

Schueller

Riley

, et al. Trials of intervention principles: evaluation methods for evolving behavioral intervention technologies. J Med Internet Res 2015; 17e166.

16.

Hyder

Wosu

Gibson

, et al. Noncommunicable disease risk factors and mobile phones: a proposed research agenda. J Med Internet Res 2017; 19: e133.

17.

Bashi

Fatehi

Fallah

, et al. Self-management education through mHealth: review of strategies and structures. JMIR Mhealth Uhealth 2018; 6: e10771.

18.

Torous

Firth

Mueller

, et al. Methodology and reporting of mobile heath and smartphone application studies for schizophrenia. Harv Rev Psychiatry 2017; 25(3): 146–154.

19.

Panic

Leoncini

de Belvis

, et al. Evaluation of the endorsement of the preferred reporting items for systematic reviews and meta-analysis (PRISMA) statement on the quality of published systematic review and meta-analyses. PLoS ONE 2013; 8(12): e83138.

20.

L’Engle

Mangone

Parcesepe

, et al. Mobile phone interventions for adolescent sexual and reproductive health: a systematic review. Pediatrics 2016; 138(3): e20160884.

21.

Richardson

Wilson

Nishikawa

, et al. The well-built clinical question: a key to evidence-based decisions. ACP J Club 1995; 123(3): A12–A13.

22.

Burla

Knierim

Barth

, et al. From text to codings: intercoder reliability assessment in qualitative content analysis. Nurs Res 2008; 57(2): 113–117.

23.

McHugh

ML.

Interrater reliability: the Kappa statistic. Biochem Medica 2012; 22: 276–282.

24.

Matthew-Maich

Harris

Ploeg

, et al. Designing, implementing, and evaluating mobile health technologies for managing chronic conditions in older adults: a scoping review. JMIR Mhealth Uhealth 2016; 4: e29.

25.

Ginsburg

Delarosa

Brunette

, et al. mPneumonia: development of an innovative mHealth application for diagnosing and treating childhood pneumonia and other childhood illnesses in low-resource settings. PLoS ONE 2015; 10: e0139625.

26.

Lopez

Ramirez

Valenzuela

, et al. Sexual and reproductive health for young adults in Colombia: teleconsultation using mobile devices. JMIR Mhealth Uhealth 2014; 2: e38.

27.

Benski

Stancanelli

Scaringella

, et al. Usability and feasibility of a mobile health system to provide comprehensive antenatal care in low-income countries: PANDA mHealth pilot study in Madagascar. J Telemed Telecare 2017; 23(5): 536–543.

28.

Gunawardena

Jackson

Robinett

, et al. The influence of the smart glucose manager mobile application on diabetes management. J Diabetes Sci Technol 2019; 13: 75–81.

29.

Islam

Alvi

Al Mamun

. DiaHealth: a smart app for complete diabetes lifestyle management. In: Proceedings of the 2016 international conference on medical engineering, health informatics and technology (MediTec), Dhaka, Bangladesh, 17–18 December 2016. New York: IEEE.

30.

Bull

Thomas

DSK

Nyanza

, et al. Tanzania health information technology (T-HIT) system: Pilot test of a tablet-based system to improve prevention of mother-to-child transmission of HIV. JMIR Mhealth Uhealth 2018; 6: e16.

31.

Guo

Hong

Qiao

, et al. Run4Love, a mHealth (WeChat-based) intervention to improve mental health of people living with HIV: a randomized controlled trial protocol. BMC Public Health 2018; 18: 793.

32.

Soto-Perez-De-Celis

Kim

Rojo-Castillo

, et al. A pilot study of an accelerometer-equipped smartphone to monitor older adults with cancer receiving chemotherapy in Mexico. J Geriatr Oncol 2018; 9(2): 145–151.

33.

Rishikesh

Leveraging technology by digitalization using “I report app” for safety at construction sites. In: Proceedings of the 26th annual conference of the international group for lean construction, Chennai, India, 18–20 July 2018, pp. 943–951, https://iglcstorage.blob.core.windows.net/papers/attachment-3c1b9667-ff79-4673-b27c-b42d9b49afcf.pdf

34.

Francis-Lyon

Attiga

Manjunath

, et al. Tablet app for child cognitive assessment in low and middle income countries. In: Proceedings of the global humanitarian technology conference (GHTC), San Jose, CA, 19–22 October 2017, pp. 1–5. New York: IEEE.

35.

García

Tomás

Parra

, et al. An m-health application for cerebral stroke detection and monitoring using cloud services. Int J Inf Manage 2019; 45: 319–327.

36.

Praveen

Patel

Raghu

, et al. SMARTHealth India: development and field evaluation of a mobile clinical decision support system for cardiovascular diseases in rural India. JMIR Mhealth Uhealth 2014; 2: e54.

37.

Han

Zhang

Hser

Y-I

, et al. Feasibility of a mobile phone app to support recovery from addiction in China: secondary analysis of a pilot study. JMIR Mhealth Uhealth 2018; 6: e46.

38.

Ilozumba

Dieleman

Kraamwinkel

, et al. “I am not telling: the mobile is telling”: factors influencing the outcomes of a community health worker mHealth intervention in India. PLoS ONE 2018; 13(3): e0194927.

39.

Bernardes-Souza

De Assis Pires

FPA

Madeira

, et al. Facial-aging mobile apps for smoking prevention in secondary schools in Brazil: appearance-focused interventional study. JMIR Public Heal Surveill 2018; 4: e10234.

40.

McNabb

Chukwu

Ojo

, et al. Assessment of the quality of antenatal care services provided by health workers using a mobile phone decision support application in northern Nigeria: a pre/post-intervention study. PLoS ONE 2015; 10(5): e0123940.

41.

O’Connor

O’Sullivan

Hardy

, et al. A mobile health technology intervention for addressing the critical public health issue of child mortality. Electr J Inform Syst Dev Countr 2018; 84: e12011.

42.

Alsharif

Philip

. A framework for smoking cessation in the Kingdom of Saudi Arabia using smart mobile phone technologies (Smoke Mind). In: Proceedings of the 2015 second international conference on computer science, computer engineering, and social media (CSCESM), Lodz, 21–23 September 2015, pp. 96–102. New York: IEEE.

43.

Soong

Chen

Borzekowski

DLG

. Using ecological momentary assessment to study tobacco behavior in urban India: there’s an App for that. JMIR Res Protoc 2015; 4: e76.

44.

Afroze

Shezan

Sharmin

. Poster: HeartFit: an intuitive smartphone application for well-being of hypertensive patients. In: Proceedings of the 15th annual international conference on mobile systems, applications, and services, New York, 23 June 2017. New York: ACM.

45.

Lwin

Jayasundar

Sheldenkar

, et al. Lessons from the implementation of Mo-Buzz, a mobile pandemic surveillance system for dengue. JMIR Public Heal. Surveill 2017; 3: e65.

46.

Musyimi

Mutiso

Haji

, et al. Mobile based mhGAP-IG depression screening in Kenya. Community Ment Health J 2018; 54(1): 84–91.

47.

Lestantri

Sabiq

Suherlan

Developing and pilot testing mHealth care application for pregnant and toddlers based on user experience. J Phys: Conf Ser 2018; 978: 12067.

48.

Borsari

Stancanelli

Guarenti

, et al. An innovative mobile health system to improve and standardize antenatal care among underserved communities: a feasibility study in an Italian Hosting Center for asylum seekers. J Immigr Minor Health 2018; 20(5): 1128–1136.

49.

Alnasser

Sathiaseelan

Al-Khalifa

, et al. Development of “Twazon”: an Arabic app for weight loss. JMIR Res Protoc 2016; 5: e76.

50.

Malik

Bibi

Khan

, et al. Mr. Doc: a doctor appointment application system. arXiv. 2017, https://arxiv.org/ftp/arxiv/papers/1701/1701.08786.pdf

51.

Mishori

Anastario

Naimer

, et al. mJustice: preliminary development of a mobile app for medical-forensic documentation of sexual violence in low-resource environments and conflict zones. Glob Heal Sci Pract 2017; 5: 138–151.

52.

Fang

Gao

, et al. Decision support system for the response to infectious disease emergencies based on WebGIS and mobile services in China. PLoS ONE 2013; 8(1): e54842.

53.

Miah

Hasan

, et al. Healthcare support for underserved communities using a mobile social media platform. Inf Syst 2017; 66: 1–12.

54.

Sobrinho

da Silva

Perkusich

, et al. Design and evaluation of a mobile application to assist the self-monitoring of the chronic kidney disease in developing countries. BMC Med Inform Decis Mak 2018; 18: 7.

55.

Wang

C-J

Chaovalit

Pongnumkul

A breastfeed-promoting mobile app intervention: usability and usefulness study. JMIR Mhealth Uhealth 2018; 6: e27.

56.

Marcolino

Oliveira

JAQ

D’Agostino

, et al. The impact of mHealth interventions: systematic review of systematic reviews. JMIR Mhealth Uhealth 2018; 6: e23.

57.

Gong

, et al. A smart and multifaceted mobile health system for delivering evidence-based secondary prevention of stroke in rural China: design, development, and feasibility study. JMIR Mhealth Uhealth 2019; 7: e13503.

58.

Bockting

CLH

Williams

Carswell

, et al. The potential of low-intensity and online interventions for depression in low- and middle-income countries. Glob Ment Health 2016; 3: e25.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.15 MB

0.12 MB