Maturity assessment and maturity models in health care: A multivocal literature review

Abstract

Background

The maturity of practices and infrastructure in the health care domain directly impacts the quality and efficiency of health care services. Therefore, various health care administrations (e.g. from hospital management to the nationwide health authority) need to assess and improve their operational maturity.

Objective

This study aimed to review and classify studies that propose/use maturity assessment or maturity models (MMs) as a vehicle to achieve operational excellence in the health care domain.

Method

To achieve this objective, we performed a multivocal literature review (MLR) – a form of systematic review that includes data from the grey literature (e.g. white papers and online documents) in addition to formal, peer-reviewed literature.

Results

Based on 101 sources, 80 from peer-reviewed literature and 21 from the grey literature, we identified 68 different MMs on, for example, telemedicine, care pathways and digital imaging. We reviewed them with respect to various aspects, including types of research and contribution, list of MMs proposed/used with their subject areas, elements of maturity/capability and application scope or scale. In the synthesis of empirical benefits of using MMs, two were found to be significant: (a) identifying issues and providing guidance for improvement in health care contexts, and (b) improving efficiency, effectiveness, performance and productivity.

Conclusion

This MLR provides an overview of the landscape and serves as an index to the vast body of knowledge in this area. Our review creates an opportunity to cope with the challenges in obtaining an overview of the state-of-the-art and practice, choosing the most suitable models or developing new models with further specialties.

Keywords

Maturity models maturity assessment health care multivocal literature review

Introduction

The maturity of practices, operations and infrastructure in the health care domain is of high importance.¹ However, these qualities often vary among health care provider units (e.g. hospitals) and in some cases are far from perfect. Establishing process thinking and achieving quality management is vital to assure service maturity and to improve service maturity continually in such a complex, dynamic and multidisciplinary domain.^2,3

The notion of ‘maturity’ was first proposed by Phillip Crosby⁴ and is defined as ‘the state of being complete, perfect, or ready’.⁵ In a more general view, a maturity model (MM) is a conceptual framework that consists of a sequence of discrete maturity levels for a class of processes in one or more business domains, and represents an anticipated, desired or typical evolutionary path for these processes.^6,7 MMs (also referred to as ‘capability frameworks’) have been widely used in many domains (such as industrial engineering, software engineering and information technology (IT)) for the purpose of process assessment and improvement.^8–10

MMs have also been proposed in the health care domain with the purpose of assessing and improving the maturity of health care practices, operations and infrastructure.^11,12 Some of these MMs focus on the specific sub-areas, such as telemedicine, care pathways, digital imaging, picture archiving and communication systems and facilities management. Given the large body of knowledge in this area, there is a need to review the scope and characteristics of available models together with their use and usefulness. health care practitioners, decision makers and researchers can then eliminate ‘reinventing the wheel’ to obtain the state-of-the-art and practice, choose the ‘right’ models that are most suitable for their needs or develop new models with further specialties. Therefore, this study is aimed at providing insight into and creating awareness on the MMs proposed as a vehicle to achieve operational excellence in the health care domain.

To achieve the above-mentioned objective, we performed a comprehensive literature review on MMs proposed and/or used in the health care domain. We used a multivocal literature review (MLR)¹³ as the research method, which is a form of a systematic literature review (SLR) and includes grey literature (e.g. blog posts, white papers, presentations) in addition to the published (formal) academic literature. While a SLR is a common method used to conduct a literature review on published (peer-reviewed) literature on a given topic, the significance of including the grey literature in addition to the academic literature has been emphasised in the health care community.^14–16

We used systematic guidelines^13,17 to search for the sources and to categorise and synthesise them with respect to a number of review questions (RQs). We searched the academic literature using Google Scholar, PubMed and ScienceDirect, and we searched the grey literature using the regular Google search engine. In the remainder of this paper, we use the term ‘source’ to refer to a formally published paper/article or a resource from the grey literature. Through the RQs, we addressed the following properties of the sources: (a) types of research and contribution, (b) newly proposed versus existing MM used in the studies, (c) the aspects of maturity/capability addressed by the MMs (technology, business process, people, etc.), and (d) the application scope or scale of the MMs (e.g. single departments of a single hospital, multiple hospitals in a city, region or province). More specifically, the main contributions of this article can be summarised as providing:

An overview of the state-of-the-art and practice of 101 sources that propose or use MMs in the health care domain;

A list of 68 MMs proposed or used in this domain, with details of selected examples;

A classification scheme of the sources with respect to research type, contribution type, subject area, contained aspect, applied scope, scale of empirical evidence and reported benefits of the MMs; and

A classification and synthesis of the sources with respect to the attributes mentioned above.

The remainder of this paper is structured as follows. Related work is presented in the following section. Next, we describe the research method and the review planning. The results of the literature review are then presented. Subsequently, we discuss our observations and suggestions, and potential threats to the validity of this MLR. Finally, we draw conclusions and make suggestions for future work.

Related work

Since this paper is an MLR on MMs in health care, the related works are the existing secondary studies on MMs in health care. A secondary study is a study of regular (primary) studies.

Using a literature search, we were able to identify 12 secondary studies on MMs in health care, as listed in Table 1. There are usually four types of secondary studies: systematic mapping (SM) studies, SLRs, MLRs and regular reviews (surveys). For each paper in Table 1, we also provide the title of the paper to show its scope, its type (SLR, SM, MLR or regular review) and the number of papers reviewed by the study.

Table 1.

Secondary studies on MMs in health care (ordered by year of publication).

Reference	Year of publication	Title	Type of study	# Primary studies (papers) reviewed
¹⁸	2009	A PACS maturity model: a systematic meta-analytic review on maturation and evolvability of PACS in the hospital enterprise	SLR	34
¹⁹	2011	Evolution of information systems and technologies (IST) maturity in healthcare• As part this paper, the authors provided a survey of MMs focusing on the IST management in health care	A brief regular review (survey) as part of the paper	5
²⁰	2013	Composite quality of care scores, electronic health record maturity models, and their associations; preliminary literature review results	SLR	53
²¹	2013	Towards a business intelligence (BI) maturity model for healthcare• Presented a review of existing BI MMs to determine their adequacy for use in health care	Regular review	15
²²	2015	Quality system maturity model for medical devices-medical device innovation consortium• Provided an overview of various MMs in the medical device industry and how they have been implemented and leveraged	Regular review	22
²³	2016	A patient-centred framework for evaluating digital maturity of health services: a systematic review	SLR	28
²⁴	2016	Information systems and technologies maturity models for healthcare: a systematic literature review	SLR	14
²⁵	2016	Maturity models for hospital information systems management: are they mature?	Regular review	3
¹¹	2016	Maturity models of healthcare information systems and technologies: a literature review	SLR	14
¹²	2016	The use of maturity/capability frameworks for healthcare process assessment and improvement	SM/SLR	29
²⁶	2017	A review and comparison of maturity/capability frameworks for healthcare process assessment and improvement	SLR	6
²⁷	2017	A maturity model for hospital information systems	A brief SLR as part of the study	14
This study	2020	Maturity assessment and maturity models in healthcare: a multi-vocal literature review	MLR	101

MM: maturity model; PACS: picture archiving and communication systems; SLR: systematic literature review; SM: systematic mapping; MLR: multivocal literature review.

Our current MLR differs from the existing works in that it is the first comprehensive review on maturity assessment and MMs in health care, which covers both the formal and the grey literature and clusters subject areas and aspects studied so far. In terms of number of papers reviewed by a secondary study, it is seen from Table 1 that our MLR reviews also the highest number of papers (101 sources) compared to the existing secondary studies and derives a broad classification scheme.

Research method

We performed our study based on the guidelines for conducting SLR/MLR studies provided in the health care domain^28,29 and also in other domains.^13,17

The area of health care MMs is cross-disciplinary in nature, as it includes various stakeholders from health services and hospital management, to information systems (IS) and IT, each on the side of academia or practice. Furthermore, we found that some models in this area have been presented in the grey literature and have not been formally published. Conducting a typical SLR without including the grey literature would not have included many important and useful MMs that have emerged from practice and which are available in the grey literature. Therefore, we followed a natural approach by including the grey literature in the review and conducted an MLR. This decision is also in alignment with the literature considering the use of MLRs as a means to close ‘the gap between academic research and professional practice’.¹³

Below, we discuss various aspects of our research method.

Goal and review questions

The goal of this study was to review and classify studies that propose/use maturity assessment or MMs in the health care domain. Based on this goal, we raised the following review questions (RQs):

• RQ 1 – Contribution and research types:

○ RQ 1.1 – Contribution types: How many sources have presented new MMs, methods/techniques, tools, metrics or processes in this area?

○ RQ 1.2 – Research types: What types of research methods were used by authors of the sources? Some sources only propose solutions without extensive validations, while some other sources present in-depth evaluation of their approach (e.g. rigorous empirical studies).

• RQ 2 – Properties of the proposed MMs:

○ RQ 2.1 – MMs in this area: What models have been proposed in this area? What are their subject areas?

○ RQ 2.2 – Aspects of maturity/capability framework: What aspects (e.g. business process, organisational, technology and people) does each of the proposed MM address?

○ RQ 2.3 – Applicability scope/scale: What is the applicability scope/scale of the MMs? The models could be applied within the scope of, for example, single/multiple department(s) or single/multiple hospital(s). In this RQ, we consider the expected/possible applicability of the models, and not necessarily the scale of the actual empirical study presented in a given source.

• RQ 3 – Properties of the empirical studies:

○ RQ 3.1 – Scale of empirical study: On what scales have the empirical studies been conducted, that is, what is the number of hospitals that the proposed MM has been applied in? Different from RQ 2.3, here we consider the ‘actual’ reported empirical results presented in a source.

○ RQ 3.2 – Reported benefits of MM applications: What benefits (quantitative or qualitative) have been reported as a result of applying the models? The application of such a model is sometimes claimed to be costly. Thus, enough benefits should be provided to ensure justifying the associated costs.

Multivocal review process

Figure 1 outlines the process that lies at the basis of this study, which consists of three phases: (a) search and selection of sources, (b) development of the classification scheme and data extraction and (c) systematic classification, synthesis and review. The process starts with the selection of sources in the academic and grey literature. Next, a classification scheme (map) is developed following a structured approach. The map is then used to conduct systematic classification, which is followed by the synthesis and reporting of the results.

Figure 1.

An overview of the multivocal literature review process.

Search and selection of sources

To search for formally published (peer-reviewed) papers, we used Google Scholar, PubMed and ScienceDirect search engines. There have been numerous studies^30–34 comparing the effectiveness, strengths and weaknesses of these academic search engines. We carefully reviewed these studies to be as objective and systematic as possible in our choice of search engines.

We also conducted a number of experimental search activities for papers in this area. We observed that the topic of MMs in health care is a cross-disciplinary subject and that many papers in this area (e.g. Sources 10, 17 and 24 in the pool of studies³⁵) are published in venues that address the domains outside of ‘core’ health care (e.g. management, IT and computer science), and thus are not covered (i.e. could not be found) in PubMed. Since Google Scholar has a much larger index (search space) than PubMed, first we used Google Scholar and then PubMed to populate the candidate pool of sources.

Through an iterative process, we developed the search string for this MLR study as: (maturity model OR maturity assessment) AND (health care OR healthcare OR health). We should note that the search process and source selection of this study was conducted in March 2018. Thus, sources published by that time had a chance to be included in our pool.

We performed the searches in our list of search engines using the above keywords. Using Google Scholar, we found 76 candidate formally published papers. Using PubMed, we found four additional candidate formally published papers which were not found through Google Scholar. They were Sources 13, 25, 80 and 90. Full details can be found in the study’s transparent online spreadsheet (https://goo.gl/Z4jxBd). Our search of ScienceDirect did not yield any additional papers.

To search for grey-literature sources, we used the Google search engine, which provides comprehensive access to the grey literature.³¹ This popular web search engine (also the most widely used one according to eBizMBA, Inc.³⁶) has been used in many MLR studies in the past (e.g. in health³⁷ and management sciences³⁸).

As a result of the first search phase, we obtained an initial pool of 148 sources on which we applied the criteria for inclusion/exclusion and quality assessment.

Criteria for inclusion/exclusion and quality assessment of sources

We carefully defined the inclusion and exclusion criteria to ensure all relevant sources were included but not those which were out of scope. Our inclusion criteria were sources that proposed or used MMs in the health care domain; sources which were directly related to maturity assessment in the health care domain; and sources that were in English and with their full text accessible via our institutions’ subscriptions or freely available on the Internet. (Please note that one author of this article has a medical affiliation, giving us good enough to health care publication venues.)

In addition to the criteria above, we also defined and applied criteria for checking the quality of the sources. Quality assessment is specifically focused on determining the sources that report sufficient information to compare other studies in terms of answering defined review questions.³⁹ Therefore, we developed and used the following quality assessment criteria to ensure that sufficient information was reported in the sources:

Is there a clear statement that the study proposes/uses an MM for health care?

Can it be inferred from the study that the MM is newly proposed, or it is only applied as already available?

Can the organisational aspects that the MM involves (e.g. process, technology and people) be extracted from the study?

Can the scope/scale in which the MM could be applied be extracted from the study?

All sources were examined in detail by the first three researchers in the voting stage with respect to the inclusion/exclusion and quality assessment criteria given above, and the sources which did not meet these criteria were excluded. Each of the first three researchers voted independently on to include or exclude each source. In the case of disagreements, discussions were conducted, and the fifth author who is a medical practitioner (pathologist) was consulted to reach a consensus.

As previously stated, the search process and source selection for this study was conducted in March 2018. Only a single paper from 2018 was found,⁴⁰ which was excluded in order not to provide a partial view for 2018. As a result, the time window for the pool sources was set to the yearly range of 1996–2017. After applying the inclusion/exclusion and quality criteria, 47 sources were excluded (see Figure 1), and our review pool was finalised with 101 sources.

Development of the classification scheme and data extraction

To conduct a structured and systematic classification of the sources, a classification scheme (systematic map) is needed.^28,29 We developed our classification scheme through an iterative process of analysing the sources in the pool, identifying relevant list of attributes and refining them to derive the final map. Our goal was to categorise the sources to build a complete picture of the research area. Driven by our review questions, we identified the key attributes of the sources and refined them into a classification scheme using an iterative approach. Table 2 shows the classification scheme that we developed after applying the process described above.

Table 2.

Classification scheme (systematic map) developed and used in this study.

RQ		Attribute/aspect	Categories	Multiple (M)/single (S) value
RQ 1	1.1	Contribution type	New (maturity) model, method/technique, tool, metric, process, empirical study only, other	M
RQ 1	1.2	Research type	Validation research (weak empirical study), evaluation research (strong empirical study), solution proposal, philosophical paper, experience paper, opinion paper, other	S
RQ 2	2.1	MMs proposed with subject focuses	Name of the MM proposed in the source, with its subject area(s)	S (name)/M (subject focus)
	2.2	Aspects covered by the MM	Business process, technology, people, other	M
	2.3	Scope of MM application	Single departments of a hospital/health care institution; multiple departments of a hospital; single health care institution; multiple health care institutions; city, region or province; government (nation-wide); international; other	M
RQ 3	3.1	Scale of the empirical study	Number of hospitals/health care institutions, any other size/scope related metric (e.g. number of experts, number of survey or interview participants): numerical values	M
RQ 3	3.2	Reported benefits of MM application	Quantitative benefits, qualitative benefits: open text	M

Taking our RQs and the classification scheme as the basis, we thoroughly reviewed the sources in our pool. In tagging each study based on the classification scheme, we incorporated as many explicit ‘traceability’ links between our mapping and the primary sources as possible. That is, we added comments to the cells of our mapping matrix by verbatim copy/paste of text from the source acting as the ‘traceability’ link. Such comments facilitated peer reviewing (that we conducted among the authors) and are also useful for other researchers reviewing the resulting paper pool and the Supplemental Material.

As in the initial voting stage, during the detailed analysis phase, three researchers extracted and analysed data from the portion of the sources and then peer reviewed each other’s extracted data. In the case of disagreements among the researchers, issues were resolved by holding discussions with all authors to ensure quality and validity of the data and their relevance in the health care domain.

Results

Our final pool included a total of 101 sources: 80 from the formal literature and 21 from the grey literature. For a complete list of these sources, please refer to Tarhan et al.³⁵ Below, we report a summary of the trends in the final pool of sources, followed by the results with respect to our RQs.

Figure 2 shows the cumulative number of sources per year by literature type (formally published vs. grey literature). We can see that the sources in both literature categories have been increasing steadily. While many scientists are publishing their work in this area, many practitioners/professionals are using online platforms such as blogs to post their work in this area as grey literature. We can see from Figure 2 that the topic has received active attention in recent years.

Figure 2.

Number of sources per year.

Out of the 80 sources from the formal literature, 60 were journal papers, 13 were conference papers, four were theses, two were book chapters and one was a workshop paper. Out of the 21 sources from the grey literature, 13 were white papers, four were presentations, three were web pages and one was a blog post. Journal papers formed the majority (69%) of the pool, denoting a considerable level of maturity in the research field.

RQ 1: Contribution and research types

RQ 1.1: Contribution types

Figure 3 shows the annual cumulative breakdown of sources by contribution types. Under the figure, we also provide the references of the sources for each type. Please note that a source may contribute more than one contribution type. The top three contribution facets are new MMs, empirical (case) studies and methods/techniques, which have appeared in 68, 26 and 4 sources, respectively.

Figure 3.

Sources by contribution type.

We can see that across different years, different contribution facets have been studied. The general observable trend is that new MMs are continuously being offered in this area. In addition, the trend with sources that have major contributions as empirical (case) studies is a promising sign of practice by the community.

In Figure 4, we present a bird’s-eye view of MMs by showing a word cloud of the MM names (these are discussed further below in a review of these MMs). As we can see, there have been many MMs focusing on the topics of IT, electronic systems and telemedicine. These topics highlight the importance of increasing digitalisation in this domain.

Figure 4.

A word cloud of the maturity model (MM) names presented in the literature.

Four sources contributed methods and frameworks in this area. Source 2 presented a framework for a domain-specific business intelligence (BI) MM and applied it in health care. Source 22 developed a maturity assessment tool and technique in the subject of clinical governance. Source 59 proposed a MM for identity management and a procedure for applying the MM, and reported from practice on its successful use in two large Swiss hospitals. Source 74 proposed a method to identify a selected number of items from a previously validated quality improvement (QI) maturity tool as the basis for calculating organisational and system-level QI maturity scores in time.

As mentioned above, the contributions of 26 sources were empirical (case) results only. For example, the research in Source 12 sought further empirical evidence of IS and IT maturity in the context of the UK’s National Health Service (NHS) acute trust hospitals. A survey was used to collect data from more than 70 top and middle managers, representing four trust hospitals in the UK. Statistical analysis of these data provided evidence that 6/23 maturity characteristics identified in existing models can currently be used to differentiate the maturity of NHS acute trust hospitals. The aims of the study in Source 17 were to investigate the challenges of assessing process maturity in health-care institutions using a generic business process MM (i.e. OMG’s Business Process Maturity Model), and to explore the opportunities for future work that would facilitate a process maturity assessment and improvement in the health care domain. Source 20 reported an empirical audit of a health agency for preparedness with respect to the implementation of e-health management using the COBIT (Control Objectives for Information and Related Technology) framework.⁴¹ Further insights into the pool of empirical studies in response to RQ 3 are discussed later in this article.

Two sources offered tools as one of their main contributions. In Source 54, the proposed health care quality maturity assessment model was incorporated in an interactive Microsoft Excel worksheet that visually displayed the quality maturity-level risk meters, which we considered under the ‘tool’ category. Another tool-presenting work was Source 64, which presented a software tool assisting the users in applying the General Practice Information Maturity Model.

Two sources contributed processes. Source 29 developed a capability maturity model (CMM)-based security risk assessment ‘process’ for patient-centred health care systems, and applied it in the context of personal health records. The CMM¹⁰ is a framework developed based on the data collected from organisations that contracted with the US Department of Defense. The term ‘maturity’ in the CMM relates to the degree of formality and optimisation of processes, from ad hoc practices to formally defined steps to managed result metrics to active optimisation of the processes. The CMM’s aim is to improve software-development processes, but it has also been applied in other domains. Source 39 proposes a process for maturity assessment based on the MM introduced in that paper (i.e. the health care IT maturity model).

Five sources focused on ‘other’ types of contributions. Source 10 proposed a software process improvement model for the UK’s health care industry by using organisational maturity (based on ISO/IEC 15504⁹) as the guideline. Source 16 discussed authors’ experiences in applying the CMMI for Services model to health care services, specifically for in-hospital pharmaceutical and respiratory services. Similarly, Source 28 shared the encountered experience and challenges that the authors faced during the design and implementation of three MMs for distinct improvement purposes in hospitals. Source 29 mapped the components of the CMM into the regulations of the American Health Insurance Portability and Accountability Act. Source 34 discussed opinions on using digital maturity assessment in order to understand the level of readiness for using technology in the UK’s NHS.

RQ 1.2: Research types

Based on the classification scheme described earlier, we classified the sources into seven types of research methods. Figure 5 shows the classification of the sources according to the type of research method they followed. We adapted these classifications from the SLR/MLR guidelines.^13,17,28,29 Papers ranked under ‘solution proposal’ and ‘opinion papers’ used the least rigorous research methods, while papers conducting weak and strong empirical research were the most rigorous in terms research methods in general. Papers falling under the ‘experience’ category reported a form of experience in the topic.

Figure 5.

Sources by type of research method.

Recall that for the research facet type, each study could be classified into only one category. In showing the classification in Figure 5, we also divided the sources by the two types of literature: academic literature and grey literature.

A large ratio of sources fell under ‘solution proposals’ (29 sources) and weak and strong empirical studies (31 and 20 sources, respectively), which is a positive sign of empiricism in this community. Twelve sources reported ‘opinions’. There were nine experience reports based on the ‘experience’ of their authors. No source was classified as a ‘philosophical paper’ or ‘other’.

Division of the sources by the two types of literature shows that most grey-literature sources were of ‘solution proposal’ or ‘opinion paper’ types and less of ‘weak empirical’ studies. No grey-literature sources used ‘strong empirical’ research methods. These observations are as one would expect, since grey-literature sources tend to utilise less rigorous research methods.

RQ 2: Various properties of MMs

RQ 2.1: Proposed MMs with their subject areas

We compiled a list of all the proposed MMs and collated them in Table 3. In total, 68 sources proposed 68 new models in this area. As can be seen from Table 3, there is vast diversity in the topic of MMs in this area. Thus, we also extracted the subject area of these models and provide an index of these subjects, as addressed by the sources, in alphabetical order in an online resource.³⁵ The most frequently occurring subjects by the models were: health services and IT (n = 6 each); public health, QI and safety culture (n = 4 each); BI, data analytics, e-health, electronic medical records (EMR), EMR adoption, hospital information system, integrated care, patient safety, security and telemedicine (n = 3 each).

Table 3.

List of proposed MMs by sources.

#	Name of MM	Source # in online pool
1	State-wide master person index	1
2	BI maturity model	2
3	Business process model	3
4	Care pathway MM	4
5	MM for hospital information systems	5
6	MM for telemedicine implementation	6
7	Multi-step MM for electronic and computable diagnostic clinical prediction rules (eCPRs)	7
8	Picture archive and communication system MM	8
9	Digital maturity of health services	9
10	Act on oncology model	11
11	A process MM for governance	13
12	Inter-professional practice capability framework	14
13	Open government MM	15
14	Case management MM	21
15	Cloud MM	23
16	Business intelligence maturity in health care	24
17	Analytics assessment MM	27
18	Health profession regulation strengthening framework	30
19	Telemedicine MM	31
20	QI maturity index	32
21	Digital imaging adoption model	33
22	E-healthcare MM	35
23	Electronic health care MM	37
24	Healthier cities MM	38
25	health care IT MM	39
26	Inherited cardiac conditions (ICC) MM	41
27	A PACS MM for strategic situational planning	43
28	Game MM	44
29	Data-management maturity	45
30	Hospitals cooperation MM	46
31	health care network MM	47
32	health care analytics MM	50
33	health care breach security MM	51
34	health care information security adoption model	52
35	health care quality MM	54
36	High-reliability health care MM	55
37	Hospital medicine NN	56
38	Identity management MM	59
39	Informatics capability MM	60
40	IT capacities MM	62
41	Governance, risk and compliance MM	63
42	A MM for Hospital Information Systems management	65
43	Field hospital MM	66
44	MM for integrated care	67
45	EMR adoption framework	68
46	B3-MM	69
47	Use of EMR MM	70
48	QI maturity tool	71
49	Health in all polices MM	72
50	Medicaid information-technology architecture MM	73
51	Patient safety culture improvement tool	76
52	Manchester patient safety framework	77
53	Process management MM	78
54	EMR-MM	80
55	health care usability MM	81
56	Organisational public–private partnership MM	82
57	Safety culture assessment in community	85
58	Social media MM	87
59	Spatial maturity in health care	88
60	Software process capability/MMs for the asynchronous store-and-forward telemedicine systems	89
61	Computerised medical records MM	90
62	health care paperless MM	91
63	health care security MM	92
64	Public health IT maturity	96
65	IT in nursing homes	98
66	Telemedicine service MM	99
67	Hospital cooperation MM	100
68	Networkability MM	101

QI: quality improvement; EMR: electronic medical record.

RQ 2.2: Aspects covered by the MMs

We classified the aspects covered by the proposed MMs. The possible aspects (categories of classification) were process, technology, people and other aspects. Figure 6 shows the distribution of the sources with respect to these aspects. Note that the MM presented in a given source may cover more than one single aspect (e.g. Sources 1 and 2). Thus, the sum of the four categories in the figure is greater than number of sources (N = 101).

Figure 6.

Mapping of sources with respect to the aspects of the proposed MMs.

The ‘business process’ aspect is covered in 65 sources. For example, Source 15 proposed an open government MM for participation/collaboration processes between public and government agencies. Source 30 developed a framework to measure maturity of regulations in health care, and Source 40 presented an MM for enterprise architecture and applied it in the context of the Veterans Health Administration. The ‘technology’ aspect of maturity is also covered in 65 sources. For example, Source 5 defined an MM to assess hospital IS. Source 23 proposed a cloud MM for the health care industry, and Source 33 described a digital imaging MM. The ‘people’ aspect of maturity is addressed in 46 sources. For example, Source 19 defined an assessment tool for patient safety culture in Manchester. Source 42 proposed a people capacity MM in medical record wards of Iranian hospitals. Source 100 defined a cooperation MM for hospitals. Under the ‘other’ category, 30 sources covered various aspects, including work culture, strategy, governance, leadership, interoperability, and data. Readers are encouraged to use Figure 6 to trace the references to the sources in each category and to access details of them via the study’s online spreadsheet (https://goo.gl/Z4jxBd).

Table 4.

Three example MMs focusing on different aspects of health care.

MM name	Source #	Staged or continuous?	Levels (or dimensions) of the model
Cloud MM for the health care industry	Source #23	Staged	• Level 1: Departmental, niche applications ○ Medical imaging archiving ○ Personal health records ○ Analytics • Level 2: Core health IT systems ○ Electronic health records (EHR)/health information exchange (HIE) ○ Scheduling/practice management ○ Clinical decision support ○ Quality reporting • Level 3: Virtualised, integrated health networks ○ Health plans ○ Hospitals, clinics and labs ○ Pharmacies ○ Patients and caregivers • Level 4: Seamless care delivery ○ Anywhere, anytime access ○ Personalised care plan ○ Real-time visibility (cost, quality)
Care pathway MM	Source #4	Continuous	• Design • Owner and performers • Performance • Management • Culture
A MM for the implementation of eCPRs	Source #7	Staged	• 1. Literature-based CPRs • 2. Electronic document-based CPRs • 3. Electronic computable individual CPR tools • 4. Service-oriented generalised CPR • 5. CPR with terminology services integration • 6. Learning, versionable CPR

Table 4 shows the levels in the staged representation (or dimensions in continuous representation^42,43) of three example models in different aspect categories.

RQ 2.3: Applicability scope/scale of the MMs

For RQ 2.3, we investigated the applicability scope of the MMs. As discussed earlier, the scope could be one of the following: a single department of a hospital/health care institution; multiple departments of a hospital; a single hospital/health care institution; multiple health care institutions; a city, region or province; government (nationwide); international; or ‘other’. Based on the classification of the sources into the above categories, Figure 7 shows a histogram. Remember that in eliciting the applicability scope of the MMs, we considered the expected/possible applicability of the models and not necessarily the scale of the actual empirical study presented in a given source. The sum of all bars in Figure 7 is 85. Please note that some sources did not report any applicability assessment of the MMs. Thus, their data (scope/scales) are not presented in Figure 7. Furthermore, for some other sources, the scope/scale classification could be multiple (e.g. for Source 15, the applicability scope/scale was both multiple hospitals and nationwide). We found that the most common application scope/scale of MMs was ‘multiple hospitals’, as addressed by 35 sources. For example, Source 4 applied a MM for care pathways to 11 hospitals in The Netherlands. To enable ‘precision medicine’,⁴⁴ Source 24 measured the maturity of BI in six hospitals in Italy. As another example, Source 75 assessed the relationship between organisational maturity and project success in health care based on data from seven hospitals in Portugal.

Figure 7.

Sources with respect to the applicability scope/scale of the MMs.

RQ 3: Properties of empirical studies

RQ 3.1: Scales of empirical studies

To characterise the scale of the reported empirical studies, we focused on the number of health care institutions in which a proposed MM was applied. We also looked into other attributes of the empirical context, for example the number of physicians or other types of experts involved in the surveys, or regular participants who were interviewed, or the number of workshops, focus groups or expert panels that were organised to gather data for the maturity assessment. Note that the applicability scope/scale in RQ 2.3 differs from the empirical scale in RQ 3.1 in that in the case of the former, we assessed the ‘potential’ applicability, while in the latter we looked at the ‘actual’ reported empirical results presented in a source.

Out of the 101 sources, 56 reported quantitative information regarding the scale of their empirical studies. There were 33 sources that performed their applications in health care institutions (hospitals, medical centres or clinics), while the remaining 23 sources performed empirical work based on other forms, such as opinion surveys, workshops or focus groups with health care specialists, or interviews with experts. In the last group (‘other’ forms), seven sources used surveys with diverse stakeholders in the health care domain (e.g. Sources 70, 71, 84 and 88). Five other studies applied focus groups/expert panels for the evaluation of the MMs (e.g. Sources 76 and 77).

These sources typically structured their evaluations as case studies and involved experts from multiple departments within an institution (e.g. Sources 2 and 19). The sources that performed their empirical evaluations in multiple institutions (22 sources), on the other hand, varied greatly in terms of the number of institutions. Four sources applied their MMs on a large scale (i.e. more than 50 institutions). Sources 35 and 78 involved the most institutions in our list of sources (538 and 129 hospitals, respectively). Sources involving high numbers of institutions in their research typically conducted opinion surveys, which incorporated questions from MMs to gain an understanding of the state-of-the-maturity of health care in particular geographical regions. These included, for instance, the maturity of QI systems across Europe (Source 32), e-health care maturity in hospitals in Taiwan (Source 35) or the process management capability of hospitals in Switzerland (Source 78).

It is also important to note that a large majority of sources that reported quantitative data regarding the scale of their empirical studies originated from academia (52/55 sources). Only 3/21 sources (Sources 84, 88 and 97) in the grey literature provided quantitative information about the empirical studies that were conducted.

RQ 3.2: Empirical benefits of applying MMs

In addressing RQ 3.2, we extracted and synthesised the benefits reported by sources as the results of their empirical studies. In doing so, we focused particularly on the benefits reported through the application of the MMs, and whether these benefits were stated in quantitative or qualitative forms (or both). We observed that more than half of the sources (62/101) reported the benefits of applying MMs in qualitative terms, and only two sources reported benefits in quantitative terms (Sources 70 and 79). The remaining 37 sources did not provide any explicit discussions on the benefits of applying MMs.

As an example of quantitative benefits, Source 70 measured maturity of use for EMRs. A comparison of the quantitative data from before and after applying the maturity assessment showed an increase from 21% to 83% of physicians who would use EMRs as the principal method of record keeping. As another example, Source 79 compared procurement maturity with procurement performance in three Dutch hospitals. The gathered numerical measurements supported the hypothesis that an increase in maturity of organisation, processes and IT in hospitals would lead to better procurement performance, thus again showing a quantitative benefit of the proposed MM.

We then looked at the 62 sources which reported the benefits of applying MMs in qualitative terms. To understand the vast amount of qualitative data that we extracted from the papers better, we classified them using the ‘open coding’ (grounded theory) method⁴⁵ which is widely used in qualitative analysis. The process resulted in three classifications: (a) identifying issues and providing guidance for improvement (n = 17); (b) improving efficiency, effectiveness, performance and productivity (n = 12); and (c) ‘other’ benefits (n = 33). The classifications of the sources that report these types of benefits, together with selected examples, are shown in Table 5.

Table 5.

Qualitative benefits of applying MMs.

Qualitative benefits	Number of sources	References to the sources	Example qualitative benefit
Identifying issues and providing guidance for improvement	17	[Sources 1, 3, 4, 17, 19, 22, 30, 36, 41, 46, 55, 68,74, 85, 87, 92, 101]	‘The employees noted that the capabilities included in the model were highly relevant to their situation and provide plenty of discussion points in regards to areas for improvement’. (Source 3)‘The results indicate the usefulness of the proposed model in assessing pathway’s maturity and its potential to provide guidance for its improvement’. (Source 4)‘The MM provides support for identifying deficiencies in strategic, organizational, and technical cooperation capabilities’. (Source 46)
Improving efficiency, effectiveness, performance and productivity	12	[Sources 18, 21, 26, 39, 60, 63, 81, 83, 84, 95, 97, 99]	‘The MMs allow healthcare organisations systematically improve performance’. (Source 39)‘The MMs could enable hospital to deliver benefits quicker and more economically’. (Source 95)
‘Other’ benefits	33	[Sources 6, 11, 12, 14, 21, 22, 23, 24, 31, 41, 43, 44, 48, 51, 52, 53, 56, 60, 63, 68, 73, 74, 78, 79, 80, 81, 82, 84, 85, 87, 88, 96, 100]	‘Sustaining healthcare delivery’ (Source 6)‘Allowing a quick and reproducible analysis of a (health care) centre within a few days’ (Source 11) ‘More ready access to healthcare’ (Source 23)‘Providing useful elements for developing an action plan’ (Source 24)‘A means of internal and external benchmarking, self-assessment, change management, and organisational learning’ (Source 31)

Based on these numbers, we can see that although the number of sources that reported on the benefits of applying MMs in concrete quantitative terms was limited, a vast number of sources reported on the benefits of using MMs in health care in qualitative terms.

Discussion

Observations and suggestions

In response to RQ 1, we reviewed sources by contribution and research type. The general observable trend showed that new MMs are continuously being offered in this area, and there were 68 new MMs proposed. Also, the trend of sources with an emphasis on and having major contributions as empirical (case) studies was an indicator of the progressive nature of MM adoption. In terms of research-method types, a large ratio of sources fell under the categories of ‘weak empirical study’ and ‘solution proposal’. Although this is noted as a good sign of emergence and empiricism in this area, there is a need for studies with stronger empirical evidence.

For RQ 2, we reviewed sources by various properties of MMs, including the identity of the proposed models, aspects covered and applicability scope/scale. We identified and listed MMs proposed by the sources, and also extracted and indexed the subject areas on which they focused. We believe the catalogue describing the available MMs and the index given in Tarhan et al.³⁵ could be useful to practitioners, decision makers and researchers who are interested in assessing the maturity of a given health care entity (e.g. a hospital) from a particular point of view.

The list of subject areas demonstrated a wide range of research and application, which is promising for further research and practice. Since the index of subject areas is very broad, detailed analysis of critical areas for potential future work is needed for various specialties in health care and technical domains. For example, we highlight areas such as practical aspects of patient safety (e.g. incorrect labelling on patients’ drug charts) and the clinical safety in medical and surgical departments. We were able to locate and review three MMs related to patient safety: (a) patient safety culture improvement tool (Source 76), (b) Manchester patient safety framework (Source 77) and (c) safety culture assessment in community (Sources 19 and 85). While these models are useful, we see a need for improvement to cover a broader range of clinical safety.

The MMs also varied in the aspects that they covered (e.g. technology, business processes and people). The majority of the models addressed both the ‘process’ and ‘technology’ aspects, and about half of the models covered the ‘people’ aspect. Almost one third of the models widened their coverage to include ‘other’ aspects such as culture, strategy, governance, leadership, interoperability and data. As the included aspects demonstrated, this area is multidisciplinary in nature and requires organisation of research and practice in such challenging settings. We believe that the increasing amount of digitalisation in health care will make the advantages of using guiding models as MMs even more significant in the upcoming years.

According to the applicability scope/scale of MMs, the majority of the models were targeted for use in ‘multiple hospitals’, followed by ‘a single hospital’, ‘city, region or province wide’, ‘nationwide’, ‘single departments of a hospital, and ‘multiple departments of a hospital’. As a result, there was a balanced variety in the applicability scale of the models. We suggest that the use of MMs among multiple hospitals or within governmental bodies (i.e. nationwide) will be useful, especially for the benchmarking of health care services.

RQ 3 focused on the application scale of the empirical studies performed and the benefits observed. Findings on the empirical applications of MMs showed that there was variety in the number of institutions involved and in the empirical research methods employed (e.g. case studies, surveys, interviews, workshops/focus groups). More than half of the sources reported on the scale of their works, and more than half of those reporting information about the scale of application performed their work directly in one or more health care institutions. More specifically, the MMs were empirically applied in various contexts, ranging from a single hospital to multiple hospitals and to health bodies of countries. Therefore, we argue that an important number of sources rely on a solid empirical basis. On the other hand, a considerable percentage of works, particularly in the grey literature, lacked application or empirical results.

Regarding the benefits of empirical applications, many of the sources reported on the usefulness and utility of the models through qualitative methods, but few presented concrete evidence of the benefits in using them. These benefits included, for instance, increased effectiveness and efficiency in health care service provisioning, improved patient satisfaction and improved service quality. Moreover, there is a need to investigate the relation between qualitative and quantitative benefits in future studies. Designing longitudinal studies to evaluate the effects of increasing maturity in combination with understanding these effects in quantitative terms (e.g. by using process analytics techniques such as process mining^46–48) might be useful for this purpose.

Threats to validity

Here, we discuss the potential validity threats in the context of the four types adopted from Wohlin et al.⁴⁹

Internal validity

In the systematic approach utilised for source selection, search engines, search terms and inclusion/exclusion criteria were carefully defined to ensure that this review is repeatable. Still, the selection of search terms and search engines and the bias in applying exclusion/inclusion criteria can be considered as the limitations during this process. In order to mitigate the risk of missing some relevant sources, formal searching using defined keywords was conducted, and an adequate and inclusive basis has been collected while building the study pool. In order to minimise the bias related to researchers’ judgement and experience in applying inclusion/exclusion criteria, joint voting was applied after the initial source inclusion, and only sources passing the joint voting were selected for review.

Construct validity

This type of validity is concerned with the suitability of the RQs and the categorisation scheme used for the data extraction. Review questions were designed to cover our goal and were answered according to a categorisation scheme. To design a good categorisation scheme, we adapted standard classifications and also finalised the schema through several iterations. Another threat comes from the lack of empirical evidence in the primary sources. The majority of the grey literature was opinion or experience based, and as the source of knowledge was not typically revealed, we are faced with an epistemological problem, that is, we do not know how we know what we know. However, handling this limitation in study design by excluding the grey literature would leave an important source and related voice of practice out of the scope. Therefore, we decided to include these sources as they are in order to have a more complete profile in our review.

Conclusion validity

This type of validity requires reaching appropriate conclusions through rigorous and repeatable treatment. In order to ensure reliability of our treatments, the entire pool of primary sources was analysed, and the data were reviewed, extracted and synthesised by the authors. Following the systematic approach and describing the review questions and procedure ensured replicability of this study and created confidence that the results of a similar study would not have major deviations from our classification.

External validity

As described earlier, search terms in the source-selection approach resulted in having primary sources all written in the English language. However, a good proportion of industrial and collaborative works, in addition to academic studies, exists in our sources due to including the grey literature. This means that our inclusive process of article selection has led us to have an adequate basis for concluding results and that our pool contained sufficient information to represent the knowledge reported by other researchers and professionals. Nevertheless, the findings of this study are mainly within the specific area under study, and we have no intention of generalising our results beyond this field.

Conclusion

The complex and multidisciplinary nature of the health care domain brings significant challenges in terms of process and people management, and IT and IS. Moreover, the increasing digitalisation of health care services demands well-defined guidelines and success reports to rationalise and manage the transition process effectively. In response to those challenges, a wide variety of maturity assessment and MMs has been proposed and applied in the domain.

In this study, we have provided an overview of these models and classified them based on their various properties and applications using MLR as the research method. The results of our review indicate that MMs are continuously being offered under various subjects of health care and also empirically applied, and that the body of sources presents a good balance between research and practice, which is promising for conducting further studies in this area. Future research, however, is required to include results from a stronger empirical basis, especially for the reports in the grey literature.

The index of the subject areas concerned by the MMs is so broad that there is a need for deeper analysis of the sources with respect to detailed specialties of health care and technology. It is suggested that researchers should check the list of subject areas by the MMs reported in this study prior to proposing new models. This is especially important for eliminating identical studies (i.e. reinventing the wheel) as well as for creating opportunity on using or adapting already available MMs.

Finally, we believe that conducting longitudinal studies in different contexts by using quantitative or mixed methods (e.g. process mining) could better demonstrate the advantages of using MMs and therefore serve the community with stronger empirical evidence to advocate the utility of these models – particularly on the side of the practitioners.

Footnotes

Conflict of interest

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Contributorship

A.K.T., V.G. and O.T. performed the search, selection and pilot data extraction regarding the literature review, and S.G. was consulted for her medical knowledge in discussions as to whether to include/exclude the sources. A.K.T., V.G., O.T. and M.S. carried out the data extraction and analysis of all the included studies in the review, during which S.G. was actively consulted and involved in discussions in case of conflicts. All of the authors were involved in the manuscript design, writing, review and revision.

Ethical approval

Not applicable.

Funding

The author(s) received no financial support for the research, authorship and/or publication of this article.

Guarantor

A.K.T.

ORCID iD

Ayça Kolukısa Tarhan

Peer review

This manuscript was reviewed by reviewers, the authors have elected for these individuals to remain anonymous.

References

Quaglini

Information and communication technology for process management in healthcare: a contribution to change the culture of blame. J Software Mainten Evol Res Pract 2010; 22: 435–448.

Kirchmer

Laengle

Masías

Transparency-driven business process management in healthcare settings. IEEE Technol Soc Mag 2013; 32: 14–16.

Tarhan

Turetken

Reijers

Do mature business processes lead to improved performance? – A review of literature for empirical evidence. In: European Conference on Information Systems, Münster, Germany, 26–29 May 2015, paper 178.

Crosby

Quality is free. New York: McGraw-Hill, 1979.

Stevenson

Waite

(eds) Concise Oxford English dictionary. 12th ed. Oxford: Oxford University Press, 2004.

Becker

Knackstedt

Poeppelbuss

Developing maturity models for IT management – a procedure model and its application.

Business Inform Syst Eng 2009; 1: 213–222.

Tarhan

Turetken

Reijers

HA.

Business process maturity models: a systematic literature review. Inform Software Technol 2016; 75: 122–134.

OMG. Business Process Maturity Model (BPMM), v1, https://www.omg.org/spec/BPMM/1.0/PDF (2008).

ISO/IEC 15504-5: Information Technology-Process Assessment. Part 5: An exemplar software life cycle process assessment model, https://www.iso.org/standard/60555.html (2012).

10.

Paulk

MC.

The capability maturity model: guidelines for improving the software process. Harlow, UK: Addison-Wesley, 1995.

11.

Carvalho

Rocha

Abreu

Maturity models of healthcare information systems and technologies: a literature review. J Med Syst 2016; 40: 131.

12.

Söylemez

Tarhan

The use of maturity/capability frameworks for healthcare process assessment and improvement. In: Software Process Improvement and Capability Determination: 16th International Conference (eds MP Clarke, VR O’Connor, T Rout and A Dorling), Dublin, Ireland, June 9–10 2016, pp.31–42. Cham: Springer, 2016.

13.

Ogawa

Malen

Towards rigor in reviews of multivocal literatures: applying the exploratory case study method. Rev Educ Res 1991; 61: 265–286.

14.

McAuley

Pham

Tugwell

, et al. Does the inclusion of grey literature influence estimates of intervention effectiveness reported in meta-analyses? Lancet 2000; 356: 1228–1231.

15.

Alberani

De Castro Pietrangeli

Mazza

AM.

The use of grey literature in health sciences: a preliminary survey.

Bull Med Libr Assoc 1990; 78: 358–363.

16.

Benzies

Premji

Hayden

K A

, et al. State‐of‐the‐evidence reviews: advantages and challenges of including grey literature. Worldviews Evid Based Nurs 2006; 3: 55–61.

17.

Garousi

Felderer

Mäntylä

MV.

Guidelines for conducting multivocal literature reviews in software engineering. Inform Software Technol 2019; 106: 101–121.

18.

Van De Wetering

Batenburg

A PACS maturity model: a systematic meta-analytic review on maturation and evolvability of PACS in the hospital enterprise. Int J Med Inform 2009; 78: 127–140.

19.

Álvaro

Evolution of information systems and technologies maturity in healthcare.

Int J Healthc Inform Syst Inform 2011; 6: 28–36.

20.

Joukes

De Keizer

Cornet

Composite quality of care scores, electronic health record maturity models, and their associations; preliminary literature review results.

Stud Health Technol Inform 2013; 192: 981.

21.

Brooks

El-Gayar

Sarnikar

Towards a business intelligence maturity model for healthcare. In: 2013 46th Hawaii International Conference on System Sciences, Wailea, Hawaii, 7–10 January 2013, pp.3807–3816. Washington, DC: IEEE Computer Society.

22.

Medical Device Innovation Consortium. MDIC case for quality program: maturity model research report, https://mdic.org/wp-content/uploads/2015/06/MDIC-Deloitte_Maturity-Model-Research-Report_June-2015.pdf (accessed 1 October 2019).

23.

Flott

Callahan

Darzi

, et al. A patient-centered framework for evaluating digital maturity of health services: a systematic review. J Med Internet Res 2016; 18: 4: e75.

24.

Carvalho

Rocha

Abreu

Information systems and technologies maturity models for healthcare: a systematic literature review. In: Á

Rocha

Correia

Adeli

Reis

Mendonça Teixeira

(eds) New advances in information systems and technologies. Vol. 2. Cham: Springer, 2016, pp.83–94.

25.

De Carvalho

Rocha

De Vasconcelos

JB.

Maturity models for hospital information systems management: are they mature? In: Y-W

Chen

Torro

Tanaka

Howlett

Jain

(eds) Innovation in medicine and healthcare 2015. Cham: Springer, 2016, pp.541–552.

26.

Söylemez

Tarhan

A review and comparison of maturity/capability frameworks for healthcare process assessment and improvement.

Software Qual Prof 2017; 19: 28–42.

27.

Carvalho

Rocha

Van De Wetering

, et al. A maturity model for hospital information systems. J Bus Res 2019; 94: 388–399.

28.

Van Tulder

Assendelft

WJJ

Koes

, et al. Method guidelines for systematic reviews in the Cochrane Collaboration Back Review Group for Spinal Disorders. Spine 1997; 22: 2323–2330.

29.

Garrard

Health sciences literature review made easy. Burlington, MA: Jones & Bartlett, 2016.

30.

Falagas

Pitsouni

Malietzis

, et al. Comparison of PubMed, Scopus, Web of Science, and Google Scholar: strengths and weaknesses. FASEB J 2008; 22: 338–342.

31.

Shultz

Comparing test searches in PubMed and Google Scholar. J Med Libr Assoc 2007; 95: 442–445.

32.

Jacso

As we may search: comparison of major features of the Web of Science, Scopus, and Google Scholar citation-based and citation-enhanced databases.

Curr Sci 2005; 89: 1537–1547.

33.

Giustini

Barsky

A look at Google Scholar, PubMed, and Scirus: comparisons and recommendations. J Can Health Libr Assoc 2005; 26: 85–89.

34.

Samadzadeh

Rigi

Ganjali

AR.

Comparison of four search engines and their efficacy with emphasis on literature research in addiction (prevention and treatment). Int J High Risk Behav Addict 2013; 1: 166–171.

35.

Tarhan

Garousi

Turetken

, et al. Dataset for paper: maturity assessment and maturity models in healthcare: a multivocal literature review, https://zenodo.org/record/2600722#.XnKBBa-hnqu (2019).

36.

eBizMBA, Inc. Top 15 most popular search engines, www.ebizmba.com/articles/search-engines (accessed 1 October 2019).

37.

McGrath

Sumnall

Edmonds

, et al. Review of grey literature on drug prevention among young people. London: National Institute for Health and Clinical Excellence, 2006.

38.

Adams

Hillier-Brown

Moore

, et al. Searching and synthesising ‘grey literature’ and ‘grey information’ in public health: critical reflections on three case studies. Syst Rev 2016; 5: 164.

39.

Kitchenham

Charters

Guidelines for performing systematic literature reviews in software engineering. Evidence-Based Software Engineering, Technical Report, Ver 2.3, EBSE, 2007.

40.

Gastaldi

Pietrosi

Lessanibahri

, et al. Measuring the maturity of business intelligence in healthcare: supporting the development of a roadmap toward precision medicine within ISMETT hospital. Technol Forecast Soc Change 2018; 128: 84–103.

41.

Bernard

COBIT® 5 – a management guide. Zaltbommel, The Netherlands: Van Haren, 2012.

42.

Weber

Curtis

Chrissis

MB.

The capability maturity model, guidelines for improving the software process. Harlow, UK: Addison-Wesley, 1994.

43.

Lee

Kang

An overview of the Business Process Maturity Model (BPMM). In: Chang KCC, et al. (eds) Advances in web and network technologies, and information management. APWeb 2007, WAIM 2007. Lecture Notes in Computer Science. Vol. 4537. Berlin: Springer, 2007, pp.384–395.

44.

Deigner

H-P

Kohl

Precision medicine: tools and quantitative approaches. London: Academic Press, 2018.

45.

Miles

Huberman

Saldana

Qualitative data analysis: a methods sourcebook. 3rd ed. Thousand Oaks, CA: Sage, 2014.

46.

Van Der Aalst

Process mining discovery, conformance and enhancement of business processes. Berlin: Springer, 2011.

47.

Erdogan

Tarhan

Systematic mapping of process mining studies in healthcare. IEEE Access 2018; 6: 24543–24567.

48.

Gurgen Erdogan

Tarhan

A goal-driven evaluation method based on process mining for healthcare processes. Appl Sci 2018; 8: 894.

49.

Wohlin

Runeson

Höst

, et al. Experimentation in software engineering. Berlin: Springer Science & Business Media, 2012.