Construction of Evaluation Index System of Electronic Archives Management System (EAMS) Based on Analytic Hierarchy Method

Abstract

This study focused on the assessment of electronic archive management systems for digital archives. The objective is to address the challenges encountered in archives informatization and archives management systems, while advancing the development of archives informatization and standardization. This study utilizes the Delphi Method and Analytic Hierarchy Method to determine the weighting and combined weighting of indicators across all levels of the assessment index system for electronic archives management systems (EAMS). It is concluded that the evaluation index system should be consistent with the standard system of electronic archive management system of digital archives, as well as the development strategy and status quo of archive informatization construction, which is itself the result of a complex research process involving the intersection of multiple disciplines. The problems with archives informatization and archives management system are addressed, and the development of archives informatization and archives standardization is encouraged by this paper, which has significant theoretical significance for enriching the evaluation system of electronic archives management (EAMS) system of digital archives. This study provides a complete analysis of assessment standards and evaluation research, proposing a novel evaluation index system that encompasses key variables such as condition guarantee, construction and management, quality assurance, and sustainable development.

Plain Language Summary

Developing a way to assess electronic archives management systems using analytical hierarchy method

This study advanced the development of archives informatization and standardization for future sustainability in this field while focusing on the evaluation of electronic archive management systems (EAMS) for digital archives. The weighting and combined weighting of indicators across all tiers of the electronic archives management system assessment index system were determined using the Delphi Method and the Analytic Hierarchy Method. Using the Delphi method, experts in the fields of archive research and electronic archive management system (EAMS) standardization were polled. Experts were contacted via email and in-person interviews. When choosing specialists, diversity in professional backgrounds and areas of expertise was desired. Twelve eminent experts were chosen after careful consideration. Professionals and workers from standardized institutes and archives who have strong theoretical backgrounds and a lot of expertise managing standards are included in this category. The assessment index system should, it is concluded, be in line with the standard electronic archive management system of digital archives, as well as the development strategy and current state of archive informatization construction, which is itself the outcome of an intricate research process that crosses several disciplinary boundaries. The electronic archive management system (EAMS) framework, comprehensive analysis of assessment standards, practical application, and evaluation research are all included in this study. Additionally, a novel evaluation index system that takes into account important factors like condition guarantee, construction and management, quality assurance, and sustainable development is proposed.

Keywords

digital archives analytic hierarchy delphi method electronic archives management system (EAMS)evaluation index system

Introduction

Within the realm of digital transformation, the electronic archives management system (EAMS) holds significant importance in the construction of archive informatization. The establishment of an electronic archive management system (EAMS) aims to enhance the capacity for managing files in an informative manner and improve the overall quality & effectiveness of EAMS. Currently, there exist numerous projects and findings pertaining to the investigation of such system in China, including the Open Archival Information System (OAIS). The “14th Five-Year Plan for the Development of National Archives” was released in June 2021 by the General Office of the CPC (Communist Party of China) Central Committee and the General Office of the State Council China (SOCC, 2022). This plan emphasizes the necessity of improving and developing digital and intelligent archives management, expedite the development of the digital archives management system, and achieve comprehensive digital transformation of archives (State Office of the CPC Central Committee (n.d); Wu, 2022). In recent years, the study and implementation of smart archives have emerged as a novel research domain and a progressive path in the field of archive informatization (J. Li, 2020; Zhu, 2023). Currently, there exist certain challenges in the implementation of EAMS. These challenges primarily encompass the absence of comprehensive strategic planning, the lack of standardized protocols, and varying levels of prioritization, leading to uneven levels of development (Cui, 2022; N. Liu & Kong, 2013). Additionally, the infrastructure development pertaining to software and hardware remains incomplete, thereby indicating a deficiency in advanced information technology. There is a need for improvement in the management system and measures, as well as a lag in the development of the security system. The absence of effective integration between information systems and company development, inadequate presence of skilled individuals, and insufficient allocation of funding for building are some of the key challenges identified in this context.

This study focuses on constructing an evaluation index system for the EAMS. summaries the evaluation standards and research on EAMS both domestic and international level. It develops an evaluation model for EAMS, designs evaluation indicators, and constructs an evaluation index system using the Delphi Method and Analytic Hierarchy Method to determine the weighting and combined weighting of indicators across all levels of the assessment index system for EAMS. According to the study’s findings, an implementing of EAMS should make use of cutting-edge information technology to make it easier to create an effective information platform. The careful management of the system, as well as improving security oversight and enhancing the use of information resources, should be emphasized. Improve the level of service while enhancing the user experience

Review of Relevant Literature

Status of Evaluation Indicators for Electronic Archives Management System (EAMS)

The International Organization for Standardization (ISO) released ISO 16363, titled “Audit and Certification of Trusted Digital Collections,” in 2012. This international standard is widely employed across the globe to assess and certify reliable storage systems utilized by long-term preservation institutions (CRL & OCLC, 2022). The ISO 16363 standard offers a comprehensive array of assessment and certification frameworks for digital repositories, aiming to guarantee the enduring preservation and effective management of digital content, while also delivering dependable services to users and stakeholders of such repositories (Poveda et al., 2021). The ISO 16363 standard offers comprehensive principles and standards that encompass several facets of digital repositories, such as organizational and managerial characteristics, technological infrastructure, digital objects, and metadata, access and interaction, as well as sustainability and long-term viability (Barros et al., 2018). The ISO 16363 Trusted Warehousing Certification System encompasses three main categories of requirements: infrastructure assurance, digital management, and risk management. These categories are further divided into 13 sub-categories, each containing over 100 specific certification indicators.

The Open Archival Information System (OAIS) is a standardized information system that aims to safeguard and preserve digital information. It is founded on the principles of the electronic document life cycle and is widely regarded as the most comprehensive and structured document outlining the functional criteria for electronic document management. The provided framework offers a reference model and conceptual framework that enhances the management and security of data information within information systems (Barbau, 2013; Richard, 2002). In 2003, it was formally incorporated into the International Organization for Standardization (ISO) as ISO 14721:2003. Currently, the most up-to-date and effective version of this standard is ISO 14721:2012. The OAIS reference model encompasses a framework structure with six functional modules: ingestion, file management, data management, system management, preservation planning, and access functions (The Library of Congress [LC], 2022). Each of these functional entities encompasses over 30 sub-functional activities. Scholars have undertaken study in the field of long-term preservation technology, focusing on specific objects such as mail, journals, computer-aided products, multimedia, and other unique entities (Ball, 2023; Beagrie, 2022; Digital Preservation Coalition, 2023; Prom, 2023; Wright, 2022).

The European Union introduced the “Model Requirements for the Management of Electronic Records” standard in March 2001, also known as “General Requirements for Electronic Document Management” or Moreq (Weisinger, 2023). It has been observed that the specifications provided by Moreq lack specificity, and MoReq2 was released in early 2008, However, it is important to note that Moreq2 does have certain limitations (Fresko, 2007, 2008, 2009; Wilhelm, 2009) . As a result, in 2011, a new iteration of the “File System Modular Requirements System” was introduced, known as MoReq2010. The primary objective of MoReq2010 was to overcome the limitations of Moreq2 and broaden its applicability across various industries, such as healthcare, finance, and law. This expansion was intended to cater to the diverse requirements of these industries and beyond (Wilhelm, 2002; Zhang, 2013). The MoReq2010 framework initially introduced two fundamental principles: the modular description of needs and the concept of a service-based architecture. The MoReq standards have gained significant recognition on a global scale, prompting countries to actively assess the compatibility of their own standards with MoReq standards during the development process (Hanger & Lupprian, 2005). Efforts are made to ensure consistency and alignment with MoReq standards. An instance of this can be observed in the case of the British National Archives, where they made use of the MoReq version of the standard during the revision process of their national electronic document management system requirements, known as TNA2002 (Wilhelm, 2002). Similarly, Finnish scholars utilized the MoReq2 version of the standard as a reference point while developing their own electronic document management system requirements standard, named SHAKE (Henttonen, 2009).

Status of Evaluation Indicators for Electronic Archives Management System (EAMS) China

The “Digital Archive System Test Measures” and the “Digital Archive System Test Index Table” were released by the Office of the National Archives Administration in November 2014. These documents serve as the foundation for conducting tests and include 99 specific indicators. The scope of the tests is not limited to the information system alone, but also encompasses infrastructure, system functions, and archive resources, among other aspects. The test indicators of the digital archive system will undergo future updates as a result of the progress in archive informational and technical advancements. The future versions of the “Digital Archive System Test Measures” should include network security standards, enable the opening of mobile ports, and enhance user experience and other relevant metrics. Currently, there is a substantial body of literature dedicated to the evaluation of electronic archives management systems is considered as an application system evaluation within the context of digital archives. It incorporates a thorough evaluation index consisting of four indicators: infrastructure, resource creation, system function, and security assurance. This evaluation system aims to provide a comprehensive assessment of digital archives (Ni & Jin, 2013; Xue, 2019) conducted an analysis of the elemental composition and fundamental features of digital archives. As a result, a comprehensive evaluation index system was presented for assessing digital archives. This system encompasses four key areas, namely collection resources, system technology, service quality, and collection management (Zhou, 2005). The literature review revealed a scarcity of assessment studies exclusively focused on the digital archive system. Instead, the available studies mostly relied on the interpretation of indications outlined in the test criteria for the digital archive system. In the “Measures for the Evaluation of Digital Archives,” conducted a comprehensive examination of the evaluation indicators pertaining to the application system function of digital archives (Ouyang, 2019).

The assessment criteria and research perspectives of digital archives exhibit a wide range of diversity. By applies the analytic hierarchy technique using triangular fuzzy numbers to ascertain the weightage and presents an assessment approach for the comprehensive risk and risk level of the university cloud digital archive system (L. H. Liu, 2020). A study focused on the evaluation system technology, specifically examining four primary indicators: system technology, information retrieval technology, user service technology, and interface design. Analysis, constructed an evaluation index system for digital archives technology (H. Jiang, 2016). The evaluation of construction performance in China’s digital archive with the research objective was to develop a performance evaluation index system using the BSC (Balanced Scorecard) model (X. Gao, 2019). The index system consists of first-level indicators, namely resource allocation, performance of duties, user evaluation, and comprehensive benefit. Simultaneously, the assessment of digital archives focuses increased significance on the notion of user-centricity, and lays more emphasis on the consideration of users’ requirements, perceptions, and satisfaction when designing evaluation indicators (C. Gao & Huang, 2017). Study presented a design of evaluation indicators for digital archives, taking into consideration the requirements of digital archive development and design, including system development and design rules and standards. The evaluation indicators include security, performance, operability, and function, therefore establishing a comprehensive evaluation index system for digital archives (Zhao, 2013). The primary entity responsible for assessing the service quality of digital archives’ information utilization is the general public. Additionally, the public plays a crucial role in guiding the sustainable development of service quality. However, the level of public participation in the actual evaluation process remains low. To address this issue, the researchers have developed a comprehensive evaluation index system that focuses on public-facing aspects. This system comprises five primary indicators, namely collection resources, availability, special services, effectiveness, and service results (P. Jiang & Zhang, 2015).

Furthermore, scholars have undertaken comprehensive study on the subject of assessment and the theoretical implementation of digital archives. The research theory of archive pluralism informs the design of four first-level indicators: condition guarantee, resource formation, business management, and integrated service. These indicators are used to systematically construct an evaluation index system for the construction of national digital archive resources (Hao et al., 2017). Based on the theoretical framework of user information demand behavior, an analysis was conducted of the various factors that influence user perception and satisfaction during the provision of digital archive services. Consequently, a comprehensive evaluation index system is developed, comprising first-level indicators such as supporting conditions, service construction, retrieval functions, and collection resources. This evaluation index system serves as the basis for assessing the effectiveness and quality of digital archives (Shi, 2011). The distinctions between various methods of service performance evaluation, namely the input evaluation method, task evaluation method, comprehensive evaluation method, and CIPP (background, input, process, and achievement) evaluation theory as well as the comparative analysis were conducted to assess these methods (Y. Li, 2011). Furthermore, utilizing the CIPP evaluation method as a framework, an evaluation index system is developed for assessing the service performance of digital archives. This index system is constructed based on four primary indicators: use evaluation performance, service level, service quality of archivists, and user satisfaction. An examination on the inherent developmental characteristics of digital archives, as well as the many roles and components that comprise them was conducted (Sun & Liu, 2018). It was observed that digital archives had distinct core process areas that align with the requirements outlined in the Capability Maturity Model (CMM). The digital preservation capability maturity model (DPCMM) is offered as a means to assess the efficacy of the capability maturity model (CMM) in the context of digital resources. The feasibility of incorporating the Capability Maturity Model (CMM) into the assessment system for digital archives was observed. A scholarly examination of the pertinent theories surrounding the development of archive information systems was conducted and study involves a comprehensive analysis of the knowledge level, using the capacity maturity assessment model approach. Furthermore, an evaluation system is developed, consisting of two primary indicators: functional operation and influencing variables (Zheng, 2020).

Furthermore, compared with the existing standards or frameworks, the evaluation indicators of the electronic archive management system established in this paper are more scientific and reasonable. The indicators are derived from the standards and norms of two aspects: “Systems and Software” and “Electronic Archive Management.” In terms of “Systems and Software,” the national standard requirements of GB/T 25000.51-2016 “Systems and Software Engineering - System and Software Quality Requirements and Evaluation (SQuaRE) - Part 51: Quality Requirements and Test Details for Ready-to-Use Software Products (RUSP)” are referred to. In terms of “Electronic Archive Management,” the requirements of international digital archive long-term preservation projects (such as the OAIS model, ISO 14721 standard, ISO 16363 standard, etc.), the national standard GB/T 39784-2021 “General Functional Requirements for Electronic Archive Management Systems,” and the industry standard “Testing Methods for Digital Archive Systems” are referred. Authors, have conducted an in-depth analysis and critical comparison of the commonly used electronic archive management system evaluation standards at home and abroad, clarified the deficiencies of the existing evaluation methods in terms of comprehensiveness of indicators, system structure and scientific weight, and further clarified the differentiated advantages and unique contributions of the system proposed in this article (Cain, 2002; Sundqvist & Svärd, 2016). Specifically, we pointed out that the existing standards have insufficient consideration of the integration of emerging technologies and have ignored the sustainability and dynamic optimization of the system, while the indicator system constructed in this paper has made up for these shortcomings (Oliver & Foscarini, 2014).

Research Methodology

Literature Research Method

This study utilizes Chinese electronic books, periodical databases, and foreign language electronic journal databases available on platforms such as Web of Science and CNKI. A keywords approach employed to retrieve the related articles “electronic archive management system,”“digital archive system,”“electronic document management system,”“evaluation,”“index,” and “test.” Relevant articles are collected and subjected to literature collation and analysis. The objective is to gain insights into the research conducted by scholars both domestically and internationally on the evaluation system of electronic archive management systems. The methodology used, the outcomes attained, the unresolved issues, and afterward, a pragmatic study proposal was devised.

Comprehensive Evaluation Method

In order to develop the evaluation index of EAMS and calculate its weight and combination weight, this study used the Delphi method (Okoli & Pawlowski, 2004) and the analytic hierarchy technique. The research object was a representative electronic archives management system; questionnaires were designed, investigated, and collected; a thorough evaluation was conducted; evaluation results were analyzed; the constructed electronic archives management system’s evaluation index system was put to the test; and recommendations were made to address the system’s shortcomings.

Delphi Law

(a) Selection of expert groups.

The Delphi method was employed to gather insights from experts who possess a strong theoretical foundation or practical expertise in the domains of archive research and standardization of electronic archive management systems. These experts were engaged through various means such as face-to-face interviews and email correspondence. The selection of experts aimed to ensure diversity in terms of their professional backgrounds and fields of work. Following extensive deliberation, a cohort of 12 esteemed professionals was ultimately chosen. Specifically, this group comprises specialists and personnel hailing from archives and standardized institutes, possessing a solid foundation in theory and a wealth of expertise in standardization administration.

(b) Prepare an expert consultation questionnaire.

Based on the initially established comprehensive evaluation index system of the electronic archives management system, opinions were collected on the first-level indicators, second-level indicators and third-level indicators, and an expert consultation questionnaire on evaluation indicators was compiled. The collection of opinions on the first-level indicators is as shown in the table. The questionnaire survey content mainly includes the importance, judgment basis and familiarity evaluation of the comprehensive evaluation indicators of the electronic archives management system. Members of the expert group will conduct their own evaluations respectively. The expert group members independently evaluate these indicators. After the questionnaire survey is completed, the questionnaire result data is imported into Excel for analysis and calculation, and the arithmetic mean, full score frequency and coefficient of variation of the scores for each indicator are calculated. The arithmetic mean indicates the degree of concentration of expert opinions. When the arithmetic mean is higher, it indicates that the indicator has strong importance, representativeness and feasibility. The coefficient of variation reflects the experts’ views on the indicator, and it can be used to measure whether experts’ views on the indicator are consistent or fluctuate. When the coefficient of variation is smaller, experts are more capable of coordinating the indicator.

The result data of the first-level indicators in the questionnaire survey has shown in Table 1. The calculation results of the critical values of arithmetic mean, full score frequency and coefficient of variation are as follows:

(1) The critical value of the arithmetic mean is: 4.664 − 0.170 = 4.494. If the arithmetic mean of the indicator is higher than this critical value, then the indicator is retained while being adjusted based on the modification opinions proposed by the experts.

(2) The critical value of the full score frequency is: 0.664 − 0.170 = 0.494. If the full score frequency value obtained by the indicator is higher than this critical value, then the indicator will be retained and adjusted according to the modification opinions proposed by experts.

(3) The critical value of the coefficient of variation is: 0.093 + 0.021 = 0.114. If the coefficient of variation value obtained by the indicator is lower than this critical value, then the indicator is retained and adjusted according to the modification opinions proposed by experts (Table 2).

Upon receipt of the questionnaire survey feedback from the respondents of the study, author employed MS Excel for data arrangements and SPSS to conduct data processing and statistical analysis. This involved evaluating the representativeness of the experts, as well as calculating the expert positivity coefficient, expert authority degree, and expert coordination coefficient. These calculations were instrumental in assessing the credibility and scientific reliability of the responses to the questionnaire, ultimately leading to the determination of the final index system.

Table 1.

Correlation Coefficients of First-Level Evaluation Index Elements.

Code	Index	Arithmetic mean	Full score frequency	Coefficient of variation
A1	Conditions guarantee	5.000	1.000	0.000
A2	Construction and management	4.667	0.667	0.101
A3	Quality assurance	4.917	0.917	0.056
A4	Sustainable development	4.833	0.833	0.077

Table 2.

Level 1 Evaluation Index Questionnaire.

Sr. No.	Indicators	Importance Base Score					Judgment Base Selection				Familiarity Base Selection
Sr. No.	Indicators	Very Unimportant	Not Important	Moderately Important	Somewhat Important	Very Important	Based On Practical Experience	Based On Theoretical Analysis	Based On The Opinions Of Domestic And Foreign Peers	Based On Intuition	Particularly Familiar	Relatively Familiar	Generally Familiar	Unfamiliar,	Very Unfamiliar
1	Conditions Guarantee (A1)	1	2	3	4	5
2	Construction and Management (A2)	1	2	3	4	5
3	Quality Assurance (A3)	1	2	3	4	5
4	Sustainable Development (A4)	1	2	3	4	5

Note. Please put “√” in the corresponding spaces in the “Importance” column, the “Judgment Basis” column and the “Your Familiarity with the Survey Content” column respectively.

The experts’ positivity coefficient has been used to measure degree of cooperation of the experts who participated in the questionnaire survey for this study, and it can reflect the enthusiasm of the experts. The degree of expert authority used to measure the authority of the experts who participated in questionnaire survey for this research. The reliability of the assessment generally the degree of expert authority is Cr ≥ 0.50 which indicated that the authority of experts and scholars is higher. Cr is derived from the average number of experts and scholars’ evaluation coefficient Ca and familiarity coefficient Cs on the questionnaire content, that is, Cr = 1/2 (Ca + Cs). The judgment coefficient Ca is the expert’s basis for judging the evaluation content. The judgment basis is obtained by the expert’s own evaluation. According to the convention, it is divided into four categories: Based on Practical Experience, Based On Theoretical Analysis, Based On The Opinions Of Domestic & Foreign Peers and Based On Intuition. The familiarity coefficient Cs is the expert’s familiarity with the evaluation content. The degree of familiarity is determined by the expert’s own evaluation. According to the convention, it is divided into five categories: particularly familiar, relatively familiar, generally familiar, not very familiar, and unfamiliar.

The degree of expert coordination used to reflect the degree of consistency between the experts participating in the questionnaire on the importance of each indicator and to test the reliability of the questionnaire results. The degree of coordination of experts is reflected by Kendall’s coordination coefficient (Kendall’s W). Through the calculation of Kendall’s coordination coefficient, the consistency level of evaluation can be measured scientifically and objectively. Its value range is between 0 and 1. The closer the Kendall coordination coefficient is to 1, the higher the degree of recognition of the evaluation experts and the better the degree of coordination.

Moreover, based on the initially established comprehensive evaluation index system for the electronic archives management system, opinions were solicited on the first-level, second-level and third-level indicators, and an expert consultation questionnaire for the evaluation indicators was compiled. It is planned to conduct two rounds of Delphi method questionnaires. The content of the questionnaire mainly includes the importance, judgment basis and familiarity evaluation of the comprehensive evaluation indicators of the electronic archives management system, which are self-evaluated by the members of the expert group respectively. After receiving the questionnaire results, we used tools such as Excel and SPSS to process and statistically analyze the data, evaluate the representativeness of the experts, calculate the expert positive coefficient Caj, expert authority degree Cr and expert coordination coefficient W, etc., to assess the credibility and scientificity of the questionnaire survey, and determine the final index system. Authors have supplemented the detailed explanation of the Delphi method questionnaire design, expert screening criteria (such as professional title, years of experience, research experience, etc.) and the content validity and reliability of the questionnaire in the article to improve the transparency of the research method (Saaty & Vargas, 2012). In addition, we have further clarified the subjectivity of the AHP method, and reduced the risk of subjectivity through consistency testing and multiple rounds of expert consultation (Ishizaka & Labib, 2011; Saaty, 2008).

Analytic Hierarchy

In order to ensure the effectiveness of evaluating the electronic archives management system, it is crucial to provide weights to the evaluation index system, thereby expressing the relative importance of each index. The effectiveness of the evaluation findings obtained from the evaluation index system is directly influenced by the fairness of the weight distribution. The Analytic Hierarchy Process (AHP) is a method that entails the subjective allocation of weights to measurements with the aim of ascertaining their relative significance. The weights are determined based on the subjective evaluation of experts, who rely on their personal experiences. The system exhibits advantages in terms of its basic functionality, precision, and efficiency.

Construct a matrix with the objective of facilitating well-informed assessments. The objective was to ascertain the average indicator scores across all levels by employing data obtained from the expert consultation questionnaire table. The important scale below table will be utilized as a point of reference in order to build a scale that is based on the disparity between the means. It displays the assignment and interpretation of the scale. It is imperative to acknowledge that the reciprocal relationship exists between the scale value with respect to A and B, and the scale value with respect to B and A.

It was considering the indices S1, S2, S3, and S4 as exemplars for the purpose of illustration. Let us consider a scenario where the mean scores of the expert consultation questionnaire table are reduced by two pairs. A judgment matrix for the indicators S1, S2, S3, and S4 is generated based on the provided information, as depicted in below Table 3.

Table 3.

Indicator S1, S2, S3, S4 Matrix.

	S1	S2	S3	S4
S1	1	2	1	2
S2	1/2	1	1/2	1/2
S3	1	2	1	2
S4	1/2	2	1/2	1

Step 1: The comparison results as a matrix according to the scales in Table 4:

[\begin{matrix} 1 \\ 1 / 2 \\ 1 \\ 1 / 2 \end{matrix} | \begin{matrix} 2 \\ 1 \\ 2 \\ 2 \end{matrix} | \begin{matrix} 1 \\ 1 / 2 \\ 1 \\ 1 / 2 \end{matrix} | \begin{matrix} \begin{matrix} \begin{matrix} 2 \\ 1 / 2 \end{matrix} \\ 2 \end{matrix} \\ 1 \end{matrix}]

Step 2: Normalize the vector columns of the matrix, and the processing result is:

[\begin{matrix} 0.333 \\ 0.167 \\ 0.333 \\ 0.167 \end{matrix} | \begin{matrix} 0.286 \\ 0.143 \\ 0.286 \\ 0.286 \end{matrix} | \begin{matrix} 0.333 \\ 0.167 \\ 0.333 \\ 0.167 \end{matrix} | \begin{matrix} \begin{matrix} \begin{matrix} 0.364 \\ 0.091 \end{matrix} \\ 0.364 \end{matrix} \\ 0.182 \end{matrix}]

Step 3: Normalize the vectors of each row of the matrix, and the processing result is:

[\begin{matrix} 1.316 \\ 0.567 \\ 1.316 \\ 0.801 \end{matrix}]

Table 4.

Average Random Consistency Index RI Values.

	Matrix order
Particulars	1st level	2nd level	3rd level	4th level	5th level	6th level
RI value	0	0	0.52	0.89	1.12	1.26

The upper matrix is normalized to obtain the weights of each indicator:

WA1 = 1.316, WA2 = 0.567, WA3 = 1.316, WA4 = 0.801.

Step 4: Calculate the maximum feature root of the matrix λ_max, Calculation results λ_max = 4.062.

(2) Consistency test of matrix.

CI is the consistency index of the judgment matrix, and its calculation formula is:

CI = \frac{λ_{\max} - n}{n - 1}

(1)

Where: n is the order of the matrix of the judgment matrix.

CR is the random consistency ratio of the judgment matrix, and its calculation formula is:

CR = \frac{CI}{RI}

(2)

Among them, RI is the average random consistency index, and the value is calculated according to Table 5.

Table 5.

Evaluation Indicators for Index System of Electronic Archives Management System.

Level 1 indicators	Level 2 indicators	Level 3 indicators
Condition Guarantee (A1)	Funding guarantee B1	Budget C1
	Funding guarantee B1	Expenses C2
	Institutional guarantees B2	Management system C3
	Institutional guarantees B2	Strategic planning C4
Construction and Management (A2)	Resources building B3	Diversified archive resources C5
		File naming is prescriptive C6
		File format & quality standardization C7
		Archive metadata integrity C8
	Security management B4	Physical device security C9
		Cyber-security C10
		Operating system & platform security C11
		Hacking and virus prevention C12
		Database security C13
Quality Assurance (A3)	System function B5	File Reception C14
		File management C15
		Archival preservation C16
		Archival Exploitation C17
		System management features C18
		Personalization features C19
	System performance B6	Reliability C20
		Scalability C21
		Openness C22
		Convenience C23
		Advancement C24
	Quality of service B7	File utilization C25
		Retrieval quality C26
		Retrieval speed C27
Sustainable Development (A4)	Management performance B8	Plan extent to achieved goals C28
	Management performance B8	Management innovation C29
	Service performance B9	External personnel satisfaction C30
		Staff satisfaction C31
		Economic benefits C32
		Social benefits C33

When the order of the matrix is less than the third order, it is unnecessary to compute the CR value of the judgment matrix, as the consistency of the judgment matrix is deemed adequate. When the order of the matrix is equal to or more than three, it is necessary to compute the CR value of the matrix. When the calculated CR value is below 0.1, typically, a RI number below 0.1 is regarded as favorable, while a RI value ranging from 0.1 to 0.25 is deemed acceptable. Conversely, a RI value between 0.25 and 0.5 is considered subpar, and a RI value over 0.5 is categorized as highly unsatisfactory. The RI values can be used to identify the matrices that are most inconsistent and to focus on improving the consistency of these matrices. This can be done by re-evaluating the judgment of the experts or by providing them with additional information about the factors that are being evaluated. By reducing the inconsistency of the judgment matrices, it is possible to improve the reliability and validity of the results of the evaluation of the electronic archive management system.

Determine the Combined Weights of Indicators at All Levels

In this research the product approach used as a means of determining the aggregated weights for each indicator. The proposed approach involves assessing the weights and distributions of various evaluation levels using the product technique. This allows for the determination of the aggregated weight for each evaluation index, hence enabling a precise evaluation of the overall assessment outcomes. The weight and combined weight of the first-level index, second-level indicator, and third-level indicator are determined using mathematical calculations.

Design of Evaluation Indicators of Electronic Archives Management System (EAMS)

The assessment of the developed electronic archives management system is based on of planning, construction, management, service, and sustainable development of said system. First guarantees level required essential hardware, software, network, and other infrastructure, as well as financial resources, to support the electronic archives management system. Furthermore, it ensures that the construction of the electronic archives management system adheres to the predetermined plan and objectives. The first step is formulating the design requirements for the EAMS. This entails identifying the specific functionalities and features that the system should possess. Next, it is necessary to carefully select a suitable system supplier who will be responsible for the design, development, operation, and other construction aspects of the system. This selection process should involve a thorough evaluation of potential suppliers based on their expertise, experience, and track record in similar projects. Once the system supplier has been chosen, it is crucial to initiate and oversee the system security management work. This entails implementing measures and protocols to ensure the confidentiality, integrity, and availability of the system and its data. This includes establishing access controls, encryption mechanisms, and regular security audits to identify and address any vulnerabilities or threats to the system’s security. Subsequently, it is imperative to offer consumers a file utilization service that enhances satisfaction by focusing on both system quality and service quality. Ultimately, by recognizing the significance of managerial performance and implementing cutting-edge technology, the EAMS may ensure its long-term viability and consistently provide economic and social advantages, thereby establishing a self-reinforcing cycle. The evaluation index system that has been developed has four primary indicators, including condition guarantee, construction and management, quality assurance, and sustainable development. These primary indicators are further broken down into nine secondary indicators and a total of 33 third-level indicators. Hence an in-depth analysis of the interactive relationship between indicators. In particular, it is pointed out that the funding and institutional guarantees in the “condition guarantee” indicator have a direct or indirect positive effect on the system performance and service quality in the “quality assurance” indicator (Purcell, 2019). At the same time, we cited relevant research support and pointed out that infrastructure construction and system improvement in electronic archive management systems are important prerequisites for realizing system functions and improving service quality (Oliver & Foscarini, 2014).

The evaluation index system constructed in this article takes into account the standards and normative requirements of “system and software” and “electronic archives management” in the selection of evaluation indexes. The indexes of “system and software” refer to ISO/IEC 25051:2014 “Software.” Engineering-Systems and software Quality Requirements and Evaluation (SQuaRE)-Requirements for quality of Ready to Use Software Product (RUSP) and instructions for testing standard requirements, and the indicators of “electronic archives management” refer to ISO 16363, OAIS and other archive digital resources Long-term preservation projects, GB/T 39784-2021 “General Functional Requirements for Electronic Archives Management Systems,” archives industry standard “Digital Archives System Test Methods” and other standard requirements. At the same time, the literature analysis method is used to draw on the evaluation indicators of relevant journal documents to construct Evaluation index system, so the evaluation index system constructed is more standardized, scientific and effective.

Condition Guarantee A1

The perfection of the condition guarantee is the fundamental guarantee of the EAMS. The author considers “condition guarantee” as the primary index in the evaluation index system of the EAMS. This primary index is further divided into two secondary indicators: financial guarantee B1 and system guarantee B2.

Funding Guarantee

The concept of a “funding guarantee” (B1) refers to a commitment or assurance of financial support. The establishment and functioning of an EAMS necessitate a substantial financial commitment, and an adequate capital investment is a prerequisite for the successful implementation of such system.

(a) Budget C1: It is the system construction, including management, operation and maintenance, upgrade and other expenses included in the budget of the unit;

(b) Expenses C2: The system hardware configuration encompasses various components such as computers, servers, storage facilities, network technology equipment, input and output equipment, and other facilities that are equipped to fulfill the operational requirements. Additionally, the software configuration includes the expenses associated with operating systems and application software, as well as their operation and maintenance.

Institutional Guarantee B2

The establishment and enhancement of archives rules and regulations are necessary for the electronic archives management system to assure the standardization, systematic approach, and institutionalization of archives management.

(a) Management system C3: The system complies with relevant laws, regulations, rules, standards, and existing unit management systems;

(b) Strategic Planning C4: It is the strategic planning of the systematic formulation of phased goals and overall objectives.

Construction and Management A2

The primary focus of the creation and management of an EAMS is the establishment and administration of said system. The author includes “construction and management” as the second primary index in the evaluation index system of the EAMS. This primary index consists of two secondary indicators: resource construction B3 and security management B4.

Resource Building (B3)

The security management of resource construction is an important means to ensure the data security of the EAMS.

(a) Diversification of archive resources C5: All kinds of electronic archives and traditional carrier archives are included in the system for centralized and unified management;

(b) Normative C6 of file naming method: Electronic archives or digital copies of traditional carrier archives should be automatically named using file numbers;

(d) Archive Metadata Integrity C8: Complete and complete metadata for various categories of electronic archives.

Security Management (B4)

The implementation of a network security system necessitates the establishment of a robust security strategy and management framework. This entails enhancing security training for both institutions and personnel, as well as intensifying oversight and management of security infrastructure and operations. Ultimately, this will culminate in the development of a comprehensive security management system.

(a) Physical device security C9: Is the secure storage location and access control of system supporting equipment;

(b) Network security C10: The system has network isolation and access control management;

(c) Operating system and platform security C11: Prevent unauthorized access to systems and application platforms, with identity authentication, access control, and security audit functions;

(d) Hacker intrusion and virus prevention C12: The system can effectively prevent malicious attacks carried out by overseas countries and organizations;

(e) Database Security C13: The system ensures data integrity and confidentiality, and has efficient data backup and recovery functions after disasters.

Quality Assurance A3

The administration of electronic archives is assisted by the use of electronic records management systems. The assessment of service quality is an essential component in evaluating the efficacy of these systems. The inclusion of “quality assurance” as the third primary index in the assessment index system of the EAMS is emphasized by the author. The primary index is composed of three secondary indicators, namely system function B5, system performance B6, and service quality B7.

System Function (B5)

The collection, management, storage, and utilization of the electronic archives management system encompass various functions, such as navigation, assistance, result output, interactive services, customization, specialized services, information dissemination, and personalization. The functionality of the electronic archives management system is subject to the advancements in information technology and the evolving policies, regulations, and standardization in archives management. Consequently, the system continually expands its capabilities and adapts to emerging requirements.

(a) File reception C14: Meet the needs of online and offline reception and detection of different data types of files;

(b) Archives management C15: Meet the orderly management of digital archives and assist the management of physical archives;

(c) Archive preservation C16: According to the requirements of long-term preservation, classify and store archives, format conversion, and backup;

(d) File utilization C17: To meet the different utilization needs of LAN, government network, Internet, and other environments;

(e) System management function C18: With user management functions such as identity authentication, user setting and permission assignment, audit, and log analysis;

(f) Personalized function C19: Navigation function, help function, result output, customized service, special service information push, interactivity.

System Performance (B6)

The effectiveness of the file management system is contingent upon the performance of the software architecture employed in the EAMS. This encompasses the dependability, scalability, accessibility, user-friendliness, and advanced functionality of the program.

(a) Reliability C20: The software system is required to be able to complete the predetermined function within the specified time and continue to operate under the specified conditions without serious errors and failures.

(b) Scalability C21: Software systems that can effectively support business scale expansion, such as supporting large data volumes and concurrent access, while meeting business needs.

(c) Openness C22: The system structure is required to have the characteristics of openness, and functional integration, data exchange and data sharing with other systems can be realized when needed.

(d) Convenience C23: The system is required to provide full-text online access, Internet of Things and mobile terminal and other convenience services.

(e) Advanced C24: It reflects the degree of application of systems knowledge management, artificial intelligence, digital humanities, and other technologies.

Quality of Service (B7)

Service quality pertains to the extent of service provision by archives to users. A superior level of service quality has the potential to enhance user contentment, thus fostering heightened user engagement and involvement in using the records management system.

(a) Archival utilization rate C25: The degree of utilization of archival information resources;

(b) Search quality C26: Search quality indicators that comprehensively consider recall rate and precision rate. Search quality = 2 × precision rate × recall rate/(precision rate + recall rate);

(c) Retrieval speed C27: The consistency between the retrieval response time of the system and the reasonable response time of the system.

Sustainable Development A4

The digital archive the environment is currently encountering both opportunities and challenges in its development. As a result, there has been a discourse on the management of this ecosystem, with a focus on the concept of “sustainable development” as the prevailing imperative and inevitable trajectory for its advancement (Ni & Jin, 2019).

Management Performance (B8)

The management performance of the EAMS system refers to the management ability and management performance of the system administrator, management people and management team in the execution of the file management system. The capacity to proficiently strategize, arranges, synchronize, oversee, and assess the operations of the archives management system is encompassed inside this capability.

(a) Level of Attainment of Planned Objectives C28: The intended strategic planning objectives encompass various aspects, including the determination of the number and scope of phases to be accomplished, the overarching objectives to be pursued, and the precise indicators to be utilized for measuring progress and success;

(b) Management Innovation C29: This initiative focuses on implementing reform and innovation initiatives within the management model, organizational structure, business processes, and departmental decision-making of the system. The objective is to enhance the overall efficiency and effectiveness of file management.

Service Performance (B9)

The service performance of the file management system pertains to the quality and level of service offered by the system, namely its capacity to fulfill user requirements. The evaluation of service performance can encompass various dimensions, including but not limited to customer satisfaction, economic considerations, and social impacts.

(a) External Satisfaction C30: Satisfaction of users receiving archival utilization services with the process and results of the service, including response time, corresponding quality;

(b) Staff satisfaction C31: Satisfaction of system staff with the use of the system, supplier training and after-sales service;

(c) Economic benefits C32: Advantages in the realm of economics the electronic archives management system (C32) has been found to yield both direct and indirect benefits in enhancing the quality and efficiency of archived information resource utilization services. Additionally, it has been observed to contribute to the reduction of various expenditures and prices;

(d) Social benefits C33: Provide valuable information for social development and scientific research by providing the public with archival information resource utilization services.

The results of determining the weight of the evaluation index system of the electronic archives management system of digital archives are shown in Table 6.

Table 6.

Weights and Combined Weights of Primary, Secondary and Tertiary Indicators.

Primary indicators	Weight	Secondary indicators	Combined weights	Tertiary indicators	Weight	Combined weights
Condition Guarantee (A1)	0.329	Funding guarantee B1	0.75/0.2468	Budget C1	0.667	0.165
		Funding guarantee B1	0.75/0.2468	Expenses C2	0.333	0.082
		Institutional guarantees B2	0.25/0.0822	Management system C3	0.75	0.062
		Institutional guarantees B2	0.25/0.0822	Strategic planning C4	0.25	0.021
Construction and Management (A2)	0.142	Resources building B3	0.2/0.0284	Diversified archive resources C5	0.141	0.004
				File naming prescriptive C6	0.263	0.007
				File format & quality standardization C7	0.141	0.004
				Archive metadata integrity C8	0.455	0.013
		Security management B4	0.8/0.1136	Physical device security C9	0.2	0.023
				Cyber-security C10	0.2	0.023
				Operating system & platform security C11	0.2	0.023
				Hacking and virus prevention C12	0.2	0.023
				Database security C13	0.2	0.023
Quality Assurance (A3)	0.329	System function B5	0.5/0.1644	File Reception C14	0.098	0.016
				File management C15	0.259	0.043
				Archival preservation C16	0.339	0.056
				Archival Exploitation C17	0.066	0.011
				System management features C18	0.098	0.016
				Personalization features C19	0.140	0.023
		System performance B6	0.25/0.0823	Reliability C20	0.377	0.031
				Scalability C21	0.172	0.014
				Openness C22	0.112	0.009
				Convenience C23	0.260	0.021
				Advancement C24	0.080	0.007
		Quality of service B7	0.25/0.0823	File utilization C25	0.142	0.012
				Retrieval quality C26	0.334	0.027
				Retrieval speed C27	0.525	0.043
Sustainable Development (A4)	0.200	Management performance B8	0.333/0.0666	Plan extent to achieved goals C28	0.667	0.044
		Management performance B8	0.333/0.0666	Management innovation C29	0.333	0.022
		Service performance B9	0.667/0.1334	External personnel satisfaction C30	0.313	0.042
				Staff satisfaction C31	0.124	0.017
				Economic benefits C32	0.124	0.017
				Social benefits C33	0.439	0.059

According to the data shown in Table 6, it has evidenced that Condition Guarantee (A1) with the weight (0.329) and Quality Assurance (A3) with the weight (0.329) having the most significance relation from all four primary indicators (A1, A2, A3, and A4). This means that the experts believe that the condition guarantee and Quality Assurance are the most important factor in the evaluation of the electronic archives management system. While from the combine weight of the all nine secondary indicators, funding guarantee (B1) with the weight of (0.2468) is on top which expressed its highest influence as compare to other eight indicators at second level. From the combine weight of all nine secondary indicators, funding guarantee (B1) with combine weight (0.2468), security management (B4) with combine weight (0.1136), system function (B5) with combine weight (0.1644), and service performance (B9) with combine weight (0.1334) have emerged as the top four indicators. This observation suggests that these four indicators exert the most significant influence on electronic archives management. Out of 33 tertiary indicators, the five that carry the highest significance are budget (C1) with weight (0.667) & combine weight (0.165), expenditure (C2) with weight (0.333) & combine weight (0.082), management system (C3) with weight (0.75) & combine weight (0.062), social benefits (C33) with weight (0.439) & combine weight (0.059), and archive preservation C16 with weight (0.339) & combine weight (0.056). Among the various indicators, it is noteworthy that the two third-level indicators pertaining to the condition guarantee of the secondary indicators rank among the top five. This observation suggests that the condition guarantee has the most significant influence on the EAMS. The above analysis identified that the administration of electronic archives is most significantly influenced by the four second level and five third level factors. Therefore, the guarantee of advantageous conditions greatly impacts the electronic archives management system. Consequently, in perspective of China has already significant increase in its investment to electronic archives management systems by focusing on influencing indicators to improve the efficiency and effectiveness.

To ensure its reliability and validity, this article conducts reliability and validity tests on the index system established based on the analytic hierarchy process. Reliability testing and validity testing of the evaluation index system can help improve the accuracy and reliability of measurement results. This article adds this reliability test and validity test method to the article. SPSS software was used to conduct a reliability test on the questionnaire results. The Cronbach’s alpha coefficients of the three-level indicators all met the requirements of higher than .7. In summary, it can be concluded that the internal consistency of the questionnaire is good. SPSS software was used to conduct validity testing. The KMO index of the importance of the three-level indicators of the questionnaire was 0.604, and the approximate chi-square of Bartlett’s sphericity test was 64.974, p < .05. The data obtained from the questionnaire met the requirements of factor analysis.

Discussion

In light of the current requirements for the development of the informatization and standardization of archives management systems, as well as the ongoing global trend toward harmonized standards for electronic archives management systems, there is a need to enhance the establishment of a standardized system for electronic archives management. This includes the development of an evaluation standard system and the utilization of qualitative and quantitative research methods for assessing the effectiveness of electronic archives management systems in the future.

(1) The study has expanded the scope of research on the utilization of electronic archives management systems in archives, enterprises, institutions, and grassroots components in China. Additionally, it seeks to empirically test the evaluation index system proposed in this paper. By doing so, the study aims has gathered more comprehensive data and results. Furthermore, it intends to conduct statistical analysis to compare and summaries the variations in EAMS across different units, units of varying scales, and units with different business types. Based on the findings, the study will make necessary adjustments to the evaluation index system and modify the weightage assigned to basic indicators. The assessment index system for EAMS with universal applicability has been derived based on the prevailing circumstances.

(2) In order to enhance the comprehensiveness of research on the subjects pertaining to the electronic archives management system, it is recommended to incorporate a wider array of perspectives from experts in the field of archives management, as well as from various user departments, system users, and system suppliers. By gathering relevant suggestions and opinions, this study founded that to refine and modify the evaluation index system, thereby ensuring its wider recognition and alignment with the requirements of the current policy and technological advancements in electronic archive management.

(3) This study intentions to improve the depth of qualitative and quantitative research in evaluating electronic archives management systems. It seeks to gain a deeper understanding and examine the influence of each key evaluation index on the assessment of electronic archives management systems. Additionally, it aims to determine the comprehensiveness of the constructed evaluation index system in evaluating the successful implementation of specific electronic archives management systems, as well as its effectiveness in providing guidance for electronic archives management.

(4) In future research, the author will continue to conduct in-depth theoretical research by combining theories such as improving Delone and McLean information system success models to establish and improve the evaluation index system.

(5) When improving the evaluation indicators, the Delphi method is mainly used to collect expert opinions. In this process, the number of experts selected is not enough. In future research, the relevant subjects of the electronic archives management system will be evaluated. Conduct a larger-scale survey, collect relevant suggestions and opinions, and further improve and adjust the evaluation index system of this article. At the same time, a specific electronic archives information system was selected as the research object to conduct empirical research to further understand and study the impact of each key evaluation index on the evaluation of EAMS, and the impact of the constructed evaluation index system on the success of various specific electronic archives management systems. Is the implementation evaluation comprehensive. Hence, we have specifically elaborated on the human, financial and technical difficulties that this system may face in the actual promotion process, such as funding shortages, lack of technical personnel, insufficient infrastructure and other practical problems, and proposed corresponding strategies and practical application guidance (Purcell, 2019).

Conclusion

It is concluded that the analytic hierarchy method to allocate weights and combined weights to the primary, secondary, and tertiary level indicators of the index system, following the development of a comprehensive evaluation index system for EAMS and the outcomes of Delphi expert consultation. The essential aspects for evaluating the secondary indicators of the EAMS are the cost guarantee, system function, service performance, and security management, as seen from the weight comparison. The EAMS encompasses various fields and the development of an assessment index system is a multifaceted research endeavor. Consequently, this study acknowledges certain shortcomings in the present evaluation index system. Through ongoing research and practical application, the evaluation index system for electronic archives management systems is expected to mature and play a crucial role in promoting the informatization and standardization of China’s electronic archives management system. This system will provide substantial support for the effective implementation of archival informatization construction.

Footnotes

Acknowledgements

We sincerely appreciate the assistance of Zhang Juan, director of Nanjing Urban Construction Archives in valuable advice in the construction process of index system framework.

ORCID iD

Majid Ali

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: School of Information Resource Management, Nanjing University financially supported this research.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Data Availability Statement

Data sharing not applicable to this article as no datasets were generated or analyzed during the current study.

References

Ball

(2023). Preserving computer-aided design. Retrieved February 10, 2023, from https://www.dpconline.org/advice/technology-watch-reports

Barbau

(2013). Toward a reference architecture for archival systems. Product Lifecycle Management for Society, 2013(1), 68–77.

Barros

D. B. S.

Ferrer

I. D.

Maia

C. M. D. S.

(2018). Audit of preservable digital repositories. Revista Ibero-Americana De Ciencia Da Informacao, 11(1), 300–313.

Beagrie

(2022, August 19). Preservation, trust and continuing access for e-journals. http://www.dpconline.org/advice/technology-watch-reports

Cain

(2002). Model requirements for the management of Electronic Records (MoReq): A critical evaluation. Records Management Journal, 12(1), 14–18.

CRL, OCLC. (2022). Trustworthy repositories audit & certification: Criteria and checklist. Retrieved December 8, 2022, from http://www.crl.edu/PDF/trac.pdf

Cui

(2022). Research on the status quo and countermeasures of archives information management. Lantai Inside and Outside, 26(26), 4–6.

Digital Preservation Coalition. (2023). DPC technology watch reports. Retrieved January 13, 2023, from http://www.dpconline.org/advice/technology-watch-reports.

Fresko

(2007). A very non-standard standard. Records Management Society Bulletin, 137(1), 6–7.

10.

Fresko

(2008). MoReq2 in detail. Records Management Society Bulletin, 144(1), 8–10.

11.

Fresko

(2009). MoReq2: A European contribution to the preservation of electronic records [Conference session]. DigCCurr 2009 Conference.

12.

Gao

Huang

(2017). Research on security assessment system of digital archives in cloud computing environment. Archives Research, 154(01), 77–83. https://doi.org/10.16065/j.cnki.issn1002-1620.2017.01.13

13.

Gao

(2019). Construction of performance evaluation index system of digital archive information system. Lantai World, 2019(4), 21–23.

14.

Hanger

Lupprian

K. E.

(2005). Archiving of the electronic documentation of administration: Selection criteria, long-running stability and authenticity. Zeitschrift Fur Bibliothekswesen Und Bibliographie, 52(3-4), 137–142.

15.

Hao

C. H.

X. M.

Bai

W. L.

(2017). Construction of the evaluation index system for national digital archives resource construction based on the archive pluralism. Archival Science Research, 154(1), 31–41.

16.

Henttonen

(2009). A comparison of MoReq and SÄHKE metadata and functional requirements. Records Management Journal, 19(1), 26–36.

17.

Ishizaka

Labib

(2011). Review of the main developments in the analytic hierarchy process. Expert Systems with Applications, 38(11), 14336–14345.

18.

Jiang

(2016). The role of digital archives evaluation and the construction of technology evaluation. Heilongjiang Science and Technology Information, 20(20), 168.

19.

Jiang

Zhang

(2015). A preliminary study on the service quality evaluation system of digital archive information utilization. Lantai World, 466(8), 38–39. https://kns.cnki.net/kcms2/article/abstract?v=3uoqIhG8C44YLTlOAiTRKibYlV5Vjs7ir5D84hng_y4D11vwp0rrtUyG95RuqVS2OgUTWT039MT5VAtuszZCX2qnkuE4CmWN&uniplatform=NZKPT

20.

(2020). Development trend and hot spot tracking analysis of domestic archives informatization research. Shanxi Archives, 2020(4), 180–188.

21.

Liu

L. H.

(2020). Security risk evaluation method of cloud digital archive system in colleges and universities based on fuzzy FMEA method. Journal of Inner Mongolia University, 51(2), 135–140. https://kns.cnki.net/kcms2/article/abstract?v=3uoqIhG8C44YLTlOAiTRKibYlV5Vjs7i8oRR1PAr7RxjuAJk4dHXogwEFiCOrtW03_I0qnMWGPXf2qqzBgfkZvFwWYqWubih&uniplatform=NZKPT

22.

Liu

Kong

(2013). Review of archival informatization research. Shanxi Archives, 2013(3), 61–63.

23.

(2011). Analysis on service performance evaluation of digital archives. Beijing Archives, 245(5), 22–23. https://kns.cnki.net/kcms2/article/abstract?v=3uoqIhG8C44YLTlOAiTRKgchrJ08w1e7tvjWANqNvp-xHdVLu-kSggSLD7sbxUhqUF5btj50dlTiMdu6kyGyTJiUrEZnEbnr&uniplatform=NZKPT

24.

D. C.

Jin

(2013). On the management of digital archive ecosystem. Archival Management, 2013(2), 13–15. https://kns.cnki.net/kcms2/article/abstract?v=3uoqIhG8C44YLTlOAiTRKgchrJ08w1e7xAZywCwkEELkQVjzNN_VMwzM76hxL79O_Bz7sLQ6L7kzR-qx–F7VHDRHPuSz0z2&uniplatform=NZKPT

25.

D. C.

Jin

(2019). Analysis on sustainable development of digital archive ecosystem. Archives and Construction, 2019(7), 19–22.

26.

Okoli

Pawlowski

S. D.

(2004). The Delphi method as a research tool: An example, design considerations and applications. Information Management, 42(1), 15–29.

27.

Oliver

Foscarini

(2014). Records management and information culture: Tackling the people problem. Facet Publishing.

28.

Ouyang

H. F.

(2019). Analysis of functional evaluation index of digital archive application system. Shanxi Archives, 2019(4), 114–118.

29.

Poveda

E. B.

De Giusti

Alarcon

C. M.

(2021). Long-term digital preservation and the basis for strategic planning. 3C TIC, 10(3), 17–39.

30.

Prom

C. J.

(2023, February 10). Preserving email. https://www.dpconline.org/docs/dpc-technology-watch-publications/technology-watch-reports-1/739-dpctw11-01-pdf/file

31.

Purcell

A. D.

(Ed.) (2019). The digital archives handbook: A guide to creation, management, and preservation. Rowman & Littlefield.

32.

Richard

(2022, August 19). Preserving moving pictures and sound. http://www.dpconline.org/advice/technology-watch-reports

33.

Saaty

T. L.

(2008). Decision making with the analytic hierarchy process. International Journal of Services Sciences, 1(1), 83–98.

34.

Saaty

T. L.

Vargas

L. G.

(2012). Models, methods, concepts & applications of the analytic hierarchy process (Vol. 175). Springer Science & Business Media.

35.

Shi

Y. J.

(2011). Research on evaluation index system of digital archives based on user perspective. Shanghai Jiao Tong University. https://kns.cnki.net/kcms2/article/abstract?v=3uoqIhG8C44YLTlOAiTRKibYlV5Vjs7iAEhECQAQ9aTiC5BjCgn0RmaeqwnigC2s6mNaMQB70AG4z3JOl5Fbqrzm1jZsQm0X&uniplatform=NZKPT

36.

SOCC. (2022). 14th Five-year plan national archives development plan. http://www.saac.gov.cn/daj/toutiao/202106/ecca2de5bce44a0eb55c890762868683.html

37.

State Office of the CPC Central Committee. (n.d). 14th Five-year plan for the Development of National Archives. Retrieved June 12, 2022, from http://www.saac.gov.cn/daj/toutiao/202106/ecca2de5bce44a0eb55c890762868683.html

38.

Sundqvist

Svärd

(2016). Information culture and records management: A suitable match? Conceptualizations of information culture and their application on records management. International Journal of Information Management, 36(1), 9–15.

39.

Sun

Liu

K. F.

(2018). Thinking on introducing maturity model into the evaluation of digital archives. Chinese Archives, 2018(8), 68–69. https://kns.cnki.net/kcms2/article/abstract?v=3uoqIhG8C44YLTlOAiTRKibYlV5Vjs7i0-kJR0HYBJ80QN9L51zrP4uRgU01NAUAVqlZtKN6xe3wc-EzW6KygEZ7djOb7nk6&uniplatform=NZKPT

40.

The Library of Congress [LC]. (2022). Program background. Retrieved August 13, 2022, from http://www.digitalpreservation.gov/about/background.html.

41.

Weisinger

(2023, February 10). Records management: Progress for Europe’s MoReq2010. Retrieved from https://formtek.com/blog/records-management-progress-for-europes-moreq2010

42.

Wilhelm

(2009). An evaluation of MoReq2 in the context of national EDRMS standard developments in the UK and Europe. Records Management Journal, 19(2), 117–134.

43.

Wilhelm

(2002). EDRMS Standards-a critical evaluation of the benefits of Superseding National Standards with European Models Focusing on TNA 2002 Replacement by MoReq2. https://cdn.ymaws.com/irms.org.uk/resource/resmgr/moreq/papers/anonymized_edrms_standards_m.pdf

44.

Wright

(2022, August 19). Preserving moving pictures and sound. http://www.dpconline.org/advice/technology-watch-reports

45.

(2022). Thoughts on the future development of digital archives management. Inside and Outside Lantai, 2022(26), 4–6.

46.

Xue

(2019). Research on the construction of index evaluation system of digital archives [Doctoral dissertation, Heilongjiang University].

47.

Zhang

(2013). Introduction of the EU “File System Modular Requirements System”. Lantai World, 397(11), 21–22. https://kns.cnki.net/kcms2/article/abstract?v=3uoqIhG8C44YLTlOAiTRKgchrJ08w1e7xAZywCwkEEKXgIl6PLLDa6pgeEk3dh6bSQ8WIBGGCCn9uKSxSIDd6ti4twGlRpj1&uniplatform=NZKPT

48.

Zhao

M. C.

(2013). Design and implementation of digital archive system [Doctoral dissertation, Shaanxi Normal University].

49.

Zheng

(2020). Implementation and evaluation of HF company file information system based on CMM. Heilongjiang University.

50.

Zhou

L. X.

(2005). On the construction of evaluation index system of digital archives. Hubei Archives, 2005(4), 10–12.

51.

Zhu

(2023). Analysis on the status quo and future development path of intelligent archives management construction. Lantai Inside and Outside, 20(4), 25–27.