Study on Net Assessment of Trustworthy Evidence in Teleoperation System for Interplanetary Transportation

1. Introduction

Interplanetary transportation system is a general term for the transportation system that transports various payloads between different planets. It is also the foundation to develop space resources, ensure the security of space assets and maintain national security [1]. And the teleoperation system is indispensable during the rover travels over the surface of a planet, which has a vital role in the deep space mission. The teleoperation system consists of four modules: 2D data display, 3D data display, manipulation of geometric elements and communication between configuration items. It mainly includes four important capabilities: reconstruction of lunar terrain, vision-based navigation and localization, path planning for safe movement and verification planning of instruction

In the information age, many critical and complex functions of interplanetary transportation system are implemented by software. Along with the growth of system size and complexity, some related information processing software often doesn’t work in unexpected way. There are a lot of aerospace accidents caused by software-related errors. For example, the European Space Agency’s Ariane-5 launcher ended in failure in 1996. About 40 seconds after initiation of the flight sequence at an altitude of 2,700 m, the launcher veered off its flight path, broke up, and exploded; the NASA’s “Mars Climate Orbiter” was lost when it entered the Martian atmosphere in a lower than expected trajectory in 1999; the NASA’s “Mars Polar Lander” was lost after performing the landing process in 1999, the “Titan/Centaur” military satellite was launched at an incorrect and unusable low elliptical orbit instead of the intended geosynchronous orbit in 1999; because of loss of roll axis control and yaw control resulted by an incorrect roll rate filter constant zeroed the roll rate data, the Russian “Alliance-TMA1” manned spacecraft deviated from the scheduled landing site about 460 km in 2003; the NASA’s “Demonstration of Autonomous Rendezvous Technology (DART)” satellite detected that its propellant supply was depleted ahead of schedule, etc. [2]

At present, qualitative methods such as holding project review meetings or software testing have been unable to provide enough evidences for whether the system is trustworthy. Therefore, the question that whether aerospace information systems have trustworthy quality is plaguing people increasingly. In this paper, the “net assessment” theory is introduced into the field of aerospace information systems for the first time to solve this problem. Net assessment was first adopted by the US military, due to its good performance on comparative analysis, diagnostic and prospective evaluation for multi-disciplinary. The US Department of Defense (DOD) has a deeply comprehension on the definition, core content and architecture of net assessment, which was generated from decades of practical application. For example, the DOD submitted military strength net assessment report on its competitors regularly, such as the former Soviet Union and China.

Following the net assessment theory, a net assessment model based on the evidence formed by the data in the development process is established and the net assessment method (NAM) is proposed to assess system quality quantitatively. The analysis and experimental results show that the net assessment model and method can assess the information system objectively and comprehensively.

2. Information System Quality Model

2.1 The Definition of Trustworthy Information System

According to the basic trustworthiness mechanism, trustworthiness is the trust relationship between subject and object, and it is a concentrated expression of many attributes. Generally, it is believed that “trustworthy” means the object behavior always conforms to subject expectation in the process of achieving the given goal, which emphasizes the predictability of the object behavior.

From the users’ perspective, the definition of trustworthy emphasizes whether the system is worthy to be trusted and what it expresses is “trustworthy”. From the system’s perspective, the definition focuses on the objective ability of the system, which emphasizes the abilities that system should have in order to get the trust from users and what it expresses is “trustworthiness” [3]. In other words, “trustworthy” is the user’s subjective recognition for system objective quality, but “trustworthiness” is the objective quality of system. It is the ability of the system to meet users’ expectation, and it is the concentrated expression of several features such as availability, reliability, security, correctness, integrity, etc. [4]. Therefore, there are two definitions for the trust relationship of information system: subjective and objective definition

Subjective definition: The definition of America’s National Science and Technology Commission (NSTC) is “high confidence”, which is a measure for the predictability of system behavior, emphasizing the system behavior is predictable [5]. The definition of Microsoft is “trustworthy”, and it means that customers can always rely on the system. Trustworthy systems are available, reliable and secure just like the power and water services, which emphasizes that the behaviors of system are trustworthy [6]. The definition of Trusted Computing Group (TCG) is “Trusted”, which means if the system behaviors can follow the desired approach fully when achieving a given goal, the system is trusted. This definition emphasizes the compliance between system behavior and the goal [7]. The definition of ISO/IEC25010 is “dependability”, and it represents the ability that provides designated services to users and ensures them trust that they can trust the services provided by system, which emphasizes the system behavior is dependable [8].

Objective definition: The McCall model defines software quality from three aspects—product revision (ability to change), product transition (adaptability to new environments), and product operations (basic operational characteristics)—and there are 11 features including maintainability, testability, flexibility, portability, reusability, interoperability, correctness, reliability, efficiency, usability, and integrity distributed in the above aspects [9]. Boehm model defines the quality of software as a layered model. It includes 15 features: device independence, self-contentedness, accuracy, completeness, robustness integrity, consistency, accountability, device efficiency, accessibility, communicativeness, self-descriptiveness, structuredness, conciseness, legibility, and augmentability [9]. The definition of ISO/IEC15408 is “security”, and the behavior of a trustworthy component, operation, or process is predictable in any conditions and it can resistant to damage caused by application software, viruses, and certain physical interference excellently. Its features include availability, reliability, safety, robustness, testability, and maintainability [10]. ISO/IEC9126 quality model consists of 6 major features and 27 sub-features. The 6 major features are functionality, reliability, usability, efficiency, maintainability and portability [11]. The TRUSTIE model is proposed by the “High Trusted Software Production Tools and Integrated Environment” research group funded by China’s National High Technology Research and Development Program. It also consists of 6 major features and 27 sub-features. The 6 major features are availability, reliability, security, real-time, maintainability and survivability [12].

In summary, the definition of system trustworthiness is “trustworthy” in this paper. It is the confidence level that allows users to predict the system behaviors in any conditions.

2.2 Process Oriented Model

With the growth of system size and complexity, whether the system is trustworthy has become an international issue. Since the first Trusted Computer System Evaluation Criteria (TCSEC) was set by the US DOD in 1983, a lot of countries and international organizations have developed their own research projects about trustworthy software. The America’s National Software Development Strategy (2006–2015) places the high-reliability software in the most important position. And the Network and Information Technology Research and Development Program (NI-TRD) invests about $134 million in the field of “Highly Trusted Software and Systems” per year. In 2006, Europe launched the “Open Trustworthy Computing” project participated by 23 scientific institutions to study the trustworthy mechanism of software. In 2007, the “Major Research Plan of Trustworthy Software” was set up by the National Natural Science Foundation of China to promote the trustworthy software research. At the same time, the National Program on Key Basic Research Project of China (973 program) took “the common theory and construction method of safety-critical software system” as a major funding direction. It holds that trustworthy safety-critical software is the deeply integrated of software, hardware, systems, physical world, people and other factors [13].

The quality is built in processes, so the trustworthy evidences for system quality are also in the process. A series of theories have been formed based on software development process to measure the system quality. Moreover, some international and national standards have been established, mainly including the Software Life Cycle Processes Standard (ISO/IEC 12207), Software Process Improvement and Capability Determination Standard (ISO/IEC 15504), Capability Maturity Model Integration (CMMI) and Capability Maturity Model for Military Software Development of China (GJB5000A). ISO/IEC 12207 [14] consists of seventeen sub-processes including acquisition, supply, development, operation, maintenance, documentation, configuration management, quality assurance, verification, validation, joint review, audit, problem resolution, management, infrastructure, improvement and training. ISO/IEC 15504 [15] divides the software capability into 6 levels: incomplete, performed, managed, established, predictable, and optimizing. It includes 29 sub-processes: software acquisition, customer demand management, software provision, operating software, customer service, system requirements and design, software requirements analysis, development software design, software design implementation, integration and debugging software, integration and debugging systems, systems and software maintenance, documentation, configuration management process, quality assurance process, verification product, confirmation product, joint review, review, problem solving, project management, quality management, risk management, subcontract management, business planning, definition process, process improvement, provision of skilled personnel and provide software engineering architecture. CMMI [16] divides software capability maturity into 5 levels including initial level, managed level, defined level, quantitative management level, and continuous optimization level. It includes 22 process areas such as requirement management, project planning, project monitoring and control, supplier agreement management, measurement and analysis, process and product quality assurance, configuration management, requirement development, technology solution, product integration, verification, validation, organizational process focus, organizational process definition, organizational training, integrated project management, risk management, decision analysis and resolution, organizational process performance, quantitative project management, organizational performance management, and causal analysis and resolution. China’s GJB5000A-2008 is consistent with CMMI in the division of software capability maturity and process domain definition basically, so, no more details here. The trustworthy evidences of the multi-dimensional net assessment model proposed in this paper are extracted from the above standards/models mainly. In addition, some evidences that are not included in the traditional system models are added.

3. Net Assessment

3.1 Theory of Net Assessment

According to the definition of “attribute+differentia”, the attribute of the net assessment is assessment and the differentia is “net”. Net assessment is a comparative analysis between two or more related parties during the long-term competition or interaction process in a specific field [17]. In 1971, the US President Nixon signed a memorandum to order the establishment of net assessment agency. The following year, the US National Security Council officially established the net assessment team, and then the Assistant Secretary of Defense’s Office established a net assessment department. After that, the head of the Net Assessment Team of the National Security Council named Andrew Marshall transferred to the director of the Department of Defense’s Net Assessment Office. At the beginning, the net assessment is aimed at military analysis between different countries. Then, it continued to expand and covered the cases such as technical, political, economic, social, and ideological factors. Especially, the net assessment of economic and the net assessment of enterprise are derived from the net assessment of national security strategy [18]. Currently, a lot of countries such as Japan, India, Australia, and Israel are developing and applying net assessment. Net assessment emphasizes the strategic, comprehensive, and comprehensive of assessment. The key of net assessment is not only to assess the current situation, but more important to anticipate potential long-term trends. The NAM was extensively used by the RAND Corporation, and with multi-disciplinary integrated and multi-analysis tool adopted, its masterpiece consists of “The Straits of Terror”, “China and India in 2025”, and so on [19], which have great popularity and influence for the whole world. However, the research on NAM was not popular during the last decades in China, and the majority of papers published just focus on the introductory aspect, lacking of practical analysis. Lin [20] introduced application cases of net assessment in military field, with experimental mechanism of modeling proposed by “The Straits of Terror” covered. Yan [21] presented the detail definition, analytical framework and implementation steps of net assessment method introduced by the DOD. Based on SWOTCLPV matrix analysis mode, Yi and Li [22] provided the flow chart of net assessment and statement axis from the point view of scene analysis and model simulation.

This paper introduces the net assessment theory into the field of information system for the first time. The system multi-attribute decision-making (MADM) problem is abstracted into the comparison problem of two or more related parties, and the information system trustworthiness assessment problem is divided to a multi-attribute decision problem that can be quantified directly. Through the comparison of the trustworthy evidences at the same level, the weights of each evidence class can be determined. The above process is the net assessment of the information system. Its purpose is to take all factors related to system quality as much as possible into consideration to ensure the comprehensiveness and rationality of net assessment.

3.2 Net Assessment Model

Following the principles of net assessment, a multi-dimensional net assessment model taking ISO standards, national military standards and industry standards as trustworthy evidence source is established in this paper.

Then, the weight of each evidence element could be calculated using the NAM to obtain the quantitative result that whether the system is trustworthy. As shown in Fig. 1, the multi-dimensional net assessment model is a layered model (it is not always three layers), and its leaf nodes can be quantified directly. In order to reduce the people’s interference factors, the evidence sub-classes must be divided until the type of the leaf node evidence element is “BOOL”.

The trustworthiness of the system can be defined as Eq. (1).

According to the total quality management theory, there are 5 elements of “People, Machine, Material, Law, and Environment” that affect system quality directly. So the net assessment model proposed in this paper includes five aspects. The first is the organization, such as the measurement of organizational maturity, personnel ability, quality awareness, synergy efficiency, etc. The second is about the system environment, such as the social environment (international situation, laws and regulations and national policies), physical environment, hardware, development support environment, etc. The third is about the quality management, such as risk management, document management, configuration management, etc. The fourth is about the measurement of software ontology based on the coding rules for the source code. The fifth is about the software process documentation, such as requirements documents, design documents, coded documents, etc. Table 1 shows the above correspondence. Due to the large number of evidence elements, there lists a total of 40 evidence subclasses only. Some representative evidence subclasses and their elements will be discussed in more detail later.

Taking the development support environment subclass of environmental evidence class as example, the development support environment includes 5 Boolean evidence elements: operating system, integrated development environment, version control tool, third-party library, and compiler, as shown in Table 2

Taking the configuration management subclass of management evidence class as example, it contains 20 Boolean evidence elements, as shown in Table 3

Net Assessment Method

The quantification of the elements in net assessment model belongs to the MADM problem, which mainly solves the multi-attribute/element ranking problem. A quantitative trustworthiness of software not only allows users to choose software products, but also helps developers to develop higher quality software. But, there is no mature assessment method that can get the quantitative trustworthiness accepted by most people now

At present, the method of MADM can be divided into two categories: subjective weighting method and objective weighting method. The subjective weighting method mainly includes expert investigation method, analytic hierarchy process, binomial coefficient method and comparison matrix method, etc. The objective weighting method mainly includes principal component analysis method, entropy right method, deviation maximization method and mean square deviation method, etc. The subjective weighting method is more mature. Its weights are determined by the expert experience. So it is subjective and arbitrary. The objective weighting method calculates the weight of the indicator following the specific algorithm based on the original data. But it ignores the subjective cognition of the decision maker. So the weight of the indicator often differs from the actual importance of the indicator. In order to overcome the above difficulties, this paper adopts the combination of subjective and objective weighting method, and combines fuzzy analytic hierarchy process (FAHP) and entropy weight method (EWM) to form NAM [24,25].

The four steps of the NAM are as follows.

Step 1. The establishment of fuzzy complementary judgment matrix.

Each element would be compared with others in the fuzzy analytic hierarchy process. The 0.1–0.9 scale method is shown in Table 4. This paper uses the 0.1–0.9 scale method to construct the fuzzy complementary judgment matrix as Eq. (2) [25].

According to Table 4, the factors will be compared with each other to obtain the fuzzy judgment matrix A.

If matrix A meets the following condition:

Then, the fuzzy judgment matrix A is called the fuzzy complementary judgment matrix.

Step 2. The solution of fuzzy consistency matrix.

DEFINITION 1. If the fuzzy complementary judgment matrix R meets the following condition:

Then, the fuzzy matrix R is a fuzzy consistency matrix.

THEOREM 1. Sum the fuzzy judgment matrix A by the line, the formula is Eq. (5).

And, the matrix element r_ij can be calculated as Eq. (6).

Step 3. Eliminate cognitive blindness.

DEFINITION 2. The fuzzy consistency matrixes given by the 5 experts are A, B, C, D, and E as Eq. (7).

Assuming the five experts have the same understanding of the elements in the net assessment model, the average awareness matrix can be defined as Eq. (8).

Define the uncertainty of the expert on an element is “Cognitive Blindness”, the “Cognitive Blindness” matrix is recorded as Q.

Define the total recognition matrix of five experts on an element is X, it can be calculated as Eq (10).

Step 4. Process of normalized.

Defining the weight vector [TeX:] $$W=\left(w_{1}, w_{2}, \dots, w_{n}\right)$$，it can be calculated as Eq. (11).

The symbol n is the number of rows in the matrix.

Then vector W is called the weight vector of the elements in NAM.

Compared with the EWM, the advantage of NAM is consistency. To ensure the credibility of the judgment matrix, the fuzzy consistent matrix is calculated based on the fuzzy complementary judgment matrix established in step 1, which avoids the inconsistency between different experts, such as the case A>B, B>C, C>A. Compared with the FAHP, its advantage is the convergence in the quantitative conversion process. The fuzzy complementary judgment matrix is established using the FAHP. And then the experts' “Cognitive Blindness” in the establishment stage of the fuzzy judgment matrix is eliminated in step 3, which makes the experts’ opinion is convergent. The solution of “Cognitive Blindness” Q avoids the “Noise Data” in the FAHP, and ensures the consistent convergence of results.

4. Case Analysis

The interplanetary transportation teleoperation system is the regular application software system deployed in the Beijing Aerospace Flight Control Center for each lunar and deep exploration mission. It belongs to the ground application system in the whole lunar exploration project. It provides a three-dimensional information display and operation platform for the configuration items such as terrain construction, visual positioning, mission planning, activity organization planning, planning verification, etc. Under the data interaction and processing support of the related configuration items, users can complete the image data monitoring and judging of the lunar detector, the path planning of the rover, the task planning key space point selection, teleoperation planning verification and other business work through the teleoperation system. At the same time, the parameters of each configuration item can be changed and set, and the running status information can be monitored in real time. Figs. 2–4 provide several control interfaces of teleoperation system in China’s CE-3 mission.

This paper takes the teleoperation system as the net assessment case, and takes China’s Quality Management Systems Requirements standard (GJB9001B-2009) certification audit work as the verification means to illustrate the feasibility of the net assessment model and the practicability of the NAM. There are 5 experts who are invited to weight the trustworthy evidences of the teleoperation system according to the net assessment model, and the NAM is used to solve the weight of the trustworthy evidences. Due to the limited space, the specific calculation process is not given in the paper. Firstly, the five experts used the 0.1–0.9 scale method to assign value to the evidence classes. There are 5 classes including organization, environment, management, code and documentation as shown in Table 1. The fuzzy judgment matrices given by the 5 experts are A1, A2, A3, A4, and A5 (Table 5). The cognitive blindness matrix and the weight vector are calculated using the NAM introduced in Section 3.3 (Table 6).

4.1 Environmental Evidence Class

Most traditional information system quality model contains the trustworthy evidences such as documents, codes and management. Therefore, this paper mainly introduces the evidence classes that the net assessment model is different from the traditional quality model. There are four evidence subclasses: social environment, physical environment, hardware environment, and development support environment in the environmental evidence class shown in Table 1. The fuzzy judgment matrices given by the 5 experts are A1, A2, A3, A4, and A5 (Table 7). The cognitive blindness matrix and weight vector are calculated using the NAM introduced in Section 3.3 (Table 8).

There are five evidence elements including operating system, integrated development environment, version control tool, third-party library, and compiler in the subclass of development support evidence shown in Table 2. The fuzzy judgment matrices given by the 5 experts are A1, A2, A3, A4, and A5 (Table 9). The cognitive blindness matrix and weight vector are calculated using the NAM introduced in Section 3.3 (Table 10).

According to the actual situation of the teleoperation system, the BOOL evidence elements are assigned value through fuzzy judgment matrix. The five evidence elements of operating system, integrated development environment, version control tool, third-party library, and compiler in the subclass of development support evidence are all “false”. According to formula (1), the loss to the trust value is 0.0552 = (1*0.2158+1*0.1974+1*0.1767+1*0.1884+1*0.2058)*0.2719*0.2031.

4.2 Management evidence class

Taking the management evidence class as example, there are eleven subclasses including Project management, Development process management, Configuration management, Document management, Resource management, Risk management, Quality management, Agreement management, Acceptance management, Stereotype management, and Contractor management as shown in Table 1. Due to the limited space, this paper only gives the judgment matrix of 2 experts, as shown in Table 11. The weight vector W = (0.0969 0.0974 0.0869 0.0929 0.0904 0.0979 0.0879 0.0899 0.0929 0.0839 0.0829).

As shown in Table 3, the judgment matrix of configuration management evidence elements can be constructed through the experts’ comparison. Due to the limited page, the cognitive blindness matrix is not given, and the fuzzy judgment matrix is calculated using the NAM introduced in Section 3.3 (Table 12). The weight vector W = (0.0340 0.0338 0.0332 0.0337 0.0330 0.0337 0.0339 0.0338 0.0334 0.0331 0.0332 0.0329 0.0333 0.0342 0.0333 0.0331 0.0341 0.0334 0.0343 0.0330).

According to the actual situation of the teleoperation system, the evidence elements are assigned value through fuzzy judgment matrix. The 20 evidence elements in the subclass of configuration management evidence are all “True”. According to formula (1), its trust value is (0.0340*1+0.0338*1+0.0332*1+ 0.0337*1+0.0330*1+0.0337*1+0.0339*1+0.0338*1+0.0334*1+0.0331*1+0.0332*1+0.0329*1+0.0333*1+0.0342*1+0.0333*1+0.0331*1+0.0341*1+0.0334*1+0.0343*1+0.0330)*0.0904*0.1868 = 0.0169.

The trust value of the teleoperation system can be got by assigning 0|1 to each evidence element. That is the “net result”. According to the China’s Military Software Product Evaluation standard (GJB2434A-2004), the interplanetary transportation teleoperation system is the A-level software. In GJB2434A-2004, if the trust value is greater than 0.9, it means that the system is trustworthy. The untrustworthy factor in the project process can be found based on the “net result”, which will make further improvement on the system quality. Due to the limited space, it is inconvenient to give all judgment matrixes and weight vectors. Finally, the final trust value of the interplanetary transportation teleoperation systems is 0.9154. So, the teleoperation system is trustworthy.

4.3 Comparisons with FAHP and EWM

By comparison, it is proved that NAM proposed in this paper is more accurate than FAHP and EWM. The matrix E can be solved based on the weight vector W using Eq. (12).

Then the correlation coefficient r between the matrix E corresponding to the three methods and the fuzzy judgment matrix given by the 5 experts can be calculated by the MATLAB command: corrcoef(A,E). A is the fuzzy judgment matrix given by expert. E is the matrix solved based on the weight vector W. Taking the five fuzzy judgment matrixes in Table 5 as testing data, the weight vector W obtained by the three methods and Correlation Coefficient Matrix (CCM) calculated in MATLAB are shown in Table 13.

It illustrates that the weight vector calculated by the NAM has higher correlation with the fuzzy judgment matrix given by 5 experts in Table 13. So it can reflect the expert’s intention better. Therefore, the quantitative result obtained by the NAM is more accurate and reasonable.

5. Conclusions

Acknowledgement

Interplanetary transportation mission teleoperation system plays a vital role in the whole lunar exploration mission and its trustworthy evidence must be assessed quantitatively before and during its implementation. In China’s CE-3 and CE-4 mission, the availability and practicability of the net assessment model and method proposed in this paper have been verified. In addition, this research work achieves the Weapons and Equipment Quality Management System Certification successfully. The main innovations include the following two cases:

The net assessment has been applied in the field of national defense and it is a comprehensive assessment method for the military strength of the two countries or organizations. This paper applies the net assessment theory to the field of military information system engineering for the first time. Following the net assessment theory, this paper takes personnel capabilities, organizational maturity, social environment, physical environment and other factors into the net assessment model. The net assessment model is more comprehensive than other traditional quality models. So, the “net result” will be more accurate.

The NAM proposed in this paper combines the advantages of FAHP and EWM. Firstly, the fuzzy judgment matrix is used to ensure the fuzziness of the assessment method. Secondly, the fuzzy consistency matrix is solved to ensure the consistency of fuzzy matrix. Finally, the cognitive uncertainty is reduced by eliminating the “Cognitive Blindness”, which can make the weights vector more convergent.

However, according to the net assessment results in this paper and the development experience of teleoperation system for multiple deep space missions, there are four suggestions for the development of interplanetary transportation mission teleoperation system.

Cross-platform: The capability of deep space exploration is an important indicator of a country’s national defense strength. So the relevant information processing systems must be independently controllable. In order to ensure the trustworthy of national lunar exploration project, it is necessary to complete the design, development, testing and application of the teleoperation system of lunar exploration project on the operating system having independent intellectual property rights to achieve the independent of the software system in national aerospace engineering [26]. And it must be compatible with other major operating systems.

Open source: At present, the development environment of teleoperation system in the lunar exploration engineering build the integrated development environment based on the domestic NeoKylin operating system. All the related tools are open source. It consists of Qt creator, Qt 4.7.4, OpenSceneGraph (OSG), Cmake and XML. Qt is used to achieve the functions of system interface design, display 2D data and manipulation of geometric elements. OSG is used to realize display of 3D terrain, detector model, path planning and planning verification of patrol mechanism. The XML file is used for communication between different configuration items.

Formal proof: According to the GJB2434A-2004 in China, the teleoperation system should be proved using formal method. The formal proof is based on formal description. But the requirements and design documents of the teleoperation system are all written in natural language in fact. So, it is necessary to describe and verify them through formal method [27]. The Z notation is a requirement description language based on first-order predicate logic and set theory. It abstracts the pre/post conditions of state transitions based on the mathematical theories such as predicate, sets, sequences, packages, relations, functions, and classes. It forms the formal specification by describing the relationship that the input/output variables should be satisfied. Formal specification ensures the flexibility of the software structure extremely, so it minimizes the impact of requirement changes on the original system. In addition, the formal specification of the system can be reasoned and validated following the logic algorithm.

Net assessment of system lifecycle: The application scope of net assessment has covered military, environment, performance, economic, etc., and it has been applied and promoted by scholars in various fields. However, there is still no any research on the net assessment of information systems. The net assessment is more comprehensive, forward-looking and pure compared with the traditional evaluation methods. An intensive study of net assessment is of great significance for the construction of military information system assessment theory and the formulation of corresponding development process improvement measures. Therefore, the research on the net assessment of information systems must be paid attention by the relevant experts and organizations.

The work reported in this paper is carried out in the Institute of Complex Networks and Visualization at Shijiazhuang Tiedao University in China. The research is partially funded by the third batch of innovation teams and leading talents plan, funding number “JIZIBAN [2018]33” (Hebei, China).