A machine-learning-based hardware-Trojan detection approach for chips in the Internet of Things

Abstract

With the development of the Internet of Things, smart devices are widely used. Hardware security is one key issue in the security of the Internet of Things. As the core component of the hardware, the integrated circuit must be taken seriously with its security. The pre-silicon detection methods do not require gold chips, are not affected by process noise, and are suitable for the safe detection of a very large-scale integration. Therefore, more and more researchers are paying attention to the pre-silicon detection method. In this study, we propose a machine-learning-based hardware-Trojan detection method at the gate level. First, we put forward new Trojan-net features. After that, we use the scoring mechanism of the eXtreme Gradient Boosting to set up a new effective feature set of 49 out of 56 features. Finally, the hardware-Trojan classifier was trained and detected based on the new feature set by the eXtreme Gradient Boosting algorithm, respectively. The experimental results show that the proposed method can obtain 89.84% average Recall, 86.75% average F-measure, and 99.83% average Accuracy, which is the best detection result among existing machine-learning-based hardware-Trojan detection methods.

Introduction

The Internet of Things (IoT) is a huge network that combines everything with the Internet through a variety of information sensing devices, such as radio frequency identification devices, infrared sensors, and global positioning systems. The IoT is called the third wave of the world information industry after computers and the Internet. With the development of the IoT technology, its related applications have become more and more diversified, such as the Internet of Vehicles, ¹ smart medical, ² and smart home, ³ as shown in Figure 1, making our life more convenient and intelligent. However, a large amount of user sensitive information is stored and processed in a smart device. If it is exploited by an attacker, it may seriously damage or endanger the privacy of the user, and even threaten the life safety of users.

Figure 1.

The model of the IoT.

The logical architecture of the IoT is divided into the sensing layer, the transport layer, and the application layer. In the past few years, some researchers have conducted related research on different layers of security in the IoT logical architecture. For the security of the sensing layer, the previous literature ⁴ established a lightweight trusted execution environment in embedded systems to protect system security. For the security of the transport layer, the previous literature^5–7 defended against sensor network attacks by lightweighting cryptographic algorithms and protocols. For the security of the application layer, the previous literature^8,9 protected sensitive data in the application program by designing multiple access control models.

However, the hardware is the foundation of the IoT, and the core device of the hardware is the integrated circuits (ICs), or called chips. Obviously, the security of the ICs is the key issue of the IoTs’ security. Nowadays, as the amount of data processed increases, the equipment of each layer needs more complex chips to enhance the processing power. Therefore, the integration of IoT chips is getting higher and higher, while it is very difficult to ensure the security of the very large-scale integration (VLSI).

Hardware Trojans refer to a special module that is intentionally implanted in the ICs or a defective module that the designer has no intention of leaving during the IC design process. Under special conditions, the module can be exploited by an attacker to achieve destructive functions. The typical hardware Trojans consists of Trojan trigger and Trojan payload. Trojan trigger monitors the rare signals from normal circuits and converts the rare signals into the effective trigger signal to Trojan payload. Trojan payload performs specific functions of the hardware Trojans. Hardware Trojans can cause many serious risks which include, but are no limit to, leakage of information, alteration of the functionality of a device, and denial-of-service (DoS). Hardware Trojans are extremely concealed and normally do not work. They are usually only activated when triggered by rare signals. Therefore, the detection of hardware Trojans is very difficult.

Currently, most of the researches on hardware-Trojan detection methods are reverse engineering,^10,11 logic testing,^12–14 and side-channel signal analysis,^15–18 which called post-silicon detection methods. However, due to the difficulty of the acquisition of golden chips, influenced by noise easily, high cost, the requirement of special equipment, and other reasons, the post-silicon detection methods are not effective in practical applications. Especially in VLSI, the Trojan circuits are extremely smaller than normal circuits. Thus, it is very hard to find the hardware Trojans by the post-silicon detection methods. The pre-silicon detection methods do not have the limitation compared with the post-silicon detection methods, and these are the effective detection methods for very large-scale ICs.

In this study, we focus on hardware-Trojan detection using machine learning in gate-level netlists. Based on the 51 existing Trojan-net features, ¹⁹ we analyze the features of Trojan circuits and propose new hardware-Trojan features. In order to remove some features which are less relevant, we use the scoring mechanism of the eXtreme Gradient Boosting (XGBoost) to further extract the effective set of 49 Trojan-net features out of 56 features and use XGBoost to detect hardware Trojans for the first time.

We summarize the main contributions of this study as follows:

• By further analyzing the existing 51 Trojan circuit features, we propose 5 new features. In addition, the appropriate number of features is a prerequisite for effective training of classifiers, so we use the scoring mechanism of the XGBoost algorithm to extract 49 effective features out of 56 features.

• In the process of hardware-Trojan detection, the Trojan net should be detected as much as possible. Therefore, we aim to maximize Recall and use the XGBoost algorithm to train the hardware-Trojan classifier.

• The comparison experiment shows that the XGBoost algorithm is more suitable for hardware-Trojan detection, and also shows that our proposed features can further improve the hardware-Trojan detection effect. In the process of detecting the Trojan-free netlists, we can achieve perfect detection results without any mistakes. For the Trojan-inserted netlists, we also can obtain 89.84% average Recall, 86.75% average F-measure, and 99.83% average Accuracy.

The rest of this study is organized as follows. The necessary preliminaries are introduced in section “Preliminaries.” We give the detection model and the design goals in section “Problem formulation.” The introduction and selection of Trojan-net features is in section “Trojan-net features.” Then, we propose the detail of our method in section “The XGBoost-based hardware-Trojan detection method.” We give the experiments and analysis in section “Experiments and analysis.” Finally, we introduce the related work in section “Related work” and conclude our work in section “Conclusion.”

Preliminaries

In this section, we introduce the necessary preliminaries, including the hardware-Trojan features of the gate-level netlists and the XGBoost algorithm.

Gate-level netlists

The gate-level netlists are written in Verilog-HDL (hardware description language) to describe the connections between circuit components, which are shown in Netlist 1. The gate-level netlists start with module and end with endmodule. The name indicates the name of the modified module (Line 1). All input and output signals of the module are listed after the name. The input of the circuit module is defined in input, and the output of the circuit module is defined in output (Line 2–3). The wire defines the name of the net that connects the components inside the circuit (Line 4). Then, each circuit component will be described, including the circuit component function, name, input, and output (Line 5–8). For example, Line 5 indicates that the logic-gate is an AND gate. The input of the AND gate is signal $a$ and signal $b$ , and its output is signal $x$ .

Figure 2 lists the relevant gate-level netlists used in the experiments in this study, which are provided by the Trust-Hub. ²⁰ At the same time, according to the data information provided by the platform, we also counted the Trojan nets and normal nets in different netlists. It should be noted that in this study, we consider that the nets inside the Trojan circuits can represent Trojans. Therefore, the number of Trojan nets in the statistical process is counted according to the number of nets inside the Trojan circuits.

Figure 2.

Statistical results of the normal nets and the Trojan nets in the Trojan-included netlists.

XGBoost

The XGBoost is one of the supervised machine-learning methods. It optimizes the boosting algorithm based on the traditional gradient-boosted decision trees (GBDTs). Due to its fast speed, good classification results, ability to handle large-scale data, support for multiple languages, and so on, it is popular in domestic and foreign data competition platforms such as Kaggle, DataCastle, and Kesci.

Netlist 1 Schematic diagram of gate-level netlist.
1: module name ( a, b, c, cout);
2:  input a, b, c;
3:  output cout;
4:  wire x, y, z;
5:  and gate1 ( x, a, b);
6:  or gate2 ( y, a, c);
7:  and gate3 ( z, b, c);
8:  or gate4 ( cout, x, y, z);
9: endmodule

The XGBoost is made up of several classification and regression trees (CARTs). Since the value corresponding to the leaf node of CART is a score, not a certain class, the final predicted value of XGBoost is the sum of the predicted scores for each CART. The mathematical model of XGBoost is defined as

$$\hat{y}=\sum _{k=1}^K{f}_k\left(x_i\right),f_k\in F$$

(1)

where $\hat{y}$ corresponds to the prediction result of $x_i$ , $K$ is the number of trees, $F$ represents all possible CART trees, and $f$ represents a specific CART tree.

The objective function of XGBoost can be defined as

$$\mathrm{obj}\left(\theta \right)=\sum _{i=1}^{n}l(y_i,{\hat{y}}_i)+\sum _{k=1}^K\mathrm{\Omega }\left(f_k\right)$$

(2)

The first half of the objective function $\sum _{i=1}^{n}l(y_i,{\hat{y}}_i)$ represents the loss function of the model, and the second half $\sum _{k=1}^K\mathrm{\Omega }(f_k)$ represents the regularization term of the model. The regularization term represents the total complexity of the K trees. The optimization of the XGBoost model is mainly to optimize these two parts. XGBoost uses an incremental learning approach to solve the loss function instead of the traditional stochastic gradient descent algorithm. The method of incremental learning is not to directly optimize the entire objective function but to optimize it in steps. The first is to optimize the first tree, and then optimize the second tree on this basis, step by step until the k tree is optimized. So in step t, the objective function is expressed as

$$\begin{array}{cc}\multicolumn{1}{c}{\mathrm{obj}(\theta {)}^{(t)}}& =\sum _{i=1}^{n}l(y_i,{\hat{y}}_i^{(t)})+\sum _{i=1}^t\mathrm{\Omega }\left(f_i\right)\hfill \\ \multicolumn{1}{c}{}& =\sum _{i=1}^{n}l(y_i,{\hat{y}}_i^{(t-1)}+f_t(x_i))+\mathrm{\Omega }(f_t)+C\hfill \end{array}$$

(3)

where $C$ is the complexity of the former t – 1 trees and is a constant.

According to Taylor’s theory, the new objective function is shown as equation (4), where $g_i$ and $h_i$ are defined as

$$\begin{array}{cc}\multicolumn{1}{c}{\mathrm{obj}(\theta {)}^{(t)}}& \cong \sum _{i=1}^n\left[l(y_i,{\hat{y}}_i^{(t-1)}+g_i{f}_t(x_i)+\frac{1}{2}{h}_i{f}_t^2(x_i))\right]\hfill \\ \multicolumn{1}{c}{}& +\mathrm{\Omega }(f_t)+C\hfill \end{array}$$

(4)

$$g_i={\partial }_{{\hat{y}}_i^{(t-1)}}l(y_i,{\hat{y}}_i^{(t-1)})$$

(5)

$$h_i={\partial }_{{\hat{y}}_i^{(t-1)}}^{2}l(y_i,{\hat{y}}_i^{(t-1)})$$

(6)

The regularization term is very important to XGBoost, here we define the tree as

$$f_t\left(x\right)={\omega }_{q(x)},\omega \in R^T,q:R^d\to 1,2,3\dots ,L$$

(7)

where $T$ represents the number of leaf nodes of a tree, and the values of the $T$ leaf nodes constitute a T-dimensional vector $\omega$ . $q(x)$ is a map that maps samples to a value from 1 to $T$ . Based on equation (7), the regularization term is expressed as

$$\mathrm{\Omega }(f)=\gamma T+\frac{1}{2}\lambda \sum _{j=1}^T{\omega }_j^2$$

(8)

where $\gamma$ and $\lambda$ are defined by XGBoost. As shown in equation (8), the regularization term contains two parts, one is the total number of leaf nodes, and the other is the L2 regularization term obtained by the leaf node. From the perspective of the bias–variance tradeoff, the regularization term reduces the variance of the model and makes the learned model simpler.

Problem formulation

In this section, we describe the hardware-Trojan detection model based on machine learning. At the same time, we also introduced the design goals of this method proposed in this study.

Detection model

We use $L_t$ to represent the netlist, where $t$ is the number of nets in the netlist. In this study, we consider that the inside nets of the Trojan circuit are defined as effective Trojan nets, which are more representative. Then, we divide the nets in the netlist $L_t$ into normal net $N_0$ and Trojan net $N_1$ . We use $F(x_1,x_1,\dots x_n)$ to represent feature vector with n features. According to the Trojan-net features $F(x_1,x_1,\dots x_n)$ , we extract feature data for each net in the netlist. Therefore, we get the data set $D_{t\times n}$ corresponding to the netlist $L_t$ . The data set $S_{m\times n}$ is obtained by extracting feature data from several known netlists, where m is all nets of known netlists. Next, we divide the data set $S_{m\times n}$ into a training set $T_r$ and a testing set $T_e$ . After that, the classifier $C$ is trained by the training set $T_r$ and the testing set $T_e$ . Finally, the $D{\prime }_{t\times n}$ of the unknown netlist is provided to the trained classifier $C$ to detect if the netlist contains hardware Trojans.

Design goals

Our goal is to design a machine-learning-based hardware-Trojan detection model with higher detection accuracy in gate-level netlists. Specifically, we have the following goals:

• Effectiveness. Our main goal is to detect any parts of hardware Trojans in gate-level netlists. In the process of security detection for the gate-level netlists, as long as any part of the hardware Trojans is detected, it can be judged that the gate-level netlist contains the hardware Trojan.

• Accuracy. During the process of detecting hardware Trojan in gate-level netlists, the Trojan nets should be detected as much as possible. And the number of normal nets identified to be Trojan nets mistakenly should be as little as possible.

Trojan-net features

In this section, in order to more clearly describe the features of the Trojan net, we introduce the existing Trojan-net features by way of examples. At the same time, by analyzing the features of the existing Trojan and circuit structures, we propose a new Trojan-net feature.

Existing Trojan-net features

Fifty-one hardware-Trojan features have been proposed by Hasegawa et al., ¹⁹ which are listed in Table 1. This study gives a brief example of the existing 51 Trojan-net features.

Table 1.

The extracted features from a netlist (1≤x≤5).

Label	Trojan-net feature candidate	Description
f0–f4	fan_in_x	The number of logic-gate fan-ins x-level away from the net $n$ .
f5–f9	in_flipflop_x	The number of flip-flops up to x-level away from the input side of the net $n$ .
f10–f14	out_flipflop_x	The number of flip-flops up to x-level away from the output side of the net $n$ .
f15–f19	in_multiplexer_x	The number of multiplexer up to x-level away from the input side of the net $n$ .
f20–f24	out_multiplexer_x	The number of multiplexer up to x-level away from the output side of the net $n$ .
f25–f29	in_loop_x	The number of up to x-level loops from the input side of the net $n$ .
f30–f34	out_loop_x	The number of up to x-level loops from the output side of the net $n$ .
f35–f39	in_const_x	The number of constants up to x-level away from the input side of the net $n$ .
f40–f44	out_const_x	The number of constants up to x-level away from the output side of the net $n$ .
f45	in_nearest_pin	The minimum level to the primary input from the net $n$ .
f46	out_nearest_pout	The minimum level to the primary output from the net $n$ .
f47–f48	{in, out}_nearest_flipflop	The minimum level to any flip-flop from the input or output side of the net $n$ .
f49–f50	{in, out}_nearest_multiplexer	The minimum level to any multiplexer from the input or output side of the net $n$ .
f51–f55	in_gate_x	The number of logic-gate x-level away from the net $n$ .

As shown in Figures 3 and 4, the target Trojan net is the net $n$ marked with red. In Figure 3, $\mathrm{in}\_\mathrm{multiplexer}\_3=1$ indicates there is 1 multiplexer up to three-level away from the input side of the net $n$ . $\mathrm{in}\_\mathrm{nearest}\_\mathrm{pin}=3$ means the minimum level to the primary input from the net $n$ is 3. $\mathrm{in}\_\mathrm{flipflop}\_3=1$ indicates the number of flip-flops up to three-level away from the input side of the net $n$ is 1. $\mathrm{in}\_\mathrm{const}\_3=1$ means the number of constants up to three-level away from the input side of the net $n$ is 1. $\mathrm{fan}\_\mathrm{in}\_1=2$ indicates there are 2 logic-gate fan-ins one-level away from the net $n$ .

Figure 3.

The introduction of hardware-Trojan features (A).

Figure 4.

The introduction of hardware-Trojan features (B).

In Figure 4, $\mathrm{in}\_\mathrm{loop}\_3=1$ means the number of up to three-level loops from the input side of the net $n$ is 1. $\mathrm{out}\_\mathrm{nearest}\_\mathrm{flipflop}=2$ indicates the minimum level to any flip-flop from the output side of the net $n$ is 2. $\mathrm{out}\_\mathrm{nearest}\_\mathrm{multiplexer}=2$ means the minimum level to any multiplexer from the output side of the net $n$ is 2. $\mathrm{out}\_\mathrm{nearest}\_\mathrm{pout}=2$ indicates the minimum level to primary output from the net $n$ is 2.

Logic-gate

Dataset and features determine the upper limit of a machine-learning method. Therefore, extracting the effective features of the Trojan net can further improve the hardware-Trojan detection effect.

The feature $\mathrm{fan}\_\mathrm{in}\_x$ which is proposed by Hasegawa et al. ¹⁹ represents the number of logic-gate fan-ins x-level away from the net $n$ , but by analyzing the structure of the circuit, a malicious designer can change the value of this feature by designing an equivalent circuit structure as shown in Figures 5 and 6. Figures 5 and 6 are equivalent circuits, the circuit functions they implement are the same. In Figure 5, $\mathrm{fan}\_\mathrm{in}\_2$ of the net $n$ is 8, while $\mathrm{fan}\_\mathrm{in}\_2$ is 4 and $\mathrm{fan}\_\mathrm{in}\_3$ is 8 in Figure 6.

Figure 5.

The score of different features.

Figure 6.

The example of Trojan-net features for logic-gate.

According to the discussion above, we extract the numbers of logic-gate x-level away from the net n as Trojan-net features which are named $\mathrm{in}\_\mathrm{gate}\_x$ . In this study, we set x = 1, 2, 3, 4, 5. For example, $\mathrm{in}\_\mathrm{gate}\_2$ of the net n which is shown in Figure 5 is 2.

Selecting Trojan-net features using XGBoost

According to the introduction of XGBoost, the XGBoost is a gradient boosting algorithm. One advantage of using the gradient boosting algorithm is that after the CARTs are created, the importance score for each feature can be obtained directly. In general, the higher the importance score, the more importance the feature. In a single CART, the closer a feature is to the root node, the greater the weight of the feature. The more the feature is selected by the CARTs, the more important it is. Finally, the results of a feature in all CARTs are weighted and then averaged to get the importance score.

Using the built-in scoring function of XGBoost, we can obtain the scores of 56 features which are list in Table 1 for every classifier. We summarize the feature scores of different classifiers for Trojan-included netlists, which is shown in Figure 7. Among the 56 Trojan-net features, 16 Trojan-net features can get more than 10,000 score and f46 (out_nearest_pout) gets the highest score. The scores of f53 (in_gate_3), f55 (in_gate_5), f54 (in_gate_4), and f52 (in_gate_2) which are proposed in this study are in the top 15. The results prove that the new Trojan-net features proposed in this study are effective.

Figure 7.

The example of Trojan-net features for logic-gate fan-ins.

According to the statistical results, f25 (in_loop_1), f30 (out_loop_1), and f40–f44 (out_const_x) still get 0 score, meaning that these features do not work for XGBoost-based hardware-Trojan detection method. Although the scores of features such as f34, f15, and f16 are very low compared to the scores of other features, these scores are all non-zero, meaning that they are useful for the training of the classifier. Therefore, we believe that these low score features are also effective. As a result, we select the best set of 56 Trojan-net features, which contains 49 features except for f25, f30, and f40–f44.

The XGBoost-based hardware-Trojan detection method

In this section, we introduce the hardware-Trojan detection method based on the XGBoost algorithm for gate-level netlist, including the entire hardware-Trojan detection process and related evaluation indexes. At the same time, we also summarize the experimental results of some existing related methods.

Hardware-Trojan detection process for gate-level netlist

This study first uses XGBoost to detect the hardware Trojans. The process of hardware-Trojan detection based on the XGBoost algorithm for gate-level netlists is shown in Figure 8. The hardware-Trojan detection process is as following four steps:

• Step 1: analysis of circuit structure features of gate-level netlists. In the gate-level netlist, the structure of the normal circuit and the Trojan circuit is further analyzed based on the existing net features. Summarize the corresponding net features, feature 1, feature 2, …, feature n.

• Step 2: extracting feature values from netlists. Based on the net features of the gate-level netlist summarized in Step 1, the feature values of the known netlists and the unknown netlists are extracted through Python and Pyverilog toolkits. The feature data extracted from the known netlist are divided into a training set and a verification set of the classifier. The classifier is trained through the training set. The quality of the trained classifier is measured by the validation set. The data set extracted from the unknown netlist is the testing set. Through the trained classifier, the testing set is classified and identified, thereby completing the security detection of the unknown netlist.

• Step 3: training the hardware-Trojan classifier. For the first time, we applied the XGBoost algorithm to hardware-Trojan detection of gate-level netlists. For the detection of Trojan-included netlists, we use the leave-one-out cross-validation method ²¹ where each one of the Trojan-included netlists is considered to be a testing set, and other 11 Trojan-included netlists are considered to be the training set. For the detection of Trojan-free netlists, each one of the Trojan-free netlists is used to be a testing set. We use 11 Trojan-included netlists as the training set, which are not related to the testing set. For example, when testing the free-RS232-T1000 netlist, the Trojan-included netlists except the RS232-T1000 netlist are used to be the training set. While testing free-s15850 and free-s38584 netlists, we use all 12 Trojan-included netlists to be the training set. During the training process, the quality of the trained classifier is evaluated by relevant evaluation indicators. Specific indicators are introduced in a later section. If the detection effect of the trained classifier is not satisfactory, the classifier is retrained by adjusting the parameters of the XGBoost classifier until the desired detection effect is achieved.

• Step 4: detecting the hardware Trojans on gate-level netlists. Based on the XGBoost classifier trained in Step 3, the testing set extracted from the unknown gate-level netlists in Step 2 is detected. If the Trojan net is not detected in the results, we consider that the unknown gate-level netlist is Trojan-free netlist. If a Trojan net is detected, we believe that the unknown gate-level netlists are Trojan-included netlists.

Figure 8.

The process of hardware-Trojan detection based on the XGBoost algorithm for gate-level netlists.

Experimental results of existing machine-learning-based hardware-Trojan detection methods

So far, some existing machine-learning hardware-Trojan detection methods are as follows: support vector machine (SVM)-based hardware-Trojan detection method, ²² neural network (NN)-based hardware-Trojan detection method, ²³ random forest (RF)-based hardware-Trojan detection method, ²⁴ and multi-layer neural network (MNN)-based hardware-Trojan detection method. ²⁵ The true positive rate (TPR) and the true negative rate (TNR) are used by Hasegawa et al. ²² and Inoue et al. ²⁶ to evaluate the detection results. In addition to the TPR and the TNR, Hasegawa et al. ²⁴ also used the Accuracy, the Precision, and the F-measure to evaluate the detection results and proposed that the F-measure is the best to measure the results very well. In order to further improve the TPR and the TNR, Hasegawa et al. ²⁵ used the MNN to detect hardware Trojans on the basis of the 11 Trojan-net features which are proposed by Hasegawa et al. ²⁴ For better comparison, we have compiled the experimental results of these related methods. The experimental results of above methods are shown in Tables 2 and 3.

Table 2.

Experimental results of the existing machine-learning-based methods (A).

Test data	Recall		F-measure		Precision		Accuracy
	SVM-5 22 (%)	NN-5 23 (%)	SVM-5 22 (%)	NN-5 23 (%)	SVM-5 22 (%)	NN-5 23 (%)	SVM-5 22 (%)	NN-5 23 (%)
RS232-T1000	53.33	42.22	18.79	25.00	11.54	17.76	34.08	63.34
RS232-T1100	58.33	100.00	5.88	17.52	3.10	9.60	28.21	63.78
RS232-T1200	80.00	70.00	6.53	8.59	3.40	4.58	27.30	52.70
RS232-T1300	88.89	22.22	6.72	4.40	3.49	2.44	27.69	71.66
RS232-T1400	83.33	100.00	7.84	14.12	4.12	7.59	24.44	53.05
RS232-T1500	83.33	66.67	7.91	13.01	4.15	7.21	25.80	65.92
RS232-T1600	88.89	77.78	6.72	11.29	3.49	6.09	27.69	64.17
s35932-T100	93.33	100.00	1.07	2.94	0.54	1.49	59.85	84.59
s35932-T200	100.00	87.50	1.21	3.21	0.61	1.64	59.29	86.88
s35932-T300	27.03	100.00	0.73	2.71	0.37	1.37	57.82	58.82
s38417-T100	100.00	100.00	1.67	1.47	0.84	0.74	75.70	72.28
s38417-T200	73.33	73.33	1.53	1.13	0.78	0.57	75.75	67.06
Average	77.48	78.31	5.57	8.78	3.04	5.09	43.63	67.02

SVM: support vector machine; NN: neural network.

Table 3.

Experimental results of the existing machine-learning-based methods (B).

Test data	Recall		F-measure		Precision		Accuracy
	RF-11 24 (%)	MNN-11 25 (%)	RF-11 24 (%)	MNN-11 25 (%)	RF-11 24 (%)	MNN-11 25 (%)	RF-11 24 (%)	MNN-11 25 (%)
RS232-T1000	100.00	100.00	93.51	25.08	87.80	14.34	98.43	32.58
RS232-T1100	52.78	78.00	64.41	20.27	82.61	11.65	93.44	30.96
RS232-T1200	94.12	91.00	96.97	31.74	100.00	19.22	99.38	58.79
RS232-T1300	100.00	86.00	100.00	32.31	100.00	19.89	100.00	66.93
RS232-T1400	100.00	100.00	100.00	27.95	100.00	16.24	100.00	27.03
RS232-T1500	97.44	82.00	97.44	28.95	97.44	17.57	99.38	51.24
RS232-T1600	96.55	97.00	91.80	21.04	87.50	11.80	98.44	34.23
s35932-T100	73.33	80.00	84.62	26.35	100.00	15.77	99.94	98.96
s35932-T200	8.33	67.00	15.38	2.04	100.00	1.04	99.83	87.96
s35932-T300	94.59	100.00	97.22	27.80	100.00	16.15	99.97	97.02
s38417-T100	41.67	83.00	58.82	14.44	100.00	7.91	99.88	97.97
s38417-T200	40.00	93.00	57.14	1.82	100.00	0.92	99.85	74.05
Average	74.90	88.08	79.78	21.65	96.28	12.71	99.04	63.14

RF: random forest; MNN: multi-layer neural network.

In Tables 2 and 3, SVM-5 represents hardware-Trojan detection method using the SVM based on 5 Trojan-net features, NN-5 represents hardware-Trojan detection method using NN based on 5 Trojan-net features, SVM-11 represents hardware-Trojan detection method using SVM based on 11 Trojan-net features, RF-11 represents hardware-Trojan detection method using RF based on 11 Trojan-net features, and MNN-11 represents hardware-Trojan detection method using a multi-layer NN based on 11 Trojan-net features.

In Table 2, the experimental results of NN-5 ²³ are better than SVM-5 ²² in terms of the average Recall, the average F-measure, the average Precision, and the average Accuracy. This comparison shows that the detection effect of hardware Trojans can be improved by selecting the appropriate machine-learning method based on the same Trojan-net features. Then, we compare the experimental results of SVM-5 ²² and SVM-11. ²⁴ The average F-measure, the average Precision, and the average Accuracy of SVM-11 ²⁴ are better than the results of SVM-5. ²² And the average Accuracy of SVM-11 ²⁴ is the best. According to the comparison results, the detection effect can be improved by extracting the effective Trojan-net features based on the same machine-learning method.

According to the experimental results of RF-11 ²⁴ and MNN-11 ²⁵ which are shown in Table 2, the F-measure, the Precision, and the Accuracy of RF-11 ²⁴ are better than the results of MNN-11. ²⁵ The F-measure of RF-11 ²⁴ is the best among existing methods, and MNN-11 ²⁵ can obtain the best Recall. Therefore, this study is mainly compared with RF-11 ²⁴ and MNN-11. ²⁵

Summarizing the experimental results of the above existing machine-learning-based methods, the best average Recall is 88.08%, the best average F-measure is 79.78%, the best average Precision is 96.28%, and the best average Accuracy is 99.04%. Although the existing machine-learning-based methods for hardware-Trojan detection in gate-level netlists can reach a good Precision and Accuracy, the F-measure and the Recall are not very satisfied. We consider that for ensuring the IC security, it is the most important to detect the Trojan nets as much as possible. Therefore, the Recall and the F-measure are more important than the Precision and the Accuracy. Therefore, when training the XGBoost classifier in this study, it mainly adjusts the parameters by measuring the two indicators of Recall and F-measure.

Evaluation indexes of machine learning for hardware-Trojan detection

There are four values to evaluate the machine-learning-based hardware-Trojan detection results: the true positive (TP) value indicates the number of Trojan nets identified to Trojan nets; the false negative (FN) value indicates the number of Trojan nets identified to normal nets mistakenly; the false positive (FP) value indicates the number of normal nets identified to Trojan nets mistakenly; and the true negative (TN) value indicates the number of normal nets identified to normal nets. In addition, there are five more evaluation values to evaluate the experimental results: the TPR which is known to be the Recall (R), the TNR, the Precision (P), the F-measure, and the Accuracy. TPR indicates the probability that all Trojan nets in the netlist are correctly detected as Trojan nets by the classifier, that is, the prediction accuracy of the Trojan net. The calculation formula is as follows

$$\mathrm{TPR}(R)=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}$$

(9)

TNR indicates the probability that the normal nets in the netlist are correctly detected as normal nets by the classifier, that is, the prediction accuracy rate of the normal net. The calculation formula is as follows

$$\mathrm{TNR}=\frac{\mathrm{TN}}{\mathrm{FP}+\mathrm{TN}}$$

(10)

Precision indicates the proportion of the actually Trojan nets in the nets of the Trojan nets predicted by the classifier. The calculation formula is as follows

$$\mathrm{Precision}(P)=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}$$

(11)

F-measure, also known as F-score, represents the weighted harmonic average of Precision and Recall. The calculation formula is as follows

$$F\mathrm{-}\mathrm{measure}=\frac{2\mathrm{PR}}{P+R}$$

(12)

Accuracy represents the proportion of correctly detected nets among all predictions output by the classifier. The calculation formula is as follows

$$\mathrm{Accuracy}=\frac{\mathrm{TP}+\mathrm{TN}}{\mathrm{TP}+\mathrm{FN}+\mathrm{FP}+\mathrm{TN}}$$

(13)

All of the evaluation values above are important. Figure 2 shows the statistics of the normal net and Trojan nets for some netlists, which are provided by Trust-Hub. ²⁰ According to the number of normal nets and Trojan nets in Figure 2, we can see that the number of normal nets is far more than the number of Trojan nets. So, it is unreasonable to evaluate the detection effect of machine-learning-based method only by the Precision and Accuracy while the F-measure is suitable. Furthermore, when detecting the hardware Trojans, we should pay more attention to detect all Trojan nets. Even if a few normal nets are mistakenly detected as Trojan nets, it is better than the security problem caused by the omission of the Trojan nets. Therefore, it is also reasonable to use Recall to evaluate the detection effect of hardware Trojans. In this study, we focus on F-measure and Recall to evaluate the detection results.

Experiments and analysis

In this section, first, we use XGBoost for experiments based on 51 Trojan-net features to prove that the XGBoost-based hardware-Trojan detection method is more suitable than existing machine-learning-based hardware-Trojan detection methods. Second, based on the best Trojan-net features, we use XGBoost for experiments to prove that the features proposed in this study can further improve the detection effect of hardware Trojans. Third, in order to better explain that the proposed method can further improve the detection effect, we compare it with some related methods. Finally, in order to prove the practicality of the proposed method, we explain the time-consuming situation.

Experiments based on existing Trojan-net features

In this section, we give 51 Trojan-net features ¹⁹ to XGBoost for experiments to prove that the XGBoost is effective for the detection of hardware Trojans. Tables 4 and 5 show the experimental results with 51 Trojan-net features.

Table 4.

The experimental results based on 51 Trojan-net features (A).

Test data	TN	FP	FN	TP	TPR (Recall) (%)	TNR (%)	Precision (%)	F-measure (%)	Accuracy (%)
RS232-T1000	298	7	1	9	90.00	97.70	56.25	69.23	97.46
RS232-T1100	309	1	0	11	100.00	99.68	91.67	95.65	99.69
RS232-T1200	310	0	0	13	100.00	100.00	100.00	100.00	100.00
RS232-T1300	309	0	0	7	100.00	100.00	100.00	100.00	100.00
RS232-T1400	306	0	0	12	100.00	100.00	100.00	100.00	100.00
RS232-T1500	308	3	0	11	100.00	99.04	78.57	88.00	99.07
RS232-T1600	311	0	1	9	90.00	100.00	100.00	94.74	99.69
s35932-T100	6409	0	2	11	84.62	100.00	100.00	91.67	99.97
s35932-T200	6405	0	11	1	8.33	100.00	100.00	15.38	99.83
s35932-T300	6403	2	2	35	94.59	99.97	94.59	94.59	99.94
s38417-T100	5790	9	2	9	81.82	99.84	50.00	62.07	99.81
s38417-T200	5797	5	2	9	81.812	99.91	64.29	72.00	99.88
Average	–	–	–	–	85.93	99.68	86.28	81.94	99.61

TN: true negative; FP: false positive; FN: false negative; TP: true positive; TNR: true negative rate.

Table 5.

The experimental results based on 51 Trojan-net features (B).

Test data	TN	FP	FN	TP	TPR (Recall) (%)	TNR (%)	Precision (%)	F-measure (%)	Accuracy (%)
free-RS232-T1000	312	1	0	0	100.00	99.68	0.00	0.00	99.68
free-RS232-T1100	306	8	0	0	100.00	97.45	0.00	0.00	97.45
free-RS232-T1200	310	6	0	0	100.00	98.10	0.00	0.00	98.10
free-RS232-T1300	302	6	0	0	100.00	98.05	0.00	0.00	98.05
free-RS232-T1400	304	8	0	0	100.00	97.44	0.00	0.00	97.44
free-RS232-T1500	308	8	0	0	100.00	97.47	0.00	0.00	97.47
free-RS232-T1600	306	4	0	0	100.00	98.71	0.00	0.00	98.71
free-s15850	2419	0	0	0	100.00	100.00	100.00	100.00	100.00
free-s35932	6405	0	0	0	100.00	100.00	100.00	100.00	100.00
free-s38417	5798	0	0	0	100.00	100.00	100.00	100.00	100.00
free-s38584	7343	0	0	0	100.00	100.00	100.00	100.00	100.00
Average	–	–	–	–	100.00	98.81	–	–	98.81

TN: true negative; FP: false positive; FN: false negative; TP: true positive; TNR: true negative rate.

Table 4 shows the detection results of Trojan-included netlists. We can obtain 100.00% TPR in RS232-T1100, RS232-T1200, RS232-T1300, RS232-T1400, and RS232-T1500; all the Trojan nets in these netlists are correctly identified to be Trojan nets. Because the FP values in all the netlists are less than 10, so the normal nets are also correctly identified, the average TNR of the proposed method is 99.68%. The average F-measure of our proposed method is 81.94%, and the best F-measure of existing methods is 79.78%. The average Precision of ours is 86.28%, which is also the best. In terms of Accuracy, almost all netlists obtain more than 99.00% or more. This means that most nets can be correctly identified to Trojan nets and normal nets.

We can see that s35932-T200 has the smallest TP value. Although the detection result of s35932-T200 is relatively poor compared to other netlists, this result also proves that our proposed method is effective for the detection of hardware Trojans, which achieves our goal 1.

The detection results of Trojan-free netlists, which are shown in Table 5, the TNR, and the Accuracy can reach 98.00%, which are good detection results. The FP values of free-s15850, free-s35932, free-s38417, and free-38584 are 0, and this means that the proposed method is effective for the detection of Trojan-free netlists.

Experiments based on 49 Trojan-net features

According to the above discussion, we use the 49 effective Trojan-net features for XGBoost-based experiments to prove that the detection effect can be improved by the proposed Trojan-net features. Tables 6 and 7 show the experimental results with 49 Trojan-net features.

Table 6.

The experimental results based on 49 Trojan-net features (A).

Test data	TN	FP	FN	TP	TPR (Recall) (%)	TNR (%)	Precision (%)	F-measure (%)	Accuracy (%)
RS232-T1000	301	2	1	9	90.00	99.34	81.82	85.71	99.04
RS232-T1100	309	1	0	11	100.00	99.68	91.67	95.65	99.69
RS232-T1200	310	0	0	13	100.00	100.00	100.00	100.00	100.00
RS232-T1300	309	0	0	7	100.00	100.00	100.00	100.00	100.00
RS232-T1400	306	0	0	12	100.00	100.00	100.00	100.00	100.00
RS232-T1500	310	1	0	11	100.00	99.68	91.67	95.65	99.69
RS232-T1600	311	0	0	10	100.00	100.00	100.00	100.00	100.00
s35932-T100	6409	0	1	12	92.31	100.00	100.00	96.00	99.98
s35932-T200	6405	0	10	2	16.67	100.00	100.00	28.57	99.84
s35932-T300	6402	3	1	36	97.30	99.95	92.31	94.74	99.94
s38417-T100	5788	11	0	11	100.00	99.81	50.00	66.67	99.81
s38417-T200	5802	0	2	9	81.82	100.00	100.00	90.00	99.97
Average	–	–	–	–	89.84	99.87	92.29	87.75	99.83

TN: true negative; FP: false positive; FN: false negative; TP: true positive; TNR: true negative rate.

Table 7.

The experimental results based on 49 Trojan-net features (B).

Test data	TN	FP	FN	TP	TPR (Recall) (%)	TNR (%)	Precision (%)	F-measure (%)	Accuracy (%)
free-RS232-T1000	313	0	0	0	100.00	100.00	100.00	100.00	100.00
free-RS232-T1100	314	0	0	0	100.00	100.00	100.00	100.00	100.00
free-RS232-T1200	316	0	0	0	100.00	100.00	100.00	100.00	100.00
free-RS232-T1300	308	0	0	0	100.00	100.00	100.00	100.00	100.00
free-RS232-T1400	312	0	0	0	100.00	100.00	100.00	100.00	100.00
free-RS232-T1500	316	0	0	0	100.00	100.00	100.00	100.00	100.00
free-RS232-T1600	310	0	0	0	100.00	100.00	100.00	100.00	100.00
free-s15850	2419	0	0	0	100.00	100.00	100.00	100.00	100.00
free-s35932	6405	0	0	0	100.00	100.00	100.00	100.00	100.00
free-s38417	5798	0	0	0	100.00	100.00	100.00	100.00	100.00
free-s38584	7343	0	0	0	100.00	100.00	100.00	100.00	100.00
Average	–	–	–	–	100.00	100.00	100.00	100.00	100.00

TN: true negative; FP: false positive; FN: false negative; TP: true positive; TNR: true negative rate.

First, we analyze the detection results of Trojan-inserted netlists, which are shown in Table 6. The FP values in almost all netlists become less than 3, and more than half of netlists can get 100.00% TNR. This means that almost all normal nets can be identified correctly. We can obtain 100.00% Recall in RS232-T200, RS232-T300, RS232-T400, RS232-T600, s35932-T100, s35932-T200, and s38417-T200, which is more than half of netlists, and the FN values of almost all netlists are less than 2. The results mean that the Trojan nets can be identified to be Trojan nets correctly. We can obtain 100.00% Precision in more than half of netlists, which means that all the nets identified to be Trojan nets are actually Trojan nets in these netlists.

In terms of F-measure, most netlists achieve more than 90.00%, and 100.00% can be obtained in RS232-T1200, RS232-T1300, RS232-T1400, and RS232-T1600. The average F-measure is 87.75%. The Accuracy of all netlists is more than 99.00% or more, and the average Accuracy can get 99.83%. The average F-measure and the average Precision of our proposed method are quite good results in hardware-Trojan detection.

Table 7 shows the detection results of Trojan-free netlists. The FP values in all netlists are 0, which means that all the normal nets can be identified to be normal nets correctly. This means that our proposed method can achieve excellent detection results for the Trojan-free netlists.

Comparison of experimental results

In order to prove that the XGBoost algorithm and the proposed Trojan-net features are more suitable for hardware-Trojan detection, we first use the XGBoost algorithm to train the classifier based on the existing features, labeled XGBoost-51. Second, combined with the features proposed in this study, the feature set of the feature selection is used to train the corresponding XGBoost classifier, labeled XGBoost-49. According to the analysis in the previous sections, we know that the best Recall indicator can be obtained using the MNN-11 ²⁵ method. Therefore, in terms of the Recall indicators comparison, we compare to the detection effect of the MNN-11 ²⁵ method. As for the comparison of F-measure, Accuracy, and Precision indicators, the comparison of the previous sections shows that the best results can be obtained using RF-11. ²⁴ Therefore, this study mainly compares these three indicators with the RF-11 ²⁵ method. Since the 11 Trojan-net features used in the RF-11 method are further extracted based on 51 features, XGBoost-51 is comparable with RF-11 ²⁴ and MNN-11. ²⁵

First, compare the experimental results of Recall indicators, as shown in Figure 9. We can see that the RS232-T1100, RS232-T1200, RS232-T1300, RS232-T1400, RS232-T1500, and s35932-T100 netlists use the XGBoost-51 method, which is better than the MNN-11 ²⁵ detection effect in the Recall indicators. Although the average Recall of XGBoost-51 is lower than that of MNN-11, ²⁵ it is also a good result. While using the XGBoost-49 method for the netlists in the experiment, 66.7% of the netlists were better than MNN-11. ²⁵ Furthermore, on the average Recall, XGBoost-49 has achieved the current best Recall, which means that the method proposed in this study can further improve the detection effect of the hardware-Trojan net.

Figure 9.

The comparison in terms of Recall.

Second, a comparison of the F-measure indicators is shown in Figure 10. We can see that the average F-measure obtained using the XGBoost-51 method is better than that obtained by the RF-11 ²⁴ method. It can be seen that the XGBoost algorithm is more suitable for the detection of hardware Trojans of gate-level netlists. Moreover, with XGBoost-49, 75% of the netlists are superior to RF-11 ²⁴ in F-measure indicators. They are RS232-T1100, RS232-T1200, RS232-T1300, RS232-T1400, RS232-T1600, s35932-T100, s35932-T200, s38417-T100, and s38417-T200. At the same time, the average F-measure of XGBoost-49 is the best.

Figure 10.

The comparison in terms of F-measure.

Third, a comparison of Accuracy indicators. From the average Accuracy comparison in the last column of Figure 11, we can see that the average Accuracy obtained by the XGBoost-51 method is better than that obtained by the RF-11 ²⁴ method. This shows that the XGBoost algorithm is more suitable than the RF algorithm in hardware-Trojan detection. Furthermore, using the XGBoost-49 method, the Accuracy indicators of all netlists in the experiment are superior to the RF-11 method, ²⁴ which is a good illustration of the superiority of the proposed method.

Figure 11.

The comparison in terms of Accuracy.

Finally, compare the Precision indicator. As can be seen from the comparison of the last column of Figure 12, the proposed method is not as good as RF-11 in the Precision indicator. However, using the XGBoost-51 method, the Precision of 58.3% netlists is no lower than that using the RF-11 ²⁴ method. And using the XGBoost-49 method, there are 66.79% netlists can obtain better Precision.

Figure 12.

The comparison in terms of Precision.

In summary, using the XGBoost-49 method proposed in this study can further improve the detection effect of hardware Trojans in the gate-level netlist.

Comparison of time-consuming

We know that under the same conditions, the method that takes less time will be an excellent method. To prove that the XGBoost algorithm is also very time-consuming, we compare it to the NNs, RF, and SVM methods.

The comparison experiment is performed on the Win7 server with an Intel i5-6500 central processing unit (CPU), running at 3.20 GHz and 4 GB memory. The experiment was carried out with the training set and testing set corresponding to the RS232-T1000 netlist. The program is written in Python 2.7 using scikit-learn ²⁷ as a machine-learning library. During the experiment, the classifiers were trained using the default parameters. As we know, the efficiency of a machine-learning-based approach depends on its training phase. Therefore, we only count the training time of different classifiers. At the same time, in order to avoid random interference, we run each method 10 times under the same conditions and then record the average time as the time consumption of the method. The experiment results are shown in Table 8.

Table 8.

The time-consuming of different methods.

Method	Time (s)	Recall	F-measure
NN-51	27.04	0	0
RF-51	0.39	0.48	0.20
SVM-51	150.1	0	0
XGBoost-51	5.51	0.3	0.07
XGBoost-49	5.38	0.3	0.07

NN: neural network; RF: random forest; SVM: support vector machine.

We can see that under the same conditions, the RF method takes the least time, and the average time is only 0.39 s. Furthermore, under the default parameters, the RF method can get the best Recall and F-measure. The SVM method takes the most time, and the average time is as much as 150.1 s. The average time of the XGBoost method costs 5.51 s, which is also a very good result. The XGBoost method performs worse in terms of time-consuming, Recall, and F-measure than the RF method. However, the XGBoost method is more stable than the RF method, as shown in Figure 13. We can see that the XGBoost method is very stable during the 10 training sessions, and the results are the same every time. The RF method is not stable, and sometimes, the detection results are better, but sometimes, the detection results are not very good. Therefore, we believe that the XGBoost method is more suitable for the training of hardware-Trojan classifiers.

Figure 13.

The comparison of the random forest and XGBoost methods.

At the same time, in order to illustrate the time-consuming effects of features on classifier training, we compare the time spent on classifier training for 49 features and 51 feature data sets. As shown in Table 8, the XGBoost-based classifier trained with 49 features took less time and obtained the same detection effect.

Related work

At present, machine learning can achieve excellent results in pre-silicon detection. It is an excellent technical method, while few researchers have applied machine-learning methods to the detection of hardware Trojans in gate-level netlists.

Hasegawa et al. ²² extracted five Trojan-net features from the netlist and then used the SVM to classify the normal nets and the Trojan nets. The experimental results show that using the SVM can achieve 100% TPR in some cases. Inoue et al. ²⁶ designed three hardware Trojans, and then the Trojans were inserted into the netlist. The experimental results show that the SVM-based hardware-Trojan detection method is effective. Hasegawa et al. ²³ used the NN to identify the hardware Trojans on the basis of five hardware-Trojan features. This method can obtain 81% of the average TPR and 69% of the TNR. Hasegawa et al. ¹⁹ proposed 51 Trojan-net features and used the RF classifier to extract 11 features that can effectively detect and maximize the F-measure. Hasegawa et al. ²⁴ used the RF classifier with 11 Trojan-net features to classify the normal nets and the Trojan nets, and obtained 94.9% the average Precision, 79.3% the average F-measure, and 99.2% the average Accuracy. Hasegawa et al. ²⁵ extracted 11 Trojan-net features from the netlist and detected the Trojan nets using MNNs. Using this method can obtain 85% of the average TPR and 70% of the average TNR.

Conclusion

The security of the ICs is the key issue in the security of the IoT. In this study, we propose a machine-learning-based hardware-Trojan detection method for hardware Trojans in gate-level netlists. We propose new Trojan-net features based on the existing 51 Trojan-net features. ¹⁹ To remove some less important features, we use the scoring mechanism of the XGBoost algorithm to further extract the effective set of 49 Trojan-net features out of 56 features. After that, the hardware-Trojan classifier was trained based on 49 features combined with the XGBoost algorithm. The experimental results show that the proposed method can obtain the best Recall, F-measure, and Accuracy. Furthermore, through the comparison experiments, it is proved that the XGBoost algorithm and the features proposed in this study can further improve the detection effect of hardware Trojans. In future, we will further study the structure of gate-level Trojans and extract new Trojan-net features to get a better Trojan detection result.