Sage Journals: Discover world-class research

Abstract

Commonly used fastener positioning methods include pixel statistics (PS) method and template matching (TM) method. For the PS method, it is difficult to judge the image segmentation threshold due to the complex background of the track. For the TM method, the search in both directions of the global is easily affected by complex background, as a result, the locating accuracy of fasteners is low. To solve the above problems, this paper combines the PS method with the TM method and proposes a new fastener positioning method called local unidirectional template matching (LUTM). First, the rail positioning is achieved by the PS method based on the gray-scale vertical projection. Then, based on the prior knowledge, the image of the rail and the surrounding area of the rail is obtained which is referred to as the 1-shaped rail image; then, the 1-shaped rail image and the produced offline symmetrical fastener template is pre-processed. Finally, the symmetrical fastener template image is searched from top to bottom along the rail and the correlation is calculated to realize the fastener positioning. Experiments have proved that the method in this paper can effectively realize the accurate locating of the fastener for ballastless track and ballasted track at the same time.

Keywords

Fastener positioning pixel statistics template matching LUTM

Introduction

Railway fasteners are fasteners that fix the rails to the sleepers. Loss and fracture of the fasteners are likely to cause major accidents such as train derailment. In the actual laying of high-speed railway lines, about 7200 fasteners are used per kilometer. By the end of 2019, the operating mileage of high-speed railways in China has exceeded 35,000 km, accounting for about 70% of the world’s. With the rapid development of China’s railway, the problems of low efficiency and high-risk factor caused by ordinary manual inspections have been unable to meet the needs of frequent inspections of the railway network represented by high-speed railways. Therefore, the automatic detection of railway fasteners^1–4 is of great significance for rail operation and maintenance.

In recent years, many researchers have made some efforts to apply vision-based technology to the detection tasks of railway fasteners.^5,6 However, if the original image taken on the track line is used for detection, because the original track image area is too large, which will lead to poor real-time detection results due to long recognition calculation time, and the complex background in the original track image will lead to the decrease of detection accuracy. To solve these problems, research on the track fastener positioning is necessary for the fastener fault diagnosis. Many researchers have carried out in-depth research on the positioning algorithm of track fasteners. At present, a locating method based on PS, a locating method based on TM, and a locating method derived from the above methods combining image texture features⁷ and edge features^8–10 are the main methods that are used in fastener positioning.

Wei et al.¹¹ statistics the grayscale characteristics of the track image in the vertical direction and the grayscale characteristics in the horizontal direction, selected appropriate thresholds to locate the rail area and the sleeper area, and indirectly based on the geometric relationship of the track, sleeper, and fasteners to locate the fastener area; Xia et al.¹² used the method of PS to first locate the sleeper area and then use TM to achieve accurate positioning of the fastener. Although the PS method can easily locate the rail area based on the gray-scale characteristics, however, when the above methods are used to the ballastless track, The gray value of the sleeper area changes little, it is difficult to judge the threshold for segmenting the fastener area, which affects the locating accuracy of fasteners.

Yang et al.³ extracted the direction field as the feature descriptor of the fastener and realized the positioning of the ballasted track fastener through the TM method. Fan et al.¹³ proposed to use the improved local binary pattern algorithm to obtain the feature of the fastener, then use the TM method to realize fastener positioning. Although the TM method can be applied to ballasted track and ballastless track, however, only using a single template to match in the two global directions (vertical and horizontal directions) increases the difficulty of positioning, resulting in a decrease in the accuracy and efficiency of fastener positioning.

This paper proposes a method based on LUTM, combining the algorithm of PS and the algorithm of TM. First, the gray-scale vertical projection is used to locate the rail area, and the 1-shaped rail image containing fasteners is segmented according to prior knowledge because the gray level of the rail area is significantly different from the gray levels of other areas, the grayscale vertical projection is used for rail positioning, the threshold is easy to judge, and it can avoid the interference of complex backgrounds and other objects in the image. Then, because of the quality problems of shooting and the interference factors such as rust, oil, and gravel in the actual line, the corresponding preprocessing of the image can effectively improve the image quality; Finally, the symmetrical fastener template is used to search and match from the top to the bottom of the 1-shaped rail image to achieve the precise locating of the fastener.

Our approach has the following innovations: (1) When the TM method is applied to locate the fastener’s area, the only fastener template is used to search and match in two directions of the original track image, the LUTM method used in this paper, single-direction (vertical) search on partial images can avoid false matching of templates in the other direction (horizontal direction), meanwhile, it can reduce the preprocessing of the image outside the fastener area, which can save calculation time and improve real-time positioning of fasteners. (2) Experiments show that the method in this paper can be effectively applied to ballastless track and ballasted track, and improve the locating accuracy of fasteners.

This paper is organized as follows. Section 2 introduces the commonly used fastener positioning algorithm. Section 3 introduces the theory of the proposed LUTM fastener positioning method. The experimental results and analysis are provided in section 4. Finally, section 5 summarizes our work.

Common fastener positioning algorithm

This section will briefly introduce the pixel statistics-based positioning method and template matching-based positioning method currently used in fastener positioning methods. It provides corresponding theoretical support for the subsequent LUTM algorithm.

Fastener positioning based on PS

The method of locating track fasteners based on PS is to find the threshold value that can divide the rail region in the vertical direction and the threshold value that divides the sleeper region in the horizontal direction according to the gray value projection rule of the fastener image. Rail area and sleeper area, to form a “cross” shape where the rail and sleeper intersect, get the coordinate position of the intersection point in the original track line image, and finally use the geometric position relationship of the sleeper, rail and fastener and with the prior knowledge which is the size of the fastener, the fastener area can be located. The traditional process is shown in Figure 1 below. There is a process of determining the threshold for the statistical gray value in the process of fastener positioning, and the interference of complex backgrounds such as oil and gravel in the original track image makes it difficult to determine the threshold.

Figure 1.

Fastener positioning process based on PS.

Fastener positioning based on TM

The TM algorithm uses the template image to perform left-to-right and top-to-bottom correlation operations on the original track image, and finally obtains a coefficient matrix after the correlation operation: the parameter in the coefficient matrix is used to measure the similarity between the template image and the sub-image in the original track image. The principle is shown in Figure 2 below. The template $T (N, M)$ is superimposed on the searched original track image $S (W, H)$ and slides. The area where the template covers the searched image is called the sub-image $S_{ij}$ , and $i$ , $j$ are the coordinates of the upper left corner of the sub-image location, the search range is: $1 \leq i \leq W - N$ , $1 \leq j \leq H - M$ . The similarity between T and S can be measured by the following formula (1):

D (i, j) = \sum_{m = 1}^{M} \sum_{n = 1}^{N} [S_{ij} (n, m) - T (n, m)]^{2},

(1)

Where $S_{ij} (n, m)$ is the gray value at coordinate $(n, m)$ in the sub-image $S_{ij}$ , and $T (n, m)$ is the gray value at coordinate $(n, m)$ in the template $T (N, M)$ . The correlation coefficient¹⁴ of the normalized template matching can be obtained from the following formula (2):

R (i, j) = \frac{\sum_{m = 1}^{M} \sum_{n = 1}^{N} S_{ij} (n, m) \times T (n, m)}{| S_{ij} (n, m) | | T (n, m) |},

(2)

Where $| S_{ij} (n, m) | = \sqrt{\sum_{m = 1}^{M} \sum_{n = 1}^{N} {[S_{ij} (n, m)]}^{2}}$ , and $| T (n, m) | = \sqrt{\sum_{m = 1}^{M} \sum_{n = 1}^{N} {[T (n, m)]}^{2}}$ . When the template image is the same as the sub-image, the correlation coefficient is 1. After the search image is completely searched, $R_{\max}$ is found, and its corresponding sub-image $S_{ij}$ is the matching target.

Figure 2.

Template matching diagram.

Fastener positioning based on LUTM

This section will explain in detail the fastener positioning method based on LUTM and the principle of how this method can improve accuracy and efficiency. The overall flow of the LUTM algorithm proposed in this paper is shown in Figure 3.

Figure 3.

Flow chart of fastener positioning based on LUTM.

Split 1-shaped rail image

Each track line image contains at least the following three features: (1) The rail area is distributed vertically in the original track image, and the fastener area is perpendicular to the rail.¹⁵ (2) The position of the rail area is basically fixed in all track images. (3) Friction causes the rail area to become brighter, and it is clearly distinguished from other areas in the original track image.

By setting a sliding window with the same length as the rails (the size of the window is set to $h \times w$ , where $h$ is the length of the rails and $w$ is the width of the selected sliding window) to slide horizontally in step = 1 on the original track line image, the average gray value in each sliding window is counted to find the change characteristics of the average gray value of the image at the rail area, to realize the locating of the rail area. By calculating the average gray value $A$ of the pixels in each sliding window, the average gray value distribution curve of the pixels can be obtained. Due to the remarkable high gray value performance of the rail, the positioning problem of the rail area can be converted into an image threshold segmentation problem. The value of calculating the average gray level $A$ of the pixels in the sliding window can be obtained by the following formula (3):

A (x, y) = \frac{1}{w \cdot h} \sum_{x = 1}^{w} \sum_{y = 1}^{h} f (x, y),

(3)

Where $f (x, y)$ is the gray value of the pixel at the location. After calculation, the average gray value curve of the pixel projected in the vertical direction of the original track line image is obtained, and the average gray value curve of the pixel has an obvious peak $A_{\max}$ , which corresponds to the rail area in the original track image. Select a suitable threshold value $A_{t}$ ( $\bar{A} < A_{t} < A_{\max}$ ), and after threshold segmentation, the rail area can be positioned through the program, and the column coordinates of the 1-shaped rail image of the rail including the fastener area in the original image can be determined by the following formula (4):

\begin{matrix} c_{l} = C_{l} - w_{f} + α \\ c_{r} = C_{r} + w_{f} + α \\ α = \frac{1}{2} [w_{t} - (C_{r} - C_{l} + 2 w_{f})], \end{matrix}

(4)

Where $c_{l}$ and $c_{r}$ are the column coordinates of the 1-shaped rail image in the original track image, $C_{l}$ and $C_{r}$ are the column coordinates of the left and right rails in the original image after segmentation according to the threshold, and $w_{f}$ is the fastener known according to prior knowledge in the original image, $α$ is an adaptive parameter that adjusts the width of the 1-shaped rail image to match the width of the template, and $w_{t}$ is the width of the template.

Image preprocessing

In this section, the image pre-processing involved in this paper mainly includes image graying,¹⁶ image histogram equalization,¹⁷ and image Gaussian filtering and denoizing¹⁸ will be briefly introduced.

To quickly obtain features and reduce the amount of calculation, the image uses the weighted-average method for grayscale processing.

Since the track’s original image is dark as a whole, and the contrast between the fastener area and the background is not obvious, the histogram equalization method is used to adjust the gray distribution of an image, the original image can be reconstructed into a new image with gray values consistent with uniform distribution, which can improve the contrast of the track image and enhance the image details.

Gaussian filtering can remove noise and other interference and can keep the edges of the image very well. The Gaussian filter has a smaller degree of blurring of the image than the traditional filter. Because the template coefficient of the Gaussian filter decreases with increasing distance from the center of the template, more original features of the image can be retained.

Accurate positioning of fasteners

Analysis of the characteristics of the fastener image shows that the fastener is a standard component and has a high degree of axial symmetry,¹⁹ so a pair of normal fasteners are used as templates, and the width of the symmetry template is the same as the width of the rail image, Finally, to achieve accurate locating of fasteners, the symmetrical fastener template is scanned from top to bottom in a 1-shaped rail image with the same columns and rows changed. The schematic diagram is shown in Figure 4 below. The algorithm uses a normalized correlation coefficient to measure the similarity,^20,21 and its calculation formula is shown in the following formula (5):

R (j) = \frac{\sum_{m = 1}^{M} \sum_{n = 1}^{N} S_{j} (n, m) \times T (n, m)}{| S_{j} (n, m) | | T (n, m) |},

(5)

Where $| S_{j} (n, m) | = \sqrt{\sum_{m = 1}^{M} \sum_{n = 1}^{N} {[S_{j} (n, m)]}^{2}}$ , $| T (n, m) | = \sqrt{\sum_{m = 1}^{M} \sum_{n = 1}^{N} {[T (n, m)]}^{2}}$ , $R (j)$ refers to the normalized correlation coefficient between template $T$ and the original image $S$ at coordinates $(0, j)$ , 1 means a perfect match, −1 means bad match, 0 means no Correlation, find $R_{\max}$ , and its corresponding subgraph $S_{j}$ is the position of the fastened fastener.

Figure 4.

LUTM diagram.

Experimental results

The images used in this paper are all taken by line scan cameras with a resolution of 2048 × 4096. There are a total of 1364 original track images, of which 733 were ballasted line fasteners and 631 were ballastless line fasteners. A computer with a main frequency of 2.4 GHz is used to carry out simulation experiments.²² It is verified by experiments that the LUTM method proposed in this paper can effectively achieve the accurate positioning of actual line fasteners.

Experimental parameter settings

According to experimental verification, to obtain the best results of the experiment, according to prior knowledge, the width of the fastener area is 800 pixels. The sliding window width $w = 1$ is used in the gray vertical projection of this paper. The $3 \times 3$ , $σ = 0.8$ template used by Gaussian filtering is as follows:

\frac{1}{16} [\begin{matrix} 1 & 2 & 1 \\ 2 & 4 & 2 \\ 1 & 2 & 1 \end{matrix}]

Experiment analysis

For the consideration of the actual scene application, the experimental part analyzes the ballasted and ballastless track images collected on the actual line, as shown in Figure 5 below, where (a) is the ballastless track image, (b) is a ballasted track image.

Figure 5.

Original view of track fasteners.

To improve the image quality, this paper first uses the vertical projection method to determine the rail area. As shown in (a) and (b) of Figure 6 below, where the x-axis represents the column coordinates of the track image, and where the y-axis is the average gray value, it can be seen that the average pixel gray value of the image between 2200 and 2400 columns has an obvious wave peak which corresponds to the rail area in the original track image. Select an appropriate threshold to binarize the image, the processed images of the ballasted and ballastless track lines are shown in (c) and (d) of Figure 6 below, the rail area can be located, and then the formula (4) can be combined with the fixed value of the fastener width in the prior knowledge to obtain the 1-shaped rail image as shown in (e) and (f) of Figure 6 below, where the green rectangular frame is the rail area being located, and the red rectangular frame is the 1-shaped rail image.

Figure 6.

1-Shaped rail image segmentation process.

This paper uses image preprocessing methods such as image graying, histogram equalization, and image Gaussian filtering and denoizing, as shown in Figure 7 below, where (a) and (b) are the 1-shaped rail image that obtained from the original track line image, (c) and (d) are images after image enhancement and Gaussian filtering and denoising. After the above image preprocessing, the image contrast is significantly improved, the fastener features are more obvious.

Figure 7.

Image preprocessing.

The difficulty of precisely locating fasteners is increased by oil-covered fasteners, broken fasteners, and lost fasteners. The results of fastener position based on LUTM method in this paper are shown in Figure 8 below, where (a) is the fastener looseness image of ballastless track, (b) is the fastener fracture image of ballastless track, (c) is the fastener covered by oil in the ballasted track, and (d) is the fastener losing image of the ballasted track. It can be seen that the above problems have been effectively solved.

Figure 8.

Location of the damaged fasteners.

Evaluation of experimental methods

To evaluate the performance of the algorithm in this paper, the accuracy rate of fastener locating is used as the evaluation criteria of the algorithm, the evaluation results are shown in Table 1 and in Table 2 below. Table 1 is a comparison of the locating accuracy of a ballastless track, and Table 2 is a comparison of the locating accuracy of ballasted track. From the comparison between Tables 1 and 2, it can be seen that due to the complex background of ballasted track image, the locating accuracy of the TM method in the ballasted track is only 85%, while the locating accuracy of the TM method in the ballastless track is 92.4%, so the TM method is better in ballastless tracks; while in ballastless tracks, since there are no sleepers and the threshold value of the fastener backing plates is difficult to be determined, so the locating accuracy of PS method in the ballastless track is only 86.1%. Compared with Xia et al.,¹² the accuracy rate of the PS method used in the ballasted track with sleepers is 93.9%, therefore, the PS method is better in the ballasted track. The LUTM method proposed in this paper makes the locating accuracy of the fastener significantly improved compared to using TM and PS alone, for the ballastless track, the accuracy of this method in this paper is 7% and 13.3% higher than the TM method and PS method respectively, and for the ballasted track, the accuracy of this method in this paper is 14.6% and 5.7% higher than the TM method and PS method respectively. It shows that LUTM method can be applied to both ballastless tracks and ballasted tracks and can achieve a very reliable performance.

Table 1.

Comparison of locating accuracy of Ballastless track.

Track line category	Method	Accuracy (%)
Ballastless track	TM	92.4
	PS	86.1
	LUTM	99.4

Table 2.

Comparison of locating accuracy of Ballasted track.

Track line category	Method	Accuracy (%)
Ballasted track	TM	85.0
	Xia’s et al.¹²	93.9
	LUTM	99.6

Conclusion

This paper proposes a fastener locating method based on LUTM, which is used to solve the problems of existing fastener locating methods. When the method of PS is used, it is susceptible to the complex background interference of the track line, and it is difficult to judge the threshold value, which causes the image segmentation algorithm to become more complicated. When the global two-direction TM method is used, it is affected by the state of fasteners and the complex background of the track, which results in the problem of low locating efficiency and accuracy of fasteners. This method first divides the 1-shaped rail image containing the fastener area from the original track image, reduces the interference information of the original image’s complex background, improves the locating accuracy of the method, and then through image preprocessing improves the image quality and the grayscale characteristics of the fastener are more prominent. Finally, the symmetrical fastener template is used to search and match the 1-shaped rail image only in the vertical direction to achieve accurate locating of fasteners. It is experimentally verified that this method can be effectively applied to both ballastless track and ballasted track without the need for a systematic replacement method, which can effectively meet the requirements of railway engineering departments.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: A: The support foundations: National Natural Science Foundation of China (Grant No. 51975347). B: Shanghai Science and Technology Committee (Grant No. 18030501300). C: Shanghai Sailing Program (18YF1409200). D: Talent Program of Shanghai University of Engineering Science. E: Supported by the Opening Project of Shanghai Trusted Industrial Control Platform (TICPSH202103003-ZC).

ORCID iD

Zhaomin Lv

Author biographies

Liang Li, Master of College of Urban Railway Transportation, Shanghai University of Engineering Science, is mainly engaged in the research of intelligent detection, machine learning and image processing of rail transit.

Zhaomin Lv received Ph.D. degree in control science and engineering from East China University of Science and Technology in 2017. In the same year he joined Shanghai University of Engineering Science as a lecturer. His research interests include fault diagnosis and machine learning. He won the Shanghai Youth Science and technology talents sailing program in 2018.

Xingjie Chen, Master of Nanjing University of Science and Technology. Now he works at College of Urban Railway Transportation, Shanghai University of Engineering Science His current research interests include signal detection and image processing of rail and vehicle detection.

Yijin Qiu, Master of College of Urban Railway Transportation, Shanghai University of Engineering Science. is mainly engaged in the research of intelligent detection, machine learning and image processing of rail transit.

Liming Li received M.Phil in Vehicle Engineering from Shanghai University of Engineering Science, Shanghai, China, in 2010. Now he works at College of Urban Railway Transportation, Shanghai University of Engineering Science. His current research interests include image processing and computer vision.

Anqi Ma, Master of College of Urban Railway Transportation, Shanghai University of Engineering Science, is mainly engaged in the research of intelligent detection, deep learning and image processing of rail transit.

Shubin Zheng received Ph.D degree in Transportation Engineering from State Key Laboratory of Traction Power, Southwest Jiaotong University, Chengdu, China, in 2008. Now he works at College of Urban Railway Transportation, Shanghai University of Engineering Science. His current research interests include railway track and vehicle condition simulation, inspection and monitoring.

Xiaodong Chai received PhD degree in Electrical Engineering from Anhui University, Hefei, China, in 2003. Now he works at College of Urban Railway Transportation, Shanghai University of Engineering Science. His current research interests include automatic control, signal processing and computer vision.

References

Luo

, et al. A classification model of railway fasteners based on computer vision. Neural Comput Appl 2019; 31(12): 1–13.

Zhang

Yang

Tao

, et al. Vision method of inspecting missing fastening components in high-speed railway. Appl Optics 2011; 50(20): 3658–3665.

Yang

Tao

Liu

, et al. An efficient direction field-based method for the detection of fasteners on high-speed railways. Sensors Basel 2011; 11(8): 7364–7381.

Marino

Distante

Mazzeo

, et al. A real-time visual inspection system for railway maintenance: automatic hexagonal-headed bolts detection. IEEE Trans Syst Man Cybern C Appl Rev 2007; 37(3): 418–428.

Resendiz

Molina

Hart

, et al. Development of a machine-vision system for inspection of railway track components. In: 12th World conference on transport research, Lisbon, Portugal, 11–15 July 2010, pp.1–22.

Aytekin

Ulusoy

Doğru

, et al. Railway fastener inspection by real-time machine vision. IEEE Trans Syst Man Cybern 2015;45(7): 1101–1107.

Dai

Peng

Wang

KCP

, et al. Railway fastener detection method based on 3D images. In: 1st International conference on rail transportation, SW Jiaotong University, Chengdu, China, 10–12 July 2017, pp.938–946.

Feng

Jiang

Xie

, et al. Automatic fastener classification and defect detection in vision-based railway inspection systems. IEEE Trans Instrum Meas 2014; 63(4): 877–888.

Hsieh

Chen

Liao

. Visual recognition system of elastic rail clips for mass rapid transit systems. In: ASME/IEEE joint rail conference/ASME internal combustion engine division spring technical conference, Pueblo, CO, 13–16 March 2007, pp.319–325.

10.

Prasongpongchai

Chalidabhongse

Leelhapantu

. A vision-based method for the detection of missing rail fasteners. In: IEEE International conference on signal and image processing applications, Kuching, Malaysia, 12–14 September 2017, pp.419–424.

11.

Wei

Yang

Liu

. Railway track fastener defect detection based on image processing and deep learning techniques: a comparative study. Eng Appl Artif Intel 2019; 80: 66–81.

12.

Xia

Xie

Jiang

. Broken railway fastener detection based on adaboost algorithm. In: 2010 International conference on optoelectronics and image processing, Haikou, China, 11–12 November 2010, pp.313–316.

13.

Fan

Cosman

Hou

, et al. High-speed railway fastener detection based on a line local binary pattern. IEEE Signal Process Lett 2018; 25(6): 788–792.

14.

Debella-Gilo

Kääb

. Sub-pixel precision image matching for measuring surface displacements on mass movements using normalized cross-correlation. Remote Sens Environ 2011; 115(1): 130–142.

15.

Liu

Huang

Zou

, et al. Learning visual similarity for inspecting defective railway fasteners. IEEE Sens J 2019; 19(16): 6844–6857.

16.

Kanan

Cottrell

. Color-to-grayscale: does the method matter in image recognition. PLoS One 2012; 7(1): 133–-139.

17.

Celik

. Two-dimensional histogram equalization and contrast enhancement. Pattern Recognit 2012; 45(10): 3810–3824.

18.

Pan

. Bias error reduction of digital image correlation using Gaussian pre-filtering. Opt Laser Eng 2013; 51(10): 1161–1167.

19.

Yang

, et al. Integrate the original face image and its mirror image for face recognition. Neurocomputing 2014; 131: 191–199.

20.

Yang

Gui

Chen

, et al. Voltage difference residual-based open-circuit fault diagnosis approach for three-level converters in electric traction systems. IEEE Trans Power Electron 2020; 35(3): 3012–3028.

21.

Chen

Yang

, et al. A data-driven ground fault detection and isolation method for main circuit in railway electrical traction system. ISA Trans 2019; 87: 264–271.

22.

Peng

Tao

Yang

, et al. A uniform modeling method based on open-circuit faults analysis for NPC-three-level converter. IEEE Trans Circuits Syst II Express Briefs 2019; 66(3): 457–461.

Research on track fastener positioning method based on local unidirectional template matching

Abstract

Keywords

Introduction

Common fastener positioning algorithm

Fastener positioning based on PS

Fastener positioning based on TM

Fastener positioning based on LUTM

Split 1-shaped rail image

Image preprocessing

Accurate positioning of fasteners

Experimental results

Experimental parameter settings

Experiment analysis

Evaluation of experimental methods

Conclusion

Footnotes

Declaration of conflicting interests

Funding

ORCID iD

Author biographies

References