Automatic road marking recognition for intelligent vehicle systems application

Abstract

Automatic road markings recognition is important for the research of intelligent vehicle which is used for both automotive navigation and advanced driver assistance system. In most previous researches, the markings such as lane have been used for localizing and serving the vehicle along the road. However, in fact, the markings such as guide arrows and warnings are necessary for automotive navigation. Therefore, this article presents an automatic road markings recognition method based on support vector machine to reduce the impact of external environment such as viewpoint, brightness, and background. In which, the input vector comprises four improved Hu moments and two affine invariant moments obtained from reconstructed image. The presented method has been tested with experiment images, and the results show that the accuracy of recognition can be reached over 97% and time consumption per frame is 0.26 s. It is clear that the proposed method has strong potential effectiveness to be applied for intelligent vehicle systems application.

Introduction

Machine vision has greatly promoted the development of intelligent vehicle (IV)-related technology and effectively reduced the traffic accidents caused by driver’s fatigue and distraction. However, on the other hand, machine vision has shown limited success because of its reliability in real driving scenarios. Considering road markings recognition as the core issues in IV research field, this article presents an approach with information fusion for road markings recognition. The presented method is capable of classifying the markings with minimum false alarms and miss-detections.

Road marking recognition is a challenging task for real-time applications, which has become a research highlight for decades. Many studies have been conducted on this topic, but many of them are often restricted to a simple target such as lane mark. Partial of the achievements are already commercially available in vehicles. At present, the research can be roughly classified into monocular and binocular methods according to vision information obtained.

In monocular research, road markings are always recognized in two-dimensional images. On one hand, the characteristics of brightness difference between markings and background are often applied. Different classification methods are proposed. Wang et al. ¹ presented a lane detection and tracking system based on lane features extraction method and the Gaussian sum particle filter. In order to overcome the shortcomings of traditional algorithms such as not adaptive to changing environments and hard selection of threshold, Xiao et al. ² proposed an adaptive structured learning method based on road markings detection algorithm. In which, a structured random forest was learned to classify each image patch first and then the contextual information of the images and the structural information of the labels were effectively exploited to reduce the ambiguity.

Tang et al. ³ proposed a method for real-time lane detection and anti-rear-end collision system for drivers. The work uses a monocular camera to estimate the distance with the forward vehicles. The system can distinguish lane departures if the vehicle is too close with the forward vehicle. However, in these methods, the lane mark is often approximated with line model of Hough transform (HT) method. When the markings do not have the characteristics of line shape appearance especially the curve lane and other traffic markings, the methods cannot work properly. Traditionally, B-snake curve fitting is used for this purpose. ⁴ But it does not work well at the truncation points of the images because this method always uses vanishing point to extrapolate the line segment which represents lane data. So, Gupta and Merchant ⁵ proposed an algorithm to achieve lane detection applying K-means clustering method to report data in a manner suitable to create a solvable map. The presented method overcomes the shortcomings of B-snake as it did not depend on the perspective transform.

In binocular research, the stereo pairs make it possible that the road markings are reconstructed in three-dimensional (3D) spaces. Soheilian et al. ⁶ presented an approach to road marking reconstruction using stereo pairs acquired by a mobile mapping system in a dense urban area. In the research, zebra crossings and dashed lines were studied. The other authors use a sequence of stereo pairs for road marking reconstruction.^7–11 The advantage of these methods is that the marking information can be obtained in an absolute coordinate system. But on the other hand, the binocular method is always time consuming and unsuitable for real-time applications.

At present, most of the related methods mentioned above are only researched in detail in theoretical and model approaches, but there is still a long distance to go from theory approaches to practical applications. The automatic extraction of road markings from images is a complex problem due to the following issues: (1) partially damaged or occluded markings, (2) complex road environment and marking contrasts, and (3) perspective distortion. Thus, to fuse possible features from the image extraction is essential for making an accurate recognition effectively.

Support vector machine (SVM) has proved its success in many research fields including both data classification and pattern recognition. ¹² It shows very resistance to over-fitting problem and achieves high performance in recognition problems.^13–15 These successful researches motivate us to apply SVM in the road markings detection for natural environments. Therefore, this article aims to establish a road marking recognition model. In which, SVM is used as a classifier to recognize the markings which types they belong to. To achieve this, in this article, measurable visual characteristics such as improved Hu moments and affine inconvenience moments will be identified.

The rest of this article is organized as follows: first, the system overview including on-board image acquisition system and sketch map of this article is described; second, the perspective distortion of the image is corrected using non-linear mathematical model; then, within the established area of interest (AOI), the combined feature vector used for SVM is obtained; finally, auto-detection of road markings with the presented method has been tested with experiment and the conclusions are derived respectively.

System overview

On-board system setup

To acquire the markings image accurately, it is essential to set up an on-board image acquiring system. Figure 1 shows the configuration for this article’s research, which include a camera mounted on the windshield and an on-board computer. When system starts, the camera captures the front environment image and transmits it to the computer real time. Then, the proposed method of this article is conducted.

Figure 1.

Hardware configuration for the proposed method.

Overview of the proposed method

Considering the practical applications required with performance, the presented method refined road marking recognition and is shown in Figure 2.

Figure 2.

Sketch of the road marking recognition.

First, perspective distortion correction process is conducted and then the image preprocessing steps are applied. Second, the region of AOI which limits the road markings search area to a region as small as possible is defined. In this area, most of the “noise” is excluded. Then, invariant combined vector of candidates for SVM recognition are extracted. In the following section, the main steps for the presented approach are described in detail.

Perspective scene reconstruction

In photography and cinematography, perspective distortion is determined by the relative distances between the viewer and the target. For road marking recognition, such a distortion would result in visual deviation and have influences on the image information extraction. Therefore, this article adopts a non-linear mathematical model to conduct the reconstruction process. According to Zhang et al., ¹⁶ given an image point (x, y) and its 3D corresponding point (X, Y, Z), the relationship between them is

$$\{\begin{array}{c}\multicolumn{1}{c}{X=\frac{C_{1}x+C_{2}y+C_3}{C_{7}x+C_{8}y+1}}\\ \multicolumn{1}{c}{Y=\frac{C_{4}x+C_{5}y+C_6}{C_{7}x+C_{8}y+1}}\end{array}$$

(1)

where C₁–C₈ are undetermined coefficients which can be determined with four couple points by undetermined coefficients method.

With equation (1), indoor and outdoor simulation experiments were conducted, and the results are shown in Figure 3. In which, the red circles indicate the control points used in the scene reconstruction process. Based on our experience, it is important to note that when the control points are distributed over the entire image, a better reconstruction effect can be obtained. On the contrary, when distributed within the local area, the effect decreases.

Figure 3.

Simulation experiments.

Some conclusions can be obtained through the above experiments: First, the accuracy of image reconstruction depends on the distribution of the control point. Second, the road markings are reconstructed to their relative spatial location.

Lane marking detection

In this step, the lane detection phase includes three steps: feature extraction, lane modeling, and parameter estimation. Taking into account that most of the road markings are located in the current lane, we conduct the road marking recognition in the current lane region.

Lane marking feature extraction with one-dimensional entropy segmentation

In information theory, entropy is used for measuring the uncertainty problems associated with random variable. This term is often referred to Shannon entropy. In general, the value of entropy information reaches maximum at the border between target and background.

Given the image with gray level L (0 < L ≤ 256), the probability of pixels with gray level i is

$$p_i=f_i/(M\times N)$$

(2)

where f_i denotes the number of pixels with gray level i and M, N denotes the image dimension.

Assumed t as the threshold for segmentation, the target and background entropies are

$$\{\begin{array}{c}{E}_O=-\sum _{i=0}^t[p_i/p\left(t\right)]·\ln [p_i/p\left(t\right)]\hfill \\ \multicolumn{1}{c}{E_B=-\sum _{i=t+1}^{L-1}\{p_i/[1-p\left(t\right)]\}·\ln \{p_i/[1-p\left(t\right)]\}}\end{array}$$

(3)

where

$$p\left(t\right)=\sum _{i=0}^t{p}_i$$

(4)

then the optimal threshold is

$$T={\mathrm{Arg}}_{i\subseteq C_L}^{\max }(E_O+E_B)$$

(5)

With the algorithm mentioned above, examples for feature extraction are shown in Figure 4. In which, the first row is the original reconstruction images, the second row is the images after preprocessing with gray balance, the third row is the binary image using one-dimensional (1D) entropy, the fourth row is the results with area-filtering algorithm, and the fifth row is the gradient image using Canny operator. The experiment shows that the marking edge features are extracted completely (the fifth row). Within the image processing process, the gray balance algorithm was introduced to enlarge the difference between foreground and background gradation thereby enhancing the image contrast. The area-filtering algorithm was conducted to reduce some unwanted image information for marking detection.

Figure 4.

Lane feature extraction results.

Lane marking detection

In this article, HT is used for lane marking detection. The HT method is widely used in the lane marking detection which is not sensitive to noise, so it can handle the condition that target is incomplete or be covered partially. ¹⁷

The lane-related parameters are extracted with HT method. In this article, the AOI is divided into two regions; the origin of coordinates is set at the bottom line center which is shown as Figure 5.

Figure 5.

HT coordinate systems.

For the left and right regions, given y = mx + b as the line equation. HT method is conducted with the following formula

$$r=x\cos \left(\theta \right)+y\sin \left(\theta \right)$$

(6)

where (r, θ) denotes the vector from the origin to the nearest point of y = mx + b. In r − θ space, every line in the space xoy can be transformed to a point. In the same way, every point in xoy can be transformed to a line. The relationship between (r, θ) space and (r, θ) space is shown as Figure 6. Hough line transformation is the method which is based on this duality and the process is that calculating the line intersection in r − θ space (shown as (r₀, θ₀) in Figure 5) and then determining the line with the maximum point in the parameter space.

Figure 6.

Hough transform.

With the method described above, lane markings in Figure 4 are detected which are shown as Figure 7.

Figure 7.

Lane marking detection results.

With the AOI established, the image is segmented with 1D entropy algorithm. In experimental result shown as Figure 8, region index algorithm is used for recognizing the markings. Then, the target with maximum index length is considered as the possible candidate. In which, the first column is the segmentation images in the AOI, the second column is the images after noise filtering, and the third column is the region index images for the candidate markings.

Figure 8.

The candidate target detection.

Features extraction

Road marking has many characteristics including circle, rectangle, texture feature, and so on. Hu moment and affine invariant moment are the two important features in pattern recognition because they are not sensitive to translation, stretch, and rotation. Thus, this article adopts the combined-moment invariants (including four Hu moments and two affine invariant moments) as the main features used for road marking recognition.

Improved Hu invariant moment

Hu presented a theory of two-dimensional moment invariants for plane geometric figures and gave the descriptors of seven invariant moments in the literature. ¹⁸ In discrete space, the moment with p + q order and central moment are described as

$$\begin{array}{c}{m}_{\mathrm{pq}}=\sum _{(x,y)\in R}\sum x^p{y}^{q}f(x,y)\hfill \\ \multicolumn{1}{c}{{\mu }_{\mathrm{pq}}=\sum _{(x,y)\in R}\sum {(x-x_c)}^p{(y-y_c)}^{q}f(x,y)}\end{array}$$

(7)

where

$$\{\begin{array}{c}{x}_c=m_{1,0}/m_{0,0}\hfill \\ \multicolumn{1}{c}{x_c=m_{0,1}/m_{0,0}}\end{array}p,q=0,1,\dots$$

(8)

It is clear that the moments have the invariance properties with translation and rotation. However, this property would be affected with scale factor. ¹⁹ Given the image coordinate (x, y) and the coordinate (x′, y′) with scale factor ρ, the central moment is

$$\begin{array}{c}\multicolumn{1}{c}{{\mu }_{\mathrm{pq}}^{\prime }=\sum _{(x,y)\in R}\sum {(x\prime -x_c^{\prime })}^p(y\prime -y_c^{\prime }{)}^{q}f\prime (x\prime ,y\prime )}\\ \multicolumn{1}{c}{=\sum _{(x,y)\in R}\sum {\rho }^p{(x-x_c)}^p{\rho }^q{(y-y_c)}^{q}f(x,y)={\rho }^{p+q}{\mu }_{\mathrm{pq}}}\end{array}$$

(9)

This equation describes the relationship between the central moment $({\mu }_{\mathrm{pq}})$ and the scaled central moment $({\mu }_{\mathrm{pq}})$ . It is concluded that in discrete space the central moment are not only associated with factor ρ but relevant to the order p and q. Thus, in this article, a scale normalization method is used for eliminating the scale factor influence

$$\begin{array}{c}{M}_1=\lg \left|\frac{{\phi }_2}{{\phi }_1^2}\right|,M_2=\lg \left|\frac{{\phi }_3}{{\phi }_1^3}\right|,M_3=\lg \left|\frac{{\phi }_4}{{\phi }_1^3}\right|\hfill \\ M_4=\lg \left|\frac{{\phi }_5}{{\phi }_1^6}\right|,M_5=\lg \left|\frac{{\phi }_6}{{\phi }_1^4}\right|,M_6=\lg \left|\frac{{\phi }_7}{{\phi }_1^6}\right|\hfill \end{array}$$

(10)

where ${\phi }_1-{\phi }_7$ denote the seven Hu invariant moments.

Affine invariant moment

Affine invariant moment is derived from the algebraic invariant theory, which has the invariance characteristic with translation, shearing, scaling, and rotation. Flusser and Suk ²⁰ presented three affine moment descriptors with the invariant property

$$\begin{array}{c}{I}_1=\lg |({\mu }_{20}{\mu }_{02}-{\mu }_{11}^2)/{\mu }_{00}^4|\hfill \\ I_2=\lg |\left(\begin{array}{c}{\mu }_{30}^2{\mu }_{03}^2-6{\mu }_{30}{\mu }_{03}{\mu }_{21}{\mu }_{12}\hfill \\ \multicolumn{1}{c}{+4{\mu }_{03}{\mu }_{21}^3-3{\mu }_{12}^2{\mu }_{21}^2}\end{array}\right)/{\mu }_{00}^{10}|\hfill \\ I_3=\lg |\left(\begin{array}{c}{\mu }_{20}({\mu }_{21}{\mu }_{03}-{\mu }_{12}^2)-{\mu }_{11}({\mu }_{30}{\mu }_{03}-{\mu }_{21}{\mu }_{12})\hfill \\ \multicolumn{1}{c}{+{\mu }_{02}({\mu }_{30}{\mu }_{12}-{\mu }_{21}^2)}\end{array}\right)/{\mu }_{00}^7\hfill \end{array}$$

(11)

where ${\mu }_{\mathrm{ij}}$ is the central moment of the image, i and j are integers. Considering the features of road markings can be changed with different geometric transformations, this article introduces the affine moment to eliminate the influence of image affine transformation.

Features dataset construction

In this phase, a feature vector reflecting the difference between markings is essential for road markings recognition. Thus, this article extracts the features M₁–M₅ and I₁–I₃ with three types of road markings, including straight, straight right, and straight left markings. Then, the comparison analysis is conducted, and the results are shown in Figure 9. In which, because the difference of M₄ and I₁ is not obvious between the three types, the comparisons of them are not listed.

Figure 9.

Statistics of the features with different road marking.

From the above analysis, some important cues can be drowned. First, M₁ and M₂ are different obviously between the marking samples. Thus, they are considered the main important features for identifying process. Second, shown as features M₃, M₅, I₂, and I₃, although the markings recognition cannot be conducted with a certain feature, we can recognize the marking with the combination of all features obtained. Thus, in this article, the vector used for SVM is constructed as follows

$$V={(M_1,M_2,M_3,M_5,I_2,I_3)}^T$$

(12)

Road marking recognition based on SVM

SVM is introduced by Cortes and Vapnik ²¹ in 1995. It is a powerful method which has proved its success in many research fields ranging from freeway incident detection, ²² target recognition, ²³ bus arrival time, ²⁴ facial expression recognition, ¹³ and handwritten character recognition ¹⁴ to intrusion detection. ²⁵ Here, only a very brief introduction to SVM is described.

Given the dataset with instance label (x_i, y_i), where x_i∈Rⁿ and y∈(−1,1) ^l , the solution of SVM is the result with the following problem

$$\begin{array}{c}\underset{\omega ,b,\xi }{\min }\begin{array}{c}\frac{1}{2}\hfill \end{array}{w}^T·w+C\sum _{i=1}^l{\xi }_i\hfill \\ \multicolumn{1}{c}{\mathrm{s}.\mathrm{t}.\begin{array}{c}{y}_i[w^T·\phi \left(x_i\right)+b]\hfill \end{array}\ge 1-{\xi }_i}\\ \multicolumn{1}{c}{{\xi }_i\ge 0,i=1,2,\dots ,l}\end{array}$$

(13)

where vectors x_i is mapped into a higher dimensional space by kernel function Φ. C > 0 denotes a penalty parameter which is used for evaluating the error of instance.

With Lagrange multiplier ${\partial }_i$ introduced, the dual optimization problem is obtained^26–28

$$\begin{array}{c}\underset{\partial }{\max }\begin{array}{c}W=\hfill \end{array}\sum _{i=1}^l{\partial }_i-\frac{1}{2}\sum _{i=1}^l\sum _{i=1}^l{\partial }_i{\partial }_j{y}_i{y}_{j}K(x_i,x_j)\hfill \\ \multicolumn{1}{c}{\mathrm{s}.\mathrm{t}.\begin{array}{c}\sum _{i=1}^l{\partial }_i{y}_i=0\hfill \end{array},0\le {\partial }_i\le C,i=1,2,\dots ,l}\end{array}$$

(14)

Then, the decision function of SVM is

$$f\left(x\right)=\mathrm{sgn}(\sum _{i=1}^l{\partial }_{i}k(x_i,x)+b)$$

(15)

In this article, the presented SVM model will be trained with feature vectors extracted from image. Then, it is used for the classification process of road markings with test dataset. The structure of the presented SVM model is shown in Figure 10. In which, V₁…V_s denote the combined feature vector extracted with equation (12) in section “Features dataset construction.” All the features are mapped to the high-dimensional space through the kernel function K and classified with linear classification criteria. Finally, the decision result is given by the discriminant function f(x).

Figure 10.

Structure of the SVM model for the road marking recognition.

Based on the SVM model, radial basis function (RBF) kernel function is selected. With the grid search strategy on the training dataset, the parameters of SVM can be obtained: $(C,\gamma )$ is (512, 0.0078) and the accuracy is 97.14%. Figure 11 shows the SVM model training process with the optimized parameter results.

Figure 11.

The SVM model for marking recognition training process.

Table 1 shows the partial training and validate data in the SVM training process. The road marking recognition experiment results are shown in Table 2. It is clear that the recognition rate could reach 97% and the time consumption per frame is 0.26 s. Therefore, it indicates that the proposed method has strong potential effectiveness to be applied to the intelligent driver assistance system.

Table 1.

Training and validate dataset.

	M 1	M 2	M 3	M 5	I 2	I 3	Training set	Validate set
Straight	−0.2	−1.59	−1.19	−1.3	−1.58	−1.74	50	20
	−0.18	−2.07	−1.68	−1.78	−2.59	−3.26
	⋮	⋮	⋮	⋮
Straight with right	−0.5	−0.77	−2.59	−3.12	0.2	1.77	50	15
	−0.48	−0.9	−1.55	−1.79	3.29	2.44
	⋮	⋮	⋮	⋮
Left with right	−0.2	−0.61	−1.19	−1.29	3.72	2.57	50	15
	−0.15	−1.15	−2.69	−2.79	4.26	3.13
	⋮	⋮	⋮	⋮
Amount to	–	–	–	–	–	–	150	50

Table 2.

Experimental results.

Target	Identify results			Accuracy (%)	Average time (ms)
	Straight	Straight right	Left right
Straight	48	1	1	96	240
Straight right	0	49	1	98	260
Left right	0	2	48	96	270

Conclusion

Road marking recognition in real driving environments is an important task for the vehicle autonomous navigation and the robotic location. This article presents a road marking recognition method to reduce the impacts of environment changes such as viewpoint, brightness, or background changes. The method adopts SVM for classifying the obtained markings from each other. In which, the features vector includes four improved Hu moments and two affine inconvenience moments. Experimental results show that the proposed method for road marking recognition in natural environment has good reliability and robustness.