Integration of scanning and image processing algorithms for lane detection based on fuzzy method

Abstract

In order to detect lane rapidly and accurately, the integration of scanning and image processing algorithms (SIP) based on the fuzzy method is proposed. Further, combination of the proposed algorithm with an adaptive threshold value for image binarization, the least-square method and Bessel curve algorithm are proposed for detection and to fit the lane. The proposed SIP algorithm was evaluated by various tests. The experimental results indicate that the average time consumed for the detection of lane in each frame is 16.7639 ms, and the accuracy of lane detection is 95% . The proposed algorithm demonstrates good robustness, and can be used as the core algorithm for further application in lane departure warning systems.

1 Introduction

Within the last few years, statistics have indicated that traffic accidents are fa major concern around the world, and that single-vehicle road departure crashes lead to more fatalities than any other type of crash [14, 16]. Several technical solutions have been proposed, e.g., lane departure warning (LDW) systems to use computer vision and highly-specialized algorithms to detect lane markings, which would then generate a warning to the driver [12]. An unintentional lane-departure is typically due to the involuntary fade of the driver’s vision caused by use of a mobile phone, falling asleep [10], chatting, or other distractions [3]. Extraction of lane-related information is the first step in lane-departure detection. Detecting road markings is also a formidable problem. Computer vision approaches [5] can be used to perform lane edge detection, and provide a measure of distance between the vehicle and the lane edge [7, 19]. Roads can be marked by well-defined solid-line markings [18], segmented-line markings [6], circular-reflector markings, or other methods [4]. Most paved roads in China have lane marks painted in white. However, road surface environmental conditions such as glare and snow, unpredictable weather conditions and time of day, may inhibit widespread adoption.

Pilutti and Ulsoy [15] proposed a system identification approach by applying an ARX model to estimate the relationship between vehicle lateral position and the steering wheel angular position. Lee [9] proposed a feature-based lane-departure detection system using edge information to define an edge distribution function (EDF). Kwon and Lee [17] developed a lane departure warning system utilizing heuristic decision-making strategies, a time-to-lane crossing (TLC) strategy and a lateral offset (LO) strategy to improve decision-making performance. Hsiao, et al. [11] proposed an improved lane detection algorithm based on peak finding using a spatiotemporal mechanism to detect lane boundaries and then generate warning signals. Wang, Lin, and Chen [8] developed a vision-based lane detection and departure warning system which combines fuzzy rules, the fuzzy C-mean and the self-clustering algorithm (SCA) to process spatial information, and Canny algorithms to enhance lane boundary information to suit various light conditions throughout the day. López, Serrat, and Cañero [1] proposed a generic robust estimation method based on ridge orientation to extract lane features. Jiang, et al. [13] proposed a new weak lane model based on particle filtration to detect all types of lanes. Wu, Lin, and Lee [2] proposed a special-purpose lane-detection method based on a functional neurofuzzy network, used the fan-scanning-detection method to extract lane boundary information.

Almost all LDW systems follow a similar flow. First, a road model and vehicle model are proposed. Second, a sensing system is used to gather data information regarding the traveling environment of the vehicle. Third, data from the sensors are fused to estimate lane position. Finally, the vehicle model can be used to refine the estimates and determine whether to issue a warning. This paper proposes a framework for the development of a lane-detection algorithm, focused on a computer vision method combined with lane data to extract features. Features information such as edges and textures are used in combination with the road model to detect the lane position.

Much of the information about the road captured by camera when a vehicle is driving is useless to lane detection. As a result, this paper proposes an algorithm which combines the scanning area algorithm to obtaining points and the image processing algorithm to detect the lane. The algorithm obtains the efficient points of the lane line in the scanning area which the lane lays, so that the computing time of the algorithm can be greatly reduced. This paper is organized as follows. In Section 2, a lane detection algorithm is proposed, including the image binarization algorithm which can self-adaptively obtain a threshold value, the image processing algorithm which can obtain the intersection of the lane and the image edge, and the scanning algorithm. Additionally, it introduces the algorithm selection with the application of a fuzzy controller, as well as the lane fitting method. In Section 3, the experimental results and comparative studies are provided. Finally, conclusions are discussed in Section 4.

2 Lane detection

Image preprocessing is a key component to lane detection, which can greatly increase system performance. Road modeling depends on the road environment. A video camera used to identify the lane can be installed at the top right of the windscreen. The adjustment principle of the camera overlooking angle is to ensure that the horizontal middle line of the camera screen meets the vanishing line of the road far in the distance. Figure 1 shows the lane detection process used in the proposed algorithm.

2.1 Adaptive image binarization threshold algorithm

For the lane detection algorithm based on computer vision, the accuracy of the lane detection is affected by changes in weather and light intensity on the road surface. To improve the real-time performance of lane detection, binarization processing is conducted on the RGB image captured by the camera. The binarization threshold of the image must therefore be rapidly and accurately determined. For this reason, this paper proposes an adaptive image binarization threshold algorithm, as follows:

Adaptive Image Binarization Threshold Algorithm

Step 1:n = 1;//To initiate the binarization threshold value.

i = 255;//To initiate the controlling variable of the loop.

Step 2: Fori >  = 0 do;//To determine the last coordinates of the fastest falling point.

ifArray[i] >  500 then;//Storage of binarization threshold. break; endif;

i = i-1;//endfor.

Step 3:n = i+10;//Access to the fastest falling point coordinate values, then read its 10th points’ value in the array.

Step 4: forArray[n] >  50 andn <  255 do;//To obtain the correct binarization threshold.

n = n+5; end.

Effects are induced by scenery changes ahead of the vehicle, including: clouds, houses beside the road, trees, road signs, etc. The proposed algorithm is based on statistical information to adaptively obtain the image binarization threshold.

2.2 Obtain the intersection of lane and image edge

Ideally, after binarization, the lane will extend to the bottom of the image or to the margin of the broadside image. A lane depicted by a dotted line is shown in Fig. 2, in which the green portion depicts the edge area of the image.

Because the image captured by the camera represents a certain range of road environment, conditions will not be acceptable when three lane lines simultaneously intersect with the bottom edge of the image. If the four lane lines closest to the vehicle are all dotted lines, then some lanes will intersect with the image bottom, while some lanes will intersect with the image side. An algorithm is proposed to determine the intersection between the lane and the image side by scanning, expressed as follows:

Obtain the Intersection of Lane and the Image Edge Algorithm

Step 1: //On the lower part of the image block processing, as shown in Fig. 3, the b² block pattern is used to count all pixels for the binary b² block, with b representing the length of the block.

In the block, if N _{point w}  ⩾ N _{point b} , k = 1, else k = 0 (N _{point w} is the number of the white points, and N _{point b} is the number of black points). Due to the fact that the edge of the green area is only at the image edge, the number of the blocks processed by the algorithm is shown in Equation (1): $N_{\mathit{block}}=(L+2W-16)/8$(1) where L is the length of the image, and W is the width of the image. According to Equation (1) the calculation required by the algorithm is greatly reduced.

Step 2:// Scan image edges.

Step 2.1.i = 0;

Step 2.2. Scan image bottom from left to right, i = i+1 when meeting Point _w , then retreat to the state of the white line area.

Step 2.3.i = i-1/ When black spots (Point _b ) is met, the points should be saved. If δ ≤ 5 and the distance from the identified points in the previous frame is less than 5 (d≤5), repeat Step 2.2 until all the edge points on the image edge have been scanned.

Step 3:// Scan the left side of the sidebar from the bottom up.

Step 3.1. Check k = 1. In the left points in the sidebar at the bottom of the image, if k = 1, then move up five pixels, which illustrates that these points may overlap.

Step 3.2. Reaches an end if a white point is obtained in the left sidebar in the image.

Step 4:// Scan right sidebar from the bottom up.

Step 4.1. Check k = 1. In the right points, which are at the bottom sidebar of the image, if k = 1, then move up five pixels, which illustrates that these points may overlap.

Step 4.2. Reaches an end if a white point is obtained in the right sidebar of the image.

With the application of this algorithm, the intersections of the lane and the image edge can be obtained. However, when there is a strong light on the road, the result of image binarization will be inaccurate. When the lane is denoted by a dotted line, all intersections of the lane and the image edge cannot be determined. One node, at least, can be obtained with this algorithm; some nodes can be obtained in most cases. For this reason, based on the above algorithm to get the intersection of the lane and the image side by scanning, this paper further proposes an algorithm to obtain lost intersections, described as follows.

Obtaining Lost Intersections Algorithm

Step 1: Obtain two nodes, Point _G and Point _H , intersections of the lane and the image edge, by the scanning algorithm, as shown in Fig. 4.

Step 2:Line _  _BC : turn 90 degrees counterclockwise to cobtain Line _  _AC ; turn Line _  _DE 90 degrees clockwise to obtain Line _  _DF .

Step 3: As the greatest width of the lane in the image is 5/7 of the image (the area between Point _I and Point _J in Fig. 4), the vanishing point P _K of the default lane is described as follow:

The horizontal coordinate is expressed in Equation (2): $K_{\mathit{abscissa}}=L/7+45L/77$(2)

The vertical coordinate is expressed in Equation (3): $K_{\mathit{ordinate}}=W/2$(3)

Step 4: Calculate the coordinates of Point _M according to the position of Point _K and Point _H . When Point _H is mapped to Line _  _D F, the lane slope r is obtained based on the coordinate of Point _H . Then, the coordinate of Point _M can be calculated.

Step 5: Repeat Steps 1 through 4 to calculate the coordinates of ‘Point _L , Point _I , Point _J , Point _M ’.

According to the scanning algorithm, the position of ‘Point _L , Point _I , Point _J , Point _M ’ will be easily obtained by the further application of the algorithm to obtain lost nodes as long as any two nodes of ‘Point _G , Point _I , Point _J , Point _H ’ have been obtained. In the driving process, the position of ‘Point _L , Point _I , Point _J , Point _M ’ will shift left or right with vehicle movement. Once the positions of ‘Point _L , Point _I , Point _J , Point _M ’ are obtained, they can be used to calculate all intersections of lane and image edge in the next frame.

Based on the above analysis, this algorithm can be applied to calculate all intersections on the condition that some intersections of the lane and image edge were not detected as shown in Fig. 5.

In Fig. 5(a) and (b), the blue area between the two lines is the scanning area of the left lane, the light blue area between the two lines is the scanning area of right lane, the black circle is the lane points found in the scanning area, and the red circle is the points calculated by the locations of other points. In Fig. 5(a), the white circle represents the nodes between a lane of the current frame and the bottom edge of the image calculated according to the lane position in the past frame. In Fig. 5(b), the pink points represent the nodes between the lane and left edge of the image.

2.3 Fuzzy controller

This paper uses the fuzzy method to determine whether to choose the intersection algorithm by scanning area or the intersection algorithm during image processing. ‘IF X AND Y THEN Z’ fuzzy rules were used to construct a rule-based fuzzy system [2, 8]. As the image is being processed by the algorithm looking for intersections between the lane and image edge and the algorithm to obtain lost nodes, the algorithm can obtain both coordinates of ‘Point _L , Point _I , Point _J , Point _M ’ (the nodes of the lane and the image edge) and the coordinates of the lane vanishing point. In this study, the moving distance of intersection of lanes and the images was divided into three member functions by the algorithm, as follows: $\{\begin{array}{c}\hfill \mathrm{DPC}\to \mathit{Distance}\_min\to [0,2);\hfill \\ \hfill \mathrm{DPN}\to \mathit{Distance}\_\mathrm{mid}\to [2,4);\hfill \\ \hfill \mathrm{DPF}\to \mathit{Distance}\_max\to [4,5].\hfill \\ \hfill \hfill \end{array}$ where e is the distance travelled by the vanishing points in these two frames, which can be classified into three member functions, as follows: $\{\begin{array}{c}\hfill \mathrm{DVC}\to \mathit{Distance}\_min\to [0,1);\hfill \\ \hfill \mathrm{DVN}\to \mathit{Distance}\_\mathrm{mid}\to [1,2);\hfill \\ \hfill \mathrm{DVF}\to \mathit{Distance}\_max\to [2,3].\hfill \\ \hfill \hfill \end{array}$

The fuzzy controller uses the Centroid-method to defuzzify the output variable without losing all the information of fuzzy rules. The Centroid-method is described by Equation (4): $e^{*}=\sum _{i=1}^n{e}_{\mathit{ci}}\left({\omega }_i\right){\omega }_i/\sum _{i=1}^n{e}_{\mathit{ci}}\left({\omega }_i\right)$(4) where e^* is the determination of the output value, ω _i is the i-th value in the output field, and e _ci  (ω _i ) is the membership function of ω _i .

According to the fuzzy rules, the output is divided into three levels, as follows: $\{\begin{array}{c}\hfill \mathrm{SAN}\to \mathrm{To}\mathrm{choose}\mathrm{scanning}\mathrm{point}\mathrm{algorithm};\hfill \\ \hfill \mathrm{MS}\to \mathrm{To}\mathrm{choose}\mathrm{the}\mathrm{same}\mathrm{algorithm}\mathrm{applied}\hfill \\ \hfill \mathrm{in}\mathrm{former}\mathrm{frame};\hfill \\ \hfill \mathrm{CV}\to \mathrm{algorithm}\mathrm{of}\mathrm{image}\mathrm{processing}.\hfill \\ \hfill \hfill \end{array}$

As a result, nine fuzzy logic rules were established, and are listed in Table 1.

In summation, the algorithm can accurately obtain the node coordinates of the lane and the image edge location, as well as the coordinates of the lane’s vanishing points. The algorithm can further establish the corresponding image scanning area to detect the lane with the application of the obtained information.

2.4 Obtain the efficient points of the lane line in the scanning area

In this paper, the lane scanning area is divided into four categories: (1) when the left edge of the image and the lane line intersect, the scanning area is from left to right with the intersection point as a starting point; (2) when the lane and the image bottom intersect, the scanning area is from the bottom to the top, with the intersection point as a starting point; (3) when the right edge of the lane line intersects the image to the intersection as the starting point of the scanning area from right to left; (4) when the vehicle is moving on a larger curved lane, in order to facilitate the use of the Bezier algorithm to fit the curved lane, it is necessary to establish the lane scanning area in the distance.

After establishing the scanning area, the algorithm will obtain the efficient points of the lane line in the scanning area in three ways: (1) When the scanning area extends from the left to the middle or from the right to the middle, the algorithm will scan the points from the image bottom along the edge of the domain, and will stop after scanning nine efficient points. Because the scene in the lower part of the lane line is easier to observe than that in the upper part, when the lane line is on the edge, the scene on the left upper part of the lane line extends beyond the road, while the one in the right lower part of the domain area still is still in the scope of the, and the road is of a single color. As a result, this scan mode can greatly reduce disturbance of the road surface and the surrounding sceneries; (2) When the scanning area extends from the left to the middle, the algorithm will scan points from the right to the left, and will stop after scanning nine efficient points; (3) When the scanning area extends from the bottom of the image to the middle, the algorithm will stop after scanning nine efficient points.

3 Experiment

The experiment described below was conducted on flat asphalt road in the city on a clear day. In the experiment, the algorithm processing of a single-frame image takes up to 31 ms, which indicates that the algorithm demonstrates high real-time performance. The effects of the SIP algorithm is discussed: a total processing time (TT), a total number of frames (NF), the average consuming time for the detection of lane in each frame (TF), and the maximum consuming time for the detection of lane in a single frame (MaxTF). The lane detection accuracy is 95% . Most proposed lane detection algorithms have reported simply selected images of the algorithm results, but the selected images cannot be used to quantitatively compare different algorithms. Therefore, in order to measure the effectiveness of lane detection, this paper illustrates how the algorithm performs in a variety of experimental conditions. The results are listed in Table 2. The average lane detection time is described by Equation (5): $T_{\mathit{FA}}=\frac{{\sum }_{i=1}{\mathit{TF}}_i}{i}=16.7639\mathit{ms}$(5)

In this paper, the least square method is applied to fit the linear lane line as shown in Fig. 6, while the Bessel algorithm is used to fit a curvy lane line as shown in Fig. 6.

4 Conclusion

In this paper, an integrated scanning and image processing algorithms (SIP) based on the fuzzy method is proposed to detect lanes rapidly and accurately. To evaluate the performance of the SIP algorithm and to measure the effectiveness of lane detection, the algorithm was applied to various test experiments. The experimental results reveal very encouraging results in terms of the quality and efficiency of detections. There are many research directions that can be considered as useful extensions of this research work. Further work will concentrate on combining the newly developed algorithm with warning algorithms, and experiments will be conducted on cloudy or rainy days to improve the performance of the LDW system.