Sage Journals: Discover world-class research

Abstract

In this article, we propose a fast method for detecting the horizon line in maritime scenarios by combining a multi-scale approach and region-of-interest detection. Recently, several methods that adopt a multi-scale approach have been proposed, because edge detection at a single is insufficient to detect all edges of various sizes. However, these methods suffer from high processing times, requiring tens of seconds to complete horizon detection. Moreover, the resolution of images captured from cameras mounted on vessels is increasing, which reduces processing speed. Using the region-of-interest is an efficient way of reducing the amount of processing information required. Thus, we explore a way to efficiently use the region-of-interest for horizon detection. The proposed method first detects the region-of-interest using a property of maritime scenes and then multi-scale edge detection is performed for edge extraction at each scale. The results are then combined to produce a single edge map. Then, Hough transform and a least-square method are sequentially used to estimate the horizon line accurately. We compared the performance of the proposed method with state-of-the-art methods using two publicly available databases, namely, Singapore Marine Dataset and buoy dataset. Experimental results show that the proposed method for region-of-interest detection reduces the processing time of horizon detection, and the accuracy with which the proposed method can identify the horizon is superior to that of state-of-the-art methods.

Keywords

Horizon detection region-of-interest multi-scale edge detection autonomous ships

Introduction

The interest in remote-controlled and autonomous ships has been increasing in the fields of marine industry and information technology. A key technology in autonomous ships is the situational awareness equipment required to safely operate and navigate. The radar equipment comprises a primary sensor system, but it is insensitive to small, non-metallic objects. Therefore, complementary sensor systems are necessary.

Electro-optical (EO) sensors can capture images of greater resolution than any sensor system. In addition, the images acquired by EO sensors can be recognized intuitively by crews or remote operators. Many video analytic methods that automatically analyze image frames to detect and identify the events have been developed in the field of computer vision. Generally, video analytics assume the existence of a fixed camera. However, video analytics in maritime scenes need to process a moving camera, because the camera is mounted on a moving vessel. Therefore, the first step of video analytics in maritime scenes is detecting the horizon, because the horizon is the important clue for registering the consecutive frames and for compensating the motion of the moving platform.¹ In addition, the result of horizon detection can be used for estimating the distance between another ship and the EO sensor^2,3 and for increasing the processing speed by reducing the search area in the maritime detection system.^2,4 Furthermore, a recent study⁵ pointed out that horizon information can reduce the false positive rate by 50% for a given detection rate of the maritime surveillance system. Likewise, horizon detection is the first step of video analytics, and the result of horizon detection greatly affects the performance of overall maritime surveillance. Therefore, there is a need for a method to accurately detect the horizon without burdening the processing time of the entire system.

Typical approaches for horizon detection in maritime scenes assume that the horizon is a straight line and include projection-based methods,^5–9 intensity analysis (including the vertical direction),^10,11 and statistical analysis of the sea and sky regions.^10,12 These methods have relatively low complexity, but they have a limitation on the accuracy of horizon detection. Recently, hybrid methods,^13,14 combining different approaches, have been proposed to increase the accuracy of horizon detection. However, these methods mainly extract edge information from a single-scale image. Relatedly, the non-stability of edge detection is a challenging issue. To solve the non-stability issue, several methods,^15–17 adopting a multi-scale approach, have been proposed to mitigate sensitivity to parameters of edge detection and to reduce the effect of noisy edges. Most previous works, using multi-scale approach, performed the edge detection and horizon estimation on each scale image, later finding the optimal solution. Thus, these methods have a disadvantage of being computationally costly. Extant methods require processing time on the order of tens of seconds; they are unsuitable for real-time processing during maritime scene analysis. However, horizon detection should be performed quickly and accurately because it is the first step in maritime scene analysis. Thus, we devise a method that reduces the computational time, while taking advantage of multi-scale edge detection.

As the resolution of the images captured from contemporary cameras increases, the amount of information to be processed to detect the horizon also increases. Recent datasets¹ for maritime scene analysis consist of full high-definition resolution images. Unfortunately, high resolution is a factor that decreases the processing speed of horizon detection. Using the region-of-interest (ROI) is an efficient way of reducing the amount of information necessary for processing. Thus, we explore a way to efficiently use ROI detection for maritime scene analysis.

Therefore, in this article, we propose a fast method for detecting the horizon line in maritime scenarios. Our approach is based on three steps. First, ROI detection is performed to reduce the processing region, using the characteristic that the color change is large around the horizontal line in maritime images. Then, multi-scale edge detection performs edge extraction at each scale, combining the results to produce a single edge map, which suppresses small noisy edges and retains the prominent edges. The initial candidate line is estimated by applying the Hough transform to combined edge images, and median filtering is applied to remove outliers. Finally, the least-square method is used to find the optimal line, using the inliers.

In summary, this article contributes the following. First, in a maritime scenario, we explore the effectiveness of ROI recognition in a horizon detection task, and we demonstrate that our proposed method of ROI recognition can improve the processing speed without sacrificing the accuracy of the horizon detection. Second, we propose a fusion method to improve the processing speed by effectively combining the edge information of various scales and we develop a high-accuracy, high-speed method for detecting the horizon. Third, unlike previous works using their own dataset for performance comparison, we use two publicly available databases for maritime scenarios, namely, Singapore Marine Dataset (SMD)¹ and buoy dataset,¹³ and compare the performance of the proposed method and that of state-of-the-art methods.

The remainder of this article is organized as follows. The “Related works” section briefly reviews related works on horizon detection in maritime scenarios. The “Proposed method” section outlines the proposed horizon detection method. The “Experimental results” section discusses the various experiments conducted and the results obtained. Finally, the “Conclusion” section concludes this article.

Related works

Several approaches have been developed and reported for detecting the horizon using EO sensors mounted on vessels. A horizontal line is generally represented as a straight line in a maritime scenario. Typical approaches for horizon detection include projection-based methods,^5–9 intensity analysis (including the vertical direction),^10,11 and statistical analysis of the sea and sky regions.^10,12

In the projection-based methods, an edge detector is first applied to generate an edge map. Then, the horizon line is determined by projecting the edges to another parametric space where the dominant line can easily be detected. Hough transform and Radon transform are the representative projection methods. These methods are most popular, because they are simple and mathematically well-defined. However, they are sensitive to prepossessing and frequently fail to detect the horizon when the horizon is not the most dominant line.

Intensity variation analysis methods first calculate the gradient map (including the vertical direction), after which they identify the maximal edges in each column of the image. Various optimization techniques, such as the least-squares method and dynamic programming, are used to calculate the optimal horizon line, using maximal edges. These methods have relatively low complexity, but their performance, in terms of accuracy, decreases when there exists a boundary having larger intensity change than the horizon, or the horizon lines are occluded by vessels.

Statistical analysis methods identify the horizon line by segmenting an image into regions containing the sea and sky. These methods are more robust to preprocessing than the methods using the gradient information at a pixel level, because they use the local statistics information. In addition, they can detect the blurred horizon. However, these methods must calculate the statistical distribution of the regions. Thus, they require more computation time.

The methods above are complementary to each other. Thus, several methods combining different approaches have been proposed to increase the accuracy of horizon detection. Fefilatyev et al.¹³ proposed a hybrid approach, combining a Hough transform and statistical analysis. They first selected a limited number of candidate lines, using the Hough transform, and calculated the separation criteria to find the optimal line, maximizing the difference between the regions containing the water and sky. The Gaussian model in RGB color space was used to represent the color distribution of each region, and the Bhattacharyya distance¹⁸ was used to measure the similarity of color distributions. Lipschutz et al.¹⁴ also proposed a method like that of Fefilatyev et al. They used a morphological filter as the pre-processing stage and calculated the histograms of both the sky and sea regions to represent the color distribution of each. These methods require statistical analysis because of the possibly large number of candidate lines. Thus, the computation time increases drastically. In addition, hybrid methods mainly extract edge information from a single-scale image, in relation to which, there is a challenging issue about the non-stability of edge detection.

Recently, horizon detection methods,^15,16 using multi-scale edge detection, have been proposed to solve the non-stability of edge detection issue. A new method, multi-scale cross-modal linear feature (MSCM-LiFe) was proposed by Prasad et al.¹⁵ MSCM-LiFe adopts the multi-scale approach for detecting edges, and two modalities (i.e. Hough transforms and intensity variation analysis) are used to estimate the candidate horizon line for each scale image. The final solution is estimated by analyzing the goodness score and geometric proximity for each candidate pair of two different methods. A novel method, multi-scale consistence of weighted edge Radon transform (MuSCoWERT), was proposed by Prasad et al.¹⁶ MuSCoWERT first generates multi-scale images by applying the median filter with different sizes. Then, for each scale image, weighted edge map is computed by analyzing the length of the edges. Radon transform is applied to estimate the parameter of the candidate line for each weighted edge map. Then, the final solution is selected by voting on the estimated line parameter from all scales. Whereas previous works showed that using the multi-scale approach can improve the accuracy of horizon detection, they require processing times on the order of tens of seconds. Therefore, we should to devise a way to reduce the processing time while maintaining the accuracy of horizon detection, because horizon detection is the first step and an essential element in maritime scene analysis.

Proposed method

In this section, we present a fast method for detecting the horizon line in maritime scenarios. A flowchart of the proposed method is shown in Figure 1. First, ROI detection, using the property of maritime scenes, is performed on an input image. The proposed method effectively reduces the processing time by restricting the search region, because the processing time of horizon detection method is proportional to the region to be processed. Then, multi-scale edge detection is applied to the ROI image, and the combined edge map is generated using edge images from various scales. Finally, the Hough transform and a least-squares method are sequentially used to find the optimal parameter of the horizon line. Although the proposed method detects the edges on each scale independently, as in previous works, it can reduce the computation time by applying the horizon parameter estimation to the combined edge map at once.

Figure 1.

Flowchart of proposed method.

ROI detection

As cameras capable of capturing high-resolution images become popular, the amount of information to be processed to detect the horizon rapidly increases. Furthermore, the horizon detection methods that adopt multi-scale edge detection require repetitive processing for a high-resolution image, and therefore, the amount of computation is increased. Thus, the ROI can help to reduce the amount of information to be processed. Many methods^19–21 for detecting ROI, based on visual saliency, have been proposed to identify the salient regions. However, applying it to detect the horizon is inadequate because the horizon is not an attended area in maritime scenes.

In maritime scenes, the sky and sea have homogeneous color regions, and the horizon is the apparent line that divides the sky and sea. Thus, the region containing the horizon has a different mean color vector than the mean color vector of sea and sky region, because it includes both the color of sky and sea regions. Using this property of maritime scenes, we propose a method for ROI detection to effectively detect the horizon. As a first step, an input image is resized smaller to reduce the processing time and smooth the regions. Then, the proposed method divides the resized image into $N$ regions horizontally with $50 %$ overlapping. For each region, the mean vectors and the covariance matrices of the color distribution of the region are calculated. The difference between the color distributions of the two consecutive regions (including the vertical direction) is calculated using the Bhattacharyya distance,¹⁸ as follows

D (R_{1}, R_{2}) = (μ_{1} - μ_{2})^{T} (Σ_{1} - Σ_{2})^{- 1} (μ_{1} - μ_{2})

(1)

where $μ$ and $Σ$ are the mean vectors and the covariance matrices of the color distribution of the region, respectively.

The proposed method chooses a region having the largest distance as the ROI for horizon detection. If the horizon line exists at the boundary of the horizontally divided region, it is difficult to determine to which of the two consecutive regions the horizontal line belongs. To mitigate the boundary effect, our proposed method divides the region into overlapping segments. We define ROI as a rectangle to improve the speed in subsequent tasks. Whereas the ROI image is defined as a rectangle, it does not contain the entire horizon, because of its large gradient. The proposed method can accurately identify the horizon using the partial line segments of horizon in ROI images, because the horizon is represented as a straight line.

The resized image and the divided horizontal regions are shown in Figure 2. In Figure 2(b), the red rectangle indicates the ROI, confirming that the average color of the region abruptly changes near the horizon.

Figure 2.

(a) Resized image and (b) horizontally divided regions and the mean color vector of each region.

Multi-scale edge detection

The scale in edge detection is related to the size of the area where brightness changes are analyzed or to the size of the smoothing filter applied before edge detection. Maritime scenes contain many edges generated by wakes from ships, sunglint, and waves. Thus, small-scale edge detection suffers from false positives. Large-scale edge detection can identify reliable edges related to the horizon, but it loses detailed structures. In addition, recent research²² pointed out that detecting edges at multiple scales can reduce the inherent ambiguity of edge detection at a single scale. Therefore, several methods,^15,16 adopting a multi-scale approach, have been proposed to mitigate sensitivity to parameters of edge detection and to reduce the effect of noisy edges.

Multi-scale edge detection is distinguishable by the method of analyzing the information detected at different scales.²³ Previous works have independently processed edge information detected on different scales, so that horizon line estimation is applied to the number of scales. This was one reason for increasing the processing time of the horizon detection method, when adopting multi-scale edge detection. In addition, MusCoWERT¹⁶ detected edges on different scales and analyzed their length, because it suffers from processing times requiring an order of tens of seconds. Therefore, the proposed method adopts the different multi-scale edge detection method, which is different from previous methods.

The resized image is used in the ROI detection step to reduce the processing time, but the ROI region of an original input image is used in the multi-scale edge detection step. The proposed method detects edges from the images by applying a smoothing filter of various sizes. Then, it synthesizes the edge images at different scales to a single edge map. Detecting a horizon by analyzing a combined edge map can reduce the inherent ambiguity of edge detection using a single scale, while increasing the processing speed of horizon detection. The proposed method uses the median filter as a smoothing filter because the median filter can effectively reduce noise, while preserving the prominent edges. The median filters are applied to an input images to generate the multi-scale images, as follows

I_{s} (x, y) = f_{s} \otimes I (x, y) = \underset{i, j \in [- 5 s, 5 s]}{median} I (x + i, y + j)

(2)

where $\otimes$ is the convolution operator, and $s$ is the scale of the median filter. The proposed method uses the median filter with three different scales.

The Canny edge detector²⁴ is applied to the multi-scale images independently. Then, the weighted edge map is synthesized using the edge maps to which the Canny edge detector is applied, as follows

W (x, y) = \sum_{s = 1}^{N} w_{s} \cdot E_{s} (x, y)

(3)

where $N$ is the number of median filters, $ω_{s}$ is the weight of the scale $s$ , and $E_{s}$ is the edge maps of the scale $s$ . In the experiments conducted, we set the same weight for all edge maps.

The edges related to the horizon were consistently detected on edge maps at various scales because of the strong brightness changes near the horizon. Thus, the proposed method applies thresholding to the weighted edge map to suppress noisy edges, while keeping the edges associated with the horizon. The thresholding to the weight edge maps is applied as follows

W_{T} (x, y) = {\begin{matrix} 255, & if W (x, y) \geq t \\ 0, & if W (x, y) < t \end{matrix}

(4)

where $t$ is the threshold for filtering the noisy edges; we set the threshold to 170.

The example images of the edge maps generated from the multi-scale images and the weighted edge map applying the thresholding are shown in Figure 3. To improve the readability of the edge maps in Figure 3, a dilation filter with a 5 × 5 rectangular structuring element is first applied to the edge maps; then, image inversion is applied. The dilation filter is used to increase the visibility of the reader and does not affect the edge detection results.

Figure 3.

Multi-scale edge detection: (a) ROI image; (b) edges images with different-sized median filters ( $s = 1$ , $s = 2$ , $s = 3$ ); and (c) weighted edge map with thresholding.

Figure 3 shows that the proposed multi-scale edge detection can preserve the edges associated with the horizon while suppressing the noisy edges. In Figure 3, the proposed method reduces edges unrelated to the horizon, but there still exist outlier edges. Therefore, a method is necessary to reduce the effect of outlier edges when estimating the horizon line.

Horizon line estimation

The representative methods for estimating the horizon from the edge image are the Hough transform²⁵ and the least-squares method. The method using Hough transform can robustly estimate the parameter of horizon, even when there are small numbers of edges related to the horizon or there are many noisy edges. Whereas the least-square method is highly affected by outlier edges, it can more accurately estimate the horizon when there are few outlier edges. A recent study showed that the least-squares method better accurately estimates the parameter of the horizon than the method using the Hough transform.²⁶ Therefore, the proposed method applies the Hough transform and a least-squares method, sequentially, to calculate the best parameter for the horizon line.

The proposed method first applies the Hough transform to the weighted edge map, applying thresholding to find an initial candidate for the horizon. Then, it calculates the residual between the candidate horizon line and the edges for applying the median filtering. Finally, the parameters of horizon line are calculated using the least-squares method and the inlier edges. The proposed method can accurately estimate the horizon because the outlier edges are removed via median filtering.

The inlier edges and outlier edges classified by the Hough Transform and median filtering are shown in Figure 4(a). In addition, the result of horizon detection, using the least-square method, is shown in Figure 4(b). Figure 4 shows that Hough transform and median filtering can effectively remove outlier edges, which helps the least-squares method accurately estimate the horizon.

Figure 4.

(a) Edges classified by the result of Hough transform and median filtering. Red dots show the outlier edges, and green dots show the inlier edges; (b) horizon detection result (yellow line) from applying the least-square method to inlier edges.

Experimental results

Dataset

We conducted experiments on the SMD¹ and buoy dataset.¹³ SMD is the most recent database in maritime scenarios, including 39 videos with 15,376 frames. SMD consists of the onboard videos acquired by a camera mounted on moving vessels and the onshore videos acquired by a static camera installed onshore. All frames in the SMD have a resolution of 1920 × 1080 pixels and contain various conditions of ships, wakes from ships, sunglint, and waves. The buoy dataset consists of the videos acquired by a camera mounted on a floating buoy, and all frames in the buoy dataset have a resolution of 800 × 600 pixels. The buoy dataset includes a large motion change between adjacent frames.

The ground truth annotation for the horizon line and example images are shown in Figure 5. The ground truth of horizon line is represented by the vertical position, $Y$ , and the gradient of the horizon line, $α$ , at the middle position of the image. The difference between the estimated and ground truth parameter of horizon is used to measure performance. In addition, the main properties of the dataset are summarized in Table 1.

Figure 5.

(a) Ground truth annotation; (b) sample frame of onboard videos; (c) sample frame of onshore videos; and (d) sample frame of buoy videos.

Table 1.

Details of datasets.²⁷

	SMD		Buoy
	Onboard	Onshore
No. of videos	11	28	10
No. of frames	2772	12,604	996
Min(Y − mean(Y)) (pixels)	−436.30	−13.54	−281.68
Max(Y − mean(Y)) (pixels)	467.86	9.95	307.82
Standard deviation of Y (pixels)	145.10	1.52	107.98
Min(α − mean(α)) (°)	−26.34	−9.99	−15.72
Max(α − mean(α)) (°)	12.99	0.51	20.72
Standard deviation of α (°)	1.11	0.04	4.40

SMD: Singapore Marine Dataset.

Performance of ROI detection

In this section, we demonstrate the extent to which the proposed ROI detection method reduces the processing time without loss of horizon detection accuracy. In the experiment, the input image was resized to a quarter of the original image. To find the optimal parameter for ROI detection, the proposed method measured the processing time and accuracy of horizon detection for various $N$ , the number of regions horizontally divided. We implemented the proposed method using the Python and executed it on an Intel E5-1680 CPU. I/O time was excluded when measuring the processing time, and the average of the positional errors for all frames was used as an accuracy measure. When the ROI region did not contain the horizon, we set a positional error to the height of the images as a penalty.

Experimental results on SMD are shown in Figure 6. The ROI detection clearly reduced the processing time without loss of horizon detection accuracy. When comparing results using the entire frame and those using less than a quarter of the images area, the processing speed was four times faster, but the accuracy remained the same. Therefore, the proposed method divided the image into nine horizontal regions with $50 %$ overlap to select the ROI.

Figure 6.

Processing time and accuracy of horizon detection according to the ROI area on SMD.

Figure 7 shows the ROI detection results using the proposed method. Thus, it is possible to improve the processing speed by reducing the area to be processed and to improve the accuracy of horizon detection by reducing the influence on the edges generated by wakes from ships, sunglint, and waves.

Figure 7.

Sample frames of ROI detection (red rectangle) using the proposed method: (a) ROI detection from onboard frames; (b) ROI detection from onshore frames; and (c) ROI detection from buoy frames.

Horizon detection performance

In this section, we compare the performance of the proposed method and that of state-of-the-art methods, including MusCoWERT,¹⁶ MSCM-Life,¹⁵ the method of Fefilatyev et al. (FGSL),¹³ and the method of Lipschutz et al., (LHSL).¹⁴ The method using Hough transform (Hough)¹⁰ and the method based on intensity variation analysis (IntV)¹⁰ were considered for performance comparison.

Hough

This method applied the median filter to smooth the input image.¹⁰ Then, it extracted the edge image using the Canny edge detector. Then, the Hough transform was used to find the optimal parameter of horizon line from the edge map.

IntV

This method first performed the median filtering.¹⁰ Then, it searched the points having the maximum edge magnitude in each column of the smoothed image. Then, the least-square method was applied to maximal edges to find the optimal line.

FGSL

This method selected $N$ candidate horizon lines by performing the Hough transform on the edge image.¹³ Gaussian distribution was then used to model the color information of two regions separated by the candidate line. The similarity between the color distribution of the sea and the sky regions was measured using the Bhattacharyya distance. The candidate horizon line having the lowest similarity was determined to be the final solution. The number of candidate lines greatly affected the processing speed of the system. Thus, 10 candidate lines were selected for the computational efficiency.

LHSL

This method also selected $N$ candidate horizon lines, like the FGSL method.¹⁴ However, the different color model (e.g. color histogram) was used to model the color distribution of the regions. The optimal line was then selected by calculating the distance between the two histograms. In the experiment, the number of candidate lines equal to the FGSL method was selected.

MSCM-LiFe

10 vertical median filters having different sizes were used to generate the multi-scale image.¹⁵ For each scale image, Canny edge detection and the Hough transform were used to find the 10 candidate horizon lines. For each scale, the mean multi-scale images were generated by accumulating the scale image. Then, the method found the points having the maximum intensity variation in each column of the mean multi-scale images. The optimal horizon line was then selected by analyzing the goodness score and geometric proximity of pairs consisting of the candidate line of the Hough transform and the candidate line of the intensity variation approach.

MuSCoWERT

The multi-scale image was generated by applying median filters of various sizes.¹⁶ For each scale image, edge detection was performed to generate the edge map. Then, lengths were calculated for all edges to consider their weight. Then, the Radon transform was applied to each weighted edge map to select $N$ candidate horizon parameters. The optimal horizon line was then determined by voting on the candidate horizon parameters of each scale.

We measured the difference between the estimated and ground-truth parameter of horizon to compare the performance of horizon detection. The positional error of the horizon has defined as the difference between the vertical position of estimated line, $Y$ , and that of the ground truth, $Y_{GT}$ . In addition, the angular error of the horizon was used to compute the difference between the gradient of the estimated line, $α$ , and that of the ground truth, $α_{GT}$ .

We implemented the comparison methods, except for MuSCoWERT, and calculated the horizon parameters, $Y$ and $α$ , to measure performance. The statistical results from errors in parameters $Y$ and $α$ on the SMD are listed in Tables 2 and 3, where the data of MuSCoWERT were taken from their results.¹⁶ The experimental results show that the proposed method outperformed the other methods for both scenarios. The proposed method had a median positional error of less than 2 pixels from the center of the horizon and a median angular error of approximately 0.1°. Particularly, the 95th percentile error of the proposed method, as listed in Table 2, was relatively small, compared to the other methods.

Table 2.

Horizon detection, using different methods, on onboard videos from the SMD.

	$\| Y - Y_{GT} \| (pixels)$			$\| α - α_{GT} \| (\circ)$
	25th percentile	50th percentile	95th percentile	25th percentile	50th percentile	95th percentile
Proposed	0.51	1.23	3.99	0.05	0.12	0.39
MuSCoWERT¹⁶	0.54	1.49	8.17	0.06	0.25	0.88
MSCM-LiFe¹⁵	1.16	2.84	505.78	0.17	0.38	5.50
LHSL¹⁴	13.78	25.65	507.92	0.88	1.37	6.52
FGSL¹³	5.28	10.85	581.44	0.67	1.00	3.88
IntV¹⁰	13.36	24.89	498.17	0.87	1.35	6.12
Hough¹⁰	2.27	221.67	520.34	0.25	1.00	4.57

MuSCoWERT: multi-scale consistence of weighted edge Radon transform; MSCM-LiFe: multi-scale cross-modal linear feature.

Table 3.

Horizon detection, using different methods, with onshore videos from the SMD.

	$\| Y - Y_{GT} \| (pixels)$			$\| α - α_{GT} \| (\circ)$
	25th percentile	50th percentile	95th percentile	25th percentile	50th percentile	95th percentile
Proposed	0.99	2.09	12.87	0.04	0.10	0.67
MuSCoWERT¹⁶	1.14	2.63	11.41	0.14	0.21	1.07
MSCM-LiFe¹⁵	1.63	3.88	81.59	0.11	0.18	1.14
LHSL¹⁴	14.96	27.92	109.00	0.75	1.03	3.86
FGSL¹³	5.88	11.53	64.70	0.75	1.00	2.87
IntV¹⁰	2.08	5.82	39.89	0.14	0.52	5.37
Hough¹⁰	3.12	165.02	460.24	0.14	0.36	3.80

SMD: Singapore Marine Dataset; MuSCoWERT: multi-scale consistence of weighted edge Radon transform; MSCM-LiFe: multi-scale cross-modal linear feature.

The statistical results from errors in parameters $Y$ and $α$ on the buoy dataset are listed in Table 4. The proposed method had a median positional error of less than 1 pixels from the center of the horizon and a median angular error of approximately 0.15°. Furthermore, the 95th percentile error of the proposed method, as listed in Table 4, was relatively small, compared to the other methods. This implies that the proposed method could robustly detect ROI regions and accurately identify the horizon on a moving platform.

Table 4.

Horizon detection, using different methods, with buoy videos.

	$\| Y - Y_{GT} \| (pixels)$			$\| α - α_{GT} \| (\circ)$
	25th percentile	50th percentile	95th percentile	25th percentile	50th percentile	95th percentile
Proposed	0.53	1.07	2.98	0.07	0.15	0.45
MSCM-LiFe¹⁵	1.54	2.97	11.56	0.33	0.57	11.56
LHSL¹⁴	0.66	1.50	3.73	0.17	0.33	0.67
FGSL¹³	0.60	1.35	3.84	0.18	0.36	0.79
IntV¹⁰	0.84	1.91	55.06	0.14	0.32	13.24
Hough¹⁰	0.77	1.76	4.46	0.18	0.37	0.89

MSCM-LiFe: multi-scale cross-modal linear feature.

When comparing methods using multi-scale edge detection (e.g. MuSCoWERT, MSCM-LiFe, and the proposed method), to those using single scale edge detection, experimental results on SMD show that multi-scale edge detection helps increase the accuracy of the horizon detection by mitigating the instability of edge detection.

Sample horizon detection frames from the SMD and buoy dataset are displayed in Figure 8, showing that the proposed method accurately detects the horizon in maritime scenes containing great diversity. The proposed method is more robust to noisy edges, owing to wakes from ships, sunglint, and waves, because the ROI region can effectively reduce the noisy edges occurring away from the horizon. Although the proposed method has shown the reliable performance for horizon detection, the performance of the proposed method was degraded when the boundary of the horizon changed gradually. Thus, the edge could not be detected. In addition, the proposed method could not correctly identify the horizon when the number of edges constituting the horizontal line was relatively small, because the proposed method basically uses edge information.

Figure 8.

Sample frames of horizon line detection (red line) using the proposed method: (a) horizon detection from onboard frames; (b) horizon detection from onshore frames; and (c) horizon detection from buoy frames.

The average processing time-per-frame for each method is provided in Table 5. To fairly measure the time of the horizon detection methods, all methods except MuSCoWERT were implemented using Python. All methods were executed on an Intel E5-1680 CPU, and the average processing time-per-frame used the mean values from a run time of 10 measured times. The results of MuSCoWERT were taken from Prasad et al.,¹⁶ which was implemented using MATLAB 2015b and was executed on an Intel i7-3770 CPU. Conventional methods, such as IntV and Hough, had relatively fast processing times, but they had a low accuracy of horizon detection. The methods using the multi-scale approach had high computational cost, in the tens of seconds, but they showed reliable accuracy of horizon detection. In Table 5, experimental results showed that the proposed method is the only way to accurately detect the horizon at high speed.

Table 5.

Average processing time-per-frame in seconds.

	Onboard	Onshore	Buoy
Proposed	0.07	0.07	0.02
MuSCoWERT¹⁶	9.2	9.5	−
MSCM-LiFe¹⁵	6.73	6.83	2.26
LHSL¹⁴	13.75	13.76	2.30
FGSL¹³	36.58	36.63	8.61
IntV¹⁰	0.30	0.30	0.01
Hough¹⁰	0.11	0.10	0.01

MuSCoWERT: multi-scale consistence of weighted edge Radon transform; MSCM-LiFe: multi-scale cross-modal linear feature.

Conclusion

In this article, we proposed a novel approach that enables a seafaring vessel to quickly detect the horizon using a multi-scale approach with ROI detection. The ROI detection method was used to restrict the search region and to reduce the computational time using the characteristics of maritime images. In addition, unlike previous works about processing multi-scale images independently, an approach, using edge information from different scales, was combined to improve the processing speed, while taking advantage of the multi-scale approach. We conducted experiments on the SMD and buoy dataset, which are publicly available databases of maritime scenarios. The results from our experiments showed that the proposed method for ROI detection can reduce the processing time by restricting the search region without degrading the accuracy of horizon detection. In addition, we compared the performance of the proposed method with that of state-of-the-art methods, and the experimental results showed that the proposed method outperforms the other tested methods in terms of accuracy of horizon detection and processing speed. The proposed method can process a high-definition image at about 15 fps, has a median positional error of less than 2 pixels from the center of the horizon, and has a median angular error of approximately 0.15°. These results demonstrate that the proposed method is the only way to accurately detect the horizon at high speed.

Whereas the proposed method showed reliable performance for horizon detection, its performance degraded when the edges related to the horizon could not be detected. The two reasons why the edges associated with the horizon could not be detected are as follows: (1) the maritime images acquired by a camera mounted on moving vessels were blurred by the motion of the vessels. To compensate for the motion of vessels, more sophisticated filtering methods, such as mean curvature motion filtering, are necessary; and (2) the boundary between the sky and sea region changed smoothly. This is because the proposed method depended on the edge information. Thus, in the future, we plan to develop a way to detect horizons without edge information using a scene segmentation method.

Footnotes

Handling Editor: Luca Reggiani

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Chi Yoon Jeong and KyeongDeok Moon were supported by the Industrial Strategic Technology Development Program (10077551, Development of Original Technology for Artificial Intelligence Systems for Autonomous Navigating Ships) funded by the Ministry of Trade, Industry & Energy (MOTIE), Korea. Hyun S Yang was supported by the ICT R&D program of MSIP/IITP. [2017-0-00162, Development of Human-care Robot Technology for Aging Society] and National Research Foundation of Korea (NRF) grant funded by the Korean government (MSIP) (NRF-2018R1A2B6002565).

References

Prasad

Rajan

Rachmawati

et al . Video processing from electro-optical sensors for object detection and tracking in a maritime environment: a survey. IEEE T Intell Transp 2017; 18(8): 1993–2016.

van den Broek

Bouma

Degache

MAC

. Discriminating small extended targets at sea from clutter and other classes of boats in infrared and visual light imagery. ProcSPIE 2008; 6969: 10–34.

Gladstone

Moshe

Barel

et al . Distance estimation for marine vehicles using a monocular video camera. In: European signal processing conference (EUSIPCO), Budapest, Hungary, 28 August–2 September 2016, pp.2405–2409. DOI: 10.1109/EUSIPCO.2016.7760680.

Voles

Smith

AAW

Teal

MK.

Nautical scene segmentation using variable size image windows and feature space reclustering. In: European conference on computer vision, Dublin, Ireland, 26 June–1 July 2000, pp.324–335. DOI: 10.1007/3-540-45053-X_21.

Bloisi

Iocchi

Fiorini

et al . Automatic maritime surveillance with visual target detection. In: Proceedings of the international defense and homeland security simulation workshop, Rome, 12–14 September 2011, pp.141–145. http://www.dis.uniroma1.it/bloisi/papers/bloisi-maritime-surveillance.pdf

Tang

Sun

he Wang

et al . Research on infrared ship detection method in sea-sky background. ProcSPIE 2013; 8907: 8907–8910.

Zhang

Yin

Zhang

et al . A robust adaptive horizon recognizing algorithm based on projection. T I Meas Control 2011; 33(6): 734–751.

Wei

Nguyen

Ramu

et al . Automated intelligent video surveillance system for ships. Proc SPIE 2009; 7306: 7306–7312.

Alpatov

Babayan

Shubin

. Weighted radon transform for line detection in noisy images. J Electron Imaging 2015; 24(2): 023023.

10.

Gershikov

Libe

Kosolapov

. Horizon line detection in marine images: which method to choose? Int J Adv Int Sys 2013; 6(1 and 2): 79–88.

11.

Bouma

de Lange

DJJ

van den Broek

et al . Automatic detection of small surface targets with electro-optical sensors in a harbor environment. ProcSPIE 2008; 7114: 7114–7118.

12.

Ettinger

Nechyba

Ifju

et al . Vision-guided flight stability and control for micro air vehicles. In: IEEE/RSJ international conference on intelligent robots and systems, Lausanne, 30 September–4 October 2002, pp.2134–2140. DOI: 10.1109/IRDS.2002.1041582.

13.

Fefilatyev

Goldgof

Shreve

et al . Detection and tracking of ships in open sea with rapidly moving buoy-mounted camera system. Ocean Eng 2012; 54(Suppl. C): 1–12.

14.

Lipschutz

Gershikov

Milgrom

. New methods for horizon line detection in infrared and visible sea images. Ocean Eng 2013; 3(8): 226–233.

15.

Prasad

Rajan

Prasath

et al . MSCM-life: multi-scale cross modal linear feature for horizon detection in maritime images. In: IEEE region 10 conference (TENCON), Singapore, 22–25 November 2016, pp.1366–1370. DOI: 10.1109/TENCON.2016.7848237.

16.

Prasad

Rajan

Rachmawati

et al . MuSCoWERT: multi-scale consistence of weighted edge radon transform for horizon detection in maritime images. J Opt Soc Am A 2016; 33: 2491.

17.

Liu

Liang

Zheng

et al . Ship infrared image edge detection based on an improved adaptive canny algorithm. Int J Distrib Sens N 2018; 14(3). DOI: 10.1177/1550147718764639.

18.

Bhattacharyya

. On a measure of divergence between two multinomial populations. Sankhya 1946; 7(4): 401–406.

19.

Cheng

Mitra

Huang

et al . Global contrast based salient region detection. IEEE T Pattern Anal 2015; 37(3): 569–582.

20.

Yan

Liu

Chen

. Microscopy image fusion algorithm based on saliency analysis and adaptive m-pulse-coupled neural network in non-subsampled contourlet transform domain. Int J Distrib Sens N 2017; 13(6). DOI: 10.1177/1550147717711620.

21.

Jiao

Deng

. Deep combining of local phase quantization and histogram of oriented gradients for indoor positioning based on smartphone camera. Int J Distrib Sens N 2017; 13(1). DOI: 10.1177/1550147716686978.

22.

Witkin

. Scale-space filtering. In: International joint conference on artificial intelligence, Karlsruhe, 8–12 August 1983, pp.1019–1022. http://dl.acm.org/citation.cfm?id=1623516.1623607

23.

Lopez-Molina

Baets

Bustince

et al . Multiscale edge detection based on Gaussian smoothing and edge tracking. Knowl-Based Sys 2013; 44(Suppl. C): 101–111.

24.

Canny

. A computational approach to edge detection. IEEE T Pattern Anal 1986; PAMI-8(6): 679–698.

25.

Duda

Hart

. Use of the Hough transformation to detect lines and curves in pictures. Commun ACM 1972; 15(1): 11–15.

26.

Shin

. A statistical method for peak localization in Hough space by analysing butterflies. Berlin: Springer, 2014, pp.111–123.

27.

Prasad

Prasath

Rajan

et al . Challenges in video based object detection in maritime scenario using computer vision. In: International conference on connected vehicles, Zurich, Switzerland, 13–14 January 2017, pp.1–6.

Fast horizon detection in maritime images using region-of-interest

Abstract

Keywords

Introduction

Related works

Proposed method

ROI detection

Multi-scale edge detection

Horizon line estimation

Experimental results

Dataset

Performance of ROI detection

Horizon detection performance

Hough

IntV

FGSL

LHSL

MSCM-LiFe

MuSCoWERT

Conclusion

Footnotes

Declaration of conflicting interests

Funding

References