A novel algorithm of ship structure modeling and target identification based on point cloud for automation in bulk cargo terminals

Abstract

Achieving port automation of machinery at bulk terminals is a challenging problem due to the volatile operation environments and complexity of bulk loading compared to the situations in container terminals. In order to facilitate port automation, we present a method of hull modeling (reconstruction of hull’s structure) and operation target (cargo holds under loading) identification based on 3D point cloud collected by Laser Measurement System mounted on the ship loader. In the hull modeling algorithm, we incrementally register pairs of point clouds and reconstruct the 3D structure of bulk ship’s hull blocks in details through process of encoder data of the loader, FPFH feature matching and ICP algorithm. In the identification algorithm, we project real-time point clouds of the operation zone to spherical coordinate and transforms the 3D point clouds to 2D images for fast and reliable identification of operation target. Our method detects and complements four edges of the operation target through process of the 2D images and estimates both posture and size of operation target in the bulk terminal based on the complemented edges. Experimental trials show that our algorithm allows us to achieve the reconstruction of hull blocks and real-time identification of operation target with high accuracy and reliability.

Introduction

As the world economics develops, maritime transportation flourishes and accordingly the demand in capacity of port increases. In order to deal with the increasingly huge volume of transportation, more and more ports attach great importance to the automation at port terminals. Unmanned cargo loading system is a significant constitute of port automation. Therefore, a lot of researchers and engineers focus on developing advanced measurement systems and intelligent control systems for automated cargo loading system and a significant amount of related technologies and methods^1,2–6 have come up and been applied to facilitate port automation at container terminals. However, compared to rapid development of automated loading system for container ships, the degree of automation in bulk terminals is still low and develops slowly due to the complexity of bulk cargo loading and volatile operation environments. The specification of cargo containers are standard, while shapes of bulk hatches vary a lot according to different structures and capacity of bulk ships. Thus, it is challenging to design one automatic loading system which can undertake the loading task for all the bulk ships with different structures. Currently, bulk ship loading still highly relies on manual operation of loading machine drivers, hatch conductors and loading instructors in most of bulk terminals. Two significant problems need to be solved in order to achieve automated loading operation at bulk terminals and so far, researchers have done some exploratory and enlightening work.

Firstly, a reliable measurement system and an algorithm of bulk ship modeling are necessary for acquiring 3D hull structure of bulk ships, especially the detailed structure of hulls under operation. The main stream of sensing modules for measuring ships structure includes 3D depth cameras and lidars, which describes 3D structure of entities in terms of point cloud while the modeling algorithms are varied.^2–8 As cargo ships are often large and the horizon of those sensors have limits, sensors often need to scan different pieces of cargo ships and then put the sectional scanned point clouds together to model the structure of the whole hull blocks. In Guan et al., ⁹ present an automatic registration method of hull blocks based on given CAD model of ships. They use principal component analysis (PCA) for rough registration and then refine the result through iterative closest point (ICP) algorithm. Their method works efficiently but it requires ships’ CAD model as reference to register those sectional measured point clouds, which makes it inapplicable when ships CAD models are not accessible due to common reasons like confidentiality. In Xi-Liang et al., ⁷ present a reconstruction method based on a single-wire Laser Measurement System (LMS) and a single axis rotating servo mechanism. The measurement system is mounted on the loader above the middle of cargo ship. The servo mechanism drives the LMS rotating so that LMS can scan from one end of ships to the other, as the position of the LMS is fixed in 3D space and it can acquire point cloud of only one sector once a scan. Then, the hull structure can be built by combining point clouds of different slices of ships cross sector collected by the LMS with different directions. However, the drawback of this method lies in that it cannot acquire complete structure of hull blocks especially the ships hatches because the LMS undertakes the measurement from only one position and thus the edges of hatches can be easily sheltered by the lid of hatches. As a result, the effectiveness and accuracy of the method in recognition of bulk cargo holds are undermined. In Mi et al., ⁸ use four single-wire LMS mounted on the loader above bulk ships. Compared with the method in Xi-Liang et al., ⁷ instead of using only one LMS fixed at one position, they use four LMS scanning linearly from above ship’s bow to stern and merge point clouds measured by multiple lidars at different positions and rebuild ship’s structure by registration of point clouds. However, they use the ratio of loaders moving speed to the LMSs scanning frequency as interval between coordinates of point clouds of adjacent cross sections and register the adjacent point clouds directly with the estimated interval. Using the roughly estimated interval as final transformation for registration can result in inescapable errors in ship’s reconstruction. That’s because the instant scanning frequency of LMS can often be different with the designated value and the estimation of the loaders speed also has errors. Whats more, the method does not consider ship’s movement caused by tide or other outer forces, which also jeopardizes the accuracy of reconstruction modeling.

Second problem to be solved is the real-time identification of operation targets. If the position and size of bulk cargo holds within the operation zone are identified in real time, it would greatly facilitate the loader to locate and load bulk materials into holds automatically. In Xi-Liang et al., ⁷ achieve the identification by transforming the 3D point cloud into 2D image and processing the image for identification of cargo holds. They project the point cloud of hull blocks to XOY plane (considered as the horizontal plane) as an 2D image and the pixel values of the image represents the Z coordinates value of original points in the point cloud. They distinguish zone in the deck from zone in the holds by setting a height threshold of pixel values, as they assume that Z coordinate values of points in the holds are always lower than the height of coastline while the values of points in the deck are higher. Then, they use a seed rectangle which starts in the corner of cargo hold and grows until it reaches borders of the two different zones mentioned above and the grown seeding rectangle is recognized as the operation target. In Mi et al., ⁸ apply similar projection method but use a different recognition algorithm. They project the 2D images into both X and Y coordinates and record how many pixels of points in the deck zone are at each X and Y coordinate value respectively. By setting a threshold on the number of pixels of points in the deck zone, their algorithm can detect the rising edges and falling edges of the 1D pixel value, and thus the position of borders between hold zones and deck zones are estimated and the position and size of holds under operation are identified. Methods in Refs.^7,8 work efficiently and provides enlightening thoughts of achieving automation in bulk terminals. However, their identification algorithms are not technically real-time. In order for the dynamic identification, their systems have to scan the whole hull blocks every time before the identification. That’s because their identification algorithms are completely based on the knowledge of ship structure of the whole hull blocks. What’s more, their methods are based on several unwarranted assumptions, which makes it difficult to apply those methods in the volatile operating conditions of real cases:

1. The plane of bulk hold’s hatch is assumed parallel to XOY plane in lidar’s coordinate. However, a lot of factors in real operation cases like tide and the uneven distribution of cargo materials in the hold can makes this assumption invalid and thus cause distortion of hatchs size and position in the projection of the 3D structure to the XOY plane;

2. It is assumed that the height of the point cloud in the hold zone is significantly lower than the cost line and the deck, which is not necessarily true when the holds are highly loaded with a huge volume of materials;

3. The borders of hold hatches is assumed parallel or perpendicular to the X coordinate. But such a condition isnt met when bulk ship berths with its main axis not strictly parallel with the coastline of the port.

In summary, in order to facilitate the automation at bulk cargo terminals, it is necessary to reconstruct the 3D structure of hull blocks and then achieve identification of operation targets in real time. Current solutions to part of these two problems provide enlightening insights and exploratory examples but have disadvantages in terms of practicability, accuracy and capabilities of adapting to volatile operating environments in bulk terminals. The main contribution of this paper is concluded as follows:

1. A moving scan and hull modeling system is established, which can reconstruct structure of the whole hulls in an accurate and integrated way compared with existing techniques.

2. A novel algorithm of identifying operation targets is presented, which can achieve the real-time recognition of bulk hatches with high accuracy in volatile operating situations.

Hull reconstruction system

This section describes our laser measurement system which can scans cargo ship from ship’s stern to bow and capture 3D structure information of the whole hull blocks and our reconstruction algorithm which can merge all the 3D information to rebuild the 3D model of the whole ship.

Moving-scan lidar measurement system

Our system used a multi-wire lidar (Livox Horizon, developed by DJI) with a field of view (FOV) of 81.7° horizontally and 25.1° vertically. Compared with the single-wire LMS used in Refs.,^7,8 the lidar with large FOV allows us to collect information of hull structure with more details and higher resolution. The lidar is mounted on the loader, facing toward berth of bulk cargo ship but not vertically. In order to measure the structure of the whole hull blocks, the lidar with loader moves straightly from above ships stern to the bow at a uniform speed (0.5 m/s in our experiment) along the rail on bulk terminal. The lidar scans 10 times a second, which is enough to collect details of structure of the operation targets for identification. The system is shown in Figure 1, with the red rail, green loader, blue bulk ship, purple lidar and its pink FOV.

Figure 1.

Moving-scan lidar measurement system.

Hull reconstruction algorithm

The output of the above measurement system consists of point cloud streams in 10 frame per second. Among the streams, two timely adjacent point clouds are also spatially adjacent. However, coordinates of the two are not coincident due to the movement of lidar and change in ships posture when scanning. In order to acquire the structure information of the whole ship, we incrementally register pairs of point clouds to the coordinate of the point cloud of the first frame.

To achieve that purpose, we register each pair of adjacent frames, calculate the transformation matrix of coordinates of the two frames. As the difference between two coordinates is mainly caused by the movement of lidar, we do the rough registration first with the estimation of translation movement based on the data of encoder on the loader which can estimate the position of the loader.

$$\begin{array}{cc}{R}_i^{i-1}\hfill & ={R_{\mathrm{encoder}}}_i^{i-1}\hfill \\ \multicolumn{1}{c}{{R_{\mathrm{encoder}}}_i^{i-1}}& =\left(\begin{array}{cccc}1\hfill & 0\hfill & 0\hfill & -{x_{\mathrm{encoder}}}_i^{i-1}\hfill \\ \multicolumn{1}{c}{0}& 1\hfill & 0\hfill & -{y_{\mathrm{encoder}}}_i^{i-1}\hfill \\ \multicolumn{1}{c}{0}& 0\hfill & 1\hfill & -{z_{\mathrm{encoder}}}_i^{i-1}\hfill \\ \multicolumn{1}{c}{0}& 0\hfill & 0\hfill & 1\hfill \end{array}\right)\hfill \end{array}$$

(1)

$$P_i\prime =R_i^{i-1}\times P_i$$

(2)

$R_i^{i-1}$ is the rough transformation matrix from coordinate of $i^{\mathrm{th}}$ frame to coordinate of $(i-1{)}^{\mathrm{th}}$ frame. ${x_{\mathrm{encoder}}}_i^{i-1}$ , ${y_{\mathrm{encoder}}}_i^{i-1}$ , ${z_{\mathrm{encoder}}}_i^{i-1}$ are the linear movement of lidar from $(i-1{)}^{\mathrm{th}}$ frame to $i^{\mathrm{th}}$ frame in the directions of global $X$ , $Y$ , $Z$ coordinates recorded by the encoder in the loader. $P_i$ is the original point cloud of hull block at $i^{\mathrm{th}}$ frame and $P_i\prime$ is the point cloud transformed from $P_i$ at the roughly registered $(i-1{)}^{\mathrm{th}}$ frame, as shown in Figure 2.

Figure 2.

$P_i\prime$ (purple) and $P_{i-1}$ point clouds (blue).

Though the roughly registered point cloud is closely coincident with the former-frame point cloud, as shown in Figure 2, due to the error of data in the encoder and the potential movement of ship during scanning, rough registration is not precise enough to reconstruct the structure of hull blocks of high accuracy: such small errors can accumulate when the transformation matrixes of adjacent frames are multiplied and result in great divergence between point clouds of same part of hull block. Therefore we do the refined registration. With the special scanning mode, Livox Horizon lidar produce the point clouds of the same part of the hull structure with different points and point density at different scanning positions. And thus we need to use the Fast Point Feature Histograms (FPFH) ¹⁰ for feature matching in order to acquire refined transformation matrix between $P_i\prime$ and $P_{i-1}$ .

As the calculation of FPFH is computationally expensive, we only calculate the FPFH of key points of those adjacent point clouds. In order to choose key points, we calculate the normal of each point $p_m$ in point cloud:

$$N_n^{p_m}=\{p_j(x_j,y_j,z_j)|j=1,2,\dots ,n\}$$

(3)

$N_n^{p_m}$ is the set of $n$ points neighbored to point $p_m$ in point cloud. Then we calculate the covariance matrix based on the neighbor set:

$$\begin{array}{c}\mathit{C}=\frac{1}{n}\sum _{j=1}^n({\mathit{p}}_{\mathbf{j}}-{\overline{\mathit{p}}}_{\mathbf{m}})·({\mathit{p}}_{\mathbf{j}}-{\overline{\mathit{p}}}_{\mathbf{m}}{)}^T,\\ \mathit{C}·{\mathit{v}}_{\mathit{k}}={\lambda }_k{\mathit{v}}_{\mathit{k}},k=1,2,3\end{array}$$

(4)

$\mathit{C}$ is the covariance matrix, ${\overline{\mathit{p}}}_{\mathit{m}}$ is coordinates of the centroid of $N_n^{p_m}$ , ${\mathit{v}}_{\mathit{k}}$ is the eigenvector, and ${\lambda }_k$ is the corresponding eigenvalue.

The estimated normal at $p_m$ is as follows:

$${\mathit{n}}_{\mathit{m}}=\arg \underset{{\mathit{v}}_{\mathit{k}}}{min}|{\lambda }_k|,k=1,2,3$$

(5)

By setting a $\mathrm{Threshold}1$ in the norm of normal of points, we select key points from the adjacent point clouds for refined matching:

$${P_i}_{\mathrm{key}}^{\prime }=\{p_m\in P_i\prime ||{\mathit{n}}_{\mathit{m}}|>\mathrm{Threshold}1\}$$

(6)

$${P_{(i-1)}}_{\mathrm{key}}=\{p_m\in P_{i-1}||{\mathit{n}}_{\mathit{m}}|>\mathrm{Threshold}1\}$$

(7)

Then we calculate FPFH of each point of ${P_i}_{\mathrm{key}}^{\prime }$ and ${P_{(i-1)}}_{\mathrm{key}}$ and match them for refined transformation matrix. For more details about FPFH matching, please refer to Ref. ¹⁰

$$R_i^{i-1^{\prime }}=\mathrm{FPFH}\_\mathrm{MATCH}({P_i}_{\mathrm{key}}^{\prime },{P_{(i-1)}}_{\mathrm{key}})$$

(8)

$$P_i\prime \prime =R_i^{i-1^{\prime }}\times P_i^{\prime }$$

(9)

To further refine the registration, we use ICP algorithm ¹¹ to acquire further refined transformation matrix:

$$R_i^{i-1^{″}}=\mathrm{ICP}({P_i}^{\prime \prime },P_{i-1})$$

(10)

Through above algorithm, we have the final transformation matrix of adjacent frames:

$$R_i^{i-1\mathrm{final}}=R_i^{i-1}\times R_i^{i-1^{\prime }}\times R_i^{i-1^{″}}$$

(11)

$$R_i^1=\underset{k=2}{\overset{i}{\Pi }}{R}_k^{k-1\mathrm{final}}$$

(12)

By multiplying those adjacent matrixes from $R_i^{i-1}$ to $R_2^1$ , we acquire the transformation matrix of coordinate of each frame to the first frame $R_i^1$ and through those matrix, we incrementally register point clouds of all frames to the coordinate of first frame and thus acquire the whole structure of bulk ship $P_{\mathrm{total}}$ :

$$P_{\mathrm{total}}=P_1+\sum _{k=2}^i{R}_k^{k-1\mathrm{final}}\times P_i$$

(13)

Figure 3 show the whole structure of a bulk ship registered by our algorithm. As shown in the figure, the ship’s bow is only partially reconstructed, and that’s because of the limit of lidar’s FOV and range of loader’s movement. However, it does not affect the identification of operation targets since all the structure bulk holds is scanned and rebuilt.

Figure 3.

Example of our reconstruction algorithm.

Identification of operation targets

This section describes our algorithm of identifying the operation targets based on the real-time point clouds of operation zones and the bulk cargo ships 3D structure built above. Once the reconstruction of the hull blocks structure is finished, the LMS moves to operation zones and collects real-time point clouds of those zones. By processing those point cloud streams and using the built 3D structure a reference, our algorithm identifies the size and posture of operation targets and thus the slip barrel in the loader can be guided to load bulk materials into the ships bulk holds automatically. In our framework, we apply classical image processing methods instead of machine learning method because as mentioned before, bulk terminals have to load various kinds of bulk cargo ships with different structures and there is not enough qualified dataset for us to build up some framework of artificial Intelligence for target identification of all those different kinds of ships.

Dimensionality reduction of 3D point cloud

Identification of operation targets relies highly on detection of four edges of the bulk ships hatch which is rectangular. Meanwhile, as points of the 3D point cloud are not well structured spatially like 2D images, detecting the four edges directly from the point cloud is difficult and computationally expensive. Thus, the 3D point clouds need to be transformed to 2D images for fast and precise target identification in real time. It is a natural thought to project the point cloud to the deck or horizontal plane, as in Refs.^7,8 However, as the posture of ships can change due to tide and uneven distribution of materials in the holds, the deck plane is often not coincident to the horizontal plane and the plane coefficients are difficult to be estimated. Thus, we use the method presented in Steder et al. ¹² Instead of plane projection, we project the point cloud to spherical coordinate and transform it to a 2D image.

For points in 3D space, their coordinates are defined as $p_m=(x,y,z)$ in Cartesian Coordinate System and $p_m=(\theta ,\phi ,r)$ in Spherical Coordinate System. And transformation formulation from Cartesian Coordinate System to Spherical Coordinate System is as follows:

$$\begin{array}{c}r=\sqrt{x^2+y^2+z^2}\hfill \\ \multicolumn{1}{c}{\theta =\arctan \frac{y}{x}}\\ \multicolumn{1}{c}{\phi =\arcsin \frac{z}{r}}\\ \multicolumn{1}{c}{\theta \in (-\frac{\mathrm{\Theta }}{2},\frac{\mathrm{\Theta }}{2}),\phi \in (-\frac{\mathrm{\Phi }}{2},\frac{\mathrm{\Phi }}{2})}\end{array}$$

(14)

$\mathrm{\Theta }$ and $\mathrm{\Phi }$ are the horizontal and vertical FOV of the lidar respectively.

The angular resolution is set as $(\mathrm{\Delta }\mathrm{\Theta },\mathrm{\Delta }\mathrm{\Phi })$ , and the resolution of transformed 2D image is $\frac{\mathrm{\Theta }}{\mathrm{\Delta }\mathrm{\Theta }}\times \frac{\mathrm{\Phi }}{\mathrm{\Delta }\mathrm{\Phi }}$ . We use $r$ as pixel value of the 2D image. And thus the transformation formulation from 3D point cloud to 2D range image with single-channel pixel value is as follows:

$$\begin{array}{c}\multicolumn{1}{c}{{\mathrm{row}}_m=\frac{1}{\mathrm{\Delta }\mathrm{\Phi }}(\arcsin \frac{z}{r}+\frac{\mathrm{\Phi }}{2})}\\ \multicolumn{1}{c}{{\mathrm{col}}_m=\frac{1}{\mathrm{\Delta }\mathrm{\Theta }}(\arctan \frac{y}{x}+\frac{\mathrm{\Theta }}{2})}\\ \multicolumn{1}{c}{f({\mathrm{row}}_m,{\mathrm{col}}_m)={\mathrm{pixel}}_m=r}\end{array}$$

(15)

The advantage of spherical projection lines in that compared to plane projection, no matter what posture of bulk ship relative to lidar is, points of point cloud do not overlap and information of shup’s structure will not be lost during project. Figure 4 show a frame of point cloud of the operation zone and its projected 2D image.

Figure 4.

Example of projected.

Identification of operation targets

As the operation targets are rectangular, the core of identifying those targets lies in detection of four edges of the rectangle. To achieve that purpose, our algorithm convolves the 2D images with Canny edge detect filter.

Firstly, we apply Gaussian filter for denoizing:

$$\begin{array}{c}\multicolumn{1}{c}{f_g(\mathrm{row},\mathrm{col})=f(\mathrm{row},\mathrm{col})*K,}\\ \multicolumn{1}{c}{K=\frac{1}{2\pi {\sigma }^2}{e}^{-\frac{x^2+y^2}{2{\sigma }^2}}}\end{array}$$

(16)

Then the gradient of the filtered image $f_g(\mathrm{row},\mathrm{col})$ is calculated:

$$\begin{array}{c}{G}_{\mathrm{row}}=\left(\begin{array}{ccc}-1\hfill & 0\hfill & +1\hfill \\ \multicolumn{1}{c}{-2}& 0\hfill & +2\hfill \\ \multicolumn{1}{c}{-1}& 0\hfill & +1\hfill \end{array}\right)*f_g(\mathrm{row},\mathrm{col}),\hfill \\ \multicolumn{1}{c}{G_{\mathrm{col}}=\left(\begin{array}{ccc}+1\hfill & +2\hfill & +1\hfill \\ \multicolumn{1}{c}{0}& 0\hfill & 0\hfill \\ \multicolumn{1}{c}{-1}& -2\hfill & -1\hfill \end{array}\right)*f_g(\mathrm{row},\mathrm{col})}\end{array}$$

(17)

$G_{\mathrm{row}},G_{\mathrm{col}}$ are the components of pixel gradients in $\mathrm{row}$ coordinate and in $\mathrm{col}$ coordinate respectively. And thus we can acquire the norms and directions of image’s pixel gradients:

$$\begin{array}{c}|\mathit{G}(\mathrm{row},\mathrm{col})|=\sqrt{{G_{\mathrm{row}}}^2+{G_{\mathrm{col}}}^2}\hfill \\ \alpha =\arctan \frac{G_{\mathrm{row}}}{G_{\mathrm{col}}}\hfill \end{array}$$

(18)

Then we apply Non-Maximum Suppression to $\mathit{G}(\mathrm{row},\mathrm{col})$ . As shown in Figure 5, according to the direction of ${\mathit{G}}_{\mathit{m}}$ , the gradient of $p_m$ , we search the neighbored gradients $G_{\mathrm{temp}1}$ and $G_{\mathrm{temp}2}$ which are calculated through linear interpolation:

$$\begin{array}{c}\multicolumn{1}{c}{G_{\mathrm{temp}1}=\mathrm{weight}\times G_{g1}+(1-\mathrm{weight})\times G_{g2}}\\ \multicolumn{1}{c}{G_{\mathrm{temp}2}=\mathrm{weight}\times G_{g5}+(1-\mathrm{weight})\times G_{g6}}\\ \multicolumn{1}{c}{\mathrm{weight}=\frac{d1}{d}}\end{array}$$

(19)

After that, we apply Non-Maximum Suppression for refined edge detection:

$$\begin{array}{c}{\mathit{G}}_{\mathrm{nms}}(\mathrm{row},\mathrm{col})=\hfill \\ \{\begin{array}{c}\mathit{G}(\mathrm{row},\mathrm{col}),|\mathit{G}(\mathrm{row},\mathrm{col})|=\max \{|\mathit{G}(\mathrm{row},\mathrm{col})|,\hfill \\ \multicolumn{1}{c}{|{\mathit{G}}_{\mathrm{temp}1}(\mathrm{row},\mathrm{col})|,|{\mathit{G}}_{\mathrm{temp}2}(\mathrm{row},\mathrm{col})|\}}\\ \multicolumn{1}{c}{0,\mathrm{else}}\end{array}\hfill \end{array}$$

(20)

By setting a $\mathrm{Threoshold}2$ on the gradient map ${\mathit{G}}_{\mathrm{nms}}(\mathrm{row},\mathrm{col})$ , we apply binarization to $f(\mathrm{row},\mathrm{col})$ and obtain the 2D image $f_e(\mathrm{row},\mathrm{col})$ with only edge features:

$$f_e(\mathrm{row},\mathrm{col})=\{\begin{array}{c}1,|{\mathit{G}}_{\mathrm{nms}}(\mathrm{row},\mathrm{col})|>\mathrm{Threoshold}2\hfill \\ \multicolumn{1}{c}{0,\mathrm{else}}\end{array}$$

(21)

Then, we use the method presented in Suzuki and Abe ¹³ and extract the edges of operation target based on $f_e(\mathrm{row},\mathrm{col})$ . This result in two contour and we choose the inner one. Given the fact the straight edges in point cloud remains straight after projection to 2D image, the rectangle made up of the four straight edges becomes parallelogram after projection. The minimum enclosing rectangle of the parallelogram is calculated and four sides of parallelogram can be separately identified based on the distance from points of parallelogram to the different sides of the minimum enclosing rectangle. As there is a one-to-one match between pixels of the 2D images and points of the 3D point cloud, the four sides of parallelogram are remapped to the corresponding point clouds of four edges of the hatch in the operation zone, as shown in Figure 6.

Figure 5.

Non-maximum suppression.

Figure 6.

Point cloud of four edges of operation target.

With the Sample Consensus Function of PCL Library, we calculate parameters of corresponding four straight line as the edges of hatch based on the identified four edges of the point cloud:

$$\begin{array}{c}\underset{\mathrm{coefficients}\mathrm{of}{\mathrm{line}}_k}{min}\sum _{p_m}d({\mathrm{line}}_k,p_m)\hfill \\ \mathbf{s}\mathbf{.}\mathbf{t}\mathbf{.}\hfill \\ d({\mathrm{line}}_k,p_m)=\mathrm{distance}\mathrm{between}{\mathrm{line}}_k\mathrm{and}{p}_m\hfill \\ d({\mathrm{line}}_k,p_m)\le \mathrm{Threshold}3\hfill \\ p_m\in P_e^k\hfill \end{array}$$

(22)

$P_e^k(k=1,2,3,4)$ represents the point clouds of four detected edges.

For real-time identification of operation targets, our algorithm only process point cloud of one frame once a time, and thus the points of hatchs edge detected in real time are not dense enough for accurate fitting of edge lines. To solve this problem, we use the 3D point cloud of the whole hull blocks built in former section as reference to complement the four edges point clouds and redo the straight line fitting in order to refine our estimation. To be specific, we build a KdTree with the point cloud of whole hull blocks as input, and search the KdTree for points neighbored to the four fitted straight lines and then add those neighbored points to the point clouds of four edges respectively:

$$\begin{array}{c}\multicolumn{1}{c}{P_e^{k^{\prime }}=P_e^k+\mathrm{KdTree}(P_{\mathrm{total}}).\mathrm{NearestSearch}((x,y,z)),}\\ \multicolumn{1}{c}{x_{\min }\le x\le x_{\max }}\\ \multicolumn{1}{c}{y_{\min }\le y\le y_{\max }}\\ \multicolumn{1}{c}{z_{\min }\le z\le z_{\max }}\end{array}$$

(23)

$x_{\min },x_{\max },y_{\min },y_{\max },z_{\min },z_{\max }$ are the range of hatch under operation calculated through the four fitted lines. By doing so, we acquire four new point clouds which include complemented points of edges $P_e^{k\prime }$ , as shown in Figure 7. From the figure, we can see that the point clouds of edges are much denser than those in Figure 6, which can improve the fitting accuracy.

Figure 7.

Complete point cloud of four edges of operation target.

We redo the fit based on complemented point clouds $P_e^{k\prime }$ and obtain refined edge lines, as shown in Figure 8.

$$\begin{array}{c}\underset{\mathrm{coefficients}\mathrm{of}{\mathrm{line}}_k^{\prime }}{min}\sum _{p_m}d({\mathrm{line}}_k^{\prime },p_m)\hfill \\ \mathbf{s}\mathbf{.}\mathbf{t}\mathbf{.}\hfill \\ d({\mathrm{line}}_k^{\prime },p_m)=\mathrm{distance}\mathrm{between}{\mathrm{line}}_k^{\prime }\hfill \\ \multicolumn{1}{c}{\mathrm{and}{p}_m}\\ d({\mathrm{line}}_k^{\prime },p_m)\le \mathrm{Threshold}3\hfill \\ p_m\in P_e^{k^{\prime }}\hfill \end{array}$$

(24)

Figure 8.

Refined edge lines.

Experiment

Experiment for our hull reconstruction algorithm and target identification algorithm is implemented on the ship loader at Tian Jian’s Shen Huang Coal Terminal. The lidar is mounted on the load as shown in Figure 9. The speed of moving scan system is $0.5m/s$ . A laptop with $\mathrm{IntelCorei}7-8750\mathrm{KCPU}$ is used for computation.

Figure 9.

Experiment environment of the coal terminal.

The 3D structure of the whole hull blocks is shown in Figure 3, and the result of hull reconstruction’s experiment is shown in Table 1. And the result of experiment on identification algorithm of operation target is shown in Table 2.

Table 1.

The result of hull reconstruction.

Option	Overall length	Overall width
Reconstruction value	179 m	28 m
Real value	178 m	27.6 m
Error rate	0.6%	1.4%

Table 2.

The result of target identification.

Hold No.	1 1	2	3	4	5
Estimated length (m)	14.7	19.65	19.5	19.43	19.63
Real length (m)	14.4	19.2	19.2	19.2	19.2
Error rate (%)	2.1	2.3	1.6	1.2	2.2
Estimated width (m)	14.23	14.64	14.57	14.63	14.52
Real width (m)	14.4	14.4	14.4	14.4	14.4
Error rate (%)	1.2	1.7	1.2	1.6	0.8
Computation time (ms)	≤400

1

As the size of Hold NO.1 shown in the ship plan is not reliable, the “real size” of Hold NO.1 is measured manually and the “real size” of other holds are from the ship plan.

As shown in the Tables 1 and 2, the computation time of target identification is less than $400\mathrm{ms}$ , which makes our algorithm applicable to process point cloud streams and guide the operation of loading in real time. As material loading is the main reason that the posture of bulk ships changes and the loading period takes time, change in the posture is slow (it often takes several minutes during loading before such change is enough to be detected). Thus, the rate of target identification (more than 2.5 Hz) is enough for real-time target identification.

The error in reconstruction of hull blocks is comparatively low (1.4% in width and 0.6% in length). The maximum error in target identification is 1.7% in width and 2.3% in length. In the result of target identification, the error in length estimation is generally larger than that in width estimation. The main reason of that is the estimated edge of width in the left (the left edge in purple shown in Figure 7) is deviated from the real edge not because of flaws in algorithm but because the lidar does not face straightly to the deck plane due to the operation environment, as shown in Figure 1 and thus the features of the left edge is not complete even in the rebuilt model of the whole blocks. To solve this problem, in the future, we will use multiple lidars at different angles facing toward the deck plane of cargo ships and merge point clouds collected by those lidars for identification, and then the accuracy of our target identification algorithm can be further improved.

Conclusion and future work

This paper presents a algorithm of reconstructing hull blocks of cargo ships based on a moving scan measurement system and a algorithm of identifying operation targets in real time with high accuracy. Our system and algorithm are simple and reliable. Firstly, LMS moves and scans the whole cargo ship at a uniform speed and then rebuild the 3D structure of hull blocks. Then, the LMS moves to operation zone of the ship, face to the operation targets and scan the zone constantly. Our identification algorithm processes the real-time point cloud streams for estimation the size and dynamic posture of operation target at a refreshing frequency of more than 2.5 frame per second, which makes it possible to guide the automatic ship loading in real time. The experiment results show that our system and algorithm works effectively and efficiently.

In order to further improve our system and algorithm and to apply them in real operation scenes in bulk terminals, in the future, we will do further research as follows:

1. In order to achieve higher accuracy of reconstruction of hull blocks and identification of operation targets, we plan to apply multiple lidars in measurement system with different directions toward the deck plane and to explore new algorithms of reconstruction and identification based on merged point clouds from multiple lidars.

2. As the loader switches between operation targets, obstacles like cranes and hatch lids on the deck may get in the way of the loader and result in collision. Thus, in order to facilitate the movement of the loader and avoid collision, the obstacles on the deck need to be mapped and the safe path free of obstacles for the loader can be planned.

3. The shape of bulk materials in hatches can be modeled to guide the loading system for the even distribution of materials and improving the utilization of space in the hatch.