Sage Journals: Discover world-class research

Abstract

This paper analyzes the problems of existing container positioning methods and proposed a vision-based container position measuring system to provide precise parameters for container lifting operations. This system uses camera to get container information, then detects container corners by the method that combined with convolutional neural network and traditional image processing algorithm. This system is used to provide specific parameters associated with container lifting operation. In the first of detection, it uses the modified SSD (Single Shot MultiBox Detector) neural network to detect the coarse position of container corners in the image, second stage detection uses the usage of rectangle fitting to detect the precise position of corner holes in the coarse position. In the last step the offset distance and deflection angle were calculated by precise corner position. The experiment shows the detection rate of the proposed system reach 94%. The positioning errors between 14.3 and 19.6 mm for a frame rate of 10 fps are obtained.

Keywords

Automated container terminal container lifting operation container corner detection vision-based measurement

Introduction

Container terminals are an important node, for providing storage and distribution services for container transportation. In the 21st century, the automation of container terminals has become a very important research topic to container terminals. According to the Drewry shipping consultants and Port Equipment Manufacturers Association,¹ until mid-2019, 9.1% of all major container terminals have been automated or semi-automated. The advantages of ACT (Automated container terminal) are obvious. For example, labor costs usually can account for 33% of the total operating costs in container terminal,² but because ACT reduced a large number of on-site workers, their operating costs have been greatly reduced.

ARMG (Automated Rail-Mounted Gantry Crane) is a type of container lifting equipment, which is used to transfer containers between transportation equipment and container storage areas (such as yard blocks). And the transfer process is called container lifting operation. Container lifting operation has high requirements for reliability and accuracy, which causes difficulty in the automation of container lifting operation. Figure 1(a) shows the container lifting operations in the yard block. During this operation process, ARMG will move the spreader to the approximate area on the container first, then fine-tune the position of the spreader and connect the twist lock on the spreader with the corner holes on the container corners, this is called spreader alignment, as shown in Figure 1(b). Traditionally, spreader alignment operation is manually operated by ARMG drivers, which causes some efficiency problem.

Figure 1.

Container lifting operation: (a) container loading and uploading operation and (b) spreader position adjustment.

Container lifting operations usually take about 80 s, it including several stages:

Move RMG to a suitable position to lift the container in the yard (10 s);

Align the spreader with the container and lift the container (30 s);

Transfer the container to the truck road (10 s);

Wait for the spreader to stop swinging (5 s);

Align the container with the container truck chassis, and put down the container (30 s);

Unlock the spreader and lift it (5 s).

In this process, the spreader alignment operation of processes (ii) and (v) consumes the most time. The reason is spreader alignment operation is manually operated in most terminals, but due to the far range between driver room and spreader, the operation accuracy of manual alignment is low. Generally, once alignment requires multiple adjustments.

With the automation trend in container terminals, the automation equipment is expected to improve the operation efficiency and reduce the involved operation times, but it needs a system to measure the relative position of the spreader and the container to calculate operation parameters.

The common container positioning technology is based on laser radar,³ which positioning container by detecting the shape of container. Laser radar is not easily affected by weather and light conditions, but it is costly and lacks accuracy. Some other technology is based on computer vision, vision-based measuring systems is cheaper than laser radars, and it has better accuracy. Such device has been widely used in automated container terminal.⁴

To positioning containers, Yoon et al.⁵ proposed a container positioning method based on stereo vision, but the accuracy is still insufficient, the minimum position error has reached 60 mm. This result is because the measuring accuracy of stereo vision depends on the length of the baseline (the distance of two cameras), but the space on the lifting equipment is limited.

Traditional image processing algorithm has another problem, image information is easily affected by environmental changes. In the actual terminal, containers have many colors and different contamination conditions (such as container corrosion, damage and mud stains, etc.), and there are also unstable lighting effects in this open-air operating area, these factors will interfere with the recognition of image information. As an example, Dai et al.⁶ proposed a method of using images to locate containers in the cabin, but in order to extract container information from a complex environment, this method performs a large calculation on the whole image, the calculation time for a single photo reached 0.6 s.

For this problem, machine learning shown a new direction. Some methods that combine machine learning and traditional image algorithms have been applied.⁷ Mi et al.⁸ proposed a container corner recognition method based on SVM(Support vector machine) classifier. This method uses SVM classifier to detect containers in first, and then detects container’s precise positioning by morphological operations. This method achieves a higher recognition rate and recognition accuracy with lower processing time.

The feature recognition technology based on convolutional neural network (CNN) has been widely used in the field of industrial measurement, because it has high robustness and strong versatility.^9,10 CNN can extract higher-dimensional information in the image, it has a strong ability to eliminate light and color interference.

Kitayama et al.¹¹ proposed a container corner detection method based on SSD (Single Shot MultiBox Detector).¹² In this reported work, the detection rate of the neural network reached 94.57%, while the Intersection-Over-Union (IOU) reached 87.79%. Li et al.¹³ proposed another container corner detection method based on YOLO,¹⁴ while the IOU reached 80.43%. The accuracy of CNN based positioning method is insufficient.

Taking into account the reported solutions, this paper proposed a vision-based positioning system to measure the position of containers. It is used to provide operation parameters for the container lifting operation in RMG. The measurement principle of this system is combined CNN and traditional image processing algorithms. This paper organized as follows. Section 2 introduces the measuring system and its principles. Section 3 gives a method of calculating operation parameters. Section 4 is the experiment part.

Vision-based measuring system

The system structure of vision-based measuring system are presented in Figure 2, this system used to measure the position of containers. First it would detect container corners in images, then use those data to calculate container position data. At last, system would send container position to ACCS (Automatic Crane Control System) for control the movement of spreader.

Figure 2.

System structure of vision-based positioning system.

Image capture device

This positioning system requires some cameras to capture images on the top surface of containers. The installation of these cameras as shown in Figure 3(a). Two sets of cameras are installed on the ARMG beam and on the truck road. It is about 15 m high from the ground to avoid the movement route of spreaders. Two cameras with different views will captured images of the front and rear of container, which could eliminate the measurement error caused by the image distortion. On the other hand, the measurement system will measure at a high rate, which can effectively reduce the probability of detection failure caused by unexpected situations. Figure 3(b) shown one captured image, the image processing part use the corner holes on the container top surface to position the container, because the container corner is a standard part of the container.

Figure 3.

Installation of cameras and the image captured by cameras: (a) camera installation position and (b) captured image.

In some studies, the camera is installed on the spreader,¹⁵ but the shaking of spreader will cause some additional errors, so we install cameras on the beam of ARMG, and two sets of cameras are used to cover the entire operation area.

First stage detection based on modified SSD

The image processing part is a two-stage target detection method, then it will spend more calculation time than once detection. To reduce the detection time, we chose SSD as the first stage detector. SSD is a Convolutional Neural Network (CNN) model. The main idea of SSD is to use anchor boxes of different scales and aspect ratios to perform uniform and dense sampling at different positions of the image. Then use the CNN layer to extract features, and classify the feature information. This design made SSD faster than other traditional two-stage methods, such as R-CNN (regions with Convolutional Neural Networks).¹⁶

Figure 4 shows the basic model of the SSD-300 (input image size is 300 × 300). SSD uses the VGG-16¹⁷ as the backbone layer, and it adds several convolutional feature layers to the end of the backbone, then SSD can detect target on multiple scales. This design makes it have better detection accuracy than other one-stage method, such as YOLO.

Figure 4.

The basic model of SSD.

In order to adapt to container corner target, the SSD model needs some modifications to strengthen the recognition ability of small targets and improve the detection speed, there are two main modifications:

(i) Backbone layer change:

DSSD (Deconvolutional Single Shot Detector)¹⁸ is an improved SSD detector, it improves the shallow characterization ability by replacing the VGG-16 with a newer ResNet¹⁹ backbone. The higher depth of Resnet can save more characterization information, then improving the robustness ability for small features.

There are two types of Resnet models: ResNet-101 and ResNet-50. The latter has a faster detection speed while accuracy has slightly reduced. Based on the above description, we use ResNet-50 to replace the original VGG-16 backbone layer.

(ii) Feature layer change:

The basic SSD model has a total of six feature layers, the higher feature layer has a larger receptive field, it used to extract larger feature information, but not sensitive to small size features. The container corner belongs to that small size features in this distance. According to this, we removed two higher feature layers, Conv10_2 and Conv11_2, which are not sensitive to small size features, to improve the detection speed. The modified SSD model is shown in Figure 5.

Figure 5.

The modified SSD model.

Second stage detection based on image processing algorithm

Define the container corner position detected by SSD is:

P_{0} = [x_{0}, y_{0}, d_{0}, l_{0}]

(1)

d and l are length and width of detection result, it represents the position and extent of the corner area in the image. But the first detection maybe excluded some part of corner features from the detection result, which makes it necessary to expand the size of detection result before the second detection. The expand calculation as in (2), and a is the expand pixel length.

P_{1} = [x_{1}, y_{1}, d_{1}, l_{1}] = [x_{0}, y_{0}, d_{0} + a, l_{0} + a]

(2)

Image pre-processing

Container lifting equipment works in an open-air environment, then different container image has different color and light conditions, some typical situation is shown in Figure 6.

Figure 6.

The process of image pre-processing.

In first detection, SSD detector has excellent feature extraction capability, so these environmental differences will not significantly affect the detection results. But this is not the case for traditional image process algorithm. To the latter, it is necessary to enhance image information before morphological detection.

The process of image pre-processing is shown in Figure 7, It used to enhance and extract the feature information in the image.

Figure 7.

Lock hole image in different situations.

The first step of image pre-processing is to enhance color information by MSRCR (Multi-Scale Retinex with Color Restoration), which was first proposed by Jobson et al.,²⁰ and based on Land’s Retinex theory.²¹ MSRCR is often used in the restoration of complex illumination images.²²

First step of MSRCR calculation is calculated the enhanced color value $R_{i} (x, y)$ , as shown in (3). $I_{i} (x, y)$ is the original color value of the point with pixel coordinates $(x, y)$ on the color channel i, $F_{k} (x, y)$ ) is the Gaussian wrap, and its calculation is shown in (4), $C_{k}$ is the scale value of Gaussian wrap, it means the neighborhood size of $(x, y)$ during convolution operation.

Next step is the normalization of enhanced color value to adjust color cast, it based on the average value $Mea n_{i}$ and means square error $Vai l_{i}$ of $R_{i} (x, y)$ , as (4) and (5) shows, Dynamic is the adjustment value.

At last, the enhanced image is combined by the processed images of each color channels.

\begin{matrix} \log [R_{i} (x, y)] = \sum_{k = 1}^{N} ω_{k} (\log I_{1} (x, y) - \\ \log (F_{k} (x, y) * I_{i} (x, y))) \end{matrix}

(3)

F_{k} (x, y) = λ e^{- \frac{x^{2} + y^{2}}{c_{k}}}

(4)

\begin{matrix} Mi n_{i} = Mea n_{i} - Dynamic * Va r_{i} \\ Ma x_{i} = Mea n_{i} + Dynamic * Va r_{i} \end{matrix}

(5)

R_{i} (x, y) = \frac{R_{i} (x, y) - Mi n_{i}}{(Ma x_{i} - Mi n_{i}) * 255}

(6)

The second step is to convert pictures in the BGR (Blue Green Red) color space into HSV (Hue, Saturation, Value) color space, and thresholding it in V space. The corner hole is generally the darker area in the image, which can be extracted by the brightness thresholding, its calculation is shown in (7).

T_{V} (x, y) = {\begin{matrix} Valu e_{V}, & if HS V_{V} (x, y) > Thres h_{V}, \\ 0, & otherwise . \end{matrix}

(7)

$Valu e_{V}$ and $Thres h_{V}$ are related to $HS V_{V 0.5} (x, y)$ , which is the median value of the V channel value of the image, for adapt images in different brightness. The calculation as shown in (6), (7), and $ε_{V}$ is the adjust value obtained from experiments.

Valu e_{V} = \max (0, (1 - ε_{V}) * HS V_{V 0.5} (x, y))

(8)

Thres h_{V} = \min (255, (1 + ε_{V}) * HS V_{V 0.5} (x, y))

(9)

The last step is the binarization of the image, and the Gaussian blur is performed again to eliminate noise pixel. MSRCR enhancement, threshold segmentation and other preprocessing on the corner hole images effectively eliminate the irrelevant information in the image and retain a relatively complete corner hole area. These images can be used for corner hole fitting, which helps to improve the measurement accuracy.

Rectangle fitting

The main problem of corner hole detection is that the hole has different shapes and defects, it needs to be restored to rectangular features. The process is shown in Figure 8.

Figure 8.

The process of rectangle fitting.

First step is to detect all closed contours in the image, and extract the position point sets $C_{i}$ . Second step is to find the max area contour $C_{\max}$ , as shown in (10).

C_{\max} = maxarea C_{i} | i = 1, 2, \dots, n

(10)

The last step is to calculate the minimum circumscribed rectangle $C_{\max}$ , and the process is based on Graham-Scan. The specific steps are as follows: Firstly, find the maximum and minimum points of the abscissa and ordinate in the contour point set, and use these parameters to create a rectangle $Re c_{0}$ . Secondly, enumerate the silhouette edges existing in the point set, and calculate the included angle $α_{i}$ . Thirdly, rotate the contour with the angle $α_{i}$ , re-find the maximum and minimum points of the abscissa and ordinate and establish a new rectangle $Re c_{i}$ . Finally, calculate the minimum area rectangle and the corresponding rotated angle.

The pre-processed corner hole features usually have partial shape defects, but the hole is a rectangular feature in actual, calculate the minimum circumscribed rectangle of contours can partially restore the original corner hole features. The detection result of corner hole is recorded as $P_{2} = [x_{2}, y_{2}]$ , but this is not the final positioning result. $P_{2}$ is the lock hole detection result in $P_{1}$ , the final position P is calculated by $P_{1}$ and $P_{2}$ , as shown in (11).

P = (x_{1} + x_{2} - a, y_{1} + y_{2} - a)

(11)

Container position calculation

The container position information is calculated by container corner position. There are two position parameters, one is the offset distance, another is the relative deflection angle of the container.

Calculation offset distance

As shown in Figure 9, the container offset distance means the distance vector between the detected container center position $P_{c 0}$ , and the standard operation position $P_{s 0}$ , as shown in (12). But the center position of container cannot be directly detected, we use the four detected corners position $P_{c}$ and the standard corners positions $P_{s}$ to calculate the offset distance, the calculation as shown in (13), (14).

l = (Δ x, Δ y)

(12)

Δ x = \frac{1}{4} (x_{ca} + x_{cb} + x_{cc} + x_{cd} - x_{sa} - x_{sb} - x_{sc} - x_{sd})

(13)

Δ y = \frac{1}{4} (y_{ca} + y_{cb} + y_{cc} + y_{cd} - y_{sa} - y_{sb} - y_{sc} - y_{sd})

(14)

Figure 9.

Calculation of container offset distance.

Calculation deflection angle

As shown in Figure 10, The deflection angle of container means the counterclockwise rotation angle of the container relative to the standard container angle. The deflection angle $θ$ is calculated by the detected corner position vector $a_{c}$ and the standard position vector $a_{s}$ , as shown in (15), and these vector’s calculation as shown in (16), (17).

θ = \arccos (\frac{1}{4} * (\frac{a_{c 1} \cdot a_{s 1}}{| a_{c 1} | \cdot | a_{s 1} |} + \frac{a_{c 2} \cdot a_{s 2}}{| a_{c 2} | \cdot | a_{s 2} |}))

(15)

a_{c 1} = (x_{ca} - x_{cd}, y_{ca} - y_{cd}), a_{c 2} = (x_{cb} - x_{cc}, y_{cb} - y_{cc})

(16)

a_{s 1} = (x_{sa} - x_{sd}, y_{sa} - y_{sd}), a_{s 2} = (x_{sb} - x_{sc}, y_{sb} - y_{sc})

(17)

Figure 10.

Calculation of deflection angle.

Convert the pixel distance to actual distance

The offset distance calculated by (12) is pixel distance, but the operation parameters need actual distance. We use the triangulation distance measurement to calculate the actual distance, the principle is shown in Figure 11. This method based on the characteristics of pinhole camera, its calculation is simple and accurate.

Figure 11.

Calculation of actual distance.

In Figure 11, $D$ is the vertical distance between measure target and camera, Point $F$ is the focus point of camera. The light reflected by point $T_{1}$ and $T_{2}$ will pass through the focus point, and form virtual image $t_{1}$ and $t_{2}$ on the CCD (Charge Coupled Device) of camera. Then the triangle formed by $T_{1}$ , $T_{2}$ , $F$ is similar to the triangle formed by $t_{1}$ , $t_{2}$ , $F$ . When the measure target is parallel to the CCD, and the distance $D$ is known, this method could use the pixel distance of two point in the image to calculate the actual distance. The actual distance calculation is shown in (15), x₁is the total pixel lens of image.

d_{1} = \frac{D}{fc} \cdot \frac{x}{x_{1}} \cdot d_{3}

(18)

If set $γ = \frac{D}{fc} \cdot \frac{x}{x_{1}}$ , then (16) can be simplified to (17), where $γ$ is defined as the distance conversion factor between pixel distance and actual distance.

d_{1} = γ \cdot d_{3}

(19)

Before distance calculation, it should be noted that most cameras have some image distortion, which is caused by lens distortion and coordination problems in the assembly process. Therefore, before calculate the position parameters, the image needs be calibrated, we use the image calibration method based on Zhang’s²³ method, the calibrated image as shown in Figure 12. But image calibration cannot completely remove the distortion of image, it will cause some positioning errors, which can be measured in the experiment.

Figure 12.

Image calibration: (a) original image and (b) calibrated image.

Experiments

Experiment preparation

The positioning accuracy of this system depends on the accuracy of container corner detection and the accuracy of distance convert method. Experiment part will test these parts.

The images used in the experiment were captured by the camera shown in Figure 13, and the installation position of the camera is shown in Figure 3(a), there is about 15 m from the ground. The resolution of image is 2560 × 1440, and the fps is 24.

Figure 13.

The installation of camera.

The hardware equipment for calculation as follows, which is a common industrial computer configuration:

CPU: Intel i7-6700;

GPU: Nvidia GTX970-4GB.

And the software as follows:

Operating System: Ubuntu18.04;

Machine learning library: Pytorch1.3.0²⁴;

Image process library: OpenCV 4.0²⁵;

Programming language: Python3.6.

According to ISO-1161-2016,²⁶ the length of lock holes is about 124 mm, and the width is 63.5 mm. In order to perform the operation requirements, the positioning error of corners in the heading direction of container should not exceed 60 mm, and the positioning error in the lateral direction of container should not exceed 30 mm, and for the real-time positioning, the fps of positioning method should be greater than the fps of camera.

Evaluation of measurement model

We used 200 images of the top surface of container, each image labeled the position of the two corner holes on the front of container. The width of container is determined, then the distance conversion factor $γ$ can be calculated by normal fitting the distance between the two container corners. The calculation result are shown in T1, this error is mostly caused by the distortion of camera (Table 1).

Table 1.

Evaluation of measutement model.

Evaluation item	Result
$γ$ mean	5.6584
95% confidence interval	(5.6224, 5.6951)
Maximum measuring range	(±4670 mm, ±2335 mm)

The maximum measurement range means the measurement range that meets the accuracy of 95% confidence interval with the standard operation position as the origin.

Evaluation of modified SSD detection

The training of modified SSD uses 8700 container images, each with 2–4 corner hole targets was labeled. The detection results are shown in Figure 14, and some typical detection result is shown in Figure 15. The Figure 15(a) is good, but other results have some positioning errors. The detection result of SSD is not the container corner target itself, it is the area with the greatest probability of containing the corner holes, this is the largest source of error in SSD detection.

Figure 14.

Detection results of modified SSD.

Figure 15.

Some typical detection results of modified SSD: (a) example of detection result with an accuracy of 0.93, (b) example of detection result with an accuracy of 0.99, (c) example of detection result with an accuracy of 0.98, and (d) example of detection result with an accuracy of 0.99.

There are two parts to the evaluation of our modified SSD, the first part is compare the performance of our modified SSD model with origin SSD model. We uses 500 images taken under different lighting conditions to test the detection performance. The evaluation result is shown in T2, and the SSD-300-Resnet-50 is another model that only change the backbone layer to Resnet50, as the comparison group. The experimental results show that the detection accuracy of the modified SSD is not significantly different from comparison group, but the calculation speed is improved by about 5 ms (Table 2).

Table 2.

The performance of modified SSD detection.

Item	SSD-300	SSD-300-Resnet-50	Modified SSD
AP	0.8672	0.9005	0.9016
Time	25.21 ms	21.24 ms	19.96 ms

The second part is tested the detection errors of modified SSD, the result is shown in T3. The calculation of heading error and lateral error is as follows: Count the distance between the center point of the detection result and the center of corner hole marked in the test samples. Divide the distance vector into two components: lateral (Y-axis) and heading (X-axis) of the screen. Then normalize the statistical data of the two components (as shown in Figures 17 and 18) respectively. Finally, calculate the maximum error value of the fitted normal distribution curve within the 95% and 90% probability interval. These data can be transfer to actual distance by (19).

Evaluation of morphological detection

Our two-stage positioning method uses the detection result of the image processing algorithm to replace the result of SSD detection, its accuracy depends on the accuracy of former. According to this, the experiment mainly test the positioning error of image processing algorithm.

The implementation of image processing algorithm is based on Python and OpenCV. In the experimental part we used a set of 500 corner hole image, these images were captured in different time from day to night. The length and width of images also expanded by 10 pixels to ensure those images include entire corners, the average image size was 65 pixels length and 55 pixels width.

Some typical detection result of image processing algorithm is shown in Figure 16. To images that were captured with different angles of the camera and different light levels, this image algorithm has a good detection result.

Figure 16.

Detection results of corner holes in morphology detection.

T4 shows the influence of MSRCR to the morphological detection. When the detection result lock on the corner hole area, then this detection was been defined as a successful detection. Experiment shows that MSRCR has effectively improves the detection rate of container corners, but the calculation time is increased by about 33 ms (Table 4).

The positioning performance of the two-stage positioning method is shown in Table 5. The calculation time of the two-stage method is calculated from the image that has two corner hole targets. The detection rate of two-stage method is the product of the detection rate of image processing and the detection rate of SSD detection (Table 5).

The detection rate shows that the image processing has a good correction ability for detect corner holes with irregular shapes and different light conditions. The calculation time spend is also less, this is mainly because the SSD detection has filtered most of irrelevant areas in image.

The accuracy comparison of two-stage detection and SSD detection is shown in Figures 17 and 18, the distribution of experiment data is calibrated by normal fitting .The results show that positioning accuracy has been greatly improved in second detection.

Figure 17.

Distribution of detection error in heading direction.

Figure 18.

Distribution of detection error in lateral direction.

Because the detection rate of the image algorithm is 95.6%, 2.8% of lock hole targets would be detected in the first SSD detection, but not detected by the second detection. Then, the accuracy of these detection results should be equal with the accuracy of the first detection, which is shown in Table 3.

Table 3.

Positioning error of modified SSD detection.

Item	Heading error	Lateral error
95% confidence level	8.52 px/48.2 mm	4.44 px/25.1 mm
90% confidence level	7.27 px/41.3 mm	3.52 px/19.9 mm
Detection rate	98.4%

Table 4.

Improvement of using MSRCR.

Evaluation Item	Detection rate (%)	Average calculation time
Detection with MSRCR	95.6	35.6 mm
Detection only with HSV	83.4	2.1 ms

Table 5.

Performance of two-stage detection.

Evaluation Item	Heading error	Lateral error
95% confidence level	3.765 px/19.6 mm	2.81 px/14.3 mm
90% confidence level	2.02 px/11.4 mm	1.80 px/10.2 mm
Calculation time	91.16 ms
Detection rate	94.0%

Conclusion

To solve the automation problem in container lifting operations, we analyze the situation of container lifting operation, proposed a vision based measurement system to positioning container. The experiment results show the positioning error is great than operation requirements. Meanwhile the system’s positioning rate reached 10 fps, which proves the system can be used for real-time positioning.

The experimental results also show that the totally detection rate is not ideal. This is because two detection are performed during the image detection process, and each of these two detections has some error detection. But the lower calculation time shows the image processing algorithm can still be further enhanced.

Footnotes

Acknowledgements

We appreciate Shanghai SMUVision Smart Technology Ltd about the data sharing in this research work.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported by the Science and Technology Commission of Shanghai Municipality (No. 22ZR1427700), China (Shanghai) Pilot Free Trade Zone Lin-gang Special Area Administration (No. SH-LG-GK-2020-21).

ORCID iDs

Yage Huang

Chao Mi

References

Limited

IR.

Global automated container terminal market 2020-2024. New York: TechNavio, 2020.

Ryu

Ahn

Yoon

YS.

A study on the cost analysis for the container terminal services based on abc approach. J Navig Port Res 2011; 35(7): 589–596.

Perkovic

Gucma

Luin

, et al. Accommodating larger container vessels using an integrated laser system for approach and berthing. Microprocess Microsyst 2017; 52: 106–116.

Huang

, et al. Vision-based measurement: actualities and developing trends in automated container terminals. IEEE Instr Meas Magz 2021; 24(4): 65–76.

Yoon

Hwang

Cha

. Real-time container position estimation method using stereo vision for container auto-landing system. In: ICCAS 2010, 2010, pp.872–876. IEEE.

Dai

Liu

Wang

. An auxiliary container loading location algorithm based on computer vision. In: 2019 34rd youth academic annual conference of Chinese association of automation (YAC), 2019, pp.280–284. IEEE.

Mei

Guo

Liu

, et al. A novel framework for container code-character recognition based on deep learning and template matching. In: 2016 International conference on industrial informatics-computing technology, intelligent technology, industrial information integration (ICIICII), 2016, pp.78–82. IEEE.

Zhang

Huang

, et al. A fast automated vision system for container corner casting recognition. J Mar Sci Technol 2016; 24(1): 54–60.

Shen

. Crack detection of track plate based on YOLO. In: 2019 12th international symposium on computational intelligence and design (ISCID), vol. 2, 2019, pp.15–18. IEEE.

10.

Shi

, et al. Vehicle detection under uav based on optimal dense YOLO method. In: 2018 5th international conference on systems and informatics (ICSAI), 2018, pp.407–411. IEEE.

11.

Kitayama

, et al. Detection of grasping position from video images based on ssd. In: 2018 18th international conference on control, automation and systems (ICCAS), 2018, pp.1472–1475. IEEE.

12.

Liu

Anguelov

Erhan

, et al. SSD: single shot multibox detector. In: European conference on computer vision, 2016, pp.21–37. Springer.

13.

Fang

Container keyhole positioning based on deep neural network. Int J Wirel Mob Comput 2020; 18(1): 40–50.

14.

Kawai

Kim

Choi

Measurement of a container crane spreader under bad weather conditions by image restoration. IEEE Trans Instrum Meas 2011; 61(1): 35–42.

15.

Redmon

Divvala

Girshick

, et al. You only look once: unified, real-time object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, Las Vegas, NV, USA, 27–30 June 2016, pp.779–788. New York: IEEE.

16.

Girshick

. Fast R-CNN. In: Proceedings of the IEEE international conference on computer vision, 2015, pp.1440–1448. IEEE.

17.

Simonyan

Zisserman

Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:14091556, 2014.

18.

Liu

Ranga

, et al. DSSD: deconvolutional single shot detector. arXiv preprint arXiv:170106659, 2017.

19.

Zhang

Ren

, et al. Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778. IEEE.

20.

Jobson

Rahman

Woodell

GA.

A multiscale retinex for bridging the gap between color images and the human observation of scenes. IEEE Trans Image Process 1997; 6(7): 965–976.

21.

Land

EH.

An alternative technique for the computation of the designator in the retinex theory of color vision. Proc Natl Acad Sci U S A 1986; 83(10): 3078–3080.

22.

Liu

Gao

Kong

, et al. Restoration algorithm for noisy complex illumination. IET Comput Vis 2019; 13(2): 224–232.

23.

Zhang

A flexible new technique for camera calibration. IEEE Trans Pattern Anal Mach Intell 2000; 22(11): 1330–1334.

24.

Ketkar

Santana

Deep learning with Python, vol 1. Berkeley, CA: Springer, 2017.

25.

Bradski

Kaehler

. Learning OpenCV: computer vision with the OpenCV library. Sebastopol, CA: O’Reilly Media, Inc., 2008.

26.

ISO. ISO 1161:2016. Series 1 freight containers - corner and intermediate fittings - specifications. Geneva: ISO, 2016.

A vision-based container position measuring system for ARMG

Abstract

Keywords

Introduction

Vision-based measuring system

Image capture device

First stage detection based on modified SSD

Second stage detection based on image processing algorithm

Image pre-processing

Rectangle fitting

Container position calculation

Calculation offset distance

Calculation deflection angle

Convert the pixel distance to actual distance

Experiments

Experiment preparation

Evaluation of measurement model

Evaluation of modified SSD detection

Evaluation of morphological detection

Conclusion

Footnotes

Acknowledgements

Declaration of conflicting interests

Funding

ORCID iDs

References