Sage Journals: Discover world-class research

Abstract

To enhance the automation level of foreign object removal in bird’s nest processing and reduce the costs and misjudgment rates associated with manual inspection, this paper proposes and develops a parallel system for automatic foreign object removal in bird’s nest based on machine vision. A two-stage foreign object detection algorithm is proposed: the first stage introduces an attention mechanism to improve the U-Net++ model, enhancing segmentation accuracy under texture interference; the second stage combines binary mask processing to achieve precise classification of foreign object and non-foreign object regions. Experimental results show that the algorithm achieves an F₁ score of 94.80% and a recall rate of 97.90%, outperforming traditional methods in overall performance. An evaluation system for bird’s nest foreign object removal is established, incorporating criteria such as cleanliness, loss rate, structural integrity, and removal time. A set of automated equipment with identification, localization, and removal functions is designed and implemented. The algorithm and equipment were deeply integrated, and through prototype construction and performance testing, the system’s ability to accurately identify and stably remove various types of foreign objects under complex operating conditions was validated. This study provides a feasible intelligent solution for the bird’s nest impurity removal field, with good application prospects and industrial promotion value.

Keywords

bird’s nest removal robot structural design simulation optimization hair picking needle

Introduction

Bird’s nest is formed by the saliva secreted by swiftlets intertwined with feathers to create a nest. Natural bird’s nests typically contain foreign objects such as feathers, eggshells, and sand particles, with down-like impurities being the most common.¹ Currently, impurity removal in bird’s nest processing primarily relies on manual visual inspection and manual removal, which not only results in low efficiency and high labor intensity but also carries significant risks of misjudgment and missed detection, severely hindering the automation development of the bird’s nest reprocessing industry. Therefore, achieving efficient and precise identification and automatic removal of impurities in bird’s nest has become a critical technological challenge that urgently needs to be addressed in the fields of food intelligent detection and automated processing.

In terms of bird’s nest impurity detection, traditional image processing methods struggle to handle its complex textured background and semi-transparent structure. Subramaniam et al.² used edge detection methods for identification, achieving a detection rate of only 68.18%; Meng et al.³ employed the C-K mean fusion algorithm, improving the detection rate to 84%, but the misclassification rate remained as high as 57%. Yee et al.⁴ combined image fusion technology to improve recall rates to some extent, but were still limited by image shadow interference, resulting in a misclassification rate exceeding 30%. The saliva chain structures in bird’s nest are entangled, with semi-transparent components and uneven intensity,⁵ making it difficult for traditional image segmentation methods to extract effective features. As a result, researchers have gradually introduced deep learning methods to enhance segmentation accuracy. Hui et al.⁶ utilized the U-Net architecture for segmentation of objects with uneven intensity in medical images, achieving good results; Yeo et al.⁷ proposed a segmentation model based on autoencoders, achieving a recall rate of 92.82%, but the misclassification issue remains unresolved.

On the other hand, with the continuous increase in market demand, various types of automatic weed removal equipment have been developed and applied. Generally speaking, the configurations of such robots can be divided into serial robots,^8,9 parallel robots,^10,11 and hybrid robots.^12,13 METİN derived the general inverse Jacobian matrix equations for 195 six-degree-of-freedom GSP mechanisms under specific constraints and proposed a generalizable construction method, providing a theoretical foundation and engineering tools for the systematic design and analysis of parallel robot systems. Additionally, a software tool was developed for the design, analysis, and simulation of 195 six-degree-of-freedom GSP parallel mechanisms.^10,14 Serdar proposed a novel planar hybrid robot composed of a 2-degree-of-freedom planar parallel robot manipulator and two planar serial robot manipulators, enabling easier material grasping.¹⁵ Based on the aforementioned three robot configurations, multiple studies have attempted to combine image recognition with robotic sorting systems to remove impurities from bird’s nest. Subramaniam et al.² developed a bird’s nest impurity removal device based on a machine vision system and a serial robot platform, targeting specific bird’s nest specimens with a thickness of 2 mm. They employed traditional image processing methods based on edge detection and image enhancement, achieving an impurity detection rate of only 68.18%. Seenivasan et al.¹⁶ optimized brushing, bubble, and microbubble technologies to reduce cleaning time and improve purity for small impurities in bird’s nest. The structure was designed as a brush attached to the end of an arm to clean original bird’s nest impurities; Negatively charged microbubbles attract microscopic impurities, and the microbubbles contract and collapse under water pressure, leaving debris on the water surface. Fai et al.² designed a debris removal device based on machine vision and a robotic platform, using image processing to drive the end-effector to remove foreign objects; Kin et al.³ designed a biomimetic end-effector based on the principle of artificial tweezers, improving the flexibility of the removal operation. However, most of these studies still face bottlenecks in terms of recognition accuracy or removal reliability, making it difficult to meet the complex and variable requirements of actual debris removal.

Based on the aforementioned issues, this paper proposes a systematic solution for the automatic removal of impurities from bird’s nest using a parallel robot configuration. By incorporating an attention mechanism based on the U-Net++ architecture, we have developed an impurity segmentation algorithm suitable for low-contrast and complex backgrounds. We have designed an automated impurity removal robot with integrated identification, localization, and removal functions, completing structural modeling, control system development, and end-effector optimization. A multi-indicator quality evaluation system is established to quantify removal effectiveness and validate the system’s overall performance. A collaborative “machine vision recognition + intelligent actuator” automated impurity removal platform is developed to address the core issues of low efficiency and poor accuracy in current bird’s nest impurity removal processes. This provides theoretical support and engineering implementation pathways for the intelligent upgrading of the bird’s nest industry, demonstrating significant research significance and application value.

Results and discussions

Design of the robot

According to the design requirements of the foreign object removal, the actuator for the multi-needle picking needle is designed including driving structure of the single-needle picking needle, the motor container of the multi-needle picking needle, the pressure plate structure and the self-cleaning device, which can achieve the fast and accurate removal of the foreign matter of the swallow in the bird’s nest.

As presented in Figure 1(a), with reference to the removal method of the picking needle in the manual needle-rolling method, a novel picking needle with three heads structure is designed to solve the problems of insufficient adhesion and low coaxiality between picking needle and the driving motor, which lead to the high-speed rotation of the picking needle to fly away the foreign matters in bird’s nests. Since high precision positioning is required for the removal of foreign matter from bird’s nests, the design of the picking needle drive structure, the surface treatment of the materials used and the assembly precision of the parts are all required to meet the design requirements.

Figure 1.

Design of the foreign object removal device: (a) structure of the picking needle, (b) structure of the array structures of picking needles, (c) structure of the platen, and (d) structure of the whole picking device.

The foreign matter in the bird’s nest can be effectively rejected by single needle in a one-to-one manner, while the number of downy foreign matter in the bird’s nest pictures can be identified more than 1 with the assistance of machine vision. Therefore, it is necessary to arrange the single picking needle structure in arrays in order to improve the overall rejection efficiency without increasing the complexity of the mechanical structure. It is verified in experiments that the tiny picking needles do not damage the structural integrity of the bird’s nest filaments, and thus the array arrangement of the multi-needle picking needle structure can effectively increase the rejection efficiency of the bird’s nest foreign matter. The linear arrangement scheme is adopted, the mounting cover plate is used for effective fixation with the Z-axis motion mechanism, and the motor container is arranged in a 3 × 3 arrangement to fix nine picking needles (shown in Figure 1(b)). Besides, in order to facilitate the electrical connection, a specific motor driver board is designed to drive the nine single-needle head motors as a whole, and all the electrical controls are connected through the alignment channel on the mounting cover plate.

Besides, as shown in Figure 1(c), the platen is fastened to the motor container structure by four fixed rods. The compression spring, built-in sliding sleeve which can realize the travel translation and the end fastening cover are designed on each fixed rod. The designed end actuator is obtained as shown in Figure 1(d). The initial state is the natural recovery state of the spring, at this time the tip of the picking needle is parallel to the bottom surface of the platen. When the foreign matter rejection in the bird’s nest is carried out, the end actuator reaches the rejection position, and as the end actuator continues to move downward in the Z-axis direction. At this moment, the press plate touches the bird’s nest and then as the press plate rises, the picking needle realizes the rejection of the foreign matter. When the picking needle rolls up the bird’s nest foreign matter, the end actuator moves upward in the Z-axis direction, and the platen is restored to the initial state under the joint action of the spring return force and the platen’s own gravity. While any bird’s nest filament on the picking needle away from the bird’s nest is detected, it will be separated from the adhered bird’s nest filament into the bird’s nest container by the smaller beveled opening on the platen.

The structure of the whole robot is shown in Figure 2(a), which can be divided into robot motion module, foreign object removal module, vision module, and control module. The motion module contains the main architecture, drive structure of the upper end actuator and the motion of the lower bird’s nest container. Foreign object reject module can achieve the efficient removal of contaminants with the assistance with vison and control module. As shown in Figure 2(b), in the bridge structure of robot, the end actuator can realize the composite motion of X, Y, and Z axes in the upper three-dimensional spatial structure, and the bird’s nest container realizes the composite motion of X and Y axes in the lower planar structure. The X-axis of the upper and lower layers and the Z-axis of the upper layer achieve motion through a single stepping motor, and the Y-axis motion of the upper and lower layers achieve synchronized motion through two stepping motors respectively, which can effectively increase the stability and precision of the motion mechanism. Based on the specified operating space, the actual effective travel lengths of the upper X-axis, Y-axis, and Z-axis shall not be less than 600, 700, and 340 mm, respectively. Considering the replacement of external equipment, to effectively improve space utilization, the maximum spatial dimensions of the robot’s equipment are proposed to be 930 mm × 700 mm × 650 mm. To facilitate subsequent testing and further structural optimization, the experimental platform will not be externally enclosed during its initial setup.

Figure 2.

Design of the whole robot: (a) design of the robot including motion module, foreign object removal module, vision module, and control module and (b) design of the bridge structure of robot.

The load-bearing components of the X-axis are the lead screw and the mounting plate, so it is necessary to verify the structural strength of these components to ensure the reliability of the mechanism. The end effector and Z-axis structure are mounted on the X-axis slide block, so the X-axis bears the full mass of the end effector and Z-axis structure. Both ends of the X-axis structure are fixed to the Y-axis moving slide block. Based on the force distribution of the X-axis, it can be simplified to a simply supported beam problem, as shown in Figure 3. To facilitate the strength verification of the X-axis, it is simplified to a uniformly distributed load scenario.

Figure 3.

Simplified diagram of forces acting on the upper X-axis.

To calculate the load results under uniformly distributed loads on the simply supported beam shown in the figure above, it is simplified to act on the midpoint of the beam. In the actual calculation, the length of the slider on the X-axis is approximated as the distribution distance of the uniformly distributed load, and the calculation formula shown in x is obtained.

q = \frac{Q}{l_{h}}

(1)

In the equation, q represents the calculation result of the uniformly distributed load, Q represents the total gravitational force along the Z-axis acting on the simply supported beam, and l_h represents the length of the sliding block along the X-axis. Based on the X-axis structure, the cross-section of the simply supported beam is approximated as a rectangle. According to the distribution of uniformly distributed loads and the action form of the rectangular cross-section, and based on the material parameters, the bending moment M₁, the moment of inertia I₁, the static moment S₁, and the resistance moment W₁ of the cross-section of the simply supported beam can be further calculated, as shown in equation (2).

{\begin{matrix} M_{1} = \frac{q \times l_{h}}{4} (l_{0} - \frac{l_{h}}{2}), I_{1} = \frac{b h^{3}}{12} \\ S_{1} = b (\frac{h}{2} - y) (\frac{h}{4} + \frac{y}{2}), W_{1} = \frac{b h^{3}}{6} \end{matrix}

(2)

In the equation, h represents the height of the rectangular cross-section, y represents the distance from the cross-section to the longitudinal axis, and b represents the width of the rectangular cross-section. Based on the above equation, the maximum bending deflection max f, bending normal stress σ, and shear stress τ of the X-axis are calculated, as shown in equation (3).

{\begin{matrix} f_{max} = \frac{5}{384} \times \frac{q \times l_{0}^{4}}{E \times I_{1}} \\ σ = \frac{M_{1}}{W_{1}} \\ τ = \frac{Q \times S_{1}}{I_{1} \times b} \end{matrix}

(3)

In the formula, E represents the elastic modulus of the material. This project uses a combination of aluminum alloy sliders and lead screws, so the calculated results for the bending deflection of the X-axis structure are compared with the allowable deflection results in the reference manual, as shown in Formula (4).

{\begin{matrix} f_{max} = 0.68 mm < [f] \\ σ = 9.31 MPa < [σ] \\ τ = 1.25 MPa < [τ] \end{matrix}

(4)

Based on the calculation results, it can be seen that the simplified simply supported beam on the X-axis meets the strength requirements for material and design structure applications in terms of reliability and safety. According to the load capacity of the module in the horizontal and vertical directions during selection, which are 30 and 20 kg respectively, it meets the actual application conditions.

Based on the design of the proposed mechanism, we can complete the kinematic modeling. The parallel robot for bird’s nest cleaning and debris removal constructed in this paper has independent serial motion chains, which are simultaneously connected to the ground and the motion platform. However, due to the complexity of closed-loop chain mechanisms, the kinematic analysis of parallel robots is relatively challenging. The D-H^15,17 method is a common kinematic modeling convention that uses a 4 × 4 homogeneous transformation matrix. The main advantage of this method is that it facilitates a physical interpretation of the robot mechanism.

Given that research on Cartesian coordinate robots is already well-established, based on robotics theory,¹⁸ we assume that any point P in the Cartesian coordinate system can be represented as a column vector, which can be expressed using the three coordinate components PX, PY, and PZ. Using the DH method, the coordinates of each joint of the Cartesian coordinate robot are transformed using rotation and translation matrices. Through the DH parameter model, the transformation between adjacent coordinate systems can be expressed as shown in equation (5).

\begin{matrix} {}_{i + 1}^{i}T = Rot (z, θ_{i + 1}) Trans (0, 0, d_{i + 1}) \\ Trans (a_{i + 1}, 0, 0) Rot (x, α_{i + 1}) \\ = [\begin{matrix} c θ_{i} & - c α_{i} s θ_{i} & s α_{i} s θ_{i} & a_{i} c θ_{i} \\ s θ_{i} & c α_{i} s θ_{i} & - s α_{i} c θ_{i} & a_{i} s θ_{i} \\ 0 & s α_{i} & c α_{i} & d_{i} \\ 0 & 0 & 0 & 1 \end{matrix}] \end{matrix}

(5)

The pose transformation between the end-effector pose of the Cartesian coordinate robot and the polar coordinate system is shown in equation (6). According to the design requirements, the initial positions of the sliding joints of the Cartesian coordinate robot are d₁ = 45 mm, d₁ = 20 mm, and d₁ = 60 mm. Therefore, the initial pose of the end-effector is derived as follows:

{}_{3}^{0}T = {}_{1}^{0}T {}_{2}^{1}T {}_{3}^{2}T = [\begin{matrix} 0 & 1 & 0 & 45 \\ 0 & 0 & 1 & 20 \\ 1 & 0 & 0 & 60 \\ 0 & 0 & 0 & 0 \end{matrix}]

(6)

Parameter analysis and optimization

As shown in Figure 4, the triple-needle structure consists of fixed, transitional, and rejection segments. Due to that the picking needle is driven by a motor for high speed rotation, the fixed segment of the picking needle is used to connect with the end of the motor shaft to ensure that the center maintains a consistent coaxiality with the motor shaft. The transitional segment is used to adjust the scaling tolerance of the platen section in the end actuator and rejection segment can achieve the removement of foreign objects. It should be noted that the length (L) of the picking needle must be more than the average thickness of bird’s nest threads (4 mm) to ensure the removement of foreign objects.

Figure 4.

Parameters of triple-needle structure consisting of fixed, transitional, and rejection segments.

Based on the basic mechanical theory, there will be a variety of forces between the high-speed rotation of the picking needle with the foreign body, such as the centripetal force generated by the high-speed rotation (F_c), the natural adhesion force in contact (F_l), and torsional force under instantaneous action (F_n). F_c can be calculated through the equation (7):

F_{c} = \int dF = \int r \times ω^{2} d m

(7)

Herein, m denotes the mass of the test foreign object, r denotes the distance between the test foreign object and the center of rotation axis, and ω denotes the angular velocity.

Since that the contact area between the swallow down and the needle tip is very small and the fiber structure of swallow down is complex, the actual friction will be affected by many factors such as bending, compression, and stretching of the fibers. Therefore, an approximation will be taken to calculate the friction force:

F_{l} = μ \times N

(8)

where μ is the friction coefficient between swallow down and needle tip, pressure N can be measured through miniature pressure sensors. Besides, F_n can be calculated through the DC motor:

F_{n} = (9550 \times U \times I \times K) / (n \times r)

(9)

Herein, U, I, K are the voltage, current, and power factor of the drive motor, respectively. n and r denote the motor speed and distance between swallow down and shaft center, respectively.

Moreover, in order to further study the effect of the above variables on the action force of the picking needle, the uniform design method^19–21 as well as simulation method are designed to calculate the force and provide theoretical support for the related research. The specific optimization design and experimental steps are described in detail in Section “Parameter analysis and optimization.”

Parameter analysis and optimization

The design focus of the bird’s nest foreign object removal robot lies in its end-effector, and the core of the end-effector is the design of the feather-picking needles. For the three-needle-tip feather-picking needle, parameter analysis and calculations indicate that the needle tip spacing C, the needle tip-to-bird’s nest fiber contact depth h, and the feather-picking needle rotation speed n are the key variables affecting the adhesion force during the removal process. Among these, an excessively large needle tip spacing C may cause the needle tips to become stuck in the bird’s nest fibers, thereby damaging the structural integrity of the bird’s nest. The contact depth h directly affects the contact radius a between the needle tips and the bird’s nest fibers; when h is determined, a is also determined accordingly. Additionally, an excessively high rotational speed n may cause the torque generated by the feather-removal needle to exceed the shear or tensile limits that the bird’s nest fibers can withstand, thereby also damaging their structural integrity. Given the complex influence of the aforementioned variables on the force exerted by the feather-picking needle, as well as the difficulty in deriving the optimal solution through pure calculation, it is necessary to conduct parameter optimization design. By systematically adjusting the key design parameters C, h, and n, the aim is to determine their optimal combination and ultimately derive the functional relationship expression between the instantaneous adhesion force between the feather-picking needle and the nest fibers and C, h, and n.

Optimization framework and experimental design

For the design of optimization frameworks, intelligent algorithms such as PSO, are currently popular. This study adopts an optimization strategy combining the Uniform Design Method (UDM) with multivariate regression analysis. UDM can efficiently explore the multi-factor design space with the minimum number of experiments and sets the needle tip distance (C): the single-needle action range (range: 0.20–0.45 mm); contact depth (h): determines the extent to which the needle tip is embedded in the nest matrix (range: 1.00–2.00 mm); and rotational speed (n): directly affects adhesion generation (range: 10,000–20,000 rpm). These three key parameters are used as design variables.

Based on the above optimization framework, the experiment was designed with the following specific steps:

(1) Determine the indexes for experimental analysis. The main research index of the experiment is the maximum adhesion force at the contact between the picking needle and the bird’s nest F.

(2) Determine the experimental factors. The factors are picked as picking needle tip distance C, contact depth h, and picking needle rotation speed n.

(3) Determine the level of each experimental factor. According to the homogeneous design method using three factors and six levels of the homogeneous design method, the level table of the influence variables is shown in Table 1.

(4) Design of uniform design table. In order to reduce the number of experiments, the uniform design table is designed. Based on Admas simulation software, the tip of the picking needle in the adhesion of the different influencing factors can be acquired through setting of different depth of contact and drive speed. The statistical results are shown in Table 2.

Table 1.

Factors and level distribution.

Factors	Level
Factors	1	2	3	4	5	6
A Distance (mm)	0.20	0.25	0.30	0.35	0.40	0.45
B Depth (mm)	1.00	1.20	1.40	1.60	1.80	2.00
C Speed (r/min)	10,000	12,000	14,000	16,000	18,000	20,000

Table 2.

Results of uniform design method.

Testing number	Factor			Result
Testing number	A	B	C	Adhesion force (N)
1	1	2	3	0.386
2	2	4	6	0.604
3	3	6	2	0.465
4	4	1	5	0.562
5	5	3	1	0.485
6	6	5	4	0.590

Through the above experiments, a 3-factor, 6-level experimental matrix was constructed based on UDM, as shown in Table 1. Only six groups of simulation experiments were required to cover the design space. By simulating the needle tip-impurity interaction under different parameter combinations, the maximum adhesion force F was collected, as shown in Table 2.

Regression modeling and parameter optimization

Moreover, multivariate regression analysis is designed to calculate the correlation through SPSS software, the result is shown in Table 3.

Table 3.

Analysis of multiple regression results.

Parameter	Non-standardized coefficient		Standardized coefficient	t	Significance level (S)
Parameter	Variable coefficient	Standard error	Standardized coefficient	t	Significance level (S)
Constant	0.216	0.047	–	0.453	0.001
A	6.12e-002	0.008	0.355	1.298	0.001
B	1.41e-005	0.005	0.396	5.263	0.004
C	3.84e-002	0.017	0.871	0.757	0.005

It can be obtained from the experimental results that the correlation coefficient R of the model at 95% confidence level is 0.961, which indicates that the independent variable is highly correlated with the dependent variable and the regression equation is highly accurate. The regression equation is shown in the following equation:

\begin{matrix} F_{max} = 6.12 \times 10^{- 2} A + 1.41 \times 10^{- 5} B \\ + 3.84 \times 10^{- 4} C + 0.416 \end{matrix}

(10)

As presented in Figure 5(a), the average value of the effect of each factor on the results and the response contribution rate of each factor are obtained as well. The response contribution rate of each factor indicates the normalized value in the regression model. According to the sample fitting, the red color indicates the positive effect, and the larger the value of the parameter, the closer it is to the target value. Therefore, it can be obtained that the most influential factor on adhesion is rotational speed, followed by tip distance and contact depth.

Figure 5.

Analysis of multiple regression results and simulations: (a) average value of the effect of each factor on the results and the response contribution rate of each factor, (b) the simulation of total deformation displacement and equivalent force and (c) total deformation displacement and equivalent atress calculation results.

Static analysis of the pick needle

Through the optimization design in the previous section, the optimal parameter design for the hair-picking needle structure was adopted. This was used for mechanical simulation verification, where the drive motor secures the hair-picking needle to the motor shaft via a connector, thereby transmitting power to the hair-picking needle, driving it to rotate at high speed and start/stop. During simulation, the 3D model .x_t file is first imported into the static structure of Workbench, and the material properties of the plucking needle and end-effector are configured to match the material characteristics of the simulation, as shown in Table 4.

Table 4.

Modal analysis material properties table.

Material property	Parameter	Parameter
Material name	304 stainless steel	PLA material
Young’s modulus (MPa)	1.93 × 10⁵	3.8 × 10³
Material density (kg m⁻³)	7930	1250
Poisson’s ratio μ	0.29	0.35
Yield strength (MPa)	200	77

Finally, the optimal parameters are obtained: the speed of the picking needle is 16,000 r/min, the tip distance is 0.45 mm, and the contact depth is 1.5 mm, respectively, and the resulting adhesion force is about 0.539 N. Therefore, the optimal design of the structure of the triple-tip picking needles is optimized using this optimal combination of factors. The simulation of total deformation displacement and equivalent force is achieved through Ansys software shown in Figure 5(b). It can be obtained from Figure 5(c) that while under 20,000 r/min rotational speed, the deformation of picking needles can be almost negligible. And the equivalent force results in 0.5 MPa, which is far less than the yield strength value of stainless steel materials. Therefore, it can be concluded that it is fully capable of adapting to the needs of practical work.

Visual discussions and applications

After optimizing the structural design, we conducted research on visual applications. First, we collected raw bird’s nests, which are referred to as “raw nests.” Raw nests can be classified into extremely light raw nests, light raw nests, medium raw nests, and heavy raw nests based on their impurity content. Medium raw nests account for the largest proportion of imported raw materials in the bird’s nest processing industry. According to application requirements, the selected medium raw bird’s nests were soaked in water at 20°C for 2 h, after which images of the target bird’s nests were captured (Figure 6).

Figure 6.

RGB image of bird’s nest collected.

Image preprocessing

In order to enhance the collected images and reduce the deformation of foreign objects in the images caused by enhancement, radiographic transformation and elastic deformation are used to achieve data enhancement.²² The image data is prepared as 512 × 512 pixels, which satisfies the requirements for extracting impurity features while ensuring the integrity of the image. The images captured and stored by industrial cameras are in RGB-32 output format, which takes up a lot of storage space, has low transmission efficiency, and results in long computer processing times, which does not meet practical usage requirements. Therefore, it is necessary to convert color images into grayscale images for model training through grayscale conversion (Figure 7).²³

Figure 7.

Gray-scale image of bird’s nest.

Improved U-Net++ method for detecting foreign objects in bird’s nest

U-Net++ is an improvement and extension of the U-Net model, as shown in Figure 8, aimed at improving the segmentation performance of image boundaries to reduce boundary blurring.²⁴ Due to the tangled saliva chain structure of bird’s nest, its strength exhibits significant non-uniformity. Additionally, bird’s nest impurities vary in type, shape, and depth, and are influenced by factors such as lighting, making this model unsuitable for bird’s nest images.

Figure 8.

Improved U-Net++ architecture.

The improved network model retains certain structural components from U-Net++, such as skip connections and dense convolutional blocks, as shown in Figure 8. The encoder section employs 3 × 3 same convolutions and 2 × 2 max pooling layers to progressively reduce the spatial resolution of the input image, ensuring that the feature extraction section and upsampled image have consistent sizes, which facilitates feature fusion.²⁵ The decoder section restores the resolution through upsampling and skip connections, and performs multi-scale feature fusion by combining small-scale features with large-scale features. This enables the model to obtain more information from both upper and lower layers in feature maps at different levels, thereby enhancing its ability to segment objects at different scales. In the figure, $x^{i, j}$ represents the $j$ th convolution unit of the jump connection of the $i$ th down sampling layer of the encoder, which contains two 3 × 3 Conv (padding = 1, stride = 1, ReLU). Assuming that $x^{i, j}$ is the output feature map of this node, the connection point $x^{i, j}$ is expressed as follows.

x^{i, j} = {\begin{matrix} R (x^{i - 1, j}), & j = 0 \\ R ([{[x^{i, n}]}_{n = 0}^{j - 1}, u (x^{i + 1, j - 1})]), & j > 0 \end{matrix}

(11)

In the equation, R(-) denotes a convolution operation with an activation function; u(-) denotes upsampling; [-] denotes feature concatenation. When j = 0, it is an encoder node; when j > 0, the node receives the first j skip connections from layer i and the upsampled output from layer i + 1. Due to the introduction of significant interference noise from the dense convolutional units in the model, an attention module (AG) is proposed to be introduced before feature fusion to maximize the suppression of background feature noise responses generated during feature fusion across channels.

To highlight the foreground target, a nested jump link structure of the AG module is introduced, as shown in Figure 9. When j > 0, the input values of the convolution unit are $x^{i, 0}$ , $x^{i, j - 1}$ , $x^{i, 1}$ , and the output values of the next layer $x^{i + 1, j - 1}$ , are the calculated $x^{i, j}$ . The expression of the attention coefficient introduced in AG is:

η = σ_{2} [τ {σ_{1} (x^{i} + φ (x^{i + 1, j - 1})), K}], η \in [0, 1]

(12)

When outputting $x_{i} = η x_{i}$ , it represents the element-wise product of the feature map c obtained after merging and modifying the number of channels of $η$ with the feature map of the i-th layer. In the expression, $x^{i} = τ {[x^{i, 0}, x^{i, j - 1}, \dots, x^{i, j}], L_{x}}$ , [-] denotes feature fusion performed with a fixed number of channels, L_x and K are convolution kernels, $τ {-}$ denotes the convolution operation of the specified convolution kernel, and $φ {-}$ denotes bilinear upsampling. First, the additive attention map is obtained by adding the decoding matrix $φ$ and the encoding matrix $σ_{2}$ ; then, the soft attention map is obtained by applying the ReLU activation function, 1 × 1 convolution, and the Sigmoid activation function 101010; Finally, the weighted feature map $x^{i}$ is fused with $φ$ to form the feature map fusion, which serves as the input feature map for the convolutional unit. To mitigate the shadowing issues generated during the original deconvolution process, $x^{i + 1, j - 1}$ employs bilinear upscaling as an alternative to the deconvolution process in the original U-Net++ model. The introduction of the AG module in the skip connection section of the neural network not only enhances the network’s ability to extract foreign object features but also effectively suppresses the impact of noise on the segmentation of bird’s nest images.

Figure 9.

AG flowchart.

Two-stage foreign object detection algorithm

To meet the demand for efficient and accurate detection of impurities in bird’s nest, the U-Net++ network model was specifically improved to propose a two-stage impurity detection algorithm for bird’s nest, as shown in Figure 10.

Figure 10.

Two-stage impurity detection algorithm.

In the first stage, the image segmentation model outputs a tensor of size 512 × 512 × 2. The first channel contains the probability values of all pixels classified as impurities, and the second channel contains the probability values of all pixels classified as non-impurities. In the second stage, pixels are classified into impurity and non-impurity regions based on the probability values, as shown in equation (13), ultimately yielding a binary mask image for classification. Pixels with a mask intensity value of 255 represent impurity regions, while those with a value of 0 represent non-impurity regions.

S_{i, j} = {\begin{matrix} 0, & p_{i, j} (impurity) < p_{i, j} (non impurity) \\ 255, & p_{i, j} (impurity) > p_{i, j} (non impurity) \end{matrix}

(13)

In this context, i represents the row coordinate of the image pixel; j represents the column coordinate; p(impurity) denotes the probability that the pixel at coordinate (i, j) belongs to the impurity class; p(non-impurity) denotes the probability that it belongs to the non-impurity class.

Analysis of test results

A confusion matrix is a matrix used to evaluate the performance of a classification model,²⁶ showing the correspondence between the model’s predicted images and the actual labels on the test data. Bird’s nest images are labeled into two parts: foreign object parts and non-foreign object parts. For a binary classification problem, foreign object parts are classified as the positive class, and non-foreign object parts are classified as the negative class. The segmentation results of bird’s nest images are compared to assess the performance of the foreign object detection algorithm. For all pixels in the image, the predicted image results are compared with the expert manual segmentation values. If the instance is a positive class and the predicted result is positive, it is a true positive (TP); if the instance is negative and the predicted result is negative, it is a false negative (FN); if the instance is positive and the predicted result is positive, it is a false positive (FP); if the instance is negative and the predicted result is negative, it is a true negative (TN). To objectively evaluate the performance of the detection algorithm, we introduce accuracy (Acc), recall (RE), precision (PR), F₁ score, and prediction time (T) as evaluation metrics. Acc represents the percentage of correctly predicted samples out of the total, that is, the proportion of correctly detected pixels out of all pixels, as shown in the following formula.

Acc = \frac{TP + TN}{TP + FP + FN + TN}

(14)

RE, also known as sensitivity (SE), refers to the proportion of correctly predicted positive samples among actual positive samples. High sensitivity indicates that the algorithm has good detection capabilities, as shown in the following formula.

RE = \frac{TP}{TP + FN}

(15)

PR, also known as positive predictive value, refers to the proportion of correctly predicted positive samples among the predicted positive samples. A high precision is ideal, while a low precision indicates that the foreign object detection algorithm is overly sensitive, as shown in the following formula.

PR = \frac{TP}{TP + FP}

(16)

F ₁ represents the harmonic mean of PR and RE, indicating the overlap between the predicted segmentation results and the actual segmentation results. The closer the value is to 1, the more similar the contours of the two areas are, as shown in the following equation.

F_{1} = \frac{2 * PR * SE}{PR + SE} = \frac{2 * TP}{2 * TP + FN + FP}

(17)

Among them, T is used to evaluate the image prediction efficiency of the model.

The improved detection algorithm was compared with U-Net, U-Net++ neural network models, and traditional image processing methods. Under the same experimental conditions, ablation experiments were used to verify that the improved U-Net++ model with the AG module was more feasible and effective when applied to bird’s nest images.

By calculating the sum and average of the TP, TN, FP, and FN values for the three test images, the confusion matrices for the test images are obtained, as shown in Figure 11. Using parameters such as PR, RE, F₁ coefficient, Acc, and T as evaluation metrics, the U-Net, U-Net++, two-stage detection algorithm, and traditional image method are evaluated, with the respective parameter values shown in Table 5. The improved detection algorithm achieved a RE of 97.90% and a PR of 91.89%, indicating that the algorithm has good foreign object detection capability and high accuracy; the F₁ value was 94.80%, an improvement of approximately 2.78% compared to U-Net and approximately 1.12% compared to U-Net++. Therefore, incorporating the AG module into the U-Net++ model effectively suppresses background noise caused by dense connections and enhances the detection performance of foreign objects in bird’s nest images. Additionally, the improved detection algorithm achieves an F₁ score 20.71% higher than traditional image-based methods, thereby demonstrating that the improved neural network segmentation algorithm significantly outperforms traditional image-based methods in foreign object detection for bird’s nests.

Figure 11.

Confusion matrix for binary classification of bird’s nest images: (a) test figure 1, (b) test figure 2, and (c) test figure 3.

Table 5.

Performance indicators for detecting foreign objects in bird’s nest using different models.

Method	Acc	PR	RE	F ₁	T/s
U-Net	0.9907	0.8851	0.9582	0.9202	0.5532
U-Net++	0.9958	0.9067	0.9690	0.9368	0.6256
Improved algorithm	0.9993	0.9189	0.9790	0.9480	0.6721
Traditional methods	0.8051	0.7137	0.7696	0.7409	–

Removal test of the robot

Based on the reliable design of the robot, the test of foreign object removal is carried out to verify the function. As presented in Figure 12(a), the whole process can be divided into four steps: Recognition of the foreign object inside the bird’s nest; Route planning of the needle structures; Removal of foreign objects; Self-cleaning of the device. Every time the end-actuator cleans the bird’s nest on the picking needle, it needs to collect real-time images through the vision, and complete the rapid rejection of all the foreign objects in the bird’s nest after several cycles.

Figure 12.

Foreign object removal test of the robot: (a) the whole process including recognition, route planning, removal and self cleaning, (b) comparison of robot coordinates with the actual removal coordinates, and (c) comparison of bird’s nest before and after the cleaning by foreign object removal robot.

The control system acquires images of bird’s nests and recognizes the shaped centers of a plurality of bird’s down foreign objects, which can obtain the pixel coordinates of the shaped centers of the foreign objects. Based on the proportionality relationship of the results of the hand-eye calibration,^10,14 the pixel coordinates of the foreign objects are converted to the actual spatial coordinates, and then the picking needles are used to reject the foreign objects. The data for the seven foreign body shaped centers are shown in Table 6 and the comparison of robot coordinates with the actual removal coordinates are shown in Figure 12(b). Through the fixed-point removal of multiple groups of specific foreign objects, the pixel coordinates of the optimal group of detected foreign object centers highly coincide with the trajectory of the rejection coordinates of the centermost picking needle, errors of witch are lower than 0.5 mm, demonstrating the accurate automated detection, identification, localization, and removal of foreign objects from bird’s nest.

Table 6.

Foreign body shaped centers.

Test number	Pixel coordinate	Robot coordinate	Shift value of X/Y/Z axes	Removal coordinate	Error (mm)
1	[325, 2037]	[7.54, 47.24]	60.49/67.24/208	[7.49, 47.24]	0.34
2	[613, 729]	[14.22, 16.90]	66.93/36.74/208	[13.93, 16.74]	0.33
3	[877, 1826]	[20.34, 42.34]	72.99/62.10/209	[19.99, 42.10]	0.41
4	[1244, 1188]	[28.85, 27.55]	81.85/47.27/208	[28.85, 27.27]	0.28
5	[1671, 2262]	[38.75, 52.45]	91.75/72.45/208	[38.75, 52.45]	0.24
6	[1501, 970]	[34.80, 22.49]	87.8/42.49/207	[34.8, 22.49]	0.20
7	[1395, 507]	[32.35, 11.76]	85.35/31.76/208	[32.35, 11.76]	0.24

And as presented in Figure 12(c), it can be obtained that after the removal process through the robot, most of the foreign objects are cleaned away from the nest. The cleanliness rate can be as high as 92.78% with the loss rate as 1.96%. Working time of the robot is about 20 min, which is 7.07% faster than that of manual operation, demonstrating the reliable performance of our robot in foreign object removal applications.

Conclusions

In this study, we propose an automatic needle removal scheme for clearing foreign objects swallowed by birds. According to the design requirements of the scheme, we designed the overall structure of the needle removal robot, the key components of the end effector, and the structure of the vision module. Specifically targeting the need to remove foreign objects from bird nests, we conducted computational analysis on the key mechanisms of the end effector and performed specialized design on the parameters of the needle. Finally, we analyzed and quantified the key mechanisms of the end effector. This study successfully developed an automatic impurity removal system for bird’s nest based on parallel robots, overcoming the limitations of traditional manual removal methods, which are inefficient and lack precision. An innovative attention mechanism-enhanced U-Net++ algorithm was proposed, achieving an F₁ score of 94.80% and a recall rate of 97.90% for impurity identification in complex texture backgrounds, representing an improvement of over 10% compared to existing methods. A novel three-needle array actuator was developed, with key parameters optimized using a uniform design method (needle tip spacing of 0.45 mm, contact depth of 1.5 mm, and rotational speed of 16,000 rpm), achieving an optimal adhesion force of 0.539 N. This design maintains the structural integrity of the bird’s nest (with a loss rate of 1.96%) while achieving a cleanliness rate of 92.78%. The constructed “vision-control-execution” closed-loop system achieves sub-millimeter positioning accuracy (error <0.5 mm) with a single batch processing time of 20 min, improving efficiency by 7.07% compared to manual operations. This solution provides the first fully automated process solution for the bird’s nest processing industry, driving the industry’s intelligent upgrading.

Footnotes

Handling Editor: Guanglei Wu

ORCID iD

Ziheng Zhan

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported by the National Key R&D Program of China (2024YFD2100202) and the National Key R&D Program of China (2022YFD2100304).

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

References

Chua

Zukefli

SN.

A comprehensive review of edible bird nests and swiftlet farming. J Integr Med 2016; 14(6): 415–428.

Subramaniam

Fai

Ming

ESL

. Edible bird nest processing using machine vision and robotic arm. Jurnal Teknologi (Sciences & Engineering) 2015; 72(2).

Goh

Lai

Ting

, et al. Size characterisation of edible bird nest impurities: a preliminary study.Procedia Comput Sci;2017; 112:1072–1081.

Yee

Yeo

Cheng

, et al. Impurities detection in edible bird’s nest using optical segmentation and image fusion. Mach Vis Appl 2020; 31(7): 68.

Gwee

Cheng

Yen

KS.

Optimization of lighting parameters to improve visibility of impurities in edible bird’s nest. J Electron Imaging 2019; 28(2): 023014–023014.

Sun

Jiang

, et al. Structure boundary-preserving U-Net for prostate ultrasound image segmentation. Front Oncol 2022; 12: 900340.

Yeo

Yen

KS.

Development of a hybrid autoencoder model for automated edible bird’s nest impurities inspection. Journal of Electronic Imaging 2022; 31(5): 051603–051603.

Ayiz

Kucuk

The kinematics of industrial robot manipulators based on the exponential rotational matrices. 2009 IEEE international symposium on industrial electronics, IEEE, 2009.

Toz M and Kucuk

A comparative study for computational cost of fundamental robot manipulators. In: IEEE international conference on industrial technology,Auburn, AL, USA,14–16 March 2011, pp.289–293. New York: IEEE

10.

Toz M and Kucuk

Development of derivation of inverse Jacobian matrices for 195 6-DOF GSP mechanisms. Turk J Electr Eng Comput Sci 2016;24:4142–4153.

11.

Sun

Lian

Stiffness and mass optimization of parallel kinematic machine. Mech Mach Theory 2018; 120: 73–88.

12.

Tanev

TK.

Kinematics of a hybrid (parallel–serial) robot manipulator. Mech Mach Theory 2000; 35(9): 1183–1196.

13.

Kucuk

Dexterous workspace optimization for a new hybrid parallel robot manipulator. J Mech Robot 2018; 10(6): 064503.

14.

Toz M and Kucuk

Parallel manipulator software tool for design, analysis, and simulation of 195 GSP mechanisms. Comput Appl Eng Educ 2015; 23(6):931–946.

15.

Kucuk

The inverse kinematics of a new planar hybrid robot manipulator. In: Exploring the world of robot manipulators. Intech Open, 2024.

16.

Seenivasan

Chan Sin

Optimization of brushing, bubble, and microbubble techniques using Taguchi method for raw edible bird nest cleaning purpose. Pertanika J Sci Technol 2022;30 (2):1273–1288.

17.

Kucuk

Kinematics, singularity and dexterity analysis of planar parallel manipulators based on DH method. In: Lazinica

Kawai

(eds) Robot manipulators new achievements. Intech Open,2010, pp.387–400.

18.

Jazar

RN.

Theory of applied robotics. Springer Science+ Business Media, LLC, 2010.

19.

Chen

S-H

Hsu

C-H.

Using uniform design and regression methodology of turning parameters study of nickel alloy. Int J Adv Manuf Technol 2021; 116 (11–12): 3795–3808.

20.

Qin

Gao

, et al. Optimization of total flavonoids purification process in rose by uniform design method. Prep Biochem Biotechnol 2024; 54(7): 946–957.

21.

Zhao

Y-G

P-P.

Efficient approach for dynamic reliability analysis based on uniform design method and Box-Cox transformation. Mech Syst Signal Process 2022; 172: 108967.

22.

Liu

Research on liver image segmentation method based on deep convolutional neural networks. Harbin Institute of Technology, China,2019.

23.

Ẑeger

Grgic

Vuković

, et al. Grayscale image colorization methods: overview and evaluation. IEEE Access 2021; 9: 113326–113346.

24.

İncetaş

Uçar

, et al. A novel image Denoising approach using super resolution densely connected convolutional networks. Multimed Tools Appl 2022; 81 (23): 33291–33309.

25.

Riyue

. PyTorch deep learning concise practical guide. Beijing:Tsinghua University Press, 2022.

26.

Helmud

Fitriyani

Romadiana

Classification comparison performance of supervised machine learning random forest and decision tree algorithms using confusion matrix. J Sisfokom (Sistem Informasi dan Komputer) 2024; 13(1):92–97.

Design and optimization of a bird’s nest foreign object removal robot

Abstract

Keywords

Introduction

Results and discussions

Design of the robot

Parameter analysis and optimization

Parameter analysis and optimization

Optimization framework and experimental design

Regression modeling and parameter optimization

Static analysis of the pick needle

Visual discussions and applications

Image preprocessing

Improved U-Net++ method for detecting foreign objects in bird’s nest

Two-stage foreign object detection algorithm

Analysis of test results

Removal test of the robot

Conclusions

Footnotes

ORCID iD

Funding

Declaration of conflicting interests

References