High precision assembly and efficient dispensing approaches for millimeter objects

Abstract

In this article, high precision alignment and efficient dispensing approaches for millimeter objects are proposed. Firstly, a thin component and a cylindrical component are assembled together based on microscopic vision and laser range sensor. If the normal vector of upper surface of thin component is not perpendicular to the optical axis of side camera, the region around the lower edge line will be blurred easily in image. In order to measure the orientation of the thin component with high precision, a novel method based on laser range sensor is proposed. The two components are assembled after orientation and position alignment. In order to bond the two components, the glue is dispensed into all the 16 holes on the components. For keeping the two components firm after being dispensed, the volume of glue dispensed into the 16 holes should be controlled precisely. A laser range sensor is employed to measure the volume of fluid glue. The dispensing process should be finished in a limited time for the characteristics of fluid glue that will change with time. The fluid glue should be dispensed into the hole for some times to reach the expected volume. However, it is necessary to wait a little time in order to obtain stable level of glue after the glue is dispensed into the hole. Therefore, a dual dispensing strategy with higher efficiency is developed which fully utilizes the waiting time. Experimental results validate the effectiveness of the proposed system and methods.

Introduction

Precision assembly and manipulation are widely used in biotechnology, medical science, and microelectromechanism system, which address the manipulation problems of objects with size ranging from tens of microns to several millimeters. ^{1 –4} Usually, after being assembled, the components will be bonded by binders in order to keep stable and the precision and efficiency of dispensing are both highly required.

Alignment of two or more components is a key problem in precision assembly tasks where the orientation measurement is always the first step in the whole process. ^{5 –8} Vision, as an effective tool, is widely used in orientation measurement. ^{9 –13} In the study by Liu et al., ⁶ the orientation of thin annular component is measured by two side microscopic cameras. The lower edge line of component is detected to indicate its orientation in image space. Similarly, the side edge lines of cylindrical component are also detected to indicate its orientation. A coarse-to-fine relative orientation measurement method is proposed in the study by Liu et al. ⁵ where the relative orientation errors between two long cylindrical components are estimated by side view cameras. The invisible feature information of component is fully utilized for improving measurement accuracy. A method combined with Perspective-n-Point (PnP) and orthogonal iteration (OI) algorithms for orientation measurement is given by Chen et al., ¹⁴ which considered the real time, accuracy, and robustness, simultaneously. The aforementioned methods are all based on the geometrical features of components. Color information is also a feasible choice, based on which a three-dimensional orientation estimation method was proposed by Kyriakoulis and Gasteratos. ¹⁵ Several colored markers are arranged in a known geometry for helping orientation vector estimation. Besides vision information, laser sensor is another powerful tool which has been used in orientation measurement. A novel six-degree-of-freedom (DOF) orientation and position measurement system based on laser one-dimensional sensors and a camera has been designed which can measure the orientation of an object away from a long distance in the study by Kim et al. ¹⁶ A laser interferometry-based sensing and measurement method has been developed for high precision orientation measurement in the study by Clark et al. ¹⁷ As for orientation measurement of thin object, vision-based methods do not work in some cases for two reasons. One is the side edge lines occupy few pixels and the other is the region around lower edge line will be blurred easily when the normal vector of object is not perpendicular to the optical axis of camera. Therefore, laser-based orientation measurement methodology is needed.

Usually, after being alignment, the two or more components will be assembled together and one of the typical tasks is peg-in-hole assembly. ^18,19 There are many approaches to bond the assembled components and one of which utilizes the friction between components. For example, in the study by Liu et al., ⁶ the external diameter of cylindrical component is larger than the internal diameter of thin component. After inserting the cylindrical component into the thin component, two components can be bonded tightly by friction. The object can be grasped by dedicated mechanism where the tip of compliant arms is pushed into the slot. ¹¹ The binder, another available choice, is utilized to bond the microgripper to the end of probe of the micro-assembly robot. ²⁰ The fluid glue will solidify under the curing light. However, in some applications, especially in an extreme environment, the volume of fluid glue should be controlled with high precision in order to bond the components stably. And the volume measurement with high precision is the vital step. Laser range sensor is quite suitable to measure the volume of fluid glue in a hole with known size. ⁸ It is a challenge that how to align the laser range sensor with the center of fluid glue precisely. If the laser aligns with a noncentral point, the laser beam may be reflexed away rather into the sensor, which will cause measurement fail. Therefore, it is necessary to improve the alignment precision with novel strategy.

In industry, the binders are always dedicated which are light sensitive, ultraviolet sensitive, ²¹ and so on. Therefore, the glue dispensing process should be accomplished in a limited time. Generally, it is necessary to wait for a little time for the glue to become stable after being ejected from pipe or other ejectors. The reason is that the fluid glue will spread from the center to the edge owing to gravity. There will be two cases: the height of edge region will increase slowly but the height of central region will decrease instead. It means that the height measured during this time is invalid. Usually, if the viscosity of fluid glue is high, the waiting time can’t be neglected. For example, the dispensing strategy which dispenses each hole serially can’t meet the requirement of efficiency when using high viscosity binders. ⁸ Therefore, a dispensing strategy with high efficiency is highly needed.

The motivation of this article is to develop high precision alignment and efficient dispensing strategies for millimeter objects based on microscopic vision and laser range sensor. Two millimeter-level components are assembled and bonded with binder. First, in order to achieve precision orientation alignment, a novel orientation measurement method is proposed based on laser range sensor. Then, the two components are assembled under the guidance of microscopic vision and force sensor. Second, for aligning the laser range sensor with target hole of component, a new alignment strategy is presented. Finally, considering the property of the binder, especially high viscosity, a dual dispensing strategy with high efficiency is developed which fully utilizes the waiting time of the fluid glue.

The rest of this article is organized as follows. The second section introduces the system configuration and tasks specification. Automatic assembly methods including features extraction, orientation and position alignment control methods, and insertion strategies are detailed in the third section. Fluid glue dispensing including laser range sensor alignment strategy and efficient dispensing strategy is described in the fourth section. The fifth section presents the experiments and results. Finally, this article is concluded in the sixth section.

System configuration and tasks specification

System configuration

The automated precision assembly system is designed as shown in Figure 1. It consists of a four-DOF adjusting platform, a three-DOF manipulator, a five-DOF manipulator, three microscopic cameras, a laser range sensor, a dispenser, a high precision force sensor, corresponding lighting system, and a host computer.

Figure 1.

System configuration and its coordinates.

The optical axes of three microscopic cameras are approximately orthogonal to each other and the three microscopic cameras can move along their moving platform to adjust the distance between objective lens and objects for capturing clear images. Manipulator A can move along with $X_{\text{mA}}$ , $Y_{\text{mA}}$ , and $Z_{\text{mA}}$ axes and rotate around $X_{\text{mA}}$ and $Y_{\text{mA}}$ axes. Manipulator B can move along with $X_{\text{mB}}$ , $Y_{\text{mB}}$ , and $Z_{\text{mB}}$ axes. The four-DOF adjusting platform consists of three rotation DOFs around $X_{\text{w}}$ , $Y_{\text{w}}$ , and $Z_{\text{w}}$ axes, respectively, and a translation DOF along $Z_{\text{w}}$ axis. The laser range sensor can move along with $X_{\text{c1}}$ , $Y_{\text{c1}}$ , and $Z_{\text{c1}}$ axes.

The world coordinate {W} is established on the adjusting platform. The manipulator coordinates {M_A} and {M_B} are established on manipulator A and B, respectively. The camera coordinates {C₁}, {C₂}, and {C₃} are established on the three cameras, respectively. The laser coordinate {M₃} is established on the moving platform of camera 1. The force sensor coordinate {F} is established on the force sensor.

Tasks specification

The two components to be assembled are shown in Figure 2. Component A is a thin annular object, whose external diameter and thickness are 11 and 0.5 mm, respectively. Component B is a cylindrical object, whose external diameter and height are both 6 mm. The external diameter of component B is 20 µm larger than which of component A. Therefore, the assembly of components A and B is interference fit. There is an inclined surface on the top edge of component B, whose height is 60 µm and inclined angle is 30°. The internal diameter of component A can be merely stretched about 40 µm. There are 16 holes distributed evenly on components A and B.

Figure 2.

Components: (a) component A and (b) component B.

Components A and B are mounted on manipulator A and adjusting platform, respectively. By using microscopic vision and force sensor, the first task is to insert component B into component A with the requirement that the surfaces of two components are parallel to each other and all the 16 holes are aligned with high precision. The next task is to dispense glue into the 16 holes with a micro pipe which is mounted on manipulator B. Laser range sensor is employed to measure the height of glue in each hole.

Automation assembly

Calibration

The calibration items include image Jacobian matrices, the displacement vector V _L in {M₃} between the laser range sensor and the camera 1, and the transformation matrix between the movements in {M_A} and the forces in $\left\{\text{F}\right\}$ . The image Jacobian matrices are used to adjust the pose of components. V _L is used to guide the glue height measurement with laser range sensor. The image Jacobian matrices concerned include rotation matrix J ₁ and translation matrices J ₂ and J ₃. J ₁ represents the transformation from incremental rotation of feature lines in the Cartesian space to which on the images of three cameras. J ₂ represents the transformation from incremental translation of feature points in {M₃} to which on the images of cameras 2 and 3. J ₃ represents the transformation from incremental translation of feature points in {M_A} to which on the images of cameras 1 and 2. ${}^{\text{A}}{J}_{\text{F}}$ represents the transformation matrix between {F} and {M_A}. The calibration method is available in the study by Liu et al. ⁶

Feature extraction

Images features are mainly lines and points extracted from components A and B in three microscopic cameras which are used to modify the orientation and position of components A and B. The image features of components A and B in camera 1 are shown in Figure 3. The methods of image feature extraction are detailed in the study by Liu et al. ⁶

Figure 3.

Image features of components A and B, (a) component A in camera 1, (b) component B in camera 1, and (c) line features of component B in cameras 2 and 3.

In order to compute the orientation of component A, line features are extracted in two side cameras detailed in the study by Liu et al. ⁶ There exist two problems to be solved. First, the region around the lower edge line will be blurred easily if the normal vector of upper surface is not perpendicular to the optical axis of side camera. The lower edge line can’t accurately represent the orientation of component A in the case, as shown in Figure 4. Second, the lengths of left and right edge lines are quite short in image because the thickness of component A is quite small. The side lines are not credible enough to indicate the orientation of component A. Therefore, lines’ features in cameras 2 and 3 are not used in orientation measurement.

Figure 4.

Components A and B in side view. d is the thickness of component A.

In order to solve the problems aforementioned, a novel method for orientation measurement by using laser range sensor is proposed, as shown in Figure 5. The normal vector of the upper surface of component A is measured by laser range sensor which is then converted to cameras 2 and 3 by using image Jacobian matrixes.

Figure 5.

Sketch of orientation measurement based on laser range sensor.

A plane can be determined by three noncollinear points and the coordinates of points can be measured by using laser range sensor, as shown in Figure 5. P _ai represents the point on the surface of component A. The general equation of the surface plane of component A is expressed as equation (1) since the origin of { ${\text{M}}_3$ } is not on the plane

$ax+by+cz=1$

1

where a, b, and c are parameters of plane.

The plane parameters a, b, and c can be computed with at least three points via least square method. Then the normal vector V _A of the plane can be obtained. V _A can be converted to the image space of cameras 2 and 3 as

$\left[\begin{array}{c}{\mathit{V}}_{\text{Ac2}}\\ {\mathit{V}}_{\text{Ac3}}\end{array}\right]=J_2{\mathit{V}}_{\text{A}}$

2

${\mathit{V}}_{\text{Aci}}=\left[\begin{array}{c}{\mathit{V}}_{\text{Aci}}(x)\\ {\mathit{V}}_{\text{Aci}}(y)\end{array}\right]$

3

${\mathit{V}}_{\text{Ac2}}$ and ${\mathit{V}}_{\text{Ac3}}$ can represent the orientation of component A in image spaces of cameras 2 and 3, respectively. The angle error in image spaces of cameras 2 and 3 is expressed as

$\Delta {\theta }_{ci}=\frac{1}{2}({\theta }_{li}+{\theta }_{ri})-\text{arctan}\left(\frac{{\mathit{V}}_{\text{Aci}}(y)}{{\mathit{V}}_{\text{Aci}}(x)}\right),i=2,3$

4

where ${\mathit{V}}_{\text{Aci}}(x)$ and ${\mathit{V}}_{\text{Aci}}(y)$ are the two components of ${\mathit{V}}_{\text{Aci}}$ and i represents the number of camera. ${\theta }_{li}$ and ${\theta }_{ri}$ represent the angle of lines $l_{li}$ and $l_{ri}$ , respectively.

Alignment control and insertion strategies

Orientation and position alignment between components A and B are achieved sequentially under the guidance of vision and laser range sensor. The proportional–integral (PI) control law is employed to realize alignment, as given in the study by Liu et al. ⁶

During insertion procedure, the force direction of force sensor should be calibrated elaborately. Transformation matrix ${}^{\text{A}}{J}_{\text{F}}$ is calibrated, based on which an insertion strategy is proposed.

The insertion process is guided by force sensor, and its procedure is shown in Figure 6. In order to protect the components from damage, the radial force should be reduced to a safe range. Component B is moved up along with $Z_{\text{w}}$ axis by adjustment platform. The forces F _X , F _{Y ,} and F _Z along with X_f , Y_f , and Z_f axes are obtained from the force sensor in real time. F _XY is the resultant force of F _X and F _Y . If | F _XY | is larger than the threshold $F_{\text{T1}}$ , the component A should be adjusted with the step $\Delta d_x$ and $\Delta d_y$ along with $X_{\text{mA}}$ and $Y_{\text{mA}}$ axes according to equation (5), respectively, until | F _XY | is less than or equal to $F_{\text{T1}}$ .

Figure 6.

The flowchart of insertion process.

The force F _Z is checked if | F _XY | is less than the threshold $F_{\text{T1}}$ . The insertion process will finish if | F _Z |is larger than the threshold $F_{\text{T2}}$ stably. If F _Z is less than or equal to the threshold $F_{\text{T2}}$ , | F _XY | is checked again. In this situation, if | F _XY | is large than the threshold $F_{\text{T3}}$ , the component A should be adjusted according to equation (5) before the component B is moved up. Otherwise, the component B is moved up without the adjustment of component A.

$\left[\begin{array}{c}\Delta d_x\\ \Delta d_y\end{array}\right]=-\lambda {\cdot }^A{J}_F\left[\begin{array}{c}{F}_x\\ F_y\end{array}\right]$

5

where λ is a constant which is within the range from 0 to 1. F_x and F_y represent the expected force along X_f and Y_f axes, respectively.

In fact, the threshold $F_{\text{T1}}$ is larger than $F_{\text{T3}}$ . The insertion strategy is divided into two cases. When | F _XY | is larger than the threshold $F_{\text{T1}}$ , the insertion should be stopped and component A should be adjusted until | F _XY | is less than $F_{\text{T1}}$ . When | F _XY | is larger than $F_{\text{T3}}$ but less than or equal to $F_{\text{T1}}$ , the radial force is within the safe range, and the insertion can continue while the component A is being adjusted. The control system is designed, as shown in Figure 7.

Figure 7.

The block diagram of insertion control system.

Fluid glue dispensing

After being assembled, the two components A and B should be bonded by binder in order to keep the relative orientation and position unchanged when being used in practice. The fluid glue will be dispensed into the 16 holes. The dispensing precision and efficiency are quite important which will determine whether the components can work stably in extreme environment. In this section, a time–pressure dispensing model is first described. In order to align the laser range sensor with target hole with high precision, a searching-based method is proposed. Besides, a dual dispensing strategy with high efficiency is presented.

Time–pressure dispensing model

The time–pressure-based dispensing model is described in Figure 8 which is developed by Liu et al. ⁸ for nanoliter dispensing with high precision. Here, we provide a brief overview for the dispensing model. As shown in Figure 8(a), the fluid glue is ejected into the target hole from the pipe by putting the pressurized air into the pipe with specified pressure and time range. The altitude of the fluid glue in the hole is measured by laser range sensor.

$Q=\frac{\pi R_0^4}{8\mu L_n}(\Delta P+\rho g{H}_n)-\frac{\pi R_0^2{R}_1\sigma }{4\mu L_n}$

6

where Q represents the flow rate of the fluid glue; ρ represents the density of fluid glue; µ represents the viscosity of fluid glue; g represents gravitational acceleration; $\Delta P=P_u-P_l$ , where P_u and P_l are the pressure on the upper and lower surface of fluid glue, respectively; H_n is the relative altitude between the upper and lower surface of fluid glue; L_n is the length of fluid glue; and σ represents the tension coefficient of the adhesive layer between the pipe and the fluid glue.

Figure 8.

The time–pressure-based dispensing model. (a) Dispensing sketch map. (b) Force status of the flow with radius of r.

In equation (7), the total dispensing volume can be computed by integrating dispensing time

$\mathit{V}=\left[\frac{\pi R_0^4}{8\mu L_n}\left(\Delta P+\rho g{H}_n\right)-\frac{\pi R_0^2{R}_1\sigma }{4\mu L_n}\right]\times T$

7

where T represents the dispensing time.

Then, the pressure difference can be computed as

$\Delta P=\frac{8\mu L_n\mathit{V}+2\pi R_0^2{R}_1\sigma T}{\pi R_0^{4}T}-\rho g{H}_n$

8

Dispensing control strategies

After components A and B are assembled, fluid glue should be dispensed into 16 holes with expected volume. As the radius of each hole is known, the dispensing volume can be computed according to the fluid glue height measured by laser range sensor, as shown in Figure 8(a). The procedures of dispensing process are given in Figure 9. The initialization includes injecting the fluid glue into the pipe, adjusting angle of pipe, and so on. Then, the pipe is moved into the view of camera 1 and aligns with the target hole. Next the laser range sensor aligns with the target hole for measuring the altitude of fluid glue in the hole. Finally, the fluid glue dispensing is completed with the dispensing strategies detailed as follows.

Figure 9.

The flowchart of dispensing process.

Laser range sensor alignment

In order to measure the fluid glue altitude by the laser range sensor, the laser range sensor should align with the target hole guided by microscopic camera 1. Because the viscosity of fluid glue is high, the higher precision of laser range sensor alignment is acquired. The shape of the fluid glue surface in the hole is cambered. If the laser range sensor does not align with the central region of fluid glue, the laser beam may be reflexed away rather than into the sensor, which will cause measurement fail. The alignment method employed in the study by Liu et al. ⁸ can’t meet the requirement of high precision. A new alignment method is proposed, as shown in Figure 10. l_p is the optical axis of laser range sensor. $P_{\text{c0}}$ is the central point of the image captured by camera 1. The vector V _L is the translation vector between l_p and $P_{\text{c0}}$ in $\left\{{\text{M}}_3\right\}$ , which is calibrated off-line.

Figure 10.

Sketch of laser range sensor alignment, (a) coarse alignment and (b) fine alignment.

Laser range sensor alignment includes coarse and fine alignment. In coarse alignment stage, the camera is firstly moved for making $P_{\text{c0}}$ overlaps $P_{\text{h0}}$ . $P_{\text{h0}}$ is the center of target hole. The corresponding translation vector V _H can be gotten by equation (9). Then, according to V _L and V _H, the laser range sensor can be moved to the center of the target hole. Because V _L and V _H may have errors, laser range sensor may pass through the point $P_{\text{A}}$ , shown in Figure 10(b), rather than $P_{\text{h0}}$ . Therefore, fine alignment is necessary. The laser range sensor is moved along $X_{\text{M3}}$ axis with step d_L . The edge points $P_{\text{B1}}$ and $P_{\text{B2}}$ of the hole can be gotten. Then, laser range sensor is moved to $P_{\text{B}}$ , which is the midpoint between $P_{\text{B1}}$ and $P_{\text{B2}}$ . Next the laser range sensor is moved along $Y_{\text{M3}}$ axis and the edge points $P_{\text{B3}}$ and $P_{\text{B4}}$ of the hole can be gotten. $P_{\text{h0}}$ is the midpoint between $P_{\text{B3}}$ and $P_{\text{B4}}$ . The laser range sensor finally is moved to $P_{\text{h0}}$ . The edge points can be discriminated by their altitudes. The point is considered as an edge point if its altitude deviation to $P_{\text{A}}$ exceeds the threshold $\Delta h_{\text{T}}$

${\mathit{V}}_{\text{H}}=\left[\begin{array}{c}\Delta x_{\text{M}3}\\ \Delta y_{\text{M}3}\end{array}\right]={\left(J_4^T{J}_4\right)}^{-1}{J}_4^T\left(\left[\begin{array}{c}\Delta u_{\text{h}0}\\ \Delta v_{\text{h}0}\end{array}\right]-\left[\begin{array}{c}\Delta u_{\text{c}0}\\ \Delta v_{\text{c}0}\end{array}\right]\right)$

9

where J ₄ represents the transformation from incremental translation of feature points along with $X_{\text{M3}}$ and $Y_{\text{M3}}$ axes to which on the image of camera 1.

The altitudes at $P_{\text{h1}}$ , $P_{\text{h2}}$ , $P_{\text{h3}}$ , and $P_{\text{h4}}$ are firstly measured and the average altitude is served as the reference altitude. Subtracting the reference altitude from the value measured by laser range sensor provides the fluid glue altitude.

Dispensing control

The dispensing control system is shown in Figure 11, which is based on the time–pressure model described in the fourth section. A PI controller is used for the dispensing control. $h_{\text{de}}$ represents the expected fluid glue altitude in the target hole. $h_{\text{dc}}$ is the current fluid glue altitude measured by laser range sensor in real time. The output of the PI controller is computed by the following equation

$\Delta h_{\text{do}}=K_{\text{pd}}(\Delta h_{\text{d}}\left(k\right)-\Delta h_{\text{d}}\left(k-1\right))+K_{\text{id}}\Delta h_{\text{d}}\left(k\right)$

10

where $K_{\text{pd}}$ and $K_{\text{id}}$ are the proportional and integral factors of the PI controller and k represents the sampling moment.

Figure 11.

The diagram of dispensing control.

The fluid glue altitude increment $\Delta h_{\text{do}}$ is converted to volume increment $\Delta V$ based on the geometric information of the hole. According to equations (11) and (12) and the dispensing time $T_{\text{s}}$ , the final dispensing pressure $P_{\text{d}}$ and time $T_{\text{d}}$ are computed based on P ₁ and $\Delta h_{\text{d}}$

$P_{\text{d}}=\{\begin{array}{l}{P}_{\text{s}},\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\text{if}\hspace{1em}\Delta h_{\text{d}}>\Delta h_{\text{T}}\\ \text{min}(P_{\text{s}},P_1),\hspace{1em}\hspace{1em} \text{if}\hspace{1em}\Delta h_{\text{d}}\le \Delta h_{\text{T}}\hspace{1em}\text{and}\hspace{1em}{P}_{\text{1}}>P_{\text{L}}\\ P_{\text{L}},\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\text{if}\hspace{1em}\Delta h_{\text{d}}\le \Delta h_{\text{T}}\hspace{1em}\text{and}\hspace{1em}{P}_{\text{1}}\le P_{\text{L}}\end{array}$

11

$T_{\text{d}}=\{\begin{array}{l}{T}_{\text{s}},\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\text{if}\hspace{1em}\Delta h_{\text{d}}>\Delta h_{\text{T}}\\ T_{\text{s}},\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\text{if}\hspace{1em}\Delta h_{\text{d}}\le \Delta h_{\text{T}}\hspace{1em}\text{and}\hspace{1em}{P}_{\text{1}}>P_{\text{L}}\\ T_{\text{s}}\Delta V/V_{\text{L}1}\hspace{1em}\hspace{1em}\hspace{1em}\text{if}\hspace{1em}\Delta h_{\text{d}}\le \Delta h_{\text{T}}\hspace{1em}\text{and}\hspace{1em}{P}_{\text{1}}\le P_{\text{L}}\end{array}$

12

where $V_{\text{L1}}$ is the dispensing volume calculated by equation (7) with $\Delta P=P_{\text{L}}$ and $T=T_{\text{s}}$ . P ₁ is the pressure difference computed from equation (8) with dispensing time $T=T_{\text{s}}$ . $P_{\text{L}}$ represents the lowest pressure the fluid glue can be ejected out. $P_{\text{s}}$ is the given constant pressure. $T_{\text{s}}$ is the given constant dispensing time. $\Delta h_{\text{T}}$ is a threshold of altitude.

If the altitude difference $\Delta h_{\text{d}}>\Delta h_{\text{T}}$ , the dispensing process belongs to the rapid dispensing stage and the fluid glue is dispensed with parameter $P_{\text{d}}=P_{\text{s}}$ and $T_{\text{d}}=T_{\text{s}}$ . If $\Delta h_{\text{d}}\le \Delta h_{\text{T}}$ , the dispensing strategy is changed to precision dispensing stage. When $P_{\text{1}}>P_{\text{L}}$ , the minimum of $P_{\text{s}}$ and P ₁ is selected as the working pressure for guarantee the output pressure within a safe range, and the dispensing time $T=T_{\text{s}}$ . If $\Delta h_{\text{d}}\le \Delta h_{\text{T}}$ and $P_1\le P_{\text{L}}$ , then the pressure $P_{\text{d}}=P_{\text{L}}$ and the dispensing time $T_{\text{d}}$ should be adjusted to $T_{\text{s}}\Delta V/V_{\text{L}1}$ .

Dual dispensing strategy

Generally, the fluid glue should wait a little time to become stable after being dispensed into the target hole from the pipe. The fluid glue will spread from the central region to the edge region owing to gravity.

Figure 12(a) shows the pipe aligns with the target hole. In rapid dispensing stage, the pipe is moved along $Z_{\text{mB}}$ axis for d ₁, as shown in Figure 12(b). The pipe is moved to the position, as shown in Figure 12(a), after rapid dispensing. In precision dispensing stage, the altitude of fluid glue in the hole should be fed back in real time for guiding high precision dispensing. However, once the fluid glue is dispensed into the hole, the fluid glue will flow slowly from the central region to the edge region which will cause altitude measurement errors. Therefore, it is necessary to wait a little time before measuring the altitude of fluid glue. If the waiting time can be fully utilized, the efficiency of dispensing will be improved.

Figure 12.

Sketch of pipe movement strategies in one-time dispensing procedure. (a) The pipe aligns with the target hole, (b) the pipe movement in one hole, and (c) and (d) the movement of pipe and laser range sensor in the dual dispensing strategy.

A dual dispensing strategy with high efficiency is proposed. The fluid glue is dispensed into two holes simultaneously. Firstly, fluid glue is dispensed into two holes sequentially by using the rapid dispensing method. Then dual dispensing strategy works. As shown in Figure 12(c), the laser range sensor measures the altitude of the fluid glue in target holes 1 and 2 sequentially. The corresponding positions are saved for fast movement from one hole to the other hole. Then, the pipe is moved to align with target hole 1. According to the aforementioned dispensing control strategies, fluid glue is dispensed into target hole 1. Then, the pipe is moved to align with target hole 2 and laser range sensor is moved to align with target hole 1, as shown in Figure 12(d). The pipe and laser range sensor obey this moving strategy until the fluid glue altitude of any one hole reaches expected altitude. The pipe movement in one hole is shown in Figure 12(b). The pipe is moved along $Z_{\text{mB}}$ axis for d ₁. Because the total time of pipe movement time t _m and one dispensing time $t_{\text{d}}$ in a target hole is longer than the time $t_{\text{w}}$ of fluid glue becoming stable, the pipe will keep working, which can achieve fast dispensing.

Experiments and results

Experiment system

An experiment system was established according to the scheme given in the second section, as shown in Figure 13. In this experiment system, there are three microscopic cameras with Baumer TXG50. All the three cameras are equipped with a Navitar zoom lens with magnification 0.7–4.5×, which capture images 15 f/s with an image size of 2448 × 2050 in pixel. The adjusting platform is composed of KOHZU PK544PMB for rotation around $X_{\text{w}}$ , $Y_{\text{w}}$ , and $Z_{\text{w}}$ axes and translation along $Z_{\text{w}}$ axis. The rotation and translation resolutions of adjusting platform are 0.002° and 1 µm, respectively. The manipulator B is composed of KOHZU PK523HPMB for translation along $X_{\text{mB}}$ , $Y_{\text{mB}}$ , and $Z_{\text{mB}}$ axes and the resolution is 1 µm. The manipulator A is composed of KOHZU PK566-B for translation along $X_{\text{mA}}$ , $Y_{\text{mA}}$ , and $Z_{\text{mA}}$ axes and KOHZU PK544PMB for rotation around $X_{\text{mA}}$ and $Y_{\text{mA}}$ axes. The rotation and translation resolutions of manipulator A are 0.002° and 1 µm, respectively. Microscopic cameras 2 and 3 are mounted on SURUGA C005C-90215P-1 and microscopic camera 1 is mounted on KOHZU PK566-B. The laser range sensor is STIL CCS PRIMA, which can be moved together with camera 1. The measuring range and resolution of the laser range sensor are 1400 µm and 3 nm, respectively. The force sensor is Nano-43, whose measuring range is 18 N and its resolution is 1/128 N.

Figure 13.

Experimental system.

The image Jacobian matrices J ₁, J ₂, J ₃, and J ₄ and the force transformation matrix ${}^{\text{A}}{J}_{\text{F}}$ were calibrated, which are shown as follows

$J_1=\left[\begin{array}{lll}0.936& 0.048& -0.023\\ -0.114& 1.132& -0.032\\ 0.011& -0.118& -0.960\end{array}\right]$

$J_2=\left[\begin{array}{ccc}-0.0006& -0.2581& 0.0030\\ 0.0003& -0.0025& -0.2572\\ 0.2611& -0.001& -0.0025\\ -0.0031& 0.0008& -0.2522\end{array}\right]\text{pixel/µm}$

$J_3=\left[\begin{array}{ccc}-0.0032& -0.2583& 0.0005\\ -0.2618& 0.0008& 0.0002\\ 0.0030& -0.0012& -0.2556\end{array}\right]\text{pixel/µm}$

$J_4=\left[\begin{array}{cc}0.0041& 0.2599\\ 0.2619& -0.0036\end{array}\right]\text{pixel/µm}$

${}^{\text{A}}{J}_{\text{F}}=\left[\begin{array}{cc}-0.7551& 0.6558\\ -0.6679& -0.7444\end{array}\right]$

Automatic assembly experiments

The features were extracted as described in the third section. The parameters of PI controllers were set as follows: $K_{\text{p}}$ = 0.15 and $K_{\text{i}}$ = 0.5. $K_{\text{p}}$ is the proportional coefficient. $K_{\text{i}}$ is the integral coefficient. The expected accuracy was 0.05° and 4 µm for orientation alignment and position alignment, respectively. The orientation and position alignment finished once the errors were less than their corresponding expected accuracy. The parameters in insertion process were set as follows: $F_{\text{T1}}$ = 50 mN, $F_{\text{T2}}$ = 500 mN, $F_{\text{T3}}$ = 30 mN, and λ = 0.07. The two thresholds $F_{\text{T1}}$ and $F_{\text{T3}}$ were determined according to the acceptable radial force of component A.

The automatic assembly process includes orientation alignment, position alignment, and insertion process with the methods discussed in the third section. The component A was aligned with the component B in orientation and position with high precision. The alignment errors in one assembly experiment are shown in Figure 14. Figure 15 shows the images captured by three cameras before alignment, after alignment, and after insertion. The final orientation errors around $X_{\text{w}}$ , $Y_{\text{w}}$ , and $Z_{\text{w}}$ axes were −0.01°, −0.01°, and −0.04°, respectively, after seven control steps. The position errors along with $X_{\text{mA}}$ , $Y_{\text{mA}}$ , and $Z_{\text{mA}}$ axes were −3.7, −2.0, and −4.0 µm after nine control steps. It can be seen that the whole control process of orientation and position alignment was quickly and steadily.

Figure 14.

The errors in (a) orientation alignment and (b) position alignment.

Figure 15.

Images captured by (a1) microscopic camera 1, (b1) microscopic camera 2, (c1) microscopic camera 3 before alignment, (a2) microscopic camera 1, (b2) microscopic camera 2, (c2) microscopic camera 3 after alignment, (a3) microscopic camera 1, (b3) microscopic camera 2, and (c3) microscopic camera 3 after insertion.

Dispensing experiments

The fluid glue should be dispensed into all 16 holes and the results in one dispensing experiment are given as follows. According to the fast dispensing strategy described in the fourth section, two holes are dispensed in one dispensing batch.

The parameters of dispensing control are given as follows: $P_{\text{s}}$ = 25 kPa, $T_{\text{s}}$ = 1.0 s, $P_{\text{L}}$ = 4 kPa, R ₀ = 18 µm, $K_{\text{pd}}$ = 0.1, $K_{\text{id}}$ = 0.3, and $\Delta h_{\text{T}}$ = 100 µm. The expected altitude is 360 µm.

After the rapid dispensing stage, the relative altitude error of fluid glue in the two holes became 33.5 and 37.8 µm, respectively. The relative altitude errors during precision dispensing stage are given in Figure 16. The parameters $P_{\text{d}}$ and $T_{\text{d}}$ in each step of precision stage are show in Table 1. The time $t_{\text{m}}$ and $t_{\text{d}}$ are about 4 and 16 s, respectively. $t_{\text{w}}$ is 15 s. The total time for two holes fluid glue dispensing is about 400 s. If only each hole is dispensed independently, the total time of two holes dispensing is about 640 s. By using the fast dispensing strategy, the efficiency can be improved about 37%. The final altitude relative errors of whole 16 holes are given in Table 2.

Figure 16.

Relative altitude errors in precision dispensing stage.

Table 1.

$P_{\text{d}}$ and $T_{\text{d}}$ in precision dispensing stage.

	Target hole 1		Target hole 2
Step number	$P_{\text{d}}$ (kPa)	$T_{\text{d}}$ (s)	$P_{\text{d}}$ (kPa)	$T_{\text{d}}$ (s)
1	25.0	1.0	25.0	1.0
2	25.0	1.0	25.0	1.0
3	25.0	1.0	25.0	1.0
4	25.0	1.0	25.0	1.0
5	25.0	1.0	25.0	1.0
6	25.0	1.0	25.0	1.0
7	25.0	1.0	17.9	1.0
8	22.4	1.0	11.8	1.0
9	15.8	1.0	6.3	1.0
10	12.0	1.0
11	8.0	1.0
12	6.6	1.0

Table 2.

Final relative altitude errors of 16 holes.

Hole number	Errors (µm)	Hole number	Errors (µm)
1	1.4	9	1.5
2	1.3	10	1.4
3	1.0	11	1.6
4	−1.2	12	−1.8
5	1.8	13	−1.2
6	1.2	14	1.7
7	−1.7	15	1.4
8	−1.5	16	1.5

Comparative experiments

The orientation measurement method in the study by Liu et al. ⁶ is selected as the comparative method. In the method by Liu et al., ⁶ the orientation of component A is measured by extracting the lower surface edge in images. When the component A is rotated around $X_{\text{mA}}$ and $Y_{\text{mA}}$ axes, the angle variations in the images of cameras 2 and 3 are $\Delta {\alpha }_2$ and $\Delta {\alpha }_3$ , respectively. Then the rotation angles around $X_{\text{mA}}$ and $Y_{\text{mA}}$ axes can be computed by the following equation

$\left[\begin{array}{c}\Delta {\theta }_{x1}\\ \Delta {\theta }_{y1}\end{array}\right]=J_{r1}\left[\begin{array}{c}\Delta {\alpha }_2\\ \Delta {\alpha }_3\end{array}\right]$

13

where $J_{r1}$ represents the relationship between the angle increments on the image and in $\left\{{\text{M}}_{\text{A}}\right\}$ . ${\left[\Delta {\theta }_{x1},\Delta {\theta }_{y1}\right]}^T$ represents the corresponding angle variations in $\left\{{\text{M}}_{\text{A}}\right\}$ . ${\left[\Delta {\alpha }_2,\Delta {\alpha }_3\right]}^T$ represents the angle variations in the images of cameras 2 and 3.

The method in this article can directly measure the normal vector of component A in $\left\{{\text{M}}_3\right\}$ . When the component A is rotated around $X_{\text{mA}}$ and $Y_{\text{mA}}$ axes, the normal vector V_A is rotated for $\Delta {\beta }_x$ and $\Delta {\beta }_y$ around $X_{\text{m3}}$ and $Y_{\text{m3}}$ , respectively. Similarly, the rotation angles around $X_{\text{mA}}$ and $Y_{\text{mA}}$ axes can be computed by the following equation

$\left[\begin{array}{c}\Delta {\theta }_{x2}\\ \Delta {\theta }_{y2}\end{array}\right]=J_{r2}\left[\begin{array}{c}\Delta {\beta }_x\\ \Delta {\beta }_y\end{array}\right]$

14

where $J_{r2}$ represents the relationship between the angle increments in $\left\{{\text{M}}_3\right\}$ and $\left\{{\text{M}}_{\mathrm{A}}\right\}$ . ${\left[\Delta {\theta }_{x2},\Delta {\theta }_{y2}\right]}^T$ represents the corresponding angle variations in $\left\{{\text{M}}_{\mathrm{A}}\right\}$ . ${\left[\Delta {\beta }_x,\Delta {\beta }_y\right]}^T$ represents the angle variations in $\left\{{\text{M}}_3\right\}$ .

$\Delta {\theta }_{x1}$ , $\Delta {\theta }_{y1}$ , $\Delta {\theta }_{x2}$ , and $\Delta {\theta }_{y2}$ can be obtained according to equations (13) and (14), after rotating the component A for $\Delta {\theta }_{x0}$ and $\Delta {\theta }_{y0}$ around $X_{\text{mA}}$ and $Y_{\text{mA}}$ axes, respectively. Comparing $\Delta {\theta }_{x1}$ , $\Delta {\theta }_{y1}$ , $\Delta {\theta }_{x2}$ , and $\Delta {\theta }_{y2}$ with $\Delta {\theta }_{x0}$ and $\Delta {\theta }_{y0}$ , the measurement errors can be computed. Three experiments are conducted: (a) rotating around $X_{\text{mA}}$ for five times with angle step 0.5°, (b) rotating around $Y_{\text{mA}}$ for five times with angle step 0.5°, and (c) rotating around $X_{\text{mA}}$ and $Y_{\text{mA}}$ simultaneously for five times with angle step 0.5°. Figure 17 shows the measurement results of rotating around $Y_{\text{mA}}$ for 2.5° by using the method by Liu et al. ⁶ It can be seen that the edge line is inaccurate because the region of lower surface is blurred. The measurement errors are −0.33° and −3.76° around $X_{\text{mA}}$ and $Y_{\text{mA}}$ axes. On the contrast, the measurement errors by using the method in this article are −0.01° and −0.31°, respectively, which are much smaller than those of method by Liu et al. ⁶ All the experiment results are shown in Table 3.

Figure 17.

The edge lines detected results by using the method by Liu et al. ⁶ (a) The result in camera 2, (b) the amplified image of (a), (c) the result in camera 3, and (d) the amplified image of (c).

Table 3.

Comparative experiment results (in degrees).

Actual angle	$\Delta {\theta }_{x0}$	0.5	1.0	1.5	2.0	2.5
	$\Delta {\theta }_{\text{y0}}$	0.0	0.0	0.0	0.0	0.0
Method by Liu et al. 6	$\Delta {\theta }_{x1}$	1.31	2.05	2.99	3.66	4.21
	$\Delta {\theta }_{y1}$	−0.28	−2.69	−10.33	−12.57	−12.97
Our method	$\Delta {\theta }_{x2}$	0.43	0.94	1.25	1.55	2.11
	$\Delta {\theta }_{y2}$	0.02	0.23	0.18	0.03	-0.11
Actual angle	$\Delta {\theta }_{x0}$	0.0	0.0	0.0	0.0	0.0
	$\Delta {\theta }_{y0}$	0.5	1.0	1.5	2.0	2.5
Method by Liu et al. 6	$\Delta {\theta }_{x1}$	−0.11	−0.13	−0.09	−0.19	−0.33
	$\Delta {\theta }_{y1}$	0.52	−0.69	−1.91	−3.47	−1.26
Our method	$\Delta {\theta }_{x2}$	−0.00	−0.14	−0.18	−0.09	−0.01
	$\Delta {\theta }_{y2}$	0.34	0.86	1.45	1.70	2.18
Actual angle	$\Delta {\theta }_{x0}$	0.5	1.0	1.5	2.0	2.5
	$\Delta {\theta }_{y0}$	0.5	1.0	1.5	2.0	2.5
Method by Liu et al. 6	$\Delta {\theta }_{x1}$	1.33	2.23	2.88	3.13	3.52
	$\Delta {\theta }_{y1}$	−1.26	−6.20	−10.85	−10.23	−10.08
Our method	$\Delta {\theta }_{x2}$	0.53	0.96	1.24	1.61	2.03
	$\Delta {\theta }_{y2}$	0.31	0.87	1.38	1.81	2.24

The average angle errors around $X_{\text{mA}}$ and $Y_{\text{mA}}$ axes are 0.88° and 6.62° with the method by Liu et al. ⁶ In contrast, the corresponding angle errors with our method are 0.19° and 0.15°. Obviously, our method outperforms the method by Liu et al. ⁶

Conclusion

The main contribution of this work is the high precision alignment and efficient dispensing approaches for millimeter objects based on microscopic vision and laser range sensor. An orientation measurement method based on laser range sensor is developed to measure the norm vector of upper surface. Then, it is converted to the increments on the images captured by the two side cameras via image Jacobian matrixes. The measure accuracy is satisfactory even if the normal vector of upper surface of thin component is not perpendicular to the optical axis of side camera. Two components are aligned in orientation and position with high precision according to the measurement results. After assembling the two components, fluid glue dispensing is used to bond them. For keeping the two components stable, the volume of fluid glue dispensed into the 16 holes should be controlled precisely. Primarily, a high precision searching-based method is proposed to align the laser range sensor and target hole. Furthermore, in order to keep the fluid glue reliability, the dispensing process of 16 holes should be finished in a limited time which means the efficiency is quite important. A dual dispensing strategy with high efficiency is developed, which utilizes the waiting time for the fluid glue spreading in a hole to dispense in another hole. The efficiency can be improved about 37%. Experimental results demonstrate the effectiveness of the proposed methods.

In the future, we will focus on intelligent control methods in assembly and dispensing process.