Sage Journals: Discover world-class research

Abstract

UAV application research has attracted more and more attention of researchers. However, in the field of UAV transportation, especially in the case of multi-UAV cooperation, there is little research or design on the overall system of multi-UAV cooperative transportation due to the complexity of the task and many factors need to be considered. In this paper, we first introduce a 2-UAV cooperative transportation task scenario and we call it Co-TS. Then according to the scenario, we design a task planning system, which describes the entire process of 2-UAV performing transportation tasks; in order to solve the design complexity of this UAV upper-layer software, we propose a model architecture of 2-UAV transportation application, design the functions of each module and analyze the relationship between modules in the model architecture; in addition, to realize the correct execution of the 2-UAV cooperative transportation task, the task execution is divided into four states which include preparation, rising, forwarding, and landing states, and the corresponding cooperative transportation control flow algorithm is designed for four states. Finally, we develop this system and conduct experiments. The quantitative and qualitative results demonstrate the effectiveness of our method.

Keywords

multi-UAV transportation system task planning model architecture control flow algorithm

1. Introduction

With the development of UAV (Unmanned Aerial Vehicle), UAVs are widely used in various fields, such as UAV formation control,¹ traffic monitoring,² crowd surveillance,³ and so on. Especially in transportation tasks,^4,5 UAVs have the advantages of fast deployment, low maintenance costs, and strong mobility. Also, compared with the traditional transportation mode such as unmanned vehicle transportation, UAV transportation is not limited by ground conditions, and can choose relatively short air routes. However, a single UAV operation in this field has several limits. On the one hand, the weight of objects that can be transported by a single UAV is limited. On the other hand, for specially shaped payload, a single UAV cannot maintain balance and cannot well operate the attitude and position of the payload in the suspended state.^6,7 In this sense, to address these problems, multi-UAV is selected as one of the solutions for transportation tasks which is paid more attention than ever before.

There are some existing research on this topic. Gassner et al.⁸ propose a collaborative transportation scheme, in which two UAVs transport a suspension cable load, but this is a static process so that UAVs can't perceive the external environment and have no autonomy. Once the external environment changes dynamically, the static methods will not work. In,⁹ Webb et al. propose and develop a modular solution for handling larger packages. By allowing small UAVs teams to cooperatively lift packages using adaptive control algorithms, this strategy could allow a combination of several standard-sized UAVs to deliver a variety of packages. However, their method needs to consider the knowledge of dynamics, the weight of the payload, and the center of gravity, which increases the difficulty of UAV application deployment and reduces autonomous of the UAV. In recent years, with the hardware processing capacity of edge computing equipment constantly improving, the next generation of UAVs will have the ability to perceive plan, reason, share and act to accomplish their tasks.¹⁰ Therefore, AI models and application software can be deployed on the onboard computer such as Jetson-nano, which promotes the design and development of intelligent UAVs.^11–15 Kim et al.¹⁶ design the scenario of multi-UAV lifting the payload and propose a method of lifting the payload autonomously by quadrotor UAVs based on vision. Though computer vision technology is applied in this study, this paper does not propose a method of transporting the payload to the designated location. To the best of our knowledge, few studies have been done to design and develop an intelligent multi-UAV transportation system due to both the complexity of a multi-UAV collaborative transportation task and the uncertainty of environmental conditions. In this sense, how to address the above problems are the initial motivation of this paper.

We summarize the three main challenges when developing an intelligent multi-UAV collaboration system. (1) Decisions executed autonomously: Multi-UAV transportation payloads are faced with complex tasks, multi-source information processing, and many uncertain factors. (2) Model design: How to design corresponding functional modules for each UAV according to the task goal, and how each functional module interacts to ensure the accuracy of the UAV's action. (3) Stability of system: The distance between the two UAVs is too large or too small, making the payload unbalanced or in an unsafe state. UAVs need to have the ability to respond to interference in real time.

To address the above challenges, we developed the Co-TS (Cooperative Transportation System) that UAVs to collaborate and work together autonomously to achieve the transportation task. In order to let UAVs be autonomous, firstly, we propose a task planning system, which determines that our Co-TS system is based on a distributed architecture. Each UAV is an independent agent that has the ability such as environmental perception, data management, and collaborative control to achieve the ultimate task goal. Secondly, based on the task planning system, we design a 2-UAV cooperative transportation model architecture with different modules. The model architecture includes the management module, internal data management module, data assessment module, autonomous decision module, collaborative module, and communication module. We implement the function of each module and analyze the relationship and interaction between modules. The task planning system and model architecture provide the software foundation for UAV autonomy. Thirdly, based on the task planning and model architecture, we divide the task into four states which include preparation, rising, forwarding, and landing states, and design the control flow algorithm respectively. In every state, the UAV can obtain and perceive the external environment information and other UAV information in real-time. Based on the data information, the UAVs make or change the corresponding actions through the control flow algorithm to complete the task goal autonomously. To keep system stability such as keeping payload balance and a safe distance between UAVs under external environmental factors. We add the algorithm to maintain the balance of the payload in the case that the two ends of the payload rope may not be equal in length, and we add the distance constraint to the UAV in the control flow algorithm. Finally, the experiment is verified in our real experiment scene.

Four main contributions of this paper are summarized as follows:

We analyze the preconditions and task goals of our scenarios that realized are more general. The multi-UAV can transport the payload to the designated place in a balanced manner, and the designated place is more arbitrary.

According to the scenarios and goals, we propose a task planning system from a singer UAV to multi-UAV. UAVs have complete autonomy without considering dynamics knowledge.

The model architecture is designed based on the task planning system. The function of each module and the relationship between modules are proposed.

The control flow algorithm of the complete task system is designed based on the task planning and model architecture. The algorithm not only enables multi-UAV to complete tasks cooperatively but also makes the Co-TS system have the anti-interference ability under the influence of the external environment such as wind, so it can maintain a safe distance between drones and a balance of payload.

This paper is organized as follows: Section 2 shows the scenario of our 2-UAV cooperative transportation task, the preconditions, and the goals of this task. Section 3 introduces the task planning system and model architecture, and analyzes every module's function and the relationship between modules. Section 4 demonstrates the task control flow algorithm. Section 5 shows implementation details and the real-scene verification experiment. Section 6 concludes this paper.

2. Scenario

In this paper, we call our 2-UAV cooperative transportation system as Co-TS, which is a cyber-physical system (CPS)¹⁷ connecting 2 UAVs and a ground station via LAN. The scenario of the task is that 2 UAVs can autonomously cooperate to transport a payload to a designated place. UAVs are designed to make autonomous decisions based on their current goals, capabilities, and real-time data. They build and maintain data of the task by directly observing the environment (for example, by using onboard sensors) and by receiving a message from other UAVs, and ground station.¹⁸ The UAVs then perform a series of actions to achieve their goals.

The Co-TS system has the following specific goal that UAVs land the left end of the payload on the red marker and the right end on the yellow marker. Figure 1 shows two UAVs working together to balance and accurately lift, carry, fly, identify, and drop the payload at a certain location. We use markers as target position points to guide two UAVs in the air. The red marker is randomly located at any position in a straight line. The yellow marker is randomly located at any position on the circle where the red marker is the circle center. The radius is equal to the payload length. The purpose of this setting is to make the target point have a delivery uncertainty and make the scene application more generic so that the intelligence of the UAV can be reflected.

Figure 1.

2-UAV cooperative transport scenario. (a) The red and yellow markers are located in the center and on the circle, respectively, and L equals to R; (b) real scenario.

With the above goals, the Co-TS has the following pre-conditions. The UAV uses various information as the basis for decision-making in Co-TS system. In order to acquire the information, firstly, the UAV's status information is based on GPS, IMU, and other sensors; secondly, to obtain visual information without a blind area, each UAV is equipped with 2 cameras in the front and below respectively; thirdly, the balance sensor is attached to the payload so that the UAV can obtain the balance state of the payload. In addition, each UAV is equipped with an onboard computer and a flight controller, which are the hardware foundation for UAVs to make intelligent decisions. All the modules designed in this paper are deployed on the onboard computer. The flight controller as an intermediary accepts the upper-level commands from the onboard computer and outputs UAV hardware instructions.

3. Detailed design for the co-ts

This section first designs the task planning system according to the scene and task purpose. Then the overall system model architecture is introduced.

3.1. Task planning

A centralized control system^19,20 is controlled by a single processor or a ground station center, which controls all task execution. However, in this mode, a UAV does not have the independent ability in a dynamic environment which does not meet our transportation task requirement for autonomy. A fully distributed control system^21,22 is the opposite of centralized control, in which all UAVs are equal and can negotiate and coordinate with each other UAVs without any center stations. Therefore, agents not only can quickly understand and learn the environment, but also have a good cooperation ability to solve contradictions and conflicts with other agents, to plan their behavior, and to make the final decision. In our Co-TS, the two UAVs both equipped onboard computers respectively as an independent agent in a distributed system

The research on multi-UAV cooperative task planning in a distributed system is based on the development of a single UAV, so it still has the characteristics of a single UAV task planning system. First, clear task purpose: including task type and task restrictions; second, state analysis including environmental information, target information, and situation assessment; third, task planning according to the constraints under considering the evaluation data and the calculation of the flight path. Essentially, multi-UAV have the ability to cooperate compared to a single UAV.²³

In the past UAV task planning system,^24,25 the control station completes the task planning and then gives the task command to the UAV. A UAV only needs to complete the task command. In order to make UAV autonomous, in,²⁶ some basic system modules are established in a single UAV task planning system, including task allocation, flight path planning, evaluation, control module, etc., but their composition is too simple to satisfy our Co-TS.

To realize our Co-TS, the research ideas from local to overall and from single UAV to multi-UAV are adopted. As shown in Figure 2, based on a deep analysis of the scenario, we build a single UAV task planning system where the data management module is based on a multi-data fusion system, then the UAV analyzes the transporting task based on the data, path planning is also divided into the pre-planning and online planning that the UAV can quantify the real-time environmental change. Therefore, the UAV can conduct online flight path planning independently and has the ability of evaluation and autonomy in dynamic environment. The above completes a single UAV task planning system. For a multi-UAV task planning system, conflict elimination and cooperation functions are added based on the single UAV task planning system. As shown in Figure 3, according to the task requirements, the ground station completes the task allocation and distributes it to each UAV. Each UAV is equipped with an onboard computer so that each UAV is an agent in a distributed system. The multi-UAV control system is deployed on the onboard computer. The multi-UAV control system senses the flight environment information and UAV state information through the information fusion system. Then UAVs complete the task of detecting the landing site and planning the path. Through the network, two UAVs can perform tasks cooperatively and eliminate conflicts, arrange the paths reasonably to ensure the maximum revenue, and finally complete the transportation task.

Figure 2.

Details in a single UAV task planning system.

Figure 3.

Multi-UAV task planning system based on the single UAV.

3.2. Model architecture

To improve the development efficiency of application software of a multi-UAV cooperation system, and enhance the scalability and maintainability of the software. Model-driven Architecture (MDA) as an important methodology provides effective guidance for our Co-TS. MDA separates our upper algorithm from the implementation technology platform and reduces the complexity of the application system. In this section, we design a model architecture of Co-TS based on task planning in Section 3.1. We present the module function, the relationship between modules, the module external interface, and how to call each other between modules under the guidance of model-driven methodology. In this way, it is easy to analyze the interaction, cooperation, synchronization, and asynchronous relationship of the whole system.

Figure 4 defines the modules of our Co-TS.

Management module: Since each module is executed concurrently, this module tracks the real-time status of each module, schedules the control flow of each module, and eliminates the conflict of each module. On the one hand, the UAV decision-making in our system is driven by real-time data. Therefore, when the autonomous decision module calls the internal data management module for the data, the management module needs to keep decision command and data in a synchronous relationship; otherwise, if the data on which the decision based is in the past, will lead to a wrong decision execution. For example, when the path planning module gets the marker position information that is not synchronized with the current situation, that will lead to a wrong path. To solve this problem, we use a pipeline synchronization,²⁷ which ensures the synchronization between data and decision. On the other hand, the acquisition of different data sources such as sensors or cameras does not interfere with each other, so the management module enables data acquisition in an asynchronous mode.

Internal data management module: This module can receive, transform, store and send all information data related to the task planning, such as marker information detected by the computer vision model, UAV state information, payload information obtained by various sensors, and the other UAV status information obtained by the communication module. Then, this module updates the results of the task planning and accepts the final results of the task planning in the database.

Data assessment module: This module gets the latest information from the internal data management module. Since all kinds of information from the database are primitive, the data assessment module analyzes and processes them so that they can be recognized and analyzed by the autonomous decision module. The autonomous decision module uses processed data to make decisions. The ground station can obtain the situational information of the task in this module.

Autonomous decision module: This module is divided into two sub-modules: the task allocation module and the path planning module. They determine the instructions of the UAV. Due to the different task states of the system, different planning needs to be completed. The complex states of the task include preparation, rising, forwarding, and landing. The task allocation sub-module completes the task assignment based on the task purpose and requirements. The path planning sub-module determines the takeoff point, landing point, and flight path according to the position of markers and the UAV's track, completes the path planning from the takeoff point to the landing point, and selects the optimal path.

Communication module: Since two UAVs are required to cooperate in the execution process, they need to transmit information in real time for cooperation, such as current location information, current task status, etc. Therefore, in order to meet the real-time of communication requirements, this module is based on the UDP network protocol for information packaging, distribution, and interaction.

Collaborative module: This module keeps the stability of the payload, removes the conflict, and keeps the safe distance between the two UAVs based on the communication module.

Figure 4.

Multi-UAV model architecture of the Co-TS.

4. Task control flow algorithm

In this section, we first divide the task execution process into four states based on the task planning system. Then, based on the model architecture, we design the algorithm for each state by calling the function of each module.

In Section 3, UAVs are not explicitly represented. The underlying implicit assumption is that a UAV can determine its task instructions based on the environment and the state when the task is executed. In addition, a UAV can access one or more instances of the database, which stores the sensor information, the state information of UAVs, and the location information of markers, etc. In order to let multi-UAV perform the transportation task autonomously and cooperatively based on the information, the overall task control flow algorithm is designed based on the model architecture. Also, since we divide the task into four states, the concept of the finite state machine (FSM) is introduced to express the transition relationship between different states.²⁸ In general, an FSM is a standard quintuple $F S M = (Q, E, Σ, Q_{o}, F)$ , where Q is a finite state set, E is a finite event set, $Σ$ is a state transition function $Q \times E \to Q$ , $Q_{o} \in Q$ is an original state; $F \in Q$ is a final state set. As shown in Figure 5, we set the control flow for four states: preparation, rising, forwarding, and landing where the landing state is the terminal state set. To maintain two UAVs cooperative during the task execution process, the two UAVs are in the same state at the same moment. When a UAV switches its states, it not only needs to satisfy the completion in the current state but also needs to satisfy that another UAV has received a synchronization signal which presents the completion of the current state. The state is switched if the two conditions are satisfied.

Figure 5.

Transitions to different states in FSM.

In Figure 6 the control flow algorithm of each state is designed based on the task planning system and model architecture established in Sections 2 and 3.

Figure 6.

Control flow algorithm in Co-TS.

4.1. Preparation\landing state

Before takeoff, UAVs first enter the state of preparation and receive task start command from the ground station. Then UAVs complete the situation analysis and estimation by loading computer vision network data and all kinds of sensor data. After completion of task analysis, they establish the task priority and situational analysis.

When the UAVs are above the markers at the end of the task, the landing state is carried out such that the two UAVs land at the same speed through the synchronization signal.

4.2. Rising state

In the rising state, the two UAVs work together to maintain the balance of the payload. We assume that the length of the two ropes connected to the two ends of the payload may be unequal as shown in Figure 7. Thus, we can't keep the payload balanced by setting the altitude value in advance. To find the altitude difference by UAVs autonomously and collaboratively, the payload balance data is sensed in real-time by the sensor attached to the payload. The data is stored and updated by the internal data management module. The attitude difference between the UAVs is dynamically adjusted based on real time sensor data. Algorithm 1 shows how to control the speed of one of the UAV and how to compute the altitude difference between the UAVs to achieve the payload balance state. The balance sensor data is named angle whose range is [-90,90]. 0 degree is the ideal balance number. because of errors, we set the interval [-5,5] to indicate that the load is balanced.

Figure 7.

Rise and keep the payload balance.

Algorithm 1.

The adjustment in the rising state

1. If the payload is lifted then

_2.

a n g l e \leftarrow receive sensor value

3. while True do

4. If

a n g l e > 5

then

5. UAV2 maintains its original speed and UAV1 accelerates

6. endif

7. If

a n g l e < - 5

then

8. UAV2 maintained its original speed and UAV1 decelerates

9. endif

10. If

- 5 \leq a n g l e \leq 5

then

11. The UAVs hover and record the current altitude difference

12. break

13. endif

14. end while

4.3. Forward state

In the forwarding state, the UAVs find the path and fly forward autonomously. The path planning problem is a scheduling technology that arranges all UAVs to complete tasks in an orderly manner. Under the constraints of UAV performance, natural environment, and task requirements, this technology can achieve the goals of the shortest distance and minimum cost.^29,30

In our Co-TS, the path is based on the markers’ location and the UAV location data. The UAV firstly gets the marker position and the distance to another UAV from the internal data management module. Then, the UAV analyzes and evaluates the flight path. In the flying process, UAVs check if they need to re-plan the path according to the current state. If yes, they coordinate and modify; if no, they go on the original plan. This dynamic planning can ensure that the UAVs have autonomous ability.

We define a path plan as a problem with constraints that can be formulated in Eq. (1):

J = k J_{s h o r t e s t_{p a t h}} + (1 - k) J_{d i s t a n c e_{t h r e a t}}

(1)

where the UAV flight path cost is composed of the shortest path cost and the distance threat cost: the coefficient value reflects the tactical tendency of the decision-maker.

k = 0

means that the decision maker only wants the safest path regardless of path length, and

k = 1

means that the decision maker only wants the shortest path regardless of safety factors. When the path length cost or the safety distance changes, it must be replanned online.

Algorithm 2 shows the process of finding the shortest path based on the marker position in the camera image, which is divided into two steps as shown in Figure 8(a) - (b). First, the target information is obtained from the front camera. When the UAV approaches the marker, the front camera cannot capture the marker so that the below camera is switched to obtain the marker information.

Figure 8.

Forwarding state. (a), (b) front and below camera view respectively. (c) final path planning model.

Algorithm 2.

Find the shortest path based on vision information

1. If the marker is detected then

(x_{min}, x_{max}, y_{min}, y_{max})

coordinates of the detection bound-box

W_{I} \leftarrow Initial widith, H_{I} \leftarrow Initial height

c e n t r e = (x_{min} + \frac{x_{max} - x_{min}}{2}, y_{min} + \frac{y_{max} - y_{min}}{2})

c e n t r e_{n o r m a l i z e d} = (\frac{c e n t r e [0]}{W_{I}}, \frac{c e n t r e [1]}{H_{I}})

6. Case1: front camera

7. If

c e n t r e_{n o r m a l i z e d} [0] > 0.5 + e O R c e n t r e_{n o r m a l i z e d} [0] < 0.5 - e

then

8. Adjust yaw to make

c e n t r e_{n o r m a l i z e d} [0] into [0.5 - e, 0.5 + e]

which

9. determine shortest path with the UAV's yaw facing the marker as shown

10. in Figure 8(a)

11. end if

12. If front camera cannot detect the marker then

13. Switch to the below camera

14. end if

15. Case2: below camera

16. If

c e n t r e_{n o r m a l i z e d} [0] a n d c e n t r e_{n o r m a l i z e d} [1] n o t i n [0.5 - e, 0.5 + e]

17. respectively then

18. determine forward path and landing point as shown in Figure 8(b)

19. endif

While in the safe distance cost, the UAV gets the original information of two UAV latitude and longitude $(U A V_{1 - l o n g}, U A V_{1 - l a t})$ and $(U A V_{2 - l o n g}, U A V_{2 - l a t})$ from the GPS. The GPS we use is based on RTK (Real-time kinematic) technology,³¹ and the accuracy between them is centimeter-level positioning, which provides a hardware basis for us to obtain and calculate the accuracy between the two UAVs. The distance and azimuth Angle of the two UAVs are calculated by Eq. (2) and Eq. (3), respectively, where $a = U A V_{1 - l a t} - U A V_{2 - l a t}$ , $b = U A V_{1 - l o n g} - U A V_{2 - l o n g}$ and 6378.137 is the equatorial radius of the Earth. If the distance is more than 2.5 + e meters, a UAV approaches the other UAV in the azimuth direction. If the distance is less than 2.5-e meters, then they keep away in the azimuth direction Finally, the total cost is calculated, and the final path trajectory is calculated by considering the path length and distance cost, as shown in Figure 8(c).

\begin{aligned} d i s t a n c e = \\ 2 * \sqrt{\sin^{2} \frac{a}{2} + \cos (U A V_{1 - l a t}) * \cos (U A V_{2 - l a t}) * s i n^{2} \frac{b}{2}} * 6378.137 \end{aligned}

(2)

a z i m u t h = \arctan (\frac{b * \cos U A V_{1 - l o n g}}{a})

(3)

5. Experiment and analysis

This section presents the experience platform and implementation details, and then we analyze the system performance based on the quantitative and qualitative results.

5.1. Platform Introduction

Co-TS consists of two subsystems: flight controller and onboard computer. Figure 9 shows the hardware selection, GUI, and architecture for our Co-TS. We select Pixhawk/PX4 flight controller. The core of the PX4 flight controller is an ARM-based Nuttx real-time operating system. GPS, IMU, and other sensors are connected to PX4. We select Jetson-Nano onboard computer which includes a GPU of 128-core Maxwell, the CPU of Quad-core ARM A57, 4GB memory. The operating system on Jetson-nano is Ubuntu16.04. Every UAV is equipped with an onboard computer, a flight controller, and two cameras. One camera is fixed at the UAV's front and one is fixed below as Figure 9(c) shows for supporting the computer vision network. In addition, the PX4 Offboard mode allows the onboard computer to send inputs directly to the flight controller stack by the MAVLink protocol. In order to monitor the task status of the UAV, as shown in Figure 9(c) - (d) we design a GUI software of the ground station to interact with the UAV to create and view the task planning, monitor the task progress, provide interactive guidance, and maintain situational awareness.

Figure 9.

Hardware selection, GUI and architecture for our Co-TS.

5.2. Implementation details

To implement the Co-TS, we use Python to write each module and the control flow algorithms. For data acquisition, we use the python standard library asyncio which supports asynchronous IO and makes the acquisition of various data of the UAV more efficient.

For multi-source data acquisition and processing. Firstly, to obtain the balance data of the payload, we use a Bluetooth sensor to transmit the level information as Figure 9(b) shows. Compared to Wi-Fi, Bluetooth has a stronger anti-interference ability and a lower power consumption. It only needs a few batteries to maintain a relatively long work time. Secondly, to obtain the marker position in an image, we adopt the Yolo-Tiny^32–34 computer vision model trained by our marker dataset. Then, we get marker datasets from UAV perspectives at different altitudes and distance, creating 200 training datasets and 100 validation datasets labeled by ourselves. The Yolo-Tiny is 4MB after training that can recognize markers in the camera at 0.1 s per frame on jetson-nano and achieve over 95% accuracy. Thirdly, to obtain UAV native data is through Mavlink, a very lightweight messaging protocol. Mavlink sends the UAV data stream as a topic and builds the Mavlink service component on the onboard computer to parse the data sent through the serial port. Fourthly, the communication between UAVs uses the UDP protocol to ensure real-time data transmission.

For algorithm and module implementation. UAV‘s actions are data-driven in the Co-TS system. When synchronization between actions and data is required, we use the pipe function of the multiprocessing library in Python to implement it. Since the UAV control flow algorithm needs to control the attitude of the UAV, we adopt the MAVSDK third-party library which provides APIs to manage multi-UAV. Also, the MAVSDK provides programmatic access to UAV information and telemetry. In this way, we can write a program to let the UAVs control movement and other operations autonomously without considering the knowledge of dynamics. Finally, each module is realized through the encapsulation of classes, and each module class is called through the main program to realize the control flow algorithm.

5.3. Results and discuss

Figure 10 (a) - (d) shows the process of 2-UAV cooperative transportation task qualitatively. The ends of the payload tied to UAVs need to be placed on red and yellow markers, respectively. The ground station sends the start command, and then the UAVs successfully execute transportation tasks cooperatively in preparation, rising, forwarding, and landing states. The video is available at https://flml.tongji.edu.cn/TOOL.htm

Figure 10.

Overview of task execution of multi-UAV cooperative transportation in Co-TS.

The experiment results were obtained and plotted in Figure 11 quantitatively, which proves that our control algorithm is effective.

Figure 11.

Results of the UAV's altitude, the balance value of payload sensor, and the distance between two UAVs during the execution of Co-TS.

In the rising state, as shown in Figure 7, the angle value of more than 5 means that the left end of the payload is lower than the right. Each UAV's rope length is unknown in advance. To keep the payload balance suspended in the air, the two UAVs need to compute the altitude difference to offset the length difference of the two ropes. Between 0 and 6 s, the two UAVs rise at the same speed, the ropes tied to the payload both are not tightened at first, so that the balance angle value is around 0 baseline. Then, after around 6 s, as shown in Figure 11(b) the UAV detects that the sensor data value on the payload exceeds 5. The data means one end of the rope is tightened while the other end is not tightened which leads the payload to unbalanced. If the left end of the payload is lower than the right, the left UAV changes the rising velocity autonomously to make the altitude difference offset the rope length difference. As we can see from Figure 11(a) - (b), between 6 and 14 s, the altitude difference becomes bigger and the balance angle value from the high point becomes smaller until 0. Then at around 14 s, when the altitude difference is around 0.25-meter which is the final altitude difference, and the payload sensor returns to around baseline 0; after that, the two UAVs hover. In the following states, the UAVs’ altitude was kept at 2.0 meters and 1.75 meters, respectively, to keep the height difference at around 0.25 meters. The above process shows that the two UAVs can autonomously cooperate to keep the payload balance.

When entering the forwarding state, the distance between the two UAVs needs to be kept at about 2.5 meters. Since the shortest path of the marker position and the path of the threat distance jointly synthesize the final trajectory, as shown in Figure 8(c), it oscillates around 2.5 meters as shown in Figure 11(c). This 2.5 meters is equal to the length of the payload. But at around 30 s, the distance between the UAVs fluctuates greatly to around 2.9 meters. It may be due to the interference of wind or other environmental factors that make the distance between UAVs too large. However, we can see that it returns to around 2.5 m later because the distance constraint algorithm works as described in section 4.3. Finally, after around 84 s, both UAVs are above the markers and they enter the landing state, the altitude quickly becomes 0.

6. Conclusion

In this paper, we propose Co-TS that designs and implements 2-UAV transport payload to the designed location cooperatively. We make 2-UAV task planning according to the scene goals to be achieved. To solve the complexity of the upper-layer application software of UAVs, we design the model architecture, introduce the function of different modules and analyze the relationship between the modules. Then the UAV task execution is divided into four different states, and the corresponding control flow algorithm is designed. Finally, experiments in real environment show that our system is effective and feasible.

However, our Co-TS employs only two UAVs. Future work will focus on expanding to more UAVs for more general application prospects.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Shanghai Science and Technology Committee, National Natural Science Foundation of China, (grant number 22511105500, 62032019, 62172299).

ORCID iD

Qian Li Weng

References

Wang

Xin

. Integrated optimal formation control of multiple unmanned aerial vehicles. IEEE T Contr Syst T 2013; 21: 1731–1744.

Wang

, et al. Car detection from low-altitude UAV imagery with the faster R-CNN. J Adv Transport 2017; 2017: 2823617.

Motlagh

Bagaa

Taleb

. UAV-based IoT platform: a crowd surveillance use case. IEEE Commu Mag 2017; 55: 128–134.

Menouar

Guvenc

Akkaya

, et al. UAV-enabled intelligent transportation systems for the smart city: applications and challenges. IEEE Commu Mag 2017; 55: 22–28.

Villa

Brandao

Sarcinelli-Filho

. A survey on load transportation using multirotor UAVs. J Intell Robot Syst 2020; 98: 267–296.

Mohiuddin

Tarek

Zweiri

, et al. A survey of single and multi-UAV aerial manipulation. Unmanned Syst 2020; 8: 119–147.

Zhang

Huang

, et al. Self-triggered based coordinate control with low communication for tethered multi-UAV collaborative transportation. IEEE Robot Autom Let 2021; 6: 1559–1566.

Gassner

Cieslewski

Scaramuzza

. Dynamic collaboration without communication: vision-based cable-suspended load transport with two quadrotors. In: 2017 IEEE international conference on robotics and automation (ICRA). Singapore: IEEE, June 2017, pp.5196–5202.

Webb

Rogers

. Adaptive control design for multi-UAV cooperative lift systems. J Aircraft 2021; 58: 1302–1322.

10.

Nahavandi

. Trusted autonomy between humans and robots: toward human-on-the-loop in robotics and autonomous systems. IEEE Syst Man Cybern 2017; 3: 10–17.

11.

Liu

, , et al. The application of deep learning in computer vision. In: 2017 Chinese automation congress (CAC). Jinan, China: IEEE, October 2017, pp.6522–6527.

12.

Tijtgat

Van Ranst

Goedeme

, et al. Embedded real-time object detection for a UAV warning system. In: Proceedings of the IEEE international conference on computer vision workshops. Venice, Italy: IEEE, October 2017, pp.2110–2118.

13.

Chung

Paranjape

Dames

, et al. A survey on aerial swarm robotics. IEEE T Robot 2018; 34: 837–855.

14.

Kraus

. Negotiation and cooperation in multi-agent environments. Artif Intell 1997; 94: 79–97.

15.

Maza

Caballero

Capitán

, et al. Experimental results in multi-UAV coordination for disaster management and civil security applications. J Intell Robot Syst 2011; 61: 563–585.

16.

Kim

Choi

Lee

, et al. Vision-based collaborative lifting using quadrotor UAVs. In: 2014 14th international conference on control, automation and systems (ICCAS). Gyeonggi-do, Korea (South): IEEE, 2014, pp.1169–1174.

17.

Chen

. Applications of cyber-physical system: a literature review. J Ind Integr Manag 2017; 2: 1750012.

18.

Jennings

. On agent-based software engineering. Artif Intell 2000; 117: 277–296.

19.

Zhan

Casbeer

Swindlehurst

. A centralized control algorithm for target tracking with UAVs. In: Conference record of the thirty-ninth asilomar conference onSignals, systems and computers. Pacific Grove, CA, USA: IEEE, November 2005, pp.1148–1152.

20.

Zheng

Wang

, et al. UAV Formation flight and collision warning with centralized control of ground control station. In: 2019 IEEE international conference on unmanned systems (ICUS). Beijing, China: IEEE, October 2019, pp.103–108.

21.

Chandler

Pachter

Rasmussen

. UAV Cooperative control. In: Proceedings of the 2001 American control conference. Arlington, VA, USA: IEEE, June 2001, pp.50–55.

22.

Pham

Feil-Seifer

, et al. A distributed control framework for a team of unmanned aerial vehicles for dynamic wildfire tracking. In: 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS). Vancouver, BC, Canada: IEEE, September 2017, pp.6648–6653.

23.

Maza

Kondak

Bernard

, et al. Multi-UAV cooperation and control for load transportation and deployment. J Intell Robot Syst 2010; 57: 417–449.

24.

Gancet

Hattenberger

Alami

, et al. Task planning and control for a multi-UAV system: architecture and algorithms. In: 2005 IEEE/RSJ international conference on intelligent robots and systems. Edmonton, AB, Canada: IEEE, August 2005, pp.1017–1022.

25.

Zhou

Wang

, et al. A research framework on mission planning of the UAV swarm. In: 2017 12th system of systems engineering conference (SoSE). Waikoloa, HI, USA: IEEE, June 2017, pp.1–6.

26.

Atyabi

MahmoudZadeh

Nefti-Meziani

. Current advancements on autonomous mission planning and management systems: an AUV and UAV perspective. Annu Rev Control 2018; 46: 196–215.

27.

Seizovic

. Pipeline synchronization. In: Proceedings of 1994 IEEE symposium on advanced research in asynchronous circuits and systems. Salt Lake City, UT, USA: IEEE, 1994, pp.87–96.

28.

de Araujo

Almeida

APG

Miranda

, et al. A parallel hierarchical finite state machine approach to UAV control for search and rescue tasks. In: 2014 11th international conference on informatics in control, automation and robotics (ICINCO). Vienna, Austria: IEEE, September 2014, pp.410–415.

29.

Bortoff

. Path planning for UAVs. In: Proceedings of the 2000 American control conference. ACC (IEEE Cat. No. 00CH36334). Chicago, IL, USA: IEEE, June 2000, pp.364–368.

30.

Souissi

Benatitallah

Duvivier

, et al. Path planning: a 2013 survey. In: Proceedings of 2013 international conference on industrial engineering and systems management (IESM). IEEE, October 2013, pp.1–8.

31.

Sun

Slaughter

Ruiz

, et al. RTK GPS mapping of transplanted row crops. Comput Electron Agr 2010; 71: 32–37.

32.

Redmon

Divvala

Girshick

, et al. You only look once: unified, real-time object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition(CVPR). Las Vegas, NV, USA: IEEE, June 2016, pp.779–788.

33.

Liu

Wang

Zhou

, et al. Uav-yolo: small object detection on unmanned aerial vehicle perspective. Sensors 2020; 20: 2238.

34.

Shi

, et al. Vehicle detection under UAV based on optimal dense YOLO method. In: 2018 5th international conference on systems and informatics (ICSAI). Nanjing, China: IEEE, 2018, pp.407–411.

Co-TS: Design and Implementation of a 2-UAV Cooperative Transportation System

Abstract

Keywords

1. Introduction

2. Scenario

3.1. Task planning

4.2. Rising state

5.1. Platform Introduction

5.3. Results and discuss

Footnotes

Declaration of conflicting interests

Funding

ORCID iD

References