Abstract
In the years to come, large power grid operators will operate and maintain an ever-increasing asset base. New innovative solutions are needed to increase the quality and efficiency of asset management to avoid corresponding growth in resources and cost. To this end, autonomous unmanned aerial vehicles (UAVs) provide a range of possibilities. Here, we present a novel prototype solution for autonomous and remotely operated inspection missions with resident drones on electrical substations, comprising: (1) an autonomous drone with sense and avoid and robustness to harsh weather capability; (2) a drone hangar for remote operations; and (3) drone operations and data acquisition management software. Further, we discuss the possibilities and challenges that such a system offers and give an overview of requirements that are key to realizing the potential of drones for improved asset management. These requirements are based on years of operational experience with electrical substations combined with the lessons learned during the development and testing of our drone system. We also experimentally investigate safety distances between the drone and high-voltage infrastructure. We demonstrate the usefulness of our autonomous inspection solution through extensive field testing at one of Statnett’s fully operational electrical substations.
Introduction
Stable and reliable electrical substations are crucial for ensuring reliable power distribution in all countries worldwide. In the years to come, large power grid operators such as Statnett in Norway will operate and maintain an ever-increasing asset base. New innovative solutions are needed to increase the quality and efficiency of asset management to avoid corresponding growth in resources and cost. To this end, autonomous unmanned aerial vehicles (UAVs) provide a range of possibilities. In this article, we present and evaluate a prototype resident drone system (Figure 1) as a means for meeting the industry’s needs.
Drone and hangar at Aura substation, residing below the mountains. Photo by Thomas Negård.
There are several “drone-in-a-box” systems available commercially, such as Airobotics 1 and Percepto, 2 and other companies, such as DJI, offer various supporting technologies. However, none of the existing solutions on the market meet Statnett’s specific requirements. Many of Statnett’s electrical substations are located at northern latitudes, and thus, harsh weather conditions during winter present significant operational challenges. To be able to perform remote inspections effectively, Statnett requires new technologies that enable safe and robust autonomous operation in extreme wind, cold, and precipitation conditions.
In this article, we present a holistic system for remote inspection and monitoring of capital-intensive assets that meet strict requirements on weather robustness, space availability, operation in high-voltage environments, and geographical challenges. One of the requirements for this system is that the drone must reside on-site, by which we mean that the system should be able to deploy safely and automatically without personnel on-site at any time of day or year. We term this system an Autonomous Resident Drone-based Inspection System (ARDIS). Our implementation comprises three main components: (1) an autonomous drone platform with sense & avoid (S&A) capability and robustness to harsh weather; (2) a drone hangar capable of automatically deploying and retrieving the drone as well as automatically recharging and providing other support for permanent on-site availability; and (3) a communication architecture and user interface for remote mission management. In addition to developing the drone-based system, we also carry out live testing at SINTEF’s high-voltage laboratory to investigate safe operating distances when flying our drone around high-voltage equipment. We validate our autonomous inspection system through extensive field testing at Statnett’s operational high-voltage substation “Aura,” located at Sunndalsøra in western Norway.
The remainder of this article is comprised as follows: in the next section, we review the main challenges and requirements for our use case, followed by an overview of the system as a whole and a more detailed overview of the various components of the autonomous navigation system; in the following sections, we then discuss the results of our high voltage testing and on-site automatic inspection tests, before detailing some of our lessons learned and conclusions from this work.
Related work
As the usage of drones as a work tool for day-to-day operations within the industry has grown dramatically over the past 3–5 years, the need for increasing degrees of automation and self-reliance to reduce the resources has required to support and train personnel for drone operations. In 2017, a drone solution developed by Airobotics was the first system to be certified for fully automated flight (beyond visual line-of-sight, BVLOS), 1 and since then, many similar BVLOS systems have been certified by authorities worldwide. However, drone systems certified for BVLOS operations typically do not operate in close proximity to structures due to the dangers involved and the limited accuracy of GNSS-based localization. Consumer camera drones such as the Skydio R1 3 have an impressive degree of onboard autonomy and are able to navigate safely through cluttered environments, however, such platforms are not typically employed for industrial tasks due to their high degree of specialization and limited extensibility for inspection or maintenance operations. Furthermore, industrial drones require enhanced protection against weather and extreme conditions (e.g. DJI Wind series UAS 4 ). There is thus a need for new solutions within autonomy and harsh environments to close the “autonomy gap” for industrial drones.
To be able to support remote operations with short call-up times, drone-based systems must have a permanent presence on-site. Hangars or drone garages are designed to support this need, facilitating automatic launch and retrieval, automatic battery (re-)charging as well as data-link with mission control, protection against harsh environments, and some diagnostic and troubleshooting capabilities (such as an internal camera for visual inspection of the UAV). These systems are sometimes referred to as “drone-in-a-box” solutions. A state-of-the-art example is the ground station for the Airobotics UAS, 1 which includes a robotic arm for replacement of batteries and other payloads.
Challenges and requirements
In this section, we outline the main challenges for a resident drone-based solution to autonomous inspection and monitoring on electrical substations as well as the associated requirements that need to be met to address these challenges. Possible challenges related to new work processes, acceptance of new technology, and so on, are outside the scope of this article.
Challenges for drone-based operations on electrical substations
Requirements for drone-based operations on electrical substations
Our holistic system concept is designed to meet the aforementioned challenges and comprises three components: A drone hangar that houses, charges, provides for remote inspection, and deploys/retrieves our drone platform 24/7 without a human operator on-site in windy, rainy, snowy weather, and subzero temperatures. A drone system that can navigate safely and accurately within the available space of a high-voltage substation under all expected weather and environmental conditions. An architecture for remote management, programming, deployment, and operation of the hangar-drone system that is adapted for use by Statnett’s personnel. The architecture also provides for real-time retrieval and review of payload sensor data captured by the drone.
General system requirements
General performance: The systems shall be able to operate automatically, while the system operator shall always be capable of over-riding the system. The systems also need requirements for system initiation time and proven uptime of the systems in stand-by mode.
Operating conditions: The drone platform is required to work in high average winds of up to 18 m/s to achieve sufficient uptime. In addition, tolerance for operations in rain and snow is required.
Electromagnetic resilience: The drone platform needs to prove that it will not be negatively affected by the expected electromagnetic radiation from the substation.
Certification: All systems created need to be within regulatory requirements.
Functional safety: The ARDIS implementation needs to provide functional safety mitigation to avoid negative effects of loss of motors, loss of communication links, degradation of localization systems, and so on.
Subsystem requirements
Drone platform: The drone platform needs to be resistant toward ingress of water and dust to achieve the operational requirements. It also needs to have sufficient endurance both for single flights and for prolonged operations, where recharging between missions is required.
S&A: The system needs to be capable of detecting and navigating around objects found at the substation. This includes thin wires with dimensions greater than 5 mm.
Path planning: The system needs to be able to find a valid and safe path to any arbitrary point inside the defined allowable flight volume of the substation.
Localization: The localization system needs to provide a localization accuracy of ≤5 cm. The system shall ensure the quality of the localization for ≥10 s in case of any degradation event.
Communication: The communication system needs to provide coverage at any point in the substation to provide the capability of operator override. In case of required roaming, the system shall be capable of seamless transitions between access points.
User interface: The user interface needs to provide the operator with the necessary information to perform safe and informed decisions in flight. The system also needs to provide acquired payload data automatically to the operator in a transferable format.
Hangar: The hangar system needs to protect the drone system in all expected weather conditions. The system needs to provide automatic charging of the drone without human intervention on-site. The system also needs to provide the necessary capability for the operator to perform required preflight inspections of the drone.
System overview
In this section, we provide an overview of the various system components in our ARDIS implementation.
User interface
One of the key requirements for the resident drone system was that the control system should have a user interface for operators that have limited experience flying drones. To achieve this, the drone control is decoupled from the user control. Instead of direct control, the operator intuitively moves a camera platform in 3D space.
The drone then acts autonomously in response to the input from the operator with the S&A system, ensuring that the operator is not able to crash the drone into obstacles or to fly outside of specified zones. Any desired movements by the operator are transformed from the camera view in the operator interface to velocity and yaw commands in the drone frame and verified by the S&A system that the desired command is safe before it is sent to the drone flight controller. An unsafe area in the desired direction will act as a virtual wall that the operator cannot move through.
In addition to the semimanual control scheme, the operator must be able to select and launch a preplanned inspection, where the drone executes the flight and captures inspection data autonomously. These missions are created by specifying georeferenced points of interest (PoIs) and their associated viewpoints (position from which the PoI is viewed). Immediately after capturing inspection data for each PoI, the data are uploaded automatically to the cloud-based inspection database, where it can be reviewed by the remote operator. The inspection objects are tagged with an object ID such that all gathered data can be tracked over time, avoiding time-consuming tasks of manually sorting data postinspection.
Hangar and communication
Another key requirement was the residence capability for the drone platform. The drone needs to reside and charge in a hangar, ready for deployment 24/7 under a large range of weather conditions without any manual operator interaction. The drone performs inspections and interacts with the hangar automatically without any manual work to place the drone outside for takeoff or charge batteries. As shown in Figure 1, the drone is launched and retrieved via an automatic landing platform, which retracts into the hangar.
As shown in Figure 2, the main responsibilities of the hangar are to control the hangar platform, charge the drone batteries, and transmit real-time kinematic (RTK) corrections from the RTK base station to the drone localization board. This is in addition to providing weather status to the user interface for situational awareness, and being able to visually inspect both the inside and outside of the hangar by controllable pan–tilt–zoom cameras. The drone batteries have a connection to the charger through the drone feet, where a centering mechanism and magnetic connectors on the drone platform ensure that the drone feet are connected and locked to the charger when the hangar retracts. The operator can start the charging when the drone is back in the hangar, while the internal battery management system in the drone stops charging when batteries are full or a failure occurs.
Overview of the connection between the different components in the ARDIS. Operator control refers to the user interface to control the drone in addition to the inspection cloud, where the operator can create inspections and process the gathered data. The hangar system contains the necessary components to allow a resident drone system and automatic release and recover of the drone during inspection flights. The drone system refers to all the components and subsystems on the drone needed for fully autonomous inspections. RTK corrections are sent from the hangar base station to the drone localization board, where the position estimate is further used by the flight controller to control the drone. ARDIS: Autonomous Resident Drone-based Inspection System; RTK: real-time kinematic.
To allow for real-time video stream from the drone to the user interface, it is of important that the communication link has high enough bandwidth and signal strength. Due to the large metal structures spread around in the substation, this proved challenging. The solution was to use a 5-GHz Wi-Fi network with three Wi-Fi access points covering the substation area and one covering the inside of the hangar.
Onboard modules
On the drone, there are systems with three main objectives: (1) Stable drone flight, (2) keeping the drone within a safe flight envelope, and (3) gathering data in automatic inspections. The first two objectives are covered by the navigation framework and flight controller, as described in the following sections, while the onboard computer is responsible for inspection execution and interaction between the individual subsystems on the drone, hangar, and the operator control, as shown in Figure 2.
The drone payload consists of a three-axis gimbal with both a Sony UMC-R10C 20 Mpx optical camera and a Workswell WIRIS 640 thermal camera. The high-resolution optics provide a video stream to control room operator and capture actionable data, while the radiometric readings from the thermal camera provide sensing capability to identify overheating components and enable data-driven decisions. The gimbal provides a stable camera platform that can be controlled manually and tracks desired georeferenced inspection points automatically. Finally, the battery management system monitors the battery state of charge and ensures safe charging of the batteries.
Drone platform
The drone used in the project was a Camflight FX8, a heavy lift octocopter drone manufactured by Nordic Unmanned, with an autopilot based on Ardupilot (https://ardupilot.org/), running on a Pixhawk 3 Pro (https://docs.px4.io/v1.9.0/en/flight_controller/pixhawk3_pro.html) flight controller.
An octocopter platform was chosen due to redundancy of the motor setup. This setup provides sufficient power and maneuverability to allow for safe landing of the drone, even after losing power on one motor. This has been demonstrated during the project, where the pilot was able to safely land the drone on the ground. The use of a heavy lift octocopter also allows more robust operation in heavy wind scenarios. To increase the robustness when landing, a landing algorithm was developed that takes into account horizontal drift due to wind.
With two 12-cell batteries, the maximum flight time was up to 50 min, depending on the payload weight. These could then either be changed and charged manually, or charged automatically through a connection in the drone feet when resident in the hangar.
The drone has been weatherproofed during the project to allow operation in most of the expected weather conditions, such as wind and precipitation. With operation of the drone in adverse and cold weather, there have been several considerations for the drone platform. Especially, this goes for the lithium polymer batteries with regard to cold weather efficiency and with 3D-printed plastic parts with regard to cold weather embrittlement.
Navigation
The onboard navigation system was responsible for guiding the drone through the mission objectives safely and efficiently based on knowledge of the substation layout and real-time position, attitude, and 3D sensor data. The key challenges that influenced the design of our navigation solution were
Precision: The drone’s 6D pose had to be measured accurately to enable precise navigation in close proximity to dangerous equipment and also to allow accurate steering of the payload sensors.
Robustness: The operational envelope had to include all accessible areas within the substation and all expected weather conditions in Northern Norway, including darkness, rain, snow, and wind.
Safety: The drone had to be able to safely complete its mission despite encountering unexpected obstacles (e.g. vehicles, ladders, and personnel).
The design of our onboard navigation system is shown in Figure 3. The software modules comprising the navigation framework were implemented in C++ on a multicore UDOO x86 ULTRA II (https://shop.udoo.org/eu/x86/udoo-x86ii-ultra.html) running Windows 10. This platform was selected due to processing power, support for Windows development environments, and I/O configuration. The key navigation modules are described in further detail in the following sections.
Navigation system design showing data flow between the various software modules comprising the navigation framework and external modules and/or hardware. All modules in the figure except those marked with “off-board” are on board the UAV. The navigation controller is responsible for communicating mission status and objectives with the mission controller (ground station) and for transmitting guidance commands to the flight controller (on board); the path planner continuously replans the optimal route to the current mission objective based on an offline graph and an online 3D obstacle map generated by the sense and avoid module; the localization module computes current 6D pose; and the navigation framework handles initialization, synchronization, visualization, and logging for the navigation components. Note that this figure shows logical data flow, but, in practice, the onboard computer acts as a gateway for all communication between the navigation computer and the flight controller and ground station, and the localization module includes an onboard hardware component (localization board), which interfaces with the RTK-GNSS/IMU and transmits current 6D pose directly to the flight controller for low-latency control feedback (see Figure 2). RTK: real-time kinematic.
Localization
The localization module is responsible for providing other system components with low-latency measurements of the drone’s position and attitude (i.e. 6D pose) for control feedback, path planning, obstacle mapping, and aiming of the payload cameras. Due to its critical role in almost all other onboard systems, it is crucial for the localization method to be robust to varying environmental conditions, to provide data that degrades gracefully, and to allow quantitative accuracy estimates. For these reasons and due to the scale of the outdoor areas to be supported, it was decided to design the localization solution around RTK-GNSS, rather than an onboard SLAM-based 5 approach or sensor network. The hardware selected was the Novaltel OEM7720 dual-antenna quad-constellation triple-frequency GNSS receiver (https://www.novatel.com/products/gnss-receivers/oem-receiver-boards/oem7-receivers/oem7720/) with integrated inertial navigation system (INS) via Analog Devices ADIS16488 inertial measurement unit (IMU) (https://www.analog.com/en/products/adis16488.html). Our GNSS antenna baseline is approximately 0.8 m. Under nominal conditions, this system can provide centimeter-level positioning accuracy and fractional-degree-level heading accuracy and additionally propagate pose estimates with little loss in accuracy during transient GNSS interruptions of up to 5 s duration. However, the substation environment is far from nominal and both onboard and offboard factors contribute to pushing our localization solution to its limits. In the remainder of this section, we briefly discuss some of the key factors affecting localization.
The most influential factor is terrain masking due to the location of the primary test site in a deep valley at Statnett’s facility in Sunndalsøra (Figure 1). This has the effect of greatly reducing the number of satellites visible and ultimately leads to an increased dilution of precision (DOP), depending on the position of the constellations at any particular time (Figure 4(a) and (b)). This issue is not isolated to this particular test site, as the combination of Norway’s northern latitude and dramatic landscapes creates many widespread areas with reduced GNSS-constellation visibility (Figure 5(a) and (b)). Compounding the adverse effects of terrain masking is the drone’s close proximity to concrete and metal structures during flight on the substation, which can also lead to GNSS signal reflections and multipath issues. To combat these issues, we found it essential to employ a receiver capable of tracking all available GNSS constellations, including US GPS, Russian GLONASS, European Galileo, and Chinese Beidou, and also to exploit modernized GNSS signal bands such as L5 and E5, which offer vastly superior multipath resistance compared to the legacy L1CA GPS band used in most consumer GPS receivers.
DOP over a period of 24 h at 62.6 north in Norway. TDOP relates how accurately the receiver can synchronize its clock to the constellations, while the GDOP gives an estimate of the total 3D uncertainty. (a) We see the DOP over a period of 24 h at 62.6 north in Norway with an open sky coverage. As the DOP is proportional with the approximate position uncertainty, this figure would give us an indication that the position estimate is stable and has a low uncertainty. (b) The simulated approximate satellite coverage of a mountain valley is displayed. With the mountains blocking the satellite coverage, the uncertainty would rise rapidly at specific points during the day. Both plots were generated using Trimble Office. 6 TDOP: time dilution of precision; GDOP: geometrical dilution of precision; DOP: dilution of precision.
Satellite coverage over a period of 24 h at 62.6 north in Norway. (a) The coverage with an open sky is shown. (b) The coverage is blocked by simulated mountains. Both plots were generated using Trimble Office. 6
As a result of the size and complex 3D layout of the substation and flight paths for inspection necessitating that the drone flew underneath and in between structures, it was very challenging to guarantee a continuous communications link between the drone and the base station. For the localization system, this meant that there could be periods of up to 1 min, where the RTK correction data stream from the base station to the drone might be interrupted. It was therefore necessary to employ an RTK-GNSS system that supported “corrections propagation” to ensure centimeter-level localization accuracy even during these communication interruptions.
The error magnitude during loss of lock is a realization of a stochastic process in that both the core INS produces outputs based on the single/double integral of noise bearing gyroscopes and accelerometers, and in that the information from the satellite observables has nonstationary characteristics. Typically, we evaluate the loss of accuracy as the output state uncertainties from the integration filter within the SPAN system executing the GNSS+INS sensor fusion relative to steady state in open sky conditions (For system dimensioning purposes, we consider the case of total loss (blockage of all satellites), in which conditions the noise characteristics of the IMU drive the uncertainty in accuracy over time. For a numerical representation of accuracy over time during outage, see the Novatel SPAN documentation: https://hexagondownloads.blob.core.windows.net/public/Novatel/assets/Documents/Papers/SPANBrochure/SPAN%20D16507%20v24.pdf), where anything less than 15 cm sigma on any of the position states is considered acceptable, while less than 2 cm is ideal in clear sky steady state.
The ARDIS was designed to support remote installations. This meant that the drone had to be capable of launching and carrying out missions autonomously without personnel on-site. To ensure safe operation, the drone also had to perform a calibration and initialization procedure before flight. Typical calibration of the RTK-GNSS/IMU system requires a series of controlled maneuvers designed to enable the system to confidently estimate the inertial biases and other variables that change with each startup and over time, as well as to calibrate alignment between the inertial sensors and RTK baseline. Performing these maneuvers safely and autonomously within the confines of the substation facility and prior to complete initialization of the localization system was very challenging. Through extensive field testing, we were able to develop an initialization flight comprising alternating circular paths conducted within a relatively open area within the substation that could be completed automatically and could reliably allow proper initialization of the localization system, typically within 1–2 min from takeoff.
The final factor influencing the design of the localization system was its close proximity to other onboard systems and noise sources. In particular, USB3 interfaces on the onboard computers generated significant levels of L-band electromagnetic interference (EMI), effectively masking GNSS signals within 1–2 m range. 7 To overcome this issue, GNSS antennas were mounted on “ground planes” elevated above the main drone body and away from onboard processing hardware; all USB3 interfaces were insulated with layers of laptop and copper tape; and the waterproofing shell for the main drone body was encased in a thin layer of metal to reduce emissions. These actions were able to reduce masking of the GNSS carrier signal by more than 90%, giving an average SNR loss of less than −3 dB.
Path planning
The purpose of the path planning module was to guide the drone from its current position to the next mission objective safely and efficiently. To avoid unnecessary exploration during missions, the planner had access to a coarse map of the substation that was generated offline and indicated flyable and nonflyable areas. To enable the drone to navigate around unexpected obstacles, the optimal route to the goal was continuously replanned during flight based on both the offline map and an online map of nearby obstacles, generated by the S&A module (see “Sense and avoid” section).
The offline data generation produces all the necessary components for the online planning step, that is, the planning graph and collision mapping. The graph consists of nodes and edges in three dimensions, where the nodes are the possible drone positions in the volume enclosing the substation, and the edges are the connections between the nodes. The online planning step will use the produced graph to find a path from start to goal. We define the edges using a set of motion primitives. Here, we use motion primitives that give linear motion between the nodes. The simplest primitives define movement along each of the dimensions and we also define primitives along two or three dimensions simultaneously, producing diagonal paths. Finally, to allow for smooth paths, we use primitives with a combination of one and two steps along different dimensions, giving a total of 74 unique motion primitives. Our graph supports subvolumes with different resolutions to enable precise control of drone position in confined spaces while reducing computation complexity (and therefore increasing maximum flight speed) in more open spaces.
The collision mapping specifies the relationship between graph edges and nodes and voxels in the obstacle map (generated by the S&A module). This is used in the online search for quick collision checking without the need for online ray tracing. The mapping is created by placing a model of the drone in the voxel-based obstacle map and finding all covered voxels. The swept volume of voxels is also precalculated for all motion primitives minus the volumes covered by nodes themselves. The relationship between the node graph and obstacle map is shown in Figure 6.
An illustration of the node graph and voxel-based obstacle map used by the path planning module to compute guidance commands for the drone. The graph is composed of regularly spaced nodes and connecting edges. Nontraversable nodes and edges (e.g. nonflyable areas) are marked as such (e.g. node E) offline. At run time, the planner finds the optimal route to the goal (A) through the graph (F-D-A). The path is then checked for collisions by consulting corresponding voxels in the obstacle map (right) generated by the sense and avoid module. In this example, the edge D-A passes through occupied voxels in the neighborhood of node B; thus, this edge is marked as nontraversable and the optimal path is recomputed (F-D-C-A). If this path is collision-free, it is sent to the flight controller. The planned path is continuously rechecked for collisions while the drone is underway such that the path can be updated to account for unexpected obstacles as they are detected.
The online planning system runs continuously during flight and operates at two levels—the planner level, which runs at a low rate and continuously replans a global path to the destination, and the updater level, which runs at a high rate, and takes the path from the planning level as input, and continuously rechecks the corresponding voxels in the obstacle map for collisions. The two online planning steps are described in more detail below (see also Figure 7).
The figure shows the steps needed for the path planning module to calculate a collision-free shortest-path for the UAV. UAV: unmanned aerial vehicle.
The
Connecting to the graph: Since the drone is almost never at the location of a node, we need to connect its current position to the graph. This is done by finding the N closest nodes and collision-checking the paths to them. All connected nodes are updated with their distance from the drone’s current position. Similarly, the goal configuration is also connected to the graph.
Searching through the graph: When all N closest starting nodes are updated, an A* graph search algorithm 8 is used to find the shortest path through the graph. For each node and edge that is evaluated, the collision mapping is used to find the voxels that need to be checked for collisions. When a node connecting to the goal is reached, the search returns.
Smoothing of the path: After the path through the graph is found, we do a simple smoothing by searching the path to find the furthest waypoint along the path that can be reached in a direct collision-free step.
We provide further comments to the planner as follows: (a) The graph search algorithm (A*) is optimal, that is, gives the shortest path for a predefined graph. Since the graph has a limited number of nodes and edges, it can, in theory, be possible to find a shorter path (unless the shortest path is a straight line, which coincides with the predefined nodes) by redefining the spatial position of the nodes in the 3D environment in which the UAV operates. (b) The smoothing in the above step 3 removes nodes along the path. Thus, removing any of them will, due to the triangle inequality, produce a path that is shorter or the same length as the original path.
The
The navigation system also supports assisted manual control, whereby the operator provides velocity commands via a joystick. In this case, the updater checks a certain distance along the direction of the desired (3D) velocity for collisions—if there are none, the velocity command is passed to the flight controller; if the path is obstructed, the drone is halted.
Sense and avoid
The purpose of the S&A module is to detect unexpected obstacles, that is, those obstacles that are not represented in the path planner’s offline map, thereby enabling the drone to move safely in 3D to negotiate narrow spaces on the substation under all weather conditions. The key requirements for the design of the S&A module are as follows:
An effective field of view (FoV) covering the full sphere around drone to enable safe movement in 3D.
Resolution and detection range giving the drone time to avoid even the smallest obstacles.
Robustness to poor visibility conditions.
To overcome these challenging design constraints and to make the problem more tractable, we have made the following system design decisions: (1) we enforce that the drone always travel “forward,” that is, the platform must yaw to face the direction of travel, thus reducing the required FoV for the S&A system to cover the forward, upward, and downward axes and (2) we leverage the dense graph of nodes used for path planning to constrain the task for the S&A module to simply verifying that the next node is “free” prior to assigning it as a waypoint for the flight controller, that is, at a minimum, the S&A module must only verify that the straight-line path (actual volume swept by the drone model) from the current position to the next node is free from obstacles.
This design philosophy reduces the required detection range of the S&A module to the spacing between graph nodes (which can be set arbitrarily, we use 1.5 m) and also allows us to make much stronger guarantees about safety. For example, it is difficult to guarantee that we have accurately detected all obstacles at any particular point in time (e.g. due to poor sensing conditions), but we can be much more confident that the space between the sensor and the obstacles we have detected is free, and directions for which we do not have reliable range measurements can be marked as uncertain. By building up a probabilistic free-space map over time, we can be confident as to whether the path to each successive node is traversable or not.
Our implementation is based around three Intel RealSense depth cameras (https://www.intelrealsense.com/depth-camera-d435) and a Hokuyo scanning lidar (https://www.hokuyo-aut.jp/search/single.php?serial=170) (Table 1). The three RealSense cameras were mounted parallel to the forward, downward, and upward axes, respectively, to enable high-resolution, wide-FoV obstacle detection around any direction of travel (including vertical flight). The Hokuyo scanning lidar was mounted upright to provide long-range and redundant obstacle detection in the horizontal plane (primary mode of locomotion). All sensors provide dense point cloud data in their local reference frame. To construct the obstacle map, point clouds are first transformed into the substation coordinate frame using synchronized pose data from the localization module before being fused by projecting them into a probabilistic voxel-based 3D representation. Our map implementation is conceptually based on OctoMap, 9 although we have made several optimizations based on the observation that an obstacle map needs only to capture the immediate volume surrounding the drone. Our optimized quadtree-based implementation, ModMap, uses the modulus operator to map continuous real space to a fixed memory footprint and maps the occupancy likelihood for each voxel in a finite volume surrounding the drone, whereby “free” voxels are inferred by sampling the view rays to inserted 3D points. To reduce computation load, we run the three RealSense cameras at a resolution of 848 × 480 pixels @ 6 FPS and downsample the point clouds using a 5 × 5 kernel. We found a voxel dimension of 20 cm and a maximum map dimension of 20 m (±10 m) to provide a good trade-off between obstacle detection capability and map update rate (6 Hz).
Specifications for the two sensors used by the sense and avoid module.a
FoV: field of view.
a Note that these values correspond to our configuration, and not the maximum performance capabilities of these sensors.
Our selected sensor configuration meets the design requirements for the S&A module but also enabled us to investigate the relative strengths and weaknesses of the two sensing modalities during flight testing under challenging conditions in the substation environment. We briefly summarize some of our key findings below.
Electromagnetic interference
Having a drone with advanced embedded electronics and critical communication subsystems, operating safely in a substation environment requires great emphasis on protection against EMI and operation close to potential high-voltage air breakdown. The substation is a hostile environment with regard to EMI, having very high voltages and magnetic fields varying at 50 Hz in addition to the occasional corona discharges that generate powerful EMI in a wide spectrum. High-voltage breakdown occurs when the applied voltage exceeds the breakdown voltage for the material. The resulting electrostatic discharge is typically a current arc through the material. Both the frame and propellers of the drone are made in carbon fiber, a material with conductive properties, which will be charged when placed in an electrical field and carry surface currents. The current through the drone will select routes that include the conductive parts of the frame, propellers, and also the antennas and electronics for navigation and communication, and it may damage both the drone and the electronics.
One of the key issues is to establish how close to a power line the drone can operate safely. A mock-up drone without the electronics was used in an experimental set up at SINTEF Energy Lab, where the separation distance to a high-voltage cylinder was controlled, and the onset of charging (partial discharge inspection value (PDIV)) was measured as a function of voltage settings from 80 kV to 400 kV. The drone was also moved closer to the high-voltage cylinder until high-voltage breakdown occurred, and a sparkling arc was observed between the closest propeller tip and the high-voltage cylinder. The minimum safety distance at 330 kV was measured to 3.2 m, defined by the onset of charging (PDIV). For the same voltage, high-voltage breakdown occurred at 0.5 m separation distance.
GNSS measurements are not measurably influenced by the EMI of the power grid due to the relatively high frequency of the satellite carriers between 1.1 GHz and 1.6 GHz, and the use of a 2.4-GHz Wi-Fi signal for sending RTK corrections. Past field measurement campaigns under 500-kV lines
10
have not evidenced any measurable impact of line adjacent charge distributions. While no anomalous inertial sensor behavior was detected, magnetometers are not recommended for use in or around electrical substations due to both the time-varying field as well as the high concentration of steel structure and magnetic transformers. The lack of disruption to the GNSS signal can be understood as from the point of view of the GNSS signals, the magnetic field of the substation is static and extremely far outside of the band to which the receiver is sensitive. The GNSS signals and the referenced USB3 EMI occur at 1.2–1.6 GHz, which is in the magnitude of
Flight test results
We performed over 200 experimental test flights onsite at Sunndalsøra with approximately six technology iterations over the space of 13 months. In this section, we discuss just two of the final test flights that demonstrated automatic inspection and collision avoidance.
Automatic inspections
By geotagging objects, we are able to create an automatic inspection. The position of both the desired object and the position of the drone creates an inspection view, such that the data gathered are consistent. By having comparable data, it is possible to track the state of a component over time. In a test at Statnett’s facility at Sunndalsøra, the inspection of 10 gauges was measured up, as seen in Figure 9, with the drone position in the numbered circles, and the desired object represented by the camera symbol to the right.
Mock-up drone without electronics in an experimental set up at SINTEF Energy Lab, measuring the minimum safety distance from a high voltage cylinder. In the image, a high voltage breakdown occurred between the closest propeller tip and the high voltage cylinder. Photo by Thomas Negård, Statnett, June, 2018. (image captured by one of the authors during one of the experiments in this project).
Inspection plan with 10 points of interest. The drone will launch from the red hangar, fly in a GNSS calibration circle, and then move to the numbered circles from 1 to 10, representing the individual drone positions in an inspection view and gathering data from the desired point of interest located to the right represented by the camera symbols. After the 10th point, the drone flies back to point 5 and back in to the hangar.
As the ARDIS relies only on GNSS localization both for navigation and landing on the hangar platform, the accuracy must be within the accepted limit before any inspection close to critical infrastructure can begin. This is by no means an easy criterion to guarantee with autonomous operations. To account for this, each inspection commences with the drone flying around in a calibration maneuver to align both the inertial and the GNSS solution. Normally, one full circle with a radius of 8 m at a velocity of 1.5 m/s was enough, depending on the GNSS constellations at the flight time.
In Figure 10, an example of data gathered during one of the inspections is shown. The camera platform is able to track the GNSS position of a gauge, giving consistent and high-quality images in each inspection. Due to the variety of depth in the surrounding structures, using autofocus proved to be a challenge. To account for this, a manual focus algorithm was developed by utilizing the distance to the desired object. The result of this is shown in Figure 10(a). In addition, Figure 10(b) shows an example of a thermal image of the surroundings at the same inspection point. During this test, the inspection was repeated three times, where the drone was charged in the hangar in between inspections. The measured flight path can be viewed in Figure 11. (A video demonstration of the system in operation is made available at https://youtu.be/BeKaO6u5yKE.)
Example of data gathered during an inspection. (a) The optical camera is able to capture high-resolution images of desired components and (b) the IR camera is able to capture thermal data of the component. IR: infrared.
Three consecutive missions, using flight plan from Figure 10. All inspections start with a GNSS calibration circle, where the drone flies in circle until the height standard deviation drop below a threshold of 4 cm. The first inspection started on the ground outside of the hangar, while the remaining two launched and landed on the hangar platform.
Example of network latency showing the frequency of RTK corrections received onboard the drone from the hangar during a mission with the same flight profile, as shown in Figure 11. All messages are received at the desired rate of 1 Hz without any significant transmission latency. RTK: real-time kinematic.
Collision avoidance
Collision avoidance was tested by placing obstacles in the drone’s flight path. Figure 13 shows flight data from a typical avoidance maneuver. The S&A system continuously updates the obstacle map and the path planner dynamically adapts the flight path to safely avoid the detected obstacle (see “Navigation” section for details). The object (ladder) is first detected at 10 m despite appearing in silhouette against sunlight reflected from smooth snow. The drone begins the avoidance maneuver approximately 8 m away from the obstacle and maintains a distance of at least 2.5 m to the obstacle from the drone’s center of mass at all times.
Flight test demonstrating collision avoidance maneuver: (a, c) the 3D perspective and top-down views of the requested (red dash) and actual (black solid) flight path as well as the reconstructed scene voxels from this short flight segment, including detected obstacle (blue), is shown and (b) a NIR image from the onboard RealSense camera during the flight test showing the obstacle (ladder). The drone was requested to proceed from start to goal but automatically adjusted flight path underway to avoid detected obstacle. Note that for this test, only the forward-facing RealSense depth camera was used, which lead to patchy ground reconstruction in smooth snowy areas due to reflected sunlight. The downward-facing sensor would have provided complete ground coverage if enabled. (a) 3D perspective view of flight path and detected obstacle, (b) NIR image from onboard RealSense, and (c) birds eye view of flight. NIR: near-infrared.
Lessons learned
In this section, we will discuss the challenges encountered during this project, including the system complexity of an ARDIS, localization accuracy and redundancy, fail-safes to ensure the safety in a fully autonomous system, and S&A sensor hardware selection.
During the 1-year test-phase, a substantial effort was made to predict and handle unexpected events. For the operator to be able to trust the ARDIS, there was a need for safe-guards and fail-safes to make sure that any failure in the system could be handled properly. The topographical effects of the location at Aura Substation provided more challenges for safe navigation between the structures than anticipated. To ensure the stability and deterministic behavior of the system, a series of fail-safe behaviors were implemented. In the event that the RTK-GNSS position uncertainty goes above a threshold during landing, the sequence will be aborted and an alternative landing position with greater tolerance in regard to horizontal error will be used. This will, in addition, handle the cases, where the communication between drone and hangar is lost. Related to that point is the redundancy of the ARDIS. In the extreme case, there were two different occasions the GNSS antennas had a hardware fault, resulting in complete GNSS loss and falling back on the INS solution. When flying close to the ground and surrounding objects, any significant accuracy drop may be dangerous. Especially, the height estimate degrades dangerously fast when switching from RTK-GNSS-based INS to only relying on the IMU. From this, there are two interesting lessons. The first addresses the lack of redundancy. The ARDIS was designed with high reliability on GNSS to measure the height, without any redundant height sensor such as laser or barometer. With the result that the recovery time to trigger a fail-safe and land the drone safely is in the range of a few seconds.
Despite these challenges, using an RTK-GNSS system to precisely land a drone on a hangar platform has proven to be a robust solution. In harsh weather such as heavy rain, snow, or wind gusts up to 15 m/s, the drone has performed multiple landings on the landing platform. During the landing sequence, the drone continuously corrects its horizontal position to counteract any disturbances from the wind. If the horizontal error grows above a threshold, the descent will halt while the drone corrects the horizontal position. As the drone nears the landing platform, the attitude of the drone is also constrained to ensure that the drone does not unintentionally hit the platform. In the event that the wind/wind gust is too strong and the drone is unable to successfully stay within the threshold for horizontal position, a timer will trigger after a configurable amount of time and land at the backup position. Similarly, the drone has been able to navigate close to critical infrastructure in all kinds of conditions. There are challenges related to the GNSS constellations and topographical aspects, redundancy in case of hardware failure, and loss of RTK corrections. But, in these rare events, it is possible to recover using the proper fail-safes, proving that RTK-GNSS is a reliable solution for localization in a substation.
The design of the drone can also be modified to reduce the safety distances and to make the drone more robust against some discharging in case of high voltage breakdown. It is recommended to add conductive guards around the propellers, which can be in the form of toroids with smoothly curved surfaces. The smooth surface will work to avoid locally high electric fields in addition to guiding the unwanted currents around the installed electronics. For the design and placement of the electronics, good electromagnetic compatibility design is ensured by (1) well-defined ground throughout the drone, (2) short cables, run cables close to ground (to avoid pick-up loops), (3) shielded coax cables, (4) include feed through capacitors and EMI filters for coax interconnections, (5) shielding of electronics, separate boxes, and (6) noisy boards and cables separated from sensitive components.
The ARDIS was designed highly reliant on a continuous access to correction data received from an encrypted customer provided Wi-Fi network. This required seamless transition between access points for the system to operate smoothly. This was complicated by the number of steel structures at the site, creating a rather challenging environment for continuous wireless network coverage. To account for this, the Wi-Fi access points were placed and oriented to optimally cover the whole area, ensuring that the communication link on the drone was able to roam between access points without any delay.
The selection of S&A sensors (RealSense depth cameras and 2D scanning lidar) provided a good tradeoff between FoV and reliability for obstacle detection. Complete FoV around and above/below the drone was important to meet the requirements for 3D flight paths among obstacles and necessitated the use of wide-FoV 2D sensors such as the RealSense depth camera to provide out-of-plane depth data. However, we found the depth cameras to be less robust than the scanning lidar (see S&A for details) and in particular could detect false-positive obstacles that sometimes prevented the drone from completing a mission. Since the completion of this project, rotating scanning 3D lidars (e.g. Emesent Hovermap) has become commercially available, which would provide a more robust, complete FoV obstacle detection solution, although at significantly higher hardware cost. Alternatively, we found that inspection missions could be largely constrained to a single flight altitude, requiring only obstacle detection around the horizontal plane, which can be provided by a 3D scanning lidar.
Conclusions and future challenges
Main conclusions are derived from the experimental results achieved at Aura Substation in Sunndalsøra Norway. The project has shown that an ARDIS is able to generate repeating inspection flights inside a high-voltage substation, capturing information from the asset with the human operator involved only for selecting and launching the drone from a remote location. This, however, requires emphasis on how the fail-safes will be handled in a location, where a straight return to a set location, or even flying straight down can lead to catastrophic results.
Critical problems with the technology include the limited areas of safe operations within a substation. If parts of the ARDIS fail, the fail-safes need to be programmed to always find a safe route to landing. Unsafe areas located in a 3D space present further challenges to safe navigation, especially if some of the sensor data are lost.
In summary, the project has provided important findings and results to better achieve resident drones for remote inspection on electrical substations.
Footnotes
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed the receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by Statnett.
