Drive-by infrastructure monitoring: a workflow for rigorous deformation analysis of mobile laser scanning data

Abstract

This paper presents a practical and efficient workflow for deformation monitoring of transport infrastructure. We propose using commercially available mobile laser scanning (MLS) systems to scan civil infrastructure while driving by in a car or rail vehicle. Our processing pipeline corrects for MLS-specific systematic deviations and models deformations from point clouds of two epochs. Following the concept of rigorous deformation analysis, we statistically test the deformations for significance. The required point cloud uncertainty may be obtained in two ways. First option is empirically by multiple passes and, secondly, by prediction with a learned stochastic model. We apply the method to three retaining structures and evaluate results based on ground truth geodetic surveys. The deviations did not exceed 10 mm, even for complex object surfaces or when traveling at 80 km/h. We demonstrate that the method is capable of revealing displacements in the centimeter range without relying on any installations on the structure. The approach shows great potential as a novel, efficient tool for detecting and quantifying defective structures in a road and railway network.

Introduction

Structural Health Monitoring (SHM) aims to observe a structure through measurements, extract damage-related parameters, and analyze data to assess the current state of health. ¹ Depending on the structure or mechanical system, different damage-sensitive features may be relevant for monitoring. In transport infrastructure maintenance, geometry is a frequently monitored parameter for bridges, tunnels, roads, rails, and retaining walls. ² The term deformation monitoring refers to the process of observing structural geometry changes, that is, strain, bending, torsion, displacement, and rotation. ³

Geodetic monitoring established itself as a versatile solution for safety-critical applications.^4,5 It includes optical surveys with total stations to distinct points mounted on the structure of interest and of stable points placed outside the deformation area. The processed angle and distance measurements provide 3D coordinates of these points in the established reference coordinate frame. A notable advantage of optical surveys with total stations is that by observing the same points from multiple setup locations, it is possible to assess the reliability and quality of the results. Consequently, the engineer can decide whether displacements of the points are significant or within the measurement uncertainty. The term rigorous deformation analysis stands representative of this approach.

Another related technology that has become increasingly popular for monitoring recently is terrestrial laser scanning (TLS). Many aspects are identical to traditional geodetic monitoring, including the measurement parameters and the establishment of a stable coordinate frame by reference points with known and stable coordinates. The main difference is that terrestrial laser scanner sample the entire structural surface in angular increments. The resulting point clouds provide rich geometry information. In contrast to total station monitoring, there are no point correspondences in laser scans acquired at different times, though. Due to the irregular object sampling by TLS, a direct comparison of 3D coordinates is thus not possible. So different strategies have evolved to derive information about geometry changes from point clouds. Some are very flexible and applicable to almost any object while others are designed for specific-use cases but require a high degree of manual interaction. Another side effect of no point being measured twice is that no information about quality or reliability can be derived, as is the case with total station measurements. This fact raises the question of how to carry out rigorous deformation analysis if no uncertainty information is available for the point clouds. So far, no standard operating procedure is available, and research is still ongoing.

In recent years, new mobile laser scanning (MLS) devices came onto the market, promising increased productivity. The workflow differs greatly from TLS as point clouds are captured while moving around with a backpack and handheld devices, remotely steering drones and robots or operating systems on land-based vehicles. In particular, MLS systems mounted on a car or railway vehicle seem promising for infrastructure monitoring (see Figure 1). Collecting data without impeding traffic flow or closing lanes brings huge cost and time benefits.

Figure 1.

Concept of deformation monitoring using mobile laser scanning and the quality influencing factors: trajectory (t), distance (d), angle of incidence ( $\phi$ ), and surface curvature (c); the red dots represent points of the point cloud.

Despite the clear advantages, there are few to no case studies in which this relatively young and expensive measurement technology is used for such safety-critical applications. In other words, there is a lack of practical experience. In this publication, we summarize the key findings from 6 years of experience surveying 24 retaining structures with four commercial MLS systems and processing 300-point clouds with approximately 7 billion points.

In this paper, we focus on three retaining structures A, B, and C (see Figure 2). The selection of structure A is motivated by the fact that extensive reference data with total stations and high-frequency tilt values with a low-cost wireless sensor system is available for more than 1 year. B represents a typical retaining structure with ongoing geodetic monitoring by a service provider and C is a structure where large deformations have actually occurred due to nearby construction works.

Figure 2.

Three retaining structures A, B, C serve as examples for the proposed workflow in this paper. The illustrations also show the structural dimensions and the extent of vegetation growing above the objects.

Our aim is to achieve a workflow that provides reliable data on deformation and uncertainty using MLS (see Figure 1). The proven rigorous deformation analysis of point-wise displacements by total stations presents the benchmark for the new method. This paper elaborates the individual components of the workflow, focusing on the following challenges:

1. Objects of transport infrastructure vary in shape and size. A deformation modeling strategy for point clouds should thus be flexible and highly automated for large-scale application.

2. The uncertainty of the results depends on data acquisition and processing and is unknown a priori. However, knowledge about precision and accuracy is crucial to obtain trustworthy results and to distinguish between significant deformations and measurement uncertainty.

3. MLS enables fast data collection but the question remains of how to establish a stable reference frame efficiently. The old-proven method of measuring reference points with total stations proves too time-consuming and thus costly.

This article first introduces the reader to the concept of rigorous deformation analysis of total station measurements, applications of TLS, and the challenges associated with using MLS. Building on existing work, we then provide details of our tailored workflow to address the challenges identified above. Finally, we present the results and evaluate them using geodetic monitoring data for the three retaining structures A, B, and C (see Figure 2).

State-of-the-art in laser scanning and deformation monitoring

Rigorous deformation analysis of total station measurements

Total stations are optical surveying instruments that measure horizontal, vertical angles, and distances $(\mathrm{hz},v,d)$ from a setup location to distinct points, mostly retro-reflective targets. These measurements allow the computation of 3D coordinates that refer to a local coordinate system with the origin at the instrument’s center and the z-axis aligned with the plumb line. Relative movements of object points may reveal strain, bending or torsion and may indicate structural stress. An advantage of geodetic monitoring over other established monitoring sensors such as strain gauges and fiber optic sensors is its ability to determine rigid-body displacements and rotations. A rotation corresponds to a change in orientation. Accordingly, one can describe a rotation by vectors between object points in 3D space. A prerequisite is a stable coordinate frame throughout the entire monitoring campaign, to which the measurements of all setup locations refer. The realization of such a reference frame requires fixed points outside the expected deformation area. Therefore, expert knowledge of structural behavior, predicted deformations of the structure, and the surroundings is beneficial.

Optical surveys provide empirical uncertainty information for the 3D coordinates when carried out from multiple setup stations (see Figure 3, left). From a statistical point of view, this procedure is equivalent to drawing multiple samples independently from a population. Hence, faulty observations are detectable mathematically and results are more trustworthy. The typical processing procedure establishes an over-determined, linear equation system $\mathbf{Ax}\mathbf{=}\mathbf{l}+\mathbf{e}$ to estimate the unknown coordinates $\mathbf{x}={[{(x,y,z)}_1,\dots ]}^T$ based on the observations $\mathbf{l}={[{(\mathrm{hz},v,d)}_1,\dots ]}^T$ and covariance of observations ${\mathbf{\Sigma }}_{\mathbf{ll}}$ . ⁶ $e$ stands for the residuals. The solution of the equation system yields $\mathbf{x}$ and the corresponding covariance matrix ${\mathbf{\Sigma }}_{\mathbf{xx}}$ at each epoch. The rigorous deformation analysis incorporates the displacement vectors between epoch 1 and 2 ${\mathbf{d}}_{12}\mathbf{=}{\mathbf{x}}_2-{\mathbf{x}}_1$ and the covariance of the deformations ${\mathbf{\Sigma }}_{\mathbf{dd}}={\left({\mathbf{\Sigma }}_{\mathbf{xx}}\right)}_1+{\left({\mathbf{\Sigma }}_{\mathbf{xx}}\right)}_2$ in a two-stage procedure. ⁷ First, the global congruence test ⁸ determines statistically whether any displacement vector deviates from zero with a predefined confidence level. Vectors that cross the ellipses in Figure 3 on the right indicate significant displacements with 95% confidence (red-colored ellipses). If the binary test of congruence reveals significant deformations within the group of object and reference points, an iterative process begins to identify the actual stable and moved points. This is an essential quality assessment measure, since putative reference points can move and thus may affect the quality of the entire reference frame and the final displacement vectors. It should be noted that the congruence model is the simplest model to apply, while more sophisticated models incorporate time and external influences to identify abnormal structural behavior.^9,10

Figure 3.

Rigorous deformation analysis geodetic surveys at retaining wall A: schematic representation of observing distinct points on the structure from three setup stations (white, black, and red lines; left); horizontal displacement vectors and 95% confidence ellipses (right); red ellipses indicate significant deformations while the green ones represent stable points.

Geodetic monitoring has been around for decades and much experience has been built up regarding its performance and the accuracy of the calculated displacement vectors. Knowing the quality of the measurement data is crucial to obtaining trustworthy and interpretable results and ultimately adding value to the assessment of the health condition of a structure.

Reference frame for laser scanning point clouds

Like total stations, laser scanners record angle and distance measurements, but with the big difference of sampling the entire object surface instead of selectively measuring individual points.

Another similarity to total station measurements is that a reference frame is also needed to register the local point coordinates of each scan. There are two approaches to realizing the reference frame. The first involves installing reference points and determining their coordinates by performing optical surveys with total stations in advance. The prism targets are then replaced by circular or flat targets that are recognizable in the laser scans. Representative case studies use this registration method for monitoring of dams,^11,12 lock gates, ¹³ bridge decks, ¹⁴ and television towers. ¹⁵ The second method relies on computing transformation parameters between scans by minimizing discrepancies for stable reference surfaces. This method is preferable if reference points are very close to the setup station or do not exist at all. Such a situation is typical for rockface or landslide monitoring. The challenge is finding reference surfaces, especially for data sets that were collected with a large time gap. There are multiple publications following similar strategy . First, the point cloud is subdivided using an octree data structure before computing centroid displacements ¹⁶ or applying the Iterative-Closest-Point (ICP) algorithm ¹⁷ for aligning the point cloud patches.^18,19 Those octree cells with similar parameters provide potential reference surface candidates for point cloud registration.

Deformation modeling using point clouds

The second challenge in using TLS for monitoring tasks, besides the realization of the reference frame, is the modeling of deformations. A direct point-to-point comparison is not possible, since the point cloud sampling will differ in general. It depends on the device settings, the setup location, and the object geometry. Literature provides a multitude of solutions to this specific task. We categorize them according to their outcome:

1. 3D displacement vectors in distinct points,

2. magnitude of deformations in each point,

3. transformation parameters for subgroups of point clouds and

4. parameters of fitted geometric primitives and their changes.

Category 1 algorithms: 3D displacement vectors in distinct points

The point sampling of two epochs can differ completely due to different station setups, yet there is a chance that at least some points of both clouds match. So the strategy of the first category algorithms is to define similarity criteria, search for similar points, and then proceed as with signalized target points in geodetic monitoring. The idea of point matching stems from photogrammetry and computer vision. ²⁰ Research provides numerous key point detectors and feature descriptors for image data. Consequently, researchers sought to simplify the 3D to a 2D matching problem. Wagner. ²¹ introduces additional RGB data for the matching process and then uses the relative orientation to identify the corresponding points in the clouds. Another approach aims to map the polar elements ( $\mathrm{hz},v,d$ ) to a planar representation and using the range or laser intensity as gray scale information. ²² However, recent publications show the trend toward direct 3D point cloud matching with hand-crafted and learned 3D feature descriptors. ²³ The latter provide a compact representation at low matching time and high accuracy using deep learning. Gojcic et al. ²⁴ evaluate such descriptors’ feasibility for TLS-based monitoring of a landslide. In contrast, human-made structures consist of regular surfaces, providing less distinct points for matching. In such a case, the strategy of segmenting and intersecting planes for tracking virtual corner points may prove more viable. ²⁵

Category 2 algorithms: Magnitude of deformations in each point

Representative algorithms of the second category are the most applied. They work de facto for any type of structure, whether natural or artificial. Therefore, implementations of these algorithms are included in many software programs for point cloud processing. Given a point cloud A, the Cloud-to-Cloud (C2C) algorithm searches for the nearest neighbor in B and calculates the Euclidean distance. Similarly, the cloud-to-mesh (C2M) method searches for the smallest distance between each point of point cloud A and a surface $S_{\mathrm{b}}$ . However, the distances are signed, as the normal of the mesh is incorporated as additional information. The principle idea originates in inspection of as-built mechanical structures. ²⁶ In practice, computer-aided design models are missing for deformation monitoring, though . This method thus requires mesh generation in advance, which depending on the technique, is mostly time-consuming and smooths out details . ²⁷ Reported use cases include monitoring of an artificial concrete specimen ²⁸ as well as a rockfall ²⁹ among others. The so-called M3C2 (Multiscale Model to Model Cloud Comparison) algorithm ²⁷ is another popular representative that constitutes a mixed form of C2C and C2M. It models the surface locally, computes the signed distance in normal direction, and performs a smoothing. Noteworthy, the M3C2 algorithm provides uncertainty estimates for identifying significant deformations. However, this measure depends on the surface roughness and the registration errors only. Roughness may be a dominant error cause for landslide monitoring, but for structural monitoring, neglecting point cloud correlations yields too optimistic uncertainty estimates in general. ³⁰

Category 3 algorithms: Transformation parameters for subgroups of point clouds

The third category of deformation models follows a strategy already mentioned. The idea of computing rigid-body transformations via ICP for point cloud registration is applicable to the deformed area as well. The estimated translations and rotations provide an intuitive representation of the deformed scene. The question is how to partition the point cloud into areas with homogeneous deformations. The most reliable solution is to define it by hand, ³¹ but also the most costly. The most efficient way is choosing all points within an octree cell for alignment. ³² However, the larger the cell size, the more likely it is that the assumption of rigid-body motion is violated.

Catergory 4 algorithms: Parameters of fitted geometric primitives and their changes

The approaches of the last listed category are especially relevant for artificial structures with distinct geometries. Point clouds provide redundant data for fitting geometric primitives and thus a much higher accuracy for the deformation parameters than the specified single point accuracy would suggest. ³³ Representative case studies analyze parameter changes including focal length of a parabolic radio telescope, ³⁴ bending of concrete and timber beams, ³³ tilt of tall chimneys, ³⁵ and the bending line of a cylindrical television tower. ¹⁵ While these examples focus on structures with a clear geometry, other objects exist that are described best by multiple mathematical surfaces. Examples include planar and hyperbolic parts of a bridge, ³⁶ lock doors of a sea lock, ³⁷ or concrete elements of an anchored retaining wall. ³⁸ All these methods propose to first segment and then parameterize the individual elements.

Uncertainty in laser scanning point clouds

Four groups of error sources impede the quality of laser scanning point clouds: (i) registration and geo-referencing, (ii) instrumental insufficiencies, (iii) meteorological conditions, and (iv) the interaction of laser beam and object surface.^39,40 Literature provides exemplary publications that attempt to model the point cloud uncertainty for simulated and practical examples from empirical values and data sheets.^41,42 Creating such a model is challenging because many factors systematically influence uncertainty. For the sake of simplicity, researchers thus focused on individual effects.^43,44 Wujanz et al. ⁴⁵ discovered a way to indirectly relate all influences (angle of incidence, distance, and surface properties) to range noise. They derived a mathematical function for the raw intensity of the reflected laser beam and the range precision of a terrestrial laser scanner. Inspired by this work, Heinz et al. ⁴⁶ applied this method for modeling uncertainty of a 2D profile laser scanner, and Schmitz et al. ⁴⁷ used point clouds with scaled intensities of another terrestrial scanner.

Range precision presents a small portion of the entire error budget of laser scanning, though. Wujanz ⁴⁸ demonstrated that errors due to an unfavorable measurement configuration dominate the overall uncertainty of C2M comparisons of an arch bridge, making the range noise negligible. ⁴⁸ He argues that the object sampling is more important than the single point precision. Heinz et al. ⁴⁹ confirm this theory in experiments scanning a white, planar specimen. They observe that increasing point density has more impact on the precision of estimated plane parameters than choosing a higher quality setting.

It is tempting to assume that increasing the number of points will positively affect the quality of the point clouds in any case. The distribution of points turns out critical for deriving reliable results. Holst et al. ⁵⁰ demonstrated that in case of TLS point clouds, the irregular sampling may bias the parameter estimation for a locally deformed plane. One proposed solution is to reduce data to points on a nearly regular grid or to incorporate spatial correlations with constant radii to achieve a similar effect. The latter strategy is reasonable since measurement configuration and surface properties affect neighboring points in a similar way. ³⁰ The estimated precision from geometric surface fitting, from ICP methods, or from the local modeling component in the M3C2 algorithm thus provide too optimistic measures for the overall scanning uncertainty.

There is room for further research on modeling laser scanning uncertainty. Knowledge about the stochastic and the systematic influences is crucial for identifying significant deformations and thus for the concept of rigorous deformation analysis. The influencing factors are manifold, and, until today, there is no feasible approach to derive quality information for laser scans of arbitrary objects under practical, outdoor conditions.

Systematic effects in MLS point clouds

Vehicle-based MLS systems utilize 2D profile scanners that capture the surroundings while driving (see Figure 1). Although the point sampling also depends on the vehicle motion, it is generally more homogeneous than in point clouds acquired with static TLS. This is beneficial, since laser scans with regular spacing are less prone to biased estimates. ⁵⁰

In MLS, however, there are many other sources of systematic deviations. The reason is that point clouds are the result of fusing data from multiple sensors. Each sensor is subject to errors, as is the combination of these. Researchers emphasized that missing functional models and error distributions complicate the uncertainty modeling. ⁵¹ Considering the geo-referencing equation (1), we highlight the three main sources of error: geo-referencing, calibration, and scanning itself.

$$\begin{array}{c}\multicolumn{1}{c}{\left[\begin{array}{c}{p}_x\hfill \\ \multicolumn{1}{c}{p_y}\\ \multicolumn{1}{c}{p_z}\end{array}\right]=\stackrel{\mathrm{geo}-\mathrm{referencing}}{\overbrace{\left[\begin{array}{c}{t}_x\hfill \\ \multicolumn{1}{c}{t_y}\\ \multicolumn{1}{c}{t_z}\end{array}\right]+\mathbf{R}(r,p,y).}}}\\ \multicolumn{1}{c}{\left(\left[\begin{array}{c}{\mathrm{\Delta }}_x\hfill \\ \multicolumn{1}{c}{{\mathrm{\Delta }}_y}\\ \multicolumn{1}{c}{{\mathrm{\Delta }}_z}\end{array}\right]\underset{\mathrm{calibration}}{\underbrace{+\mathbf{R}(\alpha ,\beta ,\gamma ).}}\underset{\mathrm{scanning}}{\underbrace{\left[\begin{array}{c}{s}_x\hfill \\ \multicolumn{1}{c}{s_y}\\ \multicolumn{1}{c}{s_z}\end{array}\right]}}\right)}\end{array}$$

(1)

Systematic deviations related to scanning were already part of the discussion in the previous section. Object properties, as well as distance and incidence angle affect MLS point clouds systematically. However, the influence of geo-referencing and calibration is much higher.

Direct geo-referencing means that position and attitude are derived from GNSS (Global Navigation Satellite System), IMU (Inertial Measurement Unit), and odometer data. Theoretically, it is possible to calculate the position with an accuracy at lower centimeter level by incorporating additional GNSS signals from control stations nearby. However, under unfavorable practical conditions in alpine valleys and near tunnels, two geo-referenced MLS point clouds showed deviations of up to 20 cm. ⁵² The quality of position and attitude depends, among other factors, on location and time, respectively. Geo-referencing thus varies with time, reportedly generating variations of up to 10 cm within 200 m. These discrepancies are the reason why two MLS point clouds of a retaining wall cannot be aligned in terms of rigid-body motion. In other words, neither the ICP method nor the use of ground control points prove suitable to realize a stable reference frame.

System calibration presents another potential error source in MLS. It implies determining the position and orientation of the scanners relative to the IMU coordinate system. The six parameters are three boresight angles $\alpha ,\beta ,\gamma$ and the three components of the lever arm $\mathrm{\Delta }={[{\mathrm{\Delta }}_x,{\mathrm{\Delta }}_y,{\mathrm{\Delta }}_z]}^T$ . The calibration strategies differ greatly among manufacturers of MLS systems. For example, some rely on annual factory calibration while others provide customers with procedures for field calibration. Both approaches have their strengths and weaknesses. Suboptimal conditions during field calibration may result in inaccurate parameters. The environmental conditions can also have a considerable impact on stability of the factory-calibrated parameters. The consequence could be that with dual-scanner systems, the two scanners of the same MLS system deviate significantly from each other. Not just the purchased components, but also the operation of the MLS systems determines the quality of the sampling. In case of dual-scanner systems, before merging clouds of left and right scanner of a service provider, it is thus highly recommended to check the co-registration quality first. ⁵²

Summary and scope of work

A standard operating procedure for rigorous deformation analysis of laser scanning point clouds does not yet exist. The main reason is that uncertainty information is missing for the scan points, and it is thus impossible to decide whether deviations are significant or explainable by measurement uncertainty. There are numerous research publications on the derivation of covariance matrices of point clouds. Some of them offer complex mathematical approaches that may not be easily applicable in practice.

In the case of MLS, uncertainty modeling is even more complex. Measurement errors are time-dependent, and for some of them the magnitude and distribution is unknown. For example, when calculating the trajectory without costly ground control points, the resulting point clouds may show errors in the decimeter range that vary within a short time. ⁵² The lack of knowledge about accuracy and precision is why this promising measurement technology has not yet established itself for SHM.

However, in this paper, we propose a working procedure for rigorous deformation analysis of MLS data. We build upon existing work and therefore highlight the overlap of our approach and previous publications.

• Under challenging conditions, direct geo-referencing produces trajectory errors that dominate the overall uncertainty budget. Since errors are time-dependent, point cloud registration using ground control points^11–14 or point cloud-based methods such as the ICP algorithm ¹⁸ will fail in case of MLS. We adopt the idea of Kalenjuk and Lienhart ⁵² to determine the time-variable deviations between mobile laser scans. We extend the approach by starting a search for the putative deformed objects ³⁸ to be excluded from the point cloud registration.

• We co-register point clouds of two scanners of one drive or scans of two drives to remove the major systematic error component in MLS (see equation (1)). The remaining errors are mainly due to the scanning process itself and thus are similar as for TLS. We follow the suggestion by Kuhlmann ⁵¹ to quantify uncertainty of MLS empirically. The strategy is to collect a set of point clouds by passing by a scene several times within a short interval and deriving the precision from multiple measurements.

• As proposed by Kalenjuk and Lienhart, ⁵² we use trajectory data to reconstruct the acquisition geometry of MLS point clouds. We exploit this information along with surface parameters such as roughness ³⁸ to build a stochastic model for a commercial MLS system. It enables rigorous deformation analysis of a structure without requiring to collect point clouds multiple times. Such models could therefore bring additional time and cost savings.

Method

Overview

Before going into detail, we first give an overview and clarify notations. Our approach enables rigorous deformation analysis of point clouds captured with a commercial MLS system in a two-stage procedure (see Figure 4). In the quality evaluation and modeling phase, we first determine the quality, i.e. uncertainty of the used MLS system empirically. In the monitoring phase, we incorporate this information to decide whether the computed point cloud deviations represent significant deformations or are due to measurement uncertainty (black, solid rectangle in Figure 4).

Figure 4.

Processing mobile laser scanning data for rigorous deformation analysis requires two phases: the quality evaluation and modeling phase (top), and the monitoring phase (bottom).

In the quality evaluation and modeling phase, the recommended workflow is as follows:

Data collection

When using MLS for capturing structures for the first time, pass by multiple times to collect at least three independent data sets ( $n\ge 3$ ). These are used to determine precision and improve quality, as will be elaborated later.

Scanner alignment

Before merging point clouds of left and right scanner, check the co-registration quality and correct it if necessary. ⁵² Otherwise, deformation modeling will be less accurate.

Co-registration

Align the point clouds using the co-registration method that accounts for time-variable geo-referencing errors. ⁵²

Deviation/deformation modeling

Choose a point cloud modeling strategy that works automatically and repeatably. Compute deviations between two point clouds for all $\left(\begin{array}{c}n\hfill \\ \multicolumn{1}{c}{2}\end{array}\right)=\left(\begin{array}{c}3\hfill \\ \multicolumn{1}{c}{2}\end{array}\right)=3$ combinations.

Statistical analysis

The expectation value for the derived parameters is zero. Compute the root-mean-square deviation (RMSD) for each point, region, or segment individually. Since the RMSD values vary in the point cloud, we denote the color-coded point cloud as precision map. The precision map is valid for a given scene, a specific MLS system, and approximately equal conditions (driving lane and velocity).

Stochastic modeling

The last step of this phase is optional, but recommended. Create a stochastic model based on the empirical precision map. Calculate features related to the acquisition geometry and surface roughness. All that is needed is the time-tagged point cloud and the trajectory. Both are typical results of modern MLS systems.

Most processing steps are similar in both phases. Some specialties should be noted in the monitoring phase:

Reference frame realization

The realization of the reference frame requires stable regions for co-registration. Assuming a non-deforming road surface or stable guardrails, we first search for the allegedly deformed structure in a highly automated manner. By doing so, we can exclude it from the co-registration process. However, if nothing can be considered stable, this approach will not work. Then we must rely on the idea that high-frequency trajectory errors occur differently in point clouds of multiple runs and average the results. We have observed both scenarios and report on them in the “Evaluation in practical case studies” section.

Statistical testing

With the empirical precision map available, we can test deformations for significance. If only one point cloud is acquired from a scene, we can leverage the stochastic model learned from another setting to predict precision. The prerequisite is features calculated based on the current point clouds and trajectories.

In the next subsections, we will cover data processing in more detail. We conclude this section with a concise specification list of requirements for data collection and processing.

Scanner alignment

The first thing to check after obtaining geo-referenced point clouds is how well the data of the two scanners match. Remaining calibration errors or uncertainty in the geo-referencing might be reasons for time-variable deviations. If left untreated, the merged point clouds would provide lower accuracy than the scanner data sheets suggest. The solution to align them relies on the same procedure as for two point clouds from different passes. More details on the co-registration method are thus available in the next section.

Reference frame realization

The term reference frame originates from traditional geodetic monitoring. It implies that all measurements have a common coordinate system to which they refer. Although the coordinate system remains the same for multi-temporal mobile laser scans, the point clouds may be shifted due to uncertainties in geo-referencing. Therefore, both the position and the shape are not stable within the coordinate frame. The reason for the latter are the time-dependent measurement errors in MLS.

Realizing a reference frame is thus a matter of obtaining mobile laser scans that overlap in areas where no movements occurred. To accomplish this task, we need to distinguish between two scenarios:

• I. Besides the structure to monitor, engineers expect the surrounding environment to be stable. In such a case, we exploit this prior knowledge to co-register the point clouds of two epochs based on these stable areas.

• II. Due to mass movements or construction works, there may be no stable region at all. We demonstrate an example of a retaining structure where we faced this challenge. Having this kind of prior knowledge is critical to processing the data correctly. We opted for the workaround of creating a quality-enhanced point cloud by averaging multiple clouds, which proved viable in this case.

Scneario I: We assume two point clouds, a reference cloud $P=\{{\mathbf{p}}_1,{\mathbf{p}}_2,\dots ,{\mathbf{p}}_{\mathbf{n}}\}$ and a query cloud $Q=\{{\mathbf{q}}_1,{\mathbf{q}}_2,\dots ,{\mathbf{q}}_{\mathbf{m}}\}$ (Figure 5, left). Both result from mobile laser scans, providing time information for each point ${\mathbf{p}}_{\mathbf{i}}=[x_i,y_i,z_i,t_i]$ so that $(P,Q)\in {\mathbb{R}}^4$ . Our goal is to transform $Q$ such that deviations from $P$ become minimal, resulting in a new cloud $Q^{\mathrm{Reg}}$ (Figure 5, right). Since there are boundary conditions to consider, it is not an ordinary point cloud transformation. Concretely, we need to identify stable regions in both clouds $P,Q$ and consider the time-variable errors.

Figure 5.

Realization of a reference frame by co-registration of stable regions in two mobile laser scanning point clouds.

The first question is how to decide which regions are stable and which are moving based on point cloud data alone. Unless we receive other information from civil engineers or geotechnical engineers, we assume that the retaining structure could move while, for example, the road and guardrails should be stable. Our strategy is to segment the retaining wall from the query point cloud $Q$ . As a first step, we identify the ground as the surface with the highest point density and with the lowest height variation by statistical analysis of the XYZ coordinates. ³⁸ Then, we compute the difference of normals in each point of the remaining set. It allows to filter the difference vectors by their magnitude $\left|\mathrm{\Delta }\mathbf{n}\right|$ to retain smooth surfaces. Finally, the remaining points are clustered into objects using Euclidean clustering. The object with the highest point number is interpreted as the structure to monitor $Q_{\mathrm{unstable}}$ (black region in Figure 5, top left). Consequently, the stable region defines the relative complement $Q_{\mathrm{stable}}\leftarrow Q\backslash Q_{\mathrm{unstable}}$ . More details on the automatic segmentation algorithm are available in Kalenjuk et al. ³⁸

Time-variable errors in MLS are why co-registration methods fail to align two point clouds accurately. Despite the time dependency, one may consider the errors constant within sufficiently small time intervals. The strategy is thus to use the nanosecond timing information in mobile laser scans to partition the point cloud. ⁵² Since most MLS systems use profile scanners, the temporal partitioning will yield subsets $Q_{\mathrm{prof}}$ with points aligned within profiles (red lines in Figure 5, top left). The minimum time interval is the reciprocal of the profiling rate as at least one profile is necessary for the alignment via rigid-body transformation. For example, with two profile scanners with 200 Hz rotation and 1 MHz scanning rate, theoretically an MLS system would scan 10,000 points within 0.005 s. However, the final number of points used for registration depends on the scenery and the extent of the stable region. Therefore, it may be advisable to choose a larger time interval. The cloud used for computing the rigid-body transformation is formally defined as the set intersection of the stable region and the partitioned subcloud $Q_{\mathrm{sub}}=Q_{\mathrm{prof}}\cap Q_{\mathrm{stable}}$ (blue lines in Figure 5, top left).

With $Q_{\mathrm{sub}}$ , we have found small portions of the query cloud that presumably present stable areas. The corresponding counterparts of $Q_{\mathrm{sub}}$ in the reference cloud $P_{\mathrm{sub}}$ are required for co-registration. We obtain these subsets not by semantic interpretation (as for $Q$ ) but by searching for the nearest neighbors in $P$ (lower left in Figure 5) for all points of $Q_{\mathrm{sub}}$ . Despite having the point clouds coarsely aligned, the nearest neighbor search may produce faulty point correspondences that influence point cloud registration with the ICP algorithm. Therefore, we instead use the registration method proposed by Kalenjuk and Lienhart ⁵² to minimize discrepancies between $Q_{\mathrm{sub}}$ and $P_{\mathrm{sub}}$ in the surface normals direction (Figure 6). This approach proved robust to erroneous point correspondences, a small number of points, and the unfavorable point distribution of the profile clouds $Q_{\mathrm{sub}},P_{\mathrm{sub}}$ , where the samples lie in a plane. Before translating and rotating the subsets $Q_{\mathrm{sub}}$ to best match $P_{\mathrm{sub}}$ , we perform a smoothing of the transformation parameters to avoid discontinuities in the transition zones. The transformation of the subsets $Q_{\mathrm{prof}}$ with the estimated, smoothed parameters represents our profile-based co-registration method.

Figure 6.

Co-registration of $Q_{\mathrm{sub}}$ and $P_{\mathrm{sub}}$ (gray and black) by minimizing the orthogonal discrepancies (black arrows); the dotted lines indicate the nearest neighbors.

Scenario II: Reference cloud $P$ and query point cloud $Q$ do not contain stable regions at all. A reason may be large-scale settlements that affect both, the structure, and the surroundings including road surface, and so on. Hence, the proposed co-registration method will likely underestimate the true deformations. Registration of point clouds based on stable areas outside the deformation region is also no option. Geo-referencing errors evoke systematic deviations that may vary within short time intervals and distances, respectively. ⁵² It should be noted that for this approach to work, data outside the deformation area would be needed in the first place. Therefore, the exchange with civil engineers and geotechnical engineers is crucial before conducting MLS campaigns.

In such a scenario, possibilities are limited for realizing a reference frame. We opt for the direct geo-referencing solution but enhance the quality by multiple measurement passes. Let us consider a point ${\mathbf{p}}_{\mathbf{i}}$ for which we can establish a point correspondence between point clouds from two epochs $t_1$ and $t_2$ (equation (2)). The observation of the true signals, that is, the surfaces, will be falsified by the time-dependent systematic errors and the measurement noise.

$$\begin{array}{c}{\mathbf{p}}_{\mathbf{i}}\left(t_1\right)=\mathrm{signal}\left({\mathbf{p}}_{\mathbf{i}}\right)+\mathrm{sys}.\mathrm{error}\left(t_1\right)+\mathrm{noise}\left({\mathbf{p}}_{\mathbf{i}}\right)\hfill \\ \multicolumn{1}{c}{{\mathbf{p}}_{\mathbf{i}}\left(t_2\right)=\mathrm{signal}\left({\mathbf{p}}_{\mathbf{i}}\right)+\mathrm{sys}.\mathrm{error}(t_2)+\mathrm{noise}\left({\mathbf{p}}_{\mathbf{i}}\right)}\end{array}$$

(2)

Following the findings of previous work, ⁵² we must assume systematic deviations due to geo-referencing between two point clouds, no matter whether collected within 5 min or a year. Accordingly, the point clouds recorded in several passes in one epoch will be influenced by systematic errors differently. Although the distributions of the errors are unknown, ⁵¹ we expect $E\{\mathrm{sys}.\mathrm{errors}\}=0$ . Note that due to correlations in laser scanning, $E\left\{\mathrm{noise}\left({\mathbf{p}}_{\mathbf{i}}\right)\right\}\ne 0$ .^30,50

It is worth mentioning that this strategy neglects the atmosphere as a traditional influence factor. In contrast to tachymetric observations, the mapping geometry remains the same along the entire wall. Even for tall structures, the distances are no greater than 35 m (c.f. Figure 9). Range distortions are thus negligible. Moreover, one can reduce refraction-related angle distortions by scheduling the mapping campaign on cloudy days.

Our strategy is to reduce the geo-referencing-related systematic component by scanning the scene multiple times and averaging the point clouds. However, we do not average the coordinates of corresponding points individually, but use a modified version of the profile-based co-registration method (see Figure 5). The motivation for this procedure is robustness and precision in the presence of noise. In general, the workflow is similar to the one presented in “Scenerio I” (see also Figure 5). In a nutshell, it adopts the following procedure:

1. Establish direct point correspondences by searching for the nearest neighbors in both MLS point clouds,

2. Perform temporal partitioning,

3. Estimate transformation parameters by minimizing point cloud deviations in normal direction,

4. Smooth the transformation parameters and

5. Apply the transformation to both point clouds. More precisely, rotate both clouds by half the angle, but in opposite directions, and shift them by half the translation vector, each with a different sign.

The proposed procedure reduces systematic deviations between two point clouds. Since we captured at least three laser scans of each structure in each epoch, we can apply the method to all $\left(\begin{array}{c}3\hfill \\ \multicolumn{1}{c}{2}\end{array}\right)=3$ point cloud pairs. Finally, we “average” these results again to obtain a point cloud in which the signal ( ${\mathbf{p}}_{\mathbf{i}}$ ) predominates and the systematic errors $(\mathrm{sys}.\mathrm{error}(t))$ are largely removed. For a detailed explanation on the concept of the quality-enhanced point cloud, we refer the reader to Kalenjuk and Lienhart. ⁵²

Deformation modeling

Through the previous processing steps, the point clouds $P$ and $Q$ should be aligned at regions that have not moved in the time between data collections. Thus, the point clouds are referenced in a stable coordinate frame and ready for comparison. However, due to the generally different sampling, a direct point-to-point comparison of $P$ and $Q$ is not possible. Modeling deformations from point clouds requires a processing strategy that is flexible to the structure type, surface, and expected type and pattern of deformations. For example, it should work for the three different retaining walls and reveal structure-specific parameters including local settlements, lateral displacements, and tilting. To fulfill these requirements, we draw on concepts from existing strategies, including the M3C2 algorithm and point cloud-based methods providing transformation parameters (see Section “Deformation modeling using point clouds”).

Our strategy is to model the deformation pattern of the entire structure as a composite of many rigid-body motions. The procedure consists of two steps (Figure 7). We first subdivide the mobile laser scans into point groups that adhere to object boundaries. Then, we estimate the translation and rotation parameters that best align the surface patches of cloud $P$ and $Q$ .

Figure 7.

Visualization of normals of a cloud of a crib wall in the HSV color space, labeled as normal map (top); voxel clustering of the cloud with a voxel size of 1 m (bottom left); and cascading region growing-based segmentation and voxel clustering (bottom right) to obtain small point sets that adhere to object boundaries (c.f. object details in normal map). HSV = hue, saturation, value.

Step 1: Point cloud partitioning

A prerequisite for the partitioning algorithm is surface normals. We compute them at each point using the surrounding neighbors within a specified radius. The radius choice depends on the structure’s surface and the minimum feature size to resolve (see normal map in Figure 7). By decreasing the search radius, the object description becomes more detailed, but the variance also increases. Considering retaining wall C with 30 cm wide reinforced concrete elements or retaining wall B with 30 cm wide block joints, we propose a radius of 15 cm for normals computation. We leverage the descriptiveness of normals for point cloud segmentation. Compared to prior work, ³⁸ we do not aim to precisely cluster groups into perceptually meaningful regions. The reason is that the strategy would fail in practice for structures like retaining wall C (Figures 2 and 7). The elements are thin, there is heavy vegetation, and with a wall height of 16 m, there are also differences in the point density between the top and bottom of the structure. Therefore, we instead apply a generous angle threshold of 5°–10° to obtain over-segmented point sets $P_{\mathrm{seg}}$ . As a result, these subsets extend over many concrete elements. Nonetheless, we establish regions with similar properties and thus gain a distinction between vegetation, ground, structure, and other objects.

Next, we spatially cluster the segmented point cloud $P_{\mathrm{seg}}$ . We seek small, spatially connected subsets in the over-segmented regions for which we can assume rigid-body behavior. Given the dimensions of the point cloud, we create a three-dimensional grid with equal-sized volumes, commonly referred to as voxels. We exploit the centers of the voxels for clustering. All points of the segmented cloud closest to the centroid belong to the same cluster. In this way, we obtain small surface patches $P_{\mathrm{cluster}}$ that conform to the object boundaries and comply with the assumption of rigid-body motion (Figure 7, bottom right). Note that without the region growing segmentation, voxels would contain mixed points of vegetation and structure (Figure 7, bottom left) and thus falsify the estimated deformations. The choice of the voxel size depends on the surface characteristics and the structural deformation pattern.

Step 2: Estimating transformation parameters

Given a clustered reference point cloud $P_{\mathrm{cluster}}$ of a concrete element (Figure 8, left), the next is to find corresponding points in the query cloud $Q$ . The popular ICP algorithm establishes correspondences by searching for the closest point in ${ℝ}^3$ (black arrows in Figure 8, center). Without any information on the scanned surface, the closest point might not be the perceptually corresponding point. Figure 8 center illustrates such a case, where the nearest neighbor at the edge of the concrete element is associated with one at the vertical face. Consequently, such erroneous correspondences also affect the transformation parameters. One can think of the subsequent optimization process as rotating and shifting the query cloud so that the difference vectors between the corresponding points (black arrows in Figure 8) are minimized as best as possible (in a least-squares sense).

Figure 8.

Deformation modeling of a concrete element of retaining wall C: clustered point cloud (left) and visualization of the working principle using Iterative-Closest-Point (ICP) (middle) and our algorithm (right).

Our method for estimating transformation parameters works differently. Like the M3C2 algorithm, we search for the corresponding points within a cylinder whose axis is aligned with the surface normal at this point (dashed line in Figure 8, right). Note that surface normals ${\mathbf{n}}_{\mathbf{i}}$ are already available as a side product after region growing (see previous section). This procedure can be considered as a local modeling of the surface, which helps to minimize false correspondences. The key difference, however, is that our method minimizes orthogonal distance (black arrows in Figure 8, right) rather than Euclidean distance. The idea is motivated by following facts:

1. The point sampling has less influence on the orthogonal distances than on the 3D displacements.

2. Much of the transport infrastructure consists of flat surfaces. Typical deformation patterns occur in the lateral direction or cause structural tilting that is measurable perpendicular to these surfaces.

For each point ${\mathbf{p}}_{\mathbf{i}}\in P_{\mathrm{cluster}}$ , we define the local tangent plane using the Hesse normal form with $d_i=〈{\mathbf{p}}_{\mathbf{i}},{\mathbf{n}}_{\mathbf{i}}〉$ being the distance to the coordinate origin. By optimizing

$$\underset{\mathbf{t},q}{min}〈{\mathbf{n}}_{\mathbf{i}},(L_q({\mathbf{q}}_{\mathbf{i}})+\mathbf{t})〉-d_i,$$

(3)

we realize a minimization of orthogonal distances between ${\mathbf{p}}_{\mathbf{i}}$ and ${\mathbf{q}}_{\mathbf{i}}$ by rotating and translating the query cluster $Q_{\mathrm{cluster}}$ . It should be noted that $q$ refers to a unit quaternion with the associated rotation operator $L_q$ , while bold ${\mathbf{q}}_{\mathbf{i}}$ is a point of the query cloud $Q_{\mathrm{cluster}}$ . Note that the notation stems from Kalenjuk and Lienhart, ⁵² where this approach is described in detail. Based on the estimated translation, we compute the lateral displacement as $\delta =〈\overline{\mathbf{n}},\mathbf{t}〉$ . $\overline{\mathbf{n}}$ is the normal vector that represents the orientation of the entire cluster $P_{\mathrm{cluster}}$ .

In summary, our method for deformation modeling incorporates concepts from M3C2 and the ICP algorithm. The basic idea is to compute transformation parameters between two point subsets, just as the ICP algorithm does. However, our algorithm differs in two aspects. First, we identify corresponding points as the closest only if they lie within a cylinder whose axis coincides with the surface normals. This idea originates from the M3C2 algorithm, which searches for corresponding points orthogonal to the scanned surface. The second difference is that we minimize these orthogonal distances instead of the Euclidean distance. The benefit of doing so is less sensitivity to varying point sampling and in-plane movements. As a result, we obtain translation and rotation parameters in 3D space.

Stochastic modeling

So far, we have described the first three steps of both phases, the quality evaluation, and monitoring phase (see Figure 4). We successfully aligned point clouds of the left and the right scanner and co-registered the mobile laser scans of two passages. Moreover, an automatic procedure for modeling 3D deformations is at our fingertips. Now all is prepared for the derivation of uncertainty in MLS. Of the three systematic sources of error in MLS (see “State-of-the art in laser scanning and deformation monitoring” section), the proposed preprocessing steps eliminate the geo-referencing and calibration component. These two are specific to MLS and dominate the overall error budget of point clouds.

After their elimination, we can focus on the remaining source of error, namely the scanning process itself. The strategy is as follows. In the course of data collection, scan a retaining structure three times minimum. Compute the lateral displacements $\delta$ for each of the $\left(\begin{array}{c}n\hfill \\ \multicolumn{1}{c}{2}\end{array}\right)=\left(\begin{array}{c}3\hfill \\ \multicolumn{1}{c}{2}\end{array}\right)=3$ clustered point cloud pairs $P_{\mathrm{cluster}}$ and $Q_{\mathrm{cluster}}$ . Since $E\{\delta \}=0$ , we can empirically determine the RMSD as

$$\mathrm{RMSD}\left(P_{\mathrm{cluster}}\right)=\sqrt{\frac{1}{n}\sum _{i=1}^n{\delta }_i^2},$$

(4)

with $n\ge 3$ mobile laser scans. RMSD depends on the cluster $P_{\mathrm{cluster}}$ and thus also on its location. If we plot RMSD for each cluster, we obtain a “precision map” (see Figure 9, top left). According to the plotted RMSD, the precision is variable at different spots of the retaining wall. However, it is not random but tends to get worse at guardrails, vegetation, or regions that are more distant from the vehicles trajectory. The pattern is plausible as the literature provides numerous studies on the correlation of scanning precision, surface properties, and acquisition geometry.^43–45 Yet, it is noteworthy that these effects are so pronounced in the empirical precision map.

Figure 9.

The precision map correlates with surface properties and the acquisition geometry. Visualization of the key parameters for retaining wall A: empirical root-mean-square deviation (top left), surface curvature $c$ (bottom left), scanner-to-object distance $d$ (top right), and angle of incidence $\phi$ (bottom right).

So far, researchers have investigated the factors influencing laser scanning in elaborate experiments under controlled conditions. Here, we show how the influencing factors (see Figure 1) can be also derived directly from MLS data. The requirement is the trajectory, that is, position ( $\mathbf{t}$ ) and orientation ( $\alpha ,\beta ,\gamma$ ) in space, with high frequency and a precise time stamp. Another requirement is that point clouds have timing information after export. By linear interpolation of the trajectory, we obtain the cars position and orientation at the time of sampling of each point. Thus, the trajectory frequency increases from 200 Hz to 1 MHz.

We adopt the shorthand notation of the geo-referencing equation (1)

$$\mathbf{p}=\mathbf{t}+\mathbf{R}(r,p,y)(\mathbf{\Delta }+{\mathbf{p}}_{\mathrm{scanner}})$$

(5)

and rearrange it to

$${\mathbf{p}}_{\mathrm{scanner}}=\mathbf{R}{(r,p,y)}^T(\mathbf{p}-\mathbf{t})-\mathbf{\Delta }.$$

(6)

${\mathbf{p}}_{\mathrm{scanner}}$ corresponds to the point $\mathbf{p}$ whose reference frame is moved into the laser scanner’s center. Knowing the lever arms $\mathbf{\Delta }$ from calibration is advantageous to precisely reconstruct the line-of-sight during the point scanning. At greater distances, the parallax will have a negligible effect. We derive the scanner-to-object distance $d$ (Figure 9, top right)

$$d=\Vert {\mathbf{p}}_{\mathrm{scanner}}{\Vert }_2,$$

(7)

and the incidence angle $\phi$ (Figure 9, bottom right) using the computed surface normals $\mathbf{n}$

$$\phi =\arccos \frac{〈-{\mathbf{p}}_{\mathrm{scanner}},\mathbf{n}〉}{d}.$$

(8)

Principal component analysis is the mathematical basis for calculating surface normals, which provide valuable byproducts for another crucial point cloud feature. The ratio between the smallest eigenvalue and the sum of eigenvalues describes the variation of the surface. This measure is also commonly referred to as surface curvature $c$ ⁵³ (Figure 9, bottom left)

$$c=\frac{{\lambda }_2}{{\lambda }_0+{\lambda }_1+{\lambda }_2},$$

(9)

whereas ${\lambda }_0\ge {\lambda }_1\ge {\lambda }_2$ . The curvature describes the variability of the surface. One can thus consider it as a measure for surface roughness.

Equations (7) to (9) provide the formulae to compute three features for each individual point. Similar to the idea of Extended Gaussian Images for visualizing surface normals, ⁵⁴ we interpret the three features as point coordinates in ${\mathbb{R}}^3$ and plot them as a point cloud (Figure 10, left). While point clouds are less intuitive to interpret in feature space, they offer interesting insights into the correlation of point features. With prior knowledge of the scene, we can identify the curb (A, blue) and the guardrails (C, yellow) as objects with increased surface curvature $c$ (i.e., roughness) and that road surface (B, orange) is sampled at larger angles of incidence $\phi$ than the retaining wall (D, purple) at the same distance. The point clouds are not randomly distributed in feature space but show a clear pattern, that is, there is a correlation between the features. Figure 10 right shows the correlation heat map between the point features and the RMSD. In case of the retaining wall A, Figure 9 shows that the distance and the angle of incidence are correlated with $\mathrm{corr}(d,\phi )=0.58$ .

Figure 10.

Correlation of point features and the root-mean-square deviation (RMSD) for retaining wall A: 3D visualization of the features as point cloud with specific areas highlighted with A-D (left) and the correlation heat map of features and RMSD (right).

The correlation between point features and RMSD is much lower. So the question is whether we can use these features to build a model that estimates RMSD for scenes that have only been scanned once. As the three features and the target variable appear to have a non-linear relationship, we aim for a non-parametric model. Among the supervised machine learning algorithms, the random forest (RF) regression algorithm promises good predictive performance with little hyperparameter tuning. ⁵⁵ Nevertheless, before feeding the RF with data, we first preprocess the feature cloud. Let us consider a cluster $P_{\mathrm{cluster}}$ with 1000 points. We can compute features for each, thus providing us 1000 values of $d,\phi ,c$ . However, we get one only RMSD per cluster (see equation (4)). Consequently, we average the 1000 data points of each feature to obtain a representative triplet of $\overline{d},\overline{\phi },\overline{c}$ . This step is advantageous for several reasons. First, it reduces the data amount by a factor of 1000 and, thus, the time required to train the model considerably. It also reduces noise, resulting in more representative features, which in turn, result in a model that is less susceptible to overfitting. We train a standard RF regression model using bootstrapped samples to generate an ensemble of 100 decision trees. The outcome of the decision trees is aggregated to achieve more stable and accurate predictions. ⁵⁵

Intuitively, the trained RF model provides rules on partitioning the feature space $d,\phi ,c$ to obtain the average RMSD that has been observed during training. Moreover, the grown tree provides measures on the importance of the features. ⁵⁵ According to our model, the curvature $c$ is the most important feature indicated by 80% relevance, followed by the distance (16%) and the incidence angle (4%). It is noteworthy that the distance correlates less with RMSD than $\phi$ does (see correlation heat map in Figure 10) but still proves more relevant for the RF model. This fact may be due to the non-linearity of the model. These measures may be different when training the RF model with data from another wall or a different setting, though. We will evaluate the predicted precision maps for the three retaining walls A, B, C in the “Evaluation in practical case studies” section and in the Appendix using the RF model of retaining wall A.

Summary—requirements for data collection and processing

We propose a method for rigorous deformation analysis of MLS data. It is not state-of-the-art and requires multiple processing steps that can be automated. To sum up, following requirements apply for the data collection or delivery, respectively:

• Point coordinates with time stamp: $X,Y,Z,t$

• Point clouds separated by left and right scanner

• Information from geotechnical or civil engineers on stable areas

• Three collected data sets if nothing is expected to be stable near the structure

• Uncertainty information as

• ○ a valid stochastic model for the MLS system and the scene or

• ○ three point clouds of one epoch to determine the precision map empirically for the MLS system and the scene

• Trajectory data exported with the highest possible frequency and including timing information: $t_x,t_y,t_z,r,p,y,t$ to build a stochastic model

• The lever arm $\mathbf{\Delta }$ , that is, the three-dimensional offset of scanner to IMU.

The method requires following considerations regarding data processing:

• Merging data without checking the co-registration of left and right scanner will introduce additional uncertainty

• MLS point clouds suffer from time-variable errors

• The proposed method performs a profile-based co-registration that accounts for these time-dependent errors

• A prerequisite is that road surface and inventory are stable near the structure in-between the data collections

• If nothing is expected to be stable due to, for example, construction works, one can obtain a stable reference frame by averaging multiple data collections of one epoch, albeit with lower accuracy

• The computationally most intense processing step is calculation of surface normals

• A complex structural deformation pattern is approximable by a composite of rigid-body motions of many small surface patches

• Focusing on the deformations perpendicular to the structure’s surface $\delta$ (road, retaining wall, tunnel, etc.) is reasonable in most scenarios

• Multiple data collections of one epoch allow deriving the empirical RMSD

• If a stochastic model is available from previous campaigns, one can predict the precision map, a color-coded point cloud with RMSDs using point cloud features

• The time-tagged trajectory and point cloud, enables us to compute two relevant point cloud features for each point: distance and angle of incidence

• The third feature is surface curvature (or roughness), described by the ratio of eigenvalues.

Finally, we can formulate the null hypothesis $H_0:E\{{\delta }_i\}=0$ and alternative hypothesis $H_{\mathrm{a}}:E\{{\delta }_i\}\ne 0$ for every voxel. We rearrange the $t$ -distributed test statistics ³⁸ to obtain the thresholds for testing the deformations for significance with

$$\mathrm{significant}\left({\delta }_i\right)=\{\begin{array}{cc}0\hfill & \left|{\delta }_i\right|<t_{f,1-\alpha /2}·\mathrm{RMS}{\mathrm{D}}_i\hfill \\ \multicolumn{1}{c}{1}& \left|{\delta }_i\right|\ge t_{f,1-\alpha /2}·\mathrm{RMS}{\mathrm{D}}_i\hfill \end{array}$$

(10)

The choice of $\alpha =5\%$ as significance level (two-sided test) is motivated by standards in deformation monitoring. ⁵⁶

Evaluation in practical case studies

We present the application of our method to three practical use cases. Retaining structures A, B, and C provide examples of the feasibility of rigorous deformation analysis of MLS data and the associated challenges. The selected three differ in surface properties, structural type, and, thus, the deformation behavior. In this section, we, therefore, focus on three key aspects of the method:

• The complete workflow: We show the results from intermediate steps and evaluate the accuracy using geodetic surveys for retaining structure A.

• The efficiency: Building upon the results from the previous example, we highlight how to increase efficiency further for structure B by using a stochastic model.

• The large-scale deformation case: While most structures show little to no deformations, retaining structure C did experience extensive deformations, including its surrounding. Such a case is rare and poses challenges to processing MLS data.

Retaining structure A—evaluating the complete workflow

Retaining structure A is an ideal testbed. With its 18 m height, it stabilizes a slope next to a state road that is less heavily trafficked than the highway next to it. However, failure would affect both traffic routes, making the maintenance of the structure of great importance. Although such failures are seldom, an incident on an Austrian Highway in 2012 ⁵⁷ raised the operator’s awareness. The last inspection attested retaining wall A to be in a poor structural condition. The main reason was the concrete surface, which spalls and poses a danger to pedestrians using the nearby sidewalk. Another problem was the old anchors, originally not designed for detailed inspections. Their exact condition and, thus, the expected total service life of the structure were unknown. Parallel to ongoing geodetic monitoring, we conducted total station surveys from 3 setup stations to 19 prisms (see Figure 3) on our own. Hence, the classical rigorous deformation analysis served as a benchmark for our proposed, novel approach. The MLS campaigns took place on November 27, 2019 and November 10, 2020. During road closures, we drove by the structure four times, twice in each direction at about 40 km/h.

Precision map

Using the four MLS point clouds, we computed the RMSDs for each cluster (see equation (4)). Figure 11 top depicts the distribution of the RMSDs in the empirical precision map. The bottom visualization of Figure 11 shows the modeled precision map. It represents the prediction of the RF regressor trained on the three features, $d$ , $\phi$ , and $c$ for retaining wall A. The modeled RMSDs are plausible and the two precision maps show a similar pattern. At the top of the structure, where the angle of incidence reaches 79°, the RMSD is at the maximum with $\pm 5.4$  mm. For the rest of the wall, both indicate consistent RMSDs between adjacent voxels in the lower millimeter range on the structure. The differences near the anchors or the curbs are due to our preprocessing steps that simplify the model. We could train an RF model to reproduce the observed precision map more accurately, but we would then sacrifice generalizability. According to Figure 11, the model tends to provide too pessimistic uncertainty predictions. Looking at the histogram of the RMSD differences (model—empirical) in Figure 12, we notice a positive 1–2 mm shift in the distribution, confirming our perception. A slight accumulation is visible at 11–13 mm, reflecting the regions of high surface curvature but low empirical RMSD. Remember that our RF model attributes high importance to curvature (see Section “Method”). While predictions are accurate for the retaining structure, deviations increase for regions of high curvature, that is, high surface roughness. The trees and vegetation above the wall would all be red in Figure 11 (RMSD > 10 mm), so we excluded them from the visualization.

Figure 11.

Precision of MLS monitoring retaining structure A: empirical (top) and modeled (bottom) precision map.

Figure 12.

Histogram of differences of modeled and observed root-mean-square deviations for retaining wall A.

Deformation and significance map

Following the assessment of geotechnical engineers, we considered road surface and infrastructure inventory stable for co-registering MLS cloud from 2019 and 2020. The deformation map obtained from the two MLS surveys is visible in Figure 13, left.

Figure 13.

MLS-based rigorous deformation analysis of retaining wall A for the period between November 2019 and November 2020: lateral displacements with locations of geodetic reference targets (left) and binary plot indicating significant displacements (dark gray, right).

The color bar is limited to ±5 mm, and yet most parts of the plot are green. However, the red pattern at the top of the wall indicates positive lateral displacements of approximately +5 mm. The direction of displacements follows the convention defined by the reference frame for geodetic monitoring. In this case, positive deformations mean displacements toward the slope, that is, away from the road and highway. We have observed seasonal variations of this magnitude with total stations, but the variations shown are due to uncertainties in laser scanning. We can draw this conclusion with the precision map at our fingertips (Figure 11). Using either the empirical or modeled map, we can interpret the point cloud discrepancies and classify them as significant or insignificant. The binary significance map (as visualization of equation (10)) in Figure 13 (right) identifies most of the structure as not displaced between 2019 and 2020. Individual clusters flagged significant may be considered outliers.

Benchmarking MLS using geodetic surveys

The black dots and numbers in Figure 13 (left) visualize the location of the 19 prisms. For each of them, we plot the lateral displacements obtained from conventional and MLS monitoring in Figure 14. The graph additionally shows the confidence intervals for both methods. Since we conducted geodetic surveys from three setup stations (see Figure 3), we derived the 95% confidence of the lateral displacements. Thanks to our precision map, we can plot the 95% confidence region for MLS by scaling the precision map by $t_{f,0.975}$ (equation (10)). All points whose confidence bands (light gray) intersect with the zero displacement line (solid black) can be considered as not significantly displaced. Accordingly, the MLS approach showed that only point 2071 moved. By contrast, geodetic monitoring identified half the points (2001–2053) as moved. However, if we overlay the confidence intervals of both methods, they overlap at each point. The MLS monitoring results conform to geodetic surveys in the range of the measurement uncertainty.

Figure 14.

Rigorous deformation analysis of structure A at the location of the 19 prisms obtained by static, geodetic moni-toring (top) and MLS monitoring (bottom); deformations are significant for points whose 95% confidence bands do not overlap with the zero displacement line.

To sum up, the MLS approach enables monitoring the spalling, rough concrete surface of structure A with a precision of better than 5.4 mm. The established stochastic model predicts RMSDs at the structure that conform in the millimeter range to empirical values. The absolute displacement values of the 19 points agree with geodetic surveys within 5 mm.

Retaining structure B—increasing efficiency

Retaining structure B is representative of the many anchored structures of medium height in the Austrian highway network. ³⁸ It consists of several anchored concrete elements of 5.4 m × 5.4 m size with a relatively smooth surface. A local surveyor performs geodetic monitoring of 19 retro-reflective points twice a year. During these measurements, the road administration closes one lane of the busy highway. For safety and efficiency reasons, we thus consider the deliverables from the external surveyor as our benchmark. The dates of the geodetic and MLS surveys (November 27, 2019 and November 10, 2020) roughly coincide at both epochs.

Time savings by reducing amount of data

To prevent any significant impediment to the traffic flow, we drove past the structures at the minimum permissible speed of 80 km/h. So we moved twice as fast as when scanning structure A, completing the point cloud acquisition after 7 s. A single pass with such a short data collection is sufficient to apply our workflow for deformation monitoring. The underlying idea is to predict the precision map of structure B based on the learned stochastic model from structure A. The transfer of the learned relations between the features and the RMSDs from A to B (and C), works well, as we have shown in the Section “Evaluation of the stochastic model” in the Appendix. By reducing the number of scans needed, we can spare time in collecting data and processing the precision maps.

Maintaining the workflow

The rest of the workflow is the same as for structure A. We first derive the deformation map (Figure 15, top left). Again, we received the information that, except for the structure, everything else should not have moved between the two epochs. Accordingly, we established the reference frame for comparison by co-registering the point clouds over stable regions. Then, we incorporated the modeled precision map (Figure 15, bottom left) to statistically test the deformations (lateral displacements) for significance (Figure 15, right).

Figure 15.

The three key components of rigorous deformation analysis of structure B using mobile laser scanning (MLS) data: deformation map (top left), the modeled precision map (bottom left), and the binary significance map (right).

Compared to structure A, the acquisition geometry is more favorable for structure B. Note the larger scanner-to-object distance at a highway and the lower height (Figure 2). The wall points thus result from scanning at roughly equal distances and angles of incidence. As a result, the RMSDs do not differ considerably between the toe and top of structure B (see precision map in Figure 15). Regions of high surface curvature, including joints and vegetation, are clearly evident in the modeled precision map.

The deformation map (Figure 15, top left) shows a specific pattern for the highway and the structure (same colorbar limits as for structure A). One could spot local settlements at the pavement or displacements at the top of the wall, but we can classify them as insignificant after statistical testing.

Maintaining accuracy and completeness

We evaluated the MLS approach based on the external geodetic surveys of the 19 prisms visualized as block dots in Figure 15 (top left). Since there were no uncertainty values stated in the monitoring report, we assumed a constant precision of ±1.5 mm for all points. According to the statistical test, the geodetic surveys revealed significant displacements for point 9.0 only (Figure 16, top), whereas MLS flagged all points as unmoved (see also significance map in Figure 15). Except for the mentioned point 9.0, MLS and geodetic surveys provide lateral displacements that deviate less than ±5 mm.

Figure 16.

Rigorous deformation analysis of structure B at the location of the 19 prisms obtained by static, geodetic monitoring (top) and mobile laser scanning (MLS) monitoring (bottom); deformations are significant for points whose 95% confidence bands do not overlap with the zero displacement line.

One interesting observation is that data points are missing for both methods. According to the monitoring report, the bi-reflective targets of points 7.0, 8.0, and 8.1 were not available at epoch 2. They must have broken down sometime in the summer or fall of 2020, so the trace of their displacements was lost. MLS provides complete field information and does not rely on targets and sensors installed on the surface. However, a weakness of laser scanning becomes evident at points 7.2 and 8.2. These two points are located above the wall where shrubs obstruct the line-of-sight to them (Figure 15, top left). With dense vegetation in front of the wall (as is often the case for crib walls), quality decreases drastically, and may result in no displacement values being determined at all.

Although structure B defines a typical example among thousands of objects in the Austrian highway network, the use case provided additional insights:

• The proposed procedure does work with one point cloud per epoch only. It is not necessary to pass by multiple times and derive precision maps empirically. We can model it based on point cloud features.

• The driving speed has little to no effect on the accuracy of the lateral displacements.

• The precision of MLS-based monitoring depends on surface smoothness and acquisition geometry.

• Geodetic monitoring can lose track of displacements when optical targets break and are no longer recoverable. MLS provides deformations independent of any installations, but it may require time-consuming pruning operations if there is heavy vegetation (especially for crib walls as structure C).

Retaining structure C—the large-scale deformation case

Retaining wall C is located along an Austrian highway. It is an anchored crib wall of 330 m in length and 14 m in height. The structure consists of many precast reinforced concrete elements, each 3.4 m long and 0.3 m high. The composite of soil and anchored parts provides the supporting effect. Hence, trees and shrubs grow between the artificial elements and cover the front of the wall. Crib walls thus proved to be more challenging than anchored structures with concrete surfaces, for exmaples, structures A or B. The potential deformation behavior is also more complex, as movements can occur very locally. Some elements may rigidly shift if an anchor fails, or individual components may break due to their slender design if additional loads arise.

Background information

In our analyses, we divide structure C into two parts: C1 and C2 (Figure 2). The reason is that C1, with its 150 m, and C2, with its 180 m length, behave differently. Currently, a new tunnel with two tubes is being built, crossing under the highway and the retaining structure at a depth of 100 m (see white lines in Figure 17). As a result of the tunnel construction, terrain settlement has been observed in the surrounding area. To better assess the structural response, engineers installed total stations to permanently monitor the deformations at 94 prisms at hourly intervals during construction for a period of more than 1.5 years. We received the measurement data for evaluation at the exact time of our MLS surveys on November 27, 2019 and April 12, 2022. The observed lateral displacements (white arrows in Figure 17) show increased deformations for C2, while C1 remains stable. The west tube defines the transition zone between the two structures, C1 and C2, and the moving and stable zones.

Figure 17.

Orthophoto of the scene around retaining wall C: the arrows indicate the displacement vectors obtained from geodetic monitoring between the epochs December 2019 and April 2022; the white solid lines represent the current state of construction of the tunnel tubes as of April 2022.

Workflow adaptation

With the prior knowledge about the structure’s behavior, we could not proceed as with retaining wall A and B. The problem is that the profile-based co-registration method would compensate for the local settlements, when presuming stable surfaces in the point clouds. This approach would thus incorrectly suggest no settlements in the area of C2.

In such a case (see Scenario II in Section “Reference frame realization”), we acquire multiple datasets and compute a quality-enhanced point cloud per epoch. The proposed method averages the collected point clouds and thus minimizes low and high-frequent systematic trajectory errors. This strategy proves to be particularly feasible when the expected deformations are in the centimeter range. The remainder of the workflow is the same as for retaining wall A or B, as Figure 18 shows. We predict the precision map (2) based the stochastic model from structure A to test the deformation map (1) for significance (3).

Figure 18.

The three key components of rigorous deformation analysis of structure C using mobile laser scanning (MLS) data of November 2019 and April 2022: deformation map (top), the modeled precision map (middle), and the binary significance map (bottom).

Benchmarking MLS using total station data

At first glance, the deformation map resembles the behavior observed by the total station measurements (c.f. Figures 17 and 18). C1 shows almost no movements, while the comparison of the point clouds indicates extensive deformations for structure C2 and the road next to it. At the pavement, the two epochs differ by a maximum of −90 mm, roughly corresponding to the −80 mm settlements reported by geodetic monitoring. The compliance is remarkable, considering that the two quality-enhanced point clouds were not registered to each other. According to the statistical tests, C2 deformed, and the nearby road settled significantly (see significance map in Figure 18, bottom).

Figure 19 depicts the lateral displacements of the 94 prisms observed with geodetic measurements and MLS between the epochs November 2019 and April 2022. The prisms are numbered in direction from C1 to C2. For structure C1 (right of dashed line), lateral displacements are below 5 mm and thus at the limit of being flagged significant. At the transition zone between C1 and C2 (dashed line), where the west tube crosses the structure, the point movements increase up to 30 mm. The rest of the points at structure C2 indicates significant displacements of +20–+30 mm toward the highway. Note that we again assumed a precision of 1.5 mm for the statistic testing of geodetic measurements.

Figure 19.

Comparison of mobile laser scanning (MLS)-based (gray) and static, geodetic monitoring (black) at the locations of the target prisms at structure B; the bars indicate the 95% confidence levels.

The results of the MLS approach reflect the deformation pattern described (Figure 19, bottom). The confidence intervals of the C1 points overlap the zero displacement line, meaning that they have not moved statistically. By contrast, the rigorous deformation analysis of the MLS data attests 60 out of 63 points (corresponding to 95%) at C2 having displaced significantly. The magnitude of displacements ranges from +20 to +30 mm. Accordingly, the two methods agree better than ±10 mm for most points and as the confidence bands intersect (if overlayed), the differences between the MLS and geodetic monitoring are insignificant. It is worth mentioning that the precision of the MLS approach is worse for the thin concrete elements partly covered by vegetation (compared to structures A or B), so the confidence intervals are larger (e.g., ±15 mm).

Retaining structure C proved to be a valuable case study. Most of the infrastructure objects investigated showed little to no movements. Based on structure C, we could show the capability of MLS for detecting large-scale deformations. The obtained results are remarkable, since 95% of the displaced points could be identified as such. Albeit a challenging object surface with vegetation, the achieved accuracy is better than 10 mm. Note that for the evaluation at the position of the 94 prisms, we averaged values from adjacent voxels to obtain the lateral displacements at this point. Due to the wider confidence bands (compare Figure 19, top and bottom), MLS will not replace the more precise geodetic monitoring in all situations. The latter will remain the monitoring technique of choice in safety-critical situations. However, MLS has proven to provide accurate and unbiased results. It offers great potential for large-scale application in efficiently assessing the condition of many objects in transport infrastructure.

Conclusion

In this article, we have proposed a method for monitoring the structural deformations of retaining structures in a transport network using MLS. The idea is to leverage the fast data collection capability, apply efficient, automated point cloud processing, and thus establish a working procedure that scales to many objects. Remarkably, it is not limited to specific hardware but is compatible with commercially available MLS systems from four major manufacturers. We defined the requirements regarding data collection, delivery, and processing. Accordingly, following the provided guidelines, one can contract external service providers for data collection and will still obtain comparable datasets.

The proposed processing pipeline follows two basic concepts. First, remove systematic deviations, including geo-referencing and calibration errors. Second, determine the remaining discrepancies through multiple passes empirically. By incorporating the uncertainty into the deformation monitoring, we can test the modeled deformations for significance, conforming to the principle of rigorous deformation analysis. We also showed a way to circumvent the need for repeatedly scanning an object at each epoch. It is possible to predict the uncertainty of a point cloud by a supervised stochastic model based on the three main features: scanner-to-object distance, angle of incidence, and surface roughness.

We presented three case studies in which geodetic surveys served as benchmark. In the case of a more or less smooth object surface, the precision and accuracy proved better than 5 mm. An anchored crib wall that experienced large deformations of almost 100 mm turned out challenging for modeling deformation from the MLS point clouds. While the accuracy achieved is lower compared to the other two objects, our method successfully identified the same regions as moved as the geodetic monitoring measurements. This result is remarkable and highlights the potential of MLS for infrastructure maintenance.

In safety-critical cases, MLS will not replace other, high-accurate measurement techniques in SHM. However, we presented a time-efficient working procedure to inspect many structures and identify potentially defective ones in road and railway networks. We intend to give the inspecting engineer information at hand to assess the structure’s health condition more objectively.