Keywords:
Digital Twin, deep learning, 3D photogrammetry, damage inspection,
artificial intelligence, image segmentation
Introduction
The world is currently in the fourth industrial revolution, this
revolution has been taking place since the turn of the millennium.
Industry 4.0 combines the physical world with the digital and virtual
worlds. This can lead to more efficient production, smarter products,
and more efficient use of resources. The Digital Twin is an important
element of Industry 4.0. Digital Twins are often used together with
Internet of Things (IoT) and Artificial Intelligence (AI) is frequently
an integrated part of the data processing section. AI can here be both
machine learning and deep learning. The application of Digital Twins has
been reported in several different areas including healthcare [1]
and [2], smart cities [3] and [4] and manufacturing, see the
review paper [5]. The latter is the area where most Digital Twin
applications have been reported. There are many examples of small-scale
Digital Twins but a lack of very large-scale Digital Twin projects in
the literature. The reason is a lack of specific domain knowledge of how
successful upscaling is done.
References [6] and [7] discuss the application of Digital Twins
in the wind power industry. A cloud-based Digital Twin monitoring and
analysis system is discussed in [7]. A working prototype Digital
Twin of a wind farm is presented where data are fed to the model and
both technical and business parameters are generated for the wind farm.
This can be used to evaluate the wind farm both technically and
economically. A case study for wind farm use and smart grid energy
consumption is discussed in [6].
A key enabling technology in the wind energy industry and many other
industries is digitalization. Sensors placed on wind turbines will
generate large amounts of data that when combined with other digital
technologies such as Big Data, Internet of Things, and Cloud Computing
will open up many new possibilities. The data from sensors and wind
turbine inspection measurements can be combined in a Digital Twin. Many
different definitions of a Digital Twin are used in the literature but
we will use the definition of Bolton et al. [8], where a Digital
Twin is a ‘dynamic virtual representation of a physical object or system
across its lifecycle, using real-time data to enable understanding,
learning and reasoning’. A Digital Twin can be both data-driven and
based on physical models. Sensor data from the operating access for
example a wind turbine are used to update the state of the Digital Twin.
Advanced numerical tools must be further developed to better understand
sensor signals and inspection data and simulate the consequence of
structural and material deterioration. Some examples can be found in
[9, 10]. The structural performance of each component of the wind
turbine running in the field can be evaluated using its counterpart
Digital Twin in a control center.
The Digital Twin for the wind turbine can be used as a prognostic health
management system. This will lead to a change in the maintenance
strategies from fixed schedules and intervals towards predictive
maintenance. The Digital Twin will make it possible to economically find
the balance between wind turbine utilization and lifetime reserve and
maintenance.
Currently, wind turbines and their components are often inspected
manually using lifts, and sometimes it is needed that a person climbs
onto the structure. This type of inspection is expensive,
time-consuming, and can potentially also be dangerous. Drones can be
used to collect a huge amount of images. The drone can be programmed to
autonomously follow a predetermined route and acquire images at specific
waypoints. This approach significantly reduces the time needed for the
inspection. It is very time-consuming for a person and sometimes also
inaccurate to go through massive amounts of images and make reports
where for example the defects found in the images are documented.
Combining AI with the Digital Twin concept produces a more efficient
approach. Deep learning can find the defects in the images in near
real-time and is therefore much faster than a human. Defects from
multiple images can be categorized by AI and the 3D location can be
calculated and mapped onto the Digital Twin. One specific defect that is
observed in several images can then be seen on the Digital Twin from
different angles and distances. AI is also cable of finding defects too
small to be observed by humans. Reference [11] provides an example
of wind turbine surface damage detection using AI where drone inspection
was used.
This study builds on the findings reported in [12], in which a
transition piece was selected as an example of a large-scale structure.
In the current study, both a transition piece and a rotor blade are used
to demonstrate newly developed functionalities. As such, novel
contributions of this study to the current knowledge base are:
- Image segmentation to isolate the structure of interest from complex
image background. In the previous study [12], AI can occasionally
find damage in the images on distant structures in the background and
report the damages that do not belong to the structure of interest.
These damages should not be mapped to the Digital Twin of the
concerned structure in the center of the images.
- Real geometric features of defects/damages mapped to the Digital Twin.
To achieve this, image segmentation is performed on the small section
of the total image that is inside the bounding box calculated by deep
learning algorithm. The pixels from the image segmentation,
representing realistic damage shape and size, are mapped to the
structure.
- A more precise new algorithm using surface normals for mapping the
damage pixels is presented in this study, enabling realistic
representation of unique defects/damage of the structure.
This paper is organized as follows. Section 2 presents the visual
Digital Twin and how it has been used in the study. Section 3 describes
in detail the methodology of the visual Digital Twin. This includes
algorithms for pre-processing drone images and algorithms that map
damages found in images to a 3D model of a structure. The image
pre-processing and 2D to 3D damage mapping techniques are applied in
section 4 which demonstrates the techniques on a set of images from one
drone flight using a transition piece as demonstration. Section 5 apply
the mapping algorithms to a composite wind turbine blade where
subsurface delamination damage is visually inspected. The main
conclusions of the study are summarized in section 6.
The visual Digital Twin
The arrival of powerful and cost-effective drones has opened up many new
applications. Drone inspection of very large structures is an example of
a new application type that gives better results and is more
cost-effective than previously used methods, see references [13] and
[14]. Wind turbine transition pieces are currently inspected at the
factory for different types of damages. This is accomplished by an
inspector from a crane. The objectives of this study are to develop and
demonstrate fully automatic intelligent drone inspections based on RGB
images and to find and recognize paint damages and defects of wind
turbine transition pieces TPs, see [15] and [16]. The starting
point is a description of the physical position of the TP in a
georeferenced coordinate system. A detailed meshed CAD model of the TP
is moved to the same georeferenced coordinate system used during drone
flights. This makes it easier to compare it to the reconstructed 3D
model. An AI algorithm is applied to the RGB images to detect and
classify paint imperfections and damages, respectively. The RGB images
from the drone flight are used to generate a 3D model of the TP. This
reconstruction model is produced with the use of photogrammetry and is
an important part of the digital twin. Information from the AI algorithm
is used in the pre-processing of the images. The 3D Digital Twin is
updated with the positions, types, and sizes of the identified paint
imperfections and damages identified in the images by AI. These
different processing steps are summarized in the flowchart shown in
Figure 1 and explained in detail in the next section. Information from
the Digital Twin is used to update the physical structure. Information
on the position and type of paint damage from the Digital Twin can be
used to determine if maintenance is needed on the physical structure.
After the offshore installation of the TPs, drones can be used to
inspect the transition pieces at regular time intervals. The Digital
Twin will then be updated based on the new images, providing essential
information for asset management. Green boxes are used in the flowchart
to indicate the new algorithms and processing steps compared to the
reference [12]. The image segmentation steps and the 2D to 3D
mapping technique that uses the projection along the surface normals
method are all introduced in this paper. A major difference in the
mapping algorithm is that the paint damage pixels will be mapped to the
tower instead of the bounding box corners which was the case in
reference [12].