Abstract
Digital twins (DTs) are transforming business operations across industries through accurate replication of physical entities using the Internet of Things and big data analytics. Despite booming progress in the manufacturing, aerospace and buildings sectors, the adoption of DTs in the minerals industry has been slow, and integration with efficient visualisation and user interactions has not been fully optimised to achieve maximum fidelity and usability. One promising avenue for enhancing DT capabilities is the utilisation of extended reality (XR) technologies, which also hold great potential for realising an industrial metaverse where real-world business activities can be conducted in a virtual space. This article proposes a cost-effective and scalable approach to developing a DT with real-time monitoring and control capabilities for a ball mill operation, a widely used processing equipment in the minerals industry. The case study showcases two approaches with different levels of system integration by leveraging serious game development platforms, toolkits and workflows.
Keywords
Introduction
Mining in the twenty-first century presents a complex set of challenges. With shifted focus from a booming phase into a steady operational phase in countries across the globe as well as a rising concern in its contribution towards climate change, and the ongoing labour shortage among younger generations at remote locations, mining companies are looking for ways to maximise long-term sustainability. In turn, the minerals industry is investing heavily in digital transformation of the workspace to retain young talents, and improve safety, as well as productivity (Elbazi et al., 2022).
The term ‘digital twin’ (DT) came from the pioneering vision of having a data-driven, digital replica that accurately represents a real-world entity in terms of its attributes and real-time behaviours (Rasheed et al., 2020). The concept was first introduced as an idea for product lifecycle management in 2002 by Dr Michael Grieves who later illustrated a DT as the ‘Mirrored Spaces Model’ in 2005 (Grieves, 2005). In the age of Industry 4.0 powered by the Internet of Things (IoTs), DTs have also demonstrated their values in assisting design, production, maintenance, risk management, as well as process optimisation across numerous industries such as manufacturing and aerospace.
Defined by Kritzinger's publication in 2018, a true DT should have two-way integration of real-time data flows between the digital space or the ‘mirrored space’, and the physical space containing real-world entities as shown in Figure 1. In such configuration, a true DT may also serve as a controlling instance of objects in the real world (Kritzinger et al., 2018).
Classification of data integrations in a two-space analogy (Qu et al., 2023).
Ultimately, advancements in DT research can also contribute towards realising a future of industrial metaverse (Liu et al., 2023; Stothard, 2023) – a hypothetical future iteration of the Internet as a collective, immersive and collaborative virtual space where people can work, entertain, trade as well as other things they could do in life using technologies such as virtual reality (VR), augmented reality (AR) and mixed reality (MR) to provide human-centred experiences which had also been used to deliver effective industrial training (Bise, 1997; Denby et al., 1998; Kerridge et al., 2003; Kizil et al., 2001; Schofield et al., 1994; Squelch, 1997) and human-centred cyber-physical systems (Brune, 2010; Deryabin et al., 2020; Huang et al., 2022; LeBlanc-Smith et al., 1998; Stothard et al., 2019; Xie et al., 2019).
This article provides the details of developing a proof-of-concept DT with real-time monitoring and control capability (Figure 2) on a lab-scale ball mill operation. The case study introduces a cost-effective approach to developing an operation-oriented DT by integrating a legacy control system with VR visualisation, locomotion and interaction techniques in a 3D virtual space powered by Unity – a free modern real-time graphics engine used extensively for serious games and modern XR applications developments (Unity Technologies, 2023).
Real-time DT operation from XR-based human–computer interfaces (Qu et al., 2023).
Developing the DT
Digital modelling
Development of the DT started with creating digital models in the virtual domain. According to Stothard et al. (2019), there are different ways to model a mining system, and each would result in differences in the amount of human overhead, time and quality of the modelling process. The author of this article considered using a LiDAR scanner for rapid point-cloud generation, which would reduce human overhead and modelling time (Perez et al., 2020). However, importing point-cloud models into Unity would require specialised software add-ons that would cost hundreds of dollars (Baggett and McMillan, 2018), and in the context of this project, the simplicity of a proof-of-concept led to the decision to employ a conventional approach to modelling the physical aspects of the mill and its surrounding workspace. The cost benefit of employing point-cloud generation would be justified when modelling large-scale mining systems such as processing plants, buildings and mine sites (Kharroubi et al., 2019). Alternatively, the author considered using the Unreal Engine for its built-in point-cloud supports. However, it was removed from the scope of this study in favour of the streamlined XR development workflows offered by the Unity Engine.
Modelling the ball mill
The physical dimensions of the ball mill were first obtained through existing AutoCAD drawings supplied by the Julius Kruttschnitt Mineral Research Centre (JKMRC). Missing information was filled in from measurements conducted on-site, as shown in Figure 3. From here, a 3D model of the ball mill assembly was built using Autodesk Inventor, with preliminary textures applied to each component and later enhanced with improved textures and details obtained from site measurements that were absent from the original 2D drawings. It is worth noting that such manual approach would only make sense in a small-scale project due to increased amount of human overhead (Stothard et al., 2019). The time required to fill in missing data for a large-scale project where 3D digital documentations are not always present would make point-cloud modelling a compelling approach.
Ball mill CAD modelling in inventor.
The Inventor assembly model (.iam) was then exported as a STEP 3D CAD file (.stp) and further converted to Autodesk's proprietary Filmbox (FBX) format to ensure compatibility with the Unity workflow which involves retexturing, lighting, animation design and VR/MR integration (Ardolino et al., 2014). Figure 4 illustrates the file conversion workflow used for this particular study where Autodesk 3DMax was used as a middleware to export FBX models to Unity with pre-processed meshes and textures.
CAD format conversion for DT modelling workflow.
FBX format is one of many 3D formats that allow developers to individually place, orientate, re-texture and animate components of the model with higher degree of controls which would become extremely helpful when trying to visualise the asset's real-world behaviours with the DT. It is worth noting that depending on the needs, an organisation could opt for alternative approaches if visual fidelity, and VR/MR support are not focused on. For example, the ISO-ratified X3D graphical format supported by the Web3D Consortium offers excellent compatibility with existing industry standards while offering good adaptability thanks to its web-based runtime architecture (Web3D Consortium, 2023). Combining the X3D format and free-to-use animation software such as Blender as a substitute to 3DMax could also dramatically reduce the entry cost for future developments.
Modelling the workspace
For the purpose of developing a proof-of-concept to demonstrate a DT's values in operation monitoring and control, the ball mill was considered the centre piece during the modelling process with a high level of emphasis on replicating its dimensions, details and surface appearances in the virtual domain. However, having the DT placed inside of an empty space would not be sufficient in delivering the sense of presence and spatial awareness for human operators which they develop from real-world experience (Bellanca et al., 2019). Therefore, the effort to present a suitable workspace for ball mill operators in the virtual space was considered an integral part of the DT operation experience.
Modelling of the virtual workspace had gone through multiple iterations throughout the modelling stage of the project. Initially, the idea was to create a point-cloud scan of the actual pilot plant where the ball mill is located at JKMRC to replicate the working environment for the plant operators closely. The idea was later ditched considering the low level of complexity the proof-of-concept was trying to achieve and could not justify the costly hardware and software needed for a point-cloud workflow (Baggett and McMillan, 2018). Instead, a workspace was recreated in Autodesk Inventor based on measurements taken from the Visualisation Laboratory at the University of Queensland, built in collaboration with Glencore and Rio Tinto as shown in Figure 5. The lab was designed as a Class 5 immersive visualisation system described in Stothard et al. (2008), which consists of a large 180-degree semi-immersive opaque display capable of showing live video feed as well as computer generated graphics simultaneously (Stothard et al., 2008). This wall-projected visualisation system can also be deployed in conjunction with modern head-mounted VR/MR systems to deliver fully immersive visual experiences.
Reconstruction of the workspace in unity (photo supplied by Qu et al. 2023).
It is worth noting that when it applies to large-scale mining projects where technical drawings of assets are not always accessible or complete, point-cloud technologies would be essential to reduce the amount of human overhead during the modelling process (Stothard et al., 2019).
Monitor and control the ball mill
After modelling the ball mill and the workspace, the next step was to establish the connections between the physical assets and the digital models in the virtual domain. In this study, achieving both real-time monitoring and control on the ball mill from the virtual space is considered an essential part of demonstrating a true DT with two-way, real-time data connections as illustrated previously in Figure 1.
Assessing the ball mill
Real-world projects in the minerals industry usually go through lengthy lifecycles from site discovery to decommissioning. Unlike those that are seen in other industries with a high adoption rate of IoT and digital technologies, planning, design, as well as construction of mining operations often date back decades before the introduction of the concept themselves (Farrelly and Davies, 2021a; Farrelly and Davies, 2021b). During a plant's active lifecycle, changes to technologies, standards and personnel presented a unique challenge for DT integration in the mineral processing industry, often amplified by the absence of original technical documentation, scattered data management and loss of skilled persons and corporate knowledge. In addition, mineral processes of the same kind differ from site to site in terms of their layouts and specifications due to variations in location, age, geology and ore grade. Therefore, it is important to assess each case accordingly when applying the DT concept to the minerals industry. To effectively achieve operation monitoring and control of legacy systems using DT, one must understand the pre-existing set-ups in terms of hardware and software specifications, intended use and limitations.
In the context of this study, the ball mill was originally built in 2015 as a lab-based test platform to study the size-specific energy (SSE) consumption of different mineral commodities under controlled grinding conditions. During its operation, a torque sensor (Burster Model 86413-5500-V501) measures the induced strain in the mill shaft and sends a voltage signal between 10 and −10 volts to a NI-USB6008 data acquisition (DAQ) device.
Torque was calculated from the voltage reading using the following linear equation:
Torque sensor calibration chart.
In addition to torque, shaft speed was measured using the following equation:
Pulse (edge) counting illustration.
The hardware configuration is shown in Figure 8. Besides processing input signals from the torque sensor, the on-board DAQ device is also tasked with outputting a voltage signal between 0 and 5 v to a variable frequency drive (VFD) which in turn, powers a 3 kW three-phase motor and controls the mill's speed through a 25:1 ratio gearbox.
Ball mill hardware configuration.
In its original designed configuration, the NI-USB6008 DAQ device served as the brain for the ball mill's control system. The data acquisition (DAQ) device records analogue signals from the torque sensor and converts them to digital signals using the built-in analog to analogue-to-digital converter (ADC) shown in Figure 9. It is worth noting that open-source hardware such as Arduino microcontroller and Raspberry Pi microcomputer were also considered for the task thanks to their excellent accessibility, expandability and cost-effectiveness. However, the author was instructed to digitalise the plant equipment without modifying its original hardware configuration due to ongoing projects conducted by other research groups.
Working principle of analogue to digital converter (NI 2022a).
At the front end, a 2D GUI application was used to monitor and control the ball mill's operation. The application was developed using National Instruments’ proprietary LabView software which is widely adopted in laboratories around the world (Chouder et al., 2013). Upon assessing the ball mill's design configuration and capabilities, the article presents two different approaches to integrate its legacy system with a Unity-based DT.
Frontend approach – interfacing with LabView
A frontend approach was implemented to interface a Unity-based DT with the existing LabView application developed for the ball mill. A schematic of this approach is shown in Figure 10. By keeping the DT application in the front end, this approach was able to enable ball mill operation within the boundary of the virtual space by taking advantage of Unity's VR and network multiplayer capability, while retaining the ball mill's existing hardware configuration and programming logic.
Interfacing with LabView through file I/O.
The existing communication logic between LabView and the ball mill remained untouched on the right-hand side of the schematic. Meanwhile, a DT was created in Unity following the procedures documented under Section 2.1. According to National Instruments, several methods could be used to establish data communication between third-party software and LabView (NI, 2022a). Here, a method called File I/O was chosen to connect Unity and LabView. Refer to Figure 11 for the detailed programming flowchart.
Data access flowchart.
To control the mill, an operator would initiate an input to the ball mill from the virtual workspace. The input settings containing the desired operational parameters was written to a data file located on the computer's hard drive which is updated when a new command was given by the operator. Meanwhile, LabView checks the time stamp of when the input file was last modified and reads the file when it detects a change. The input parameters such as mill speed, grinding duration, number of revolutions and energy target were parsed through the existing control loops in LabView, which in turn, controls the motor via the USB6008 DAQ device. To monitor the mill, LabView calculates the amount of torque on the main shaft based on the voltage reading from the torque sensor. Together with other parameters such as RPM and energy consumption, these output values were written to a second data file then displayed to the operator in the virtual workspace. During this interaction, the operator can be fully immersed in a virtual space in an accurate 1:1 scale. Figure 12 provides a comparison between LabView's original 2D interface and the reimagined user interface in Unity. In remote operations, such a level of immersion enabled by fast-growing XR technologies is a major selling point of the industrial metaverse which has the potential to facilitate effective team communication through the implementation of multiplayer with features such as animated virtual avatars and real-time voice chat. Furthermore, presenting an operational scenario in a 3D spatial format allows an easy understanding of real-life states and behaviours of the plant as naturally intended (Xie et al., 2018).
Reimagine of ball mill control UI in unity.
A File I/O interface would allow the developer to easily integrate this workflow to a wide range of fixed-plant applications with legacy PLC by keeping the data exchange at the front end of the control loop. It is simple to implement as well as troubleshoot, and it is currently implemented by industrial OEM applications that are running in integrated remote operation centres (IROCs) today. Examples of these applications include while not limited to haul route dispatching, ship loading, plant process monitoring, as well as mine planning (Benton, 2020; Guthrie, 2021).
To prevent access conflict from occurring between two instances trying to access the same data file at a given time, file reading follows a ‘Check-Access-Close’ logic. The programming flow diagram of this process is shown in Figure 11.
By following this programming logic, the file I/O method worked seamlessly between LabView and the Unity3D-based DT application during testing, enabling the user to navigate, interact, monitor and control the actual ball mill from the 3D virtual environment with minimal delay. While the DT program is running on the front end, the output data containing the current states of the asset was used only by the animator, audio and UI components, hence no input will be sent back to the PLC without explicit actions from the operator. To prevent unwanted behaviours from the asset, input parameters were parsed on a separate data file which could only be generated once the operator confirmed the intention to take control. Besides the existing safety guards programmed into the original LabView interface, Unity also ensures only a predefined selection of text formats and a range of values that are known to be safe for each input parameter. The PLC will not log any value outside of this range or format. To prevent dangerous actions from possible human errors, the DT interface was designed to require several precise inputs for the user in order to activate, edit and execute (Stone, 2016).
Alternatively, parsing data through network interfaces such as TCP/IP or hardwired serial buses could achieve a similar outcome without the concern for file corruption and access conflict during runtimes (NI, 2022a). From a metaverse perspective, doing so would reduce process latency and improve the responsiveness of data communication. However, transmitting operational data persistently through wireless serial interfaces could also present a higher risk to cyber security in the form of man-in-the-middle (MITM) attacks under an insecure network that could even hinder human safety when data are used to control the physical space. Practices such as virtual private network (VPN) and web-based protocols with data encryption such as HTTPS and WSS could help here but in the long run, industries will have to commit to the never-ending battle in the cyberspace.
Backend approach – custom data handler
After successfully demonstrating a working DT that is capable of monitoring and controlling a ball mill in real time from a 3D virtual environment, a second approach was implemented to explore the possibility of achieving a similar outcome from the backend using a combination of data files as well as serial communication, eliminating the reliance on the LabView software.
In this approach, a custom-made data I/O handler middleware application was programmed in Visual Studio C# using NI's native .NET Application Programming Interface (API) which provides an object-oriented class library that allows third-party software to interface with NI hardware (NI, 2022b). A schematic of this intermediate approach is shown in Figure 13.
Intermediate file I/O between unity and NI DAQ device.
The manufacturer's API was packaged as a Dynamic Link Library (DLL) file in the NI DAQmx driver installation along with official documentation and sample codes written in both Visual Basic .NET and C# programming language. The approach seamlessly integrated the native DAQmx class library to a user-friendly C# programming workflow in Visual Studio, therefore fully replacing the need for the LabView software which requires a separate licence subscription and pre-established knowledge of its unique graphical programming language. In theory, the same approach can be universally applied to a wider range of industrial hardware manufacturers if there is an official API published that allows programmers to access full hardware functionalities from the backend. Ideally, using open-sourced hardware for new projects with IoT capability in mind would also eliminate the need to rely on proprietary API from hardware manufacturers such as National Instruments and Siemens.
In the case of utilising NI's native .NET API to monitor and control the USB6008 DAQ device installed on the ball mill, a standard programming format specified by the manufacturer was followed to communicate with the NI hardware from a third-party .NET application. Depending on the levels of API support provided by different hardware manufacturers, DT developers may need to follow different steps when integrating real-world functionalities into a real-time graphics engine such as Unity. As mentioned previously, the use of open-source options would greatly improve the compatibility and future expandability of a DT system from a longevity standpoint. However, there is also a pressing need to ensure the security of the system in cyberspace. This is where standards for XR and industrial metaverse development become essential as outlined by Standards Australia (Wallace et al., 2023).
For data connections at the backend, two-way data flows in and out of the ball mill communicate over a serial interface. During operations, the data handler application shown in Figure 14 would be running in the background on an edge computer where the NI-USB6008 DAQ device is connected via a USB cable. The operator would specify two separate paths for data logging in the file explorer for input and output parameters. The data handler application also features a stop/start control to allow site workers to manually disconnect and restart the data transmission with the remote DT. This adds an extra layer of safety during maintenance.
Dt data handler application interface for the ball mill.
At the frontend, the operator uses a floating UI to control the ball mill from the Unity3D-based DT. At the same time, the mill's actual behaviour was visualised in the virtual environment in the form of animations and coloured texts. During this process, the command actions taken by the operator were written to the input data file and parsed to the mill via the serial bus where sensor data from the ball mill was parsed to the output file for visualisation in Unity. The data handler program also displays the logged values for both the input file and the output file which the plant operator could use to debug data transmission between the ball mill and the DT.
The simplicity of using files to transmit data between the ball mill and the Unity3D-based DT provides flexibility for future expansions and changes to the plant. Text formatting conventions of the input and output parameters were hardcoded into the File Handler application, while the locations of which the data files were saved could be modified using the built-in Windows file explorer interface. For long-distance remote operations, the data files can be saved and updated to network storage such as OneDrive, an industrial-grade Building Information Modelling (BIM) platform like Autodesk BIM360, or the corporate's internal network drive if available in order to reduce long-term licensing cost and potentially protect sensitive data hence adequate cybersecurity measures are taken. For many advantages such as ease of implementation, debugging, reliability and backend compatibility with legacy hardware, data files were used extensively by BHP's IROC operation to exchange real-time data with remote mine sites (Benton, 2020; Jurgens, 2018). For the purposes of this proof-of-concept, OneDrive was chosen for its accessibility and compatibility with the university's network firewall.
During pilot plant testing (Figure 15), the file I/O handler approach demonstrated a high level of responsiveness and reliability. A webcam was installed to capture the ball mill's behaviour in the pilot plant while the times between several control inputs and physical responses were measured in seconds. The results show an approximately five-second response time for control inputs, and the monitored parameters were accurately represented as 3D animations and floating texts in the Unity-based DT environment on a ten-second update interval.
Pilot plant testing (left) of the ball mill digital twin (right).
The actual speeds of the mill were measured on-site using a handheld RPM gauge and compared against the speed values set by the DT operator. The results showed an error averaging around 5 RPM which could be caused by multiple factors such as internal friction of the motor drive train and calibration (Table 1).
Ball mill DT test run results.
Overall, the outcome of the DT testing was satisfactory as the ball mill operation monitoring and control capabilities were achieved from a Unity3D-based DT application with reasonable accuracy.
Future considerations
From a programming perspective, the ability to access the manufacturer API directly from a Unity3D-based DT application (Figure 16) would allow DT developers to integrate the full suite of functionalities of hardware without the need for an intermediate data handler mentioned in the previous section.
Direct serial communication between unity and NI DAQ device.
Unfortunately, system integrations in the real world often present problems related to incompatibility with established workflows and standards in industries (Qi et al., 2021). In this case, NI's native .NET API was programmed to work with the Visual Studio development workflow, whereas Unity Editor could not import the DAQmx class library DLL file as a ‘managed plugin’ at the time when this article was produced. This means that developers could not directly call the functions from the class library to interface with NI hardware from Unity. To bypass this compatibility issue without changing the hardware itself, programmers would need to invest long hours in importing as well as translating the manufacturer's class library to a custom C# class library which can then be accessed seamlessly in the Unity editor environment. In a case where the hardware manufacturer does not provide its official API, the additional cost of additional labour or upgrading hardware without a favourable business case for XR-enabled DT in operational scenarios could deter mining companies from investing in further development of the concept.
Future developments of DTs will need to rely on the joint effort between hardware manufacturers and industry members as well as regulatory bodies towards improving interoperability. While the concept of industrial metaverse is still in its infant stage, the International Standards Organisation (ISO) and Standards Australia are working proactively towards creating standards for enabling technologies such as virtual reality, augmented reality, as well as mixed reality (ISO, 2020; Wallace et al., 2023). Initiatives from the Institute of Electrical and Electronics Engineers (IEEE) have also recognised the pressing need to develop standards to guide future developments towards industrial metaverses of different kinds (IEEE, 2023; Koziol, 2022; Saracco, 2022). For the mining metaverse, international and domestic standards should form the governing layer for future research and developments to improve compatibility with legacy systems, rapid virtual modelling, human comfort and safety, accessibility to technologies, privacy and security, interoperability and communication, as well as cost-effectiveness (Figure 17).
The seven dimensions of industrial metaverse (Qu et al., 2023).
An industry-led engagement approach should also be considered to facilitate expert knowledge between peak bodies from different sectors. Specifically, major mining companies such as BHP, Rio Tinto and Anglo American should work more closely with regulatory bodies such as ISO as well as Standards Australia, and hardware manufacturers such as Siemens, National Instruments, Caterpillar and Komatsu who provide products, services and expert knowledge in the IoT space (Figure 18).
Industry engagement for mining metaverse.
In addition, professional societies such as Engineers Australia, the Australasian Institute of Mining and Metallurgy (AusIMM) and IEEE could serve well as the middle body to bridge talents from different sectors while universities and research institutions like CSIRO provide dedicated resources and capacity for R&D. The push towards realising the metaverse was initiated by the leading bodies in the tech industries such as Meta, Nvidia, Apple and Microsoft. This ecosystem provides the disruptive technologies that enable the metaverse concept. Currently, there is not enough collaboration between the established circle of the mining community with these technology companies. Therefore, it is vital to include these players in the future development of the mining metaverse by providing more networking and employment opportunities for talents from this space.
Lastly, although Unity currently provides the best-in-class support for XR development workflows, adopting web-based solutions such as WebXR, WebGL, X3D and Three.js would not only reduce entry requirements for the industry metaverse experience but also improve persistency of the virtual space where permitted users can join from anywhere at any time. The implementation of multiplayer features such as animated avatars and voice chat with spatial audio would also highlight the advantages of a mining metaverse in a 3D spatial format.
Conclusion
A true DT is one that allows two-way, real-time state synchronisation between a virtual space and the real world. The minerals industry is putting a large emphasis on sustainable operation in response to the rising concern in its contribution towards climate change, and the ongoing labour shortage among younger generations at remote locations. This article has explored the concept of using DTs in conjunction with spatial visualisation and interaction to monitor as well as control a ball mill in a real-time operational scenario. By developing a proof-of-concept using two different design approaches, the study demonstrated that it is possible to integrate real-world functionalities of legacy processing equipment into a virtual space powered by a popular real-time graphical engine such as Unity. Despite reliance on API availability by the manufacturer, the backend approach has considerably lower implementation cost compared to the frontend approach which relies on an OEM software platform such as LabView. The article also suggests that future developments of an industrial metaverse for the mining industry need closer collaboration between mining companies, equipment manufacturers and suppliers, research bodies, professional societies, as well as tech companies outside of the mining community. Crucially, an industry-led approach to build a mining metaverse cannot succeed without clear written standards by a peak body to uphold safety and security, transparency, accessibility, efficiency and compatibility for existing as well as future mining projects.
Footnotes
Abbreviations
Declaration of conflicting interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.