Development of a virtual manufacturing assembly simulation system

Abstract

Assembly operations are a key component of modern manufacturing systems. Designing, planning, and conducting assembly operations represent an important part of the cost of a product. Virtual reality provides an efficient and cost-effective solution to manufacturing design, planning, and prototyping. Still there are certain issues (such as data translation, integration of various hardware and software systems, and real-time collision detection) faced while applying this advanced technology to the assembly domain. For example, existing works focus on using virtual reality systems and environments mainly to design new products and to plan for assembly. Little focus has been given to develop virtual reality environments that contribute to train operators on assembly operations and to bridge the gap between design and implementation/execution of assembly. Therefore, the research work presented in this article focuses on developing a fully functional virtual manufacturing assembly simulation system that solves the issues related to virtual reality environments. The proposed system uses a virtual environment to create an interactive workbench that can be used for evaluating assembly decisions and training assembly operations. It is a comprehensive system that provides visual, auditory, tactile, as well as force feedback. The system works successfully even with large components.

Introduction

In the era of globalization, one of the significant factors for manufacturing organization to remain competent and flourish in the market is to provide cost-effective and innovative products. Therefore, to fulfill the customers’ demands, these industries are always looking for advancements in the technological domain. The technological requirements of these firms are very straightforward, such as, for manufacturing industries, the technology should assist in either providing designing innovative products or it helps them to improve their processes in terms of time and cost. Mitigation of product development cycle time and cost have been areas of strategic focus of the industry. ¹ Quality requirements have also led to an increased emphasis on product and process validation and require efficient and effective product design changes. ² Assembly operations are a key component of modern manufacturing systems. ³ It has been estimated that assembly processes account for a large part of time and cost of product development. ⁴ Planning and training assembly operations during the early stages of product design can ensure that a product is manufactured in the most efficient way. The principle of design for assembly is concerned with making assembly decisions during the early product design stages^4,5 and is an integral part of computer-aided process planning.^6,7 Assembly sequence planning plays an important role in a successful assembly procedure, and an efficient sequence will result in improvements in terms of time and cost. Traditionally, assembly planning is achieved using two-dimensional (2D) drawings or costly physical prototypes. ⁸ Increase in product complexity make it difficult to make sequence planning manually; thus, computer-aided processes are applied. ⁹

In this context, virtual reality (VR) appeared as a technology that enables advanced forms of human–computer interaction based on a wide range of inputs, such as voice recognition and movement tracking, and an even wider range of outputs, such as advanced graphics, auditory displays, and haptic interfaces.^3,10,11 Therefore, VR-based systems are becoming popular in the manufacturing field. ¹² VR can be applied to different domains of manufacturing, such as design, prototyping, assembly, and maintenance.^9,13–21

At the present time, VR is recognized as a capable tool for examining the overall performance of a product, and it can prevail over the limitations in conventional processes of designing and manufacturing. ²² Manufacturing assembly processes impact product quality, time to market, and cost; ³ consequently, they can benefit from VR applications development. The VR applications can support the integration of natural human interaction into a computer-aided assembly planning system. ²³ There have been several notable contributions in the development of VR applications for manufacturing, including virtual assembly, ²⁴ disassembly reasoning, ²⁵ assembly and maintenance verification, ²⁶ the virtual assembly design environment (VADE), ²⁷ immersive virtual assembly, ²⁸ process simulation, ¹² and product assemblability analysis. ²⁹

VR research in manufacturing in general, and assembly operations in particular, has focused on applications development. Existing works focus on using VR systems and environments mainly to design new products and to plan for assembly. Little focus has been given to developing VR environments that contribute to train operators on assembly operations and to bridge the gap between design and implementation/execution of assembly operations. Technologies that allow using virtual assembly as an analysis, evaluation, assessment, training and operator skill acquisition, and improvement tool are not yet fully utilized by industry. Although this emerging technology is not completely understood in regard to applications within commercial industries, the technology as a whole is viewed as viable and valuable. There are several issues associated with the proper development of virtual environments (VEs) in manufacturing assembly simulation, including the translation of data generated by computer-aided design (CAD) systems, modeling the physical behavior and constraints of objects, collision detection, integration of various hardware peripherals and software platforms, and sometime limited field of view make VE unrealistic to user. Therefore, there is a need to develop a comprehensive VR-based assembly planning system that takes into consideration product CAD data, provides physical modeling, and offers intuitive feedback to users.

Therefore, this research work focuses on developing a fully functional system for virtual manufacturing assembly simulation system (VMASS), which is mainly intended to bridge the gap between assembly design and planning on one hand and assembly training, implementation, and execution on the other hand. Our focus is to provide a tool that enables training operators and letting them acquire and improve their assembly skills using VEs, which has received little research interest in the literature. The research work extends VR applications in manufacturing by integrating concepts and studies from training simulations, such as the evaluation of assembly training effectiveness and transfer of training. ³⁰ Therefore, the subsequent sections explain the related work and the development steps of VMASS and provide preliminary results of evaluation of the system for its effectiveness in the assembly training.

Related work

VR techniques have been extensively explored by the researchers to support assembly design and planning. The National Research Academy (part of the United States National Research Council ³¹ ) identified the following mission for VR applications development:

The most promising configuration will be designed by a computer-aided design system, manufactured in a virtual factory, and tested in a synthetic environment.

According to the authors, now the VR triangle that were suggested by Burdea and Coiffet ²² is transformed into VR quadrilateral because the fourth dimension is now added to the newly developed VR systems, that is, intelligence.^14,32–36 Figure 1 shows this transformation.

Figure 1.

Virtual reality triangle transformation to virtual reality quadrilateral.

Some of the research works mainly related to application of VR for the assembly training purpose are presented in this section. Assembly processes have a direct influence on product quality, time to market, and cost; consequently, they require special validation. ³ Assembly process validation includes determining (1) whether a human operator can assemble the part or component, (2) whether a human operator can disassemble the part or component for maintenance, (3) the difficulty of the assembly or disassembly process, and (4) the time required for the assembly or disassembly process. ²⁶ Planning and training assembly operations during the early stages of product design can ensure that a product is manufactured in the most efficient way^4,5 and enables design for assembly.

Durlach and Mavor ³¹ state that manufacturing systems are typically of national importance. They are intellectually challenging and can benefit from VR application development. VR applications can benefit manufacturing assembly processes by immersing an operator in a VE and providing feedback information to determine the ease of assembly or disassembly as the operator performs an assembly or maintenance task. ³ The resulting VR applications can integrate natural human interaction into a computer-aided assembly planning system. ²³ Jayaram et al. ²⁴ identified five issues that must be addressed for the acceptance of VR in manufacturing assembly applications:

• Can the system provide a solid view of assembly problems?

• Can the system aid engineers in decision-making?

• Can the technologies be applied to real design needs today and in the future?

• Can the system be used regularly by engineers?

• Can system data be exchanged with engineering design (i.e. CAD) systems?

There exist several notable VR applications that have been developed for manufacturing, which include the following:

• Disassembly reasoning. ²⁵ This article discusses the generation of a disassembly process for a product (a Chrysler LHS center console) using VR, in which (1) a CAD model is used to generate the virtual prototype product model and (2) automated reasoning techniques are used to generate the disassembly process model. The automated reasoning system creates a range of feasible directions for component removal to facilitate disassembly.

• Assembly/maintenance verification. ²⁶ This article discusses using VR to verify assembly and maintenance operations in vehicle prototyping. The authors discuss CAD-VR data integration and identify how engineers and designers can interact with the prototype naturally.

• VADE. ²⁷ This article presents a VE for assembly design. The VADE allows the efficient manipulation of multiple parts for assembly evaluation. The authors state that constrained motion and dynamic simulation aid in evaluating assembly operations.

• Immersive virtual assembly. ²⁸ This article describes a multi-modal immersive virtual assembly system. The authors view the VR assembly system as a finite state machine and incorporate different tracking devices and input and output hardware to enable optimal interaction with complex assembly models.

• Process simulation. ¹² This article discusses different VR applications in manufacturing process simulation. The authors conclude that VR can be used to model manufacturing processes in a way that allows engineers to solve problems and evaluate assembly decisions by interacting with the VR environment.

• Collaborative design. ³⁷ This article describes about a collaborative VR-based system augmented by a multi-agent system (MAS). The knowledge of designers is stored by the system in a product life cycle management environment. This knowledge was used by VR tools to investigate various aspects of the virtual prototype such as manufacturing, maintenance, reliability, or ergonomics.

• Product assemblability analysis. ²⁹ This article describes a visual assembly environment created in DELMIA^™ for a blower assembly. The authors conclude that “virtual assembly can be used for the planning and validation of the actual assembly process as well as feedback for design changes.”

There have been several other efforts in developing VEs for manufacturing assembly simulation.^3,38–40 Each environment is focused on a particular case study for a particular application (e.g. assembly or maintenance). Some environments utilize CAD data. The developed applications utilize different interaction paradigms and different input and output hardware and focus on assembly task simulation. There are several issues associated with the proper development of VEs in manufacturing assembly simulation, including (1) the translation of data generated by CAD systems, (2) modeling the physical behavior and constraints of objects, (3) collision detection, and (4) integration of various hardware and software systems.

First, translating CAD data into a VR scene requires producing polygonal representations of each part via the tessellation of parametric surfaces and/or B-rep solids.^3,41 An issue inherent in this translation process is that parametric information included in the CAD model and the pre-existing texture maps may not be imported into the VR scene graph. ⁴² Common CAD-VR translation issues include the “orientation of normal, missing geometry, and deletion of interior or other unwanted geometry.” ⁴³

Second, after the CAD data are translated into a VR scene graph, information regarding the physical behavior and the constraints on parts typically needs to be added.^44,45 Physical behavior information includes the weight of objects, their relative size, and representative object behavior such as hardness. ⁴⁶ Several types of constraints need to be added via VR modeling software, such as (1) positional constraints to affix parts to assemblies and (2) geometric constraints to maintain relative spatial data between various parts in the model.^42,47,48

Third and last, algorithms for collision detection between various parts, sub-assemblies, and assemblies need to be added to the VR model to represent the real-time interactions between parts during manufacturing assembly simulation.^34,49,50 Proper implementation of collision detection algorithms should (1) prevent part interpenetration and (2) provide feedback to users regarding how to change the position and orientation of the parts to properly complete the assembly task. ³ The integration of various hardware and software systems is also a matter of concern when developing a VR system.

Another area in manufacturing assembly simulation that can benefit from VR application development is training assembly operations. ⁵¹ The National Academy of Engineers ⁵² recommends the integration of technology-based tools to enhance learning; in fact, Dede ⁵³ introduced a pedagogy to enhance user learning called learning by doing in a virtual environment (LDVE). LDVE emphasizes that “hands on” activities are especially important in learning psychomotor-based skills such as those involved in assembly operations. There have been limited efforts toward building virtual manufacturing simulation environments for training. For example, Schenk et al. ⁵⁴ describe the creation of fully interactive and immersive three-dimensional (3D) visualizations of assembly lines and factories that can be used for factory design and for planning factory operations (without a specific focus on assembly operations). Additionally, Peniche et al. ⁵⁵ discuss the implementation of an immersive training system for mechanical assembly based on VR, and they conclude that the system is as effective as conventional methodologies for training mechanical assembly. Abidi et al. ⁵⁶ described the development and evaluation of a virtual assembly training system. A comparison between traditional method of training and semi-immersive VR-based training was made, and it was concluded that VR-based training was more effective.

Another important dimension in the manufacturing assembly VR-based systems is the haptic feedback. To make digital environment closer to the real environment, it is important to have a sense of touch and force feedback. The previous research works^57–62 appear to lack features in one area or another for the ideal haptic-based desktop VR system; therefore, still there is a need of comprehensive systems that will provide tactile as well as kinesthetic feedback, in an immersive environment. This system should take into consideration translation of product data from CAD, model physical properties and constraints, and provide feedback to users.

Creating a manufacturing assembly training system entails incorporating feedback regarding the degree to which trainees know “when, where, and how to use newly acquired knowledge.”^63,64 Feedback has been shown to be important in training tasks. ⁶⁵ Ying and Sotudeh ⁶⁶ studied the effects of feedback (e.g. auditory and visual) on task performance in a virtual assembly simulation environment. Objective performance data, such as task completion time and error rate, were collected. Additionally, questionnaires were used to capture the participants’ subjective opinions. The results showed that providing feedback improves assembly task performance. The authors concluded that “combined auditory and visual feedback offered better assembly task performance” than either form of feedback alone. A key finding reported by the authors was that the participants’ comments indicated that “non-realistic or inappropriate feedback had a negative effect on the task performance.”

There are several reviews of VR and assembly simulation, including reviews by Wang et al. ⁴² on collision detection and physically based modeling in VR; Stanney ¹⁰ on the design, implementation, and applications of VEs; Wang ¹⁸ on virtual prototyping; Schmorrow et al. ³⁰ on VEs in training and education; Abdulaziz and Mousavi ⁶⁷ on using haptic feedback for VR in manufacturing; and Seth et al. ³ on using VR in prototyping assembly methods. Based on the existing literature (to the best of the authors’ knowledge), there have been limited attempts to integrate training system design and evaluation with manufacturing assembly process simulation. The goals of this research consist of twofold: to build VMASS that (1) facilitates the investigation of the optimal translation of CAD data to VR software and (2) integrates the various devices and software to build a system, which is comprehensive in terms of feedback mechanisms, and provides an efficient training platform.

Methodology

This section summarizes the methodology that will be used to develop the VMASS.

Structural design of the system

The proposed system is based on three modules: product data management system (PDMS), product data conversion system (PDCS), and VMASS. Figure 2 illustrates the relationships between the three modules. The PDMS module generates CAD files, which serve as an input to the PDCS module. The output of the PDCS module is a VR software file (i.e. a *.VRNative file) that is used for creating the virtual assembly simulation environment.

Figure 2.

Relationship between PDMS, PDCS, and VMASS.

PDMS module

The function of this module is to generate product descriptions (in eXtensible Markup Language (XML)) using a specification taxonomy for data coming either from design efforts or via reverse engineering. The product description is then used to create a CAD model that is exported in IGES (initial graphics exchange specification) format, STL (standard tessellation language) format, CATIA (computer-aided three-dimensional interactive application) format, SolidWorks format, and so on. A taxonomy for specifying the appropriate level of detail for parts, sub-assemblies, and assemblies will be developed. The taxonomy will use the XML file format to represent (1) product dimensional information; (2) positional information with respect to a global reference system; (3) spatial relationships between different product parts, sub-assemblies, and assemblies; and (4) inter-part relationships. The taxonomy will include (1) product data in the form of low-level geometric entities, such as lines, planes, and circles, and (2) the relationships between the product geometric information. The XML taxonomies for the selected case studies will be used to create 3D object models in a CAD system, such as Pro/E or CATIA. Figure 3 illustrates the structure and operation of the PDMS module.

Figure 3.

Structure and operation of the PDMS module.

PDCS module

The function of this module is to convert the geometrical format derived from a CAD system to the VR software file format. In this task, design-of-experiments techniques are used in conjunction with the ProductView adapter to convert CAD files to a VR software format. There are several parameters associated with the conversion process, such as the tolerance, chord length, edge length, and number of levels of detail. Sequential experimentation is used to determine the parameter settings that result in optimal data conversion, as measured by the quality of the resulting VR representation. Figure 4 illustrates the structure and operation of the PDCS module.

Figure 4.

Structure and operation of the PDCS module.

VMASS module

The function of this module is the virtual simulation of manufacturing assembly operations. The VMASS module interfaces with VR hardware, such as the Intersense^™ tracking system, the 3D rear-projection display, data gloves, haptic device, and the 3D surround sound system. The VMASS uses the optimally converted VR software file. The VMASS has several sub-systems, such as (1) an input manager to handle communication with various input devices such as the keyboard, mouse, haptic, glove, and wand; (2) a VE world manager that manages task interaction and graphical scene updates; (3) a visual manager that updates a graphical 3D display; (4) an audio manager that updates a 3D surround sound system; (5) an assembly simulation event handler that manages inter-part constraints and detects inter-part collisions in real time; and (6) haptic rendering engine. Figure 5 illustrates the structure and operation of the VMASS module.

Figure 5.

Structure and operation of the VMASS module.

System description

In this section, hardware and software system of VMASS is described.

Hardware

The description of VMASS hardware system is explained in this section. The different hardware peripherals include the following:

• A 3.1 m × 2.1 m power-wall to provide a platform for 3D projection;

• A Christie Mirage rear-projected high-resolution (S + 6K) projector to produce an immersive 3D environment;

• Active shutter glasses for stereoscopic viewing;

• Nvis^™ head-mounted display (HMD) for fully immersive VE; moreover, it is used to assess collaborative VE;

• Dell Precision T5400 workstation with 2.33 GHz Intel Xeon processor and 1.5 GB Nvidia FX5800 graphics card, which runs the software and enables interaction between human and machine;

• Dell Precision T5400 workstation with 2.33 GHz Intel Xeon processor and 1.5 GB Nvidia FX5800 graphics card, which makes collaborative environment and controls HMD;

• An inertial tracking system (i.e. Intersense^™ IS900) that provides dynamic, real-time measurement of the position and the orientation of the user’s head and hands;

• Phantom Desktop haptic device to provide the force feedback;

• 5DT data gloves to provide the tactile feedback;

• AMX controller to control the lights, display, and sound system.

A schematic diagram of the VMASS is shown in Figure 6. The rear projection–based system provides a benefit that all the messed up hardware and wires lie behind the screen and the user and the viewers have a clear view of the VE.

Figure 6.

VMASS schematic diagram.

Software

The system depends on a set of software involved in constructing the virtual assembly environment:

• CATIA, SolidWorks, PTC Pro-engineer, and so on can be used to construct the various components of the assembly.

• VisRender is used to employ a number of aesthetic features and supplementary behavioral properties to the CAD models, so that the subsequent VE will evenly match with the real world.

• Virtalis stereo-server is used to achieve interactive functionality, real-time visualization, and integration of various VR devices.

• Several developed plugins to support and integrate the various hardware devices and to run the collaborative design environment.

Figure 7 shows the software architecture of the developed system.

Figure 7.

Software architecture of VMASS.

Collision detection

Detecting the collision in the virtual assembly environment is vital to the success of the system. Within the VE, when virtual hand grabs the component, it is requisite to determine the collision status among the virtual hand and the digital object. ⁶⁸ Otherwise, through virtual hand, the user cannot grab the digital components. The two important criteria for a successful collision detection algorithm are response time and accuracy. VMASS uses an axis-aligned bounding box (AABB) algorithm for real-time collision detection. The bounding box in this is determined by the minimal hexahedron which includes this object and parallels to the coordinate axis. Two AABB transect with each other if and only if the projective zones of their three-dimension axis overlay. Another issue arises in collision detection is to prevent penetration of the virtual hand through the digital components. Therefore, to handle this issue, the restriction algorithm suggested by Ding et al. ⁶⁸ is applied. Moreover, to detect the interference between the components, a visual feedback is applied, so that once there is interference detected between two components, then their color changes to different color, for example, red. This provides the visual feedback to the user while conducting the assembly simulation. In addition, auditory feedback is also provided when components have interference.

Characteristics of VMASS

Semi-immersive environment gives a user sense of presence in the digital environment. It helps a user to analyze the design and assembly as close as possible to the real-world environment. The feature such as collision detection can be used to validate the assembly interferences. The motion simulation provides the user with assembly sequence and thus will be used for training purpose. Figure 8 shows the graphical user interface of VMASS.

Figure 8.

Graphical user interface of the VMASS welcome screen.

Figure 9 shows a user assessing the VMASS for a large-component assembly.

Figure 9.

Virtual manufacturing assembly simulation system (VMASS): (a) user analyzing the design, (b) collision detection visual feedback, (c) collaborative environment, and (d) user working with data gloves.

As shown in Figure 9(a)–(d), the developed system can successfully handle large components and products, such as an aircraft engine. The system performs well with all the cases and provides visual feedback, auditory feedback, tactile feedback, as well as kinesthetic feedback. Therefore, it may represent a comprehensive VR system applied in the research that provides different options such as immersiveness option (semi or fully), haptic feedback (kinesthetic, tactile, or both), collaborative, or individual. The system also performs efficiently for real-time collision detection.

The VMASS provides the following features to answer the above-mentioned assembly verification issues:

• Human operator can easily and intuitively assemble a part and component;

• A human operator can innately disassemble a part or component for maintenance;

• CAD data can be imported to VR environment in a systematic manner;

• The time duration of the assembly or disassembly process can be studied;

• The ergonomics analysis during the assembly or disassembly process can be conducted.

The VMASS is responsible for assembly task simulation. The VMASS is used for evaluating assembly alternatives and for training assembly operators. The collision detection feature is used to assess product spatial relationships in real time to determine if the assembly procedure (or disassembly procedure) is performed in the best possible way. The feedbacks routine is used to provide visual, haptic, and auditory cues to the users of VMASS and is triggered using the collision detection routine. Feedback is a necessary component of training. ⁶⁵ There are different types of feedback (e.g. cognitive and corrective feedback) and different points at which the feedback can be given (e.g. during the action or after the action). Feedback during the action provides information to trainees regarding their performance while they are engaged in a training exercise. ⁶⁵ This feedback can take the form of “training wheels,” in which substantial feedback is provided at the beginning of a learner’s interaction with the system, and the feedback is withdrawn gradually. After-action feedback provides information to learners about their performance and provides strategies for improving their performance in future training opportunities. ⁶⁹ VMASS is especially designed for manufacturing assembly training purpose; however, its flexibility allows to use it in other applications also. The potential application domain of VMASS is shown in Figure 10.

Figure 10.

Potential application domains of VMASS.

Conclusion and future work

In this article, we detailed the design of a VMASS, and its main functions include (1) evaluating assembly alternatives, (2) planning for assembly operations, and (3) training assembly operators. The VMASS uses a VE to create an interactive workbench that can be used for evaluating assembly decisions and training assembly operations. The developed system provides visual feedback, auditory feedback, tactile feedback, as well as kinesthetic feedback. Therefore, the system is comprehensive and flexible as it provides different options such as enables immersiveness (semi or fully) and haptic feedback (kinesthetic, tactile, or both) and provides collaborative or individual interaction modes. Our suggestions bring together two different VR applications: (1) manufacturing systems simulation and (2) training system design and evaluation. The developed system contributes to bridge the gap between design and implementation/execution of assembly operations. It allows using virtual assembly as an analysis, evaluation, assessment, and training tool, to allow training operators, and letting them acquire new skills and improve existing ones. The developed system also contributes to solve some of the challenges associated with integrating training systems into manufacturing assembly process simulation. Potential applications for the resulting VMASS include the aerospace, automotive, and defense industries, among others.

VMASS improves the computer-integrated manufacturing (CIM) by integrating CAD and computer-aided assembly planning in a VR environment. VMASS is a comprehensive system, which includes almost all features and feedbacks of VR, except olfactory, which is of not much relevance to assembly tasks.

VMASS offers a collaborative design environment that can be a very useful tool in this era of globalization. Through this feature, design teams which are geographically apart can work on a same design within a common VE. It incorporates collision detection algorithm to provide real-time feedback to users.

VMASS can be used for manual manufacturing assembly training, since it provides risk free platform, in which user can conduct assembly without the risk of damaging the components. The system offers an efficacious solution for training when there are very large components involved such as in aerospace and marine industry, in which the physical prototypes involve a large amount of cost and risk. Hence, VMASS provides a cost-effective and time-saving solution for assembly training and validation.

Future work includes a series of user-based evaluation studies that will be conducted to assess the effectiveness of the VMASS in the training domain. The study will consist of a pre-test and a post-test type of experimentation. In addition, several other haptic feedback mechanisms such as friction and gravity will be added to the environment.