A smart performance measurement approach for collaborative design in Industry 4.0

Abstract

Industry 4.0, the fourth industrial revolution, focuses on intelligent and smart manufacturing. This article investigates a smart design performance measurement approach, which can be utilized to support performance measurement implementation during a collaborative design process. First, we develop a smart product design framework with Industry 4.0 enabling technologies to support key design stages in an iterative fashion. Second, based on this framework, we propose a smart design performance measurement approach to potentially support a smart product design project management via its performance management. Third, we adapt our existing design performance measurement, for a traditional design environment into a smart design environment at its early stage to test its feasibility. This approach features integration of a flexible performance measurement setup, a multi-feedback design performance measurement mechanism and a multiple design performance measurement results presentation which allows the design performance measurement approach to produce flexible and customized operations by connecting design performance measurement with the stage-based design objectives, balancing design performance measurement feedbacks through interoperability between collaborative design team members and providing real-time design performance measurement results to guide design activities. An empirical industrial evaluation case study indicates that the proposed design performance measurement approach can support design team members in improving their collaborative design performance.

Introduction

A new industrial era, Industry 4.0, also called the fourth industrial revolution, focuses on intelligent and smart manufacturing. One of its application areas is to apply advanced information technologies to support design collaborations (between user to machines, user to user and machines to machines) and to improve a product development performance, such as the Cyber Physical Systems (CPS) and Internet of Things (IoT), smart factories and smart services.^1,2 Applications of smart information technologies in product design and development have had a significant impact on peoples’ behaviour in communication, negotiation, coordination and collaboration in a design team.^3,4 Thus, several works highlighted that Industry 4.0 is changing the operational process of design collaboration and innovation in organizations to foster knowledge flows via an open-approach knowledge management system. ⁵ Meanwhile, it has been widely agreed that the complexity of design projects increased dramatically in the last 10 years, because of multidisciplinary design collaborations, dynamic design processes and intangible and unpredictable design results. ⁶ Therefore, it is essential for companies to improve their design collaborations via smart technologies to allow product design and development cooperation beyond industry and sectorial boundaries. ⁷

In Industry 4.0–related research fields, such as collaborative design and computer-aided design, studies have tended to focus on supporting and improving design collaborations from computer-supported collaborative design, smart collaboration platforms, smart factories^8,9 and simultaneous engineering management^10,11and project management. ¹² Although the existing studies are notable, there is a scarcity of research that has looked at improving the collaborative design by a performance measurement (PM) approach in the context of Industry 4.0. Researchers have demonstrated the positive impact of PM on project team management, such as promoting self-motivation, supporting decision making and facilitating training. ¹³ Companies operate PM to examine and reflect on their staff performance, organization culture and strategies positively. ¹⁴ This affirmative impact is especially beneficial to support design collaboration. Industry 4.0 will enable PM operations to engage project staff from cross-functional and industrial sectors, customize PM, conduct PM activities at any time in any places and receive real-time feedback via technologies such as web-based systems, IoT and cloud-based infrastructure. It breaks boundaries and limitations of people, time and place in the traditional and off-line-based PM operations.

Thus, this study explores ‘How to improve collaborative design via a performance measurement approach in Industry 4.0?’ More specifically, it aims to investigate and develop a smart design performance measurement (DPM) approach to measure and improve collaborative design performance from a process perspective and at a project level. The contributions of this article are as follows:

1. A smart product design framework is developed with identified Industry 4.0 enabling technologies to support every key design stage.

2. A smart DPM approach is proposed, which can partially support a smart product design process.

3. Adaptation of our existing DPM approach into a smart DPM.

4. Implementation of the proposed smart DPM has been validated via a case study.

The proposed smart DPM approach contributes to simplify and customize PM to support and improve design collaboration. Based on a validation case study, it has been confirmed that the smart DPM approach can support design managers to monitor the design development performance during a design process, help design team members to learn from the PM results, and in turn to drive collaborative design performance towards achievement of strategic objectives. Furthermore, it can realize a flexible and customized project management and quality control operations by integrating smart DPM with stage-based design objectives, balancing DPM feedback through interoperability between collaborative design team members and providing real-time DPM results to guide design activities.

Literature review

Collaborative design

Collaborative design has been intensively discussed in the last decade. Li et al. ³ defined collaborative design as an activity that requires involved project stakeholders to work together, share information, interact to organize design tasks and make decisions jointly. As a key factor of the success of new product development (NPD) and business, collaborative design contributes to a reduction of product development costs, access to new skills, technologies and new markets. ¹⁶ Thus, many scholars have concentrated on improving collaborative design from multiple aspects.^8,11,16,17

Among these studies, computer-based and web-based technologies have been applied to construct new functions to improve design collaboration, such as interoperability, virtualization, decentralization of decision making and sharing knowledge. For example, some researchers concentrated on improving design team communication, information sharing, cooperation, coordination and negotiation via web-based collaborative design applications.^16,17 These applications support design project members to access to design information on components and sub-assemblies, to communicate among multidisciplinary design teams in multimedia formats and to authenticate access to design tools, services and documents. In order to improve design interoperability, El-Diraby et al. ¹⁸ developed an online system ‘Green2.0’ that supports stakeholders/users to share comments and views about building design in the construction industry. Furthermore, a Constraint Satisfaction Problem (CSP) platform helped design managers to decentralize decision making by integrating supply-chain constraints for product design problems. ¹⁹ Although the existing research on collaborative design tools has created significant contributions to improving design collaboration, there is a lack of research exploring how PM can support collaborative design during a design process with technologies that Industry 4.0 provide.

PM

In the last two decades, PM has been intensively discussed and explored from both theoretical and practical perspectives in the academy. From the theoretical aspect, the traditionally PM research focused on a narrow or unidimensional measurement. ²⁰ Applications of this single PM were widely applied to measure the success and failure of NPD. ²¹ Then, Kaplan and Northon ²² developed a balanced set of measures to guide PM from finance, internal business, the customer, innovation and learning, so as to overcome shortages of the single-dimension PM measurement. Griffin and Page ²³ applied this balanced measurement idea into product development studies. They developed a project-based balanced measurement matrix to measure product development success and failure from customer-oriented success, financial success and technical performance aspects. By following a similar root, other measurement criteria such as innovation, creativity, project planning, product life-cycle time, customer participation, efficiency and effectiveness have been considered as a key element to evaluate the success of product development in recent studies.^24–27

From the practical perspective, PM system design, evaluation and applications have been well researched to support PM implementations in organizations.^28–33 An effective PM system enables project managers to quantify efficiency and effectiveness of past actions via PM activities to collect, sort, analyse and interpret PM results to support decision making processes. ³⁴ Several recommendations for successful PM system design have been proposed from previous studies, such as PM should be positioned at a strategic level in organizations; selected measures should be clear and easy to understand by those being evaluated; performance data should be collected, where possible, by staff whose performance is being measured; performance should be reported daily or weekly; measures should consider all members of the organization so as to understand how they affect the entire business; PM systems feedback should be provided at numerous levels of the organization and PM systems feedback should be integrated cross-functionally to ensure it supports and not inhibits strategy implementation.^35,36

Although a large body of research has explored various PM criteria and illustrated how to design a PM system, most of this work only provided some strategy-based recommendations for creating a successful PM tool. Furthermore, there is a lack of research exploring how to implement a PM system as a project management tool for collaborative design projects, especially with consideration of the Industry 4.0 context. Comparing with this body of related work, this study concentrated on design and development of a smart DPM approach that provides a flexible, customized and real-time feedback PM operation to improve a collaborative design.

Smart product design and PM framework under Industry 4.0

Key smart product design enabling technologies

Industry 4.0 enables product design, development and manufacturing ecosystems driven by smart technologies such as CPS, web-based platforms, IoT and cloud-based infrastructure. ³⁷ Applications of these smart technologies during a product design and development process have supported design teams to improve communication, interoperability, flexible operation, virtualization, co-design, concurrent engineering for design, decentralization of decision making, customized project management and data management.^12,38 For example, virtualization focused on developments of simulation models to test design strategies and management decisions before a product design reaches production stages. ³⁹ Cloud-based manufacturing supports auto communications between temporary, reconfigurable CPS production infrastructures based on a networked manufacturing model to improve manufacturing efficiency, reduce cost and optimize resource allocation based on a customers’ requirements. ⁴⁰ In addition, design collaborations have been supported by various web-based platforms to facilitate communication and decision making during product design and development processes (Figure 1). ¹⁷

Figure 1.

Technologies in Industry 4.0.

Smart product design framework

Product design and development is a complex process and always involves multi-stages and experts with various knowledge. Although there is no universal design process that is suitable to all types of product design, traditionally, there are four broad phases of a design process in essence: planning, design, testing and production. ⁴¹ Applications of Industry 4.0 technologies will have significant impact of the product design and development process (Figure 2). First, from a planning perspective, technologies such as Product Lifecycle Management systems and Enterprise Resource Planning (ERP) tools can provide an integration of comprehensive and timely updated product data that allow design managers to develop and optimize project plans. ⁴² Other tools such as digital project planning and workflow management systems assist design managers to reduce time requirement on project management by cutting down administration tasks and necessary paperwork. Second, during the product design stage, technologies such as web-based collaboration platforms, visualization tools, open innovation platforms for customer/user engagement, digital Failure Mode and Effect Analysis and three-dimensional (3D) scanning can be used to improve design efficiency and effectiveness. Co-design/co-creation has been identified as one of the most important activities in product design and development because it contributes to improving design quality, reducing time and cost via integrating key stokeholds in design projects and promoting data sharing. ⁴³ Traditionally, co-design activities are conducted mainly off-line and companies had to travel around the world to collect insights from stakeholders, such as targeted customers/users, business partners and suppliers. Web-based design collaboration platforms provide functions that not only support communication but also allow them to review design works in diverse formats (such as two-dimensional (2D) or 3D), make immediate changes on design and preview the likely impacts of the changes on cost and production. ¹⁷ As a result, design efficiency would be improved dramatically. Third, regarding the testing stage, 3D printing, virtual design review tools, rapid prototyping and digital mock-up techniques can help designers to develop physical and digital product prototypes for evaluation. Web-based customer surveys and social media would also support the design team to collect feedback from customers/users. Finally, in the production stage, technologies such as smart sensors, smart factories, cloud computing and intelligent CPS can be applied to improve manufacturing.

Figure 2.

Smart product design framework.

Apart from the product design and development tasks, project management–related tasks such as resources management, team communication, decision making and project review also need to be considered during the project development. From a resource management aspect, Industry 4.0 technologies such as Enterprise Resources Plan systems and SharePoint enable design managers to review and plan business resources with product technique information (such as engineering and production information) on a common platform, which provides resources information across different sectors to support the project team in their daily activities. ⁴² Emails, shared folders, Google Doc and videoconference systems (such as Skype and Zoom) are making team communication and collaboration more efficient. Web-based collaboration platforms not only support product design and development activities, but are also decentralizing decision making in the project team by generating information from both team members and machines/sensors based on CPS and IoT. These technologies also bring a greater potential for artificial intelligence to contribute more to project decision making in the future. ⁴ Regarding quality control, Industry 4.0 can provide real-time key principle indicator cockpits and mobile monitoring solutions for design mangers to monitor the project development processes. As discussed at the end of the literature review section, there is a scarcity of study focused on implementation of DPM tools for collaborative design projects, especially in the context of Industry 4.0. It is worthwhile exploring how Industry 4.0 technologies can be integrated into DPM operation and to support design collaboration during the design process.

Smart design performance measurement (S-DPM) framework

In order to develop a smart DPM application approach, four design principles have been chosen. The selection of these principles was based on the features of Industrial 4.0, collaborative design projects and their matching with existing PM system design suggestions. They are: (1) flexible operation: supporting dynamic project management; (2) customization: strategic links between DPM operation and stage-based collaborative design objectives; (3) interoperability: allowing all stakeholders into the operation and (4) real-time feedback: providing a holistic analysis of the project development for decision making during a design process.

Flexible operation: supporting a dynamic project management

Many organizations are outsourcing their work to ensure design quality and product productivity to satisfy dynamic market changes. Thus, collaborative design project team members might come from different organizations and may not be located under the same roof. A team member, with various responsibilities, may join and leave the project team at different points in time. These features make a design team very dynamic and require dynamic management of a collaborative design project. But, very few systems consider the dynamic features of PM as Nudurupati et al. ⁴⁴ summarized; ‘today most PM systems are historical and static’. These static PM systems are not sensitive to changes in the internal and external environments of the project. Hence, the effectiveness of these PM systems can be questionable. Therefore, based on the dynamic nature of collaborative design projects, it is necessary to develop a DPM tool with a flexible operation feature. The tool would enable design managers to administer the project dynamically, such as create or remove members from the DPM operation during a collaborative DPM process. By doing so, it will also be able to support design managers to control the complexity of the project development processes, especially large-scale and long-term projects. Industry 4.0 technologies will be able to support the manager to develop a DPM operation platform that is able to facilitate functions of DPM team setup and be flexible to team member management, based on the design development process.

Customization: linking DPM operation with design project strategies

As suggested by the literature, a PM tool should be able to link with project strategies.^34,35,44 Because of the complexity and uncertainty in a collaborative design process, different design stages may have distinctive sub-level project strategies and the overall project strategies may vary during a project development process in order to match market changes or respond to competitor actions. Thus, the required DPM criteria for different stages in a collaborative design process might vary. For instance, according to Sun and Wing’s ⁴⁵ work, the ‘clearly defined target market’ criterion is the top criterion at the ideas generation design stage, while the ‘implementation of quality standards’ criterion is more important for the design specification stage in the context of the toy industry in Hong Kong. Therefore, a customization feature, which allows PM to link with the project strategy, is crucial to the success of a PM tool design. Furthermore, ‘to define a set of measures that can be clearly linked to operational strategies of the project’ has been identified as one of the major challenges in PM design. ⁴⁶ In the same vein, Wouters and Sportel ⁵⁰ confirmed that the design and setting of performance measures were concrete formulations of the firm’s strategic choices. Therefore, our second design principle is that the DPM tool should offer sufficient flexibility for design managers to customize DPM measures for matching the dynamic collaborative design strategies. Industry 4.0 technologies can support the customized feature of DPM tool via an Internet-based operation platform. This platform allows project managers to set up measurement criteria during the design process via multiple devices such as computers, tablets and mobile phones.

Interoperability: allowing all interested stakeholders into the operation

Collaborative design projects always recruit staff with diverse professional skills for tasks such as design research, insights, concept design, research and development, manufacturing, marking and product services.^3,16–19 In this cross-functional design team, members are expected to collaborate together towards shared design goals, to identify design issues, to organize design activities, to make a joint decision and to determine design constraints. ¹⁵ Goh ⁴⁷ highlighted that the success of PM implementation requires the engagement of all stakeholders of a project. Thus, with an intention to produce effective and accurate DPM results, not only design managers but all the collaborative design team members (including key stakeholders such as suppliers, outsources partners, clients and end-users) should participate and add value to the DPM operation process and to improve the project from multiple subject perspectives. By doing so, it will create the ownership of a DPM system among the collaborative design team members and ensure a transparent, consistent and fair performance implementation. Moreover, collecting PM data from multiple sources could improve the accuracy of DPM results.^34,35 As some collaborative design projects might involve clients, users or consumers into the project development process to ensure that the design project is developing towards the excepted direction and to improve design quality. These stakeholders should be involved in the DPM operation process as well. Therefore, it is essential to allow every team member and related people from both internal and external, such as designers, clients or users, to participant in the DPM operation process. Industry 4.0 technologies will be able to provide flexible access to project team members that allows them to be engaged with the DPM operation process without minimized limitation of location and data input devices.

Real-time feedback: providing a holistic analysis of the project development

One important objective of a PM system is to improve performance by learning and changing. ³⁵ Only calculating and publishing PM results cannot improve collaborative design performance; the real success relies on staff’s positive behavioural changes and responses to the performance information during the DPM operation process. ⁴⁸ If PM feedback cannot effectively contribute to decision making activities, then it defeats the purpose of developing performance measures. Therefore, a DPM tool should be considered as an integral element in a project management operation process and able to provide real-time feedback. ³⁵ Meanwhile, design managers should operate a DPM exercise not only to monitor the design process and but also to get real-time practical insights, which can support the team to identify existing and potential design issues and encourage them to fix any problems as early as possible. Therefore, the final recruited design principle is that the DPM tool should be included in a project management process and able to provide a holistic analysis of the project development and real-time feedback for decision making during a design process. Industry 4.0 technologies will enable the DPM approach to analyse collected DPM data during a design process and provide measurement results via multiple media formats to the team.

The DPM implementation process under Industry 4.0

Based on Industry 4.0 technologies, the proposed smart DPM approach could be operated to measure collaborative design performance by following the procedure below during a design process:

1. A web-based platform can support a collaborative design project manager to set up project objective, plan and a project team based on the required skills in the DPM system. Based on the team structure, the manager should define members’ role as the top design manager, middle managers or individual designers based on their job roles in order to clarify their positions for the multi-feedback component in the DPM approach. This function allows the design managers to include all project team members to engage with PM activities no matter where they are located to from which groups or companies.

2. The design manager can design PM criteria and set up stage-based weightings for each staff member based on their job responsibilities during the collaborative design process. This function enables the DPM approach to provide precise PMs at an individual staff level with consideration of possible shifts of staff’s job responsibilities during the design development process.

3. The design manager can decide the frequency and set up milestones for DPM operations. Based on this setting, project team members will receive notices that guide them to provide PM information to the DPM online system via their computers or mobile devices.

4. The design manager can explain the DPM matrix and DPM operation process to all project team members in order to make sure they share a common understanding of the DPM criteria and the process. The DPM online-system provides various communicating channels for the manager and project staff to communicate with each other, such as sharing documents, dialogue charts, a discussion forum and videoconferences.

5. Depending on the chosen DPM matrix, the DPM approach can collect PM data from the project team members via the multi-feedback DPM mechanism. Project team members need to login to the online platform and then complete PM data input tasks for themselves, their managers, the same level colleagues and lower level staff if applicable. During this PM process, they have access to the DPM matrix, historical DPM data and information of team member’s work progress through shared documents and machinery data via IoT technologies.

6. Next, the DPM data are saved in the Database component for performance calculation and historical analysis.

7. Then, the DPM engine calculates and produces DPM scores for each design team member.

8. Subsequently, DPM results can be presented with multiple formats to support decision making in product development, such as to identify issues or problems in design collaboration and explore the strengths and weaknesses of the design team members.

9. The DPM outcomes can help the design manager to better monitor and supervise the design team during the design process. For example, the manager can offer specific training to staff if his or her performance outcomes reflect a lack of management skills. The team members can also have a better understanding of their performance when comparing their self-evaluation scores and feedback with other colleagues.

10. After the DPM approach has been optional multiple times, the collaborative design team members can review their performance holistically and decide on their responsive actions early in the project stages. This will have a positive impact on the design development.

11. The frequency of DPM operation during a design process can be determined by the project features, such as size, number of team members and time scales.

Development of a smart DPM approach

According to the design principles outlined above, an operational DPM application approach (Figure 3) was developed with five major components: (1) DPM implementation setup; (2) multi-feedback DPM mechanism; (3) DPM database; (4) DPM engine and (5) DPM result presentation. The following sections will describe details of these components.

Figure 3.

DPM application approach.

DPM implementation setup

Applying the design principle 1 – flexible operation: supporting a dynamic project management) – a DPM implementation setup component was developed. This component enables design managers to create or remove members from the DPM operation team, to set up and update DPM criteria during a collaborative DPM process through a flexible DPM team setup and strategic DPM criteria weighting setup function. Here, we used a DPM matrix that has 25 criteria and represents five indicators: Efficiency, Effectiveness, Collaboration, Management Skills and Innovation (Table 1). This matrix has been chosen because it was particularly designed and developed for collaborative DPM. ⁴⁹ Industry 4.0 technologies will be able to allow project managers to build up a team and set up responsibilities for each team member via a web-based DPM platform. The project set up can also be accessed via multiple devices to set measurements and collect data via Internet of Thing technologies.

Table 1.

Design performance measurement matrix. ⁴⁹

	Most important Less important
Efficiency	Decision making efficiency	Problem solving	Personal motivation	Ability to work undertake pressure	R&D process well planned
Effectiveness	Delivering to the brief	Personally responsible/work ownership	Understand design rationale	Fast and detailed feedback	Managing mistakes
Collaboration	Clear team goal/objectives	Information sharing	Communication quality	Cross-functional collaboration	Shared problem-solving
Management Skills	Decision making	Define/fully understand role/s and responsibilities	Build high morale within the team	Conflict management	Monitor/evaluate team performance
Innovation	Competitive advantage	Select the right creativity concept to implementation	Products lead to future opportunities	High-quality product design	Perceived value

Flexible DPM team setup

As collaborative design team members might join or leave the project at different times, the flexible DPM team setup function allows design managers to register new staff into or remove a staff from the initially established project team during the project development process. For newly added members who join the project in the middle of the design development process, the DPM approach creates a new member package, which demonstrates relationships between the new staff and other team members and creates PM data collection, analysis and recording. Their performance will be assessed from the next measurement point after they join the project team. By doing so, the new staff’s first round DPM can be conducted based on their interactions and contributions to the collaborative design project between the previous and the next incoming measurement points during the design process. DPM operation times can be set up by the top design leader/project manager based on the project strategies and development plan.

Strategic DPM matrix weighting setup

A strategic DPM matrix weighting setup function was designed according to the second design principle – customization: strategic links between DPM operation and the collaborative design project. This function can support the design manager to set up weightings of the DPM criteria based on individual team member’s roles and responsibilities. Neely ³⁵ recommended methods to link DPM operations with a project’s strategy. They emphasized the importance of rational links between project objective and measurement criteria at each level of an organization. By following this theory, the strategic DPM matrix weighting setup function was developed to facilitate a customized feature that allows PM operations to be closely integrated with the project strategies. It allows the design managers to set up three layers of weighting: (1) project strategy–based weighting which is based on the project features; (2) stage-based weighting which is based on design objectives for each design and development stage and (3) individual-based weighting which takes into account the design staff’s job role and responsibilities. It also provides a flexible manner for design managers to emphasize priorities of each DPM criterion in the DPM matrix during a DPM operation process. To this end, the linkage between DPM and design strategies supports the DPM approach to produce trustworthy DPM results, in turn, to lead team members’ behaviour changes positively and encourage design collaboration towards the strategic project objectives.

Multi-feedback DPM mechanism

Based on the third design principle – interoperability: allowing all interested stakeholders into the operation – a multi-feedback DPM mechanism was developed to construct a comprehensive and balanced PM. The mechanism includes four DPM data collection channels: by the individual staff, by higher level design manager/directors, by colleagues at the same project structure level and by colleagues from a lower project structure level (Figure 4). This mechanism allows the collaborative design performance to be calculated objectively and fairly as it supports all the involved team members to participate and contribute to DPM operation from diverse disciplines. Meanwhile, each team member can compare their self-evaluation data and feedback from their colleagues so as to establish a better understanding of their working performance, strengths and weaknesses and respond positively via proper actions or behavioural changes. The web-based DPM platform will allow team members to provide PM feedback via multiple devices such as a computer, laptop or mobile phone.

Figure 4.

Multi-feedback DPM mechanism.

DPM database

After collecting all DPM data from collaborative design team members, the data will be used for measuring design performance and then saved in a DPM database. In the database, each design team member has a data set covering personal information data, DPM data and DPM results data. The personal information data include the member’s ID, name, position, job role and address. In terms of DPM data information, it has self-assessment DPM data, DPM data from a manager, DPM data from colleagues, DPM data from a lower level team member and a record of time and date. For management level staff, this part also has their DPM criteria design and weighting setup data in the DPM database. The third part, DPM result data, includes single DPM results based on the selected DPM criteria, total DPM results based on the criteria weighting setting, historical DPM data and a record of time and date. These data will be applied to identify team member’s features and calculate and present DPM results.

DPM engine

The DPM engine is responsible for calculating collaborative design performance based on the DPM input data from users and the DPM matrix weightings. Project managers need to build up a DPM matrix based on the collaborative design project’s objectives and features. Based on the DPM criteria, the DPM engine collects DPM criteria weighting setup data from different levelled managers. Then, it computes DPM scores for each indicator. The calculation method for different indicators is the same.

For each DPM indicator (dimension) P_d (d = 1,2, …, 5), based on the multi-feedback mechanism, the calculation method involves five steps. The first step is to get a design team member’s self-evaluation score for the indicator. The second step is to obtain a score from his or her line manager. In step 3, an average score from his or her peer colleagues is calculated. Similarly, in step 4, an average score from his or her lower level staff under his or her leadership will be obtained. Finally, the scores for the indicator will be summarized with weightings, which have been set up by managers. For example, if we want to calculate a middle manager’s performance for efficiency, there are five DPM sub-criteria in the efficiency indicator according to the DPM matrix. Here, W_i (i = 1,2, …, 5) means weightings for each sub-criterion, and W₁+W₂ + … + W₅ = 1. Now, we assume that the middle manager has N same level colleagues and Q staff under his or her leadership. And of course, he or she has a line manager as a leader. Here, N means the number of the same level colleagues and Q represents the number of lower level individual staff who is under the middle manager’s leadership. The calculation process is as follows:

• Step 1. A self-evaluation: the middle manager’s efficiency performance calculation based on his self-evaluation scores. S_i represents DPM data from the middle manager’s self-evaluation for each sub-criterion

$$E_s=\sum _{i=1}^5(S_i\times W_i)$$

(1)

• Step 2. An evaluation from manager: the middle manager’s efficiency performance calculation based on scores from his or her line manager (only one). Here, M_i means scores from his or her manager for each sub-criterion

$$E_M=\sum _{i=1}^5(M_i\times W_i)$$

(2)

• Step 3. An evaluation from peer colleagues: the middle manager’s efficiency performance calculation based on scores from his or her peer colleagues. Here, C_ji means scores from his or her colleague j for the sub-criterion i

$$E_C=\sum _{i=1}^5(\sum _{j=1}^N{C}_{\mathrm{ji}}\times \frac{W_i}{N})$$

(3)

• Step 4. An evaluation by the lower level staff: the middle manager’s efficiency performance calculation based on scores from his or her individual lower level staff. Here, I_ji means scores from his or her individual staff j for the sub-criterion i

$$E_I=\sum _{i=1}^5(\sum _{j=1}^Q{I}_{\mathrm{ji}}\times \frac{W_i}{Q})$$

(4)

• Step 5. Combining DPM scores from all four channels: as DPM data from each of the four channels may have a different impact on the final measurement outcome, the design manager can set up another layer of weightings W_s, W_c, W_m and W_I to indicate different weightings for different DPM input data groups: self-evaluation, colleagues, manager and lower staff. And W_s + W_c + W_m + W_I = 1. Thus, the middle manager’s final score for efficiency can be calculated via

$$E=E_s\times W_s+E_c\times W_c+E_M\times W_M+E_I\times W_I$$

(5)

After calculating scores for all indicators for each staff: efficiency (E), effectiveness (EE), collaboration (C), management skill (M) and innovation (I), a total DPM score for the staff will be figured out as an overall performance score with assigned priorities for each indicator from managers. Thus, the overall DPM score for a single PM round can be worked out via

$$P=E\times W_E+\mathrm{EE}\times W_{\mathrm{EE}}+C\times W_C+M\times W_{\mathrm{Ma}}+I\times W_N$$

(6)

where W_E, W_EE, W_C, W_MA and W_N present the priorities for the five DPM items, and $W_E+W_{\mathrm{EE}}+W_C+W_{\mathrm{Ma}}+W_N=1$ .

Finally, with an intention to reduce the differences between diverse team members’ and managers’ marking styles, the DPM tool can normalize the final DPM scores so as to provide a meaningful performance comparison across the design team members.

Here, A_N means the normalized design performance score and X presents the member of staff in the design team

$$A_N=\frac{P}{Max{(P_K)}_{K=1,\dots ,X}}\times 100\%$$

(7)

DPM results presentation

According to the design principle 4 – real-time feedback: providing a holistic analysis of the project development for decision making – a DPM results presentation component was designed and developed. It provided functions of displaying DPM results with multiple forms. Design managers can use a DPM tool to get support and recommendations from other team members enabling better decision making. Therefore, a DPM operation process can be embedded (or incorporated) in a project management loop to provide a holistic analysis of the project development. In this way, the DPM results presentation component can support design managers and designers to get real-time feedback to improve the collaborative design performance.

After the DPM engine calculates DPM data for each of the design team members, results can be displayed by numbers, single curve and comparison curves. According to the DPM calculation formulas within the DPM Engine, the DPM outcomes include six performance scores to present design team members’ performance, which include efficiency score, effectiveness score, collaboration score, management skill score, innovation score and total performance score. DPM operation timeframe can be planned according to a stage-gate-based design process. DPM data collection time points can be embedded into the design development process. Once the DPM has been run more than twice, the DPM outcomes can be visualized as lines or curves to present the overall DPM data that include historical and present data. These visual presentations can help design managers to predict a future trend of the project development from different perspectives, such as overall performance or just focus on innovation performance. Furthermore, based on the DPM data, the tool can also help the manager to analyse individual staff’s strengths and weakness by comparing their performance in efficiency, effectiveness, collaboration, management skill and innovation or compare performance of different team members. This will support the design managers to better understand their staff in a shorter time and take management actions accordingly, such as providing training programmes or changing team structures. The impact of these management actions on collaborative design performance can be reflected in the next round of DPM operations. After multiple rounds of DPM exercises, the curves will clearly indicate whether the project collaborative design performance has been improved.

Case study evaluation

The proposed DPM approach aims to measure project team’s collaborative design performance and in turn to provide useful information for the design manager and team members to identify and solve potential design issues during the design development process. In order to explore whether the DPM approach can be used to measure and improve collaborative design performance, a case study has been conducted with one of the top international design agencies in the United Kingdom (Table 2). The DPM approach was applied for a 4-month period with a collaborative design project, Project A. The Project A aimed to develop a new/updated product for a market extension.

Table 2.

Information about the case study sample.

Project	Organization type	No. of Staff in company	Operates in no. of countries	No. of Staff in the project
A	Design agency	160	32	7

In Project A, the project team had one top manager, two middle managers and fours designers, in total seven members (Figure 5). The top manager was in charge of the day-to-day project operation management and to make sure the project was running smoothly towards the strategic objective. The two middle managers A-1 and A-2 were responsible for leading creative teams to fulfil the brief requirement from their client. The four designers focused on design tasks such as product structural design, graphics design and materials for the project.

Figure 5.

Team structure of Project A.

The development process of the project was divided into five stages (Figure 6): strategy design, R&D, concept design, design development, design finalization and production. The DPM was implemented in the project management from the strategy design stage to the ending part of the production stage. Due to limited resources and technological support, the case study evaluation was implemented based on an off-line DPM approach demonstration on the authors’ laptop. All DPM activities, such as team setting up, matrix design, weighting design and PM data collection, were completed on the authors’ laptop. After the project manager A set up the DPM design for Project A, staff’s PM information was collected at their stage-based design review meetings. Instructions and support were provided from the researchers to all team members during the DPM activity process. In total, there were five rounds of DPM data collections. DPM feedback was presented to the project manager A first and then to the other team members. The feedback was also discussed in their team meetings to support project decision makings.

Figure 6.

DPM application during the Project A processes.

In order to uncover the case project design team’s opinions and attitudes towards to the proposed DPM approach, interviews were conducted at the end of the project. The interviews focused on evaluation of key features of the DPM approach that include whether the DPM approach could be (1) applied to measure collaborative design performance during a design process; (2) utilized to support all users, such as managers and designers; (3) implemented to produce reliable and fair DPM results through the multi-feedback DPM mechanism; (4) applied to a design process with different stage-based design strategies; (5) easily and flexibly implemented and (6) the overall quality of the proposed DPM approach and recommendations for further development. Six team members from Project A attended the interviews.

Based on the evaluation interviews, all the participants (N = 6) confirmed that the DPM approach helped the team to measure their collaborative design performance during the design development process. The flexibility feature allowed the top manager to integrate the DPM operation with the project team structure. One middle manager commented that the DPM approach supported him to comprehensively review and track the team members’ design performance according to the DPM criteria. This helped him to examine potential design problems and weakness of the team at an early stage in the design process, so they could fix the problem before it became too serious. Thus, they believe the DPM approach did help them to improve their design performance. All participants agreed the DPM approach supported not only the design managers but also the four designers to measure the team performance, gain a better understanding of their own performance and their colleagues’ expectations to them and direct them to improve their collaborative design performance during a design development process. The top manager indicated that the DPM approach was providing insightful information for him to better monitor and control quality of the design and the projects development. Most of the participants believed the DPM provided objective and real-time feedback via the multi-feedback mechanism. One designer further explained that the DPM approach avoided the measurement to be driven by a single person, which may lead to unfair results. It was also reported that this approach was more effective than the measurement approaches they had used previously, which was a manager-oriented measurement system. Participants agreed that the customized feature of this DPM approach allowed the managers to embed stage-based project strategies into the PM operation and improved accuracy and reliability of the DPM outcomes. One middle manager highlighted the weighting setup feature which could allow the DPM to work well with more complex and dynamic design projects. Three designers indicated that the weighting system helped them to understand priority tasks in their roles, which was not an expected benefit from the DPM approach.

In summary, based on the evaluative case study, most of the participants agreed that the DPM proposed approach can be used to measure and improve collaborative design performance by providing flexible and customized PM set up, balanced and multi-feedback measurement data collection, real-time feedback during a design development process and accurate and reliable DPM outputs.

Conclusion

Based on the trend of Industry 4.0, this study developed a smart DPM approach that offers flexible and customized operation, interoperability and intelligent real-time feedback features to measure, monitor and improve design team collaboration during a design development process. This approach was composed of a PM implementation setup mechanism, a multi-feedback DPM mechanism and a multiple DPM results presentation mechanisms. These mechanisms enable the proposed DPM approach to support dynamic team and project management based on the approach structure design. In contrast to traditional PM systems, the mechanism’s design allows the DPM approach to be used during a product design process and evidence found that it supported both design managers and designers in improving their design collaboration. Moreover, it also contributes to producing reliable DPM results by linking DPM with the stage-based design objectives, balancing DPM feedback from both design managers and designers and providing intelligent DPM results to guide design activities. The empirical industry evaluation case studies positively indicated that the proposed DPM approach could be used to support design managers and designers in measuring and improving collaborative design performance during a collaborative design project development process. Based on the suggestions from the evaluation case study, the future research will focus on investigations of impacts of collaborative design project features, such as big-/small-sized teams, long-/short-term projects and business types on DPM approach designs, if a generic computational intelligence–aided design framework could be utilized in a smart design process, and a further development of the DPM software.