A multiple triangulation analysis on the role of product development activities on product success

Abstract

The relationship between product development process and product success remains to confound many. Although most researchers agree that product development is one of the main factors affecting product success, many case studies performed since 1980s have shown different results. Some studies find a strong relationship between a product development activity and the product success, while some others discover the opposite. There is no generic result to be referred to further study or used for practices. Therefore, this study explores further the effect of product development activities on product success by conducting a multiple triangulation approach. This approach cross-examines three methods with three different data: meta-analyses that use data from a number of case studies previously published; analytic hierarchy process that takes data from product development experts’ judgment; and resource allocation analysis that utilizes companies’ operational data. The results show that the relationship between product development process and product success is unique for every case or group of cases. The relationship can be determined by utilizing resource allocation data and experts’ judgment, as shown by a general procedure proposed.

Introduction

Background

Product development is the first stage in the process of making products, comprising of creating the products’ idea, developing the design of the products, as well as designing the consecutive processes of making the products real and beneficial. Thus, the product development will drive product success, as proven by many case studies. ^{1 –13}

Researches on product development began in 1960s and started to significantly increase in 1980s, since the most cited article on product development and industrial success written by Cooper ¹⁴ was published. Some of those researches specifically discuss the effect of product development process on success through case studies. ^{1,3,4,7 –10,15} However, although the case studies basically refer to the same model, that is, a model of product development activities developed by Cooper and Kleinschmidt, ¹ the results are varied and are contradictive, as described in the following illustration.

The case studies mentioned above ^{1,3,4,7 –10,15} identify the relationship between quality or performance level of every activity in product development process and the product/industrial success or failure. Table 1 compares the results of the studies. The reference number in the first column refers to the references list. The activities presented by numerical code are defined according to Cooper and Kleinschmidt, ¹ as it is the most cited article, and the other articles are fully or partially referred to it. The activities are (1) initial screening, (2) preliminary market assessment, (3) preliminary technical assessment, (4) detailed market study/research, (5) business/financial analysis, (6) product development, (7) in-house process testing, (8) customers test of product, (9) test market/trial sell, (10) trial production, (11) pre-commercialization business analysis, (12) production start-up, and (13) market launch. Notation Y means that an activity influences success, F means that an activity may or may not influence success, and N means that an activity could not be proven to influence success.

Table 1.

Findings about the influence of product development activities on success.

Reference number	Object, industry type	Research region	Identification method	Activity
				1	2	3	4	5	6	7	8	9	10	11	12	13
1	Manufacture	Canada	% successful projects conducting the activity	Y	Y	Y	N	Y	Y	Y	Y	N	F	N	Y	Y
3	Chemical	USA, Canada, UK, Germany	Correlation between quality of activity and project success	Y	Y	N	Y	Y	N	N	Y	Y	Y	Y	Y	Y
4	Manufacture mix	Korea, Canada, China	Correlation between activity proficiency and project success	Y	Y	Y	Y	Y	Y	Y	Y	Y	Y	Y	Y	Y
15	Manufacture and services	USA	% best performance conducting high-quality activity	F	Y	Y	N	Y	Y	Y	Y	N	Y	F	Y	Y
7	Manufacture mix	China, Canada, Japan, Slovenia, South Korea	Correlation between activity proficiency and success measures	Y	Y	Y	Y	Y		Y	Y	Y				Y
8	Manufacture and services	USA	Association of new product quality and activity		Y	Y			Y	Y			Y		N
9	Manufacture	Taiwan, Indonesia	ANOVA between activities and success	N				Y		Y		Y				Y
10	IT companies	Swedish, South Africa, USA	Influence of activities on product success	Y	Y	N	N	N	Y	N	N	N		Y

ANOVA: analysis of variance.

As shown in Table 1, Cooper and Kleinschmidt ^1,3 and Cooper ¹⁵ show that the different observation object groups will give different conclusions. In manufacturing industries, ^1,15 market-related activities could not be proven to affect success, while in chemical industries, ³ the activities not proven to affect success are technical activities.

In other part of Table 1, the studies conducted by Mishra et al. ⁴ and Jin and Li ⁷ present the same conclusions, as they observed similar objects, that is, industry mix, including services, that are mostly located in Asia. In another part, Millson and Wilemon ⁸ who observe industry mix including services in US region show different results. However, the similar object groups may also give different results, such as Cooper ¹⁵ and Millson and Wilemon. ⁸ They both observe manufacturing and service industries in the same region, United States, but the results are different.

In the last part of Table 1, Wang et al. ⁹ and Khurum et al. ¹⁰ give different findings in details, as they observed different objects: manufacturers ⁹ and IT companies. ¹⁰ However, they show similar characteristic as the recent studies, that is, that the most important is about market and customer.

Problem and research benefit

There is no general conclusion could be taken from the aforementioned comparison. The way the product development drives product success remains uncharted. Those who need an information about the most influencing product development activity on success for a further study, for example, will stumble on choosing which results of the studies could be referred to. For practices, it is important to know which activities of product development will drive the success of the product, because the product development cost is high ¹⁵ and should be effectively allocated. Thus, a deeper analysis on the role of product development activities on success is required to generate a beneficial result.

Research questions

The findings from the comparison through some examples shown in Table 1 lead to an indication that every case may have its own characteristic of the relationship between product development process and success. If this indication is right, it is important to know how to determine the relationship for every case. Hence, these indications lead to the following questions:

Q1: Is the role of the product development activity on the product success specific for every case?

Q2: How does a general procedure to determine which activities of product development affect the product success?

To answer the questions, a cross-examination on three data sources and methods is performed in this study. The symptom, the behavior, as well as the pattern are elaborated. The approach applied to find the answers is a triangulation.

The following discussion begins with the triangulation approach and the context of the research. The next part is the data collection, the analyses, and the findings of the three methods consecutively. A cross-evaluation on the three methods is performed, followed by the discussion of the result, and closed by the conclusion section.

Triangulation approach

The lexical definition of triangulation is the division of a map or plan into triangles for measurement purposes, or the calculation of positions and distances using this method, working from a fixed base. ¹⁶ In the context of research, triangulation means the use of two or more approaches in research aspects to strengthen the research finding. Triangulation could decrease the weaknesses of a single approach and contribute to understanding further the phenomenon. ¹⁷ Theoretical perspectives, ¹⁸ methodological approaches, ^18,19 data sources, ²⁰ investigators, ¹⁸ or data analysis methods ²¹ are some aspects usually triangulated.

Regarding the questions Q1 and Q2 that could not be answered by previous studies, ^{1,3,4,7 –10,15} this study analyzes three different data types from three different sources using three different approaches. The three data are as follows: (1) the results from a number of published researches which are analyzed by meta-analyses; (2) experts’ judgment taken from experienced product development practitioners and analyzed using analytic hierarchy process (AHP); and (3) product development operational data taken from some product development projects in three companies and evaluated through resource allocation analysis using activity base costing (ABC) approach. Thus, this is a triangulation across recognized data, subjective data, and objective data.

Context

The concern of this study is about the role of product development activities on product success. Thus, the discussion in this article is focused on and limited to the relationship between product success and product development process or activities. The model of product development process analyzed here refers to a model developed in the previous work, ²² in which the model is developed based on other established models ^{1,15,23 –26} and observations in some companies. Table 2 presents this model.

Table 2.

Product development activities. ²²

Activity		Sub-activity
D 1	Development of product idea	D 11	Idea exploration from national forum
		D 12	Idea exploration from international forum
		D 13	Analysis of customer voice
		D 14	Technical evaluation on current product
		D 15	Review on market chance/opportunity
D 2	Establishment of product idea	D 21	Development and selection of product alternatives
		D 22	Detail analysis on market condition
		D 23	Business feasibility analysis
D 3	Detail design of product	D 31	Virtual design of product
		D 32	Physical prototyping
		D 33	Virtual design of manufacturing process
		D 34	Simulation of production process and management
D 4	Test of the design of product	D 41	Physical test of product
		D 42	Test of product by customer
		D 43	Improvement of the design
		D 44	Financial analysis
		D 45	Test of production
D 5	Launch of product	D 51	Production start-up
		D 52	Promotion

Meta-analyses

The mechanism of meta-analysis

Meta-analysis is a technique to comprehensively analyze the relationship between two variables by statistically wrapping up the values resulted from an adequate number of studies/cases. ²⁷ It is usually utilized as a tool for a systematic literature review. Meta-analysis tries to diminish the effect of the artifacts of the findings from a number of cases, in order to acquire a general and reasonable measure of relationship. ^27,28 The artifacts include, for instance, sampling error, measurement error, computational error, or typographical errors. The meta-analysis recognizes at least sampling error and measurement error.

A meta-analysis is performed through the following steps: ²⁷ (1) collecting the data from an adequate number of samples and cases; (2) calculating the weighted average of the data; (3) correcting the average by sampling errors; (4) correcting the average by measurement errors; and (5) determining the level of significance. In case the expected level of significance has not been reached, the process goes back to step 1. A case defined in step 1 means a study in which a datum can be taken. Thus, a research article may contain more than one case. Every case generates the datum based on a number of samples.

The meta-analysis commonly performed is a correlation meta-analysis, which is applied in this study. A series of meta-analyses are performed to find the correlation between every product development activity defined in Table 2 and product success.

The concern of the meta-analyses conducted here is to answer the research question Q1, addressing the confusing phenomena shown in Table 1. The indication shown by the phenomena is that the role of product development activity on success may be influenced by industry type (as the representation of the product type); or geographical area (as the representation of the customer type); or time (as the representation of the trend). This indication will figure out by hypotheses testing. Supposed that D_i is activity i, D_ij is sub-activity j of activity i (see Table 2) and S is product success, the hypotheses to be tested are

H1: All D_i and D_ij are correlated to S.

H2: The correlations between D_i or D_ij and S are varied by industry type.

H3: The correlations between D_i or D_ij and S are varied by geographical area.

H4: The correlations between D_i or D_ij and S are varied by time.

The data for meta-analyses

The data for meta-analyses came from a number of articles ^{1 –9,11 –14,21,29 –97} acquired through a systematic review that considers some criteria, that is, the required content, the year of publication, the publication sources, and the citation number. The followings are the explanation of the review criteria.

The content

The articles acquired for the meta-analyses must contain quantitative information about the relationship between at least one D_ij and S, in the form of correlation values, F-statistic values, t-statistic values, R ² values, or other effect sizes. All the values other than correlation will be converted into correlation. ^27,98

The year of publication

The range of publication of the articles involved in meta-analyses is from 1980s, the publication time of the most cited article on product development activities, ¹ until 2015. This long range of publication year will support the analysis of time variation. Around 75% of the collected articles are published after year 2000.

The publication sources

The articles involved in meta-analyses are firstly taken from the Science Direct, in which reputable science, engineering, and business articles are available. Once a high cited and qualified article found, the other relevant articles are searched from the related articles in the reference list and the citation data.

The number of citation

The average number of citation of the articles involved in theses meta-analyses are 714, 255, 121, and 23 for the articles published during 1979–1990, 1991–2000, 2001–2010, and 2011–2015, respectively. Around 95% of the articles are cited publication. The other 5% noncited publication are dissertation and the last three year articles.

The finding from meta-analyses

Tables 3 to 6 show the results of the meta-analyses. K is the number of cases involved in the analysis, N is the number of samples, ρ is the cleansed correlation of D_i or D_ij and S, and σ_ρ is the standard deviation of ρ. The average reliability of the D_i , D_ij , and S is denoted by α_D and α_S , respectively.

Regarding hypothesis 1, Table 3 presents the overall results of the meta-analyses. It can be seen in Table 3 that all the D_i and D_ij are significantly correlated to S. Thus, hypothesis 1 is accepted.

Table 3.

Meta-analyses on the correlation between D_i or D_ij and S.

Di , Dij	K	N	αD	αS	ρ	σρ
D 1	28	5195	0.93	0.95	0.30***	0.08
D 2	21	3726	0.92	0.96	0.34***	0.08
D 3	23	3280	0.90	0.87	0.32*	0.13
D 4	26	3798	0.93	0.93	0.38**	0.14
D 5	17	3016	0.93	0.96	0.34*	0.16
D 11	30	4419	0.90	0.90	0.43*	0.17
D 12	23	4254	0.93	0.95	0.43*	0.19
D 13	36	5538	0.90	0.91	0.34**	0.10
D 14	71	7853	0.87	0.90	0.49*	0.20
D 15	42	6144	0.92	0.93	0.46**	0.17
D 21	48	6341	0.89	0.87	0.48*	0.21
D 22	42	6455	0.93	0.94	0.51**	0.18
D 23	30	4702	0.91	0.92	0.42*	0.18
D 31	40	4749	0.91	0.91	0.42*	0.17
D 32	44	5655	0.90	0.90	0.40*	0.17
D 33	31	3406	0.91	0.91	0.38**	0.13
D 34	62	6897	0.93	0.91	0.42*	0.18
D 41	60	6039	0.93	0.93	0.40*	0.16
D 42	52	6709	0.94	0.93	0.45**	0.15
D 43	31	3684	0.90	0.93	0.43**	0.15
D 44	44	4726	0.89	0.88	0.53*	0.21
D 45	37	3481	0.92	0.94	0.36**	0.13
D 51	27	3258	0.89	0.92	0.41*	0.16
D 52	65	10,704	0.90	0.92	0.57**	0.19

*Significantly correlated at p < 0.05.

**Significantly correlated at p < 0.01.

Table 4 to 6 present the results of the meta-analyses based on industry type, geographical area, and publication year, respectively, to test hypotheses 2, 3, and 4. The difference between two correlation values from two different groups (ρ ₁ and ρ ₂) are identified using the tests of two means procedure, ⁹⁹ through the z-statistic value. The z-statistic is calculated from the related ρ, σ_ρ , and K. The analyses are performed on some D_i and D_ij , regarding to the availability of the decent data.

Table 4.

The difference (z-test) between two related D_i -S and D_ij -S correlations based on industry type.

Di , Dij	Type 1 (all cases)					Type 2				z1–2
	K1	N1	ρ1	σρ 1	Type	K2	N2	ρ2	σρ 2
D 13	36	5538	0.34**	0.10	Chemical based	4	1328	0.38**	0.13	–0.61
D 34	62	6897	0.42*	0.18		2	902	0.20*	0.08	3.64c
D 44	44	4726	0.53*	0.21		2	206	0.38***	0.04	3.67c
D 14	71	7853	0.49*	0.20	Mechanical based	3	241	0.59***	0.06	–2.45a
D 51	27	3258	0.41*	0.16		2	192	0.56***	0.05	–3.36c
D 3	23	3280	0.32*	0.13	Electrical based	7	815	0.38	0.11	–1.20
D 11	30	4419	0.43*	0.17		6	488	0.61***	0.07	–4.10c
D 14	71	7853	0.49*	0.20		4	296	0.57***	0.05	–2.28a
D 21	48	6341	0.48*	0.21		8	624	0.41*	0.12	1.41
D 23	30	4702	0.42*	0.18		6	460	0.47***	0.05	–1.31
D 42	52	6709	0.45**	0.15		10	1044	0.48**	0.14	–0.55
D 45	37	3481	0.36**	0.13		4	296	0.54***	0.04	–6.20c
D 51	27	3258	0.41*	0.16		2	192	0.56***	0.05	–3.36c
D 52	65	10,704	0.57**	0.19		8	380	0.51***	0.07	1.97a

^aSignificantly different at p < 0.05.

^bSignificantly different at p < 0.01.

^cSignificantly different at p < 0.001.

*Significantly correlated at p < 0.05.

**Significantly correlated at p < 0.01.

***Significantly correlated at p < 0.001.

In Table 4, the results of the z-tests show that 9 out of 14 D_i -S and D_ij -S correlations of the chemical-based industries, mechanical-based industries, and electrical-based industries (ρ ₂) differ from the all-cases’ correlations (ρ ₁). In other words, there is a variation among the differences of the correlations. Thus, hypothesis 2 is accepted.

Table 5 compares the D_i -S and D_ij -S correlations based on geographical area. There are four areas to be considered: America, Europe, Asia, and Africa. The America-Europe comparison shows that the D_i -S and D_ij -S correlations from the cases in both areas are nearly the same. Only 2 out of 10 z-tests show the difference. The America-Asia comparison shows that 13 out of 16 z-tests indicate different correlations, whereas the Europe-Asia comparison shows three different correlations from the seven z-tests performed. In summary, the D_i -S and D_ij -S correlations varied by geographical area, and hypothesis 3 is accepted.

In Table 6, the D_i -S and D_ij -S correlations are compared based on time or publication year of the articles in this case. The cases are classified into two groups, that is, case studies published in year 2000 and earlier and case studies published after year 2000. The z-tests show that 10 out of 12 D_i -S and D_ij -S correlations of the two groups are significantly different. This result supports hypothesis 4 to be accepted.

Table 5.

The difference (z-test) between two related D_i -S and D_ij -S correlations based on geographical area.

Di, Dij	Area 1	K1	N1	ρ1	σρ 1	Area 2	K2	N2	ρ2	σρ 2	z1–2
D 1	America	19	3431	0.22**	0.07	Europe	7	1298	0.17***	0.02	3.18b
D 4		16	2551	0.23***	0.08		11	1436	0.43*	0.18	–1.76
D 13		13	2158	0.28***	0.07		10	2161	0.31*	0.14	–0.69
D 14		53	5497	0.48*	0.22		12	1562	0.44*	0.19	0.61
D 21		19	2222	0.55***	0.11		8	1045	0.48*	0.22	0.90
D 22		14	2073	0.36*	0.14		3	332	0.45***	0.06	–1.74
D 32		30	3687	0.38**	0.14		7	962	0.38*	0.15	0.04
D 41		39	3555	0.41***	0.10		12	1562	0.44*	0.17	–0.58
D 43		13	1369	0.32***	0.06		13	1700	0.39*	0.19	–1.18
D 52		27	3561	0.40**	0.15		10	3201	0.78***	0.14	–7.18c
D 2	America	9	1686	0.28**	0.09	Asia	11	1969	0.37***	0.02	–2.82b
D 11		18	2241	0.63**	0.23		11	1868	0.41**	0.16	3.04b
D 12		13	1848	0.31**	0.12		9	1606	0.43***	0.11	–2.45a
D 14		53	5497	0.48*	0.22		7	954	0.50***	0.08	–0.47
D 15		15	2128	0.32**	0.12		25	3858	0.51**	0.18	–4.19c
D 21		19	2222	0.55***	0.11		18	2411	0.54**	0.21	0.15
D 22		14	2073	0.36*	0.14		27	4256	0.55**	0.17	–3.81c
D 31		24	2753	0.38**	0.12		12	1678	0.51**	0.17	–2.32a
D 32		30	3687	0.38**	0.14		6	726	0.39***	0.09	–0.25
D 33		14	1358	0.32***	0.05		13	1685	0.45***	0.13	–3.26b
D 34		24	2631	0.29***	0.05		30	3709	0.54***	0.16	–8.08c
D 42		28	3517	0.37***	0.10		22	2940	0.47*	0.18	–2.43a
D 43		13	1369	0.32***	0.06		7	855	0.44***	0.07	–3.64c
D 45		34	3190	0.34**	0.11		2	288	0.54***	0.08	–3.35c
D 51		22	2625	0.34**	0.13		5	633	0.52***	0.02	–6.06c
D 52		27	3561	0.40**	0.15		31	4381	0.56***	0.13	–4.37c
D 13	America	13	2158	0.28***	0.07	Africa	3	660	0.47***	0.04	–6.45c
D 14	Europe	12	1562	0.44*	0.19	Asia	7	954	0.50***	0.08	–0.92
D 21		8	1045	0.48*	0.22		18	2411	0.54**	0.21	–0.71
D 22		3	332	0.45***	0.06		27	4256	0.55**	0.17	–2.06a
D 32		7	962	0.38*	0.15		6	726	0.39***	0.09	–0.21
D 43		13	1700	0.39*	0.19		7	855	0.44***	0.07	–0.90
D 44		4	476	0.42***	0.10		19	2438	0.59***	0.10	–3.05b
D 52		10	3201	0.78***	0.14		31	4381	0.56***	0.13	4.35c
D 13	Europe	10	2161	0.31*	0.14	Africa	3	660	0.47***	0.04	–6.45c

^aSignificantly different at p < 0.05.

^bSignificantly different at p < 0.01.

^cSignificantly different at p < 0.001.

*Significantly correlated at p < 0.05.

**Significantly correlated at p < 0.01.

***Significantly correlated at p < 0.001.

Table 6.

The difference (z-test) between two related D_i -S and D_ij -S correlations based on publication year.

Dij	Year 1 (≤2000)				Year 2 (>2000)				z1–2
	K1	N1	ρ1	σρ 1	K2	N2	ρ2	σρ 2
D 1	11	2225	0.25***	0.06	17	2970	0.31***	0.09	–1.80
D 2	7	1421	0.29***	0.07	14	2305	0.35***	0.08	–1.88
D 3	4	757	0.23***	0.05	19	2523	0.31*	0.14	–1.87
D 4	9	1827	0.27***	0.05	17	1971	0.45**	0.16	–4.37c
D 5	12	2428	0.24***	0.05	5	588	0.63***	0.13	–6.33c
D 12	7	1313	0.28*	0.09	16	2941	0.48*	0.20	–3.34c
D 15	16	2386	0.36*	0.15	26	3758	0.49**	0.16	–2.56a
D 21	15	2328	0.37*	0.15	33	4013	0.48*	0.22	–1.97a
D 22	16	2417	0.41*	0.18	26	4038	0.53***	0.16	–2.31a
D 31	12	1525	0.31*	0.13	28	3224	0.44**	0.17	–2.63b
D 33	11	1113	0.24***	0.02	20	2293	0.41**	0.14	–5.08c
D 34	24	2078	0.43*	0.20	38	4819	0.43*	0.18	0.06
D 41	6	659	0.26**	0.09	54	5380	0.41*	0.16	–3.47c
D 42	14	2088	0.41***	0.07	34	4237	0.41**	0.16	–0.07
D 43	6	882	0.40***	0.09	25	2802	0.42**	0.16	–0.44
D 51	15	1608	0.33*	0.13	12	1650	0.40**	0.14	–1.32
D 52	20	2472	0.34*	0.14	45	8232	0.62***	0.17	–7.08c

^aSignificantly different at p < 0.05.

^bSignificantly different at p < 0.01.

^cSignificantly different at p < 0.001.

*Significantly correlated at p < 0.05.

**Significantly correlated at p < 0.01.

***Significantly correlated at p < 0.001.

To sum up, all the comparisons show that the role of product development activities on product success is specific for every group of cases. It confirms the phenomena shown in Table 1 and answers a part of question Q1. However, still, the meta-analysis results as well as every study involved could not definitely answer if the role of product development activities on product success is specific for every case. In other words, the results of meta-analyses are still confusing to be referred to, especially for practices.

Meta-analyses presented above provide the most likely D_i -S and D_ij -S correlations, but they cannot explain the unique character of a case. Therefore, the upcoming analyses will use observational data.

AHP for experts’ judgment

The mechanism of AHP

AHP is basically a process for decision-making. ^100,101 This approach is designed to combine the rational and the intuitive aspects that could not be quantitatively presented, to choose a best decision upon some alternatives based on some criteria. Some problems usually solved by AHP are priority/weight setting, alternative generation, best policy decision, requirement determination, resource allocation, risk assessment, performance measurement, system design, system stability evaluation, optimization, planning, and conflict resolution. ¹⁰⁰

In this study, the AHP is used to find weights, that is, the influence level of product development activities on product success, according to experts’ judgment. The results of AHP will be used to elaborate the special characteristic of the relationship between the product development activities and product success, to accomplish answering question Q1.

The AHP performed in this study begins with the data collection through a questionnaire of pairwise comparison among the influence level of product development activities and sub-activities (as defined in Table 2). The respondents (the experts) are asked to compare all combination or pairs of activities. The results then are converted into a scoring system and input to a pairwise comparison matrix. Next, the weighted averages of the influence level are computed and the results are normalized. The AHP computation in this study used a software designed by Goepel. ¹⁰² The result of AHP computation is feasible if the consistency ratio (CR) is less than 0.1.

Data for AHP

The questionnaires described above are accomplished by nine product development experts from three local companies in Indonesia. The experts meant here are the companies’ personnel who have long years of experience in doing practices related to product development. The judgments from the experts are assumed to be a reference, as their long experiences give strong intuition related to the projects performed in their company. The three companies are hospital equipment manufacturer, special paper manufacturer, and furniture manufacturer, named as A, B, and C, respectively. Table 7 shows the profile of the experts.

Table 7.

Experts’ profile.

Expert	Occupation at the time of observation	Years of experience in product development- related field	Company	Business
E1	Sub-division head of engineering	23	A	Hospital equipment manufacturer
E2	Engineering division head	12
E3	Vice director of engineering	16
E4	QA division head	12
E5	Export marketing manager	9
E6	Technical support	7	B	Special papers manufacturer
E7	Technical support	5
E8	Owner and marketing manager	12	C	Furniture manufacturer
E9	Marketing staff	10

Finding from AHP

The results of the weight computation using AHP ¹⁰² are shown in Table 8 with 0.001 ≤ CR ≤ 0.068, fulfilling the minimum required CR. The weights represent the level of influence of activities (D_i ) and sub-activities (D_ij ) on S.

Table 8.

The importance of activity for product success according to the experts’ judgment.

Di	Weight, at company			Dij	Weight, at company
	A	B	C		A	B	C
D 1	0.27	0.20	0.10	D 11	0.05	0.03	0.05
				D 12	0.10	0.02	0.05
				D 13	0.41	0.35	0.27
				D 14	0.15	0.31	0.45
				D 15	0.29	0.29	0.18
D 2	0.19	0.20	0.14	D 21	0.38	0.13	0.40
				D 22	0.35	0.12	0.33
				D 23	0.27	0.75	0.27
D 3	0.22	0.22	0.24	D 31	0.20	0.10	0.12
				D 32	0.61	0.54	0.62
				D 33	0.10	0.17	0.08
				D 34	09	0.19	0.18
D 4	0.16	0.21	0.46	D 41	0.48	0.08	0.10
				D 42	0.06	0.10	0.27
				D 43	0.23	0.11	0.38
				D 44	0.13	0.44	0.11
				D 45	0.10	0.27	0.14
D 5	0.16	0.17	0.06	D 51	0.37	0.83	0.73
				D 52	0.63	0.17	0.27

As shown in Table 8, every company has its unique result, as it produces different product characters. The experts from company A say that the development of product idea (D ₁) is the most influencing activity on product success. The experts of company B opine that the detail design of product (D ₃) is giving the highest impact on product success. In company C, the experts think that the test of the design of product (D ₄) is the most important activity to trigger product success.

The characteristic of product is one of the causal factors of the uniqueness. Company A, which produces hospital equipment, must be focused on the development of product idea. Innovation on the function and features of the hospital equipment is the most important concern to win the market. Technical aspect is not too difficult to handle because the technology change in the production process is relatively slow. In company B, detail design of product is the most important to trigger product success. Special paper products like security paper and smart card must be highly accurate and highly precise. The technical aspects take the main role in this case. Company C, a wood products manufacturer, concerns on the test of the design of product, as it has to serve its customer regarding to the customized order. Many corrections, modifications, and adjustments are required even during installation and finishing of the products.

According to the industry grouping in the meta-analyses, company A and company C are included in the same group (mechanical-based industry). However, the most influencing activities of those companies are different. This result answers the question Q1: The role of product development activity on product success is specific for every case.

Up to this point, the question Q2 has not been answered. The procedure to decide the most influencing activity on product success must be evaluated by analyzing the behavior of the activities in detail. The analysis must use operational data, as the following analysis.

Resource allocation analysis

The mechanism of resource allocation analysis

The resource allocation analysis here is dedicated to answer question Q2, to find a procedure to determine which product development activities influence product success. Determining the influencing activity in a case can be conducted by identifying the pattern of the relationship between the product development activities and product success, in order that the way the activities influencing the product success can be found out. Thus, both the level of the activities and the level of product success must be measured as the following.

Some researchers calculate the resource consumed for a product development process to measure the level of the process. ^{74,103 –105} In this study, the resources allocated for every activity of some finished product development project are measured. The approach used to compute the allocated resources is ABC. ¹⁰³ The ABC is modified for product development process and the steps are as follows: (1) identification of the cost centers, (2) identification of the costs, (3) allocation of the costs in every cost center, (4) identification of activities involved, (5) identification of the activity-related cost drivers, (6) computation of the activity costs in every cost center, and (7) computation of the overall activity cost. Furthermore, the product success is measured based on the cumulative sales volume, the market share, the cumulative profit, the market life, and the payback period, as recommended by some previous studies. ^15,106,107

Analyzing the resources consumed/allocated for every activity of product development process and then correlating them to the product success will give information about the importance of every activity on product success. Then, by observing some product development projects of similar products, the pattern of the relationship between the level of product development activity and the level of success can be measured in order to answer the question Q2.

Data for resource allocation analysis

The resources consumed by every activity of product development are measured by a series of direct and indirect interviews using structured questions list. The data are collected from nine product development projects in the three companies previously mentioned. Table 9 presents these projects.

Table 9.

Projects observed.

Company	Business	Project	Product
A	Hospital equipment manufacturer	A1	Manual bed type 1
		A2	Manual bed type 2
		A3	Manual bed type 3
		A4	Automated bed
B	Special papers manufacturer	B1	Certificate paper
		B2	Voucher card
C	Furniture manufacturer	C1	Hotel interior
		C2	Mass order furniture
		C3	Personal order furniture

In general, the data consist of the man power involved, the machines and tools utilized, the material and supplies consumed, the duration of activity, and the time utilization.

Finding from resource allocation analysis

The computations of resource allocation on activities are carried out for every project. The profile of resource allocation on product development process for four projects in company A, two projects in company B, and three projects in company C is presented in Figure 1. Due to the space limitation, the resource allocation on the activities is not presented here.

Figure 1.

The profile of resource allocation on product development activities.

The influence of the activity on product success will be portrayed by analyzing the pattern of the relationship between the level of resource allocation on every activity and sub-activity to the related product success level. The level of resource allocation on an activity may significantly relate to product success, or may weakly relate to product success, or may not have a relation to product success.

A scoring system in the range of 0–10 is applied to evaluate the relationship, as presented in Table 10. Score 10 means the possible highest value of resources allocated in an activity, defined for every company. Score below 10 is obtained by linear interpolation. The blank cell in Table 10 means the associated project does not perform the activity.

Table 10.

The existence of the product success–resource allocation (S–D_i ) relationship.

Item	Company A					Company B			Company C
	A1	A2	A3	A4	S–Di	B1	B2	S–Di	C1	C2	C3	S–Di
S	7.94	4.50	6.57	5.07		0.51	7.19		2.15	9.43	6.67
D 1	8.76	3.00	8.96	6.47	*	0.14	0.14		1.06	0.20	0.01
D 2	2.44	1.13	6.22	0.10	*	0.14	0.14		0.06	0.03	0.00
D 3	2.67	1.01	3.22	2.53	*	0.02	0.04	*	0.58	0.07	0.04
D 4	1.13	1.45	1.61	3.04		4.51	8.85	*	8.84	0.09	0.05
D 5	2.59	0.94	1.92	3.91		4.69	1.66		0.52	0.93	0.31	*
D 11	7.16	0.42	4.08	3.68	*	–	–		1.05	0.14	–
D 12	6.76	2.80	7.92	6.32	*	–	–		–	–	–
D 13	–	0.23	0.15	0.08		–	–		0.01	0.00	0.00
D 14	–	0.23	1.96	0.23		–	–		0.01	0.06	0.01	*
D 15	0.09	1.13	0.23	0.05		0.14	0.14		–	–	–
D 21	3.87	0.68	9.95	0.15	*	–	–		0.05	0.02	–
D 22	–	0.90	–	–		–	–		–	–	–
D 23	0.04	0.23	–	–		0.14	0.14		0.01	0.01	0.00
D 31	0.51	0.31	2.10	1.89	*	–	–		0.10	0.02	0.02	*
D 32	1.17	0.79	2.00	2.14	*	0.01	0.03	*	0.48	0.04	0.03
D 33	0.18	0.23	1.05	0.02		–	–		–	–	–
D 34	2.41	0.30	–	–		0.01	0.01		–	0.01	–
D 41	1.08	0.30	1.23	0.94	*	–	–		0.43	0.01	0.01
D 42	–	1.35	–	1.88		4.51	8.85	*	0.05	–	0.01
D 43	0.66	0.11	1.28	2.05		–	–		8.35	0.06	0.03
D 44	0.06	0.23	0.07	0.00		0.00	0.00		0.00	0.00	0.00
D 45	–	0.34	–	–		–	–		–	0.03	–
D 51	2.25	0.38	3.07	3.24	*	4.69	1.66		0.52	0.83	0.21	*
D 52	1.90	1.13	–	3.02	*	–	–		–	0.10	0.10

*Relationship exists.

The existence of the relationships (* signs in Table 10) are taken from regression analysis. Figure 2 shows the examples of the relationship pattern of an influencing activity (D ₁) and a noninfluencing activity (D ₄) on product success in company A.

Figure 2.

Example of the relationship between resource allocation and product success.

As illustrated in Figure 2, resource allocation on D ₁ in company A is in line with product success (R ² = 0.79). Otherwise, the resource allocated to D ₄ does not seem related to product success (R ² = 0.29). To sum up, by plotting the relationship between resource allocation on an activity or sub-activity and its related product success, the way an activity influencing product success could be examined.

Additionally, in case the profile of the relationship between resource allocation level and product success level is not linear, a procedure of linear transformation before the regression process is required. Figure 3 gives an example of this case, the activity D ₂ of product development project in company A.

Figure 3.

Example of the nonlinear relationship.

In summary, the variations of the existence of the S–D_i and S–D_ij relationship among the cases presented in Table 10 affirm the findings from the AHP. Moreover, the pattern of the S–D_i and S–D_ij relationship as illustrated in Figures 2 and 3 can be used to identify the influencing activity on product success. It answers the question Q2.

Cross-examination on experts’ judgment and resource allocation analysis

Although both question Q1 and question Q2 have been accomplished, there is a phenomenon to be noticed further, that is, that the influencing activity identified by AHP differs from the influencing activity identified by resource allocation analysis. Thus, a cross-examination on the two approaches is conducted to explore this phenomenon through the following analysis.

In Table 11, the influence levels of product development activities on product success according to the experts’ judgment are compared to the resources allocation levels and connected to the existence of S–D_ij relationship. The adjusted scores (0–10) of the influence level of product development activities or sub-activities (are computed from Table 8) and the adjusted scores of the level of resource allocation on activities or sub-activities (are computed from Table 10). Score around 0–5 is categorized as “low,” and score around 5–10 is categorized as “high.” There are four types of combination of activity influence score (from experts’ judgment) and resources level score (from resource allocation analysis). The first is the combination of high influence score and high resources level. The second is the combination of high-importance score and low resources level, and the third is the combination of low-importance score and low resources level. The last is the combination of low-importance score and high resources level.

Table 11.

Experts’ judgment, resource allocation level, and existence of S–D_{i .}

Company A				Company B				Company C
SA	EJ	RA	S–Di	SA	EJ	RA	S–Di	SA	EJ	RA	S–Di
D 11	2.74	7.98	*	D 15	8.47	2.97		D 11	0.57	5.17
D 12	5.29	10.00	*	D 23	9.30	2.97		D 13	3.23	2.51
D 13	10.00	2.69		D 32	10.00	2.55	*	D 14	5.13	2.70	*
D 14	6.02	4.56		D 34	5.88	2.52		D 21	5.60	2.66
D 15	8.25	3.68		D 42	5.02	10.00	*	D 23	5.03	2.55
D 21	8.06	7.81	*	D 44	9.61	2.50		D 31	2.58	2.85	*
D 22	7.75	3.11		D 51	8.65	7.05		D 32	7.01	3.93
D 23	6.84	2.50						D 34	3.81	2.51
D 31	5.36	5.46	*					D 41	5.37	3.68
D 32	8.80	5.77	*					D 42	8.09	2.65
D 33	3.34	3.66						D 43	10.00	10.00
D 34	3.11	4.84						D 44	5.55	2.50
D 41	7.50	5.16	*					D 45	6.11	2.56
D 42	1.77	5.09						D 51	3.19	5.42	*
D 43	5.50	5.29						D 52	1.21	3.01
D 44	4.20	2.59
D 45	3.14	2.57
D 51	5.93	6.45	*
D 52	7.52	5.76	*

SA: sub-activity; EJ: experts’ judgment; RA: total resource allocation.

*Relationship exists.

In general, with some exceptions, when a high-importance activity is highly funded, this activity looks to influence product success. Otherwise, a high-importance activity looks to have no impact on product success when the resources allocated on the activity are low. High level of resource allocation on low-importance activity may or may not affect product success. Low level of resource allocation on low-importance activity will give no effect on product success. It is not necessary to allocate high level resources on low-importance activity, as it will be a waste. In summary, to trigger the product success, resources should be highly allocated to the most important activities. The resources for the low-important activity, otherwise, should be minimized for effective funding.

Figure 4 shows the quadrant presentation of these phenomena. In Figure 4, 5 out of 41 points (12.2%) are exceptions from the conclusion above, as shown by the red balls far below the middle line and two blue boxes far above the middle line. This deviation may be caused by the error in data collection, or the “not yet”-relevant experts’ judgment because of some radical changes in the system.

Figure 4.

Activity importance, resource allocation level, and contribution to product success.

From the cross-examination on experts’ judgment and resource allocation analysis, it can be concluded that to determine the influencing product development activities on product success, the two aspects, that is, experts’ judgment and the level of resource allocation history, must be involved. The level of resource allocation history will portray the inner behavior of the system, while the experts’ judgment will intuitively accommodate the nonhistorical character of the system.

Furthermore, the following procedure is developed to clear up the idea described above (Figure 5). This preliminary procedure basically determines the influencing product development activities by experts’ judgment and resource allocation analysis.

Figure 5.

A preliminary procedure to determine the influencing product development activities on product success.

Further discussion

As mentioned in the beginning of the article, the main problem discussed in this article is that the previous studies on the role of product development activities on product success are confusing to be referred to, especially for practices. A manufacturing company, for example, wanting to decide which part of product development process should the money spend on mostly, will get confused in finding which study to be referred to. Some related case studies in manufacturing field conclude different results.

Hence, the way to solve the problem can be performed by developing a general mechanism to identify the influencing activities for every case. No study discusses this mechanism this far. Thus, this study triangulates three different data sources using three different methods to explore the behavior of the relationship between product development activities and product success, in order to find the answer for the problems.

An important and beneficial insight drawn from this triangulation process is that historical data of resource allocation on product development process, altogether with the judgment from the experts, can be used to analyze the role of product development activities on product success. The result of the analysis can help companies to decide the focus of product development funding, or in other words, to decide which activities of product development should be highly funded because they potentially increase the product success.

Conclusions and further opportunity

A triangulation performed in this study on meta-analyses result, experts’ judgment, and resource allocation analysis on the role of product development activities on product success draws these two following conclusions.

First, the role of product development activities is specific for every case, so that the use of an aggregate data from a wide variation of cases as usually conducted by many researchers this far may mislead the conclusion.

Second, the findings in the resource allocation analysis and the experts’ judgment analysis show that although all the activities in product development process are mandatory, some of the activities potentially trigger the product success more than the others. A procedure of determining influencing activities by the resource allocation analysis and the experts’ judgment (Figure 5) can help companies to decide the focus of their product development funding.

Moreover, the findings from this study lead to some further research opportunities. The results of meta-analyses, as an updated works, can be used for many related studies. The mechanism of the resource allocation analysis is a valuable idea for further research on management accounting. The preliminary procedure proposed at the end of this article still needs to be tested and developed further to be practically perfect.