This paper describes how, based upon item response theory (IRT) and its differential item functioning (DIF), two studies were designed to address two important issues – adopting effective items or inviting proper respondents – involved in the identification of successful new concepts, to test new concepts with different levels of newness. Study One shows that some items in a multi-item scale better discriminate among concepts with low or high viability, and that tailored selections of items are necessary when testing major innovation or minor improvement concepts. Study Two pinpoints that choosing an effective source of respondents is important to identify popular movies. Although evaluations from ordinary moviegoers are generally more discriminating among movies with different popularity, those from professional critics are more effective for movies of unfamiliar genres. The implementation of IRT and DIF in both studies demonstrates an effective two-step benchmarking procedure for picking up winners for different new concepts.
Get full access to this article
View all access options for this article.
References
1.
AinslieA., DrèzeX., & ZufrydenF. (2005) Modeling movie life cycles and market share. Marketing Science, 24, 3, pp. 508–517.
2.
BasuroyS., ChatterjeeS., & RavidS.A. (2003) How critical are critical reviews? The box office effects of film critics, star power & budgets. Journal of Marketing, 67, 4, pp. 103–117.
3.
BaumgartnerH., & HomburgC. (1996) Applications of structural equation modeling in marketing and consumer research: a review. International Journal of Research in Marketing, 13, 2, pp. 139–161.
4.
BeardenW.O., & NetemeyerR.G. (1999). Handbook of Marketing Scales: Multi-item Measures for Marketing and Consumer Behavior Research.Sage: Newbury Park, CA.
5.
BlytheJ. (1999) Innovativeness and newness in high-tech consumer durables. Journal of Product & Brand Management, 8, 5, pp. 415–429.
6.
ChakravartyA., LiuY., & MazumdarT. (2010) The differential effects of online word-of-mouth and critics' reviews on pre-release movie evaluation. Journal of Interactive Marketing, 24, 3, pp. 185–197.
7.
ChurchillG. A. (1979) A paradigm for developing better measures of marketing constructs. Journal of Marketing Research, 16, 1, pp. 64–73.
8.
CrawfordC.M., & Di BenedettoC.A. (2015). New Products Management (11th edn). Boston, MA: McGraw-Hill Education.
9.
CuiG., LuiH.-K., & GuoX. (2012) The effect of online consumer reviews on new product sales. International Journal of Electronic Commerce, 17, 1, pp. 39–58.
10.
de JongM.G., SteenkampJ.-B.E., & FoxJ.-P. (2007) Relaxing measurement invariance in cross-national consumer research using a hierarchical IRT model. Journal of Consumer Research, 34, 2, pp. 260–278.
11.
de JongM.G., SteenkampJ.-B.E., & VeldkampB.P. (2009) A model for the construction of country-specific yet internationally comparable short-form marketing scales. Marketing Science, 28, 4, pp. 674–689.
12.
DellarocasC., ZhangX.Q., & AwadN.F. (2007) Exploring the value of online product reviews in forecasting sales: the case of motion pictures. Journal of Interactive Marketing, 21, 4, pp. 23–45.
13.
DesaiK.K., & BasuroyS. (2005) Interactive influence of genre familiarity, star power and critics' reviews in the cultural goods industry: the case of motion pictures. Psychology & Marketing, 22, 3, pp. 203–223.
14.
EliashbergJ., & ShuganS.M. (1997) Film critics: influencers or predictors?Journal of Marketing, 61, 2, pp. 68–78.
15.
EliashbergJ., ElberseA., & LeendersM.A. (2006) The motion picture industry: critical issues in practice, current research & new research directions. Marketing Science, 25, 6, pp. 638–661.
16.
GodesD., & MayzlinD. (2004) Using online conversations to study word-of-mouth communication. Marketing Science, 23, 4, pp. 545–560.
17.
GoldenbergJ., LehmannD.R., ShidlovskiD., & BarakM.M. (2006) The role of expert versus social opinion leaders in new product adoption. Marketing Science Institute Report 06-124.
18.
HambletonR. K. (1991). Fundamentals of Item Response Theory (Vol. 2). Sage: Newbury Park, CA.
19.
HolbrookM. B. (1999). Consumer Value: A Framework for Analysis and Research.Psychology Press: London.
20.
HolbrookM. B. (2005) Customer value and autoethnography: subjective personal introspection and the meanings of a photograph collection. Journal of Business Research, 58, 1, pp. 45–61.
21.
JiangH., & StoutW. (1998) Improved Type I error control and reduced estimation bias for DIF detection using SIBTEST. Journal of Educational and Behavioral Statistics, 23, 4, pp. 291–322.
22.
KamakuraW.A., RamaswamiS.N., & SrivastavaR.K. (1991) Applying latent trait analysis in the evaluation of prospects for cross-selling of financial services. International Journal of Research in Marketing, 8, 4, pp. 329–349.
23.
KlinkR.R., & AthaideG.A. (2006) An illustration of potential sources of concept-test error. Journal of Product Innovation Management, 23, 4, pp. 359–370.
24.
LeeY., & O'ConnorG. (2003) The impact of communication strategy on launching new products: the moderating role of product innovativeness. Journal of Product Innovation Management, 20, 1, pp. 4–21.
25.
LiuY. (2006) Word of mouth for movies: its dynamics and impact on box office revenue. Journal of Marketing, 70, 3, pp. 74–89.
26.
LordF. M. (1980). Applications of Item Response Theory to Practical Testing Problems.Lawrence Erlbaum Associates: Mahwah, NJ.
27.
MooreW. L. (1982) Concept testing. Journal of Business Research, 10, 3, pp. 279–294.
28.
PaasL.J., & SijtsmaK. (2008) Nonparametric item response theory for investigating dimensionality of marketing scales: a Servqual application. Marketing Letters, 19, 2, pp. 157–170.
29.
PandelaereM., BriersB., DewitteS., & WarlopL. (2010) Better think before agreeing twice: mere agreement: a similarity-based persuasion mechanism. International Journal of Research in Marketing, 27, 2, pp. 133–141.
30.
PengL., & FinnA. (2008) Concept testing: the state of contemporary practice. Marketing Intelligence & Planning, 26, 6, pp. 649–674.
31.
PengL., & FinnA. (2010) How cloudy a crystal ball: a psychometric assessment of concept testing. Journal of Product Innovation Management, 27, 2, pp. 238–252.
32.
PengL., CuiG., & LiC. (2013) The comparative impact of critics and consumers applying the generalisability theory to online movie ratings. International Journal of Market Research, 55, 3, pp. 413–436.
33.
RoussosL.A., & StoutW.F. (1996) Simulation studies of the effects of small sample size and studied item parameters on SIBTEST and Mantel-Haenszel Type I error performance. Journal of Educational Measurement, 33, 2, pp. 215–230.
34.
SantorD.A., RamsayJ., & ZuroffD.C. (1994) Nonparametric item analyses of the Beck Depression Inventory: evaluating gender item bias and response option weights. Psychological Assessment, 6, 3, p. 255.
35.
ShealyR., & StoutW. (1993) A model-based standardization approach that separates true bias/DIF from group ability differences and detects test bias/DTF as well as item bias/DIF. Psychometrika, 58, 2, pp. 159–194.
36.
SinghJ. (2004) Tackling measurement problems with item response theory: principles, characteristics and assessment, with an illustrative example. Journal of Business Research, 57, 2, pp. 184–208.
37.
SochayS. (1994) Predicting the performance of motion pictures. Journal of Media Economics, 7, 4, pp. 1–20.
38.
SwaminathanS., & BawaK. (2005) Category-specific coupon proneness: the impact of individual characteristics and category-specific variables. Journal of Retailing, 81, 3, pp. 205–214.
39.
ThissenD. (1991). Multilog User's Guide: Multiple, Categorical Item Analysis and Test Scoring Using Item Response Theory.Scientific Software International: Lincolnwood, IL.
