This paper demonstrates the need for systematic approaches to assess design tasks for experimental studies of idea generation and conceptual design. We first examine a collection of 160 design tasks used in the literature revealing a high variance of task characteristics. Variance across studies is captured by analysing operating variables including task elaboration, task orientation, task selection, participants and time allocated for ideation. Four metrics are then proposed to assist researchers to objectively characterise design tasks. These metrics help identify the semantic and lexical dimensions of problem statements, evaluate precedent ideas and assess task wording. The value of this approach is demonstrated by applying the metrics to a subset of design tasks showing their usefulness to assess the design task objectively as part of the wider experimental design process. Armed with these metrics, researchers can tune both the task and the wider experiment to best match the purposes and the operating variables of their studies. The paper concludes by formulating concrete recommendations to assist in the selection of design tasks using these metrics.
GoelVPirolliP. The structure of design problem spaces. Cognitive Sci1992; 16: 395–429.
2.
GuilfordJP. Measurement and creativity. Theory Pract1966; 5: 185–189.
3.
MednickSA. The remote associates test. J Creat Behav1968; 2: 213–214.
4.
VasconcelosLACrillyN. Inspiration and fixation: Questions, methods, findings, and challenges. Des Stud2016; 42: 1–32.
5.
ViswanathanVTomkoMLinseyJ. A study on the effects of example familiarity and modality on design fixation. Artif Intell Eng Des Anal Manuf2016; 30: 171–184.
6.
GlierMWTsennJLinseyJSet al.Evaluating the directed intuitive approach for bioinspired design. J Mech Des Trans ASME2014; 136: 071012-1–071012-9. DOI: 10.1115/1.4026825.
7.
GencoNHölttä-OttoKSeepersadCC. An experimental investigation of the innovation capabilities of undergraduate engineering students. J Eng Educ2012; 101: 60–81.
8.
ShahJJKulkarniSVVargas-HernandezN. Evaluation of idea generation methods for conceptual design: Effectiveness metrics and design of experiments. J Mech Des2000; 122: 377–384.
9.
KrippendorffKValidity in content analysis. In: MochmannE (ed). Computerstrategien fur die kommunikationsanalyse, Frankfurt, Germany: Campus, 1980, pp. 69–112.
10.
KrippendorffK. Content analysis: An introduction to its methodology, Los Angeles; London: SAGE, 2013.
11.
LazarJFengJHHochheiserH. Experimental design. Research methods in human-computer interaction, Cambridge: Morgan Kaufmann, 2017, pp. 45–69.
12.
GoldschmidtGSmolkovM. Variances in the impact of visual stimuli on design problem solving performance. Des Stud2006; 27: 549–569.
13.
ChanJDowSPSchunnCD. Do the best design ideas (really) come from conceptually distant sources of inspiration?Des Stud2015; 36: 31–58.
14.
EastmanCMOn the analysis of intuitive design processes. In: MooreGT (ed). Emerging methods in environmental design and planning, Cambridge, MA: MIT Press, 1970, pp. 21–37.
15.
RittelHWJWebberMM. Dilemmas in a general theory of planning. Policy Sci1973; 4: 155–169.
16.
CoyneRDRosenmanMARadfordADet al.Innovation and creativity in knowledge-based CAD. In: GeroJS (ed). Expert systems in computer-aided design, New York: Elsevier, 1987, pp. 435–465.
17.
GeroJS. Design prototypes: A knowledge representation schema for design. AI Mag1990; 11: 26–36.
18.
SchmittGNChenC-C. Classes of design — classes of methods — classes of tools. Des Stud1991; 12: 246–251.
19.
FrostRB. A suggested taxonomy for engineering design problems. J Eng Des1994; 5: 399–410.
20.
Durand F, Helms ME, Tsenn J, et al. In search of effective design problems for design research. In: 27th international conference on design theory and methodology, Boston, MA, USA, 2–5 August 2015, pp.V007T06A11.
21.
HarfieldS. On design ‘problematization’: Theorising differences in designed outcomes. Des Stud2007; 28: 159–173.
DorstKCrossN. Creativity in the design process: co-evolution of problem–solution. Des Stud2001; 22: 425–437.
24.
VasconcelosLACardosoCCSääksjärviMet al.Inspiration and fixation: The influences of example designs and system properties in idea generation. J Mech Des Trans ASME2017; 139(3):031101-1–031101-13. DOI: 10.1115/1.4035540.
25.
ZahnerDNickersonJVTverskyBet al.A fix for fixation? Rerepresenting and abstracting as creative processes in the design of information systems. Artif Intell Eng Des Anal Manuf2010; 24: 231–244.
26.
LiikkanenLAPerttulaM. Inspiring design idea generation: Insights from a memory-search perspective. J Eng Des2010; 21: 545–560.
27.
Kumar V and Mocko G. Similarity of engineering design problems to enable reuse in design research experiments. In: 28th international conference on design theory and methodology, Charlotte, NC, USA, 21–24 August 2016.
28.
LinseyJSWoodKLMarkmanAB. Modality and representation in analogy. Artif Intell Eng Des Anal Manuf2008; 22: 85–100.
29.
GonçalvesMCardosoCBadke-SchaubP. Inspiration peak: Exploring the semantic distance between design problem and textual inspirational stimuli. Int J Des Creat Innov2013; 1: 215–232.
30.
WodehouseAIonW. Augmenting the 6-3-5 method with design information. Res Eng Des2012; 23: 5–15.
31.
OkudanGEMohammedS. Task gender orientation perceptions by novice designers: Implications for engineering design research, teaching and practice. Des Stud2006; 27: 723–740.
32.
TohCAMillerSR. How engineering teams select design concepts: A view through the lens of creativity. Des Stud2015; 38: 111–138.
33.
YangMC. Consensus and single leader decision-making in teams using structured design methods. Des Stud2010; 31: 345–362.
34.
ArrighiP-ALe MassonPWeilB. Addressing constraints creatively: How new design software helps solve the dilemma of originality and feasibility. Creat Innov Manage2015; 24: 247–260.
35.
Lefebvre C, Manheimer E and Glanville J. Searching for studies. In: Julian Higgins and James Thomas (eds), Cochrane handbook for systematic reviews of interventions version 5.1.0. London: Cochrane Information Retrieval Methods Group, 2011. Available online: http://handbook-5-1.cochrane.org.
36.
CheongHShuLH. Using templates and mapping strategies to support analogical transfer in biomimetic design. Des Stud2013; 34: 706–728.
37.
BonnardelNMarmècheE. Towards supporting evocation processes in creative design: A cognitive approach. Int J Human-Comput Stud2005; 63: 422–435.
38.
DahlDWMoreauP. The influence and value of analogical thinking during new product ideation. J Market Res2002; 39: 47–60.
39.
MohedasIDalySRSienkoKH. Requirements development: Approaches and behaviors of novice designers. J Mech Des Trans ASME2015; 137: 071407-1–071407-10. DOI: 10.1115/1.4030058.
40.
LemonsGCarberryASwanCet al.The benefits of model building in teaching engineering design. Des Stud2010; 31: 288–309.
41.
MuletEVidalR. Functional requirements for computer-based design support systems, derived from experimental studies. Knowl-Based Syst2006; 19: 32–42.
42.
HuangY. Investigating the cognitive behavior of generating idea sketches through neural network systems. Des Stud2008; 29: 70–92.
43.
SunLXiangWChaiCet al.Creative segment: A descriptive theory applied to computer-aided sketching. Des Stud2014; 35: 54–79.
44.
RexfeltOAlmefeltLZackrissonDet al.A proposal for a structured approach for cross-company teamwork: A case study of involving the customer in service innovation. Res Eng Des2011; 22: 153–171.
45.
GoelAKShuLHZhangGet al.On the benefits of digital libraries of case studies of analogical design: Documentation, access, analysis, and learning. Artif Intell Eng Des Anal Manuf2015; 29: 215–227.
46.
FrankeNPoetzMKSchreierM. Integrating problem solvers from analogous markets in new product ideation. Manage Sci2014; 60: 1063–1081.
47.
CashPSniderC. Investigating design: A comparison of manifest and latent approaches. Des Stud2014; 35: 441–472.
48.
DalySRYilmazSChristianJLet al.Design heuristics in engineering concept generation. J Eng Educ2012; 101: 601–629.
49.
PerttulaMSipiläP. The idea exposure paradigm in design idea generation. J Eng Des2007; 18: 93–93.
50.
ChakrabartiASarkarPLeelavathammaBet al.A functional representation for aiding biomimetic and artificial inspiration of new ideas. Artif Intell Eng Des Anal Manuf2005; 19: 113–132.
51.
ChristensenBTBallLJ. Creative analogy use in a heterogeneous design team: The pervasive role of background domain knowledge. Des Stud2016; 46: 38–58.
52.
López-MesaBMuletEVidalRet al.Effects of additional stimuli on idea-finding in design teams. J Eng Des2011; 22: 31–54.
53.
Alcaide-MarzalJDiego-MásJAAsensio-CuestaSet al.An exploratory study on the use of digital sculpting in conceptual product design. Des Stud2013; 34: 264–284.
54.
CashPHicksBCulleyS. Activity Theory as a means for multi-scale analysis of the engineering design process: A protocol study of design in practice. Des Stud2015; 38: 1–32.
55.
HauptG. Learning from experts: Fostering extended thinking in the early phases of the design process. Int J Technol Des Educ2015; 25: 483–520.
56.
BurnapAGerthRGonzalezRet al.Identifying experts in the crowd for evaluation of engineering designs. J Eng Des2017; 28: 317–337.
57.
OzkanODoganF. Cognitive strategies of analogical reasoning in design: Differences between expert and novice designers. Des Stud2013; 34: 161–192.
58.
Vargas HernandezNShahJJSmithSM. Understanding design ideation mechanisms through multilevel aligned empirical studies. Des Stud2010; 31: 382–410.
59.
SarkarPChakrabartiA. A model for the process of idea generation. Des J2017; 20: 239–257.
60.
ChaiCCenFRuanWet al.Behavioral analysis of analogical reasoning in design: Differences among designers with different expertise levels. Des Stud2015; 36: 3–30.
61.
StarkeyETohCAMillerSR. Abandoning creativity: The evolution of creative ideas in engineering design course projects. Des Stud2016; 47: 47–72.
62.
McMahonKRuggeriAKämmerJEet al.Beyond idea generation: The power of groups in developing ideas. Creat Res J2016; 28: 247–257.
63.
NouriRErezMLeeCet al.Social context: Key to understanding culture’s effects on creativity. J Organ Behav2015; 36: 899–918.
64.
MoalosiRPopovicVHickling-HudsonA. Culture-orientated product design. Int J Technol Des Educ2010; 20: 175–190.
65.
JainVKSobek IiDK. Linking design process to customer satisfaction through virtual design of experiments. Res Eng Des2006; 17: 59–71.
66.
GoelAKShuLH. Analogical thinking: An introduction in the context of design. Artif Intell Eng Des Anal Manuf2015; 29: 133–134.
67.
HäggmanATsaiGElsenCet al.Connections between the design tool, design attributes, and user preferences in early stage design. J Mech Des Trans ASME2015; 137: 071408-1–071408-13doi: 10.1115/1.4030181.
68.
ViswanathanVKLinseyJS. Design fixation and its mitigation: A study on the role of expertise. J Mech Des Trans ASME2013; 135. pp. 051008-1–051008-15. DOI: 10.1115/1.4024123.
69.
Badke-SchaubPGoldschmidtGMeijerM. How does cognitive conflict in design teams support the development of creative ideas?Creat Innov Manage2010; 19: 119–133.
70.
Leblebici-BaşarDAltarribaJ. The role of imagery and emotion in the translation of concepts into product form. Des J2013; 16: 295–314.
71.
MuletEChulviVRoyoMet al.Influence of the dominant thinking style in the degree of novelty of designs in virtual and traditional working environments. J Eng Des2016; 27: 413–437.
72.
LinseyJSClaussEFKurtogluTet al.An experimental study of group idea generation techniques: Understanding the roles of idea representation and viewing methods. J Mech Des Trans ASME2011; 133: pp. 031008-031008-15. DOI: 10.1115/1.4003498.
73.
JinYBenamiO. Creative patterns and stimulation in conceptual design. Artif Intell Eng Des Anal Manuf2010; 24: 191–209.
74.
JohnsonDGGencoNSaundersMNet al.An experimental investigation of the effectiveness of empathic experience design for innovative concept generation. J Mech Des Trans ASME2014; 136: ▪▪.
75.
RahmanNChengRBayerlPS. Synchronous versus asynchronous manipulation of 2D-objects in distributed design collaborations: Implications for the support of distributed team processes. Des Stud2013; 34: 406–431.
76.
van der LugtR. Developing a graphic tool for creative problem solving in design groups. Des Stud2000; 21: 505–522.
77.
DaalhuizenJPersonOGattolV. A personal matter? An investigation of students’ design process experiences when using a heuristic or a systematic method. Des Stud2014; 35: 133–159.
78.
NikanderJBLiikkanenLALaaksoM. The preference effect in design concept evaluation. Des Stud2014; 35: 473–499.
79.
TangHHLeeYYGeroJS. Comparing collaborative co-located and distributed design processes in digital and traditional sketching environments: A protocol study using the function–behaviour–structure coding scheme. Des Stud2011; 32: 1–29.
80.
ViswanathanVLinseyJ. Examining design fixation in engineering idea generation: The role of example modality. Int J Des Creat Innov2013; 1: 109–129.
81.
ChulviVGonzález-CruzMCMuletEet al.Influence of the type of idea-generation method on the creativity of solutions. Res Eng Des2013; 24: 33–41.
82.
WorinkengEJoshiSSummersJD. An experimental study: Analyzing requirement type influence on novelty and variety of generated solutions. Int J Des Creat Innov2015; 3: 61–77.
83.
StonesCCassidyT. Seeing and discovering: How do student designers reinterpret sketches and digital marks during graphic design ideation?Des Stud2010; 31: 439–460.
84.
GregoryRJ. Validity and test development. Psychological testing: History, principles, and applications, Boston: Allyn & Bacon, 2011, pp. 136–153.
85.
Snyder C. Task design. In: Paper prototyping. Burlington: Morgan Kaufmann, 2004, pp.121–143.
86.
FellbaumCWordNet and wordnets. In: BrownK (ed). Encyclopedia of language and linguistics, 2nd ed. Oxford: Elsevier, 2005, pp. 665–670.
87.
Pedersen T, Patwardhan S and Michelizzi J. WordNet:: Similarity: measuring the relatedness of concepts. In: Proceedings of the 19th national conference on artificial intelligence, 2004. Menlo Park, CA: AAAI Press.
88.
SpeerRChinJHavasiC. ConceptNet 5.5: An open multilingual graph of general knowledge. Association for the Advancement of Artificial Intelligence (AAAI), Cambridge, MA: Luminoso Technologies, 2017, pp. 4444–4451.
89.
Gallagher E. Measuring semantic ambiguity. Centre for Computation, Mathematics and Physics in the Life Sciences and Experimental Biology, University College London, Contract No.: MRes Case Presentation 3, 2013.
90.
RönnM. A theory for assessing quality in architecture competitions. Nordic J Archit Res2012; 24: 24.
91.
Laham D and Steinhart D. Latent Semantic Analysis @ CU Boulder. http://lsa.colorado.edu/ (1998, accessed 17 March 2018).
92.
CrossleySSalsburyTMcNamaraDSValidating lexical measures using human scores of lexical proficiency. In: JarvisSDallerM (eds). Vocabulary knowledge: Human ratings and automated measures, Amsterdam: John Benjamins, 2013, pp. 105–134.
93.
KohJHLHerringSCHewKF. Project-based learning and student knowledge construction during asynchronous online discussion. Internet Higher Educ2010; 13: 284–291.
94.
SimontonDK. Creativity in science: Chance, logic, genius, and zeitgeist, Cambridge, UK: Cambridge University Press, 2004.
95.
CrossleySASkalickySDascaluMet al.Predicting text comprehension, processing, and familiarity in adult readers: New approaches to readability formulas. Discour Process2017; 54: 340–359.
96.
ZamanianMHeydariP. Readability of Texts: State of the art. Theory Pract Lang Stud2012; 2: 43–53.
97.
BonnardelN. Towards understanding and supporting creativity in design: Analogies in a constrained cognitive environment. Knowl-Based Syst2000; 13: 505–513.
98.
BonnardelNMarmècheE. Evocation processes by novice and expert designers: Towards stimulating analogical thinking. Creat Innov Manage2004; 13: 176–186.
99.
GulikersJTMBastiaensTJKirschnerPA. A five-dimensional framework for authentic assessment. Educ Technol Res Develop2004; 52: 67.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
