Sage Journals: Discover world-class research

Abstract

Computer- and web-based survey technology has enabled the use of question formats and layouts that would be difficult or even impossible to implement with traditional paper questionnaires. The present article investigates context effects—specifically assimilation versus contrast effects—in connection with four web survey question designs. Three traditional types of matrix questions are experimentally tested against a new two-dimensional question format referred to as ZEF. This new question format allows survey practitioners to present two questions at the same time, the responses to which are entered on the x- and y-axes of a two-dimensional chart. The chart also displays a respondent’s answers to the preceding two-dimensional questions. This approach accentuates the simultaneous application of two dimensions of a question and encourages the comparison of pairwise items across the dimensions. The two-dimensional format generates a higher number of assimilation effects as compared with the three traditional matrix question formats. This suggests that instead of conceptual interconnection of items, context effect formation in ZEF is dominated by the visual proximity of the item scales and the common region of the two-dimensional chart.

Keywords

Web survey context effect assimilation effect contrast effect two-dimensional question format experimental psychology cognitive psychology

Introduction

Survey and web survey responding are particular tasks requiring careful consideration and cognitive effort from respondents. Therefore, web survey responding should be understood as cognitive activity, based on formulating a cognitive representation of the target question and available contextual information about the various interface elements of web surveys (see Couper et al., 2004; Selkälä et al., 2020; Tourangeau, 2018). In the present paper, we examine the implications of particular interface elements of the ZEF two-dimensional web survey format (Figure 1) on the cognitive response process by comparing the ZEF format with three other, more conventional formats.

Figure 1.

Presentation of a two-dimensional task in the ZEF interface.

The abbreviation ZEF refers in this context to Z-scored Electronic Feedback, and it has now become its own trademark (ZEF®). ZEF was originally developed as a tool for strategy workshops to find common values, strengths, weaknesses, opportunities, and threats. Using survey platforms to conduct online surveys is generally recommended (Reips et al., 2015). The two-dimensional question format combined with the z-scoring method provides a way to select the main components for new strategies in workshops. The method has been used in numerous ways, mostly in employee engagement and customer satisfaction surveys. The z-scoring method applied in combination with the two-dimensional question format was extensively studied and evaluated by Selkälä et al. (2011). The two-dimensional ZEF format presents associated information on the x- and y-axes of a two-dimensional chart as two separate matrix questions (Selkälä et al., 2011). Answering a two-dimensional question is done by clicking a square-shaped chart, in which the x-axis represents the response scale of the first dimension and the y-axis the response scale of the second dimension of an item (Figure 1). The chart displays a respondents’ answers to all items equivalently on PCs and smartphones (see Selkälä et al., 2011). A respondent also has the opportunity to change the location of a response on the chart before submitting it, a feature found in many other web-based surveys (Couper et al., 2004).

The three other examined designs in the present study were implemented using the traditional matrix question format. In each design, including the ZEF two-dimensional question format, two experimental question batteries were embedded in the web survey. The items of the first question battery concern the importance of student union services to university students, while the second battery focuses on the need to improve the services. Therefore, in each design, two dimensions of an item are represented equally, yet by applying different visual and functional formats. These types of judgmental tasks are common in market research, where customers are, for instance, requested to evaluate the price and the quality of products simultaneously. A conjoint analysis is a good example of judgmental tasks, emphasizing the interrelationship of two or more aspects of items to the respondents (Imthorn et al., 2016). In particular, the ZEF two-dimensional question format reconstitutes the way in which respondents are able to compare to each other their responses on scales representing different dimensions of the same topic. Therefore, the question arises as to how the two separate question dimensions impact underlying cognitive response processes in terms of comparing pairwise items across the dimensions. While comparing task elements to each other, both conceptual and visual cues may have an impact on how the interrelationship of particular elements is interpreted. This further contributes to the emergence of context effects, indicating underlying cognitive processes connected with the comparable elements. These underlying processes constitute a cognitive representation or a cognitive map enabling participants to navigate through the task completion requirements (Behrens et al., 2018; Schiller et al., 2015; Tolman, 1948).

A comparison setting involving two separate question batteries can be created in multiple ways in web survey applications. The matrix format is particularly suitable for this purpose, it has the same response options for the items, as indicated by shared verbal labels in the headings of the columns of radio buttons (see Appendix A1). However, despite the wide popularity of the matrix format, it remains controversial for multiple reasons. For example, the item nonresponse rate has been found to be higher with matrix questions than with item-by-item questions (Liu and Cernat, 2016; Toepoel et al., 2009). In addition, the matrix format increases the likelihood of survey satisficing, implying respondent behavior to take cognitive shortcuts during responding instead of providing an optimized answer (Hamby and Taylor, 2016; Vannette and Krosnick, 2014; Zhang and Conrad, 2018). Survey satisficing is proposed to be determined by three major factors: task difficulty, respondent ability and respondent motivation (Krosnick, 1991; Krosnick and Presser, 2010). Weak survey satisficing occurs when all of the four cognitive stages of survey responding—comprehension, retrieval, judgment, and response—are performed, but incompletely. In the case of strong survey satisficing, some of these cognitive stages are entirely skipped (Krosnick, 1991, 1999; Tourangeau et al., 2005).

Given the discussed cognitive basis of survey responding, cognitive load requirements to complete certain survey tasks can exceed the processing capacity of the respondent, affecting negatively problem-solving, learning, and the quality of the data (Brosnan et al., 2021; Hamby and Taylor, 2016; Liu and Cernat, 2016). In particular, extensive batteries of questions can be cognitively demanding and likely to increase survey satisficing (Vannette and Krosnick, 2014). The matrix format is a clear representative of extensive question batteries and can be more daunting for respondents than an item-by-item question format (Liu and Cernat, 2016). We expect this tendency of matrix format to increase when applying two matrix questions simultaneously. Therefore, the ZEF two-dimensional question design seems more advantageous in creating a comparison setting for two question batteries. This is partly due to the fact that when the item pairs are compared to each other across the traditional matrices, two separate mouse clicks are required to give the responses, whereas in ZEF, only one mouse click is required to give two responses simultaneously. Consequently, ZEF demonstrably requires less functional effort from the respondents to reply to two questions. It has been shown that such burden-free fluency experience is a distinct feature that increases the likelihood of tasks being solved (e.g. Dantlgraber et al., 2019). Market researchers may find this beneficial, because in their product evaluations, the response burden is to be avoided whenever possible. Overall, given the expected higher survey satisficing and the functional effort of two matrix questions, ZEF seems a more advantageous design in creating a comparison setting for two question batteries. However, the question arises as to how cognitive representations are formed under the influence of the different designs and what the emerging context effects are. This is crucial because context effects have the potential to indicate more than just cognitive effort. They reflect how respondents interpret the sets of information from separate question batteries in relation to each other. In this respect, context effects offer an insight into the cognitive response processes related to the different designs.

Theoretical background

In order to understand any comparison task in terms of the possible outcome, assimilation and contrast effects have been found highly relevant. As Barker and Imhoff (2021) argue:

In broad terms, the integration of comparison information can logically only affect the final judgment in one of two ways; either the judgment is moved closer towards the standard (i.e., an assimilation effect) or it is pushed away from the standard (i.e., a contrast effect).

Assimilation and contrast effects have been studied in disciplines such as product evaluation research to examine how users either “assimilate” or “contrast” to historical ratings under different scenarios (Zhang et al., 2019). Assimilation and contrast effects are also important in web surveying (Reips, 2002; Smyth et al., 2009). Given the high relevance of assimilation and contrast effects in understanding and describing any type of comparison task, we examine the ZEF two-dimensional question design particularly from the perspective of assimilation and contrast theory.

Contrast effect

Contrast effect refers to a phenomenon in which the distributions of the target variable and contextual variable diverge (e.g. after an experimental manipulation), resulting in a larger difference between the mean values of the target variable and the contextual variable or lower correlation between the variables. In most theories regarding the cognitive basis of assimilation and contrast effects, an assimilation effect is proposed to occur relatively effortlessly, whereas a contrast effect is expected to require motivation and capacity from the participants (Biernat and Eidelman, 2007). The likelihood of a contrast effect increases when the participants become aware of priming manipulation or the influence of priming on the target task (Lombardi et al., 1987; Newman and Uleman, 1990; Strack et al., 1993). The previous research has also shown that when participants in a judgmental task hold a strong attitude towards the task, are familiar with the topic or find it important, their memory inferences are generated based on more chronically accessible information (Herr et al., 1983; Tourangeau et al., 2005: 172). As a consequence, the cognitive representation of a target question or a comparative standard becomes more context independent (Bless and Schwarz, 1998, 2010). This increases the likelihood of a contrast effect between the contextual information and a target (Palmer and Gore, 2014). Dissimilarity testing has been proposed to be responsible for the particular cognitive representation generating a contrast effect. In dissimilarity testing, participants focus more on differences than similarities between the target and the standard (Barker and Imhoff, 2021; Mussweiler and Damisch, 2008). A contrast effect is also suggested to occur as a result of more extensive and complex information processing (Dijksterhuis et al., 1998; Kolańczyk and Pawłowska-Fusiara, 2002) or when the participants consciously compare or discriminate between the contextual information and the target variable (Lombardi et al., 1987; Martin et al., 1990; Wedell et al., 2007). What follows is that the outcome of any given task can be manipulated by varying the participants’ awareness of the relationship between context and target. When the relationship between the context and target is explicitly accentuated, it increases the participants’ awareness of the relationship, making a contrast effect the more likely outcome. For instance, Strack et al. (1988) found that placing two questions consecutively and presenting them with the same lead-in or using the same response format emphasizes the shared conversational context and gives rise to a contrast effect (Strack et al., 1988).

Assimilation effect

Assimilation effect refers to a phenomenon in which the distributions of the target variable and contextual variable converge (e.g. after an experimental manipulation), resulting in a smaller difference between the mean values of the target variable and the contextual variable or higher correlation between the variables. The likelihood of an assimilation effect increases and it is the more likely outcome when the participants remain unaware of the contextual information (Clore, 1992; Schwarz and Bless, 1992; Schwarz et al., 1991; Strack et al., 1993). On the other hand, when participants’ memory inferences are based on more temporarily accessible information, the cognitive representation of the target question becomes susceptible to contextual factors (Herr et al., 1983). This increases the probability of the emergence of an assimilation effect (Schwarz and Bless, 2007; Tourangeau et al., 2005: 224). When it comes to the cognitive basis of the assimilation effect, similarity testing has been proposed to be responsible for the particular cognitive representation generating an assimilation effect. In this process, participants tend to focus more on similarities than dissimilarities between the comparison target and the standard (Barker and Imhoff, 2021; Mussweiler and Damisch, 2008).

Gestalt principles

In addition to assimilation-contrast theory, Gestalt principles are essential to understanding the ZEF two-dimensional question format in relation to the traditional matrix format. Gestalt principles describe how humans organize visual information by grouping elements together (Wong, 2010). From the Gestalt principles (see Montoro et al., 2017; Palmer, 1992, 1999), similarity, visual proximity and common region are the most relevant ones in terms of the present study. The similarity principle states that elements are grouped together if their attributes are perceived as related, while the visual proximity principle states that elements which are close to each other will be grouped together (Chang et al., 2007; Fraher and Boyd-Brent, 2010; Goldstein, 2010; Koch and Oulasvirta, 2016). Grouping by a common region refers to the fact that elements located within the same bounded region tend to be grouped together (Montoro et al., 2017). Although the Gestalt principles are generally closely related (see Wong, 2010), the visual proximity principle has the most direct implications for the assimilation-contrast theory, because when items are presented visually in proximity to each other, the probability of an assimilation effect increases (Couper et al., 2001; Toepoel et al., 2009; Tourangeau et al., 2004, 2013). On the other hand, when it comes to the relative strength of the Gestalt principles to influence perceived grouping, common region has been found more effective than the similarity or proximity principles (Palmer, 1992). Montoro et al. (2017) found that grouping by common region dominates the perceived organization of the display when the grouping factors were not explicitly attended. Wong (2010) argues that grouping by enclosure resulting in elements bounded in a common region is powerful enough to overcome the similarity, proximity and connection principles. According to Wong (2010), this is due to the fact that while objects grouped by similarity and proximity are seen as loose confederations, grouping by connection and enclosure leads us to associate them as a unified whole.

With regard to the applicability of the Gestalt principles to the present study, it should be noted that in ZEF, the distance between the response scales of pairwise items representing different dimensions (question batteries) is the shortest of the tested experimental conditions. This is because in ZEF, the response scales are presented on the x- and y-axes of a square-shaped chart of a two-dimensional question, both starting at the origin (Figure 1), while in the other conditions they are located in visually separated question batteries (Appendices A1 to A4). Consequently, in ZEF, the scale values of pairwise items can be detected in close proximity on the x- and y-axes, while in the matrix conditions they must be sequentially verified from the separate matrices. Therefore, we expect the visual proximity of the response scales in ZEF to have a significant impact on interpreting their mutual relationship. In addition, as soon as responses start to accumulate within the square-shaped chart of ZEF, we expect the respondents to increasingly start focusing on their interrelationships. Consequently, the visual context of task completion in ZEF starts to contribute to the interpretation of the relationships between responses, characterizing a common region with clear boundaries. This occurs because the response scales of the different dimensions are presented in ZEF as the x- and y-axes of a square-shaped chart enclosing a common region. This implies that in ZEF, the visual proximity of the response scales is eventually enabled by the common region principle. In ZEF, we expect this combination of visual interface features to lead respondents to interpret the relationships between individual responses as a unified whole, which is a less likely outcome in the other conditions (see Wong, 2010). Overall, ZEF seems to facilitate a coherent comparison of pairwise items across the dimensions, while in the case of two separate matrices, further mouse clicks and a longer distance between the response scales of pairwise items reduce this opportunity. Therefore, the examined solutions to create a comparison setting for two question batteries differ from each other in terms of functionality and visual perception, most likely generating different context effects representing underlying cognitive representations.

Research questions and hypotheses

As discussed above, respondents’ inferences about the task elements or survey questions can be influenced either visually or conceptually, resulting in potentially different context effects. As a consequence, in determining the direction of a context effect, it becomes essential to ascertain whether visual or conceptual features affect the responses. In the present study, we examine how context effects differ between item pairs representing the same topic in different dimensions and matrices across the experimental conditions. The previous research has shown that the visual proximity of items makes an assimilation effect more likely. On the other hand, when the relationship between the context and target is explicitly accentuated, a contrast effect is a more likely outcome. Given that the ZEF two-dimensional question design represents both of these aspects, we expect (1) the emergence of an assimilation or contrast effect to depend on whether the visual proximity of item pairs enabled by the common region principle or the explicitly accentuated relationship between the item pairs will dominate the context effect formation.

Most importantly, we compare the ZEF two-dimensional question format with the traditional matrix question formats in order to find out how assimilation and contrast effect formation differs between them. Given that the key features of ZEF, visual proximity and explicitly accentuated relationship between item pairs, are represented to varying degrees in the experimental matrix formats, we expect (2) to find results that are approximately similar to those of ZEF from the matrix formats that represent these features most distinctively. As discussed above, we also expect (3) to find stronger contrast effects in connection with items that are familiar or important to the respondents or potentially trigger strong attitudes, making memory inferences chronically more accessible.

Method

Experimental design

The web survey experiment was embedded in a larger web survey, the purpose of which was to evaluate the quality of various services offered by the Student Union of the University of Lapland. The web survey questions covered topics that the Student Union finds important: its customer service quality and public relations. The experimentally manipulated questions of the survey pertained to the importance of the Student Union services and the need to improve them. When reproducing a two-dimensional logic in a matrix format, the same items are asked twice from different substantive perspectives in separate matrices representing separate dimensions. The joint lead-in text is then added above the matrices, based on which the respondents perceive the pairwise items from the different perspectives and respond accordingly.

The experiment consisted of four treatments (Table 1). The treatments varied by the visual proximity of the matrices, the joint lead-in text and whether a two-dimensional chart was applied or not. The first treatment (Matrix 1) was a control condition. In this condition, the “importance” and the “need for improvement” matrices were presented without a joint lead-in text on separate web screens accessible with a Forward and a Back button (Appendix A1 and Appendix A2). In the second treatment (Matrix 2), the matrices were presented on the same web screen (Appendix A3). The third treatment (Matrix 3) was identical to the second, with the exception of an added joint lead-in text (Appendix A4). The fourth treatment was ZEF, presenting both the visual proximity of the matrices and the joint lead-in text analogously to treatment 3, but differing from all the other treatments in functionality and visual format (Figure 1).

Table 1.

Experimental design: Treatment features.

Characteristics	Treatment 1	Treatment 2	Treatment 3	Treatment 4
Characteristics	Matrix 1	Matrix 2	Matrix 3	ZEF
Visual proximity of matrices	−	+	+	+
Joint lead in text	−	−	+	+
Two-dimensional chart	−	−	−	+

Matrix 1, Matrix 2, and Matrix 3 were one-dimensional treatments comprised of two separate matrices conducted by Webropol web survey application. ZEF was functionally and visually a two-dimensional treatment.

The joint lead-in text was applied in the Matrix 3 and ZEF conditions in order to direct the respondents’ attention to the fact that they needed to consider two dimensions simultaneously. In particular, the expression “on the other hand” increases the likelihood that a respondent will consciously compare the two dimensions to each other (Strack et al., 1988). In the Matrix 3 layout, the joint lead-in text was located above the first matrix of the questionnaire. In ZEF it was located above the list of two-dimensional items.

The experimental conditions differed in respect to how they accentuated the substantive connection of the item pairs located in the separate matrices or dimensions. In the first condition (Matrix 1), the matrices were presented on separate web screens without an orienting text. The result lacked visual and conceptual cues enhancing comparison between the items located in separate matrices. In the second condition (Matrix 2), the respondents could see the visual proximity of the matrices but there was no joint lead-in text. In conditions 3 (Matrix 3) and 4 (ZEF), the respondents could use both cues, visual and conceptual, but the conditions differed in function and visual format (Table 1). We expected the substantive connection of item pairs to increase if the layout includes either conceptual or visual cues, and we expected it to increase most likely if there are both types of cues. Therefore, we expected to observe the strongest context effects in the item pairs of the Matrix 3 and ZEF conditions.

The first two questions of the questionnaires concerned the importance of the Student Union and the overall quality of its customer services, while the third question was an experimentally manipulated question. The experimental question was presented according to the treatments (see Appendices A1 to A4 and Figure 1). The experimental items covered various services of the Student Union and were presented twice; the first matrix concerned the importance of the Student Union services and the second the need to improve them. Both matrices in the three traditional formats included ten items each. In ZEF, the total number of items was only 10, because they were asked two-dimensionally, that is, one two-dimensional ZEF item represents two one-dimensional matrix items. In ZEF, the x-axis represented the importance of the Student Union services and the y-axis the need to improve them. In the three traditional matrix formats, the response options were presented as radio buttons. In ZEF, the response was selected by clicking the square-shaped chart, by which the respondent rated simultaneously both items representing the two dimensions.

Sample and data collection

A total of 3507 university students were invited by e-mail to participate in the survey (Table 2). Non-responders received two reminders after 2 and 3 weeks following the initial invitation. The total of 800 respondents included 787 undergraduate students and thirteen postgraduate students. The students were randomly assigned to the four experimental treatments, and the response rate varied from 20 to 30% (Table 2).

Table 2.

Number of respondents, number of dropouts, and response rates.

Characteristics	Treatment 1	Treatment 2	Treatment 3	Treatment 4
Characteristics	Matrix 1	Matrix 2	Matrix 3	ZEF
Sample size	876	883	877	871
Number of respondents	174	183	181	262
Number of dropouts	41	52	45	70
Response rate (%)	19.9	20.9	20.6	29.9

The Webropol applications applied in treatments 1–3 and ZEF differ in how they manage dropouts. The dropouts started the survey by submitting a partially completed questionnaire. ZEF includes these respondents in the final data, whereas Webropol excludes them. Because of this difference, a ZEF response rate represents the RR2-standard definition method, whereas the response rates of treatments 1–3 represent the RR1 method (The American Association for Public Opinion Research, 2008: 34, 48). It should also be noted that the “No answer” option was only available for respondents in the ZEF treatment, in which the responses to this option were treated as missing values.

Measurement

Directional assimilation and contrast effects are typically measured using mean values (Kaplan et al., 2013; Tourangeau et al., 2005). For instance, if the mean values of context and target variables converge after an experimental manipulation, this is the evidence of a directional assimilation effect. Correspondingly, if those values diverge, a directional contrast effect has occurred (Tourangeau et al., 2005: 201; Zhang et al., 2019).

In order to operationalize the directional context effect measurement appropriately, the details regarding how the responses are captured in the ZEF and matrix conditions should be considered. In ZEF, a single mouse click on the square-shaped chart results in two separate values representing responses on the x- and y-axes. The response scales in ZEF are thus measured at the interval level, whereas in the matrix conditions, the measurement level is ordinal. In order to measure the directional context effect in terms of diverging or converging responses to item pairs, we subtracted the values of the importance items from the values of the need-for-improvement items pairwise in all conditions. This results in negative and positive values: 4 − 2 = 2 or 2 − 4 = −2, yet indicating the same distance between the responses. While measuring the directional context effect, the distance between the responses to item pairs appears to be essential, rather than the content of the responses. If the original subtraction variables were applied without an absolute value transformation, the negative and positive values, still equal in the absolute value, would represent different locations in the distribution, although they should be treated as equal in terms of directional assimilation and contrast effect. Therefore, the subtracted values were further transformed to absolute values. This operation produces distributions with lower values indicating an assimilation effect and higher values indicating a contrast effect. In these distributions, the value “zero” represents the highest possible assimilation effect, while the value “four” represents the highest possible contrast effect.

Results

As discussed above, in the experiment, the differences in the absolute values of subtracted values between item pairs are measures of directional assimilation and contrast effects. Table 3 shows these values for each condition. Lower values indicate an assimilation effect and higher values a contrast effect. As shown in Table 3, the mean values of absolute value differences appears to be smallest in the ZEF condition for 7 out of 10 item pairs, suggesting the largest assimilation effect in ZEF as compared with the other conditions. According to the Kolmogorov-Smirnov tests that we performed, no one variable expressing the difference in absolute values in any experimental condition can be treated as normally distributed (p < 0.005). Accordingly, we applied the one-way ANOVA test as a relatively robust method whenever the variables were skewed in the same direction (Appendix B1) and whenever the variances between the conditions were equal. As shown in Table 4, the dependent variables 2–7 fulfilled both of these requirements. The number of respondents in each condition was also large enough to support this procedure (Table 2). Whenever the variances between the conditions were not equal or whenever the variables were skewed in an inappropriate manner, the Kruskal-Wallis non-parametric analysis of variance, which compares medians, was applied (Dependent variables: 1 and 8–10).

Table 3.

Mean values of differences in absolute value between item pairs in experimental groups.

Items	Matrix 1	Matrix 2	Matrix 3	ZEF
1. Academic year calendar	1.55	1.51	1.47	1.20
2. Student card	2.25	2.17	2.36	1.87
3. Promoting the degree program	1.12	1.14	0.96	0.88
4. Promoting the profession	0.99	1.08	0.98	0.96
5. Advising services	0.96	1.02	0.93	0.91
6. Finnish Student Health Service	1.64	1.72	1.60	1.56
7. Local student discounts	1.11	1.18	1.02	1.05
8. Short-term loan services	0.54	0.56	0.60	0.49
9. Development co-operation and environmental activities	0.55	0.79	0.60	0.65
10. International activities (Erasmus student network)	0.91	0.71	0.66	0.75
Total mean of mean differences	1.16	1.19	1.12	1.03

Table 4.

Results of ANOVA and Kruskal-Wallis test values for absolute value differences between the experimental groups.

Dependent variables	DF within groups	Levene test (sig.)	F	χ²
1. Academic year calendar	781	0.016		4.178
2. Student card	777	0.126	7.306***
3. Promoting the degree program	764	0.386	3.830*
4. Promoting the profession	757	0.346	0.542
5. Advising services	757	0.321	0.514
6. Finnish Student Health Service	758	0.186	0.765
7. Local student discounts	759	0.183	0.956
8. Short-term loan services	751	0.000		64.598***
9. Development co-operation and environmental activities	753	0.001		28.081***
10. International activities (Erasmus student network)	756	0.000		19.150***

DF: degrees of freedom; F: F-ratio; χ²: Kruskal–Wallis chi-squared value.

p < 0.05. ***p < 0.001.

As shown in Table 4, we found statistically significant absolute value differences between the experimental conditions in 5 of the 10 items. The absolute value differences between the experimental groups are statistically significant at the p < 0.001 level in the following items: the student card F(3, 777) = 7.306, short-term loan services χ²(3, 751) = 64.598, development co-operation and environmental activities χ²(3, 753) = 28.081 and international activities χ²(3, 756) = 19.150. At the p < 0.05 level, the differences are statistically significant for the item “promoting the degree program” F(3, 764) = 3.830. The results were further elaborated to identify which particular differences between pairs of means are significant. In order to reach this, we performed ANOVA multiple post hoc tests using the Fisher’s least significant difference test (LSD) (p < 0.05) with Bonferroni corrections to compare the mean values of the “Student card” and “Promoting the degree program” variables (Table 5). Kruskal-Wallis pairwise comparisons to compare the mean ranks of the differences in absolute value between the treatments were applied to the “Short-term loan services,” “Development co-operation and environmental activities,” and “International activities (Erasmus Student Network)” variables (Table 6).

Table 5.

ANOVA post hoc multiple comparisons for mean item pair differences between the treatments.

Dependent variables	Subtraction	Mean difference
1. Academic year calendar	Matrix 1–Matrix 2	0.04
	Matrix 1–Matrix 3	0.07
	Matrix 1–ZEF	0.35
	Matrix 2–Matrix 3	0.04
	Matrix 2–ZEF	0.31
	Matrix 3–ZEF	0.27
2. Student card	Matrix 1–Matrix 2	0.09
	Matrix 1–Matrix 3	−0.11
	Matrix 1–ZEF	0.38**
	Matrix 2–Matrix 3	−0.19
	Matrix 2–ZEF	0.29
	Matrix 3–ZEF	0.49***
3. Promoting the degree program	Matrix 1–Matrix 2	−0.01
	Matrix 1–Matrix 3	0.16
	Matrix 1–ZEF	0.24*
	Matrix 2–Matrix 3	0.18
	Matrix 2–ZEF	0.26*
	Matrix 3–ZEF	0.08
4. Promoting the profession	Matrix 1–Matrix 2	−0.09
	Matrix 1–Matrix 3	0.01
	Matrix 1–ZEF	0.03
	Matrix 2–Matrix 3	0.10
	Matrix 2–ZEF	0.12
	Matrix 3–ZEF	0.02
5. Advising services	Matrix 1–Matrix 2	−0.05
	Matrix 1–Matrix 3	0.04
	Matrix 1–ZEF	0.06
	Matrix 2–Matrix 3	0.09
	Matrix 2–ZEF	0.11
	Matrix 3–ZEF	0.02

Post hoc tests: LSD tests with Bonferroni correction applied to the “Student card” and “Promoting the degree program” variables.

p < 0.05. **p < 0.01. ***p < 0.001.

Table 6.

Kruskal-Wallis pairwise comparisons (mean ranks) for mean item pair differences between the treatments.

Dependent variables	Subtraction	Mean difference
6. Finnish Student Health Service	Matrix 1–Matrix 2	−0.08
	Matrix 1–Matrix 3	0.04
	Matrix 1–ZEF	0.08
	Matrix 2–Matrix 3	0.12
	Matrix 2–ZEF	0.16
	Matrix 3–ZEF	0.04
7. Local student discounts	Matrix 1–Matrix 2	−0.07
	Matrix 1–Matrix 3	0.08
	Matrix 1–ZEF	0.06
	Matrix 2–Matrix 3	0.15
	Matrix 2–ZEF	0.13
	Matrix 3–ZEF	−0.02
8. Short-term loan services	Matrix 1–Matrix 2	−0.02
	Matrix 1–Matrix 3	−0.06
	Matrix 1–ZEF	0.05***
	Matrix 2–Matrix 3	−0.04
	Matrix 2–ZEF	0.07***
	Matrix 3–ZEF	0.11***
9. Development co-operation and environmental activities	Matrix 1–Matrix 2	−0.24*
	Matrix 1–Matrix 3	−0.05
	Matrix 1–ZEF	−0.1***
	Matrix 2–Matrix 3	0.19
	Matrix 2–ZEF	0.14
	Matrix 3–ZEF	−0.05***
10. International activities (Erasmus student network)	Matrix 1–Matrix 2	0.2*
	Matrix 1–Matrix 3	0.25*
	Matrix 1–ZEF	0.16
	Matrix 2–Matrix 3	0.05
	Matrix 2–ZEF	−0.04**
	Matrix 3–ZEF	−0.09**

Kruskal-Wallis pairwise comparisons for mean ranks (adj. sig.) applied to the “Short-term loan services,” “Development co-operation and environmental activity,” and “International activities (Erasmus Student Network)” variables.

p < 0.05. **p < 0.01. ***p < 0.001.

The test values indicated that ZEF differed from the other treatment conditions. ZEF was involved in eleven of the fourteen statistically significant mean differences. ZEF differed from the Matrix 1 and Matrix 3 conditions the most. Four significant differences were found between the ZEF and Matrix 1 conditions and between the ZEF and Matrix 3 conditions. Three significant differences were found between the ZEF and Matrix 2 conditions. In addition, two significant differences were found between the Matrix 1 and Matrix 2 conditions and one significant difference between the Matrix 1 and Matrix 3 conditions.

The occurrence of context effects between the experimental groups was evaluated by all differences between the pairs of means in Tables 5 and 6. The summary of these findings is shown in Table 7. An assimilation effect takes place if the mean value difference in the absolute values of item pairs is smaller in one group than in the opposite group (see Table 3). If the mean value difference is larger than the equivalent mean value difference in the opposite group, then a contrast effect occurs. The results regarding ZEF indicate that an assimilation effect occurs 25 out of 30 times. In seven of these cases, the assimilation effect is statistically significant, while a statistically significant contrast effect occurs four times (Table 7). As compared with the rest of the conditions, ZEF tends to induce an assimilation effect in mean values more than it does a contrast effect. Of the matrix conditions, Matrix 1 demonstrably increased the occurrence of a contrast effect (five sig. contrasts), whereas the rest of the conditions seemed to induce neither type of effect when compared to all the other conditions.

Table 7.

The number of assimilation and contrast effects in mean values per condition.

Groups	Assimilations (total)	Sig. assimilations	Contrasts (total)	Sig. contrasts
Matrix 1	11	2	19	5
Matrix 2	6	2	24	3
Matrix 3	18	3	12	2
ZEF	25	7	5	4

The numbers of assimilation and contrast effect were calculated by comparing the mean values of differences in absolute value between the experimental conditions per each item pair. This equals to 30 comparisons to each condition.

Conclusions

The most prominent result is that ZEF generates an assimilation effect more frequently than any other of the examined conditions, whereas a contrast effect occurs most frequently with the Matrix 1 condition. This was expected, given that the ZEF and Matrix 1 conditions represent visually and functionally the opposite extremes. However, it was unexpected that context effect formation in the Matrix 3 condition corresponds much closer to Matrix 2 context effect formation than the context effect formation of ZEF. Given that the participants in Matrix 3 and ZEF conditions were equally exposed to visual and conceptual cues, we expected more similarity between their results. However, these findings can be explained by the fact that ZEF differs visually and functionally to a great extent from the other conditions, also from the Matrix 3 condition. In this respect, the most prominent feature of ZEF is the very short distance between the response scales representing the two dimensions. This feature is enabled by the particular design of a square-shaped chart characterizing the common region principle. These interface features combined allow the responses to become perceived as a unified whole, the tendency pronounced by the functionality of ZEF to allow two separate responses on pairwise items simultaneously. Overall, the results support the expectation that instead of an accentuated conceptual relationship of items, the visual proximity of items enabled by the common region principle dominates context effect formation in ZEF. This is due to the fact that an accentuated conceptual relationship of items has typically resulted in a contrast effect, while increased visual proximity of items has resulted in an assimilation effect.

The results offer guidelines for survey practitioners who wish to direct respondents’ inferences about the task elements and their mutual interrelationships. In case a survey practitioner wishes to trigger a contrast effect between the comparable items, then the Matrix 1 design should be adopted. In the opposite case, when the assimilation effect is pursued, the ZEF two-dimensional question format should be preferred. If a contrast effect is an unwanted outcome, the Matrix 1 design should be avoided and the ZEF two-dimensional question design adopted and vice versa.

In addition, we found the response pattern occurrence across all of the treatments consistent with the assimilation-contrast theory (Table 3), based on the highest observed contrast effect in connection with items two and six, which the respondents rated as the most important services. As discussed above, the emergence of contrast effects with these types of questions suggests more conscious and more complex information processing by the respondents.

Market researchers may benefit from our results, given that in market research, customers are often requested to compare different features of products simultaneously to each other in terms of price and quality, among other things. This is a typical situation, for example, in conjoint analysis (Imthorn et al., 2016). Our results show that the ZEF two-dimensional question format should be considered as a promising candidate for these types of research questions. The results of the present study can be used in related future studies to provide a more realistic interpretation of participant behavior.

The participants in the present study were university students, which might challenge the generalizability of the results especially in terms of the learning capabilities of the target population regarding complex interfaces, such as ZEF. However, the apparent complexity of the ZEF interface is misleading, given the very short distance between the response scales of the dimensions as well as the opportunity to give two responses simultaneously on pairwise items. Both of these features likely reduce the respondents’ cognitive effort during responding. Therefore, despite the apparent complexity of ZEF, its applicability goes beyond the target population of the present study. With appropriate instructions, ZEF can reliably be used with the general population as well. Overall, ZEF offers an interesting and reliable solution to gather information in the realm of two-dimensional research designs. In particular, it has great potential in market research.

Footnotes

Appendices

Acknowledgements

Special thanks are due to Suvi Ronkainen for enabling the participation in the project focusing on web survey methodology at the University of Lapland. The authors would also like to express their appreciation for the assistance provided by Esko Alasaarela, the creator of the ZEF method. A long-term support of Markku Alasaarela from ZEF is also warmly appreciated.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Arto T Selkälä

Ulf-Dietrich Reips

Author biographies

Arto Selkälä, PhD, is a University Lecturer in Quantitative Methods at the University of Lapland, Finland. He is also an Adjunct Professor of Economic Sociology at the University of Turku. His research interests cover web survey methodology, UI design issues, interface design optimization under uncertainty and valid measuring of user behaviour in various interfaces.

Ulf-Dietrich Reips is a Full Professor at the University of Konstanz. His research focuses on methods of data collection on the Internet, especially Internet-based experimentation and web surveys. He is one of the pioneers and experts in this field, he and his team (https://iscience.uni-konstanz.de/) have developed several techniques and tools to improve the quality and validity of online data. Ulf has published more than 190 scientific publications, including six books, see https://www.uni-konstanz.de/iscience/reips/pubs/publications.html. Ulf and his team received numerous awards for their web applications (available from the iScience Server at () and methodological work serving the research community.

References

The American Association for Public Opinion Research (2008) Standard definitions: Final dispositions of case codes and outcome rates for surveys, 5th edn. Lenexa, Kansas: AAPOR.

Barker

Imhoff

(2021) The dynamic interactive pattern of assimilation and contrast: Accounting for standard extremity in comparative evaluations. Journal of Experimental Social Psychology 97: 104190.

Behrens

TEJ

Muller

Whittington

JCR

, et al (2018) What is a cognitive map? Organizing knowledge for flexible behavior. Neuron 100(2): 490–509.

Biernat

Eidelman

(2007) Standards. In: Kruglanski

Higgins

(eds) Social Psychology, Second Edition: Handbook of Basic Principles. New York, NY: The Guilford Press, pp.308–333.

Bless

Schwarz

(1998) Context effects in political judgement: Assimilation and contrast as a function of categorization processes. European Journal of Social Psychology 28(2): 159–172.

Bless

Schwarz

(2010) Mental construal and the emergence of assimilation and contrast effects: The inclusion/exclusion model. In: Zanna

(ed) Advances in Experimental Social Psychology, vol.42. Cambridge, USA: Academic Press, pp.319–373.

Brosnan

Grün

Dolnicar

(2021) Cognitive load reduction strategies in questionnaire design. International Journal of Market Research 63(2): 125–133.

Chang

Nesbitt

Wilkins

(2007) The Gestalt principles of similarity and proximity apply to both the haptic and visual grouping of elements. In: AUIC proceedings. Eighth Australasian User Interface Conference (AUIC2007), Ballarat, Australia. Conferences in Research and Practice in Information Technology (CRPIT), Vol.64., Ballarat, Australia, pp.79–86. Australian Computer Society, Inc.

Clore

(1992) Cognitive phenomenology: Feelings and the construction of judgment. In: Martin

Tesser

(eds) The Construction of Social Judgments. Hillsdale, NJ: Lawrence Erlbaum Associates, pp.133–163.

10.

Couper

Tourangeau

Conrad

(2004) What they see is what we get. Response options for web surveys. Social Science Computer Review 22: 111–127.

11.

Couper

Traugott

Lamias

(2001) Web survey design and administration. Public Opinion Quarterly 65: 230–253.

12.

Dantlgraber

Kuhlmann

Reips

U-D

(2019). Conceptual fluency in inductive reasoning. PLoS One 14(11): e0225050.

13.

Dijksterhuis

Spears

Postmes

, et al (1998) Seeing one thing and doing another: Contrast effects in automatic behaviour. Journal of Personality and Social Psychology 75: 862–871.

14.

Fraher

Boyd-Brent

(2010) Gestalt theory, engagement and interaction. In: CHI EA ’10: CHI ’10 extended abstracts on human factors in computing systems, Atlanta, GR, 10–15 April 2010, pp.3211–3216. New York, NY: ACM.

15.

Goldstein

(2010) Sensation and Perception, 8th edn. Belmont, CA: Wadsworth, Cengage Learning.

16.

Hamby

Taylor

(2016) Survey satisficing inflates reliability and validity measures: An experimental comparison of college and Amazon Mechanical Turk samples. Educational and Psychological Measurement 76(6): 912–932.

17.

Herr

Sherman

Fazio

(1983) On the consequences of priming: Assimilation and contrast effects. Journal of Experimental Social Psychology 19(4): 323–340.

18.

Imthorn

Kemp

Nobel

(2016) Using conjoint analysis in merger control. ACM working papers. Available at: https://www.acm.nl/sites/default/files/old_publication/publicaties/15747_acm-working-paper-conjoint-analysis-2016-04-25.pdf (accessed 20 November 2021).

19.

Kaplan

Luchman

Mock

(2013) General and specific question sequence effects in satisfaction surveys: Integrating directional and correlational effects. Journal of Happiness Studies 14: 1443–1458.

20.

Koch

Oulasvirta

(2016) Computational layout perception using Gestalt laws. In: CHI EA '16: Proceedings of the 2016 CHI conference extended abstracts on human factors in computing systems, San Jose, CA, 7–12 May 2016, pp.1423–1429. New York, NY: ACM.

21.

Kolańczyk

Pawłowska-Fusiara

(2002) Automatic correction or controlled processing of affective priming. Polish Psychological Bulletin 33(2): 35–44.

22.

Krosnick

(1991) Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology 5(3): 213–236.

23.

Krosnick

(1999) Survey research. Annual Review of Psychology 50: 537–567.

24.

Krosnick

Presser

(2010) Question and questionnaire design. In: Marsden

Wright

(eds) Handbook of Survey Research, 2nd edn. Bingley: Emerald Group Publishing Limited, pp.263–314.

25.

Liu

Cernat

(2016) Item-by-item versus matrix questions: A web survey experiment. Social Science Computer Review 36(6): 690–706.

26.

Lombardi

Higgins

Bargh

(1987) The role of consciousness in priming effects: Assimilation versus contrast as a function of awareness of the priming task. Personality and Social Psychology Bulletin 13: 411–429.

27.

Martin

Seta

Crelia

(1990) Assimilation and contrast as a function of people’s willingness and ability to expand effort in forming an impression. Journal of Personality and Social Psychology 59: 27–37.

28.

Montoro

Villalba-García

Luna

, et al (2017) Common region wins the competition between extrinsic grouping cues: Evidence from a task without explicit attention to grouping. Psychonomic Bulletin & Review 24(6): 1856–1861.

29.

Mussweiler

Damisch

(2008) Going back to Donald: How comparisons shape judgmental priming effects. Journal of Personality and Social Psychology 95: 1295–1315.

30.

Newman

Uleman

(1990) Assimilation and contrast effects in spontaneous trait inference. Personality and Social Psychology Bulletin 16: 224–240.

31.

Palmer

Gore

(2014) A theory of contrast effects in performance appraisal and social cognitive judgments. Psychological Studies 59: 323–336.

32.

Palmer

(1992) Common region: A new principle of perceptual grouping. Cognitive Psychology 24(3): 436–447.

33.

Palmer

(1999) Vision Science. Photons to Phenomenology. Cambridge, MA: The MIT Press.

34.

Reips

U-D

(2002) Context effects in web surveys. In: Batinic

Reips

U-D

Bosnjak

(eds) Online Social Sciences. Seattle, WA: Hogrefe & Huber, pp.69–80.

35.

Reips

U-D

Buchanan

Krantz

, et al (2015). Methodological challenges in the use of the Internet for scientific research: Ten solutions and recommendations. Studia Psychologica 15(2): 139–148.

36.

Schiller

Eichenbaum

Buffalo

, et al (2015) Memory and space: Towards an understanding of the cognitive map. Journal of Neuroscience 35(41): 13904–13911.

37.

Schwarz

Bless

(1992) Constructing reality and its alternatives: An inclusion/exclusion model of assimilation and contrast effects in social judgment. In Martin

Tesser

(eds) The Construction of Social Judgments. Hillsdale, NJ: Lawrence Erlbaum Associates, pp.217–245.

38.

Schwarz

Bless

(2007) Mental construal processes: The inclusion/exclusion model. In: Stapel

Suls

(eds) Assimilation and Contrast in Social Psychology. New York, NY: Psychology Press, pp.119–141.

39.

Schwarz

Strack

Mai

(1991) Assimilation and contrast effects in part-whole question sequences: A conversational logic analysis. Public Opinion Quarterly 55: 3–23.

40.

Selkälä

Callegaro

Couper

(2020) Automatic versus manual forwarding in web surveys – A cognitive load perspective on satisficing responding. In: LNCS 12194: Social computing and social media. design, ethics, user behavior, and social network analysis. 12th International Conference, SCSM 2020, Held as Part of the 22nd HCI International Conference, HCII 2020, (ed Meiselwitz

), Copenhagen, Denmark, July 19–24, 2020, pp.130–155. Cham, Switzerland: Springer Nature Switzerland.

41.

Selkälä

Ronkainen

Alasaarela

(2011) Features of the Z-scoring method in graphical two-dimensional web surveys: The case of ZEF. Quality & Quantity 45: 609–621.

42.

Smyth

Dillman

Christian

(2009) Context effects in Internet surveys: New issues and evidence. In: Joinson

McKenna

KYA

Postmes

Reips

U-D

(eds.) Oxford Handbook of Internet Psychology. New York, NY: Oxford University Press, pp.429–446.

43.

Strack

Martin

Schwarz

(1988) Priming and communication: Social determinants of information use in judgments of life satisfaction. European Journal of Social Psychology 18: 429–442.

44.

Strack

Schwarz

Bless

, et al (1993) Awareness of the influence as a determinant of assimilation versus contrast. European Journal of Social Psychology 23: 53–62.

45.

Toepoel

Das

Van Soest

(2009) Design of web questionnaires. The effects of the number of items per screen. Field Methods 21: 200–213.

46.

Tolman

(1948) Cognitive maps in rats and men. Psychological Review 55(4): 189–208.

47.

Tourangeau

(2018) The survey response process from a cognitive viewpoint. Quality Assurance in Education 26(2): 169–181.

48.

Tourangeau

Conrad

Couper

(2013) The Science of Web Surveys. New York, NY: Oxford University Press.

49.

Tourangeau

Couper

Conrad

(2004) Spacing, position, and order. Interpretive heuristics for visual features of survey questions. Public Opinion Quarterly 68: 368–393.

50.

Tourangeau

Rips

Rasinski

(2005) The Psychology of Survey Response. New York, NY: Cambridge University Press.

51.

Vannette

Krosnick

(2014) Answering questions: A comparison of survey satisficing and mindlessness. In: Ie

Ngnoumen

Langer

(eds) The Wiley Blackwell Handbook of Mindfulness. Chichester: Wiley, pp.312–327.

52.

Wedell

Hicklin

Smarandescu

(2007) Contrasting models of assimilation and contrast. In: Stapel

Suls

(eds) Assimilation and Contrast in Social Psychology. New York, NY: Psychology Press, pp.45–74.

53.

Wong

(2010) Gestalt principles (part 1). Nature Methods 7(11): 863–863.

54.

Zhang

Conrad

(2018) Intervening to reduce satisficing behaviors in web surveys: Evidence from two experiments on how it works. Social Science Computer Review 36(1): 57–81.

55.

Zhang

Xie

Zhao

, et al (2019) Understanding assimilation-contrast effects in online rating systems: Modelling, debiasing, and applications. ACM Transactions on Information Systems 38(1): 1–25.

A new two-dimensional question format in web survey design: Assimilation and contrast effects

Abstract

Keywords

Introduction

Theoretical background

Contrast effect

Assimilation effect

Gestalt principles

Research questions and hypotheses

Method

Experimental design

Sample and data collection

Measurement

Results

Conclusions

Footnotes

Appendices

Acknowledgements

Declaration of conflicting interests

Funding

ORCID iDs

Author biographies

References