Effect of knowledge differentiation and state space partitioning on subjective probability estimation

Abstract

A common technique for eliciting subjective probabilities is to provide a set of exclusive and exhaustive events and ask the assessor to estimate the probabilities of such events. However, such subjective probabilities estimations are usually subjected to a bias known as the partition dependence bias. This study aims to investigate the effect of state space partitioning and the level of knowledge on subjective probability estimations. The state space is partitioned into full, collapsed, and pruned trees, while the knowledge is manipulated into low and high levels. A scenario called “Best Bank Award” was developed and a 2 × 3 experimental design was employed to explore the effect of the level of knowledge and the partitioning of the state space on the subjective probability. A total of 627 professionals participated in the study and 543 valid responses were used for analysis. The results of two-way ANOVA with the Tukey HSD test for post hoc analysis indicate a mean probability of 24.2% for the full tree, which is significantly lower than those of the collapsed (35.7%) as well as pruned (36.3%) trees. Moreover, there is significant difference in the mean probabilities between the low (38.1%) and high (24.9%) knowledge levels. The results support the hypotheses that the partitioning of the state space as well as the level of knowledge affects subjective probability estimation. The study demonstrates that regardless of the level of knowledge, the partition dependence bias is robust. However, the subjective probability accuracy improves with more knowledge.

Introduction

Decision making processes usually involve subjective probability assessment in different business situations, such as consumer behavior toward promotion ¹ and project risk management. ² Subjective probability assessment under uncertainty is exposed to several cognitive biases. ³ One such bias is the partition dependence bias^4–7 in which the assessed subjective probabilities are determined by the state space partitioning. Hence, differences in the state space partitioning result in different probability assessments. Literature suggests that assessors use the anchoring and adjustment heuristic, when assessing subjective probabilities. ⁴ Specifically, assessors tend to anchor their subjective probability on 1/n for each event, where n is the number of options available in the state space. Therefore, if only two options are available, the anchor is 1/2. However, if the state space is partitioned into three options, the anchor is 1/3. This estimation is in contrast with the typical assumption of rational probability assessment, in which the state space selection should have no impact on the assessed subjective probabilities.

Fox and Clemen ⁴ conducted five studies to investigate the effect of partition dependence bias on subjective probability estimate. In each study, they created a scenario with different state space partitioning, and asked the participants to estimate the subjective probability. The vigorousness of partition dependence bias was observed in all the five studies. It is worth noting that the information provided to the participants in all the five studies was limited. The participants utilized this limited information, in addition to their intuitions, to estimate the probabilities. In studies 3, 4, and 5, the populations selected were assumed to be knowledgeable on the topics covered by the studies. However, the level of knowledge or information on the study scenarios were the same, and was not explicitly manipulated. Therefore, is unclear whether the subjective probability estimate will be affected by this bias, if additional information is provided. This study addresses this lack of understanding on the effect of the level of knowledge on subjective probability estimation.

This study has two objectives. The first is to investigate the effect of the differences in the partitioning of the state space on subjective probability estimation, using a business scenario called the “Best Bank Award.” Three levels of partitioning are employed: full, collapsed, and pruned trees. The second is to investigate the impact of the level of knowledge regarding the scenario on the subjective probability estimate by applying two levels of knowledge, high and low.

The remainder of the paper includes four more sections. Section 2 reviews the relevant literature in brief. Section 3 describes the experimental study. Section 4 presents the obtained results and discussions. Section 5 concludes the study and provides directions for future research.

Literature review

In this section, the partition dependence bias and knowledge level are briefly discussed.

Partition dependence bias

Probability includes three different interpretations: classical, frequency, and subjective. ⁸ In the classical viewpoint, the sample space is divided into a set of outcomes with equal probability. The probability of an event is determined by counting the number of outcomes of the event and dividing this count by the total number of outcomes. The dice is a typical example of the classical viewpoint. The second interpretation of probability is the frequency viewpoint. An experiment is repeated several times under the same conditions. The probability of the event is estimated as the relative frequency of this event in comparison with the other events in the experiment. ⁸ The last interpretation is the subjective viewpoint in which the probability is estimated based on personal belief of the likelihood of the event. In this viewpoint, the person may assign the probability based on intuition, experience, or current information. It is subjective because the probability assigned to the same event may vary based on the person. ⁸

Partition dependence bias is associated with subjective probability assessment. The subjective probabilities assigned to the events are dependent on the partitioning of the state space by the analyst. ⁴ Therefore, different state space partitioning result in different subjective assessments of the same event. The effect of partition dependence was first demonstrated by Fischhoff et al. ⁹ who called this effect “the pruning bias.” They conducted an experiment in which people were asked to assign points out of 1000 to different categories of reasons for a car to not start. Two forms of trees were presented. The full tree included seven categories:

1. Battery charge insufficient

2. Starting system defective

3. Fuel system defective

4. Ignition system defective

5. Other engine problems

6. Mischievous acts or vandalism

7. All other problems

The second tree removed or pruned three categories, and four reasons remained:

1. Starting system defective

2. Ignition system defective

3. Mischievous acts or vandalism

4. All other problems

This study showed that the points allocated to the last category (all other problems) were higher for the pruned tree compared to the full tree. Ideally, if there was no effect of the partitioning of the state space, the sum of the points of the three pruned categories should be added to the points of the last category, which was not the case in this study. Subsequent studies have attempted to justify this phenomenon, and four justifications have been proposed: availability, ambiguity, credibility, and anchoring and insufficient adjustment. A comprehensive discussion of these justifications can be found in Fox and Clemen. ⁴

Five studies were conducted by Fox and Clemen ⁴ in which the availability, ambiguity, and credibility can be largely reduced. In the first study, the participants were informed that the MBA program for Wharton was ranked #1 among the other programs including Chicago, Harvard, Kellogg, Stanford, and others. The participants were asked to report their probability judgment of the program that will be ranked #1 the following year. Three tree forms of state space partitioning, full, collapsed, and pruned, were presented. The results showed that the full tree probability estimation that a program other than Wharton would win was significantly higher than that of the pruned or collapsed tree. The second study was on estimating the probability of the Jakarta Stock Exchange (JSX) closing value on December 31. The participants were presented with two sets of ranges: one set included three ranges of closing values, whereas the other set contained six ranges. The results showed that the probability estimations were significantly different between the two sets. The third study was on salary brackets. MBA students at Duke University, assumed to be more knowledgeable on the starting salaries in Duke, were asked to state the probability of the starting salary of a graduate from Duke University versus a graduate from Harvard Law. Two forms were provided, low partition and high partition. Again, the probability estimations were significantly different for the low partition compared to the high. The fourth study asked the participants to estimate the probability of the closing values of NASDAQ and JSX using low and high partitioning. As in the findings of previous studies, the probability estimations depended on the state space partitioning. The fifth and final study involved members of the Decision Analysis Society (DAS) of INFORMS, who have extensive training in decision analysis. The probability estimations of these experts were significantly affected by state space partitioning. ⁴ In all the five studies, it was observed that the assessors were affected by state space partitioning. As a result of the phenomenon’s persistence, Fox and Clemen ⁴ called it the partition dependence bias.

In this study, the author investigates through a scenario, whether the subjective probability is affected by partition dependence bias. Therefore, the following hypothesis is formulated:

• Hypothesis 1 (H1): Subjective probability assessment is influenced by state space partitioning.

Knowledge

In all the studies conducted by Fox and Clemen, ⁴ the scenario information provided was limited. In studies 3, 4, and 5, participants with knowledge on the investigated scenarios were selected. However, the impact of different levels of knowledge on subjective probability estimation was not explicitly tested. To the best of the author knowledge, the impact of manipulating the knowledge level on the accuracy of the subjective probability estimations affected by this specific bias has not yet been reported. However, some studies have shown that knowledge moderates the anchoring bias. Smith et al. ¹⁰ conducted four studies across different domains on the relationship between knowledge and the anchoring effect. The topic of the first study was American football. The level of knowledge on American football was determined through an objective quiz that included 14 questions and a subjective assessment of football knowledge. In the second study, two samples from India and USA were used to answer questions on topics in India and the USA, respectively. The third study tested the within-participant relationship between knowledge and the anchoring effect. The participants were questioned on 14 different domains and their knowledge level of these domains. The fourth study was on the average price of midsized sedans, where the knowledge level of the participants was manipulated. The results of these four studies were consistent and suggested that knowledge moderated the anchoring effect. ¹⁰ Barrera-Causil et al., ¹¹ performed two experiments to investigate the elicitation of subjective distributions. In the first experiment, they tested whether instructions on the elicitation methods would help in estimating the true value of the parameter more accurately, compared to the case where no such instructions were provided. The results of the experiment showed that the group with instructions estimated the parameter more accurately than the group without. ¹¹

The author investigates the effect of knowledge on subjective probability estimation in this study. Based on literature review, the second hypothesis is formulated:

• Hypothesis 2 (H2): The level of knowledge provided to the participants affects subjective probability estimation.

Experimental study

The impact of state space partitioning as well as knowledge level on subjective probability estimation is investigated using the “Best Bank Award” scenario.

“Best Bank Award” scenario

The Asian Banker is a financial institution established in 1996. This institution provides strategic intelligence on financial services, and generally offers several awards to banks in Asia, Middle East, and North Africa. One such award is the best retail bank in Saudi Arabia, which is the focus of this scenario. This award was selected because the study was conducted in Saudi Arabia and it was expected that the participants would be familiar with the banks in the country.

Saudi Arabia has several large banks, such as AlInma, AlRajhi, Riyad, Sabb, and Samba. The participants were asked to estimate the probability that Samba or the other banks would win the award the next time. To test the impact of partition dependence on probability estimation, three forms were developed. The first form included a full tree with six options, arranged alphabetically:

• AlInma      ________________%

• AlRajhi      ________________%

• Riyad       ________________%

• Sabb       ________________%

• Samba       ________________%

• None of the above   ________________%

The second form included a collapsed tree in which all the banks other than Samba were merged with the residual category:

• AlInma, AlRajhi, Riyad, Sabb, or another bank other than Samba ________________%

• Samba ________________%

The last form had a pruned tree in which only the Samba and a bank other than Samba were provided:

• A bank other than Samba  ________________%

• Samba       ________________%

To test the impact of knowledge, two levels of knowledge were manipulated. The first level was basic, labeled low-level knowledge, where the only information given was the following statement: “In the latest version of the awards, Samba bank won the award for the best retail bank in Saudi Arabia and was ranked #1.” It should be noted that this statement is considered an explicit anchor, which may affect the subjective probability. This is in contrast with implicit anchor 1/n that depends on the state space partitioning, which is the focus of this study. This low-level knowledge was considered as the control group. The second level of knowledge had the same statement along with information on the criteria used for evaluating the winner of the award. The participants were informed that the award would be given based on four criteria: capital in billions in Saudi Riyals, profitability in 2015 and 2016 in Saudi Riyals per share, number of branches, and number of ATMs. The information on these four criteria are listed in Table 1.

Table 1.

Criteria used to estimate the probabilities of the banks winning in 2017.

	AlInma	AlRajhi	Riyad	Sabb	Samba
Capital (billions Saudi Riyals)	15.00	16.25	30.00	15.00	20.00
Profitability 2015 (Saudi Riyals/share)	0.99	4.39	1.35	2.89	2.61
Profitability 2016 (Saudi Riyals/share)	1.01	5.00	1.11	2.60	2.50
Number of branches (inside Saudi Arabia)	134	500	330	99	72
Number of ATMs (inside Saudi Arabia)	1320	4100	2700	950	540

Rationally, it can be concluded that Samba would have low probability of winning. It is ranked number 2 in capital, number 3 in profitability, and number 5 in both the number of branches and number of ATMs. Although Samba had won the award last year, the current information indicates an opposite trend, which should lead participants to conclude that another bank has higher probability of winning the award. The group that received high-level knowledge was the experimental group.

Six versions of the survey were developed, which were combinations of the two levels of knowledge and the three forms of state space partitioning. Each participant received only one version, and was asked to read the consent form and provide consent to participate in the survey. The six versions of the survey are shown in the Appendix.

Population

The population of interest were professionals working in private or public organizations in Saudi Arabia. Such professionals would usually encounter situations in which they might need to use their judgment in probability estimation. The paper-based survey was provided to MBA students, who were asked to share it with their colleagues in their respective organizations. The survey included a total of 627 participants, among which 52 responses were removed because the probability summation was not equal 100%. Moreover, an additional 32 responses were removed for one of the following reasons: The English level reported was less than average, there was no work experience, or demographic information was missing. Therefore, 543 valid responses were used to test the hypotheses. The original data are available in figshare https://doi.org/10.6084/m9.figshare.14034791. The demographics of this sample are shown in Table 2.

Table 2.

Demographics of the sample.

	Count	Mean	St Dev
Age		32.79	7.81
Gender
Male	459
Female	71
Not specified	13
Academic status
High school	6
Diploma	35
Bachelor	379
Master	113
PhD	7
Other	3
Years of experience		8.46	7.53

Results and discussion

The dependent variable was the subjective probability that Samba bank could win the subsequent award. The two independent variables were knowledge: low versus high and partitioning: full versus collapsed versus pruned. The data were analyzed using IBM SPSS Statistics 20 software and verified using Excel. Table 3 shows the mean, standard deviation, and the sample size for each condition.

Table 3.

Statistics of the six experimental conditions: mean (standard deviation) [sample size].

Condition\knowledge	Low	High	Average
Full	0.289 (0.214) [102]	0.184 (0.140) [84]	0.242 (0.191) [186]
Collapsed	0.432 (0.249) [100]	0.267 (0.202) [83]	0.357 (0.243) [183]
Pruned	0.430 (0.226) [88]	0.294 (0.245) [86]	0.363 (0.245) [174]
Average	0.381 (0.240) [290]	0.249 (0.205) [253]

Three values are depicted in each cell. For example, 102 participants received the version of the full tree under low knowledge; the mean probability that Samba could win the award is 28.9% with a standard deviation of 21.4%. The mean, standard deviation, and sample size for the other conditions can be inferred similarly. The mean probability for the full tree under low knowledge is 28.9%, whereas it is 18.4% under high knowledge. For the collapsed tree under low knowledge, the mean probability is 43.2%, whereas it is 26.7% under the high knowledge condition. Finally, the mean probability is 43.0% for the pruned tree under low knowledge, whereas it is 29.4% under high knowledge.

The two hypotheses were tested both graphically and statistically. Figure 1 shows the data distribution using a boxplot. From this graph, the medians of the Low/Collapsed and Low/Pruned categories are different from that of the Low/Full category. Similarly, the medians of the High/Collapsed and High/Pruned categories differ from that of the High/Full category. Therefore, hypothesis 1 is supported because the state pace partitioning influences the assessment of the subjective probability. Furthermore, the medians of the three levels of partitioning under high-level knowledge are lower than those of the three levels of partitioning under low-level knowledge, respectively. Hence, the level of knowledge influences the assessment of the subjective probability, supporting hypothesis 2.

Figure 1.

Data distribution across the six conditions.

The two-way ANOVA technique was used to test the two hypotheses statistically. The results showed that the overall model was significant with R² = 14.4 %. The main effect of state space partitioning was significant (F (2, 537)  = 18.470, p < 0.001). Therefore, hypothesis 1 is supported. Partitioning the state space has significant impact on the probability estimation. For the full tree, the probability that Samba bank would win the award next time is 24.2%. The mean probability for the collapsed tree was 35.7%, whereas it was 36.3% for the pruned tree. Using the Tukey HSD test for post hoc analysis, significant difference was found between the full tree and collapsed tree (p < 0.001). In addition, there was significant difference between the full tree and pruned tree (p < 0.001). However, there was no significant difference between the collapsed tree and pruned tree (p = 0.970).

Moreover, the main effect of knowledge was significant (F (1, 537)  = 52.384, p < 0.001). Therefore, hypothesis 2 is supported. Providing participants with more knowledge on the scenario had a positive impact on improving the probability estimation accuracy. For low knowledge, the mean probability was 38.1%, while it was 24.9% for high knowledge. This finding is consistent with that of Barrera-Causil et al., ¹¹ for example, providing instructions to participants on the elicitation method results in more accurate parameter assessment.

Finally, no interaction was found between partitioning and knowledge (F (2, 537)  = 0.860, p = 0.424). It is worth noting that partition dependence bias was not completely eliminated. As shown in Table 1, the mean probability for the full tree is lower than that of the collapsed and pruned trees for both levels of knowledge. Therefore, increasing the level of knowledge will not completely debias the partition dependence bias. However, it assists in improving the accuracy of the subjective probability.

Conclusion

The field of judgment and decision making incorporates three types of models: normative, descriptive, and prescriptive. ¹² Normative models use mathematical or philosophical arguments for identifying the best possible solution to a specific problem. The responses of respondents are evaluated against this best solution. If the solution of a particular respondent is close to the best solution, the respondent has no bias in his/her judgment. Otherwise, the solution given by the respondent is considered biased. Descriptive models use psychological theories to explain the judgment and decisions made by respondents. Prescriptive models are used to improve the quality of the decisions made by respondents. Baron ¹² discusses these tree types of models in detail. This study, which investigated the partition dependence bias associated with subjective probability assessment, falls under the descriptive model category. An experiment was conducted, in which the levels of knowledge and the state space partitioning forms were manipulated. Statistical analysis showed that the partition dependence bias was robust and existed regardless of the level of knowledge or information provided. This is a cause of concern for decision makers regarding the accuracy of their subjective probability estimations when presented with a list of options. Differences in the presentation of options may lead to different subjective probability assessments. Despite the robustness of partition dependence bias, the accuracy of probability estimation increased with the level of information provided.

One possible application of this study is the diversification of an individual’s finances among different investment options. An investment agency may provide five specific stocks and inform the investor that stock X had performed best the previous year. Without any additional information, the method of presentation of the stock list will impact the investor decision on allocating his/her money among these options. If the options are presented in a full tree form, the money is expected to be distributed among these options. However, if the options are presented in a collapsed or pruned form, investment allocation is expected to be higher for the stock with the best performance the previous year. To improve the judgment accuracy, the investor should ask the investment agency for a more objective criteria to evaluate the different investment options. This study shows that even with more knowledge, the investor is expected to be affected by the partitioning or presentation of the stocks. However, his/her judgment is expected to improve.

A potential path for future research is to investigate the possibilities for limiting this inherent bias, which falls under prescriptive models. A possible strategy is to ask the subjects to estimate the probabilities using different techniques such that they can be aware of their inconsistent estimations. Another area of research is to provide a group of people with a problem and compare their performance against individual performances. Group performance might be better than individual performance because groups may deliberate and correct each other. These possible ideas for research may improve the quality of probability estimation, when making decisions. As a final research direction, three or more levels of knowledge may be provided, and it can be investigated whether the subjective probability estimates differ across these levels. This study mostly involves only two levels of knowledge, with and without information. Future research should explore this area by providing more than two levels of information to the participants.