The importance of data-driven decision-making is rapidly increasing thanks in part to the growing availability and accessibility of data sets and analysis tools. Yet, applicable insight can be difficult due to biases and anomalies in data. An often overlooked phenomenon is mix effects, in which subgroups of data exhibit patterns opposite to the data as a whole. This phenomenon is widespread and often leads inexperienced analysts to draw incorrect statistical conclusions. In this paper we present Wiggum, an interactive visual analysis system for uncovering both mix effects and special cases known as Simpson’s paradox. A Python-based web implementation of Wiggum lets users interactively analyze multidimensional data sets to reveal various forms of mix effects. Through use cases, we describe how Wiggum supports the examination of mix effects in three real data sets and demonstrate how a combination of visualization techniques—heatmaps, trend plots, small multiples, coordinated multiple views, and dynamic queries for multi-attribute drill-down—are effective for analyzing mix effects. We conducted a user study to evaluate Wiggum, focusing on users’ ability to comprehend the statistical concepts, identify the corresponding visual patterns, and perform common analysis tasks correctly and efficiently. We discuss usability issues, utility limitations, and outline future directions to improve Wiggum.
BickelPJHammelEAO'ConnellJW. Sex bias in graduate admissions: Data from Berkeley. Science1975; 187(4175): 398–404.
2.
PearlJ.Comment: understanding Simpson’s paradox. Am Stat2014; 68(1): 8–13.
3.
ArmstrongZWattenbergM.Visualizing statistical mix effects and Simpson’s paradox. IEEE Trans Vis Comput Graph2014; 20(12): 2132–2141.
4.
van WijkJ. The value of visualization. In: VIS 05. IEEE Visualization, 2005. IEEE, pp. 79–86.
5.
PirolliPCardS. The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis. In Proceedings of International Conference on Intelligence Analysis, vol. 5. McLean, VA, USA, 2005, pp. 2–4.
6.
CookKAThomasJJ.Illuminating the path: The research and development agenda for visual analytics. Technical Report.Richland, WA: Pacific Northwest National Lab.(PNNL), 2005.
7.
GaheganMWachowiczMHarrowerM, et al. The integration of geographic visualization with knowledge discovery in databases and geocomputation. Cartogr Geogr Inf Sci2001; 28(1): 29–44.
8.
GaheganM. Beyond tools: visual support for the entire process of Giscience. In: DykesJMacEachrenAMKraakMJ (eds) Exploring Geovisualization. Elsevier, 2005, pp. 83–99.
9.
SachaDStoffelAStoffelF, et al. Knowledge generation model for visual analytics. IEEE Trans Vis Comput Graph2014; 20(12): 1604–1613.
10.
BuolamwiniJGebruT. Gender shades: Intersectional accuracy disparities in commercial gender classification. In: Conference on Fairness, Accountability and Transparency. PMLR, 2018, pp. 77–91.
11.
CabreraÁAEppersonWHohmanF, et al. FairVis: Visual analytics for discovering intersectional bias in machine learning. In: 2019 IEEE Conference on Visual Analytics Science and Technology (VAST)2019, IEEE, pp. 46–56.
12.
BlythCR.On Simpson’s paradox and the sure-thing principle. J Am Stat Assoc1972; 67(338): 364–366.
13.
NorrisF.Median pay in us is stagnant, but low-paid workers lose. N Y Times2013; 26.
14.
FreitasAA. On objective measures of rule surprisingness. In: European Symposium on Principles of Data Mining and Knowledge Discovery. Springer, 1998, pp. 1–9.
15.
SalimiBGehrkeJSuciuD. Bias in OLAP queries: Detection, explanation, and removal. In Proceedings of the 2018 International Conference on Management of Data. ACM, 2018, pp. 1021–1035.
16.
WardropRL.Simpson’s paradox and the hot hand in basketball. Am Stat1995; 49(1): 24–28.
17.
PearlJMackenzieD.The Book of why: the New Science of cause and effect. Basic Books, 2018.
18.
KievitRAFrankenhuisWEWaldorpLJ, et al. Simpson’s paradox in psychological science: A practical guide. Front Psychol2013; 4: 513.
19.
XuCBrownSMGrantC. Detecting Simpson’s paradox. In Florida Artificial Intelligence Research Society Conference (FLAIRS), 2018, pp. 221–224.
20.
HernánMAClaytonDKeidingN.The Simpson’s paradox unraveled. Int J Epidemiol2011; 40(3): 780–785.
21.
TuYKGunnellDGilthorpeMS.Simpson’s paradox, Lord’s paradox, and suppression effects are the same phenomenon–the reversal paradox. Emerg Themes Epidemiol2008; 5(1): 2.
22.
AlipourfardNFennellPGLermanK. Can you trust the trend?: Discovering Simpson’s paradoxes in social data. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining. ACM, 2018, pp. 19–27.
23.
AlipourfardNFennellPLermanK.Using Simpson’s paradox to discover interesting patterns in behavioral data. Twelfth International AAAI Conference on Web and Social Media2018; 12(1): 2–11.
24.
LermanK.Computational social scientist beware: Simpson’s paradox in behavioral data. J Comput Soc Sci2018; 1: 49–58.
25.
DaySM.Simpson’s paradox and Major League Baseball’s hall of fame. Red1994; 4: 8.
26.
CharigCRWebbDRPayneSR, et al. Comparison of treatment of renal calculi by open surgery, percutaneous nephrolithotomy, and extracorporeal shockwave lithotripsy. Br Med J1986; 292(6524): 879–882.
27.
JuliousSAMulleeMA.Confounding and Simpson’s paradox. BMJ1994; 309(6967): 1480–1481.
28.
FroelichW. Mining association rules from database tables with the instances of Simpson’s paradox. In: MorzyTHärderTWrembelR (eds) Advances in databases and Information Systems. Springer, 2013, pp.79–90.
29.
ArahOA.The role of causal reasoning in understanding Simpson’s paradox, Lord’s paradox, and the suppression effect: covariate selection in the analysis of observational studies. Emerg Themes Epidemiol2008; 5(1): 5.
30.
PearlJ.Simpson’s paradox: An anatomy. Department of Statistics, UCLA; 2011.
31.
SalimiBColeCLiP, et al. HypDB: a demonstration of detecting, explaining and resolving bias in OLAP queries. Proceedings VLDB Endowment2018; 11(12): 2062–2065.
32.
WangJMuellerK.The visual causality analyst: an interactive interface for causal reasoning. IEEE Trans Vis Comput Graph2016; 22(1): 230–239.
33.
WangJMuellerK. Visual causality analysis made practical. In: 2017 IEEE Conference on Visual Analytics Science and Technology (VAST). IEEE, 2017, pp. 151–161.
34.
PearlJ.Causality. Cambridge University Press, 2009.
35.
BandyoapdhyayPSNelsonDGreenwoodM, et al. The logic of Simpson’s paradox. Synthese2011; 181(2): 185–208.
36.
PavlidesMGPerlmanMD.How likely is Simpson’s paradox?Am Stat2009; 63(3): 226–233.
37.
FabrisCCFreitasAA. Discovering surprising patterns by detecting occurrences of Simpson’s paradox. In: BramerMMacintoshACoenenF (eds) Research and development in Intelligent Systems XVI. Springer, 2000, pp.148–160.
38.
RückerGSchumacherM.Simpson’s paradox visualized: the example of the rosiglitazone meta-analysis. BMC Med Res Methodol2008; 8(1): 34.
39.
ChenABengtssonTHoTK.A regression paradox for linear models: Sufficient conditions and relation to Simpson’s paradox. Am Stat2009; 63(3): 218–225.
40.
YanJNGuZLinH, et al. Silva: Interactively assessing machine learning fairness using causality. In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 2020, pp. 1–13.
41.
XieXDuFWuY.A visual analytics approach for exploratory causal analysis: exploration, validation, and applications. IEEE Trans Vis Comput Graph2021; 27(2): 1448–1458.
42.
JinZGuoSChenN, et al. Visual causality analysis of event sequence data. IEEE Trans Vis Comput Graph2021; 27(2): 1343–1352.
43.
ChoudhryASharmaMChunduryP, et al. Once upon a time in visualization: understanding the use of textual narratives for causality. IEEE Trans Vis Comput Graph2021; 27(2): 1332–1342.
44.
ElmqvistNTsigasP. Causality visualization using animated growing polygons. In IEEE Symposium on Information Visualization 2003 (IEEE Cat. No. 03TH8714). IEEE, pp. 189–196.
45.
GuoYBinnigCKraskaT. What you see is not what you get!: Detecting Simpson’s paradoxes during data exploration. In Proceedings of the 2nd Workshop on Human-In-the-Loop Data Analytics. ACM, 2017, p. 2.
46.
CrottyAGalakatosAZgraggenE, et al. Vizdom: interactive analytics through pen and touch. Proceedings VLDB Endowment2015; 8(12): 2024–2027.
47.
WexlerJPushkarnaMBolukbasiT, et al. The what-if tool: interactive probing of machine learning models. IEEE Trans Vis Comput Graph2020; 26(1): 56–65.
48.
AhnYLinYR.FairSight: Visual analytics for fairness in decision making. IEEE Trans Vis Comput Graph2020; 26(1): 1086–1095.
49.
WangQXuZChenZ, et al. Visual analysis of discrimination in machine learning. IEEE Trans Vis Comput Graph2021; 27: 1470–1480.
50.
LiptonZC.The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery. Queue2018; 16(3): 31–57.
51.
SahaDSchumannCMcelfreshD, et al. Measuring non-expert comprehension of machine learning fairness metrics. In: International Conference on Machine Learning. PMLR, 2020, pp. 8377–8387.
52.
PedregosaFVaroquauxGGramfortA, et al. Scikit-learn: machine learning in Python. J Mach Learn Res2011; 12: 2825–2830.
53.
AllisonJSSantanaL(Jaco) VisagieIJH. (Jaco) A primer on simple measures of association taught at undergraduate level. Teach Stat2022; 44(3): 96–103.
54.
de WinterJCGoslingSDPotterJ.Comparing the Pearson and spearman correlation coefficients across distributions and sample sizes: A tutorial using simulations and empirical data. Psychol Methods2016; 21(3): 273–290.
55.
CrouxCDehonC.Influence functions of the spearman and Kendall correlation measures. Stat Methods Appl2010; 19: 497–515.
56.
ShneidermanB. The eyes have it: A task by data type taxonomy for information visualizations. In: The craft of Information Visualization. Elsevier, 2003, pp.364–371.
Ngrok: Secure introspectable tunnels to localhost. https://ngrok.com (2013).
61.
AmarREaganJStaskoJ. Low-level components of analytic activity in information visualization. In IEEE Symposium on Information Visualization, 2005. INFOVIS 2005. IEEE, 2005, pp. 111–117.
62.
LamHToryMMunznerT.Bridging from goals to tasks with design study analysis reports. IEEE Trans Vis Comput Graph2018; 24(1): 435–445.
63.
CrawfordK.The trouble with bias. Conference on Neural Information Processing Systems, Keynote2017.
64.
HolsteinKWortman VaughanJDauméHIII, et al. Improving fairness in machine learning systems: What do industry practitioners need? In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, 2019, pp. 1–16.
65.
DworkCHardtMPitassiT, et al. Fairness through awareness. In Proceedings of the 3rd Innovations in Theoretical Computer Science Conference, 2012, pp. 214–226.
