Assessing the causal effects of interventions on ordinal outcomes is an important objective of many educational and behavioral studies. Under the potential outcomes framework, we can define causal effects as comparisons between the potential outcomes under treatment and control. However, unfortunately, the average causal effect, often the parameter of interest, is difficult to interpret for ordinal outcomes. To address this challenge, we propose to use two causal parameters, which are defined as the probabilities that the treatment is beneficial and strictly beneficial for the experimental units. However, although well-defined for any outcomes and of particular interest for ordinal outcomes, the two aforementioned parameters depend on the association between the potential outcomes and are therefore not identifiable from the observed data without additional assumptions. Echoing recent advances in the econometrics and biostatistics literature, we present the sharp bounds of the aforementioned causal parameters for ordinal outcomes, under fixed marginal distributions of the potential outcomes. Because the causal estimands and their corresponding sharp bounds are based on the potential outcomes themselves, the proposed framework can be flexibly incorporated into any chosen models of the potential outcomes and is directly applicable to randomized experiments, unconfounded observational studies, and randomized experiments with noncompliance. We illustrate our methodology via numerical examples and three real-life applications related to educational and behavioral research.
AgrestiA. (2010). Analysis of ordinal categorical data (2nd ed.). Hoboken, NJ: John Wiley.
2.
AgrestiA.KateriM. (2017). Ordinal probability effect measures for group comparisons in multinomial cumulative link models. Biometrics, 73, 214–219.
3.
AngristJ. D.ImbensG. W.RubinD. B. (1996). Identification of causal effects using instrumental variables (with discussion). Journal of the American Statistical Association, 91, 444–455.
4.
BakerS. G. (2011). Estimation and inference for the causal effect of receiving treatment on a multinomial outcome: An alternative approach. Biometrics, 67, 319–323.
5.
BeranR. (1988). Balanced simultaneous confidence sets. Journal of the American Statistical Association, 83, 679–697.
6.
BeranR. (1990). Refining bootstrap simultaneous confidence sets. Journal of the American Statistical Association, 85, 417–426.
7.
BickelP. J.GotzeF.Van ZwetW. R. (1997). Resampling fewer than n observations: Gains, losses, and remedies for losses. Statistica Sinica, 7, 1–31.
8.
BickelP. J.SakovA. (2008). On the choice of m in the m-out-of-n bootstrap and confidence bounds for extrema. Statistica Sinica, 18, 967–985.
9.
BradleyR. A.KattiS. K.CoonsI. J. (1962). Optimal scaling for ordered categories. Psychometrika, 27, 355–374.
10.
ChengJ. (2009). Estimation and inference for the causal effect of receiving treatment on a multinomial outcome. Biometrics, 65, 96–103.
11.
ChengJ.SmallD. S. (2006). Bounds on causal effects in three-arm trials with non-compliance. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 68, 815–836.
12.
ChernozhukovV.LeeS.RosenA. (2013). Intersection bounds: Estimation and inference. Econometrica, 81, 667–737.
13.
DemidenkoE. (2016). The p-value you can’t buy. The American Statistician, 70, 33–38.
14.
DempsterA. P.LairdN.RubinD. B. (1977). Maximum likelihood estimation from incomplete data using the EM algorithm (with discussion). Journal of the Royal Statistical Society: Series B (Statistical Methodology), 39, 1–38.
15.
DiazI.ColantuoniE.RosenblumM. (2016). Enhanced precision in the analysis of randomized trials with ordinal outcomes. Biometrics, 72, 422–431.
16.
DingP.DasguptaT. (2016). A potential tale of two by two tables from completely randomized experiments. Journal of the American Statistical Association, 111, 157–168.
17.
DjebbariH.SmithJ. A. (2008). Heterogeneous impacts in PROGRESA. Journal of Econometrics, 145, 64–80.
18.
EfronB.TibshiraniR. J. (1994). An introduction to the Bootstrap. Boca Raton, FL: Chapman & Hall.
19.
FanY.ParkS. S. (2009). Partial identification of the distribution of treatment effects and its confidence sets. Advanced Economies, 25, 3–70.
20.
FanY.ParkS. S. (2010). Sharp bounds on the distribution of treatment effects and their statistical inference. Econometric Theory, 26, 931–951.
21.
FanY.ShermanR.ShumM. (2014). Identifying treatment effects under data combination. Econometrica, 82, 811–822.
22.
FrankM. J.NelsenR. B.SchweizerB. (1987). Best-possible bounds for the distribution of a sum: A problem of Kolmogorov. Probability Theory and Related Fields, 74, 199–211.
23.
FreedmanD. A. (2008). Randomization does not justify logistic regression. STASIS Trailer, 23, 237–249.
24.
FrumentoP.MealliF.PaciniB.RubinD. B. (2012). Evaluating the effect of training on wages in the presence of noncompliance, nonemployment, and missing outcome data. Journal of the American Statistical Association, 107, 450–466.
25.
GadburyG. L.IyerH. K. (2000). Unit-treatment interaction and its practical consequences. Biometrics, 56, 882–885.
26.
GrilliL.MealliF. (2008). Nonparametric bounds on the causal effect of university studies on job opportunities using principal stratification. Journal of Educational and Behavioral Statistics, 33, 111–130.
27.
HeckmanJ. J.SmithJ.ClementsN. (1997). Making the most out of programme evaluations and social experiments: Accounting for heterogeneity in programme impacts. The Review of Economic Studies, 64, 487–535.
28.
HiranoK.ImbensG. W.RidderG. (2003). Efficient estimation of average treatment effects using the estimated propensity score. Econometrica, 71, 1161–1189.
29.
HiranoK.PorterJ. (2012). Impossibility results for nondifferentiable functionals. Econometrica, 80, 1769–1790.
30.
HoffP. D. (2009). A first course in Bayesian statistical methods. New York, NY: Springer Science & Business Media.
31.
HorowitzJ. L.ManskiC. F. (2000). Nonparametric analysis of randomized experiments with missing covariate and outcome data. Journal of the American Statistical Association, 95, 77–84.
32.
HuangE. J.FangE. X.HanleyD. F.RosenblumM. (2017). Inequality in treatment benefits: Can we determine if a new treatment benefits the many or the few. Biostatistics, 18, 308–324.
33.
ImbensG. W.RubinD. B. (2015). Causal inference for statistics, social, and biomedical sciences: An introduction. New York, NY: Cambridge University Press.
34.
JiangZ.DingP. (2018). Using missing types to improve partial identification with missing binary outcomes. Annals of Applied Statistics, in press.
35.
JuC.GengZ. (2010). Criteria for surrogate end points based on causal distributions. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 72, 129–142.
36.
KimJ. H. (2014). Identifying the distribution of treatment effects under support restrictions. arXiv. Retrieved fromhttps://arxiv.org/abs/1410.5885
37.
KreiderB.PepperJ. V. (2007). Disability and employment: Reevaluating the evidence in light of reporting errors. Journal of the American Statistical Association, 102, 432–441.
38.
LaberE. B.MurphyS. A. (2011). Adaptive confidence intervals for the test error in classification. Journal of the American Statistical Association, 106, 904–913.
39.
LeeD. S. (2009). Training, wages, and sample selection: Estimating sharp bounds on treatment effects. The Review of Economic Studies, 76, 1071–1102.
40.
LiuR. C.BrownL. D. (1993). Nonexistence of informative unbiased estimators in singular problems. Annals of Statistics, 21, 1–13.
41.
LongD. M.HudgensM. G. (2013). Sharpening bounds on principal effects with covariates. Biometrics, 69, 812–819.
42.
LuedtkeA. R.Van der LaanM. J. (2016). Statistical inference for the mean outcome under a possibly non-unique optimal treatment strategy. Annals of Statistics, 44, 713–742.
43.
MakarovG. D. (1982). Estimates for the distribution function of a sum of two random variables when the marginal distributions are fixed. Theory of Probability and Its Applications, 26, 803–806.
44.
ManskiC. F. (1997). Monotone treatment response. Econometrica, 65, 1311–1334.
45.
ManskiC. F. (2003). Partial identification of probability distributions. New York, NY: Springer.
46.
ManskiC. E.PepperJ. V. (2000). Monotone instrumental variables with an application to the returns to schooling. Econometrica, 68, 997–1010.
47.
ManskiC. F.PepperJ. V. (2009). More on monotone instrumental variables. The Econometrics Journal, 12, 200–216.
48.
MealliF.PaciniB. (2013). Using secondary outcomes to sharpen inference in randomized experiments with noncompliance. Journal of the American Statistical Association, 108, 1120–1131.
49.
NelsenR. B. (2006). An introduction to copulas (2nd ed.). New York, NY: Springer.
50.
NewcombeR. G. (2006a). Confidence intervals for an effect size measure based on the mann–whitney statistic. Part 1: General issues and tail-area-based methods. Statistics in Medicine, 25, 543–557.
51.
NewcombeR. G. (2006b). Confidence intervals for an effect size measure based on the mann–whitney statistic. Part 2: Asymptotic methods and evaluation. Statistics in Medicine, 25, 559–573.
52.
NeymanJ. (1923). On the application of probability theory to agricultural experiments. essay on principles. Section 9. Statistical Science, 5, 465–472.
53.
OenemaA.BrugJ.LechnerL. (2001). Web-based tailored nutrition education: Results of a randomized controlled trial. Health Education Research, 16, 647–660.
54.
ParrW. C. (1983). A note on the jackknife, the bootstrap and the delta method estimators of bias and variance. Biometrika, 70, 719–722.
55.
PearlJ. (2009). Causality: Models, reasoning and inference (2nd ed.). New York, NY: Cambridge University Press.
56.
PraetM.DesoeteA. (2014). Enhancing young children’s arithmetic skills through non-intensive, computerised kindergarten interventions: A randomised controlled study. Teaching and Teacher Education, 39, 56–65.
57.
RichardsonA.HudgensM. G.GilbertP. B.FineJ. P. (2014). Nonparametric bounds and sensitivity analysis of treatment effects. Statistical Science, 29, 596–618.
58.
RomanoJ. P.ShaikhA. M. (2008). Inference for identifiable parameters in partially identified econometric models. Journal of Statistical Planning and Inference, 138, 2786–2807.
59.
RomanoJ. P.ShaikhA. M. (2010). Inference for the identified set in partially identified econometric models. Econometrica, 78, 169–211.
60.
RosenbaumP. R. (2001). Effects attributable to treatment: Inference in experiments and observational studies within a discrete pivot. Biometrika, 88, 219–231.
61.
RosenbaumP. R.RubinD. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70, 41–55.
62.
RubinD. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66, 688–701.
63.
RubinD. B. (1978). Bayesian inference for causal effects: The role of randomization. Annals of Statistics, 6, 34–58.
64.
RubinD. B. (1980). Comment on “Randomization analysis of experimental data: The Fisher randomization test” by D. Basu. Journal of the American Statistical Association, 75, 591–593.
65.
RüschendorfL. (1982). Random variables with maximum sums. Advances in Applied Probability, 14, 623–632.
66.
ScharfsteinD. O.ManskiC. F.AnthonyJ. C. (2004). On the construction of bounds in prospective studies with missing ordinal outcomes: Application to the good behavior game trial. Biometrics, 60, 154–164.
67.
SchochetP. Z.CaoJ. B. R.GlazermanS.GradyA.GritzM.McConnellS.JohnsonT.BurghardtJ. (2003). National job corps study: Data documentation and public use files, Volume I. Documentation. Washington, DC: Mathematica Policy Research.
68.
SennC. Y.EliasziwM.BarataP. C.ThurstonW. E.Newby-ClarkI. R.RadtkeH. L.HobdenK. L. (2015). Efficacy of a sexual assault resistance program for university women. The New England Journal of Medicine, 372, 2326–2335.
69.
SmithT. W.MarsdenP.HoutM.KimJ. (2013). General social surveys, 1972–2010. Chicago, IL: National Opinion Research Center.
70.
StrassenV. (1965). The existence of probability measures with given marginals. Annals of Mathematical Statistics, 36, 423–439.
71.
VolfovskyA.AiroldiE. M.RubinD. B. (2015). Causal inference for ordinal outcomes. arXiv. Retrieved fromhttps://arxiv.org/abs/1501.01234
72.
WilliamsonR. C.DownsT. (1990). Probabilistic arithmetic. i. numerical methods for calculating convolutions and dependency bounds. International Journal of Approximate Reasoning, 4, 89–158.
73.
YangF. (2014). Causal inference methods for addressing censoring by death and unmeasured confounding using instrumental variables(PhD thesis). University of Pennsylvania. Retrieved fromhttps://repository.upenn.edu/dissertations/AAI3622136/
74.
YangF.SmallD. S. (2016). Using post-quality of life measurement information in censoring by death problems. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 78, 299–318.
75.
ZhangJ. L.RubinD. B.MealliF. (2009). Likelihood-based analysis of causal effects of job-training programs using principal stratification. Journal of the American Statistical Association, 104, 166–176.
76.
ZhouW. (2008). Statistical inference for P(X < Y). Statistics in Medicine, 27, 257–279.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
