Robustly estimating heterogeneity in factorial data using Rashomon
  Partitions
Aparajithan Venkateswaran Anirudh Sankar Arun G. Chandrasekhar Tyler H. McCormick
Many statistical analyses, in both observational data and randomized control
trials, ask: how does the outcome of interest vary with combinations of
observable covariates? How do various drug combinations affect health outcomes,
or how does technology adoption depend on incentives and demographics? Our goal
is to partition this factorial space into ``pools'' of covariate combinations
where the outcome differs across the pools (but not within a pool). Existing
approaches (i) search for a single ``optimal'' partition under assumptions
about the association between covariates or (ii) sample from the entire set of
possible partitions. Both these approaches ignore the reality that, especially
with correlation structure in covariates, many ways to partition the covariate
space may be statistically indistinguishable, despite very different
implications for policy or science. We develop an alternative perspective,
called Rashomon Partition Sets (RPSs). Each item in the RPS partitions the
space of covariates using a tree-like geometry. RPSs incorporate all partitions
that have posterior values near the maximum a posteriori partition, even if
they offer substantively different explanations, and do so using a prior that
makes no assumptions about associations between covariates. This prior is the
$\ell_0$ prior, which we show is minimax optimal. Given the RPS we calculate
the posterior of any measurable function of the feature effects vector on
outcomes, conditional on being in the RPS. We also characterize approximation
error relative to the entire posterior and provide bounds on the size of the
RPS. Simulations demonstrate this framework allows for robust conclusions
relative to conventional regularization techniques. We apply our method to
three empirical settings: price effects on charitable giving, chromosomal
structure (telomere length), and the introduction of microfinance.