The Unintended Consequences of Discount Regularization: Improving
  Regularization in Certainty Equivalence Reinforcement Learning
Sarah Rathnam Sonali Parbhoo Weiwei Pan Susan A. Murphy Finale Doshi-Velez
Discount regularization, using a shorter planning horizon when calculating
the optimal policy, is a popular choice to restrict planning to a less complex
set of policies when estimating an MDP from sparse or noisy data (Jiang et al.,
2015). It is commonly understood that discount regularization functions by
de-emphasizing or ignoring delayed effects. In this paper, we reveal an
alternate view of discount regularization that exposes unintended consequences.
We demonstrate that planning under a lower discount factor produces an
identical optimal policy to planning using any prior on the transition matrix
that has the same distribution for all states and actions. In fact, it
functions like a prior with stronger regularization on state-action pairs with
more transition data. This leads to poor performance when the transition matrix
is estimated from data sets with uneven amounts of data across state-action
pairs. Our equivalence theorem leads to an explicit formula to set
regularization parameters locally for individual state-action pairs rather than
globally. We demonstrate the failures of discount regularization and how we
remedy them using our state-action-specific method across simple empirical
examples as well as a medical cancer simulator.