Vlearn: Off-Policy Learning with Efficient State-Value Function
  Estimation
Fabian Otto Philipp Becker Vien Ang Ngo Gerhard Neumann
Existing off-policy reinforcement learning algorithms typically necessitate
an explicit state-action-value function representation, which becomes
problematic in high-dimensional action spaces. These algorithms often encounter
challenges where they struggle with the curse of dimensionality, as maintaining
a state-action-value function in such spaces becomes data-inefficient. In this
work, we propose a novel off-policy trust region optimization approach, called
Vlearn, that eliminates the requirement for an explicit state-action-value
function. Instead, we demonstrate how to efficiently leverage just a
state-value function as the critic, thus overcoming several limitations of
existing methods. By doing so, Vlearn addresses the computational challenges
posed by high-dimensional action spaces. Furthermore, Vlearn introduces an
efficient approach to address the challenges associated with pure state-value
function learning in the off-policy setting. This approach not only simplifies
the implementation of off-policy policy gradient algorithms but also leads to
consistent and robust performance across various benchmark tasks. Specifically,
by removing the need for a state-action-value function Vlearn simplifies the
learning process and allows for more efficient exploration and exploitation in
complex environments