Theorem [Duality Formula for Variational Inference] — Let Θ Θ be a set endowed with an appropriate σ σ -field F , and two probability measures P and Q, which formulate two probability spaces ( Θ Θ , F , P ) and ( Θ Θ , F , Q ) , with Q ≪ ≪ P . ( Q ≪ ≪ P indicates that Q is absolutely continuous with respect to P.) Let h be a real-valued integrable random variable on ( Θ Θ , F , P ) . Then the following equality holds