Automatic Deduction Path Learning via Reinforcement Learning with
  Environmental Correction
Shuai Xiao Chen Pan Min Wang Xinxin Zhu Siqiao Xue Jing Wang Yunhua Hu James Zhang Jinghua Feng
Automatic bill payment is an important part of business operations in fintech
companies. The practice of deduction was mainly based on the total amount or
heuristic search by dividing the bill into smaller parts to deduct as much as
possible. This article proposes an end-to-end approach of automatically
learning the optimal deduction paths (deduction amount in order), which reduces
the cost of manual path design and maximizes the amount of successful
deduction. Specifically, in view of the large search space of the paths and the
extreme sparsity of historical successful deduction records, we propose a deep
hierarchical reinforcement learning approach which abstracts the action into a
two-level hierarchical space: an upper agent that determines the number of
steps of deductions each day and a lower agent that decides the amount of
deduction at each step. In such a way, the action space is structured via prior
knowledge and the exploration space is reduced. Moreover, the inherited
information incompleteness of the business makes the environment just partially
observable. To be precise, the deducted amounts indicate merely the lower
bounds of the available account balance. To this end, we formulate the problem
as a partially observable Markov decision problem (POMDP) and employ an
environment correction algorithm based on the characteristics of the business.
In the world's largest electronic payment business, we have verified the
effectiveness of this scheme offline and deployed it online to serve millions
of users.