RL-MPCA: A Reinforcement Learning Based Multi-Phase Computation
  Allocation Approach for Recommender Systems
Jiahong Zhou Shunhui Mao Guoliang Yang Bo Tang Qianlong Xie Lebin Lin Xingxing Wang Dong Wang
Recommender systems aim to recommend the most suitable items to users from a
large number of candidates. Their computation cost grows as the number of user
requests and the complexity of services (or models) increases. Under the
limitation of computation resources (CRs), how to make a trade-off between
computation cost and business revenue becomes an essential question. The
existing studies focus on dynamically allocating CRs in queue truncation
scenarios (i.e., allocating the size of candidates), and formulate the CR
allocation problem as an optimization problem with constraints. Some of them
focus on single-phase CR allocation, and others focus on multi-phase CR
allocation but introduce some assumptions about queue truncation scenarios.
However, these assumptions do not hold in other scenarios, such as retrieval
channel selection and prediction model selection. Moreover, existing studies
ignore the state transition process of requests between different phases,
limiting the effectiveness of their approaches.
  This paper proposes a Reinforcement Learning (RL) based Multi-Phase
Computation Allocation approach (RL-MPCA), which aims to maximize the total
business revenue under the limitation of CRs. RL-MPCA formulates the CR
allocation problem as a Weakly Coupled MDP problem and solves it with an
RL-based approach. Specifically, RL-MPCA designs a novel deep Q-network to
adapt to various CR allocation scenarios, and calibrates the Q-value by
introducing multiple adaptive Lagrange multipliers (adaptive-$\lambda$) to
avoid violating the global CR constraints. Finally, experiments on the offline
simulation environment and online real-world recommender system validate the
effectiveness of our approach.