A mixed policy to improve performance of language models on math
  problems
Gang Chen
When to solve math problems, most language models take a sampling strategy to
predict next word according conditional probabilities. In the math reasoning
step, it may generate wrong answer. Considering math problems are
deterministic, we propose a mixed policy exploration approach to solve math
problems with reinforcement learning. In peculiar, we propose a two level token
exploration policy: the abstract level explores next token with probability and
the second level is deterministic. Specifically, the abstract level policy will
decide whether the token is operator or operand with probability sampling,
while the second level is deterministic to select next token with the highest
score in a greedy way. We test our method on GSM8K dataset with GPT-2 model,
and demonstrate more than $2\%$ performance gain. Our implementation is
available at https://github.com/vividitytech/math_lm_rl.