Learning Not to Spoof
David Byrd
As intelligent trading agents based on reinforcement learning (RL) gain
prevalence, it becomes more important to ensure that RL agents obey laws,
regulations, and human behavioral expectations. There is substantial literature
concerning the aversion of obvious catastrophes like crashing a helicopter or
bankrupting a trading account, but little around the avoidance of subtle
non-normative behavior for which there are examples, but no programmable
definition. Such behavior may violate legal or regulatory, rather than physical
or monetary, constraints.
  In this article, I consider a series of experiments in which an intelligent
stock trading agent maximizes profit but may also inadvertently learn to spoof
the market in which it participates. I first inject a hand-coded spoofing agent
to a multi-agent market simulation and learn to recognize spoofing activity
sequences. Then I replace the hand-coded spoofing trader with a simple
profit-maximizing RL agent and observe that it independently discovers spoofing
as the optimal strategy. Finally, I introduce a method to incorporate the
recognizer as normative guide, shaping the agent's perceived rewards and
altering its selected actions. The agent remains profitable while avoiding
spoofing behaviors that would result in even higher profit. After presenting
the empirical results, I conclude with some recommendations. The method should
generalize to the reduction of any unwanted behavior for which a recognizer can
be learned.