Visual Active Tracking (VAT) aims at following a target object by autonomously controlling the motion system of a tracker given visual observations. Previous work has shown that the tracker can be trained in a simulator via reinforcement learning and deployed in real-world scenarios. However, during training, such a method requires manually specifying the moving path of the target object to be tracked, which cannot ensure the tracker’s generalization on the unseen object moving patterns. To learn a robust tracker for VAT, in this paper, we propose a novel adversarial RL method which adopts an Asymmetric Dueling mechanism, referred to as AD-VAT. In AD-VAT, both the tracker and the target are approximated by end-to-end neural networks, and are trained via RL in a dueling/competitive manner: i.e., the tracker intends to lockup the target, while the target tries to escape from the tracker. They are asymmetric in that the target is aware of the tracker, but not vice versa. Specifically, besides its own observation, the target is fed with the tracker’s observation and action, and learns to predict the tracker’s reward as an auxiliary task. We show that such an asymmetric dueling mechanism produces a stronger target, which in turn induces a more robust tracker. To stabilize the training, we also propose a novel partial zero-sum reward for the tracker/target. The experimental results, in both 2D and 3D environments, demonstrate that the proposed method leads to a faster convergence in training and yields more robust tracking behaviors in different testing scenarios. For supplementary videos, see: https://www.youtube.com/playlist?list=PL9rZj4Mea7wOZkdajK1TsprRg8iUf51BS 
 The code is available at https://github.com/zfw1226/active_tracking_rl Visual Active Tracking (VAT) aims at following a target object by autonomously controlling the motion system of a tracker given visual observations. VAT is demanded in many real-world applications such as autonomous vehicle fleet (e.g., a slave-vehicle should follow a master-vehicle ahead), service robots and drones (e.g., a drone is required to follow a person when recording a video). To accomplish the VAT task, one typically needs to perform a sequence of tasks such as recognition, localization, motion prediction, and camera control. However, conventional visual tracking BID0 BID29 BID24 BID11 BID3 BID13 aims to solely propose a 2D bounding box of the target frame by frame, and does not actively take into consideration the control of camera. Thus, compared to the problem of "passive" tracking, VAT is more practical and challenging.With the advancement of deep reinforcement learning BID35 BID25 BID26 , training an end-to-end deep neural network via reinforcement learning for VAT is shown to be feasible BID21 BID20 . The authors learn a policy that maps raw-pixel observation to control signal straightly with a Conv-LSTM network. Such an end-to-end approach could save the effort of tuning an extra camera controller. Meanwhile, it also outperforms the conventional methods where the passive tracker is equipped with a hand-engineered camera controller. However, the performance of the deep reinforcement learning based tracker is still limited by the training methods. Due to the "trial-and-error" nature of reinforcement learning, it is infeasible to directly train the tracker in the real world. Alternatively, virtual environments are always utilized to generate sufficient data for training without tedious human labeling. Nevertheless, to deploy the trained tracker in the real world, one has to overcome the virtual-to-real gap. One solution can be building numbers of high-fidelity environments . However, it is expensive and tedious to build such environments for VAT. Both the visual rendering (illumination, texture, etc.) and the physical properties should be carefully designed to emulate the real world. Suppose we carry out VAT where the target is a pedestrian. To build the environment, one has to not only model the human's appearance, but also design physical rules and the pedestrian's trajectory so that it moves naturally like a human beings. Recently, BID21 tried to overcome the virtual-to-real gap by applying the so-called environment augmentation technique. They diversify the visual appearance by changing the placement of the background objects and by flipping left-right the screen frame. However, they neglect another important factor, that is, the motion of the target for VAT task. Intuitively, the complexity and diversity of the target motion in training will impact the generalization of the data-driven tracker. For example, if the target only moves forward during training, the tracker may over fit to move straightly and fail to track other motion patterns, like a sharp turn. In this paper, we have proposed an asymmetric dueling mechanism for visual active tracking (AD-VAT). Within AD-VAT, agents of tracker and target are learned in an adversarial manner. With the design of the partial zero-sum reward structure and tracker-aware model, the reinforced active tracker outperforms baseline methods. Experiments including ablation study in both 2D and 3D environments verify the effectiveness of the proposed mechanism.As future work, we would like to: 1) investigate the theoretical justification of applying modern Multi-Agent RL methods BID18 BID32 to solving Partially Observable Markov Game and finding Nash Equilibrium. 2) further develop the mechanism/model for active tracking in more complex environment (e.g., environments with a number of obstacles and moving distractors); 3) adapt the mechanism to other tasks (e.g., learning to grab a moving object).
We propose AD-VAT, where the tracker and the target object, viewed as two learnable agents, are opponents and can mutually enhance during training. This work aims to address the visual active tracking problem with a training mechanism in which the tracker and target serve as mutual opponents This paper presents a simple multi-agent Deep RL task where a moving tracker tries to follow a moving target. Proposes a novel reward function - "partial zero sum", which only encourages the tracker-target competition when they are close and penalizes whey they are too far.