In this paper, we consider the problem of autonomous lane changing for self driving vehicles in a multi-lane, multi-agent setting. We present a framework that demonstrates a more structured and data efficient alternative to end-to-end complete policy learning on problems where the high-level policy is hard to formulate using traditional optimization or rule based methods but well designed low-level controllers are available. Our framework uses deep reinforcement learning solely to obtain a high-level policy for tactical decision making, while still maintaining a tight integration with the low-level controller, thus getting the best of both worlds. We accomplish this with Q-masking, a technique with which we are able to incorporate prior knowledge, constraints, and information from a low-level controller, directly in to the learning process thereby simplifying the reward function and making learning faster and data efficient. We provide preliminary results in a simulator and show our approach to be more efficient than a greedy baseline, and more successful and safer than human driving. In recent years, there has been a growing interest in self driving vehicles. Building such autonomous systems has been an active area of research BID23 BID27 BID3 for its high potential in leading to road networks that are much more safer and efficient. One of the fundamental skills a self driving vehicle must possess is an ability to perform lane change maneuvers, which is especially critical on a multi-lane highway in the presence fast moving traffic (as shown in FIG0 . A bad decision at best leads to congestion and at worst leads to accidents BID8 . Reasoning about interactions with other agents and forming an efficient long term strategy while maintaining safety makes this problem challenging and complex.Prior work on lane changing consists of a diverse set of approaches with early work considering vision based control BID21 . Other methods track trajectories BID14 BID19 , use fuzzy control BID6 , model predictive control BID5 , generate a steering command with adaptive control BID15 , consider planning BID23 BID26 , and mixed logic programming BID4 . However majority of the prior work considers the problem only from a local perspective, i.e. changing between adjacent lanes while avoiding the few neighboring vehicles. There is no notion of a goal, like reaching an exit, which would require reasoning about long term decisions on a strategic level when present on a multi-lane highway among traffic. Formulating a control or optimization based problem to handle such a scenario is not straight forward, would require a lot of hand design and tuning, may work only on a subset of cases, and would generally be intractable. The primary roadblock is that there is no abstraction of what the overall ideal policy should look like, only the ideal outcome is know: reaching the exit safely and efficiently (in least amount of time).Reinforcement learning provides a way to learn arbitrary policies giving specific goals. In recent years learning based methods have been used to address similar or related problems, like learning from human driving BID22 , inverse reinforcement learning BID18 , end-to-end methods that map perception inputs (mainly images) directly to control commands BID13 BID9 BID0 BID25 , and methods that understand the scene via learning to make driving decisions BID2 BID17 . Along these lines deep reinforcement learning has had great success in learning policies from raw sensor information BID12 .In this work, we investigate the use and place of deep reinforcement learning in solving the autonomous lane changing problem. In general learning a full policy than can reason about tactical decisions while at same time address continuous control and collision avoidance can be exceedingly difficult with large notorious to train networks. Thus, an ideal approach would strike the right balance by learning the hard to specify high-level tactical policy while relying on established optimization or rule based method for low-level control.We propose a framework that uses deep Q-learning to learn a high-level tactical decision making policy, and also introduce, Q-masking, a novel technique that forces the agent to explore and learn only a subspace of Q-values. This subspace is directly governed by a low-level module that consists of prior knowledge about the system, constraints of the problem, and information from the lowlevel controller. Not only does Q-masking provide the tight integration between the two paradigms: learning high-level policy and using low-level control, but also heavily simplifies the reward function and makes learning faster and data efficient. By relying on a controller for low-level decisions, we are also able to completely eliminate collisions during training or testing, which makes it a possibility to perform training directly on real systems. We present preliminary benchmarks and show that our framework can outperform a greedy baseline in terms of efficiency and humans driving in the simulator in terms of safety and success, while also generalizing to several unseen scenarios without any extra training. We proposed a framework that leverages the strengths of deep reinforcement learning for high-level tactical decision making, and traditional optimization or rule-based methods for low-level control, by striking the right balance between both domains. At the heart of this framework lies, Q-masking, that provides an interface between the two levels. Using Q-masking we are able to incorporate prior knowledge, constraints about the system and information from the lower-level controller, directly in to the training of the network, simplifying the reward function and making learning faster and more data efficient, while completely eliminating collisions during training or testing. We applied our framework on the problem of autonomous lane changing for self driving cars, where the neural network learned a high-level tactical decision making policy. We presented preliminary results and benchmarked our approach against a greedy baseline and humans driving in the simulator and showed that our approach is able to outperform them both on different metrics with a more efficient and much safer policy. Finally, we demonstrated zero shot generalizations on several unseen scenarios.
A framework that provides a policy for autonomous lane changing by learning to make high-level tactical decisions with deep reinforcement learning, and maintaining a tight integration with a low-level controller to take low-level actions. Considers the problem of autonomous lane changing for self-driving cars in multi-lane multi-agent slot car setting, proposes a new learning strategy Q-masking - coupling a defined low level controller with a high level tactical decision making policy. This paper proposes a deep Q-learning approach to the problem of lane change using "Q-masking," which reduces the action space according to contraints or prior knowledge. Authors propose a method which uses a Q-learning-based high-level policy that is combined with a contextual mask derived from safety-contraints and low-level controllers, which disable certain actions from being selectable at certain states. 