Traffic Signal Control Using Lightweight Transformers: An
  Offline-to-Online RL Approach
Xingshuai Huang Di Wu Benoit Boulet
Efficient traffic signal control is critical for reducing traffic congestion
and improving overall transportation efficiency. The dynamic nature of traffic
flow has prompted researchers to explore Reinforcement Learning (RL) for
traffic signal control (TSC). Compared with traditional methods, RL-based
solutions have shown preferable performance. However, the application of
RL-based traffic signal controllers in the real world is limited by the low
sample efficiency and high computational requirements of these solutions. In
this work, we propose DTLight, a simple yet powerful lightweight Decision
Transformer-based TSC method that can learn policy from easily accessible
offline datasets. DTLight novelly leverages knowledge distillation to learn a
lightweight controller from a well-trained larger teacher model to reduce
implementation computation. Additionally, it integrates adapter modules to
mitigate the expenses associated with fine-tuning, which makes DTLight
practical for online adaptation with minimal computation and only a few
fine-tuning steps during real deployment. Moreover, DTLight is further enhanced
to be more applicable to real-world TSC problems. Extensive experiments on
synthetic and real-world scenarios show that DTLight pre-trained purely on
offline datasets can outperform state-of-the-art online RL-based methods in
most scenarios. Experiment results also show that online fine-tuning further
improves the performance of DTLight by up to 42.6% over the best online RL
baseline methods. In this work, we also introduce Datasets specifically
designed for TSC with offline RL (referred to as DTRL). Our datasets and code
are publicly available.