Towards Model-Free LQR Control over Rate-Limited Channels
Aritra Mitra Lintao Ye Vijay Gupta
Given the success of model-free methods for control design in many problem
settings, it is natural to ask how things will change if realistic
communication channels are utilized for the transmission of gradients or
policies. While the resulting problem has analogies with the formulations
studied under the rubric of networked control systems, the rich literature in
that area has typically assumed that the model of the system is known. As a
step towards bridging the fields of model-free control design and networked
control systems, we ask: \textit{Is it possible to solve basic control problems
- such as the linear quadratic regulator (LQR) problem - in a model-free manner
over a rate-limited channel?} Toward answering this question, we study a
setting where a worker agent transmits quantized policy gradients (of the LQR
cost) to a server over a noiseless channel with a finite bit-rate. We propose a
new algorithm titled Adaptively Quantized Gradient Descent (\texttt{AQGD}), and
prove that above a certain finite threshold bit-rate, \texttt{AQGD} guarantees
exponentially fast convergence to the globally optimal policy, with \textit{no
deterioration of the exponent relative to the unquantized setting}. More
generally, our approach reveals the benefits of adaptive quantization in
preserving fast linear convergence rates, and, as such, may be of independent
interest to the literature on compressed optimization.