---
abstract: 'For complex real-world systems, designing controllers are a difficult task. With the advent of neural networks as a proxy for complex function approximators, it has become popular to learn the controller directly. However, these controllers are specific to a given task and need to be relearned for a new task. Alternatively, one can learn just the model of the dynamical system and compose it with external controllers. Such a model is task (and controller) agnostic and must generalize well across the state space. This paper proposes learning a “sufficiently accurate” model of the dynamics that explicitly enforces small residual error on pre-defined parts of the state-space. We formulate task agnostic controller design for this learned model as an optimization problem with state and control constraints that is solved in an online fashion. We validate this approach in simulation using a challenging contact-based Ball-Paddle system.'
bibliography:
- 'paper.bib'
---

Introduction
============

[[One of the fundamental problems in many fields, such as robotics, is the design of controllers for complex dynamical systems. For the most part, controllers rely on the availability of a mathematical model that describes the system. However, deriving the models and estimating all their parameters (e.g., mass, torques, friction) is often impossible. This is especially true for systems with contact dynamics.]{}]{} With the advent of deep learning, an alternative paradigm has emerged which proposes learning the controller directly [@ho2016generative; @khan2017memory; @silver2017predictron]. The drawbacks with this approach are twofold; supervised expert data is often needed and the learned controller may not generalize to new tasks.

[An alternative to directly learning a controller is to instead learn a model for the complex dynamical system under consideration. One may learn a *forward* model that produces a mapping from the control inputs and current state to predicted behavior of the system. This is in contrast with the previously described direct controller learning approach where an *inverse* model is learned that maps current state and desired behaviour to control signals. One of the advantages of a forward model is that it eliminates the need for a carefully curated set of expert demonstrations across different tasks. This is due to the fact that the behaviour of the controller is completely defined by the model and the optimization problem posed to solve for the controller. ]{}

![**Optimizing a sufficiently accurate model:** The solid line represents the true trajectory of the ball, while the dotted lines represent predicted trajectories. The paddle actions, $a$, are optimized with gradient descent on a defined loss function. For the task of bouncing the ball consistently, a sufficiently accurate model that guarantees prediction error within some bound $\epsilon$ may be sufficient. []{data-label="fig:main"}](images/icmldrawings.png){width="35.00000%"}

In cases where the task parameters, such as goal location or actuation limits, change, a new controller involves simply solving a different optimization problem instead of relearning an inverse model. While there has been some work to learn task robust policies for inverse models, such as using Goal-Conditioned Policies [@nair2018visual; @pathak2018zero], it is not clear how one can enforce general constraints for the inverse model.

Learning a model is fundamentally different from learning a controller in that the controller is an end in itself but a model is useful only as an intermediate step to learn a controller. Therefore, learning a model with arbitrary accuracy is not necessarily warranted. Rather, we want to learn a model that is sufficiently accurate for controller design. This paper formulates the problem of learning a model as a constrained optimization problem in which the required accuracy of the model is imposed as a set of constraints (Section \[sec:model\_learning\]). The constraints that are imposed in the model accuracy are intended to ensure that the model is [*sufficiently accurate*]{} for a variety of control tasks. The main advantage of learning sufficiently accurate models is an appropriate tailoring of the sample complexity. A bonus advantage derived from our problem formulation is that the accuracy constraints that are imposed in model learning allow for trading off the accuracy of the model in different parts of the state space. For example, in Fig. \[fig:main\] we consider the problem of determining the dynamical trajectories followed by a ball that is hit by a paddle in order to design a controller that would allow us to keep the ball in the air by repeatedly hitting it when the vertical position crosses a certain threshold. We argue that for this problem it is advantageous to learn models that predict with an accuracy dependent on the velocity of the ball.

Contributions
-------------

This paper proposes a constraint-based formulation for learning and controlling dynamical systems. Contact dynamics is difficult to model and we show in this paper how optimizing a controller with constraints is a more viable solution to even an analytic controller on the learned model. The contributions of this paper are: (i) a novel constrained objective function for model learning with neural networks and the adjoining constrained optimization problem for learning the controller, and (ii) a primal-dual method to solve both these problems that has small duality gap.

The paper is organized as follows: Section \[sec:model\_learning\] describes the sufficiently accurate formulation for model learning and Sec. \[sec:control\] utilizes this learned model for designing controllers with state and action constraints. We demonstrate the efficacy of this approach on a challenging Ball-Paddle system in Sec. \[sec:experiments\].

Related Work {#sec:relatedwork}
============

The idea of learning a model from data and using it to control systems is not new. PILCO [@deisenroth2011pilco] learns a probabilistic forward model with Gaussian Process Regression, and is later extended to Bayesian Neural Networks [@gal2016improving]. Guided Policy Search [@levine2014learning; @levine2016end] uses a Gaussian Mixture Model for use as an expert policy. Both formulate an optimization problem with the task of maximizing an expected reward. This allows a policy to be trained by backpropagating through the forward model. [@jordan1992forward] and [@gillespie2018learning] both learn neural network models. The latter then formulates a convex optimization problem by linearizing the neural network. However, it is has been observed that linearizing highly nonlinear systems often performs poorly [@diehl2005real]. The aforementioned methods learn models with an objective function similar to $$\min_\theta \mathbb{E}\left\| \phi_\theta(s, a) - f(s, a) \right\|$$ where $f(s, a)$ represents the true model dynamics and $\phi_\theta(s, a)$ is the learned model. @byravan2017se3 design a specific neural network architecture for their forward model which uses a normalized objective. However, there are slack terms introduced to avoid numerical instability caused by dividing with small numbers. [@amos2018differentiable] presents a way to differentiate through the controller so that a model can be learned end to end. However, this method requires that the policy has converged to a fix point which can be hard to achieve in complicated systems.

Learning models is also of great interest in reinforcement learning where a forward model can increase sample efficiency [@sutton1990integrated; @levine2014learning]. In addition, it has been found that learning forward and inverse models can provide additional rewards to help train a reinforcement learning agent [@pathak2017curiosity]. [@achiam2017constrained] introduced a policy search method for reinforcement learning that can impose expectation constraints on states and actions.

There has also been some work in multi task learning that tries to obtain task agnostic policies. One way to do this is with meta learning algorithms such as MAML [@finn2017model] which learns a policy that can be adapted to different task parameters. [@pathak2018zero; @nair2018visual] learn policies that include the goal as an input to encourage the policy to generalize across different targets. These methods work for small numbers of task parameters but may have difficulty scaling up as each additional parameter added to a policy will decrease the sample efficiency of the algorithm.

Sufficiently Accurate Model Learning {#sec:model_learning}
====================================

Let us denote the state of the system by ${s} \in \mathcal{S}$ and the control inputs or actions denoted by ${a}\in \mathcal{A}$. The forward dynamics model is defined by a function $f:\mathcal{S}\times\mathcal{A}\to \mathcal{S}$ that takes as inputs the state and action at time $n$ and whose output is the state at time $n+1$ $${s}_{n+1} = f({s}_n, {a}_n).$$ We denote $\phi_\theta : \mathcal{S} \times \mathcal{A} \to \mathcal{S}$ as the neural network approximation of the true model, where $\theta\in\mathbb{R}^N$ represents the network parameters. The classical approach to model learning consists of finding the parameters $\theta$ that minimize the expectation of a defined scalar-valued loss function. Hence, the problem is written as $$\label{eq:model_learning}
     \min_\theta  \mathbb{E}_{( {s},  {a}) \sim D} l(s,a,\phi_\theta),$$ where $D$ denotes the sampling distribution over the state-action space. In this paper we formulate the problem of learning a model as a constrained optimization problem in which the required accuracy of the model is imposed as a set of constraints. The constraints that are imposed in the model accuracy are intended to ensure that the model is sufficiently accurate for a variety of control tasks. To that end, we consider a partition $\cup_{i=0}^P\mathcal{K}_i=\mathcal{S}\times\mathcal{A}$ of the state-action space and constraints $h:\mathcal{S}\times\mathcal{A}\times\mathbb{R}^N\to \mathbb{R}^P$, where each component of the constraint is imposed on a different region of the space-action space. With these definitions, we propose the following optimization problem $$\begin{aligned}
\label{eq:constrained_model_learning}
    P^*_\theta &=\min_\theta  \mathbb{E}_{( s,  a) \sim D} l(s,a,\phi_\theta) \mathbb{I}_{\mathcal{K}_0}(s,a) \\
    & \text{s.t. } \mathbb{E}_{(s,a) \sim D} h_i(s,a,\theta) \mathbb{I}_{\mathcal{K}_i}(s,a) \leq \epsilon \quad \forall i\ldots P, \nonumber\end{aligned}$$ where $\mathbb{I}_{\mathcal{K}_i}(s,a)$ with $i=0\ldots P$ are indicator functions taking the value $1$ if $(s,a)\in\mathcal{K}_i$ and $0$ otherwise. In the next section we present a primal dual algorithm to solve the optimization problem . Before doing so, we consider an example to illustrate the sufficiently accurate learning framework.

**Example 1.** As an example of the previous formulation we consider the minimization of a normalized error. The error tolerance in a forward model is directly related to the magnitude of the quantity being estimated. For example, 1 unit of error when the output is 100 is very different from 1 unit of error when the output is 1. A natural way to mitigate these difference is to consider a normalized error, $\left\| \phi_\theta( {s},  {a}) - f( {s},  {a}) \right\| / \left\| f( {s},  {a}) \right\|$. It is often the case that small values of $f( {s},  {a})$ are hard to even measure accurately, thus, it matters only that the error is bounded rather than minimized. The set of non-small values will be defined as $\mathcal{K} = \{ ( {s},  {a}) : \left\| f( {s},  {a}) \right\| \geq \delta \}$, where $\delta > 0$. One can then pose the following model learning objective $$\begin{aligned}
\label{eq:constrained_model_learning_example}
    \min_\theta & \mathbb{E}_{( {s},  {a}) \sim D} \frac{\left\| \phi_\theta( {s},  {a}) - f( {s},  {a}) \right\|}{ \left\| f( {s},  {a}) \right\| } \mathbb{I}_{\mathcal{K}} \\
    & \text{s.t. } \mathbb{E}_{( {s},  {a}) \sim D} \left\| \phi_\theta( {s},  {a}) - f( {s},  {a}) \right\| \mathbb{I}_{\mathcal{K}^C}  \leq \epsilon . \nonumber\end{aligned}$$ This objective simply states that for all large enough $f( {s},  {a})$, the normalized error should be minimized, and all small values should be bounded by $\epsilon$.

Primal-Dual algorithm {#Methodology}
---------------------

The problem of sufficiently accurate learning can be formulated as the constrained optimization problem . A possible approach to solve said problem is through primal-dual methods. Let us start by defining a vector of multipliers $\lambda\in\mathbb{R}^P_+$ and the Lagrangian of associated to problem $$L(\theta, \lambda) = \mathbb{E}[ \mathcal{L}( {s},  {a}, \phi_\theta)] + \lambda^\top \mathbb{E} [g( {s},  {a}, \phi_\theta)].$$ where to simplify notation we have defined $\mathcal{L}( {s},  {a}, \phi_\theta)  = l(s,a,\phi_\theta)\mathbb{I}_{\mathcal{K}_0}$ and where each component of $g$ is defined as $g_i(s,a,\phi_{\theta}) = h_i(s,a,\phi_{\theta})\mathbb{I}_{\mathcal{K}_i}-\epsilon$ and we have dropped the distribution $D$. The Lagrangian allows us to define the dual problem as $$\begin{aligned}
\label{eq:opt_problem_dual}
    D_\theta^* &= \max_\lambda \min_\theta L(\theta, \lambda)\\
   &  \text{s.t. } \lambda \succeq 0 \nonumber. \end{aligned}$$ The duality gap is defined by the difference $P_\theta^* - D_\theta^*$. When an optimization problem has zero duality gap (we show in Section \[sec\_zero\_duality\_gap\] that this is the case for the problem of learning sufficiently accurate models), then the solutions of both optimization problems in and are the same. The optimal primal variable, $\theta^*$, must necessarily minimize $L$ given the optimal dual variable, $\lambda^*$. Likewise, the optimal dual variable must maximize the Lagrangian given the optimal primal variable. This leads to the widely used primal-dual method, where the Lagrangian is iteratively minimized with respect to the primal variable and maximized with respect to the dual. This minimization/maximization can be solved by computing gradient descent steps with respect to $\theta$ and gradient ascent steps with respect to $\lambda$. The gradient of the Lagrangian with respect to $\theta$ takes the form $$\begin{aligned}
    \nabla_\theta L(\theta, \lambda) &= \mathbb{E} [\nabla_\theta \mathcal{L}( {s}, {a}, \phi_\theta)] +\sum_{i=1}^P \lambda_i \mathbb{E} [\nabla_\theta g_i( {s}, {a}, \phi_\theta)], \end{aligned}$$ and the gradient of the Lagrangian with respect to the multiplier $\lambda$ yields $$\nabla_\lambda L(\theta, \lambda) = \mathbb{E} [g( {s}, {a}, \phi_\theta)].$$ Each iteration must be followed by a projection of $\lambda$ onto the positive orthant to make sure it is non negative. Given a static distribution of states and actions to optimize this model, the algorithm is summarized in Algorithm \[alg:model\_learning\].

\[alg:model\_learning\] $\lambda \leftarrow \lambda_0$ , set dual variable to an initial value $\theta \leftarrow \theta_0$ $\theta$

Note that gradient ascent and descent steps can be modified to include momentum or include more complicated algorithms such as ADAM [@kingma2014adam]. In the next section we show that the sufficiently accurate learning formulated in has almost zero duality gap which motivates the use of the primal-dual algorithm to solve it.

Almost Zero Duality Gap {#sec_zero_duality_gap}
-----------------------

By definition of the dual problem, it follows that the dual solution $D^*_\theta$ is always a lower bound for the primal solution $P^*_\theta$. That is, $P^*_\theta \geq D^*$ [@boyd2004convex]. The converse is however not true, but we can show that in the case of sufficiently accurate learning the duality gap is small. To that end we consider the following generalization of the problem $$\begin{aligned}
\label{eq:opt_problem_func}
P^* &= \min_{\phi \in \Phi} \mathbb{E}_B [l( {s},  {a}, \phi)] \\
& \text{s.t. } \mathbb{E}_B [g( {s},  {a}, \phi)] \leq 0, \nonumber\end{aligned}$$ where instead of optimizing the weights of a function approximator, the optimization is done over the space of all possible integrable functions $\Phi$. We now have the following result shown in @ribeiro2012optimal.

Given the optimization problem in , if (i) the distribution $B$, is non-atomic, (ii) the inequality constraints define a compact region within $\Phi$, and (iii) there exists a strictly feasible solution $(\phi, \lambda)$ (Slater’s condition) then the duality gap is zero.

With this result, a natural question to ask is how the parameterization of functions $\phi \in \Phi$ affects the duality gap. The following theorem from @eisen2018learning describes sufficient conditions under which a proxy of the duality gap is bounded.

\[thm:parameterized\_gap\] For the optimization problems  and , if,

-   there exists a strictly feasible solution $(\phi, \lambda)$ to the primal problem ;

-   the parameterization $\theta$ of the function space $\Phi$ is a universal approximator within error $\delta$, i.e. $\mathbb{E}_{D} \left\| \phi - \phi_\theta \right\| \leq \delta$ for some $\theta$ for all $\phi \in \Phi$;

-   the loss function, $l( {s},  {a}, \phi)$ is expectation-wise Lipschitz continuous, i.e., there exists a $K$ such that $\mathbb{E} \left\| l( {s},  {a}, \phi_1) -  l( {s},  {a}, \phi_2)\right\|_\infty \leq K \mathbb{E} \left\| \phi_1 - \phi_2 \right\|_\infty$ for all $\phi_1, \phi_2 \in \Phi$;

then the optimal parameterized dual value $D_{\theta}^*$ is bounded by $$\label{eq:dual_gap}
        P^* \leq D_\theta^* \leq P^* + \| {\lambda}^* \|_1 K \delta,$$ where $\lambda^*$ is the solution of .

The following proposition formalizes that the problem of sufficient accurate learning in has small duality gap. **Proposition 1.** *Sufficiently Accurate Learning and its dual, defined in and respectively, satisfy the assumptions of Theorem 2. Hence holds for sufficiently accurate learning.*

First we look at the Lipschitz condition. $$\begin{aligned}
    \mathbb{E} \left\| l( {s},  {a}, \phi_1) - l( {s},  {a}, \phi_1) \right\|_\infty &\leq
    \mathbb{E} \left\| \frac{\mathbb{I}_{\mathcal{A}}}{| f( {s},  {a})| }\right\| \left\| | \phi_1( {s},  {a}) - f( {s},  {a}) | - | \phi_2( {s},  {a}) - f( {s},  {a}) | \right\|_\infty \label{eq:lemma_1}\\
    &\leq \mathbb{E} \left\| \frac{\mathbb{I}_{\mathcal{A}}}{| f( {s},  {a})| }\right\| \left\| | \phi_1( {s},  {a}) - f( {s},  {a}) - \phi_2( {s},  {a}) + f( {s},  {a})| \right\|_\infty \label{eq:lemma_2} \\
    &\leq   \frac{1}{\min_{( {s},  {a})} | f( {s},  {a})| }\mathbb{E} \left\| \phi_1( {s},  {a}) - \phi_2( {s},  {a}) \right\|_\infty \label{eq:lemma_3} \\
    &= \frac{1}{\epsilon} \mathbb{E} \left\| \phi_1( {s},  {a}) - \phi_2( {s},  {a}) \right\|_\infty \label{eq:lemma_4}\end{aligned}$$

Where step to comes from the triangle inequality and step to comes from the definition of the set $\mathcal{A} = \{ ( {s},  {a}) : \left\| f( {s},  {a}) \right\| \geq \delta \}$. Thus, the loss is expectation-wise Lipschitz continuous.

A strictly feasible solution, $\phi$, exists since the ground truth model, which is representable in $\Phi$, is strictly feasible.

Lastly, the parameterization used is a neural network which is a universal function approximator.

Therefore the conditions for Theorem 2 are fulfilled.

This proposition is easy to show and a proof of it along with Theorem \[thm:parameterized\_gap\] is given in the supplementary material. This theorem states that for a large enough neural network, the gap between the optimal solution with no function approximation and the optimal solution to the parameterized dual problem can be made as small as desired. While this does not mean that the optimal parameterized dual problem can be solved, it does motivate the use of the primal dual method. A sample training curve is shown in Fig. \[fig:model\_learning\].

![**Model training.** The orange curve shows the value of the objective function, $l$. Teal curve shows the value of the constraint function, $g$. Dark blue curve shows the dual variable, $\lambda$. The objective and constraint functions are smoothed and shown in a darker color. The dotted red line shows the $y=0$ line and the constraint curve must go below that for the problem to have a feasible solution. The curves have different values and are normalized so that they can be displayed on one graph.[]{data-label="fig:model_learning"}](images/training_curve.png)

The primal and dual variables interact in a very natural manner. As the constraints are violated, the dual variable increases in magnitude which heavily penalizes constraint violations. In order to drive the constraints into the feasible region, the primal loss may suffer. However, in the end, when the constraints are satisfied, the dual variables stop increasing and converge.

When a problem has zero duality gap, the dual problem can be rewritten as $$\begin{aligned}
\label{eq:opt_problem_dual_2}
D_\theta^* &= \min_\theta L(\theta, \lambda^*)\end{aligned}$$ if the dual maximizer $\lambda^*$ is known. This is simply a minimization of the objective function with a weighted constraint term. Thus, solving the constrained optimization problem is the same as solving an unconstrained problem with soft constraints that are weighted by the optimal dual variables.

Control with Sufficiently Accurate Models {#sec:control}
=========================================

The previous section describes a method for learning a sufficiently accurate model for controller design. In this section we describe how the learned model can be used to that end. A policy is a function from the state to the action space $\pi:\mathcal{S}\to \mathcal{A}$. A specific way to describe a desired policy is to minimize some cost, $c( {s},  {a}, \phi_\theta)$ associated with the performance of the system. This could be for instance the difference between the predicted state of the model and some desired state. The action selected has to typically satisfy some constraints imposed by the model, e.g., the action is bounded by the maximum torque of the motor in a robotic system, or obstacles in an environment. Denote by $g_c:\mathcal{S}\times\mathcal{A}\times\mathbb{R}^N\to \mathbb{R}^M$ the constraints imposed to the system and define the following policy $$\begin{aligned}
    \pi( {s,\theta}) &= \operatorname*{arg\,min}_{ {a}} c( {s},  {a}, \phi_\theta) \\
    &\quad \text{s.t.  } g_c( {s},  {a}, \phi_\theta) \leq 0 \nonumber.\end{aligned}$$ Observe that the policy depends on the learned model, and thus it is also a function of the parameters $\theta$. In particular, if the residual error of learning the model dynamics is low, we can expect good performance out of such policies. Since, the model is a neural network, it is easy to obtain gradients $\nabla_{ {a}} \phi_{\theta}( {s},  {a})$ which can be used for the same primal-dual method mentioned in Sec. \[Methodology\]. The only difference is that instead of optimizing model weights, the solver is optimizing for the inputs. This procedure is shown in Algorithm \[alg:controller\] where the Lagrangian is defined as $$L_c( {a}, \lambda) = c( {s},  {a}, \theta) + \lambda g_c( {s},  {a}, \theta).$$

\[alg:controller\] $\lambda \leftarrow \lambda_0$ , set dual variable to an initial value $ {a} \leftarrow  {a}_0$, set action to some initial action $ {a}$

Learning a model and using such an optimization problem as the policy allows different controllers to be designed for different goals and constraints. This flexibility is not a feature of many end-to-end systems. While there exists some recent work that attempts to differentiate through the optimization procedure itself [@amos2018differentiable], it can be hard to train.

Model Refinement
----------------

After a model is learned, it can be improved such that it has higher accuracy on states that the controller may visit more. The simple method of using a static dataset collected from such a controller does not work due to the fact that once a new model is trained, the controller will choose different actions as it is dependent on the model itself. Thus, the distribution of states the model was trained to minimize is no longer the same distribution that the controller will induce. This is a problem that also plagues Imitation Learning. One solution is to use the DAgger algorithm [@ross2011reduction]. It involves an iterative process where a model is trained on a distribution of states induced by the controller. Then, the controller based on the new model is used to collect more data. The procedure is described in Algorithm \[alg:dagger\].

\[alg:dagger\] $\theta \leftarrow \theta_0$ $D \leftarrow \{\}$ $\theta$

Experiments {#sec:experiments}
===========

To demonstrate the effectiveness of our approach, we consider a Ball-Paddle system. In this problem, one is tasked with bouncing a ball using a paddle in three dimensional space at a specific location. This seemingly simple problem is actually quite difficult to solve well. Previously, there have been analytic controllers developed for simple models of ball dynamics [@rizzi1994further; @buhler1990stable]. Changes to the contact model or other dynamics requires further analysis. This is also a difficult problem for Reinforcement Learning alone to solve. To have any sort of positive reward, the event where the paddle hits the ball has to occur several times. The probability of such an event occurring with an initial random policy is low. In addition, it can be hard to specify a good reward function to describe the behaviour of bouncing a ball and the policy will have to be retrained for different behaviours. It can be much simpler writing out a loss function with constraints.

The state space of the system includes the 3D position and velocity of both the ball and the paddle. The paddle also has angular position and velocity for roll and pitch. Thus, this state space has 16 dimensions. The actions are the linear forces on the paddle in 3 dimensions as well as roll and pitch torques. This gives a continuous control input with 5 dimensions. A simulation of this Ball-Paddle system is created in Mujoco [@todorov2012mujoco] using DeepMind Control Suite bindings [@tassa2018deepmind].

The learned model is used with an analytic controller. One difficult part of this problem is modeling the contact or bounce of the ball. A simple semi-elastic model of the ball using a coefficient of restitution does not model the behaviour well enough. Thus, the contact model is a natural candidate to be learned. The model takes the state of the system and controller action at the point of contact as input and predicts how the ball will bounce. This prediction is in the form of an $(x,y)$ coordinate where the ball will next cross a predetermined $(x,y)$ plane. The action space of the model is the velocity and orientation of the paddle.

A simple form of how the analytic controller uses the model is the following policy, $$\begin{aligned}
\label{eq:ball_paddle_opt}
    \pi( {s}) =& \min_{ {a}} \left\| \phi_\theta( {s,a}) -  {p}_{desired} \right\| \\
    & \text{s.t. }  {a}_{min} \leq \left\|  {a} \right\| \leq  {a}_{max} \nonumber \\
    & \quad \quad \phi_\theta(s,a) \not \in P_{obstacle} \nonumber,\end{aligned}$$ where $ {p}_{desired}$ is a desired location for the ball to bounce at in the $(x,y)$ plane, $ {a}_{min}$ and $ {a}_{max}$ are constraints on the magnitudes of the action, and $P_{obstacle}$ is a set of locations for which the ball is not permitted to bounce at. The goal of this optimization problem is to get the ball as close to the target while maintaining action and state constraints. More details of this controller are given in the supplementary material.

All forward models are multilayer perceptrons with two hidden layers of 128 and 64 neurons. Leaky rectified linear activations are used for the hidden layers.

![Ball-Paddle Simulator](images/simulator.png){width="30.00000%"}

Sufficiently Accurate Models
----------------------------

To investigate the performance of our model, we first consider the effect of learning a model with an objective described by Eq. \[eq:constrained\_model\_learning\_example\] rather than Eq. \[eq:model\_learning\]. Does the normalization and constraints make a difference in the quality and performance of the model? In Fig. \[fig:model\_errs\], the magnitude of the errors is plotted against the magnitude of the output state.

![**Model errors vs. state magnitude**. On the left side, data from a model trained with Eq. \[eq:model\_learning\] is shown. The right side shows data from a model using Eq. \[eq:constrained\_model\_learning\]. The unconstrained problem leads to a good loss in expectation, however the errors are distributed poorly. There might be large errors for states with a small magnitude. The normalized loss on the right allows the large states to have large error and small states to have small error.[]{data-label="fig:model_errs"}](images/model_err.png){width="47.50000%"}

In the unconstrained objective Eq. \[eq:model\_learning\] for model learning, the errors are distributed more or less randomly. States with small magnitudes can have large errors even though, on average, the model error is small. With the sufficiently accurate formulation Eq. \[eq:constrained\_model\_learning\], smaller state magnitudes correspond to smaller errors. This suggests that perhaps the sufficiently accurate model may be easier to use in a controller as small state magnitudes do not have large errors.

To test the effect of the distribution of model errors on controller performance, 500 different Ball-Paddle tasks were generated. Each task varies in goal location of the ball as well as in velocity constraints. Each task is tested with both models using the same controller and results are shown in Table \[table:model\_errs\].

                Unconstrained   Constrained
  ------------ --------------- -------------
    Failure       $20.8 \%$      $0.8 \%$
   Mean error     $0.3136$       $0.2124$

  : **Controller performance with learned models**. A failure is when the ball falls off the paddle, or when it is hit very far away and can not recover. The first row shows what percentage of the 500 different tasks failed. The mean error is the average error of the remaining tasks that did not fail. The error is calculated as the mean position error over time. Thus, it is a crude measure of both how fast the controller gets to the goal and how well it stays on it.[]{data-label="table:model_errs"}

![**Action constraints**. The same learned model is used in two different controllers with different constraints on how large the action can be. The solid line is the trajectory of a controller with a lower action magnitudes, while the dotted line is the trajectory of a controller with higher action magnitudes. The ball can be controlled to bounced in a stable fashion for each set of constraints.[]{data-label="fig:speed_constraints"}](images/speed_constraints.png){width="40.00000%"}

![**State constraints**. The same learned model is used in two different controllers with different constraints on the state space. The orange controller has no state constraints, while the teal controller has a constraint that prevents the ball from bouncing within the blue circle. The paths of both controllers are plotted and the dots along the path represent points where the ball bounces on the paddle. Without state constraints, the controller can go straight through the obstacle region. Using state constraints, the ball takes a path that bounces over the obstacle.[]{data-label="fig:pos_constraints"}](images/pos_constraints.png){width="40.00000%"}

We hypothesize that Algorithm \[alg:controller\] can provide better results when the model errors are more consistent. If moving a small distance in the state space can lead to large changes in model errors, the controller may have a hard time finding a good solution.

Constraints
-----------

A key aspect of this framework is that different constraints can be applied easily to the same model to yield different controller behaviour. These are the state and action constraints in . First, the effect of changing the action constraints governing how hard the paddle can hit the ball is examined in Fig. \[fig:speed\_constraints\]. The action constraints are a minimum and maximum limit on the relative velocity between the ball and paddle at the point of impact.

State constraints are also examined. The constraints can specify a region of the $(x,y)$ plane that the ball can not bounce in and the controller must then respect this constraint. The results are shown in Fig. \[fig:pos\_constraints\].

Analytic and Learned Models
---------------------------

The goal of learning a model is so that it can improve in situations where an analytic model is hard or missing. For the Ball-Paddle system, an analytic model can be derived for the reflection of the ball on the paddle. This model makes a basic assumption that the velocity of the ball, when hitting the paddle, will be reflected along the normal direction of the paddle. Some energy will be absorbed by the paddle and the exact amount is determined by a coefficient of restitution. This coefficient can be measured by bouncing the ball on the paddle while it is flat and observing the successive heights that the ball will reach.

This model seems reasonable according to basic laws of physics and can be tuned carefully by measuring its physical constants. However, this model makes errors in the direction and speed of the reflected ball as it does not exactly follow the assumptions made. This can lead to errors in the controller. Fig. \[fig:analyticvslearned\] shows the path taken by the controller using both the learned model and analytic model. The analytic model overcompensates for the errors it makes and becomes unstable, while the learned model reaches the target location.

![**Imitation policy**. The position of both the ball and paddle are shown. The red line indicates the target ($x,y$) location for the ball. The dotted vertical lines indicate timesteps when the ball collides with the paddle. This policy performs better than reinforcement learning alone but performs worse than the model based method.[]{data-label="fig:imitation"}](images/inv_mode.png){width="45.00000%"}

![image](images/analyticvslearned.png){width="90.00000%"}

Reinforcement and Imitation Learning {#sec:rl}
------------------------------------

We compare the performance of a model free online reinforcement learning algorithm on the problem considered in this paper. We task TRPO [@schulman2015trust] with learning a policy that maximizes rewards for bouncing the ball at a certain height. The reward for this task is a weighted sum of the euclidean distance between the ball’s current position and the desired position and absolute difference between the ball’s current speed and a desired speed. We observe that even after a large number of iterations, the policy fails to learn anything meaningful. This can be attributed to the immensely long sequence of correct actions one must execute to even hit the ball.

Another natural question that is asked is how does Imitation Learning fare? In this case, the data used is vital. An experiment is performed by using the data collected while doing model refinement on one of the learned models. An imitation policy is trained to map the current state at impact and desired position in the $(x,y)$ plane to paddle actions. This is done by minimizing an l2 loss between the action chosen by the controller with the learned model and the action predicted by the imitation policy. The results are shown in Fig. \[fig:imitation\]. While this performs much better than the Reinforcement Learning algorithm, the position of the ball doesn’t quite converge to the desired target. Imitation Learning can be used to distill the expert policy from the forward model or multiple expert policies into a faster imitation policy much like in [@levine2014learning].

Discussion and Future Work
==========================

This paper presents a constrained learning approach to learn forward models as well as utilize them for control. This framework has the flexibility to set different constraints to induce different behaviours with the same learned model. However, due to the nature of posing an optimization problem for every action, the policy is more computationally expensive to compute than a simple forward pass through a policy network. There are several ways that this work can be extended. One will look at reducing the computational load by using the model to optimize for a whole policy rather than specific actions. This framework can be extended to solve more general Model Predictive Control problems. In addition, currently a learned model is used with an analytic controller. Reinforcement learning can be used to improve the policy from a base controller rather than from scratch.

Acknowledgements {#acknowledgements .unnumbered}
================

The authors would like to thank Pratik Chaudhari for thoughtful discussions and ideas. In addition, the authors would like to thank the following funding sources: ARL grants W911NF-08-2-0004 and W911NF-10-2-0016, ARO grant W911NF-13-1-0350, N00014-14-1-0510, DARPA grant HR001151626/HR0011516850 and IIS-1328805. This material is based upon work supported by the Na- tional Science Foundation Graduate Research Fellowship Program under Grant No. DGE-1321851. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.

Proofs
======

\[thm:parameterized\_gap\] **[@eisen2018learning]** Given the optimization problems (\[eq:opt\_problem\_func\]), (\[eq:opt\_problem\_dual\]).

$$\begin{aligned}
\label{eq:opt_problem_func}
P^* &= \min_{\phi \in \Phi} \mathbb{E}_B [l(\boldsymbol{s}, \boldsymbol{a}, \phi)] \\
& \text{s.t. } \mathbb{E}_B [g(\boldsymbol{s}, \boldsymbol{a}, \phi)] \leq 0 \nonumber\end{aligned}$$

$$\begin{aligned}
\label{eq:opt_problem_dual}
    D_\theta^* &= \min_\theta L(\theta, \lambda^*)\end{aligned}$$

and the following assumptions:

-   there exists a strictly feasible solution, $(\phi, \lambda)$, to the primal problem (\[eq:opt\_problem\_func\])

-   the parameterization $\theta$ of the function space $\Phi$ is a universal approximator within error $\delta$, i.e. $\mathbb{E}_{D} \left\| \phi - \phi_\theta \right\| \leq \delta$ for some $\theta$ for all $\phi \in \Phi$.

-   the loss function, $l(\boldsymbol{s}, \boldsymbol{a}, \phi)$ is expectation-wise Lipschitz continuous, i.e. there exist $K$ such that $\mathbb{E} \left\| l(\boldsymbol{s}, \boldsymbol{a}, \phi_1) -  l(\boldsymbol{s}, \boldsymbol{a}, \phi_2)\right\|_\infty \leq K \mathbb{E} \left\| \phi_1 - \phi_2 \right\|_\infty$ for all $\phi_1, \phi_2 \in \Phi$.

then the optimal parameterized dual value $D_{\theta}^*$ is bounded by, $$P^* \leq D_\theta^* \leq P^* + \| \vec{\lambda}^* \|_1 K \delta$$

Since (\[eq:opt\_problem\_func\]) has a zero duality gap, consider the optimal primal dual variables, $(f^*, \vec{\lambda}^*)$ that attains the solution value of $P^* = D^*$ $$\begin{aligned}
        P^* &= L(f^*, \vec{\lambda}^*)\\ 
        &\leq L(f_\theta^*, \vec{\lambda}^*) \label{upper_1}\\ 
        &\leq L(f_\theta^*, \vec{\lambda}^*_\theta) = D_\theta^* \label{upper_2}
        \end{aligned}$$ where $(f_\theta^*, \vec{\lambda}^*_\theta)$ is the primal dual variables that attains maximum to the dual parameterized problem. Line \[upper\_1\] comes from the fact that the functions a the parameterization can represent is a subset of the functions considered in the optimization, thus the value obtained by the parameterized function can always be achieved by an unparameterized function. Line \[upper\_2\] comes from the fact that $ \vec{\lambda}^*_\theta$ maximizes the value of $L(f_\theta^*, \vec{\lambda})$. This gives us a lower bound on the solution to (\[param\_dual\]). Note that the optimal dual variables for (\[func\_primal\]) and in general are not the same as the optimal dual variables for (\[param\_primal\]).

Next, we will show an upperbound to the parameterized dual problem. $$\begin{aligned}
        D_\theta^* &= \max_{\vec{\lambda}} \min_{\theta} L(f_\theta, \vec{\lambda}) \\ 
        &= \max_{\vec{\lambda}} \min_{\theta} L(f^*, \vec{\lambda}) + L(f_\theta, \vec{\lambda}) - L(f^*, \vec{\lambda}) \label{eq:thm_1}\\
        &= \max_{\vec{\lambda}} [L(f^*, \vec{\lambda}) + \min_{\theta}  [L(f_\theta, \vec{\lambda}) - L(f^*, \vec{\lambda})]] \label{eq:thm_2} \\
        &\leq \max_{\vec{\lambda}} [L(f^*, \vec{\lambda}) + \min_{\theta}  |L(f_\theta, \vec{\lambda}) - L(f^*, \vec{\lambda})|] \label{eq:thm_3} \\
        &\leq \max_{\vec{\lambda}} [L(f^*, \vec{\lambda}) + K\delta] \label{eq:thm_4}\\
        &= P^* + K\epsilon
        \end{aligned}$$ where the step from line \[eq:thm\_2\] to line \[eq:thm\_3\] comes from the fact the minimization over a function will always be less than the minimization over its absolute value. The step from line \[eq:thm\_3\] to line \[eq:thm\_4\] comes from the assumption that the loss and constraints are expectation-wise Lipschitz continuous and thus the Lagrangian (a weighted sum of the loss and constraints) is also Lipschitz continuous. The $\epsilon$ comes from the universal function approximation.

Putting the two bounds together, we obtain $$P^* \leq D_\theta^* \leq P^* + K\delta$$

**Proposition 1.** *The loss function $l$ satisfies assumptions of Theorem 2.*

First we look at the Lipschitz condition. $$\begin{aligned}
    \mathbb{E} \left\| l(\boldsymbol{s}, \boldsymbol{a}, \phi_1) - l(\boldsymbol{s}, \boldsymbol{a}, \phi_1) \right\|_\infty &\leq 
    \mathbb{E} \left\| \frac{\mathbb{I}_{\mathcal{A}}}{| f(\boldsymbol{s}, \boldsymbol{a})| }\right\| \left\| | \phi_1(\boldsymbol{s}, \boldsymbol{a}) - f(\boldsymbol{s}, \boldsymbol{a}) | - | \phi_2(\boldsymbol{s}, \boldsymbol{a}) - f(\boldsymbol{s}, \boldsymbol{a}) | \right\|_\infty \label{eq:lemma_1}\\
    &\leq \mathbb{E} \left\| \frac{\mathbb{I}_{\mathcal{A}}}{| f(\boldsymbol{s}, \boldsymbol{a})| }\right\| \left\| | \phi_1(\boldsymbol{s}, \boldsymbol{a}) - f(\boldsymbol{s}, \boldsymbol{a}) - \phi_2(\boldsymbol{s}, \boldsymbol{a}) + f(\boldsymbol{s}, \boldsymbol{a})| \right\|_\infty \label{eq:lemma_2} \\
    &\leq   \frac{1}{\min_{(\boldsymbol{s}, \boldsymbol{a})} | f(\boldsymbol{s}, \boldsymbol{a})| }\mathbb{E} \left\| \phi_1(\boldsymbol{s}, \boldsymbol{a}) - \phi_2(\boldsymbol{s}, \boldsymbol{a}) \right\|_\infty \label{eq:lemma_3} \\
    &= \frac{1}{\epsilon} \mathbb{E} \left\| \phi_1(\boldsymbol{s}, \boldsymbol{a}) - \phi_2(\boldsymbol{s}, \boldsymbol{a}) \right\|_\infty \label{eq:lemma_4}\end{aligned}$$

Where step \[eq:lemma\_1\] to \[eq:lemma\_2\] comes from the triangle inequality and step \[eq:lemma\_2\] to \[eq:lemma\_3\] comes from the definition of the set $\mathcal{A} = \{ (\boldsymbol{s}, \boldsymbol{a}) : \left\| f(\boldsymbol{s}, \boldsymbol{a}) \right\| \geq \delta \}$. Thus, the loss is expectation-wise Lipschitz continuous.

A strictly feasible solution, $\phi$, exists since the ground truth model, which is representable in $\Phi$, is strictly feasible.

Lastly, the parameterization used is a neural network which is a universal function approximator.

Therefore the conditions for Theorem 2 are fulfilled.

Ball-Paddle Analytic Controller and Model
=========================================

This section presents the analytic controller and model used for the Ball-Paddle system. Since there are many variables used in this section, whole notation completely separate from the main paper is introduced here. Vector variables will be explicitly denoted with an arrow above it, such as $\vec{x}$.

Simple Model of System
----------------------

A simple model of a ball bouncing on a moving platform is presented without any special interactions such as air friction, paddle to ball friction, magnus effect, etc. The simple model assumes that

1.  the only forces acting on the ball are gravity or an impact from the paddle

2.  $m_{\text{paddle}} >> m_{\text{ball}}$. The paddle is so massive that any impacts with the ball will not affect its velocity at all

3.  we can control the velocity and angular velocity of the paddle arbitrarily

![image](images/system.png)

The state space of the ball looks like

$$\vec{x}_{\text{ball}} = \begin{pmatrix} \vec{p}_{\text{ball}} \\ \vec{v}_{\text{ball}} \end{pmatrix}$$

where $\vec{p}$ stands for position, $\vec{v}$ stands for velocity and all vectors are column vectors. In addition, $x, y, z$ components of the a vector $\vec{v}$ will be denoted as $\vec{v}[x], \vec{v}[y], \vec{v}[z]$.

Thus normally, the ball follows very simply dynamics $$\dot{\vec{x}}_{\text{ball}} = \begin{pmatrix}
\vec{v}_{\text{ball}} \\ \vec{g}
\end{pmatrix}$$ where $\vec{g} = \begin{pmatrix}
0 \\ 0 \\ -9.81
\end{pmatrix}$ is gravity. But when it hits the paddle, we model it as a partially elastic collision. At the moment of impact, the velocity of the ball changes according to

$$\begin{aligned}
\vec{v}_{\text{ref}} &= \vec{v}_{\text{ball}} - \vec{v}_{\text{paddle}}\\
\vec{v^\prime}_{\text{ball}} &= [\vec{v}_{\text{ref}} - 2 \vec{n}_{\text{paddle}} (\vec{n}_{\text{paddle}} \cdot \vec{v}_{\text{ref}})] \alpha + \vec{v}_{\text{paddle}} \label{eq:collision_dynamics}\end{aligned}$$

where $\vec{v^\prime}_{\text{ball}}$ is the velocity of the ball after collision, $\vec{n}_{\text{paddle}}$ is normal vector of the plane of the paddle, and $\alpha$ is the coefficient of restitution that represents what that fraction of velocity in the direction of the paddle normal is preserved (so 1, would mean a perfectly elastic collision). Note that in certain contexts, the coefficient of restitution is denoted as the ratio of energies instead of velocity. In that context it is simply the square of how we denote it.

Analytic Controller
-------------------

The analytic controller assumes that we will always be hitting the ball at some height $z_{\text{hit}}$. This controller has two inputs, a reference height, $h_d$ above $z_{\text{hit}}$ and reference location $\vec{p}_d$ in the $z=z_{\text{hit}}$ plane. The goal of this controller is to keep the ball bouncing at that certain height (above the hitting plane) at a specified location.

The basic concept to control height is to use an energy shaping controller. If we can control the kinetic energy (in the z direction) we can control the height, given perfect model knowledge. Basic steps for controller are as follows (these quantities can be recomputed every timestep)

\[alg:controller\]

1.  \[base\_1\] Compute the time until the ball crosses the $z=z_{\text{hit}}$ plane, $t_{\text{impact}}$ along with the location that it crosses the plane at, $\vec{p}_{\text{impact}}$, and the velocity of the ball right before impact, $\vec{v}_{ball, impact}$.

2.  \[base\_2\] Compute the desired velocity of the ball right after impact, $\vec{v}_d$ to both achieve a certain height and to land at a specific location in the $z$ plane.

3.  \[base\_3\] Compute the paddle orientation and velocity (assuming orientation is static throughout impact) required to achieve the desired ball velocity after impact

4.  \[base\_4\] Fit a path from current paddle position to impact location that ends with the correct paddle velocity and orientation and execute the first step of this path.

Step \[base\_1\] is very simply computed. Given current position and velocity of ball (assuming the ball is above $z_{\text{hit}}$), we can write $$\begin{aligned}
\vec{p}_{\text{ball}}[z] - z_{\text{hit}} &= \vec{v}_{\text{ball}}[z] t_{\text{impact}} + \frac{1}{2} g t_{\text{impact}}^2\\
t_{\text{impact}} &= (-\vec{v}_{\text{ball}}[z] - \sqrt{\vec{v}_{\text{ball}}[z]^2 - 2g(\vec{p}_{\text{ball}}[z] - z_{\text{hit}}) } ) / g\end{aligned}$$ and obtain $t_{\text{impact}}$ by solving the quadratic equation and using the only solution that makes sense (positive time). $g=-9.81$. $\vec{p}_{\text{impact}}$ and $\vec{v}_{ball, impact}$ are computed by simply advancing the dynamics of the ball ahead by $t_{\text{impact}}$ time.

Step \[base\_2\] can be computed by first looking at desired z velocity to achieve a certain height. $$\begin{aligned}
m_{\text{ball}} |g| h_{d} &= \frac{1}{2}m \vec{v}_d[z]^2\\
\vec{v}_d[z] &= \sqrt{2 |g| h_d}\end{aligned}$$ Then, we can compute the time that is required for the ball to cross the $z=z_{\text{hit}}$ plane again which will be denoted as $t_d$. $$\begin{aligned}
h_d &= \frac{1}{2} |g| (\frac{t_d}{2})^2 \\
t_d &= 2\sqrt{\frac{2 h_d}{|g|}}\end{aligned}$$ Using $t_d$ we can compute the desired $\vec{v}_d[x], \vec{v}_d[y]$ with $\vec{v}_d[x] = \frac{\vec{p}_d(x) - \vec{p}_{\text{impact}}(x)}{t_d}$.

Step \[base\_3\] is the trickest step. It involves solving Eq. \[eq:collision\_dynamics\] for a desired paddle velocity and paddle orientation. Plugging in the variables we have obtained so far, we get. $$\begin{aligned}
\vec{v}_{\text{ref}} &= \vec{v}_{ball, impact} - \vec{v}\\
\vec{v}_d &= [\vec{v}_{\text{ref}} - 2 \vec{n} (\vec{n} \cdot \vec{v}_{\text{ref}})] \alpha + \vec{v}\end{aligned}$$ Where the unknowns are $\vec{v}$, the velocity of paddle at impact, and $\vec{n}$, the normal vector of the paddle plane at impact. Rewriting the equation, we can obtain

$$\begin{aligned}
\vec{v}_d - \alpha \vec{v}_{\text{ball, impact}} &=  2 \vec{n} \vec{n}^T (\vec{v} - \alpha \vec{v}_{\text{ball, impact}}) + (1 - \alpha) \vec{v} \label{eq:paddle_eq}\end{aligned}$$

This represents 3 equations with 5 degrees of freedom (6 unknowns with one constraint $|| \vec{n} || = 1$), thus it is an underdetermined system, and we have some choice in how we assign choose the speed and orientation of the paddle at impact. However, we want to choose it such that the velocities of the paddle are not too great. One way to do this is to solve the optimization problem:

$$\begin{aligned}
\vec{v}^*, \vec{n}^* &= \min_{\vec{v}, \vec{n}} || \vec{v} || \\
\text{s.t. } || \vec{n} || &= 1 \nonumber \\
\vec{v}_d - \alpha \vec{v}_{\text{ball, impact}} &=  2 \vec{n} \vec{n}^T (\vec{v} - \alpha \vec{v}_{\text{ball, impact}}) + (1 - \alpha) \vec{v} \nonumber\end{aligned}$$

Note that this problem is not a convex problem simply due to the first constraint $ || \vec{n} || = 1$. There may be an analytical solution, but I was unable to find it. Thus, I have used a heuristic to assign $\vec{v}, \vec{n}$. The heuristic works as follows. First assume $\vec{n} = \begin{pmatrix}
0 & 0 & 1
\end{pmatrix}^T$ (the paddle is pointing straight up) and $\vec{v} = \begin{pmatrix}
0 & 0 & 0
\end{pmatrix}^T$ (with no velocity). Then compute where the ball will end up the next time it crosses the $z_{\text{hit}}$ plane. Call this location $\vec{p}_{\text{nominal}}$. We can then look at the difference between the reference location and norminal location $\vec{p}_d - \vec{p}_{\text{nominal}}$. Let us then set $(x,y)$ components of $\vec{n}$ to be some scaled version of this difference. The $z$ component is still 1, thus we have to then normalize $\vec{n}$. Thus, this encodes the behavior that the further the natural motion of the ball (without any assistance from the paddle) has to be modified, the harder the paddle leans towards that direction. Then, with a constant $\vec{n}$, Eq. \[eq:paddle\_eq\] can be solved for $\vec{v}$.

Step \[base\_4\] considers mainly the paddle velocity. Paddle orientation can be taken care of by feeding $\vec{n}$ into a PID controller. We have several known quantities, $\vec{v}$, velocity of paddle at impact (at time $t_{\text{impact}}$), $\vec{p}_{\text{paddle}}$, current paddle position at time $0$, and $\vec{p}_{\text{paddle, impact}}$, paddle position at time $t_{\text{impact}}$. We can model the position of the paddle as a polynomial in time in each dimension separately, for example, $\vec{p}[x](t) = a_x t^2 + b_x t + c_x$. We can then solve for $a, b, c$ in each dimension by solving the system of equations: $$\begin{bmatrix}
\vec{p}_{\text{paddle}}[x] \\
\vec{p}_{\text{paddle, impact}}[x] \\
\vec{v}[x]
\end{bmatrix} = 
\begin{bmatrix}
0 & 0 & 1 \\
t_{\text{impact}}^2 & t_{\text{impact}} & 1 \\
2 t_{\text{impact}} & 1 & 0
\end{bmatrix}
\begin{bmatrix}
a_x \\ b_x \\ c_x
\end{bmatrix}$$ Modeling as a polynomial leads to smooth velocities and accelerations. There are other choices to model the paddle position. For example, a sinusoid is an attractive alternative to model the z dimension as there is some physical motivation for choosing it (you will have repetitive up down motions in hitting the ball). You might also use higher order polynomials, and build in more constraints to the path (such as having a certain velocity at time $0$).

$$\begin{bmatrix}
\vec{p}_{\text{paddle}}[x] \\
\vec{p}_{\text{paddle, impact}}[x] \\
\vec{v}_{\text{paddle}}[x]\\
\vec{v}[x]
\end{bmatrix}
=
\begin{bmatrix}
0 & 0 & 0 & 1 \\
t_{\text{impact}}^3 & t_{\text{impact}}^2 & t_{\text{impact}} & 1\\
0 & 0 & 1 & 0\\
3t_{\text{impact}}^2 & 2t_{\text{impact}} & 1 & 0
\end{bmatrix}
\begin{bmatrix}
a \\ b \\ c \\d
\end{bmatrix}$$

Controller with Learned Model
-----------------------------

The model, $\phi$, that the Neural Network learns is $$\vec{p}_{ball, impact} = \phi(\vec{v}_{ball}, \vec{v}_{paddle}, \vec{n}_{paddle})$$ where the next impact location of the ball is predicted, given the velocity of the ball and paddle and the orientation of the paddle at the point of contact.

This can be used with the analytic controller in Algorithm \[alg:controller\], by replacing steps 2 and 3 with an optimization problem.

\[alg:controller\_learned\]

1.  \[base\_1\] Compute the time until the ball crosses the $z=z_{\text{hit}}$ plane, $t_{\text{impact}}$ along with the location that it crosses the plane at, $\vec{p}_{\text{impact}}$, and the velocity of the ball right before impact, $\vec{v}_{ball, impact}$.

2.  Solve optimization problem \[eq:controller\_learned\_opt\] to obtain paddle orientation and velocity.

3.  \[base\_4\] Fit a path from current paddle position to impact location that ends with the correct paddle velocity and orientation and execute the first step of this path.

$$\begin{aligned}
    \label{eq:controller_learned_opt1}
    \vec{n}^*, \vec{v}^* &= \operatorname*{arg\,min}_{\vec{n}, \vec{v}} \left\| \phi(\vec{v}_{ball, impact}, \vec{v}, \vec{n}) - \vec{p}_{desired} \right\| \\
    & \text{s.t} \quad \vec{v}_{min} \leq \left\| \vec{v} - \vec{v}_{ball, impact} \right\| \leq \vec{v}_{max} \nonumber \\
    & \quad \quad \left\| \phi(\vec{v}_{ball, impact}, \vec{v}, \vec{n}) - \vec{p}_{avoid} \right\| \geq R \nonumber\end{aligned}$$

Optimization problem \[eq:controller\_learned\_opt1\] has several free parameters that can be chosen by the user to invoke different behaviour of the controller.

1.  $\vec{p}_{desired}$ is the desired location of the ball to boucne at

2.  $\vec{v}_{min}, \vec{v}_{max}$ and minimum and maximum velocity limits for the relative impact velocity between the ball and the paddle

3.  $\vec{p}_{avoid}$ and $R$ is the center and radius of a circle that the ball should avoid bouncing in
