---
abstract: 'Smart buildings have great potential for shaping an energy-efficient, sustainable, and more economic future for our planet as buildings account for approximately 40% of the global energy consumption. A key challenge for large-scale plug and play deployment of the smart building technology is the ability to learn a good control policy in a short period of time, i.e. having a low sample complexity for the learning control agent. Motivated by this problem and to remedy the issue of high sample complexity in the general context of cyber-physical systems, we propose an event-triggered paradigm for learning and control with variable-time intervals, as opposed to the traditional constant-time sampling. The *events* occur when the system state crosses the a priori-parameterized *switching manifolds*; this crossing triggers the learning as well as the control processes. Policy gradient and temporal difference methods are employed to learn the optimal switching manifolds which define the optimal control policy. We propose two event-triggered learning algorithms for stochastic and deterministic control policies. We show the efficacy of our proposed approach via designing a smart learning thermostat for autonomous micro-climate control in buildings. The event-triggered algorithms are implemented on a single-zone building to decrease buildings’ energy consumption as well as to increase occupants’ comfort. Simulation results confirm the efficacy and improved sample efficiency of the proposed event-triggered approach for online learning and control.'
address:
- 'Laboratory for Information and Decision Systems, MIT, USA'
- 'D. E. Shaw Group, New York, NY 10036 USA'
- 'Center for Energy Science and Technology, Skolkovo Institute of Science and Technology, 3 Nobel Street, Skolkovo, Moscow Region 121205, Russia'
author:
- Ashkan Haji Hosseinloo
- Alexander Ryzhov
- Aldo Bischi
- Henni Ouerdane
- Konstantin Turitsyn
- 'Munther A. Dahleh'
bibliography:
- 'sample.bib'
title: 'Data-driven control of micro-climate in buildings; an event-triggered reinforcement learning approach'
---

Event-triggered learning ,smart buildings ,reinforcement learning ,data-driven control ,energy efficiency ,cyber-physical systems

Introduction {#intro}
============

Buildings account for approximately 40% of global energy consumption about half of which is used by heating, ventilation, and air conditioning (HVAC) systems [@nejat2015global; @wei2017deep], the primary means to control micro-climate in buildings. Furthermore, buildings are responsible for one-third of global energy-related greenhouse gas emissions [@nejat2015global]. Hence, even an incremental improvement in the energy efficiency of buildings and HVAC systems goes a long way towards building a sustainable, more economic, and energy-efficient future. In addition to their economic and environmental impacts, HVAC systems can also affect productivity and decision-making performance of occupants in buildings through controlling indoor thermal and air quality [@satish2012co2; @wargocki2017ten]. For all these reasons micro-climate control in buildings is an important issue for its large-scale economic, environmental, and health-related and societal effects.\
The main goal of the micro-climate control in buildings is to minimize the building’s (mainly HVAC’s) energy consumption while improving or respecting some notion of occupants’ comfort. Despite its immense importance, micro-climate control in buildings is often very energy-inefficient. HVAC systems are traditionally controlled by rule-based strategies and heuristics where an expert uses best practices to create a set of rules that control different HVAC components such as rule-based ON/OFF and conventional PID controllers [@levermore2013building; @dounis2009advanced]. These control methods are often far from optimal as they do not take into account the system dynamics model of the building i.e. the building thermodynamics and stochastic disturbances e.g. weather conditions or occupancy status. To overcome some of these shortcomings, more advanced model-based approaches have been proposed. In this category Model Predictive Control (MPC) is perhaps the most promising and extensively-studied method in the context of buildings climate control [@oldewurtel2012use; @ryzhov2019model; @afram2014theory; @smarra2018data].\
Despite its potential benefits, performance and reliability of MPC and other model-based control methods depend highly on the accuracy of the building thermodynamics model and prediction of the stochastic disturbances. However, developing an accurate model for a building is extremely time-consuming and resource-intensive, and hence, not practical in most cases. Moreover, a once accurate developed model of a building could become fairly inaccurate over time due to, for instance, renovation or wear and tear of the building. Furthermore, at large scales, MPC like many other advanced model-based techniques may require formidable computational power if a real-time (or near real-time) solution is required [@marantos2019rapid]. Last but not least, traditional and model-based techniques are inherently building-specific and not easily transferable to other buildings.\
To remedy the above-mentioned issues of model-based climate control in buildings and towards building autonomous *smart* homes, data-driven approaches for HVAC control have attracted the interest of many researchers in the recent years. The concept of *smart* homes where household devices (e.g. appliances, thermostats, and lights) can operate efficiently in an autonomous, coordinated, and adaptive fashion, has been around for a couple of decades [@mozer1998neural]. However, with recent advances in Internet of Things (IoT) technology (cheap sensors, efficient data storage, etc.) on the one hand [@minoli2017iot], and immense progress in data science and machine learning tools on the other hand, the idea of smart homes with data-driven HVAC control systems looks ever more realistic.\
Among different data-driven control approaches, reinforcement learning (RL) has found more attention in the recent years due to enormous recent algorithmic advances in this field as well as its ability to learn efficient control policies solely from experiential data via trial and error. This study focuses on an RL approach and hence, we next discuss some of the related studies using reinforcement learning for energy-efficient controls in buildings followed by our contribution.\
The remaining of this article is organized as follows. Section \[related\] reviews the related work and highlights our contributions in this study. The Problem is stated and mathematically formulated in section \[MPDframework\] after which the idea of switching manifolds for event-triggered control is introduced in section \[manifolds\]. Combining the average-reward set-up and event-triggered control paradigm in sections \[MPDframework\] and \[manifolds\], we present our event-triggered reinforcement learning algorithms in section \[RL\]. Finally, the implementation and simulation results are discussed in section \[results\] before the article is concluded in section \[conclusion\].

Related work and contribution {#related}
=============================

Tabular RL
----------

The Neural Network House project [@mozer1998neural] is perhaps the first application of reinforcement learning in building energy management system. In this seminal work, the author explains how tabular Q-learning, one of the early versions of the popular Q-learning approach in RL, was employed to control lighting in a residential house so as to minimize energy consumption subject to occupants’ comfort constraint [@mozer1997parsing]. Tabular Q-learning was later used in a few other studies for controlling passive and active thermal storage inventory in commercial buildings [@liu2006experimental1; @liu2006experimental2], heating system[@barrett2015autonomous], air-conditioning and natural ventilation through windows [@chen2018optimal], photovoltaic arrays and geothermal heat pumps [@yang2015reinforcement], and lighting and blinds [@cheng2016satisfaction].\
Given fully observable state and infinite exploration, tabular Q-learning is guaranteed to converge on an optimal policy. However, the tabular version of Q-learning is limited to systems with discrete states and actions, and becomes very data-intensive, hence very slow at learning, when the system has a large number of state-action combinations. For instance, the simulated RL training in [@liu2006experimental2] for a fairly simple building required up to 6000 days (roughly 17 years) of data collection. To remedy some of these issues, other versions of Q-learning such as Neural Fitted Q-iteration (NFQ) and Deep RL (DRL) were employed where function approximation techniques are used to learn an approximate function of the state-action (Q) function.

RL with action-value function approximation
-------------------------------------------

Dalamagkidis et al. [@dalamagkidis2007reinforcement] used a linear function approximation technique to approximate the Q-function in their Q-learning RL to control a heat pump and an air ventilation subsystem using sensory data on indoor and outdoor air temperature, relative humidity, and $\mathrm{CO_2}$ concentration. Fitted Q Iteration (FQI) developed by Ernst et al. [@ernst2005tree] is a batch RL method that iteratively estimates the Q-function given a fixed batch of past interactions. An online version that uses a neural network, neural fitted Q-iteration, has been proposed by [@riedmiller2005neural]. In a series of studies [@ruelens2015learning; @ruelens2016residential; @ruelens2016reinforcement], Ruelens et al. studied the application of FQI batch RL to schedule thermostatically controlled HVAC systems such as heat pumps and electric water heaters in different demand-response set-ups. Marantos et al. [@marantos2018towards] applied NFQ batch RL to control the thermostat set-point of a single-zone building where input state was four-dimensional (outdoor and indoor temperatures, solar radiance, and indoor humidity) and action was one-dimensional with three discrete values.\
Tremendous algorithmic and computational advancements in deep neural networks in the recent years have given rise to the field of deep reinforcement learning (DRL) where deep neural networks are combined with different RL approaches. This has resulted in numerous DRL algorithms (DQN, DDQN, RBW, A3C, DDPG, etc.) in the past few years, some of which have been employed for data-driven micro-climate control in buildings. Wei et al. [@wei2017deep] claim to be the first to apply DRL to HVAC control problem. They used Deep Q-Network (DQN) algorithm [@mnih2015human] to approximate the Q-function with discrete number of actions. To remedy some of the issues of the DQN algorithm such as overestimation of action values, improvements to this algorithm have been made resulting in a bunch of other algorithms like Double DQN (DDQN) [@van2016deep] and Rainbow (RWB) [@hessel2018rainbow]. Avendano et al. [@avendano2018data] applied DDQN and RWB algorithms to optimize energy efficiency and comfort in a 2-zone apartment; they considered temperature and $\mathrm{CO_2}$ concentration for comfort and used heating and ventilation costs for energy efficiency.

RL with policy function approximation
-------------------------------------

All the above-mentioned RL-based studies rely on learning the optimal state-value or action-value (Q) functions based on which the optimal policy is derived. Parallel to this value-based approach there is a policy-based approach where the RL agent tries to directly learn the optimal policy (control law). Policy gradient algorithms are perhaps the most popular class of RL algorithms in this approach. The basic idea behind these algorithms is to adjust the parameters of the policy in the direction of a performance gradient [@sutton2000policy; @silver2014deterministic]. A distinctive advantage of policy gradient algorithms is their ability to handle continuous actions as well as stochastic policies. Wang et al. [@wang2017long] employed Monte Carlo actor-critic policy gradient RL with LSTM actor and critic networks to control HVAC system of a single-zone office. Deep Deterministic Policy Gradient (DDPG) algorithm [@lillicrap2015continuous] is another powerful algorithm in this class that handles deterministic policies. DDPG was used in [@gao2019energy] and [@li2019transforming] to control energy consumption in a single-zone laboratory and 2-zone data center buildings, respectively.

Sample efficiency
-----------------

Despite the sea-change advances in RL, sample efficiency is still the bottleneck for many real-world applications with slow dynamics. Building micro-climate control is one such application since thermodynamics in buildings is relatively slow; it can take a few minutes to an hour to collect an informative data point. The time-intensive process of data collection makes the online training of the RL algorithms so long that it practically becomes impossible to have a plug & play RL-based controller for HVAC systems. For instance, training the DQN RL algorithm in [@wei2017deep] for a single-zone building required about 100 months of sensory data. The required data collection period for training the DDQN and RWB algorithms in [@avendano2018data] were reported as 120 and 90 months, respectively. A few different techniques have been proposed to alleviate the RL’s training sample complexity when it comes to real-world applications, in particular buildings, which are discussed next.\
Multiple time scales in some real-world applications is one reason for the sample inefficiency of many RL algorithms. For instance, for precise control of a set-point temperature it is more efficient to design a controller that works on a coarse time scale in the beginning when the temperature is far from the set-point temperature, and on a finer time scale otherwise. To address this issue, double and multiple scales reinforcement learning are proposed in [@riedmiller1998high; @li2015multi]. Reducing the system’s dimension, if possible, is another way to shorten the online training period. Different dimensionality reduction techniques such as auto-encoder [@ruelens2015learning] and convolutional neural networks (CNN) [@claessens2016convolutional] were used in RL-based building energy management control where the system states are high dimensional.\
Another approach to reduce the training period is based on developing a data-driven model first, and then use it for offline RL training or direct planning. This approach is similar to the Dyna architecture [@sutton1991dyna; @sutton2018reinforcement]. Costanzo et al. [@costanzo2016experimental] used neural networks to learn temperature dynamics of a building heating system to feed training of their FQI RL algorithm while Nuag et al. [@naug2019online] used support vector regression to develop consumption energy model of a commercial building for training of their DDPG algorithm. In [@nagy2018deep] and [@kazmi2018gigawatt] data-driven models of thermal systems are developed in the form of neural networks and partially observable MDP transition matrix, respectively, which are then used for finite horizon planning. As another example, Kazmi et al. [@kazmi2019multi] used muti-agent RL to learn an MDP model of identical thermostatically controlled loads which was then used for deriving the optimal policy by Monte Carlo techniques.

Contributions
-------------

Despite all the recent efforts, none of the proposed methods can be used for a plug & play deployment of smart HVAC systems without pre-training due to their large sample complexity. In addition, all the reinforcement learning studies in building energy management systems have formulated the problem based on *episodic* tasks, as opposed to *continuing* tasks. Micro-climate control in buildings is indeed a continuing task problem and should be formulated as such. Furthermore, the algorithms in these studies are all based on periodic sampling with fixed time intervals. This is not very sample-efficient in many cases and is certainly not desirable in resource-constrained wireless embedded control systems [@heemels2012introduction]. To remedy these issues we make the following major contributions:\

-   We develop a general framework called *switching manifolds* for data-efficient control of HVAC systems;

-   Based on the idea of switching manifolds, we propose an event-triggered paradigm for learning and control with an application to the HVAC systems;

-   We develop and formulate the event-triggered control problem with variable-duration sampling as an undiscounted continuing task reinforcement learning problem with average reward set-up;

-   We demonstrate the effectiveness of our proposed approach on a small-scale building via simulation in EnergyPlus software.

Problem statement and MDP framework {#MPDframework}
===================================

The aim of this study is to provide a plug & play control algorithm that can efficiently learn to optimize HVAC energy consumption and occupants’ comfort in buildings. To this end we first formulate the sequential decision-making control problem as a Markov decision process (MDP) in this section.\
The MDP is defined by a state space $\mathcal{S}$, an action space $\mathcal{A}$, a stationary transition dynamics distribution with conditional density $p(s_{k+1}|s_k,a_k)$ where $s_k$ and $a_k$ are state and action at time indexed by $k$ when the $k^{\mathrm{th}}$ *event* occurs, and a reward function $r: \mathcal{S} \times \mathcal{A} \times \mathcal{S} \rightarrow \mathbb{R}$. States and actions are in general continuous (e.g. temperature state or temperature threshold action). Events are occasions when control actions are taken and the learning takes place; hence, they define the transition times. These events are characterized when certain conditions are met and are explained in detail in section \[manifolds\]. Actions are taken at these events based on a stochastic ($\pi_{\theta}:\mathcal{S}\rightarrow\mathcal{P(A)}$) or deterministic ($\mu_{\theta}:\mathcal{S}\rightarrow\mathcal{A}$) policy, where $\mathcal{P(A)}$ is the set of probability measures on $\mathcal{A}$ and $\theta \in \mathbb{R}^n$ is a vector of $n$ parameters.\
Taking action $a_{k-1}$ at state $s_{k-1}$ moves the system to a new state $s_{k}$ and results in a reward of $r_{k}$. Let us assume this transition takes $\Delta t_{k}$ unit time ($\Delta t_{k}=t_{k}-t_{k-1}$). Following the policy, dynamics of the MDP evolves and results in a trajectory of states, actions, and rewards; $s_0, a_0, r_1, ..., s_{k-1}, a_{k-1}, r_k, ...$. We define the performance measure that we want to maximize as the *average rate of reward per unit time* or simply *average reward rate*: $$J(\theta) \doteq r(\pi) \doteq \lim_{h \to \infty} \mathbb{E} \left[\left. \frac{\sum_{k=1}^{h}r_k}{\sum_{k=1}^{h}\Delta t_k}  \right|s_0,a_{0:k-1}\sim \pi \right].
\label{Eq:metric}$$

This is different from and not proportional to the average rate of reward per time step if transition time periods $\Delta t_k$ are not equal, which will be the case in this study. We also define the *differential* return $G_k$ as: $$G_k \doteq r_{k+1}-r(\pi)\Delta t_{k+1}+ r_{k+2}-r(\pi)\Delta t_{k+2}+ ... \:.
\label{Eq:return}$$ In this definition of return the average reward is subtracted from the actual sample reward in each step to measure the accumulated reward relative to the average reward. Similarly, we can define the state-value $V_{\pi}(s)$ and action-value functions $Q_{\pi}(s,a)$ as: $$\begin{aligned}
V_{\pi}(s)=\sum_a \pi(a|s)\sum_{s'} p(s'|s,a)\left(r-r(\pi)\Delta t+V_{\pi}(s')\right) \nonumber \\
Q_{\pi}(s,a)=\sum_{s'} p(s'|s,a)\left(r-r(\pi)\Delta t+\sum_{a'}\pi(a'|s')Q_{\pi}(s',a')\right),
\label{Eq:VnQ}\end{aligned}$$ where, $\pi(a|s)$ is the conditional probability density at $a$ associated with the policy. Although the average reward set-up is formulated here for stochastic policies, it is applicable to deterministic policies as well with minor modification to the equations above. In the next section, we introduce the idea of switching manifolds and learning and controlling when needed.

Switching manifolds and event-triggered control {#manifolds}
===============================================

Many HVAC control devices work based on a discrete set of control actions e.g. ON/OFF switches or discrete-scale knobs. The optimal control in the system’s state space is often not very discontinuous or non-smooth in many practical applications, or at least there often exists one such control policy that is not far from the optimal. In this case optimal (or near-optimal) actions are separated by some boundaries in the state space. We call these boundaries *switching manifolds* since it is only across these boundaries that the controller needs to switch actions. Figure \[Fig:1\] illustrates the concept of switching manifolds for two simple systems with two-dimensional state vectors and 2 or 4 actions.\

Switching manifolds fully define a corresponding policy, hence, it is more sample-efficient to learn these manifolds or a parameterized version of them rather than a full tabular policy. Let us consider one such manifold parameterized by a parameter vector $\theta_g$ as $g^{\theta_g}(s)=0$. A different action is taken when the system dynamics cross this manifold, or in other words when $g^{\theta_g}(s)=0$ holds true. To make it more intuitive we rewrite this manifold equation in terms of one particular state (e.g. temperature in the HVAC example) as $s_2=f^{\theta_f}\left(s-\left\{s_2\right\}\right)$. Given the other states of the system, we can now think of $s_2$ as a threshold $s_2^{\mathrm{th}}$, i.e. if state $s_2$ of the system reaches this threshold value of $s_2^{\mathrm{th}}$ we need to switch to the new action based on the switching manifolds mapping (Fig.\[Fig:1\](a) and Fig.\[Fig:1\](b) schematically illustrate two such mappings). Also, instead of the parameters or the actual physical actions we can think of these thresholds as the actions that the learning agent needs to take.\
So far we introduced the switching manifolds or the threshold policies as a family of policies among which we would like to search for an optimal policy via e.g. reinforcement learning. The manifold/threshold learning does not need to happen at constant time intervals. In fact, here we propose controlling and learning with variable-time intervals when actions and updates take place when specific events occur. By definition, these events occur when system dynamics reach the switching manifolds or equivalently when thresholds are reached.\
Here we further illustrate these concepts with a simple example. Let us consider a 1-zone building equipped with a heating system described by its state vector $s=[T, T_o, h_s]$, where $T$ and $T_o$ are indoor and outdoor temperatures and $h_s$ is the heater status ($h_s=1$ means heater is on and $h_s=0$ means it is off). Possible physical actions we can take are; turning the heater ON, turning the heater OFF, or do nothing. Corresponding to this set of actions, we employ linear manifolds as an example and describe the parameterized temperature thresholds as: $T^{\mathrm{th}}_{\mathrm{OFF}}=\theta_1+\theta_2 T_o$ and $T^{\mathrm{th}}_{\mathrm{ON}}=\theta_3+\theta_4 T_o$. This is illustrated schematically in Fig. \[Fig:2\]. For a given parameter vector $\theta = [\theta_i|_{i=1,...,4}]^\top$ and outdoor temperature $T_o$, when the indoor temperature $T$ reaches the switch-off threshold ($T^{\mathrm{th}}_{\mathrm{OFF}}$) the heater is turned off and when it reaches the switch-on threshold ($T^{\mathrm{th}}_{\mathrm{ON}}$) the heater is turned on; otherwise, no action is taken. The deterministic action policy for the underlying MDP of this system could be written as $a=\mu_{\theta}(s)=[T^{\mathrm{th}}_{\mathrm{OFF}}, T^{\mathrm{th}}_{\mathrm{ON}}]$. Since at every event we need to decide for only one threshold (which will affect the next event), we can reduce the action dimension to only one by writing it as $a=\mu_{\theta}(s)=[1-h_s, h_s][T^{\mathrm{th}}_{\mathrm{ON}}, T^{\mathrm{th}}_{\mathrm{OFF}}]^\top$. This idea is applied to stochastic policy in a similar way to decide for only one threshold temperature when an event occurs. In the next section, we propose actor-critic event-triggered RL algorithms with both stochastic and deterministic policies based on the average-reward MDP set-up presented in section \[MPDframework\] and the concept of switching manifolds introduced in this section.

Reinforcement learning algorithm and implementation {#RL}
===================================================

Most, if not all, of the popular RL algorithms (both stochastic and deterministic) are based on episodic-task MDPs. Furthermore, transition time periods do not play any role in these algorithms; this is not an issue for applications where either the transition time intervals are irrelevant to the optimization problem e.g. in a game play, or these intervals are assumed to have fixed duration. None of these hold for the problem of micro-climate control in buildings where we want to optimize energy and occupants’ comfort in a continuing fashion with event-triggered sampling and control which result in variable-time intervals.\
Here we consider both stochastic and deterministic policy gradient reinforcement learning for event-triggered control. Our algorithms are based on stochastic and deterministic policy gradient theorems [@sutton2000policy; @silver2014deterministic] with modifications to cater for average-reward set-up and variable-time transition intervals. These theorems are as follows: $$\begin{aligned}
\nabla_{\theta}J(\pi_{\theta})=\mathbb{E}_{s\sim \rho^{\pi},a\sim \pi_{\theta}}\left[ \nabla_{\theta}\log \pi_{\theta}(a|s)Q^{\pi}(s,a) \right] \nonumber \\
\nabla_{\theta}J(\mu_{\theta})=\mathbb{E}_{s\sim \rho^{\mu}}\left[\nabla_{\theta}\mu_{\theta}(s)\nabla_a Q^{\mu}(s,a)|_{a=\mu_{\theta}(s)} \right],
\label{Eq:theorems}\end{aligned}$$ where, $\rho^{\pi}$ and $\rho^{\mu}$ are stationary state distributions under stochastic and deterministic policies, $\pi_{\theta}$ and $\mu_{\theta}$. The actor components of our proposed algorithms employ Eq.(\[Eq:theorems\]) to adjust and improve the parameterized policies. To this end we use approximated action-value $Q^w(s,a)$ and state-value $V^v(s)$ functions by parameterizing their true functions with parameter vectors $w$ and $v$, respectively. We employ temporal difference (TD) Q-learning for the critic to estimate the state-value or action-value functions. In this set-up we also replace the true average reward rate $r(\pi)$ (or $r(\mu)$) by an approximation $\bar{r}$, which we learn via the same temporal difference error. We use the following TD errors ($\delta_k$) for the stochastic and deterministic policies, respectively: $$\begin{aligned}
\delta_k & \doteq& r_{k+1}-\bar{r}_k \Delta t_{k+1} + V^{v_k}(s_{k+1})- V^{v_k}(s_{k}) \\
\delta_k & \doteq& r_{k+1}-\bar{r}_k \Delta t_{k+1} + Q^{w_k}(s_{k+1},a_{k+1})- Q^{w_k}(s_{k},a_k),
\label{Eq:TD}\end{aligned}$$ where, $\bar{r}_k$, $v_k$, and $w_k$ are the average reward and parameters at time $t_k$. With this definition of TD errors we update the average reward as follows: $$\bar{r}_{k+1} = \bar{r}_{k} + \alpha_{\bar{r}} \frac{\delta_k}{\Delta t_{k+1}},
\label{Eq:update}$$ where, $\alpha_{\bar{r}}$ is the learning rate for the average reward update. Having explained the average-reward set-up and the event-triggered control and learning, we can now present the pseudocodes for actor-critic algorithms for continuing tasks with both deterministic and stochastic policies.\
Algorithm \[Algo:stochastic\] shows the pseudocode for stochastic policies with eligibility traces while algorithm \[Algo:deterministic\] shows its deterministic counterpart. Algorithm \[Algo:deterministic\] is an event-triggered compatible off-policy deterministic actor-critic algorithm with a simple Q-learning critic (ET-COPDAC-Q). For this algorithm we use compatible function approximator for the $Q^w(s,a)$ in the form of $(a-\mu_{\theta}(s))^{\mathrm{T}}{\nabla_{\theta}\mu_{\theta}(s)}^{\mathrm{T}}w + V^v(s)$. Here $V^v(s)$ is any differentiable baseline function independent of $a$, such as a state-value function. We parameterize the baseline function linearly in its feature vector as $V^v(s)=v^{\mathrm{T}}\phi_v(s)$, where, $\phi_v(s)$ is a feature vector. In the next section, we implement these algorithms on a simple building model and assess their efficacy.\

a differentiable stochastic policy parameterization $\pi_{\theta}(a|s)$ a differentiable state-value function parameterization $V^{v}(s)$: $\lambda_v \in [0,1]$, $\lambda_{\theta} \in [0,1]$, $\alpha_v>0$, $\alpha_{\theta}>0$, $\alpha_{\bar{r}}>0$ Initialize $\bar{r} \in \mathbb{R}$ (e.g. to 0) Initialize state-value and policy parameters $v \in \mathbb{R}^{d_v}$ and $\theta \in \mathbb{R}^{d_{\theta}}$ (e.g. to [0]{}) Initialize the state vector $s \in \mathcal{S}$

$z_v \leftarrow 0$ ($d_v$-component eligibility trace vector) $z_{\theta} \leftarrow 0$ ($d_{\theta}$-component eligibility trace vector) ( forever )[ $a \sim \pi_{\theta}(.|s)$ Execute action $a$ and ; then observe $s'$, $r$, $\Delta_t$ $\delta \leftarrow r-\bar{r} \Delta t + V^{v}(s')- V^{v}(s)$ $\bar{r} \leftarrow \bar{r}+\alpha_{\bar{r}}\frac{\delta}{\Delta t}$ $z_v \leftarrow \lambda_v z_v + \nabla_v V^v(s) $ $z_{\theta} \leftarrow \lambda_{\theta} z_{\theta} + \nabla_{\theta}\log\pi_{\theta}(a|s)$ $v \leftarrow v+\alpha_v \delta z_v$ $\theta \leftarrow \theta+\alpha_{\theta} \delta z_{\theta}$ $s \leftarrow s'$ ]{}

a differentiable deterministic policy parameterization $\mu_{\theta}(s)$ a differentiable state-value function parameterization $V^{v}(s)$ a differentiable action-value function parameterization $Q^{w}(a,s)$: $\alpha_v>0$, $\alpha_w>0$, $\alpha_{\theta}>0$, $\alpha_{\bar{r}}>0$ Initialize $\bar{r} \in \mathbb{R}$ (e.g. to 0) Initialize state-value, action-value, and policy parameters $v \in \mathbb{R}^{d_v}$, $w \in \mathbb{R}^{d_w}$ and $\theta \in \mathbb{R}^{d_{\theta}}$ (e.g. to [0]{}) Initialize the state vector $s \in \mathcal{S}$ Initialize a random process $F_k$ for action exploration

( forever )[ $a = \mu_{\theta}(s)+F$ Execute action $a$ and ; then observe $s'$, $r$, $\Delta_t$ $\delta \leftarrow r-\bar{r} \Delta t + Q^{w}(s',\mu_{\theta}(s'))- Q^{w}(s,a)$ $\bar{r} \leftarrow \bar{r}+\alpha_{\bar{r}}\frac{\delta}{\Delta t}$ $v \leftarrow v+ \alpha_v \delta \nabla_v V^v(s)=v+ \alpha_v \delta \phi_v(s)$ $w \leftarrow w+\alpha_w \delta \nabla_w Q^w(s,a)=w+\alpha_w \delta \left(a-\mu_{\theta}(s)\right)^\mathrm{T}{\nabla_{\theta}\mu_{\theta}(s)}^{\mathrm{T}}$ $\theta \leftarrow \theta+\alpha_{\theta} \nabla_{\theta}\mu_{\theta}(s)\nabla_a Q^w(s,a)|_{a=\mu(s)}=\theta+\alpha_{\theta}\nabla_{\theta}\mu_{\theta}(s)\left({\nabla_{\theta}\mu_{\theta}(s)}^{\mathrm{T}} w\right)$ $s \leftarrow s'$ ]{}

Simulations and results {#results}
=======================

In this section we implement our proposed algorithms to control the heating system of a one-zone building in order to minimize energy consumption without jeopardizing the occupants’ comfort. To this end we first describe the building models that we use for simulation, followed up by designing the rewards to use for our learning control algorithms. Then we explain the policy parameterization used in the simulations before we present the simulation results.

Building models {#BuildingModels}
---------------

We use two one-zone building models: a simplified linear model characterized by a first-order ordinary differential equation, and a more realistic building modeled in the EnergyPlus software. The linear model for the one-zone building with the heating system is as follows: $$C \frac{dT}{dt}+K(T-T_o)=h_s(t) \dot{Q}_h,
\label{Eq:linear}$$ where, $C=2000 \, kJK^{-1}$ is the building’s heat capacity, $K=325 WK^{-1}$ is the building’s thermal conductance, and $\dot{Q}_h=13 \, kW$ is the heater’s power. As defined earlier, $h_s(t) \in \{0,1\}$ is the heater status, and $T_o=-10 \, \degree C$ is the outdoor temperature.\
In addition to the simplified linear building model, a more realistic building modeled in EnergyPlus is also used for implementation of our proposed learning control algorithms. The building modeled in EnergyPlus is a single-floor rectangular building with dimensions of $15.240 \times 15.240 \times 4.572 \, m^3$ ($50 \times 50 \times 15 \, ft^3$). The walls and the roof are modeled massless with thermal resistance of $1.291\, m^2\,K/W$ and $2.456\, m^2\,K/W$, respectively. All the walls as well as the roof are exposed to the Sun and wind, and have thermal and solar absorptance of 0.90 and 0.75, respectively. The floor is made up of a 4-inch h.w. concrete block with conductivity of $1.730\, W/m\,K$, density of $2242.585\, kg/m^3$, specific heat capacity of $836.800\, J/kg\,K$, and thermal and solar absorptance of 0.90 and 0.65, respectively. The building is oriented 30 degrees east of north. EnergyPlus Chicago Weather data (Chicago-OHare Intl AP 725300) is used for the simulation. An electric heater with nominal heating rate of $10\, kW$ is used for space heating.

Rewards
-------

Comfort and energy consumption are controlled by rewards or penalties. Rewards in RL play the role of cost function in controls theory and therefore proper design of the rewards is of paramount importance in the problem formulation. Here we formulate the reward with three components; one discrete and two continuous components: $$r_{k+1}=r_{sw}+\int_{t_k^+}^{t_{k+1}^+}r_e h_s(t)+r_c \left(T-T_d\right)^2 dt,
\label{Eq:reward}$$ where, $-r_{sw}=0.8 \, unit$ is the discrete penalty for switching on/off the heater to avoid frequent switching. The frequent on/off switching can decrease the system life-cycle or could result in unpleasant noisy operation of the heater. Here, *unit* is an arbitrary scale for quantifying different rewards. Having the heater on is penalized continuously in time with the rate of $-r_e=1.2/3600 \, {unit}\,s^{-1}$. This penalty is responsible for limiting the power consumption; hence, for a more intuitive meaning, $r_e$ could be chosen such that the reward unit (*unit*) equals the monetary cost unit of the power consumption e.g. dollar currency. Here we define occupants’ discomfort rate proportional to the square of deviation from their desired temperature $T_d$, and coefficient of proportionality is $r_c=-1.2/3600 \, unit\,K^{-2}\,s^{-1}$.

stochastic and deterministic policy parameterization
----------------------------------------------------

As discussed in section \[manifolds\], although we can define actions as both of the thresholds at each event, we only need one of the thresholds at each event. For instance, when the system has just hit the switch-off manifold ($T_{\mathrm{OFF}}^{\mathrm{th}}$), we only need to decide for the next switch-on manifold ($T_{\mathrm{ON}}^{\mathrm{th}}$). This helps to reduce the action dimension to one. Next, we present the parameterization for the stochastic policy approach followed up by the deterministic policy approach. In the stochastic policy method, we constrain the policy distributions to the Gaussian distributions of the form: $$\pi_{\theta}(T^{\mathrm{th}}|s) \doteq \frac{1}{\sigma_{\theta^{\sigma}}(s)\sqrt{2\pi}} \exp\left(-\frac{\left(T^{\mathrm{th}}-m_{\theta^{m}}(s)\right)^2}{2{\sigma_{\theta^{\sigma}}(s)}^2}\right),
\label{Eq:Gaussian}$$ where, $m_{\theta^{m}}(s)$, and $\sigma_{\theta^{\sigma}}(s)$ are mean and standard deviation of the action that are parameterized by parameter vectors $\theta^{m}$ and $\theta^{\sigma}$, respectively ($\theta=[\theta^m, \theta^{\sigma}]^{\mathrm{T}}$). Here, we consider constant switch-on and switch-off thresholds and parameterize the mean and standard deviation as follows: $$\begin{aligned}
m_{\theta^{m}}(s) &\doteq& {\theta^{m}}^\top \phi(s)  \nonumber \\
\sigma_{\theta^{\sigma}}(s) &\doteq& \exp({\theta^{\sigma}}^\top \phi(s)),
\label{Eq:meanstandard}\end{aligned}$$ where, $\theta^{m}=[\theta^m_{\mathrm{ON}}, \theta^m_{\mathrm{OFF}}]^\top$ and $\theta^{\sigma}=[\theta^{\sigma}_{\mathrm{ON}}, \theta^{\sigma}_{\mathrm{OFF}}]^\top$. For simplicity we later assume $\theta^{\sigma}_{\mathrm{ON}} = \theta^{\sigma}_{\mathrm{OFF}}=\theta_{\sigma}$. $\phi(s)=[1-h_s, h_s]^\top$ is the state feature vector. We also approximate the state-value function as $V_v(s)=[v_1, v_2]^\top \phi(s)$. It should be noted that with this simple parameterization the switching temperature thresholds do not depend on the outdoor temperature. This is a reasonable assumption because we know that if the outdoor temperature is fixed, the optimal thresholds should indeed be constant.\
In a similar fashion, we simplify the parameterization of the deterministic policy in the form of: $$T^{\mathrm{th}}_{\theta}(s) = \mu_{\theta}(s) \doteq \theta^\top \phi(s),
     \label{Eq:deterpolicy}$$ where, $\theta=[\theta_{\mathrm{ON}}, \theta_{\mathrm{OFF}}]^\top$ is the policy parameter vector. We approximate the action-value function by a compatible function approximator as $Q^w(s,a)=(a-\mu_{\theta}(s))^\top{\nabla_{\theta}\mu_{\theta}(s)}^\top w + V^v(s)$ with $w=[w_1, w_2]^\top$. The state feature vector $\phi(s)$ and the state-value function $V_v(s)$ are defined the same as in the stochastic policy approach.

Results
-------

Having set-up the simulation environment and parameterized the control policies and the related function approximators, we can now implement the learning algorithms \[Algo:stochastic\] and \[Algo:deterministic\]. In order to asses the efficacy of our learning control methods we would better have the ground truth optimal switching thresholds to which the results of our learning algorithms should converge. It should be noted that even with a simple and known model of the building with no disturbances, the optimal control of energy cost minimization while improving the occupants’ comfort does not fall into any of the classical optimal control problems such as LQG or LQR. This is mainly because of the complex form of the reward or the cost function defined in Eq.(\[Eq:reward\]). With that said, since we know that the optimal thresholds are constant (for fixed outdoor temperature), it is not computationally very heavy to find the ground truth thresholds by brute-force simulations and policy search in this set-up.\
To this end, we run numerous simulations where the system dynamics are described by either Eq.(\[Eq:linear\]) or the EnergyPlus model, and the control policy by Eq.(\[Eq:deterpolicy\]) with constant parameter vector $\theta$[^1]. For each such simulation, the simulation is run for a long time with a fixed pair of switching temperature thresholds at the end of which the average reward rate is calculated dividing the total reward by the total time. For the case where the system dynamics are described by Eq.(\[Eq:linear\]), results are illustrated in Fig. \[Fig:3\] based on which the optimal average reward rate is $r(\mu)=-3.70 \, unit\,{hr}^{-1}$ corresponding to optimal thresholds of $T_{\mathrm{ON}}^{\mathrm{th}}=12.5 \, \degree C$ and $T_{\mathrm{OFF}}^{\mathrm{th}}=17.5 \, \degree C$. Knowing the optimal policy for the simplified linear model of the building, we next implement our proposed stochastic and deterministic learning algorithms on this building model.\

Figure \[Fig:4\] depicts the on-policy learning of stochastic policy parameters during a training period of 10 days. Initial values of the mean of the threshold temperatures $[\theta^m_\mathrm{ON}, \theta^m_\mathrm{OFF}]$ are set to $[11.0, 19.0]\, \degree C$ and the initial standard deviation of these threshold temperatures are set to $1.0 \, \degree C$. Figure \[Fig:5\] illustrates probability distributions of the stochastic policies for switching temperature thresholds before and after the 10-day training by Algorithm \[Algo:stochastic\]. As seen in these two figures, the mean temperature thresholds have reached $12.3 \, \degree C$ and $ 17.5\, \degree C$, very close to the true optimal values. The standard deviation has decreased to $0.17 \, \degree C$ by the end of the training. According to Fig. \[Fig:6\] the average reward rate is learnt and converges to a value of $-3.73 \, unit\,{hr}^{-1}$. This learnt policy is then implemented from the beginning in a separate 10-day simulation and the average reward rate is calculated as $-3.74 \, unit\,{hr}^{-1}$. Both of these value are very close to the optimal value of $-3.70 \, unit\,{hr}^{-1}$, confirming the efficacy of the proposed event-triggered stochastic learning algorithm.\

Next, we implement our deterministic event-triggered learning algorithm (Algorithm \[Algo:deterministic\]) on the same building model. The learnt on/off switching temperatures at the end of a 10-day training are found to be $12.4 \, \degree C$ and $ 17.3\, \degree C$, again very close to the true optimal values. The implemented ET-COPDAC-Q is an off-policy algorithm; hence, to assess its efficacy we need to implement the resulted learnt policy on a new simulation where the average reward is calculated based on the learnt policy applied from the beginning. The average reward rate corresponding to the learnt thresholds is then calculated to be $-3.73 \, unit\,{hr}^{-1}$ that is very close to the optimal value of $-3.70 \, unit\,{hr}^{-1}$.\
It was explained in detail in sections \[manifolds\] and \[RL\] that the proposed event-triggered learning and control with variable time intervals should improve learning and control performance in terms of sample efficiency and variance. To back-up this via simulations we run two 10-day simulations on the same building model; one with variable intervals i.e. event-triggered learning (Algorithm \[Algo:deterministic\]) and one with constant intervals with 5-minute duration. This time the event-triggered deterministic algorithm learns the exact optimal thresholds i.e. $12.5 \, \degree C$ and $ 17.5\, \degree C$ corresponding to an average reward rate of $-3.70 \, unit\,{hr}^{-1}$, whereas the same algorithm with constant time intervals learns the thresholds to be $11.2 \, \degree C$ and $ 19.3\, \degree C$. Now if the latter threshold policy is implemented with constant time interval for controls (i.e. both learning and control have constant time intervals) it results in an average reward rate of $-6.19 \, unit\,{hr}^{-1}$; however, this value improves to an average reward rate of $-5.22 \, unit\,{hr}^{-1}$ if the learnt policy is implemented via event-triggered control (i.e. constant time interval for learning but variable time interval for controls). These numbers corroborate the advantage of event-triggered learning and controls over the classic learning and controls with fixed time intervals. To highlight this advantage even more, Fig. \[Fig:7\] shows the learnt average reward rate during a 10-day training by Algorithm \[Algo:deterministic\] with both variable and constant time intervals. It is clear that learning with constant time intervals results in a considerably larger variance.\

Last but not least we implement our learning algorithms on the more realistic building modeled in EnergyPlus software as detailed in section \[BuildingModels\]. Here the outdoor temperature is no longer kept constant and varies as shown in Fig. \[Fig:8\]. Although the optimal thresholds should in general be functions of outdoor temperature, here we constrain the learning problem to the family of threshold policies that are *not* functions of outdoor temperature. This is because (i) finding the ground truth optimal policy via brute-force simulations within this constrained family of policies is much easier than the unconstrained family of threshold policies, and (ii) based on our simulation results the optimal policy has a weak dependence on the outdoor temperature in this set-up.\
Similar to the case of the simplified building model, we first find the optimal threshold policy and the corresponding optimal average reward rate by brute-force simulations. The optimal thresholds are found to be $T_{\mathrm{ON}}^{\mathrm{th}}=12.5 \, \degree C$ and $T_{\mathrm{OFF}}^{\mathrm{th}}=17.5 \, \degree C$ resulting in an optimal average reward rate of $r(\mu)=-3.31\, unit\,{hr}^{-1}$. Here we employ our deterministic event-triggered COPDAC-Q algorithm to learn the optimal threshold policy. Starting from initial thresholds of $11.0 \, \degree C$ and $ 19.0\, \degree C$, the algorithm learns the threshold temperatures to be $12.9 \, \degree C$ and $ 17.5\, \degree C$ at the end of 10 days of training. This learnt policy results in an average reward rate of -$3.37\, unit\,{hr}^{-1}$. Time history of the building’s indoor temperature controlled via an exploratory deterministic behaviour policy during the 10-day training period is illustrated in Fig.\[Fig:8\]. The learning time history of the deterministic policy parameters, i.e. the switching temperature thresholds during the 10-day training is shown in Fig.\[Fig:9\].

Conclusion
==========

This study focuses on event-triggered learning and control in the context of cyber-physical systems with an application to buildings’ micro-climate control. Often learning and control systems are designed based on sampling with *fixed* time intervals. A shorter time interval usually lead to a more-accurate learning and more-precise control system; however, it inherently increases sample complexity and variance of the learning algorithms and requires more computational resources. To remedy these issues we proposed an event-triggered paradigm for learning and control with variable time intervals and showed its efficacy in designing a smart learning thermostat for autonomous micro-climate control in buildings.\
We formulated the buildings’ climate control problem based on a continuing-task MDP with event-triggered control policies. The events occur when the system state crosses the a priori-parameterized *switching manifolds*; this crossing triggers the learning as well as the control processes. Policy gradient and temporal difference methods are employed to learn the optimal switching manifolds which define the optimal control policy. Two event-triggered learning algorithms are proposed for stochastic and deterministic control policies. These algorithms are implemented on a single-zone building to concurrently decrease buildings’ energy consumption and increase occupants’ comfort. Two different building models were used: (i) a simplified model where the building’s thermodynamics are characterized by a first-order ordinary differential equation, and (ii) a more realistic building modeled in the EnergyPlus software. Simulation results show that the proposed algorithms learn the optimal policy in a reasonable time. The results also confirm that in terms of sample efficiency and variance our proposed event-triggered algorithms outperform their classic reinforcement learning counterparts where learning and control happen with constant time intervals.

Acknowledgements {#acknowledgements .unnumbered}
================

This work is supported by the Skoltech NGP Program (joint Skoltech-MIT project).

[^1]: We know the optimal policy should be a deterministic policy with constant switching temperature thresholds.
