 As people learn to navigate the world, autonomic nervous system (e.g., ``fight or flight) responses provide intrinsic feedback about the potential consequence of action choices (e.g., becoming nervous when close to a cliff edge or driving fast around a bend.) Physiological changes are correlated with these biological preparations to protect one-self from danger. We present a novel approach to reinforcement learning that leverages a task-independent intrinsic reward function trained on peripheral pulse measurements that are correlated with human autonomic nervous system responses. Our hypothesis is that such reward functions can circumvent the challenges associated with sparse and skewed rewards in reinforcement learning settings and can help improve sample efficiency. We test this in a simulated driving environment and show that it can increase the speed of learning and reduce the number of collisions during the learning stage. The human autonomic nervous system (ANS) is composed of two branches. One of these, the sympathetic nervous system (SNS), is "hard-wired" to respond to potentially dangerous situations often reducing, or by-passing, the need for conscious processing. The ability to make rapid decisions and respond to immediate threats is one way of protecting oneself from danger. Whether one is in the African savanna or driving in Boston traffic.The SNS regulates a range of visceral functions from the cardiovascular system to the adrenal system BID9 . The anticipatory response in humans to a threatening situation is for the heart rate to increase, heart rate variability to decrease, blood to be diverted from the extremities and the sweat glands to dilate. This is the body's "fight or flight" response.While the primary role of these anticipatory responses is to help one prepare for action, they also play a part in our appraisal of a situation. The combination of sensory inputs, physiological responses and cognitive evaluation form emotions that influence how humans learn, plan and make decisions BID14 . Intrinsic motivation refers to being moved to act based on the way it makes one feel. For example, it is generally undesirable to be in a situation that causes fear and thus we might choose to take actions that help avoid these types of contexts in future. This is contrasted with extrinsic motivation that involves explicit goals BID4 .Driving is an everyday example of a task in which we commonly rely on both intrinsic and extrinsic motivations and experience significant physiological changes. When traveling in a car at highspeed one may experience a heightened state of arousal. This automatic response is correlated with the body's reaction to the greater threats posed by the situation (e.g., the need to adjust steering more rapidly to avoid a pedestrian that might step into the road). Visceral responses are likely to preempt accidents or other events (e.g., a person will become nervous before losing control and hitting someone). Therefore, these signals potentially offer an advantage as a reward mechanism compared to extrinsic rewards based on events that occur in the environment, such as a collision. This paper provides a reinforcement learning (RL) framework that incorporates reward functions for achieving task-specific goals and also minimizes a cost trained on physiological responses to the environment that are correlated with stress. We ask if such a reward function with extrinsic and intrinsic components is useful in a reinforcement learning setting. We test our approach by training a model on real visceral human responses in a driving task.The key challenges of applying RL in the real-world include the amount of training data required and the high-cost associated with failure cases. For example, when using RL in autonomous driving, rewards are often sparse and skewed. Furthermore, bad actions can lead to states that are both catastrophic and expensive to recover from. While much of the work in RL focuses on mechanisms that are task or goal dependent, it is clear that humans also consider the feedback from the body's nervous system for action selection. For example, increased arousal can help signal imminent danger or failure to achieve a goal. Such mechanisms in an RL agent could help reduce the sample complexity as the rewards are continually available and could signal success or failure before the end of the episode. Furthermore, these visceral signals provide a warning mechanism that in turn could lead to safer explorations.Our work is most closely related to that in intrinsically motivated learning BID4 BID26 BID7 BID17 that uses a combination of intrinsic and extrinsic rewards and shows benefits compared to using extrinsic rewards alone. The key distinction in our work is that we specifically aim to build intrinsic reward mechanisms that are visceral and trained on signals correlated with human affective responses. Our approach could also be considered a form of imitation learning BID20 BID19 BID8 BID2 as we use the signal from a human expert for training. However, a difference is that our signal is an implicit response from the driver versus an explicit instruction or action which might commonly be the case in imitation learning.The structural credit assignment problem, or generalization problem, aims to address the challenge posed by large parameter spaces in RL and the need to give the agent the ability to guess, or have some intuition about new situations based on experience BID13 . A significant advantage of our proposed method is the reduced sparsity of the reward signal. This makes learning more practical in a large parameter space. We conduct experiments to provide empirical evidence that this can help reduce the number of epochs required in learning. In a sense, the physiological response could be considered as an informed guess about new scenarios before the explicit outcome is known. The challenge with traditional search-based structured prediction is the assumptions that must be made in the search algorithms that are required BID5 . By training a classifier using a loss based on the human physiological response this problem can potentially be simplified.The core contributions of this paper are to: (1) present a novel approach to learning in which the reward function is augmented with a model learned directly from human nervous system responses, (2) show how this model can be incorporated into a reinforcement learning paradigm and (3) report the results of experiments that show the model can improve both safety (reducing the number of mistakes) and efficiency (reducing the sample complexity) of learning.In summary, we argue that a function trained on physiological responses could be used as an intrinsic reward or value function for artificially intelligent systems, or perhaps more aptly artificially emotionally intelligent systems. We hypothesize that incorporating intrinsic rewards with extrinsic rewards in an RL framework (as shown in FIG0 will both improve learning efficiency as well as reduce catastrophic failure cases that occur during the training. Heightened arousal is an key part of the "fight or flight" response we experience when faced with risks to our safety. We have presented a novel reinforcement learning paradigm using an intrinsic reward function trained on peripheral physiological responses and extrinsic rewards based on mission goals. First, we trained a neural architecture to predict a driver's peripheral blood flow modulation based on the first-person video from the vehicle. This architecture acted as the reward in our reinforcement learning step. A major advantage of training a reward on a signal correlated with the sympathetic nervous system responses is that the rewards are non-sparse -the negative reward starts to show up much before the car collides. This leads to efficiency in training and with proper design can lead to policies that are also aligned with the desired mission. While emotions are important for decision-making BID12 , they can also detrimentally effect decisions in certain contexts. Future work will consider how to balance intrinsic and extrinsic rewards and include extensions to representations that include multiple intrinsic drives (such as hunger, fear and pain).We must emphasize that in this work we were not attempting to mimic biological processes or model them explicitly. We were using a prediction of the peripheral blood volume pulse as an indicator of situations that are correlated with high arousal.
We present a novel approach to reinforcement learning that leverages a task-independent intrinsic reward function trained on peripheral pulse measurements that are correlated with human autonomic nervous system responses.  Proposes a reinforcement learning framework based on human emotional reaction in the context of autonomous driving. The authors propose to use signals, such as basic autonomic visceral responses that influence decision-making, within the RL framework by augmenting RL reward functions with a model learned directly from human nervous system responses. Proposes to use physiological signals to improve performance of reinforcement learning algorithms and build an intrinsic reward function that is less sparse by measuring heart pulse amplitude