Hierarchical Reinforcement Learning: A Survey
---
paper_title: A hierarchical reinforcement learning algorithm based on heuristic reward function

paper_content:

A hierarchical reinforcement learning method based on heuristic reward function is proposed to solve the problem of “curse of dimensionality”, that is the states space will grow exponentially in the number of features, and low convergence speed. The method can reduce state spaces greatly and can enhance the speed of the study. Choose actions with favorable purpose and efficiency so as to optimize reward function and enhance convergence speed. Apply this method to the Tetris game; the experiment result shows that the method can partly solve the “curse of dimensionality” and can enhance the convergence speed prominent.
---
paper_title: Recent Advances in Hierarchical Reinforcement Learning

paper_content:

Reinforcement learning is bedeviled by the curse of dimensionality: the number of parameters to be learned grows exponentially with the size of any compact encoding of a state. Recent attempts to combat the curse of dimensionality have turned to principled ways of exploiting temporal abstraction, where decisions are not required at each step, but rather invoke the execution of temporally-extended activities which follow their own policies until termination. This leads naturally to hierarchical control architectures and associated learning algorithms. We review several approaches to temporal abstraction and hierarchical organization that machine learning researchers have recently developed. Common to these approaches is a reliance on the theory of semi-Markov decision processes, which we emphasize in our review. We then discuss extensions of these ideas to concurrent activities, multiagent coordination, and hierarchical memory for addressing partial observability. Concluding remarks address open challenges facing the further development of reinforcement learning in a hierarchical setting.
---
paper_title: Adopting animal concepts in hierarchical reinforcement learning and control of intelligent agents

paper_content:

This research integrates rigorous methods of reinforcement learning (RL) and control engineering with a behavioral (ethology) approach to the agent technology. The main outcome is a hybrid architecture for intelligent autonomous agents targeted to the Artificial Life like environments. The architecture adopts several biology concepts and shows that they can provide robust solutions to some areas. The resulting agents perform from primitive behaviors, simple goal directed behaviors, to complex planning. The agents are fully autonomous through environment feedback evaluating internal agent state and motivates the agent to perform behaviors that return the agent towards optimal conditions. This principle is typical to animals. Learning and control is realized by multiple RL controllers working in a hierarchy of Semi Markov Decision Processes (SMDP). Used model free Q(lambda) learning works online, the agents gain experiences during interaction with the environment. The decomposition of the root SMDP into hierarchy is automated as opposed to the conventional methods that are manual. The agents assess utility of the behavior and provide rewards to RL controller as opposed to the conventional RL methods where the rewards-situations map is defined by the designer upfront. The resulting learning algorithm converges to a recursively optimal solution with probability 1. Agent behavior is continuously optimized according to the distance from the agentpsilas optimal conditions.
---
paper_title: Combining Hierarchical Reinforcement Learning and Bayesian Networks for Natural Language Generation in Situated Dialogue

paper_content:

Language generators in situated domains face a number of content selection, utterance planning and surface realisation decisions, which can be strictly interdependent. We therefore propose to optimise these processes in a joint fashion using Hierarchical Reinforcement Learning. To this end, we induce a reward function for content selection and utterance planning from data using the PARADISE framework, and suggest a novel method for inducing a reward function for surface realisation from corpora. It is based on generation spaces represented as Bayesian Networks. Results in terms of task success and human-likeness suggest that our unified approach performs better than a baseline optimised in isolation or a greedy or random baseline. It receives human ratings close to human authors.
---
paper_title: Hierarchical sub-task decomposition for reinforcement learning of multi-robot delivery mission

paper_content:

In applying reinforcement learning (RL) to multi-robot control, the size of the learning state space easily explodes because the state space has a high dimension. Hierarchical reinforcement learning (HRL) is one of the most practical approaches to solve the problem; however, automatically decomposing a plain MDP state space into sub-spaces has not been studied thoroughly enough to be applied to practical robotics problems. We propose a method that automatically forms hierarchical sub-tasks for multi-robot delivery missions. The method executes sub-task decomposition and the learning process in a step-by-step manner, by widening the robot's range of movements around the load and gradually decreasing the domain of the load position. The method automatically detects the state in which cooperative motion among the robots is needed for them to accomplish the mission. The performance of the method is demonstrated by simulations.
---
paper_title: A hierarchical representation policy iteration algorithm for reinforcement learning

paper_content:

This paper presents a hierarchical representation policy iteration (HRPI) algorithm. It is based on the method of state space decomposition implemented by introducing a binary tree. Combining the RPI algorithm with the state space decomposition method, the HRPI algorithm is proposed. In HRPI, the state space is decomposed into multiple sub-spaces according to an approximate value function, then the local policies are estimated on each sub-space and finally the global near-optimal policy is obtained by combining these local policies. The simulation results indicate that the proposed method has better performance compared to the conventional RPI algorithm.
---
paper_title: Course-Scheduling Algorithm of Option-Based Hierarchical Reinforcement Learning

paper_content:

This paper proposes applying Option-based hierarchical reinforcement learning algorithm to timetable scheduling strategy learning in colleges and universities so as to enhance the performance of traditional reinforcement learning course-scheduling algorithm. From experimentation and application in course-scheduling system, it can be seen that Option-based hierarchical reinforcement learning course scheduling algorithm has a better effect than traditional reinforcement learning course-scheduling algorithm.
---
paper_title: A Hierarchical Path Planning Approach Based on Reinforcement Learning for Mobile Robots

paper_content:

In this paper, we propose a novel hierarchical path planning algorithm for mobile robots based on A* and reinforcement learning (RL) with the structure of two layers. In the first layer, we adopt the A* search algorithm to plan a geometric path and select several points as sub-target points for the planning of the next stage. In the second layer, a local path planning algorithm based on an approximate RL method called Least Square Policy Iteration (LSPI) is used to find a kinematically feasible path with these sub-targets. After learning, the local path planner in the second layer has good generalization performance. The path obtained by the proposed algorithm is smooth and safe for executing. Simulations have been carried out and the results demonstrate the validity of the proposed scheme.
---
paper_title: Recent Advances in Hierarchical Reinforcement Learning

paper_content:

Reinforcement learning is bedeviled by the curse of dimensionality: the number of parameters to be learned grows exponentially with the size of any compact encoding of a state. Recent attempts to combat the curse of dimensionality have turned to principled ways of exploiting temporal abstraction, where decisions are not required at each step, but rather invoke the execution of temporally-extended activities which follow their own policies until termination. This leads naturally to hierarchical control architectures and associated learning algorithms. We review several approaches to temporal abstraction and hierarchical organization that machine learning researchers have recently developed. Common to these approaches is a reliance on the theory of semi-Markov decision processes, which we emphasize in our review. We then discuss extensions of these ideas to concurrent activities, multiagent coordination, and hierarchical memory for addressing partial observability. Concluding remarks address open challenges facing the further development of reinforcement learning in a hierarchical setting.
---
paper_title: Multi-task reinforcement learning: a hierarchical Bayesian approach

paper_content:

We consider the problem of multi-task reinforcement learning, where the agent needs to solve a sequence of Markov Decision Processes (MDPs) chosen randomly from a fixed but unknown distribution. We model the distribution over MDPs using a hierarchical Bayesian infinite mixture model. For each novel MDP, we use the previously learned distribution as an informed prior for modelbased Bayesian reinforcement learning. The hierarchical Bayesian framework provides a strong prior that allows us to rapidly infer the characteristics of new environments based on previous environments, while the use of a nonparametric model allows us to quickly adapt to environments we have not encountered before. In addition, the use of infinite mixtures allows for the model to automatically learn the number of underlying MDP components. We evaluate our approach and show that it leads to significant speedups in convergence to an optimal policy after observing only a small number of tasks.
---
paper_title: Spatially-aware dialogue control using hierarchical reinforcement learning

paper_content:

This article addresses the problem of scalable optimization for spatially-aware dialogue systems. These kinds of systems must perceive, reason, and act about the spatial environment where they are embedded. We formulate the problem in terms of Semi-Markov Decision Processes and propose a hierarchical reinforcement learning approach to optimize subbehaviors rather than full behaviors. Because of the vast number of policies that are required to control the interaction in a dynamic environment (e.g., a dialogue system assisting a user to navigate in a building from one location to another), our learning approach is based on two stages: (a) the first stage learns low-level behavior, in advance; and (b) the second stage learns high-level behavior, in real time. For such a purpose we extend an existing algorithm in the literature of reinforcement learning in order to support reusable policies and therefore to perform fast learning. We argue that our learning approach makes the problem feasible, and we report on a novel reinforcement learning dialogue system that performs a joint optimization between dialogue and spatial behaviors. Our experiments, using simulated and real environments, are based on a text-based dialogue system for indoor navigation. Experimental results in a realistic environment reported an overall user satisfaction result of 89p, which suggests that our proposed approach is attractive for its application in real interactions as it combines fast learning with adaptive and reasonable behavior.
---
paper_title: Reinforcement Learning, Conditioning, and the Brain: Successes and Challenges

paper_content:

The field of reinforcement learning has greatly influenced the neuroscientific study of conditioning. This article provides an introduction to reinforcement learning followed by an examination of the successes and challenges using reinforcement learning to understand the neural bases of conditioning. Successes reviewed include (1) the mapping of positive and negative prediction errors to the firing of dopamine neurons and neurons in the lateral habenula, respectively; (2) the mapping of model-based and model-free reinforcement learning to associative and sensorimotor cortico-basal ganglia-thalamo-cortical circuits, respectively; and (3) the mapping of actor and critic to the dorsal and ventral striatum, respectively. Challenges reviewed consist of several behavioral and neural findings that are at odds with standard reinforcement-learning models, including, among others, evidence for hyperbolic discounting and adaptive coding. The article suggests ways of reconciling reinforcement-learning models with many of the challenging findings, and highlights the need for further theoretical developments where necessary. Additional information related to this study may be downloaded from http://cabn.psychonomic-journals.org/content/supplemental.
---
paper_title: Hierarchical reinforcement learning using path clustering

paper_content:

In this paper we intend to study the possibility to improve the performance of the Q-Learning algorithm, by automatically finding subgoals and making better use of the acquired knowledge. This research explores a method that allows an agent to gather information about sequences of states that lead to a goal, detect classes of common sequences and introduce the states at the end of these sequences as subgoals. We use the taxiproblem (a standard in Hierarchical Reinforcement Learning literature) and conclude that, even though this problem's scale is relatively small, in most of the cases subgoals do improve the learning speed, achieving relatively good results faster than standard Q-Learning. We propose a specific iteration interval as the most appropriate to insert subgoals in the learning process. We also found that early adoption of subgoals may lead to suboptimal learning. The extension to more challenging problems is an interesting subject for future work.
---
paper_title: Hierarchical Reinforcement Learning Approach for Motion Planning in Mobile Robotics

paper_content:

The motion planning task for a mobile robot aims to generate a free-collision path from an initial point to a target point. This task may be highly complex because it requires a complete knowledge of the robot's environment. In this paper an option-based hierarchical learning approach is proposed to this problem in which basic behaviors are applied in order to accomplish the robot motion planning task. Each behavior is independently learned by the robot in the learning phase. Afterward, the robot learns to coordinate these basic behaviors to resolve the motion planning task. The application of the learning approach is validated with robot motion planning tasks in simulation as well as in an experimental environment. The results show a solution to the motion planning problem that can be highly successful in new unknown environments.
---
paper_title: Course-Scheduling Algorithm of Option-Based Hierarchical Reinforcement Learning

paper_content:

This paper proposes applying Option-based hierarchical reinforcement learning algorithm to timetable scheduling strategy learning in colleges and universities so as to enhance the performance of traditional reinforcement learning course-scheduling algorithm. From experimentation and application in course-scheduling system, it can be seen that Option-based hierarchical reinforcement learning course scheduling algorithm has a better effect than traditional reinforcement learning course-scheduling algorithm.
---
paper_title: Reinforcement learning: The Good, The Bad and The Ugly

paper_content:

Reinforcement learning provides both qualitative and quantitative frameworks for understanding and modeling adaptive decision-making in the face of rewards and punishments. Here we review the latest dispatches from the forefront of this field, and map out some of the territories where lie monsters.
---
paper_title: Hierarchical multi-agent reinforcement learning

paper_content:

In this paper, we investigate the use of hierarchical reinforcement learning (HRL) to speed up the acquisition of cooperative multi-agent tasks. We introduce a hierarchical multi-agent reinforcement learning (RL) framework, and propose a hierarchical multi-agent RL algorithm called Cooperative HRL. In this framework, agents are cooperative and homogeneous (use the same task decomposition). Learning is decentralized, with each agent learning three interrelated skills: how to perform each individual subtask, the order in which to carry them out, and how to coordinate with other agents. We define cooperative subtasks to be those subtasks in which coordination among agents significantly improves the performance of the overall task. Those levels of the hierarchy which include cooperative subtasks are called cooperation levels. A fundamental property of the proposed approach is that it allows agents to learn coordination faster by sharing information at the level of cooperative subtasks, rather than attempting to learn coordination at the level of primitive actions. We study the empirical performance of the Cooperative HRL algorithm using two testbeds: a simulated two-robot trash collection task, and a larger four-agent automated guided vehicle (AGV) scheduling problem. We compare the performance and speed of Cooperative HRL with other learning algorithms, as well as several well-known industrial AGV heuristics. We also address the issue of rational communication behavior among autonomous agents in this paper. The goal is for agents to learn both action and communication policies that together optimize the task given a communication cost. We extend the multi-agent HRL framework to include communication decisions and propose a cooperative multi-agent HRL algorithm called COM-Cooperative HRL. In this algorithm, we add a communication level to the hierarchical decomposition of the problem below each cooperation level. Before an agent makes a decision at a cooperative subtask, it decides if it is worthwhile to perform a communication action. A communication action has a certain cost and provides the agent with the actions selected by the other agents at a cooperation level. We demonstrate the efficiency of the COM-Cooperative HRL algorithm as well as the relation between the communication cost and the learned communication policy using a multi-agent taxi problem.
---
paper_title: Connect-Based Subgoal Discovery for Options in Hierarchical Reinforcement Learning

paper_content:

This paper presents a new method which uses a connect-based thought to automatically discover subgoals in a dynamic environment. We argue that the states which are not only visited frequently in the whole space but also have a relative high visiting frequency in their neighboring regions performing a critical role in connecting the neighbor states are subgoals, and then propose a novel algorithm for identifying them by considering one state who has a high sum of the relative exceeding frequencies from all its neighbor states as a subgoal. Most earlier methods actually discover a frequent visited region and then filter and select the critical states in this region, but our method directly discovers the critical states of a frequent visited region in the space without generating excessive useless candidates. Combining the global and local perspectives is a key property of our algorithm and the one that differentiates it from most of existing works in this area. Experiments show that this simple and robust approach detects subgoals quickly and correctly.
---
paper_title: A hierarchical reinforcement learning algorithm based on heuristic reward function

paper_content:

A hierarchical reinforcement learning method based on heuristic reward function is proposed to solve the problem of “curse of dimensionality”, that is the states space will grow exponentially in the number of features, and low convergence speed. The method can reduce state spaces greatly and can enhance the speed of the study. Choose actions with favorable purpose and efficiency so as to optimize reward function and enhance convergence speed. Apply this method to the Tetris game; the experiment result shows that the method can partly solve the “curse of dimensionality” and can enhance the convergence speed prominent.
---
paper_title: Combining Hierarchical Reinforcement Learning and Bayesian Networks for Natural Language Generation in Situated Dialogue

paper_content:

Language generators in situated domains face a number of content selection, utterance planning and surface realisation decisions, which can be strictly interdependent. We therefore propose to optimise these processes in a joint fashion using Hierarchical Reinforcement Learning. To this end, we induce a reward function for content selection and utterance planning from data using the PARADISE framework, and suggest a novel method for inducing a reward function for surface realisation from corpora. It is based on generation spaces represented as Bayesian Networks. Results in terms of task success and human-likeness suggest that our unified approach performs better than a baseline optimised in isolation or a greedy or random baseline. It receives human ratings close to human authors.
---
paper_title: A Neural Signature of Hierarchical Reinforcement Learning

paper_content:

SUMMARY Human behavior displays hierarchical structure: simpleactionscohereinto subtask sequences,which work together to accomplish overall task goals. Although the neural substrates of such hierarchy have been the target of increasing research, they remain poorly understood. We propose that the computations supporting hierarchical behavior may relate to those in hierarchical reinforcement learning (HRL), a machine-learning framework that extends reinforcement-learning mechanisms into hierarchical domains. To test this, we leveraged a distinctive predictionarisingfromHRL.Inordinaryreinforcement learning,rewardpredictionerrorsarecomputedwhen there is an unanticipated change in the prospects for accomplishing overall task goals. HRL entails that prediction errors should also occur in relation to task subgoals. In threeneuroimaging studieswe observed neural responses consistent with such subgoalrelated reward prediction errors, within structures previously implicated in reinforcement learning. The results reported support the relevance of HRL to the neural processes underlying hierarchical behavior.
---
paper_title: A hierarchical reinforcement learning based control architecture for semi-autonomous rescue robots in cluttered environments

paper_content:

Teleoperated rescue robots designed to explore disaster scenes and find victims face serious limitations due to the cluttered nature of the environments as well as the rescue operators becoming stressed and disoriented in these scenes. An alternative to using teleoperated control is to develop fully autonomous controllers for rescue robots. However, these robots are also not capable of traversing these complex unpredictable environments. In order to address the limitations of both teleoperation and fully autonomous robotic control for urban search and rescue (USAR) environments, semi-autonomous controllers can be developed to allow task sharing and cooperation between a human operator and a robot. In this paper, a unique Hierarchical Reinforcement Learning (HRL) based semi-autonomous control architecture is proposed. The architecture provides the robot with the ability to learn and make decisions regarding which rescue tasks, exploration or victim identification, should be carried out at a given time and whether an autonomous robot or a human controlled robot can perform these tasks more quickly and efficiently without compromising the safety of the victims, rescue workers and the rescue robot. Preliminary experiments presented here evaluate the performance of the proposed HRL control approach for a rescue robot in an unknown cluttered USAR environment.
---
paper_title: Applying hierarchical reinforcement learning to computer games

paper_content:

Hierarchical finite state machine (HFSM) has proven to be a powerful tool for controlling Non-player characters (NPCs) in computer games due to its flexibility and modularity. For most implementations, however, it is often the case that the control details at all levels are hand-coded. As a result, the development process is often time intensive and error prone. In this paper, we explore the use of a hierarchical reinforcement learning approach, based on Hierarchies of Abstract Machines (HAMs), to help overcome some of these limitations. We analyse in detail both HAMs and its use for designing HFSM, propose two HAMs-related machines, and make a preliminary experiment: applying HAMs to design NPCs' behavior and implementing it in Quake2. The result shows that this method can satisfy the need for controlling NPC and has faster convergence speed than flat reinforcement learning.
---
paper_title: Transfer in variable-reward hierarchical reinforcement learning

paper_content:

Transfer learning seeks to leverage previously learned tasks to achieve faster learning in a new task. In this paper, we consider transfer learning in the context of related but distinct Reinforcement Learning (RL) problems. In particular, our RL problems are derived from Semi-Markov Decision Processes (SMDPs) that share the same transition dynamics but have different reward functions that are linear in a set of reward features. We formally define the transfer learning problem in the context of RL as learning an efficient algorithm to solve any SMDP drawn from a fixed distribution after experiencing a finite number of them. Furthermore, we introduce an online algorithm to solve this problem, Variable-Reward Reinforcement Learning (VRRL), that compactly stores the optimal value functions for several SMDPs, and uses them to optimally initialize the value function for a new SMDP. We generalize our method to a hierarchical RL setting where the different SMDPs share the same task hierarchy. Our experimental results in a simplified real-time strategy domain show that significant transfer learning occurs in both flat and hierarchical settings. Transfer is especially effective in the hierarchical setting where the overall value functions are decomposed into subtask value functions which are more widely amenable to transfer across different SMDPs.
---
paper_title: Safe State Abstraction and Reusable Continuing Subtasks in Hierarchical Reinforcement Learning

paper_content:

Hierarchical reinforcement learning methods have not been able to simultaneously abstract and reuse subtasks with discounted value functions. The contribution of this paper is to introduce two completion functions that jointly decompose the value function hierarchically to solve this problem. The significance of this result is that the benefits of hierarchical reinforcement learning can be extended to discounted value functions and to continuing (infinite horizon) reinforcement learning problems. This paper demonstrates the method with the an algorithm that discovers subtasks automatically. An example is given where the optimum policy requires a subtask never to terminate.
---
paper_title: Hierarchical reinforcement learning using path clustering

paper_content:

In this paper we intend to study the possibility to improve the performance of the Q-Learning algorithm, by automatically finding subgoals and making better use of the acquired knowledge. This research explores a method that allows an agent to gather information about sequences of states that lead to a goal, detect classes of common sequences and introduce the states at the end of these sequences as subgoals. We use the taxiproblem (a standard in Hierarchical Reinforcement Learning literature) and conclude that, even though this problem's scale is relatively small, in most of the cases subgoals do improve the learning speed, achieving relatively good results faster than standard Q-Learning. We propose a specific iteration interval as the most appropriate to insert subgoals in the learning process. We also found that early adoption of subgoals may lead to suboptimal learning. The extension to more challenging problems is an interesting subject for future work.
---
paper_title: Hierarchical reinforcement learning with movement primitives

paper_content:

Temporal abstraction and task decomposition drastically reduce the search space for planning and control, and are fundamental to making complex tasks amenable to learning. In the context of reinforcement learning, temporal abstractions are studied within the paradigm of hierarchical reinforcement learning.
---
paper_title: Hierarchical reinforcement learning and decision making

paper_content:

The hierarchical structure of human and animal behavior has been of critical interest in neuroscience for many years. Yet understanding the neural processes that give rise to such structure remains an open challenge. In recent research, a new perspective on hierarchical behavior has begun to take shape, inspired by ideas from machine learning, and in particular the framework of hierarchical reinforcement learning. Hierarchical reinforcement learning builds on traditional reinforcement learning mechanisms, extending them to accommodate temporally extended behaviors or subroutines. The resulting computational paradigm has begun to influence both theoretical and empirical work in neuroscience, conceptually aligning the study of hierarchical behavior with research on other aspects of learning and decision making, and giving rise to some thought-provoking new findings.
---
paper_title: Hierarchical Reinforcement Learning Approach for Motion Planning in Mobile Robotics

paper_content:

The motion planning task for a mobile robot aims to generate a free-collision path from an initial point to a target point. This task may be highly complex because it requires a complete knowledge of the robot's environment. In this paper an option-based hierarchical learning approach is proposed to this problem in which basic behaviors are applied in order to accomplish the robot motion planning task. Each behavior is independently learned by the robot in the learning phase. Afterward, the robot learns to coordinate these basic behaviors to resolve the motion planning task. The application of the learning approach is validated with robot motion planning tasks in simulation as well as in an experimental environment. The results show a solution to the motion planning problem that can be highly successful in new unknown environments.
---
paper_title: Dynamic analysis of multiagent Q-learning with ε-greedy exploration

paper_content:

The development of mechanisms to understand and model the expected behaviour of multiagent learners is becoming increasingly important as the area rapidly find application in a variety of domains. In this paper we present a framework to model the behaviour of Q-learning agents using the e-greedy exploration mechanism. For this, we analyse a continuous-time version of the Q-learning update rule and study how the presence of other agents and the e-greedy mechanism affect it. We then model the problem as a system of difference equations which is used to theoretically analyse the expected behaviour of the agents. The applicability of the framework is tested through experiments in typical games selected from the literature.
---
paper_title: A hierarchical reinforcement learning based control architecture for semi-autonomous rescue robots in cluttered environments

paper_content:

Teleoperated rescue robots designed to explore disaster scenes and find victims face serious limitations due to the cluttered nature of the environments as well as the rescue operators becoming stressed and disoriented in these scenes. An alternative to using teleoperated control is to develop fully autonomous controllers for rescue robots. However, these robots are also not capable of traversing these complex unpredictable environments. In order to address the limitations of both teleoperation and fully autonomous robotic control for urban search and rescue (USAR) environments, semi-autonomous controllers can be developed to allow task sharing and cooperation between a human operator and a robot. In this paper, a unique Hierarchical Reinforcement Learning (HRL) based semi-autonomous control architecture is proposed. The architecture provides the robot with the ability to learn and make decisions regarding which rescue tasks, exploration or victim identification, should be carried out at a given time and whether an autonomous robot or a human controlled robot can perform these tasks more quickly and efficiently without compromising the safety of the victims, rescue workers and the rescue robot. Preliminary experiments presented here evaluate the performance of the proposed HRL control approach for a rescue robot in an unknown cluttered USAR environment.
---
paper_title: Applying hierarchical reinforcement learning to computer games

paper_content:

Hierarchical finite state machine (HFSM) has proven to be a powerful tool for controlling Non-player characters (NPCs) in computer games due to its flexibility and modularity. For most implementations, however, it is often the case that the control details at all levels are hand-coded. As a result, the development process is often time intensive and error prone. In this paper, we explore the use of a hierarchical reinforcement learning approach, based on Hierarchies of Abstract Machines (HAMs), to help overcome some of these limitations. We analyse in detail both HAMs and its use for designing HFSM, propose two HAMs-related machines, and make a preliminary experiment: applying HAMs to design NPCs' behavior and implementing it in Quake2. The result shows that this method can satisfy the need for controlling NPC and has faster convergence speed than flat reinforcement learning.
---
paper_title: A Hierarchical Reinforcement Learning Based Artificial Intelligence for Non-Player Characters in Video Games

paper_content:

Nowadays, video games conforms a huge industry that is always developing new technology. In particular, artificial intelligence techniques have been used broadly in the well-known non-player characters (NPC) given the opportunity to users to feel video games more real. This paper proposes the usage of the MaxQ-Q hierarchical reinforcement learning algorithm in non-player characters in order to increase the experience of the user in terms of naturalness. A case study of an NPC with the proposed artificial intelligence based algorithm in a first personal shooter video game was developed. Experimental results show that this implementation improves naturalness from the user’s point of view. In addition, the proposed MaxQ-Q based algorithm in NPCs allow to programmers a robust way to give artificial intelligence to them.
---
paper_title: Object-Oriented Representation and Hierarchical Reinforcement Learning in Infinite Mario

paper_content:

In this work, we analyze and improve upon reinforcement learning techniques used to build agents that can learn to play Infinite Mario, an action game. We extend the object-oriented representation by introducing the concept of object classes which can be effectively used to constrain state spaces. We then use this representation combined with the hierarchical reinforcement learning model as a learning framework. We also extend the idea of hierarchical RL by designing a hierarchy in action selection using domain specific knowledge. With the help of experimental results, we show that this approach facilitates faster and efficient learning for this domain.
---
paper_title: Course-Scheduling Algorithm of Option-Based Hierarchical Reinforcement Learning

paper_content:

This paper proposes applying Option-based hierarchical reinforcement learning algorithm to timetable scheduling strategy learning in colleges and universities so as to enhance the performance of traditional reinforcement learning course-scheduling algorithm. From experimentation and application in course-scheduling system, it can be seen that Option-based hierarchical reinforcement learning course scheduling algorithm has a better effect than traditional reinforcement learning course-scheduling algorithm.
---
paper_title: Hierarchical reinforcement learning using path clustering

paper_content:

In this paper we intend to study the possibility to improve the performance of the Q-Learning algorithm, by automatically finding subgoals and making better use of the acquired knowledge. This research explores a method that allows an agent to gather information about sequences of states that lead to a goal, detect classes of common sequences and introduce the states at the end of these sequences as subgoals. We use the taxiproblem (a standard in Hierarchical Reinforcement Learning literature) and conclude that, even though this problem's scale is relatively small, in most of the cases subgoals do improve the learning speed, achieving relatively good results faster than standard Q-Learning. We propose a specific iteration interval as the most appropriate to insert subgoals in the learning process. We also found that early adoption of subgoals may lead to suboptimal learning. The extension to more challenging problems is an interesting subject for future work.
---
paper_title: Hierarchical Reinforcement Learning Approach for Motion Planning in Mobile Robotics

paper_content:

The motion planning task for a mobile robot aims to generate a free-collision path from an initial point to a target point. This task may be highly complex because it requires a complete knowledge of the robot's environment. In this paper an option-based hierarchical learning approach is proposed to this problem in which basic behaviors are applied in order to accomplish the robot motion planning task. Each behavior is independently learned by the robot in the learning phase. Afterward, the robot learns to coordinate these basic behaviors to resolve the motion planning task. The application of the learning approach is validated with robot motion planning tasks in simulation as well as in an experimental environment. The results show a solution to the motion planning problem that can be highly successful in new unknown environments.
---
paper_title: Preference-Aware Web Service Composition Using Hierarchical Reinforcement Learning

paper_content:

Dynamic web service composition is currently a hot issue in research. It mainly focuses on function-oriented or QoS-oriented, but the research combines with them are scanty. In this paper, we propose an HRLPLA algorithm to implement dynamic composition using hierarchical reinforcement learning and preference reasoning, which can consider both function-oriented and Qos-oriented. Experimental results will show its high effectiveness.
---
paper_title: RLPLA: A Reinforcement Learning Algorithm of Web Service Composition with Preference Consideration

paper_content:

There are many static and dynamic Web services composition strategies, however literatures about automatic composition is very rare. In this article, a new algorithm based on reinforcement learning is proposed to realize web service composition automatically and randomly. On the other hand, the existing composition prototype systems mainly focus on function-oriented composition, but not QoS-oriented composition. After understanding the function-oriented composition by reinforcement learning, this paper then introduces preference logic to seek a QoS optimization solution, which is some kind of qualitative solution. When compared with quantitative solution it has many advantages. The result is a novel algorithm RLPLA, which is an algorithm of Web services composition based on reinforcement learning and preference logic
---
paper_title: Web Service Composition Method Using Hierarchical Reinforcement Learning

paper_content:

Through web services composition, distributed applications and enterprise business processes can be integrated by individual service components developed independently. In this chapter, we concentrate on the optimization problems of dynamic web service composition, and our goal is to find an optimal composite policy. We introduce a hierarchical reinforcement learning technique, i.e., a continuous-time unified MAXQ algorithm, to solve large-scale web service composition problems in the context of continuous-time semi-Markov decision process (SMDP) model under either average- or discounted-cost criteria. Finally, this proposed algorithm is tested in a simulation, and the experimental results show that it has better optimization performance than the flat Q-learning.
---
paper_title: Multi-robot Cooperation Based on Hierarchical Reinforcement Learning

paper_content:

Multi-agent reinforcement learning for multi-robot systems is a challenging issue in both robotics and artificial intelligence. But multi-agent reinforcement learning is bedeviled by the curse of dimensionality. In this paper, a novel hierarchical reinforcement learning approach named MOMQ is presented for multi-robot cooperation. The performance of MOMQ is demonstrated in three-robot trash collection task.
---
paper_title: A combined hierarchical reinforcement learning based approach for multi-robot cooperative target searching in complex unknown environments

paper_content:

Effective cooperation of multi-robots in unknown environments is essential in many robotic applications, such as environment exploration and target searching. In this paper, a combined hierarchical reinforcement learning approach, together with a designed cooperation strategy, is proposed for the real-time cooperation of multi-robots in completely unknown environments. Unlike other algorithms that need an explicit environment model or select parameters by trial and error, the proposed cooperation method obtains all the required parameters automatically through learning. By integrating segmental options with the traditional MAXQ algorithm, the cooperation hierarchy is built. In new tasks, the designed cooperation method can control the multi-robot system to complete the task effectively. The simulation results demonstrate that the proposed scheme is able to effectively and efficiently lead a team of robots to cooperatively accomplish target searching tasks in completely unknown environments.
---
paper_title: Deep Belief Network for Modeling Hierarchical Reinforcement Learning Policies

paper_content:

Intelligent agents over their lifetime face multiple tasks that require simultaneous modeling and control of complex, initially unknown environments, observed via incomplete and uncertain observations. In such scenarios, policy learning is subject to the curse of dimensionality, leading to scaling problems for traditional Reinforcement Learning (RL). To address this, the agent has to efficiently acquire and reuse latent knowledge. One way is through Hierarchical Reinforcement Learning (HRL), which embellishes RL with a hierarchical, model-based approach to state, reward and policy representation. This paper presents a novel learning approach for HRL based on Conditional Restricted Boltzmann Machines (CRBMs). The proposed model provides a uniform means to simultaneously learn policies and associated abstract state features, and allows learning and executing hierarchical skills within a consistent, uniform network structure. In this model, learning is performed incrementally from basic grounded features to complex abstract policies based on automatically extracted latent states and rewards.
---
paper_title: A hierarchical reinforcement learning based control architecture for semi-autonomous rescue robots in cluttered environments

paper_content:

Teleoperated rescue robots designed to explore disaster scenes and find victims face serious limitations due to the cluttered nature of the environments as well as the rescue operators becoming stressed and disoriented in these scenes. An alternative to using teleoperated control is to develop fully autonomous controllers for rescue robots. However, these robots are also not capable of traversing these complex unpredictable environments. In order to address the limitations of both teleoperation and fully autonomous robotic control for urban search and rescue (USAR) environments, semi-autonomous controllers can be developed to allow task sharing and cooperation between a human operator and a robot. In this paper, a unique Hierarchical Reinforcement Learning (HRL) based semi-autonomous control architecture is proposed. The architecture provides the robot with the ability to learn and make decisions regarding which rescue tasks, exploration or victim identification, should be carried out at a given time and whether an autonomous robot or a human controlled robot can perform these tasks more quickly and efficiently without compromising the safety of the victims, rescue workers and the rescue robot. Preliminary experiments presented here evaluate the performance of the proposed HRL control approach for a rescue robot in an unknown cluttered USAR environment.
---
paper_title: Web Service Composition Method Using Hierarchical Reinforcement Learning

paper_content:

Through web services composition, distributed applications and enterprise business processes can be integrated by individual service components developed independently. In this chapter, we concentrate on the optimization problems of dynamic web service composition, and our goal is to find an optimal composite policy. We introduce a hierarchical reinforcement learning technique, i.e., a continuous-time unified MAXQ algorithm, to solve large-scale web service composition problems in the context of continuous-time semi-Markov decision process (SMDP) model under either average- or discounted-cost criteria. Finally, this proposed algorithm is tested in a simulation, and the experimental results show that it has better optimization performance than the flat Q-learning.
---
paper_title: Hierarchical reinforcement learning using path clustering

paper_content:

In this paper we intend to study the possibility to improve the performance of the Q-Learning algorithm, by automatically finding subgoals and making better use of the acquired knowledge. This research explores a method that allows an agent to gather information about sequences of states that lead to a goal, detect classes of common sequences and introduce the states at the end of these sequences as subgoals. We use the taxiproblem (a standard in Hierarchical Reinforcement Learning literature) and conclude that, even though this problem's scale is relatively small, in most of the cases subgoals do improve the learning speed, achieving relatively good results faster than standard Q-Learning. We propose a specific iteration interval as the most appropriate to insert subgoals in the learning process. We also found that early adoption of subgoals may lead to suboptimal learning. The extension to more challenging problems is an interesting subject for future work.
---

Title: Hierarchical Reinforcement Learning: A Survey

Section 1: INTRODUCTION
Description 1: Provide an introduction to Reinforcement Learning (RL) and the motivation behind Hierarchical Reinforcement Learning (HRL).

Section 2: Reinforcement Learning
Description 2: Introduce the core concepts of Reinforcement Learning (RL), including states, actions, rewards, and the agent-environment interaction.

Section 3: Hierarchical Reinforcement Learning
Description 3: Define Hierarchical Reinforcement Learning (HRL) and discuss its benefits and techniques for problem decomposition.

Section 4: Q-Learning
Description 4: Explain the Q-learning algorithm and its importance in the context of RL.

Section 5: HRL Based Control Architecture for Semi-Autonomous Rescue Robots in Cluttered Environments
Description 5: Discuss the application of HRL in control architecture for rescue robots and techniques used to address challenges in such environments.

Section 6: HRL in Computer Games
Description 6: Explore the application of HRL in controlling Non-Player Characters (NPCs) in computer games and the advantages over traditional methods.

Section 7: Course-Scheduling Algorithm of Option-based HRL
Description 7: Present an option-based HRL approach for improving traditional course scheduling algorithms.

Section 8: HRL Approach for Motion Planning in Mobile Robotics
Description 8: Examine the implementation of HRL in motion planning tasks for mobile robots to avoid obstacles and plan paths effectively.

Section 9: Web Service Composition method using HRL
Description 9: Analyze HRL-based methods for optimizing dynamic web service compositions and handling large-scale problems.

Section 10: A Combined HRL Based Approach for Multi-robot Cooperative Target Searching in Complex Unknown Environments
Description 10: Describe a combined HRL approach for improving multi-robot cooperation in target searching and exploration tasks.

Section 11: Deep Belief Network for Modeling HRL policies
Description 11: Discuss the use of Deep Belief Networks for modeling HRL policies to address Curse of Dimensionality issues.

Section 12: CONCLUSION & FUTURE WORK
Description 12: Summarize the findings of the survey and propose future research directions in the field of HRL.