Many deep reinforcement learning approaches use graphical state representations,
 this means visually distinct games that share the same underlying structure cannot
 effectively share knowledge. This paper outlines a new approach for learning
 underlying game state embeddings irrespective of the visual rendering of the game
 state. We utilise approaches from multi-task learning and domain adaption in
 order to place visually distinct game states on a shared embedding manifold. We
 present our results in the context of deep reinforcement learning agents. Games have often been used in order to experiment with new approaches with Artificial Intelligence (AI) research. Games provide flexibility to simulate a range of problems such as fully observable vs partially observable, stochastic vs discrete and noisy vs noise free environments. This first started with digital versions of board games being used such as backgammon and chess. More recently video games have begun to provide a plethora of digital environments and tasks for benchmarking AI systems. These new systems use neural networks and are usually trained using the raw pixel values of game frames, meaning the networks have to interpret these pixels into game states that can then be used to learn an optimal policy for play. Due to the fact that they use these raw pixel values they are sensitive to changes in the visuals of the game used, this results in very little knowledge transfer between visually distinct games BID25 resulting in the networks learning each game individually without any representation overlap. Games are usually very visually distinct as concepts are often abstract, especially for puzzle games. Early video games were often like this because it is very computationally expensive to create games that accurately imitate the real world, whereas more modern games may take a more abstract representation due to the financial expense.Learning representations has long been an important area of AI research. It has led to many new approaches to producing representations for many applications, such as word embeddings BID1 and style transfer BID6 . In the case of word embeddings networks can be used to solve various tasks such as predicting the context within which each word appears, the outputs of one of the hidden layers can then be used in order represent that word. This then results in word embeddings that place words that appear in similar contexts within close proximity to each other in the embedding space BID1 ). These resulting embeddings can then be used in order to solve more complex tasks without the system having to train from raw input. This could be seen as a form of knowledge transfer as the knowledge of the meaning of words has been encoded into its embedding and can then be used with new networks working on new tasks without the need to learn this mapping between words and their context again BID27 .In our work we improve knowledge representation across tasks that have underlying similarities but are represented in a visually distinct way. The architecture we propose in this paper learns representations that are independent of the visual differences of the games. This will result in the strategic elements of the game playing network to share knowledge between visually distinct games.In order to achieve this we use and extend work that has been done around domain adaption. Domain adaption seeks to produce shared representation between two separate domains a source domain and a target, such as high resolution product photos and images taken with a low resolution webcam BID5 . We present a method for using similar techniques in the domain of reinforcement learning allowing an agent to learn domain independent representations for a group of similar games that are visually distinct. This paper will first ground our work in the context of both learning representations and reinforcement learning. We will then outline the environment that the networks were trained in along with the desired outcomes. We will finally present the resulting representations and outline future work to extend this approach.2 RELATED WORK 2.1 AI AND GAMES Games provide a good benchmark for AI as they require high level reasoning and planning BID32 BID18 . This means that they have often been used to signify advances in the state of the art such as with Deep Blue the chess playing AI that beat Gary Kasparov BID2 and AlphaGo the go playing agent that beat Lee Sedol . Games also provide a nice interface for agents to be able to either look into the internal state of the games, as needed for methods such as Monte carlo tree search (MCTS) , or can provide visual representations of state that can be used by agents . Games also allow agents to process experiences much faster than would be possible in the real world. This means data hungry methods are still able to learn in a relatively short period of time BID13 ).There has also been a significant amount of research into other areas of game playing, such as competitions aimed at agents passing a form of Turing Test in order to rank agents based on their ability to mimic human play BID9 ). Most of these systems have revolved around using data from human play in order to achieve this goal BID24 BID14 . There have also been competitions organised in order to assess agents ability to generalise across a variety of games, the General Game Playing Competition BID7 focusing on playing multiple board games, and the General Video Game Playing Competition that assess agents performance across a broad range of video games. Other work has also started that uses modified versions of game engines in order to provide environments to teach AI, these include Project Malmo that uses the popular Minecraft Game in order to provide an experimentation platform for AI agents BID11 , and OpenArena, a modified version of the ID Tech 3 engine used by Quake 3, in order to train and benchmark AI at a range of tasks including path finding in a Labyrinth and laser tag BID10 ).There has also been some work in using neural evolution to evolve a network to control an agent in an first-person shooter (FPS) environment BID22 . Others have investigated the use of hierarchical approaches in the design of AI agents for FPS games BID28 that use networks in a hierarchical fashion to deconstruct the tasks into sub-skills. As our results show it is possible for our system to separate game states from raw pixel values despite the renderings of the game being so distinct. It also effectively separates out states that require a different task to be completed, with the game states that contain pickups being in a distinct but related space to the states that require the player to now head towards the goal. This shows that the learned embeddings are capturing a lot about how the game is played irrespective of how the frame is rendered. We show that this is possible in the context of deep reinforcement learning for domain adaptions to be successfully achieved using adversarial networks. Our network also manages to deal with more visually diverse inputs than was possible with a network that fully shares its parameters. As the results have shown the single shared convolutional layers did not have the capacity to deal with the differences between the visual representations of the game. We have shown that it is possible to use adversarial networks along with separate convolutional layers in order to produce shared embedding spaces for visually distinct inputs.
An approach to learning a shared embedding space between visually distinct games. A new approach for learning underlying structure of visually distinct games combining convolutional layers for processing input images, Asynchronous Advantage Actor Critic for deep reinforcement learning and adversarial approach to force the embedding representation to be independent of the visual representation of games Introduces a method to learn a policy on visually distinct games by adapting deep reinforcement learning. This paper discusses an agent architecture which uses a shared representation to train multiple tasks with different sprite level visual statistics