What makes humans so good at solving seemingly complex video games?   Unlike computers, humans bring in a great deal of prior knowledge about the world, enabling efficient decision making. This paper investigates the role of human priors for solving video games. Given a sample game, we conduct a series of ablation studies to quantify the importance of various priors. We do this by modifying the video game environment to systematically mask different types of visual information that could be used by humans as priors. We find that removal of some prior knowledge causes a drastic degradation in the speed with which human players solve the game, e.g. from 2 minutes to over 20 minutes. Furthermore, our results indicate that general priors, such as the importance of objects and visual consistency, are critical for efficient game-play. While deep Reinforcement Learning (RL) methods have shown impressive performance on a variety of video games BID17 , they remain woefully inefficient compared to human players, taking millions of action inputs to solve even the simplest Atari games. Much research is currently focused on improving sample efficiency of RL algorithms BID19 BID9 . However, there is an orthogonal issue that is often overlooked: RL agents attack each problem tabula rasa, whereas humans come in with a wealth of prior knowledge about the world, from physics to semantics to affordances.Consider the following motivating example: you are tasked with playing an unfamiliar computer game shown in FIG0 (a) . No manual or instructions are provided; you don't even know which game sprite is controlled by you. Indeed, the only feedback you are ever given is "terminal", i.e. once you successfully finish the game. Would you be able to successfully finish this game? How long would it take? We recruited forty human subjects to play this game and found that subjects finished it quite easily, taking just under 1 minute of game-play or 3000 action inputs. This is not overly surprising as one could easily guess that the game's goal is to move the robot sprite towards the princess by stepping on the brick-like objects and using ladders to reach the higher platforms while avoiding the angry pink and the fire objects. Now consider a second scenario in which this same simple game is re-rendered with new textures, getting rid of semantic and affordance BID8 cues, as shown in FIG0 (b) . How would human performance change? We recruited another forty subjects to play this game and found that, on average, it took the players more than twice the time (2 minutes) and action inputs ( 6500) to complete the game. The second game is clearly much harder for humans, likely because it is now more difficult to guess the game structure and goal, as well as to spot obstacles.For comparison, we can also examine how modern RL algorithms perform on these games. This is not so simple, as most standard RL approaches expect very dense rewards (e.g. continuously updated game-score BID17 ), whereas we provide only a terminal reward, to mimic how most humans play video games. In such sparse reward scenarios, standard methods like A3C BID18 are too sample-inefficient and were too slow to finish the games. Hence, we used a curiosity-based RL algorithm specifically tailored to sparse-reward settings BID20 , which was able to solve both games. Unlike humans, RL did not show much difference between the The same game modified by re-rendering the textures. Despite the two games being structurally the same, human players took twice as long to finish the second game as the first one. In comparison, the performance of an RL agent was approximately the same for the two games. two games, taking about 4 million action inputs to solve each one. This should not be surprising: since RL did not have any prior knowledge about the world, both these games carried roughly the same amount of information from the perspective of the agent.This simple motivating experiment highlights the importance of prior knowledge that humans draw upon to quickly solve tasks given to them BID13 BID24 . Developmental psychologists have begun documenting the prior knowledge that children draw upon in learning about the world BID23 BID4 . However, these studies have not explicitly quantified the relative importance of the various priors for problem-solving.In this work, we systematically quantify the importance of different types of priors humans bring to bear while solving one particular kind of problem -video games. We chose video games as the task for our investigation because it is relatively easy to methodically change the game to include or mask different kinds of knowledge and run large-scale human studies. Furthermore, video games, such as ATARI, are a popular choice in the reinforcement learning community.The paper consists of a series of ablation studies on a specially-designed game environment, systematically masking out various types of visual information that could be used by humans as priors. The full game (unlike the motivating example above) was designed to be sufficiently complex and difficult for humans to easily measure changes in performance between different testing conditions. We find that removal of some prior knowledge causes a drastic degradation in the performance of human players from 1 minute to over 20 minutes. Another key finding of our investigation is that while specific knowledge, such as "ladders are to be climbed", "keys are used to open doors", "jumping on spikes is dangerous", is important for humans to quickly solve games, more general priors about the importance of objects and visual consistency are even more critical. While there is no doubt that the performance of deep RL algorithms is impressive, there is much to be learned from human cognition if our goal is to enable RL agents to solve sparse reward tasks with human-like efficiency. Humans have the amazing ability to use their past knowledge (i.e., priors) to solve new tasks quickly. Success in such scenarios critically depends on the agent's ability to explore its environment and then promptly learn from its successes BID6 BID5 . In this vein, our results demonstrate the importance of prior knowledge in helping humans explore efficiently in these sparse reward environments BID11 BID7 .However , being equipped with strong prior knowledge can sometimes lead to constrained exploration that might not be optimal in all environments BID14 BID3 . For instance , consider the game shown in FIG6 consisting of a robot and a princess object. The game environment also includes rewards in hidden locations (shown as dashed yellow boxes only for illustration). When tasked to play this game, human participants (n=30) immediately assume that princess is the goal and do not explore the free space containing hidden rewards. They directly reach the princess and thereby terminate the game with sub-optimal rewards. In contrast, a random agent (30 seeds) ends up obtaining almost four times more reward than human players as shown in FIG6 . Thus, while incorporating prior knowledge in RL agents has many potential benefits, future work should also consider challenges regarding under-constrained exploration in certain kinds of settings.While our paper primarily investigated object priors (and physics priors to some extent), humans also possess rich prior knowledge about the world in the form of intuitive psychology and also bring in various priors about general video game playing such as that moving up and to the right in games is generally correlated with progress, games have goals, etc. Studying the importance of such priors will be an interesting future direction of research.Building RL algorithms that require fewer interactions to reach the goal (i.e., sample efficient algorithms) is an active area of research, and further progress is inevitable. In addition to developing better optimization methods, we believe that instead of always initializing learning from scratch, either incorporating prior knowledge directly or constructing mechanisms for condensing experience into reusable knowledge (i.e., learning priors through continual learning) might be critical for building RL agents with human-like efficiency. Our work takes first steps toward quantifying the importance of various priors that humans employ in solving video games and in understanding how prior knowledge makes humans good at such complex tasks. We believe that our results will inspire researchers to think about different mechanisms of incorporating prior knowledge in the design of RL agents. We also hope that our experimental platform of video games, available in open-source, will fuel more detailed studies investigating human priors and a benchmark for quantifying the efficacy of different mechanisms of incorporating prior knowledge into RL agents.
We investigate the various kinds of prior knowledge that help human learning and find that general priors about objects play the most critical role in guiding human gameplay. The authors study by experiment, what aspects of human priors are the important for reinforcement learning in video games. The authors present a study of priors employed by humans in playing video games and demonstrates the existence of a taxonomy of features that affect the ability to complete tasks in the game to varying degrees.