Aspiration-Based Reinforcement Learning In Repeated Interaction Games: An Overview
---
paper_title: The use of learning algorithms in telephone traffic routing-A methodology

paper_content:

A theoretical as well as conceptual framework for the use of learning algorithms in telephone traffic routing is given. The approach is distinctly different from the mathematical programming methods generally used in such cases. Learning algorithms at the network nodes update their strategies for routing traffic on the basis of success or failure in completing calls. The entire system is described as a Markov process and different learning schemes are shown to lead to different flow patterns in the steady state.
---
paper_title: The chain store paradox

paper_content:

It is the purpose of this paper to present the example of a simple game in extensive form where the actual behavior of well informed players cannot be expected to agree with the clear results of game theoretical reasoning. A story about a fictitious chain store and its potential competitors is a convenient way to describe the game. This expositionary device should not be misunderstood as a model of a real situation. In view of the story the game will be called "the chain store game". The disturbing disagreement between plausible game behavior and game theoretical reasoning constitutes the "chain store paradox". The chain store paradox throws new light on the well known difficulties which arise in connection with the finite supergame of prisoners' dilemma game. A limited rationality approach seems to be needed in order to explain human strategic behavior. An attempt shall be made to discuss the possibility of a "tree-level theory of decision making" as an explanation of discrepancies between game theoretic analysis and human behavior.
---
paper_title: Evolution, Games, and Economic Behaviour

paper_content:

This textbook for advanced undergraduate and postgraduate students of Evolutionary Game Theory covers recent developments in the field, with an emphasis on economic contexts and applications. It begind with the basic ideas as they originated within the field of theoretical biology and then proceeds to the formulation of a theoretical framework that is suitable for the study of social and economic phenomena from an evolutionary perspective. Core topics include the Evolutionary Stable Strategy (EES) and Replicator Dynamics (RD), deterministic dynamic models, and stochastic perturbations. A set of short appendices presents some of the technical material referred to in the main text. Evolutionary theory is widely viewed as one of the most promising appraoches to understanding bounded rationality, learning, and change in complex social environments. New avenues of research are suggested by Vega-Redondo, and plentiful examples illustrate the theory's potential applications. The recent boom experienced by this discipline makes the book's systematic presentation of its essential contributions vital reading for any newcomer to the field.
---
paper_title: A Stochastic Learning Model of Economic Behavior

paper_content:

Stochastic learning theory, 242. — Probability matching: an example, 244. — The model, 247. — Alternative states of the market: a monopoly model, 252. — A simple market model, 254. — Further properties of the single-firm model, 256. — The dynamics of adjustment, 258. — The firm in disequilibrium, 261. — A final note: rules of thumb, 264.
---
paper_title: Epistemic conditions for Nash equilibrium

paper_content:

Sufficient conditions for Nash equilibrium in an 'n'-person game are given in terms of what the players know and believe - about the game, and about each other's rationality, actions, knowledge, and beliefs. Mixed strategies are treated not as conscious randomizations, but as conjectures, on the part of other players, as to what a player will do. Common knowledge plays a smaller role in characterizing Nash equilibrium than had been supposed. When 'n' = 2, mutual knowledge of the payoff functions, of rationality, and of the conjectures implies that the conjectures form a Nash equilibrium. When 'n [greater than or equal to] 3 and there is a common prior, mutual knowledge of the payoff functions and of rationality, and common knowledge of the conjectures, imply that the conjectures form a Nash equilibrium. Examples show the results to be tight. Copyright 1995 by The Econometric Society.
---
paper_title: The Bayesian foundations of solution concepts of games

paper_content:

We transform a non co-operati ve game into a -Bayesian decision problem for each player where the uncertainty faced by a player is the strategy choices of the other players, the pr iors of other players on the choice of other players, the priors over priors and so on.We provide a complete characterization between the extent of knowledge about the rationality of players and their ability to successfulIy eliminate strategies which are not best responses. This paper therefore provides the informational foundations of iteratively unaominated strategies and rationalizable strategic behavior (Bernheim (1984) and Pearce (1984». Moreover, sufficient condi tions are also found for Nash equilibrium behavior. We also provide Aumann's (1985) results on correlated equilibria .
---
paper_title: Case-Based Decision Theory

paper_content:

This paper suggests that decision-making under uncertainty is, at least partly, case-based. We propose a model in which cases are primitive, and provide a simple axiomatization of a decision rule that chooses a "best" act based on its past performance in similar cases. Each act is evaluated by the sum of the utility levels that resulted from using this act in past cases, each weighted by the similarity of that past case to the problem at hand. The formal model of case-based decision theory naturally gives rise to the notions of satisficing decisions and aspiration levels.
---
paper_title: Learning in Extensive-Form Games: Experimental Data and Simple Dynamic Models in the Intermediate Term

paper_content:

We use simple learning models to track the behavior observed in experiments concerning three extensive form games with similar perfect equilibria. In only two of the games does observed behavior approach the perfect equilibrium as players gain experience. We examine a family of learning models which possess some of the robust properties of learning noted in the psychology literature. The intermediate term predictions of these models track well the observed behavior in all three games, even though the models considered differ in their very long term predictions. We argue that for predicting observed behavior the intermediate term predictions of dynamic learning models may be even more important than their asymptotic properties. Journal of Economic Literature Classification Numbers: C7, C92.
---
paper_title: Predicting How People Play Games: Reinforcement Learning in Experimental Games with Unique, Mixed Strategy Equilibria

paper_content:

The authors examine learning in all experiments they could locate involving one hundred periods or more of games with a unique equilibrium in mixed strategies, and in a new experiment. They study both the ex post ('best fit') descriptive power of learning models, and their ex ante predictive power, by simulating each experiment using parameters estimated from the other experiments. Even a one-parameter reinforcement learning model robustly outperforms the equilibrium predictions. Predictive power is improved by adding 'forgetting' and 'experimentation,' or by allowing greater rationality as in probabilistic fictitious play. Implications for developing a low-rationality, cognitive game theory are discussed. Copyright 1998 by American Economic Association.
---
paper_title: Experience-weighted Attraction Learning in Normal Form Games

paper_content:

In ‘experience-weighted attraction’ (EWA) learning, strategies have attractions that reflect initial predispositions, are updated based on payoff experience, and determine choice probabilities according to some rule (e.g., logit). A key feature is a parameter δ that weights the strength of hypothetical reinforcement of strategies that were not chosen according to the payoff they would have yielded, relative to reinforcement of chosen strategies according to received payoffs. The other key features are two discount rates, φ and ρ, which separately discount previous attractions, and an experience weight. EWA includes reinforcement learning and weighted fictitious play (belief learning) as special cases, and hybridizes their key elements. When δ= 0 and ρ= 0, cumulative choice reinforcement results. When δ= 1 and ρ=φ, levels of reinforcement of strategies are exactly the same as expected payoffs given weighted fictitious play beliefs. Using three sets of experimental data, parameter estimates of the model were calibrated on part of the data and used to predict a holdout sample. Estimates of δ are generally around .50, φ around .8 − 1, and ρ varies from 0 to φ. Reinforcement and belief-learning special cases are generally rejected in favor of EWA, though belief models do better in some constant-sum games. EWA is able to combine the best features of previous approaches, allowing attractions to begin and grow flexibly as choice reinforcement does, but reinforcing unchosen strategies substantially as belief-based models implicitly do.
---
paper_title: The Theory of Learning in Games

paper_content:

In economics, most noncooperative game theory has focused on equilibrium in games, especially Nash equilibrium and its refinements. The traditional explanation for when and why equilibrium arises is that it results from analysis and introspection by the players in a situation where the rules of the game, the rationality of the players, and the players' payoff functions are all common knowledge. Both conceptually and empirically, this theory has many problems. In The Theory of Learning in Games Drew Fudenberg and David Levine develop an alternative explanation that equilibrium arises as the long-run outcome of a process in which less than fully rational players grope for optimality over time. The models they explore provide a foundation for equilibrium theory and suggest useful ways for economists to evaluate and modify traditional equilibrium concepts.
---
paper_title: Evolutionary Games and Equilibrium Selection

paper_content:

Evolutionary game theory is one of the most active and rapidly growing areas of research in economics. Unlike traditional game theory models, which assume that all players are fully rational and have complete knowledge of details of the game, evolutionary models assume that people choose their strategies through a trial-and-error learning process in which they gradually discover that some strategies work better than others. In games that are repeated many times, low-payoff strategies tend to be weeded out, and an equilibrium may emerge. Larry Samuelson has been one of the main contributors to the evolutionary game theory literature. In Evolutionary Games and Equilibrium Selection, he examines the interplay between evolutionary game theory and the equilibrium selection problem in noncooperative games. After an overview of the basic issues of game theory and a presentation of the basic models, the book goes on to discuss evolutionary stability, the dynamics of sample paths, the ultimatum game, drift, noise, backward and forward induction, and strict Nash equilibria.
---
paper_title: The Evolutionary Stability of Cooperation

paper_content:

Known cam and roller-ramp overrunning clutches depend on frictional forces to effect a kinematic lock between the driven and driving members and they are dynamically limited to a specific engagement-disengagement cycle. In the present invention the transmission of torque between the driving and driven members is accomplished by a worm/worm-wheel combination such that dependence on frictional forces is eliminated. The congruent surfaces of the lands on the worm and the teeth of the worm-wheel are sloped so that there is normal free-wheeling of the worm upon rotation of the worm-wheel in one direction but there is locking action upon rotation of the worm-wheel in the other direction. The lead angle of the worm is such that the worm would normally be free-wheeling in either direction of rotation. Because engagement and disengagement is positively programmed the application of the locking elements is not limited kinematically to any specific cycle.
---
paper_title: The Evolutionary Stability of Cooperation

paper_content:

Known cam and roller-ramp overrunning clutches depend on frictional forces to effect a kinematic lock between the driven and driving members and they are dynamically limited to a specific engagement-disengagement cycle. In the present invention the transmission of torque between the driving and driven members is accomplished by a worm/worm-wheel combination such that dependence on frictional forces is eliminated. The congruent surfaces of the lands on the worm and the teeth of the worm-wheel are sloped so that there is normal free-wheeling of the worm upon rotation of the worm-wheel in one direction but there is locking action upon rotation of the worm-wheel in the other direction. The lead angle of the worm is such that the worm would normally be free-wheeling in either direction of rotation. Because engagement and disengagement is positively programmed the application of the locking elements is not limited kinematically to any specific cycle.
---

Title: Aspiration-Based Reinforcement Learning In Repeated Interaction Games: An Overview

Section 1: Introduction
Description 1: Briefly introduce the topic of aspiration-based reinforcement learning in repeated interaction games, its relevance, and foundational context.

Section 2: Modeling Reinforcement Learning
Description 2: Describe the framework for modeling reinforcement learning, focusing on the dynamics of how players' strategies are selected and adjusted over time.

Section 3: Constant Consistent Aspirations
Description 3: Discuss models where players have constant aspiration levels that do not change during the game and the implications of this assumption.

Section 4: Evolving Aspirations Based on Personal Experience
Description 4: Explore models in which players' aspiration levels evolve based on their personal payoff experiences over time.

Section 5: Aspirations Evolving on the Basis of Experience of Previous Generations of Players
Description 5: Analyze models where aspirations are inherited from the experiences of previous generations, highlighting the impact on long-term outcomes.

Section 6: Aspirations Based on Observed Experience of Peers, and Alternative Interaction Patterns
Description 6: Investigate models where aspirations are shaped by the observed experiences of peers and discuss how different patterns of interaction influence learning processes.

Section 7: Concluding Comments
Description 7: Summarize the main findings, discuss the distinctive features of aspiration-based reinforcement learning, and suggest directions for future research.