---
abstract: 'We introduce a kind of partial observability to the projective simulation (PS) learning method via Dirac notation. It is done by adding a projection operator and an observability parameter to the original formulation of the efficiency in PS model. Our examples are from invasion toy problem regarding a multi-agent setting.'
author:
- 'R. Kheiri'
date: 'Received: date / Accepted: date'
title: 'A Projective Simulation Scheme for a Partially-Observable Multi-Agent Game '
---

[example.eps]{} gsave newpath 20 20 moveto 20 220 lineto 220 220 lineto 220 20 lineto closepath 2 setlinewidth gsave .4 setgray fill grestore stroke grestore

Introduction {#intro}
============

Projective simulation, to put it briefly, is an embodied learning method which tries to represent a reductionist approach of a brain-like learning (thinking) scheme utilizing random walks between some network of clips via their interface edges as episodic and compositional memories. In the article titled “projective simulation for artificial intelligence” [@Briegel1], the authors formulate PS model using a toy problem as “invasion game” and checked the speed of learning, maximum blocking efficiency, etc. from which we expand our examples on it for the early step of introducing the partially observable PS method.

As the primary feature, the PS thinking process is considered to be decoupled from immediate motor action since the random walks happen between virtual (fictitious) percept and action clips in the memory (networks of clips) itself which can be modified (updated) both in the number of its clips and/or in the transition probabilities between the clips via reflections or some compositional properties before the real action $a$ takes place. Due to the ability of the creation of new clips in the network of clips,[^1]one may think of every percept clip or every action clip just as a network clip in the system. Therefore one can formulate PS model just on the network’ clips irrespective of the kind of them. Therefore the transitional probabilities between every two clips $c_i$ and $c_j$ in the time step $t$ can be written as a normalized conditional probability function build of the wight transitions ${\omega}^{(t)} (c_i ,c_j )$. $$P^{(t)} ( c_j |c_i ) = \frac{{\omega}^{(t)} (c_i, c_j )}{\sum_k {\omega}^{(t)} (c_i, c_k)},
\label{eqprob}$$ where ${\omega}^{(t)} (c_i ,c_j ) = f (h^{(t)} (c_i ,c_j ) )$ is modified as an adaptation rule of $$h^{(t+1)} (c_i, c_j ) = h^{(t)} (c_i, c_j ) - \gamma ( h^{(t)} (c_i, c_j ) -1 ) + \sum_l \delta (c_i , c_{k_l} ) \delta (c_j, c_{m_l}) {\lambda}^{(t+1)}
\label{eqtrans}$$ when the edges $(c_{k_l}, c_{m_l})$ were traversed during the last random walk, where $0\leq \gamma \leq 1$ is a forgetting factor (damping parameter), and $\lambda$ is a non-negative reward that will increment the related *h*-value function. It follows that the forgetting factor $ \gamma $ in Eq. \[eqtrans\] can show a positive effect in speed-up learning and an adverse effect on the amount of efficiency.[^2] The simplest function for $\omega$ is for $f(h)=h$ as will be used in the current study and have used in the previous studies[^3]. In addition, the weight matrix $\omega = h^{(t)} (c_i, c_j)$ is initially unit ($h^{(1)} (c_i, c_j )=1$) for all edges.

Subsequently, learning happens by changing transition probabilities of Eq. \[eqprob\] on a given pair of $(c_i, c_j )$ by updating the rule of Eq. \[eqtrans\]. One can evaluate the efficiency of learning, $r^{(t)}$, by adding together the amounts of desirable joint probabilities $P^{(n)} (a_s^{*} , s)$ at time $t$, where $ a_s^{*} $ is a pleasing action on a given percept $ s $. Assuming that $$r^{(t)} = \sum_s P^{(t)} (s)\, P^{(t)} (a_s^{*} |s) , \quad P^{(n)} (s) = \frac{1}{\textit{number of different percepts}},$$ then we have $$r^{(t)} = \frac{1}{N} \sum_{i=1}^N P^{t} (a_{s_i}^{*} | s_i ).
\label{eqoriggin}$$

There are other important properties in the PS scheme such as edge-glow mechanism $ g(c_i, c_j ) \in [0, 1] $ [@Briegel2; @Alexey1] which refers to a delayed reward $ g^{(t+1)} (c_i, c_j ) \, \lambda^{(t+1)} $ for a non-traversed edge in the right hand of Eq. \[eqtrans\]. In the current study, however, our examples contain only the invasion game for which the agent performs optimally in case we do not have such temporal correlations that are $ g=0 $ [@Briegel3]. Accordingly, the adaptation rule of Eq. \[eqtrans\] is enough for our purpose here.

Projective simulation as a new model for classical and quantum artificial intelligence needs to be expanded in all AI directions as a lot of works have been done so far. Particularly, some classical studies are [@Briegel1; @Briegel2; @Briegel3; @Alexey1] and some quantum works are [@Dunjko1; @Tiersch11; @JensClausen.; @Alexey2; @Dunjko2]. In the current study, we use Dirac notation, which is a useful theoretical method in both classical AI and quantum context, to speak of the influence of a partially observable environment on the efficiency of a given agent, though my results are presented classically here.

Partially observability and multi-agent approaches have been considered widely in AI [@KaelblingLittman; @book:Russel; @book:Sigaud] and reinforcement learning [@Fujita; @Doshi]. In this paper, we introduce the notion of a partially observable environment to the PS learning method using a set of belief states in the formulation of efficiency. As our examples, we use a multi-agent invasion setting where another agent as an interpreter can teach the percepts to the defender. Afterwards, every agent can be considered as a simultaneous learner-teacher where there could be some game theoretical aspects therein.

A one way partially observable projective simulation (POPS)
===========================================================

In a partially observable environment, the agent may be unable to observe the current state. In this approach, a fully observable environment is just a special case of the partial observability when the belief state ($b$) is equal to the current percept ($s$) for every real percept $s$ at all times. It is believed that partially observable AI can add more realistic examples to the entirely observable AI scenarios.

We track a multi-agent partially observable example in this article, though our following formulation is independent of any multi-agent hypothesis and I think it will be beneficial as an interior network structure too. For a multi-agent setting, there are two main streams in literature as interactive partially observable [@Gmytrasiewicz1; @Gmytrasiewicz2; @Panella] and also decentralized partially observable [@Bernstein1; @bookchapter:Oliehoek; @Amato1].

In the PS context, remembering efficiency of Eq.\[eqoriggin\] and using a vector notation for $N$ different percept-actions, then $$r^{(t)} = \frac{1}{N} \sum_{i=1}^N \, \langle a_{s_i}^{*} | s_i \rangle.
\label{eqfullyy}$$

Where we omitted the superscript $ t $ for the probabilities. The environments, so far, have been stochastic, fully observable (since the agent observes real percepts) and containing one learning agent. However, people may think of expanding a vector space on a different basis to make some belief states from the world states $ | s_i \rangle $. As a tangible example, considering one another intelligent agent as an interpreter (an **intelligent projector**) as illustrated in Fig. \[interpreter:1\]. We can span our real states (world states) $ | s_i \rangle  $ on the belief states $ | b_i \rangle  $ which are what is learned by an interpreter (I) concerning $ s_i $. One can write a projection operator [@book:Sakurai] $ B_j $ as $$B_j =  |  b_j \rangle \langle b_j |.$$ Then $$| s_i \rangle = \sum_j \, |  b_j \rangle \langle b_j | s_i \rangle \quad , \quad  \sum_j \, \langle b_j | s_i \rangle = 1.$$ $$r^{(t)}= \frac{1}{N} \sum_{i=1}^N \sum_{j=1}^{N'} \, \langle a_{s_i}^{*} | \, B_j \, | s_i \rangle = \frac{1}{N} \sum_{i=1}^N \sum_{j=1}^{N'} \, \langle a_{s_i}^{*}  | b_j \rangle \, \langle b_j | s_i \rangle
\label{blockeff}$$ where $ \langle \alpha | \beta  \rangle = P^{t} ( \alpha | \beta) $ and $ {N'} $ stands for the number of possible belief states which in immediate examples equal to the number of world states $ N $. One can also write the belief states’ vectors with respect to the world states using an inverse matrix for probabilities $\langle b_i | s_i \rangle $. Notice that in the probability matrix of $\langle b_k | s_k  \rangle $, the summation of elements on a row is equal to one $ \sum_j \, \langle b_j | s_i \rangle = 1 $ but it is not the case for the summation on a column $ \sum_j \, \langle b_i | s_j \rangle $. We can also, assume that $${\delta}_{ij} = \langle s_i | s_j \rangle \, = \langle a_i | a_j \rangle \ = \langle b_i | b_j \rangle$$ where needed.

As a more general case, one can consider a combinational operator $ (S+B)_j $ which regards an **observability parameter** $ \alpha $ for a given environment that is some portion $ \alpha $ of all percepts to be fully observable and the rest of them be partially observable. In other words, a given world percept is visible with the probability of $ \alpha $ and is invisible with the probability of $1 - \alpha $.

$$\alpha \, S_j + ( 1 - \alpha ) \, B_j =  \alpha \, | s_j \rangle \langle s_j | \, + \, ( 1 - \alpha ) \,   |  b_j \rangle \langle b_j | ,$$

$$r^{(t)} = \frac{1}{N} \sum_{i=1}^N \sum_{j=1}^{N'} \, \langle a_{s_i}^{*} | \, \alpha \, S_j + ( 1 - \alpha ) \, B_j \, | s_i \rangle ,$$

$$r^{(t)}= \frac{\alpha }{N} \sum_{i=1}^N \, \langle a_{s_i}^{*} | {s_i} \rangle \, + \, \frac{1 - \alpha }{N} \sum_{i=1}^N \sum_{j=1}^{N'} \, \langle a_{s_i}^{*} | b_j \rangle \, \langle b_j | s_i \rangle .
\label{kabli}$$

Further, we could include the imaginary space to define our generic belief state $ | (S+B)_{j} \rangle $ as $$| (S+B)_{j} \rangle = \sqrt{\alpha} \, | s_{j} \rangle  + i \, \sqrt{1 - \alpha } \, | b_j \rangle ,
\label{eqktswmtt}$$ $$(S+B)_j = \, | (S+B)_{j} \rangle \,\, \langle (S+B)_{j} | .
\label{thefifi}$$ Then, one can define the efficiency of Eq.\[kabli\] as the real part of $ r^{(t)} $ that is $$r^{(t)}= \frac{1}{N} \sum_{i=1}^N \sum_{j=1}^{N'} \, \langle a_{s_i}^{*} | (S+B)_j | s_i \rangle$$ $$\begin{aligned}
efficiency \equiv Real \, [ r^{(t)} ] = \frac{\alpha }{N} \sum_{i=1}^N \, \langle a_{s_i}^{*} | {s_i} \rangle \, + \, \frac{1 - \alpha }{N} \sum_{i=1}^N \sum_{j=1}^{N'} \, \langle a_{s_i}^{*} | b_j \rangle \, \langle b_j | s_i \rangle .
\label{jankan}\end{aligned}$$ Equation \[eqktswmtt\] is well-defined. Because the operators as $ | b_j \rangle \langle s_j | $ and $ | s_j \rangle \langle b_j | $ for $ b_j \neq s_j  $ cause seemingly nonsensical probabilities in the current study. The former would bring probabilities from which when a direct world state $ s_j $ was seen, the action takes place for an indirect $ b_j $, and the effect is just vice versa for the latter.

Suppose that we have asymptotic probabilities as a function of forgetting factor, $${\langle a_k | s_l \rangle}_{max} = {\langle a_k | b_l \rangle}_{max} = 
\begin{cases}
p ( \gamma_{pa} ) , \quad k = l \\
q ( \gamma_{pa} ) , \quad k \neq l 
\end{cases} \\
,{\langle b_k | s_l \rangle}_{max} = 
\begin{cases}
p ( \gamma_{I} ) , \quad k = l \\
q ( \gamma_{I} ) , \quad k\neq l 
\end{cases}$$ where $ \gamma_{pa} $ and $ \gamma_{I} $ stand for the protagonist agent’s forgetting factor and interpreter’s forgetting factor respectively. Hence, for a simple fully observable problem with $ a_{s_i}^{*} = a_i $ we have $ r_{max} = p ( \gamma_{pa} ) $ according to Eq. \[eqoriggin\] or Eq. \[eqfullyy\]. Yet, considering our partially observable two-agent model with $ N'=N $ ($ |S| = |A| = |B|= N $) one can use Eq. \[jankan\], regarding $ r_{PO} $ for the asymptotic efficiency in a partially observable environment (versus $ r_{FO} $ preserved for a fully observable one) and write

$$\begin{gathered}
r_{PO} \equiv r_{max}^{pa} (\alpha , \gamma_{pa}, \gamma_{I} ) =  \alpha \, p ( \gamma_{pa} ) + (1- \alpha ) \left[ p ( \gamma_{pa} ) . p ( \gamma_{I} )  + q ( \gamma_{pa} ) . q ( \gamma_{I} ) \right].
\label{eqrmaxxx}\end{gathered}$$

\
$$r_{PO} = p ( \gamma_{pa} ) \left\lbrace 1 - (1 - \alpha ) \left[ 1 - p ( \gamma_{I} ) - \frac{q ( \gamma_{I} )}{p ( \gamma_{pa} ) } . q ( \gamma_{I} ) \right] \right\rbrace .
\label{lalbkusab}$$ Remembering the efficiency of a given agent in a fully observable environment, $ r_{FO} = p ( \gamma_{pa} ) $, then $$r_{PO} = \beta \,\, r_{FO} , \qquad 0 \leq \beta \leq 1 \quad \Longrightarrow \quad  r_{PO} \leq r_{FO}.$$ We may recall $ \beta $ as a transparency coefficient or $ \mu = 1 - \beta $ as the reduction of transparency. Besides, one can have $ q $ with respect to $ p $ according to the specific form of $ h $-matrix within a given problem, see Eq. \[diwwonehhha\] for instance, for which $$If \quad p + q \, ( N-1 ) = 1,$$ $$\mu = (1 - \alpha ) \, q ( \gamma_{I} ) \left[ (N - 1) - \frac{q ( \gamma_{pa} )}{p ( \gamma_{pa} ) } \right]  , \quad N \geq 2.$$\
Our formulation can go beyond to include multi-agent games where an interpreter could be a sort of protagonist agent by itself and a protagonist agent, on the other hand, would be an intelligent interpreter too so that every agent become a simultaneous player-interpreter (see Sec. \[sectionmmultii\]). Then, some world percepts would be invisible for each agent while another agent can detect them and help its partner to have more efficiency. Assuming a different forgetting factor for every task to be done for a given agent $ i $ in a multi-agent setting containing $ n $ tasks, we could have $$\gamma_{i} = \sum_{k=1}^n \gamma_{ki} , \quad  0 \leq \gamma_{i} \leq 1.
\label{gamasamam}$$ In such a scenario, the parameter of observability $ \alpha $ can also differ for two given agents $ i $ and $ j $, that is $ \alpha_{i} \neq  \alpha_{j}$.

Specific examples of a multi-agent invasion toy problem
=======================================================

The original formulation of PS has been introduced using a toy problem called invasion game as elaborated in [@Briegel1; @Briegel2]. To add partial observability employing another agent to the original projective simulation, we can assume an interpreter added to the standard invasion problem. In the basic form of the invasion game, an attacker (A) send some precepts $s \in \{ \Leftarrow , \Rightarrow \}$ (in case $ N=2 $) where a defender (D) perceives and learns them by taking action $a\in \{ - , + \}$ on a percept and getting a reward ($\lambda$). $$\{ s_1 , s_2 \} = \{ \Leftarrow , \Rightarrow \}, \qquad
\{ - , + \} = \{ a_1 , a_2 \} ,$$ where we can consider $ | s_i \rangle = \{ | \Leftarrow  \rangle , | \Rightarrow \rangle \} $ and $ | a_i \rangle = \{ | -  \rangle , | + \rangle \} $ for $ N=2 $ in an invasion game.

While the theoretical asymptotic efficiency for a small amount of $ \gamma $ has been derived for a fully observable one defender invasion game [@Briegel1; @Briegel2], that derivation cannot be used for an arbitrary choice of $ \gamma $ even in the simple form of one agent toy problem. The asymptotic efficiency relating to a given agent ”$ D $” that is $r_{max}^D = r_{max}^D ( \alpha , \gamma_{D}, \gamma_{I}) $ occurs with respect to $ \lim_{t \to \infty} h^{(t)} (s,a) $ regarding the adaptation rule of Eq. \[eqtrans\]. This adaptation is changed stochastically, however, in a big enough time, $ t \to \infty $, the averaged efficiencies on a large number of agents ($ m $) reaches a certain asymptote for every forgetting factor $ \gamma $ (see Appendix. \[averagefequa\]). Then one can assume that there is an effective reward function, $ \lambda_{eff} $, for every $ \gamma $ so that it becomes the averaged rewards obtained on a large number of agents $ m $. $${\left\langle \sum_l \delta (c_i , c_{k_l} ) \delta (c_j , c_{m_l}) {\lambda}^{(t+1)} \right\rangle }_{ m \to \infty} \longrightarrow \quad \lambda_{eff}^{(t+1)} (c_i , c_j , \gamma ) ,$$ thus $$h^{(t+1)} (c_i , c_j ) = h^{(t)} (c_i , c_j ) - \gamma ( h^{(t)} (c_i , c_j ) -1 ) + \lambda_{eff}^{(t+1)} (c_i , c_j , \gamma ) .
\label{effectivee}$$\
A constant reward for every time in Eq.\[eqtrans\], $${\lambda}^{(t+1)} (c_i , c_j ) = \lambda (c_i , c_j ),
\label{conditionddq}$$ leads to $$h_{max} = \lim_{t \to \infty} h^{(t)} (c_i , c_j ) \, = \, \frac{\lambda_{eff} (c_i , c_j , \gamma ) }{\gamma} + 1.$$ Then $$h_{max} =
\begin{cases}
\frac{\lambda_{eff}^{reward}}{\gamma} + 1 & \textit{for a rewarded pair of $ (c_i , c_j) $} \\
1 & \textit{for a non-rewarded pair of $ (c_i , c_j) $}
\end{cases}$$ Due to the fact that $ \lambda_{eff}^{non-reward} = 0 $, we rename $ \lambda_{eff}^{reward} = \lambda ( \gamma ) $ from so on. Given $ N $ different percept-action but just one desirable action $ a_{s_i}^{*} $ for each percept $ s_i $, we have $$p + q \, ( N-1 ) = 1,
\label{diwwonehhha}$$ and $$p_{N} ( \gamma ) = \frac{\lambda_{N} (\gamma) + \gamma}{\lambda_{N} (\gamma) + N \gamma} \quad , \quad q_{N} ( \gamma ) = \frac{ \gamma}{\lambda_{N} (\gamma) + N \gamma} , \qquad p(0) = 1, \qquad q(0) = 0. 
\label{incsioo}$$

Equations \[incsioo\] are the evaluations of the asymptotic probabilities in our partially observable multi-agent model of problems for which Eq. \[conditionddq\] is satisfied. Otherwise, we may use the specific properties of a given scenario to have $ p $ and $ q $.

In the following, we consider an invasion with $ N=2 $. Figure \[effrewardx\] shows the effective reward of the rewarded pairs of $(c_i, c_j )$ for a fully observable one agent invasion with $ \lambda^{t+1} (s, a_{s}^{*} ) =1 $. For instance, $$p_2 (\gamma \rightarrow 1) \longrightarrow \frac{1.2}{2.2}, \quad q_2 (\gamma \rightarrow 1) \longrightarrow \frac{1}{2.2}.$$

![image](effective_reward.eps){width="75.00000%"}

**An example of an absolute partially observable environment; Regarding Equation \[blockeff\]**
-----------------------------------------------------------------------------------------------

Fully observable environments have been considered widely with respect to Eq. \[eqfullyy\]. Here, we start from Eq. \[blockeff\] which describe an environment in which all of the percepts are invisible to the protagonist agent. Here, in my invasion example, the belief percepts are produced by an interpreter (I), such as what is depicted in Fig. \[interpreter:1\] and coming indirectly to the defender (D). To be nontrivial, the interpreter is not a kind of a simple mirror or a polarizer. Instead it is another intelligent agent that learns the percepts $ s_i $ by itself and sends what it is learning to the defender as a new state $ b_i $, where the defender percieves them and take actions $ a_i $ on them. As it is assumed in the original paper [@Briegel1], the defender always reaches sooner to the next door than the attacker (here, with or without a stamp from the interpreter).

![image](interpreter1){width="75.00000%"}

Figures \[n\_027000\] and \[n\_227000\] show a reduction in the speed of learning (or learning time) in a partially observable environment (red solid curves) in comparison with a fully observable one (dashed curves) when the interpreter’s forgetting factor is zero. Whereas, we have a reduction in the efficiency of the defender as illustrated in \[n\_127000\] and \[n\_1027000\] when the interpreter’s forgetting factor is not zero. While the reduction in the learning time is caused by that the defender learning must wait for the interpreter learning, the non-vanishing decrease in the efficiency is due to the portion of permanent partially observability arising from the interpreter’s forgetting factor.[^4]

Figure \[gasbu\] illustrates the same property in Fig. \[n\_1027000\] for extreme dissipation factor of $ \gamma = 1 $.

![image](oct_0_7){width="5.6cm" height="4.7cm"}

It is worth noting that, the multiplied probabilities of Eq.\[blockeff\] reduce the maximum blocking efficiency of a partially observable environment in comparison with a fully observable one for every $ \gamma > 0 $ even in the case of the minimum efficiency of $ \gamma = 1 $. This is because the term including $ \gamma $ in the adaptation rule of Eq. \[eqtrans\] refers to forgetting what the agent learned in the previous states, but not the current state. Therefore, this adaptation rule leaves something (even small) more than nothing ($ r_{max}  > 0.5 $) for $ \gamma = 1 $ that can be reduced in a partially observable environment.

**An example of a general fully-partially observable environment; Regarding Equation \[jankan\]** {#secsecdfga}
-------------------------------------------------------------------------------------------------

In a more general scenario, there are both partially and fully observable percepts in the environment where an agent is expected to act. A physics correspondence for such a situation might be where two kinds of ray lights are coming to a given (learning!) polarizer in the condition that the $ \alpha $ portion of rays is already polarized in the same direction of the polarizer. In our invasion example, one can add some portion ($ \alpha $) of fully observability to the problem. It means that the defender can see an $ \alpha $ portion of its percepts directly coming from the attacker and $ 1 - \alpha $ portion come indirectly from the interpreter. As a tangible example of the situation, one can suppose that the attacker’s signs are being sent in two different colors from which one of these colors are invisible for a given defender.

![image](alphainter){width="75.00000%"}

It follows that, Figs. \[gasbulbdata\] and \[gasbu\] are specific examples of the current scenario with the observability parameter of $ \alpha =0 $. A comparison between the effect of the defender’s forgetting factor ($ \gamma_{D}$) and the interpreter’s forgetting factor ($ \gamma_{D}$) is illustrated in Fig. \[pilot1\], that is, the former (Gamma D) is dominated the later (Gamma I) in the amount of efficiency. Moreover, the effect of the reduction of observability $ \alpha $ in the amount of efficiency is depicted in Fig. \[kesh2\]. What apparently seems from these two, Fig. \[pilot1\] and Fig. \[kesh2\], is that the contribution of the defender itself in its efficiency is more important than the contribution of the interpreter. To be more clear, in the following, we focus on the maximum blocking efficiency (asymptotic efficiency) containing both Gamma factors ($ \gamma_{D}$ and $\gamma_{I} $).

Figure \[3d\_alpha\_1\_0\_5\] shows that, while the blocking efficiency in the plane of $ \gamma_{D} = constant $ alters drastically with changing $\alpha$, the plane of $ \gamma_{I} = constant $ has a few changes, respecting $ \alpha $ alterations, in its amounts. It is because the reduction of the observability $ \alpha $ means an increase in the contribution of the interpreter and thus its dissipation factor $ \gamma_{I} $ becomes more important. Thereby and as it might be expected, we can see that while in $ \alpha =1 $ (Fig. \[3d\_alpha\_1\]) the interpreter’s forgetting factor has no rule in the amount of maximum efficiency due to the fact that the environment is fully observable in this case, it will be as influential as the defender’s gamma factor in $ \alpha =0 $ (the absolute partially observable environment). Therefore and because of the relevant game-theoretical considerations, in the next two asymptotic figures, I will focus just on the plane of $ \gamma_{D} = constant $ to show more details about it.

Eventually, Fig. \[alpanim\] is comparing the maximum blocking efficiencies of the defender in some planes of $ \gamma_{D} = const $ in a certain amount of $\alpha = 0.5$. Obviously, there is no intersection between the lines in Fig. \[alpanim\] which refers to the domination of a smaller $ \gamma_{D} $ for a given $\alpha$.[^5] Alternatively, when we also introduce some different $ \alpha $ factor in the planes of $ \gamma_{D} = const $, it causes a bunch of plots (or a scattering) for every $ \gamma_{D} = const $ and we can see some intersection between the lines of the asymptotic efficiencies thereafter. As a result, in a certain observability ($ \alpha = const $) the line of maximum blocking efficiency dominates for a smaller $ \gamma_{D} = const $; however, the ultimate domination of a smaller $ \gamma_{D} $ can be annihilated by changes in the amount of observability of the environment.

![image](oct_2_1.eps){width="75.00000%"}

![image](oct_2_2_4.eps){width="75.00000%"}

In the next section, we use this consequence to have some discussion about a related 2-defender-interpreter game.

**Multi-agent invasion game** {#sectionmmultii}
-----------------------------

In this section, we consider an invasion including two agents $ i $ and $ j $ such as illustrated in Fig. \[fig:1\] and then, for every agent, $ i $, there is a forgetting factor $ \gamma_{i} $. Due to the fact that every agent has two different tasks of blocking and teaching, we consider $ \gamma_{i} = \gamma_{1i}  + \gamma_{2i} $ according to Eq. \[gamasamam\], where the first Gamma $ \gamma_{1i} $ stands for a forgetting factor in a blocking task belonging to the agent $ i $ and the second Gamma $ \gamma_{2i} $ stands for its forgetting factor in a teaching (helping) task. Furthermore, it assumed that every agent can select to be absolutely selfish ($ \gamma_{1i} =0 , \gamma_{2i} = \gamma_{i}  $) as a defender, sacrifice its blocking task ($ \gamma_{1i} = \gamma_{i} , \gamma_{2i} = 0  $) to be more helpful in teaching, or having every other selection among these two border options, namely, $ \gamma_{i} = ( \gamma_{1i} , \gamma_{2i} ) $, $ \gamma_{1i} + \gamma_{2i} = \gamma_{i} $.

![image](two_defender){width="75.00000%"}

Then we can have $$\begin{gathered}
r_{max}^i (\alpha_{i} , \gamma_{1i}, \gamma_{2j} ) =  \alpha_{i} \, p ( \gamma_{1i} ) + (1- \alpha_{i}) \left[ p ( \gamma_{1i} ) . p ( \gamma_{2j} )  + q ( \gamma_{1i} ) . q ( \gamma_{2j} ) \right] ,
\label{eqrmaxx}\end{gathered}$$\
according to Eq. \[eqrmaxxx\]. For example, if $$\begin{cases}
\alpha_{i} = \alpha_{j} = \alpha , \\
\forall k \quad \gamma_{ki} =  \gamma_{kj} = \gamma_{k} , 
\end{cases}
\Longrightarrow \quad r_{max}^i (\alpha , \gamma_{k}) = r_{max}^j (\alpha , \gamma_{k}),$$\
which refers to the symmetry of the problem between two agents. Otherwise, the efficiency of one agent would differ from that of its partner as a function of its own forgetting factor, its partner’s forgetting factor and also the parameter of observability of the environment for one agent. With the same variables of $ \gamma_{1i}, \gamma_{2j} $ as regarded for Eq. \[eqrmaxx\], one would have $$\begin{gathered}
r_{max}^j (\alpha_{j} , \gamma_{1i}, \gamma_{2j} ) =  \alpha_{j} \, p ( \gamma_{j} - \gamma_{2j} ) + (1- \alpha_{j}) [ p ( \gamma_{j} - \gamma_{2j} ) . p ( \gamma_{i} - \gamma_{1i} ) \\ + q ( \gamma_{j} - \gamma_{2j} ) . q ( \gamma_{i} - \gamma_{1i} ) ]. \end{gathered}$$

### classes of coalitions {#superadditiven .unnumbered}

Considering the collective efficiency of two gaents as $ r_{col} = r_{max}^i + r_{max}^j $, one may ask about the classes of coalitions (see [@book:MultiagentShoham] p. 386 for the definitions) in this game comparing two cases: $ r_{col-FO} (\gamma_{i}, \gamma_{j} ) $ in a fully observale environment versus $ r_{col-PO} (\alpha_{i},\alpha_{j} , \gamma_{1i}, \gamma_{2j}) $ in a partially observable environment. The former is straightforward as $$r_{col-FO} (\gamma_{i}, \gamma_{j} )  = p ( \gamma_i ) + p ( \gamma_j)$$ on one hand. However, regarding the latter, one the other hand, we deal with a four variable function. At first, we can see that every *selfish-selfish* coalition $ \gamma_i = (0 , \gamma_i ) \,\, \& \,\, \gamma_j = ( 0 , \gamma_j ) $ will be superadditive, $$\begin{aligned}
r_{max}^i = \alpha_{i} + ( 1 - \alpha_i ) p ( \gamma_j ) = p ( \gamma_j ) +  \alpha_{i} q ( \gamma_j ) , \notag \\
r_{max}^j = \alpha_{j} + ( 1 - \alpha_j ) p ( \gamma_i ) = p ( \gamma_i ) +  \alpha_{j} q ( \gamma_i ) . \notag \end{aligned}$$ Then $$r_{col-PO} (\alpha_i , \alpha_j , \gamma_{1i} = 0 , \gamma_{2j} = \gamma_{j} ) \geq r_{col-FO} (\gamma_{i}, \gamma_{j} ) .
\label{supsupadlll}$$\
Equation \[supsupadlll\] will be satisfied for every $ 0 \leq \gamma_{i}, \gamma_{j} \leq 1 $. Specifically, the underlying reason for being a superaddition in the *selfish-selfish* coalition of $ \gamma_{i}= \gamma_{j}= 1 $ is that it will increase the collective efficiency only due to the use of random information that two players share with each other.

Secondly, in the case of a *sacrifice-sacrifice* coalition, $ \gamma_i = (\gamma_i , 0 ) \,\, \& \,\, \gamma_j = (\gamma_j , 0 ) $, we can obtain $ r_{max}^i  = p ( \gamma_i ) $, $ r_{max}^j  = p ( \gamma_j ) $ according to Eq. \[eqrmaxx\] that is independent of $ \alpha_{i} \,\, \& \,\, \alpha_{j} $. Therefore, $$r_{col-PO} ( \alpha_{i}, \alpha_{j}, \gamma_{1i} = \gamma_i , \gamma_{2j}= 0 ) = r_{col-FO} (\gamma_{i}, \gamma_{j} ).
\label{adlkntkacx}$$ Hence, every *sacrifice-sacrifice* coalition will be an additive game.

On the contrary, while there are a plenty of selections which for them the coalition is superadditive, especially in the big $ \alpha $ factors (see Fig. \[colcoclckeh1\] for $\alpha_{i} = \alpha_{j} =1$), there are a variety of other selections for which, $ r_{col-PO} < r_{col-FO} $, according to the example of Fig. \[colclckeh0\], especially for small observability parameters.

### Some other game theoretical aspects {#some-other-game-theoretical-aspects .unnumbered}

After all, we can consider the maximum blocking efficiencies of $ i $ and $ j $ as a simple symmetric game. Furthermore, with $\gamma_{i} = \gamma_{j} =1$, we can analyze the game using the results in Sec. \[secsecdfga\] considering a small adjustment in the variables therein so that $$\gamma_D \rightarrow \gamma_{1i}, \qquad \gamma_I \rightarrow \gamma_{2j}, \qquad \alpha \rightarrow \alpha_i .
\label{moakasdi}$$ regarding the agent $ i $.

Beside this, if we assume two agents as two players, then every selection of every agent for its forgetting factor can be considered as a different pure strategy. For the condition of $ \gamma_{1i} + \gamma_{2i} = \gamma_{1j} + \gamma_{2j} = 1 $, a given pure strategy $ A $ would be a selection as $ A \equiv \gamma = [\gamma_{A}, 1-\gamma_{A}] $. Therefore we can have two players regarding 2 different pure strategies $ A, B $ and build our game such that playing $ A [\gamma_{A}, 1-\gamma_{A}] $ for two agents means that two agents have similar selections for their blocking and teaching forgetting factors. In addition, a payoff obtained by every agent could be considered as its maximum efficiency $ r_{max} (\alpha ) $ as a function of observability for every pure strategy. Having in mind that $ \alpha_{i} \neq \alpha_{j}$, then in a general situation there would be a variety of different payoffs for every agent.

Nevertheless, regarding the modification of Eq.\[moakasdi\], considering some other restrictions might be helpful. One can see that if every agent can choose between $ A ( \alpha_A ) $ and $ B ( \alpha_B ) $, then we can have a game as the following $$\begin{aligned}
 player2\,(j) \quad \: \quad \nonumber \\ [0.2 cm]
player1 (i) \; \; \;
\begin{tabular}{|c|c|c|}
\hline  & $A ( \alpha_A )$ & $B ( \alpha_B )$ \\
\hline  $A ( \alpha_A ) $ & $a,a$ & $c,d$ \\
\hline  $ B ( \alpha_B )$ & $d,c$ & $b,b$ \\ 
\hline
\end{tabular}
\nonumber\end{aligned}$$\
where the lowercase Latin letters refer to the playoffs of each player for a given pure strategy. $$\begin{cases}
a = r_{max} ( \alpha_{A}, \gamma_{A}, 1 - \gamma_{A} ) \\
b = r_{max} ( \alpha_{B}, \gamma_{B}, 1- \gamma_{B} ) 
\end{cases} \\
,\qquad
\begin{cases}
c = r_{max} ( \alpha_{A}, \gamma_{A}, 1 - \gamma_{B} ) \\
d = r_{max} ( \alpha_{B}, \gamma_{B}, 1- \gamma_{A} ) .
\end{cases}$$ Suppose that we always have $ \gamma_{A} <  \gamma_{B} $. Then, regarding the modification of Eq.\[moakasdi\], Fig. \[alpanim\] as an example of $ \alpha = constant $ and tracking other plots in Fig. \[alpamore\] for other examples of $ \alpha = constant $ show that, in a certain amount of $ \alpha $, we have $ a +c >  b + d $. Therefore, the strictly dominant strategy for every player is being absolutely selfish (selecting the lowest amount of the blocking forgetting factor) and since we have symmetry between two agents, then (absolute) selfishness is also the only Nash equilibrium. It is a pure strategy, strictly dominant for each player as well as Pareto optimal in the case of $ \alpha_A = \alpha_B $; it can also be considered as the fair share (the simplest Shapley value) between the agents in a superadditive coalition game.

In contrast, Fig. \[alpamore\] shows that regarding different amounts of observability parameter, $ \alpha_{A} \neq \alpha_{B} $, one could have more complicated games. For example, the (absolute) selfishness as the Nash equilibrium can be dominated by another Pareto optimal solution which gives a higher payoff to two agents. In particular, tracking the scattering between the maximum and the minimum curves of $ r_{max} $ for $ \gamma_{1i} = 0 $ ($ \alpha_i = 0.99$ and $\alpha_i = 0 $ respectively) in Fig. \[alpamore\] shows that there would be a variety of other selections of $ \gamma_{1i} \neq 0 $ for which the related $ r_{max} $ have some intersections with that of (absolute) selfishness of $ \gamma_{1i} = 0 $ with $ \alpha_i =0 $. In other words, regarding different amounts of observability ($ \alpha_{A} \neq \alpha_{B} $), one can build other games such as a dilemma to be played as $ A( \alpha_A ) $ or $ B( \alpha_B ) $. For instance, one can count for the payoffs if $ \gamma_A = 0 $, $ \gamma_B = 0.9 $ and $ \alpha_A = 0 $ and $ \alpha_B = 1$ which leads to

$$\begin{aligned}
\begin{tabular}{|c|c|}
\hline  $p(1), p(1)$ & $p(0.1), p(0.9)$ \\
\hline  $p(0.9), p(0.1)$ & $p(0.9), p(0.9)$ \\ 
\hline
\end{tabular}
\nonumber\end{aligned}$$

\
We can see that while $ p(1)$ is the Nash equilibrium, $ p(0.9) > p(1) $ is the Pareto optimal strategy; however, $ p(0.9) $ is not a Nash equilibrium itself.

Discussion {#discussion .unnumbered}
==========

-   In our invasion example, one can distinguish between belief states and world states by adding another property such as color to them for simplicity. That is, for example, $ | s\rangle \in \{ | \Leftarrow \rangle , | \Rightarrow \rangle \} $, $ | a \rangle \in \{ | -  \rangle , | + \rangle \} $ and $ | b\rangle \in \{ | { \color{mygreen} \Leftarrow} \rangle , | {\color{mygreen} \Rightarrow} \rangle \} $ for $ N=2 $ in a partially observable invasion toy problem. More importantly, we have used a multi-agent framework for our work. However, It is important noting that the additional properties such as different colors and also multi-agent setting have not any significant rule in the theory of the partially observable of Eq. \[jankan\]. It means that additional features such as different colors and multi-agent setting are not inherent properties and do not have vital importance in the formulation of partially observable PS.

-   As a programmer point of view, one may think that we could add temporal correlations (glow mechanism [@Briegel2]) between actions down by the interpreter and actions down by the defender to recognize the world percepts little by little via delayed rewards. However, it has no sense in case the belief states are producing by another intelligent agent. Because we have two agents with two disconnected brains as their clip networks. That is, the former action clip (interpreter’s action) happens on a separate network where the latter action clip (defender’s action) occurs in the defender’s brain. Therefore we must prevent any connection between the two detached network of clips related to two agents in a multi-agent setting.

Conclusion
==========

In this paper, we add two new concepts of the *projection operator* and the *observability parameter* to the original scheme of Projective simulation (PS) for the performance of an agent in a partially observable environment. A given projector makes some belief states from the world states to the extent of the observability parameter of a given environment. We used our method in a multi-agent invasion example where the projector is a second agent called an interpreter. In this example, the fully observable percepts are coming directly from the attacker while the partially observable percepts are conveying indirectly through the interpreter. Then, we continued our case example to include the game theoretical aspects of a 2-agent toy problem.

It is worth noting that, the projective simulation model is applying in quantum artificial intelligence likewise in the classical AI. Our partially observable PS formulation, on the other hand, founded on Dirac notation as it is utilized both in the theoretical quantum mechanic as well as in the artificial intelligence. Thereby, we hope that our partially observable method can be used uncomplicatedly in the quantum AI problems such as quantum machine learning, quantum multi-agent systems and game theory, quantum random walk and quantum neural networks.

Nevertheless, In my opinion, PS model and its partially observable method can be considered widely in psychology or behavioural economics too. Owing to the existence of the difference between the fictitious memory clips of $\textcircled{\textit{s}}$, $\textcircled{\textit{a}}$ with their actual counterparts ($s, \, a$), on the other hand, PS is a compelling context for understanding the various decision-making processes among artificial people in a certain condition via different perceptions on a specific situation. For example, merging some memory clips in some compositional memory may be applied to build an abstract clip network which can form a creative thinking or an illusion that is related to one type of personality as “openness to new experiences” in the five-factor model which we want to consider as another study.

Further, the partially observable PS might be utilized in psychology by itself. Because a projector may not be just an exterior interpreter, but it can also be considered as an interior brain structure of a given agent. For instance, a brain can have some projections from the childhood, and as a result, it affects on the perception of a given situation and thereby it affects on the performance or the decision-making. Consequently, an internal projection as a part of the whole other technical methods might make some subconscious notion in the individuals or even different cultures in the societies via some more general interior-exterior projectors.

Finally, as we saw, the transparency was dependent on three parameters, the environment parameter $ \alpha $, the agent parameter $ \gamma_{pa} $ and the belief states raised by another agent for example $ \gamma_{I} $. In a general perspective, on the other hand, the transparency or a given perception can be dependent on the environment, the agent itself and its society.

I thank Alexey Melnikov for long-term discussions. I think the final version could not have been completed without any discussion.

Appendix 1: Actions versus the probability of actions {#averagefequa .unnumbered}
=====================================================

While in the original papers of PS, averaged performing rewarded actions are depicted for the efficiency, we used the probability of doing rewarded actions $ r^{(t)} $ for the same purpose.

It is due to the fact that the probability of doing an action in a large time step $ t $, as in Fig. \[figappendix11\], is a better approximation for $ r^{t \to \infty} $ than the real actions in $ t $.

Briegel, H. J., & Cuevas, G. D. (2012). Projective simulation for artificial intelligence. *Scientific Reports*, 2, 400. Mautner, J., Makmal, A., Manzano, D., Tiersch, M., & Briegel, H. J. (2015). Projective simulation for classical learning agents: A comprehensive investigation. *New Generat. Comput.*, 33(1), 69-114. Makmal, A., Melnikov, A. A., Dunjko, V., & Briegel, H. J. (2016). Meta-learning within Projective Simulation. *IEEE Access*, 4, 2110-2122. Melnikov , A. A., Makmal , A., Dunjko, V., & Briegel , H. J. (2017). Projective simulation with generalization.*Scientific Reports*, 7, 14430. Watkins, C. J. C. H., & Dayan, P. (1992). Q-learning. *Machine Learning*, 8, 279-292. Verbeeck, K., Nowé, A., Parent, J., & Tuyls, K. (2007). Exploring selfish reinforcement learning in repeated games with stochastic rewards. *Journal of Autonomous Agents and Multi-Agent Systems*, 14, 239-269. This figure is produced by *Anna Antinori* whom we have gotten permission to be re-used in our paper. The figure is depicted in the following website. https://theconversation.com/people-with-creative-personalities-really-do-see-the-world-differently-77083. Russel, S. J., & Norvig, P. (2010). *Artifical intelligence - A modern approach*. New Jersey: Prentice Hall. Kaelbling, L. P., Littman, M. L., & Cassandra, A. R. (1998). Planning and acting in partially observable stochastic domains. *Artificial intelligence*, 101(1-2), 99-134. Ceren, R., Doshi, P., & Banerjee, B. (2016). Reinforcement Learning in Partially Observable Multiagent Settings: Monte Carlo Exploring Policies with PAC Bounds. In *Proceedings of the 15th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2016)*, 530-538. Ishii, S., Fujita, H., Mitsutake, M., Yamazaki, T., Matsuda, J., & Matsuno, Y. (2005). A Reinforcement Learning Scheme for a Partially-Observable Multi-Agent Game. *Machine Learning*, 59, 31-54. Sigaud, O., & Buffet, O. (2010). *Markov Decision Processes in Artificial Intelligence*. New York: Wiley. Panella, A., & Gmytrasiewicz, P. (2017). Interactive POMDPs with finite-state models of other agents. *Autonomous Agents and Multi-Agent Systems*, 31(4), 861–904. Gmytrasiewicz, P., & Doshi, P. (2005). A Framework for Sequential Planning in Multi-Agent Settings. *Journal of Artificial Intelligence Research*, 24, 49-79. Doshi, P., Gmytrasiewicz, P. (2006). On the Difficulty of Achieving Equilibrium in Interactive POMDPs. In *Proceedings of the 21st national conference on artificial intelligence, vol. 2, AAAI’06*, (pp. 1131-1136). AAAI Press. Bernstein, D. S., Givan, R., Immerman, N., & Zilberstein, S. (2002). The complexity of decentralized control of Markov decision processes. *Mathematics of Operations Research*, 27(4) , 819-840. Oliehoek, F. A. (2012). Decentralized POMDPs. In *Reinforcement Learning: State of the Art, Adaptation, Learning, and Optimization* (pp. 471-503). Berlin: Springer Berlin Heidelberg. Amato, C., Chowdhary, G., Geramifard, A., Ure, N. K., & Kochenderfer, M. J. (2013). Decentralized control of partially observable Markov decision processes. In *Proc. of the 52nd IEEE Conf. on Decision and Control*. Sakurai, J. J. (1994). *Modern Quantum Mechanics, Revised edn*. Reading: Addison Wesley. Shoham, Y., & Leyton-Brown, K. (2009). *Multiagent systems: algorithmic, game-theoretic, and logical foundations* (p. xiii). Cambridge: Cambridge University Press. Paparo, G. D., Dunjko, V., Makmal, A., Martin-Delgado, M. A., & Briegel, H. J. (2014). Quantum Speedup for Active Learning Agents. *Physical Review X*, 4, 031002. Clausen, J., & Briegel, H. J. (2018). Quantum machine learning with glow for episodic tasks and decision games. *Physical Review A*, 97, 022303. Tiersch, M., Ganahl, E., J., & Briegel, H. J. (2015). Adaptive quantum computation in changing environments using projective simulation. *Sci. Rep.*, 5, 12874. Melnikov, A., Nautrup, H. P., Krenn, M., Dunjko, V., Tiersch, M., Zeilinger, A., & Briegel, H. J. (2018). Active learning machine learns to create new quantum experiments. *Proceedings of the National Academy of Sciences*, 201714936. Dunjko, V., & Briegel, H. J. (2018). Machine learning & artificial intelligence in the quantum domain: a review of recent progress. *Reports on Progress in Physics*.

[^1]: I may think of also annihilation of old useless clips or decaying some portion of the network (clips or edges) as a result of a sort of neurological disorder.

[^2]: The probabilities in projective simulation change fractionally as in fictitious play models and the role of forgetting (dissipation) factor in the projective simulation can be indirectly compared with the task of the discounted reward or learning factor in Q-learning [@WatkinsDayan], or the step size parameter in the linear update scheme of reinforcement learning [@Verbeeck].

[^3]: Though there could be other alternative functions like an exponential function as mentioned in [@Alexey1] similar to that of original reinforcement learning.

[^4]: In the original works, the authors have plotted their figures for action blocking, whereas mine have been planned in respect to the probability of action blocking using the fact that the action blocking averaged on an infinite number of actions would ultimately be equal to the probability of doing an action, for more details see Appendix \[averagefequa\].

[^5]: Although Fig. \[alpanim\] is depicted just for $ \alpha = 0.5 $, this consequence is true for every $ \alpha = constant $ that can be tracked in Fig. \[alpamore\].
