A Survey of Decision-Theoretic Models for Cognitive Internet of Things (CIoT)
---
paper_title: Opportunistic Spectrum Access in Cognitive Radio Networks: Global Optimization Using Local Interaction Games

paper_content:

We investigate the problem of achieving global optimization for distributed channel selections in cognitive radio networks (CRNs), using game theoretic solutions. To cope with the lack of centralized control and local influences, we propose two special cases of local interaction game to study this problem. The first is local altruistic game, in which each user considers the payoffs of itself as well as its neighbors rather than considering itself only. The second is local congestion game, in which each user minimizes the number of competing neighbors. It is shown that with the proposed games, global optimization is achieved with local information. Specifically, the local altruistic game maximizes the network throughput and the local congestion game minimizes the network collision level. Also, the concurrent spatial adaptive play (C-SAP), which is an extension of the existing spatial adaptive play (SAP), is proposed to achieve the global optimum both autonomously as well as rapidly.
---
paper_title: Learning-Based QoS Control Algorithms for Next Generation Internet of Things

paper_content:

The Internet has become an evolving entity, growing in importance and creating new value through its expansion and added utilization. The Internet of Things (IoT) is a new concept associated with the future Internet and has recently become popular in a dynamic and global network infrastructure. However, in an IoT implementation, it is difficult to satisfy different Quality of Service (QoS) requirements and achieve rapid service composition and deployment. In this paper, we propose a new QoS control scheme for IoT systems. Based on the Markov game model, the proposed scheme can effectively allocate IoT resources while maximizing system performance. In multiagent environments, a game theory approach can provide an effective decision-making framework for resource allocation problems. To verify the results of our study, we perform a simulation and confirm that the proposed scheme can achieve considerably improved system performance compared to existing schemes.
---
paper_title: Game Theory: Analysis of Conflict

paper_content:

Preface 1. Decision-Theoretic Foundations 1.1 Game Theory, Rationality, and Intelligence 1.2 Basic Concepts of Decision Theory 1.3 Axioms 1.4 The Expected-Utility Maximization Theorem 1.5 Equivalent Representations 1.6 Bayesian Conditional-Probability Systems 1.7 Limitations of the Bayesian Model 1.8 Domination 1.9 Proofs of the Domination Theorems Exercises 2. Basic Models 2.1 Games in Extensive Form 2.2 Strategic Form and the Normal Representation 2.3 Equivalence of Strategic-Form Games 2.4 Reduced Normal Representations 2.5 Elimination of Dominated Strategies 2.6 Multiagent Representations 2.7 Common Knowledge 2.8 Bayesian Games 2.9 Modeling Games with Incomplete Information Exercises 3. Equilibria of Strategic-Form Games 3.1 Domination and Ratonalizability 3.2 Nash Equilibrium 3.3 Computing Nash Equilibria 3.4 Significance of Nash Equilibria 3.5 The Focal-Point Effect 3.6 The Decision-Analytic Approach to Games 3.7 Evolution. Resistance. and Risk Dominance 3.8 Two-Person Zero-Sum Games 3.9 Bayesian Equilibria 3.10 Purification of Randomized Strategies in Equilibria 3.11 Auctions 3.12 Proof of Existence of Equilibrium 3.13 Infinite Strategy Sets Exercises 4. Sequential Equilibria of Extensive-Form Games 4.1 Mixed Strategies and Behavioral Strategies 4.2 Equilibria in Behavioral Strategies 4.3 Sequential Rationality at Information States with Positive Probability 4.4 Consistent Beliefs and Sequential Rationality at All Information States 4.5 Computing Sequential Equilibria 4.6 Subgame-Perfect Equilibria 4.7 Games with Perfect Information 4.8 Adding Chance Events with Small Probability 4.9 Forward Induction 4.10 Voting and Binary Agendas 4.11 Technical Proofs Exercises 5. Refinements of Equilibrium in Strategic Form 5.1 Introduction 5.2 Perfect Equilibria 5.3 Existence of Perfect and Sequential Equilibria 5.4 Proper Equilibria 5.5 Persistent Equilibria 5.6 Stable Sets 01 Equilibria 5.7 Generic Properties 5.8 Conclusions Exercises 6. Games with Communication 6.1 Contracts and Correlated Strategies 6.2 Correlated Equilibria 6.3 Bayesian Games with Communication 6.4 Bayesian Collective-Choice Problems and Bayesian Bargaining Problems 6.5 Trading Problems with Linear Utility 6.6 General Participation Constraints for Bayesian Games with Contracts 6.7 Sender-Receiver Games 6.8 Acceptable and Predominant Correlated Equilibria 6.9 Communication in Extensive-Form and Multistage Games Exercises Bibliographic Note 7. Repeated Games 7.1 The Repeated Prisoners Dilemma 7.2 A General Model of Repeated Garnet 7.3 Stationary Equilibria of Repeated Games with Complete State Information and Discounting 7.4 Repeated Games with Standard Information: Examples 7.5 General Feasibility Theorems for Standard Repeated Games 7.6 Finitely Repeated Games and the Role of Initial Doubt 7.7 Imperfect Observability of Moves 7.8 Repeated Wines in Large Decentralized Groups 7.9 Repeated Games with Incomplete Information 7.10 Continuous Time 7.11 Evolutionary Simulation of Repeated Games Exercises 8. Bargaining and Cooperation in Two-Person Games 8.1 Noncooperative Foundations of Cooperative Game Theory 8.2 Two-Person Bargaining Problems and the Nash Bargaining Solution 8.3 Interpersonal Comparisons of Weighted Utility 8.4 Transferable Utility 8.5 Rational Threats 8.6 Other Bargaining Solutions 8.7 An Alternating-Offer Bargaining Game 8.8 An Alternating-Offer Game with Incomplete Information 8.9 A Discrete Alternating-Offer Game 8.10 Renegotiation Exercises 9. Coalitions in Cooperative Games 9.1 Introduction to Coalitional Analysis 9.2 Characteristic Functions with Transferable Utility 9.3 The Core 9.4 The Shapkey Value 9.5 Values with Cooperation Structures 9.6 Other Solution Concepts 9.7 Colational Games with Nontransferable Utility 9.8 Cores without Transferable Utility 9.9 Values without Transferable Utility Exercises Bibliographic Note 10. Cooperation under Uncertainty 10.1 Introduction 10.2 Concepts of Efficiency 10.3 An Example 10.4 Ex Post Inefficiency and Subsequent Oilers 10.5 Computing Incentive-Efficient Mechanisms 10.6 Inscrutability and Durability 10.7 Mechanism Selection by an Informed Principal 10.8 Neutral Bargaining Solutions 10.9 Dynamic Matching Processes with Incomplete Information Exercises Bibliography Index
---
paper_title: A Game Theoretic Perspective on Self-organizing Optimization for Cognitive Small Cells

paper_content:

In this article, we investigate self-organizing optimization for cognitive small cells (CSCs), which have the ability to sense the environment, learn from historical information, make intelligent decisions, and adjust their operational parameters. By exploring the inherent features, some fundamental challenges for self-organizing optimization in CSCs are presented and discussed. Specifically, the dense and random deployment of CSCs brings about some new challenges in terms of scalability and adaptation; furthermore, the uncertain, dynamic, and incomplete information constraints also impose some new challenges in terms of convergence and robustness. For providing better service to users and improving resource utilization, four requirements for self-organizing optimization in CSCs are presented and discussed. Following the attractive fact that the decisions in game-theoretic models are exactly coincident with those in self-organizing optimization (i.e., distributed and autonomous), we establish a framework of gametheoretic solutions for self-organizing optimization in CSCs and propose some featured game models. Specifically, their basic models are presented, some examples are discussed, and future research directions are given.
---
paper_title: Decentralized Learning of Nash Equilibria in Multi-Person Stochastic Games With Incomplete Information

paper_content:

A multi-person discrete game where the payoff after each play is stochastic is considered. The distribution of the random payoff is unknown to the players and further none of the players know the strategies or the actual moves of other players. A learning algorithm for the game based on a decentralized team of learning automata is presented. It is proved that all stable stationary points of the algorithm are Nash equilibria for the game. Two special cases of the game are also discussed, namely, game with common payoff and the relaxation labelling problem. The former has applications such as pattern recognition and the latter is a problem widely studied in computer vision. For the two special cases it is shown that the algorithm always converges to a desirable solution. >
---
paper_title: Markov Decision Processes with Applications in Wireless Sensor Networks: A Survey

paper_content:

Wireless sensor networks (WSNs) consist of autonomous and resource-limited devices. The devices cooperate to monitor one or more physical phenomena within an area of interest. WSNs operate as stochastic systems because of randomness in the monitored environments. For long service time and low maintenance cost, WSNs require adaptive and robust methods to address data exchange, topology formulation, resource and power optimization, sensing coverage and object detection, and security challenges. In these problems, sensor nodes are used to make optimized decisions from a set of accessible strategies to achieve design goals. This survey reviews numerous applications of the Markov decision process (MDP) framework, a powerful decision-making tool to develop adaptive algorithms and protocols for WSNs. Furthermore, various solution methods are discussed and compared to serve as a guide for using MDPs in WSNs.
---
paper_title: A Roadmap to the Programmable World: Software Challenges in the IoT Era

paper_content:

The Internet of Things (IoT) represents the next significant step in the evolution of the Internet and software development. Although most IoT research focuses on data acquisition, analytics, and visualization, a subtler but equally important transition is underway. Hardware advances are making it possible to embed fully fledged virtual machines and dynamic language runtimes virtually everywhere, leading to a Programmable World in which all our everyday things are connected and programmable dynamically. The emergence of millions of remotely programmable devices in our surroundings will pose significant software development challenges. A roadmap from today's cloud-centric, data-centric IoT systems to the Programmable World highlights the technical challenges that deserve to be part of developer education and deserve deeper investigation beyond those IoT topics that receive the most attention today.
---
paper_title: Decision-Theoretic Distributed Channel Selection for Opportunistic Spectrum Access: Strategies, Challenges and Solutions

paper_content:

Opportunistic spectrum access (OSA) has been regarded as the most promising approach to solve the paradox between spectrum scarcity and waste. Intelligent decision making is key to OSA and differentiates it from previous wireless technologies. In this article, a survey of decision-theoretic solutions for channel selection and access strategies for OSA system is presented. We analyze the challenges facing OSA systems globally, which mainly include interactions among multiple users, dynamic spectrum opportunity, tradeoff between sequential sensing cost and expected reward, and tradeoff between exploitation and exploration in the absence of prior statistical information. We provide comprehensive review and comparison of each kind of existing decision-theoretic solution, i.e., game models, Markovian decision process, optimal stopping problem and multi-armed bandit problem. We analyze their strengths and limitations and outline further research for both technical contents and methodologies. In particular, these solutions are critically analyzed in terms of information, cost and convergence speed, which are key concerns for practical implementation. Moreover, it is noted that each kind of existing decision-theoretic solution mainly addresses one aspect of the challenges, which implies that two or more kinds of decision-theoretic solutions should be incorporated to address more challenges simultaneously.
---
paper_title: Intelligent Planning for Developing Mobile IoT Applications Using Cloud Systems

paper_content:

IoT (Internet of Things) is increasingly becoming more popular mainly due to the fact that almost all the smart devices nowadays are network enabled to facilitate many current and emerging applications. However, some important issues still need to be addressed before fully realizing the potential of IoT applications. One of the most important issues is to have effective approaches to planning various device actions to satisfy user requirements efficiently and securely in mobile IoT applications. A mobile IoT application can be composed of mobile cloud systems and devices, such as wearable devices, smart phones and smart cars. In this type of systems, mobile networks with elastic resources from various mobile clouds are effective to support IoT applications. In this paper an effective approach to intelligent planning for mobile IoT applications is presented. This approach includes a learning technique for dynamically assessing the users' mobile IoT application and a MDP (Markov Decision Process) planning technique for enhancing efficiency of IoT device action planning. Simulation results are presented to show the effectiveness of our approach.
---
paper_title: A Game Theoretic Approach for an IoT-Based Automated Employee Performance Evaluation

paper_content:

In the present scenario, performance evaluation of employees in industries is done manually, in which there are ample chances of biases. It is observed that manual employee evaluation systems can be efficiently eliminated by using ubiquitous sensing capabilities of Internet of things (IoT) devices to monitor industrial employees. However, none of the authors have used IoT data for automating performance evaluation systems of employees. Hence, this paper proposes a game theoretic approach for an IoT-based employee performance evaluation in industry. The system infers useful results about the performance of employees by mining data collected by the sensory nodes using the MapReduce model. The information hence obtained is then used to draw automated decisions for employees using game theory. The system is analyzed both experimentally and mathematically. The experimental evaluation compares the proposed system with other techniques of data mining and decision making. The results depict that the proposed system evaluates the performance of employees efficiently and shows a performance improvement over other techniques. The mathematical evaluation shows that correct evaluation of employees by the system effectively motivates employees in favor of the industry. Thus, the proposed system effectively and efficiently automates the employee evaluation system and decision-making process in the industry.
---
paper_title: SMDP-Based Radio Resource Allocation Scheme in Software-Defined Internet of Things Networks

paper_content:

With rapid development of the Internet of Things (IoT), various machine-to-machine communications technologies have emerged in recent years to provide ubiquitous wireless connections for a massive number of IoT devices. This poses significant challenges to network control and management of large-scale IoT networks. Software-defined networking (SDN) is considered a promising technology to streamline network management due to dynamic reconfigurable network elements. Thus, the integration of SDN and IoT provides a potentially feasible solution to strengthening management and control capabilities of the IoT network. Benefit from the SDN technology, resource utilization in the IoT network can be further enhanced. In this paper, we first propose a software-defined network architecture for IoT. Then, the resource allocation problem in the proposed SDN-based IoT network is investigated. The optimal problem of maximizing the expected average rewards of the network is formulated as a semi-Markov decision process (SMDP). The optimal solution is obtained through solving the SMDP problem using a relative value iteration algorithm. Simulation results demonstrate that the proposed resource allocation scheme is able to improve the system rewards compared with other comparative resource allocation schemes.
---
paper_title: R-learning-based team game model for Internet of things quality-of-service control scheme

paper_content:

In modern times, it has been observed that Internet of things technology makes it possible for connecting various smart objects together through the Internet. For the effective Internet of things m...
---
paper_title: Enabling smart cities through a cognitive management framework for the internet of things

paper_content:

The Internet of Things (IoT) is expected to substantially support sustainable development of future smart cities. This article identifies the main issues that may prevent IoT from playing this crucial role, such as the heterogeneity among connected objects and the unreliable nature of associated services. To solve these issues, a cognitive management framework for IoT is proposed, in which dynamically changing real-world objects are represented in a virtualized environment, and where cognition and proximity are used to select the most relevant objects for the purpose of an application in an intelligent and autonomic way. Part of the framework is instantiated in terms of building blocks and demonstrated through a smart city scenario that horizontally spans several application domains. This preliminary proof of concept reveals the high potential that self-reconfigurable IoT can achieve in the context of smart cities.
---
paper_title: Cognitive Management for the Internet of Things: A Framework for Enabling Autonomous Applications

paper_content:

The handling of the number of objects that will be part of the Internet of Things (IoT) and the various networking technologies used for their interconnection requires suitable architecture and technological foundations. Despite significant work on architectures and test facilities for the IoT, there is still a lack of management functionality to overcome the technological heterogeneity and complexity of the underlying networks and IoT infrastructure, so as to enhance context/situational-awareness, reliability, and energy-efficiency of IoT applications. This article presents a cognitive management framework for the IoT aiming to address these issues, comprising three levels of functionality: virtual objects (VOs), composite VOs (CVOs), and service levels. Cognitive entities at all levels provide the means for self-management (configuration, optimization, and healing) and learning. Three fundamental processes of this framework are presented: dynamic CVO creation, knowledge-based CVO instantiation, and CVO self-healing. A first prototype implementation of this framework and corresponding derived results are presented.
---
paper_title: Cognitive Machine-to-Machine Communications for Internet-of-Things: A Protocol Stack Perspective

paper_content:

Machine-to-machine (M2M) communications enables networked devices to exchange information among each other as well as with business application servers and therefore creates what is known as the Internet-of-Things (IoT). The research community has a consensus for the need of a standardized protocol stack for M2M communications. On the other hand, cognitive radio technology is very promising for M2M communications due to a number of factors. It is expected that cognitive M2M communications will be indispensable in order to realize the vision of IoT. However cognitive M2M communications requires a cognitive radio-enabled protocol stack in addition to the fundamental requirements of energy efficiency, reliability, and Internet connectivity. The main objective of this paper is to provide the state of the art in cognitive M2M communications from a protocol stack perspective. This paper covers the emerging standardization efforts and the latest developments on protocols for cognitive M2M networks. In addition, this paper also presents the authors’ recent work in this area, which includes a centralized cognitive medium access control (MAC) protocol, a distributed cognitive MAC protocol, and a specially designed routing protocol for cognitive M2M networks. These protocols explicitly account for the peculiarities of cognitive radio environments. Performance evaluation demonstrates that the proposed protocols not only ensure protection to the primary users (PUs) but also fulfil the utility requirements of the secondary M2M networks.
---
paper_title: Game Theory-Based Network Selection: Solutions and Challenges

paper_content:

In order to cater for the overwhelming growth in bandwidth demand from mobile Internet users operators have started to deploy different, overlapping radio access network technologies. One important challenge in such a heterogeneous wireless environment is to enable network selection mechanisms in order to keep the mobile users Always Best Connected (ABC) anywhere and anytime. Game theory techniques have been receiving growing attention in recent years as they can be adopted in order to model and understand competitive and cooperative scenarios between rational decision makers. This paper presents an overview of the network selection decision problem and challenges, a comprehensive classification of related game theoretic approaches and a discussion on the application of game theory to the network selection problem faced by the next generation of 4G wireless networks.
---
paper_title: Cognitive-Radio-Based Internet of Things: Applications, Architectures, Spectrum Related Functionalities, and Future Research Directions

paper_content:

Recent research and technology trends are shifting toward IoT and CRNs. However, we think that the things-oriented, Internet-oriented, and semantic-oriented versions of IoT are meaningless if IoT objects are not equipped with cognitive radio capability. Equipping IoT objects with CR capability has lead to a new research dimension of CR-based IoT. In this article, we present an overview of CR-based IoT systems. We highlight potential applications of CR-based IoT systems. We survey architectures and frameworks of CR-based IoT systems. We furthermore discuss spectrum-related functionalities for CR-based IoT systems. Finally, we present open issues, research challenges, and future direction for these CR-based IoT networks.
---
paper_title: Game Theoretic Mechanisms for Resource Management in Massive Wireless IoT Systems

paper_content:

As a result of rapid advancement in communication technologies, the Internet of Things (i.e., ubiquitous connectivity among a very large number of persons and physical objects) is now becoming a reality. Nonetheless, a variety of challenges remain to be addressed, one of them being the efficient resource management in IoT. On one hand, central resource allocation is infeasible for large numbers of entities, due to excessive computational cost as well as immoderate overhead required for information acquisition. On the other hand, the devices connecting to IoT are expected to act smart, making decisions and performing tasks without human intervention. These characteristics render distributed resource management an essential feature of future IoT. Traditionally, game theory is applied to effectively analyze the interactive decision making of agents with conflicting interests. Nevertheless, conventional game models are not adequate to model large-scale systems, since they suffer from many shortcomings including analytical complexity, slow convergence, and excessive overhead due to information acquisition/exchange. In this article, we explore some non-conventional game theoretic models that fit the inherent characteristics of future large-scale IoT systems. Specifically, we discuss evolutionary games, mean field games, minority games, mean field bandit games, and mean field auctions. We provide the basics of each of these game models and discuss the potential IoT-related resource management problems that can be solved by using these models. We also discuss challenges, pitfalls, and future research directions.
---
paper_title: Game theoretic approach on Real‐time decision making for IoT‐based traffic light control

paper_content:

Summary ::: Smart traffic light control at intersections is 1 of the major issues in Intelligent Transportation System. In this paper, on the basis of the new emerging technologies of Internet of Things, we introduce a new approach for smart traffic light control at intersection. In particular, we firstly propose a connected intersection system where every objects such as vehicles, sensors, and traffic lights will be connected and sharing information to one another. By this way, the controller is able to collect effectively and mobility traffic flow at intersection in real-time. Secondly, we propose the optimization algorithms for traffic lights by applying algorithmic game theory. Specially, 2 game models (which are Cournot Model and Stackelberg Model) are proposed to deal with difference scenarios of traffic flow. In this regard, based on the density of vehicles, controller will make real-time decisions for the time durations of traffic lights to optimize traffic flow. To evaluate our approach, we have used Netlogo simulator, an agent-based modeling environment for designing and implementing a simple working traffic. The simulation results shows that our approach achieves potential performance with various situations of traffic flow.
---
paper_title: Game theoretic decision making in IoT-assisted activity monitoring of defence personnel

paper_content:

Innovative growth of IoT Technology has enhanced the service delivery aspects of defence sector in terms of high-tech surveillance, and reliable defence mechanisms. Along with the sensing capability for ubiquitous events, IoT Technology provides means to deliver services in time sensitive and information intensive manner. In this paper, a framework for IoT based activity monitoring of defence personnel is presented to detect the precursors of suspiciousness in terms of information outflow that can compromise the national security. Though maintaining intellectual defence personnel remained a major area of concern for every nation, still investigating reports of recent terrorist attacks in different countries have discovered the number of suspicion factors from their daily activities. The work presented in this study focuses on these factors in terms of efficient monitoring of social activities and analyzing it over suspicious scale. Moreover, Suspicious Index (SI) is defined for every personnel on the basis of their activities that can compromise national security directly or indirectly. Furthermore, automated game theoretic decision making model is presented to aid the monitoring officials in suppressing the probability of information outflow. In order to validate the system, two types of evaluations are performed. In one case, an imitative environment is considered to monitor 10 college students’ daily engagements for 7 days. The results are compared with the state-of-the-art techniques of data assessment. In the second case, a mathematical evaluation for the game theoretic decision making is performed. Results in both cases show that the proposed model achieves better performance in efficient monitoring of suspicious activities and effective decision making.
---
paper_title: Cognitive Internet of Things: A New Paradigm Beyond Connection

paper_content:

Current research on Internet of Things (IoT) mainly focuses on how to enable general objects to see, hear, and smell the physical world for themselves, and make them connected to share the observations. In this paper, we argue that only connected is not enough, beyond that, general objects should have the capability to learn, think, and understand both physical and social worlds by themselves. This practical need impels us to develop a new paradigm, named cognitive Internet of Things (CIoT), to empower the current IoT with a “brain” for high-level intelligence. Specifically, we first present a comprehensive definition for CIoT, primarily inspired by the effectiveness of human cognition. Then, we propose an operational framework of CIoT, which mainly characterizes the interactions among five fundamental cognitive tasks: perception-action cycle, massive data analytics, semantic derivation and knowledge discovery, intelligent decision-making, and on-demand service provisioning. Furthermore, we provide a systematic tutorial on key enabling techniques involved in the cognitive tasks. In addition, we also discuss the design of proper performance metrics on evaluating the enabling techniques. Last but not the least, we present the research challenges and open issues ahead. Building on the present work and potentially fruitful future studies, CIoT has the capability to bridge the physical world (with objects, resources, etc.) and the social world (with human demand, social behavior, etc.), and enhance smart resource allocation, automatic network operation, and intelligent service provisioning.
---
paper_title: Markov Decision Processes with Applications in Wireless Sensor Networks: A Survey

paper_content:

Wireless sensor networks (WSNs) consist of autonomous and resource-limited devices. The devices cooperate to monitor one or more physical phenomena within an area of interest. WSNs operate as stochastic systems because of randomness in the monitored environments. For long service time and low maintenance cost, WSNs require adaptive and robust methods to address data exchange, topology formulation, resource and power optimization, sensing coverage and object detection, and security challenges. In these problems, sensor nodes are used to make optimized decisions from a set of accessible strategies to achieve design goals. This survey reviews numerous applications of the Markov decision process (MDP) framework, a powerful decision-making tool to develop adaptive algorithms and protocols for WSNs. Furthermore, various solution methods are discussed and compared to serve as a guide for using MDPs in WSNs.
---
paper_title: Decision-Theoretic Distributed Channel Selection for Opportunistic Spectrum Access: Strategies, Challenges and Solutions

paper_content:

Opportunistic spectrum access (OSA) has been regarded as the most promising approach to solve the paradox between spectrum scarcity and waste. Intelligent decision making is key to OSA and differentiates it from previous wireless technologies. In this article, a survey of decision-theoretic solutions for channel selection and access strategies for OSA system is presented. We analyze the challenges facing OSA systems globally, which mainly include interactions among multiple users, dynamic spectrum opportunity, tradeoff between sequential sensing cost and expected reward, and tradeoff between exploitation and exploration in the absence of prior statistical information. We provide comprehensive review and comparison of each kind of existing decision-theoretic solution, i.e., game models, Markovian decision process, optimal stopping problem and multi-armed bandit problem. We analyze their strengths and limitations and outline further research for both technical contents and methodologies. In particular, these solutions are critically analyzed in terms of information, cost and convergence speed, which are key concerns for practical implementation. Moreover, it is noted that each kind of existing decision-theoretic solution mainly addresses one aspect of the challenges, which implies that two or more kinds of decision-theoretic solutions should be incorporated to address more challenges simultaneously.
---
paper_title: A Game Theoretic Approach for an IoT-Based Automated Employee Performance Evaluation

paper_content:

In the present scenario, performance evaluation of employees in industries is done manually, in which there are ample chances of biases. It is observed that manual employee evaluation systems can be efficiently eliminated by using ubiquitous sensing capabilities of Internet of things (IoT) devices to monitor industrial employees. However, none of the authors have used IoT data for automating performance evaluation systems of employees. Hence, this paper proposes a game theoretic approach for an IoT-based employee performance evaluation in industry. The system infers useful results about the performance of employees by mining data collected by the sensory nodes using the MapReduce model. The information hence obtained is then used to draw automated decisions for employees using game theory. The system is analyzed both experimentally and mathematically. The experimental evaluation compares the proposed system with other techniques of data mining and decision making. The results depict that the proposed system evaluates the performance of employees efficiently and shows a performance improvement over other techniques. The mathematical evaluation shows that correct evaluation of employees by the system effectively motivates employees in favor of the industry. Thus, the proposed system effectively and efficiently automates the employee evaluation system and decision-making process in the industry.
---
paper_title: Evolution and the theory of games

paper_content:

The modern theory of evolutionary dynamics is founded upon the remarkable insights of R. A. Fisher and Sewall Wright and set forth in the loci classici The Genetical Theory of Natural Selection (1930) and ‘Evolution in Mendelian Populations’ (1931). By the time of the publication of Wright’s paper in 1931 all of the theory of population genetics, as it is presently understood, was established. It is a sign of the extraordinary power of these early formulations, that nothing of equal significance has been added to the theory of population genetics in the thirty years that have passed since that time. Yet we cannot take this period to mean that we now have an adequate theory of evolutionary dynamics. On the contrary, the theory of population genetics, as complete as it may be in itself, fails to deal with many problems of primary importance for an understanding of evolution.
---
paper_title: Enabling smart cities through a cognitive management framework for the internet of things

paper_content:

The Internet of Things (IoT) is expected to substantially support sustainable development of future smart cities. This article identifies the main issues that may prevent IoT from playing this crucial role, such as the heterogeneity among connected objects and the unreliable nature of associated services. To solve these issues, a cognitive management framework for IoT is proposed, in which dynamically changing real-world objects are represented in a virtualized environment, and where cognition and proximity are used to select the most relevant objects for the purpose of an application in an intelligent and autonomic way. Part of the framework is instantiated in terms of building blocks and demonstrated through a smart city scenario that horizontally spans several application domains. This preliminary proof of concept reveals the high potential that self-reconfigurable IoT can achieve in the context of smart cities.
---
paper_title: Cognitive Management for the Internet of Things: A Framework for Enabling Autonomous Applications

paper_content:

The handling of the number of objects that will be part of the Internet of Things (IoT) and the various networking technologies used for their interconnection requires suitable architecture and technological foundations. Despite significant work on architectures and test facilities for the IoT, there is still a lack of management functionality to overcome the technological heterogeneity and complexity of the underlying networks and IoT infrastructure, so as to enhance context/situational-awareness, reliability, and energy-efficiency of IoT applications. This article presents a cognitive management framework for the IoT aiming to address these issues, comprising three levels of functionality: virtual objects (VOs), composite VOs (CVOs), and service levels. Cognitive entities at all levels provide the means for self-management (configuration, optimization, and healing) and learning. Three fundamental processes of this framework are presented: dynamic CVO creation, knowledge-based CVO instantiation, and CVO self-healing. A first prototype implementation of this framework and corresponding derived results are presented.
---
paper_title: Coalition Formation Games for Collaborative Spectrum Sensing

paper_content:

Collaborative spectrum sensing (CSS) between secondary users (SUs) in cognitive networks exhibits an inherent tradeoff between minimizing the probability of missing the detection of the primary user (PU) and maintaining a reasonable false alarm probability (e.g., for maintaining good spectrum utilization). In this paper, we study the impact of this tradeoff on the network structure and the cooperative incentives of the SUs that seek to cooperate to improve their detection performance. We model the CSS problem as a nontransferable coalitional game, and we propose distributed algorithms for coalition formation (CF). First, we construct a distributed CF algorithm that allows the SUs to self-organize into disjoint coalitions while accounting for the CSS tradeoff. Then, the CF algorithm is complemented with a coalitional voting game to enable distributed CF with detection probability (CF-PD) guarantees when required by the PU. The CF-PD algorithm allows the SUs to form minimal winning coalitions (MWCs), i.e., coalitions that achieve the target detection probability with minimal costs. For both algorithms, we study and prove various properties pertaining to network structure, adaptation to mobility, and stability. Simulation results show that CF reduces the average probability of miss per SU up to 88.45%, relative to the noncooperative case, while maintaining a desired false alarm. For CF-PD, the results show that up to 87.25% of the SUs achieve the required detection probability through MWCs.
---
paper_title: Opportunistic Spectrum Access in Cognitive Radio Networks: Global Optimization Using Local Interaction Games

paper_content:

We investigate the problem of achieving global optimization for distributed channel selections in cognitive radio networks (CRNs), using game theoretic solutions. To cope with the lack of centralized control and local influences, we propose two special cases of local interaction game to study this problem. The first is local altruistic game, in which each user considers the payoffs of itself as well as its neighbors rather than considering itself only. The second is local congestion game, in which each user minimizes the number of competing neighbors. It is shown that with the proposed games, global optimization is achieved with local information. Specifically, the local altruistic game maximizes the network throughput and the local congestion game minimizes the network collision level. Also, the concurrent spatial adaptive play (C-SAP), which is an extension of the existing spatial adaptive play (SAP), is proposed to achieve the global optimum both autonomously as well as rapidly.
---
paper_title: A Game Theoretic Perspective on Self-organizing Optimization for Cognitive Small Cells

paper_content:

In this article, we investigate self-organizing optimization for cognitive small cells (CSCs), which have the ability to sense the environment, learn from historical information, make intelligent decisions, and adjust their operational parameters. By exploring the inherent features, some fundamental challenges for self-organizing optimization in CSCs are presented and discussed. Specifically, the dense and random deployment of CSCs brings about some new challenges in terms of scalability and adaptation; furthermore, the uncertain, dynamic, and incomplete information constraints also impose some new challenges in terms of convergence and robustness. For providing better service to users and improving resource utilization, four requirements for self-organizing optimization in CSCs are presented and discussed. Following the attractive fact that the decisions in game-theoretic models are exactly coincident with those in self-organizing optimization (i.e., distributed and autonomous), we establish a framework of gametheoretic solutions for self-organizing optimization in CSCs and propose some featured game models. Specifically, their basic models are presented, some examples are discussed, and future research directions are given.
---
paper_title: Decentralized Learning of Nash Equilibria in Multi-Person Stochastic Games With Incomplete Information

paper_content:

A multi-person discrete game where the payoff after each play is stochastic is considered. The distribution of the random payoff is unknown to the players and further none of the players know the strategies or the actual moves of other players. A learning algorithm for the game based on a decentralized team of learning automata is presented. It is proved that all stable stationary points of the algorithm are Nash equilibria for the game. Two special cases of the game are also discussed, namely, game with common payoff and the relaxation labelling problem. The former has applications such as pattern recognition and the latter is a problem widely studied in computer vision. For the two special cases it is shown that the algorithm always converges to a desirable solution. >
---
paper_title: Decision-Theoretic Distributed Channel Selection for Opportunistic Spectrum Access: Strategies, Challenges and Solutions

paper_content:

Opportunistic spectrum access (OSA) has been regarded as the most promising approach to solve the paradox between spectrum scarcity and waste. Intelligent decision making is key to OSA and differentiates it from previous wireless technologies. In this article, a survey of decision-theoretic solutions for channel selection and access strategies for OSA system is presented. We analyze the challenges facing OSA systems globally, which mainly include interactions among multiple users, dynamic spectrum opportunity, tradeoff between sequential sensing cost and expected reward, and tradeoff between exploitation and exploration in the absence of prior statistical information. We provide comprehensive review and comparison of each kind of existing decision-theoretic solution, i.e., game models, Markovian decision process, optimal stopping problem and multi-armed bandit problem. We analyze their strengths and limitations and outline further research for both technical contents and methodologies. In particular, these solutions are critically analyzed in terms of information, cost and convergence speed, which are key concerns for practical implementation. Moreover, it is noted that each kind of existing decision-theoretic solution mainly addresses one aspect of the challenges, which implies that two or more kinds of decision-theoretic solutions should be incorporated to address more challenges simultaneously.
---
paper_title: Learning to Compete for Resources in Wireless Stochastic Games

paper_content:

In this paper, we model the various users in a wireless network (e.g., cognitive radio network) as a collection of selfish autonomous agents that strategically interact to acquire dynamically available spectrum opportunities. Our main focus is on developing solutions for wireless users to successfully compete with each other for the limited and time-varying spectrum opportunities, given experienced dynamics in the wireless network. To analyze the interactions among users given the environment disturbance, we propose a stochastic game framework for modeling how the competition among users for spectrum opportunities evolves over time. At each stage of the stochastic game, a central spectrum moderator (CSM) auctions the available resources, and the users strategically bid for the required resources. The joint bid actions affect the resource allocation and, hence, the rewards and future strategies of all users. Based on the observed resource allocations and corresponding rewards, we propose a best-response learning algorithm that can be deployed by wireless users to improve their bidding policy at each stage. The simulation results show that by deploying the proposed best-response learning algorithm, the wireless users can significantly improve their own bidding strategies and, hence, their performance in terms of both the application quality and the incurred cost for the used resources.
---
paper_title: Game Theory-Based Network Selection: Solutions and Challenges

paper_content:

In order to cater for the overwhelming growth in bandwidth demand from mobile Internet users operators have started to deploy different, overlapping radio access network technologies. One important challenge in such a heterogeneous wireless environment is to enable network selection mechanisms in order to keep the mobile users Always Best Connected (ABC) anywhere and anytime. Game theory techniques have been receiving growing attention in recent years as they can be adopted in order to model and understand competitive and cooperative scenarios between rational decision makers. This paper presents an overview of the network selection decision problem and challenges, a comprehensive classification of related game theoretic approaches and a discussion on the application of game theory to the network selection problem faced by the next generation of 4G wireless networks.
---
paper_title: Opportunistic Spectrum Access in Cognitive Radio Networks: Global Optimization Using Local Interaction Games

paper_content:

We investigate the problem of achieving global optimization for distributed channel selections in cognitive radio networks (CRNs), using game theoretic solutions. To cope with the lack of centralized control and local influences, we propose two special cases of local interaction game to study this problem. The first is local altruistic game, in which each user considers the payoffs of itself as well as its neighbors rather than considering itself only. The second is local congestion game, in which each user minimizes the number of competing neighbors. It is shown that with the proposed games, global optimization is achieved with local information. Specifically, the local altruistic game maximizes the network throughput and the local congestion game minimizes the network collision level. Also, the concurrent spatial adaptive play (C-SAP), which is an extension of the existing spatial adaptive play (SAP), is proposed to achieve the global optimum both autonomously as well as rapidly.
---
paper_title: Convergence to equilibrium in local interaction games

paper_content:

We study a simple game-theoretic model for the spread of an innovation in a network. The diffiusion of the innovation is modeled as the dynamics of a coordination game in which the adoption of a common strategy between players has a higher payoff. Classical results in game theory provide a simple condition for the innovation to spread through the network. The present paper characterizes the rate of convergence as a function of graph structure. In particular, we derive a dichotomy between well-connected (e.g. random) graphs that show slow convergence and poorly connected, low dimensional graphs that show fast convergence.
---
paper_title: Spectrum Sharing as Spatial Congestion Games

paper_content:

In this paper, we present and analyze the properties of a new class of games - the spatial congestion game (SCG), which is a generalization of the classical congestion game (CG). In a classical congestion game, multiple users share the same set of resources and a user's payoff for using any resource is a function of the total number of users sharing it. As a potential game, this game enjoys some very appealing properties, including the existence of a pure strategy Nash equilibrium (NE) and that every improvement path is finite and leads to such a NE (also called the finite improvement property or FIP). While it's tempting to use this model to study spectrum sharing, it does not capture the spatial reuse feature of wireless communication, where resources (interpreted as channels) may be reused without increasing congestion provided that users are located far away from each other. This motivates us to study an extended form of the congestion game where a user's payoff for using a resource is a function of the number of its interfering users sharing it. This naturally results in a spatial congestion game (SCG), where users are placed over a network (or a conflict graph). We study fundamental properties of a spatial congestion game; in particular, we seek to answer under what conditions this game possesses the finite improvement property or a Nash equilibrium. We also discuss the implications of these results when applied to wireless spectrum sharing.
---
paper_title: Game Theoretic Channel Selection for Opportunistic Spectrum Access with Unknown Prior Information

paper_content:

The issue of distributed channel selection in opportunistic spectrum access is investigated in this paper. We consider a practical scenario where the channel availability statistics and the number of competing secondary users are unknown to the secondary users. Furthermore, there is no information exchange between secondary users. We formulate the problem of distributed channel selection as a static non-cooperative game. Since there is no prior information about the licensed channels and there is no information exchange between secondary users, existing approaches are unfeasible in our proposed game model. We then propose a learning automata based distributed channel selection algorithm, which does not explicitly learn the channel availability statistics and the number of competing secondary users but learns proper actions for secondary users, to solve the proposed channel selection game. The convergence towards Nash equilibrium with respect to the proposed algorithm also has been investigated.
---
paper_title: Opportunistic Spectrum Access in Cognitive Radio Networks: Global Optimization Using Local Interaction Games

paper_content:

We investigate the problem of achieving global optimization for distributed channel selections in cognitive radio networks (CRNs), using game theoretic solutions. To cope with the lack of centralized control and local influences, we propose two special cases of local interaction game to study this problem. The first is local altruistic game, in which each user considers the payoffs of itself as well as its neighbors rather than considering itself only. The second is local congestion game, in which each user minimizes the number of competing neighbors. It is shown that with the proposed games, global optimization is achieved with local information. Specifically, the local altruistic game maximizes the network throughput and the local congestion game minimizes the network collision level. Also, the concurrent spatial adaptive play (C-SAP), which is an extension of the existing spatial adaptive play (SAP), is proposed to achieve the global optimum both autonomously as well as rapidly.
---
paper_title: Distributed Learning in Secondary Spectrum Sharing Graphical Game

paper_content:

Secondary users sharing primary users' spectrum is modeled as a graphical game. Users located in random graphs and a regular lattice are considered. Secondary users are assumed to differentiate the ``quality" of the primary spectrum while interacting within their local neighborhood to minimize interference and congestion. The learning algorithm is also shown to be effective in punishing malicious users that violate spectrum etiquettes. An equivalence between spectrum sharing neighborhood interaction and the spin-glass model in statistical physics is established. A distributed exponential learning algorithm is used to arrive at an evolutionary stable solution to the game. Some theoretical properties of the system are studied and simulation results are presented to illustrate price of anarchy, convergence of the learning algorithm and asymptotic invariance of the system performance with respect to spectrum quality.
---
paper_title: Markov Decision Processes with Applications in Wireless Sensor Networks: A Survey

paper_content:

Wireless sensor networks (WSNs) consist of autonomous and resource-limited devices. The devices cooperate to monitor one or more physical phenomena within an area of interest. WSNs operate as stochastic systems because of randomness in the monitored environments. For long service time and low maintenance cost, WSNs require adaptive and robust methods to address data exchange, topology formulation, resource and power optimization, sensing coverage and object detection, and security challenges. In these problems, sensor nodes are used to make optimized decisions from a set of accessible strategies to achieve design goals. This survey reviews numerous applications of the Markov decision process (MDP) framework, a powerful decision-making tool to develop adaptive algorithms and protocols for WSNs. Furthermore, various solution methods are discussed and compared to serve as a guide for using MDPs in WSNs.
---
paper_title: Evolution and the theory of games

paper_content:

The modern theory of evolutionary dynamics is founded upon the remarkable insights of R. A. Fisher and Sewall Wright and set forth in the loci classici The Genetical Theory of Natural Selection (1930) and ‘Evolution in Mendelian Populations’ (1931). By the time of the publication of Wright’s paper in 1931 all of the theory of population genetics, as it is presently understood, was established. It is a sign of the extraordinary power of these early formulations, that nothing of equal significance has been added to the theory of population genetics in the thirty years that have passed since that time. Yet we cannot take this period to mean that we now have an adequate theory of evolutionary dynamics. On the contrary, the theory of population genetics, as complete as it may be in itself, fails to deal with many problems of primary importance for an understanding of evolution.
---
paper_title: Cooperative Control and Potential Games

paper_content:

We present a view of cooperative control using the language of learning in games. We review the game-theoretic concepts of potential and weakly acyclic games, and demonstrate how several cooperative control problems, such as consensus and dynamic sensor coverage, can be formulated in these settings. Motivated by this connection, we build upon game-theoretic concepts to better accommodate a broader class of cooperative control problems. In particular, we extend existing learning algorithms to accommodate restricted action sets caused by the limitations of agent capabilities and group based decision making. Furthermore, we also introduce a new class of games called sometimes weakly acyclic games for time-varying objective functions and action sets, and provide distributed algorithms for convergence to an equilibrium.
---
paper_title: Evolutionary Games in Wireless Networks

paper_content:

We consider a noncooperative interaction among a large population of mobiles that interfere with each other through many local interactions. The first objective of this paper is to extend the evolutionary game framework to allow an arbitrary number of mobiles that are involved in a local interaction. We allow for interactions between mobiles that are not necessarily reciprocal. We study 1) multiple-access control in a slotted Aloha-based wireless network and 2) power control in wideband code-division multiple-access wireless networks. We define and characterize the equilibrium (called evolutionarily stable strategy) for these games and study the influence of wireless channels and pricing on the evolution of dynamics and the equilibrium.
---
paper_title: Distributed Channel Selection in CRAHNs with Heterogeneous Spectrum Opportunities: A Local Congestion Game Approach

paper_content:

This letter investigates the problem of distributed channel selection in cognitive radio ad hoc networks (CRAHNs) with heterogeneous spectrum opportunities. Firstly, we formulate this problem as a local congestion game, which is proved to be an exact potential game. Then, we propose a spatial best response dynamic (SBRD) to rapidly achieve Nash equilibrium via local information exchange. Moreover, the potential function of the game reflects the network collision level and can be used to achieve higher throughput.
---
paper_title: Competitive Spectrum Access in Cognitive Radio Networks: Graphical Game and Learning

paper_content:

Competitive spectrum access is studied for cognitive radio networks. Based on the assumption of rational secondary users, the spectrum access is modeled as a graphical game, in which the payoff of a secondary user is dependent on only other secondary users that can cause significant interference. The Nash equilibrium in the graphical game is computed by minimizing the sum of regrets. To alleviate the local knowledge of payoffs (each secondary user knows only its own payoff for different channels), a subgradient based iterative algorithm is applied by exchanging information across different secondary users. When information exchange is not available, learning for spectrum access is carried out by employing stochastic approximation (more specifically, the Kiefer-Wolfowitz algorithm). The convergence of both situations is demonstrated by numerical simulations.
---
paper_title: Atomic Congestion Games on Graphs and Their Applications in Networking

paper_content:

In this paper, we introduce and analyze the properties of a class of games, the atomic congestion games on graphs (ACGGs), which is a generalization of the classical congestion games. In particular, an ACGG captures the spatial information that is often ignored in a classical congestion game. This is useful in many networking problems, e.g., wireless networks where interference among the users heavily depends on the spatial information. In an ACGG, a player's payoff for using a resource is a function of the number of players who interact with it and use the same resource. Such spatial information can be captured by a graph. We study fundamental properties of the ACGGs: under what conditions these games possess a pure strategy Nash equilibrium (PNE), or the finite improvement property (FIP), which is sufficient for the existence of a PNE. We show that a PNE may not exist in general, but that it does exist in many important special cases including tree, loop, or regular bipartite networks. The FIP holds for important special cases including systems with two resources or identical payoff functions for each resource. Finally, we present two wireless network applications of ACGGs: power control and channel contention under IEEE 802.11.
---
paper_title: A Game Theoretic Perspective on Self-organizing Optimization for Cognitive Small Cells

paper_content:

In this article, we investigate self-organizing optimization for cognitive small cells (CSCs), which have the ability to sense the environment, learn from historical information, make intelligent decisions, and adjust their operational parameters. By exploring the inherent features, some fundamental challenges for self-organizing optimization in CSCs are presented and discussed. Specifically, the dense and random deployment of CSCs brings about some new challenges in terms of scalability and adaptation; furthermore, the uncertain, dynamic, and incomplete information constraints also impose some new challenges in terms of convergence and robustness. For providing better service to users and improving resource utilization, four requirements for self-organizing optimization in CSCs are presented and discussed. Following the attractive fact that the decisions in game-theoretic models are exactly coincident with those in self-organizing optimization (i.e., distributed and autonomous), we establish a framework of gametheoretic solutions for self-organizing optimization in CSCs and propose some featured game models. Specifically, their basic models are presented, some examples are discussed, and future research directions are given.
---
paper_title: Evolutionarily Stable Spectrum Access

paper_content:

In this paper, we design distributed spectrum access mechanisms with both complete and incomplete network information. We propose an evolutionary spectrum access mechanism with complete network information, and show that the mechanism achieves an equilibrium that is globally evolutionarily stable. With incomplete network information, we propose a distributed learning mechanism, where each user utilizes local observations to estimate the expected throughput and learns to adjust its spectrum access strategy adaptively over time. We show that the learning mechanism converges to the same evolutionary equilibrium on the time average. Numerical results show that the proposed mechanisms achieve up to 35 percent performance improvement over the distributed reinforcement learning mechanism in the literature, and are robust to the perturbations of users' channel selections.
---
paper_title: Distributed Channel Selection in CRAHNs with Heterogeneous Spectrum Opportunities: A Local Congestion Game Approach

paper_content:

This letter investigates the problem of distributed channel selection in cognitive radio ad hoc networks (CRAHNs) with heterogeneous spectrum opportunities. Firstly, we formulate this problem as a local congestion game, which is proved to be an exact potential game. Then, we propose a spatial best response dynamic (SBRD) to rapidly achieve Nash equilibrium via local information exchange. Moreover, the potential function of the game reflects the network collision level and can be used to achieve higher throughput.
---
paper_title: Coalition Formation Games for Collaborative Spectrum Sensing

paper_content:

Collaborative spectrum sensing (CSS) between secondary users (SUs) in cognitive networks exhibits an inherent tradeoff between minimizing the probability of missing the detection of the primary user (PU) and maintaining a reasonable false alarm probability (e.g., for maintaining good spectrum utilization). In this paper, we study the impact of this tradeoff on the network structure and the cooperative incentives of the SUs that seek to cooperate to improve their detection performance. We model the CSS problem as a nontransferable coalitional game, and we propose distributed algorithms for coalition formation (CF). First, we construct a distributed CF algorithm that allows the SUs to self-organize into disjoint coalitions while accounting for the CSS tradeoff. Then, the CF algorithm is complemented with a coalitional voting game to enable distributed CF with detection probability (CF-PD) guarantees when required by the PU. The CF-PD algorithm allows the SUs to form minimal winning coalitions (MWCs), i.e., coalitions that achieve the target detection probability with minimal costs. For both algorithms, we study and prove various properties pertaining to network structure, adaptation to mobility, and stability. Simulation results show that CF reduces the average probability of miss per SU up to 88.45%, relative to the noncooperative case, while maintaining a desired false alarm. For CF-PD, the results show that up to 87.25% of the SUs achieve the required detection probability through MWCs.
---
paper_title: Opportunistic Spectrum Access in Unknown Dynamic Environment: A Game-Theoretic Stochastic Learning Solution

paper_content:

We investigate the problem of distributed channel selection using a game-theoretic stochastic learning solution in an opportunistic spectrum access (OSA) system where the channel availability statistics and the number of the secondary users are apriori unknown. We formulate the channel selection problem as a game which is proved to be an exact potential game. However, due to the lack of information about other users and the restriction that the spectrum is time-varying with unknown availability statistics, the task of achieving Nash equilibrium (NE) points of the game is challenging. Firstly, we propose a genie-aided algorithm to achieve the NE points under the assumption of perfect environment knowledge. Based on this, we investigate the achievable performance of the game in terms of system throughput and fairness. Then, we propose a stochastic learning automata (SLA) based channel selection algorithm, with which the secondary users learn from their individual action-reward history and adjust their behaviors towards a NE point. The proposed learning algorithm neither requires information exchange, nor needs prior information about the channel availability statistics and the number of secondary users. Simulation results show that the SLA based learning algorithm achieves high system throughput with good fairness.
---
paper_title: Price-Based Resource Allocation for Spectrum-Sharing Femtocell Networks: A Stackelberg Game Approach

paper_content:

This paper investigates price-based resource allocation strategies for two-tier femtocell networks, in which a central macrocell is underlaid with distributed femtocells, all operating over the same frequency band. Assuming that the macrocell base station (MBS) protects itself by pricing the interference from femtocell users, a Stackelberg game is formulated to study the joint utility maximization of the macrocell and femtocells subject to a maximum tolerable interference power constraint at the MBS. Two practical femtocell network models are investigated: sparsely deployed scenario for rural areas and densely deployed scenario for urban areas. For each scenario, two pricing schemes: uniform pricing and non-uniform pricing, are proposed. The Stackelberg equilibriums for the proposed games are characterized, and an effective distributed interference price bargaining algorithm with guaranteed convergence is proposed for the uniform-pricing case. Numerical examples are presented to verify the proposed studies. It is shown that the proposed schemes are effective in resource allocation and macrocell protection for both the uplink and downlink transmissions in spectrum-sharing femtocell networks.
---
paper_title: A Game Theoretic Perspective on Self-organizing Optimization for Cognitive Small Cells

paper_content:

In this article, we investigate self-organizing optimization for cognitive small cells (CSCs), which have the ability to sense the environment, learn from historical information, make intelligent decisions, and adjust their operational parameters. By exploring the inherent features, some fundamental challenges for self-organizing optimization in CSCs are presented and discussed. Specifically, the dense and random deployment of CSCs brings about some new challenges in terms of scalability and adaptation; furthermore, the uncertain, dynamic, and incomplete information constraints also impose some new challenges in terms of convergence and robustness. For providing better service to users and improving resource utilization, four requirements for self-organizing optimization in CSCs are presented and discussed. Following the attractive fact that the decisions in game-theoretic models are exactly coincident with those in self-organizing optimization (i.e., distributed and autonomous), we establish a framework of gametheoretic solutions for self-organizing optimization in CSCs and propose some featured game models. Specifically, their basic models are presented, some examples are discussed, and future research directions are given.
---
paper_title: Repeated Games with Intervention: Theory and Applications in Communications

paper_content:

In communication systems where users share common resources, selfish behavior usually results in suboptimal resource utilization. There have been extensive works that model communication systems with selfish users as one-shot games and propose incentive schemes to achieve Pareto-optimal outcomes. However, in many communication systems, due to strong negative externalities among users, the sets of feasible payoffs in one-shot games are nonconvex. Thus, it is possible to expand the set of feasible payoffs by having users choose different action profiles in an alternating manner. In this paper, we formulate a model of repeated games with intervention. First, by using repeated games we can convexify the set of feasible payoffs in one-shot games. Second, by using intervention in repeated games we can achieve a larger set of equilibrium payoffs and loosen requirements for users' patience to achieve a target payoff. We study the problem of maximizing a welfare function defined on users' payoffs. We characterize the limit set of equilibrium payoffs. Given the optimal equilibrium payoff, we derive the sufficient condition on the discount factor and the intervention capability to achieve it, and design corresponding equilibrium strategies. We illustrate our analytical results with power control and flow control.
---
paper_title: A stackelberg game approach to distributed spectrum management

paper_content:

We consider a cognitive radio system with one primary (licensed) user and multiple secondary (unlicensed) users. Given the interference temperature constraint, the secondary users compete for the available spectrum to fulfill their own communication need. Borrowing the concept of price from market theory, we develop a decentralized Stackelberg game formulation for power allocation. In this scheme, the primary user (leader) announces prices for the available tones such that a system utility is maximized. Using the announced prices, secondary users (followers) compete for the available bandwidth to maximize their own utilities. We show that this Stackelberg game is polynomial time solvable under certain channel conditions. When the individual power constraints of secondary users are inactive (due to strict interference temperature constraint), the proposed distributed power control method is decomposable across the tones and unlike normal water-filling it respects the interference temperature constraints of the primary user. When individual power constraints are active, we propose a distributed approach that solves the problem under an aggregate interference temperature constraint. Moreover, we propose a dual decomposition based power control method and show that it solves the Stackelberg game asymptotically when the number of tones becomes large.
---
paper_title: A Q-learning based approach to interference avoidance in self-organized femtocell networks

paper_content:

The femtocell concept is an emerging technology for deploying the next generation of the wireless networks, aiming at indoor coverage enhancement, increasing capacity, and offloading the overlay macrocell traffic. Nevertheless, the detrimental factor in such networks is co-channel interference between macrocells and femtocells, as well as among neighboring femtocells. This in turn can dramatically decrease the overall capacity of the network. In addition, due to their non-coordinated nature, femtocells need to self-organize in a distributed manner not to cause interference on the macrocell, while at the same time managing interference among neighboring femtocells. This paper proposes and analyzes a Reinforcement-Learning (RL) framework where a macrocell network is underlaid with femtocells sharing the same spectrum. A distributed Q-learning algorithm is proposed in which each Femto Base Station/Access Point (FBS/FAP) gradually learns (by interacting with its local environment) through trials and errors, and adapt the channel selection strategy until reaching convergence. The proposed Q-learning algorithm is cast into high level and low level subproblems, in which the former finds in a decentralized way the channel allocation through Q-learning, while the latter computes the optimal power allocation. Investigations show that through learning, femtocells are not only able to self-organize with only local information, but also mitigate their interference towards the macrocell network.
---
paper_title: Price-Based Resource Allocation for Spectrum-Sharing Femtocell Networks: A Stackelberg Game Approach

paper_content:

This paper investigates price-based resource allocation strategies for two-tier femtocell networks, in which a central macrocell is underlaid with distributed femtocells, all operating over the same frequency band. Assuming that the macrocell base station (MBS) protects itself by pricing the interference from femtocell users, a Stackelberg game is formulated to study the joint utility maximization of the macrocell and femtocells subject to a maximum tolerable interference power constraint at the MBS. Two practical femtocell network models are investigated: sparsely deployed scenario for rural areas and densely deployed scenario for urban areas. For each scenario, two pricing schemes: uniform pricing and non-uniform pricing, are proposed. The Stackelberg equilibriums for the proposed games are characterized, and an effective distributed interference price bargaining algorithm with guaranteed convergence is proposed for the uniform-pricing case. Numerical examples are presented to verify the proposed studies. It is shown that the proposed schemes are effective in resource allocation and macrocell protection for both the uplink and downlink transmissions in spectrum-sharing femtocell networks.
---
paper_title: Transmission control in cognitive radio as a Markovian dynamic game: Structural result on randomized threshold policies

paper_content:

This paper considers an uplink time division multiple access (TDMA) cognitive radio network where multiple cognitive radios (secondary users) attempt to access a spectrum hole. We assume that each secondary user can access the channel according to a decentralized predefined access rule based on the channel quality and the transmission delay of each secondary user. By modeling secondary user block fading channel qualities as a finite state Markov chain, we formulate the transmission rate adaptation problem of each secondary user as a general-sum Markovian dynamic game with a delay constraint. Conditions are given so that the Nash equilibrium transmission policy of each secondary user is a randomized mixture of pure threshold policies. Such threshold policies can be easily implemented. We then present a stochastic approximation algorithm that can adaptively estimate the Nash equilibrium policies and track such policies for non-stationary problems where the statistics of the channel and user parameters evolve with time.
---
paper_title: A Game Theoretic Perspective on Self-organizing Optimization for Cognitive Small Cells

paper_content:

In this article, we investigate self-organizing optimization for cognitive small cells (CSCs), which have the ability to sense the environment, learn from historical information, make intelligent decisions, and adjust their operational parameters. By exploring the inherent features, some fundamental challenges for self-organizing optimization in CSCs are presented and discussed. Specifically, the dense and random deployment of CSCs brings about some new challenges in terms of scalability and adaptation; furthermore, the uncertain, dynamic, and incomplete information constraints also impose some new challenges in terms of convergence and robustness. For providing better service to users and improving resource utilization, four requirements for self-organizing optimization in CSCs are presented and discussed. Following the attractive fact that the decisions in game-theoretic models are exactly coincident with those in self-organizing optimization (i.e., distributed and autonomous), we establish a framework of gametheoretic solutions for self-organizing optimization in CSCs and propose some featured game models. Specifically, their basic models are presented, some examples are discussed, and future research directions are given.
---
paper_title: Decision-Theoretic Distributed Channel Selection for Opportunistic Spectrum Access: Strategies, Challenges and Solutions

paper_content:

Opportunistic spectrum access (OSA) has been regarded as the most promising approach to solve the paradox between spectrum scarcity and waste. Intelligent decision making is key to OSA and differentiates it from previous wireless technologies. In this article, a survey of decision-theoretic solutions for channel selection and access strategies for OSA system is presented. We analyze the challenges facing OSA systems globally, which mainly include interactions among multiple users, dynamic spectrum opportunity, tradeoff between sequential sensing cost and expected reward, and tradeoff between exploitation and exploration in the absence of prior statistical information. We provide comprehensive review and comparison of each kind of existing decision-theoretic solution, i.e., game models, Markovian decision process, optimal stopping problem and multi-armed bandit problem. We analyze their strengths and limitations and outline further research for both technical contents and methodologies. In particular, these solutions are critically analyzed in terms of information, cost and convergence speed, which are key concerns for practical implementation. Moreover, it is noted that each kind of existing decision-theoretic solution mainly addresses one aspect of the challenges, which implies that two or more kinds of decision-theoretic solutions should be incorporated to address more challenges simultaneously.
---
paper_title: Mining spectrum usage data: a large-scale spectrum measurement study

paper_content:

Dynamic spectrum access has been a subject of extensive research activity in recent years. The increasing volume of literature calls for a deeper understanding of the characteristics of current spectrum utilization. In this paper we present a detailed spectrum measurement study, with data collected in the 20MHz to 3GHz spectrum band and at four locations concurrently in South China. We examine the first and second order statistics of the collected data, including channel occupancy/vacancy statistics, channel utilization within each individual wireless service, and the temporal, spectral, and spatial correlation of these measures. Main findings include that the channel vacancy durations follow an exponential-like distribution, but are not independently distributed over time, and that significant spectral and spatial correlations are found between channels of the same service. We then exploit such spectrum correlation to develop a 2-dimensional frequent pattern mining algorithm that can accurately predict channel availability based on past observations.
---
paper_title: Markov Decision Processes with Applications in Wireless Sensor Networks: A Survey

paper_content:

Wireless sensor networks (WSNs) consist of autonomous and resource-limited devices. The devices cooperate to monitor one or more physical phenomena within an area of interest. WSNs operate as stochastic systems because of randomness in the monitored environments. For long service time and low maintenance cost, WSNs require adaptive and robust methods to address data exchange, topology formulation, resource and power optimization, sensing coverage and object detection, and security challenges. In these problems, sensor nodes are used to make optimized decisions from a set of accessible strategies to achieve design goals. This survey reviews numerous applications of the Markov decision process (MDP) framework, a powerful decision-making tool to develop adaptive algorithms and protocols for WSNs. Furthermore, various solution methods are discussed and compared to serve as a guide for using MDPs in WSNs.
---
paper_title: Reinforcement Learning: A Survey

paper_content:

This paper surveys the field of reinforcement learning from a computer-science perspective. It is written to be accessible to researchers familiar with machine learning. Both the historical basis of the field and a broad selection of current work are summarized. Reinforcement learning is the problem faced by an agent that learns behavior through trial-and-error interactions with a dynamic environment. The work described here has a resemblance to work in psychology, but differs considerably in the details and in the use of the word ``reinforcement.'' The paper discusses central issues of reinforcement learning, including trading off exploration and exploitation, establishing the foundations of the field via Markov decision theory, learning from delayed reinforcement, constructing empirical models to accelerate learning, making use of generalization and hierarchy, and coping with hidden state. It concludes with a survey of some implemented systems and an assessment of the practical utility of current methods for reinforcement learning.
---
paper_title: Decentralized control of partially observable Markov decision processes

paper_content:

The focus of this paper is on solving multi-robot planning problems in continuous spaces with partial observability. Decentralized Partially Observable Markov Decision Processes (Dec-POMDPs) are general models for multi-robot coordination problems, but representing and solving Dec-POMDPs is often intractable for large problems. To allow for a high-level representation that is natural for multi-robot problems and scalable to large discrete and continuous problems, this paper extends the Dec-POMDP model to the Decentralized Partially Observable Semi-Markov Decision Process (Dec-POSMDP). The Dec-POSMDP formulation allows asynchronous decision-making by the robots, which is crucial in multi-robot domains. We also present an algorithm for solving this Dec-POSMDP which is much more scalable than previous methods since it can incorporate closed-loop belief space macro-actions in planning. These macro-actions are automatically constructed to produce robust solutions. The proposed method's performance is evaluated on a complex multi-robot package delivery problem under uncertainty, showing that our approach can naturally represent multi-robot problems and provide high-quality solutions for large-scale problems.
---
paper_title: Decentralized cognitive MAC for opportunistic spectrum access in ad hoc networks: A POMDP framework

paper_content:

We propose decentralized cognitive MAC protocols that allow secondary users to independently search for spectrum opportunities without a central coordinator or a dedicated communication channel. Recognizing hardware and energy constraints, we assume that a secondary user may not be able to perform full-spectrum sensing or may not be willing to monitor the spectrum when it has no data to transmit. We develop an analytical framework for opportunistic spectrum access based on the theory of partially observable Markov decision process (POMDP). This decision-theoretic approach integrates the design of spectrum access protocols at the MAC layer with spectrum sensing at the physical layer and traffic statistics determined by the application layer of the primary network. It also allows easy incorporation of spectrum sensing error and constraint on the probability of colliding with the primary users. Under this POMDP framework, we propose cognitive MAC protocols that optimize the performance of secondary users while limiting the interference perceived by primary users. A suboptimal strategy with reduced complexity yet comparable performance is developed. Without additional control message exchange between the secondary transmitter and receiver, the proposed decentralized protocols ensure synchronous hopping in the spectrum between the transmitter and the receiver in the presence of collisions and spectrum sensing errors
---
paper_title: Opportunistic spectrum access for energy-constrained cognitive radios

paper_content:

This paper considers a scenario in which a secondary user (SU) opportunistically accesses a channel allocated to some primary network (PN) that switches between idle and active states in a time-slotted manner. At the beginning of each time slot, SU can choose to stay idle or to carry out spectrum sensing to detect the state of PN. If PN is detected to be idle, SU can carry out data transmission. Spectrum sensing consumes time and energy and introduces false alarms and mis-detections. The objective is to dynamically decide, for each time slot, whether SU should stay idle or carry out sensing, and if so, for how long, to maximize the expected reward. We formulate this as a partially observable Markov decision process and prove important properties of the optimal control policies. Heuristic control policies with low complexity and good performance are also proposed. Numerical results show the significant performance gain of our dynamic control approach for opportunistic spectrum access.
---
paper_title: Sequential Optimality and Coordination in Multiagent Systems

paper_content:

Coordination of agent activities is a key problem in multiagent systems. Set in a larger decision theoretic context, the existence of coordination problems leads to difficulty in evaluating the utility of a situation. This in turn makes defining optimal policies for sequential decision processes problematic. We propose a method for solving sequential multi-agent decision problems by allowing agents to reason explicitly about specific coordination mechanisms. We define an extension of value iteration in which the system's state space is augmented with the state of the coordination mechanism adopted, allowing agents to reason about the short and long term prospects for coordination, the long term consequences of (mis)coordination, and make decisions to engage or avoid coordination problems based on expected value. We also illustrate the benefits of mechanism generalization.
---
paper_title: Online Planning Algorithms for POMDPs

paper_content:

Partially Observable Markov Decision Processes (POMDPs) provide a rich framework for sequential decision-making under uncertainty in stochastic domains. However, solving a POMDP is often intractable except for small problems due to their complexity. Here, we focus on online approaches that alleviate the computational complexity by computing good local policies at each decision step during the execution. Online algorithms generally consist of a lookahead search to find the best action to execute at each time step in an environment. Our objectives here are to survey the various existing online POMDP methods, analyze their properties and discuss their advantages and disadvantages; and to thoroughly evaluate these online approaches in different environments under various metrics (return, error bound reduction, lower bound improvement). Our experimental results indicate that state-of-the-art online heuristic search methods can handle large POMDP domains efficiently.
---
paper_title: Joint Design and Separation Principle for Opportunistic Spectrum Access in the Presence of Sensing Errors

paper_content:

Opportunistic spectrum access (OSA) that allows secondary users to independently search for and exploit instantaneous spectrum availability is considered. The design objective is to maximize the throughput of a secondary user while limiting the probability of colliding with primary users. Integrated in the joint design are three basic components: a spectrum sensor that identifies spectrum opportunities, a sensing strategy that determines which channels in the spectrum to sense, and an access strategy that decides whether to access based on potentially erroneous sensing outcomes. This joint design is formulated as a constrained partially observable Markov decision process (POMDP), and a separation principle is established. The separation principle reveals the optimality of myopic policies for the design of the spectrum sensor and the access strategy, leading to closed-form optimal solutions. Furthermore, it decouples the design of the sensing strategy from that of the spectrum sensor and the access strategy, and reduces the constrained POMDP to an unconstrained one. Numerical examples are provided to study the tradeoff between sensing time and transmission time, the interaction between the physical layer spectrum sensor and the MAC layer sensing and access strategies, and the robustness of the ensuing design to model mismatch.
---
paper_title: Opportunistic Spectrum Access via Periodic Channel Sensing

paper_content:

The problem of opportunistic access of parallel channels occupied by primary users is considered. Under a continuous-time Markov chain modeling of the channel occupancy by the primary users, a slotted transmission protocol for secondary users using a periodic sensing strategy with optimal dynamic access is proposed. To maximize channel utilization while limiting interference to primary users, a framework of constrained Markov decision processes is presented, and the optimal access policy is derived via a linear program. Simulations are used for performance evaluation. It is demonstrated that periodic sensing yields negligible loss of throughput when the constraint on interference is tight.
---
paper_title: Maximum Throughput Region of Multiuser Cognitive Access of Continuous Time Markovian Channels

paper_content:

The problem of cognitive access of multiple primary channels by multiple cognitive users is considered. The primary transmission on each channel is modeled by a continuous time Markov on-off process. Cognitive access of the primary channels is realized via channel sensing. Each cognitive user adopts a slotted transmission structure, senses one channel in each slot and makes the transmission decision based on the sensing outcome. The cognitive transmissions in each channel are subject to collision constraints that limit their interference to the primary users. The maximum throughput region of this multiuser cognitive network is characterized by establishing inner and outer bounds. Under tight collision constraints, the inner bound is obtained by a simple orthogonalized periodic sensing with memoryless access policy and its generalizations. The outer bound, on the other hand, is obtained by relating the sum throughput with the interference limits. It is shown that when collision constraints are tight, the outer and inner bounds match. This maximum throughput region result is further extended by a generalized periodic sensing scheme with a mechanism of timing sharing. Under general collision constraints, another outer bound is obtained via Whittle's relaxation and another inner bound obtained via Whittle's index sensing policy with memoryless access. Packet level simulations are used to validate the analytical performance prediction.
---
paper_title: Optimal Channel Access Management with QoS Support for Cognitive Vehicular Networks

paper_content:

We consider the problem of optimal channel access to provide quality of service (QoS) for data transmission in cognitive vehicular networks. In such a network, the vehicular nodes can opportunistically access the radio channels (referred to as shared-use channels) which are allocated to licensed users. Also, they are able to reserve a channel for dedicated access (referred to as exclusive-use channel) for data transmission. A channel access management framework is developed for cluster-based communication among vehicular nodes. This framework has three components: opportunistic access to shared-use channels, reservation of exclusive-use channel, and cluster size control. A hierarchical optimization model is then developed for this framework to obtain the optimal policy. The objective of the optimization model is to maximize the utility of the vehicular nodes in a cluster and to minimize the cost of reserving exclusive-use channel while the QoS requirements of data transmission (for vehicle-to-vehicle and vehicle-to-roadside communications) are met, and also the constraint on probability of collision with licensed users is satisfied. This hierarchical optimization model comprises of two constrained Markov decision process (CMDP) formulations - one for opportunistic channel access, and the other for joint exclusive-use channel reservation and cluster size control. An algorithm is presented to solve this hierarchical optimization model. Performance evaluation results show the effectiveness of the optimal channel access management policy. The proposed optimal channel access management framework will be useful to support mobile computing and intelligent transportation system (ITS) applications in vehicular networks.
---
paper_title: Optimal Energy-Efficient Channel Exploration for Opportunistic Spectrum Usage

paper_content:

This letter studies the channel exploration problem for opportunistic spectrum usage systems, where exploring state information of each channel consumes time and energy. We formulate this problem as an optimal stopping problem and propose a myopic rule with low complexity, called one stage look ahead (1-SLA), to solve it. Moreover, the optimality of the 1-SLA rule for the energy-efficient channel exploration problem is proved, and simulation results are provided to show the effectiveness of the 1-SLA rule.
---
paper_title: Opportunistic Spectral Usage: Bounds and a Multi-Band CSMA/CA Protocol

paper_content:

In this paper, we study the gains from opportunistic spectrum usage when neither sender or receiver are aware of the current channel conditions in different frequency bands. Hence to select the best band for sending data, nodes first need to measure the channel in different bands which takes time away from sending actual data. We analyze the gains from opportunistic band selection by deriving an optimal skipping rule, which balances the throughput gain from finding a good quality band with the overhead of measuring multiple bands. We show that opportunistic band skipping is most beneficial in low signal to noise scenarios, which are typically the cases when the node throughput in single-band (no opportunism) system is the minimum. To study the impact of opportunism on network throughput, we devise a CSMA/CA protocol, multi-band opportunistic auto rate (MOAR), which implements the proposed skipping rule on a per node pair basis. The proposed protocol exploits both time and frequency diversity, and is shown to result in typical throughput gains of 20% or more over a protocol which only exploits time diversity, opportunistic auto rate (OAR).
---
paper_title: Optimal Frequency-Temporal Opportunity Exploitation for Multichannel Ad Hoc Networks

paper_content:

In multichannel system, user could keep transmitting over an instantaneous “on peak” channel by opportunistically accessing and switching among channels. Previous studies rely on constant transmission duration, which would fail to leverage more opportunities in time and frequency domain. In this paper, we consider opportunistic channel accessing/releasing scheme in multichannel system with Rayleigh fading channels. Our main goal is to derive a throughput-optimal strategy for determining when and which channel to access and when to release it. We formulate this real-time decision-making process as a two-dimensional optimal stopping problem. We prove that the two-dimensional optimal stopping rule can be reduced to a simple threshold-based policy. Leveraging the absorbing Markov chain theory, we obtain the optimal threshold as well as the maximum achievable throughput with computational efficiency. Numerical and simulation results show that our proposed channel utilization scheme achieves up to 140 percent throughput gain over opportunistic transmission with a single channel and up to 60 percent throughput gain over opportunistic channel access with constant transmission duration.
---
paper_title: Simple Channel Sensing Order in Cognitive Radio Networks

paper_content:

In cognitive radio networks (CRNs), effective and efficient channel exploitation is imperative for unlicensed secondary users to seize available network resources and improve resource utilization. In this paper, we propose a simple channel sensing order for secondary users in multi-channel CRNs without a priori knowledge of primary user activities. By sensing the channels according to the descending order of their achievable rates with optimal stopping, we show that the proposed channel exploitation approach is efficient yet effective in elevating throughput and resource utilization. Simulation results show that our proposed channel exploitation approach outperforms its counterparts by up to 18% in a single-secondary user pair scenario. In addition, we investigate the probability of packet transmission collision in a multi-secondary user pair scenario, and show that the probability of collision decreases as the number of channels increases and/or the number of secondary user pairs decreases. It is observed that the total throughput and resource utilization increase with the number of secondary user pairs due to increased transmission opportunities and multi-user diversity. Our results also demonstrate that resource utilization can be further improved via the proposed channel exploitation approach when the number of secondary user pairs approaches the number of channels.
---
paper_title: Optimal selection of channel sensing order in cognitive radio

paper_content:

This paper investigates the optimal sensing order problem in multi-channel cognitive medium access control with opportunistic transmissions. The scenario in which the availability probability of each channel is known is considered first. In this case, when the potential channels are identical (except for the availability probabilities) and independent, it is shown that, although the intuitive sensing order (i.e., descending order of the channel availability probabilities) is optimal when adaptive modulation is not used, it does not lead to optimality in general with adaptive modulation. Thus, a dynamic programming approach to the search for an optimal sensing order with adaptive modulation is presented. For some special cases, it is proved that a simple optimal sensing order does exist. More complex scenarios are then considered, e.g., in which the availability probability of each channel is unknown. Optimal strategies are developed to address the challenges created by this additional uncertainty. Finally, a scheme is developed to address the issue of sensing errors.
---
paper_title: HC-MAC: A Hardware-Constrained Cognitive MAC for Efficient Spectrum Management

paper_content:

Radio spectrum resource is of fundamental importance for wireless communication. Recent reports show that most available spectrum has been allocated. While some of the spectrum bands (e.g., unlicensed band, GSM band) have seen increasingly crowded usage, most of the other spectrum resources are underutilized. This drives the emergence of open spectrum and dynamic spectrum access concepts, which allow unlicensed users equipped with cognitive radios to opportunistically access the spectrum not used by primary users. Cognitive radio has many advanced features, such as agilely sensing the existence of primary users and utilizing multiple spectrum bands simultaneously. However, in practice such capabilities are constrained by hardware cost. In this paper, we discuss how to conduct efficient spectrum management in ad hoc cognitive radio networks while taking the hardware constraints (e.g., single radio, partial spectrum sensing and spectrum aggregation limit) into consideration. A hardware-constrained cognitive MAC, HC-MAC, is proposed to conduct efficient spectrum sensing and spectrum access decision. We identify the issue of optimal spectrum sensing decision for a single secondary transmission pair, and formulate it as an optimal stopping problem. A decentralized MAC protocol is then proposed for the ad hoc cognitive radio networks. Simulation results are presented to demonstrate the effectiveness of our proposed protocol.
---
paper_title: Optimal Energy-Efficient Channel Exploration for Opportunistic Spectrum Usage

paper_content:

This letter studies the channel exploration problem for opportunistic spectrum usage systems, where exploring state information of each channel consumes time and energy. We formulate this problem as an optimal stopping problem and propose a myopic rule with low complexity, called one stage look ahead (1-SLA), to solve it. Moreover, the optimality of the 1-SLA rule for the energy-efficient channel exploration problem is proved, and simulation results are provided to show the effectiveness of the 1-SLA rule.
---
paper_title: Exploiting Multichannel Diversity in Spectrum Sharing Systems Using Optimal Stopping Rule

paper_content:

This letter studies the problem of exploiting multichannel diversity in a spectrum sharing system, where the secondary user (SU) sequentially explores channel state information on the licensed channels with time consumption. To maximize the expected achievable throughput for the SU, we formulate this problem as an optimal stopping problem, whose objective is to choose the right channel to stop exploration based on the observed signal-to-noise ratio sequence. Moreover, we propose a myopic but optimal rule, called one-stage look-ahead rule, to solve the stopping problem.
---
paper_title: Optimal channel probing and transmission scheduling for opportunistic spectrum access

paper_content:

In this study we consider optimal opportunistic spectrum access (OSA) policies for a transmitter in a multichannel wireless system, where a channel can be in one of multiple states. Each channel state is associated with either a probability of transmission success or a transmission rate. In such systems, the transmitter typically has partial information concerning the channel states, but can deduce more by probing individual channels, e.g. by sending control packets in the channels, at the expense of certain resources, e.g., energy and time. The main goal of this work is to derive optimal strategies for determining which channels to probe (in what sequence) and which channel to use for transmission. We consider two problems within this context,allthe constant data time (CDT) and the constant access time (CAT) problems. For both problems, we derive key structural properties of the corresponding optimal strategy. In particular, we show that it has a threshold structure and can be described by an index policy. We further show that the optimal CDT strategy can only take on one of three structural forms. Using these results we present a two-step lookahead CDT (CAT) strategy. This strategy is shown to be optimal for a number of cases of practical interest. We examine its performance under a class of practical channel models via numerical studies.
---
paper_title: Energy-efficient exploration and exploitation of multichannel diversity in spectrum sharing systems

paper_content:

This letter investigates the problem of energy-efficient exploration and exploitation of multichannel diversity in spectrum sharing cognitive radio systems where the secondary user sequentially explores the channel state information on the licenced channels with time and energy consumptions. As the number of the explored channels increases, the achieved multichannel diversity gain increases and so does the exploration consumption. Thus, there is a fundamental tradeoff between the multichannel diversity gain and channel exploration overhead. To maximise the expected normalised capacity of the secondary user, we formulate this tradeoff as an optimal stopping problem and propose a myopic one-stage look-ahead rule to solve it. It is shown that the one-stage look-ahead rule is optimal in the low power region; moreover, it also has good performance in general power region. Simulation results show that the achievable normalised throughput differs greatly for different exploration overhead, which can be regarded as a distinct feature of spectrum sharing systems. Copyright © 2012 John Wiley & Sons, Ltd.
---
paper_title: Multi-Agent Q-Learning for Competitive Spectrum Access in Cognitive Radio Systems

paper_content:

Resource allocation is an important issue in cognitive radio systems. It can be done by carrying out negotiation among secondary users. However, significant overhead may be incurred by the negotiation since the negotiation needs to be done frequently due to the rapid change of primary users' activity. In this paper, an Aloha-like spectrum access scheme without negotiation is considered for multi-user and multi-channel cognitive radio systems. To avoid collision incurred by the lack of coordination, each secondary user learns how to select channels according to its experience. Multi-agent reinforcement leaning (MARL) is applied in the framework of $Q$-learning by considering other secondary users as a part of the environment. A rigorous proof of the convergence of $Q$-learning is provided via the similarity between the $Q$-learning and Robinson-Monro algorithm, as well as the analysis of the corresponding ordinary differential equation (via Lyapunov function). The performance of learning (speed and gain in utility) is evaluated by numerical simulations.
---
paper_title: Finite-time Analysis of the Multiarmed Bandit Problem

paper_content:

Reinforcement learning policies face the exploration versus exploitation dilemma, i.e. the search for a balance between exploring the environment to find profitable actions while taking the empirically best action as often as possible. A popular measure of a policy's success in addressing this dilemma is the regret, that is the loss due to the fact that the globally optimal policy is not followed all the times. One of the simplest examples of the exploration/exploitation dilemma is the multi-armed bandit problem. Lai and Robbins were the first ones to show that the regret for this problem has to grow at least logarithmically in the number of plays. Since then, policies which asymptotically achieve this regret have been devised by Lai and Robbins and many others. In this work we show that the optimal logarithmic regret is also achievable uniformly over time, with simple and efficient policies, and for all reward distributions with bounded support.
---
paper_title: Convergence to equilibrium in local interaction games

paper_content:

We study a simple game-theoretic model for the spread of an innovation in a network. The diffiusion of the innovation is modeled as the dynamics of a coordination game in which the adoption of a common strategy between players has a higher payoff. Classical results in game theory provide a simple condition for the innovation to spread through the network. The present paper characterizes the rate of convergence as a function of graph structure. In particular, we derive a dichotomy between well-connected (e.g. random) graphs that show slow convergence and poorly connected, low dimensional graphs that show fast convergence.
---
paper_title: Distributed Learning in Multi-Armed Bandit with Multiple Players

paper_content:

We formulate and study a decentralized multi-armed bandit (MAB) problem. There are M distributed players competing for N independent arms. Each arm, when played, offers i.i.d. reward according to a distribution with an unknown parameter. At each time, each player chooses one arm to play without exchanging observations or any information with other players. Players choosing the same arm collide, and, depending on the collision model, either no one receives reward or the colliding players share the reward in an arbitrary way. We show that the minimum system regret of the decentralized MAB grows with time at the same logarithmic order as in the centralized counterpart where players act collectively as a single entity by exchanging observations and making decisions jointly. A decentralized policy is constructed to achieve this optimal order while ensuring fairness among players and without assuming any pre-agreement or information exchange among players. Based on a Time Division Fair Sharing (TDFS) of the M best arms, the proposed policy is constructed and its order optimality is proven under a general reward model. Furthermore, the basic structure of the TDFS policy can be used with any order-optimal single-player policy to achieve order optimality in the decentralized setting. We also establish a lower bound on the system regret growth rate for a general class of decentralized polices, to which the proposed policy belongs. This problem finds potential applications in cognitive radio networks, multi-channel communication systems, multi-agent systems, web search and advertising, and social networks.
---
paper_title: Cognitive Medium Access: Exploration, Exploitation, and Competition

paper_content:

This paper considers the design of efficient strategies that allow cognitive users to choose frequency bands to sense and access among multiple bands with unknown parameters. First, the scenario in which a single cognitive user wishes to opportunistically exploit the availability of frequency bands is considered. By adopting tools from the classical bandit problem, optimal as well as low complexity asymptotically optimal solutions are developed. Next, the multiple cognitive user scenario is considered. The situation in which the availability probability of each channel is known is first considered. An optimal symmetric strategy that maximizes the total throughput of the cognitive users is developed. To avoid the possible selfish behavior of the cognitive users, a game-theoretic model is then developed. The performance of both models is characterized analytically. Then, the situation in which the availability probability of each channel is unknown a priori is considered. Low-complexity medium access protocols, which strike an optimal balance between exploration and exploitation in such competitive environments, are developed. The operating points of these low-complexity protocols are shown to converge to those of the scenario in which the availability probabilities are known. Finally, numerical results are provided to illustrate the impact of sensing errors and other practical considerations.
---
paper_title: Opportunistic Spectrum Access with Multiple Users: Learning under Competition

paper_content:

The problem of cooperative allocation among multiple secondary users to maximize cognitive system throughput is considered. The channel availability statistics are initially unknown to the secondary users and are learnt via sensing samples. Two distributed learning and allocation schemes which maximize the cognitive system throughput or equivalently minimize the total regret in distributed learning and allocation are proposed. The first scheme assumes minimal prior information in terms of pre-allocated ranks for secondary users while the second scheme is fully distributed and assumes no such prior information. The two schemes have sum regret which is provably logarithmic in the number of sensing time slots. A lower bound is derived for any learning scheme which is asymptotically logarithmic in the number of slots. Hence, our schemes achieve asymptotic order optimality in terms of regret in distributed learning and allocation.
---
paper_title: Decentralized Online Learning Algorithms for Opportunistic Spectrum Access

paper_content:

The fundamental problem of multiple secondary users contending for opportunistic spectrum access over multiple channels in cognitive radio networks has been formulated recently as a decentralized multi-armed bandit (D-MAB) problem. In a D-MAB problem there are M users and N arms (channels) that each offer i.i.d. stochastic rewards with unknown means so long as they are accessed without collision. The goal is to design a decentralized online learning policy that incurs minimal regret, defined as the difference between the total expected rewards accumulated by a model-aware genie, and that obtained by all users applying the policy. We make two contributions in this paper. First, we consider the setting where the users have a prioritized ranking, such that it is desired for the K-th-ranked user to learn to access the arm offering the K-th highest mean reward. For this problem, we present the first distributed policy that yields regret that is uniformly logarithmic over time without requiring any prior assumption about the mean rewards. Second, we consider the case when a fair access policy is required, i.e., it is desired for all users to experience the same mean reward. For this problem, we present a distributed policy that yields order-optimal regret scaling with respect to the number of users and arms, better than previously proposed policies in the literature. Both of our distributed policies make use of an innovative modification of the well known UCB1 policy for the classic multi-armed bandit problem that allows a single user to learn how to play the arm that yields the K-th largest mean reward.
---
paper_title: Online learning in opportunistic spectrum access: A restless bandit approach

paper_content:

We consider an opportunistic spectrum access (OSA) problem where the time-varying condition of each channel (e.g., as a result of random fading or certain primary users' activities) is modeled as an arbitrary finite-state Markov chain. At each instance of time, a (secondary) user probes a channel and collects a certain reward as a function of the state of the channel (e.g., good channel condition results in higher data rate for the user). Each channel has potentially different state space and statistics, both unknown to the user, who tries to learn which one is the best as it goes and maximizes its usage of the best channel. The objective is to construct a good online learning algorithm so as to minimize the difference between the user's performance in total rewards and that of using the best channel (on average) had it known which one is the best from a priori knowledge of the channel statistics (also known as the regret). This is a classic exploration and exploitation problem and results abound when the reward processes are assumed to be iid. Compared to prior work, the biggest difference is that in our case the reward process is assumed to be Markovian, of which iid is a special case. In addition, the reward processes are restless in that the channel conditions will continue to evolve independent of the user's actions. This leads to a restless bandit problem, for which there exists little result on either algorithms or performance bounds in this learning context to the best of our knowledge. In this paper we introduce an algorithm that utilizes regenerative cycles of a Markov chain and computes a samplemean based index policy, and show that under mild conditions on the state transition probabilities of the Markov chains this algorithm achieves logarithmic regret uniformly over time, and that this regret bound is also optimal. We numerically examine the performance of this algorithm along with a few other learning algorithms in the case of an OSA problem with Gilbert-Elliot channel models, and discuss how this algorithm may be further improved (in terms of its constant) and how this result may lead to similar bounds for other algorithms.
---
paper_title: Optimally Sensing a Single Channel Without Prior Information: The Tiling Algorithm and Regret Bounds

paper_content:

We consider the task of optimally sensing a two-state Markovian channel with an observation cost and without any prior information regarding the channel's transition probabilities. This task is of interest in the field of cognitive radio as a model for opportunistic access to a communication network by a secondary user. The optimal sensing problem may be cast into the framework of model-based reinforcement learning in a specific class of partially observable Markov decision processes (POMDPs). We propose the Tiling Algorithm, an original method aimed at reaching an optimal tradeoff between the exploration (or estimation) and exploitation requirements. It is shown that this algorithm achieves finite horizon regret bounds that are as good as those recently obtained for multi-armed bandits and finite-state Markov decision processes (MDPs).
---
paper_title: On Optimality of Myopic Policy for Restless Multi-Armed Bandit Problem: An Axiomatic Approach

paper_content:

Due to its application in numerous engineering problems, the restless multi-armed bandit (RMAB) problem is of fundamental importance in stochastic decision theory. However, solving the RMAB problem is well known to be PSPACE-hard, with the optimal policy usually intractable due to the exponential computation complexity. A natural alternative approach is to seek simple myopic policies which are easy to implement. This paper presents a generic study on the optimality of the myopic policy for the RMAB problem. More specifically, we develop three axioms characterizing a family of generic and practically important functions termed as regular functions. By performing a mathematical analysis based on the developed axioms, we establish the closed-form conditions under which the myopic policy is guaranteed to be optimal. The axiomatic analysis also illuminates important engineering implications of the myopic policy including the intrinsic tradeoff between exploration and exploitation. A case study is then presented to illustrate the application of the derived results in analyzing a class of RMAB problems arising from multi-channel opportunistic access.
---
paper_title: Multi-Agent Q-Learning for Competitive Spectrum Access in Cognitive Radio Systems

paper_content:

Resource allocation is an important issue in cognitive radio systems. It can be done by carrying out negotiation among secondary users. However, significant overhead may be incurred by the negotiation since the negotiation needs to be done frequently due to the rapid change of primary users' activity. In this paper, an Aloha-like spectrum access scheme without negotiation is considered for multi-user and multi-channel cognitive radio systems. To avoid collision incurred by the lack of coordination, each secondary user learns how to select channels according to its experience. Multi-agent reinforcement leaning (MARL) is applied in the framework of $Q$-learning by considering other secondary users as a part of the environment. A rigorous proof of the convergence of $Q$-learning is provided via the similarity between the $Q$-learning and Robinson-Monro algorithm, as well as the analysis of the corresponding ordinary differential equation (via Lyapunov function). The performance of learning (speed and gain in utility) is evaluated by numerical simulations.
---
paper_title: Evolutionarily Stable Spectrum Access

paper_content:

In this paper, we design distributed spectrum access mechanisms with both complete and incomplete network information. We propose an evolutionary spectrum access mechanism with complete network information, and show that the mechanism achieves an equilibrium that is globally evolutionarily stable. With incomplete network information, we propose a distributed learning mechanism, where each user utilizes local observations to estimate the expected throughput and learns to adjust its spectrum access strategy adaptively over time. We show that the learning mechanism converges to the same evolutionary equilibrium on the time average. Numerical results show that the proposed mechanisms achieve up to 35 percent performance improvement over the distributed reinforcement learning mechanism in the literature, and are robust to the perturbations of users' channel selections.
---
paper_title: Decision-Theoretic Distributed Channel Selection for Opportunistic Spectrum Access: Strategies, Challenges and Solutions

paper_content:

Opportunistic spectrum access (OSA) has been regarded as the most promising approach to solve the paradox between spectrum scarcity and waste. Intelligent decision making is key to OSA and differentiates it from previous wireless technologies. In this article, a survey of decision-theoretic solutions for channel selection and access strategies for OSA system is presented. We analyze the challenges facing OSA systems globally, which mainly include interactions among multiple users, dynamic spectrum opportunity, tradeoff between sequential sensing cost and expected reward, and tradeoff between exploitation and exploration in the absence of prior statistical information. We provide comprehensive review and comparison of each kind of existing decision-theoretic solution, i.e., game models, Markovian decision process, optimal stopping problem and multi-armed bandit problem. We analyze their strengths and limitations and outline further research for both technical contents and methodologies. In particular, these solutions are critically analyzed in terms of information, cost and convergence speed, which are key concerns for practical implementation. Moreover, it is noted that each kind of existing decision-theoretic solution mainly addresses one aspect of the challenges, which implies that two or more kinds of decision-theoretic solutions should be incorporated to address more challenges simultaneously.
---
paper_title: Multichannel Opportunistic Spectrum Access in Fading Environment Using Optimal Stopping Rule

paper_content:

This paper studies the tradeoff between throughput and multichannel diversity in multichannel opportunistic spectrum access (OSA) systems. We explicitly consider channel condition as well as the activities of the primary users. We assume that the primary users use the licensed channel in a slotted fashion and the secondary users can only explore one licensed channel at a time. The secondary users then sequentially explore multiple channels to find the best channel for transmission. However, channel exploration is time-consumed, which decreases effective transmission time in a slot. For single secondary user OSA systems, we formulate the channel exploration problem as an optimal stopping problem with recall, and propose a myopic but optimal approach. For multiple secondary user OSA systems, we propose an adaptive stochastic recall algorithm (ASRA) to capture the collision among multiple secondary users. It is shown that the proposed solutions in this paper achieve increased throughput both the scenario of both single secondary user as well as multiple secondary suers.
---
paper_title: Optimally Sensing a Single Channel Without Prior Information: The Tiling Algorithm and Regret Bounds

paper_content:

We consider the task of optimally sensing a two-state Markovian channel with an observation cost and without any prior information regarding the channel's transition probabilities. This task is of interest in the field of cognitive radio as a model for opportunistic access to a communication network by a secondary user. The optimal sensing problem may be cast into the framework of model-based reinforcement learning in a specific class of partially observable Markov decision processes (POMDPs). We propose the Tiling Algorithm, an original method aimed at reaching an optimal tradeoff between the exploration (or estimation) and exploitation requirements. It is shown that this algorithm achieves finite horizon regret bounds that are as good as those recently obtained for multi-armed bandits and finite-state Markov decision processes (MDPs).
---
paper_title: Multi-Agent Q-Learning for Competitive Spectrum Access in Cognitive Radio Systems

paper_content:

Resource allocation is an important issue in cognitive radio systems. It can be done by carrying out negotiation among secondary users. However, significant overhead may be incurred by the negotiation since the negotiation needs to be done frequently due to the rapid change of primary users' activity. In this paper, an Aloha-like spectrum access scheme without negotiation is considered for multi-user and multi-channel cognitive radio systems. To avoid collision incurred by the lack of coordination, each secondary user learns how to select channels according to its experience. Multi-agent reinforcement leaning (MARL) is applied in the framework of $Q$-learning by considering other secondary users as a part of the environment. A rigorous proof of the convergence of $Q$-learning is provided via the similarity between the $Q$-learning and Robinson-Monro algorithm, as well as the analysis of the corresponding ordinary differential equation (via Lyapunov function). The performance of learning (speed and gain in utility) is evaluated by numerical simulations.
---
paper_title: Decision-Theoretic Distributed Channel Selection for Opportunistic Spectrum Access: Strategies, Challenges and Solutions

paper_content:

Opportunistic spectrum access (OSA) has been regarded as the most promising approach to solve the paradox between spectrum scarcity and waste. Intelligent decision making is key to OSA and differentiates it from previous wireless technologies. In this article, a survey of decision-theoretic solutions for channel selection and access strategies for OSA system is presented. We analyze the challenges facing OSA systems globally, which mainly include interactions among multiple users, dynamic spectrum opportunity, tradeoff between sequential sensing cost and expected reward, and tradeoff between exploitation and exploration in the absence of prior statistical information. We provide comprehensive review and comparison of each kind of existing decision-theoretic solution, i.e., game models, Markovian decision process, optimal stopping problem and multi-armed bandit problem. We analyze their strengths and limitations and outline further research for both technical contents and methodologies. In particular, these solutions are critically analyzed in terms of information, cost and convergence speed, which are key concerns for practical implementation. Moreover, it is noted that each kind of existing decision-theoretic solution mainly addresses one aspect of the challenges, which implies that two or more kinds of decision-theoretic solutions should be incorporated to address more challenges simultaneously.
---
paper_title: Multichannel Opportunistic Spectrum Access in Fading Environment Using Optimal Stopping Rule

paper_content:

This paper studies the tradeoff between throughput and multichannel diversity in multichannel opportunistic spectrum access (OSA) systems. We explicitly consider channel condition as well as the activities of the primary users. We assume that the primary users use the licensed channel in a slotted fashion and the secondary users can only explore one licensed channel at a time. The secondary users then sequentially explore multiple channels to find the best channel for transmission. However, channel exploration is time-consumed, which decreases effective transmission time in a slot. For single secondary user OSA systems, we formulate the channel exploration problem as an optimal stopping problem with recall, and propose a myopic but optimal approach. For multiple secondary user OSA systems, we propose an adaptive stochastic recall algorithm (ASRA) to capture the collision among multiple secondary users. It is shown that the proposed solutions in this paper achieve increased throughput both the scenario of both single secondary user as well as multiple secondary suers.
---
paper_title: Game Dynamics and Cost of Learning in Heterogeneous 4G Networks

paper_content:

In this paper, we study game dynamics and learning schemes for heterogeneous 4G networks. We introduce a novel learning scheme called cost-to-learn that incorporates the cost to switch, the switching delay, and the cost of changing to a new action and, captures the realistic behavior of the users that we have experimented on OPNET simulations. Considering a dynamic and uncertain environment where the users and operators have only a numerical value of their own payoffs as information, we construct various heterogeneous combined fully distributed payoff and strategy reinforcement learning (CODIPAS-RL): the users try to learn their own optimal payoff and their optimal strategy simultaneously. We establish the asymptotic pseudo-trajectories as solution of differential equations. Using evolutionary game dynamics, we prove the convergence and stability properties in specific classes of dynamic robust games. We provide various numerical examples and OPNET simulations in the context network selection in wireless local area networks (WLAN) and Long Term Evolution (LTE).
---
paper_title: Multi-Agent Q-Learning for Competitive Spectrum Access in Cognitive Radio Systems

paper_content:

Resource allocation is an important issue in cognitive radio systems. It can be done by carrying out negotiation among secondary users. However, significant overhead may be incurred by the negotiation since the negotiation needs to be done frequently due to the rapid change of primary users' activity. In this paper, an Aloha-like spectrum access scheme without negotiation is considered for multi-user and multi-channel cognitive radio systems. To avoid collision incurred by the lack of coordination, each secondary user learns how to select channels according to its experience. Multi-agent reinforcement leaning (MARL) is applied in the framework of $Q$-learning by considering other secondary users as a part of the environment. A rigorous proof of the convergence of $Q$-learning is provided via the similarity between the $Q$-learning and Robinson-Monro algorithm, as well as the analysis of the corresponding ordinary differential equation (via Lyapunov function). The performance of learning (speed and gain in utility) is evaluated by numerical simulations.
---
paper_title: Decision-Theoretic Distributed Channel Selection for Opportunistic Spectrum Access: Strategies, Challenges and Solutions

paper_content:

Opportunistic spectrum access (OSA) has been regarded as the most promising approach to solve the paradox between spectrum scarcity and waste. Intelligent decision making is key to OSA and differentiates it from previous wireless technologies. In this article, a survey of decision-theoretic solutions for channel selection and access strategies for OSA system is presented. We analyze the challenges facing OSA systems globally, which mainly include interactions among multiple users, dynamic spectrum opportunity, tradeoff between sequential sensing cost and expected reward, and tradeoff between exploitation and exploration in the absence of prior statistical information. We provide comprehensive review and comparison of each kind of existing decision-theoretic solution, i.e., game models, Markovian decision process, optimal stopping problem and multi-armed bandit problem. We analyze their strengths and limitations and outline further research for both technical contents and methodologies. In particular, these solutions are critically analyzed in terms of information, cost and convergence speed, which are key concerns for practical implementation. Moreover, it is noted that each kind of existing decision-theoretic solution mainly addresses one aspect of the challenges, which implies that two or more kinds of decision-theoretic solutions should be incorporated to address more challenges simultaneously.
---
paper_title: Opportunistic Spectrum Access in Cognitive Radio Networks: Global Optimization Using Local Interaction Games

paper_content:

We investigate the problem of achieving global optimization for distributed channel selections in cognitive radio networks (CRNs), using game theoretic solutions. To cope with the lack of centralized control and local influences, we propose two special cases of local interaction game to study this problem. The first is local altruistic game, in which each user considers the payoffs of itself as well as its neighbors rather than considering itself only. The second is local congestion game, in which each user minimizes the number of competing neighbors. It is shown that with the proposed games, global optimization is achieved with local information. Specifically, the local altruistic game maximizes the network throughput and the local congestion game minimizes the network collision level. Also, the concurrent spatial adaptive play (C-SAP), which is an extension of the existing spatial adaptive play (SAP), is proposed to achieve the global optimum both autonomously as well as rapidly.
---
paper_title: A Game Theoretic Perspective on Self-organizing Optimization for Cognitive Small Cells

paper_content:

In this article, we investigate self-organizing optimization for cognitive small cells (CSCs), which have the ability to sense the environment, learn from historical information, make intelligent decisions, and adjust their operational parameters. By exploring the inherent features, some fundamental challenges for self-organizing optimization in CSCs are presented and discussed. Specifically, the dense and random deployment of CSCs brings about some new challenges in terms of scalability and adaptation; furthermore, the uncertain, dynamic, and incomplete information constraints also impose some new challenges in terms of convergence and robustness. For providing better service to users and improving resource utilization, four requirements for self-organizing optimization in CSCs are presented and discussed. Following the attractive fact that the decisions in game-theoretic models are exactly coincident with those in self-organizing optimization (i.e., distributed and autonomous), we establish a framework of gametheoretic solutions for self-organizing optimization in CSCs and propose some featured game models. Specifically, their basic models are presented, some examples are discussed, and future research directions are given.
---
paper_title: Decision-Theoretic Distributed Channel Selection for Opportunistic Spectrum Access: Strategies, Challenges and Solutions

paper_content:

Opportunistic spectrum access (OSA) has been regarded as the most promising approach to solve the paradox between spectrum scarcity and waste. Intelligent decision making is key to OSA and differentiates it from previous wireless technologies. In this article, a survey of decision-theoretic solutions for channel selection and access strategies for OSA system is presented. We analyze the challenges facing OSA systems globally, which mainly include interactions among multiple users, dynamic spectrum opportunity, tradeoff between sequential sensing cost and expected reward, and tradeoff between exploitation and exploration in the absence of prior statistical information. We provide comprehensive review and comparison of each kind of existing decision-theoretic solution, i.e., game models, Markovian decision process, optimal stopping problem and multi-armed bandit problem. We analyze their strengths and limitations and outline further research for both technical contents and methodologies. In particular, these solutions are critically analyzed in terms of information, cost and convergence speed, which are key concerns for practical implementation. Moreover, it is noted that each kind of existing decision-theoretic solution mainly addresses one aspect of the challenges, which implies that two or more kinds of decision-theoretic solutions should be incorporated to address more challenges simultaneously.
---
paper_title: A Game Theoretic Perspective on Self-organizing Optimization for Cognitive Small Cells

paper_content:

In this article, we investigate self-organizing optimization for cognitive small cells (CSCs), which have the ability to sense the environment, learn from historical information, make intelligent decisions, and adjust their operational parameters. By exploring the inherent features, some fundamental challenges for self-organizing optimization in CSCs are presented and discussed. Specifically, the dense and random deployment of CSCs brings about some new challenges in terms of scalability and adaptation; furthermore, the uncertain, dynamic, and incomplete information constraints also impose some new challenges in terms of convergence and robustness. For providing better service to users and improving resource utilization, four requirements for self-organizing optimization in CSCs are presented and discussed. Following the attractive fact that the decisions in game-theoretic models are exactly coincident with those in self-organizing optimization (i.e., distributed and autonomous), we establish a framework of gametheoretic solutions for self-organizing optimization in CSCs and propose some featured game models. Specifically, their basic models are presented, some examples are discussed, and future research directions are given.
---
paper_title: Decision-Theoretic Distributed Channel Selection for Opportunistic Spectrum Access: Strategies, Challenges and Solutions

paper_content:

Opportunistic spectrum access (OSA) has been regarded as the most promising approach to solve the paradox between spectrum scarcity and waste. Intelligent decision making is key to OSA and differentiates it from previous wireless technologies. In this article, a survey of decision-theoretic solutions for channel selection and access strategies for OSA system is presented. We analyze the challenges facing OSA systems globally, which mainly include interactions among multiple users, dynamic spectrum opportunity, tradeoff between sequential sensing cost and expected reward, and tradeoff between exploitation and exploration in the absence of prior statistical information. We provide comprehensive review and comparison of each kind of existing decision-theoretic solution, i.e., game models, Markovian decision process, optimal stopping problem and multi-armed bandit problem. We analyze their strengths and limitations and outline further research for both technical contents and methodologies. In particular, these solutions are critically analyzed in terms of information, cost and convergence speed, which are key concerns for practical implementation. Moreover, it is noted that each kind of existing decision-theoretic solution mainly addresses one aspect of the challenges, which implies that two or more kinds of decision-theoretic solutions should be incorporated to address more challenges simultaneously.
---
paper_title: A stackelberg game approach to distributed spectrum management

paper_content:

We consider a cognitive radio system with one primary (licensed) user and multiple secondary (unlicensed) users. Given the interference temperature constraint, the secondary users compete for the available spectrum to fulfill their own communication need. Borrowing the concept of price from market theory, we develop a decentralized Stackelberg game formulation for power allocation. In this scheme, the primary user (leader) announces prices for the available tones such that a system utility is maximized. Using the announced prices, secondary users (followers) compete for the available bandwidth to maximize their own utilities. We show that this Stackelberg game is polynomial time solvable under certain channel conditions. When the individual power constraints of secondary users are inactive (due to strict interference temperature constraint), the proposed distributed power control method is decomposable across the tones and unlike normal water-filling it respects the interference temperature constraints of the primary user. When individual power constraints are active, we propose a distributed approach that solves the problem under an aggregate interference temperature constraint. Moreover, we propose a dual decomposition based power control method and show that it solves the Stackelberg game asymptotically when the number of tones becomes large.
---
paper_title: Price-Based Resource Allocation for Spectrum-Sharing Femtocell Networks: A Stackelberg Game Approach

paper_content:

This paper investigates price-based resource allocation strategies for two-tier femtocell networks, in which a central macrocell is underlaid with distributed femtocells, all operating over the same frequency band. Assuming that the macrocell base station (MBS) protects itself by pricing the interference from femtocell users, a Stackelberg game is formulated to study the joint utility maximization of the macrocell and femtocells subject to a maximum tolerable interference power constraint at the MBS. Two practical femtocell network models are investigated: sparsely deployed scenario for rural areas and densely deployed scenario for urban areas. For each scenario, two pricing schemes: uniform pricing and non-uniform pricing, are proposed. The Stackelberg equilibriums for the proposed games are characterized, and an effective distributed interference price bargaining algorithm with guaranteed convergence is proposed for the uniform-pricing case. Numerical examples are presented to verify the proposed studies. It is shown that the proposed schemes are effective in resource allocation and macrocell protection for both the uplink and downlink transmissions in spectrum-sharing femtocell networks.
---
paper_title: Simple Channel Sensing Order in Cognitive Radio Networks

paper_content:

In cognitive radio networks (CRNs), effective and efficient channel exploitation is imperative for unlicensed secondary users to seize available network resources and improve resource utilization. In this paper, we propose a simple channel sensing order for secondary users in multi-channel CRNs without a priori knowledge of primary user activities. By sensing the channels according to the descending order of their achievable rates with optimal stopping, we show that the proposed channel exploitation approach is efficient yet effective in elevating throughput and resource utilization. Simulation results show that our proposed channel exploitation approach outperforms its counterparts by up to 18% in a single-secondary user pair scenario. In addition, we investigate the probability of packet transmission collision in a multi-secondary user pair scenario, and show that the probability of collision decreases as the number of channels increases and/or the number of secondary user pairs decreases. It is observed that the total throughput and resource utilization increase with the number of secondary user pairs due to increased transmission opportunities and multi-user diversity. Our results also demonstrate that resource utilization can be further improved via the proposed channel exploitation approach when the number of secondary user pairs approaches the number of channels.
---
paper_title: Multichannel Opportunistic Spectrum Access in Fading Environment Using Optimal Stopping Rule

paper_content:

This paper studies the tradeoff between throughput and multichannel diversity in multichannel opportunistic spectrum access (OSA) systems. We explicitly consider channel condition as well as the activities of the primary users. We assume that the primary users use the licensed channel in a slotted fashion and the secondary users can only explore one licensed channel at a time. The secondary users then sequentially explore multiple channels to find the best channel for transmission. However, channel exploration is time-consumed, which decreases effective transmission time in a slot. For single secondary user OSA systems, we formulate the channel exploration problem as an optimal stopping problem with recall, and propose a myopic but optimal approach. For multiple secondary user OSA systems, we propose an adaptive stochastic recall algorithm (ASRA) to capture the collision among multiple secondary users. It is shown that the proposed solutions in this paper achieve increased throughput both the scenario of both single secondary user as well as multiple secondary suers.
---
paper_title: Distributed Algorithms for Learning and Cognitive Medium Access with Logarithmic Regret

paper_content:

The problem of distributed learning and channel access is considered in a cognitive network with multiple secondary users. The availability statistics of the channels are initially unknown to the secondary users and are estimated using sensing decisions. There is no explicit information exchange or prior agreement among the secondary users and sensing and access decisions are undertaken by them in a completely distributed manner. We propose policies for distributed learning and access which achieve order-optimal cognitive system throughput (number of successful secondary transmissions) under self play, i.e., when implemented at all the secondary users. Equivalently, our policies minimize the sum regret in distributed learning and access, which is the loss in secondary throughput due to learning and distributed access. For the scenario when the number of secondary users is known to the policy, we prove that the total regret is logarithmic in the number of transmission slots. This policy achieves order-optimal regret based on a logarithmic lower bound for regret under any uniformly-good learning and access policy. We then consider the case when the number of secondary users is fixed but unknown, and is estimated at each user through feedback. We propose a policy whose sum regret grows only slightly faster than logarithmic in the number of transmission slots.
---
paper_title: Cooperative Game in Dynamic Spectrum Access with Unknown Model and Imperfect Sensing

paper_content:

We consider dynamic spectrum access where distributed secondary users search for spectrum opportunities without knowing the primary traffic statistics. In each slot, a secondary transmitter chooses one channel to sense and subsequently transmit if the channel is sensed as idle. Sensing is imperfect, i.e., an idle channel may be sensed as busy and vice versa. Without centralized control, each secondary user needs to independently identify the channels that offer the most opportunities while avoiding collisions with both primary and other secondary users. We address the problem within a cooperative game framework, where the objective is to maximize the throughput of the secondary network under a constraint on the collision with the primary system. The performance of a decentralized channel access policy is measured by the system regret, defined as the expected total performance loss with respect to the optimal performance in the ideal scenario where the traffic load of the primary system on each channel is known to all secondary users and collisions among secondary users are eliminated through centralized scheduling. By exploring the rich communication structure of the problem, we show that the optimal system regret has the same logarithmic order as in the centralized counterpart with perfect sensing. A decentralized policy is constructed to achieve the logarithmic order of the system regret. In a broader context, this work addresses imperfect reward observation in decentralized multi-armed bandit problems.
---
paper_title: Observation vs statistics: Near optimal online channel access in cognitive radio networks

paper_content:

We investigate efficient channel learning and opportunity utilization problem in cognitive radio networks (CRN). We find that the sensing order of multiple channels and channel accessing policy play a critical role in designing effective and efficient scheme to maximize the throughput. Leveraging this important finding, we propose a near optimal online channel access policy. We prove that, our policy can converge to an optimal point in a guaranteed probability. Further, we design a computational efficient channel access policy, integrating optimal stopping theory and multi-armed bandit policy effectively. The computational complexity is reduced from O(K NK) to O(K), where N is the number of channels, and K is the maximum number of sensing/probing times in each procedure. Our simulation results validate our policy, showing at least 40% performance improvement over statistically optimal but fixed policy.
---
paper_title: Opportunistic Spectrum Access in Cognitive Radio Networks: Global Optimization Using Local Interaction Games

paper_content:

We investigate the problem of achieving global optimization for distributed channel selections in cognitive radio networks (CRNs), using game theoretic solutions. To cope with the lack of centralized control and local influences, we propose two special cases of local interaction game to study this problem. The first is local altruistic game, in which each user considers the payoffs of itself as well as its neighbors rather than considering itself only. The second is local congestion game, in which each user minimizes the number of competing neighbors. It is shown that with the proposed games, global optimization is achieved with local information. Specifically, the local altruistic game maximizes the network throughput and the local congestion game minimizes the network collision level. Also, the concurrent spatial adaptive play (C-SAP), which is an extension of the existing spatial adaptive play (SAP), is proposed to achieve the global optimum both autonomously as well as rapidly.
---
paper_title: Decision-Theoretic Distributed Channel Selection for Opportunistic Spectrum Access: Strategies, Challenges and Solutions

paper_content:

Opportunistic spectrum access (OSA) has been regarded as the most promising approach to solve the paradox between spectrum scarcity and waste. Intelligent decision making is key to OSA and differentiates it from previous wireless technologies. In this article, a survey of decision-theoretic solutions for channel selection and access strategies for OSA system is presented. We analyze the challenges facing OSA systems globally, which mainly include interactions among multiple users, dynamic spectrum opportunity, tradeoff between sequential sensing cost and expected reward, and tradeoff between exploitation and exploration in the absence of prior statistical information. We provide comprehensive review and comparison of each kind of existing decision-theoretic solution, i.e., game models, Markovian decision process, optimal stopping problem and multi-armed bandit problem. We analyze their strengths and limitations and outline further research for both technical contents and methodologies. In particular, these solutions are critically analyzed in terms of information, cost and convergence speed, which are key concerns for practical implementation. Moreover, it is noted that each kind of existing decision-theoretic solution mainly addresses one aspect of the challenges, which implies that two or more kinds of decision-theoretic solutions should be incorporated to address more challenges simultaneously.
---
paper_title: Opportunistic Spectrum Access with Multiple Users: Learning under Competition

paper_content:

The problem of cooperative allocation among multiple secondary users to maximize cognitive system throughput is considered. The channel availability statistics are initially unknown to the secondary users and are learnt via sensing samples. Two distributed learning and allocation schemes which maximize the cognitive system throughput or equivalently minimize the total regret in distributed learning and allocation are proposed. The first scheme assumes minimal prior information in terms of pre-allocated ranks for secondary users while the second scheme is fully distributed and assumes no such prior information. The two schemes have sum regret which is provably logarithmic in the number of sensing time slots. A lower bound is derived for any learning scheme which is asymptotically logarithmic in the number of slots. Hence, our schemes achieve asymptotic order optimality in terms of regret in distributed learning and allocation.
---

<format>
Title: A Survey of Decision-Theoretic Models for Cognitive Internet of Things (CIoT)

Section 1: INTRODUCTION
Description 1: This section provides an overview of the Internet of Things (IoT) and introduces the concept of Cognitive IoT (CIoT), emphasizing the need for intelligent decision-making capabilities in IoT systems.

Section 2: MOTIVATION AND REAL WORLD SCENARIO
Description 2: This section discusses the inefficiencies in current CIoT applications and presents a real-world scenario to highlight the benefits of multi-agent decision theoretic models for intelligent decision-making.

Section 3: NOVEL FRAMEWORK FOR CIoT
Description 3: This section outlines the proposed hierarchical framework for CIoT, which includes a new decision-making layer and describes the functionalities of each layer in the framework.

Section 4: DECISION MAKING FOR CIoT
Description 4: This section analyzes the suitability of different decision-theoretic models (Game Theory, Markovian Decision Process, Optimal Stopping Problem, and Multi-Armed Bandit Problem) for CIoT applications, detailing their components and applications.

Section 5: COMPARATIVE ANALYSIS OF DECISION THEORETIC SOLUTIONS
Description 5: This section compares the four decision-theoretic models in terms of information selection, performance, and convergence speed, providing insights into their practical implementation in CIoT.

Section 6: OPEN ISSUES AND FUTURE DIRECTIONS FOR INTELLIGENT DECISION MAKING IN CIoT
Description 6: This section highlights the open challenges and issues in CIoT, and outlines future research directions aimed at developing practical, scalable decision-theoretic solutions for large-scale CIoT systems.

Section 7: CONCLUSION
Description 7: This section summarizes the key points discussed in the paper and reinforces the importance of decision-theoretic models in enhancing the interoperability and efficiency of CIoT systems.

</format>

The outline: