Hello and thank you for joining this talk on the Security Issues and Challenges in Deep Reinforcement Learning. I'm Vahid Behzadan, I'm an Assistant Professor of Computer Science and Data Science at the University of New Haven, and I also direct the Secure and Assured Intelligent Learning Lab, or SAIL for short, working on AI, safety, security and applications of machine learning to cybersecurity and safety of complex systems. So, here's the outline of my talk. I'm going to quickly go over the basics of Deep Reinforcement Learning and Reinforcement Learning. There will be some math, but I'll keep it to a minimum. Then we'll talk about vulnerabilities of Deep Reinforcement Learning and whether DPRL is susceptible to classical adversarial machine learning attacks like adversarial examples and such. We'll talk about and develop a threat model for Deep Reinforcement Learning. We'll identify different attack models, attack surfaces. We'll identify different types of vulnerabilities. And we'll introduce a number of attack mechanisms and corresponding defenses that have been developed in recent years. And we'll talk about the frontiers and areas of future research and work in this area. So, just a quick overview. I assume most of the audience here is already familiar with the terminology used here. We can classify machine learning algorithms as supervised, unsupervised and reinforcement learning algorithms. These supervised learning algorithms are those where the training dataset includes labeled data, meaning that each data point in that dataset comes with the correct label or the correct output expected from a model trained on that dataset. Then there is unsupervised learning where classification, where clustering and anomaly detection and some other algorithms fall under where there are no labels available for data points. There's no feedback available on the data. However, the goal is to find some underlying structure in the data. And then there is reinforcement learning, which is concerned with the problem of sequential decision making. There's a reward system included in reinforcement learning, but there are fundamental differences between the settings of reinforcement learning and supervised learning. And that's why it merits its own category. We'll talk about those differences in the next few slides. So, you can see the general settings of reinforcement learning. We're in general sensory motor agent problems and settings. In RL, we have an agent, which is to be trained by a reinforcement learning algorithm. This agent interacts with the environment by performing some action. This action causes the state of the environment to change. Then the new state is observed by the agent. There's also some sort of reward associated with this change that is provided or inferred by the agent based on this change in state. Well, you can think of this in terms of playing a game. The environment can be the game environment where different actions may result in scoring or loss of score. And the actions are, well, essentially the actions that the player can take. The state of the environment can be the state of the game. For example, in Breakout, the configuration of these breaks can be one of those where the agent is, where the ball is. If you recall the game of Breakout, of course, if you're old enough to remember Breakout, you're probably familiar with the dynamics. But you can think of any other game and this setting still applies. What's the goal here? The goal is to learn how to take actions in order to maximize cumulative rewards, not instantaneous rewards, but cumulative rewards. So an RL agent is not just concerned with maximizing the current score. It wants to learn how to act so that at the end of the game its total sum of rewards is maximized. What are the different applications of RL? This game playing is well publicized because it's a very good testbed and it has become one of the common testbeds and experimental settings for RL research. However, there are some major real-world applications for reinforcement learning. In essence, RL, or reinforcement learning, is the machine learning response to the need for data-driven control problems. If we encounter such problems in robotics, like autonomous navigation, object manipulation and such, algorithmic trading, this is one of the areas that my research group has recently become active in. Critical infrastructure, controlling smart cities, resource allocation in smart cities, traffic management, intelligent traffic systems, smart grid management, and control healthcare, such as clinical decision-making. This is actually one of the better-known applications of RL in the real world. Other types of resource management and applications in operations research and such. RL is either envisioned or is already heavily adopted in various industries, many of which are critical and may become targets of malicious actions. Let's formalize the RL problem a little more. I'm just going to quickly go over this. There will be some math, but this is just to introduce the basic settings. The underlying framework to formulate the RL problem and provide a framework to think about and reason about the RL problem is the Markov decision process. Why do we call it the Markov decision process? Because it's based on the Markovian assumption or the Markov property, which says the current state completely characterizes the state of the world. So if you have the current state and you perform an action, the next state is only going to depend or be a function of the previous state and nothing before it. You don't need to know the history of the environment. All you need to know is what the previous state has been to infer what the coming state is going to be. Now, in Markov decision processes, a problem or setting is formulated by a tuple of five main parameters. One is the set of possible states, S, set of possible actions, A, distribution of reward given state action pair or R. This distribution can also be a function. It doesn't necessarily have to be probabilistic. It can be just a function that tells you if at a particular state, S, I, an action, A, I is taken, then the reward is going to be, let's say, R, I. Then there is transition probability or transition dynamics, P, which represents the dynamics of the environment. Which state are we going to end up in if we are in a particular state and we perform a particular action? And there is finally a discount factor which defines how myopic the agent is. How much it values rewards that may occur further in the future, further down the line. Now, in general, the RL setting is based on the following process. At time step T equals zero in the beginning. The environment samples initial state S zero, then for T equals zero, this is a loop. Until done, the agent selects action AT based on some criteria. It can be completely random in the beginning and then slowly becomes more targeted and more policy driven. It performs some action. The environment produces a reward signal and the next state and the agent receives those signals. It may be partial, it may be incomplete or it may be noisy but the agent receives some signal resulting from the transition to the new state ST plus one and a reward that has emerged from that transition that has resulted from that transition. Now we define a new entity here called Pi. A policy Pi is a function or a distribution in the probabilistic case that maps from state to action. So it tells the agent what action to perform given any state. And the objective is to find the optimal policy Pi star that maximizes the cumulative discounted reward. We also call this return. So cumulative meaning the sum of rewards in the entire duration of interaction. It can be one episode of a game, it can be throughout the training period or the training horizon and it's discounted. This gamma is the discount factor that defines how myopic agent is or how much the values events that occur further down the line. It's typically a constant value between zero and one. And as you can see as T increases this gamma to power T decreases. So the value, the preference or the observed value of something that happens further down the line which means with greater T is going to decrease as T increases. So again the objective is to find the optimal policy Pi star that maximizes the sum here. Alright we are going to quickly define two other definitions here. If we want to evaluate a particular state in these settings, one approach is to measure the value of that state using the value function. The value function of state S is the expected cumulative reward from following the policy Pi from this state. So let's assume we already have a policy. What will be the expected cumulative reward if you start in state S and keep following policy Pi? Now we also define a Q value or a Q function which tells us how good is a particular action A if it's performed in state S. In other words, if we are in state S, if the agent is in state S and performs action A, the Q value of this setting is the expected cumulative reward from taking action A in state S and then following the policy. Remember the policy is a function that tells us, tells the agent what action to perform given any state. If the agent performs action A in state S, it goes into a state S prime or the next state and then we can use this policy function here to see what the action should be, what action should the agent take at that state which takes it to another state and then the policy is used to figure out the action to take in that state and this goes on until the termination of the episode or the horizon. Now that we are familiar with the value function and the Q function, let's go a bit deeper. The policy Pi, the value function, value functions both V and Q and model, the transition dynamics or the reward model are all functions. We want to learn at least one of these from experience. This is the essence of RL. If there are too many states, however, we cannot just tabulate everything and try to experience every state and the corresponding result from that state or that state and any action performed because as a size of the problem, as a state space and the action space increase in settings like let's say playing GTA 5 or a self-driving navigation policy in the real world, these state spaces just explode. The dimensionality is too high and it's just not feasible to store every possible state. In those cases we need to approximate. In general and traditionally this is called RL with function approximation. If function approximation is done using deep neural networks then we call the training setting or the RL setting deep reinforcement learning. The term is relatively new. It was, I believe, introduced in late 2013, early 2014 by Nie and David Silver and others at DeepMind in their deep Q learning paper. However, the concept is not very new. That said, there have been fantastic and even mind-blowing advances in this area in the past three or four years. We'll talk about some of those as we move forward. A quick overview of the taxonomy of different RL approaches and RL agents. Remember we have value function, policy and model. There are different approaches to solving the RL problems. Sometimes we just want to find a policy directly. The approaches that respond or satisfy that need are called policy-based RL. Sometimes we want to find a Q function or a V function first and then derive the policy from those functions. These approaches are called value-based. Sometimes we want to first learn a model of the environment and then solve the problem. These are called model-based approaches. Sometimes we don't have the model and we don't want to learn the model explicitly and those are called model-free. When we're dealing with both value function and policy at the same time, in other words, we have one agent learning the value function and another learning the policy and then contrasting those with each other in a zero-sum setting. We call those active-critic RL approaches. As you can see, there are different approaches to different settings and different problems. However, all of those can still be grounded on top of the Markov decision process framework and the general solution approach we looked at before. One of the better known approaches to RL, which falls under the value-based approaches, is Q learning. The objective in Q learning is to derive the optimal policy pie star based on optimal Q. The optimal Q is one that maximizes the value of each state and action. From that point onward, with an iterative formulation based on Bellman equations or dynamic programming, it becomes possible to find this through bootstrapping and iterative re-estimation of the Q value. There are different approaches to Q learning in large state spaces or large action spaces where function approximation is required. One such approach is to parametrize Q as SA and an estimation parameter theta. If we solve this parametrized Q function by neural networks where theta corresponds to the waves of that neural network, the approach is called Q networks. This solution approach was proposed very early in 2000s, even earlier, but was perfected in some sense in 2013-2014 by David Silver and his team, which resulted in the proposal of deep Q networks, or DQNs. We're talking about deep networks. What does that mean? It means that these networks use deep neural networks like CNNs, which help with both function approximation and also end-to-end feature learning. One of the advantages of deep learning is its superior performance in learning feature representations, especially from images. For those of you who are more statistically oriented, you may have seen the term IID. It means that the data are independent and identically distributed. It means that one data point does not depend on another data point, and also each data point is equally likely to occur. These do not hold for the RL settings. We know that a particular state, for example, is highly correlated with its previous state and action. Why does this matter? Well, a lot of our supervised and deep learning approaches are based on the assumption that the data, the training data, is IID. When this is not possible, what happens is, in response to this problem, what happened was the DQN approach introduced experience replay. It's like a bag of all data, which is randomly sampled in each training iteration to reduce the correlation, the sequential correlation, the temporal correlation of data points and also make it more likely for data to be evenly distributed. Also, to reduce the effect of oscillation during training, DQN uses fixed parameters for a target network. The target of optimization is fixed and is updated every few thousand iterations, so that reduces the oscillation problem. Of course, the rewards are normalized to minus one to one to make sure the performance, the reward signals are bounded. Now that we have a preliminary understanding of deep reinforcement learning and one of its implementations, one of the parallel approaches called DQN, let's take a quick look at adversarial machine learning. I assume that by now, those of you who are not familiar with adversarial examples have been introduced to this concept. The idea here in adversarial examples, we are now talking in the realm of, we are speaking in the context of image classifier supervised learning. Let's say we have an image classifier trained on a set of images of pandas, cats and others to identify the object in those images. In the beginning, we passed the image of a panda to the classifier and it correctly classifies it as panda. Now it's been demonstrated, it's been actually established by now that it is possible to make the classifier to induce incorrect classifications in deep learning models or machine learning models in general by adding minute minimal perturbations to the original image. As you can see, it's almost impossible to detect or see any changes in this final image. These perturbations, these pixel perturbations are very small. Now, this is one example of different attacks or different vulnerabilities in the classical realm of adversarial machine learning, which is mostly concerned with supervised learning and sometimes unsupervised learning. In general, the adversarial objectives in AML or adversarial machine learning can be classified under the traditional CIA triad, confidentiality, integrity and availability. So with respect to confidentiality and privacy, an adversary may wish to target the confidentiality of the model parameters, model architecture, no intellectual property theft is an issue in larger models, or it may target the privacy, the adversary may target the privacy of the training and test data. If medical, for example, if medical records were used in the training data, there have been proof of concept attacks showing that it is possible to infer whether a particular patient or the records pertaining to a particular patient were used in the training data or not, or sometimes it's possible to reconstruct the training data set by just having access to the model itself. And that is a major hip violation in essence, it's a privacy violation. Also, with regards to integrity and availability, the attacker may target the integrity predictions or the outcome of the model, the performance of the model. For example, adversarial examples are an attack on the integrity of image classifiers or supervised machine learning models, and also an adversary may target the availability of the system that is deploying machine learning, for example, a facial recognition system or an autonomous navigation system in a driverless car. Now that we know so much about adversarial machine learning and security vulnerabilities of classical machine learning, supervised learning and unsupervised learning, there is a major question and it's whether deep RL is immune to those attacks. In late 2016 when the research community was just beginning to pay attention to both deep reinforcement learning and the issue of adversarial examples, I came up with this question and decided to experiment with it a little to find out whether deep RL can also be vulnerable to such attacks. I started from a simple observation. The deep neural networks in DQN models and classifiers are both function approximators. At training time, there are function approximators that test times are just functions. I came up with this hypothesis. If classifiers are vulnerable to adversarial examples, then action value approximators of DQNs may also be vulnerable. I started to set up an experiment where the aim was to perform adversarial attacks. The adversary's goal was twofold, test time attack to perturb the performance of target's learned policy. So the target at this point is fully trained and is deployed in the environment and the adversary wants to somehow cause the agent to perform incorrectly to manipulate its policy. What does the adversary know about the target? It knows the type of input to the target. For example, it knows whether the target policy is looking at image data, text, audio and such. Why? Because it helps with estimating the architecture. For example, if it's images, the adversary can come up with a good guess that the architecture includes the target architecture includes CNNs or convolutional neural networks. I also assume that the adversary knows the reward function. So the adversary may have access to the environment and if it's, for example, a game environment, it knows what the scores are, how the scores are generated. What is not known? The knowledge of target's neural network architecture is not known. So it's a black box attack. And also the initial parameters, the initialization of the neural network, the target neural network is also not known. What is available to adversary in terms of actions? The adversary may perturb the environment where the target performs. For example, it can change pixel values in a game environment through a man in the middle attack. I consider in this work two techniques for perturbing that environment. One is the classical fast gradient sign method for generating adversarial examples. And the other is the Jacobian-based saliency map attack or the JSMA approach introduced by Papernot in 2015, I believe. Now, in that experience experiment, I used the classical DQN approach, the classical DQN architecture introduced by Nie in an Atari game, the game of Pong. And this was all implemented in open AI gym with TensorFlow, back then PyTorch wasn't really a thing. And train the agent against heuristic AI. And here's the initial proof of concept result. This is for the white box attack. Of course, later on I will show you the results for the black box attack. And you can see that for FGSM and JSMA, regardless of how far along the training the agent is, both policies, the policy for the Pong agent is highly vulnerable to adversarial perturbations through simple techniques like FGSM and JSMA. You can see that for JSMA, the success rate was 100% for all of the cases. For FGSM, it was slightly lower. And it was mostly because of the termination criteria and the perturbation threshold that I had defined. But you can see it can be observed that the policy can be very easily manipulated through adversarial example attacks. So at 100 random observations perturbed with FGSM and JSMA, the results were fed. The perturbed images were fed to the trained neural network representing the agent's policy as test input, and then the success rate was measured. Now, is this type of attack practical? Is this really something that we should be worried about? So a few years later, in 2018, Clark and his co-authors published the report of a similar attack on an autonomous robot based on DQN policy using ultrasonic collision sensory input for collision avoidance. And they had shown that they can use adversarial perturbations to manipulate the trajectory of the robot and make it follow a path defined or desired by the adversary, not one that the robot itself wants to follow. There are more recent examples of how this sort of attack on Dparl or in general attacks on Dparl can be of concern. One of the recent works by my graduate students is on attacks on automated trading algorithms based on reinforcement learning. The paper is coming out in a couple of months, so I can't go into more details, but this is one of the more severe and urgent cases for security researchers to consider. Dparl is already being used by many major financial players and stock traders, and it can be easily manipulated in the real world. There are other cases, of course, but this is one of the examples that demonstrates the practicality and the applicability of this attack to real-world scenarios. Now, before we go further into different types of attacks, let's develop a threat model for deep reinforcement learning. Again, the adversary's objectives can follow those of the CIA triads. The adversary may wish to access the internal configurations like model parameter, reward function, policy and such to still the model for intellectual property theft and such. There can be an attack on integrity, which means compromising the desired learning or enactment of the policy. There can be attacks on availability, which are essentially compromises of the ability of the agent to perform training or actions when needed. Now, let's look at the attack surface of Dparl. This is the general block diagram of a Dparl agent or Dparl system. Now, we have the agent. The agent typically has some memory where it stores its experiences during training and then those experiences help with function approximation, data-driven policy learning and such. And then there is an exploration controller, which controls how the agent explores the environment during training. There is an experience selector, how to select experiences from the bank of data or observations stored in this dataset. There is an actuator, which then enacts the actions of the agent inside an environment. The environment is connected back to this agent block through an observation channel, observation of the state, and reward channel. And it's of no surprise to the more seasoned security researcher and professional that all of these components can be a subject or target of adversarial attack. As we go forward, we'll cover some examples of attacks that can occur on each of these components. We've already seen an attack on the observation channel. There are attacks on the reward channel, which we'll hopefully touch on. And there are attacks on the agent during training and the actuator. Let me give you a quick example of the actuator attack. Let's assume we have a robot, an actual robot, learning to navigate in an environment while avoiding obstacles. If the robot commands or decides to, let's say, move the left wheel forward, but there is some sort of obstacle in front of the left wheel and the left wheel doesn't actually move, then the resulting observation is going to be skewed because the agent is going to assume that the actuation has happened and then look at the changes in the observation and use that to retrain its policy, to optimize its policy based on faulty data. What are the adversarial capabilities? Well, we first look at different attack modes. The attacker can perform a passive attack where it's only observing the target, it's not changing anything, or it can perform an active manipulation. In passive attacks, the attacker can perform inverse reinforcement learning to learn about the reward function of the agent, or later on we'll see it can perform imitation learning to steal the policy. Active measures include attacks on the actuation, observation, or the reward channel, and attacks on observation can be targeting the representation model, how the agent sees the environment, or perturbing the transition dynamics, how the agent sees the changes in the environment. Okay, going back to our initial proof of concept, remember our original goal was to perform a black box attack. So, to achieve this objective, we introduce an approach based on the transferability of adversarial examples. So, we create a second DQN. The adversary creates a second DQN with similar architecture, but different initial parameters. When I say similar, it doesn't necessarily have to be imagined, it just needs to be a convolutional neural network with some functional approximation techniques, but it doesn't need to have the same parameters or the exact same architecture. And it trains that agent, that model on the same environment. The assumption is the adversary has access to that environment, and then uses the knowledge of that architecture and the trained replica policy. It crafts adversarial examples the same way it did for the white box case, and we know that many of those adversarial examples can transfer to similar models trained on similar data, which also you can see applies to DQN policies as well. And this is how we implemented a black box attack against D-parallel policies. Now, what about training time attacks? In the same paper we introduced the policy induction attack, where the attack is of the adversarial example type against a DQN agent during training. Now, what are the different steps in this attack? First, the adversary derives an adversarial policy from the adversarial goal by training on the same environment where the target is going to perform or be trained in. So if the adversary wants to minimize the reward gained in a game by the agent, then the goal, the optimization goal is going to be the exact opposite of that of the target policy. The target wants to maximize the reward and the adversary wants to minimize it. There are of course different ways of formulating this adversarial goal. Then the adversary creates a replica of target DQN and initializes it randomly, and then comes the exploitation phase, where the attacker observes the current state and transitions in the environment, then estimates best action according to the adversarial policy derived in step one, then the attacker crafts perturbations to induce adversarial action based on the replica of target DQN. This is exactly the same as our black box test time attack. The attacker applies the perturbation as a man in the middle in the observation channel. The perturbed input is revealed to the target and the attacker waits for target's actions. And this is a loop. This loop can go on until the target, either the training process of the target, either converges to a suboptimal policy or up to a certain number of iterations. This is a very rough plot. This is not smoothed yet, but you can see that the unperturbed agent moves towards, this is in the Game of Pong style, moves towards convergence to an optimal total sum of rewards, while the attack agent moves towards a convergence to the minimum possible return of zero. It's getting closer and closer to zero, which indicates that the training process of DQN and D-PARL in general can also be targeted through adversarial attacks. Now I'm going to introduce another type of training time attack. Again, this aims to induce some form of misbehavior. We call this misbehavior addiction, and this is a follow-up to a work I did with my colleague Roman Niempolski on psychopathological modeling of AI safety problems. This is a proof of concept. We consider the Game of Snake, many of you probably remember Snake from older Nokia phones. The DQN agent is a snake and is learning to play in this environment. What the attacker does is it adds a drug seed with more instantaneous reward than the typical seed, but it also results in more increase in the length of the tail, and you can see that this can end up, well, if the increase in the tail length is more than a certain amount, then a longer-tailed snake is bound to eat its own tail sooner rather than later. So we show that we actually derive some theoretical close-form solutions for what the additional reward and increase in the tail length should be for addiction to emerge, meaning that the agent learns the more myopic policy instead of the optimal policy, and you can see that it's actually possible to make the agent addicted to the drug seed, as we call it, and this results in learning a suboptimal policy. And, of course, due to time limitations, I'm going to introduce only one more type of attack, and it's that of targeting the confidentiality of a depraved policy. The problem here is, or the question is, is it possible to extract a depraved policy from observations of its actions? Why does this matter? Well, the security challenge posed by this sort of action is, of course, Model Theft, a company like Google or, let's say, Uber or Waymo may have spent billions or millions of dollars on coming up with a very accurate depraved policy for autonomous navigation, if it can be stolen by an adversary, then the intellectual property becomes worthless. And also, a stolen policy, an extracted policy, can be leveraged in integrity attacks in the same way that we mounted black box attacks on DQM policies. So, let's see. As it happens, a branch of reinforcement learning, or in general, one solution to the sequential decision-making problem is not in the URL domain, but in the supervised learning domain, and it's called imitation learning. Imitation learning is the supervised learning of policies from observed behavior of an expert, and by behavior on in-state action behavior. What is the policy of an expert? Based on this concept in 2018, Hester et al. proposed DQF FD, a DQ learning from demonstrations, which is DQN, where the initial training is done based on observed data using deep learning. So, they have data from human players playing a certain game, or human performance doing a certain task that they want the agent to learn. The initial step of training for a DQFD agent is supervised learning on observed data, and then it starts building on top of it through reinforcement learning approaches, and it was shown that it can result in faster convergence, better sample complexity, and sometimes more interesting and robust policies. As security researchers, you can probably see where this is going. This wonderful algorithm, DQFD, can also be used to replicate policies instead of applying it on observed data collected from human performance. It can be applied on observed data from a target policy. So, here's a proof-of-concept attack procedure. The attacker observes and records and interactions of the sarsotype, state action, the next state, and the reward based on this transition of the target agent in a particular environment. And then the attacker applies DQFD to learn an imitation of the target policy and Q function. Now, at this point, the attacker may either just go away and sell the extracted policy, or it may decide to target it using different adversarial perturbation attacks, some of which we've covered so far in this talk. So, as a proof-of-concept, we consider a slightly less complex environment, that of cart pull, where the objective is to stabilize this pull on this cart by moving the cart to right and left. The reason for choosing this simple environment is merely economical, because we didn't want the experiment to take days or weeks. We start with a simple case, and we consider different types of policies. DQM with prioritized replay, a enhanced version of the classical DQM proximal policy optimization, and asynchronous actor critic. And, of course, we also train an adversarial RL agent, a DQM agent, whose objective is to incur maximum loss of reward, or in other terms, in more technical terms, maximize the regret of its target. And here are the results. First, with regards to replication progress, based on only 5,000 demonstrations, 5,000 state action, next state reward observations, we see that all three policies are almost exactly replicated. We can see convergence to the optimal performance of those policies in the environment.  We train an adversarial RL agent to attack and maximize the regret of those policies. And you can see that this can also be easily achieved within very few iterations of training in cart pull. I believe we can see that for PPO2, which is a somewhat robust DPRL algorithm or approach, it's possible to incur maximum damage or find a policy that incurs maximum regret on the target within 60,000 iterations, which is a relatively low amount. Now, what about defenses? So, of course, similar to adversarial examples, one approach or one technique for reducing the impact of adversarial example attacks is through regularization. And one common type of regularization in supervised learning for mitigating adversarial example attacks is adversarial training, training the model on adversarily perturbed samples to make sure that it sees different perturbations of the same image and knows that all of those results in the same label, in the correct label. This is called essentially data augmentation in general as a regularization technique. So in 2017, when I was just starting to look into this problem or this domain, I had gone through the adversarial training literature and thought that the same may also hold true for DPRL. I made a hypothesis, actually two. One is with regards to recovery. If training time attacks are not continuous, if not all of the observations are perturbed, then DRL adapts to the environment and adjusts the policy to overcome the attacks. This is training time. And with regards to robustness, I made another hypothesis, such policies, policies trained under attack are more robust to test time attacks. And this particular investigation is published in a paper titled Whether It Does Not Kill, Deep Reinforcement Learning Makes It Stronger. So, similar as before, we are looking at DQN in Atari games, Breakouts, Enduro, and Pong. Now, the way I designed the experiment was based on the probability of attack. So as an attacker, I assign a certain probability for each state, for each observation during training time to be perturbed. I perform different experiments with different values of this P attack, 20%, 40%, 80%, and 1, which means continuous attack. And it's interesting to see that for values of P, less than 50%, the agent actually recovers. But for values greater than 50%, the training process plummets and either does not converge or converges to a very, very low mean return value, or mean total reward value. I later on publish a theoretical analysis of why this happens. It's available in my PhD dissertation, which I'll reference in the final slide. Also, it was interesting to see that the robustness hypothesis is also true. And we hear that after training, if we attack the test time policy with probability of 1, the plane, or vanilla policy, a policy that was not trained adversarily, performs very poorly. However, policies trained under adversarial attacks with probability 0.2 or 20%, and 0.4 perform really well. For 80%, it's even for policies trained at 80%, as you can see, the policy itself is already performing very poorly. It's surprising to see that at P equals 1, the performance gets slightly better. It's comparable with 40%. To this day, I'm not entirely sure why this happened. I've repeated the experiments a number of times and still get the same result. I still don't know why this happened. This is one of the interesting problems that we are looking at right now. As a research group. Another difference that we introduce is based on parameter space noise. This is very much like dropout. The idea of parameter space noise was introduced in 2017, I believe independently by Plamper et al. and Fortinotto et al. And the idea here is, again, similar to dropout, to introduce zero mean random noise to the learnable parameters of neural network in deep RL to enhance exploration and convergence in deep RL benchmarks. Now, in another paper in 2018, we investigate whether this approach can be used to mitigate the impact or severity of policy manipulation attacks on DQM. And it was shown that it actually performs very well compared to vanilla or classical DQM. And these are the training time results. You can see that if parameter space noise is used, the performance degradation for all environments is at a much lower slope, at a much lower rate than the vanilla architecture. Finally, to have proposed a solution for the policy extraction problem, I along with William Hsu at K-State came up with the idea of watermarking the RL policies. Watermarking has already been introduced in deep learning in general. The idea is to come up with a unique signature that is both difficult to remove and does not impact the performance of the policy itself or the model itself. But it still provides a unique signature, which proves that a model is the same as another model or is a replica of the suspected model. So we introduce an interesting watermarking procedure. I know it's arrogant of me to call my own work interesting, but I still get excited when I think about the moment I came up with this idea. The idea is to create a second environment whose state space is disjoint from the main environment. So create an environment. If you're training an agent to play a game, let's create another environment which has no states. None of these states are the same as the original training environment or deployment environment of the agent. But the dimensionality of the states are the same. So if each state in the original environment is represented by, say, three values, three features, then each state in the second environment is also represented by three features. It doesn't really matter what the second environment looks like. It's just some other environment that the agent may interact with. And then we craft the transition dynamics and reward procedure for the second environment such that the optimal policy follows a looping trajectory. So an optimal policy for an agent trained in the second environment is going to be one that follows a loop, goes to, let's say, state one, then state two, state three, and then goes back to state one. During training, what happens is we periodically alternate between the two environments. So let's say at every end iterations of the training process, we take our RL agent from the original environment, we drop it in the second environment, train it for a few iterations, and then bring it back to the original environment. Now, once trained, if you want to examine the authenticity or whether a policy is copied or not, we apply the policy in the second environment and measure the total reward. Here's the experimental setup. Again, we are working with cart pool. So the watermarking environment is defined with five states, states one to four plus a terminal state, which should never be reached if the policy is optimal in this environment. And none of these states, as represented here, can be found in the original, can occur in the original cart pool environment. These are all highly impossible, not highly, definitely and absolutely impossible to occur in the original environment. As for the transition dynamics, this is how we've defined it. Let a particular state be A0 and A1, if we are, I'm sorry, actions, actions A0 and A1, if the agent is in state I and performs action I modulo two, if I is even, this is going to be action zero, if I is odd, this is going to be action one, then the next state is going to be state I star or I modulo four plus one, and it receives a reward of one. If the agent performs any other actions, so instead of going from one to two, it performs any other action, it will immediately go to the terminal state and receives a reward of zero. So the optimal trajectory is state one to two to three to four, back to state one and so on. All right, let's see how it works. Let's look at the test time performance comparison of watermark and nominal, non-watermarked policies. The watermark policy performs exactly as well as on-watermark policies. It reaches the optimal or best performance of 500 and when it's applied to the watermark environment, the second environment, it also performs optimally. It gets the maximum reward possible. However, you can see that if we try to apply the non-watermarked policies to the watermark environment, we'll see very, very small values of score or total reward. So you can see that it's possible to determine whether a policy is authentic or not or whether a policy is an exact copy of another policy by just applying it to a second environment and see whether it performs optimally or not. Now there are many other things that I really wanted to touch upon in this talk, but unfortunately we are a little short in time. For practitioners, it may be of interest to have some way of benchmarking or evaluating the resilience and robustness of policies and compare different policies, different approaches with regards to their resilience and robustness. Some of my work already introduces or proposes an RL-based approach to perform this evaluation and benchmarking. I've also done some work on investigating the impact of hyperparameter choices on resilience and robustness of DQNs in particular but also other model-free and active critic approaches. This can be very helpful to those who want to engineer and design the new RL agents to be deployed in critical environments. Also, something that I wanted to mention but unfortunately I don't have time to do so is that adversarial training is not a silver bullet. It's not an answer to all of the problems in DQN agents or certain limitations for robustness and resilience obtained from adversarial training of DQN agents. Also, adversarial training is very costly in general, especially when it comes to real-world scenarios, real-world environments and actions. Some of my recent work is focused on improving the sample efficiency and computational cost of adversarial training via a new exploration mechanism called adversarial-eguided exploration or H. All of this work can be found in my PhD dissertation which bears the same name as this talk, false and apply star security of deep reinforcement learning. You can find it if you search my name on Google Scholar. If you're interested, of course, all of these are published in separate papers in slightly more details under the same titles. And finally, some of the open areas of research in this domain. With regards to training time resilience and robustness, not much has been done with regards to policy search and actor-critic methods as well as model-based and hybrid methods. Of course, when we talk about model-based, there are some approaches from optimal control theory and approximate dynamic programming that may be applied here, but very few have looked at this problem from a security point of view. So if you're interested, this is one of the areas that is in dire need of security-oriented investigation. As for mitigation of policy replication, one of the ideas that my research group is currently working on is constrain randomization of policy. So randomize the policy such that the replication through techniques like imitation learning becomes more costly, more samples, more observations will be required while preserving the performance of the policy. There is almost no work done in multi-agent settings. Of course, adversarial reinforcement learning has been investigated in settings where there are zero-sum agents, but not really where there is an external adversary or adversaries trying to exploit the inner workings of the agents, the RL components of the agent. One more thing that is of note is the importance of discounting. The addiction problem that I demonstrated earlier in the Snake agent is mostly due to the constraint discounting solution. For those of us who come from a reinforcement learning background and are familiar with the basics of reinforcement learning, you probably know that the discount factor is typically chosen to be 0.99 or something and the same ballpark has left the same. It's treated as a constant throughout the training process. But this is very far from how our brain works and very far from the optimal approach or accurate approach to discounting. Our research group has recently started looking into this problem and is working on developing adaptive discounting solutions to enhance the resilience and robustness of RL agents, particularly deep RL agents in complex environments for AI safety and security purposes. Now, of course, our naturally-inspired approaches that can be looked at, for example, approaches coming from, let's say, TD Lambda models or dopamine models of psychopathological problems or neurological problems and the solutions prescribed to those, as well as approaches in social sciences which may help with the security problems arising in multi-agent RL settings. Very well, thank you very much. And I believe that this time it should be available for your questions.


A cytoskeletal protein might be involved in promoting breast cancer by making cancer cells acquire some features of stem cells. Researchers have found that fassin, a protein that bundles together the actin filaments that serve as the backbone of cells, is important for keeping breast cancer stem cells alive. The results shed light on what is going on inside cells as they break from normalcy to become cancerous, and highlight fassin as a potential diagnostic marker and therapeutic target. One reason why cancer can be so hard to treat is that certain tumor cells are resistant to chemotherapy and can sell for new, providing an endless supply of cells to reseed tumors. They're known as cancer stem cells because they hijack several properties of stem cells, like expressing transcription factors that allow for cell renewal to push their cancerous agenda. Fassin is found in higher amounts in tumors and has been linked to breast cancer chemoresistance and spread. Two key features of cancer stem cells, raising the possibility that this protein might promote cancer by regulating cancer stem cells. To find out if that's the case, researchers first experimentally lowered the amount of fassin in a human breast cancer cell line. When treated with chemotherapy, they found that these cells died more readily than those with the protein. The surviving cells, which were more common among cells with fassin, were more likely to be cancer stem cells, expressing high levels of certain markers. The reverse experiment, giving cells extra fassin, allowed more cells to survive chemotherapy and increased the number of apparent stem cells. When the scientists checked for stem cell-related transcription factors, they saw a similar trend. The cells without fassin had reduced amounts of cell renewal factors, while cells with extra fassin had higher amounts. The researchers then looked to see if fassin was associated with cancer cell invasion and spread. As cancer cells prepared to spread, they become more spindle-shaped and mobile. In a cell model of this transition, scientists found that fassin levels increased, while removing fassin reversed this process. Fassin was also found to be critical for tumor formation in the Petri dish, strongly supporting fassin's role in regulating cancer stem cells. But the real test is whether fassin can help cells self-renew and create tumors in animals. To find out, researchers injected small doses of breast cancer cells into mice with compromised immune systems. Just 50 cells with fassin were needed to produce tumors in a third of the mice. Without fassin, though, none of the mice developed tumors with so few cells. How might fassin be doing all this? Notch is an important signaling pathway that controls cell fate in the breast. The scientists found that in cells without fassin, notch signaling was low, and that that signaling was necessary for robust tumor formation in the Petri dish. In fact, blocking notch signaling made cells with fassin behave as if they didn't have the protein. They were less able to form tumors and more likely to die after chemotherapy. The findings suggest that fassin makes it easier for cancer stem cells to form tumors and stay alive by activating the notch cell for neural pathway. Paying attention to fassin levels in patients might be a valuable way of monitoring cancer and personalizing therapy. And one day, targeting fassin with a drug, in combination with other chemo, might even provide a more effective way to treat the disease.


What's going on, everyone? Sam here with laying swords cards. Refin 22 for any select football hobby, three box break, number 18, random team style. There's a look again at your random results. As always, you can find those on the websites, under the results tab, as well as the link in chat. Good luck. See, this is Coma Reneers right here. That's Sam the Salmon. That's why I bought this. Really? It's pretty cool. I like the hat. Sam the Salmon right there. Right when I saw that, I was like, purchase. Let's just dump it out. Algeer, K-Von, Tibido, Silver, Malik Willis. Silver, Kadyrika Howe. Jalen Hurts, Otto, Kenny Pickett, to $199. Going to the Steelers. There we go. Congrats, Pittsburgh. Edward R. Congrats, Edward. Is KP coming your way? Luke Fortner, to $199. Brady, sweet level. Kenneth Walker, Silver, and Fred Warner. Got a full case of this coming up after. Silver, Herbert, sweet level. Ramondre, field level. Nice, artistic selection. Super short print. That is Jalen Hurts. Going to the Eagles. There we go, Philly. He's got Philly in this one. Greg R. Let's go, Greg. Congrats, buddy. I think you'd probably mag that one, too. Kenny Pickett, rookie. Silver, Prescott, and Logan Thomas. Blue, Friderian Mathis, to $149. Silver, John Mechie, and James Conn. Breece Hall, sweet level. John Taylor, Silver, Justin Fields, Chris Olave, ETN, Johan Dotson, 66 out of 75 on the Jumbo White Jersey. Nice one there for the commanders. Silver, John Taylor. Yeah, let's go, Snell. Snell liked that one right there. That's my reader, rookie. That re-killed. Father Algier, to $75. Superboloado, Dwight Freeney.  Going to the Colts. Congrats, Indianapolis. Such a beast. So good in its prime. Connecticut native. Dawson Knox, to $149. Jonathan Taylor. Silver, Minka, and Damien Pierce. Yeah, the Hurts was, Hurts is sweet. Tideye, field level. Brian Robinson, Jr. rookie for the commanders. That is the 25, one of 25. Silver, scary Terry. Silver, Donovan Wilson, and Jonathan Taylor. Gotta stay number one somehow. Box two. Silver, Drake, London. Silver, Brock, Perry, nice one there. Matt, Drew, Don. Yeah, Bobby, we've had a bad day. Best case of the day, Garrett Wilson, Silver rookie, was 23 Bowman Baseball. Bowman Baseball Sapphire. In one case, there was a purple Drew Jones base to 25, a green Drew Jones auto to 50, and a purple Drew Jones auto to a 10. All in the same case. Unreal, dude. Gold, Derek Henry, field level. That's sick. Five of 10, nice one there, Tennessee. Silver, Tyler Linderbaum, and Stefan Diggs. I've put myself in the charge since apparently I'm a bot. According to you. Traylon Burke, Drake, London rookie. There's a rookie signatures prism, Garrett Wilson. Go to the Jets. There you go, New York. Nice one there, New York Jets. Oh, let's go, Vila showed up today. Hell yeah. Pat Sertan Blue, C-U-99. Mack Jones, Silver, Kenny Pickett. That's a card that deserves to be where it is right now, Bobby. Definitely a card that is, that's the one I was telling you about with the Vila Jones Black Shield Arcade, that's Bobby. Bobby. Silver is Michael Pittman, and Patch Mahomes field level. Oh, another gold. Joe Burrow, white to 35, sweet level. That's gonna be 10 of 10, Justin Herbert. Gold Jersey Auto. Nice one there, congrats on that, Chargers. That's nasty, he's got LA. DW, let's go D. Congrats, buddy. Good half case right here. Good Lord. There's Tyler Lockett to 249. Roquan Smith, Silver, David, and Joku, and Petit Ferret. Zebra, as a Cleveland Brown, Miles Garrett. Going to the Browns, Joe C, congrats, man, that's sick. Miles Garrett on the Zebra. Silver, Damian Pierce, Coralinsa, and Dalvin Cook. Kenny Pickett, Silver, Roquan, and James Cook. And a green in this, geez, dude. That's gonna be five of five, Zach Wilson. On the Patch, nice one there on the color match for the Jets. Silver, Brandon, I can pat Frymuth. Select is so good, dude. It's really kind of crazy. Oh, Juju went down. Justin Fields to 75 on the Purple, the Bears. Silver, Cam Jordan, and Taken, Quinteriano. One sec, guys. Last box of the break, really good half case overall here. It's kind of crazy. Silver, Darnold, and AJ Dillon. That is a Coral Patterson to 149. Got 46 of 125, Javon Holland, Jersey Auto for the Dolphins. There it is. CJ Mosley. Again, guys, 17 to go on 2020 Bowman. There's an XRC. Silver, Davante Adams, Silver Channing Tindall, and Joe Mixon. Tight end one, Dalton Kincaid. Going to the Bills. Nice one there, Buffalo, Buffalo with the Dalton Kincaid. Silver, Kenny Pickett. Silver, Trey Hendrickson, and Gino Smith. There's a Matt Stafford to 75, Field Little. Silver, DJ Moore, and DeAndre Swift. Mike White to 199, Andrew Thomas. Silver, Tyler Smith-Rookie for the Cowboys. And two up. Tie-dive, some sort. Christian Watson is a filler. Ew, gross. David Bell, 25 out of 25. Tie-dive, two-colored RPA. Congrats, Browns. Very nice. Going to Joe C. Let's go, Joe. Joe. Bass. Is it still going? Damn. This has been on since we got here. It's going to be Desmond Bitter. Like, holy cow. Jermaine Johnson, Daniel Jones. Silver, Christian Wilkins, little Mack. Blue, David and Joku to 149. Brian Inger, Silver, Sam Howell. Nice one there for the Commanders and the Josh Jacobs. Two packs left here, everybody. Tyler Smith, 97 of 99. Trevor Lawrence, Jersey Guard, second year for the Jags. Silver, Derek Stingley, Jr. Anacor, Roel Patterson. Last pack. Going to be a red. Greg Dulcich, 45 of 49. A Silver, Alante, Taylor. And Akeem Hicks. Real quick recap. Really, really, really good three boxes. I mean, we averaged a mag per box, which is pretty solid. We had a Trevor Lawrence, Green Zach Wilson of five. Jahan Dotson, David Bell RPA. XRC Tide N1, Dalton Kincaid for the Bills. Javon Holland, Dwight Freeny, Garrett Wilson. We had Artistic Selection, Jalen Hertz. A Kenny Pickett, 2199. And Gold, Jumbo, Jersey Auto, 10 of 10. Justin Herbert. Yep, that's going to do it for the break, guys. Thanks again for the filp. Up next is Full Case of Select.


When you take your body through all this right now, you know, when you get tired in that fourth quarter, let's think about that time when you were training extra hard. Waking up every day, doing what we're doing, going out there, heating on each other. This is what it's training our mind for, and the mental aspect of it. Come with it, especially in the offseason when you're training super hard, you know, getting yourself ready for the season. The offseason is where all the hard work and, you know, even training camp, you pushing yourself. In the season, you're supposed to just let it loose, have fun with yourself.


Which one of you guys started the fire on YouTube with all the trolls? What? We didn't start the fire. Yeah. Yeah. It was always burning since the world of turning. We're going to blame it on Justin. Why? Which one? Good. The CrossFit one is going. Oh, I see. Of course. What do you mean? Which one? Of course it's going nuts right now. Yeah, of course. We knew that was going to happen. Yeah, you know what? That's where they hang out, apparently. You know what I find? You know what's funny to me is, OK. So I guess if I had to identify with a modality of training, it would be bodybuilding. If I had to. Right. If you had to put me in a category of how I train most often or what I gravitate towards the most, it would be bodybuilding. Which I find it funny that people get insulted and they feel the need to defend CrossFit when we talk about these things.  That's who I am. Right. I wouldn't feel the need to come rescue it. And we're like, well, that is so wrong, and you don't know what you're talking about, Sal. There's one thing to attack. Talk about drinking the cooling. Well, here's the deal. OK, you have to get more specific.  OK, but why? Why do you say that?  It's aesthetic focused. So you don't focus so much on mobility and on functional strength and movement. Those are actual good critiques. They can be very good critiques. Absolutely. And we talk about how many people get into the gym because they're insecure about their body. And I could make the case that training for aesthetics and using the mirror and the way you look as your main motivator is actually a terrible way for most people to train and most clients that I train. But I wouldn't get offended by that. No. I poke holes in every method that's out there. Look. Look. But only when we talk about that one. OK, this is what it reminds me of. Years ago, I don't know. This is probably, I think it was 1994, I want to say. I remember I was watching TV and this commercial comes on TV. And it's like this cartoon character. And he's like punching the ground. And it's like ultimate fighting championship. I remember that. Find out which martial art is the toughest. Boxing versus judo, taekwondo versus karate. And it was like this huge debate back in the day. When I was a kid growing up, you'd watch martial arts movies. And it was always kung fu versus karate or boxing. And it was this huge debate. Which style is the most effective? Who's going to use the five finger death punch? And you know we ended up figuring out through mixed martial arts that all of them have strengths and all of them have weaknesses. And the best.  The one that can do all of them. The one that utilizes the strengths and nullifies the weaknesses. OK, similar with training modalities. Does yoga have strengths that are superior to other forms of modalities? Yes. Does yoga have weaknesses? Absolutely. What about bodybuilding? What about powerlifting? What about crossfit? What about kettlebell training? All these things have things you can learn from and use together to construct the mixed martial art of training essentially for the average person. Now if you want to be special. Let me put it this way. Using the same martial arts argument. If you want to be the world champion at Shotokan Karate does it make any sense to train in wrestling or devote any time to wrestling? No. No, the sport of Shotokan Karate is specifically Karate. Your best bet is to focus all your time on Karate. So if you want to be the best crossfit athlete in the world makes perfect sense to go focus all your time across. Same thing with powerlifting, bodybuilding, etc. But if you want to train your body the average person would be fit, well rounded. Your best bet is to take a little bit from each one and of course you're going to lean more in one direction and other because of your preferences. If you like squatting and deadlifting a lot you're probably going to do more powerlifting than the other types of lifts or modalities. But you can take a little bit from each one. Yeah, but we've done what? 1,500 episodes and we've poked holes in every modality. Every modality.  But only when we talk about that one do we get this, the feedback is crazy. It's always people that get so butt hurt about talking about crossfit. There's a bit of a cult. A bit? Yeah.  Well, and here's the thing, because people will say things like oh, you know, I work out a crossfit box and the way that they train they teach technique and they focus a lot on form and it's very appropriate to the person's level and there's lots of individualization of the training. My question for those people is functional training. Yeah, well my question is always this what makes a form of training crossfit? What are the things that make it crossfit? Yeah, please define it. To my best estimation, and I'm pretty good. I know this. I understand crossfit quite well and I know you guys do too. To my best estimation, what makes something crossfit literally is the sport of crossfit. Training for the sport. Other than that, what they're utilizing are deadlifts, squats, presses, cleans, they're running, they're doing exercises and doing them all right doesn't make it crossfit. What makes it crossfit is when you make it the sport of crossfit. And even if you want to go into the modality of it, they've pulled from every other functional training method that already existed. So it's like this culmination of all these other training methods out there that does have legitimacy into it but what makes it crossfit is the intensity, it's the competition of it, it's the actual sport of it that differentiates it from everything else. I think it's the feeling of superiority that I love to just poke at because I think they think that many people that take it believe that. Now mind you, there's a lot of people that listen to mind pump also do crossfit, are in our community or in our forum that are not like that, that are like understand the points and they can say, hey, I like doing it. I love the community. I've been very consistent with it. I totally hear all the points that you guys make. I try and make adjustments into my routines and add mobility days and do things like that. But yeah, I totally get it but it doesn't mean we're telling people, don't do it. It's just when we think, when I talk on this podcast, the person I think I'm talking or who I'm trying to communicate to are the people that I trained for 20 years. Those people. Yeah, average person wants to get in shape. Yeah, yes. Not the supreme athlete who is trying to compete in crossfit. I mean, by all, you should be doing that if that's what you want to do. I'm talking about, you know, Susie who's 55, had three kids, tried to lose weight for the last 20 years. Yo-Yo dieted most of her life, which by the way, this is like 70% of the clientele that would come through the door of a gym that would hire a personal trainer. That's who I'm talking to. I'm not talking to you 20-year-old kid who's in great shape, great mobility and don't have weight issues. And there was a comment in there where one guy's like, I lost, I don't remember what it was, like 80 pounds doing crossfit. Therefore, it's amazing. Okay, let's just use something else. I lost 80 pounds playing basketball. I lost 80 pounds. Crash dieting. Playing soccer. I lost 80 pounds doing jiu-jitsu. Does that make it the best form? Maybe for that person, look, if you love it and it's working for you, actually, I'm never going to argue against that, right? Yeah. Unless you're getting lots of injuries, unless you're, you know, complaining about problems. I'm not going to argue that. It's obviously worked for you. It's obviously brought you more benefit than good. Yeah, but you've got to be even careful saying that, because the sustainability of it, as a coach that's experienced, you know that. Right. Like somebody could easily... Right, I don't recommend any hardcore sports for weight loss.  Right. And just because they liked doing that during that time and that's got them those results, me as a professional would still not recommend that as an ideal weight. Right. Because what I know is the sustainability of that long-term is just not realistic. Most people, okay, will not run five miles a day and eat 1,300 calories forever. And so even if it did work for you and you do like doing that, that's the problem that I have with that. Well, you know what the issue is, and you guys, we've seen this in diet culture as well. If somebody does something and it gets them to lose weight and change the way they look or the way they feel. They're a fervorous evangelist at that point. Yes, they're so married to it. So you could talk to somebody who's lost a lot of weight doing keto or going vegan or paleo or whatever, cabbage juice diet, whatever. And what you'll get is a, somebody's very religious about what they just did. Well, you could bring it instead of diet, bring it back to training modalities. Sure. People get the same way with that. That's why there's a community of power lifters, there's a community of crossfitters, a community of bodybuilders. And so it's no different. I talk to them the same way. It's like that when I was bodybuilding, one of the biggest flaws that I saw in my peers was that's the way they always trained. It's like, dude, you guys know that if you just moved out of this, you know, super setting, isolation exercise, pumping exercises all the time, you would see huge benefits if you power lifted for a little while. Like you have no idea. And that's actually a great point is that even if you are extreme in one of the sports, there are some things you can learn. It doesn't have to be a ton, but there are some things you can learn that will also benefit you. For example, bodybuilders that take from powerlifting, they build more muscle as a result of doing that. Some of the best power lifters in the world, bodybuilders in the world, were power lifters at one point. Ronnie Coleman is a great example of that. You know, you could do that for most of these modalities and I think you're doing yourself a huge disservice when you put yourself in this camp and it's us versus them mentality because you're no longer open to growth. You're no longer open to progress or even just seeing what's not working for you. Make no mistake, when you get stuck in a mentality, you can actually do yourself quite a bit of harm. I mean, going back to the diet thing, I can't tell you, I would get messages from people, especially when we first started the podcast, I get messages from people who are like, hey, Sal, I've been doing keto for four months. I feel terrible. I'm very constipated. When is my body gonna transition? When's it gonna start feeling good? Like it's not, it's been four months.  You see this with training too. Hey, Sal, I lost 40 pounds doing CrossFit, but I had to get a shoulder injury. I feel really run down. I got my testosterone levels checked, it's down. So when is that gonna start to reverse? Well, maybe that's the wrong modality for you right now. It's literally everything we've learned over the past year or two is how tribal everybody is and how much they don't wanna hear a counterpoint or invite a discussion of admitting that there may be some flaws in the methodology. They don't wanna examine that and think critically about things. Yeah, I know. It cracks me up though. It always cracks me up. And I think part of me likes to trigger it. Yeah, yeah, of course. It's like, let's start this conversation. Yeah, because I feel like, again, we've talked about all the other modalities in this podcast, but nobody brings that up.  It's like, bro, we talk about every modality. And everybody is, by the way, everyone is just as guilty and so gravitating towards one modality and sticking to it. I mean, that was a lot of the motivation of starting the show was that we wanted to break those barriers and teach the average person that there is something to take from power lifters. There is something to take from kettlebells. There is something to take from CrossFit. It's easier to stay in that one, train of thought in that same pattern because it's your body, it likes that. I wanna keep doing what I like to do. I don't wanna challenge myself. I get it. We're the same way too. We're just as guilty. That's a human condition. Yes, when I assess the way we all train, we all tend to gravitate towards the things we like most, but we're all aware of it. And we all know, like, okay, it's been a little while. I've been pushing the weight too much. I need to get out of here, go into my body builder way. Or I need to go mobility focused. Or hey, I'm gonna pull the kettlebells out and get working on some rotational stuff. So yes, I mean, I'm just as guilty too, but I'm also aware enough to be okay that when someone points out one of those modalities as having flaws, I don't get attacked personally. You're coming after me. You're like, all I can say is like, yeah, you're right. That is a flaw in that training modality. Yeah, yeah. Well, I mean, look, again, just to hit the other side, for as long as I've been in fitness professionally, which is over two decades, and non-professionally, which is much longer, no strength modality at all was able to get people to squat and deadlift and to use bumper plates. None. Bodybuilding failed at that. Powerlifting failed at that. Weightlifting failed at that. It was CrossFit. It was CrossFit that literally got people to squat and to deadlift and to use platforms. Before that, you would not find a platform. They certainly didn't invent it, though. No, you would not find it in any gym, and literally, I would manage these 40,000-square-foot facilities. There would be one squat rack, and nobody would use it. And deadlifts, God forbid you saw a deadlift, people would freak out. So CrossFit single-handedly, that's why I'm grateful for CrossFit, it single-handedly got people to do some of the most effective exercises known to man. Does that mean I can't critique some of the other shit? Now evolve and be better. That's all I'm asking.


Hello and welcome back to the craft box my name is Brent and today we're doing some more completion as crafter today we are crafting the cherry pie which is a rank 3 snack on screen now everything you'll need to know everything you need in order to craft this um as normal um yeah let's hit over I believe it's basic crafting station again I'm assuming obviously you will know um I don't yes it is um so yeah very very simple it's not particularly a good snack it's only rank 3 it's pretty trash but um yeah that's kind of just how how it works some of them are good some of them are bad we will be getting into the mega snacks soon which you can actually craft so that's fun um but yeah anyway thank you for watching remember like and subscribe if you're a subscribed person will about so you know exactly when I press another video and remember craft outside the box


I was having panic attacks two, three times a day to the point where, like, my sense of reality was just completely washed out because most of my day was spent in this panic state of being, in this state of being where, like, I feel like I have no control over my environment. To me, brain education was what got me out of that cycle and gave me the tools to recreate and reclaim my mental health. I was a depressed teen with severe panic disorder and I reclaimed my mental health through brain education. My name is Rachel Ammar and I'm from Westchester, New York. I got started with brain education when I was a junior in high school. It was the end of my junior year and needless to say, there was a lot of pressure coming from my teachers and parents about college applications and grades and extracurricular activities and X, Y and Z. Aside from the school stress and pressure, I had a lot of my own personal things going on with family and my social life and I was really, really depressed and anxious to the point where I could hardly even focus and concentrate at school because my breath would just become so short and tight. And that was when I decided I really need to do something to better myself and then I found brain education and got started with brain education. My favorite brain education principle is water up, fire down.  When I learned and felt water up for the first time, I was just like, wow. So this is what it feels like to not have a busy mind. So this is what it feels like to just think clearly. And I was like, this is what I was longing for for so long is just to feel focused and clear and centered. And it's a principle that I really try to apply every day to just come back to being grounded. That's my favorite principle. I think everybody will benefit from brain education. But specifically, I feel like if you are a person who just feels so drained and washed out, like, you know, you have this feeling of like every day is just like you're living day to day to day. And you don't really get get a chance to like rejuvenate and restore and replenish yourself. And it just feels like day and night, day and night, day and night, like the same thing over and over again. And the same like voices, negative chatter in your mind over and over again, the same people. And it's just like you feel like your life is kind of like purposeless, kind of like purposeless. Brain education can help you and can help you find, you know, a space to refill your cup and find a space to like let go of stress and tension. And that anxiety, anxious feeling and just, you know, find what your purpose is and live a more purposeful and fulfilling happy, healthy life. Seeing where I was two years ago before I started with this practice, I would have never thought that I would be able to make it out of that space. I was having panic attacks two, three times a day to the point where like my sense of reality was just completely washed out because most of my day was spent in this panic state of being in this state of being where like I feel like I have no control over my environment. So to me, brain education was what got me out of that cycle and gave me the tools to recreate and reclaim my mental health and learn how to breathe deeply and center my mind and calm down and slow down my thinking. So watching how I was able to pull myself out of that state and seeing where I am now, to me brain education is like hope. It is literally like saving grace. So for the world and for the youth today, brain education is like what will save our generation from being consumed by toxic and fake, superficial and hate and all that stuff like on social media and TikTok and Instagram and like all of that material world. Brain education is what can reprogram us and help us come back to love, acceptance and harmony. That is what brain education means to me. I was a depressed teen with severe panic disorder and I reclaimed my mental health through brain education. If you're looking to get started with brain education, I personally got started at a body and brain yoga and Tai Chi studio where I grew up in Westchester. We have many locations all over. I'm sure you can find one that's close to where you live. And currently I work at Honors Haven, which is our retreat center in upstate New York. It is so incredibly beautiful up here. So come and visit me. Don't put that. No, but seriously, it's really incredible here.


Let's see, is there a certain strategy tool that should be focused on more closely during slower speeds? Saying strategy slash tools on slower speeds. So again, you got a weird mindset. I would change the mindset of why are you opening your business on a Tuesday morning? Now you can. If you are just like, I need to be there, well, that's fine. What you should expect. All you have to do is then adjust your expectations. What are your expectations?  Well, if you are locked in and that mental capacity is what I tell you, don't limit yourself on thinking in an area of like, OK, I'm going to get my profit target. I'm going to quit for the day. That's not what you should do.  Winner, winner, chicken dinner. After two winners, boom, that's your chicken dinner. That's what you've got. Well, how much did you make on those two winners? It shouldn't be, oh, every day I'm making exactly this. The markets are not that consistent.   I understand it is our responsibility to be available to trade the market when it is strong and not force it. Any insight would be greatly appreciated. Well, I have coded all that for you. You don't have to think about it as much. Now I'm saying, don't think about it. OK, look, if you want to think about it, it's fine. However, I have made it easier. I have created the psychology module inside of the algal box speedometer. So my answer to that question is this right here. My advice is be wise and let the tools do the work. Let the tools tell you if you should be trading or not. OK, now you can adjust this. I have them out of the box set to defaults and you can adjust those on your own. You are welcome to do that. But I have them set to the settings that I would have set. And what do I mean by that? I'll show you. So inside of this speedometer, you're going to have this is the average. So this is the average over the last four minutes. And this is now. Now that's not to the penny. There are we're doing a we're doing some calculations in here, but it should be a per bar item. But there there's we don't want to be too chatty. So we have some adjustments in here. This is not adjusting every single millisecond. Well, I want it to be readable because I did. We haven't we had it at one point where I was just just showing it constantly like a perfect speedometer all the time. And I don't think that was helpful. I want to know what it was. I want to check it and then I want to check in another minute from now. I don't want to be getting it every single second. So, you know, within a minute's time, you will have an update on what the speed is. Makes sense. So psychology module. There is a high side and the low side to those. I'm going to go into my indicators and I'll show you the settings that you can adjust on the speedometer. And the speedometer should be your guide, in my opinion, on how you decide if you should be or should not be inside of the play. There you go. Now, by default, remember I said the four minutes you can change that if you want to see the average over more time or less time, however you want to do this, you can change this. And it is per minute and it is an integer value. So you will need to do it in increments of one, but minutes. There it is. Four minutes there. The high value, the low value, that's right there. So I want to draw a X if the value is too low. The low is defined by 25. Now, if you have a higher time frame and you want to lower that, it totally makes sense. But in general, I am, you know, adjusting this for the lowest time frame chart, which is for us going to be alga bars, ones, twos, or threes. And watch all the videos around that. I'm not going to go into detail on that here tonight. But here it is. High value, 170. If it's higher than 170, I want to draw an X. It's lower than 25. I want to draw an X to remind myself that, hey, it's probably not a good time to trade. Makes sense. Press the like button, share, subscribe, and comment below with your thoughts and what you'd learn. Watch this next video.


On the Mustang post game report. I'm Christian Tanner with Nikki Kendrick after today's six zero loss Nikki again, it seemed like there was a lot of good things in this game today But Nazareth ended up winning. What were they doing so effectively against you guys? I mean, they're just a great team overall It's always a great competition playing against them. You know, they have some strong players But they were just moving the puck really well getting shots off and capitalizing on some of the mistakes that we made You know, they just did a great job overall. Yeah, so I mean really in this game There were a lot of positives it seems what will you take from the positives and work it into this weekend against Alvarnia? I think the biggest positive was that we worked hard until the end, you know We never gave up even when the score, you know, wasn't in our favor We just kept pushing forward and you know, I think it shows a lot about us as a team and how much faith we have in each other So obviously this week there have been some sicknesses and it's been a shortened bench But what does it say about this team to still fight in minute and minute outs? I mean overall, I think it shows that we have a lot of heart, you know, like we just Never gave up and kept pushing even though we didn't have that many players, you know It just it was a lot, but we kept going. Nikki. Thank you very much. Good luck this weekend. Thank you very much for Nikki Kendrick I'm Christian Taylor on go mustang sports.com


Welcome to the ITU Plenty Potentiary Conference 2018 in Dubai in the United Arab Emirates where I'm very pleased to be joining the studio today Mr. Malcolm Johnson who is Deputy Secretary-General of the International Telecommunication Union. Malcolm welcome to the studio. Thank you very much Max. Now you've just been re-elected as Deputy Secretary-General you've got another four years which is fantastic I just really wanted to ask you a little bit about the fact that this Plenty Potentiary Conference has seen calls to modernize ITU and over the next four years perhaps you could tell us where could ITU make the biggest strides. Yes well you'll recall that the last Plenty Potentiary Conference adopted a number of efficiency measures 30 in fact. So we've implemented most of those I'm pleased to say and made quite a lot of gains through doing that. However I still believe that we can go further on some of those in particular with regard to the duplication of activities and functions in the ITU and also do more to centralize some of the administrative and finance services in the ITU so that we we have more specialized workforce on these areas rather than having it spread across the different bureaus as well as the general secretariat so that's one area. We've also done a lot of work on simplifying and digitizing our internal processes and a lot of these processes were paper based. We can do it we still have a lot more to do in that area but we've also reduced the use and dependency on paper quite significantly I'm pleased to say since since I started we're now using three million less pages a year than we were doing. And it's noticeable at this conference too. Exactly so that's another area that we can do more of especially you know in in the headquarters and then also we are looking at trialling some machine translation and machine captioning tools. These technologies are developing rapidly so I think in the next few years they're going to be sufficiently the quality will be sufficient to be able to use them and implement them and thereby increase the amount of translation we do for the same budget. We're also trialling outsourcing the translation of the web pages you know that's a trialous ongoing at the moment with a number of member states and that's also proven to be quite beneficial. And the captioning will make ITU much more accessible as well. Exactly yes we do provide captioning at the moment but it's manual captioning. We've already done a trial comparing manual captioning with machine captioning and it's rapidly catching up with the quality and accuracy of the manual captioning. So there's that area and another area that I'm particularly keen on taking further is the remote participation to our meetings because this will reduce the cost of course of participating, cost of traveling and cost of accommodation for participating in meetings most of them being in Geneva. But also for those countries that are far from Geneva and with limited resources and not able to come you know to the meetings it mean they'll be able to participate at no cost. So it will increase the inclusiveness you know the opportunity for all our members to participate in our work. So that's also something I'm very keen on. But the one thing that will give us a huge move forward on modernizing the ITU I believe is the new building. Now the new building of course is going to be a smart building environmentally sustainable. It's the design that's been chosen and provides tremendous amount of light the way it's been designed and also capacity so this spacious accommodation. It'll be an environment I believe that will inspire staff and delegates for future generations. So that's an opportunity for us to introduce modern working methods, flexible working. So not only will that improve efficiency but it'll also enhance staff life work balance. It's proved to be very effective and many studies have shown that to be the case. So I think that's that's something that's really looking forward to taking that project forward. And as you know we've been very fortunate to get to generous sponsorship from Saudi Arabia, UAE and Czech Republic and made a donation. So with their support you know we will be able to enhance this building and make it something that we're all very proud of. And I would encourage of course other member states and sector members to follow their example and make a contribution to this new building. And that's a brand new building that will be replacing a building in Geneva that's been there for how many years now? Well since the late 50s yes that building no longer meets a number of requirements, safety requirements, accessibility requirements. So yes that building will be demolished and the new building will be built on that site. So it's a complex project because we have to relocate the 300 staff in that building will need to be relocated. But the main thing is the main challenge I think is this change of culture of the organization moving into an open space environment, paperless working, modern working methods, flexible working. So it would be quite a change for ITU but I think definitely a change for the better. Absolutely and talking about change you brought gender parity to the standardization bureau when you were director of it. I wanted to ask you what would it take to get gender parity across the whole of the organization? Yes well as we know less than half the world population is currently connected and statistics show that over 250 million more women are offline than men. So a lot of women and girls they're missing out on that opportunity of empowerment, social and health benefits, educational benefits and the opportunities that will come from future jobs. I mean most of the jobs that will be available for today's generation of girls don't exist at the moment but one thing we can be sure of is that whatever career they go into they're going to need digital skills. So it's very important all the work that we're doing to encourage more women and girls to come into the sector and acquire digital skills. And yes in the TSB when I was there I'm very proud to say that we increased the percentage of women professionals from less than 20% to over 40% while I was there. And I always like to say that that's not because I chose women candidates because they were women. I chose them because I thought they were the best candidate on merit. And I'm pleased to say they proved me right you know. But we did introduce a number of other measures that helped. I mean we introduced leadership courses for the women professionals. We introduced unconscious bias courses especially for those that are going on to the interview panels. And I myself completed the UN Women online course I Know Gender and I encouraged all the staff to take that course. So I think you know leading by example is very important in this area and if the women can see that they have the support needed to progress their careers in the ITU then that makes a huge difference. So this is what we're looking to extend to the whole of the organization. And we can see that we're making progress. I mean perfect example here is Doreen being elected as the new BDT director. Doreen started as a junior professional in the ITU. So it's perfect role model. And role models are very important of course. But also we've seen in this conference that a number of women taking leadership positions. I mean we have more women chairing the committees than men. So we're definitely making progress and we have the ITU gender policy very sound policy. So implementing that I'm sure we'll continue down the road of reaching gender parity. Finally after your reelection what message would you like to convey to both the people here and to the wider audience as well. Yes well as we know ICT is going to be essential for the implementation of the Sustainable Development Goals. And ITU as a lead UN agency on ICT is clearly got a very important role to play there. But there are many organizations all involved in implementing the SDGs and using ICTs to help implement the SDGs. So I would say that collaboration, coordination and cooperation are the key to ensure that we all bring our own specific competencies to the table. That we don't overlap and duplicate our efforts and pool our resources to the common good. So that we do achieve the implementation of the SDGs and bring this wonderful technology to everyone everywhere. Well congratulations once again for being re-elected as Deputy Secretary-General. We look forward to catching up with you again over the next four years and thank you very much indeed for joining us in the studio today. Thank you, it's a bit of a pleasure. Thanks very much.


If you are interested in using plastic mulch for sweet corn to increase the root zone soil temperature and extend the growing season in cold climate regions, you may have a few questions about the process. We will try to answer some of those in this video. What is mulching? Mulching is the process where the soil surface is covered directly by a material, such as straw, grass clippings, wood chips, or in our case, various forms of plastic sheeting. How many types of mulches are there? There are several different types of mulches you could use. The natural or organic forms of mulch consist of materials like straw or gravel. The plastic sheeting mulch varies in color, thickness, and transparency, and each variation of plastic mulch serves a different purpose. There are also different types of degradable mulches, such as photodegradable or biodegradable mulch. Why plastic mulch? Plastic mulch is more convenient to install and remove than natural or organic mulches, and they are relatively cheaper. What is biodegradable mulch and why would you use it? Biodegradable mulches are designed to act like plastic mulches when first installed. The only difference is after harvest, this type of mulch can be plowed into the soil and will degrade over time, usually in one to two years. These types of mulches can be made out of a wide variety of products from plant starches mixed with different polymers to different types of paper coated with plant starches. It is safer for the environment than plastic mulch. What's the purpose of using plastic mulch? It depends on the types of mulches you use. Clear mulch can increase soil temperature, extend the growing season for cold regions, and improve the crop yield, while black mulch can control the weeds, improve the near surface air temperature, and improve crop yield. Both mulches can conserve water by reducing the evaporation. White and reflective mulches can cool the air temperature, control weeds, and conserve soil moisture. Is there a sweet corn yield increase with plastic mulch? It depends on the seasons, mulch types, and time of application. In our study for sweet corn, the yield range was from a 0% increase with a late planting date on June 19th to a 150% increase when it was planted in mid-May with clear plastic mulches. A higher temperature increase with clear mulch is expected during the early growing season. How much does it cost to use mulch? The cost is about $326 an acre for plastic mulch. As a 4,000-foot roll of plastic mulch costs about $150. It's about $526 an acre for degradable mulch. As a 5,000-foot roll of biodegradable mulch costs about $300. A mulch planter can cost $20,000 if you want to grow in a large area. And if you just want to buy a 100-foot roll for your garden, it will cost you about $25. The bigger the roll, the cheaper the price per foot. Where can you get the mulch? You should be able to find mulch for sale near you as most stores with a gardening section have them. Nursery stores are also a good place to look. The bigger rolls of mulch, like the 4,000-foot ones, can be purchased online. How long is the season extended by using mulch? Well, it depends on when you use it. If you apply earlier in the season, it can extend the season for at least two weeks. But if you apply later, you may see no difference between the areas with and without the mulch. How long do you use the mulch? You have options here as you can apply before planting and take it off at the end of the season, leave it for a second season, or even take it off in the middle of the growing season. What are the disadvantages of mulch? Cost is higher with mulch because of materials and the additional labor cost. Also for the general public, the plastic mulches are considered as white pollution and have to be taken from the ground. They are either recycled or trashed, which can cause more environmental concerns to the public.