We have discussed the what is called LQG problems and finally, we have come with an expression that dynamic equation of the system or process dynamics and estimated dynamics we have come across with this expression, where v of t small v of t is the measurement noise. This equation can be rewritten into a error that state equation argumented with a error equation, which will come into this form. And if you look at the equation number 12, you see this the matrix this matrix is a diagonal matrix. And this we can control the what is called closed loop system Eigen values and observer or estimated Eigen values or Kalman filter Eigen values separately one does not depend on the others. So, this a of b k can be designed independently from the a minus l c of the Kalman filter gain. Both are controller gain and Kalman filter gain can be designed independently that means separation principle holds for this one. So, now let us look at this stage that what about the it is loop gain that means if you talk about the robustness of this controller as we have seen in the what is called earlier case LQR controller design. When the states are we are assuming available to us then we have seen that controller LQR controller gives the what is the good stability margin that means gain margin is varies from what is called greater than equal to half to infinity. And phase margin is greater than 6 degree whether this properties of the systems holds in case of LQG problem or not that we will study now. So, that is called the loop transfer recovery. So, let us start with loop transfer recovery that means LTR and if you see this one and this loop let us find out the loop gain of these systems. So, we know the our estimated LQG problem we know our LQG problem the estimated equation LQG problem the estimated equation is dynamic equation is x dot hat x hat dot of t this is x hat dot of t is equal to a minus b k minus l c x hat of t plus l of y of t. This l of l is the Kalman gain Kalman filter gain and this k is the controller gain gain and where u is equal to width you can write width u is equal to minus k x hat of t u of t is equal to this. Now, if you take the Laplace transform of this quantity that u of s is equal to minus k x hat of s. So, which you can write it because this even if you take the Laplace transform of this equation then we will get it x hat of s we can write it this minus that will be a s i minus a minus b k minus l c this whole inverse agree then this will be a l y of s because you take the Laplace transform of this one you will get it s i x hat of s then Laplace transform this one is x hat of s bring to this side and take the value of x hat of s then x hat of s i minus a minus b k minus l k whole inverse this will be multiplied by l into y s taking Laplace transform from this one we will get it that quantity from this to this. So, x hat of s is replaced by this one from this equation by taking Laplace transform both. So, let us call this equation is one and we have we have s is equal to s is equal to minus this whole quantity I am writing is k s into y s the whole quantity from this to up to this this is I am connected by y s. So, now, where we can write where y s as a where k of s is equal to k s i minus a minus b k minus l c this whole inverse into l this is our k s. Now, let us this whole thing this we denoted by let us call phi r of s phi r of s this whole thing is nothing but a what is called regular resultment result you this phi r of s is called phi phi r of s is called regular resultment resultment matrix regulated resultment matrix regulated resultment matrix regulated resultment matrix phi r of s agree because this is controller is there if you see in this one controller and observer gain matrix also involved in this one that sense it is called regulated resultment matrix phi of s. So, now, this is let us call this equation is that is what is we consider that equation is to this equation is to agree. Now, if you draw this circuit configuration or block diagram form this is our reference input r which is equal to assume 0 then this is our in transaction domain if you block diagram we put it then g this is our plant this is equal to s i minus a whole inverse b is nothing but a this I am written by phi of s into b this s i minus a resultment matrix of the system is phi of s and b as it is this is the and this is our symmetric the output of this is x of s this is c this is our y of s agree and from this one we have a that our carbon filter filter output of the carbon filter is our x hat of s and this is this agree then this will go with a controller gain then this is minus of that one and this is our u of s agree and this inside this one that carbon filter structure is there that input information also there output information also there. So, this is in built in there so basically in transaction domain it is a s i minus a minus b k minus l c this whole inverse into l agree that is our phi l into k that will be our k of s.  So, now find out the loop what is called this is this is u of s find out the what is called loop transfer function for this carbon filter based controller loop transfer functions. So, if you loop gain or loop transfer loop gain let us call we have denoted by l r of s is nothing but a use from here to here if you just move it here this loop if you break it find out the loop transfer function just like in our basic block diagram if you see basic block diagram if you see this is our g of s and this is our feedback path h of s and this is our minus this. So, this is y of s in order to study this one then or sensitivity or compartmentalized loop transfer function our loop transfer function is g s h of s. Similarly, in this case also we have a loop transfer function if you loop gain of this one I can write it this equal to k into this quantity I have considered is equal to phi this quantity if you see it is a phi r s into l then c then phi s into b. This is the loop gain of this carbon filter based controller when we have designed the carbon filter based controller the loop gain is k phi r this is phi r then l then l then c then b phi.  So, next is the let us call this equation is 3 this equation is 3.  This is our loop gain and clearly you see this one this loop gain of the carbon filter based controller design and the l q r design l q g l q r loop gain is different and we have seen in what is called l q r design that controller what we have designed the k controller design the system what is called gain it can be varied from k is equal to greater than equal to half or you can increase the gain up to infinity from the present value of gain whereas phase margin it will give at least 60 degree for the design gain k of the closed loop systems. So, that we have seen in other words if you draw the block diagram form of this one it is look like this g s and g s g s this is nothing but a s i minus a whole inverse whole inverse into b that output will be that output will be x of s and this states are all measurable. So, this and this loop gain if you see the loop gain for this one it is k into g s g s is what this is phi s result matrix of the originals from the given matrix s i minus a inverse this is the inverse of s i minus a this is inverse this the two loop gains are different. So, we may expect that l q r l q g controller they may lose the what is called the excellent properties of l q r what is the excellent property we got in l q r excellent property in the sense of gain margin and phase margin what we got it we may lose that one. So, remarks l q g does not have the same properties as l q r design method due to the introduction of Kalman filter the introduction of Kalman filter because the loop gain is different and that that part is coming additional this part if you see this part from here to here sorry from here to here this from here to phi l s l c this part is coming extra and this is definitely there will be role of what is called gain role as frequency is changing. So, first remarks is this one second it has lower stability margin it has lower stability margin then the l q r design l q r design and sensitivity properties that means sensitivity properties sensitivity properties are not good as those of l q r sensitive properties are not good as that of as that those of l q r design because sensitivity is stability margin that means gain margin will lose it will not able to maintain as we got l q r problems due to the introduction of this Kalman filter again and not only this that sensitivity disturbance rejection or noise rejection case it will not be good as it is in case of l q r is to third one is your that third one is it is possible to recover the it is possible to recover the what is called stability margin or good properties of l q r what we got it that properties will be able to recover while we have designed the l q r design l q g problem when you have designed the l q g problem we have seen that that is the loss of what is called stability now what is called and what is stability properties stability margin it will be reduced. So, that we can regain or recover by introducing some noise in the process signal. So, it may be possible it might be possible to recover some of the desire some of the desirable properties like gain and phase margin l q r by selecting the noise process noise process noise by selecting the process noise covariance matrix variance. Let us see how we can regain our what is called stability margin by introducing what is called by what is called selecting the process noise covariance matrix. So, let us define there are some mathematical manipulation we have to done and we have seen how the process noise if you can select properly then it is possible to regain the stability properties of l q r. So, define the following quantities it is. So, it is conclusion is that that if you design the l q r l q g problem that stability properties of l q r is not retained again. So, we can able to recover the stability margin of l q r by introducing or by selecting the noise process noise covariance matrix. So, let us define phi of s is equal to s i minus a inverse that is the result of matrix of the system. So, next we consider phi of c is equal to s i minus a minus v k whole inverse with this identity with this symbol using any equation number 1. Let us see equation number 1 this I will use that one then what will in this equation I use the phi phi c. So, from 1 loop gain 1 loop gain expression this is the loop gain expression this from equation 3 from 3 loop gain l r is equal to k then you have a that is if you see this one s i minus a minus v k minus l c whole then whole inverse that whole inverse minus l. So, this l then phi of s into b. So, this I can write it this k into s i minus a minus v k a minus v k this I am club together a minus v k plus l c whole inverse l then c that I missed here c c then phi of s into b. So, this equation I can write further l r of s then I can write it this part you see s i minus a this inverse is a your phi c that is what phi c. So, I can write this is nothing but our phi c s inverse plus what is left l c then whole inverse l c into phi of s into b let us call this is equation number 4. So, from this equation this is equal to I have written phi c inverse then l c this one. So, this and using now in equation 4 will be using one matrix inversion lemma using matrix inversion lemma what is this material let us call we have a matrix a bar b bar c bar and d bar whole inverse is equal to a bar inverse minus a bar inverse a bar inverse then your b bar then d bar then a bar inverse then b bar plus c bar inverse then whole inverse d inverse d bar a bar inverse this one this matrix inversion can be written into this form. So, you are using in this expression this matrix inversion lemma for that one we set a bar is equal to phi c inverse of s then b bar is equal to your l and c bar is equal to your identity matrix and d bar is equal to is equal to c. So, I will use this lemma in this inversion. So, our using this one in equation 4 from 4 using the above lemma in 4 we get l r of s is equal to then this is phi inverse of a a bar. So, our you see a bar phi inverse so this is phi c of s phi c of s then minus and you see this one there is a k is there k is missing here k bracket minus a bar inverse that means phi c of x s b bar is what l then bracket d bar is what c a bar of this phi c of s b bar is what l then your c bar is i. So, this whole inverse d bar is what c and then a bar is phi c inverse that inverse is phi c of s. So, this equation is rewritten into this form into that into star into l c phi of s into b. So, this equation is complete now now we are doing some manipulation of this one. So, I push it see this one I push l in this expression agree. So, phi c l and this l also go here. So, phi c l so if you take it phi c l common then it will be remain here i minus. So, this will be a i plus c phi c s l whole inverse then c phi c s into l into l bracket closed then c l I pushed inside. So, remaining part is c phi of s into b agree. So, this now so this I can write it further I can write it this phi c s into l. So, what will come here phi c l this and this I can write it this I can write it this matrix into inverse of this matrix. So, I am taking common of that inverse of this matrix i c phi c x l whole inverse then what is left here because I am writing this matrix into i plus c phi c l. So, it is left is i c phi c s into l minus this one left because this I have taken common c phi c s into l. So, close this one then c phi c s then c phi of s into b. So, you see this cancelled so ultimately it is we will get ultimately we will get it here if you see this one the simplification that l c r l r of s is equal to k phi c of s l i c phi c of s l whole inverse c phi of s b. Again this is the equation let us call this equation is equation number 5 this is the equation number 5. Now, recall our what is called our basic equation system equation if you recall our system equation recall the system dynamic equation system dynamic equation from just recall from our previous lecture the dynamic equation. So, our equation number 1 there so x dot is equal to a x plus b u plus gamma w of t. So, in the process in the system equation this is the process noise w of thing process noise again we know the and its dimension is let us call it dimension r cross 1 other dimension we have already specified in the previous statement of the LQG problem. So, this process noise covariance matrix so if you recall this one so now in l t r loop transform transfer recovery approach loop transfer recovery approach in loop it is assumed that the process noise that the process noise covariance matrix that in the process noise covariance matrix we have considered w in our last lecture matrix is nothing but it from here you see it is a process noise covariance matrix gamma w into w transpose is equal to q that gamma transpose this is the noise covariance matrix of w or noise control is small omega that is q agree. So, this is the noise covariance matrix if you recall our LQG problem statement w is the process noise and its covariance matrix is your q. So, since it is going through a channel w gamma so the noise process covariance matrix process noise covariance matrix is gamma q gamma transpose. So, this is replaced by now by gamma q gamma transpose is now replaced by gamma w 0 gamma transpose plus small q b into b transpose. This is any positive scalar quantity and the gamma 0 is the initial guess of process noise initial guess of process noise covariance matrix process noise covariance matrix initial guess again. So, let us call this is equation number six now using this thing using this expression in our algebraic Riccati equation of Kalman filter using this equation then equation 6 in A R E algebraic Riccati equation of our Kalman filter that we have discussed last class if you see this one Kalman filter then we have it is the dual of what is called algebraic Riccati equation of controller design. So, there is A transpose P here it will be A transpose replaced by A. So, A P E P E for algebraic Riccati equation for the Kalman filter and this is the error covariance matrix then P E please refer to our last lecture note. So, P E then there is P B B is transpose by C transpose.  So, this is the algebraic Riccati equation for the Kalman filter of that one. So, let us see what we have considered Q or W I am not able to recollect what is considered let us see in the algebraic Riccati equation of the Kalman filter the noise covariance matrix what we have considered. The noise covariance matrix I think I have considered W. So, better this is W I have considered there W. So, this is W agree this is W with earlier I have considered W. So, please write it same notation here W noise covariance matrix I have considered is a W there. So, this gamma W double term now replaced by a two parts one is the initial guess and another is Q transpose this one and this is the positive semi definite matrix that is one. So, all these things and W B is a greater than 0 means positive definite matrix. So, this equation we can rewrite by dividing by Q square. So, you if you divide by Q square that means A P E P suffix E P suffix A Q square P E suffix Q square this is whole A transpose then P E Q square agree that I am dividing by C transpose B inverse agree C P E I divided by Q square multiplied by Q square it does not change anything. So, that will be a gamma W 0 gamma transpose Q square plus B B transpose is equal to 0 agree. Let us call this equation is equation number 7 this equation when what I did it I divided by Q these things. So, now we made an assumption the plant we have assumed this minimum phase system the plant is if the plant is minimum phase system means all zeros are stable and B and C B and C full rank with number of inputs is equal to number of outputs B P is equal to M number of outputs is equal to number of then one can write it limit Q tends to infinity P E divided by Q square is equal to 0 why the way we are increasing the Q we are the P is increasing P is increasing more slowly then covariance matrix Q is increasing the and which in turn when Q is very large that will tends to 0. That means, if we increase in the Q again the P is increasing the error covariance matrix P is increasing more slowly then the Q is increasing. So, this implies that this will be Q is very large infinity this will be 0. So, now from equation 7 if you use this expression in equation 7 what is the logic behind this Q the way Q is increasing the increasing in P is more slowly than increasing in Q. So, with this logic we can rewrite the equation from 8 or from 7 sorry from 7 you can say this is 0 this is 0 this is 0, but this quantity is 0, but you see P is also increasing. So, this part I cannot write is a 0. So, what is the term is left here is Q square P E by Q square C transpose V inverse C P E Q square see what part is left this part C A divided by B square into Q minus B B transpose is equal to 0 because Q Q cancel now this is approaching 0, but P is also large. So, I cannot write this is equal to 0. So, further we can write it if you see this one that Q square P E by Q square C transpose V inverse C P E Q square tends to this I can write it tends to B B transpose when Q are increasing that this tends to this quantity tends to B B transpose which I can write it also P E C transpose this is not transpose this is V inverse C transpose V inverse. Then I can write it V into V inverse is identity matrix this is identity matrix then C P E into 1 by Q square tends to B B transpose this I have written just this cancel then every this is if you see this is nothing, but a Kalman gain L. So, this will be a L that will be also V is there if you see this one this L and this will be your L transpose and this is V. So, I can write it that L V L transpose this is L. So, from here to here this is L transpose then V and 1 by Q square tends to B B transpose then V is a noise covariance matrix of the measurement noise covariance matrix. So, I can write V half L V is a symmetric matrix this and I can write it V half agree L this I can write L V half whole transpose tends to B B transpose because V is a symmetric matrix the square root of V I am writing V half transpose then manipulating this one I am writing this one. So, and there is a Q square 1 Q I keep it here another Q I keep it here. So, this expression remains same. So, if you look at this expression expression number 8 the general this quantity tends to B B transpose if you select the L if you select the general solution of equation 8 is L tends to when L tends to Q B B half L tends to B B minus half if it is L is Q B Q B B minus half minus plus half is cancelled. So, that will be a B Q Q Q cancelled similarly this B. So, B B transpose it approach if you selection of this one L solution is that one P you know half as Q tends to infinity. So, that is the solution of is that one then Q B B inverse of this Q tends to infinity. So, substituting using this value 9 using equation 9 in 5 or L R expression L R expression the loop gain expression of L Q G expression. So, I am using loop gain expression in this expression I am using the value of L is what Q B B and V half that means square root of your noise covariance measurement noise covariance. If you use this one in this expression we get L R of S is equal to K phi C S L in place of L I am writing Q B B minus half then bracket I C then phi C of S Q B B minus half whole inverse then star multiplied by this C phi of S into B as Q tends to infinity as Q tends to infinity look this is our L we are substituting L this is L and this is our L in the expression 5. So, this further we can write it this finally we can write it since this is the large quantity of this one compared to one in diagonal elements then we can neglect this one compared to this one. So, that tends to phi phi C of S Q phi C of S Q then V B of half then C phi C of S Q B B half whole inverse star C phi of S B as Q tends to infinity now look at this one this inverse. So, Q Q cancel then V V is the identity matrix. So, this expression finally will come if you see this expression finally will come K phi C of S B then C phi C S B phi C S B whole inverse C phi of S this is phi of S B as it is from there C phi C B inverse this as Q tends to infinity let us call this equation is equation number 10. So, now we are using another identity in this expression 10 using the identity phi C of S we know is nothing but a S i minus A minus B K whole inverse which I can write it S i minus A this plus B K whole inverse which I can write it this is nothing but a phi S inverse this one plus B K whole inverse. So, I can write it this is nothing but a if you write it this one phi of i phi of S phi S if you take it common then i B K phi of S i B K i B K phi of S whole into phi of S this inverse. So, push it phi B K this is a phi X. So, this is nothing but a phi of S because A into B whole inverse B inverse A inverse reverse order. So, this inverse phi of S then it is a i plus B K into phi of S whole inverse. So, this let us call this equation number 1 this identity I am just writing S i minus A minus B K inverse I can write phi S plus 1 plus B K into phi S whole inverse this expression I am using in equation number 10 sorry. So, using 11 in 10 in 10 is this one wherever there is a phi C I will use that one using this one in 10 we get L S of this is K phi S into i plus B K phi S whole inverse B C phi S into i plus B K phi S whole inverse this whole inverse into B look at this expression this K phi C I am using that one phi S this expression this expression is nothing but a phi C of S. Then your what is this expression B B I have written then B I have written C this one B I have written then C phi C phi C expression is what from here to here is phi C expression this is phi C then B then this multiplied by this thing then this thing multiplied by multiplied by C phi S of B as Q tends to infinity let us call this is equation number 1. That means in equation number 10 I use this expression phi whenever there is a phi C is there phi C is there I am using in equation 1 and we got this expression. So, finally and once again we are using this identity using the identity i plus B K phi of S whole inverse B you can write it this reverse order B i plus K phi S B inverse. So, i plus B K phi S whole inverse B is B i plus B will go later phi S this identity I am using in using this identity using in 12 we get that limit Q tends to infinity L R of S is equal to K say in phi K phi S K phi S then instead of that one instead of this A this I am using the identity that one B first then I am a B then I plus K phi S into B whole inverse star. So, this is then B then B again C C phi S C phi S that is it then this identity I am using this part identity. So, that is B that will be B then I plus K phi S into B whole inverse B this i plus this that inverse is here agree that inverse is here. So, this inverse is here inverse then whole inverse this inverse into C phi S B. So, what is left in here this will this inverse and again that I miss something. So, this will be finally, it will be because we have considered that number of inputs and output is same. So, this inversion is exist that whole inversion is exist this inversion exist. So, ultimately we will see we will derive this thing in next class some two three lines is left. So, we will discuss in this next class that what is the our choice of that our loop gain choice through noise process what will be the ultimately loop gain will come it will come same as the linear quadratic regulator loop gain. So, one can regain or recover the loop gain of LQG by selecting the properly the process noise covariance. So, the rest of the portion we will discuss in next class.


Hey, welcome back to the channel. If you've been following me for any amount of time, you probably know just how obsessed I am with email marketing. It's my favorite channel for building a relationship with my audience and generating sales and making money online. So today what I wanna do is reveal to you what I believe is the fastest and simplest way to start making money online with an email marketing newsletter. Now the best part about this is you don't even have to be creating 100% of your own content to make this work or to start doing this. You can actually curate content from other creators and other individuals and other businesses to grow your own audience and grow your own list and grow your own email marketing newsletter. Now before we jump into the step-by-step, the tactical stuff, how do you start making money from an email marketing newsletter? Well, there's a few different ways, right? Number one is of course, if you're selling your own products, you can link to it in your email marketing newsletter. The second way is you can link to other people's products as an affiliate, right? So do an affiliate marketing, link into other people's products and maybe you get a 15, 20, 30 or even up to 50% or more cut of whatever the sales are from your links and whoever buys from your link. The third way that you can make money with an email newsletter is through sponsorships, right? So if you grow your viewership, if you have enough people reading your newsletter in a certain category or a certain niche, you might have companies reaching out to you saying, hey, if you mention our company in your newsletter, we're gonna pay you X amount, right? That's a sponsorship deal. And these happen when you have more and more people on your email subscriber or your email newsletter. So those are a few of the ways to make money with an email newsletter, but you have to first start your email newsletter up, which is what we're gonna talk about today. So let's jump into my computer and dive into how to do that right now. All right, so here we are on my computer and there's a couple things that we need before we get started. The first thing is some kind of software to collect email addresses, to categorize people into lists and to send the actual newsletter out to them. So I use Active Campaign to do all of that. I've been using it for about a year and a half, very happy with the software and the platform. And this is what I use to collect people's email addresses to categorize them into lists and to send out the email newsletter to them. So that's the first thing. The second thing you need is, of course, people to send your email newsletter to. So this is not a video about list building, but I wanna give you a quick overview of my strategy on how I build my list. You can see there's 6,674 people on my list right now. So what I do is I post content on social media. So YouTube, Twitter, Facebook groups, I post content for free content on those platforms that gets people interested in hearing more about what I have to say. They join my list. Sometimes they download some kind of free report or a guide or some kind of freebie. They give me their email address and exchange for that. They join my list and now they get to see my emails and now they get to see my weekly newsletter, which we're gonna talk about today. Now, shameless plug, if you guys wanna join my email list, go to emailrainmaker.com, join my free Facebook group. And if you say yes to getting the report and you put your email address, you're gonna start to get my daily emails and of course my weekly newsletter that I'm gonna talk about here in this video. So those are the two things that you need, right? Number one, again, you need a software to collect emails and to send the emails out. I use Active Campaign. Number two, you need a list of people to send emails to, right? If you don't have anyone on your list, there's no point in even doing this newsletter here. So now with that out of the way, let's talk about the newsletter itself. How do you actually create one? How do you structure it? How do you plan it? Let's talk about that now. So first thing is you need to have some kind of theme around your newsletter and you wanna give it a name. So I'm gonna go into my campaigns here and show you what mine is. So if I jump into here, mine is called Friday Fire. So today is the 11th. I just sent out an email, a newsletter. I send it out every Friday and the theme of this Friday Fire newsletter is the best internet marketing and online business heat from the week. So all I'm doing in this newsletter is I'm making a collection of my content that they might have missed, right? YouTube videos or Twitter posts or Facebook group posts. And I'm also making a collection of curated content from other people that I enjoyed or got value from. So I'm gonna open this up for you. But that's step number one, right? Pick a theme around your newsletter and also give it a name, give it an exciting name. So Friday Fire is the name and the theme is the best internet marketing and online business heat from the week. Now the second thing is to create an outline for your email marketing newsletter. So my outline is I start with some kind of story, right? Some kind of story that ties in with the theme of the week. So the theme of this week is the importance of your environment and how it affects your performance. So I have a little story about how I was in Hawaii and how I spent two and a half weeks there and I had a hard time doing calls because I was at my parents' house and I don't have the best internet there. So I use that to talk about the importance of your environment and I always start these newsletters with a short story here of something that happened during the week. So that is the piece number one is a story. The second part of this newsletter is I post your compadre, that's me, that's how I sign off every email and I have three pieces of content here that they might have missed. So this is a YouTube video. This is another YouTube video. I usually post two videos a week and if they missed it, I link to it here so they can click and it brings them directly to the video and I also have little short blurbs here of what they can expect in this video to get them excited. And then I also, I usually write one Twitter thread per week so I'll link to that as well on this third bullet here. So this is a great way for me to just take the content that I created during the week and put it in this newsletter so if they missed it they can click through and catch it if they didn't already watch it or read it. So that's the first section of my content. The second section here is anything that I enjoyed, content that I enjoyed from other creators and other companies, I put that right here. So web heat from the week. There's two videos that I have. There's a guy on Twitter named Julian Shapiro. He talks about growing an audience and writing online so I was reading a lot of his content this week so I linked to it here. This is a great way, this is what I mean but you don't have to create all of your own content, 100% original content. You can link to other people's stuff here and be more of a curator. You're going out and you're finding good content from around the internet and you're posting it here. And I also wanna note, this doesn't have to be just about business. You can create a newsletter about pretty much anything in any niche. So it doesn't have to be just business, whatever it is, maybe you have a hobby or a passion that you wanna create a newsletter around. You can do this exact same framework or this outline for whatever niche that you're in. So that's the third section actually, the story, I have my links for my content and then I also have links to other people's content that I enjoyed from the week. And then what I also do is I end it with three things that I learned from the week. So I have three lessons here of ideas or insights or things that I learned from my experiences during the week. So you can see my three lessons here. And at the end I ask them a question, right? What's your biggest growth challenge? I don't reply to every email but I do read them. And then what I do is I link to my content and some of my products here. So I say, if you wanna sign up for this, maybe you were referred by a friend, you can sign up here. This brings them to www.SeanAnthony.co where they can give me their email address and they can get the future issues of Friday Fire. I also link to my company productized profits. I link to my free Facebook group and then I link to a couple of my products. This is my super fast sales email challenge, that's $29. Shows you how to write emails that sell. And then I also have an e-book here on how to start an online service business. So I send this out every Friday and this is the outline, right? So that's the next step is after you've decided on a theme and a name for your newsletter, give it an outline. What are you gonna cover in it? What's gonna be included? You can basically copy exactly what I'm doing here, right? Short story, three pieces of content that you might have created from the week, two to three pieces of content that you enjoyed from other people, three things you learned and then links to your offers, your content and your group or community if you have that. Now the last step and this is probably the most important thing is to create a schedule and stick to it. Consistency is everything with this, right? If you just create one newsletter one week, if I were to just send this out one week and just forget about it the next two weeks, no one's gonna tune back in. So you need to have a schedule and stick to it. So my schedule is I write this up every Thursday, usually in the evening, so it's ready to go. And then I schedule it out for Friday at noon, noon Pacific, I'm on the West Coast. So I write this newsletter on Thursday so that I'm not scrambling in the morning and I schedule it out for noon on Fridays. And that's why it's called Friday Fire. So have a schedule, create a consistent schedule and actually stick to it because if you do that over time, it becomes a habit for your readers to tune in and read your newsletters. And that's when you start to gain momentum, that's when you start to grow your viewership and that's when you start to make a lot more money from the sales of your own products or affiliates or maybe you're doing sponsorships in your emails as well, if you're getting a good amount of traffic to your newsletter. So that's really all there is to creating an email marketing newsletter for yourself or maybe you're doing this for a client. It all comes down to picking your theme and the name for your newsletter, creating an outline for what's gonna go into it, the content that's gonna go into your newsletter and then creating a schedule and sticking to it, right? Picking a day where you're gonna create the newsletter and write it out and picking a day where you're gonna actually send out the newsletter and sticking to it so it becomes a habit for your readers to know what to expect from you, to know what to expect that newsletter every week or however often you're sending. Now again, if you missed it, you don't have to create 100% of your own original content. That's why I love email newsletters. You can curate other people's stuff and make affiliate money off of that. If you're selling someone else's product, you can sell your own products or you can get sponsorship deals as well as you start to grow your traffic on your newsletter. Now if you enjoyed this, make sure you give it a like and let me know in the comments. Are you gonna start an email newsletter and if so, what topics do you think you'll create it around? Or if you have ideas, let me know in the comments, I'd love to hear from you. Now next week, I'm gonna do a video around six and seven-figured newsletters so that you can see what that looks like and also model your newsletters off of them if that's what you wanna do. So stick around for that. Make sure you subscribe so you don't miss that video next week. So that's it for this video on the fastest and simplest way to make money online with an email newsletter. If you guys enjoyed this, make sure you let me know in the comments if you wanna see more content around email marketing and email newsletters. And yeah, that's it for this. I'll see you in the next one. Take it easy.


Thanks. My name is David Nikolayev. I work on the Department of Energy's Exascale Computing project, and I'm going to talk to you today about Federation and Zero Trust continuous integration in GitLab. They asked me to give a little spiel on who I am, so I don't have any social media of any kind, and I figured I'd just do something random from my recent trip to Germany. This is a panda. I like to fancy myself a panda, rolling from project to project, so I somehow found myself working on Federation in GitLab. My background's in quantum foundations and mathematics, so I'm very interested about measurement theory and sort of the context in which you run physics experiments, and I find Federation very fascinating because of that, because I feel like it's setting up the measurement apparatus to study programs that could build themselves. But if you're interested in knowing my background more, we can talk after the talk. But if you'd like to follow up after this, I'm on GitLab, GitLab.com, and then we have a big issue here where we're trying to merge all these Federation enhancements into GitLab, so that's issue 33665. I'll present it right at the end so you guys can check that out later. So very quickly, I'm just going to run through what high-performance computing is, what the Exascale Computing project is, how we're trying to use GitLab CI in the ECP, talk high-level what the enhancements are, and then I'm going to do a live demonstration. So fingers crossed, everything flows as expected. So very quickly, kind of by the numbers, the Exascale Computing project is this big research project that was signed into Executive Order by Obama, and it's about $2 billion over seven years, includes all the national laboratories, staff from all across the country, and there's three technical focus areas. Today we're going to focus on the hardware and integration side of that. So kind of visually to give you a sense of the complexity of the project. So when you run CI for something like a web application, oftentimes you can write very small unit tests and have them run in parallel by themselves. With computational physics, you have this challenge that you have multiple timescales, multiple size scales, the coupling between something like a fluid dynamics flow or a radiation transport. It becomes a very complex and it's just not sufficient to run simple unit tests. You have to couple everything and you have to write very large integration tests. So to kind of achieve the level of testing we want for something as intricate as compressible flow, we need to actually test our codes in the environments they're going to run. So that involves not just running on the big supercomputers that will run the simulations but also the diverse ecosystem of software that exists there. So different compiler versions, different libraries, tools, all of these need to be exactly what the real simulation is going to use. So keeping that in mind now we'll look at the challenges. So really when you think about CI you want to have pure automation. No people involved whatsoever but there's challenges to that. Namely CI jobs run arbitrary code, right? You don't know who checked in the code oftentimes. You're pulling in code from a third party and you want to build in real time. So in typical multi-cloud environments, right, we have Docker containers that can isolate your environment but unfortunately those don't get bare metal performance and even still most Docker containers are just run as root. So in the HPC world, right, we need this bare metal performance. I mentioned that we want to run on heterogeneous hardware architectures. We have supercomputers that use Power 9 systems, that use ARM systems, x86 and then we have this rich software ecosystem where modules are built at very specific versions. The combinatorial nightmare of getting them all to work together. You really have to test exactly with how the big simulation is going to run. So trying to enclave it away to a container doesn't really work. On top of that, to tap into the HPC resources, we use batch schedulers and that in itself introduces some new complexity because of the time limits. You have to have certain money allocated to run for that amount of time. Then of course there's security and really that to me is kind of the fascinating part about Federation is it solves a lot of the security challenges. So kind of simplify it down. You cannot have jobs run by user A interact whatsoever with jobs run by user B. So you have this idea of an enclave. So how do we ensure that that happens? Well, we need to have really fine grained user access control and so I'm going to talk about that in the Federation enhancements. And ultimately what this leads to is this idea of zero trust.  Every single action you take needs to fundamentally be authorized. And so we're going to show how our Federation enhancements are working towards achieving that. I'm not going to talk about why, of course, we need automation. Hopefully that's self-evident. This high level is sort of the model that we're working with here. Projects host their code in GitLab or their own self-hosted GitLab. That's then mirrored to our central GitLab in the cloud at Oste. And then those jobs are picked up by federated runners at the different facilities argon and Oakridge. And you can see here, I'm going to talk a fair amount about the identity providers. This is the key source here. And I'll get into it in the next few slides. But we're really tapping into the omniott functionality of GitLab to trigger the jobs and know who's running them. So I'll get right into the technical enhancements we need to make. And it can be kind of summarized by analyzing what the challenges are. So really, the three key challenges were, how do we refine the user access control? Normally, when a CI job is run, it's run as root. And we can't have that at the national laboratories. So how do we refine that control? How do we directly leverage the HBC resources? I won't talk about that. That gets into some of the set UID and batch runners that we've been working on. And then what I like to call the three-body administration problem. We have server admins, runner admins, and, of course, admins at the identity providers. Thankfully, we all communicate, but there's still a challenge there of, well, federation. So the key insight to how we moved to enable this workflow is to realize we needed to make omniott the first-class citizen in GitLab. So omniott is this middle layer in Ruby that standardizes multi-provider authentication. And in particular, there is a schema for the auth hash. Well, inside a GitLab, currently, only the required fields are actually leveraged. So you have a provider, a user ID, and some basic info like a name. But as part of that schema, there are additional fields inside of the credentials hash, inside of the auth hash. So there's a token, that's an access token. Secret is a deprecated value from OAuth 1. And then a Boolean for expires, and then it expires time. So we take advantage of that credentials information as provided by an OAuth 2-type identity provider. But we also actually support SAML and other providers. Of course, it required a fair amount of tinkering around, which, again, if you're interested in the details of that, you can come talk to me after. But fundamentally, we're taking advantage of this authorization code flow from OAuth 2. So steps 1 through 9 here, it's all standard OAuth 2 code flow. So I won't dive into it in detail. And in the demo, you'll see me walking through these steps. Really, the key enhancements start at step 9. So once the provider sends over some information, you get a payload of auth information. As I said before, GitLab just grabbed the provider, external username, and that's it. Really, it was merely meant to handle social sign-in. But we want to do more with that. We want to take advantage of the credentials information that's provided. So in step 9, we extract that credentials information, put it into the database. And now, when you want to trigger CI, you can actually pass down that user's ID to the server. So you can see that it's all the way down to the runner. So you can see there in step 10, all that data is passed down to the runner. And now, whereas before, you would just have the runner execute the CI job and send the response in step 12, now you can actually do step 11. The runner can communicate directly with the identity provider. So now you can see where that three-body administrator problem kind of comes into play here. Before, you'd have to get the token register, and then you just have this implicit trust all the time. But now, we give power to the server admin still has the same power they had. But now the runner admin has more power to say, you know what, maybe I don't want to run that job. I want to communicate with the identity provider I trust first. So they can take that access token and validate it against the identity provider. I won't go into the digital identity guidelines stuff too much. I'll just briefly mention it because you'll see some numbers in the demo. But we try to be NIST compliant. So there's things called identity assurance level and authenticator assurance level. It basically just says, did I show you my driver's license when I got my form of authentication? And do I use a crypto card or a UB key? Something that's a higher level than just username password. And another critical aspect that unfortunately, I'm not going to have time to dive into in this version of the talk is how we really audit the Federation events. But I will briefly show what we can see. But suffice it to say, it's enough to know that our enhancements actually directly take advantage of an enterprise edition feature called the audit events. So every Federation event literally looks like an audit event. All right, so I'm going to do a live demo now. The progression is going to be I'm going to show this transition from what a normal CI job looks like when you use just a normal shell runner, then we progress to a set UID enhancement, then full Federation time permits, they'll show some auditing stuff, and then of course lessons learned and where we're at now. Okay. So this first part is going to use a bunch of virtual machines that I have. Let me zoom in here. Make sure you guys can see that's maybe a little bit too big. That's still good.  So you'll see how the 1026 for all of them. Okay, so in my example repo here, I have a very simple CI YAML file. Okay, so very simple CI YAML file here. I have a tag that associates it with one of those VMs, which is running a very simple standard shell executor runner, and I'm just going to run simple command ID. I want to know who's running the CI job. So if I come over here to my pipelines, I look at the latest job, I see as I expect root ran the job. That's because the runner's running is root, and so of course the CI job is going to run as root. So the next iteration here is to take advantage of set UID. So that's where I'm going to actually try and change the user that's triggering the job to actually run it. Let's test. I'm going to cheat here for a second, actually, because I need to make sure I get the right tag. You can see here as part of the demo, there's quite a few runners here. GitLab runners at UID shell. GitLab runners. Yes, okay. So same thing. I'm going to run a very simple command here. Just want to know who's running the job. Okay. So now if I come over to my pipelines, oh, it's failing. Okay. Well, it's telling me because there's some unknown user DINIC trying to run the job. Well, that's because set UID, at first pass it, you know, moving towards Federation is literally just taking my GitLab account, DINIC. So that's my username in the server. It's DINIC. But if you look at my username here, yes, it's in a VM, but I kept it the same as my laptop. I'm just, I'm DINIC-alive. So the user names don't match. All right. So let's very quickly make this job pass. Okay. Let's change the CI file. So now I'm not going to use a VM. I'm actually going to use my raw laptop, if you will. Okay. So if we come down to pipelines now, ooh, maybe I got the tag wrong. Well, that's why I have this open. Fuego set UID show. Oh. I think maybe I hosed my config Toml file. Oh, no, that's not why. My favorite issue, I need to be running as a root. Hopefully that's the only snag in this demo. Okay. Yes. Okay. So now you see the job passed as DINIC. Well, for the purpose of this demo, you can see I created a username DINIC just so it would pass. Okay. So you can see there's some limitations there with set UID, right? The fact that it's grabbing the username straight from the server. So now let's move to federation. So federation is going to tap into Omnioth, like I had mentioned. So I come to my profile account page, and you'll see the social sign in. Another way to get there, you come up to settings, and then you click account. So I have two identity providers here. I'm just going to show off the ones. Very simple, JSON web token-based IDP that I have a VM running for. All right. So you can see I have an active session now with my identity provider. And now when I edit my CIEml file for federation, you can see, again, I've got this tag specific to that federated runner. One of those VMs I have is using a federated runner that needs to talk with that identity provider.  Who's running the job? So now when we come down to the pipelines, you see it's running, federation passed. And this time, we see it actually ran as Dnicolif, which is not my GitLab server username, it's the one that was provided by the identity provider. So all as well. I have delegated the control down to the runner to talk with the identity provider and say, you know, who should be running the CI job? And so we've kind of, you know, elevated the role of Omnioth to manage user access control and fundamentally tap into the identities table that exists in GitLab. Okay. So make sure I'm on track here. I showed you guys the shell runner, set UID, moving to the federated runners. And okay. So now I'm going to show you sort of our pre-production testing environment. So this is no longer on my VMs. This is at austi.gov. It's our development instance. So here you can see the social sign-in. We have a few. One ID. This is a really cool authentication hub where we can tap into all the national labs, really. Not all, but many of them. I'm not going to use that for this demo here because I don't feel like grabbing my most crypto card out. But for the case of multiple sign-ins, I will show you guys NERSC quickly, since it's just right across the bay. Pull out my one-time password for MFA. Just NERSC here. Okay. But for the actual next step in the demo, I want to show you something that happens with Google. So I'm going to also activate my Google social sign-in. And let me just make sure that this is indeed broken purposefully. No. Okay. So I want to show you a failure first. I forgot to turn on the failure on purpose. So all right. So now when I come to my project, same deal. I'm just going to run ID. I've added some other stuff here. These are some variables that actually are passed down to the runner, but it's not really important to see. So I'll remove that so we can trigger the job easily. Okay. So now, you can already see here on the runner, something's failing. Well, it looks exactly like that failure we had with SetUID. Namely, it's trying to run the job as some user that doesn't exist on the system. So it's trying to run as user 116487. Well, that's my Google user account. So I, of course, don't have a local account that matches my Google user account. So how do I fix this problem? Well, as I showed in the authorization code flow diagram, really wanted to give a lot of power to the runner admins. So what you can see here in the config.toml is that we actually allow you to specify a validate auth script, which I'll show you in a second. Or actually, I might as well just show you this right now before I do the run. So that shell script just calls a Python script, which is very simple. Okay. So for this example, very basic, just have this white list that is going to say, oh, okay, I'm going to remap this user ID to the local one that I care about. We also take the access token and hit the endpoint that we need to validate against. So in this Google example, you can see I'm going to actually communicate with Google and validate the access token that they give me. So I'll restart the runner now with that validate script, retry, and you can see the job has succeeded on the runner. And if we refresh the s, there you go. You can see that not only did it remap my Google user account, which is that long integer to the actual local one that I cared about. Again, an operation performed by the runner admin. But in addition, as I showed you in that script, we took the access token and hit Google's endpoint to in fact ensure, hey, this is this is a legitimate session that they have with Google. So showed you the federated runner stuff, showed you at Austin. As I said, we really focused on what the runner admins can do with the credential information, showed you that script here. I still have a couple minutes, so I'll just very quickly show you some auditing stuff. So this is tapping back into my VMs. So if you have enterprise edition, you can actually tap directly into the GUI, you go to admin audit events. But we also dump to standard logs here. If you've never played with this before, I highly recommend it. It's a very cool parser of JSON. So you can see here, we have full entries in the log. We give these unique identifiers to each federation event. You can see, oh, hey, a CI job passed. Perhaps we put a little too much information in it, but you can see here under the target details, okay, it's a type CI, here's the job ID, what was the project, project ID, the commit hash. So if anything malicious does happen in your federated environment, you can always audit and trace back and see what to do about it. So I believe that should cover everything I wanted to go over for the demo. All right. So a few seconds left. High level lessons learned. I can see my notes. Okay. This is perhaps more of a personal reflection on the project. But I feel like one of the biggest issues we had with trying to get to the state of federation that we're at was not really having a full formal specification up front. Security is really challenging, right? They're just, you can't expose every part of your attack surface. An attacker will eventually, but you know, you trying to find that is very hard. So it's nice to have a high level kind of description of what you want, but I think it's worthwhile to take the months necessary to really expose as many parts as you can so that when it comes time for development, you don't expect anything more than just a few iterative changes. We had a challenge where, you know, we went one way with set you ID in the beginning and it ultimately didn't really mesh as well with federation as we wanted. And so it took a lot of catch up to kind of evolve it to where we need to get to. This next lesson is going to be kind of preaching to the choir hopefully, but really you should just test, test, test. We didn't start making like exponential advances in our development until we wrote that full virtual environment had Docker containers testing everything.  So I mean, I can't undersell how important it is to just test your own tests. And then the third biggest lesson, especially since we want to upstream this to the community is getting involved early. Again, one of the sort of failures, perhaps early on was that we didn't engage the GitLab team early enough with the set you ID enhancements, but with federation, we've reached out to our points of contact, met with everyone in the community, we're here at GitLab commit. And so if you really want to make an impact on the open source community, I think getting involved as early as possible is really critical. In terms of current state, we have a lot of our ECP projects. I showed some of those pretty pictures of computational physics simulations. They're all starting to use our CI infrastructure. And we're nearly at a place where everyone's going to be running through our production OSD instance. In terms of next steps, the user experience is something we want to work on. So obviously going to slash profile slash account to connect your social sign ins isn't the most convenient way to do this. So we're looking at commit as assurance where you can actually sign your commits with tokens in real time. Code scanning is a big part of where we want to go as well. And actually tapping into new schedulers is another big deal. As I said, bare metal performance is critical to us. And some of the schedulers are getting a little bit older. So tapping into things like flux on the roadmap. So that's my talk. I hope you all enjoyed it. If you have any questions, please come up and talk to me after. Thank you.