Good morning. Good afternoon. Good evening and welcome to another episode of the get-ups happy hour I am Chris short executive producer of open shift TV I am joined by the one and only get-ups extraordinaire Christian Hernandez. How are you doing today, buddy? Yeah, I'm doing good doing good. It's been It's been a long year so far We're only we're only a few weeks in It feels like a few weeks is actually only the second week, but yeah, but I mean I'm doing doing well doing well everything Hope everyone else is doing there doing well out there. You're staying safe and yeah, so before Before we get started actually do want to do a quick shout-out, right? So I think And I think I'm gonna kind of want to do more of these right kind of just give General A view of get-ups land, right? So a few of my Couple of my colleagues are doing a talk, right? So they're doing a joint talk red hat and get lab get labs one of our partners, right? So we do a lot of work with them at worldwide technologies. They're actually going to do a kind of DevOps Kind of doing a DevOps get-ups Talk there, I'll put the link in the chat there. It's on so if you know if you want to go there if you're You know interested in learning more about get-ups with get lab. I'm using red head technologies It's gonna be on Tuesday the 26th, right? So not next week, but the week after I believe And uh, you know, hey, yeah, I got an opening in my calendar. I might yeah See there you go. Yeah. Yeah, so that's yeah, I do too. So it's kind of like hey, you know It'd be a good thing to uh to check out there. So um, so yeah, so today I kind of want to talk about you know originally or You know past episodes I kind of talk about get-ups in general, right? Today, I'm going to talk about Argo specifically because There are some tools and some features of Argo CD That I think are very helpful, especially when When trying to kind of get some of your brownfield applications to Get-opsible, right? So I'm gonna turn get-ops from a noun show verb get-opsible, right? So, um I'll be talking about it. So let me share my screen real quick here I don't want to inundate, uh anyone with With slides here, but uh, where did my button go? There we go. I changed my setup. So like now Oh Yeah, that's context. Good news to try. Yeah, I think I think it'll be I think it'll be better Right in the long run, but like my muscle memory now is a little off. So yeah when I move my office into this room I flipped Two screens and it was like, whoa, this is super awkward. Yeah Exactly, but now I'm used to it. So yeah, so So let me uh, do you see that on my presentation? Yeah, and I'm gonna share the length of the slides right now and chat for everybody Yeah, so uh, the slides are available. So go, um, so feel free to take a look at that afterwards If you want a copy of those, so I'm going to talk about things like, um Sink waves and hooks right so sink waves and hooks. So this is What I like to call controlling non declarative things declaratively right so not everything in You know, not every application not every, um Infrastructure stack can be described declaratively right that's just the nature of the beast, right? That's whether it's through just kind of like You know any number of snowflake applications that you may have or the fact that you know You're used to doing like waterfall or some other process and you're trying to you know go through the The the the tribulations of getting this in a kubernetes environment. Um There are just there's just instances where you need to deploy something that you just can't do it in declarative way right and I and I call it You see the slide there get oppsing the un-get-opsable So through sheer wheel on force. Yeah, yeah for sure.  So sort of sort of thing, right? So, um and just kind of just some of the some of the things that That are that are difficult to do in a declarative way, right?  You want to like do a do a backup or something? Um, that's just one thing right and really how do you do that declaratively? Um, you need to create directory structures, right health checks to other systems or any other custom scripts, right like there are I did I did a talk once uh At um dev conf by the way dev conf also now virtual. We're just talking about virtual. Um Yeah, um sessions but dev conf now search last time I was at dev conf It was in burno right in check republic. I made a bunch of great people I got people walking up to me. I did a get up session there Walking up to me. He goes well, I have this process and they they Ended up describing this process. I'm like, oh my god people have just like these Weird setups with her and it's like not through the fault of their own but just kind of just like they get um Uh Tech debt right they have some oh, yeah, and here they heritage some systems like or like they have a system that it's like Hey, it's it's always been there. It's always been humming running. We don't want to get off of it or we can't get off of it um so You know, that's just kind of some of the things that are um That just happened right you just have a deployment process right or you have like a maintenance, right? You can do like database backup block rotating, you know, you need to do some sort of um Some sort of maintenance, right? So like just just something, you know You have to you know, make sure you're splitting the the plates in the right directions and Making sure you do the sacrifices just right in order to get your um application deployed, right? So right So in general, you know, you want to use um As much kubernetes primitives as possible, right? It's like especially if you're moving over to like a cloud native A way of deploying applications if you're looking into get ops to um, you know, accelerate that process There are things like init containers, right? So init containers If you guys don't know, um A pod is a representation of one or many containers, right? So a pod in itself can have one or you know hundreds of containers, right? It just depends. Yeah It's an it's an atomic unit, right? And part of that part of those containers is there's something called an init container That runs before Your actual container runs, right? So it's something that runs in a pod That does something and then goes away before your Your actual container runs, right? So example if there's a directory structure that needs to be created Right right before or you need to download something that needs to be local Whatever right you can use an init container for that for your application. So um other than that there's jobs, right? So jobs are Kind of like a batch process, right? So jobs is essentially a separate pod. So now we're taught, you know, we go from in pod Maintenance or setup to outside of pod. There's like a separate pod that runs that does something, right? That connects to an external system that does some garbage collection Maybe it'll mount the file system and do it does a backup Whatever, right? And then There's also operators, right? Was that I'm sorry. It was it's funny because we had uh one of my first jobs Out of the Air Force and tech, uh, we had this entire fleet of servers for Quote batch processing and it's literally just like a kubernetes object now. It's just fun to me Yeah Well, yeah, yeah technology advances. Yeah Yeah, you have a you know, you have a farm to do batch processing, right? And now Now it's just an object that comes and goes away and you're like, whoa, how far we've come Yeah, that's like that t-shirt. I'll replace you with a small shell script. This is exactly it. Exactly. Yeah um So and then the operators, right? So like if if I think you know go go lang operators are are great, right? There's very extremely There's if you if you want to pull all the levers that are there Yep, I think ansible operators though. Um in terms of like maintenance or you know, whatever you need to do um I think it's perfect for this, right? So like especially if you're already using ansible, it just makes sense, right? Put it in an operator and just, you know off you go So, um, I think that ansible operators Also is another way to do this, right? You can have an operator that spins up and does some ansible process whatever you want to do um That does that, right? So, um So in general, you want to use some of these primitives, right to go along, right? So but um Argo CD has also tool sets, right? So I say Argo CD to the rescue, right? It's not just Argo CD, right? Like a lot of sync tools have this here, but we're talking about Argo here specifically in this use case, right? So Argo CD does Does the sync operation, right? So like if you've seen past episodes or if you use Argo before when you click sync You know, it seems like magic, right? Or it seems pretty simple, but actually in the background It does the sync in things that they call phases, right? so, um These phases also have hooks, right? So they're they're so let's just kind of break this down a little bit so there's this three basic phases, right? So there's the um So if you take this little thought experiment is like you have a yaml, right? So you have something simple pod service route, right? So kind of or deployment service routes is something dead simple Right. Um, that's your that's what you want to sync There's a precinct phase, right? So precinct phase is stuff that happens before those three yaml manifests gets applied so things It could be any kubernetes resource, but you know nine times out of ten. It's usually like a job Right. Um, that does something, right? So you can have precinct. So before you sync my manifests Do these steps, right? And then there is the sync phase, which is what you would normally would see in Argo is You're taking the manifest and applying it, right? So you're making sure they're they're in sync What you have in get is in sync with What you have in On your cluster so And then there's the post sync, right? So the post sync as the name implies syncs after um Your sync processes happen, right? So this this um and all these processes process processes people pronounce it different I guess um processes processes um Wait for the other one to Uh finish successfully before the other one. So like precinct happens waits for everything to go healthy Then sync happens waits for everything goes healthy and then once everything is healthy um The post sync runs, right and then um, and then your application Um Will report. Okay, you know everything ran just fine. So so some of the sample use cases, right? So like the precinct function Can perform things like a the database schema migration, right or before deploying a new version of your application You have something that happens before um that uh That your your application does Um, like a deployment of a new version, right? So, um the sync hook, you know can be used um That hook say okay. Well, you know once the precinct happens. I need some something um Something to happen that's kind of more sophisticated than the rolling update strategy, right? Because the way Kubernetes deploys your application and by extension open shift is that it'll spin up the new pod um add it to the um Uh the services endpoint and then it will just delete the other pod, right? So kind of like a rolling update Okay And some applications require something a little bit more sophisticated than that. So yeah, um, so it's just Not all applications are just plain simple, right? So, um If they were if they were we wouldn't need these tools. Yeah, exactly Exactly So, uh the post sync hook Can be used to run like integrations chess or just kind of check the overall health of your deployment, right? So um So part of hooks, right? So if you are um running things in hooks, um You can you can delete them, right? So like if you have a job that does like one task that needs to come up run a task and then go away You don't want it lingering you can um You can delete that job, right? So for instance, if it if you send like a slack uh a slack hook, right or slack message, right? You're Um saying hey, you know after this hook succeeded just delete it. I don't need it hanging around You can also have yeah get it out of here. You can also have if something failed, right? So if something failed, maybe you do want to get notified, right? So if something failed You can have a job You know spin up and and notify you and then delete deleted, right? So, um, and then uh, there's something else called before hook creation, right? So it'll delete the previous one. So maybe you do want to hanging around, right? You want a job hanging around so you can look at the logs But you want to delete it when the new one runs, right? So that's another before Before hook creation. So So that's kind of the uh the idea behind sync, right sync Sync phases and hooks, right? So There's another mechanism, right is what we call sync waves, right? So um So sync waves along with the hooks sync hooks The waves is just kind of just put it simply the order in which manifest gets applied Within that phase, right? So, um, it's ordered by number, right? So like in your Pre-sync if you have like one if you have three jobs you can have zero run And then when that's finished then number one runs and then when that finished number two runs So it's kind of it essentially it's an order of operations, right? And there And each wave is within each phase, right? So if I go back So you can have a precinct, right that has zero one two a sync that has zero one two And a post sync has zero one two and it'll run in that order, right? So, um If without specifying the default value is zero meaning that if you don't specify it at all Everything gets applied at the same time, right? So which you may you may not want, right? So for example If I am deploying, you know A front-end web and a database I want the database up even before the web gets Splint up, right? So if I have an application that I haven't written That wait, you know because some application you can write to where it'll wait Until the database is up before it'll come up. Um, this is a way to actually, um, make sure the The manifest gets applied in the order that you want, right? So you want database to come up And then the front-end database may be Manifest have a value of zero And then the front-end has a value of one, right? So then right And then your job has a value of two, right? So like you then you can have it, um Run in that order you can also have negative, right? So like if you want Um, if you want something like if you deploy something at zero one two and then You need to deploy something else.  This actually needs to run before zero So you can set it to negative one. Yeah, so it's only like it's like a weight at that point, right? Like just yeah, yeah so go negative three and you know You give yourself room to put in one and two if you need to right one and two. Yeah, exactly. Yeah Or negative 10, right? So So there's there's a lot of ways to say this. Yeah, some people do it like 10 base, right? Yeah, I would do a 10 base thing, right? Like that's actually what do I do that is like that I forget but yeah, there is something that I do a very 10 base No, it's the menuing systems on all my website. It's very much. Yeah 10 2030 that way if I need to squeeze something in the middle You're planning, right? So that way you don't have to shuffle things. You can just stick it in where you need to um, so It's just an integer, right? So it's you know, whatever Whatever makes sense to you. So that's that's another that's another way to do it, right? So question is How does this work, right? So like you're yeah, if you're watching if you're watching this, it's like, okay, so All right. So how do how do I put this together? So Um, so first when argo starts to sync It orders the resources in a particular way, right? So first it'll order them by phase Right, so it'll take all your manifest and okay, it'll look at it. Okay I'm gonna put all the precincts in one batch. I'm gonna put all the sinks in another batch and I'm gonna put Um, the post sync in another batch, right? So once it does that So it'll sort of that way And then within each phase it'll sort them by the way that they're in right so it'll start from the lowest number And then it'll apply it going to the highest number In the precinct and then it'll do that for the sink and then so that's how I'll order the yaml and then it'll um Order it by kind right so this is a byproduct using customize It'll put the namespace first and then you know the namespace and service and deployment or I forget the actual one But I know they do the the namespace first um and then by name right so then it'll do an alphabetical sorting of the things right so even before Even before it even starts the um the sync process it has to order them and this is like the process It uses to order those manifests And then um and then argo determines which number of the next wave to apply right so it'll it'll do the phase It'll look at the wave and I'll say okay, I'll apply zero one two three right um It'll apply it only to anything that it notes that it's um That it's out of sync or I don't know Yeah, so it doesn't like reapply things. It just only applies what it sees that is out of sync. So And then um and then it'll apply those resources right so it'll check to see what's out of sync and it'll apply to those resources And then it basically repeats this until everything's healthy, right so So important thing to note Is that since your application could be healthy at any point in your wave um Since it waits for the health to report on 100 healthy Um if it fails at a certain wave it may be your applications will never get to healthy, right? So You want to make sure yeah, you want to make sure that when you're writing a job that you're writing the The logic right in the the job because the job could be anything right could be Could be an enhanceable playbook. It could be a script, right? You're writing the logic to where If you don't care if it fails You may want to make it report healthy even though, you know it failed, right? So so just just note that any phase It could if it fails any at any point It'll yeah if it's like a failure is acceptable You still want to set that exit code to zero kind of deal, right? Like exactly. Yeah. Yeah So it's like to clean up the directory and like an error comes back because the directory is empty. Mm-hmm. That's probably okay Yes, you know exactly. There's nothing here. I can't do anything. No, just keep going Yeah, just keep going. So you kind of want to just keep that in mind that um within all these phases you just want to make sure um You want to report healthy when you know because sometimes, you know errors are okay. So and you need to program that in, right? So, um Argo assumes, you know, what you're doing. So that's the uh good and bad, right? So, um So, um, so yeah, so people say yeah, how does this work? But like But no, we're really like, can you like walk me through it because for me Yeah, I'm the type of person that's like we can talk about it for hours But can you just like walk me through it? Right? So like same. We'll go through a simple. Yeah. So we'll go through a simple use case, right? So, um So this arrow Going from left to right um depicts the passage of time, right? So, um, and in each phase This is how the phases happen, right? So you get the the precinct sink and post sink, right? So let's just take a simple use case And um, let's say you have a job, right? So job This job manifest runs in your precinct And and you'll see in the in the demo, right? Well, I'll walk you through kind of an example that I have You'll see that each hook has an anchor notation to it. So you know, it's a hook Instead of an actual manifest you want to apply. So that's how it differentiates, right? So first phase happens You apply the job little hook comes up and then it'll report, uh green, right? Okay. So that that succeeds So once precinct happens, uh, then the actual sink happens, right? So, uh, I want to have a deployment a service and a route Um, and then maybe I have a uh, in in this sink in in this, uh, in the sink phase Uh, and also in the sink phase, I have a hook that runs a job, right? So, um So I have here, uh, a wave, right? So the sink wave here is a zero one two three Right. So I so like I so it basically orders them this way, right? So what ends up happening is that first the deployment runs That turns green and then the service gets applied and that turns green, right? So as you can see and then Um, that number two comes along, right? So since service Turn green now the route turns green and then finally since I have my job set at three It runs that last, right? So, uh, if I back up a little bit, you'll see that I'll it'll apply it in that order, right? Um Since this job, I didn't annotate it with anything um It'll uh, it'll stick around after after the sink finished So after the sink finishes Now we have the post sync, right? Um This post sync I have a job that does something right and I have it annotated. Whatever. Yeah. Yeah, clean up Maybe send us um a slack message, which is kind of like the demo I have I have said Yeah, I have a slack message. Um Or whatever, right? Whatever you want.  Yeah Yeah, or touch a file, whatever you want, right? Yeah, could be anything Give me anything you want. Um, this particular job has a hook But it but I also has an annotation called delete on success meaning when I run I want it to run do something and then delete itself, right? So it'll then run successfully And at the end it'll delete itself, right? So, um Pretty simple, pretty straightforward Um, this is how Argo applies its manifest, right? And you can see um How this, uh, how this works, right? So, um So, uh, I do have, uh, some hacking to do. So, uh, right before we get to the Hackery Some hackery, right or our keyboard time or as I like to say hacking away until it works, right? Um, I don't see, um I've ever seen the those O'Reilly books Those fake O'Reilly books that people make. Oh, yeah. I've got a good one. Yeah They're like, yeah, there's there's a few of them. There's one about my favorite one I like two of them. One of them says trying things until it works by O'Reilly or Copying pasting from a stack overflow. Yes. Yes. That's a great one Let me uh, I'll I'll get mine shared out and I'll share the link in a second here. Yes, it's pretty it's pretty, um Which is funny because we've all done those things Let's especially the the copy and paste is like a copy paste change the parameters. Oh, it does work. Okay, cool right so, um So I do have, uh a quick example, but, um Before we get to that example here there was kind of like a, um A lively conversation, right? So is is as I've said before there is like the the argo user groups Um, within red hat, right? We have like a little what I like to call the argonauts That we kind of had a lively conversation is like shouldn't some of this stuff happen in ci and not in cd, right? Good point So stuff like precinct like like, uh, you know Or like a sync process or a post sync process Um It was really more the precinct, right? So like if I have a job that does A database backup or database maintenance shouldn't that happen in ci. So, um That's a very interesting conversation. I would it I'd be interested in like what other users have Um, you know, feel free to you know, either chat it up or hit me up on on twitter Um Talk about the experiences, right? Um Um I think There's a case to be a there's a case to be made to have things done In ci versus cd, right? Because remember argo cd is a is a cd process is delivery tool Stuff like maintenance and things kind of almost seems like a ci process. So, um Still you can do a lot of things. I I am Of the opinion that if you're doing maintenance You might want to put like an under construction page up or something or Yeah, like some kind of status Yeah, some kind of status or like a like a banner or something. Um, the argument was made because it's dev ops, right? And I think I think it's a good argument. It's like shouldn't the application tolerate maintenance windows, right? Um, right So that's that's also a very valid valid good point.  So like I mean I've been there before I've been an admin. I know chris.  Yes, it is but We are we have tech debt like what do you want me to do? Like I I agree with you But we have to do it this way. Um, yeah, and that very much has to do with uh The the fake orally cover I sent Yeah, that's that's really funny I said that many times in my career. The mess I just walked into Yeah, yeah. Yeah, what what what is this pile? I just run into. Um So so yeah, so um You know, if you guys have opinions, yeah, feel free to reach out put in the chat. Uh, I would love to hear What you guys experience out there? Um in terms of like where should this happen in the process, right? Especially especially get ops is so like, you know, we're still in that like an exploring phase with get ops that Uh, I'd be interested in where we end up in terms of where should some of these things happen So Cool. So I have right here Let's get some hacking going on here. So I have Um, so I have this application, right? So like I always use a sample application It's one of those things. Um About you chris like you wrote once a long time ago and you can just use a crap familiar with it You just go with it. Yeah. Yeah, so a simple code application, but I think I get It'll trace the point here. So I have um An application here is a simple code application, right? I have some entries in the database um Right, uh, it's a functioning app, right? They even have the chris short t-shirt, right? So get it while you still can Um It's a hot item Yeah, like I've had, you know, a cold tablet 15 bucks or 140 or whatever, right? Um Right so I can um You know, uh use this application here. So, um So, um, I have this app synced with argo, right? Um Yeah, right. So this is all synced. It's all all up and running It's it's just funny how many objects it takes to have a simple application I always look at this. It's like wow, there it takes a lot To have only a two tier application, but there we go. Um Well, it's not it's not just a two tier application, right? Like it's a two tier application That's highly available and you know redundant and you know, it's got all the bells and whistles of running on kubernetes, right? Like that's well, that's the thing that makes it complex, right? Well, and also um I think it illustrates the point of the power of kubernetes, right because Right. I with two manifests I declaratively say, you know, it sets up the networking for me. It sets up the um The ports for me. It sets up the the The dns for me.  Absolutely 100 right on that one. Yeah, so, um So cool, um, so I have this application, right? So then I have um again here. I'm using uh again Octotree Pretty cool. Um, finally Yeah, y'all I can't believe I like it's one of those tools is like I can't believe I lived my life The way I did up until now, right? Um without this tool So I have a um This application this customization file, right? So I have my resources uh synced up, right? So I'm actually pulling in the manifests the um the actual application from another repo So that's another cool thing from customize. I can um I can pull the manifest from wherever I want, right? So and this is I'm like, okay, cool I'll just take an application. I already have um, but there's a few things that happen, right? So I have A a precinct job, right? So if I look at that precinct job You'll note that here line 10 um I have uh, sorry here to start at eight, right? So I have post sync, right? Um, is this oh, this is a post sync. Where's my precinct? Uh precinct, sorry. Yeah, so my pre alphabetical order not the right order Not the right order. Yeah, exactly So I have a precinct, right? So this is this job is going to happen to precinct Right? It's the first thing that's going to happen in this wave because I set it to zero Um, and it's going to delete itself once it succeeded, right? And what it's doing It's just a maintenance script, right? So part of the maintenance script is like I'm going to throw up a um A banner saying that this application is going to come under maintenance, right? So um So that's just one, right? So then the next thing that happens is that I have a Uh db backup, right? So that's my database backup happens in the precinct Phase uh and wave one, right? So it'll happen after the maintenance script fires That'll finish it'll run this one um and And then after it's done, it'll delete, right? So this deletion policy is hook succeeded once once it Once it succeeds get rid of it. I don't want to hang it around Um and then finally in the last thing in the precinct Since it's number two, it'll happen third Is a log rotating, right? So I have a script that'll rotate the logs for me And then again hook succeeded it'll delete, right? So that's the precinct phase Now in the sync phase I have to go back to my customization I have this, um application, right? So it'll So it'll pull in all the manifests from this repo, right? And if you look in this repo If it exists One second I hope there's there we go. There you go. There it is. So, um, if I look at here, um This application is this is uh actually has Um a sync wave applied to it, right? So so it'll actually deploy in the order of I think first the db Service no no actually the quota gets deployed first No, I'm dumb obviously the namespace needs to be cut that's first then the quota, right? Because namespace always has to go first But I have the this application itself in another repo Has the sync waves, right? So So you take that into account, right? So this is kind of like a like a like like one of those russian dolls, right? Like I have a repo within a repo within a repo, right that reflects those things so you kind of can kind of get Need to like in your brain map it out in your head, but uh, right I To be fair and blunt and honest oftentimes what I'm trying to understand what I'm doing like I have to draw it out still Right. So I have this little like tiny little whiteboard Tons of stuff sitting on top of it just right here in my desk because it's like Wait a minute. How's this gonna work again? Yeah Well, um, what uh someone I know someone actually One of the kind of Not too much of a side note, right? Like I work on this project, you know, chris called helper node, right? And we're in the we're in the process of rewriting it and golang and one of the um, one of the guys You know, we have you know bi-weekly sync calls about it one of the guys actually has a whiteboard In his office at home his home office has a whiteboard in my office downstairs The old on had a whiteboard in it. Yeah Yeah, he needs to he needs to like sometimes he'll need to like get up and like he'll like draw it out What he was trying what he's trying to say so Yeah, I totally get I totally get it. I my finished office will have like the desk will be on wheels It'll be hardwood floors. I'll spin the desk around and I'll be writing on the whiteboard during shows, right? Like he'll be awesome. Yeah What we need I know we discussed this before is we need The lightboard I think I think there's one in the box very expensive kind of hard to use Maybe overkill, but yeah, I like I would like to have one if they're free Yeah Yeah, if anyone wants to send us send chris one, um, feel free to Yeah My house will suddenly become a studio for lightboard users across southeast michigan There you go. There you go. There's a little side business there. Um, yeah, so so So yeah, so I'm pulling this in and this repo has its own sink wave So that that'll and you'll see the example. It'll it'll get a deployment such a way. Um If I look at The where is my posting so then I have a posting job, right? So, um Posting job it runs post sync zero. There's nothing else running it. So I could probably left that out and it'll delete and hook succeeded and this post job all it does is Uh It'll um send us a slack web hook Right telling the the deployment's done Um, right and I have a secret there and actually if you want to take a look at the secret you can Um, where is Where's my secret? I wish I would have named these consistently here. Hold on you still can I guess I I guess I guess I still can right right There it is. Um, it's actually a sealed secret. So I'm kind of using um, because I don't want random people posting to the red hat slack. Um, what? Really? Yeah, so I have so I'm using multiple technologies. So, um By the way, so I do I do dog food So, um, but it does send a slack hook right here. So, um Um So, yeah, so it'll happen in this order, right? So, uh, let's fire this off here um Is there any way to move this? I know you don't see it on your screen. You know the uh the um the zoom The bar bar. Yeah, I don't want I don't want to ask them and close it Click the green part and you can like drag it left or right. I feel like oh, yeah, you can Oh man. This is a game changer. Can I there you go? Oh, perfect. Cool Um, although now this thing All right, uh, let's fire this off So, let's uh, let's click sync here I'm going to sync everything So, um fair warning, you know, since I'm doing like maintenance and stuff like that, um This is gonna be a lot of uh watching paint dry, but you'll see here that there's a precinct that fires off Right, so you see the little um, I don't know if you can see it Let me know if you needed to get bigger, but um, there's yes, I would just say yes people needed to see it So there's a little little anchor that denotes that it's uh, it's a hook, right? So, um, right now it's in the precinct phase and it's uh launching this pod that's doing that maintenance script, right? So part of that maintenance script as you see here it launches um It launches a pod here and It'll, uh Put this under maintenance mode, right? So I have this under construction page Right, so um, it's like the worst thing for seo by the way, but yeah So like I have this under yeah, exactly. Um my app is not working right now because I just is under maintenance mode, right? So then here Um This little hook here It does a database backup Right. So as that's as that's running My application is down construction. Makes sense. Makes total sense makes total sense, right? You don't want You know people writing tables writing to something that's unrightable bad. Yeah How would you solve that in a more production environment though? Would you have like some kind of proxy or cash just to like Yeah, hold on to things. Yeah, that's tough. Yeah highly redundant databases or a thing The database is always a problem, right? Um, right well state is always a problem, right state's always a problem. Yeah, so a christian poster, right? he he He wrote a microservices book and he's he does microservices talk and he always says the issue always comes down to your state That's gonna be your biggest problem, right? So, um data gravity all those. Yeah, it's There are terms there for a reason Yeah, exactly. So, um Once yeah, so once now I have a post sync here. Uh, this post sync is actually if I go here It'll uh, I have this little Channel here that I created for you and me chris you're part of it see so now you see here that it just came up Uh deployment was successful, right? So it was just uh, I actually do have you have you have you added Did you really? Yeah It says you're on I don't know. I'm no, I am in it. I see it now. Yeah, it's you. Yeah. Yeah, so, um, you you just got a slack message Yeah, so you just got a slack message That uh, the deployment was successful, right? So at the end I send a slack message and then those go then they go away as you see here in, um in our go, uh the, um The uh the jobs not only came but they left, right? So if I actually Drop down here if I do oc get pods under a price list Right, uh, none of the pods lingered, right? Uh, except the post deploy because the post deploy um I have it set to I don't have it annotated, right? So let's look at, um back Let's go to get ops examples Get a price list And then where's post deploy? Yeah, post deploy. I just have it a sync way, but I don't have it deleted, right? So that's the only one that sticks around. Um, if I do oc get jobs um There's only one job instead of the the three or four that I That I deployed there. So, um So, yeah, so that's, um Right, I use a slack. Yes. Yes, we do. Yes, we do. We use a lot of tools for communication actually We use a lot a lot of tools. Yeah, there's usually For companies I've been around for a while, there's not, um, there's Very rarely you end up using only one tool. So, right So, well for it and in red hat, that's very very true So IBM also uses slack, right? They have the little they had made a big slack deal. Um We still can't talk to each other, but there we go. Um, you know What's um What's really cool is you can invite other slack channels in specific channels, right? So I know the um the ansible slack. So ansible has yes, they have their own sack Oh, yes, us open had their own slack before they were owned by red hat. Yeah Yeah, yeah Well, and also the core os slack, right? So our Opus shift slack that when we're on you have our own account. So it's So don't don't don't act so surprised. You'll be more surprised finding out how actually We have it set up. Um Um any matrix usage then? Oh, yeah, totally totally plug-in. Yeah, that's why my neck hurts all the time Yeah, that's that's that's what these slots are. Um It's for your razors. No Yeah, it's for your razors. Yeah, exactly. Um, so yeah, so I have, um Um So yes, as you can see some of the you know, you can use like sink waves and and and Phases to do a lot of things. I actually use it For a little bit more complex setup. So if I go to What is this here? Cluster config. Yeah, so I actually use Um So here let me go here. I'm manifest. There you go. So deploying the efk stack Um, actually gets kind of Complex, right? So I actually have you know a sink wave So, um, you know kind of airing a little bit of a little bit of dirty laundry In terms of operators. So the operator lifecycle management isn't um It's it's one of those cases where it's not get ops friendly And it's it's not a knock on olm. A lot of things aren't get ops friendly, right? Oh, yeah I mean, which is Which is the reason yeah Yeah, which is a reason why there's sink waves, right? It's it's not just because get ops is so so new like it's obviously not friendly um I was running into an issue where I am uh Trying to deploy our logging stack and for those of you who haven't deployed our logging stack Our logging stack is actually a meta operator. It's an operator that launches three other operators, right? And so I got into this kind of limbo where It was trying to apply a manifest for a crd that doesn't exist yet, right? So um, just because you create the crd it the it takes a while for the cluster to um To see that crd, right? And I figure the crd and the cr get applied at the same time. You'll get an error and that's just just in general just in kubernetes in general um and so I basically came up And this is another shout out to the canadiens, right? They gave me this idea Where basically i'm doing an oc wait Right for the crd to exist Essentially, right? So I basically I I wait for The crd to be established. That's all that's all this job is doing, right?  So before step seven, I need to wait I need to wait until That that resource exists, right? So even in a even in kubernetes you run into race conditions, right? um So i'm basically saying until this um This crd exists I'm just gonna wait, right? And then I'll I'll sleep, you know Things or whatever and I just keep ju ju ju ju still waiting and then when it's done cool. It's done. This job goes away Well, it doesn't go away six around so I can see the logs, but um six runs and then seven will run, right? Which seven I think it's this Yes, so then I can create my cr right so this step six waits for The crd to get defined in the job step seven says um Okay, now create the custom resource against this. Uh, the create the opera and right from this operator And so um and I used to have this um Because so this here Here's another another cool option. This is a sink option some of the little known sink options, right? So since we're talking about sink waves um Argo does a kind of sanity check well Argo slash customize kind of does a sanity check before It tries to apply something, right? So if you try to apply kind cluster logging and cluster logging doesn't exist It'll error out um, it won't even try to apply it, right?  I kind of know what I'm doing ignore it for now Right because I'm going to fix it later, right? I'll fix it. Like they say I'll fix it in post production I'll fix it with sink waves, right? I'll I'll write a job for it. So that's another thing um You need to look out for is Argo does kind of like a dry run of everything before does something And you can actually with this annotation you can let me make this a little bigger with this annotation here You can You can tell it to skip that on dry run. So just took another another Tip, uh, you know tricks from the road, right? Some things that you can do Especially if you're you know, I think kubernetes in general is going to move into a crd centric, right? Like Um, I saw this coop. I saw this kubekan talk Uh, given by uh, I forget the name of the gentleman. He's from from google He says I want everything to be a crd like pods like I don't want I don't even want pods in tree like for Right, like yeah, everything needs to be controlled somehow, right like exactly So I want that to be a crd, you know, like that's I think That's very interesting at that point kubernetes just becomes an api Like an object management Kind of tool.  It's just gonna be So that that that'll be interesting to to have that um to have that unfold here. So So cool, so um last 10 minutes here. I don't have anything else. I don't know if there's been I know you've been uh, oh, we're talking about, you know Matrix is a replacement for irc not actually the matrix the movie kind of thing. So this is news to me Oh, interesting. Yeah So i'm looking at I guess they I guess the fedora community is trying to move over to a matrix based Tool and it looks like it's called element. Um So yeah, check this out That reminds me of uh of this, uh Yeah Exactly. No, yeah So no matter like right no matter what you have you'll have this one person say look I just have my rc set up the way I like right. I'm gonna move off of it Yeah, well, yeah, look you you did you did put it in chat. So sorry I missed that but you did you said, uh, where's that xkc I'll put that in chat so like these other guys can laugh at it, but um, yeah, it's just like No And it never fails right like your communication systems will Never go away unless you have an intentional design behind it from the get go that is replaceable, right? Yeah, like You know everything needs to be thought of and built in a way that it can be replaced Right, like that's the way of the future because the thing you're running right now is not going to be the thing You're running five years from now Right. Yeah. It needs to be composable and constructable in ways that you know are Well, you know And pluggable for lack of a better term Yeah, what it's it's just funny how we uh, I like this um My git commit shout out to the uh to the canadian guys My git commit is Gerald Gerald's always right. So that's kind of like an inside joke Yeah Gerald's an architect in uh in in our in our canada team and um We have this inside joke that we say well, what does Gerald think and like well I mean, you know nine times out of ten whatever he says I go. Oh, yeah, maybe I should have done it that way So it's kind of like an inside joke. Um, anyway makes sense um Yeah, so it's just funny how we you know, we as engineers We as architects we're always trying to push like our client base like we're trying to push them forward push them forward Like try try the new and better thing, but yet us We stick to irc. So it's just kind of weird It's kind of weird like no like, you know, I'm holding on to this irc. I mean, I do it too like I've been trying to use um Vs code, but yeah, I just have vim set up So like Just the way I like it you could do the same like literally the vim plugin For vs code should give you most of the functionality. Yeah, I've been I've been trying like I've been actually trying Like I had to do the same thing and yeah, I haven't like looked back since making the conversion And now I'm trying to use more of the plugins and extensions that you know, we make like the server connector, right? Like oh never leave your ide, you know like the the open shift connector You never leave your ide and you can do things on your clusters pretty wild It that's it is really nice and the vs code does have some of these plugins where it's like, oh, that's that's neat Um, so I've been trying to use it right like I've been but I've been a vingy like most of my career And I have all these plugins like I have all these you have all the vim plugins and everything and this is like And you know, I but like I've been making a conscious effort to try to make these uh vs code as much as possible. Um, so Yeah, I mean do your best. That's all you can ask for You can ask for you So just just some of the things because I use uh buffers in vim like I'm probably like a few people, right? Like I I have a buffer. I loaded it, you know loaded in memory I can jump into another file open that buffer like some of those things just It's hard to replicate right with the plugins, right? with the vs code plugins like I found workarounds and they do plugins mostly work, but Yeah, so there's my yeah, so I'm even guilty. So my only point is I've even guilty of like holding on to Technologies right like irc. So yeah Yeah, you know Could you give some more uses use cases for pre sync and post sync type jobs? So clean up is the easy one for post sync, right?  I want to get rid of them You know that kind of thing Yeah, so uh, so I'm trying to think some of the so some of the use cases that That I thought of are the ones that that I presented, right? That's why I clean up You know tagging certain images um I'm trying to think of doing something with storage. I would imagine Doing some storage cleanup or like mounting some file systems Well, I could you know, I could see you take like a backup of the current state of the application just to make sure, right? Yeah, you're doing a new deployment kind of deal like there's there's all kinds of crazy things you could do pre-hook Yeah, pre-hook especially with ansible like with the ansible. Oh, yeah operator Yeah, if you're gonna launch a net like a like a job or an operator that'll Has ansible that you can do a lot of a lot of things. I I've I've got these physical volumes. I want them backed up to s3 Yeah, you could totally do that Yeah, yeah, definitely. Yeah, you can you can you know throw up tagging images, right like working with images I think that is is a lot. Um, you we can do a lot with image maintenance image maintenance on the For your images on a precinct phase, right? So like you have like a version Um, yeah database migration is like probably the most clear cut. Yeah. So like migrations is probably The most clear cut way of doing Doing this right and and also especially if you're using like an external load balancer You can do like like blue green sort of things. I mean our argo does have a Um Maybe this is something we can We can talk about in a future episode with argo. Yeah doing blue green with argo. They have Argo rollouts, right that do nice that do some of this stuff, right that do some of this blue green for you. Um But I can see blue green happening also with like a precinct phase, right? Like you You know deploy a new version, right?  So Yeah, someone says they're using uh everything is latest for them and it's like, whoa Latest hopefully in your registry not like docker hub or some other larger public registry, right? Like Yeah, so actually, um, yeah image management is definitely like tagging images Like production and stuff like that, right? So, um It's like one of the biggest things, right? I I can see right because like in development You'll probably have it tagged as dev Right and then in production. You'll have it tagged as prod sometimes are the same sometimes they differ but you know, you you want to make sure, um because You know, that's part of my maintenance the way I put up the maintenance page. I just tag my um my production image I tag that I make sure that tag of production matches my under construction tag, right because all it is is just like a tag And then it does, you know, it does a deployment. So, um Yeah, if the helm chart defaults the latest we're running latest Well, there's that now if your automation and tools are calling for latest, okay Well, and that that's actually uh, glad glad I brought up helm. Helm's a An episode I'm gonna do it in in a later show how to work with argo because Here before we just been working with static yaml. Um, but you can actually use helm in the background um For some of this stuff. So, um There's in some helm charts, you're able to supply which version of the image you want to use so that's true and you know if if you're using I always say like you always need to keep your own version around, right? So that gives you the flexibility to do whatever but Yeah, if you're just gonna pull from a public, you know helm repository of some sort Okay Do you do you You do you is exactly whatever the I mean, whatever works best for y'all just do that and you know, there's no right or wrong answer It's what works best for that organization That's definitely true. Yeah talking about tech debt, right? So All right, cool So all right top of the hour top the hour. I gotta go to a meeting good fun stuff And we all have to go meetings. Yeah meeting meeting meeting everywhere. So yeah, thanks for thanks for joining us today everybody Thanks for uh, your engagement in chat. We always enjoy, uh, you know hearing your stories and everything and, uh We look forward to seeing you next time here on the get off to happy hour Yep. Yep. Thank you everyone. See you next time


This afternoon, we are going to go over the sort of the basics of differential gene expression analysis for RNA-seq data. Note that CVW offers a more in-depth workshop for differential gene expression analysis, RNA-seq analysis. So here we are just going to give you sort of the conceptual overview for how you would start with RNA-seq data. We're assuming it's been processed already, meaning that you've started the raw read, they've been aligned to the genome, and you've used tools to call accounts for gene expression. And we are going to use the EdgeR library to call differentially expressed genes, where this we're going to make use of the concepts that we've been learning all along, notably the GLM concept from this morning. So let's get into it. So at the end of this lecture, you will understand sort of the key steps in identifying differentially expressed genes using RNA sequencing data. You will learn how to use QQ plots, we actually decided the p-value histograms would be too much, to gauge how much signal you have after multiple hypothesis testing. We tend to do a lot of multiple hypothesis testing when we do genome-wide studies and we are trying to find a signal amidst all our genes, so how do you adjust for that. And then we will create a very common visualization called a volcano plot to look at the results of differential expression analysis. So here in a nutshell is the workflow for differential gene expression analysis. You start with a raw expression count matrix, and remember that in an RNA-seq, you are measuring, you know, how many molecules of a particular gene you found in your library.  And so this is where we start. And then, you know, we exclude the genes that have a low read count to limit the number of tests we are doing, because of multiple testing burden, we'll talk about that. Then we have to equalize the samples for some sources of technical variation between them. And then finally, we build our statistical model, and EdgeR just does this out of the box, but it just helps for you to know what's happening a bit under the hood. We run, you know, a gene by gene hypothesis test, which is to try to see which of the genes expressions is explained by our variable of interest. Say, you know, disease, not disease, responder, non-responder. And then we correct for multiple testing. And the genes that survive multiple testing correction, they have an adjusted p-value, you get one for each of your genes. You apply a cutoff. And the genes with the adjusted p-value below that cutoff, those are your differentially expressed genes.  And then you can go and do fund downstream things like what pathways are these genes in, what else can I say about them. And that is a topic of other courses that are covered under the CVW umbrella. So let's start here at the top. When we do our NAC analysis, instead of measuring one response variable, we are measuring as many variables as there are genes, right? So those are all of our measures we're looking at. We tend to do this in genomics. If you're doing DNA methylation analysis, you might be measuring anywhere from half a million to half a million, yeah, half a million to almost a million number of measures, right? And at the end of the day, you're looking for what parts of the genome seem to have values that can be explained by your factors of interest, okay? So you've got 20,000 measures. And at the heart of it, you're doing 20,000 individual statistical tests. So this model that we've conceptually put up here, you're fitting one model per gene, right? And that's why you get, and then the p-value that you get is for your treatment effect, your effect of interest. So you've got 20,000 p-values. Now, can somebody define what a p-value is to me? We talk about them all the time. What is a p-value? What does that mean? So a p-value, you guys are all getting in that area, but the definition is a p-value is the probability that you would see a value as large as the one in your data if there was no effect, okay? If the null hypothesis is true. It's a probability that your effect size would be this large under the null hypothesis. The null hypothesis means the treatment effect is zero, okay? So a p-value of 0.05 means there is a 5% chance you would get a value this big if the null hypothesis is true. If there was no effect, if there was no effect, 5% of the time you would see a value this large. So what happens if you do 100 tests? 5% of the time you could get a value this large, right?  So this comes from doing something again and again and again, doing the tests over and over again, right? So this is a multiple testing burden and this is something we deal with when we do a large number of statistical tests such as happens in genomics, okay? So what do you do with multiple testing burden? You have to limit the number of tests you do and you need to correct for as many tests of Utah. So for example, if I did do 100 tests, then a very conservative way of correcting it is I need to take that 5% p-value and I need to divide it by 100 because I did 100 tests, right? And that is a very conservative method of correction and it's called the Bonferroni method of correction. So that's what it's very simple, it's very simple mathematically and it follows from this definition of p-value but it tends to be conservative. So there are other methods of correcting and a very popular method is called Benjamin E. Hodgeberg correction, okay? And all we're going to say is when you do multiple tests like this, you're going to have a multiple testing burden. And if you want to ask the question, how many of my genes are differentially expressed? You do not want the uncorrected p-value, you want the corrected or adjusted p-value, okay? And how do you try to limit this burden? You try to reduce the number of tests you do and then you do some other things like there are some tricks built into edge R, right, which we'll go into in a bit. Okay, so one way of reducing multiple testing burden is to remove genes from the tests which are unlikely to have any effect. For example, if I have a gene that's practically unexpressed in all the samples, I'm just, it's a waste of my time and my multiple testing whatever buffer to test that gene, right? So one thing we do is it's a heuristic, you say, I'm going to exclude genes that are expressed very, very low. And then what is expressed very low means it's like a field convention, but I'm going to exclude them. So if 20,000 of my genes, 5,000 of them aren't expressed very highly, you've lost, you've already removed 25% of your tests, right? So you'll see a multiple testing burden is that much less. Okay, so that's, that's why we exclude genes with low rate count. Chances are you're going to find a signal in there, you know, you'd have to think of a scenario where you could have a low expressing gene, you know, that's so low you don't even have enough measures and you're still going to see an effect. Can you believe that effect? Could it be sampling variation, right? There's other assays as well. So in methylation analysis when you have close to a million data points, sometimes you will rank your measures based on their variance. Variance is standard deviation square. And you say, oh, I'm only going to test the top 50% of varying probes. I'm not going to test the others.  But, you know, you kind of try to balance it based on your experimental design, like if I have only 10 samples, right? Then you're trying to fit this model with 10 data points for a million points. Multiple testing issue. How big is your signal going to be? Is half the genome changing? Or are you expecting one or two genes to change or five, 10 genes to change? Or do you not know? So that's what, that's why we exclude genes with low count. Remember, try to think about where the counts and the count matrix are coming from. You are sequencing, sampling RNA molecules in your sample, right? And that sampling is going to be variable from samples to sample. Some samples happen to get sequenced a little bit more. Some samples happen to get sequenced a little bit less. Your sequencing assay is not guaranteeing exact same number of total reads coming from a sample. So there's going to be some variation. If, you know, think about it this way, if your controls happen to be sequenced twice as much as your cases, then the RNA counts are just going to look, they're going to be twice as high across the board for all your controls, right? So if you have five molecules in your controls, sorry, what was I saying? Let's say your cases got sequenced twice as much. Okay, let me start over. Suppose in the sequencing, just by chance, a lot of your cases got sequenced a lot more than your controls, say factor of two. Now you look at the RNA count matrix, raw count matrix, then all of your cases have a factor of two compared to your controls. What happens if you don't normalize for that variation? You're going to call everything a significantly different, right? But it's not real. It's because of variation. So you need to normalize for that. Then you build your statistical model.  And then you have the full model, which is, no, the expression is a function of, you know, the intercept and a function of the disease state, right? This is what we want. These are the genes we want. This is a full model. So what RNA seek, you know, tools like HR do is they, for each gene, they will fit this model and this model, and they will, they will use a likelihood ratio test to ascertain which model explains the data better. Do the data better fit the full model or the null model? And the p-value that you're getting is from this likelihood ratio test. Okay. And because this is for count-based data, if you want to test that there is some kind of additional variation in count-based data that's coming because of biological factors, you use a negative binomial distribution, which is a type of generalized linear model. So if you use, depending on what genomic assay you use, your model has to make sure to follow the assumptions of those data, right? You're not trying to fit lines with your RNA seed data. And then you need to correct for multiple tests.  Okay. So now we are going to go look at a worked example for RNA seek analysis.