All right, I'm going to start working and go find a beer. 20 minutes. Yep, I'm going to find a room at the stop. OK, hi. All right, so I'll be as brief as possible. I tend to speak very softly, so if you cannot hear me or my English is not good, just tell me. Yeah, show me. So, yeah, I'm going to talk about how we build feature ads serves. When you're starting from stories, like the traditional scrum stories, you will have many things happening at the same time and the way you usually will build this or build it into like staging servers or whatever, it's either everything goes in or nothing goes in. But so it doesn't make us very reactive. The day we are on microservices, it's going to be another story because we can really decouple things as much as possible. But in the meantime, we had to find a solution because we have business pressure. So how do we manage to build a magnetic app but still be able to deliver by feature or at least to the product guys? So that's the talk I'm going to give tonight. So, yeah, it's a teamwork. So I'm not alone. There is people behind. So I want to just mention it about me. How are we going to skip this, this or so? So that's the agenda. So what do we do at serve, the stack we have? What is the problem we are solving? And so the feature build, I have a few slides explaining what's happening. I can also do a live demo of it. Then back to the problem, like, okay, let's have some retrospective on it. Do we resolve a problem? What's the trade-offs on this kind of thing? What are the limits of this solution and what are the potential enhancement? So there's a lot of DevOps in this thing. It's very trendy to use the word DevOps. I'm not a DevOps guy. I'm just a CTO, which in the end can be described as doing whatever the team doesn't want to do. So I ended up doing DevOps. So what do we do at serve?  We are building a SaaS platform for SMEs. So typically the air conditioning company is going to serve service your icon in your place three times a year. Most likely they're using a system. So for us, these are the kind of customers we have. Those small industrial companies that nobody really cares about when it comes to technology. So in the end, before us, they're using a whiteboard and paper to organize their business. With us, they use technologies. So we have also mobile aspect and so on and so forth. So I'm going to skip on this. So that's our technology stack. So we have everything in Java Spring on the backend. AngularJS on the front end, the test range on MongoDB. The Tomcat is where we execute the code, the web code. The rest, bootstrap, we don't care. So yeah, we don't have native mobile. So that's why we have the Ioniq for now. But we are moving to native. DevOps, what we need, what we use to build. So we're using, of course, the Amazon Web Services. We're using those tools called Packer Terraform from the Hashicorp group. They're doing cool stuff. We're using Jira for the project management and this kind of thing. So an acro for the Quatee Jenkins to build, of course. So all these tools that I'm mentioning here are actually working on the hood to do what I'm going to present today. So that's why we try to solve. So having many features come from the build everything or nothing state to we build feature by feature and we can present them feature by feature. So that's how it works for us. So for us, it starts from a Jira ticket, which is a feature. It goes in GitHub. Then we will trigger some command on Slack. Slack will talk to Jenkins. Jenkins will build actually one-time throwable build job for that specific feature. Then we invoke Terraform. Terraform will build actually from the scratch the whole infrastructure. So we are not recycling environment. We are building everything every time and we're destroying at the end. So on your branch, I will show you in detail, don't worry. Provisioning wise, so then, sorry, then the job is created. Then we execute the job. The developer executes the jobs when the code is ready to be shipped. Then we provision actually through Enciable. I mean, Enciable is provisioning the stuff. And then in the end, what we want is to provide to the product guys the name of the feature.server.sg.slash system. Then you can look at what has been built. And you can map it directly to the story described in the enjar. If it's good, then it's passing it. We merge the code and we can actually push it directly into the other hotfix, for instance. And we can make it happen in the end production. So that's the way we bypass actually the limitation we have by building a monotica. So when we did the branch, then it will destroy everything. So the infrastructure that has been set up just for this is getting rid of as well as the Jara job, the Jenkins job. So should I... Where is it? I didn't track the time, so... So yeah, all starts in Jara. We have a branch. The developer will branch out. Name of the branch. In Slack, simple command slash serve branch build. So what will happen is some scripts will do that. So find as a new branch has been created then start to build the Jenkins job properly, configure it. Then building infrastructure. So it takes a bit of time for Terraform to know how the technology stack is to actually build it up. So it takes a few minutes. Then taking care of the DNS and all the stuff on Amazon through Route 53. And then once that's done, then the thing is ready to be deployed. So the first time, the developer has to ship. He has to build the specific branch using the Jenkins that has been just created. So that's the infrastructure we're building actually every time. So from zero, we're getting this. So the provider, we have three security groups. We have two instances for now, simple. One web instances, then a DB. The EIPs associate to each of the stack. Few DNS. So that's what is done in the background. And this is only for this specific feature. So in Jenkins, it looks like that. We provisioned the branch through NCFR. So that's what we give to the product guy actually. That URL. So that's the end result we're going to have. And when we destroy it, you just go in GitHub. You either merge the branch or you delete it. Then it will actually run the Slack command again. And it will destroy everything and clean stuff. So in Slack, it's like, so maybe I'll show you how it looks like in real life. So I'm going to start in GitHub. So I have a clone of our project. And I'm just going to do something simple. I'm going to create a new branch. So let's call it jara-687. Also have my new branch. Now I'm going to go in Slack. I'm just going to run the command. We also did something more for the fun of it. We actually have Slack, I mean Jenkins, the scripts to push information on what's going on. So it's going to detect our branch. It's going to build stuff. So it's going to take like 2-3 minutes. What is important is if we're looking at the previous run, you will get this link actually. So that's the Jenkins link. So I'm going to get one soon. So if I'm clicking on the link, then I'm getting in the unique... Is this the Jenkins integration into Slack? Yeah. No, not directly. That's something we wrote ourselves. We have also a Jenkins integration, but it happens somewhere in Slack, but we're using it. So that's just our script, getting the URL that has been created for the occasion through the Jenkins API and pushing it to Slack. So that's the... So now the infrastructure is being built. And once this will be done, then I'm going back to Jenkins. I was just there. And I just need to run that Jenkins job and it will build the binary, provision the... finish to provision the stack, because now we are just building from scratch with Terraform servers, but we don't have anything inside. We didn't really set the DNS, the hostname for instance, on the database side, we didn't load the dump that we are interested in. So this is what happens at the Jenkins job stage. And once that's done, we are just ready to connect to the server. So yeah, taking a bit of... a few seconds. Any questions in the meantime? Your Jenkins is by your job name. Yeah. What happens if you run two jobs of the same branch? Two jobs of the same branch. The job is... so the... when you invoke the Jenkins... sorry, the Slack command right, it will check whether there is already a job running for this or not. And if it's already running, it will stop. Okay, cool. So now I'm going back there. I can run this one. I can build, actually. So the thing is, yeah, we will have only one build or maybe five builds for the same stuff. So if you have any back and forth between the product and the dev, you did this, but I'm not very happy. So you just need to... you just need to work again on this branch. The same branch is going to modify stuff and rebuild again from that job until everybody is happy. And that job will disappear as soon as either the feature is deemed ready to ship or acceptable from a product perspective or to be dropped, so then they did it. So if we're looking and I guess you all, are familiar with Jenkins, right? So that's what's going on here. So now it's Enciable, actually. So we also chose Enciable rather than those puppet or chef or whatever, because we didn't want to have those Master 7 kind of relationship. We don't have something as straightforward and Enciable is good for this. So you don't need to have the same kind of infrastructure behind. So what it does you can see, it's what it does. So we're using some binaries like SupervisorD for instance to run some on the web stack, some Node.js kind of thing. So configuration is happening at that point. So on both servers and it crashed. And it's the first time it crashed. But usually it crashed when you have the Jenkins as this cargo feature to ship a Java into Tomcat. Sometimes it does that. And he hasn't done that for a while. So what if I run again? He doesn't resolve the host. My screen is very small so what happens if I run it again? So if I'm looking back at the slide. I was afraid of the Murphy's Law. But we've been using for three months. It's the second time it happens. Honestly. So I need to check that. But in the end you will have seen the maybe I can have. No I didn't. Okay another run and if it doesn't work then I'll move on. You will see that things have been created. So that's also the power of those tools like Packer and Terraform. You don't have to go in AWS and do things manually. So you just use those binaries. You describe for Terraform for instance how your infrastructure looks like. I have this load balancer that's behind this. I mean under this VPC then I have this security group. You describe that, you link them together in a configuration file. And once you're done with this you just run through Terraform and Terraform will take charge to build this. It does that for AWS but the same kind of execution you can apply it for many actually providers in parallel. So you can do that on Digital Assurance and all those at the same time. So if you have using one web services in one case and you're using one for whatever reason, you can actually integrate that together and work it concurrently. So if you look at this our stuff are here. So we have one machine here. We have the DB here. And if I'm looking at the for instance the Route 53 I will find the same thing. The good thing with Terraform also is once I destroy the branch or measure branch when I destroy a command I will clean that. So she's cool. There's many ways to do this in another manner using other technologies. This is one of the things we try. I'm not saying that's the best, the top but that worked. Yeah, okay now it works. So if I'm going back to this one so what's the address? So it's jira-687 So now I have actually a check and login. So I don't know why it crashed but So now I have our platform with just that feature. And result, that's what we wanted actually. So then the product knows exactly that whatever is described in this feature on the jira side, it is what it's seeing. It doesn't even have to wonder whether it's a disk plus that or whether a bug is going to find can be related and push at the same time. So for us it's a good trade-off and a good in-between from monolithic to macroservices. I think this is not going to last for macroservices for instance. We have to redefine it. So now let's imagine that the product guy is okay with this. I'm just going to go and delete the branch. I'm running the Slack job again. How do you measure state for each Terraform view? Terraform, so you know Terraform, right? So he has the state file that is created first run. So we don't change the actual infrastructure in the meantime. So he's just referring to the file he created the first run. So when he's destroying he's just going to look at that file that has been saved on our integration server. He's just going to say, okay, this one I'm destroying it. So are we at the Jenkins workspace? No, so Jenkins. So the way it works, we have an overlay of a series of scripts that would actually for each feature we create a directory with the stuff in the directory directory for Terraform the stuff would be run there. We have a lot of templating happening in the background to match the DNS kind of stuff and modify the... So we don't even... Terraform is working in such a way that we have one master file with a lot of placeholders and then we have profiles and the profile is generated automatically. And then behind there is a... I mean templating run. We're using Jinja2. So now if I'm looking back there if I'm looking back at Jenkins this one doesn't exist anymore. It shouldn't be existing anymore. Yeah. So I don't have it anymore, right? You don't see it if you have. If I'm looking back at Amazon I shouldn't see anything anymore. So my instances should be terminated. So it will be the same with all the resources we created for this thing. So if I'm going back to the... back to the slide I'll finish quickly. So, yeah, so what we are learning from this, it's more like a walking prototype that something else, right? It's a proposed approach but as I said you have many. Yeah, deep learning by feature and nothing is new. The thing we are tending towards here is immutable infrastructure. So the idea we are trying to illustrate is what about instead of updating your different services with your new update, your patch or whatever even, a new binary whatever we just actually rebuild everything from scratch every time. How does it work? So tool like Packers and Terraform helps to do that. And you have others that have been around for some time like there's this new .io that went last month you have this one called Well, I forgot the name but you have a few doing this. So that's maybe a way to go towards immutable infrastructure. We are in between actually. So we deliver by feature, not new combined with immutable infrastructure. I haven't seen that so many times actually. I'm not even sure I've seen it actually. So anyone doing it I'm curious to know. So there's benefits from a business project perspective. We did it to sustain the product actually. So it's cool because we are very responsive compared to what we were before. In this kind of context when you do Java we are a bit more agile. Not confusion what's actually deployed. You know it's that feature and only that feature. So it's getting a lot of questions out of the equation. And yeah, so a branch can be hotfixed when it's approved. So that's a good thing if we need to push it quickly. Technical perspective the image on Amazon Web Services which is super good. And it's more predictable since we are addressing it 100% programmatically. So we are not going in the console, which is nice. We don't do environment recycling. We hold the problems you might find with this kind of approach. So we are exactly sure what we have. And we ask for as professional as we can of course. We're using Packera to craft. The image that is being used by Terraform to build the system and run the system. It's actually a pre-baked image. So we're not using the Amazon CentOS 7 base image. We already process that image into something else that has the most as many common things that we will have in our image. For instance, we update you at that point. We assign the IOPS at that point of time also for the image. And a few things like this. A few limits of course. How to keep track of the full life cycle. Yes, so there is a lot of we are increasing the single bottom failure doing this. Because the more you add on the rain systems, the more you have the kind of primary. On the Packer side when you craft the image, you need to many describe the targets which is a bit tedious to describe all these things. And Cable is super hard to maintain to the first time you do it, it's fine but when you need to update it, it's becoming a bit more complex and it's hard to test. It's very hard to test actually. And you have to do everything manually and test by doing it. So that one is a bit annoying. Terraform also you have to describe everything manually. So it's very tedious. Dependency on Jenkins. Jenkins can be quite tricky. You can lose track of how many 10,000 euros you put in. You can't really show what your Jenkins does anymore. So it could be also a problem. A lot of templating also. And that's one of the things that could be avoided. And especially all the dynamic stuff like the EIP addresses and names and DNS and so on and so forth. So many pieces in movement. I'm not sure. So the question is, is it worse? I mean do we benefit enough to take this kind of risk? Because we have so many pieces in movement. So you can question it actually. I wouldn't I wouldn't blame you for thinking actually it's maybe overkill. Yeah, thinking with the product team. There's a lot of connection with the product team to be done. And yeah, what else enhancements? Yeah, we could use more tools to orchestrate. So something a bit higher level. Stuck storm, dust dust, pinnaker maybe. I don't know. Or build a macro service to this. We have a lot of things. Choose the branch to build the Slack. So now the branch is just the thing is just going to look for all the branch that have been branched out and it's going to build the Slack. We didn't code it in such a way that we can just do one specific branch. We need to work more on the Slack side. You cannot Slack argument. So it doesn't work like this. You need to have a mine in the middle services that will actually get that command translated into something else and then push the argument for now. So Enciable is a program that might be a solution, I don't know. And yeah, we are moving to macro services so how do we move that to cater for micro services? And that's pretty much it. Sorry I was a bit... Thank you very much. Yeah.


I can spin? That's the best feature of this whole game!  Did anyone just see that? That was amazing! Hop! Oh my god. One last thing before we raise our... Ralph. See, I've got some great advice for you. Welcome back, by the way. You know, I've seen Pokemon walking around under the tall grass, yeah? If you move real slow, like, and crouch down, these Pokemon won't pay any notice. Wow! But on the other hand, if you whistle at them, you'll be sure to catch their attention. If you're a bit rubbish at whistling, just remember this. Need a foolproof trick? Press the left stick! Try it out. Let's see who can catch the most Pokemon, Ralph. What if I don't want to? What if I don't want to spend any time with you, Hop? What if I never want to see you ever again? I do want this Foxboy. I do. Hello, Foxboy. How are you? Nick Tits.  Prinzuko! It's your time to shine, my dear boy! Your time to shine. Your time to shine. Prinzuko used Amber. Wow. Look at that. Do you look at that? Use Tail Whip. That does nothing. Back time? Pokeball time? Get him. Use the Pokeball. Get him. Get him. Get him. Get him. Get him. We're gonna have him. And his name will be Nick Tits. Nick's Tits has been added to your party. Perfect. Perfect. I'm so happy. All right. Yeah, I'll catch you. I don't see, uh, what's this fuck? Catching more than, like, three Pokemon. I want something different. I don't want to catch these squirrels.   Is it worth wasting my balls? What's over here? A revive! Wow! We get sparkly stuff. You'll never beat me into the station at this pace.   Here's a technical machine. With a T.M. like that, you can teach your Pokemon to move in a flash. You can use them again and again, too. That one, teach them to move swift. Lea gave it to me ages ago. Now it's yours. Gym challenge will begin. And our training is waiting. So let's go. I wish that I had said no to you. I don't- I don't want to do anything with you. Not quite so fast you two. Who is that? Moms! I knew this day was coming since the moment you chose Grookey as a partner, darling. So today, Professor Magnolia was kind enough to let us know that you two were setting out. So here, a last little gift to see you well said on your way. All the kit you'll need for camping. A set of camping gear. Wow. Wait, so did I choose to do this whole thing with Hop? Because I actually don't want to. Like, I know it's kind of a bit that I hate Hop and I think he's annoying, but I legitimately don't want to at all. Have fun, take care of yourselves. You'll be back home any time. We're off. Bye-bye. Which is luck, not that we'll need it.  Don't ever talk to your mother like that. If we have to travel with him forever, I might restart the game. Because I really don't want to. I wish it was a little more clear about, about, like, when he was like, oh, we'll do this together. And then it was like, yes, together or together. I wish it was more clear to be like, I don't want to do this with you. Plus, you can camp out and do a lot of spishing. I don't think you could ever get tired of it. I'm tired of you. It's a thing. I camp all week. I cannot spend another minute with you. Hey, Mr. Station Master. Isn't this the wild area? What are we stopped for? Yes, this is the wild station area. Make no mistake, I'm sorry to tell you that the train is halted due to a flock of Wooloo on the tracks. What's there to be sorry about? This is brilliant. What is? The wild area is massive. You don't get it, do you, Ralph? Don't talk to me like that. Hop?   Yeah, that's what I said. The greatest team. All right, bye. I hope I never see you again. Hold on, I want to see something real quick. I know this thing is playing. Just let it play. But how to abandon Hop? I can't abandon him. I don't know if I'm stuck with him forever, but you know, oh my gosh, you have cool overalls. Would you like to rest your Pokemon? Sure. Sure, why not? I'll take your Pokemon for a few seconds.  They're all happy and healthy. I hope you're happy and healthy. Oh, hello. If it isn't Sonia. Sonia, what are you doing out here? Wow. My grand gave me a proper earful in her own way. Those two trainers are sitting out on a journey, but what are you doing with your life? That sounds rough. Ouch, Professor. Never mind that. Nothing to worry yourselves over. Besides, I'd be quite curious to see what Pokemon you two met in the forest. I've been thinking I should look into it, so the time will work out fine. If I discover something really huge, then maybe even Grannel admit that I got some talent. Yikes, being an adult has got its own challenges, eh? Just wait until you hear about taxes, Hop. You'll be fucked. I'm glad to be on the road, though, really. It's been ages. It'll be great to fish and camp and rough it outside again.  I'm going to battle the snot out of a bunch of Dynamax Pokemon and fill in another page in the tale of my legend. I hope you achieve nothing. What cheek. Dynamax Pokemon are really on a lower level, you know? You'd better Dynamax your partner Pokemon, too, if you want to take on opponents that strong. You'll find spots here and there in the wild area that emit a red beam of light. If you think you're up to it, check them out. They'll be Dynamax Pokemon lurking inside. So with that, how about I send you off to play for me? Pokemon Box Link. A device that allows you to access Pokemon's storage system. There are some places where it won't work. Wow! Thank you so much, Son. You're with a Pokemon lock. That's nice. Yeah, I want to just do those things. Wow, my destiny awaits. Out in the woods. On the open air. I can achieve anything. Anything is possible.  A large Pokemon see water? If you're trying to battle, it seems like you're over your head. Try to run. Run like the wind. No, dude. I'm ready for you. I'm ready for you, Onyx. I want to catch you. A very strong looking Onyx. Sunlight is harsh. Pokemon. I'm actually going to swap out for Awful because, uh, water is good against rock. Prince Zuko, come back, my dear. Go, Awful! Awful! Onyx, you screech. Oh, is this going to scare me? Am I paralyzed? Awful. Are you Awful? Fight. Water gun. Do it. Big time. Why'd you get to go twice? Why'd you get to go two times in a row?  A critical hit? That, that, that, that is a critical hit? Alright, well, we're fucked. Yeah. He's gone. He's done. I can't catch him yet. What level is that? I'm level eight. Or this Pokemon's level eight. What is that? Like 25 or some shit? Yeah. Yeah, yeah, yeah, yeah, yeah. We're probably not going to be able to do this boy for a little bit. We are not ready for this. Oh, my God. I'm out of here, dude. I'm out. Couldn't get it. I couldn't get away? Am I about to be dominated by this Onyx? Nix Tits, you're up. I've never used you before, but now it's your time to die.  Not very good. Oh, it hits a couple times. Four times, Nix Tits. Can I run? You couldn't get away. Great.   Hello, nice lady. I would love to rest my Pokemon. Thank you so much. I love your overalls and I love you. What is this? What is this little? Oh, it's a bunny. You look just like the squirrel, but old me.  All right. You know what? I'm going to catch him because Pokedex things. We've got to expand our Pokedex, even though I don't want these garbage Pokemon. But old me was caught. Thanks, muddle me. Oh my gosh. Yes. I'm going to catch you. Vulpix. Yes. I want a nine tails immediately. Prince Zuko, give it to him. Oh yeah. Hit two times. Yes. Yes. Yes. All right. Let's see if we can get a sweet little Pokemon up in here. Okay. Get my ball. Get my ball. Get my ball. Yes. Yes. Yes. Yes. I'm going to put this Vulpix in with me. I love how everybody levels up at the same time. It's real nice. You know, this Vulpix seems like a ronda. I think ronda is a nice name.  All right. Well, let's continue going on a Pokemon adventure. I do kind of want to try. You find a note. Did someone drop it here? Use the right six to rotate the camera when you're in the wild area. If you press right six, you'll find, you'll see you can find a wire view of things. Put that all by the left straight ahead. What? Your ears are fists? Diggersby? Nah, not that one. I'm going to capture you for sure, but I hate that name. I also don't really like this Pokemon.  I don't know if it'll be able to catch him right away though because I didn't damage him all that much. It was so close. God, it was so close too. Mud shot. Oh no. Oh no. What even are you? Normal and brown?   Use water gun. He's going to hit me with mud shot. Not yet. Jesus. Jesus, and it did nothing. Alright, come on. Don't fall right away. Oh no. Rhonda, you want to take a swing? See what you can do, Rhonda. Do you even know anything yet other than Quick Attack? You know Amber, okay. We're just going to hit you with mud shot, and that's not going to be good for you. It's going to get me all the way. Ah! No! Air Bud, you're all I have left. You're it. You're it. Come on, Air Bud, and you're not going to be good against him. Come on, Air Bud.  Well, it lasted, guys. That's the whole let's play. That's the end of the series. Thanks. It's been good. I gave my player 96, I gave the Pokemon money. That Pokemon, no, that Pokemon mugged me. You have no purpose for coin. Don't come near me, don't come near me. I want to touch this thing. Pillars of red light, you'll see in a while there. Pokemon dens. Dynamics Pokemon well in these dens. I want to investigate the den. I gained 300 watts. Max raid battles? Oh my God, it's like Pokemon Go. Don't invite others. I don't have anyone to invite. Oh my God. Whoa, who are you? Who are all you people? What are you? Ninkada. Go, Prince Zuko. Big fire. And make a big. Yeah. Ember. Ember. Should I have... Should I have a... Should I have made you big? Oh God. Oh no, we are done. Oh my God. This is so cool though. I should Dynamax myself because otherwise I'm not going to do anything. Oh, I can't use Dynamax yet. Okay, I'm just going to keep using Enber because I think that's going to be the biggest one. This is so cool. I really like these like raid kind of things. It's dope. Oh yes. Oh yes, halfway through a critical hit too. We'll be Dynamax in no time. Alright, it's almost time. I wonder if this Pokemon will even be good. Who knows? It's just big. That doesn't mean it's good. You know? Oh yes. Now it's time to capture it. Wow. Use. Use that Pokeball. Go, go, go. Make a big, big ball. Oh look how big that ball is. Is it always a premier ball too? Come on. Give it to me. And Kata was caught. Cool. What level is it? Oh, I thought the first second when it said number 104. I thought that said the level. I was like it's 104. We're going to name it Cricket Wireless. Which I don't even know. I can't. I can't name it Cricket Wireless. Okay, whatever. We'll just name it Cricket. We'll swap out Air Bud. Not a huge fan of Air Bud. Alright, what do you got? Your level 19. That's good. Wait. Mystery gift. Hey, how do you hold? That's pretty nifty. I gotta me out. Nope. Get away.  It's a horse? Are you seeing this? Mudsdale? Come on guys. Pick it up with the names.  Let's double kick him. Please don't let him defeat me. Oh my God. This is so bad. This is so bad. I've never seen anything so awful. Let's see if this works. He's level 19, so at least the level is higher. I don't think he's going to do well against this boy. For the shot though. Strength. What's that? Alright, let's run Ronda. Oh God. These guys are big, big beefy boys. And I can't do anything with them. Oh my God. Let me. Let me have you. Oh my God. That would have been so cool. It's a little panda. Oh my God. Oh, I contacted something else. Minchino. You're so cute. You're so cute. Look at you. I'm just trying right away. Pokeball it. Do what happens. Thank you little rat boy. Okay, it was this guy that I was trying to get. Pan Cham.  Look at you. Pan Cham's mold breaker. He breaks the mold. He does indeed. Alright. Let's use Ember. Let's see how good it is on you. Alright, not too bad. Not too bad. I'm going to hit him a couple times. And then we'll get him. And then we'll say, hey, there it is. There you go. Take me home. Take me home. Come on. Take me home. Yes. Good. Good job. Yes. Pan Cham. I want you. Yes. Naming him Jack Black after Kung Fu Panda. I want to add to my party. I want to swap somebody.  Like awful. Now I don't have any water type, but I don't care because awful is awful. Hey, come here. Come here. God, that's horrifying.  Yeah. Yeah, we get killed. Yeah, okay. We're going to leave. I don't think that this is a good time to we're going to run. Oh, great. We can't get away. Awesome.  I know you can do it. Give him a little bit of that tenacious D. You've got the, you break the mold. You know? You got him. Arm thrust. So much damage. I'm pushing him away. Get back. Get back. Get back. You can't bully me. Teleport. Are you teleporting away or into my soul and then making me explode from the inside out? I'm going to go up these big steps. I'm going to go up these big, big, big, big steps. Wow. Who are you? Well, isn't this nice? Find a way through the aisle. I did it. I've done it. I'm in the big, I'm in the big area now. Wow. Incredible. Look at that. Look at this big area.  I wish I could be your friend, but not this time. Guys, there's gears everywhere. It's my brand. I will surely thrive here. I will surely thrive. All right.  I have a doctor's appointment, so I need to leave. Thank you guys so much for watching. Hope you enjoyed. If you did, leave a little like, leave a little comment. Let me know what you thought. Let me know what you want more. Thank you for watching. I appreciate it so much. Am I in frame? I am. Thank you guys so much for watching. I will see you later. I've got a doctor's appointment. Yeah.


Hello, hello. No, it should work. No. Hey there. So we're still trying to start on time, but as neither my co-chair or the tech lead is here, I am poking people off. Also, FYI, but I'll also raise this once we start for real. Maybe we can consider moving this call around a little bit because it's directly after CNC FTOC call, which means I'm not super well-prepared for either call and I hate that. And it doesn't do either of those calls justice because you don't have any time to just do the last fix ups before major external calls. Also known as Thanos. Free Zoom. Exactly. Also, as per usual, remember to write yourself into the attendee list would be great. Wasn't the plan that we start punctually that we don't wait any five minutes anymore? Or did I miss the plan? But people still keep appearing on the other hand. I'm torn. On the one hand, I don't want anyone to be left out starting too early. On the other hand, it's not great to lose those five minutes times currently almost 20 people. But we are at five minutes or almost anyway. So let's just get started. So two high-level things. One, FYI, there is now Prometheus Working Group within OpenTelemetry for anyone who has an interest there. And the other thing, this call is right after the CNCF TOC call, which means I'm usually jumping in between the two. And I can't really do either call full justice because all of you notice when you have two major external calls, something happens right before you are out of time. So long story short, if the feeling of the room is that we can try and find another slot, I would start talking to CNCF. But we can also move this discussion to the mailing list. I just wanted to see what people think. It's fine, right? Is that we're just finding the right time that works for folks? So as long as we can find something that's common, I think. OK, yeah, I'll talk to you. Yeah, can you take an extra item that would you to come up with a few slots? And if you send it out on the mailing list and on the Slack channel, I don't know, whatever the current way to survey that is, I think you have a nice. It's always only more channels. It's not that we can actually get rid of any old channels. It's just more channels, mailing list and Slack. Good. So the other thing which we currently have on the agenda is the due diligence talk for open telemetry. So I guess let's get started. So I'm happy to cover that. You want me to share the screen or does someone else want to share? I mean, just walk through. How do you want to handle it? No, go ahead and share that for me makes sense. Wait one second. All right. So I mean, for most of the people on the call, this is another example of doing the due diligence work. So same exact template that we have followed before. The link is in the agenda. Comments welcome. So please add feedback in here. We can be sure to incorporate all of these changes. But this is specifically for the open telemetry project, which is currently a sandbox project. It is the joining of two previous projects, open tracing, which is currently an incubation, and open census, which was not part of CNCF, but was an open source project. So the best way to think about open telemetry is kind of the next major version of both open tracing and open census. And because I'm sure it'll be asked, part of the plan here will be as open telemetry moves into incubation, open tracing will be removed. So we'll kind of work through the details of that. And of course, the idea here is not to break customers or end users. So there is something known as a SHIM that is available to both open tracing and open census. So you can continue to use that with open telemetry. And open telemetry will provide support for two years. So customers or end users of these products have a path forward without having to jump ship and re-instrument everything to kind of make it work. Now the doc that we have here has gone through the open telemetry governance board. We'll talk about governance in a while. And a little bit, you can see different members of open telemetry that have currently reviewed it. There's still a few outstanding, but they will add their review information in here kind of soon. And we don't have the TOC sponsor yet, but we will of course sort out those details. So let's make this as interactive as possible. So I'm assuming we'll go kind of section by section. And then Richie, I'm assuming you'll do call for consensus. And we can kind of mark things off as we go. So section number one, first, I guess, any questions before we get started? Can I interrupt? So usually it was kind of there was some time for review before, like for offline review or whatever. Did you share this maybe earlier or just maybe we can have the time, I don't know, to do it offline as well? Not before, I mean, before jumping into the straight consensus and stuff like that. We did, you had exactly the same point on previous incubation docs as well. Yeah, so this was shared last week in the sick observability Slack channel with the link. And we don't have to do consensus today. Like I assume this is going to take at least two meetings. So any items that folks are not comfortable with, let's definitely table. And we can definitely take feedback and come back in two weeks and address that feedback. So no rush here. If there are any concerns or areas that people want to dig into more, we have some time. So I doubt we'll get through everything today. Yep, yeah. Just next time, it would be nice to use the mating case. Otherwise, yeah, it's good. Cool, I'll make a note of that. And I'll send it out after the call. If I haven't seen anything on the Slack channel, that's why I also asked there. You said you're going to share something, but there was nothing unless I overlooked something. Yeah, I'll put the exact link. So maybe I just put the link to the agenda to sick observability because that's where the link for this document was. So I probably just wasn't clear. So I'll send that up to the mailing list and to the Slack channel to make sure it's super clear so people have access to it. Yep, that would be a pleasure. So normally, I think somewhere in the documentation, we say that the mailing list is the primary because that can be consumed completely asynchronously. On the other hand, Slack is basically half email by now, four by now, four words. So yeah, I guess we are just walking through stuff. And if there's anything which needs to be followed up as you say, Steve, then we just mark it as such. And yeah, okay. So then let's get started, I guess. So the question and also I need to copy the template which I used or which we used last time because I don't have it by heart. On the self-governance, I think there will not be too many questions. So open symmetry has governance committee charter. It follows kind of similar things to other CNCF projects. People can kind of review it, it's a little long here. One thing worth noting is that we're kind of going through a transition. So the board has kind of grown and is going to reduce again in size. Again, that's all outlined on kind of the read me here. But there is a governance committee, there are standards on how many representatives per company there can be to ensure that it's not controlled by a single entity. And yeah, I mean, there's a whole community with GitHub issues kind of outlying what the government's board is kind of responsible for, areas of focus and what have you. So definitely review it, let me know if there's any kind of comments. It follows a similar model to what Kubernetes has been using. So I guess any questions on governance other than people probably want to review it. Just one. So you said there is a limit on the total amount of people from one company on the governance board or on what level is that limit imposed? Correct, it's on the governance board, there's also a technical steering committee. So where is the note here? Maximum representation, so right here. So there's a max representation of one third from any company. Right now, given that we are in 2021 the board is going to be nine people. So that means no company could have more than three representatives in the board total. And if you have more, then someone has to concede a seat and then a special election occurs. Okay, okay, and that is on governance committee level. And governance committee is the board? Correct, yeah. Okay. And then there's a separate technical steering committee which is another board that handles the technical aspects of it. And that follows a slightly different process, but same idea. There's a max representation for that board and they handle kind of the technical decisions across all of the SIGs within open telemetry. Okay. So I will need edit access. Oh, do you not have it? I can fix that. Now RichieH at Grafana.com or RichieH at RichieH.org Boulevard. RichieH, you said Grafana.org? No, com. Com, com, com. Sorry, my bad. Editor, you have it. Perfect. On its way. Cool. Second one is around code of conduct. We just follow up CNCS. Sorry, sorry, sorry. Just to make. Oh, sorry. I think there will be no further questions regarding the governance, but maybe someone has some. So just to make sure call for consensus, is there anyone who wants to look into into the governance document more deeply? If yes, now's the time to speak else. The suggested call for consensus is already highlighted. Good. So SIG observability is happy with the section above. All agreed. Anyone disagreeing? Very good. Awesome. Code of conduct. We follow CNCF. So it's a direct link to CNCF. Nothing special for open telemetry, but yes, definitely have one. Definitely ensure it's followed and not diverging from anything else that CNCF does. Okay. So it's direct copy, I presume. I think it links directly. It links directly. Okay. That's perfect. I do missed wrong one. So given that, I don't expect much discussion, but still to make sure call for consensus, SIG observability is happy with the section above. All agreed. Anyone disagreeing? Very good. Okay. Next one. Does the project have production deployments that are high quality and high velocity? So I've included a link to our adopters file. There are a variety of both vendors and end users that are listed here. Anyone listed as an adopter is stating that they are running this in production today. What we have not articulated is what specific subcomponents of open telemetry are being used. We're actually going to update this page to make it more of a table that actually outlines that, but there's a broad range of both client library and open symmetry collector support, which of course means that the specifications are being used because that's a prereq in order to use the client library and instrumentation aspects. So yes, both end users and vendors are using open telemetry in production. Some are, of course, even trialing. Maybe they're not in production yet, but we do believe that there are production deployments that are high quality and high velocity today in open telemetry. Okay, perfect, yeah, but that would have been one of my questions regarding which component. Of course, if I remember correctly, when you talked about this last year, you considered having different tracks for different parts of open telemetry. That's maybe also something which we should be talking about. So for the update here, then we say that we request that table to be made and we will look at it next time. Sounds great, go for it. Yep, I'll take a note on that so I can make sure it's added on there. Yeah, I would love some details as well, right? Because especially Prometheus, we have like exporters, right? And they are not part of the Prometheus which is graduated. So I wonder if the kind of incubation, all those kind of street rules actually apply, yeah. All the small projects that maybe are part of open telemetry, huge ecosystem are kind of fitting the incubation essentially, if that matters even. So that's a good question. Yeah, I guess I will make one note on this. Well, I guess two notes. One is, I mean, observability, three pillars of observability. Open telemetry is focused primarily on tracing to start, metric second and logs are not even part of really scope today. There is some amount of log support but I'd say it's very early. So most of the production deployments that you're seeing here are primarily around the distributed tracing aspects. The other comment I'll make is we kind of group the client libraries or the languages together based on maturity. So if you think of like a Java or a JavaScript or a .NET those are typically like the top tier languages that are most commonly used. So those are definitely production ready. Where say like a PHP for example is like a tier two or tier three language like there isn't as much adoption of it today. So maybe those are not used in production environments yet. But for the things that we consider to be like top languages or like the collector implementation there are definitely production examples today with both vendors and end users. Cool, makes sense. Cool, so happy to provide that information and we can update this for next meeting. Next is the project committing to achieving CNCF principles and do they have a committed roadmap to address any areas of concern raised by the community? So the answer is absolutely. Open telemetry is quite a large project. So there are many repositories. The way that we handle this is that every repository is basically its own SIG and every SIG kind of runs independently with their own approvers and maintainers. So if you think about issues being raised by the community they're usually raised by individual into individual repositories. So maybe I have an issue with Java instrumentation. I will go add that to the Java repository. Each of the SIGs also maintain their own roadmaps. So for example, if you were to go into the collector repository there's a docs folder it lists roadmap information. Usually these are around major milestones. So for example, most of the roadmaps today are talking about the GA of open telemetry because that is the primary focus but a longer-term roadmap like two, three, five year type thing will also be laid out going forward. But yes, both CNCF principles and addressing community concerns as well as roadmap information are all top priorities for the project and constantly reviewed. Okay. Can you link to the roadmap documents as well? Sure, yeah, can I link? Yeah. Okay, perfect. Just make a comment and I will happily add that link for you. The next one, document that the project has a fundamentally sound design without obvious critical compromises that will inhibit potential widespread adoption. So the project is actually, as I mentioned the maturing of two other projects, open census and open tracing. Both the bridge have extensive support in the community today. Open telemetry is basically the next major version of these. From a community support perspective both contributions as well as adoption we have seen that all three major cloud providers are actively on board and have their own kind of announcements around support and compatibility as well as roadmap information. A lot of major vendors, I mean, not all but many, most I would say that are in the observability or monitoring space have some amount of support for open telemetry today. Open source projects are also adopting it. So for example, Yeager has moved off of the Yeager collector and now uses the open telemetry collector as the collector for Yeager. Fluent BIT has support in the open telemetry collector. So I mentioned there is some log support. There's an example of it. And even outside communities like SpringSleuth now has experimental support for open telemetry. So it kind of goes beyond just the CNCF. End users both are approvers and maintainers of the project which means they are actually committing engineers to develop open telemetry as well as deploying it in their production environment. And I've listed four examples of such companies here. These are also reflected in the adopters part. Now from a support perspective, open telemetry is really looking to address the primary data sources and observability. So traces, metrics and logs are where it's at. As I mentioned, traces and metrics are more mature. Logs are a little bit longer out there but there is roadmap information for this. There is kind of a path forward. So we don't see anything that would inhibit any adoption today. There's also support for all popular open source things as well. So Zipkin and Yeager in the case of traces, Prometheus and of course the collaboration with OpenMetrics for the metrics set of the house and then logs, Fluent D, Fluent BIT and very recently actually there was just a blog post on devops.com, Stanza from ObserveIQ is being added to the open telemetry project as well for native go log support. Cool, I have a question. So I feel like you are right that, I don't know, like those supported kind of projects there will be supported in future and like there is a good collaboration already to do so but it's good to mention that it's not done or like right now, right? There's definitely, I'm not sure if that's even related to incubation or not but would be nice to at least as far as I know it's not OpenMetrics compatible right now. Like there is good effort to do that but I mean it's not now. So I would love to also know if the same is with the Fluent D and Yeager in Zipkin so maybe it's just some research on our side or maybe you can tell what exactly, you know is it gonna be supported or like supported already and like, you know, to be kind of more realistic here. Yep, yep. So from a, I can comment at a high level and I'm happy to add more comments in the doc here. So Yeager and Zipkin should be fully supported from both a receiving and an exporting perspective. I mean, technically the same applies to Prometheus and OpenMetrics. The problem is the translation layer in the collector. So for example, we're using the same Go client library that Prometheus uses. So that means that the receiver should be the same. The problem today is the translation into OTLP format does not natively support into and out of. So we can definitely add a caveat for that needs to be addressed. And I'd also argue that say like the Prometheus receiving capabilities and the collector are not ideal and need to be rewritten. So there are definitely gaps from that perspective. So yeah, if you want to, I can definitely articulate on here areas of like we know these gaps need to be addressed. I can definitely add that here. Yep, that's totally what I would love to see. And for OpenMetrics, I got that because I kind of was part of the working group and I love kind of the collaboration there. I'm curious about the logs and FluentD if they have kind of the same long minds to be solved essentially, or just to be essentially transparent, nothing else. Yeah, absolutely. And the answer is yes. So I mean, we're working directly with the FluentD and FluentBit maintainers and the stanza maintainers right now. And we actually have log support. So we have elastic representations, SumoLogic, Splunk. I'm sure I'm missing someone, but there are major vendors involved in this conversation too. But logs are very early, I would say. So I would call them alpha best case scenario. So there's definitely gaps in things that will need to be worked out there. So would it be fair to say that those two metrics and logs are in the future and Jager and Zipkin is already supported as of today? Sure. I mean, there's some amount of support for all three but the way to think about it is we are planning to GA traces in the next month. So that is very, very close. And there is full support for Jager and Zipkin. Metrics would be on the next roadmap which is probably another quarter or so out to get at least the initial support in. So let's call it August. Logs, we're talking end of year earliest to have something and probably not until early next year that we'd have the first GA support for logs. Okay. Okay. Yeah, okay. So just on the, not to drag this out endlessly but just for anyone who might not be deep in this as the two of us are or the three of us who just talked there's still some data format in combat abilities between the metrics side of open telemetry and Prometheus but this is actively addressed. And I think the timeline which you just mentioned, Steve is absolutely realistic. So maybe just a little bit case of the wordsmithing. Yep. Yeah, just add a comment and I'll clean it up. Yep. Thank you. Yep. Okay, next one. Document that the project is useful for cloud native deployments and degree and that it's architected in a cloud native style. So yes, open telemetry was built from the very beginning to support kind of cloud native frameworks for traces metrics and eventually logs. You can see it listed in the assess category from the CNCF observability and user survey. Part of the reason why it's only assessed today is because open telemetry has not GA'd or offered a stable API yet. We just talked at a very high level of what that looks like. So stable API for traces, let's say the next month, stable API is for metrics, let's say the next quarter or quarter and a half, stable API for logs, at least six months probably a bit longer. So there is a path forward for that. And as I mentioned, we're already seeing other like library framework owners preparing for this. Spring-Slooth is a great example. They're only offering experimental supports because the stable API hasn't been released yet. They've already committed that once the stable API is released, they'll move from experimental to actually supported in Slooth. So there's a path forward for this as well. Clearly the same applies for Prometheus and open metrics. We have that working group to kind of work out the details and make sure it's fully compatible. So that will be addressed. It will clearly take a little bit of time, but we do have a path forward here. So my only concern on that section would be that then it's not only, like the word only is not correct, but the rest, so a reason. Oh, I see. Oh, sure, yeah. All right, add a comment. Yeah, yeah, good point. Yep, you can fix that. Also by APIs, you mean the actual client code APIs or something else? Correct. I mean, technically APIs and SDKs because the configuration happens from the SDKs. So Richie even had a comment on API and I'll have the SDK as well. But you need both. The biggest problem is like if you think of instrumentation, there are two primary ways of doing it. Automatic, which like by code manipulation type stuff, and then manual, where a user goes in and typically adds their own instrumentation. The manual aspect is where kind of the rubber hits the road. Like if you don't have a stable API and you introduce breaking changes, well, you just broke someone's application. And if you've run that on tons of different servers and production, upgrading an entire fleet takes a really long time. All the testing that's involved, like you don't want to be in that state. So once we get to the actual stable APIs here, backwards compatibility will be offered. And thus you can take a dependency and still with Semver, you can make sure you can get to next major versions as they become available. Yeah. And also just one point of order. It's just the three of us talking, but this is actually root code. So anyone else who wants to comment, you're more than welcome to also speak up. That was a good time to pause. Like any other comments with any of the sections that we've covered, please, please be careful. All right, we'll move on a bit more, but please don't hesitate to get involved. Next document that the project has an affiliation for how CNCF operates, understands the expectations of being a CNCF project. Definitely, like both in the GC and technical steering committee, we have folks from Open Tracing that's already in the incubation status. So we're very familiar with how a CNCF operates. As I've also mentioned, there's cloud support, cloud provider, vendor and end user support. Many of those members are part of CNCF in other projects too. And then there's collaboration with Yeager, Prometheus, Open Metrics, et cetera. Again, all of these people have a familiarity with the CNCF. So I don't think there's any concern from this aspect. All right. Yeah, I can fix this. I'll send in seven parts for those. I can do something in the chat. Yeager as well, yeah, and Yeager as well, that's correct. Awesome, let's move down to the next section. Document that it's being used in production by at least three independent end users, which would focus on adequate quality and scope defines. So we have the CNCF end user survey which lists customers that are using it, also listed back to the adopters page. So the three customers that I called out that I at least know are definitely running it in production would be AppDirect, Shopify and Wanderer. Again, we'll clarify the table, what specific components they're using from the previous part, but I believe that will address this section. Questions? Not really. You already anticipated the one thing to just link to the thing or refer to it up either works. Perfect. Next one, have a healthy number of committers. A committer is defined as someone with a commit bit, someone who can accept contributions to some or all of the project. Oh yeah. So according to CNCF DevStats, open telemetry is the second most active project in CNCF behind only Kubernetes.  So I have the Google, the open telemetry teams, you can only see if you're a member of the community. So that link may not work for you. The membership information is listed. The DevStats link is actually in section three right below, but it lists a number of committers based on company, based on repository groups, what have you. So DevStats is a great resource for this. There are definitely a healthy number of committers. I think that is actually something where we can already put the marble. Just a moment. So for that section, sick observability is happy with the section above. Anyone agree, anyone disagreeing or agreed? Sweet. Perfect. Next one, demonstrate a substantial ongoing flow of commits and merge contributions. Oh yeah. So there's open telemetry, currently listed as number two in kind of the project here. And if you haven't used DevStats before, really cool tool. Hopefully everyone's seen it. Open telemetry is listed as a project and there are a variety of different dashboards built in. Come on, you can do it. Usually it's very fast. Here we go. So we could see like commits based on repository group. So here are all the different repositories and you can see a lot of activity with, it looks like the collector being one of the main things today, which is not too surprising. Given that it supports all the open source work as well as say like vendor exporters and what have you. But yes, very rich contributions to this. Sounds good. Any comments? So then call for consensus. Sick observability is happy with the section above. All agreed. Anyone disagreeing? Good. I agree. Only who is reviewing all of those comments? I'm just joking. Like there was lots of work involved in this. So good job. Good job. Thanks. Okay, cool. So next one, name of the project. It's called open telemetry. Some people call it hotel, O-T-E-L. We do not call it O-T because open tracing is also O-T, but we try not to abbreviate either. So open telemetry is pretty unique. Seems to be kind of certified. To be fair, someone owned the repository on GitHub. So we have open dash telemetry instead of open telemetry. But beyond that, like it is definitely a unique name. What does it do? This is basically copied from open telemetry IO. So this is basically what the government's board has kind of defined for what open telemetry is. It's kind of high level and vague, but that's part of the point. The way to think about it is both APIs and SDKs for instrumenting your app, regardless of data source that traces metrics and logs, and it offers an end-to-end implementation. So you can do both the generating, emitting, and collecting of that telemetry data. So everything you deploy on your side of the environment for the instrumentation and data collection aspects. Open telemetry does not provide a back end today. It plugs into many back ends. It's vendor agnostic. It supports both open source and commercial back ends, but it itself does not provide a back end today. CNCF charter mission, yes aligned, TOC, already mentioned that's TBD. The entire license is Apache 2.0 based, everything is marked, all the files make that very clear. We follow the CLA, easy CLA bot type stuff. All source control is done in GitHub on the open telemetry project. For what it's worth, we do all of our communications on Gator, not on Slack. Maybe that'll change one day, but the Slack channel on CNCF for open telemetry don't use it, go to Gator. External dependencies, I mean, technically none, but this is where things get a little bit more complicated. So everything that's a core part of the project is Apache 2.0, but as I mentioned, we support like receivers and exporters, which could be other open source projects or other vendor projects. So in a way, those are dependencies if you pull in those components. So there are technically external dependencies and other licensing things that could happen depending on what you're pulling in from those projects themselves, but it's all open source based today. Bartek, remind me, did we have a list of dependencies for Cortex and Fanos, or did we also just do qualitative statements? I honestly don't remember. Whichever was done for the others, I think is correct here. I don't have a strong opinion myself. It just should be fair for everyone, so. Yeah, I'm just checking, but... Give me a moment, I'm just checking. Okay, it was explicit lists. So yeah, but I mean, that's, you can probably get some in turn to do it. It's good for you. I don't think we can add whatever's needed, so. Cool, next one, release methodology and mechanics. So as I mentioned, every SIG kind of runs independently. So SIGs determine what they want their release cadence to be. So for example, some of the client libraries are released weekly, some are bi-weekly, some are monthly. Basically the maintainers of each SIG determine what they want to do. Typically anywhere from two weeks to a month is kind of the cadence. And of course, if there are security or what have you, there can be one-off releases as well. In general, we've kind of standardized around GitHub Actions for all of the CEI type activities, but the collector still uses CircleCI. There's no real mandates to use a single thing in open symmetry today, but we are starting to see a consolidation around GitHub Actions. So that might be the long-term plan. And then publishing the bits, well, that's language specific. So like in the case of Java, like you want to publish to Maple Central, we offer kind of the common locations. And then the release page also has links to relevant information if you need like the actual compiled bits. And of course you can pile it from source if you want to run it locally. So that's kind of the end user's choice as well. But it should support all of the common formats. We also try to support all the different packaging that you would expect. So like Helm charts, for example, if you want Kubernetes and there's other examples too. So try to make it as easy as possible knowing that there's a wide variety of ways you could be deploying this. Is this taking distributions and such into account as well? Of course, for example, with AWS distribution and such, you will obviously have huge user bases on specific distributions. So are those factored into, in particular the security releases? Yeah, so they're handled completely separately today. So anyone can make their own distribution and distributions would be based on upstream, but they could also choose to diverge. So one of the things that the government's committee of open telemetry is considering is putting around a certification process around these distributions to ensure that they are actually distributions and not say forks or modifications or proprietary vendor implementations. So we are looking at ways to kind of mitigate that. But today a distribution is run by whoever controls it. So like AWS Splunk has them as well, those distributions are maintained independently. They have their own release schedule. There's no oversight from the open telemetry community today. So everything I mentioned about release methodology would be for open telemetry, SIGs and projects and repositories in the open telemetry GitHub board. So to phrase it differently, currently the distributions are out of scope, but through certification and such, you want to pull them in scope.  Exactly, yep. So for incubation, they will basically just be treated as completely external, not managed, not handled. By the time we probably reached like GA in CNCF, for example, that's when we'll probably have more guidance around what you can and cannot do for distributions and what that certification process looks like. Okay, but for the incubation phase, they are considered out of scope. That's correct, yeah. So you can add a comment and I'll make that explicit in the back. Yeah, perfect, perfect. So as I just wrote, hotel is not the official thing you really type it out every time. Well, hotel is how it's abbreviated. So you can use the abbreviation. At the beginning of the doc, we did explicitly add in the AKA hotel. So you are technically okay, but don't use OT. Yeah, no, like obviously with open tracing, but I think this is one of the few times where I actually spelled out open telemetry and didn't just write OT. But names are important, I want to get it right. All good, all good. All right, so the next one's around community size and existing sponsorship. So again, I linked to DevStats. There are over 92 companies and over 500 developers that have contributed to the project according to DevStats. So you're welcome to go take a look at kind of the links. But it's a pretty healthy community, constantly growing. I'm sure if I pull one of these things up, you'll see it's always going up and to the right, which is what you want to see. Oops, maybe that's the wrong one. Is this the one that shows to the right? Maybe it's this one. Questions about that one. Architecture design feature overview should be available. So there's basically three major components to open telemetry. There's the specification, there's a repository for that, the specification kind of defines like what is trace, what are spans? Like how are they defined? Same for metrics, same for logs. That's also where like semantic conventions go for the different data sources that exist in open telemetry. And then you have the collector and the client libraries that kind of build on top of the specification. So client libraries are language specific. Every language has their own. And the collector is a single binary that can be deployed in a variety of different formats, form factors. So you could make it be an agent, you can make it be a standalone service or a gateway or an aggregator. It really comes down to what are your use cases? And then some guidance around how you should deploy everything is listed as part of like the getting started guides for open telemetry. Beyond this, you're seeing additions being added to the project over time, but these are kind of the main components in terms of repositories, there are 50 plus. So it's a quite diverse thing. Questions? Not immediately, but I think that's one of the components which merits analysis.  The bits and pieces which I saw were fine. Like it's a long pipe, but it's a well-designed pipe. So, yeah. Do we already have volunteers who want to say yes right now? Or tech leads who want to say yes right now? I would love to read a bit before saying yes, but yeah, it was good so far. There's a ton of material. I linked to a webinar too. It's about an hour long, but it at least gives you a high-level overview if you're not familiar with open telemetry. And of course we can drill into specifics of things. So if people have pointed questions, let me know, we can pull in the right people. Bogdan's on the call here. I think Morgan was here earlier, maybe he's still here, he is. So like we can get you the right folks that can answer any follow-up questions you have. Okay, what are the primary target cloud use cases? So it can be accomplished now. So traces can definitely be handled now, as well as the generating, collecting, and processing of telemetry data. So say the collector itself, metrics are very close, but like as I mentioned, still about a quarter out, given some gaps that have been identified and at least getting stable APIs. What can be accomplished with reasonable effort? Well, I mean, we could talk about like RFC type release stuff, we haven't really planned for that yet, but with OTLP, for example, being the format, that is an essential option. Things that are in scope, but beyond the current roadmap, we've already talked about logs, there's already some experimental support in the project, but that is not part of the original GA scope that we're targeting. And what is out of scope, as of today, as I mentioned, there is no backend. This plugs into backend, it's completely vendor agnostic, but open telemetry is not providing that backend, go use another one that's available out there. And noticing the time, we have about 10 minutes, but I think the meeting stops soon. So do we wanna stop here and people can review what's left or do we wanna try to chug through a couple more? You have, we're actually at time, I can run maybe five minutes over myself. I don't know about the rest of the call. I desperately need a bio break and I have the next one right after. So this might be a good stopping place and have people like please review the entire doc, even sections we've already talked about, please comment. I'll make sure this goes out on the mailing list and in the Slack channel. And hopefully either late tonight or by tomorrow, I'll incorporate the initial feedback we had from today's meeting. Perfect. Sounds good. And once again, I want everyone to talk more. Of course, this is usually just presentation style and this should be more discussion style. But to be fair, maybe people need some time to read it, but yeah, by all means people participate, please. Good choice of five access points, Steve. Thanks. At least it works. I've gone through so many. Good, good. I appreciate the communication and collaboration folks. So yeah, definitely don't hesitate to reach out even directly, we don't have to wait till the next meeting. I'm available on Slack. We can pull in people, we're in Gitter, like come join the conversation. Perfect. Cool. Thank you very much. And see you all soon. You too. See you.


Good afternoon everyone. My name is Tomasz Poszkowski and I'm Cloud Solution Engineer from Intel and today's lightning talk is about Ironic and how we can leverage that for Bermatter workloads and what's the future with Intel RSD. So why Ironic is important. So in every modern data center Bermatter provisioning is essential thing. So whenever you deliver new hardware to your data center or you deploy something new you need to go to Bermatter provisioning and so Ironic has this advantage that it offers fast and repeatable deployments. It offers easy scale up scale down with the help of Nova. Ironic as well is having full of course with Nova having full support for multi tenancy and with support of the Nova scheduler and placement API we have really good data center wide intelligent resource orchestrators and we had really good advancement in the area of scheduler so we can safely say that Ironic together with Nova offers best in the market intelligent resource orchestration in the data center and so of course there is a lot of tools, processes and software which is running on top of OpenStack API and all this which was developed for instance VMs, VM instances works as well for Bermatter instances and why people would like to deploy their workload directly on Bermatter so the most the first and the most obvious is because they would like to have no virtualization overhead they would like to extract maximum performance out of their resources and they would like to have low as possible latency and the other thing is that with virtualization we very frequently hit the noise neighbor problem in Bermatter this problem is shifted to a different place where you can is better manage that and a lot of hardware that is introduced recently to the market offers very low level optimizations that can be accessed only when running directly on the Bermatter so within a time those solutions are being extended to offer full support for virtualization but it's not necessarily happening in the very early stages and of course a lot of hardware is having issues with the virtualization I don't know if any of you have known that QM has a limitation that VM was not able to have more than 256 cores it was fixed recently but before that it was not possible and there was a bunch of improvements in integration between OpenStack and Ironic in the last year so to go further with the future parity between VM instances and Bermatter instances the biggest was the networking carrier improvements so right now we can safely say that they are almost equal with small differences and there is also a lot of work that is happening right now so in terms of rooted networks boot from volume and a lot of stuff needs to happen which is extremely important for like intelligent resource orchestration in data center I mean mostly affinity and anti-affinity and dynamic rate configuration which is not yet in Nova I mean in Nova for Bermatter use case and for what specific use cases one can use Bermatter instances the first one is virtualization so I hope that everyone is aware about the triple O and that's exactly the use case so where you have where we have under cloud which is built with help of the Ironic and then we have in Bermatter instances on which we are deploying OpenStack itself it's called overcloud in the triple O terminology and for running containers directly on the Bermatter as I said so if you want to get maximum performance out of your compute resources going for Bermatter is the perfect use case perfect case and as Kubernetes is offering their own schedule you can put as much applications under the single Bermatter cluster and make sure that they all work well and with the essence advancement in the Kubernetes related to resource isolation which is happening right now you can also be much more secure in terms of noisy neighbor problem as well and big data and analytics so everyone probably knows Hadoop Spark so everyone knows how like CPU hungry this application is and Hadoop and Spark really likes to live as close as possible to storage and running directly on the on Bermatter offers you that you can take advantage of the direct storage protocols like NVMe which is not really possible with virtualization and for HPC so HPC is that kind of environment which is like seeing a lot of legacy deployments and it's very hard to convince those scientific guys that they should migrate to the cloud because they get used to their tools and a lot of those tools simply doesn't work on inside a VM so with Bermatter and Ironic we can simply mimic their old environment with some features that clouds offers and this example super example of building a bridge that we can show people how to migrate from legacy to the cloud environment and HPC is also really about multiple different hardware accelerators and like no virtualization helps them to extract more out of those hardware resources they have in their own data centers and historically Bermatter provisioning is not the really fast-performing one so when you compare the time which is needed to deploy a VM instance versus a Bermatter instance there is a significant difference so VM is seconds while Bermatter instance is minutes and one of the areas that we identified is simply delivering images to the Bermatter instance because as it's works right now it's being delivered from the centralized point and when you deploy let's say 100 nodes at the same time you simply overloading north-south capacity you are putting a lot of stress on your centralized storage when you store the images and we figure out that by employing peer-to-peer network like torrent we can remove a lot of stress from those components and our experiment that we did together with Mirantis on one gigabit network one gigabyte operating system image and 15 nodes which were provisioned at the same time show that we can reduce provisioning time by half so it's a significant advancement and also we removed a lot of stress on different parts of the infrastructure and the second thing is that simply booting Bermatter instance is taking time because usually when you provision a machine it needs to be powered on it needs to go over the pixie boot it needs to load the kernel RAM disk and then there is ironic Python agent which is downloading the image running to the hard drive and doing the reboot so our answer to that problem is that with the expense of additional power consumption in your data center we can you can reduce the provisioning time by simply having all unused servers already loaded ironic Python agent RAM disk and what happens when someone is requesting to launch an instance it's simply downloading the image running into the hard drive and doing the reboot so by this we are cutting the like instance deployment time by half and going even further we can remove the last reboot with kernel exec so the kernel exec is the feature that you can replace one kernel with a different one and in case when you're deploying Linux instances you can take advantage of this model and in this model Bermatter instance deployment time is even smaller than VM deployment time because you simply write image to the hard drive which is dedicated for a single machine so there is no hard drive sharing and you just reload the kernel and you are done and why Intel is investing so much in Bermatter provisioning because we believe that future lies in the Bermatter resources in data center and Intel is actively pushing development of Intel Ruxgate XAM platform which is trying to answer many problems that data center operators are reporting nowadays so it's simply disaggregated that you have a computer storage and network and whenever you would like to deploy a work load you are not trying to fit it to existing infrastructure but you build physical infrastructure or demand so you select CPUs memory storage compose the nodes and deploy a work load and Ruxgate design is built on top of standard Redfish API so everyone can leverage that API and we also have integration between OpenStack and Intel Ruxgate design and sorry I ran out of time, are there any questions? Okay so just quickly finishing so we did some POC of integration between OpenStack Ironic Nova and Intel Ruxgate design and simply deploying a work load on Intel Ruxgate design is as easy as launching an instance on OpenStack you just simply select an image you select a flavor and then you are done so if anyone knows how much effort it takes to put all the hardware resources into Ironic database with Intel Ruxgate design you forgot about that because you simply provide single Redfish endpoint URL credentials and you are done all information about the hardware is automatically populated to Ironic or it's actually when the node is composed and the node in Ironic is created and being provisioned and when if anyone is interested in hearing more about the Intel Ruxgate design we are having a demo booth at the marketplace where you can see this in action or if anyone is interested in OpenStack on Kubernetes in action we are having that as well so thank you very much