Tonight is a really fun slate for daily fantasy baseball because I think we can have our cake and eat it too There are some really good high-solid pitchers Including shouhei otani on tonight's slate and there's also a game of course field but there is a very simple path using both those options and sure they'll be popular but You know guys want to have fun sometimes and I think that we can make that happen on tonight's slate It is a delight to break down We're gonna do that for today discuss which sound which high-solid pitchers to prioritize How we can jam in course feel about breaking the bank and much more to get you ready for Wednesday night Welcome on into the solo shot. That's right here on the fan dual podcast network and number fire calm My name is Jim Sonis. I'm a senior writer and analyst for number fire here to break down Wednesday's eight game main slate With locks up for 705 p.m. Eastern for today The lone weather note on this slate is that there is a good chance of rain in Oakland For the A's and the Mariners the rain odds do decrease as a game goes along so I check back on that later That's the one game that could be a bit sketchy and I think it's actually a little bit in risk So we'll talk about a picture in that game later on and potentially a stack as well But just keep in mind that Mariners A's could be at risk to terrain out in Oakland We're gonna break down the pitching preview and stacks and more in just one second But first a reminder to make sure you're subscribed to the number fire daily fantasy podcast feed wherever you get your podcast Our PGA DFS breakdown for the Wells Fargo Championship is now posted over on the number fire daily fantasy podcast feed breaking down Studs value plays roster construction and much more to search for the number fire daily fantasy podcast feed wherever you get your Podcast hit subscribe and if you like what you hear leave us a five-star rating as well The biggest horse race of the year is here And there is no better time to get in on the action on Fandal racing because right now all customers can get a no sweat Derby bet up to $20 the means you'll get up to $20 back if your win bet doesn't win The Fandal racing app is super easy to use safe and secure and when you win you get paid fast So don't miss out the Derby is coming up this Saturday Just visit racing and dot fandal calm for your chance to get a no sweat derby bet up to $20 on Fandal racing That's racing and dot fandal calm age and residency restrictions apply offer valid on first derby win wager Refund issued in non with probable racing site credit that expires on June 12th, 2023 restrictions apply See terms at racing and dot fandal calm gambling problem call 1-800 gambler if you want some thoughts on the derby to talk to Christina blacker of Fandal TV to get her read on this year's field She did talk about a horse. She likes to win broke down the favorite forte as well That's over on the cover in the spread podcast feed and on the Fandal YouTube page later on today Pitching preview for this Wednesday main slate show. Hey, Ohtani comes in with the highest salary on Fandal His salaries 11-1 followed by Marcus Strowman at 10-5 Logan Gilbert 98 Shane Bieber is 96 Kyle Gibson 93 with Dylan Cease Louie Varlin and Alec Maneuwen as the others at $8,000 or higher Now this is not the world's Deepest pitching slate because Shane Bieber is struggling right now Kyle Gibson Not the highest upside guys throw them at the highest upside guys cease and kind of a funk Which means I think we can justify just putting Shohei Ohtani at the top of our list and figure out the Rest later, I do think Logan Gilbert is in his tier as we'll discuss later on but any night We could we can just you know close our eyes news Ohtani never a bad thing now This is not an easy matchup here Ohtani is facing the Cardinals who have a 104 WRC plus against righties They're striking right is 20% so it is a tough spot and it is also one of the warmer games on this slate But Ohtani is just so good right now now He did let up three barrels and two home runs last week against the A's Those though were the first barrels and the first home runs He has let up all year long. It has been an insane run for Ohtani He's been throwing more sliders and fewer splitters across his past 14 starts so a 14 start sample basically half a year He has a 1.63 ERA. That's absurd His peripherals aren't quite as good, but his skill interactive ERA is still 3.49 He leads the slate with a 31% strikeout rate in that time and that's impressive I think the most impressive thing about Ohtani in this 14 start sample though is that his hard-hit rate allowed is 27% if you look at the other guys on this slate in their most relevant samples Nobody else is within 8 percentage points of Ohtani So he throws off the color grading on my spreadsheet because he's been so good in that area So Ohtani get strikeouts. He suppresses hard contact. He's elite We can accept that even in a tough matchup Ohtani is our top pitcher of the slate for tonight Now with that said I do think that Logan Gilbert is an Ohtani's tier as mentioned and part of that is the matchup Ohtani facing the Cardinals. That's tough Gilbert facing the A's not as much A's have a 25% strikeout rate on the current active roster against righties this year minimal power So that helps But it's not all matchup base because Gilbert does plenty himself as well We're up to five starts on Gilbert with his new splitter that he has and In that five start sample 3.04 skill interactive ER a 30% strikeout rate and 4% walk rate the bad about data not on Ohtani's level, but it's not bad and a lot of times with splitters You can see pretty rough bad at bald data. That has not been the case for gil I Did for seven point us tonight? Which is actually a bit more than Ohtani now the reason I have Ohtani ranked high is the bad at bald data is so good That I feel like long-term. We're gonna see Ohtani outperform Logan Gilbert, but Gilbert salaries lower better matchup Gilbert is dealing with reins. Make sure you check back on that. I would say it's Ohtani 1a Gilbert 1b I don't often put guys in Ohtani's tier unless it's like you're a call or whatever So putting him in that range is a pretty big compliment to Logan Gilbert So to me, it's Ohtani 1 Gilbert 2 Nobody else really in the discussion for the top two pitching slots for the value play Dylan sees really struggling I used him as a pitcher last week and that did not go very well But his salary is down to $8,900 and I think that does open the door to use him tonight Sees his face in the twins Which is not a perfect matchup, but it's also not the end of the world the twins will strike out They have a 24% strikeout rate against righties and it's not as a cease is not getting strikeouts That's not been his issue so far this year strikeout rate is 26 percent down from 30 percent last year So it is down, but it's not awful especially on this slate. I do think the velocity is concerning for cease he's down about a mile per hour across the board from last year and it's not trending up either which is Concerning because a lot of times we can see low velocity in colder temperatures. It hasn't gotten a lot warmer I know last week for cease in that game against the rays a lot of the temperature was still low So maybe that's why the below is low but You know, it's not getting better and that's a bit concerning for me The numbers I cited above though the 26 strikeout rates are from this year with the velocity being down He still has a relatively high strikeout projection of 6.4 He's just letting up too many walks too much hard contact and that could bite him here because the twins getting healthier Has some guys who can bang that's definitely a concern Ceases 6.4 projected strikeouts ranks third behind otani and Gilbert for tonight It's enough for me to use him given the decrease salary if he were let's say $9,800 I probably wouldn't do this. I mean the same salary as Gilbert. I would just say No, thanks, but he's worth a shot. The twins offense has had its ups and downs for sure So I think cease works at 89. Uh, definitely okay with him Well, I'll talk about one more value play in things to watch But to me this slate is all about otani and Gilbert and large part because I can get there Part of the reason I can get there is because the bats that I want most at course field come with low salaries Let's talk about the stacks here for this Wednesday slate and we talked yesterday about cores I wanted just one side and that side did not do very well the brewers did not do much in that game But tonight I want both and I think both the rockies and the brewers are the top two teams for stacking tonight I'm gonna rate the brewers higher. So we'll discuss them first But we'll talk about the rockies after that the brewers facing Kyle Freeland who has come back to earth after his really nice start to this year He's left three plus earned runs in now three straight games And one of those was on the road So not a course field when we look at everything together the three good stars with the three bad ones We can stack against the profile. It's a 5.08 skill interactive erray for freeland 37 percent hard hit rate allowed it's about average, which is not what you want a course field You want better than average 45 fly ball rate. So again, you take those numbers put them at course field We can get some fireworks for sure The brewers are not good against lefties. Uh, they are Pretty bad a lot of strikeouts this year a lot of strikeouts last year But they do have some guys who can hit so I feel good stacking them here Netting the brewers do great out pretty well from that perspective the brewers the big uh ticket option here for me Is a guy who has been injured until yesterday tyrone taylor came off the aisle last night Started against a righty, which says to me he's probably gonna play against lefty And I hope he gets a good spot in the order because I will go bananas if he does taylor really good power 42 fly ball rate against lefties in his career. It was 44 last year with a 200 iso and again, that's not a core Is this is uh, you know looking primarily milwaukee If we get taylor batting eighth at $2,500 I will use him and be happy about it If we get taylor batting higher in the order, which I think he probably will I might put him every line up. That's kind of where I'm at with tyrone taylor. Um, I'm very excited here I like taylor for a pretty long time because of the power Potentially to my own detriment, but I think tonight it just makes a lot of sense. So tyrone taylor Put him at the top of the order. Let this guy eat I think that he makes a lot of sense and is a very good play for tonight So tyrone taylor is like the guy for me on this late even above otani and gilbert. I just adore him for sure now The rockies are facing eric lauer also a good matchup and I said a bit less faith in their offense Which is why they're lower than the brewers, but they are in the same tier for tonight The reason they're in the same tier despite the fact I don't like the rockies offense is that lauer is still struggling He's been letting up tons of fly balls in hard contact this year And it's led to a 5.19 er a his expected er a baseball savage is actually even higher at 6.19 And there's room for it to get worse because he has just a 7.9 percent swing and strike rate So his already low strike out rate could go down even more if that strike out rate does go down more His hard contact issues be even more concerning The rockies definitely off to a tough start this year a 68 w rc plus against lefties and we You know can definitely be annoyed by that But I would not be shocked if that number goes up in a big way for tonight So very high in the rockies here do like them quite a bit The tyrone taylor equivalent within this rockies lineup is randall richard for a couple reasons. The first one is Just came back. So there's that equivalent low salary 27 Gritchick did not have a great rehab since But he has made some hard contact and this u plate appearance in the majors He's not striking out a concerning amount yet So with gritchick at 27 and taylor at 25 I think he's in that course field Pretty easily while using otani or gilbert especially gilbert. So to me we can have our cake and eat it too I'm very into that. So to me. I think the rockies and brewers Very stackable even when you decide to go with a high salary pitcher for tonight Outside of course field. I feel like the top stack tonight has to be the cubs They're facing jake ervin who is coming up to make his debut for the nationals last year Was his first year back off an elbow injury good to see him make it through the entire year fully health And he pitched really well in high a when he first came back He got moved up to double a earlier promotion there and not a shutdown there for ervin He had a 4.7 90 array across 15 starts in double a his strike area is acceptable But a lot of fly balls that year Where he was in high a and double a was his age 25 season We saw ervin make five starts in triple a this year is age 26 season before his motion He has a 5.30 exit in triple a strike outright pretty low at 20 and it comes with a 9.4 percent swinging strike rates We also saw ervin walk too many guys down in triple a and now He has to come up to face big league batters and it's not just generic big league batters The cubs have been very good against righties so far this year with a 114 wrc plus a 165 iso Not that many strikeouts I think they're probably going to give him some issues for tonight and the bullpen and defense behind him are pretty poor So if ervin Does okay We can still get some points after that via the bullpen and the defense it comes to be facing So cubs are not on the level of course field stacks I think that they're definitely a tier below them But I think there's still a really solid option here and they do have some value plays I know a lot of the guys here for the cubs are high salary, but Seiya Suzuki comes a little low salary here and he hasn't been great since he came off the il But he does have a 51 heart hit rate Not a lot of strikeouts and the strikeouts have been decreasing The further he gets removed from that il stance We did see Suzuki show off some power last year was willing to run has not done that Well, he's run twice. He's gotten caught both times, but he hasn't been as willing to run so far I think he saw that upside in him. It is a bit of a leap of faith He hasn't shown a lot so far this year, but he's $2,900 really good stack I think that he makes sense if you did a one off on this cubs team I think Suzuki could be a worthy consideration for that spot Let's go to things to watch and mention before there was one more value play I wanted to discuss I did get some thought to louis varland for the twins against the white socks He has never been a high strikeout guy, but this year somehow varland is throwing gas His fastball velocity 96 miles per hour. It was 93.8 last year His slider velo is up three miles per hour and he's throwing that pitch a ton So he's a very different picture than what he was last year when he was a low strikeout guy He made one star in the big leagues came against the Yankees had a 14.5 percent swinging strike rate This time he's facing the white socks pretty chilly temperatures in chicago for today I think he's worth considering at 85 if you want to go kind of weird I think he's more of like a multi entry guy if you want to find ways to differentiate You want to load up on chorus field without going the full route of the otani or gilbert way I think that varland could be a fun way to differentiate without being Fully fully dumb. It is might be dumb, but maybe not fully dumb. We'll phrase it that way I'm okay with the oral snites actually pretty okay with them. They're kind of in the same tier as the cubs personally They're facing zack ranky who doesn't seem to be fully right right now Lot of hard contact this year. This is also the warmest game on the slate in kins city So I like the old the o's quite a bit if this were a normal state Slate with no course field or if both sides of course field worms attract as they are the auras would probably be a top three stack so They're in the same tier as the cubs for me for tonight The mariners viable for stacking again assuming that the weather winds up being okay in oakland They're facing jp serious who hasn't had great results so far this year But this one is more about the bullpen for the aids which might be the worst of all time I mean it's just it's like it was like 2016 or so The reds had like a hideous bullpen where you'd stack against them even when they had like okay starting pitchers Sears let's up fly balls and hard contact on top of that So This is as much about the bullpen as it is about sears The mariners salaries aren't too bad. Um, definitely can get behind that I would downgrade them if Julio Rodriguez can't play but overall the mariners fine to consider For stacking for tonight. Let's finish up here with some dinger calls I typically try not to do both that course fields But I want to do both a course field for today given that there are lower temperatures elsewhere I like the mattress of bull teams both guys are bull teams facing guys to let up a lot of fly balls On a decent amount of hard contact. So sorry. We're both going both the course field. The boring one is cj crone Crone has been I mean just like his typical self. He hasn't been like outstanding or anything But does that six home run so far this year one of those came a couple nights ago It seems like recently been hitting the ball pretty well So we'll go cj crone for the boring one. It is about as boring as can be but you know It is what it is the fun one. I got a ride with tyrone taylor Fresh off the il $2,500 again, even if that's eighth I will go hard at him for tonight because I believe in the talent like the power $2,500 good stack So even if that's eighth still in on tyrone taylor if he bats higher than that We are Full on pants off and going hard at tyrone taylor for tonight. So home run calls are cj crone and tyrone taylor That is all that we have here for today on the solo shot as mentioned If you want some pga dfs thoughts for this week check it out over on the number fire daily fantasy podcast feed Wherever you get your podcasts and why are there if you like what you hear leave us a five star rating as well If you want to check out a video version of the solo shot You can find us over on the fando youtube page shout-outs all of you watching there right now with the thumbs up button We appreciate all of you as always if you've got any questions for me. I am on twitter at jim sonnis j i m s a n n e s you can also follow the fando podcast network at fando podcast I want to thank you all for tuning in for today. Good luck to you with your mlb dfs lineups We'll talk to you once again tomorrow to break down thursday's slate This has been the solo shot right here on the fando podcast network


What's going on everyone? Sam here with late in sports cards and I am ripping one box of 2021 Penini contenders optic football for Charles L. So good luck Charles. See what we got in here for you, man Oh, yeah, for sure that pays for well over the box Well, well over the box and then I gotta figure that the trout Like the trout out of 50 is a pretty decent poll as well Like any low numbered trout's gonna sell fairly well, I'd have to imagine All right, see what we got here Got DJ Moore on the red That is to 175 for the Panthers Nice hit there and then we've got blue winning ticket to 99 back Prescott Cowboys There you go, Charles. We've got bases DK Metcalf Got rookie of Jalen Darden. Oh, that's pretty sick Got numbered to 50. How about Antonio Gates on the All-time contenders or all-time contenders. Excuse me. That's sick That is numbered 31 of 50 nice Chargers hit Charles. Wow, what a box man. We are on fire tonight 7 of 30 How about Zach Wilson on the blue Boom, let's go Heck yeah, congrats Charles Look at that. Let's go man What a break so far. I'll get that in a mag for you after the uh after the video's over It's gonna do it for your break there. Charles. Congrats on all the hits get that right out to you, man Thank you again as always


so that the pattern is to go from this to this but that's just the forking right because after you finish that you notice that the postmaster is one of the first kind of things that it nets in postgres and the very first thing it does is sets up shared memory well shared memory only really makes sense if you have multiple processes not only that but if we look at the system five IPC stuff we have shared memory we got semaphores we got shared libraries we got multiple processes we got signals we got the forking we got the IPC all this crap postgres uses and none of this makes sense in Unicolonel land once again most Unicolonels are single processes single address space systems so the I guess the learning on this is basically if you want to go down this route there it might be easier instead of trying to modify that code you can maybe potentially split it out you know postgres makes all this use of different systems maybe they can each be Unicolonels but on this note there are other systems that don't fork I think we all know which one I'm talking about windows of course now windows obviously has support for multiple processes but it doesn't actually implement the fork and so when you're diving through postgres code you're finding stuff like this internal fork exec back in fork exec and so they've already put wrappers around situations where you have to deal with this but one of the best things about doing these projects is you do what I call code dumpster diving and this is a real comment that I stole from this line like you can literally click on this link and see this code due to backward compatibility concerns the replication parameters the hybrid beast which allows the value to be either boolean or the stream database and so as a programmer you're like this is just some some mongo db bullshit right here you know but this is the fun part I like when with these projects is as you run into these little gems so so what else sucks well you know the operating system whatever you're using you want to or obsd or whatever it is you know you're given so much that you take for granted right and so so one of the things that we kind of ran into that we took for granted was the fact that we just assumed our certs would be in the the proper place we're making outgoing htp calls and those were working but our tls certs were complaining about missing x509 well it turns out and so this is the the go run time well the the go language source it turns out that every single distro out there not only has their certs in a different location but it's a completely different file name which is complete that shittery to me so you know we had to go in and modify this for something that really shouldn't exist in to begin with but this is this is kind of an example of the type of stuff that you might have to deal with I'm not saying all the time but you might have to deal with if you want to play in this ecosystem here's another kind of sample that sucks I don't know if anybody recognizes a bob on the left here but so he talks a lot about testing anyway on the right is the example of a test we were having a problem with the go run time where garbage collection wasn't actually getting ran and so eventually like um after a certain number requested just stopped responding and so after the patch was fixed and you know that was working we're like well we need to write a test you know how do we actually go about doing that and you know make sure that our fix works so what we did was we wrote a little sample go program that it boots up it prints out gc stats then it allocates like some big ass array and then it prints out the gc stats again and make sure that everything runs so this is this happens on every single build that we do with the go language and so then it compiles that as a unicolonal it boots it up into kvm it waits a little bit and then it screens scrapes the output from it to make sure that gc has ran so this is obviously a complete pain in the ass and this is totally unwarranted and this is this is definitely like one of the shittier parts of the ecosystem um so there's that well i mean that's that's a good question you know it's uh this is one of those things where the tooling could be much better this is a little bit of an outlier because it's not this is more language specific rather than you know like if it was a dynamic language it would probably work differently so it's not something that would I don't think it's a one-size-fits-all type of solution so uh so yeah it's just these are I'm just trying to be real with everybody in the room these are the actual like pain points that might suck so what the thing is like you know what's a hacker to do right you know we hack around it it's it's not a big deal at the end of the day we run into this stuff using Linux anyways so enough of that let's uh go ahead and build something once again if you download this tool you can kind of follow along uses as kind of a teaching tool get up.com slash defer panic slash Virgo if you download this you can pull down ready to go unicernals in a matter of minutes like you can literally boot your first unicernals in two minutes on your own machine if you want to download versus build them locally you can grab an account and go for it actually I don't even think you need that but basically you can access through this tool ways of running your unicernals locally and then you know also kind of going through so if we look at a sample project and it is a project that's more than just like a raw disk image right um it that's something that if you read some of the tutorials online they're just total world tutorials it's not like a full bloom you know if it's how you run it type of thing and I think that's kind of the missing part is filling in with people who are interested in running these kind of giving them the tools and the knowledge to kind of create systems for themselves but if we look at this we have a project and this is running locally we have the kernel of course which is the image and probably everything else that you want logs you know we probably want to interact with it somehow the PIDs we're just keeping track of the processes now keep in mind running this locally you're going to be using like QMU or maybe it's virtual box or something else obviously Linux is going to be a much better choice for you than OSX yep so we'll get to that we'll get to that and then then we have this manifest and then a set of volumes so in our experience once again contrary to a lot of the Hello World tutorials you'll run into out there we found that having like a single volume on the system is not the most ideal thing in most cases typically we want multiple volumes for lots of different reasons but in terms of the manifest you can see kind of a sample manifest that you would find here so we set the memory to like 64 megs you know this a lot of this is just arbitrary that the kernel is a kernel name multi-boots or not multi-boot there's some implementations that aren't they might just be raw disk images so and that does have consequences in terms of can I pass an environment variables can I do other stuff it's something to think about you know once again the command line the environment variables that you might want to set and then of course your volume and set up three volumes for this one one was for what we kind of do for system-related files this is like your SSL your host file you're like all this crap then ones for like here the Python interpreter and everything of that self and then the third ones for your actual personal project code itself and so that way like if you want to change out your new code we don't have to create you know two other we don't have to just keep on recompiling every single time we can just upload a new volume with your new code and it just immediately takes effect so this is this is one secret that we kind of found when we were playing with this stuff and it speeds up development and deploying like tremendously in terms of building these things I'm not going to kind of really talk too much about it but I am going to kind of go over our thought process when it comes to building these because there's no like set standard and I don't think there's there's never going to be like a docker file type of thing for these things it's it's way too different you know because because you have IoT you've got NFV and you've got web we're primarily interested in web and you would build unicernals for web completely different than you do for NFV so it's it's just completely different stuff but I've I've kind of divided the dependencies into a couple different layers so once again dependencies dependencies dependencies so level one dependencies are kind of just native libraries that your application uses you're probably not in the process of dealing with these most of the time this is like for your dev ops or sysadmin or whatever is going to come in afgate install or they're going to ansible they're going to install doc or whatever it is right there's probably a handful of these libraries that you have to have and I can tell you right off the bat libxml which comes with it's own ftp server by the way and libxmlt is used by almost every single dynamic language out there so if you're ruby your python your javascript any of that stuff you have to have that and so the way we build our stuff is that we actually have these binaries sitting around and we just link them in it saves you the time of compiling them we know what goes in their hash their version their save all that good stuff so that's level one level two is kind of like code specific stuff that's specific to your project so this is your bundle install your npm install your pip install like all this sort of stuff these are this is dependencies that your project needs but it's probably specific to that language so once again this is kind of different there's caveats here so like if you're rubyland and you're using our magic that actually uses the native library so you're going to need image magic installed as well so we've kind of separated those but in most cases like if it's if it's ruby code it's going to be a ruby dependency if it's python it's python that sort of thing and so so all this is to show you that this is all like automated as hell otherwise it's going to be some ugly ugly bash bullshit level three is kind of pre and post setup so like there's typically random stuff that you might need to like shell out before after you compile your image to do pre-processing or post-processing of your image so that's you know you don't have a great example here but you get the idea you can like use app getter or whatever and then you know we ourselves actually export kind of a compilation system to where we actually have it set up to where you can check in code to get hub and then you have a get hub post commit hook that sends a build trigger to our system and we immediately compile your unicolonel and if you're actually running them we'll go ahead and cycle them out and push them to prod but what I wanted to show you on this was this little script box which is kind of what I just showed you right here the script section there's another command here called the build pack and the build pack is pretty interesting now you can't kind of see where it says build pack colon roll rails so I don't know if you've ever touched a rails project but if you have you could you can kind of think about like all the crap that you have to do to make that rails project run and you know you have to pick your ruby interpreter or you're running 1.9 or you're running 2.3 you have to pick your rails version is it is it 2.x 3.x 4.x 5.x you know all your dependencies and so and this thing goes down to like is it a little bit my sequel is it postgres is it you know what are all your dependencies and so you think about the tools that you use today and how you provision these systems for your projects today but all that needs to be kind of reinvented and redone and we kind of have our opinion on how we think things should work everybody has opinions but uh so that's that's kind of what we think is you know make it as simple as possible so if uh you know once again we're at Linux talks so this slide might not be super uh important but basically on OSX you know there's a few things you need to do to get started to come up to par to a Linux system you probably are missing your ton and tap devices which don't come natively in OSX so you can download these extensions and get them installed and then the the firewall seems to change from release to release release they as of last summer they were using IPFW and then they switched to a PF control and so that's how you can do your fake NAT translation because once again you guys probably don't have ethernet on this and so you're probably using wireless and you can't really bridge on those devices not truly anyways so that's something that kind of bites you hiddenly if you're not set up to use those tools immediately but just to kind of show you example here's some nasty little shellcode and uh so but it shows you you know your your your your translation for that device and so this is all inside that tool that you can kind of look at and read to see how we're doing this automatically and then you get to CUMU and as if that code wasn't nasty enough then you then you look at CUMU's code where they take in JSON as their option parsing they also export a nice QMP protocol which you can use there but here you can attach the block devices and set up your network and point it to the kernel and all that good stuff and so like when you go to the hello world tutorials online and you're like how do I boot these things and people are like just use CUMU it's actually not as easy as that there's a whole bunch of other crap that you want to have if you want push button deploys you know for development or a problem now I showed you kind of the nasty stuff but just to show you kind of like what um more production worthy stuff looks like this is an interface taken from about a year ago from a client told us we developed and so you can kind of see and this was this already assumed like multi-tenant tendency on the system versus your development maybe not guaranteeing that and so like I said this is about a year ago old but you can kind of see all the all the different things that it's exporting to that particular client so you asked about public cloud so this is opinion I don't think the public cloud as it is today is really it doesn't gel well with unique kernels there's lots and lots of reasons why just to give you background more than a year ago back in January of last year we're on google cloud love google cloud it worked great for what we were doing before we wanted to run a unique kernel and before that I had used amazon since like 06 or something like that so long time public cloud guy I'm not like you know I've been on the bandwagon but there's some things that it was meant for linux it was not meant for unique kernels right off the bat if you try to provision an image either google cloud or azure or amazon right off the bat you run into these one gig volume limits so you're like well you know that's not a big deal I don't care how expensive it is and to me it's not the price issue it's more of the fact that if I have to compile that image and I'm probably doing that locally I don't want to be pushing up this one gig image every time I hit the fucking deploy button and I know it doesn't matter if it's a sparse file it's still going to be big because you're using all these different layers of api's and so not only is this like the minimum but the 10 gig is like they're you know kind of the default and so that's that's a huge problem like right off the bat for a real production system the other thing you know when I was showing up earlier where we were spinning up like 1500 VMs all at once you know if you were to do that with like T2 Nanos on Amazon that would be a ridiculous bill by the end of the month and T2 Nano for those of you don't know I mean that's like the cheapest you cheapest you can find on Amazon and it's literally like no resources so it's it was meant for the old VM world not for Unicurls now eventually I'm sure they'll add support but until they do you're kind of relegated to using other dedicated stuff so yeah hell no on the one gig L2 networking this is another thing that we ran into once again we're not going to go hypervisor hypervisor so like I can't just sit there and run KVM on Google Cloud there's no hardware acceleration for that right because we're already being hardware accelerated on the underlying VM now there's some academics that have and you know I think parallels or something they actually did nested hyper hypervisor pass through but you can't find that on any public cloud and so that's like you have to compile down into that base image well then that presents the other problems so if you're coming from the unique talk this is actually how they do it is they actually have another instance that kind of controls their other instances but that gives you very limited control and it it really suffers because there's a lot of security controls and there's a lot of routing and networking controls that are just they're not exposed to you it's not like they aren't present then obviously gives you that capability but you can't touch it because it's not yours to touch so this was a big impediment to us for what we wanted to do so you know that sucks and this kind of implies that resource aware scheduling is just like not even available so once again if we're talking from like a serverless point of view or if you know we have all these you know hundreds of unicernals and we want to schedule them in a way that makes even remote remote sense you know we really need to be at that hypervisor layer and so this was all sorts of stuff that kind of made us like spin off our own stuff and so um after after going from like one dedicated provider to the next we eventually just start like wrecking our own servers and so those are in Fremont now so uh so yeah that's where it stands there but you know at the end of the day like unicernals share similar problems and similar benefits with containers they also share similar problems and benefits with VMs though and this is something that a lot of people don't immediately recognize you know we have 15 20 years of like kind of enterprise virtualization out in the world today so we have all this software that's been built up to deal with VMs and we get to reuse a lot of that versus a lot of the you know obviously Bill Joy you know did she root back in 1979 so the point is that it's a little bit different from all the Docker stuff today oh on terms of workflows unicernals offer lots of different workflows that are not readily apparent and some of them get forced upon you one is the configuration management it's something that people are like well you know they deploy their first unicernal and they're like oh this is cool this is cool okay well let's go port all these systems over and then the next thing they know they're talking to the DevOps person he's like well you know how do I how do I deploy my configuration management and because you know once again most of those tools they're using shells and their SSH and in or maybe they have an agent sitting on the system and they're yanking down changes or something of that nature and none of that's available to you so just as kind of an exercise I'm not saying this is like a production thing but as an exercise just to show you this is like and you can find this on on our github like a 15 line php app all it does is export a simple little file browser that you can kind of edit and mutate and so you can kind of see here we're playing with a host file on the system and so so I guess what I'm getting at here is like just because something seems kind of impossible at first doesn't mean it's impossible you just have to kind of think about it differently to deal with it then one workflow that I actually kind of like is what we're calling the volume deploy workflow and it gets back to the parts where we found that multiple volumes were better for lots of reasons than one single volume particularly with interpreted languages because if you're applying the new code chances are you don't need to update the javascript interpreter chances are you don't need to update the php interpreter you know all this sort of stuff because it's just you know it's just new code there so what's really cool is you know once again using php as an example it's like lowest lowest amount of code to illustrate an example you know we have this little app all this is test and then it includes this other file and this is test right so we're missing code here but all we do is we we go ahead and insert you know a hello from blah blah blah we make a new file system all of it and then we upload it and and you can kind of see we just remount that file system and we're off to the races so you might be thinking wow man this sounds a little bit eerily like nfs yeah yes and no but it's it does allow you to think kind of outside the box in terms of how do I deploy changes and how do I deploy code to these systems and can I do this in a different way than what I'm used to nowadays databases so once again this is this is one of those things in Unicron land where you don't know that this is actually a issue until you actually go try to deploy something and you're like well fuck how do I actually do that you know because most people are probably used to just like they have some scripts you know some rig file or whatever and it goes and it shells out and does it you know my sequel blah blah and you know all of a sudden you have a new database schema so you can't do that in Unicron land so all your code needs to be pushed out into its own image and so I don't see this necessarily as a bad thing it's just different it's very different so in terms of this this is just an example and go you know where we literally deploy like an application now this is this is obviously a bare boat example so you could you could imagine somebody makes a tool of this and then you just have some yaml with your new schema or whatever and you you go to the races like that but this is this is kind of the stuff that when you're talking about tooling this is the type of tooling where we could use more tooling in the ecosystem rather than doing something stupid like this and then this is something like just something I've been talking to lately you know I mentioned all these companies that I know that are working with union kernels and they all have this kind of like reoccurring theme they keep on asking me and they're like you know should we be calling these these things different because it's it is kind of interesting anyways so the question is what were you build and before we get off here show you the tool so so Virgo is that tool that I kept on mentioning and can like everybody see that I can kind of bump it up a little bit so so yeah so we we have this little tool if you've ever used any of the the Docker bullshit it's very similar on purpose but basically you can you know look at your images and we can kind of pick this can run this html one and if I can take this bazard and it'll go ahead and it'll run some stuff and so so now we're deploying we have an html app so this is just statically straight but we can also deploy databases and so forth like that so in that case we might deploy multiple union kernels working together one one running in database and one running as a script and you know there's there's plenty of other stuff so here we're going to spin up another one this one's a a php one right and so now we're running a php union kernel but really the list is in in list as you can kind of see from some some of these I just searched first and test union kernels so you know my friends talked about registry list union kernels but or registry list union kernel deployment we actually do run a registry I don't necessarily agree with that deployment model but I do see it as an easy way for people to kind of try things out and kind of get started and you can literally pull down any of these and and run them locally in like a minute so it gives you some some room to play with I think that's all I have right now but we can jump into any questions or discussions or arguing if all and it is yeah so that's a good question I would say you know if you took like the average VM size for systems on our site it's going to be about 20 to 30 megs now having said that if you're running something like Ruby or Java those tend to be pretty big they can be like 200 300 there's not so much we can do those are just those languages take a lot I'm sorry now if you're doing something like include os so include os is a c++ implementation they produce stuff that's like less than a megabyte so less than a megabyte so like in the kilobyte range you know I mean I can literally if if we actually have network here yes all right so I can just show you real quick I think I have some booted up here so yeah so like this include os one I mean it's it's literally 700k when it boots up you know but you know the python stuff can be fairly lightweight and actually there's other interesting things coming out with python like I just saw one the other day it was specifically built for raspberry pi but I was thinking you can take the exact same thing and use it for what we're doing and so yeah so this is a this is kind of an interesting thing I'm booting an application here it's actually already booted but the dns takes a little bit for that but yeah so it's it really I would say your mileage varies quite a lot in new new kernel land and that's why I don't like really believe in a one-size-fits-all depending on what you're doing what language you're using what implementation you're using like all these different they're so wildly different that I don't even think you can have a different thing and so like for example this is rails with the database and everything running as a union kernel so if you're talking about like real world applications the answer is that definitively yes different parties are going to have different results so but but it is yeah I mean you can dream it you can do it so any other questions good cool Hey I'm Hans I'm one of the ones who's been helping putting on the track so I was just calling to check in so we're having all of the screen again well uh I think we just have let me find the end that doesn't work we'll get started in a second it said the voice of God I don't know I'm giving away all the all the jokes success so let's see one, two, three, four, five, six, five, six, five, six, over 10 you owe me a beer thanks for coming guys last talk of the day thank you so much for coming five, six, seven, eight, nine, 10, 11, 12, 13, 14, 15, 16, 17 absolutely not that is my beer so it's not it's not double it's 17 in the audience and I actually suggest that if we had like under 10 maybe we just go get some beers and we make it like a fireside chat it's not exactly thank you everyone for coming I'm Chris Rado I'm Clint Byram and together we're the Aristica we we did have a whole dancer team worked out but the stage doesn't yeah it's not going to work with the chairs we'll we'll next the origins we'll do that yeah yeah good to know don't let them anywhere near the stage so we're talking about transforming organizations with OpenStack and the alternate title is the story of Pirate Cloud yeah and don't maybe add some Lego so don't tell them oh yeah I'm sure they're very nice people in Denmark they're very happy that's true so both of us worked for IBM we used to be on the same team but recently we've split up but about a year ago we were working on the same project called Pirate Cloud and Chris and I took on this challenge which he's about to describe to you so we came to we came to IBM he was at HP I was at Brantus and they we kind of came in with a whole bunch of other people and we had a very simple challenge and that challenge was to deploy OpenStack show of hands quick anyone here and have you deployed OpenStack okay now keep your hands up have you deployed it for master okay continuously keep no hands all the hands went down looking looking no and without carrying any local patches or only really basically with carrying absolute emergency local patches so that was a challenge it wasn't actually simple but we were up to it yeah so and and we had learned a lot from our previous endeavors with OpenStack Hugh from Brantus me from HP and others and we decided we can do this we've seen the mistakes we've seen the pitfalls and so we started this gunworks project so it was Pirate Cloud and it was yeah definitely kind of supported by some people and those people cared about what we were trying to do and they made sure that we had the support and the money to do all the things we needed to do but it was also a little bit underground and we were it was a proof of concept and it was like I say this a bunch of different ways it was a work in progress it wasn't production ready it had some kinks to work out as any good project is right but definitely it was subversive in that not in that we were starting to subverting anything important in our business but we were subverting some of the the beliefs that had been built up around OpenStack that it's really hard to deploy that you can't do it from master that you need to carry a lot of patches to to make it work in fact we believed all these things to be opposite of the truth and so we did kind of feel like we were that pirate radio station like you know we're going to tell you the truth you know they can't shut us down and that's where we came up with the name and we have some broader aspiration too we wanted we we believed that by setting up this project this way setting up this cloud this way in some ways we kind of intentionally tied their hands and made it in a way that would force the internal contributors to increase their upstream involvement which was one of one of the key things was not letting you really carry local patches make it much easier to fix it upstream first the other thing we using the same tooling we were going to we believed get much better collaboration and introduce people to things like Zool and Garrett and other kind of really good and really great or in some cases get yes some people it was new to a lot of surprisingly and the increase of upstream involvement was was probably the most important thing we were chasing there and that we one of the things we had seen fail in open stack deployments was that there was this sort of belief like whatever just comes spraying out of open stack is going to be perfect and ready to turn into a money-making machine on day one and that absolutely isn't true and we know that should be true with any open source project the people who do the best and build the best businesses on top of them are at least present in the upstream if not also developing on it so we wanted to make sure that when we taught these teams working on it that they understood that it's the right thing to be working out there in the open and in the community and the best way to do that was basically make it the only thing which is what we did so we had some good reasons just like what we're saying about why why deploy from master which goes along with that and actually I want you to talk about most of this because the next few slides are very much right so it kind of sounds crazy like a lot of us have come up in open source you know release early release often is a thing but we still wait for the release right like unless you're crazy pants developer who wants to try all the latest crack like you wait for the release so the developers at least said it would work but open stack is different and it has been different from day one in that people were running it from as soon as it existed people were running whatever was in master this was rack space this were other companies and so there's care taken to take care of it and that combined with the fact that people like Jez Humble who have basically published what I call you know sort of my my DevOps Bibles of continuous delivery where you know limiting risk is actually easier if you just deploy a little bit of change each time instead of deploying millions of lines of code which open stack produces every six months and and definitely as you are deploying if you want to change it I want to make a patch to open stack and do it upstream if I wait for the six month cycle it might be a year before I can deploy that patch but if I'm deploying continuously from master I can go ahead and just land that upstream two or three weeks later it's out in my cloud so that is that is a huge benefit so I included here this is a graph from our actual deployment when we were deploying and this is the number of lines of code change that we were deploying every time we went to went out to production so when you see this these are a few days apart we would try to deploy every Tuesday and Thursday and we would take all the code that open stack had landed put it into intermediary step and try to test it and if it if it passed the test we would land it in our code and then on Tuesday or Thursday we would put it out to production so this was usually 40,000 40,000 there was a big patch that landed right so you see up there almost 100,000 lines of change every few days this is how we were deploying from master as soon as something went into the into the open stack code base it went into our cloud and you can see some of the the distances are longer because it failed tests and we had to spend a few days working upstream or working downstream on patches to fix it but every day 40,000 50,000 a few tens of thousands of lines of code which seems like a lot and it's like oh my god scary we're going we're sending this much out but if we'd gone with the stable model this is the release before Liberty and then up there is Newton right and you see these are I don't know if you can see the scale it's really tiny but that's four million so on release day we would have to find a way to mitigate the risk of four million lines of change instead of 40,000 right and then on the next release day Newton was there or that's actually Liberty I think this is the one before this ice house or whatever I'm losing track of the names is six million right huge releases and then this one actually came out while we were working because I don't know if you guys noticed but down there is actually the amount of change we actually we're doing on the same scale which is almost invisible this is an overlay that's right the glue is actually at the bottom that's the graph we're looking at in the previous slide so it's pretty easy to look at this and go oh my gosh this is terrifying we're sending out 90,000 lines of code today right but in here we're sending out four million right and there's a lot of stress in that and if you if you go this isn't a just humble talk unless you're here just all right and come up and talk to the rest of the time I like but it is about what he has to say which is it's way less stressful that just know that you're going to be doing a few tens of thousands of lines of code so we definitely believe strongly to follow master as close as possible and then just mitigate the risk as it comes out of the project rather than let it build up for six months and it's and you think about the people that are deploying clouds based on stable and just kind of waiting for this giant code dump to come along and when they migrate to the new version this is the amount of code they potentially have to look through when they run into problems like that's the scale the delta of things that could have could be the cause of a failed upgrade versus doing it in much much smaller chunks all the time and that was I mean that was the big thing we were pushing yeah and there's more change in here than just this is lines of code which is actually kind of an okay parameter you don't see how in there there were 15 database migrations some of them which take hours and hours it's much better to only have one day where you need to do those where you do the one three hour one in a nice downtime window instead of like let's plan for the 16 migrations and have many many many downtime windows so there's lots of change and mitigating it was a was an important factor for us and so I think the next couple sides we kind of sail out of what we already said but it's like it's the first time we've done this talk together who could have guessed no one would you shouldn't have said that oh the mics are on so the some of the big advantages right you'll be ahead of every other cloud there for a while and still and to some extent rack space is deploying for master but they have pretty big drift in and they're carrying a lot of local code so so there there was the potential here that we would have been like the only large scale cloud that was constantly deploying for master which meant we had access to all the new stuff that was coming in and we were for for the while that we were doing this were leaders in the community for some of the work we got disadvantage is a lot of the same advantages so we were had of every new cloud and so we had the first we were the first ones to find interesting breakage and and that it was actually actually happening quite a lot we were finding things and proposing patches upstream to fix them so it we were finding things in our test deployments and getting them fixed before we were deploying out to the to the other regions being the leader is always the easy path actually rarely is and yeah so the way we were doing it we were using the wall I'll go through the these bullets real quick and then I'll let Clint expand on some of this we were cloning all the open stack project repos and we would try to deploy if it worked we would test that deployment and then we were doing that about every hour and the same test that we were using basically we were using the same test all the way across so any any local change was being tested the exact same way any upstream change was coming in right and one of the the top bullet is tiny and shortest but it's actually the most important in that Zool is a system that OpenStack came up with to run commits through really fast and not give up on any quality so it's it's actually not something that OpenStack talks about enough I think and it's become the focus of my job every day now and we'll talk about it later but essentially by us using Zool in the same way that OpenStack does we're able to ingest those hundreds of patches that land for day run tests on them successfully and not have to give up on the quality not have to shrink our test coverage and also we can kind of get some transit of benefit because we know OpenStack's doing this too we can make some assumptions like our we look at what our configuration is different than what gets tested in the gate and we only have to really test that surface area really deeply so by using Zool in those two ways one by depending on upstream and one using it downstream in our own code we were able to get a huge amount of coverage and still keep the train running it wasn't easy to get that set up and in fact we took a long time and several generations of upstream sinking to get it right but we we definitely mastered that and we focused on that rather than manual testing because if we had manual tested we had seen this happen eventually you stumble and then the train just goes and it's gone and you spend the next six months trying to get caught up for that stable release so we were like we're riding the train we're staying on the train no matter how hard it gets and that was that was an extremely important part the other part the local commits was actually really difficult too if you wanted to land a a local patch you had to go through essentially more testing than if you just landed it upstream and that was that thing we were pushing our developers just go work upstream because then they can just land a patch upstream and eventually it appears in the cloud if they wanted to work local they had to run through all the same tests for their patch so there was no benefit to landing things local other than if you disagreed with the upstream community you could land your code in our our code base while you disagreed with them right but we did it was also kind of set up to make it easier even in the case of you wanting to do something that you might not be accepted by the upstream community it was still easier to propose that patch upstream and get it through all of the not not even get it accepted but just have that proposal sitting out there and then tell the system okay carry this patch like basically cherry pick that out and then if it is accepted upstream there isn't anything that we have to do later on we just know we have we don't have to cherry pick that particular patch later on you know do one more incentive for the developers to just start upstream because it's going to be a lot more work to to not and so we were using all the same those of you who are familiar with open stack at least your these are all things you've heard of so who's playing as a patch in open stack in here anybody awesome it's a pretty it's a pretty rich tool set but it's not github so for many that's their open source experience and anything that's different from that seems alien and weird and open stack doesn't use any of that yeah so the first step is like accept garret and definitely inside IBM we were like what's garret it's a code review system yeah we talk about yeah talk about code review a little bit later but in just in terms of doing collaborative distributed code review garret I think is probably one of the best systems out there for large scale and again comes kind of an Android development actually built it up so it has a pretty long history but it's one that maybe the rest of the open source world hasn't seen until now it's actually getting a lot of adoption now it is so then the other some of the other things that we use to put this together as Clint was talking about Zool and we use Jenkins and we were using Ansible partly it was another for us an efficiency to start off with blue box some of the tooling that they had developed to deploy their private clouds which were open stack clouds and IBM had acquired blue box six months or a year maybe before we started this and so we they had a really solid set of playbooks and I think it's called Ursula Ursula and so we started with that we forked it unfortunately we had to because our release cycles were different but they immediately started merging things back from us because one other benefit of doing things for master was that our deployment code was was ahead enough that when they wanted to do a stable release which makes more sense from a private cloud they were able to basically just package up ours put a bow on it to make a release from that and then there was one last tool that we that we wrote I mean Greg Haynes actually was the yep primary author on that was was upstream sync which was the thing that kept us in sync with upstream and and could bring together our local patches and anything we needed a cherry pick and I tried to get a little bit of magic upstream funk but you said no so one of the issues was the popularity of our project it turned out that we we felt we had a stretch of time to work out the kinks and we thought we'd show it to some people and let them kick the tires and they loved it and it got like they really really liked it and we hadn't intended this we were kind of said well here's this thing we're working on we think it's cool you know take a look and we'll talk to you about it some other time and then suddenly we started getting new committers and people actually starting to use the system to pull in other open stack projects that we hadn't intended or expected to carry and we were getting questions and some people thought that this was a production ready system that it was just good to go it wasn't and in that story the we in that case was between seven and 10 of us on a single team they on that grew from anywhere from 10 people looking at it up to about 90 at one point so you know that you know in the an influx of interest too early in a project maturity can be a challenge and this through that we and through this the kind of request for help and people who didn't quite understand how to use it we understood some of the things that we were going to have to solve some of the training some of the things we'd have to make easy for people to consume and these knowledge gaps so there was there was a learning curve with Garrett so just like we talked about actually you I'm sure have used Garrett anyone has anyone else actually used Garrett like to do either a code review okay it's it's got a learning curve especially if you're really used to like a get hub workflow it's there's some changes and there were plenty of people in that were that were starting to kick the tires and had no idea what to do with it which included things like patches that were 4,000 lines of code which aren't really Garrett's not meant for that and it's not not the right system really and the human brain isn't meant for that no it's really not yeah but we you know we saw these really giant patches coming in and then in a friendly way with these developers you know like I said we weren't quite ready for people to be using this anyway and we thought we expressed that and then they started getting a little bit angry like hey why aren't you merging it this is get it in there all right my boss told me to deploy this this week like what I can't even read it yeah we discovered that that we had taken a very very dev ops centric approach to all of this because we're kind of all believers and it turns out not a lot of people understand but what dev ops is and you understand okay well yeah I understand at least one definition of it yeah we had an idea what it was certainly that wasn't appearing in in the knowledge of the teams who were working with yeah and then like around Garrett the the idea of collaborative code review of small patches is also a foreign idea to some people in many large organizations so there were entire teams that that really still the way they did this with tons and tons of code and then they would get together for a week in a conference room every six weeks or two months and and review each other's code and anybody done that like the post code review good don't do that so that meant we needed to come up with a solution to this and we decided it was going to be pirate school so we would go to these different teams and teach them about the new tools the code review how they test the gating test worked and really really sell them on community engagement and fixing upstream first and our our our boss she was the one who said this I actually thought this was you but she she really kept hammering the point that people in process process matter more than the tool and so at the heart of everything we were doing was really to bring all of these disparate teams together and get them working together on the same code base definitely and when we say you know people in process matter more than tools I think he he thinks that I said it because I said it I repeated it more many many times he's often you know we're engineers and we want to talk about tools you know you know that's more power for our system but the reality was like we needed more people power and we weren't getting it and because there was just there was too much of a gap that there was such a steep cliff to climb to to join us in this sort of dev ops adventure that we weren't even able to take advantage even at one time we had you know 90 people available to us a very small percentage who were already able to to collaborate so we needed to to do something about it so that meant some of us were taking a road trip we had a couple offices the teams that were working on this were spread literally all around the world and we came up with a relatively simple curriculum that was very was not what any of the teams expected in that it was extremely collaborative we we did one pass where we had one day of slides or actually a lot of it was very slide driven so we'd come up with our outline and everything we wanted to hit and pretty quickly realized that just this wasn't going to do what we wanted so we we just didn't have slides we had a simple outline and it was whiteboards and talking to people and then that was some of this stuff and then the rest of it was just actually having them do their first commit to OpenStack and walk through some of the OpenStack upstream upstream training and some of the other things that we did and the reason for that map by the way is the face to face aspect right we also made a choice after after having a lot of phone calls you know IBM is a big company and people spread all over the world and after having a few phone phone calls we realized it's not that they're not smart enough these were absolutely some of the smartest engineers in the company it's that we don't have any way to relate on these very abstract topics let's get in a room let's get in a room let's get whiteboards up let's get this done and get a deep dive in and that definitely made all the difference which is why there's a map there's also this is the shirt from the tour you can see we went to several different cities and this is actually poor Jamie Lennox is from Sydney he's on our team so he's most of these big lines yeah but we sent people all over making sure that we got the right people in front of the right students basically right and then one of the biggest big drivers was to build an internal community and get everyone working together so there was some of the things that they loved or more yeah getting people to understand what practical devops meant at least from our for our definition which which mainly amounts to config his code and code his conversation getting people to recognize that well the and actually setting up from the very beginning the only way to deploy anything was through a commit you we weren't allowing there really wasn't any way for you to SSA tend to any of these boxes to stick something if something was wrong the only way you could fix it was go back to the code base commit something get it reviewed get someone else to merge it and then the system will pick it up and deploy it and fix it up and on staging or even on production and that so I to me what is devops that's probably one of the most key concepts and that was the thing that a lot of people were very uncomfortable with especially even the ops people they are like oh no but when things break I need to jump on and SSH to all the servers no you don't you can do it a different way and a better way I call it a a ruthless devotion to the automation yeah really what it was that there is there is a way into that system I promise that we could SSH and we're not going to we're going to work together and land code so that we can remember why we did it that way so we know what we did to the system and that is part of devops but that's also just part of being able to communicate with people and making sure that the conversation happens exactly and then other components of that where we were using the exact same tooling and the exact same playbooks across everywhere so on my laptop or anyone else's laptop you could deploy a very small version of this cloud that used the exact same deployment toolbooks and all the same bits and pieces that and it was the same thing when it would hit staging and the same thing when it would hit production which gave people an awful lot of confidence and we also had a test in the gate so everything was actually tested before it would merge which gives people also a lot of confidence about what they're putting in and alleviate some of that worry yeah it's nice when you check out master and it actually works yeah that's a really nice experience this was one of the this was in I think this was Beijing and that's that's the Jamie Lennox on the foreground and Spencer Crumb in the background who was actually here and I don't think he spoke at this he ran open in for a day oh yeah yeah he did that's right he didn't open it for a day Spencer was I would say in large part the heart of our training efforts he did all the hard work and I would help him and Jamie would sit around looking wise rubbing his chin a lot we did have a structure which I don't know that we have in our slides which which was really nice where there would there would sort of Spencer was like the head trainer and then you would be sort of the administrative like make sure everybody has everything you need to be able to listen to Spencer right and then in this case Jamie was the developer sort of the knowledgeable person to go to for the deeper questions yeah right so Spencer would be explaining and when there was a question that Spencer didn't have usually the overlap of knowledge would come to to Jamie so I did that role a couple of times Greg did that sometimes we always had somebody who was who had a deep understanding of the nuts and bolts but maybe wasn't able to be at every training right and it worked out really nicely but it was I think Spencer and I did were we were the ones that were at every area of every single lab and we we had the first day was the big knowledge dump lots of whiteboarding all the big topics getting people ready and then the second day was all lab day and that was the day that I mostly ran spending the time with everyone making sure that their systems were good that they understood how to use all the tools and mostly force them to actually walk through and use everything while we were there so that they when we left they all could launch the environment on their laptop could check code in to change it and and understand what the implications and the ramifications were of any of the things that were doing and one of the one of the favorite things that that I loved and saw Spencer this was the thing that I think had the most impact that I would see Spencer do was he would teach the levels of DevOps and this was his the way he did it far better than this one slide this doesn't even doesn't even slightly vaguely do it justice but to put it really simply he'd start with the whiteboard and he'd he'd kind of show okay your sysadmin and he would draw a stick figure and you're taking care of a computer and he would draw the computer and he would he would talk about you know your first level you'll learn to use configuration management for repeatable deploys and then oh what if you took that config management and put a crown on that so anytime you change that config it just gets automatically deployed every 15 minutes or something and what if you put all of that in in git and you you trigger that deployment when there's an update and then the last one is to collaborating the code review so he would this was he would spend 20 minutes talking carefully about this and and you'd be able to look around the room and see the light bulbs going off and the people would actually get it maybe oh my god this is this is so much better than the way we're doing it right now and and I mean like every single room every single office that we went to and every single team that we taught this changed the way they worked and got like so much more productive and collaborated so much better and I think this was one of the things that probably made the biggest difference yeah and I think the the lesson from that you know pulling back from the details was having a unified plan across these teams also helped them communicate with each other without us which increased the amount of collaboration in general so we all were just speaking the same language we all understood even though DevOps is somewhat poorly and you know morphously understood on our teams it was this it was this here and you all had Spencer show it to you so you could refer back to those concepts and pretty much know that you were saying the same thing had a huge impact on communication efficiency and like I was saying the big key was building communities and that involved officer in code review getting everyone who was working on OpenSAC to actually be on the OpenSAC mailing list and recognize that that's where the asynchronous communication happens and that you can if you tune in and read or just skim some of that stuff you'll actually have an ongoing sense of what's happening with OpenSAC and not be surprised when something you were relying on goes away or there's some big the gates on fire or whatever and then also IRC and Slack are really good for that and we were just kind of hammering all the time to these teams that we're in this together because unfortunately a lot of these teams were set up in a way that they just had a narrow responsibility and they were a little bit scared because they've been told well you're in charge of this slice of OpenSAC go make it better and some of these people didn't have any OpenSAC experience prior to this and they kind of felt like they were being thrown to the world and what we were trying to tell them and make really clear to all of them was that we're actually with every other team that's working on this stuff and the pirate team included we're all in this together and that you've got a whole group of people that you can reach out to anytime you need help and you can kind of see that release as they as they realize that we meant it at one team just one team so when it came to testing and deploying I talked about the local deployments were the same so we would use Ansible to deploy onto Vagrant and for staging it was Ansible onto Ironic which would then get out onto the bare metal and the production deployments were exactly the same Ansible with Ironic on the bare metal and across all of these it was the exact same playbooks and the exact same test so there was no like oh well it worked on my laptop why didn't it work when we got to staging it worked in DevStack for rare cases right we didn't have viadas or NetApps on our laptops but other than that everything was the same to be fair the idea was to narrow down the differences to just what's actually different from staging so that most people could debug things in that nice tight feedback loop of actually being on your local machine and so the end result that we saw for the developers where we had huge converts to Garrett people really started to appreciate appreciate reviewing code that way and sharing code with their co-workers that way and they like I said they got DevOps they really appreciated CICD and everyone loves dual I mean everyone really like people really really appreciated that well Clinton is going to talk a little bit more to you know we'll talk a little bit about it but yeah and then and it's getting getting everyone to be on the same page in one big team so here's one of the pirate teams one of the things that we did was we would bring a pirate flag to every office when we were teaching them and we would give them I present them with a pirate flag kind of as a sort of graduation and they loved it unfortunately I couldn't find any other pictures so I only have two from when we were in China which was probably the most fun trip but yeah it was one of the things that also kind of gave them the sense that hey they're they're part of they're part of the broader pirate team yeah it was like a rite of passage when your team like took the pirate flag picture it's like now we speak pirate ease so you can talk to us about DevOps levels and we'll understand what you're saying exactly there was a lot of there were quite a few of us there was a lot of pirate talk yeah I think we had a yard arm so we we we absolutely saw improved developer velocity no question once once the teams were up to speed and understood the way we were working everyone was much more integrated and they were able to move way faster and they were collaborating so much better and we had we had really very effectively broken down these kind of artificial barriers from these teams being geographically distributed and not necessarily realizing that we're all trying to solve the same problem because they were also on in different organizations within the company and there were fewer bugs and people really started to appreciate open source so we had a bunch of people that we talked to and that we trained that hadn't really thought about open source software they were some people had been IBMers their whole life and they mostly just thought about enterprise stuff and they didn't have a great appreciation for what you can do with contributing back to open source software and how important it is to be a part of that community of you know any any of the software they were consuming and there was a follow on thing that would happen had nothing to do with pirate cloud or even open stack which was that we had a number of contributors from every level like you said people who had been in IBM since they were in college up to people who had just started at IBM the week before who would come up and say you know I've always heard open source I've seen open source I've even used open source but I never got that I can actually collaborate with other people and get something real done and that there was that I had at least four or five people come to me and say the almost exact same thing which is this this was inspiring on a level that didn't matter about my day job there's actually people out there collaborating on good things and it sort of gave them a morale boost to be involved with that so this is your spy this is my my slide so another thing that was an end result for IBM is we had all those positive reactions to to pirate school and people worked once they had pirate school and once they saw the the song and dance from that they actually worked on pirate cloud for several months and we did some retrospectives we asked what did you like what did you love and there was a massive amount of people who said I I really loved working on this CI and CD system so basically Garrett Zule and and that level of testing they hadn't had that experience so the so we kind of thought if we could forklift that that chunk of the dev ops experience into something more generic we might be able to do some good not only for IBM but for the open source rule so it's it's a poorly understood idea but it's become my job to explain it to people but Zule enables much higher throughput of gated code so has anybody contributed to a thing where you have to pass the test before you land land right and there's there's one universal problem with that which is that no matter what if you have to pass the test however long the test take is how long each patch has to wait before it can land and if you have 10, 15, 20 patches waiting you have two choices you can either land all of them at once and hope that they work together those inflight patches or you can serialize them and now if you have an hour long test suite you have 10 hours to land a patch which is not fun so what OpenStack did was actually make a system that parallelizes that and we used that to to great effect during the pirate cloud in that we could now be couple approving patches and landing patches and testing patches so they all happen as fast as they can and we use the cloud to do that so we run parallelized testing to an extreme level even in gating I would be happy to explain that more in depth when we get to Q and A but I want to make sure we let people get to the end we also cross repo dependencies what that means is that if I'm working in project A and I realize there's a bug in project B that prevents my feature from landing normally what I have to do is go lamb the fix and B wait for the fix to release wait for it to get available and then it gets shipped with cross repo dependencies I can say whenever that lands land mine too and so now I get to like build the future I want and wait for it to happen so those two features are actually in Zool which is basically a replacement for Jenkins and a pipeline manager for CI and then Bonnie CI is Zool as a service for any GitHub project we're in super closed beta right now in fact our first closed beta user came online the other day sort of and reported lots of bugs so thank you to that intrepid person and but we're definitely wanting to have the conversation so if you think that sounds interesting and you have a project on GitHub where you want to be able to gate code and you want to be able to have cross repo dependencies with other projects totally go to Bonnie CI.org or just come find me and say hello and we can play with that or if you're familiar with IRC get on free note all right there's a pound Bonnie CI channel as well and you can you can do that also we have a GitHub org which is linked from the website Bonnie CI and you can see all of our deployment code because we're we're all people who work on open info projects so we are an open info project so you can go see all of our insibles and you can help us improve it you can even use it to deploy your own Zool and also possibly we might use be able to take your project if you're on Garrett Hub but that's we'll have a conversation if you're actually on Garrett Hub because we just love Garrett so we end this by encouraging everyone to do some of the same things that we did within our organization and hopefully affect some change there so you know educate top down demonstrating the value of open source and so open processes and you don't have to use the exact same tooling as long as you're doing some sort of collaborative code review and CI CD with tests any questions question was did we only do this in IBM offices or did we go around to outside offices companies outside IBM no this was only internal this was IBM what we were doing was bringing in in some sense the same upstream training that OpenStack foundation does at every OpenStack summit we're bringing that into IBM and showing basically showing the IBMers how to use the tools and the systems that were inside IBM there was another question so I'll repeat the question in case this is actually being recorded but the question was how did things work with the different orders of services and things that had to go yeah like the answer is terribly OpenStack is super complicated it's a large-scale distributed system and even though you would think that there was one agreed upon way to do anything there's 50 we absolutely tested every change including changes to the ordering and the upgrade code we didn't get great at doing certain types of updates and we would have to sort of manually hold those back so that that graph I showed where we were doing 10,000 lines some days in 40,000 there were times where we basically had our test staging deployment was really picky about if it changes the database it explodes and we would just have to manually go look at that and say how is that going to go is that are we going to be able to just let that run or is that going to take 17 hours in production and we can't do that we have to schedule downtime we never actually got a lot of users on the cloud while we were doing this so we never had to make the decision to manually hold one back more than just okay that's fine go ahead and land the change that allows that version to go forward but it is very complicated to keep an open stack system up and live and running while you're changing the code a lot we also landed a lot of patches upstream and open stack to make that work better the question was best point pointers and best case scenarios for upgrading open stack and best practices for it I mean the first question is what distribution because the truth is it's different with every distro so using like charms and juju to deploy it or the a bunch of packages and deploying it with your own deployment tooling grades manual sorry I did not want to say that today I try not say it so your problem is that you have nine million lines of code to change and you're not sure which ones actually change your deployment which is exactly the thing cd helps with because at least you only do 10,000 at a time every day but some works aren't built to have cd and it doesn't help you today it doesn't help you today so we're not good at talking about stable release updates because our whole being is to try and avoid to ever have to do one so I don't have a good answer for you yeah right so the question is how do we handle persistent data during upgrades so it's important to know we were deploying on about a thousand servers with their own local storage and we were not the planes with so we did have cinder and that was on stored on netapps that were out of our control actually we're basically just given an nfs mount like put put your volumes here so we didn't even have that in our in our sphere I bet I get the other problem too is that you were moving your the code you had control forward your cinder just kind of falling behind no no we still the code was always updated right the cinder code yeah the pursuit sorry right yeah and so netapp was just just as far as we're concerned just a disc that cinder was in front of and then the discs on an individual compute node are are what they are which is important user data but not related to the code in most cases we one time had an update to chem you which we are qmu and had to look and see if it would change any of that and it didn't so we just rolled forward but everything we're talking about was code rolling forward the only data that we really had to worry about was actually the control plane data the user data essentially like open stack exists to kind of make your VMs run and you know where they are and where the data is so it does that pretty well and we didn't do any live migration we never had to to go through a reboot of compute nodes the plan was to essentially just have downtime windows tell users your your VMs going to reboot tomorrow sorry those were those would be control plane windows only and our downtime windows only and we actually never had to open a downtime window for those because everything we were updating was basically rolling updatable yeah nothing in fact to the VMs yes so we were using the question is however we're using ansible to talk to ironic in our build step because we we did say we're using that so there's actually a project called bifrost or beef roast if you're Swedish I like beef roast actually to some bifrost I think and that is actually a project where the people developing ironic notice that it would actually be useful without the rest of open stack in bootstrappy instances so it's like ironic is very good at talking to all kinds of bare metal servers and you don't necessarily need image management and off management and volumes and network SDN you might just need something that's really good basically a cobbler replacement if they've ever used a cobbler or razor anything like that so we were using it we had a set of ansible playbooks called bifrost that just use the ironic API to deploy bare metal nodes with basically just a base image and then we would use ansible to layer on top of that IP mic seems like that's it thank you everybody and thanks for coming to scale yeah