Well, hello and everyone welcome to this sort of second open shift on OpenStack SIG meeting the first one was part of a briefing and Judd Matlin was the one who spoke at that one He gave us a wonderful overview of the reference architecture from Dell of OpenStack And we thought because it was such an interesting that we would definitely try and and host sort of bi-monthly opens open shift on OpenStack SIGs and Today we have Judd with us and Jeremy Eater Who wrote a wonderful blog post about his work testing OpenShift on OpenStack on the CNCF lab Which is the cloud native computing foundation and the CNCF has a cluster which is a wonderful thing that was donated by Intel and is I think a supernap or switch one of those places and It's been a great test bed for us for doing some scaling testing So we thought that would be a good way to open up the conversation since this is a SIG and not a briefing of sorts and Jeremy is kindly offered to do this talk We do have a chat for this so raise your hand in the chat if you want to ask questions I'm gonna unmute everybody. You can self mute if you have dogs and cats and kids and heavy equipment in the back room But then just raise your hand and ask a question and then at the end of Jeremy's talk We'll open it up for a conversation about what this SIG means to you and where we want to go with it All right, so I'm gonna turn let Judd introduce himself and then we'll kick it off Thanks a lot Diane. My name is Judd Moulton I'm a principal systems engineer at Dell and My job these days is to write a reference architecture and deployment guide for deploying OpenShift 3.3 on the latest OpenStack platform from Red Hat and address all the issues and hopes and dreams of folks delivering OpenShift on top of any stack of OpenShift on top of OpenStack in whatever vendor configuration you like and act as a an emissary into OpenStack in order to promote features that are most helpful to OpenShift Kubernetes and that whole infrastructure We As the chairman of the SIG I'm really interested to hear from you all about what you might need from OpenStack In order to accomplish what You need to do on OpenShift And also what you might want to talk about successes failures war stories to share with the group and get more Knowledge about this popular stack Um, we have the gathering coming up right before Kubecon in Seattle in early November looking forward to seeing a whole bunch of folks there and learning and teaching tons about what we're doing the My work is available through unfortunately your Dell sales rep And I'm also very proud and surprised that Intel bought the thousand nodes of of Dell gear Dell R6 30s, which are featured in my reference architecture for OpenShift So we can actually go pretty deep into the configuration of these boxes So you really know what to order when you're looking at deploying this stuff That's it I guess without further ado Again, ping me if there's stuff you want to talk about and collaborate about I Don I'd like to thank Jeremy for for going over this and I'm excited to to dig into Into what they accomplished at the CNCF lab Jeremy. Yeah. Hi. Hi everybody. My name is Jeremy. I'm engineer in the performance and scale engineering team at Red Hat and These past couple years. I've been working on OpenShift container technology in general and In the month of June or May We became aware of the CNCF environment that it was going to be become available And we had a product need to get some additional scale testing done and here we are So we were able to use that lab and I'll go over some of the details about how we used it and why we did things in a certain way and And Some of the results, you know, which are Which are I guess the most interesting part and for me, I guess the journey was equally interesting Lot of work came out of it and there's more work to go, but and we're actually right now gearing up for An additional scale test, you know, we're gonna do these as often as we possibly can So we're gearing up to do newer revise this type of a test and Hopefully achieve even greater numbers. So Hopefully in front of you, you'll see a red slide Yep Okay What I I guess I figured maybe it would be interesting to take you guys through how we turn requirements into results from our perspective And that would start with The lofty kind of marching orders from the product managers saying, you know, here's what we need to do Or here's what we'd like you to try and get to and you tell us how realistic this is and go try it anyway How CNCF got involved I want to share with you the deployment of the open shift on open stack at scale that we did which we learned a lot from and Is feeding into our product documentation and our own reference architectures along these lines I also wanted to take you into some of the details about how how we actually do our testing Because the results and the numbers have less have more meaning that way if you understand what how we are actually exercising the environment Some results. These are the ones that are shared in the blog So if you've read the blog already, you'll have seen some of these results And then hopefully some Q&A and as I'm going through this, I didn't prepare a terrible amount, you know But so so please do feel free to ask questions. Diane. I cannot see the chat while I'm looking at the slides So just, you know interrupt me if there's questions and If I'm going off in the wrong direction just refocus me here. Okay, so What did we set out to do? We get a product requirements document from the product managers in a release planning phase in advance of code freeze for open shift 3 3, which I guess was A couple of months ago now and Our goals were pretty lofty Along the way, we've been getting more and more information and feedback from the field that Density and consolidation is a very important attribute for the system So to that end we decided to try and up the number of pods that can be supported per node Open shift provides an elastic search fluent D and Kavana based stack for consolidated logging We wanted to exercise that at as big a scale as we can It also provides a Hockular, Heapster and Cassandra based metrics Stack that runs again all actually logging in metrics all run as pods on top of open shift So we wanted to test that and then we also had a like a full density kind of it's more of a control plane exercise at full scale at as dense we as we could get so Let me just say one thing before I continue The support statements that come out of open shift and out of kubernetes Are generally around how many pods you can run And how many pods you can run at a certain number of nodes? So it's it's a little bit difficult to grok, but the the idea is that fewer pods you can have fewer pods on on Nodes the more nodes you have So at full scale thousand nodes, we're not be we're not doing 250 pods we're doing half of that And so if you needed to get to 250 pods on a node, you would only be able to have 500 nodes In other words 125,000 pods currently is the maximum for open shift 3 3 So the idea was okay given these type of marketing marching orders.  How are we where are we going to do it? How are we going to do it? And we already have the why so from there we we have previous relationship with chris danzig from cncf and And dan cone. So we discussed our use of their lab Extensively and what ended up happening was We were allocated 300 of those physical nodes So everyone was bursting through doors and windows like the kool-aid man saying okay. We got this kick-ass gear It's super high-end stuff. We really can't even get higher end stuff even if we had infinite budget Because you know every system has an nbme device for example, there's just 14 000 plus cores that were available here really really awesome gear del gear and Plenty of ram basically we wanted for nothing in terms of infrastructure here 10. Well, I will say this The 10 gig nicks we probably could have used a few more of those but anyway, so it That's the hardware stack del power. It's our 6 30s With uh with haswell chips and 256 gigs of ram so really great gear for being able to do the Try and hit the targets that we are trying to get to so in other words. We didn't have any hardware limitations We were able to focus ourselves on the software which was which is really untying our hands in a lot of cases When we try and do scale testing sometimes it's a mix of well We don't have the hardware to do that or well We don't have the software to do that in this scenario the hardware question was answered for us. Thankfully by cncf So here we are. That's the left hand side of the page Right bottom quadrant talks about the versions Before we get to the software I'm curious if you know what sort of northbound Networking gear is involved for our del r. A. We have a full cross linked ha switch infrastructure both for northbound and east westbound across racks our solution In our r. A. Goes up to three full racks. This is a lot more than three full racks But you could still get a lot of processing power. So what what do you know? Who is supporting supplying the networking gear? You know the lab was turned over to us as as machines. So I don't I don't I know we had uh Problems at the network switch level with dropped packets that we were working on for a while that that ended up getting resolved We also had driver bugs in the intel I-40e driver That made it look like there might be switching problems when there weren't so I don't actually know what the uplink switch models where I believe they were I believe they were Cisco, but I could be wrong so uh This on the software side Again, this is one of those things where you know the next time we go through the cncf work One of the main requests we had was can we have read-only access to the switches? For example And they were not set up in a multi-tenant fashion that would allow that in a way that the ops guys over there were very comfortable with so One of the ideas was to dump the switch configs for us every day and and you know Process them such that they we only see what we're what we're supposed to see Or or even do it more often than that so we could have at least a tech space output.  Do we have a link? We actually had a bunch of link errors, you know Just driver bugs on you know that were related to even establishing link with the switches So those are things that were difficult to debug without access to the switching Infrastructure and this is just the nature of the way this lab was given to us at the time so part of our feedback to them has been We would like read-only access or something similar Around the switching as well as you know, we would love to have like net flow type data from the switching to understand if we're hitting across connect link contention or You know bandwidth issues along those lines that we we never did stress the The networking plane to the point where we're pushing 10 gigs of traffic But from a functional functional standpoint, it might have speeded up our debugging That's very lab specific and really didn't have much to do with open shift and open stack, but You know since you asked Okay, so we did deploy with rel 7 2 We had to upgrade to rel 7 3 development kernel in order to pick up the driver fixes that I had mentioned earlier The the bugs that were fixed upstream that are that weren't fixed at the code level we had at the time They're they're fixed at this point And then open stack Red Hat open stack platform 8, which is based on liberty if you're familiar with their naming scheme A bunch of patches that are again publicly released Arata and we used open shift 3 3 which was an alpha alpha stage at the time these tests were run. So That's the mix of hardware and software resources that we used now For private cloud, this is probably a fairly standard config Maybe it's a little bit larger. Maybe the nodes are a little bit nicer than than many people can afford, but Um Galing those down, you know, I would think this is a pretty happy path for for customers who are doing private cloud at this point There were a lot of there were a lot of things that were more nice to have than need to have in this environment The nvme devices go in that category But they were certainly we certainly put them to good use once we figured out, you know, how to wire those into open stack Okay, so any questions about the hardware software configs? Um, nope looks good to me so The logical diagram that we built is on the right And I hopefully you can see that there were the the colors are blue is a physical node and green is a virtual node And what we did was we created two different Host aggregates which are essentially like regions or zones And in each one of those regions we deployed a separate open shift installation And the reason we did that was was only because we had more gear than We needed to get to a thousand nodes And we wanted to run Different tests that would have otherwise interfered with each other open shift tests So we were able to parallelize a bunch of stuff by doing that and that's why you've got the curly braces one comma two there There were actually two mirror images of these sharing the same Open stack control plane And so you've got those for the infra Uh, I have a host aggregate zones which are open shift has the concept of masters or your api servers And scd nodes which are key value stores that that store all the persistent state cluster state and the masters talk with the scds directly And then there's also actually not on this diagram But there's a load balancer in front of those masters which is a single endpoint that all the nodes talk with So that's just another vm And we did end up with only three this diagram says it goes up to you know master And there was actually three masters and three scds And then there was a support of the availability zone or or host aggregate And in there we stuck the routers registries metrics and logging pods that were in use so The third part was a catch all which was everything else In our case it ended up being just nodes And it was a fourth actually where we stuck a bunch of servers We had in reserve where we ran our workload generators from and those are just bare metal rel nodes That and our workload generator currently uses jmeter. So it generates web traffic Some of the flavors we used are at the bottom of the diagram here All of our nodes were four vcpus 16 gigs of ram and 32 gigs of disk And those were Probably larger than we needed but it's kind of the thing where we had a bunch of hardware. We figured we would use all of it And then the the xcd servers and infra nodes those definitely need to be beefier There's a depending on the type of test or workload that's going on You generally need more cpus to handle the metrics and logging and the router and registry depending on if you're doing web traffic You know the routers would get busy on cpus The registries do a lot of cpu activity, but mostly disk activity and so the nodes tend to be Infra and fcd nodes tend to be need a little bit more cpu resources And then finally the most the most cpu intensive ones are the masters and those are the ones that are constantly busy Doing command and control traffic with the nodes Part of our effort in this time frame was to improve those command and control Channels and we've done that in a couple of ways. First of all, we've put in some some caches And we've done we've added some xcd washes If you're not familiar with those a watch is a persistent connection that a node will establish on the master That allows it to or sorry. Yeah via the master that allows it to be constantly updated with Status changes from the cluster So a node will immediately know whether or not a new node became available and it knows how to route packets to it because It has it has received that update So those watches are a current point of command and control that that has gone under optimization and and the main thing we've done is move from JSON based Packet payloads that encapsulate all of the data About state over to protocol buffer, which is binary encoded and thus much more cpu efficient to encode and decode than json is I basically we found that golang's json library was Or could use some optimization and and so the google team has invented protocol buffers Which though they're in use in more than kubernetes, but we applying them to kubernetes allowed us to reduce the cpu usage I believe from 22 cores Down to 15 cores or so so maybe a 30 or 40 percent Improvement in the number of cores it takes to manage a thousand nodes a tremendous improvement between the caches and the Protocol buffers so phenomenal scalability improvement there So for a while to get to these numbers we were just throwing more hardware at it making that's why the masters are 40 bcp Us at a point we were up to 22 You know and we really felt like we didn't want to be bottlenecked on cpu So we threw as much hardware as we could at it and at this point what we're doing is whittling down those numbers to Um to allow the masters to be just be the overall efficiency in the control plane is is what we're trying to address And in order to do all of that not only did we Involve all of the special interest groups in in kubernetes land likes particularly sig scale but also Open shift engineering quality engineering inside red hat the open stack engineering group our group the performance team and then Analogous to the teams that that Judd is likely on we have reference architecture teams, which are writing documents that Represent these best practices that we learned and finally there was a funny bit around ansible Which i'll get into in a minute that we identified some some issues with a Some scalability issues in in their code base and if you haven't seen or used open shift yet The installer for open shift is written in ansible Before you go on um i was curious about um the load balancers not pictured here um What was your load balancing solution and did you have two with a vip? Did you opt for open stacks built-in load balancer? How'd you go about it? Yeah, the so load balancer is a service and open stack is is not yet actually supported by red hat And so what's used in this installation is the native Load balancer, which is h8 proxy base which comes with open shift whenever you install Open ship with more than One master you will automatically get and you have to configure, you know a load balancer node You will get um a single load balancer with Uh with you know an ip address so there is not an uh fault tolerant config there um In our environment here So i'd have to double check i'm i'm fairly certain that that that is a tunable option to have multiple software load balancers And it's not so much for balancing a tremendous amount of load as it is for Providing a single endpoint for all of the nodes to access rather than doing maybe round robin dns or some other thing to To spread that load out But the idea is that a single master can't handle the amount of traffic that a thousand nodes generates So you have to spread it out somehow and the proxy is the load balancer does fans that out for us And then just an equal weight scenario Thank you So we have the environment let's say we have that environment built out What are some of the things that we wanted to test? our team is uh again, we're on the performance team and we collaborate with our quality assurance guys to Develop tests for open shift and um So our repository is linked here. It's called svt and I didn't name it So I don't know why it's called that but other than the fact that it stands for system verification and test Um, so those are open source. You can grab them right now and look at what we do There's four main things that are in there at least from my point of view There's actually a fair amount more but the main and interesting ones for this For uh, the subject matter we're going over are a cluster loader network tests workload generator And then we've got some reliability tests, which we're we look for memory leaks and connections that don't close properly. Um There's a fair amount of uh tcp connections HTTP connections in the system. We want to make sure that we are not leaving sockets open and You know that would potentially build off to the point where there's you've got that type of exhaustion and memory leaks, etc So we've got some reliability tests that run for a generally run for anywhere between 10 and 14 days I didn't mention it here because it's Uh in that list just because it wouldn't fit on the page But we also a key piece of what we do involves this image provisioner And what this is is it takes a rel cloud image, which is a very minimal q-cow image or an amazon am i Or any other public cloud and it turns it into an open shift node that's got no configuration on it In terms of open shift, so it'll set up the partitions make sure that the docker storage is set up correctly Bakes in you know a lot of I mean this tech this type of image gold image techniques been around forever This is just automating it through ansible and wiring in the open shift specific bits Which are really just pre pulling a lot of the images that are required to run open shift So if you've seen it before there's like ose pod ose router ose Itself and you know, there's maybe a dozen of them that we need to pre pull onto each node The reason we do that is because if you're doing thousands of nodes or a thousand nodes going over the network Becomes a point of contention to pull all of that stuff and so Baking it into the images became a necessity actually in order to get anything done In the time frames that we had so what we did is we pushed this up Into github you can use it yourself right now and It's you know as long as you configure an inventory file for your environment It will spit out a q-cow 2 or it will also spit out an amazon am i that has all of this stuff Baked in so A means of self-defense really to get work done. Uh, you don't necessarily need to do this, but it will take you a lot longer If you if you didn't do something like this this takes the world off of your Your rpm repos and your docker registry correct Those are the primary two things that you get yeah And we did some stuff like we put our monitoring solution in there We have a metric system and monitoring solution called p bench. We wanted to bake that in What that does is gather system metrics and provide graphs for us at the system level so that we can understand You know when we run these tests, what is the impact and efficiency? I mentioned you know dropping from 22 to 15 cores That was uh, we measure that with the p bench utility So yeah, there's the main two wins from a shared infrastructure standpoint or the registry and the yum repos or the yeah Okay, so now we've got open stack open shift on top and we've got our uh And we've got our images deployed the cluster is built at this. Let's assume the cluster is built at this point Again, what do we do with it? We designed some tests that are centered around a shared web hosting scenario Which is kind of the bread and butter for kubernetes and open shift at this phase in its life We expect that to change drastically over the next year or two, but right now Uh, it's centered around 12 factor application development And web based, you know, let web languages like mongo or postgres and Node and php or whatever else your favorite thing is Open shift itself ships examples, which are templates and we've used those and adjusted them to our needs So what we've developed also developed a utility called the cluster loader And it shows up in the blog what it is is a way to express an environment that consists of all the complexities of uh of kubernetes like Represent really what a customer is trying to do or a user is going to do it'll include replication controllers secrets Uh templates pods services routes all that stuff and Allow you to express it in a very human friendly way and have this script go off and Populate the environment according to your according to your wishes So the link is there and um if you were to explore it, you'd find directories like we've got uh the basic pod Manifest we've got the basic replication controller manifest and all this stuff is templated to allow us to build as if we had a box You know as if we had a a tub of legos and we could build whatever we wanted to with these with these building blocks The Config directory contains and I'll show it to you in a slide just the guts of of what the cluster loader will go off and do And it's a python script right now and we have some plans on how to improve it As you could imagine anyway, so that's the config and content directories including the templates So you have cluster loader the python script itself, which takes only two arguments you know, um what actually one argument what file am I loading it from And optionally second arguments around using the workload generator portion of cluster loader And then of course a utilities that's got out just a bunch of functions in it. So That's the framework that we use to test Open shift The architecture is pretty simple. Um, probably the most simple flow chart you can have Start it off filter the arguments. Um create a namespace or a project in open shift parlance And within that namespace create however many things you want to represent what your applications might be doing quotas users pods etc And then at the end iterate until you've reached the total number of pods and then exit and at that point We'd have an environment that's ready for the second phase, which is Actually putting work on it. So this is the first phase and oftentimes Uh, this is the only phase that matters because potentially we could be using this as not only the loading utility But also benchmarking how quickly do we load applications when the cluster is under load? What is the api response those server response times etc under load and Additionally, how many cpu's does it take us to get this a certain fixed amount of work done? The test we ran inside cncf Is the guts of it are pretty lengthy and the actual test is on the blog, but here's an example that we Here's an example that's in our repository and the the number of projects is really all that changed here This is on the left top. You'll see in numbers only five. So it would deploy five projects each with one user With three templates inside of a build configs six build templates one image stream Two replication controller templates with each of one of those with 256 byte secrets Secrets and routes and the idea here is that if you have all of this within one Namespace or one project you have pretty much what a user or what a developer might try and deploy on open shift And you know we can bike shed about The numbers and the idea here is to provide a framework and allow you to have tools to experiment on your own These configs are what we've arrived at internally for what we expect people to do, but could be right or wrong Depending on, you know, whatever you're trying to load onto the environment. So for example Java apps Malik their memory up front. They are not They they you'll you'll get less density because there's no ability to over commit with Java. So for example, your project's number if you're using Jvm's would be lower than on the same gear, but you're writing in some interpreted language like like node or Or php where you can over commit memory So instead of five we tried to go as high as we could until failure and this is roughly what we this is Well, this is exactly what we got to 1000 nodes 13,000 projects our target was 20,000. We didn't get there in the time allotted 52,000 pods, you know, I'm not going to read the rest of it, but the numbers are pretty pretty impressive. I guess and I think We learned a lot, you know, we learned a lot about where to adjust to config what to fix and the deliverables were There were bug fixes a lot of them and a lot of it actually ended up being documentation as well For example, if you're doing logging scalability Or if you're doing a big environment and you want the you know, the the elastic search to scale Here's the three or four different, you know things you need to do for example System D has a thousand messages a second rate limit by default if you have a large environment A thousand messages a second might be nothing and so you would want to adjust those types of things and That's the type of stuff we hit at scale that a smaller environment would never encounter Or we're likely, you know likely not encounter okay, so One of the stuff we did encounter on the way to those numbers where We noticed that at cd took a more just space than we were initially recommending So we've since gone from a 20 gig recommendation up to a 40 gig recommendation Even though only 13 gig is ever consumed in our test. This is the left graph Uh around I guess midnight or so The blue line where you know, we're somewhere around 13 gigs of data and so we're recommending 40 at this point and that's probably At least twice what you would actually need for a massive cluster So and actually the scd space tends to vary based on how many images you have And how many pods you have sorry no It ends it ends up varying quite a bit around the number of images that are in your registry because we do store some metadata About the image in at cd And so a lot of the disk space is not necessarily about the number of nodes routes projects, etc As much as it is around the number of images So for the next generation test, um, you can imagine we're going to Pile on the number of images and come up with the exact capacity planning formulas around Around what you can expect there And then on the right hand side towards the right hand side of the graph you can see um a fairly busy cpus that suddenly dropped off and uh, yeah, that was a bug that we encountered and um, that's actually where we ended our test pretty much because We had just run out of time So we got that's how we got to 13 000 projects instead of 20 000. That's basically the point where things fell over And Yeah, we had to bail at that point because the gear was being taken away. So that's where we ended up with uh And that bug is fixed in kubernetes as well as openshift 3 3 at this point So if we had the gear again, we would get well beyond 13 000 projects No, and that's what we're hoping for is to get the gear again and do another run through so Yep, uh a question about eight o'clock Uh in the morning follow on the left slide the left graph Um, does the seep does the disc utilization go back down because you withdrew the number of images? Yeah, so what well not images but total content. So one thing i've Forgot to mention I guess is that the cluster loader also has a tear down phase And depending on how we've configured it. Um, the growth is attempt the growth Occurs as quickly as possible and the tear down occurs as quickly as possible But there's a steady state that it tries to uh maintain Between set up and tear down so that we can kind of examine what's going on And so the fall off is because we're trying to tear it because actually Um, another data point that was asked for was how quickly can we delete stuff or not really how quickly but Are there any issues with with project deletion speed, etc. And and to be quite honest, um We did file bugs based on that and they're um being worked. So I think that was pretty useful And that that was really the whole purpose of getting this cncf cluster access to was It was a huge opportunity for us to to do this kind of testing and It's pretty pretty awesome offer on their part and on tell us part Yeah, and sending, you know requests for this type of usage is nothing more than a pr to the cncf github repo Um, we have some plans as their environment Expands to start using it again for newer versions of our product At you know, so this gave us a lot of homework to do these tests and these results and it those are not fixed overnight. They involve coordination with upstream developers and productization on our part So it's not the type of test we could immediately repeat anyway. So it kind of gives us some think time In the interim while we're scheduling ourselves for getting back on the gear where you know, this was a lot of stuff that we learned I might have a I'll just skip ahead. We we filed Somewhere around 30 plus bugs around this This effort and a lot of them were in core kubernetes. We found one kernel bug And you know, but for the most part These are solved at this point. I think there's there's some outstanding But none that are related to scale and none that we consider blockers There's there's actually there's zero blockers that we're aware of on openshift 3 3 at this point Um, given the scale numbers that we're quoting for support. So going beyond that, you know There's dragons there most likely and uh, that's actually the the scope of our next set of scale tests Which we're working on right now That's to be the name of the next blog post there be dragons there Yeah Yeah, so The idea is to push kubernetes to its to its just you know, I mean our job is pretty cool Where we get to break stuff and and try and put the shattered pieces back together as quickly as possible I think that's kind of the Or crawling over broken glass as a service So I should say this before I continue is that you know, I'm here talking with you But there was at least six people involved in this Kim Sinclair and uh, Mike Fiedler being the primaries A couple of ansible folks that helped us along the way really a team effort overall not just definitely not just me More of the coordination point of everything so The ansible issues that we encountered where between openshift 3 2 and 3 3 We pulled in a new version of ansible because we wanted some of the features And we got those features, but we also got a performance regression and what you're looking at is actually the the uh The after the before the before was so bad that you could really not even capture Stats because it never really even completed the issue is that the is linked off there At the top of this slide and you know, it was essentially a recursion of ansible tasks that was Spiraling out of control and just ending it or just being in these infinite loops where every cpu is just pegged and the system would This system would crash essentially so That bugs resolved in ansible 2 2 which is the version that will be included with openshift 3 3 So we'll we'll be on this and you can see now actually the y-axis of this graph is not is Percent of cores so 15 k is actually 1500 Uh percent which means 15 cores This is a 16 core system for a while there and what it's doing at the spike is Actually, all it's doing is reaching out and collecting all of these Facts about a thousand nodes and so that that takes a lot of cpu to gather all of that stuff together And uh, it actually takes a lot of memory as well so This we couldn't even get to this point in the installation um with some of the other versions of ansible So those haven't seen are not supported on openshift. They they work just fine in smaller scales But when you start scaling out to the thousands, do you start seeing these issues and uh So what we've done is we've got a version of 2 2 that we're packaging with uh with openshift itself Okay, those are the bugs and I think that might be it all right Let's see a bunch of people on And i'm wondering if anybody has any questions here. Nobody asked any in Dr. Dr. basically just a knowledge and creating easy to use packaging A bunch of different layers and binaries into a format That's easy to distribute. So they did that there was a first thing they did They created a runtime and made it easier to run those containers That's And then Hey, i'm having a tough time here. I didn't see anything. So essentially you had Fendi About I'm not able to make it out. I'm sorry. You're not send deep. Um, I know Can you pipe your question in? The layers For example, let's say you want to Yeah, the microphone's not working very well. No, no system copy. Well said Yeah Send deep um your your microphone is not working correctly. Can you type your question into the chat? Or your comment? Yeah, he was breaking up pretty good there In the meantime, I'd like to learn a little bit more about your open stack deployment. Uh, Was this triple o and director running typical osp stuff? Um, and did you run into anything interesting? Yes, it was typical osp stuff things we ran into were Because we were using osp eight Everything we hit had been already been fixed and uh, so we pulled some packages back We had a a small issue with heat osp d needed A newer version slightly newer version of rabid mq not osp d but the cluster needed a slightly different version of rabid mq Because there were some skill ability issues in the versions that were That we were running so the idea was that if we were Running later versions of code we would not have hit these issues and The idea was at the time we run this eight was the latest ga Um version of open stack from red hat and so we did the best we could with the latest ga You know, we weren't necessarily about Um open stack as much on this test We we were kind of just using it as a as a cloud because we wanted to get to a thousand and we only had 300 nodes Um in future versions of open of this type of testing We're going to do a lot more stack integration. For example We've got these heat templates. We'll use those, you know And we'll use the latest version of open stack newton, which is version 10 in red hat land And the latest version of open shift. We'll use those hemp heat templates to do the deploy We'll be integrating with cinder and gluster and uh, we'll likely get rid of the double vx land encapsulation so trying to do a lot more of Benefiting from running on an open stack cloud than we did in this test in this case It was mostly about providing vms for open shift to to run on um, so yeah, so the heat issues rabid mq was a little bit of an issue for a while and then To be quite honest, we were blaming software for a lot or blaming open stack and open shift for problems that were Really driver bugs in the intel driver and that would and we actually had to flash firmware on a bunch of the intel mix as well So the reason was because we were one of the first folks to get on the cncf lab And they were extremely helpful But it didn't save us any time having to you know, having to ferret those issues out because you can imagine Without access to the switches. This was a bit of a mystery for us. So uh, took a couple of extra weeks to to to um To nail all of that down and finally got to the point where everything was super stable. Um Same for those issues. Yeah, I've really just eaten rabbit were the two issues that we uh We encountered There's one more question that popped up to um, a rash Um is asking does the whole thing work on on the rdo oo oo to try it out for a poc so we've done it on eight and um I cannot imagine there's anything in In osp that You know would preclude you to be running rdo. I really don't think so You know, I have to be honest with you. We're we're using the products here for a reason. Um The the our open shift origin on rdo is not something we're we're necessarily focusing on we're more focusing on kubernetes upstream rather than you know origin scalability because we like to fix the issues in q as soon as we possibly can And from the open stack standpoint, we did the same thing where we work everything upstream and rdo becomes an integration point So I don't I'm not aware of any restrictions around that though. So it you know mileage may vary but Hopefully you're lucky And if you do it then we'll definitely get you to talk about it Let us know what happens There anybody else out there? I know sam jeep your microphone wasn't quite working So um, it looked like he bopped off. I thought he might bop back in So I would also um while we're pausing here and people are thinking if they have any questions We're going to have our um first face-to-face Meeting of the open shift and open stack sig at the open shift gathering november 7th at kubcon in seattle um, and I typed into the chat room the link to get to Registering for the gathering so we'd love to have you come Jed will be there. I think jeremy's going to be there for part of it. Um, and we're going to Have a at lunch. We have an hour and a half set aside for all the sig meetings So if you if you're going to kubcon, please come the day before and join us The url to get there is commons dot open shift dot org slash gathering and Well, we'd love to have you there And it looks like that was all the questions we have the question I now have for you, and I think jud has for you too is um, what would people like to hear for? The next Topic for then maybe we're going to do these by monthly seven two months to do another talk On open shift on open stack or something related to open staff that we should be looking at You know talking more about the load balancers or I you know if there's a topic that's near and dear your heart sign up for the mailing list Which is on also on the commons dot open shift org site and Let us know I haven't created the mailing list yet. So, um, that is one of my tasks to do today So I will add you all to it. Um, if you sign up Um, but if there's anything else jud, is there anything near and dear to your heart that we you think we ought to cover in the next one Yeah, a couple things actually, um the productization of the open shift on open stack project Um, they're their codes really good. Um I was wondering if you guys had used That project in your work to create the heat templates necessary and drive them That was really my question. How much of the under the over cloud heat templates did you have to hack up from what? um osp is delivering you out of the box So for this test, we didn't use the heat templates for the open shift on it We didn't use the open shift on open stack heat templates that I believe he was used to deploy some of the nodes that we had so Yeah, but I would say this jud is that We have prototyped open shift 3 4 On open stack 10 with those templates We have at least five patches that we're carrying now to make that happen So it's not not bad at all and our next version of this stuff will definitely be using those templates cool because i'm especially interested for my customers in upgrade paths and having well-defined templates makes upgrade somewhat tractable and so you don't want to have to redeploy your entire Your entire network in order to upgrade through osp's and through open shifts and even through say del bios updates or driver updates Thank you so much. Yeah, I'm looking forward to uh, to meeting you again. Actually we saw each other at red hat summit. Um, and uh Um, yeah folks hit me with I with ideas for future calls We've got a lot of knowledge here in within del about Managing and scaling the open stack and open shift networks. Um, the storage is is uh, Is really strong at del considering we also merged with so we have a a wide variety of of enterprise and smb mid market Storage products that we've been testing with open stack and open shift To let you know how they're working. What kind of integration is going on see whenever I deploy I'm deploying on to Equal logic boxes for all my storage. I've got a a slightly different layout and Would like to See where folks are hitting problems or or success stories around cinder integration with different storage back ends There's a lot of interesting stuff going on in this relationship and as as red hat looks more at integrating Uh, the open stack services with open shift services. There's going to be a lot more to talk about Um, so thanks so much and I really hope to see you folks in Seattle Yeah, that would be great and thank you everybody for for joining us today Um, and we will I'll reach out to Sandeep and see if I can figure out what his question was And we will do another one of these in two months. So sometime in december We'll take a date and we'll post it on the calendar. So you can always find the upcoming events on commons.openshift.org slash briefings Um, that's where my calendar hides and we add all the sigs and all the briefings and upcoming events there so Looking forward to meeting you all in person at the gathering in seattle and also going to coup con We'll take care and we'll talk to you all soon Thanks, jeremy. Thank you


Digital zoom is amazing. It allows zoom photography without any loss of picture quality. The higher the magnification, the closer you'll get to the action. A three to five times optical zoom is a standard amount. But some high zoom cameras can even reach levels of 36 times or even 42 times optical zoom. These are ideal for the serious photographer. Or if you like to take close-up pictures of bugs in the garden.


So today we're going to talk about understanding performance of Rust programs in libraries. And we're going to talk about some Rust specific pitfalls. And I'm going to talk about some tools you can use to see the performance of your Rust programs, some of them you can even use today. It's all on your laptop. I am an engineer. I work at Boyant. We make thanks to our design team for some great transitions. We make LinkerD. LinkerD is a cloud-native service proxy. We build what's called the service mesh. And here at Boyant I work on some Rust and Scalicode. One of the things that we've built is something called LinkerD TCP, which is a TCP proxy. It's written in Rust, and it's designed to work in cloud-native environments like Kubernetes or DCOS. We have more protocols coming. Right now it's TCP, but we're adding HTTP too. Just today we announced that we would call open-source the H2 repo on GitHub. You can take a look at that. LinkerD TCP, just like LinkerD is a Apache license and free for everyone. One of the things that we've built to enable us to build LinkerD TCP is something called TACO, which is a stats library. It allows you to instrument your applications. And as a stats library, it has some basic features. It has a counter, so how many times something happens. It has timings, so how long something takes to occur, or to some bit of code to run. And then gauges, which are values at a point in time. This is also Apache licensed. I'm going to use TACO. I'm going to use one of our macro benchmarks as sort of a draw. Two of our macro benchmarks as a driver for talking about the topics that I'm here to discuss today. Here is a macro benchmark that I wrote, and I'm going to talk about what macro benchmarks are here in a second. But I wanted to give you a sense of what TACO is before we start digging into it and looking at graphs and performance and charts and that sort of thing. So here is a multi-threaded macro benchmark where, here you can see we have a timing. We measure how long it takes to do an action. We have a loop about 10 million times that we do this inside of a single program. This macro benchmark driver called multi-thread.rs. And what do we do here? We set the current iteration, we increment a counter, and this is the basic work we do. And then we do this 10 million times, and at the end we send a signal to a channel as we're done. And we're going to come back to this in just a second. So here's the outline of talk. First, we're going to talk about some causes of slowness in your programs, some rough specific pitfalls tools, and then we're going to talk about this great measurement called IPC. So in native programs, there are generally three causes of performance problems. There's memory stalls, which is talking to DRAM essentially, lock intention and CPU utilization. When we talk about memory stalls, really what we're talking about is the memory hierarchy, where these aren't exact numbers. These numbers are here to give you a sense of what the order of magnitude changes when you move up levels, when you're talking to memory. So a register takes like half a nanosecond to talk to. Your last level cache is going to be like 10 nanoseconds. So if you remember in a CPU, you've got like a L1 cache, you've got an L2 cache, and then sometimes you have an L3 cache. Each layer of the cache adds another. Is larger and slower and cheaper? Well, they're the same cost. Anyway, so again, half a nanosecond, 10 nanosecond, then talking to DRAM is going to be about 100 nanoseconds. So that's just a quick overview of the memory hierarchy to remind yourselves. Then we have lock intention, things like spin loops and blocking weights. These are things that can cause performance problems in your program, and we'll see one of these here in a moment. And then CPU utilization. The reason I talked about, I want to talk about CPU utilization last is because it generally hides a lot of the performance. Usually, a CPU utilization can hide the other two things that I talked about. They can hide memory latency through a slow instruction. You can't, it looks the same. What am I trying to say? A given instruction, you can't tell if the instruction's slower if you're talking to memory when you're looking at it, is what I'm trying to say here. And instruction can hide lock intention like have a spin loop. When I say it, I really should say a function here. Idleness is often counted as useful work, like you look at top or H top and you'll see, oh, I'm using 90% of my CPU, but that can also mean you're spending 80% of your time waiting for a RAM or disk. And I want to talk a little bit about some Rust specific pitfalls. Here's a big one, like derive copy and large structs. Copy is great. Copy makes your programs much more ergonomic. They can be a real lifesaver, but if you find yourself overusing it, you can accidentally, without meaning to, the great thing about clone is it's explicit, but copy is implicit. So if you find yourself implementing copy, you can find yourself killing your DRAM bandwidth. And the most common reason is it was small when I started, and here's an example. You can see someone might make a person struct where they just have an A of an int, they have a reference to a string, and then somebody decided, well, let's put the whole user's DNA in there. And that's 800 megabytes. It should be a reference, or maybe not there at all. And then again, we're talking about, we're talking about copy, and we can talk about clone.  The nice thing about clone is it's explicit. You'll see it in a profile. You'll see clone being there at the very top. Here's just a sort of a contrived example of going through a people vector and pushing them into another friends vector. And then something that pretty much everyone in this room is familiar with is that the default hasher is cryptographically strong, so it can be a little slow. It's a well-known trade-off to experience Rust programmers. It's really surprising to brand new Rust programmers. They maybe aren't used to this, seeing this before. The nice thing is there's lots of great alternatives for using different hashers. And there's good pages with benchmarks, Rust-specific benchmarks for these different hashers. And here's an example of using a different hash, one that's a little bit faster. Here we use an FNB hash. There's no reason to show the slow version. Everyone knows what it's like to make a hash map, so I decided to show what it's like to make a slightly faster one. And again, it depends entirely on your use. You can pick from one of the great alternatives. So something else that can bite us is using expensive arguments in places where maybe we don't expect the code to be called, like expect is a good one, like you often expect your code reasons called, expect me to expect it to work. And then, but this isn't, you can put something slow and expect, like for instance, this was pulled out of one of Eliza's libraries, ended up being in the hot path, and so this format was called 10,000, the bazillion times every second, and pulled it out, did something a little cheaper, sped up the program dramatically. Again, this isn't specific to expect, it has to do with the fact that Rust is eager, and you just have to remember about the evaluation order in the program. Last, I want to talk about pre-allocating VACs. When you look at experienced Rust programs, programs you see that they pre-allocate VACs everywhere, like they hardly ever allocate. And here's an example of where we do the Selenker DTSP, we have a struct that has a buffer size and a default size, and then here it highlighted is us making the buffer that we use to shuttle data around from inbound to outbound. And now I want to talk about some of the tools that you can use. I'm going to talk about some tools in the Mac, like instruments, Cargo Bench, Gargo Bench Compare, and also some Linux tools that we use because our primary deployment target is Linux, where we use Perf and Flamegraph, VTune, and again also Cargo Bench and Cargo Bench Compare. So Cargo Bench is a micro-benchmarking tool. I'm sure many of you use it. It's part of the standard tool set. And it's really great for what's considered sort of like the high use parts of your API you should consider writing micro-benchmarks for. Here's an example of a micro-benchmark using Cargo Bench that we have in Taco where we benchmark how long does it take to make a new counter. And sort of the output will look like this. Here's a mix in with a bunch of other benchmarks that we've written, where we tend to write up, here you can see I've highlighted 23 nanoseconds, 47 nanoseconds. And those numbers look great. Everyone would probably be happy with those numbers unless you're really paying attention. Cargo Bench Compare is something that's relatively new. It allows you to compare two Cargo Bench runs. This is really great for avoiding performance regressions.   You can do a textual comparison, so Cargo Bench does. And you can see, well, this does help or doesn't help performance. In this particular example, I borrowed this graphic directly from the Cargo Bench Compare GitHub page. You can see it doesn't really make much of a difference even though there's lots of green and red. The numbers themselves are not that different. While micro-benchmarks are really useful and you should write them, what we've found is that writing macro-benchmarks is another great, really good use of time. A macro-benchmark is taking your API and using it sort of in the same context that your users would use it. The reason that micro-benchmarks are limited in utility is that you run something a handful of times and you measure it, so you don't really know how well it's running in the context of your CPU. So in the micro-benchmark that I showed you before, we exercise the API in a loop of 10 million times and we'll talk about why we do so many iterations. So when you construct a macro-benchmark, like that example that I showed earlier, you have to remember to compile with release because you wanna have optimizations turn on. It's a mistake to profile code that's built without optimizations because you might find yourself fixing something that you don't care about, that the compiler would just fix for you by turning on the optimizations. And one challenge of this is you have to add symbols in because if you don't have symbols, you're just gonna see a bunch of address spaces in your profile.  So, again, back to constructing a macro-benchmark taco, as I mentioned earlier, has two macro-benchmarks. We have a single-threaded benchmark and we have a multi-threaded benchmark because these are the two different ways that people tend to use taco. And this, again, is the same slide I showed before of the macro-benchmark, the multi-threaded macro-benchmark. So what's going on is there's a couple of threads that are being spawned. They're calculating timings and adding them. And then we have another thread that's doing the reporting work. And then here, just a really quick segue, using instruments to look at the performance of this multi-threaded benchmark. Here I use a measure called IPC, which is instructions per cycle, which I'll talk about what that means here in a second. This is really just to give you a sense of what we're gonna talk about more here in a moment. So we have stumbled upon using IPC, I shouldn't say stumbled upon, it's a fairly commonly known metric. We use IPC as a useful empirical metric. It tells you how many instructions are completed every clock cycle. When we learn CPU architecture in school, we tend to learn it as you have a program that one instruction then runs the next instruction and then the third and so on. And that's just not how it works on model until CPUs. Like this is sort of the model that we learn. As you can see here, this again is a serial model. You run the first instruction, then the next instruction, the third instruction, and here in this diagram, you see I've broken out the other stages. I got this graphic in the next one from a great page called 90-Minute Microprocessor Guide, which you might enjoy. But serially is not really how programs run anymore. They run more like this. They run, as soon as one fetch stage is run, the next fetch stage can start. As soon as one decode stage starts, or is finished, the next can start. And you don't just have one execution unit anymore inside of a single core. Now you might have three or four or five. And even more complicated, instructions can, each one of these little boxes can depend on another box. So instead of having a full CPU full of work, you can have memory stalls, the things we were talking about before, memory stalls, locks, can put a lot of empty space inside of, inside of your pipeline. So, given how complicated things are, and since instructions can depend on each other, and we can have huge bubbles in the pipeline, how do we know, how do we know we're doing well? Here we enter the realm of the performance counter. So Intel engineers had the same question, how do we know our program is running well, given how complicated a modern CPU is? They added something called the performance monitor counters. They'll let them, just like the counters in the benchmark I showed you before, just track how often things occur. And they allow you to do things like calculate ratio. An enormous number of counters inside of a modern Intel CPU. There'd be hundreds of counters, hundreds of documents, hundreds of pages of documentation that could be really tedious to pour over. And, but the nice thing is that if you find a handful that you find useful, you can use them to derive metrics for your program. And instructions per cycle is one of those metrics that we use. Instructions per cycle is how many instructions can a core retire in a cycle? Retire meaning it's finished. The useful work is done. And what we found is a good rule of thumb is that if you have an IPC score less than one, it means your memory's solved. If you have this IPC score greater than zero or greater than one, it means your instruction's solved. You maybe need to run fewer instructions. And you can, for your machine, you can learn this empirically. Like if you have a deployment target that's like a modern CPU that has like, this five wide, you can, or one that's three wide, you can see how well your program runs. But you can also learn whether 1.0 is the right metric for you empirically by writing a program. You can write one that's memory-stalled. It does nothing but DRAM fetches and writes. You can write another one that does nothing but, you know, works entirely out of registers. And then you can see whether, you can determine whether this rule of thumb is appropriate or not. Again, reminding you just how complicated a CPU is. This being a three wide, which is what you see at the very beginning, this could have an IPC of three, as I mentioned in that slide. If you have a max IPC of three instruments on the Mac, so it has IPC built in. Again, this is going back to the earlier slide. You can, what I've chosen to do here is look at the IPC for the multi-threaded benchmark. And you can see that we have an IPC score of less than 0.5. You might think, oh, it's 50% utilized. That's great. But, you know, it's a three or four wide CPU. That's like 50% of four. That's really poor. That's like maybe a zero point. That's like maybe 10% utilized. And here's, I've done the same thing with the single-threaded macro benchmark. IPC score is about 0.8, a lot better, quite a bit better. Yeah, so all these counters are available directly in instruments. What you do is you would make a new counter instrument and then you would long click on the recording button. You'd pick an event and then you can, you would have a really just daunting drop down of performance counters. It's sort of assumed that you've read these hundreds of pages of documentation. And really what you're gonna do is you're gonna find a handful of them that work well for you. But the nice thing is you can actually create formulas from these performance counters. IPC is the only one that ships with default. You can make your own. Here's an example using instruments where I'm looking at, I can tell it's hard to read from there, but I'm looking at L1 hits and L1 misses for the multi-threaded benchmark. So I specifically went through the drop down, picked L1 hits, L1 misses, because I wanted to see what they looked like for this multi-threaded macro benchmark. Here you can see we have about 20 million hits for the L1 cache and about half a million misses, which, you know, isn't great in my opinion. And then another use of instruments can be to sort of look at your typical waterfall of methods and see how long each individual function is taking. And you can see the heaviest stack trace over there. Yeah, so lots of, you know, it's an easy to use performance tool. It's a little limited because it's cocoa-specific, but since you have the performance counters available, it's actually much richer than a lot of people give it credit for. And I suggest you give it a try. On Linux, as I said, at Buoyant we deploy most of our code on Linux, so we dig in with Perf. And Perf has been part of Linux for a few years and they've been constantly improving it. They put a ton of work into Perf. It has both kernel and user space profiling. It's a sampling profile with a configurable sampling rate which you can play around with. You know, Nyquist there tells you that if you wanna see something that happens at rate X, you should sample it to X. And so the nice thing about being configurable is that you can really turn it up. And it's also pretty cheap. They designed it to run at low overhead so you can run it in production at a somewhat moderate sampling rate. Now here's an example of looking at the IPC metric again of our program using Linux Perf. And I've highlighted that it's roughly 0.5 instructions per cycle. This is a different CPU. This is a dedicated machine we use for performance work. There's a different CPU which is why the instruction scores a little bit different. And here I use Perf to look at the cache hits and misses. Here I'm looking at the L1 misses and hits and you can see that for the multi-threaded benchmark I have about a 5% miss rate which isn't great. For the single-threaded one I have a 0% hit rate which is great or excuse me 0% miss rate. The 0% hit would be pretty poor. Yeah, so I really encourage you to dig into this. If you run software that's supposed to run on Linux you should really dig into Perf. It's much, much deeper than I had time to go into considering that I'm talking about other tools. They're all very Linux specific. You can do things like you can dig into this. The Linux scheduler, you can dig into the IO network subsystems. Incredible amount of tooling being built around Perf. One of the things that we use with Perf is we build flame graphs. And a flame graph is a sample. It takes us, again like I mentioned, Perf is a sampling profiler.  And here is what one looks like for our single-threaded program. So like I said, you take, let's say this is a thousand samples and then it aggregates them. The width is what's on the very top and you can mouse over to see what that is. I know it looks incredibly tiny. You can actually drill in by double clicking and then you can mouse over to figure out exactly which function it is. What's on top is what was running on the CPU when the sample was taken and then the width of what's below it gives you a sense of how often that was seen in a sample. So here we can see we get a clock, we get the time, we make an instant, we call a map on a future. So it gives you sort of a sense of your program. That was a single-threaded benchmark. Here's the multi-threaded benchmark, much more complex as you would expect. You've got a couple threads, we've got a mutex. One thing that we do a lot is we'll just drill into some part of this and I just give things a look from time to time to make sure things are looking sane. Really useful for looking at long-running programs. Since what we ship is a proxy that runs 24-7 in people's data centers, it's really nice to be able to get a sense of how does things look at time. T versus like 30 minutes from now are things changing or as events happen inside of the system, how does the shape of your running program change? Yeah, Netflix pioneered this technique, really, and they use this to measure the health of all their online services all the time. They're always building flame graphs of all their online services. The one thing that's tricky here is you've got to remember you have to get symbols, just like I described earlier with profiling. You have to remember to add the symbols. Otherwise, what you're going to get in the graph, in the flame graph, you're going to get a bunch of addresses, and good luck. So the last tool I want to talk about is probably the most complex as VTUNE. So like I said, Intel engineers have the same question that you all have, it's like, how well is my program running? So they added performance counters. Now they have another problem, what all these performance counters do? So they made this commercial tool called VTUNE, which helps you sort of make sense of what all these performance counters do.  It has tooltips and a GUI that works with SHX forwarding, has a CLI, and if you're an open source developer, you can get a free license. Here, I'm looking at, hmm, this is the single threaded benchmark. I really should have labeled these graphs. Thankfully, I've looked at them enough to know what they are. This is the single threaded benchmark, and according to, this is the general exploration view. You can see there's a fair amount of detail here. I can tell this is memory bound. This is back and bound specifically on memory. And looking at the multi-threaded benchmark, it's also back and bound, but something really stood out to me from running this, which was that I was bound on a remote cache, and we'll come to this, because this is a lesson I learned from preparing this talk. I didn't realize this about this particular benchmark. This is the multi-threaded benchmark. Here's a view of, I believe this is the memory explorer. Here we're looking at, this is the single-threaded benchmark I can tell from the miscount, and you can see stacked CPU time, but then secondarily, what's memory bound? And what's really great about this tool is you can mouse over any cell, and it'll give you a useful tool tip of what that means. It's really helpful when you're starting to use a V-tune. It's such a daunting tool. It'll also give you the formula that it uses to drive many of these, which is something which will be like this counter, minus that counter divided by some other counter, and that's an example of a formula. You actually take those formulas that you learn and plug them into instruments for when you're doing development on your laptop. Here is the multi-threaded program. Miscount's pretty high. That's how I can tell this is the multi... Oh yeah, it also has the name at the very end. Here's the multi-threaded benchmark. Here's the locks and weight view. So typically when you use V-tune, it has a lot of different analysis types that you can run, and you would start by looking at like the general exploration view like I showed at the beginning, and then you would run it over and over again using the different types of analysis you would run your benchmark. That's another great reason to use, to write a macro benchmark, is that if you have an example application, you can hand it to Perf to run. You can hand it to V-tune to run. And then you have just an easy tool that you don't have to remember how do I run this? Like what scripts do I need? You just run your macro benchmark in the tools. Here we can see that there's very low weight count. And in the multi-threaded benchmark, you see we have a very high weight count. Like here's seven, we waited seven million times. And that's no good. This is another view. This is the event count view. You can see from the progress bar at the bottom there's a tremendous amount of detail here. This is kind of like the day trader or fantasy football view of your performance. I wouldn't encourage you to spend a lot of time here. But it is something I mentioned earlier that there's a GUI and there's also CSV you can get. This is something you can get CSV from and you can do some post-processing on to learn specific insights for yourself. So that's sort of the takeaways from V-tune is that Intel knows their CPU is better than anyone and they know these hundreds of pages of documentation and they know how to build these formula formula for the different performance counters and they can surface them in V-tune. It's really, it can be really overwhelming and the tooltips are helpful and all the different analysis modes I found it really useful. I only dug into maybe a third of the analysis modes in my screenshots here. Oh, that's, and I learned a lesson. While I'm preparing this talk, I learned something. V-tune highlighted this remote cache issue and what I learned is that I was remote cache bound. And what this means is I was talking, this machine has two physical CPUs and one thread was running on one CPU, one thread was running on another CPU. I didn't realize that when I was preparing the talk. The screenshot is what showed me that. And there was also a really great, I meant to put a screenshot in here, of a memory view that shows you for all the DRAM sockets, like the different, like the physical RAM, the stick that you'd plug in. It shows you all the memory use for that and also the memory use between the two CPUs, which was also like just a dead giveaway that there was a behavior I wasn't expecting. So going back to Perf, you can see that I had that 5% miss rate that we talked about earlier. So what I, this is the multi-threaded benchmark. What I decided to do was, well, I'll just use task set. I use task set, I'll run on one CPU. We'll see how it improves. And then the miss rate dropped to 0.01%, which was pretty fantastic. And then you can see the total runtime dropped from nine seconds to 3.8 seconds, which was pretty great. I was pretty, I was really pleased with that. IPC drop, we were talking about IPC as being a useful empirical measure. IPC increased about 10%, which tells me there's still a lot more work that we could do to improve that. Yeah, so performance is hard to understand. Like given how complex CPUs are, there's a lot of tooling. You really need to measure it empirically. You need to run your program and see how it does. There's a lot of great tools that you can use. IPC is a really great measurement that you can use. I've been really pleased at how many tools there are to use them to measure performance of programs today. You can use instruments on the Mac, which is much more powerful than people realize. You can use Perf on Linux, you can use VTunes, but ultimately the best tool is the one that you use on a regular basis. And I wanted to give a special thank out to Eliza for walking me through instruments. And thanks, thanks for listening. Thank you.


Vina dan ini adalah kedua saya untuk tahun-tahun Ya, saya mempunyai dia di Deph SG apabila saya melakukan tanda-tanda terakhir Mereka minta, kamu mahu bercakap pada metap PSP kemudian kamu mahu bercakap CFS Baiklah, jadi saya sekarang Jadi, saya akan bercakap sedikit tentang CFS Jadi, saya akan bercakap lagi Jadi, apa ini?    So entries are basically the records. Let's say in your website, you have maybe you have a list of blog posts. So the blog posts are basically the entries. And then the fields are what, later on I'll show you across CMS, the fields are what users will use to update the website blog post content, whatever like the title, the images, whatever did that. So then the main thing about cross CMS, what the whole cross CMS structures around is what we call sections. So sections are basically containers, so-called big containers for all your entries. So these are the three main types of sections in a graph. So they're single, this channel and structure. These images are not taken from the documentation. This one is basically, this is how I visualise what are the different kinds of section types in between. So single is very simple. It's basically like a one-off page, like home page. Let's say a page where it's a one-off page, like home page or contact page or whatever. Then channel is basically for kind of entries where you have a whole series of similar kind of content like let's say a post of a blog section or let's say like articles of a news articles and all that. And that structure is a more elaborate where the order of the content matters. Although personally for me, while driving around with cross CMS I've never really chance upon the need to use structure at the moment probably for more complex websites like let's say if you're writing a documentation then you have a lot of information, a lot of articles may have different orders in different manners. So some of the features that I really like about cross CMS is things like, so there's this thing called deaf mode which I will demo right now. That one was really cool because when demo mode is on, it shows you all this, I will show you. So let's say I go to my code. So right now go with, so this is example of let's say a website like this. So I actually build this template myself. And this is basically the front end of this whole CMS, this is the example of the control panel. So demoing the deaf mode if I go to suyo and then so right now my deaf mode is set to true. So let's say if I go to demo a 404 page, let's say if I go to a random page that doesn't exist, let's say random. It will show me a lot of debugging, a lot of error messages which is very good essential for debugging. But obviously as a developer and all that we know that this can be a potential security risk because it could show a lot of potential information that we don't want, once it goes to production because CMS makes it very simple. All I have to do is set this deaf mode to false. And then if I refresh the page or I get a 404 page so I don't get any debug content debugging stuff, whatever. So I'm going to set it back to true because this is on local so back to this. Dan the next one is this CP trigger CP trigger is short for control panel trigger. So what this actually means is the main route that your control panel, this is basically a control panel, where the route that the control panel sits on. So right now while my developer process I already set it to a it's not say secure in terms of I mean because CMS itself has its own internal security but it's secure in terms that so the default route when you install craft is admin which is a very guessable route for most content for a lot of admin panels and control panel. So you can actually change this value because CMS makes it very easy to change the route of your control panel. So what you have to do right now is a CPHY which is short for company so if I change this CP trigger to let's say admin right now if I go, it will show me or cannot find because I changed the route. But if I change it to admin now I get it back then I'm going to change it back to CPVH and then Okay So this is the week at the moment I kind of I forgot how to activate it but while I was going through a tutorial on craft they did feature this and this is really cool because what happens is right now I can't really show but what happens I can't really see at the corner I forgot how to activate it but basically it will appear right at the bottom and then when you expand it this is what you get and then you get to see things like resources status of your page or whatever is going on all the data and whatever is going on behind the scenes on your page There another one this is really cool CMS comes with a lot of different kinds of fields but I think the one really cool is this matrix field which basically allows a user to really structure the position the content so let me go back to a page to show so let's say I go to a product page so this is one example page that users, all this content is basically this is entire thing is a matrix field everything after the quote below so now I go to entries what product is this feature product 1 okay I go to feature product 1 oops and then let's say I do a live preview so you notice you see the item page content is basically position exactly the way all this content is position so right now if I if I move my quote down to here look at how it refreshes this quote is this right it moves back up so this is really cool and what's even cooler you think that right here I have I have a page item content here I have another item page content here but in the quote right you only need to define each block of quote once and then cross CMS is smart enough to figure out okay and then page item page content here and then I will just render the template quote that is responsible for displaying this so if I go back to my quote so actually just moving around had it first is after move around they generate a template quote base so basically when I was building up this whole thing the only part that I ever touch actively were the template section and of course the web the web to put in front end your resources and stuff so let's say for example the page content that's it so so even though page item content there's one block here and another block here behind the scenes in my quote I only need to define this one and then base on this block it's smart enough to know which thing to render so this is really cool of course there are typical standard fields like tags and stuff like that but I thought this will become the future so as you saw just now I give you a glimpse another thing I like is the tweak templating engine that craft users so it's a lot like I think I forgot blade is it's it's curly brace and exclamation mark right right curly brace okay curly brace for string interpolation but the normal like oh okay so similar so very similar to blade so here are more links because later in the demo so these are some links one thing I want to highlight is craft quest if anyone is interested in tutorials on craft cms craft quest was a website basically they have tutorials all on craft and then craft has its own stack exchange website as well and then i don't know wifi but basically I wrote an article on craftcms on dev.to so this is a link to my no but it has it's not open source but it has a free license what you can only make modifications to a template ya and any php code modification requires plugins any like php any modification to craftcms code base so you can only modify existing templates no no templates so you can build your own templates and basically a lot of times things you can ya most of your templates your templates basically what you need to get your things done maybe maybe one very good example is things like let's say let's say on your website you need user content let's say contact page let's say a contact form i think you need a plugin for that there is no i'm not here to figure out inbuilt way to even simple things like contact form but for static content ya of course it handles it pretty well so let me give you an example because we need ya maybe i'll go in and show you around so i'll leave the page so remember i thought just now a bit about death mode so even on the cp panel there's this yellow bar here this yellow bar also indicates is a visual indicator that you're in death mode so if i set it to false then this yellow bar will go away so this is what i'm talking about so there are fields, there are sections and then there are entries so sections are like i said the containers of a content just now i'll show you products so i have product item so product items is a page so this is where i define type and then this is basically the URL so it's like this products and then so this is the section where you define your entry container and then in your fields this is where you going to create the fields so that so maybe i'll go through maybe i'll go one through one process on the whole process flow on adding a field to a section and then using the field to add content to the page so let's say right now i want to add i don't know maybe a tagline here in the feature on the feature item so what i go first is i go to fields and product and i create a new field and say product tagline so and there go settings so now i created the field but right now i have not added the field yet to a section so even though i created the field if i go to entries now i go to a featured product i won't have that field available here yet because i've not yet added to the section so now i need to add it to the section so i go to sections i go to product item and this is here again cross CMS gives you a very simple interface to edit whatever fields you want to put into whatever fields you want so put in so that other users can edit entry data so here i created a new product tagline so i'm going to drag it over to let's say here and then let's say i go to entries oops need to save go to entries i go to product one i say some tagline i save ah but there's one thing if i refresh my page i'm still not going to get anything yet because i haven't yet added the template code i haven't yet added the template code to show the tagline so i'm in entry just make sure i'm in entry page okay wait i don't want it okay okay oops let's see so i'm in i'm in the entry so if i type entry.tagline okay refresh and there we go some tagline so how then then we'll be asking how it's cross here and there's no what's entry entry it's default when you define let me go back to two sections let me go back to sections okay no no here yes okay so how does it know this is the URL of the template and then i define i define here the URL of the template which is product slash entry.tweet so crossCMS knows so the template code to product item is basically in products in a plus folder and it's the underscore entry.tweet so by default entry it's a default value to call then for in this case or things like let's say product tagline product types how crossCMS knows what these are is basically when i go back to my settings and i go back to my fields is the handle so this is through the handle that crossCMS is able to pull out the content of that entry yeah so oh yeah one more thing one more thing i also want to show is image transformations another one so let me go back to you see this image it's actually i only have to upload one image source and then in settings i go to assets here image transformations based on what i define here i can based on the image transformation define here will crop or resize the image according to what you use so one good example is here the image i'm using the cut the large cut image transform and yeah so there's image transformations crossCMS you download it to composer or you can download so but it has a license so it's got 3 tiers of licenses that's the free one so the main between the different licenses is how many admin accounts you have so for a free license you only have one like i say i'm the so called the super admin of the entire control panel you see i can create fields i can create sections but generally i think in mostly you don't really want to do that you probably have one account that has access to everything but most of the time you want to restrict access to different components of the control panel based on different people you only want editors to maybe go into entries but you don't want them to start messing around with the fields and stuff like that and that's where you need there's a pro and enterprise pricing so those i hate to do that so let's say just imagine make a very trusty decision to change to put cofCMS cofCMS will this i mean i mean it will be complete so so they have the pro license is allow multiple admin accounts then they also have enterprise one which is basically a customized solution but ya okay any other questions? okay, thank you Dermia


Slow down from my sports announcing days I first got to hollywood the sound memory yelling cut They tell me that the class I was clocked on something like our gas at 200 But you mean it isn't that hard I Fellow Americans, I'd like to take a few minutes of your time to talk about some of the problems we face in this blessed land Seems to me My fellow Americans I'd like to take a few minutes of your time to talk about some of the problems We face in this blessed land of ours and what I feel we should do about them I can't cover all that territory in five minutes, so I'll be back every Saturday at the same time same station live I hope you'll tune in These aren't easy times for a great many of you Yesterday we were told that unemployment has gone up another two tenths of one percent equal to the unemployment rate We had in 1975 as we began to come out of that recession We can however take some comfort from the fact that 99 and a half million of our people are employed Now I know that's no comfort to those who want to work and can't find a job And it's no comfort to farmers independent business people auto dealers realtors and building contractors who see themselves going out of business These people want answers and so do you about what we can do to get our economy back on track The last recession before this one came in 1980 and by the time our administration took office Unemployment had almost reached 8 million the prime rate had reached 21 and a half percent The highest in more than a century and inflation was at 12.4 percent According to the polls Inflation was the number one problem in everyone's mind All of us associated it with the high interest rates and rightly so a lender must charge an interest rate high enough to cover Inflation as well as give a return on his or her money We proposed a program of economic recovery Based on the belief that high taxes had deprived people and business of incentive to the point that we'd lost much of our ability to produce Those high taxes had fueled a rate of increase in government spending that reached 17 percent in 1980 alone Even high taxes couldn't keep up with that in the few years between 1976 and 1981 federal tax revenues increased by $300 billion, but we had 318 billion in deficits So our program also aimed at reducing the rate of increase in government spending Unfortunately, the interest rates stayed up even though we began to reduce inflation People couldn't afford to buy automobiles on time Few could or would take a mortgage to buy or build a home at those rates layoffs in the automobile and construction industries increased and Farmers who borrowed a plant and repay at harvest lost money even on bumper crops By mid-July we were back in if we were ever out of a recession But for the first time in the many recessions that have taken place over the recent years We had a plan ready to go The Congress had agreed to a budget that cut the rate of increase in spending nearly in half and had passed a tax cut program which increased appreciation allowances for business and Began a 25 percent cut in income tax rates for individuals to be phased in over three years All of this only began last October 1st Even so interest rates have been reduced by 20 percent, but that's not nearly enough They have to come down more and they should because our greatest success has been in conquering inflation It's no longer double-digit for the last five months. It's been running at four and a half percent By all the rules of the game Interest rates should be down around nine or ten percent Unfortunately the increase in unemployment increased government costs and reduced revenues More money had to be spent on unemployment insurance and other benefits Fewer people working meant fewer people paying taxes up went the projected deficit for 1983 And up went the concern in the money market that this would lead to an increase in inflation as it has in all those past recessions The answer to the recession lies in bringing interest rates down To do that a signal must be sent that while the political process always requires some compromise government this time intends to stay the course that we're going to make further reductions in spending and Hold to a steady consistent growth in the money supply In short that we're going to come out of this recession not with a temporary quick fix that leads to another recession down the road But with a solid economic recovery based on increased productivity and jobs for our people Now I know you've been told by some that we should do away with the tax cuts in order to reduce the deficit That's like trying to pull a game out in the fourth quarter by punting on third down You've also been told our program hasn't worked. Well, of course, it hasn't it hasn't really started yet Our 5% cut in October was almost wiped out by the January increase in the Social Security tax called for in the 1977 tax bill The reduced budget spending and the 10% tax cut in July will be the real beginning of our program There's no instant cure, but there is a cure With your help and your prayers, we'll find it I'll be back next Saturday Thank you and God bless you My fellow Americans I'd like to take a few minutes of your time to talk about some of the problems We face in this blessed land of ours and what I feel we should do about them I can't cover all that territory in five minutes So I'll be back every Saturday at this same time same station live. I hope you'll tune in The answer to the recession lies in bringing interest rates down to do that a Signal must be sent that while the political process always requires some compromise Government this time intends to stay the course that we're going to make further reductions in spending and hold with steady consistent growth in the money supply in Short that we're going to come out of this recession not with a temporary quick fix that leads to another recession down the road But with a solid economic recovery based on increased productivity and jobs for our people Now I know you've been told by some that we should do away with the tax cuts in order to reduce the deficit Well, that's like trying to pull a game out in the fourth quarter by punting on third down You've also been told our program hasn't worked. Well, of course, it hasn't it hasn't really started yet Our 5% cut in October was almost wiped out by the January increase in the Social Security tax called for in the 1977 tax bill The reduced budget spending and the 10% tax cut in July will really be the beginning of our program There is no instant cure, but there is a cure With your help and your prayers will find it I'll be back next Saturday. Thank you and God bless you. I couldn't answer that until we have to by law go up Shortly in the next couple of weeks. I can't remember the exact date and present an updated project As I've said so many times I we do this we comply with the law I don't have the greatest faith in those projects because I think there are too many imponderables both ways that To make them really something that we should Be dependent. Well, the increase that as the recession goes on and unemployment increases Of course the deficit increases because it's that's more people who stop paying taxes and is more people who then Have to receive benefits and unemployment insurance and so forth So we've known that this recession has brought this about and the answer is We've got to Bring the recession to an end and I think our program will do that Yes, we we can get out of it and the way out of it Is not the way that's been tried on most of the recessions that have taken place in these last few decades Hyping the money supply artificially stimulating the money supply Stimulating government spending as if that somehow will be an aid to the economy and Up, of course goes inflation when you do that now inflation does have a temporary Stimulative effect just by its very nature And it's kind of like a warm bath. It feels good for a minute, but then the water gets too hot What we're trying and the difference between our plan and what's always been tried before is the wrong range plan To get the economy back on track to get America back to where we're The industrial powerhouse that we always were and to do that that requires the tax cuts Both for business and for individuals to stimulate the economy Reduce the percentage of the gross national product that the government is taking in In taxes and that the government is spending And All of these things I think our plan will do And it is the only way It isn't an instant fix But it also is a way out that simply Solves the problem by broadening the base of our economy providing the jobs that our people must have Being able to compete once again with the other industrial nations, which we haven't been able to do very well for quite some time now Thank you Well, I think that it isn't for me. I think it's time with all the confusion and all the conflicting things that seem to come out of washington Between leaks and statements by individuals and pressure groups pressing for their particular program that It's an opportunity to Try each week to based on what has taken place in that week to bring the facts to the people as succinctly as I can and as many as I can cram in five minutes And uh, I just thought I'd start with this one because this one I think happens to be the one that's the most on people's minds This thing of how are we going to get the interest rates down? How are we going to resolve the present problem? So That was the reason for it and Huh, you look pretty comfortable Yeah, it was almost like old times there. I was waiting for somebody to try and steal second Okay, thank all of you. What are you doing down here on a saturday? Don't you work up? Okay


Simon says, he asked, there should be a telephone list of employees. Previously, these were available as public folders. The problem is that the contacts in the mobile devices were missing and that the users couldn't even find the public folder. What other options are there for making such a list available? Well, I'm wondering why they're not just using the built-in functionality that's available in your profiles. So Active Directory, if it's populated correctly, will have their phone number, their supervisor, their department, and all of that. So I would advise people to go and fill out their Dell profile.  So that would be my preference. I like that operative word, the words of phrase. If it was filled out completely correctly. Yeah. I mean, there were third-party tools that were built to help that process to fill basic profile information in AD profiles just because it's so much missing. But that's one of those things where as your employee onboarding, for example, like don't allow people just to skirt by without providing that contact information. Yeah. And it gives you even more context.  Or who do I ask about that? Being able to add those attributes to your profile, say, ask me about SharePoint. Ask me about Office 365. That's the preference, right? Yeah. In fact, we had a question on exactly that just recently. Somebody was asking about building a skills database. I mean, that kind of profile information. If you think about where things are going, certainly within the Microsoft 365 world, whether or not you're using today something like the Viva topics and are building out those profile cards. I mean, the people skills and being able to identify when I've written a bunch of content, I'm involved in projects, and you're doing searching on contextual searches and getting served up data that's relevant to me. And then my contact information is not there. That's something that you need to motivate people to go and complete that information or require it. Yeah, do a little scavenger hunt, incentivize them to go fill it out and give out gift cards. I like to have them build a scavenger hunt and say, who can you find that does this, this, and this? And whoever answers the questions gets some kind of gift card. That's a good way to get people excited because what's in it for me, filling out my stuff or, you know, give card to some other place where I can spend money. Yeah, you're starting to sound like Christian. What's in it for me? I know. It's the adult trait in me. You know, it's, you always go from the adult perspective of what's in it for me. If it costs me more time, then I ain't going to do it. But if it gives me a benefit, I might think about it.  So for people to go in and do it, but there, you've got to put, you do the carrot, then have the stick ready, get people to fill out their profiles. Exactly. Other than that, you could do a SharePoint list, contact list on their main intranet, but somebody has to maintain that. That's the worst thing about it. You know, we don't want to do that. If populated correctly, Active Directory in the org chart, it's self-maintaining and that's ideal. Thank you.