I've been taking a nap every day, not today. OK, today's talk is the life and times of an open-stack virtual machine instance, quite a long title. So just to introduce myself first, I'm Mark McLaughlin. I'm from Red Hat. I'm Irish, in case you don't know from my accent. I've been working on open source for maybe 15 years now. I've been at Red Hat for 12 years. I've been working in the kind of general area of virtualization and cloud for nine years now, which is pretty crazy. And I've been working on open stack for the past four years. So I've worked on everything from the NOVA project, how to create the Oslo project. I was on the technical committee. I'm currently on the OpenStack Foundation board. So I've been around this space for quite a while. And so the point of today's talk, I think, was to give kind of a high-level overview of how NOVA works, of how NOVA creates a virtual machine, but to not just do your typical high-level overview, to also pick some kind of deep dive, interesting areas we can kind of really dig into and kind of see how it works. And I guess, from my perspective, what I'm trying to share here is I've learned a ton of really deep, interesting stuff that OpenStack touches on. It gives a second. I think we're going to have to get this sorted. Thank you. OK? Cool, thanks. OK, so OpenStack touches on a whole ton of really, really interesting technologies. And I've had the chance to learn about a bunch of different stuff throughout the years. So I kind of feel when people are looking at OpenStack and learning about OpenStack, they kind of get this high-level overview, but never really get to go down deep into some of the areas. I think that's missing an opportunity, right? There's lots of interesting stuff to learn. So I'm going to pick, I'm going to walk through OpenStack. I'm going to walk through how NOVA works and pick some kind of areas that I find interesting and maybe highlight for you that there's plenty of other things you can learn when you're working on OpenStack. OK, so to start with, what's going on here? I am going to be plagued by demons, aren't I? OK, so to start with, what's OpenStack all about? I always start with the mission statement, because I actually think unusually this is actually a really good mission statement. It's about ubiquity. It's about open source. It's about a cloud platform beyond just simple infrastructure. It's about public and private clouds, and large-scale, small-scale, simple to implement. It really captures everything that OpenStack is all about. But I think maybe the way we should all always start about OpenStack is the abstractions we're trying to provide to users of OpenStack clouds, right? Am I plagued with more? See, this is because I'm not using VGA. Can I switch to VGA? Sorry about this. Do we have VGA? OK, cool. Maybe next time I could give a talk on display technologies and VGA and I kind of feel like Kyle yesterday, which is a beautifully prepared demo that went horribly wrong. OK, so we talked about the OpenStack mission, and now I guess what I'm saying is the next thing I think about when I think about OpenStack is users of OpenStack clouds, right? The abstractions they're presented with, what those abstractions do, because that's ultimately what we're building here. So I guess if you think about you've got an OpenStack cloud, an administrator creates an account, give that account to a user, what do they have now access to, right? They've had access to all these abstractions, right? They can create storage volumes. They can create networks. They can list images. They can manage objects, storage objects, and they can create virtual machine instances. And so imagine it is a typical simple user case for OpenStack. You've got an IRC bot. You want a virtual machine. You want to run your IRC bot there. This is what we're allowing users to do. And as most of you probably know, we implement those abstractions with the following services, all pretty straightforward. The marketing view, I guess, of OpenStack is we've got a bunch of technologies. They all layer on top of each other nicely. OpenStack is an abstraction of, say, technologies like Cef and KVM and OpenV switch. All of those run on a Linux platform, and all of those run on these lovely companies and hardware platforms. That doesn't do much for me. In reality, this is bizarre. OK, we're going to have to do lots of switching here.  This is what we're actually presenting to a user. We're giving the user the opportunity to run their IRC bot on some little virtual machine running in there. So the abstractions we're creating allows us to take a bunch of physical resources, this big pile of tin, and allow us to carve up those physical resources and make them available to users. So that was a photo of three racks of gear. And so I thought it might be interesting just to start with, in a very practical sense, what's going on in those three racks. So what I did was, I don't get out much, right? I'm a developer, I'm a technologist, whatever it is. So I don't actually get to touch this kind of real gear very often. So what I did was I went to Red Hat's reference architecture with Dell, and I looked through the reference architecture that we currently recommend for running OpenStack with Red Hat and Dell Gear. And so if you had three racks, you have three 42U racks. You would have a mixture of Dell or 630 servers and two U Dell or 730 servers. You'd have three controller nodes. In total, you'd end up with something like 40 compute nodes and 12 storage nodes. You'd use the or 630s for your controller nodes and your compute nodes. And you'd use the or 730 for your storage nodes. The reason you use the or 730s is because they're nice beefy 2U boxes. You can fill them up with a whole bunch of disks, and that makes them really good storage nodes. And you end up with 11 switches in this configuration, which seems a bit crazy. But the reason for that is, let me see if I can even add this up correctly, you end up with a management switch per rack. That's three. You end up with two top of rack switches per rack. That's an additional six. That gives you seven. And I think I'm missing some, but you end up with two top of rack switches per rack. You end up with a management switch per rack. But you also end up with two aggregation switches. And these are the switches that kind of aggregates your top of rack switches and feed you out into a core network. So in an even more practical sense, just to dig into it a bit deeper, what you have, you've got three 4U racks. So three 4U racks is 1.8 meters wide, 1.2 meters deep, and 2 meters tall. So it's a pretty small space. And in that, you fit all of this gear, which adds up to about half a million dollars, as best I could tell. In there, you've got 1,600 cores. You've got 5.4 terabytes of RAM. You could run something like 2,004 gigabyte virtual machines. You'd have 16 terabytes of storage. And just as one little other data point, I decided to figure out how much power this would draw. And so it turns out the servers themselves would probably draw somewhere in the order of 25 kilowatts. But you also have to kind of take into account the rest of the infrastructure in your data center, right? So researching this a bit more, I found out that data centers have a metric called power usage efficiency. And this is the ratio of power that spent kind of powering the entire infrastructure to the amount of power that spent powering the actual servers. And so that's typically about 2.0. So that means you're drawing about 50 kilowatts of power to keep these three racks of gear running. And I did another little bit of calculation. And it turns out 50 kilowatts is enough to boil 500 liters of water every hour. And that's probably going to work out as something like $35,000 US dollars per year. So I think it's just really interesting to put a really practical view on what we're talking about here.  And you can run something like 2,000 VMs in there. Certainly opened my eyes to what's going on. So if you want to learn more about some of the stuff going on there, the OpenStack architecture design guide is quite good. Kind of goes into a bunch of different technical considerations you could just think about when you're designing an OpenStack cloud. And this Dell and Red Hat reference architecture I talked about is available there. And actually, what I'm going to do is I'm going to post these slides to Twitter later. So you'll be actually able to click those links and get that information. OK, so I pitched this talk as talking about the lifecycle of an OpenStack VM. So we all probably know all this already, but how do we go about creating an OpenStack VM? The highest level you could look at the horizon's interface for this. There's a bunch of stuff in here, but basically all you're doing is choosing a name for your VM. You're choosing a flavor, which is the kind of size of virtual machine you want. You're choosing the image you want to run, whether you want to run a Fedora image or a RL image or an Ubuntu image or whatever. And that's basically it. On the command line, it's similar. You've got an OpenStack server create command line.  And behind all of that, then, you've got the REST API. And so the REST API is posting a little JSON blob to a URL. Again, it's got the image name. It's got the virtual machine name. It's got a reference to the image. It's a reference to the flavor. It's got an authentication token.  But that's how a virtual machine is created. But what happens when that virtual machine request comes into our rack of gear? So you're sitting out there on the internet somewhere. Some magic happens that transports your request over the internet. And it comes into your rack of gear, into the top of rack switch. And then some other magic happens. And now your Python controller process handles that request and creates your virtual machine. Digging in a bit further, it's not as simple as that. It's actually your request comes in. It's addressed to a virtual IP on one of your controller nodes. That virtual IP actually maps to a HA proxy. This is in Red Hat's reference architecture. I'm talking about here. The virtual IP maps to a HA proxy instance.  So the request might come into controller one. And it's actually sent to the Python app running on controller three. And it's not even as simple as that, right? So HA proxy, the kernel on controller three is handling the request. But you don't actually just have one Python app running for handling these requests. The Nova API service, as an example, will actually typically be run so that you've got a worker process per core on your machine. So I've actually logged into a machine, a controller node, with 96 cores. And there's 96 Nova API processes running. So what's really interesting here is, I know a fair bit about what's going on here, but I actually don't know how this works. Because in this case, you have 96 Nova API processes all listening on the same socket. It's pretty interesting how that works. But how does the kernel decide which of those worker processes to give the request to? And I did a little bit of research, but not enough research to actually find an answer. It either wakes up all of your worker processes and they all rush to accept the request, or it's clever enough to wake up one of the worker processes. And so that in itself is probably a weekend project to figure out what's going on there. And it could actually result in a really interesting performance improvement to make to OpenStack if we're not doing the right thing. But it's not even as simple as that. The kernel on controller three receives the request. It wakes up, we assume, one of the worker processes. The worker processes is sitting there in what we call a main loop, calling the poll system call. So it's basically blocked, waiting for any activity on any of the sockets it cares about. Your request comes in, poll returns. And now our asynchronous IO framework called eventlet takes that request. And it knows what eventlet coroutine is waiting for a request on this socket. It dispatches that coroutine in what's called a green thread, which then basically runs something we call a whiskey pipeline. That's web services gateway interface. It's basically a standard in Python for how a HTTP request is handled by some Python code. And eventually, that code that you know in Nova is handling the request eventually gets called. So I guess what I'm trying to point out here is if you were reading the code in Nova for handling an API request, for handling that the API request for creating a server, you know that's coming across the internet somewhere, and you know you've got this rack of machines. There's actually a ton of really, really interesting stuff happening in between that's worth digging into. Which version of OpenStack? Any of the latest versions? Is there a specific reason you are wondering? Do you think some of this has changed? You remember eventlet has? OK, no, that's a good question. And actually, I'd love people to ask more questions. So the question is, has eventlet been replaced? So as I said, eventlet is an asynchronous IO handling framework.  You want to pause what you're doing and do some other useful work. And typically, you use what's called a main loop for kind of handling that. And eventlet is the framework we have for doing that. And so the question is, are we still using eventlet? And in one case, we're not. In Keystone, we currently don't use eventlet, right? In Keystone, we actually use something called Apache's modWizgi, what do you call it? ModWizgi module, I guess. And so in that case, Apache is basically handling the request and spawning off a Wizgi pipeline. But in the case of Nova or Selometer or Neutron or anything like that, they are all still using eventlet. So it's only Keystone that switched from eventlet. No problem. So in terms of more information there, there was a really good talk at the last summit about HGA and load balancing. So that would talk about a reference architecture for HGA and the load balancing. If you want to learn more about asynchronous IO, I wrote a blog a while ago about how that works in Python and how eventlet is implemented, essentially. And that detail about having multiple processes all listening on the same socket. Actually, if you go back to Richard Stevens' book, I looked it up there recently, it has a really, really good explanation of how and why that works. OK, so we've talked about your create server request coming in and being accepted by the Nova service. What happens next? So this is a pretty typical explanation of how this works. There's a typical kind of diagram that we use to explain how this works. So your HTTP request comes into the Nova API process, the Nova API service. The next thing it does is it creates a row in the database which describes your server. So it will go to the database, it'll create a row, it will give your instance a UUID, it will record the name, the flavor, and the image. And basically, its job is done. The Nova API's job is done, and it immediately returns a response to your HTTP request. What happens next is the Nova Conductor service will actually, sorry, the Nova API service creates an entry in the database, and the next thing it does before it returns your response is it actually sends a message over the message bus to the Conductor service, just letting it know that a create server request has been requested. So the Conductor service, the next job it has to do, it has to decide which of those compute nodes in your three racks of gear to run your server on. And so it needs to make a decision. Basically, what's the best server for running this virtual machine on? That's essentially what the scheduler needs to do. So the Conductor asks the scheduler which host is the best host for running the virtual machine on, and the scheduler has awareness of the resources available on all of those compute nodes in your deployment. So the scheduler will filter out all of the compute nodes that can't currently run your VM. Maybe there's no resources left available, and it'll sort the hosts that could run your virtual machine to try and make a decision about which host is the best host for running your virtual machine. So it might decide it wants to spread the virtual machines out across all of your hosts, or it might decide it wants to really pack all of the virtual machines on the smallest number of hosts possible. But whichever it decides is going to return a host to the Nova Conductor service. And then the Nova Conductor service sends a message to the Nova Compute service to say, please start the virtual machine. At that point, the Nova Compute service running on your compute node is going to download the image. It's going to create some XML for libvert, and it's going to run your virtual machine. So again, I don't want to go into too much more about that. There's plenty of other talks about this. Steve Gordon from Red Hat actually gave a really good talk in Vancouver, I think, to go a bit deeper. Rafi Cardalian from Metacloud actually, quite a few summits ago, gave a really, really good talk about really digging deep into how Nova uses the libvert in KVM. And if you want to learn more about the Conductor service and how the Conductor service works with the Scheduler service, there's a really good blog post from Russell Bryant about that. OK, I thought what I'd talk about next is a very specific type of virtual machine. Basically, posing this question, how would if you want to run a very, very specific type of virtual machine that's going to give you deterministic high performance, right? Do you want to run some workload that's really kind of latency-sensitive? You want to know that it's going to get the best out of the possible hardware available, and it's going to be really responsive to the needs of your workload. So you want a virtual machine. Maybe that has eight physical cores dedicated to its virtual CPUs. So it's a virtual machine with eight virtual CPUs. But underneath the hood, it actually has eight dedicated CPU cores for that virtual machine. It also has memory that's spread across two NUMA nodes. So NUMA nodes, if you don't know what NUMA is, basically, with many kind of modern servers, you can have a bunch of CPUs with some memory that's local to those CPUs, and a bunch of CPUs that's memory to those CPUs. And if you have some code that's running on this set of CPUs and need to access some memory over here, that request needs to go across a shared bus. To get access to that memory. So as much as possible, you want the code running on this set of CPUs to access the NUMA memory that's local to those set of CPUs, and not have to go across the bus to a different NUMA node. So in this case, we actually want a virtual machine that's so large that it's going to be spread across two NUMA nodes. But we also want the guest to be aware of that. We want the virtual machine to actually have a virtual NUMA topology of its own. So it's aware of the need to stay local or for code to be running local to those NUMA nodes. We also, for performance reasons, we want to use dedicated host memory with two megabyte pages rather than your standard four kilobyte pages. And we want to have two SRIOV virtual functions assigned to the virtual machine. But we also want to be aware that those SRIOV virtual functions have locality to the NUMA nodes as well. And finally, just to make it interesting, let's play it up again. Are we seeing any changes up there? OK, this is bugging me. OK, now, so lots of really interesting stuff. And just to throw one last interesting requirement into the mix, we want rather than the virtual machine having eight physical CPUs, we want to set that up as having two virtual sockets where each socket has four cores each. And maybe we only want to do that for Windows virtual machines. So you might be thinking, that's absolutely crazy. You don't want to do this. That's just not cloudy, right? That doesn't seem like something you ever want to do with a cloud. Just get out of here, get lost. But I think I really like this way of thinking about it. There's more to cloud these days. There's more to what we're trying to do with OpenStack than just replicating what EC2 was doing in October 2008 when EC2 first removed its beta label. The world has moved on. EC2, even if you just define cloud as EC2, EC2 is just doing a bunch of really interesting stuff these days. And when you look at my requests or my requirements for my virtual machine there, you could just think of that as an EC2 high performance flavor. It actually does make sense what we want here. So that quote wasn't from anyone, particularly insightful. That was just from me. So how do we go about doing this with OpenStack these days? It's actually pretty straightforward. The set of requirements I had sound crazy, but what we've recently added in the Kilo and Liberty release is a set of our NOVA instance type properties and also image properties for defining this stuff. So what you see here is three different instance type properties. One is called CPU policy. And we can basically say that the physical CPUs should be dedicated to the virtual CPUs, which is one of the things I asked for. We can also say how many numenodes we want. And we can also say what host memory page size we want. And basically, these are all properties on our instance type, and we just call our instance type m1.medium.performance. So you can imagine a user of an OpenStack cloud coming in and going back to that UI earlier. They're asking for a virtual machine. They're giving it a name. And they're basically just saying, I want a high performance virtual machine. And this is what's happening in the background. And the last thing I said about having a particular virtual CPU topology, we could set that up as an image property, the hardware CPU sockets image property. And we can set that as a property on the Windows image so that we actually only get this virtual CPU topology for Windows images. I have a bunch of links at the end of this. I think some of it came in kilo. It's definitely there in Liberty. But I think most of this was actually there in kilo as well. Is there any? What there is. Yes, there is. But what there is is all these links. As I have here, they're actually really good documentation for everything I described there. But actually, what I wanted to do was just dig in a little bit deeper into kind of how we've implemented this.  So in this case, you might go to one of those machines in our set up with three racks. And what you're seeing here is the new metapology of that machine. So you're seeing that there's two new missiles. We're looking at one of the new missiles here. We're seeing that there is a certain amount of memory available in that new missile. We're seeing there's a certain amount of four kilobyte pages. There's no two megabyte pages available. We're seeing a bunch of information about the CPUs and the kind of cost of a memory access going from those CPUs to this new missile. So that's kind of one aspect of this. We need to understand what the hardware platform can do or what that hardware platform has available. And then when we go to create the virtual machine, there's a libvert XML, which describes the virtual machine. Actually, it has a bunch of really interesting information for basically implementing the virtual machine this way. So the CPU Tune element basically describes each of the CPUs, which physical CPU, the virtual CPU is running on. And it also has this emulator pin setting, which is basically the QMU process has a thread which helps with IO. We want to confine that thread to just run on one of the physical CPUs that we're using for virtual CPUs. It also has a bunch of settings here for basically how the virtual numit topology is implemented and how the virtual CPU topology is implemented. So I'm running out of time here, and I just one last thing to cover. But again, I'm going to post this on Twitter, and you'll be able to dig into any of these articles or videos about how this stuff is implemented. So I'm going to throw my laptop out the window in a minute. OK, the last thing I wanted to cover was just to give a quick run through about how KVM actually works. So I think everyone has a fair idea that KVM is this technology for running virtual machines. But how does that actually work? And what I'm talking about here is based on an article that was recently published on LWN by someone called Josh Triglet. And I thought it was just a really, really good article, really worth spending the time to understand what's going on here. So to create a KVM virtual machine, it's actually as simple as this. You go to the dev KVM. You open that. You do this IocTill, which is called KVM createVM. And you get a file descriptor back. You've now created a virtual machine. But the next thing you need to do is you want to set up some memory for that virtual machine. So in this example, what we're doing is we're creating just one 4 kilobyte page. We're copying some code into that 1 kilobyte page. And then we've got a structure which basically describes that memory region. So we wanted to occupy slot 0. We want to map the 4 kilobyte page to not the first page of guest physical memory. We want to map it to the second page. We're describing that the memory size is 4 kilobyte. And we're giving a pointer to that memory space. And then we basically do this KVM set user memory region IocTill on that file descriptor that we got previously. And now our virtual machine has some memory. The next thing we want to do is we want to set up a virtual CPU for the virtual machine. Virtual machine without a CPU isn't much use. So again, we've got an IocTill which is createVCPU. We've now a file descriptor for managing that virtual CPU. We've a bunch of stuff here for basically mapping some structures which describe the state of the virtual CPU. So the kind of state of various registers and the state of the instruction pointer. We want to pass in some parameters to the AX and BX registers there. And basically all of that CPU state, we set all of that up. And then we do this KVM set registers IocTill on the file descriptor for the virtual CPU. So we almost have a virtual machine that's ready to run now. And to actually run it, there's another IocTill which is called KVM run. So what we're actually doing here is we're running that virtual CPU. And what this IocTill is going to do now is it's going to actually start running the code for the virtual machine. And that code for the virtual machine is going to keep running until it does something that needs to be handled by the hypervisor. So in the example in Josh's article, the code is very simple. The code takes the parameters that we passed into the AX and BX registers. It adds them together. And then it goes to write the result to a serial port. So it can run all of that code. But as soon as we go to output to the serial port, basically we exit from running that virtual CPU. And that IocTill runs. And now we're back in the hypervisor again. And we need to handle the reason that we exited the guest code. So in this case of writing to the virtual serial port, we'd get a KVM exit IO. And we might handle that just by outputting the result to the screen. And so what you have here is a loop where you're running the virtual CPU. You're handling the reason for the VM exit. And then you're running the virtual CPU again.  And you're doing that for one virtual CPU. So if you've got eight virtual CPUs, you're actually going to have a QMU thread for each of those virtual CPUs in a loop like this, just running the code and handling each of the exits. So if you want to learn more about that, Josh's article is quite good. But if you want to dig a bit deeper, actually, what I find really good still is Avi Kiviti's first paper about KVM, really describing the design of KVM. Everything there is still very relevant. So for me, that's just, this is going to be so annoying. So for me, that's just one example. These are just a few small examples of some of the really, really interesting stuff that OpenStack is pulling together. So when I was thinking about initially writing this talk and thinking about the kind of things that I could cover, it's the list that I initially came up with. You could talk about how to design a network that is the core network that feeds into your top of rack switches. You could talk about bonding. You could talk about in detail how OpenV switch works or how Ceph works. You could talk more about Async.io or RabbitMQ or MariaDB. Another really interesting thing is something called S-Vert, where when we're running that virtual machine, we're actually confining that process on the host side with an SE Linux context so that if there's an attacker inside your virtual machine, it can't break out to your host. We can talk about what real-time KVM is going to look like, how the Q2 image format looks like. Basically, there's a whole ton of really, really interesting stuff here. And OpenStack relies on all of this stuff, and I think it's really worth spending some time learning and understanding some of this stuff. So for me, that's really the conclusion here. When you're learning about OpenStack, don't just allow yourself to skim at the high level here. OpenStack is really, we're describing OpenStack these days as an integration engine. It's integrating a whole ton of different technologies. Most of those technologies are open source, and really there's huge opportunities out there to dig deep and really learn how these technologies work. So I'm out of time here. Hope that was interesting and useful, despite the display issues. They certainly stressed me out. But we're out of time. Anyone got some quick questions? Yeah? My machine, Robert, you should not influence that work. OK. How does this look really important? Yeah, so I think that's an interesting question. I'll repeat it so that the question is there's a concept of over-committing resources so that you could, say, maybe create virtual machines that consume more memory than you have physical memory, and that's called over-committing. And then there's the concept of resource isolation so that virtual machines shouldn't be able to influence other virtual machines. I think we could talk about that a bit further because we're gone over time here, and people are already starting leaving. But basically, the concept of over-committing is really useful to cloud administrators who want to get the best use out of the resources they have available, and the concept of resource isolation is you basically want to provide an SLA to your users, right? And the technology is there that you can enforce an SLA for your users while still doing over-committing. That makes sense. OK. Yep. OK, so people have started leaving. How about you and I just talk directly after this? That's OK. We've just run out of time. OK, thank you very much, everybody. Thank you.


Hello friends, welcome back to the channel. Very quick update this morning because Pokemon have announced an update that will be getting for the Indigo Disc which will be dropping tomorrow on the 7th of December. Quick update from Sarabee here, the next Pokemon Scarlet and Violet Indigo Disc information will release on December the 7th at 2 p.m. UTC. So this is followed up by a post on the Japanese Twitter account These days tomorrow December the 7th Thursday at 11 o'clock which translates. We'll go into the times in a minute We will release the latest information on the paid additional content Pokemon Scarlet and Violet's treasures of area 0 Which will be the Indigo Disc. We will deliver the latest videos on our official YouTube channel Please check it out. So this will be posted on the YouTube channels across Japanese sites And of course the Western sites in the US and Europe and things like that. Sarabee does go on to say the times State the times when it will be dropping. So 2 p.m. UTC for Europe, UK and places like that And you've got 3 o'clock Central, 9 o'clock Eastern, 6 a.m. PST and then 11 o'clock at night for Japan So very late for Japan, but the rest of us get it at a pretty reasonable time This is very exciting because we've only have a week to wait until the Indigo Disc actually drops on the 14th of December So very exciting that we're finally getting some news about these games pre them releasing next week Of course we'll be streaming it here on the channel. So if you want to follow along, watch along while all of this new information drops Do check out the stream tomorrow and it'll be a lot of fun I don't know what to expect. Hopefully we get some more information on the other things outside of the terrarium We've got all those previews from the content creators recently where they were kind of exploring the terrarium A lot of details around that have come out, but we don't really know too much about the main story after you leave the Blueberry Academy After you go back to Piledare and that's what I'm kind of hoping for that we get some information maybe on Area 0 And maybe we even get a special Terrorite event for, you know, Rage and Bolt and Iron Crown that we're kind of waiting for They have been released. It'd be nice to see those get an event If they all come in Terrorites, be announced before the DLCs drop But we'll wait and see what happens. It's exciting. Nonetheless, I'm very hyped for this But hopefully I'll see you all on the stream tomorrow. If you're about, let me know what you think will be revealed in the trailer tomorrow That we get for the Indigo Disc and I'll leave it there. So very quick on like I say thanks for tuning in friends Have a great rest of your day. Take care of yourselves and until the next time. Bye


community. He truly is awesome. So please give it up for Zach Gordon. Hi, everyone. Can you hear me? Is this thing on? Are we on? Yeah, okay. Hi, everyone. I'm so happy to be here. I'm coming from Washington, D.C., and I've wanted to be at WordCamp London for years now, so to have the opportunity is really cool. Y'all do a smash-up job, so thank you. So today we're going to have a quote-unquote workshop. Ideally, I was hoping we'd have a good, like, three hours to get into the code and practice building everything, but we only have 40 minutes. So what we're going to do instead is I'm going to talk at a high level and then go through a bunch of different code samples and point out a bunch of things that as you begin developing on your own, you'll want to know about and be aware of. So if you have a computer, there is a code repo, and you could also get the slides at goonberg.courses slash WordCamp London and follow along there, but it won't really be something where I'll be giving you really enough time to follow along. My apologies there. So she mentioned a little bit about me. My background is in education, and I started to really focus in-depth on Gutenberg after WordCamp US when Matt was like, it'll be out in April, by the way. It's April, right? And I'd say from following it, it has developed so quickly, and it's very feature-complete-ish and has come a long way, and there's a lot of refinement still going on, but I think it's important for all of us to know the code behind it, because in the entire time I've been working on it, the only things that have really changed are like the name of this, or this referred to that, or things have got simpler to do. So it's not like the API in major stuff is really shifting. So I feel in terms of investing your time to learn it, I think you're very safe now. Now, we're not going to get into the questions of should you use Gutenberg, or does it apply to this, or can you do that as much? I'm just going to show you behind the scenes how it works. So the goal for today is to talk at a high level about the different JavaScript libraries Gutenberg introduces. As you know, it brings React and Redux and things like that into WordPress core. And then we have to talk about the whole tooling system. We are beyond the age of writing jQuery and, you know, not using compiling tools. So you'll hear me talk a little bit about this, and I'll just say that if you are of the opinion that you don't need to learn ES6 or React, and you don't want to learn how to compile stuff, you need to change your attitude about this. The entire rest of the web world is way beyond where most people in the WordPress space are, and you have no excuse not to play catch up. The cool thing is that Gutenberg already has leveled up the entire system of the WordPress JavaScript code, so it's doing a lot of it for you. We'll talk about how to pull in the JavaScript and CSS. We'll talk about architecture of a block, and then I'll just get into a big repo of demo blocks that you could learn from and download and play with, and hopefully have time for Q&A. Now, to make this a little bit more workshoppy, if you have questions as we go, please just raise your hand, and our lovely assistant here will come and let you ask. The only thing I reserve the right to do is say this is way out of scope. Can you come ask me this later, because it's going to take me an hour to explain my opinions on this, and maybe I'll say can we talk about this at the end? Otherwise, please just raise your hand as you have questions. I couldn't find my glasses before I got on the plane, so you all are an entire beautiful blur to me, so I can't really see too well, so hopefully she'll be able to see for me. A sea of just happy blur. All right. These are the main libraries I want to talk about in terms of what Gutenberg brings to the table. The first one is wp element. So React is abstracted, and what we mean by that is that if we go to the Gutenberg repo and we look under the element library, now you might not be able to understand what all this code is, so I'm just going to explain it to you. It basically pulls in all these elements from the React library, so it loads React, pulls all this stuff in, and then goes through the process of simply exporting it exactly as it got it. So if you've used React before, then you're going to get the exact same things that you would have had before, but you're going to get them referenced as wp.element instead of capital react, but it's pretty much the exact same thing. They're not changing anything under the hood, but this is really smart because if React changes in the future or we want to modify something, the WordPress API isn't going to break, and they can shift things behind the scene, so this is, I think, a really smart approach. The next thing we see is wp blocks. And wp blocks and wp components are kind of similar, so I'm going to try to explain some of the differences between them, but in wp blocks we see a lot of functions that help us build blocks. So when we create a block, we use this function register block type that's stored in blocks. All of the core blocks are also stored in blocks, so one of the cool things that you could do is come into the Gutenberg library, go into blocks library, and here are all the blocks that come loaded with WordPress, and this list will grow and grow as more things are added, and from what I hear, I've never seen this list, but I hear there's a list of more coming, so. The other nice thing about this is that these blocks are built very much in the same way that you would build your own block, so for one of the first times in WordPress we have this really beautiful well-written code base where you can almost come in and just start copying and pasting, hey, how do they get this button here? How do they get to this to this? You just go in, you look at it, you copy and paste it over, and for the most part it really works, which is really cool and kind of a new age in WordPress development. Now, we also get inside of blocks some other helper things, so if we want to add a toolbar or we want to add the inspector bar off to the side, these things that are really tied to block specifically, all of these little snippets of code are built for us in existing blocks. Then we migrate into components, which is a kind of higher level folder, so things that maybe the editor uses or something that's super generic, like a tool tip, a button, a panel, things like that that are more generic are going to be in components, and my guess is that as Gutenberg slowly begins to evolve into more and more of WordPress, you will see this library grow. The cool thing about this is that if you go to start building new eyes, we can rely on the core WordPress UI elements instead of needing to write our own CSS, write our own HTML, and we can really just drag and drop and say, hey, I want a button, give me a button, and it gives it to you, and that's so nice. I don't know about y'all, but sometimes it gets a little under my skin when you install these different themes and plugins, and they're like, yeah, making the whole thing orange with white text would be a great idea, and we'll completely shift all of that into the next slide. So, I think hopefully what we'll see is a more uniform UI, and I'd like to personally encourage you and ask you to follow these conventions and use as many of the built-in components as possible. If you're not familiar with the term component, it's just a little snippet of code that can be reused and usually comes with its own styling, maybe a little bit of functionality built in out of the box. Go-go internationalization. So, I will let him extend on that, but hopefully if you've ever built anything in WordPress, you've used like underscore, underscore, underscore e, or escape HTML, underscore, right? Yeah, I can't really see hands, but yes, I'm going to assume everybody said yes. So, this is super that we now have this on the client side in JavaScript, so we can have all of our JavaScript strings of text be translated super easily. So, you'll see anywhere in a Gutenberg block or in Gutenberg code, you want to have text wrapped inside this underscore underscore function, which we'll look at. Finally, we have WP data and some offshoots of it. And if you've dealt with complex JavaScript applications before, at some point, you need to begin keeping track of the state of the application or what's going on in memory. You can't save every single interaction to a database and reload the page or depend on AJAX calls to save everything to some backend. So, much more is done live in memory. The cool thing about this is that the core team is starting to load up a bunch of data into this automatically. So, if you want the post ID, you want this information, you want that information, you don't have to write your own queries. You can just say, hey, data, give me this. Hey, data, give me that. And you could also begin to create your own data stores. So, if you're someone like the fabulous Yoast team that does crazy complex plugins, then you have this at your disposal to basically keep track of all your application's memory. So, these are the basic libraries. We'll get into most of these top four ones. The data is a little bit more complex, so we probably won't get that far today. And the interesting thing about these is that these are global variables, which should make every JavaScript developer in the room go, that's probably not a good idea. But I've kind of grown to like it, and I think what we're going to see here is an evolution. So, right now, there's a lot of code inside of Gutenberg that just lives in Gutenberg. And little by little, they're going through and pulling this out and throwing it up into npmjs.org. So, now we're starting to get a bunch of WordPress packages. So, the modern development workflow, whether you like it or not, is to import a ton of different packages into your application and rely on those for a lot of the heavy lifting. So, right now, when you want to build a Gutenberg block, you have to access global state. And I'll talk about what that means and show you some examples. Behind the scenes, they're starting to put more and more things in packages, pull them in for you, and then make them available in global state. We will get to the point, and it is starting to evolve to, where if you wanted to pull these in directly to your application, for maybe testing reasons or something else, or maybe you're starting to develop with Gutenberg outside of WordPress, do you all know there's a standalone Gutenberg editor? So, you could just start using Gutenberg on its own, completely outside of WordPress. So, eventually, we'll get to see more and more packages outsourced in this way. And whether we access them through a global variable or packages directly will depend on just how things go. I'm not sure. But I can say, in just the few months that I've been working on this, some of these libraries have been super helpful, like browser list config. So, our Babel files will show you used to be this big, and now it's just one line saying, well, I'm just going to use what WordPress did, and we go from there. So, we have a wp object, and if you haven't played with Gutenberg before, or if you've played with JavaScript and WordPress, you probably have never needed to access the wp object. It exists in the admin, and on the front end, you get a lot more when it's loaded into admin. But if we do wp.i189, we'll get that entire library. If we do wp.element, we'll get the entire React library. Just to show you real quick, what I mean by that is that if I come in to something that's not broken, and I open up my inspector, and I type wp, I get this huge object, which has all of this stuff built into it, including things like element. And when I open up element, this is the entire React library right there in global scope. Now, we probably don't want to be writing out all of this in super long format, so we use something called destructuring. I can't just simply refer to it as deconstructing in my entire JavaScript course. It's torched in here. Okay, there's someone here who wrote a great review of the course and pointed out that I incorrectly referred to it the entire time. But basically, what this allows you to do, if you have something called wp.element.component, you could write it in this way and then just refer to it as component by itself. So at the top of every, or most of the Gutenberg files, you'll see something like this where we just pull out everything that we need so we could refer to it by shorthand. It's not necessary. You could write wp.blocks.richText every time if you wanted to, but I don't think you want to. And I've just been following the conventions of the core devs, so that's what I would suggest for you. So let's move on to talking about tooling and enqueuing. The main tools I'm going to talk about are Babel, Webpack, and NPM. And then we have two new hooks inside of WordPress. You've all enqueued a CSS or JavaScript file, right? Okay. Otherwise, I'm not sure why you're in a Gutenberg development workshop. But hopefully this isn't new to you. And you've used wp.enqueuescripts to pull in scripts and styles before. Now we have two new ones. So enqueue.blockEditorAssets will take all your JavaScript and CSS and load it on every page where Gutenberg is active. I imagine this might adapt as Gutenberg starts to reach into more places, but right now it's just in the editor. Excuse me. So if you wanted to build a block, you hook in your JavaScript using this first hook here. The other thing about Gutenberg is that we want our styles in our blocks to match what they'll look like on the front end, right? So most of the CSS we're going to use enqueue.blockAssets so that the same styles are applied on the front end and they're also applied in the editor so that you can see them both. So what you're usually going to find is that your JavaScript will be enqueued with enqueue.blockEditorAssets and your CSS will be enqueued with enqueue.blockAssets. There's some variety to this, but in general it works that way.  So there's some discussion around how is the or what's the best way to approach this and I think it's still being talk to Tammy later, she'd be happy to give you some insight there. It's being evolved, but just be aware of that when you start styling for blocks and of course you want your theme to make all the blocks look the sexiest, right? So you might have to do some enqueuing on the back end as well for that. Or you might not. Let's talk about tools a bit. So in this plug-in repo, again, if you go to Hootenberg here, here are the slides, and then this is the plug-in repo. So you can get all the stuff I'm talking about here. Put all your blocks in a plug-in and that's what this is, is just a plug-in. Okay. So the first thing we'll start with is Babel and what Babel will do is take advanced JavaScript that cannot yet be processed in the browser and rewrite it in a way that can be processed in the browser. It'll also allow us to write HTML-like text in our JavaScript and translate that back into CreateElement and CreateTextNode and all of these things behind the scenes. The first version of this course, interestingly, was if you look at the Babel file for this, notice that we used to have to set up all of this. Actually, I've ripped out a bunch of stuff here. There should be more in here that lists exactly what kind of browsers you wanted to support, how far back you want to support, and you have to tell it that, hey, we're going to be using JSX and use WordPress's JSX to help process this and some other stuff going on. But because WordPress is pulling these out as separate packages, I can now in my configuration say, hey, just use the WordPress defaults. And this is really cool because what I imagine is over the next year or two we will be able to stop having to determine in independent agencies what our standard should be, and we could just say, hey, we're following the WordPress standards of what browsers we will support. I don't know if that's going to work for everyone, but I imagine for a lot of people it will take the load off to just blame WordPress or thank WordPress, depending on your opinion there. So more and more configuration files are going to get simpler like this. They're working on ESLint. There's an editor config that you could borrow from WordPress as well. Our Webpack, I would take the rest of the time to explain this, but basically I like this Webpack file. If you're interested in Webpack, come talk to me later and I will wrap about this with you. Why are my... This is all white. Hold on a second. Live coding examples. Let's try that, shall we? No. That's strange. Okay. I'm going to go dark on you guys because at least then we'll see it. So, I'm going to go back to the things like parentheses and curly braces are kind of important for our code. So what this is basically going to do is run through all of our JavaScript files, all of our SAS files, compile them down, and then it's going to take anything that's named editor and anything that's named style in terms of this could be SAS or CSS, and it's going to save them into two different files. So everything that's labeled as editor will be saved to a block.editor.css, and everything that's saved as a style will be saved to style.css, and then those will be loaded by our plugin on the front end and in the back end, depending on how they're named. The other thing that we'll see is a lot of these plugins that you build with a bunch of blocks will have an index file that will list all of the blocks. So this is kind of what we tell Webpack is our entry file, and then it will go from there and start finding everything. I'd really like to encourage folks to follow the naming conventions that WordPress Core does. I've seen a lot of, like, examples and kind of generator stuff that name their main block files as block.js, and I think this is a really bad idea because if in the React JavaScripty Webpack world, if you name something with index.js, I could just refer to the folder and it will find it automatically. If we're naming our main index.js file, block.js, we have to manually refer to it as block.js. The other thing is that if we look at any of the core Gutenberg blocks, find a more complicated one. Notice that they have an index.js file and it's going to be the registration of the block, and then they have a block.js file where they break out any real complex stuff for building out the edit functionality. So if you're naming your main file block.js and then you pull out something else, like I just, I don't love that pattern. So please name your stuff, follow the general React JavaScript naming conventions of index for your main files. We have a Webpack and then we have NPM. So as I mentioned, the conventions today are to reference a bunch of external things, pull those in and then run your code based off of it. Not all of these are necessarily needed for every project, but this is what a fairly common block project will look like and I gave it to you in the spoiler plate. Some of these are dealing with the transpiling of your JavaScript. Some of them are dealing with processing your JavaScript and then we'll see more and more like this that are at WordPress something that are pulling in WordPress conventions. Tooling in three minutes. So now we can start talking about the actual architecture of a block and then we'll jump into some examples. How many of you here have coded a block before? All right.  So right now most of the blocks are being registered in JavaScript. What that means is that WordPress builds your whole site, it launches the editor into the browser and then once it's in the browser it figures out what blocks are available and it makes those blocks available. It's pretty slightly problematic and there's been a lot of discussion about should blocks be registered on the server side. If we are building blocks that WordPress is not aware of at the server level then there's some things that we cannot do with them very easily like how do we figure out based on user permissions what blocks we should display or how do we figure out what are some other things? There's some security stuff. Post ID, so there are some things like how do we figure out the context of stuff that's going on. Actually post ID is now available in the data store so you can get that on the JavaScript side only because WordPress is going on the server side pulling out all the data feeding it up into the JavaScript data and making that available. But in general there are some things that you do need to register a block on the server for. So it's kind of weird. We have register block type camel case and we have register block type underscored for PHP. They both exist. This is a little confusing and there's probably going to be an evolution of naming conventions where you could use register block type in your JavaScript but if you're registering a block on the server you'd call register block type in your PHP and then in JavaScript you'll still need to add some things and maybe it'll be called extend block or do something more. So that's what we heard exactly what this will be but there are discussions on this. Now you don't, in a lot of cases you don't need to worry about the server side and the client side will work fine. The other reason that they're not going to pull everything out into the server is that Gutenberg ideally can still work as an independent editor completely outside of WordPress so if you're building your own app and you want to use this editor then you could use it and you could have inside of your app. As I mentioned register block type takes two parameters, a name and settings and these settings are as follows. I think I have them all here. We have a title, a description. Right now there are four or five set categories but I believe that categories will be extended so that if you have a suite of blocks you can kind of have them visually available in one area. I really love that you can add your own SVG icons so if you have any icon you want and just convert it into a react style SVG, throw it in there. Keywords, you have a bunch of different ways to search for block and I'll give you a little hack because you can only have three keyword strings but it will search for independent words in each one of those strings so I'm not encouraging this but I am letting you know it's possible you can completely like black SEO it and throw in a billion keywords I think and it'll find it easy. There are some things that you might want to turn on and off for block functionality for supports. There's not too much to this now but if you use Gutenberg for example it spits out an automatic class name on the front end to all your blocks you could turn this off. In editor there's always a little CSS class names thing where you could add CSS to your block. You might want to turn this off for certain users because they might start looking at you and they might want to turn this off. There's the ability to say hey don't let somebody edit the raw HTML of my block they're going to mess it up so you could turn that off things like that. Attributes are probably for me the most confusing part of Gutenberg when I got started but basically when stuff is stored oh man yeah we don't have time to explain attributes in enough depth. The basic aspect is anytime you have to keep track of that and it took me a while to figure out exactly what that model was so just rest assured that if you're sitting there like it's going on you're not alone. That is my experience too. And then the last two are kind of the heart so these last two are the biggest part of your block where you have to code out all edit functionality for your block. Like I said you can get components you can say hey give me a button to upload thing but you have to say what happens when they upload that image right do I display that image do I put it here what do you want to do with it when they click this button what's going to happen so you have to code all of that yourself which is following the React format it's not too bad but the edit setting is going to be where you code your entire block in the editor. The same setting on the other hand just determines what format it's going to be. Most blocks will write your save function in javascript there are a few blocks where if you're keeping track of metadata or something that can be changed in other places in wordpress while that data is stored in the database it actually saves as blank data in the database and renders with PHP so sometimes the save function is completely empty it really depends on what you're trying to do but for the most part this is all going to happen. This is a very limited view of it but this is what an average block registration will look like so take a moment to look this over you'll see a few things here like underscore underscore that means that my block can be translated and you'll see that the edit right here is returning a function actually so it can be called and executed and this block will do nothing in fact probably this would not be a worthy block to do but it gives you an idea of what you're going to see as we move into the demos. So that's all of Gutenberg development I have a course on this by the way where I take about three hours to go over what I just said so if you're interested in any of that in more depth you can check that out and I'll just pause real quick does anybody have questions about stuff before we jump into the code at kind of a high level here? Nailed it. All right so the most basic and probably least helpful block you will be able to build is what's called a static block and this is one that is completely uneditable and I'll just kind of show you real quick what this would look like here we have a block you can't edit it you can't do anything you can add some css but this is all just hard coded so what are some use cases for this maybe you have a site where you do want to hard code blocks maybe it's a warning maybe it's something that just doesn't change for some reason this would be what it looks like more than likely you're not going to build these however your static block is going to be your basic starting point for whatever you do so there are some things in here I just want to take a moment to go over that you probably won't build but sort of form a foundation for everything else you would want to do I like to break my blocks up pulling in files in two different ways so I usually try to put block dependencies which are my own code first so in this case I'm importing an svg icon so I have a file here called icon and I like the noun project so I just copy my stuff from there and use that and what that is is if I search static see the little lego lego and so this is what obviously this is a lego svg icon right you could all tell so I have that file being pulled in and I'll show you where I use it in a moment and then I pull in my front end styles and then my editor styles and most of the time like I said you won't really be having styles that just go on the editor side then we come down into word press dependencies so the next thing I do and notice that this is using import right because these are separate files that javascript is importing then web pack is going to go through and pull them all together with all the word press dependencies we're using that destructuring so it's not actually importing anything so we could leave all of this out and everywhere in my code where I have underscore underscore I could just type wpi 189 like that and remove this line of code okay so this is purely for convenience factor only but I would highly encourage you to do this up here so anything that you need from word press you're going to pull in up here and what we'll see is as we get into more complex things this list grows right so that's going to get bigger then we come down into our actual register block type there's two parts to a name you have to name space everything and then you have to give your block a unique name what's cool about this is that later on in word press it's going to do something like is it blocks player blocks singular it's going to apply a CSS class like this to your block so at some point you'll be like where is that name coming from and it's coming from basically the hardcore name your block you'll also need to use this as you get into some more complex blocks sometimes you need to register a block on the server side and you follow the same naming convention I don't actually know I haven't actually tried I should can you put multiple namespaces in a single block or in a single plugin like can I have some blocks with one namespace and some blocks with another okay we'll find out one day I name them all the same I imagine you can but in general this is going to be like your plugin name or something like that generic we have the title and description I'll just show you real quick where these show up in a block if we I really love that they added hover over blocks thank you design team that's so cool if we click into a block notice this description over here and we've got the title and when you're searching for a block you're going to see all the names of the blocks here so that searchable name is what the title is and the keywords here I'm going to jump down to notice that the word banner is nowhere in my block title but if I search for banner you all know this you watch the word camp us talk that that is going to be there so the next thing we have down here is category like I said there are a few default categories we've got common blocks formatting layout widgets and then we have our embedded blocks so you could define where you want your block to go sometimes it's a little confusing where your block should go and then my understanding is that eventually this will evolve I think so that you could have my blocks category and set up your own one but right now you cannot icon a little just sex for you this is basically just referring to the icon that we pulled in before your keywords and then we get down into the actual edit functionality so I'm going to skip at this point to a more complicated block that allows you to edit stuff like this one I've got five minutes left so hopefully I can get through this one example form fields okay so here's a block with a whole bunch of editable stuff you'll notice that it's got form fields here and it's got form fields here and they're mapped to each other you would never do this right but just for demo purposes to see how this would work so in a more complex block like this I'm going to break out functionality into different places so if I come down to register wait notice that I pull in this thing called edit and when I go to do to load like an inspector I'm going to pull in these other files and so at this point I've taken all the edit functionality and broken it out into its own file so eventually you're going to get to the point where your stuff becomes so complicated that you need to start pulling files out separately and this is where you would do that I'm also doing a lot of things every single one of these form fields is a built in native component in wordpress so I did no custom coding here to do any of these things and I think that that is so cool that we get all of this out of the box so I'll just show you real quick remember I said that there's the components library you can go into components and I've got a checkbox control a radio a range a text all these different ones set up and then I just want to show you one last thing here in how they work so whenever we have a form element or something like that there's some baked in like labels and things like that and then there's a better one the color palette we pass in a default color palette so if they've already saved this block we load that value automatically and then this is the very reactive way of determining how a block should be handled on change and this is probably going to handle like 90 percent of your save changes well maybe less depending on the complexity but you basically say hey every time this color palette gets changed take the new value go back to my list of attributes or changeable elements and give me the new one so this is this is all the wiring you would have to do to make this entire thing work all you have to say is appear give me the color palette and then down here set the default value and then determine what happens on change and you'll see some of these are more complex like you have to set some default options but they all follow this basic convention of value and change so at this point we have two minutes left and I wish that I could take you know like I said many more hours the course I have is about five or six hours long and you could go through that for more on this I'm also happy to answer questions afterwards I'll put this up here for anybody who's interested and then I don't know probably not time for questions or maybe one or two or let's do this come find me afterwards if you have questions I'd love to chat more about goonberg and thank you all so much for the opportunity to come here and talk with you all


Jeff Saturday in this position. Okay, have you heard all of the chatter and all the noise? It seems like everybody's against Jeff. Not me, not me, I love Jeff. Jeff has all the experience in the world to get it done. Jeff was one of the greatest sinners to ever play the game. If you play sinner, okay, that means you know front seven. Okay. That means you know what the quarterback's doing. That means you have to understand defense. You know why you have to understand defense at the center position? Because you got to protect the quarterback. You got to be able to sit there and say, that nickel's coming. So let's slide, protect, let's go. Scat protection right here. I need my runner back there. A lot of times these sinners are more savvier and have higher football IQ than quarterbacks. Why is it that the quarterbacks always get the offensive coordinator jobs or the head coaching jobs? Oh yeah, Brandon, well the last time he coached he was in high school when he went three, eight, and one. I don't care. That doesn't matter. Jeff Saturday is more than ready.


Thank you, Marianne, for that. The Director-General of the National Youth Service Corps has commended the thousands of core members working as Atrox staff of INEC for the Anambra State Governorship Election. Brigadier General Shai Bu Ibrahim has described them as patriotic. The NYSE Director-General was on tour of polling units across the 21 local government areas of the state. He said his visit was to monitor and ensure the safety of core members working in the areas. More than 5,000 core members were deployed for the governorship poll.