All right, well, let's get started. First, thanks everybody for coming out. My name is Ryan Tidwell. I'm with Hewlett Packard Enterprise. Today we're going to talk a little bit about some of the work that's gone on in Neutron over the last couple cycles. Talk about specifically dynamic routing and its applications in the cloud and what we've accomplished so far. And hopefully you'll get a chance as well, time permitting to walk through a bit of a demo to show you some of the code that we were able to get into Mataka. So to start off, I just kind of wanted to set the stage and kind of motivate the problem that we're trying to solve. And traditionally when we've thought about open stack deployments, I'm going back to even the Nova Network days before Quantum Neutron, there's kind of been this thinking that I have tenant networks, they're private networks, and all of the fixed IPs that I allocate on those networks might just hide behind a floating IP for north-south traffic. And of course I don't worry about floating IPs for east-west, because I'm just going to put everything on the same network. As things have evolved and people have tried to put open stack into production, we've found that that model doesn't quite work. And so we've evolved to things like DVR and Neutron, which will actually route your east-west traffic in a distributed way. And so we've kind of moved beyond that. But the key thing is that for north-south traffic in and out of the cloud, your best bet has always been to kind of hide behind a floating IP or SNAT, but I want you to imagine a world where floating IPs and SNAT don't exist. And when you think about IPv6, this is sort of the world that you live in. We don't support floating IPs for IPv6 and Neutron. So in that case, how am I going to route traffic down to my VMs? They're not hiding behind a floating IP. I might have subnets, prefixes that are popping up in my cloud that I need to route. And there's ways to do that. So this is kind of where we start off the talk. And you have a couple of options. Now for traffic egressing the cloud, Neutron kind of takes care of that for you, at least on the outbound flow. As I've outlined here, Neutron will carry the gateway IP for the external network as an attribute. And that gets programmed in your Neutron router as your default gateway. And so that's great, because when my VMs start sending traffic out of the cloud, that all gets routed upstream just fine. I'll say I'm trying to open a TCP connection, something bi-directional. That traffic needs to make it back to my VM. And this is kind of where the problem we've been trying to tackle comes from. And of course, this is not a new problem. Since the dawn of IP, people have been puzzling over this. And we've solved this problem basically long before I came into the industry. And it's a basic IP routing problem. Am I going to statically route traffic, or am I going to use a routing protocol to advertise subnets in their next hop? So we can statically route traffic, which involves putting manual entries for next hops in the infrastructure. Now if you've got a small cloud, this probably might work OK for you. If you don't expect to route a number of prefixes, and you kind of expect them to stay static, a routing protocol might not necessarily be for you in your deployment. However, if you have a more dynamic environment or you're scaling up a little bit bigger, this approach requires operator intervention any time a neutron router is created, any time a subnet is instantiated behind that neutron router. Conversely, as the subnet is deallocated or the router is destroyed, you might want to go clean up those routes. So there's some operator overhead involved there. Or there's dynamic routing. Going to configure a routing protocol of some kind that's going to have your routers advertise, talk amongst themselves, and kind of negotiate where next hops are for these subnets. And this is the functionality that we've been working to bring to Neutron over the last couple of cycles. And as kind of a philosophy, what we wanted to do is try and isolate the Neutron L3 agent from these changes. And in thinking about this, BGP kind of seems like a nice place to start. It allows us to kind of separate the control plane and from the data plane, in this case being the Neutron L3 agent and the namespaces the routers operate in. So what did we get into Metaka? Where are we at right now? So what we have right now is the ability for Neutron to announce next hops for tenant networks that are behind a centralized router. And I apologize for that formatting. I didn't quite get that cleaned up. You get the point. So we can advertise next hops for tenant networks that are behind a centralized router or a CVR. We can announce next hops for tenant networks that are behind a distributed router, DVR. You might have heard of that. We can also announce host routes for floating IPs. And this announcement is compatible with both centralized routers and distributed routers. And I'll talk about how that all works later. I've got some examples that will illustrate that in action. But before we go too much further, I just wanted to give a couple of shout outs to a couple of developers who weren't able to attend the summit this time. Vikram from Huawei and Neumann from Red Hat contributed significantly to getting this work done in Mitaka. So shout out to them. So the architecture. What you get in Mitaka is a service plug-in that runs inside your Neutron server. So this is going to run alongside your L3 plug-in that you use on the Neutron server that exposes all the L3 APIs. And then what you're going to have is a dynamic routing agent, or the DR agent you might see it referred to. Inside the dynamic routing agent, we've exposed a driver interface. So the back end BGP implementation is pluggable. So in Mitaka, our reference implementation drives a Ryu BGP speaker. The interface is there so we could in the future bring in support for Quagga or Bird or name your BGP speaker technology. And the dynamic routing agent, like all the other agents in Neutron, communicates with the Neutron server, gets its directives via RPC calls. So this is a simple thing you can do. This is not prescriptive, but just to illustrate one way you might deploy this in practice is this figure right here. So I have two autonomous systems that I've configured. One of them, the one on the bottom, is an autonomous system that I use for my cloud. And you can see the dynamic routing agent there, which is communicating with OpenStack via the Neutron RPC over RabbitMQ.  And it's opening that peering session with your infrastructure router in another autonomous system and exchanging route information over that peering session. In this case, this is an EBGP peering session. We can support IBGP as well, and that's no issue there. So now this is where things get interesting. What routes are we going to announce? And that's a good question to ask, because we don't want to create a BGP speaker and then be announcing prefixes that we have no business announcing. So we want to scope that and give operators a way to control that somehow. So as I mentioned, we can announce host routes for floating IPs. And prefixes that need announcement are discovered. And I'll talk about this in more detail. I've got some slides that'll walk us through it. And I have a demo as well. Prefixes that need announcement are discovered by associating a BGP speaker with an external network. And what happens is on that external network, we look at all the neutron routers that are connected to it. And then from those routers, we go and look for subnets behind those routers that need announcement and announce the neutron router as the next hop for those subnets. Now the scoping of what we announce through any given BGP speaker is heavily influenced by address scopes. So I think it would be useful to just touch on address scopes real quickly, because they're kind of fundamental to the way that our implementation in Metaca discovers routes that should be announced. If you're not familiar with address scopes, Carl Baldwin and he gave a great talk this morning. I didn't even pick that up on YouTube, I encourage that. But I'll touch on it briefly here. So this was really had the bow put on it in Metaca. And the idea is that subnets and neutron can be scoped and we can enforce uniqueness and kind of communicate the notion of a routing domain inside neutron. And if you think of what an address scope is, think about your home network. So residential broadband connectivity is probably a good context in which to try and describe this concept of address scopes. You have the public internet, which your internet service provider is bringing to you. And connecting to your home router, you typically get one IP if you're running IPv4. And behind that router, you're addressing your network inside your home, <IP_ADDRESS> or something with a 10 prefix, something that's not routable on the public internet, but is routable inside your home. So your home router here kind of creates this barrier between address scopes. And the way you traverse those address scopes is through NAT or port forwarding or tools like that. So this is kind of a quick overview of address scopes. And in this diagram, you're going to see three tenant networks. And two of these exist in the same address scope as the external network. And there's a third here that's in its own address scope. Now what gets announced by Neutron is the two subnets that exist in that same address scope as the external network. So in this case, 1521.64 and 1693.64 are going to get announced, but their next hops being the Neutron routers that they're connected to. And then on the bottom right, that tenant network doesn't get announced by our BGP speaker because it's outside the address scope. All right. So let's move on to announcement of floating IPs. With Mataka, this really isn't quite as compelling of a feature, but we went ahead and implemented this because if you're familiar with the routed networks proposal that's being worked on right now inside Neutron, this is going to become more important and more useful to you in that context. So the idea here is we're going to announce host routes for your floating IPs. When we get to routed networks in Neutron, the floating IP range isn't going to be bound to an L2 segment. So your floating IPs actually float across L2 domains. And the next hop for that floating IP may not necessarily be at a Neutron router. It may actually be right directly on the port where your fixed IP lives. And so the way you announce the next hop, it would be different. But we wanted to at least lay the groundwork in Mataka to do these announcements. And an important thing to point out here is if you don't want your network to be flooded with host routes from these floating IPs, that can all be toggled on your BGP speaker. So you can suppress announcement of host routes or even tenant network routes if all you want to announce is host routes for your floating IPs. So there's knobs that you can turn inside Neutron to kind of tweak what gets propagated through BGP. All right, so before I go into the demo, I want to kind of explain the environment that I've created here. And so you kind of know what you're looking at as I walk through it. So what I've created here is a Neutron router with three tenant networks connected to it. It has an external network, and that's connected to a Quagga router that I have in my environment. I have a Neutron dynamic routing agent, which I'm going to show you. I'm going to actually make it peer with that Quagga router. And you'll see the tenant networks get announced and see them show up inside Quagga. So let's start here. So I've gone through the trouble of kind of creating all these resources and kind of putting things mostly together so we don't have to do that here. If I run a Neutron BGP speaker list, this shows me the BGP speakers that I've created in my system. Now, an important thing to note is that this is an admin only function. So this is not exposed to your tenants by default. So this is something that admins have access to, but your tenants do not. So typically, as an operator, you might set this up on behalf of your tenants so their networks can get announced in your network. So you can see that I've created it with a local AS of 1, 2, 3, 4. So that's the autonomous system that I've created for my cloud, basically. And this BGP speaker is speaking IPv4 routes. We do support IPv6 as well, just as a note. So we've tried to keep support equivalent for both address families. Complete that command. So if I run a Neutron BGP peer list, this shows me all the BGP peers that I have available for my BGP speakers to peer with. So I have a peer IP. Again, that could be IPv4, it could be IPv6. Our reference implementation supports peering purely over IPv6 or over IPv4. And I've configured a remote ASN of 4, 3, 2, 1.  So if I just want to show you all the BGP-related CLI commands that exist, I'm just kind of gripping for anything BGP. Now you can also take a look at what routes you expect to be advertised by your BGP speaker by running this command. So now notice that in the previous slide I showed you, I'll go back to it. I had three tenant networks that I configured. But if you look at what's getting announced, you only see two subnets getting announced. Now why is that? This comes back to the discussion around address scopes. So these two subnets that you see getting announced in this list were created not only in the same address scope as each other, but in the same address scope as the external network that their router is connected to. So Neutron sees that and says, OK, this is all part of the same routing domain. And I can announce these. But there's that third tenant network, which is not part of the routing domain, so it doesn't get announced. So that's why you only see two subnets here. So I'm going to switch over here to my Quagga router. I'm going to show you what that configuration looks like. So on the Quagga end, and then this sort of configuration would apply across vendors depending on your router technology. But essentially, I've defined my BGP process that I'm going to run on my Quagga router. I've set my router ID, and then I've also configured my neighbor. In this case, the neighbor is the IP address of the system where our dynamic routing agent runs. So this is where you're going to expect Neutron to be peering from. And the remote AS is 1, 2, 3, 4. That's the ASN that I'm using for my cloud in this example. Notice I have the line here which makes the peering passive. We're not familiar with Quagga. What this does is this is going to keep Quagga from trying to reach out to that peer and initiate a session on its own.  So now let's go into the Quagga interface here and actually kind of poke around and see what we have. So now notice I don't have anything in my routing table except for the networks that are directly connected to me. Why is that? I don't have an active peering session open with my neighbor. So we're going to go ahead and change that. So we're going to get this going. Now once I, so I'm going to flip back to Neutron here and run some commands in Neutron. So we've created all this inside of Neutron. We've created the BGP speaker. We've associated that BGP speaker with an external network. And so there's these commands here. BGP speaker network add, network remove. This is where you do the association of your BGP speaker to your Neutron external network. And then this is where you remove it as well. So because I didn't want to worry about remembering UUIDs and all that, I've got a script here. This is what I'm going to run. I just wanted to show this to you. You don't think I'm cheating. I'm going to run a Neutron BGP DR agent speaker add command. Now what this is going to do is this is actually going to take that BGP process that we've constructed inside Neutron and tell it, go run on an agent. So just because we've configured all this in Neutron doesn't mean it's actually up and running it. We actually have to tell it to go run on a specific agent. We have to schedule it manually. So if I do a Neutron agent list, that's a little hard to read. Trying to blow this up so everybody can see it. This shows me all the Neutron agents that are running. And you'll notice that there's my BGP dynamic routing agent phoning home to the Neutron server saying I'm alive. So what we're going to do is I'm just going to go ahead and run that script, which is just going to associate the BGP process we've defined with an agent and get everything up and running. So you notice the second thing I did after doing the association with the agent is I listed the BGP process that has been scheduled to the agent. So I can go agent by agent and say, well, what BGP speakers do I have running here? And it'll show me that, which is kind of useful. So this has given us time for the session to open up. And we should have, yep, cool. So Quagga is now saying that I've got a neighbor, a remote router ID of <IP_ADDRESS>. That's my Neutron dynamic routing agent that has now opened a peering session with Quagga. Cool. Now let's take a look at what we've announced over that peering session. So now you'll see anything tagged with B, if you're not familiar with Quagga, that first line right here is something that was learned via BGP. So you can see we've got two routes that were learned by BGP, <IP_ADDRESS>.24 and <IP_ADDRESS>.24. So this is Neutron over a BGP session telling my upstream router the next hop for these tenant networks, which would be the Neutron routers, or the one Neutron router that I've created in this case. So that was BGP in action. So we've got some work ongoing in Newton. We have some enhancements we want to make and features we want to add. So there's a couple things in flight. First of all, when we first created this code, it was kind of a little bit of a slow process to get it into Neutron. We had to get core viewer attention. They've got a lot on their plates. This involves a lot of code that lands in the Neutron repository that's just dedicated to doing dynamic routing. And so there was kind of an idea floated, well, hey, why don't we spin this out? This is a great candidate for something to live in the Neutron stadium. And this was kind of in the middle of the Metaka release. And we said, well, that's kind of disruptive to basically none of this is going to make Metaka if we get kicked out of the Neutron repository right now. But we decided the expedient thing to do was to land this code in the main Neutron repository. And then in Newton, turn it into a stadium project. So that's the work that we're starting with in Newton is trying to move the code between repositories, which could slow us down a little bit with getting some of the features that we'd like to get completed into Newton. I'm hoping it's just easier than I think it is. But beyond that, we'd like to enable announcement of host routes for fixed IPs when you're using DVR. And the reason that you might want to do that is right now if you want DVR treatment for your north-south traffic with IPv4, you have to attach a floating IP to every fixed IP you have. And you may not want to do that. Remember, we're living in a world where you may not have floating IPs. You may not want to use floating IPs. I'm going to give people choice. Unfortunately, the only way to get traffic directly to a fixed IP with DVR is to send it through the central SNAT node and use that as the next hop. So your north-south traffic isn't really getting the same DVR treatment that your east-west traffic is. The reason that we didn't enable these announcements in Metaca, I mean, I have the code. I just didn't push it because there's still some kinks to work out with DVR inside the L3 agent. It turns out, as we're starting to look at this just over the last couple weeks, these fixes are fairly simple. And I do expect them to land easily in Newton. So once we have that, then we'll go ahead and get an announcement for fixed IPs going, such that you'll get north-south DVR with both IPv4 and IPv6 if you use BGP. The next place we'd like to go with this is to use BGP to configure L3 VPN and then use EVPN technology. We're still hashing out what that looks like. And I expect a blueprint to show up shortly that we're going to start iterating on. So those are our areas of focus for Newton. And just going back to my comments about DVR, I just wanted to illustrate it with a slide specific to that, just so you understand what I'm describing here. So what I've shown previously is that the next hop that we advertise through BGP is the gateway port on a neutron router. Well, in DVR, that doesn't really give you distributed north-south data flow. You don't get fast exit on and off the compute node because the route says send this to your central SNAT node. So what we're talking about doing is with DVR, we call the floating IP namespace, or you might hear it called the FIP gateway FIP namespace, it has an IP on the external network. And that is what you really want to use for your next hop when getting to these nodes. And so that's what we want to advertise through BGP for these nodes. So if you look at the routing table here on the right side of the screen, that's what you would expect in this scenario. So we'd be directing traffic directly to the compute nodes, bypassing any centralized network node or neutron router. Not supported in Metaca, but easily supportable in Newton, as long as we can get the associated fixes to DVR in, which, as I mentioned, seemed fairly trivial. So with that, I want to leave some time for many questions you might have. My phone's up here at the front. So thank you. Yeah, I just have a quick question. Does this dynamic routing with BGP, does it work in a triple O scenario? So if I'm running Metaca over Metaca, and my overcloud is doing, I'm trying to do BGP advertisement on my overcloud. And with my base cloud also doing BGP dynamic route, how do I make my inner, the overcloud tenant network get advertised out? That's a great question. So I guess the short answer is where are you using neutron routers?  So I mean, you could use BGP in both, I suppose. You've got neutron routers in both your overcloud and your undercloud, and you could have a BGP speaker in your undercloud and one in your overcloud. That's actually what you bring up as a scenario I've never considered. I'd have to think through that a little more, but that's the best answer I've got for you. About the BGP agent HA, if you manually associate this and you configure the IPs on the router side, is it possible to do HA here? And what's actually the impact if the BGP agents go down? Right, that's a great question, HA. We get that all the time. So given the tools we have to work with right now, I think the answer is to deploy multiple agents. So if I just need one BGP speaker, want to do some HA, maybe I might deploy two or three agents, depending on what level I want there. You can actually schedule your BGP speaker to multiple agents so you can be peering to all those routers from multiple places. So if you lose one agent, you still have a place where all your neutron routes are getting announced from. So that's the best answer that I've got for you right now. When we've got BGP to open stack, can we finally accept provided independent addresses of our customers? Can we road some random addresses directly to virtual machines? Can we give some tenant a via system for users to self-serving own IP addresses? Yes, yes, absolutely. So the way that would work is as the operator, you'd lay down your external network. But when you go to address that in neutron, what you want to make sure is that you're allocating those subnets from a subnet pool in neutron. So if you're not familiar with subnet pools, maybe go take a look at that. Because the addresses that you use here with dynamic routing need to come from an address scope. Yes, but users can have own provider independent addresses. PI allocated addresses not linked to register. That's right. So yeah, I'm getting there. So your tenants could then create their own subnet pools. Bring own addresses. And then allocate out of there. They could fill their subnet pools with whatever prefixes they want. And those would get advertised.  Every prefix they own, yes. Absolutely. Yep, yep. You can support that today. Thank you. It's about mobile IP. Can you tell me that the neutron, the agency or committee that supported mobile IP in the few days of the unit map for mobile scenarios supported in the neutron? Can you elaborate about what you mean by mobile IP? Mobile IP is when the tenant, when the VM with the public IP detached from the dedicated router and the mode gets the motivation to the different routers. Support the mobile IP, support in the current IPv6. I haven't considered that. When the IP changes, it can't be routable. Maintain the section, this route, continues the section, and must the IPv6 router can't be changed for the routable, for the outside of the internet. I have to apologize. I'm having a hard time following the scenario here. The IPv6 has a lot of solutions for supporting the mobile IP. I just wanted to ask if the neutron committee can support these advanced scenarios. Maybe let's chat offline if you've got a minute, Tom. I'd like to understand that. So a great demo. Thank you. Good, cool work. Do you, does the MATACA implementation that you have right now, does it support having two peers, two external peers, or multiple external peers? So yeah, great question. So you can actually configure a BGP speaker to pair with any number of peers you want. Any number, OK. So for simplicity of my demo, I just did one. I did one, understood. But you can peer with whatever you want, however many you want. And in fact, when you define the list of peers in neutron, those are actually reusable. So you don't have to define a peer set for each BGP speaker. Got it, got it. Can L3 agents peer to themselves, peer to each other? So right now the functionality that, so first of all, you mentioned L3 agents appearing with each other. So the L3 agents actually don't peer. It's actually the dynamic routing agents. So the BGP is kind of happening a little bit out of band. OK, got it. Now to the second part of your question, do they peer with each other? All we've enabled in MATACA is the ability for neutron to announce, not to consume routes. So you're not really, we don't support peering with other agents yet. Understood. I'd be curious to understand your use case for that, because that sounds like something we could enable for you. Kind of like a connectivity broker between tenets. Sure, yeah, absolutely. Kind of a model. But OK, very cool. One other thing, this addition of BGP state that kind of that, or how much BGP state is the dynamic routing agent keeping on this BGP peer, and how much workload is that adding to the overall dynamic routing agent to the L3 agent? Did I just reduce the scalability of the dynamic router paradigm and control? So it actually, I don't have numbers to back this up and to quantify it. But the overhead is actually pretty lightweight, because it's not in the forwarding path. So really the resources you're consuming are just the resources needed to maintain a peering session, just a TCP connection basically. So those are kind of the resources it consumes. Very cool, thank you very much. Sorry, ladies and gentlemen, we are out of time for this session. Any questions can be done on the hauler side stage. Thank you.


As pivot translation is becoming more popular, I'm wondering, do you have any thoughts on how we can improve it? Obviously, the best scenario would be to always have subtitles who translate directly. In this case, a subtitle who knows Arabic. There are subtitles like this, so very often companies have this excuse that subtitles are difficult to find in language combinations, other than English, which is not true in many countries, especially in Europe, but also for many Asian languages. We do have subtitles, so that would be the best scenario. But obviously, it's not always possible. There will be cases and there will be language combinations where it might be impractical to find a subtitle to translate directly. So we will be using pivot translation. I'm wondering, do you have any thoughts on how we can make it better? What could we do? I think we should start with raising awareness among clients that a direct translation is just bound to be better. And a pivot, try as you might, will contain a double share of errors. Every translator introduces some share of errors, none of us are perfect. And if you have two translators or more in a chain, then this will rise exponentially, kind of, because you are going to add your own share of errors and copy the errors of the previous translators. So that's one thing we should do. If pivots are, and I understand that in some cases, you can't escape a pivot. There are some exotic languages from, again, exotic, maybe that's an unpolitically correct thing to do. But from our point of view, like Poles, there are a lot of countries that are far away. And these are languages linguistically exotic, for instance. They share, they don't share, they share very little with Polish, like different grammar and so on. Different forms of address, for instance. So, if you have to work with a pivot and you have to work through English, basically good annotations are. Our key that is signaled to the to the to the to the end translator that this this kind of phrase functions in that particular way in your language, or that this is a humorous phrase or that this is that this is, say, a cultural thing. Again, show guides would be nice, like in the material I'm analyzing. There's Arabic or the Arabic language is a specific in that you have local vernaculars and you have modern standard Arabic, which is like the high variety of the language. And the main character is is a university professor. And he mixes this modern standard Arabic, the literary Arabic, the language you write, literature and poetry in it. He mixes it with this local vernacular. So he sounds more intellectual, more educated than some of the other characters. And this is a piece of information that the that is not included, I think, in the pivot that the translator did not receive. So once, you know, you have this, you have this character and the language is marked in some way. You don't usually as a translator have an idea how to mark your text in the target version. So you can visualize it this way. There's like this amount of marked text, you translate it, you leave some of it. So you end up with this amount of text and that's marked. And now you end up, you think that you end up with this and this is your source. So you have even less and because you don't realize that now it's more subtle, for instance, the market is more subtle. It's less frequent. So you do not realize that, you know, the language is marked or you've not always realized. And this is something, again, maybe a show guide could solve some peculiar ways of speaking. Same thing with humor. Humor is very often lost in translation. You have an idea how to translate a few jokes, but against some, you're powerless. OK, and they just end up flat or you omit them. And now the next translator has only the few jokes that you did translate to work with. So a comedy, for instance, loses a lot if translated through a pivot. If you think about it this way. So one thing is, again, in the form of the translators, maybe of the role of the specifics of the movie, like this is a comedy, this is bound to be funny. Humor is conveyed through such and such and such means, crank it up in your language, if you can.


Hi, everyone. Thank you, everyone, for being here. I'm very happy. If you have been in the previous talk, I'm going to be the old lady who talks about TensorFlow 1.x, but still this will be applicable to Python, to TensorFlow 3. I keep saying Python because it feels like I'm talking about Python 2 while we should be moving to Python 3. So, yeah, nevertheless. So I'm my human F. I'm a PhD student. So we are going to talk about pain. As a PhD student, I know pretty much about pain and how to deal with painful. I'm actually a last-year student, so that means that I have been dealing with pain a lot of time.   So, TensorFlow is an amazing library because you have tons of ways to do things, but this is the course for you. When everything is possible, the most probable outcome will be chaos. So if you have 100 developers and 20 different ways to do things, how many outcomes do you have? 200. And if these 20 developers will do the same 20 times, you will have 2,000 outcomes.  Like, I want to get up and I see, okay, how people who have more models do things. And I found, oh, okay, people use a train function and then they have a class with all the parameters. Oh, how to do the class? Some people build the graph within the class. Some people build the graph in the functions and train. Other people do completely different things, like there's this function and these are the layers and then they loop over the layers and create the graph while they are in this function, create net. So how should I do it? What's the correct way? I need a correct way. There should be a correct way. I'm a little bit OCD, so I want a correct way. And if you think about it, a model is just a thing that takes data, trains itself, evaluates itself and when the model is happy, say, hey, I'm happy, let's predict. So it's a thing that will take train data, train for a while, then evaluate, and then when you go to inference, it will take a new sample and then predict that outcome. It's not that hard. This should be a way, like, I don't care how is your model, how is anything, but this should be clear. When I go back to my code, I should be, no, okay, this function or this class is the train, this function is the model, wherever. And estimators came to help. So estimators became, they were released in 1.3 and became mature in 1.8. They are going to stay in 2, but things in the custom estimators are going to change a little bit. Nevertheless, since we are talking about good practices, this will stay because we still want to be good developers. We are machine learning practitioners. We're still, we are writing code. We are not writing poems. And this is the API of TensorFlow. So on top of the food chain, we are going to have the primary estimator. That's the thing that developers still use all the time because it's robust, it's very tested, so we know that it's correct. Then if you want to go lower, you have an estimator and a Keras model. We have been listening a lot about Keras models and if you want to go back, let's go back to the other talk. Let's talk previous to this one. And then let's say, oh, I actually want my estimator to be completely wildly different. So I have layers. Now in TensorFlow 2, it's in Keras layers, but still we have some way to define layers. And then we have the Python IPA, the C++, and then all the things that talks to the hardware. We don't want to talk to the hardware. Okay. So where is an estimator? An estimator is a complete model. So it was this pink thing that I had before. And an estimator will give us models for train, for predict, and for evaluate. I'm going to repeat this structure a lot because it's actually what we want in a machine learning system. And these are the functions that we need to instantiate. Train, evaluate, and predict.  I was waiting one week to my model to be trained. And now I want to use it. So I'm going to wait another week. That doesn't make sense. First thing, let's use pre-train or can estimators. This used to be called can. It's like going to the shop and buying a nice tomato soup ready to eat. So can estimators are commonly used architectures. That means if you want to write a linear classifier or linear regressor, it doesn't make sense that you need to write every single time a linear regressor. We all know how this thing is. And it represents a whole model. What kind of can estimators do we have? We have baseline classifiers. So you build your own, your lovely model and you want to test it against a baseline. So you have that. It's classifier and regressor. I'm going to say classifier for simplicity. But we have baseline classifier, linear classifier, deep neural network classifier. So a neural network with layers. Okay. We have found and can estimators are our thing. We want to use that because we are not going to do anything very fancy. We just want to use these kind of estimators. How are we going to do this? So first we need to create a template function, define the model feature column, instantiate this estimator, and then call the method, switch method, try and test and evaluate. That's all the time. So let's start. Let's create an input data. I will talk later about data, data set, but we need an efficient way to feed data to our model. TensorFlow estimators are built for parallelize, for running parallel. So we want to feed things properly. And here, in two lines of code, we have our data set ready to use. Okay. We have the data. And now we have to tell the model which kind of data is this. Like, is this an input? It's a string? It's a float? Where is it? And we simply tell, oh, this feature column is actually a numeric column. So I'm going through all the train keys and tell my model. All these models are actually numeric columns. Okay. So I said, where is the data? I tell what kind of data is this. And now I need to instantiate the estimator. Estimators are a baseline subclass from STDF estimator. And this is a deep neural network estimator, classifier. And it's within estimator. So this is a deep neural network with two hidden units. Each unit has 10 neurons. And then the output is three classes. So it has three neurons at the output. And with two lines of code, actually one line of code, I built a deep neural network. How amazing is that? We all have to agree that this is pretty good. And then we are going to use it. For using it, I just need to call train. For training, I pass the correct data set. This is the iris data set. I don't know anything about botanic. If you give me a plant, I will kill it for sure. But I know if you give me an iris, I will be able to tell you exactly what kind of virus it is. It's a classical machine learning problem where you need to decide which type of virus flower is a flower. So I train my model with inputs from a training data set. I tell how many times do I want to train my model and the steps. And now I'm going to evaluate. So I want to make sure that the model is correctly trained. So another data set is for testing, for development. And I want to see how good it is. And now I'm happy with my model. And I'm going to go to inference. And this is what I say. And this is the three classes of iris. And we have setosa, versicolor, and virginica. And we measure the petal and the sepal length. I don't know really. It's like, why do you know what a petal is? A sepal length. Yeah, whatever. But I know. And I think that most of you, if you do machine learning, you know this thing. So then the only thing that I do need to do is call predict. And I will get the prediction. And then I will figure it out. Since I know the expected, I can say, if it's correct or it's not correct. This will run in parallel for us. This has been handling sessions. This handles threads. We don't need to worry about running key runners and stopping and waiting for stop iterations. If you don't this, you know how painful it used to be. You need to wait until all the threads are dead. But not really. There's a thread that is still alive. So forget about it. TensorFlow estimators will solve it for you. And that's amazing. However, what about if like, okay, I'm a business student. I don't want to use second estimators. And maybe your problem is not this linear regressor problem, then I'm going to simply plug in.  So what are custom estimators? Custom estimators allows us to be as flexible as we want. On top of practices. So we are bringing all the things that we have built in estimator and account estimator and we bring it here. So when I go back to my code six months later, I don't know to where the custom estimator where the trainees where the model is with kind of layers. I know it will always follow the same structure. And that's good. And the only thing that we need to do is instantiate the TF estimator and write a model. That's easy. So if you want to use a custom estimator, this is how to do it. This is the recipe. And if you see, all the steps are the same. I'm actually not going to talk about how to create a input function, the final model, the model columns, because that's exactly the same way as you used to be. So as you used to do, so you can create your can estimator, test all your pipeline and then set, oh, but actually, I'm going to do my own. And then you create your custom estimator. Creating a custom estimator requires you to write a callback function with a model to use params, the parameters of your configuration. And if you're, you have, this is tricky, I'm always, because I've used it a lot. Like, you can say, oh, I want all my GPU. I don't want to share it. So you can say that in the wrong config parameters. Make sure that when you're using all the GPU, everyone who's in your lab or using the same GPU agrees. Or you're going to have a difficult conversation to handle. Okay, so this callback function is you basically write layers. So you are writing the model, stacking layers, one after each other. As I was saying, this is smooth, but never trust country. Now, it's getting more stable. At the moment, we only have country layers, stable. And I'm passing the features and the params. So here's the problem of sentiment analysis, giving an input sentence. So my feature is a sentence. And the parameters are the vocabulary size, the embedding size, and the size. So instead of writing this crazy lookup function, I'm going to use a layer that is properly tested for me. And then I'm using a convolution on your network. Again, instead of writing a convolution and figuring out that I don't have leaky memory around, I'm going to use layers. Then I have a max pool layer after that. A hidden layer and the output. So this has moved a couple of slides, but we can agree that it's pretty, see? We always know then, okay, I have a model function. This is the callback. I will always come back to here. Okay. So we have all the layers prepared. We are going to say, how do I want to train? I hear use Alarm Optimizer, but you can use whatever Optimizer fits you. Again, Keras Optimizers are there for you.  And here I'm writing directly to the summary. So no more variables than you need to make it global and then write and then remember, you just write to the summary, which is amazing. If you want to train, you have the callback will receive a signal in the mode. If it's trained, you will return estimator span with the train. If it's an evaluate, you won't compute the accuracy of the model and then you return another estimator object and it's the output of this evaluation. So in summary, estimators are a good idea because it allows us to make good decision about their code. It's easier to debug. It's easier to maintain. It's better for everyone, for your mental health. So use it. If we all agree, the estimators are good and they are. If you don't agree, we can discuss it later. So the next thing that you need to do is talk about data sets. So you have built this amazing model and you are using a fit to fit data. How not efficient is that. So if you have this model and it's parallel and can allow multiple runs and crazy things, don't use a dictionary to fit data. It's rude. It's not polite. So we are going to use a data set. How to use a data set is simply you need to import the data. You will manipulate the data. You will create an iterator and then you will consume a data from an iterator. So you don't need to have all your data ready when you are going to start training your model. You just need an iterator. That's amazing. So how to import data. You can import data directly from a generator. So you can delegate the creation of data to another generator. You can use tensor slices, different flavors of tensors or records. And then CS file, it was like temp not booted here because this is not very efficient. But nevertheless, it's an option. If you want, you can use it. Okay. So how we create a data set. I have this example generator function. Then it's a generator. Then it will be generating inputs. And I need to tell TensorFlow A, this input actually has these types and has these shapes. So that way the graph can be created properly. Okay. I have my data ready to use. But I actually want to add one. Or I want to do something. And of course, if I have this super fast thing that generates data, I don't want to apply a common function. I want to apply a map. Then we'll apply a function to all the elements. And then I want to use everything functional. So I want to be efficient. So I can apply a function to all the elements. I can shuffle my data. I can repeat. So once I see my data set once, I can see it again for another iteration. And I can tell the size of the patch. So how many examples do I need to see in order to update the parameters of my TensorFlow model? How to do it? This simple line. How amazing is that? You can concatenate or chain functions. So I said, hey, shuffle my data. And when you finish, repeat it. And the batch size is this batch size. And with this simple line, I'm creating a data set that is efficient for my TensorFlow model and can be paralyzed. Amazing. Great. I have my data ready. I just need to create an iterator. So we have two options here. One is for the iterator. So I'm going to run all the data once. Or an initializable iterator. If you want to do crazy thing, like crazier and more custom, you can use this thing. Normal users will use one-shot iterator. And this is, I said, hey, data set, please give me an iterator. And it gives me an iterator. So how to consume data? Simply call next. And get next will return me a batch of sample data. And I can feed it to the TensorFlow model. Great. So we have finished this part of the journey. We have seen that estimators are a good practical way to handle our models. We have seen that data sets are the proper way to feed data to the model. And yeah, we want to have all these things in place in order to build a proper model. And when you come back to your code, you know where exactly you are. Finally, let's say then you actually heard about Glove or WordPick or VGG. This is very old. AlexNet. I'm going to have a lot of time for questions because I was super fast. Okay. So you have heard about all these models and said, or even Bert, you have heard about Bert and say, should I write 26 layers models with attention or that's going to be very, very tricky? Don't do it. Use TensorFlow Hub. TensorFlow Hub brings all the ideas from continuous integration and software control version to machine learning. Actually, it's a thing that we should be doing and we have not been doing, but we are solving it. So TensorFlow Hub is a place where we can find models that are reusable, stackable and trainable and you can use it in your own pipelines. So let's say I use a very old one. I use a neural network language model. And how do I do that? I say I want an embedding column and this is the model specification. So this URL is the URL of the model. The last number is the version. So let's say the people who created this model updates the model and say, now I want to use the next version. It's essentially changing that. And then I feed that to my estimator. So this is a kind of estimator. I don't even need to do anything crazy. Just two lines of code. I have state of the art models, a state of the art and tested TensorFlow models. So we should be using it. In an ideal world, I am expecting them people will publish their code, their paper and I can use it because if you write science and I cannot replicate, it's not science. It's you're doing kitchen. And I can cook myself. Thank you very much. Sorry. Thanks. It was nice. Now I can finish with a lot of energy. So bits of knowledge then I want you to take away. Estimators are a good idea. Please, please, please, please when you come back to your code don't you shouldn't spend more than five minutes figuring it out where your model is, where the training is. You should be able to know where is everything. Data science allows for building high performance complex pipelines. So if you build this amazing pipeline of your TensorFlow estimator, you don't want to fuel it with this old slow data. You want to fill it fast. And finally, you have a state of the art driving models to use in TensorFlow Hub. It's kind of easy to publish. So if you have invented a really, really nice thing, please share it. The community will be very, very thankful. And now, yes, thank you very much. Thank you, May. So we have time for a few questions. Yeah, sorry, I was super fast. So I hope you get everything. I'm a very hopeful human being and I like to claim to complain about bad things. So... Anybody? Don't be shy. Yeah. So thanks for a really good talk. That was very interesting. And I will go home and write all of my estimators and see everything that's out there in TFM and everything. And so I was just thinking, do you have any kind of best practices for how to curate your model training performance on, like, what are the models? How do you have a local database where you store the, how your models have been training for the different architectures and everything? Or do you have any solution for that? Yeah, that's, that's an excellent question. There is no standard way to do it, so you need to figure it out your own. I can show you how I do it. How I do it is I write TensorFlow summaries and then I will store. So every time around when you create a summary, you can tell I want to save it in that place.  So I save, like, the layers of the model. So I remember what model I was training there on the summary. So you can write, we haven't talked about TensorFlow board, but TensorFlow board is an amazing tool. So if you have that thing, like, let's say I have, I'm training different architectures with different parameters. So I will have all the parameters, all the parameters in the same tier.  Be careful, because sometimes you let it run with four loops, four, five, four loops, and then it's like, oh, damn, I have 200 architectures and you need to figure it out which one is which. But, yeah, I didn't find a proper way to do it. This is my approach. Thank you for your question. Okay. Anybody else? Come on. We have five minutes. Sorry. Okay, then thank you for the amazing talk.


Larry hires on to a corporation as a truck driver, with intent to deliver their goods to distributors and to pick up supplies and materials at corporate provider facilities. He thinks that his employment is a contract where he agrees to be an employee and work for an employer who will pay for his services. Indeed, his purpose for signing the contract is to earn income, and the business purpose for employing him is to have him deliver incoming and outgoing materials. The challenge is the unwritten part of the contract he signs. It is contract obligations set by government instead of by agreement between the parties. Larry is also obligated to pay taxes and to have the business collect them from his earnings. He is to be registered into a social security system that is supported by even more taxes, even if there is no mention of this in the contract he signs. For its part, the corporation, again unmentioned in the contract, is obligated to collect taxes from Larry, depositing them in accord with government instructions. It agrees to pass the Internal Revenue Service information on Larry's pay and taxes information from Larry's employment. It agrees to conditions and some union agreement that is not even a viable contract under common law for lack of consideration. There are few areas of law that have greater government interference than employment. It is where government regulation has intruded into the contract adding conditions and obligations that are not agreed by the parties and are often not even written into the contract document that they sign. This behavior is patently unconstitutional interfering with obligations on private contracts is directly an absolutely banned in Article 1, Section 10 of our Constitution. The very purpose of this section is to set a duty upon Congress. It is a duty to protect we the people from even state laws that would attempt to pass bills of attainder, ex post facto laws or would impair the obligations of contract. Here we see Congress not simply failing to protect the public but creating the interference it has a duty to prevent. Larry is not going to be paid what he was agreed as the value for his services. Some of his earnings are going to be seized under tax laws. The employer does this as an agent of the government without any legal basis for agency. Larry's earnings the value he contracted to receive will not be legally secured from seizure. The government will take it without even a promise of just compensation. The business gets Larry's services as agreed but finds itself obligated to become a tax collector. This is a new obligation outside any contract and the time and effort and risks of business operation to complete this task are set upon the employer as a cost of contracting with Larry. There is no just compensation received by the business nor is one offered. How could this be? The answer is that both legislators and judges are ignoring constitutional limits. They legislated a change in the 16th amendment that would authorize them in the name of the people to seize on private income of citizens as a new authority of government. Of course our leaders were not authorized as representatives to redefine their authority to include what they were forbidden to do. Any citizen who puts his hand in his neighbor's pocket to seize upon what he wants or needs would be prosecuted for theft. No representative could as a representative put his hand in any citizen's pocket without engaging in criminal behavior in the name of the people. To make sure that this sort of violation was clearly banned, the constitution wasn't even signed until there was a 4th amendment ban on seizing private property set into the language where the ban was clear. That of normal legislation, those who were in authority only as representatives took it upon themselves to ignore the limitations of agency and document their defiance of our constitutional provision as an amendment. They violated their trust as agents of the people, relying on their authority as sovereigns. They applied the new law through legislation to interfere with the obligations of employment contracts. Having ignored the limits of constitutional government as to their responsibilities, they had little problem in proclaiming that the limits written into the original document were no longer appropriate. Like Congress, federal courts have accepted interference in contract obligations and have ignored the duties set upon Congress that might interfere with sovereignty of government over common citizens. As far as their oath to support the Constitution is concerned, the illusion from our historical agricultural foundation is appropriate. In protecting citizens under the Constitution, our courts have been as useful as tits on a bull. Our leader's general application has been support of legislative law and support of the Congress. Our courts have almost uniformly supported the actions of Congress, whether representative or not. They have assumed a right to use interpretation of law to rewrite the Constitution when it would not support what Congress would do as a sovereign body. In ignorance or sloth, take your pick, our courts do not approach the Constitution as a contract buying among the we, the people. It is only by collusion, a conspiracy to support whatever Congress is willing to put into law, that modern courts can claim even the color of legality. The courts support a constitutional process, even where it accomplishes an illegal result. Our Constitution has its faults, mainly in being dated and limited to the knowledge and intent of the people who brought it into existence. It is still an incredible and wonderful documented source for a legal government. Unlike other national constitutions, those that recognize and set limits upon a sovereign government's authority, our constitutional agreement is the open recognition of the sovereignty of citizens. It records an attempt to create a government to meet our sovereign purposes as stated in the preamble. That is what makes the United States politically unique among nations, that it is founded on the dream of self-government. It is a dream based on a reality, that the ultimate freedom of a nation is only found in the freedom of its people. It is based on the reality that the prosperity of a nation cannot be greater than the prosperity of its people. It is based on a government where the leaders serve the people, instead of a government where the citizens are required to work for its leaders. There is no greater violation of the American Dream than a government where the people work for the rulers and those who rule determine what the people get for their efforts. This is involuntary servitude for the people of the nation. And nowhere is that violation of the American Dream greater than in the open and hostile setting of obligations on those who would enter into employment. This pollutes the lifeblood of our nation's economy, our personal commitments to the economic welfare of our nation. The result is a government putting its hands in the pockets of its citizens to do what government leaders feel is right. Doing it in the name of the people does not make this any better. It is criminal behavior, and justifying crime by blaming it on someone else does not relieve it any more than claiming it is somehow necessary to be a criminal if you are to serve the public. The rule is a simple one. If the result is criminal, the act that creates the result are also criminal. This is a major element of our commercial environment, and the result of acts in violation of the principles of common law and specific provisions in our constitution. The result is real, and it will not become different unless someone makes changes. For the effective citizen of tomorrow, we again have too many potentials to offer a clear path. We are to correct what is wrong, reversing changes to our constitution that are so clearly beyond what authorities we granted to our representatives. Or do we demand our courts recognize personal sovereignty and support our authority as owners of government? Do we raise the owner flag and start cleaning house of those leaders who seem least inclined to serve we the people? Do we mandate changes to our constitution to initiate our exception management over its questionable use of our authority? We face the twin challenges of what actions by citizens can be effective, and what we can agree to do as we the people. We face a secondary challenge of inertia, of the many citizens who are unaware of the serious breaches of authority limits set on our government. We face a secondary challenge of a government that has proven quite willing to punish unwanted questioning of its authority, unless the change is fairly certain there will be no new government obligations except those pledged to those who would challenge. That is the nature of rule. Once established, rule does not lightly tolerate competition. On the other hand, when people are agreed, there is no longer any competition. There is only the authority of we the people. What we seek in this course is empowering the citizens of tomorrow. It comes from agreement, from coming together for what people agree. Our constitution, marvelous as it is, is subject to this same agreement. And the question is, can we agree that our public officers are to abide by its provisions? Our written constitutional agreement is not the true authority, unless we are agreed that it is. And we are in charge, wherever we are willing, to put our agreement into effect as directed to our public employees. U.S. citizens are also owners. The United States is your nation. The constitutional agreement proclaims itself to be your government agreement. It is to be implemented as a corporate government that is only authorized to continue through your agreement. The U.S. government that is created under this agreement exists only to serve the purposes that we, as sovereign citizens, have given to it. It should only have authorities that we agree to provide. We are the citizen owners, and where we are in agreement, everyone is in agreement. Your ownership is also a reality, even though it may now be denied by our public employees. When it comes to interfering with the obligations of employment contracts, we have a criminal violation of the authorities granted to government in our constitution. This leaves us with both the reality of being victims of criminal acts by our own government, and the reality of our authority. The reality of our being victims of crime is not going to change unless someone changes it. The challenge of change is not an authority or potency. It is in whether we, as we the people of the United States, can find sufficient agreement among ourselves so that we will mandate our change to our leaders for their action.